<html><body><div style="font-family: arial, helvetica, sans-serif; font-size: 12pt; color: #000000"><div>Dear all,</div><div><br></div><div>I made a visualization script for attentions in huggingface transformers. It works in both tensorflow and pytorch.</div><div><br></div><div><br data-mce-bogus="1"></div><div>[DEMO]</div><div><br data-mce-bogus="1"></div><div><div>from huggingface_utils import visual_atn, visual_atn_matrix</div></div><div><br></div><div>model = BertModel.from_pretrained('bert-base-uncased', output_attentions=True)</div><div>loss, logits, attentions = model(input_ids)</div><div><br></div><div><div>print(visual_atn(tokens, attentions)) # DEFAULT: last layer, average over multi-heads</div></div><div><div>print(visual_atn(tokens, attentions, layer_num=3, head_num=-1)) # third layer, last head</div></div><div><br></div><div><img src="cid:4a9b1ea80a243608d4629e02e2dbd842027fbae2@zimbra" data-mce-src="cid:4a9b1ea80a243608d4629e02e2dbd842027fbae2@zimbra"></div><div><br></div><div><div>visual_atn_matrix(tokens, attentions, layer_num=-2) # second-to-last layer, average over multi-head attention matrices</div></div><div>visual_atn_matrix(tokens, attentions, layer_num=4, head_num=7) <!--StartFragment--><span style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;"># fourth layer, seventh head attention matrix</span><!--EndFragment--><div style="clear: both;" data-mce-style="clear: both;"><br data-mce-bogus="1"></div></div><div><br></div><div><img src="cid:9b0a5d965ee45840c9518fdba9238027d85a4855@zimbra" data-mce-src="cid:9b0a5d965ee45840c9518fdba9238027d85a4855@zimbra"></div><div><br data-mce-bogus="1"></div><div>FYI</div><div><br></div><div>Regards,</div><div><br></div><div>Chiao-Wei</div></div></body></html>