[Most-ai-contest] Visualization for huggingface transformer self-attention layers

[許喬為]

Dear all, 

I made a visualization script for attentions in huggingface transformers. It works in both tensorflow and pytorch. 


[DEMO] 

from huggingface_utils import visual_atn, visual_atn_matrix 

model = BertModel.from_pretrained('bert-base-uncased', output_attentions=True) 
loss, logits, attentions = model(input_ids) 

print(visual_atn(tokens, attentions)) # DEFAULT: last layer, average over multi-heads 
print(visual_atn(tokens, attentions, layer_num=3, head_num=-1)) # third layer, last head 


visual_atn_matrix(tokens, attentions, layer_num=-2) # second-to-last layer, average over multi-head attention matrices 
visual_atn_matrix(tokens, attentions, layer_num=4, head_num=7) # fourth layer, seventh head attention matrix 



FYI 

Regards, 

Chiao-Wei 
-------------- 下一部份 --------------
抹去了一個 HTML 附加檔...
URL: <http://www.iis.sinica.edu.tw/pipermail/most-ai-contest/attachments/20200318/28a647a7/attachment-0001.html>
-------------- 下一部份 --------------
A non-text attachment was scrubbed...
Name: 4144859.4099999755
Type: image/png
Size: 3515 bytes
Desc: 〔無法取得〕
URL: <http://www.iis.sinica.edu.tw/pipermail/most-ai-contest/attachments/20200318/28a647a7/attachment-0002.png>
-------------- 下一部份 --------------
A non-text attachment was scrubbed...
Name: 4319213.85
Type: image/png
Size: 40251 bytes
Desc: 〔無法取得〕
URL: <http://www.iis.sinica.edu.tw/pipermail/most-ai-contest/attachments/20200318/28a647a7/attachment-0003.png>
-------------- 下一部份 --------------
A non-text attachment was scrubbed...
Name: example.ipynb
Type: application/octet-stream
Size: 593869 bytes
Desc: 〔無法取得〕
URL: <http://www.iis.sinica.edu.tw/pipermail/most-ai-contest/attachments/20200318/28a647a7/attachment-0001.obj>
-------------- 下一部份 --------------
A non-text attachment was scrubbed...
Name: huggingface_utils.py
Type: text/x-python
Size: 4415 bytes
Desc: 〔無法取得〕
URL: <http://www.iis.sinica.edu.tw/pipermail/most-ai-contest/attachments/20200318/28a647a7/attachment-0001.py>
-------------- 下一部份 --------------
An embedded and charset-unspecified text was scrubbed...
Name: requirements.txt
URL: <http://www.iis.sinica.edu.tw/pipermail/most-ai-contest/attachments/20200318/28a647a7/attachment-0001.txt>