[Most-ai-contest] The performance of the last version FGC QA system

[范正忠]

Dear all, 

A quick look at Date-Duration on 'test' & 'dev' data set and the results as follows. 

You can see 
1. some answers are ranked Top-2 or Top-3 
2. Some need math. operations 
3. Some need to format transform (西元 -> 民國) 
4. D267Q07 -> I think it should be correct since only "戰國末年" occurs in the passage. (golden answer : 戰國時期) 
5. D303Q05 -> "初冬" should be OK since only "初冬" occurs in the passage (golden answer: 冬) 

jjfan 

Dev 	D061Q04 	D073Q05 	D097Q06 	D103Q04 	D241Q04 	D247Q08 	D307Q03 						
	Top-2 	Top-2 		Top-2 	Top1 相減 "1506年至1626年" 		Dispatcher Error : Single-Span -> Date-Duration 						
Test 	D033Q11 	D069Q04 	D069Q05 	D069Q06 	D069Q07 	D087Q02 	D105Q02 	D105Q04 	D117Q01 	D117Q02 	D243Q07 	D267Q07 	D303Q05 
	Top-2 			Top1 相減 "1981年至纽约大学就读电影制作研究所，于1984年" 	Top1 - Top2 	Top1 前頭加上 2019 年 	Top1 1995年, output formater 沒有轉成民國 		Top-2 	Single-Span Top1 	Top2 相減 "1506年4月18日，完成于1626年11月18日" 	Top1 "戰國末年" 	Top3 "初冬" 



From: "范正忠" <jjfan at iis.sinica.edu.tw> 
To: "Simonc" <simonc at iis.sinica.edu.tw> 
Cc: "Most-ai Contest" <Most-ai-contest at iis.sinica.edu.tw> 
Sent: Friday, April 3, 2020 4:45:39 PM 
Subject: [Most-ai-contest] The performance of the last version FGC QA system 

Dear all, 

Enclosed please find the current performance of the FGC QA system. 

		Single-Span 	Kuo 	Multi-Span 	Date-Duration 		Train 	Dev 	Test 
4月2日 	ENSEMBLEModule: 12 models (new) 	train: 0.890 
dev: 0.801 
test: 0.750 	train: 0.899 
dev: 0.699 
test: 0.741 	train: 0.493 
dev: 0.476 
test: 0.350 	train: 0.589 
dev: 0.720 
test: 0.581 		0.786 	0.709 	0.658 
4月3日 	ENSEMBLEModule: 12 models (new) 
MSPE_v18_branchy27 
date_duration_module_4 	train: 0.890 
dev: 0.801 
test: 0.750 	train: 0.899 
dev: 0.699 
test: 0.741 	train: 0.548 
dev: 0.667 
test: 0.450 	train: 0.589 
dev: 0.720 
test: 0.581 		0.791 	0.725 	0.668 
Please 郭家銍 help to check Date-Duration ver.4. 

Best, 
jjfan 



From: "范正忠" <jjfan at iis.sinica.edu.tw> 
To: "Simonc" <simonc at iis.sinica.edu.tw> 
Cc: "Most-ai Contest" <Most-ai-contest at iis.sinica.edu.tw> 
Sent: Thursday, April 2, 2020 3:56:36 PM 
Subject: Re: [Most-ai-contest] Updated Performance Data with Gold Answer Inclusion Rate 

Dear all, 

Enclosed please find the performance of the last code: 

ENSEMBLEModule: 12 models (new), single_span_multi_hop_v2_1 
Multi-Spans: MSPE_v18_branchy20 
date_duration_module_3 
arithmetic_module_3 
kinship_module6 

Overall: 
train -> total: 882, correct: 668, accuracy: 0.757 
dev -> total: 247, correct: 161, accuracy: 0.652 
test -> total: 193, correct: 126, accuracy: 0.653 

Single-Span: 
train -> portion:0.619, count:546, errors: 86, accuracy: 0.842 
ensemble only -> portion:0.619, count:546, errors: 37, accuracy: 0.932 
multi-hop only -> portion:0.619, count:546, errors: 76, accuracy: 0.861 

dev -> portion:0.632, count:156, errors: 45, accuracy: 0.712 
ensemble only -> portion:0.632, count:156, errors: 37, accuracy: 0.763 
multi-hop only -> portion:0.632, count:156, errors: 43, accuracy: 0.724 
test -> portion:0.580, count:112, errors: 29, accuracy: 0.741 
ensemble only -> portion:0.580, count:112, errors: 23, accuracy: 0.795 
multi-hop only -> portion:0.580, count:112, errors: 26, accuracy: 0.768 

Multi-Spans: 
train -> portion:0.083, count:73, errors: 37, accuracy: 0.493 
dev -> portion:0.085, count:21, errors: 11, accuracy: 0.476 
test -> portion:0.104, count:20, errors: 13, accuracy: 0.350 

jjfan 

From: "Simonc" <simonc at iis.sinica.edu.tw> 
To: "Most-ai Contest" <Most-ai-contest at iis.sinica.edu.tw> 
Sent: Tuesday, March 31, 2020 6:00:52 PM 
Subject: [Most-ai-contest] Updated Performance Data with Gold Answer Inclusion Rate 

Dear all, 

The attached file contains the performance data from our results in 3/27. 
This time, the gold answer inclusion rate is also included. (That is, counting the cases where the correct answer is included in the answer candidates.) 

Regards, 
張光瑜 

_______________________________________________ 
Most-ai-contest mailing list 
Most-ai-contest at iis.sinica.edu.tw 
https://www.iis.sinica.edu.tw/mailman/listinfo/most-ai-contest 

_______________________________________________ 
Most-ai-contest mailing list 
Most-ai-contest at iis.sinica.edu.tw 
https://www.iis.sinica.edu.tw/mailman/listinfo/most-ai-contest 

_______________________________________________ 
Most-ai-contest mailing list 
Most-ai-contest at iis.sinica.edu.tw 
https://www.iis.sinica.edu.tw/mailman/listinfo/most-ai-contest 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.iis.sinica.edu.tw/pipermail/most-ai-contest/attachments/20200403/a1d6eac1/attachment-0001.html>