[Most-ai-contest] The performance of the last version FGC QA system (4/4)

Sat 4月 4 10:43:14 CST 2020

Dear Dr. Fan,

Thanks for the quick reply!

Best, Menphis.

> 范正忠 <jjfan at iis.sinica.edu.tw> 於 2020年4月4日 10:24 寫道：
> 
> 
> Dear all,
> 
> Enclosed please find the current performance of the FGC QA system.
> 
> 
> Single-Span	Kuo	Simonc	Multi-Span	Date-Duration		Train	Dev	Test
> 4月2日	ENSEMBLEModule: 12 models (new)	train: 0.890
> dev: 0.801
> test: 0.750	train: 0.899
> dev: 0.699
> test: 0.741	　	train: 0.493
> dev: 0.476
> test: 0.350	train: 0.589
> dev: 0.720
> test: 0.581		0.786	0.709	0.658
> 4月3日	ENSEMBLEModule: 12 models (new)
> MSPE_v18_branchy27
> date_duration_module_4	train: 0.890
> dev: 0.801
> test: 0.750	train: 0.899
> dev: 0.699
> test: 0.741		train: 0.548
> dev: 0.667
> test: 0.450	train: 0.589
> dev: 0.720
> test: 0.581		0.791	0.725	0.668
> 4月4日	ENSEMBLEModule: 12 models (new)
> MSPE_v18_branchy27
> date_duration_module_4	train: 0.888
> dev: 0.801
> test: 0.759	train: 0.949
> dev: 0.837
> test: 0.876		train: 0.548
> dev: 0.667
> test: 0.450	train: 0.753
> dev: 0.720
> test: 0.774		0.817	0.737	0.705
> 
> Please check the detail in Performance.txt file and Date-Duration performance is similar to  郭家銍 report.
> 
> Best,
> jjfan
> 
> 
> From: "范正忠" <jjfan at iis.sinica.edu.tw>
> To: "Simonc" <simonc at iis.sinica.edu.tw>
> Cc: "Most-ai Contest" <Most-ai-contest at iis.sinica.edu.tw>
> Sent: Friday, April 3, 2020 4:45:39 PM
> Subject: [Most-ai-contest] The performance of the last version FGC QA system
> 
> Dear all,
> 
> Enclosed please find the current performance of the FGC QA system.
> 
> Single-Span	Kuo	Multi-Span	Date-Duration		Train	Dev	Test
> 4月2日	ENSEMBLEModule: 12 models (new)	train: 0.890
> dev: 0.801
> test: 0.750	train: 0.899
> dev: 0.699
> test: 0.741	train: 0.493
> dev: 0.476
> test: 0.350	train: 0.589
> dev: 0.720
> test: 0.581		0.786	0.709	0.658
> 4月3日	ENSEMBLEModule: 12 models (new)
> MSPE_v18_branchy27
> date_duration_module_4	train: 0.890
> dev: 0.801
> test: 0.750	train: 0.899
> dev: 0.699
> test: 0.741	train: 0.548
> dev: 0.667
> test: 0.450	train: 0.589
> dev: 0.720
> test: 0.581		0.791	0.725	0.668
> 
> Please  郭家銍 help to check Date-Duration ver.4. 
> 
> Best,
> jjfan
> 
> 
> From: "范正忠" <jjfan at iis.sinica.edu.tw>
> To: "Simonc" <simonc at iis.sinica.edu.tw>
> Cc: "Most-ai Contest" <Most-ai-contest at iis.sinica.edu.tw>
> Sent: Thursday, April 2, 2020 3:56:36 PM
> Subject: Re: [Most-ai-contest] Updated Performance Data with Gold        Answer        Inclusion Rate
> 
> Dear all,
> 
> Enclosed please find the performance of the last code:
> 
> ENSEMBLEModule: 12 models (new), single_span_multi_hop_v2_1
> Multi-Spans: MSPE_v18_branchy20
> date_duration_module_3
> arithmetic_module_3
> kinship_module6
> 
> Overall:
> train -> total: 882, correct: 668, accuracy: 0.757
> dev -> total: 247, correct: 161, accuracy: 0.652
> test -> total: 193, correct: 126, accuracy: 0.653
> 
> Single-Span:
> train -> portion:0.619, count:546, errors: 86, accuracy: 0.842
>     ensemble only -> portion:0.619, count:546, errors: 37, accuracy: 0.932
>     multi-hop only -> portion:0.619, count:546, errors: 76, accuracy: 0.861
> 
> dev -> portion:0.632, count:156, errors: 45, accuracy: 0.712
>      ensemble only -> portion:0.632, count:156, errors: 37, accuracy: 0.763
>      multi-hop only -> portion:0.632, count:156, errors: 43, accuracy: 0.724
> test -> portion:0.580, count:112, errors: 29, accuracy: 0.741
>      ensemble only -> portion:0.580, count:112, errors: 23, accuracy: 0.795
>      multi-hop only -> portion:0.580, count:112, errors: 26, accuracy: 0.768
> 
> Multi-Spans:
> train -> portion:0.083, count:73, errors: 37, accuracy: 0.493
> dev -> portion:0.085, count:21, errors: 11, accuracy: 0.476
> test -> portion:0.104, count:20, errors: 13, accuracy: 0.350
> 
> jjfan
> From: "Simonc" <simonc at iis.sinica.edu.tw>
> To: "Most-ai Contest" <Most-ai-contest at iis.sinica.edu.tw>
> Sent: Tuesday, March 31, 2020 6:00:52 PM
> Subject: [Most-ai-contest] Updated Performance Data with Gold Answer        Inclusion Rate
> 
> Dear all,
> 
> The attached file contains the performance data from our results in 3/27.
> This time, the gold answer inclusion rate is also included. (That is, counting the cases where the correct answer is included in the answer candidates.)
> 
> Regards,
> 張光瑜
> 
> _______________________________________________
> Most-ai-contest mailing list
> Most-ai-contest at iis.sinica.edu.tw
> https://www.iis.sinica.edu.tw/mailman/listinfo/most-ai-contest
> 
> _______________________________________________
> Most-ai-contest mailing list
> Most-ai-contest at iis.sinica.edu.tw
> https://www.iis.sinica.edu.tw/mailman/listinfo/most-ai-contest
> 
> _______________________________________________
> Most-ai-contest mailing list
> Most-ai-contest at iis.sinica.edu.tw
> https://www.iis.sinica.edu.tw/mailman/listinfo/most-ai-contest
> <Performance.txt>
> <4.4-1.7z>
> _______________________________________________
> Most-ai-contest mailing list
> Most-ai-contest at iis.sinica.edu.tw
> https://www.iis.sinica.edu.tw/mailman/listinfo/most-ai-contest
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.iis.sinica.edu.tw/pipermail/most-ai-contest/attachments/20200404/e8332c9b/attachment-0001.html>