[Most-ai-contest] Updated Performance Data with Gold Answer Inclusion Rate

[kysu]

Thanks. How much improvement we make in this new version?

 

KY

 

From: most-ai-contest-bounces at iis.sinica.edu.tw [mailto:most-ai-contest-bounces at iis.sinica.edu.tw] On Behalf Of 范正忠
Sent: Thursday, April 2, 2020 3:57 PM
To: Simonc <simonc at iis.sinica.edu.tw>
Cc: Most-ai Contest <Most-ai-contest at iis.sinica.edu.tw>
Subject: Re: [Most-ai-contest] Updated Performance Data with Gold Answer Inclusion Rate
Importance: High

 

Dear all,

 

Enclosed please find the performance of the last code:

 

ENSEMBLEModule: 12 models (new), single_span_multi_hop_v2_1

Multi-Spans: MSPE_v18_branchy20

date_duration_module_3

arithmetic_module_3

kinship_module6

 

Overall:

train -> total: 882, correct: 668, accuracy: 0.757

dev -> total: 247, correct: 161, accuracy: 0.652

test -> total: 193, correct: 126, accuracy: 0.653

 

Single-Span:

train -> portion:0.619, count:546, errors: 86, accuracy: 0.842

    ensemble only -> portion:0.619, count:546, errors: 37, accuracy: 0.932

    multi-hop only -> portion:0.619, count:546, errors: 76, accuracy: 0.861

 

dev -> portion:0.632, count:156, errors: 45, accuracy: 0.712

     ensemble only -> portion:0.632, count:156, errors: 37, accuracy: 0.763

     multi-hop only -> portion:0.632, count:156, errors: 43, accuracy: 0.724

 

test -> portion:0.580, count:112, errors: 29, accuracy: 0.741

     ensemble only -> portion:0.580, count:112, errors: 23, accuracy: 0.795

     multi-hop only -> portion:0.580, count:112, errors: 26, accuracy: 0.768





Multi-Spans:

train -> portion:0.083, count:73, errors: 37, accuracy: 0.493

dev -> portion:0.085, count:21, errors: 11, accuracy: 0.476

test -> portion:0.104, count:20, errors: 13, accuracy: 0.350

 

jjfan

  _____  

From: "Simonc" <simonc at iis.sinica.edu.tw <mailto:simonc at iis.sinica.edu.tw> >
To: "Most-ai Contest" <Most-ai-contest at iis.sinica.edu.tw <mailto:Most-ai-contest at iis.sinica.edu.tw> >
Sent: Tuesday, March 31, 2020 6:00:52 PM
Subject: [Most-ai-contest] Updated Performance Data with Gold Answer        Inclusion Rate

 

Dear all,

 

The attached file contains the performance data from our results in 3/27.

This time, the gold answer inclusion rate is also included. (That is, counting the cases where the correct answer is included in the answer candidates.)

 

Regards,

張光瑜


_______________________________________________
Most-ai-contest mailing list
Most-ai-contest at iis.sinica.edu.tw <mailto:Most-ai-contest at iis.sinica.edu.tw> 
https://www.iis.sinica.edu.tw/mailman/listinfo/most-ai-contest

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.iis.sinica.edu.tw/pipermail/most-ai-contest/attachments/20200402/2a0f2b17/attachment-0001.html>