OVERALL ACC: (BEFORE OUTPUT FORMATTER) 485 / 752 = 0.6449468085106383 -------------------------------------------------------------------------------- OVERALL ACC: (AFTER OUTPUT FORMATTER) 488 / 752 = 0.648936170212766 Formatter Errors: ['D015Q03', 'D243Q02', 'D314Q06', 'D314Q07', 'D314Q08', 'D314Q09', 'D281Q03', 'D292Q02', 'D304Q06', 'D304Q07'] -------------------------------------------------------------------------------- DISPATCHER PERFORMANCE (ANSWER MODE TOP-1): GRND/PRED SS MS YN ARI CNT CMP CS DD KS Single-Span-Extraction 492 4 0 0 0 0 0 0 0 Multi-Spans-Extraction 0 72 0 0 0 0 0 0 0 YesNo 0 0 71 0 0 0 0 0 0 Arithmetic-Operations 0 0 0 6 0 0 0 0 0 Counting 0 0 0 0 11 0 0 0 0 Comparing-Members 0 0 0 0 0 6 0 0 0 CommonSense 0 0 0 0 0 0 0 0 0 Date-Duration 1 0 0 0 0 0 0 25 0 Kinship 1 2 0 0 0 0 0 0 61 -------------------------------------------------------------------------------- DISPATCHER PERFORMANCE (ANSWER TYPE): GRND/PRED PER D-D LOC ORG NUM YN KIN EVT OBJ MISC Person 78 0 0 0 1 0 0 0 0 0 Date-Duration 0 110 0 0 0 0 0 0 0 0 Location 0 0 110 0 0 0 0 0 0 0 Organization 0 0 0 80 0 0 0 0 0 1 Num-Measure 0 0 0 0 66 0 0 0 0 0 YesNo 0 0 0 0 0 71 0 0 0 0 Kinship 0 0 0 0 0 0 63 1 1 1 Event 1 0 0 0 0 0 0 12 0 1 Object 0 0 0 0 0 0 0 0 138 2 Misc 0 0 0 0 0 0 0 0 0 15 -------------------------------------------------------------------------------- ANSWER MODULE STATS (WHEN CORRECTLY ACTIVATED): ACC SCORE MEAN SCORE STD-VAR Single-Span-Extraction 0.6351 0.9097 0.1597 Single-Span-Multi-Hop-Extraction 0.6734 0.8979 0.1714 Multi-Spans-Extraction 0.0000 1.0000 0.0000 YesNo 0.9014 0.9737 0.0488 Arithmetic-Operations 0.3333 0.7443 0.2885 Counting 0.0000 0.0000 0.0000 Date-Duration 0.2692 0.8558 0.2561 Kinship 0.9194 0.9839 0.1260 -------------------------------------------------------------------------------- ANSWER MODULE STATS (WHEN WRONGLY ACTIVATED): ACC SCORE MEAN SCORE STD-VAR Single-Span-Extraction 0.0602 0.7649 0.2458 Single-Span-Multi-Hop-Extraction 0.0241 0.8158 0.2127 Multi-Spans-Extraction 0.0000 1.0000 0.0000 YesNo 0.0000 0.9765 0.0829 Arithmetic-Operations 0.0091 0.4262 0.3168 Counting 0.0833 0.0000 0.0000 Date-Duration 0.6064 0.8128 0.2643 Kinship 0.0000 0.1667 0.3727 -------------------------------------------------------------------------------- ANSWER MODULE ERROR SAMPLES (WHEN CORRECTLY ACTIVATED): Single-Span-Extraction: ['D001Q11', 'D006Q01', 'D006Q02', 'D013Q01', 'D013Q11', 'D031Q02', 'D033Q01', 'D033Q03', 'D033Q05', 'D033Q06', 'D033Q14', 'D033Q15', 'D035Q04', 'D035Q05', 'D038Q02', 'D038Q11', 'D042Q08', 'D042Q14', 'D042Q15', 'D045Q03', 'D048Q02', 'D048Q09', 'D070Q01', 'D070Q04', 'D070Q05', 'D070Q10', 'D070Q11', 'D071Q10', 'D071Q13', 'D071Q15'] Single-Span-Multi-Hop-Extraction: ['D001Q06', 'D001Q09', 'D001Q11', 'D006Q02', 'D008Q07', 'D008Q08', 'D014Q09', 'D031Q02', 'D033Q01', 'D033Q05', 'D033Q06', 'D033Q07', 'D033Q08', 'D033Q14', 'D035Q02', 'D035Q05', 'D038Q01', 'D038Q11', 'D042Q03', 'D042Q05', 'D042Q06', 'D042Q14', 'D045Q03', 'D048Q03', 'D048Q09', 'D061Q01', 'D061Q03', 'D061Q04', 'D062Q02', 'D068Q01'] Multi-Spans-Extraction: ['D001Q08', 'D003Q03', 'D003Q06', 'D007Q02', 'D007Q03', 'D007Q04', 'D007Q05', 'D008Q06', 'D010Q04', 'D010Q05', 'D010Q06', 'D014Q05', 'D014Q08', 'D015Q01', 'D015Q02', 'D031Q03', 'D031Q04', 'D033Q09', 'D033Q10', 'D042Q09', 'D048Q07', 'D068Q04', 'D068Q05', 'D070Q02', 'D071Q04', 'D071Q05', 'D071Q06', 'D071Q07', 'D071Q08', 'D071Q09'] YesNo: ['D014Q06', 'D116Q05', 'D116Q08', 'D313Q04', 'D313Q05', 'D315Q06', 'D301Q06'] Arithmetic-Operations: ['D085Q08', 'D113Q05', 'D246Q08', 'D248Q04'] Counting: ['D014Q04', 'D015Q05', 'D015Q06', 'D048Q01', 'D076Q02', 'D076Q03', 'D076Q06', 'D076Q07', 'D076Q08', 'D252Q02', 'D252Q06'] Date-Duration: ['D033Q11', 'D035Q01', 'D035Q09', 'D035Q10', 'D038Q09', 'D042Q10', 'D068Q02', 'D070Q07', 'D071Q14', 'D076Q10', 'D087Q02', 'D104Q02', 'D104Q05', 'D105Q02', 'D105Q04', 'D214Q05', 'D238Q07', 'D311Q07', 'D267Q07'] Kinship: ['D091Q03', 'D091Q12', 'D282Q10', 'D282Q11', 'D288Q12'] --------------------------------------------------------------------------------