OVERALL ACC: 101 / 208 = 0.4855769230769231 -------------------------------------------------------------------------------- DISPATCHER PERFORMANCE (ANSWER MODE TOP-1): GRND/PRED SS MS YN ARI CNT CMP CS DD KS Single-Span-Extraction 130 3 0 0 0 0 0 0 0 Multi-Spans-Extraction 0 25 0 0 0 0 0 0 0 YesNo 0 0 20 0 0 0 0 0 0 Arithmetic-Operations 0 0 0 1 0 0 0 0 0 Counting 0 0 0 0 3 0 0 0 0 Comparing-Members 0 0 0 0 0 0 0 0 0 CommonSense 0 0 0 0 0 0 0 0 0 Date-Duration 0 0 0 0 0 0 0 12 0 Kinship 0 1 0 0 0 0 0 0 13 -------------------------------------------------------------------------------- DISPATCHER PERFORMANCE (ANSWER TYPE): GRND/PRED PER D-D LOC ORG NUM YN KIN EVT OBJ MISC Person 25 0 0 0 0 0 0 0 0 0 Date-Duration 0 35 0 0 0 0 0 0 0 0 Location 0 0 24 0 0 0 0 0 0 0 Organization 0 0 0 15 0 0 0 0 0 0 Num-Measure 0 0 0 0 16 0 0 0 0 0 YesNo 0 0 0 0 0 20 0 0 0 0 Kinship 0 0 0 0 0 0 16 1 0 0 Event 0 0 0 0 0 0 0 1 0 0 Object 0 0 0 0 0 0 0 0 50 2 Misc 0 0 0 0 0 0 0 0 0 3 -------------------------------------------------------------------------------- ANSWER MODULE STATS (WHEN CORRECTLY ACTIVATED): ACC SCORE MEAN SCORE STD-VAR Single-Span-Extraction 0.4812 0.8466 0.1839 Single-Span-Multi-Hop-Extraction 0.6241 0.9087 0.1663 Multi-Spans-Extraction 0.0000 0.0000 0.0000 YesNo 0.6000 0.9578 0.1088 Arithmetic-Operations 1.0000 1.0000 0.0000 Counting 0.0000 0.0000 0.0000 Date-Duration 0.2500 0.7655 0.3243 Kinship 0.0000 1.0000 0.0000 -------------------------------------------------------------------------------- ANSWER MODULE STATS (WHEN WRONGLY ACTIVATED): ACC SCORE MEAN SCORE STD-VAR Single-Span-Extraction 0.0000 0.8534 0.2039 Single-Span-Multi-Hop-Extraction 0.0000 0.7637 0.2405 Multi-Spans-Extraction 0.0278 0.0000 0.0000 YesNo 0.0000 0.9612 0.0743 Arithmetic-Operations 0.4590 0.5178 0.3209 Counting 0.0238 0.0000 0.0000 Date-Duration 0.4476 0.7399 0.3072 Kinship 0.0000 1.0000 0.0000 -------------------------------------------------------------------------------- ANSWER MODULE ERROR SAMPLES (WHEN CORRECTLY ACTIVATED): Single-Span-Extraction: ['D002Q01', 'D002Q02', 'D002Q04', 'D002Q08', 'D002Q10', 'D009Q02', 'D009Q03', 'D016Q03', 'D046Q04', 'D046Q05', 'D046Q08', 'D064Q03', 'D064Q04', 'D064Q05', 'D064Q06', 'D064Q07', 'D064Q09', 'D064Q11', 'D064Q12', 'D064Q16', 'D073Q01', 'D073Q12', 'D083Q03', 'D083Q07', 'D083Q08', 'D090Q01', 'D090Q03', 'D090Q06', 'D090Q10', 'D090Q11'] Single-Span-Multi-Hop-Extraction: ['D002Q02', 'D002Q04', 'D002Q10', 'D009Q02', 'D009Q03', 'D046Q08', 'D064Q03', 'D064Q05', 'D064Q06', 'D064Q13', 'D064Q16', 'D073Q12', 'D083Q01', 'D083Q03', 'D083Q08', 'D083Q10', 'D090Q01', 'D106Q03', 'D106Q07', 'D117Q02', 'D181Q02', 'D181Q03', 'D181Q04', 'D215Q01', 'D215Q06', 'D215Q09', 'D215Q10', 'D245Q01', 'D245Q05', 'D310Q07'] Multi-Spans-Extraction: ['D009Q04', 'D016Q01', 'D016Q02', 'D016Q04', 'D037Q06', 'D037Q11', 'D037Q12', 'D046Q02', 'D046Q06', 'D064Q08', 'D073Q02', 'D073Q06', 'D073Q07', 'D073Q08', 'D073Q09', 'D073Q11', 'D090Q09', 'D096Q02', 'D117Q04', 'D310Q02', 'D318Q01', 'D268Q09', 'D268Q10', 'D296Q05', 'D303Q06'] YesNo: ['D037Q19', 'D037Q20', 'D106Q05', 'D117Q05', 'D253Q07', 'D260Q02', 'D274Q05', 'D289Q04'] Arithmetic-Operations: [] Counting: ['D117Q01', 'D245Q04', 'D283Q01'] Date-Duration: ['D083Q02', 'D215Q11', 'D245Q03', 'D310Q03', 'D310Q05', 'D310Q06', 'D318Q07', 'D318Q08', 'D318Q09'] Kinship: ['D037Q01', 'D037Q02', 'D037Q03', 'D037Q04', 'D037Q07', 'D037Q08', 'D037Q09', 'D037Q10', 'D037Q13', 'D037Q14', 'D037Q15', 'D037Q16', 'D289Q06'] --------------------------------------------------------------------------------