OVERALL ACC: (BEFORE OUTPUT FORMATTER)
	159 / 247 = 0.6437246963562753
--------------------------------------------------------------------------------
OVERALL ACC: (AFTER OUTPUT FORMATTER)
	161 / 247 = 0.6518218623481782
Formatter Errors:
	['D241Q07', 'D283Q01']
--------------------------------------------------------------------------------
AGGREGATOR PERFORMANCE:
	EASY: 154 / 156 = 0.9871794871794872
	HARD: 5 / 39 = 0.1282051282051282
--------------------------------------------------------------------------------
ANSWER MODULE STATS (WHEN CORRECTLY ACTIVATED ACCORDING TO THE DISPATCHER):
                                 ACC
Single-Span-Extraction           56 / 124 = 0.4516
Single-Span-Multi-Hop-Extraction 88 / 124 = 0.7097
Multi-Spans-Extraction           0 / 21 = 0.0000
YesNo                            19 / 28 = 0.6786
Arithmetic-Operations            13 / 23 = 0.5652
Counting                         0 / 0 = NaN
Date-Duration                    14 / 23 = 0.6087
Kinship                          18 / 23 = 0.7826
Wiki-Json-Inference              0 / 0 = NaN
Summarize                        0 / 5 = 0.0000
--------------------------------------------------------------------------------
ANSWER MODULE STATS (WHEN WRONGLY ACTIVATED ACCORDING TO THE DISPATCHER):
                                ACC
Single-Span-Extraction           25 / 123 = 0.2033
Single-Span-Multi-Hop-Extraction 33 / 123 = 0.2683
Multi-Spans-Extraction           17 / 226 = 0.0752
YesNo                            0 / 219 = 0.0000
Arithmetic-Operations            3 / 224 = 0.0134
Counting                         0 / 0 = NaN
Date-Duration                    3 / 224 = 0.0134
Kinship                          3 / 224 = 0.0134
Wiki-Json-Inference              0 / 0 = NaN
Summarize                        0 / 242 = 0.0000
--------------------------------------------------------------------------------
ANSWER MODULE ERRORS (WHEN CORRECTLY ACTIVATED):
Single-Span-Extraction:
['D001Q03', 'D007Q03', 'D013Q10', 'D013Q11', 'D013Q12', 'D031Q01', 'D031Q02', 'D037Q11', 'D037Q12', 'D049Q01', 'D049Q05', 'D049Q07', 'D073Q01', 'D073Q10', 'D073Q11', 'D079Q01', 'D079Q05', 'D079Q06', 'D079Q07', 'D085Q02', 'D085Q03', 'D085Q04', 'D085Q05', 'D085Q06', 'D091Q02', 'D091Q04', 'D097Q03', 'D115Q03', 'D115Q04', 'D115Q07', 'D127Q01', 'D181Q02', 'D181Q04', 'D247Q02', 'D247Q06', 'D253Q01', 'D253Q09', 'D253Q10', 'D253Q11', 'D253Q12', 'D259Q04', 'D271Q03', 'D271Q04', 'D271Q05', 'D271Q06', 'D283Q04', 'D283Q05', 'D283Q07', 'D289Q02', 'D289Q08', 'D289Q09', 'D289Q10', 'D289Q11', 'D295Q02', 'D295Q03', 'D295Q04', 'D295Q06', 'D295Q07', 'D295Q09', 'D301Q04', 'D301Q08', 'D301Q09', 'D301Q10', 'D301Q11', 'D307Q06', 'D307Q07', 'D325Q01', 'D325Q03']

Single-Span-Multi-Hop-Extraction:
['D007Q02', 'D007Q03', 'D013Q01', 'D013Q08', 'D013Q12', 'D031Q01', 'D031Q02', 'D037Q12', 'D049Q01', 'D049Q05', 'D061Q03', 'D073Q11', 'D079Q06', 'D079Q07', 'D085Q04', 'D091Q03', 'D097Q03', 'D115Q03', 'D115Q04', 'D115Q07', 'D127Q01', 'D181Q02', 'D211Q03', 'D211Q05', 'D247Q02', 'D247Q06', 'D253Q01', 'D253Q06', 'D289Q08', 'D295Q04', 'D295Q07', 'D301Q02', 'D307Q03', 'D307Q06', 'D325Q03', 'D325Q04']

Multi-Spans-Extraction:
['D001Q08', 'D007Q01', 'D007Q04', 'D007Q05', 'D013Q02', 'D031Q03', 'D031Q04', 'D049Q02', 'D073Q02', 'D073Q06', 'D073Q07', 'D073Q08', 'D073Q09', 'D085Q07', 'D115Q06', 'D115Q08', 'D127Q07', 'D241Q08', 'D241Q09', 'D313Q01', 'D313Q02']

YesNo:
['D037Q17', 'D037Q18', 'D115Q02', 'D127Q06', 'D253Q07', 'D301Q06', 'D313Q04', 'D313Q05', 'D325Q05']

Arithmetic-Operations:
['D049Q03', 'D073Q04', 'D085Q01', 'D085Q08', 'D097Q02', 'D247Q04', 'D247Q07', 'D247Q09', 'D253Q02', 'D295Q01']

Counting:
[]

Date-Duration:
['D001Q09', 'D061Q04', 'D097Q06', 'D097Q07', 'D097Q08', 'D103Q04', 'D247Q08', 'D259Q05', 'D283Q06']

Kinship:
['D037Q04', 'D037Q05', 'D037Q06', 'D091Q11', 'D091Q12']

Wiki-Json-Inference:
[]

Summarize:
['D001Q11', 'D049Q04', 'D073Q12', 'D103Q07', 'D289Q07']

--------------------------------------------------------------------------------