OVERALL ACC: (BEFORE OUTPUT FORMATTER)
	108 / 193 = 0.5595854922279793
--------------------------------------------------------------------------------
OVERALL ACC: (AFTER OUTPUT FORMATTER)
	110 / 193 = 0.5699481865284974
Formatter Errors:
	['D285Q04']
--------------------------------------------------------------------------------
AGGREGATOR PERFORMANCE:
	EASY: 105 / 105 = 1.0
	HARD: 3 / 39 = 0.07692307692307693
--------------------------------------------------------------------------------
ANSWER MODULE STATS (WHEN CORRECTLY ACTIVATED ACCORDING TO THE DISPATCHER):
                                 ACC
Single-Span-Extraction           54 / 97 = 0.5567
Single-Span-Multi-Hop-Extraction 65 / 97 = 0.6701
Multi-Spans-Extraction           0 / 20 = 0.0000
YesNo                            18 / 25 = 0.7200
Arithmetic-Operations            7 / 14 = 0.5000
Counting                         0 / 0 = NaN
Date-Duration                    12 / 32 = 0.3750
Kinship                          0 / 2 = 0.0000
Wiki-Json-Inference              0 / 0 = NaN
Summarize                        0 / 3 = 0.0000
--------------------------------------------------------------------------------
ANSWER MODULE STATS (WHEN WRONGLY ACTIVATED ACCORDING TO THE DISPATCHER):
                                ACC
Single-Span-Extraction           17 / 96 = 0.1771
Single-Span-Multi-Hop-Extraction 22 / 96 = 0.2292
Multi-Spans-Extraction           20 / 173 = 0.1156
YesNo                            0 / 168 = 0.0000
Arithmetic-Operations            2 / 179 = 0.0112
Counting                         0 / 0 = NaN
Date-Duration                    4 / 161 = 0.0248
Kinship                          5 / 191 = 0.0262
Wiki-Json-Inference              0 / 0 = NaN
Summarize                        0 / 190 = 0.0000
--------------------------------------------------------------------------------
ANSWER MODULE ERRORS (WHEN CORRECTLY ACTIVATED):
Single-Span-Extraction:
['D003Q04', 'D009Q02', 'D033Q01', 'D033Q02', 'D033Q03', 'D033Q04', 'D033Q05', 'D033Q06', 'D033Q07', 'D033Q08', 'D033Q12', 'D033Q13', 'D033Q14', 'D033Q15', 'D039Q03', 'D039Q04', 'D039Q05', 'D039Q06', 'D039Q07', 'D039Q08', 'D039Q09', 'D045Q05', 'D069Q08', 'D069Q09', 'D069Q12', 'D093Q01', 'D093Q02', 'D105Q07', 'D105Q09', 'D249Q03', 'D249Q04', 'D249Q06', 'D255Q09', 'D255Q12', 'D261Q05', 'D261Q07', 'D267Q02', 'D291Q08', 'D291Q09', 'D303Q03', 'D309Q04', 'D315Q07', 'D315Q09']

Single-Span-Multi-Hop-Extraction:
['D003Q04', 'D009Q02', 'D033Q01', 'D033Q03', 'D033Q04', 'D033Q05', 'D033Q06', 'D033Q07', 'D033Q08', 'D033Q14', 'D033Q15', 'D039Q03', 'D039Q04', 'D039Q05', 'D039Q06', 'D039Q08', 'D039Q09', 'D045Q04', 'D069Q03', 'D069Q09', 'D069Q12', 'D087Q01', 'D087Q06', 'D093Q07', 'D243Q04', 'D255Q04', 'D255Q10', 'D255Q12', 'D291Q09', 'D303Q03', 'D315Q01', 'D315Q09']

Multi-Spans-Extraction:
['D003Q03', 'D003Q06', 'D009Q04', 'D015Q01', 'D015Q02', 'D015Q04', 'D033Q09', 'D033Q10', 'D045Q02', 'D069Q10', 'D087Q03', 'D087Q04', 'D093Q04', 'D105Q05', 'D117Q04', 'D249Q02', 'D303Q06', 'D309Q05', 'D315Q02', 'D315Q04']

YesNo:
['D069Q11', 'D117Q08', 'D201Q05', 'D255Q07', 'D291Q03', 'D291Q05', 'D315Q06']

Arithmetic-Operations:
['D015Q05', 'D015Q06', 'D069Q13', 'D093Q03', 'D183Q02', 'D309Q02', 'D309Q03']

Counting:
[]

Date-Duration:
['D033Q11', 'D069Q04', 'D069Q05', 'D069Q06', 'D069Q07', 'D087Q02', 'D087Q07', 'D105Q01', 'D105Q02', 'D105Q03', 'D105Q04', 'D117Q01', 'D117Q02', 'D117Q03', 'D243Q07', 'D255Q01', 'D255Q11', 'D285Q05', 'D285Q06', 'D303Q05']

Kinship:
['D039Q01', 'D039Q02']

Wiki-Json-Inference:
[]

Summarize:
['D009Q03', 'D321Q03', 'D321Q04']

--------------------------------------------------------------------------------