SECTION 3.3. A BENEFIT, SINCE IT GIVES MORE OPPORTUNITY FOR ENFORC-

8. else return C

1

we’ll call these second-place questions). Given that

TREC scoring only rewards first-place answers, it

seemed that with our incremental approach we

4 Evaluation

would get most benefit there. Also, we were keen to

Due to the complexity of the learned algorithm, we

limit the additional response time incurred by our

decided to evaluate in stages. We first performed an

approach. Since evaluating the top N answers to the

evaluation with a fixed question type, to verify that

original question with the Constraints process re-

the purely arithmetic components of the algorithm

quires calling the QA system another N times per

were performing reasonably. We then evaluated on

question, we were happy to limit N to 2. In addition,

the entire TREC12 factoid question set.

this greatly reduced the number of parameters we

needed to learn.