2 FINAL THOUGHTS DETERMINE THE BEST METRIC FOR EVALUATING SUCH AP-DE...

5.2 Final Thoughts

determine the best metric for evaluating such ap-

Despite these open questions, initial trials with

proaches, which is a question for QA in general.

QA-by-Dossier-with-Constraints have been very

The task of generating auxiliary questions and

encouraging, whether it is by correctly answering

constraint sets is a matter of active research. Even

previously missed questions, or by improving confi-

for simple questions like the ones considered here,

dences of correct answers. An interesting question

the auxiliary questions and constraints we looked at

is when it is appropriate to apply QDC. Clearly, if

were different and manually chosen. Hand-crafting a

the base QA system is too poor, then the answers to

large number of such sets might not be feasible, but

the auxiliary questions will be useless; if the base

it is certainly possible to build a few for common

system is highly accurate, the increase in accuracy

situations, such as a person’s life-cycle. More gener-

will be negligible. Thus our approach seems most

ally, QDC could be applied to situations in which a

beneficial to middle-performance levels, which, by

certain structure is induced by natural temporal (our

inspection of TREC results for the last 5 years, is

Leonardo example) and/or spatial constraints, or by

where the leading systems currently lie.

properties of the relation mentioned in the question

We had initially thought that use of constraints

(evaluation example). Temporal and spatial con-

would obviate the need for much of the complexity

straints appear general to all relevant question types,

inherent in NLP. As mentioned earlier, with the

and include relations of precedence, inclusion, etc.

case of “The Beatles” being the reciprocal answer to

For certain relationships, there are naturally-

the auxiliary composition question to “Who is Paul

occurring reciprocals (if X is married to Y, then Y is

McCartney?”, we see that structured, ontological

married to X; if X is a child of Y then Y is a parent

information would benefit QDC. Identifying alter-

of X; compound-term to acronym and vice versa).

nate spellings and representations of the same name

Transitive relationships (e.g. greater-than, located-

(e.g. Clavier/Klavier, but also taking care of varia-

in, etc.) offer the immediate possibility of con-

tions in punctuation and completeness) is also nec-

straints, but this avenue has not yet been explored.

essary. When we asked “Who is Ian Anderson?”,

having in mind the singer-flautist for the Jethro Tull