1 SUBJECTIVE METRICS PROCEDURE

5.1 Subjective metrics

procedure: 1) read a short document of background

Our main resource for investigating the effect of

material, 2) took a pretest to measure initial phys-

the NM was the system questionnaires given after

ics knowledge, 3) worked through 2 problems with

each problem. These questionnaires are identical

benefit from access to the discourse structure in-

formation. These questions probe the user’s per-

and include 16 questions that probed user’s percep-

ception of ITSPOKE during the dialogue. We find

tion of ITSPOKE on various dimensions. Users

were asked to answer the questions on a scale from

that for 6 out 7 questions the NM presence has a

significant/trend effect (Table 1, column 2).