1 ASKER SATISFACTION PREDICTION FRAMEWORKWE NOW BRIEFLY REVIEW OUR A...

2.1 Asker Satisfaction Prediction Framework

We now briefly review our ASP (Asker Satisfac-

results demonstrate that when sufficient prior asker

tion Prediction) framework that learns to classify

history exists, even simple personalized models re-

whether a question has been satisfactorily answered,

sult in significant improvement over a general pre-

originally introduced in (Liu et al., 2008). ASP em-

diction model. We discuss our findings and future

ploys standard classification techniques to predict,

work in Section 4.

given a question thread, whether an asker would be

2 Predicting Asker Satisfaction in CQA

satisfied. A sample of features used to represent this

We first briefly review the life of a question in a

problem is listed in Table 1. Our features are or-

QA community. A user (the asker) posts a question

ganized around the basic entities in a question an-

by selecting a topical category (e.g., “History”), and

swering community: questions, answers, question-

then enters the question and, optionally, additional

answer pairs, users, and categories. In total, we de-

details. After a short delay the question appears in

veloped 51 features for this task. A sample of the

the respective category list of open questions. At

features used are listed in the Figure 1.

this point, other users can answer the question, vote

• Question Features: Traditional question answer-

on other users’ answers, or interact in other ways.

ing features such as the wh-type of the question

The asker may be notified of the answers as they are

(e.g., “what” or “where”), and whether the ques-

submitted, or may check the contributed answers pe-

tion is similar to other questions in the category.

riodically. If the asker is satisfied with any of the

• Question-Answer Relationship Features: Over-

answers, she can choose it as best, and rate the an-

lap between question and answer, answer length,

swer by assigning stars. At that point, the question

and number of candidate answers. We also use

is considered as closed by asker. For more detailed

features such as the number of positive votes

treatment of user interactions in CQA see (Liu et

(“thumbs up” in Yahoo! Answers), negative votes

al., 2008). If the asker rates the best answer with

(“thumbs down”), and derived statistics such as

at least three out of five “stars”, we believe the asker

the maximum of positive or negative votes re-

is satisfied with the response. But often the asker

ceived for any answer (e.g., to detect cases of bril-

never closes the answer personally, and instead, af-

liant answers or, conversely, blatant abuse).

ter a period of time, the question is closed automat-

• Asker User History: Past asker activity history

ically. In this case, the “best” answer may be cho-

such as the most recent rating, average past satis-

sen by the votes, or alternatively by automatically

faction, and number of previous questions posted.

predicting answer quality (e.g., (Jeon et al., 2006)

Note that only the information available about the

or (Agichtein et al., 2008)). While the best answer

asker prior to posting the question was used.

chosen automatically may be of high quality, it is un-

• Category Features: We hypothesized that user

known if the asker’s information need was satisfied.

behavior (and asker satisfaction) varies by topi-

Based on our exploration we believe that the main

cal question category, as recently shown in refer-

reasons for not “closing” a question are a) the asker

ence (Agichtein et al., 2008). Therefore we model

loses interest in the information and b) none of the

the prior of asker satisfaction for the category,

answers are satisfactory. In both cases, the QA com-

such as the average asker rating (satisfaction).

munity has failed to provide satisfactory answers in

• Text Features: We also include word unigrams and

a timely manner and “lost” the asker’s interest. We

bigrams to represent the text of the question sub-

consider this outcome to be “unsatisfied”. We now

ject, question detail, and the answer content. Sep-

define asker satisfaction more precisely:

arate feature spaces were used for each attribute to

Definition 1 An asker in a QA community is consid-

keep answer text distinct from question text, with

frequency-based filtering.

ered satisfied iff: the asker personally has closed the

question and rated the best answer with at least 3

Classification Algorithms: We experimented with

“stars”. Otherwise, the asker is unsatisfied.

a variety of classifiers in the Weka framework (Wit-

ten and Frank, 2005). In particular, we com-

This definition captures a key aspect of asker satis-

pared Support Vector Machines, Decision trees, and

faction, namely that we can reliably identify when

the asker is satisfied but not the converse.

Boosting-based classifiers. SVM performed the best

Feature

Description

#Questions per Asker

# Questions

# Answers

# Users

Question Features

1

132,279

1,197,089

132,279

2

31,692

287,681

15,846

Q: Q punctuation density

Ratio of punctuation to words in the question