Semi-automated classification for multi-label open-ended questions
Section 5. Experiments

• Previous
• Next

5.1  Data

We evaluated the performance of the MEPCC algorithm on three different data sets: Civil disobedience, Immigrant and Happy data (the Happy data are available upon request by contacting Marika Wenemark marika.wenemark@liu.se. The Immigrant and Civil Disobedience data are available from the GESIS Datorium http://dx.doi.org/10.7802/1795). For each data set, an open-ended question was asked to the respondents and their answers have been coded manually with possibly multiple labels.

The Civil data set was collected to study cross-cultural equivalence about Civil disobedience. Behr, Braun, Kaczmirek and Bandilla (2014) first asked respondents a closed-ended question from the ISSP (ISSP Research Group, 2012) How important is it that citizens may engage in acts of Civil disobedience when they oppose government actions? (Not at all important 1 − Very important 7). The respondents were then asked: What ideas do you associate with the phrase “Civil disobedience”? Please give examples. Answers were classified into 12 labels: non-productive, violence, disturbances, peaceful, listing activities, breadth of actions, breaking law, breaking rules, government:dissatisfaction, government:deep rift, copy/paste from the Internet, other. The survey data were collected in different languages and we use a merged data set (Spanish, German and Danish) that contains 1,029 observations.

The Immigrant data set was collected to study cross-national equivalence of measures of xenophobia. In the 2003 International Social Survey Program (ISSP) on National Identity, the questionnaire contained four statements regarding beliefs on Immigrants such as Immigrants take jobs from people who were born in Germany. After rating each statement, respondents were asked to answer to an open-ended question: Which type of Immigrants were you thinking of when you answered the question? The previous statement was: [text of the corresponding item]. Braun, Behr and Kaczmirek (2013) classified answers into 14 labels: non-productive, positive, negative, neutral/work, general, Muslim countries, eastern European, Asia, ex-Yugoslavia, EU15, sub Sahara, Sinti/Roma, legal/illegal, other. In this article, we use 1,006 observations from the German survey.

The Happy data set was collected to study the relationship between positive factors and mental health and care needs. Wenemark, Borgstedt-Risberg, Garvin, Dahlin, Jusufbegovic, Gamme, Johansson and Bjrn (2018) asked respondents “Name some positive things in your life, that are uplifting or make you Happy: (you may write several things)”. Answers were classified into 13 labels: nothing, relationships (family or romantic), working/studying, health, self-esteem, joy/happiness, well-being: drinking/eating/drugs/sex, spirituality, money, nature, hobbies, culture, and exercise. The data set contains 2,350 observations.

Table 5.1 contains summary statistics about the three data sets.

5.2  Experimental setup

We compared the proposed MEPCC method against BR and LP and PCC. For PCC, we used the uniform search to reach a predicted label set and the estimated probability of equation (3.1) for the confidence score of the prediction. EPCC was not included in the comparison because its computational cost makes it infeasible for prediction for our data sets. (In our experiment on the Immigrant data with 14 labels, running the exhaustive search for PCC $\left(m=1\right)$ for a single prediction took a single computer (Intel Core i7 CPU with 8GB RAM) over 30 minutes. This implies that predicting 200 observations using EPCC $\left(m=10\right)$ would take more than 1,000 hours.) Support vector machines (SVM) (Vapnik, 2000) were used as the base classifier on unscaled variables with a linear kernel and tuning parameter $C\mathrm{<mo>=</mo>}1.$ For probabilistic output, the SVM scores were converted into probabilities using Platt’s method (Platt, 2000). The analysis was conducted in $R$ (R Core Team, 2014) using the $e1071$ package (Meyer, Dimitriadou, Hornik, Weingessel and Leisch, 2014) for SVM.

For each data set, 5-fold cross validation (CV) was performed. That is, we randomly divided the data into five equal-sized parts and used the first four parts as the training data and the last part as the test data. Performance evaluation is only made on the test data. Each of the five parts were used as test data and the results were averaged.

5.3  Performance of the MEPCC approach

We first investigated the performance of the MEPCC. The score in equation (4.1) has two components. To demonstrate that both components are helpful, we evaluate the proposed score as well as two different scores where one of the components is missing. That is, we compared the MEPCC with three different scores $\theta ,$ ${\theta }_1$ and ${\theta }_2\text{:}$

$$\begin{array}{l}(MEPCC)\theta \mathrm{<mo>=</mo>}\left(\frac{{\displaystyle {\sum }_{i\in J}{P}_j}}{\left|J\right|}\right)\left(\frac{\left|J\right|}{m}\right)\\ (MEPCC-\text{1}\mathrm{<mo\; stretchy="false">)</mo>}{\theta }_1\mathrm{<mo>=</mo>}\left(\frac{{\displaystyle {\sum }_{i\in J}{P}_j}}{\left|J\right|}\right)\\ (MEPCC-\text{2}\mathrm{<mo\; stretchy="false">)</mo>}{\theta }_2\mathrm{<mo>=</mo>}\left(\frac{\left|J\right|}{m}\right)\mathrm{.}\end{array}$$

Prioritizing the text answers based on ${\theta }_2$ results in many ties. The tied answers were randomly reordered to be able to calculate subset accuracy at each production rate. Figure 5.1 shows the subset accuracy of each approach as a function of the production rate. The text answers with higher scores were classified first. For example, production rate 0.2 means only 20% of the test data with the highest scores were classified automatically by the models. When the production rate equals 1, there was no difference between the MEPCC models because the predicted label sets are always the same. The difference is how they prioritize the text answers from the easiest-to-classify to the hardest-to-classify answers. When the production rate was less than 1, MEPCC outperformed MEPCC-1 and MEPCC-2 for all three data. The results show that both components in equation (4.1) were helpful for prioritizing the observations.

5.4  Effect of the number of PCC models

We then investigated to what extent the number of PCC models affects the predictive performance of MEPCC. Figure 5.2 shows the performance of MEPCC for different number of PCC models $\left(m\right).$ When $m$ was low, increasing $m$ led to huge improvement of the subset accuracy of MEPCC. However, once there were enough PCC models (e.g., $m=10),$ adding more PCC models did not improve the subset accuracy. The empirical results show that MEPCC does not require many PCC models for performing well.

5.5  Comparison with other methods

At last we investigated the performance of MEPCC $\left(m\mathrm{<mo>=</mo>}10\right)$ compared to the established methods (BR, LP and PCC). For all methods, a production rate of x% refers to the x% of the data that have the highest score. MEPCC used $\theta$ as a score, while each of the other approaches used the probability of the predicted label set estimated by that method. Note when $m\mathrm{<mo>=</mo>}1,$ MEPCC and PCC are identical; the score $\theta$ coincides with the probability of the label set predicted by PCC.

Figures 5.3 and 5.4 illustrate the respective subset accuracy and Hamming loss for the different methods as a function of the production rate on the Happy, Immigrant and Civil data. For the Immigrant and Happy data, the highest subset accuracy at most production rates was obtained by MEPCC. For the Civil data, MEPCC and LP performed the best. In terms of Hamming loss, MEPCC achieved the lowest error at most production rates for all data.

Next, we consider the performance of each method given target predicted accuracy values. To decide the fraction of automatic categorization, a practitioner will typically set a threshold probability above which texts are coded automatically. For MEPCC, the relationship between true accuracy and the confidence score $\left(\theta \right)$ were estimated via cross-validation on the training data. We used Platt’s scaling to convert the confidence scores into probability outputs. Since Platt’s scaling could improve the level of calibration (Niculescu-Mizil and Caruana, 2005), the same technique was also applied to BR, LP and PCC.

Table 5.2 illustrates the tradeoff between the percentages of automated prediction and the corresponding subset accuracy of each method as a function of different thresholds. The threshold refers to the minimum predicted subset accuracy required for automated prediction. The minimum predicted subset accuracy helps us decide which text answers should be classified automatically and which should be classified manually. For example, if the client decides that at least 80% accuracy is required for automated classification, then approximately 39.3% of the Civil data, 42.5% of the Immigrant data, and 27.6% of the Happy data can be classified automatically by MEPCC with subset accuracy 0.891, 0.916 and 0.857, respectively. Note that this is a huge improvement compared to applying BR that could only automatically classify 9.3% of the Civil data, 12.8% of the Immigrant data, and 8.7% of the Happy data with lower subset accuracies. Table 5.3 shows the relationship between predicted and actual accuracy by aggregating to ranges of predictions for each method and data set. For MEPCC the actual accuracy is within the range of the predicted accuracy in most cases, much better than for the other methods.

Table 5.4 shows the runtime of each method for training the model and predicting all instances in test data (Intel Core i7 CPU with 8GB RAM). Unsurprisingly, the runtime of MEPCC at $m\mathrm{<mo>=</mo>}10$ is roughly 10 times of that of PCC in both of the training and prediction stages.

• Previous
• Article main page
• Next