Semi-automated classification for multi-label open-ended questions
Section 4. The majority-voted-based ensemble of PCC for
semi-automated classification

• Previous
• Next

The proposed method aims to ensemble multiple PCC models at much less computational cost. As mentioned in Section 3.2, the best label set (with the highest joint probability) for a single PCC can be found by a fast search strategy. In this paper, we use UCS, since the implementation is simple and the algorithm always finds the optimal solution. Using UCS, the proposed method obtains ${\widehat{Y}}_j$ $\left(j\mathrm{<mo>=</mo>}\mathrm{1,}\dots ,m\right),$ the label set predicted by the $j^{\text{th}}$ PCC model and ${\widehat{P}}_j,$ the estimated probability that ${\widehat{Y}}_j$ is the true label set. Among the $m$ predicted label sets, the proposed method chooses the most frequent label set for the final prediction. That is, $\widehat{Y}\mathrm{<mo>=</mo>}\text{mode}\left(\left\{{\widehat{Y}}_1\mathrm{,}\dots ,{\widehat{Y}}_m\right\}\right).$ In case there are ties in the mode, we choose the label set whose averaged probability estimate is the highest.

Semi-automatic classification requires a score that measures how easy/hard the prediction is. Whether a text answer is classified automatically or manually is determined based on this score. Next, a score is proposed: Let $J$ be the set that contains all indices $j$ $\left(1\le j\le m\right)$ for which ${\widehat{Y}}_j$ is the most frequent one $($ i.e., $J\mathrm{<mo>=</mo>}\left\{j\mathrm{<mo>:</mo>}{\widehat{Y}}_j\mathrm{<mo>=</mo>}\widehat{Y}\right\})$. The proposed score for the prediction is

$$\begin{array}{ll}\theta \hfill & =\left(\frac{{\displaystyle {\sum }_{i\in J}{\widehat{P}}_j}}{\left|J\right|}\right)\left(\frac{\left|J\right|}{m}\right)(4.1)\hfill \\ \hfill & \mathrm{<mtext>\hspace{0.17em}</mtext>\; <mo>=</mo>}\frac{{\displaystyle {\sum }_{i\in J}{\widehat{P}}_j}}{m}\mathrm{.}(4.2)\hfill \end{array}$$

The first factor of equation (4.1) is the average joint probability of the predicted label set. The second factor of equation (4.1) is the fraction of the PCC models that predict the predicted label set among the $m$ models. Multiplying the two components makes sense: a prediction may be more accurate if the (average) probability related to the chosen label set is high (the first factor) and more individual chain models vote for the same label set (the second component). We call this approach Majority-vote-based Ensemble of Probabilistic Classifier Chains (MEPCC). We later show empirically that combining the two factors indeed improves performance over just using a single factor. Table 4.1 illustrates an example for 5 labels $\left(L=5\right)$ and 7 PCC models $\left(m=7\right).$ The MEPCC approach stores the probability of one label set from each PCC model. Because MEPCC combines over the probabilities corresponding to the best label set from different PCC models, MEPCC can take advantage of the UCS (or any other) strategy. Note that a search strategy like UCS cannot be used for EPCC where all individual probabilities for all label combinations are required. More succinctly, MEPCC combines over the maximal probabilities of each PCC, whereas EPCC maximizes over the average probabilities, requiring evaluation of all individual probabilities. We summarize the procedure of MEPCC in Algorithm 1.

• Previous
• Article main page
• Next