7 Application

Sander Scholtus

Previous | Next

To test the new error localisation algorithm in practice, a prototype implementation was written using the R programming language. This prototype draws heavily on the existing error localisation functionality in R that was made available in the editrules package (De Jonge and Van der Loo 2011; Van der Loo and De Jonge 2011).

To test the prototype, an artificial data set was constructed by selecting twelve numerical variables $(x_1,\dots ,x_{12})$ from the Dutch structural business statistics of 2007 for the wholesale sector. We selected all records pertaining to medium-sized businesses (with 10 to 100 employees) that had been edited manually during regular production, and divided these into two data sets of 728 records each. Both of the original data sets were considered error-free. We introduced a substantial number of random errors into one of the data sets by applying the following procedure:

• in 4% of the original non-zero values, two digits were interchanged;
• in 4% of the original non-zero values, a random digit was added;
• in 4% of the original non-zero values, a random digit was omitted;
• in 4% of the original non-zero values, a random digit was replaced by another digit;
• 4% of the original non-zero values were multiplied by 25;
• 4% of the original non-zero values were divided by 25 and rounded to the nearest integer;
• 6% of the original non-zero values were replaced by zero;
• 5% of the original zero values were replaced by random integers from {1; …; 1,000};
• 10% of the original values of $x_{11}$ and $x_{12}$ were multiplied by $–$1.

This procedure was carried out in such a way that at most one change could occur in each value. The second data set was left error-free and was used as reference data.

Table 7.1 shows the hard and soft edits that were applied to the test data. The hard edits were copied from the regular production system. The soft edits were identified by examining a number of univariate and bivariate distributions in the reference data.

Table 7.1
The edits that were used in the test application
Table summary
This table displays the results of the edits that were used in the test application. The information is grouped by type (appearing as row headers), edits (appearing as column headers).

Type	Edits
hard edits:	$$\begin{array}{l}{x}_1+x_2=x_3\\ x_2=x_4\\ x_5+x_6+x_7=x_8\\ x_3+x_8=x_9\\ x_9-x_{10}=x_{11}\\ x_j\ge 0(j=1,\dots ,10\text{ and }j=12)\end{array}$$
soft edits:	$$\begin{array}{l}{x}_2\ge 0.5x_3\\ x_3\ge 0.9x_9\\ x_5+x_6\ge x_7\\ x_9\ge 50x_{12}\\ x_9\le 5,000x_{12}\\ x_{11}\le 0.4x_9\\ x_{11}\ge -0.1x_9\\ x_{12}\ge 1\\ x_{12}\ge 5\\ x_{12}\le 100\end{array}$$

The error localisation algorithm was applied to the data set with artificial errors using several different setups. Throughout, all confidence weights $w_j^N$ were chosen equal to 1, and the parameter $\lambda$ in (3.1) was chosen equal to $1/2.$ We considered the following approaches:

1. The first test used only the hard edits from Table 7.1.
2. The second test used all edits from Table 7.1, with all edits interpreted as hard edits.
3. The third test used all edits from Table 7.1, with a distinction between hard and soft edits. Each soft edit received the same fixed failure weight $s_k=1.$
4. The fourth test was similar to the third test, but with fixed failure weights that differed between soft edits. For each soft edit, $s_k$ was calculated as the fraction of records in the reference data set that satisfied the edit. Thus, a soft edit received a lower failure weight if it was failed more often in the reference data set, and vice versa. The rationale behind this is that all soft edit failures occurring in the reference data were caused by unusual but correct combinations of values. By associating low weights to soft edits that are often failed in the reference data, we ensure that these edits may also be failed more easily when editing the test data.

Since the distribution of errors in our test data set was known, we could directly evaluate the performance of each automatic error localisation approach. To this end, we used several quality indicators. Consider the following 2×2 contingency table:

$\begin{array}{rrrr}\hfill & \hfill & \hfill detected:& \hfill \\ \hfill & \hfill & \hfill error& \hfill \text{no error}\\ \hfill \text{true:}& \hfill \text{error}& \hfill TP& \hfill FN\\ \hfill & \hfill \text{no error}& \hfill FP& \hfill TN\end{array}$

The first quality indicator measures the proportion of true errors that were missed by the algorithm (proportion of false negatives):

$$\alpha =\frac{FN}{TP+FN}.$$

The second quality indicator measures the proportion of correct values that were mistaken for errors by the algorithm (proportion of false positives):

$$\beta =\frac{FP}{FP+TN}.$$

The third quality indicator measures the overall proportion of wrong decisions made by the algorithm:

$$\gamma =\frac{FN+FP}{TP+FN+FP+TN}.$$

These three indicators evaluate the performance of the algorithm with respect to identifying individual values as correct or erroneous. They have been used in previous evaluation studies; see, for instance, Pannekoek and De Waal (2005).

To evaluate the performance of the algorithm from a slightly different angle, we also calculated the percentage of records for which the algorithm found exactly the right solution $–$ that is, the solution that identifies as erroneous all erroneous values and only these. This indicator is denoted by $\delta .$ A good editing approach should have low scores on $\alpha ,\beta ,$ and $\gamma ,$ but a high score on $\delta .$

Table 7.2 shows the values of the quality indicators for editing approaches A, B, C, and D. It can be seen that approach B is outperformed by the other approaches on all measures, except for the proportion of missed errors. Thus, using the soft edits as if they were hard edits does not work well for this data set; in fact, better results are achieved by approach A, which does not use the soft edits at all. It can also be seen that approaches C and D, which use the new algorithm to take the soft edits into account, yield better results than approaches A and B, which use the old algorithm. Overall, approach D appears to achieve the best results in this experiment. Compared with approach A, approach D in fact correctly identifies more errors and more correct values.

Table 7.2
Results of automatic error localisation for the artificial data
Table summary
This table displays the results of results of automatic error localisation for the artificial data. The information is grouped by approach (appearing as row headers), quality indicators, calculated using header 1, header 2, header 3 and header 4 units of measure (appearing as column headers).

approach	quality indicators
	$$\alpha$$	$$\beta$$	$$\gamma$$	$$\delta$$
A	0.364	0.047	0.115	40%
B	0.232	0.131	0.153	37%
C	0.227	0.060	0.096	47%
D	0.253	0.037	0.083	52%

It should be noted that, under the old definition of the error localisation problem, approaches A and B represent the two extreme options available for using soft edits: either not using them, or using them all as hard edits. As a compromise between these options, one could also decide to use only a subset of the soft edits as hard edits and discard the others. We did not test this approach during the experiment. One might expect that it would lead to scores on the $\alpha ,\beta ,\gamma ,$ and $\delta$ measures in between those of approaches A and B.

Previous | Next