6 Example

Sander Scholtus

Previous | Next

To illustrate the algorithm of Section 5, we will apply it to a small example with numerical data. This is essentially an example from De Waal (2003b) to which we have added a distinction between hard and soft edits. For a somewhat larger example involving a mix of categorical and numerical variables, see Scholtus (2011).

In a fictitious business survey, there are four numerical variables: total turnover $(T),$ profit $(P),$ total costs $(C),$ and number of employees $(N).$ The following hard edits and soft edits have been identified:

$$\begin{array}{ccc}{\psi }_{0H}^1:& T-C-P=0& \\ {\psi }_{0H}^2:& T\ge 0& \\ {\psi }_{0H}^3:& C\ge 0& \\ {\psi }_{0H}^4:& N\ge 0& \\ {\psi }_{0H}^5:& 550N-T\ge 0& \\ {\psi }_{0S}^1:& 0.5T-P\ge 0& (B_{0S}^1=\{1\})\\ {\psi }_{0S}^2:& P+0.1T\ge 0& (B_{0S}^2=\{2\}).\end{array}$$

Consider the following unedited record: $(T^0,P^0,C^0,N^0)=$ (100; 40,000; 60,000; 5). This record fails the first hard edit and the first soft edit. The confidence weights of the variables are $(w_T,w_P,w_C,w_N)=$ (2, 1, 1, 3). We choose the failure weights of the two soft edits to be $s_1=s_2=2.$ Finally, we choose $\lambda =1/2$ in expression (3.1).

Suppose that the variable $P$ is selected first. In the branch where $P$ is eliminated from the original edits, we obtain the following new set of edits:

$\begin{array}{cccc}{\psi }_{1H}^1:& T\ge 0& & ({\psi }_{0H}^2)\\ {\psi }_{1H}^2:& C\ge 0& & ({\psi }_{0H}^3)\\ {\psi }_{1H}^3:& N\ge 0& & ({\psi }_{0H}^4)\\ {\psi }_{1H}^4:& 550N-T\ge 0& & ({\psi }_{0H}^5)\\ {\psi }_{1S}^1:& -0.5T+C\ge 0& (B_{1S}^1=\{1\})& ({\psi }_{0H}^1,{\psi }_{0S}^1)\\ {\psi }_{1S}^2:& 1.1T-C\ge 0& (B_{1S}^2=\{2\})& ({\psi }_{0H}^1,{\psi }_{0S}^2)\\ {\psi }_{1S}^3:& 0.6T\ge 0& (B_{1S}^3=\{1,2\})& ({\psi }_{0S}^1,{\psi }_{0S}^2).\end{array}$

We have indicated in brackets from which of the previous edits each new edit is derived. The third soft edit ${\psi }_{1S}^3$ is in fact equivalent to the first hard edit ${\psi }_{1H}^1,$ which means that it can be discarded.

Upon substituting the original values $(T^0,C^0,N^0)=$ (100; 60,000; 5) into the current edits, it is seen that all edits are satisfied except for ${\psi }_{1S}^2.$ Since all hard edits are satisfied, identifying only the original value of $P$ as erroneous is a feasible solution to the error localisation problem. Moreover, since $B=\{2\}$ is (trivially) a minimal representing set of $B_{1S}^2,$ it is possible to impute a value for $P$ which satisfies all the original edits except for ${\psi }_{0S}^2.$ Hence, the value of target function (3.1) for this solution is $(w_P+s_2)/2=3/2.$

Possibly, the current solution may be improved by eliminating another variable, say $C,$ from the current set of edits. This yields:

$\begin{array}{cccc}{\psi }_{2H}^1:& T\ge 0& & ({\psi }_{1H}^1)\\ {\psi }_{2H}^2:& N\ge 0& & ({\psi }_{1H}^3)\\ {\psi }_{2H}^3:& 550N-T\ge 0& & ({\psi }_{1H}^4)\\ {\psi }_{2S}^1:& 1.1T\ge 0& (B_{2S}^1=\{2\})& ({\psi }_{1H}^2,{\psi }_{1S}^2)\\ {\psi }_{2S}^2:& 0.6T\ge 0& (B_{2S}^2=\{1,2\})& ({\psi }_{1S}^1,{\psi }_{1S}^2).\end{array}$

Each of the two new soft edits is redundant, because both are equivalent to hard edit ${\psi }_{2H}^1.$ In fact, the remaining original values $(T^0,N^0)=(100,5)$ satisfy all the current edits. This means that $P$ and $C$ can be imputed to satisfy all the original edits, both hard and soft. The value of target function (3.1) for this solution equals $(w_P+w_C)/2=1.$ Thus, the new solution improves on the previous one. Moreover, this solution cannot be improved further by eliminating more variables in the current branch of the binary tree.

If the rest of the binary tree is explored, it eventually turns out that the best solution found so far (impute $P$ and $C)$ is also the optimal solution. A possible consistent record obtained by imputing $P$ and $C$ is: $(T,P,C,N)=$ (100, 40, 60, 5). This solution has the nice interpretation that the original values of profit and total costs were overstated by a factor of 1,000. It is of interest to note that, if only the hard edits are used in this example, then the first solution found above (impute only $P)$ is the optimal solution. In that case, there is only one way to obtain a consistent record: $(T,P,C,N)=$ (100; -59,900; 60,000; 5). This illustrates that, in this example at least, soft edits are important for finding imputations that are not only consistent with the hard edits, but also plausible.

Previous | Next