3 An error localisation problem with hard and soft edits

Sander Scholtus

Previous | Next

In the formulation of the error localisation problem given in Section 2.2, which is based on the Fellegi-Holt paradigm, it is tacitly assumed that all edits are hard edits. Consequently, the only subsets of the variables that are considered feasible solutions to this problem are those which can be imputed to make the record consistent with respect to all edits. As mentioned in the introduction, this interpretation of all edits as hard edits can lead to systematic differences between automatic editing and manual editing, because it precludes a meaningful use of soft edits. In this section, we suggest a new formulation of the error localisation problem which distinguishes between hard and soft edits.

Let $\Psi$ denote the set of edits to be used in the error localisation problem. We assume that this set can be partitioned into two disjoint subsets: $\Psi ={\Psi }_H\cup {\Psi }_S.$ The edits in ${\Psi }_H$ are hard edits; the edits in ${\Psi }_S$ are soft edits. From now on, a subset of the variables is considered a feasible solution to the error localisation problem if it can be imputed to satisfy all edits in ${\Psi }_H\text{}.$ Moreover, we want to use the status of the imputed record with respect to the edits in ${\Psi }_S$ as auxiliary information in the choice of an optimal solution. This may be done by adding another term to (2.6).

More precisely, the objective of the new error localisation problem is to find a subset of the variables which (i) can be imputed so that the imputed record satisfies all edits in ${\Psi }_H\text{},$ and (ii) minimises the following target function:

$D=\lambda D_{\text{FH}}+(1-\lambda )D_{\text{soft}},$(3.1)

where $D_{\text{soft}}$ represents the costs associated with failed edits in ${\Psi }_S.$ The parameter $\lambda \in [0,1]$ determines the relative contribution of both terms in (3.1). The original Fellegi-Holt paradigm is recovered as a special case by choosing $\lambda =1.$ Thus, the new error localisation problem can be seen as a generalisation of the old one.

In order to use (3.1) in practice, one has to choose an expression for $D_{\text{soft}}.$ Probably the easiest way to assign costs to failed soft edits is to associate a fixed failure weight $s_k$ to each edit in ${\Psi }_S,$ and to define $D_{\text{soft}}$ as the sum of the failure weights of the soft edits that remain failed:

$D_{\text{soft}}={\displaystyle \sum _{k=1}^{K_S}{s}_k{z}_k},$(3.2)

with $K_S$ the number of edits in ${\Psi }_S$ and $z_k$ a binary variable such that $z_k=1$ if the $k^{\text{th}}$ soft edit is failed and $z_k=0$ otherwise. The failure weights may be chosen by subject-matter experts, analogously to the confidence weights, to express the importance that is attached to different soft edits from a subject-matter related point of view. Alternatively, the failure weights may be based on the proportion of records that fail each soft edit in a historical data set which has been edited manually.

A drawback of using fixed failure weights is that they do not take the size of the edit failures into account: every record that fails a particular soft edit receives the same contribution to $D_{\text{soft}},$ namely $s_k.$ By contrast, a human editor sees a soft edit failure as an indication that an observed combination of values is suspicious, and the degree of suspicion depends on the size of the edit failure: a small failure is ignored more easily than a large failure. Hence, it seems interesting to take the size of the edit failures into account in $D_{\text{soft}}.$ This point will be taken up in Section 8, since it introduces certain additional difficulties. For now, we assume that expression (3.2) is used.

We should mention that taking soft restrictions into account by adding an appropriate term to a target function is a well-known technique in mathematical optimisation. The idea is related to other optimisation techniques, such as Lagrangian relaxation (see e.g., Nemhauser and Wolsey 1988). One example of a practical application with soft constraints is that of the so-called benchmarking problem for national accounts (Magnus, Van Tongeren and De Vos 2000). To our best knowledge, the application in the context of the error localisation problem is new.

We should also note that expression (3.1) is in some respects similar to the minimisation criterion of the Nearest-neighbour Imputation Methodology (NIM) developed by Statistics Canada for editing demographic census data (Bankier, Lachance and Poirier 2000; Bankier and Crowe 2009). In particular, the NIM also departs from the Fellegi-Holt paradigm by minimising a convex combination of two terms, the first measuring the amount of imputation and the second measuring the plausibility of the imputed record.

Previous | Next