1 Introduction
Sander Scholtus
An important part of every statistical process is data editing, i.e., detecting and correcting errors as well as missing values in the collected data. National statistical institutes have traditionally relied on manual editing, where the data is checked and, if necessary, adjusted by subject-matter experts. Unfortunately, this approach can be very time-consuming and expensive. Alternative methods have therefore been developed to increase the efficiency of the editing process, such as selective editing and automatic editing. This article focuses on the latter approach. We refer to De Waal, Pannekoek and Scholtus (2011) and their references for a discussion of selective editing and other forms of statistical data editing.
The goal of automatic editing is to accurately detect and correct errors as well as missing values in a data file in a fully automated manner, i.e., without human intervention. Provided that automatic editing leads to data of sufficient quality, it can be used as a partial alternative to manual editing. In practice, automatic editing implies that the data is made consistent with respect to a set of constraints: the so-called edits. Examples of edits include:
Profit = Total Turnover Total Costs;(1.1)
and
Profit ≤ 0.6 × Total Turnover.(1.2)
Most automatic editing methods proceed by solving two separate problems: first the error localisation problem, i.e., determining which variables are erroneous or missing, and subsequently the consistent imputation problem, i.e., determining new values for these variables that satisfy all edits. The present article focuses on the error localisation problem.
With respect to the two examples of edits given above, it is interesting to note the conceptual difference that exists between them. Edit (1.1) is an example of an edit that has to hold by definition, so that every combination of values that fails this edit necessarily contains an error. Edits of this type are known as hard edits, fatal edits, or logical edits. Edit (1.2), on the other hand, is an example of an edit that identifies combinations of values that are implausible but not necessarily incorrect. In this example, records for which Profit is larger than 60% of Total Turnover are considered suspicious. However, it is conceivable that such a combination of values is occasionally correct. Edits of this type, which do not identify errors with certainty, are known as soft edits or query edits.
An important limitation of existing algorithms for automatic editing is that they treat all edits as hard edits. That is to say, a failed edit is always attributed to an error in the data. In manual editing, however, subject-matter specialists also make extensive use of soft edits. During automatic editing, these soft edits are either not used at all, or else interpreted as hard edits. Both solutions are unsatisfactory: in the first case some errors may be missed during automatic editing, and in the second case some correct values may be wrongfully identified as erroneous. In fact, the inability of automatic editing methods to handle soft edits partly explains why in practice many differences are found between manually edited and automatically edited data.
The object of this article is to present a new formulation of the automatic error localisation problem which can distinguish between hard edits and soft edits. In addition, the article shows how the error localisation algorithm of SLICE the software package for automatic editing developed at Statistics Netherlands can be adapted to solve this new error localisation problem.
The remainder of this article is organised as follows. Section 2 provides a brief summary of existing methods for solving the error localisation problem. A distinction between hard and soft edits is introduced in the error localisation problem in Section 3. Section 4 extends the theory behind the algorithm of SLICE to the case that not all edits have to be satisfied. Based on these theoretical results, an algorithm that solves the error localisation problem for hard and soft edits is outlined in Section 5. In Section 6, the new algorithm is illustrated by means of a small example. Section 7 mentions the first experiences with a practical implementation of the new algorithm. Finally, some concluding remarks follow in Section 8.
- Date modified: