1. Introduction

Jeroen Pannekoek and Li-Chun Zhang

Previous | Next

We are concerned with the task of reconciling conflicting information in imputed micro data. To illustrate, consider a small part of a record from a structural business survey given in Table 1.1. Two response patterns are postulated; one with only Turnover observed and one where also Employees and Wages are observed. There are many ways to impute the missing values in such a recipient record and the proposed adjustment methods apply irrespective of the imputation method used. The use of partial donor imputation is shown in Table 1.1, where the donor record is the ‘nearest neighbour’ from the same category of economic activity and closest to the recipient record with respect to Turnover for response pattern (I) and Employees, Turnover and Wages for response pattern (II). The imputation is said to be partial because a value of the donor is transferred to the receptor if and only if the corresponding one is missing in the recipient record.

Business records generally have to adhere to a number of accounting and logical constraints. For checking of the validity of a record these are referred to as edit-rules. For the example record here, suppose the following three edit-rules are formulated:

$$\begin{array}{lll}a\text{l: }\hfill & {\text{x}}_1-{\text{x}}_5+{\text{x}}_8=0\hfill & \left(\text{Profit }=\text{ Turnover }-\text{ Total Costs}\right)\hfill \\ a\text{2:}\hfill & {\text{x}}_5-{\text{x}}_3-{\text{x}}_4=0\hfill & \left(\text{Turnover }=\text{ Turnover main }+\text{ Turnover other}\right)\hfill \\ a\text{3:}\hfill & {\text{x}}_8-{\text{x}}_6-{\text{x}}_7=0\hfill & \left(\text{Total Costs }=\text{ Wages }+\text{ Other costs}\right).\hfill \end{array}$$

Partial donor imputation leads to violation of these edit-rules, which we refer to as the (micro-level) consistency problem: for response pattern (I), the first two edit-rules involving Turnover are violated; for response pattern (II), all three edit-rules are violated. To obtain a consistent record, some of the eight values (i.e., including both the observed and imputed ones) have to be changed. Now, in the two cases here, it is possible to change only the imputed values to satisfy all the edit-rules, so let us consider adjustments of the imputed values for the moment.

Table 1.1
Data, missing data and donor values for variables in a business record. Employees (Number of employees); Turnover main (Turnover main activity); Turnover other (Turnover other activities); Turnover (Total turnover); Wages (Costs of wages and salaries)
Table summary
This table displays the results of Data. The information is grouped by Variable (appearing as row headers), Name , Response (I), Response (II) and Donor Values (appearing as column headers).

Variable	Name	Response (I)	Response (II)	Donor Values
${\text{x}}_1$	Profit			330
${\text{x}}_2$	Employees		25	20
${\text{x}}_3$	Turnover Main			1,000
${\text{x}}_4$	Turnover Other			30
${\text{x}}_5$	Turnover	950	950	1,030
${\text{x}}_6$	Wages		550	500
${\text{x}}_7$	Other Costs			200
${\text{x}}_8$	Total Costs			700

Traditional adjustment methods, such as the prorating method implemented in Banff (Banff Support Team 2008), are designed to handle one constraint at a time. In response pattern (I), the prorating method could proceed as follows: (1) adjust the imputed values for Total costs and Profit with a factor 950/1,030 so that they add up to the observed Turnover, (2) adjust the imputed values for Turnover main and Turnover other with the same factor to satisfy the second edit, and (3) adjust the imputed values of Wages and Other costs, again with the same factor to make them add up to the previously adjusted value of Total costs.

For response pattern (II): step (1) and (2) may be carried out as before, but step (3) needs to be modified unless the observed Wages is to be ‘over-written’. Notice that Total costs appears in two edit-rules: $a1$ and $a3.$ When the imputed Total costs is only adjusted according to $a1$ in step (1), the relevant information in the observed Wages is ignored. Indeed, depending on the values available it can even happen that Total costs is adjusted downwards in step (1) to the extend that there is no acceptable non-negative solution left for Other costs at step (3). In general, adjusting a variable that appears in multiple edit-rules according to only one of them is not only suboptimal in theory, it also requires an arbitrary choice of the order in which the edit-rules are to be handled, and it may unnecessarily cause a break-down of the procedure.

Under the assumption that the inconsistency is not due to systematic errors, we propose an optimization approach that treats all the constraints simultaneously. To this end it is convenient to express the edit restrictions in matrix notation, as $Cx=d,$ where $C$ is the constraint (or restriction) matrix, and $d$ a constant vector. For the restrictions $a1-a3,$ we have

$$C=\left(\begin{array}{rrrrrrrr}\hfill 1& \hfill 0& \hfill 0& \hfill 0& \hfill -1& \hfill 0& \hfill 0& \hfill 1\\ \hfill 0& \hfill 0& \hfill -1& \hfill -1& \hfill 1& \hfill 0& \hfill 0& \hfill 0\\ \hfill 0& \hfill 0& \hfill 0& \hfill 0& \hfill 0& \hfill -1& \hfill -1& \hfill 1\end{array}\right)\text{ and }d=0.$$

The non-zero elements in a row of the constraint matrix identify all the variables that are involved in the corresponding edit constraint, and the non-zero elements in a column of the constraint matrix identify all the edit constraints that involve the corresponding variable.

In addition, there are often linear inequality constraints. The simplest case is the non-negativity of most economic variables. The constraints can then be formulated as $C_{\text{eq}}x=d_{\text{eq}}$ and $C_{\text{ineq}}x<d_{\text{ineq}},$ corresponding to the equality and inequality constraints. For ease of exposition we shall, without noting otherwise, adopt the compact expression $Cx\le d.$

As mentioned earlier, not all the values need or should be adjusted. We therefore make a general distinction between free (or adjustable) and fixed (not adjustable) variables. This includes as a special case the situation where all the data values are considered adjustable. We emphasize that the distinction is not necessarily that between the imputed and observed variables, and imputation may have been carried out for missing values as well as erroneous observed ones. For instance, some imputed values may be held fixed because they are derived by logical reasoning as in deductive imputation, or they may have been obtained from external sources that are considered more reliable. Whereas some observed values may be considered unreliable and are allowed to be changed. Given the absence of systematic errors, a general approach is to identify the adjustable variables by “error localization” (e.g., de Waal, Pannekoek and Scholtus 2011), treating the imputed and observed values as equally error-prone. Nevertheless, in much of the text below we shall treat the imputed values as adjustable and the observed ones as fixed for ease of elaboration.

Given the free and fixed variables, the complete data record is accordingly partitioned into sub-vectors $x_{\text{free}}$ and $x_{\text{fixed}},$ and the constraints matrix into $C_{\text{free}}$ and $C_{\text{fixed}},$ containing the columns of $C$ that correspond to $x_{\text{free}}$ and $x_{\text{fixed}},$ respectively. The constraints for the adjustable variables are then given by $C_{\text{free}}{x}_{\text{free}}\le d-C_{\text{fixed}}{x}_{\text{fixed}}$ or, equivalently,

$$A{x}_{\text{free}}\le b(1.1)$$

where the matrix $A$ represents the constraints on the free variables and will be called the accounting matrix and $b$ the constant vector for these constraints. Notice that, while the constraint matrix $C$ is derived a priori from the edit-rules alone, without reference to the actual data, and is the same for all the records, the accounting matrix $A$ is generally different from one record to another, since the distinction between free and fixed variables varies across the units.

Our strategy to remedy the micro inconsistency problem in imputed data is to make adjustments to the adjustable values that are minimal according to some chosen distance (or discrepancy) measure, such that the adjusted record satisfies all the edit-rules. All the constraints are simultaneously handled assuming the absence of systematic errors.

The rest of the paper will contain the following. The optimization approach will be outlined in Section 2. We consider different distance (or discrepancy) measures, the adjustments they generate, and illustrate their properties and interpretations using the example record above. In Section 3 we discuss possible extensions of the basic approach to adjustments based on statistical assumptions in addition to logical constraints, treatment of categorical data, unit imputation with adjustments, and adjustments for macro-level benchmarking constraints in combination with micro-level consistency. In Section 4 we examine the pasture area data from the Norwegian Agriculture Census 2010, including an approach to the assessment of uncertainty due to editing. A final short summary is provided in Section 5.

Previous | Next