3. On possible extensions to related adjustment problems

Jeroen Pannekoek and Li-Chun Zhang

Previous | Next

3.1 Generalized ratio adjustments

The ratio model is routinely used for case weighting in business surveys under the assumption that the economic variables can all be related proportionally to a common measure-of-size of the business unit, see e.g., Särndal, Swensson and Wretman (1992). Motivated by the ratio model one could multiply all the donor values by 950/1,030 to obtain the imputed values for the example record under response pattern (I), including the variable Employees $\left(x_2\right)$ for which the initial imputed value 20 does not formally violate any constraints. This shows that there may be situations where, in addition to the logical and accounting constraints, adjustments may be introduced based on statistical assumptions.

For response pattern (II), the observed Employees $\left(x_2\right),$ Turnover $\left(x_5\right)$ and Wages $\left(x_6\right)$ can all potentially be used as the measure-of-size variable in a ratio model, so that a single ratio adjustment does not present itself. However, we may postulate the existence of a common ratio between the recipient and donor records under the ratio model, and regard the observed ratios (i.e., 20/25 for Employees, 950/1,030 for Turnover and 550/500 for Wages) as its random manifestations. Then, it seems that a plausible approach is to identify this common ratio as the value that minimizes the variance, or any other dispersion measure that is deemed suitable, of the three individual ratios. Finally, insofar as the common ratio pertains to the other variables, it becomes possible to adjust them using the following generalized ratio (GR) approach.

Assume the multiplicative adjustment model ${\tilde{x}}_j=x_{\mathrm{0,}j}{\delta }_j,$ where each ${\delta }_j$ is a random manifestation of a theoretical common ratio. Put the distance function

$$D\left(\tilde{x}\mathrm{,}{x}_0\right)=1/2\left({\delta }^T\delta -{\overline{\delta }}^2\right)(3.1)$$

where $\delta$ is the vector of ${\delta }_j’\text{s}$ and $\overline{\delta }$ the mean of them. For all the variables subjected to the common ratio, including both free and fixed ones, we now carry out the adjustment in two steps. The first step is a conceptual one, where we imagine that an adjustment ${\tilde{x}}_j/x_{\mathrm{0,}j}$ is made to the fixed variables: if ${\tilde{x}}_j=x_j$ is observed and fixed, then ${\delta }_j=x_j/x_{\mathrm{0,}j},$ whereas ${\delta }_j=1$ if ${\tilde{x}}_j$ is the imputed value $x_{\mathrm{0,}j}$ but to be held fixed from ‘further’ adjustment. At the second step, adjustments are made to the initial values of the free variables by solving the optimization problem (2.1) with (3.1) as the distance function. This yields the GR adjustments of the free variables involved.

An important condition of the GR approach is that at least one of the ${\delta }_j’\text{s}$ must relate to a fixed variable. Otherwise, ${\tilde{x}}_j\equiv x_{\mathrm{0,}j}$ would be the trivial solution because this always yields $D=0.$ Notice that we have suppressed the denotation $J$ in (3.1), and slightly abused the denotations $x_0$ and $\tilde{x}$ introduced for (2.1). Take response pattern (I) in Table 1.1, the fixed value $x_5=950$ needs to be included in (3.1), yielding ${\delta }_5={\tilde{x}}_5/x_{\mathrm{0,}j}=x_5/x_{\mathrm{0,}j}=950/1,030.$ Solving (2.1) for all the other variables yields then ${\delta }_j\equiv 950/1,030$ and $D=0.$ Whereas, without including ${\delta }_5,$ one would have merely obtained $D=0$ at ${\delta }_j=1$ and ${\tilde{x}}_j=x_{\mathrm{0,}j}$ for $j\ne 5.$

The GR adjustments for response pattern (II) are given in Table 2.1. All the three observed ${\delta }_j’\text{s,}$ for $j=\mathrm{2,5}$ and $6,$ are included in (3.1) and held fixed for the optimization problem. The results are seen to be close to the WLS/KL adjustments. The empirical variance of the multiplicative factors is 0.0270 for the GR adjustments, 0.0276 for WLS/KL and 0.1434 for LS. The relative sum of squared changes, i.e., twice the WLS distance, is 50.6 for the WLS/KL adjustments, 51.6 for GR and 78.0 for LS. Finally, the unweighted sum of squared changes, i.e., twice the LS distance, is 20,925 for the LS adjustments, 23,976 for WLS/KL and 25,090 for GR. Thus, in terms all the three distance functions, the GR adjustments are closer to WLS/KL than LS.

Now, the distance (or discrepancy) measures considered in Section 2.2 may be characterized as decomposable, since the overall distance between two vectors is given as a (weighted) sum of the ‘distances’ between the corresponding components. A consequence is that a variable that does not stand in any constraints will retain the initial value under the minimum adjustment approach. In contrast, the distance (3.1) is non-decomposable, where each adjustment is dependent on the other adjustments. As a result even the values that are not explicitly involved in any constraints will be adjusted as long as they are included in the distance function, because of the changes made to the variables that are constraint-bound. The variable Employees provides an example in Table 2.1. The GR approach provides thus a possibility for adjustments based on statistical assumptions in addition to logical and accounting constraints. Indeed, with a single fixed variable included in (3.1), the GR adjustments are reduced to a common proportional adjustment, in accordance with the ratio-adjustment intuition in this case. With multiple fixed variables included, the GR approach aims at a kind of most-uniform adjustments as a generalization of the single-ratio model. For response pattern (II) in Table 1.1, the approach at once takes into account all the three observed ratios. To achieve the same by formulating an explicit statistical model for exactly this response pattern is not as practical in a production setting.

3.2 Adjustments involving categorical data

A categorical variable carries different constraints from a continuous one. It is worth considering the extent to which categorical variables may be incorporated in the optimization approach. We shall distinguish three types of categorical data that are common in practice.

Firstly, we call a categorical/discreet variable pseudo-continuous if in practice it can be dealt with as if it were a continuous variable. Typical examples of pseudo-continuous variables are age, number of employees, household size, etc. Pseudo-continuity can affect the choice of adjustment model and distance function. For instance, both additive and proportional adjustments may be acceptable for the number of employees, whereas a proportional adjustment of household size or age seems unnatural. Still, having chosen the adjustment model and distance function, one may handle a pseudo-continuous variable just like a real one. Rounding is necessary afterwards and its effect needs to be monitored.

Secondly, what we call a nominal categorical variable indicates whether a unit falls into a particular category. A nominal variable with $M$ categories, labelled $x=1,2,\mathrm{...},M,$ carry with it the constraint

$${\displaystyle \prod _{m=1}^M}\left(\tilde{x}-m\right)=0.(3.2)$$

However, the labels (e.g., 1 = tomatoes, 2 = beans, 3 = cucumbers) are not suitable for operations such as addition, multiplication or rounding. Neither is a nominal value 3 more distant to 1 than 2. Therefore, the constraint (3.2) can not be taken into account under the minimum adjustment approach which assumes interval scale measurements. The adjustment of an observed value that does not satisfy (3.2) must be handled by marking it as missing and, then, imputing some admissible as well as suitable value, i.e., just like in case the value is missing to start with.

Thirdly, a variable may be defined to have value zero for the units that are not eligible. Depending on whether the measure is pseudo-continuous or nominal when the unit is eligible, we have a semi-continuous/-nominal variable that has a non-zero probability of being zero. The difference to pseudo-continuity above is that a semi-continuous variable may require an additional non-negativity constraint in the accounting matrix. Consider then a semi-nominal variable. In practical questionnaire design, such a variable is often split in two, say, $X_1$ and $X_2.$ Let $X_1=1$ if the unit is engaged in a certain activity, say, production of greenhouse vegetables, and let $X_1=0$ otherwise. Let $X_2$ be a nominal measure of activity when $X_1=1$ and $X_2=0$ otherwise. Formally, the logical constraint can be given as

$$\left(1-{\tilde{x}}_1\right){\tilde{x}}_2+{\tilde{x}}_1{\displaystyle \prod _{m=1}^M}\left({\tilde{x}}_2-m\right)=0(3.3)$$

Consider all the possible data patterns, including when a value is missing (indicated by “ $-$ ”):

• $\left(x_1\mathrm{,}{x}_2\right)=\left(-\mathrm{,}{x}_2\right):$ The value ${\tilde{x}}_1$ can be deduced provided admissible $x_2,$ i.e., $x_2$ is either 0 or satisfies (3.2), otherwise the situation turns into case $\left(x_1\mathrm{,}{x}_2\right)=\left(-\mathrm{,}-\right)$ below.

• $\left(x_1\mathrm{,}{x}_2\right)=\left(x_1\mathrm{,}-\right):$ If $x_1=0$ then ${\tilde{x}}_2=0;$ if $x_1=1$ then (3.3) reduces to (3.2) above.

• $\left(x_1\mathrm{,}{x}_2\right)=\left(-\mathrm{,}-\right):$ Both values need to be imputed by values that satisfy (3.3).

• $\left(x_1\mathrm{,}{x}_2\right):$ Violation of (3.3) is e.g., the case if $\left(x_1\mathrm{,}{x}_2\right)=\left(\mathrm{1,0}\right)$ or if $x_1=0$ and $x_2>0.$ We have case $\left(-\mathrm{,}{x}_2\right)$ above if $x_2$ is fixed, $\left(x_1\mathrm{,}-\right)$ if $x_1$ is fixed, or $\left(-\mathrm{,}-\right)$ if neither is fixed.

To summarize, the constraints (3.2) and (3.3) can not be handled by the minimum adjustment approach with linear constraints considered before. Instead, they need to taken care of by the imputation method. Often, donor-based imputation (e.g., Statistics Canada’s CANCEIS software that implements the Nearest Neighbour Imputation Methodology, NIM) can be designed to impute categorical data such that user specified constraints are satisfied, see e.g., Bankier, Lachance and Poirier (2000).

3.3 Adjustment of donor-based unit imputation

In donor-based unit imputation the whole record of values are taken from the chosen donor. This has advantages over joint modelling of all the target variables if there are many of them. Chen and Shao (2000) establish the consistency of survey estimator based on nearest neighbour imputation (NNI) under mild conditions. The key assumption is that the difference in the conditional expectations of any target variable between a donor and a receptor, given the variables on which the distance metric is calculated, is bounded by the ”distance” between them. That is, they have the same expectations for all the statistical variables if the “distance” between them is zero.

There is thus a need for adjusting donor-based unit imputation when the “distance” between the receptor and the donor is not zero. To illustrate with the example record in Table 1.1, suppose Turnover $\left(x_5\right)$ is always known from the administrative source and is used for donor identification, so that partial imputation under response pattern (I) becomes unit imputation. Since Turnover of the receptor differs from that of the donor, the distance between them is not zero, and it seems natural that the donor values should be adjusted to take this difference into account. Indeed, now that there are constraints involving Turnover, adjustments are necessary in any case.

Let $x$ contain the variables that may be missing. Let $z$ contain the known variables that are used for donor identification. Let $x^{\ast }={\left(x^T,z^T\right)}^T$ be the combined vector of variables. Unit imputation (giving $x_0)$ can be regarded as partial imputation of the missing sub-vector $x$ of $x^{\ast }.$ The need for adjustment of unit imputation may arise if there are edit-rules that involve both values of $x$ and $z,$ and/or if the $z-$values do not match exactly between the donor and receptor. Indeed, unit imputation without adjustment may rather be considered exceptional in practice.

3.4 Macro-level benchmarking in addition to micro-level constraints

A business census requires imputation and editing in order to arrive at a complete dataset for statistical production. Or, a statistical register may be constructed based on a combination of administrative data and one or several sample surveys. Editing and imputation are again necessary. A common feature is that, unlike survey sampling, no case weighting is needed.

When processing such data, macro-level benchmark constraints are frequently imposed due to concerns for statistical efficiency and/or macro-level consistency with external sources. A benchmark constraint is satisfied if the complete data add up to the given benchmark total, which may refer to different aggregation levels, i.e., containing both population and sub-population totals. For instance, certain key national totals may be estimated by some suitable method and imposed as benchmark constraints afterwards. Or, a set of domain-level benchmark constraints may be derived by some small area estimation technique. Also benchmark constraints from external sources are common in structural business statistics - an example from the Norwegian Agriculture Census 2010 will be described in Section 4.

Methods for imputation under benchmark constraints have been studied by Beaumont (2005), Chambers and Ren (2004), Zhang (2009) and Pannekoek, Shlomo and de Waal (2013). The approach taken here is similar to the one taken in the first two papers. In both these papers a weighted least squares distance between initial imputed values (or outlying values in the case of Chambers and Ren 2004) and adjusted imputed values is minimized subject to the constraint that sample-weighted totals based on the adjusted data are equal to the benchmark totals. Here, we assume that some suitable imputation method has been applied to yield the initial complete population dataset, which may or may not be benchmarked. The inconsistency problem on the micro-level implies that adjustments of the initial complete data set will be necessary in general.

Denote by $X$ the complete dataset of interest, where each row corresponds to a unit-level record as the one in Table 1.1, and each column corresponds to a particular variable. Let $X_0$ be the initial complete dataset after imputation and $\tilde{X}$ the adjusted dataset. Each benchmark constraint applies to a particular column vector of $X$ and over the units that fall under its domain. That is, it can be expressed generically as $r^T\text{col}\left(X\right)=t,$ where $\text{col}\left(X\right)$ is the column vector of concern, and $r$ is the indicator vector for whether a unit belongs to the domain of concern, and $t$ the benchmark total. In this way all the benchmark constraints may be summarized as

$${\left[r\right]}^T\left[\text{col}\left(X\right)\right]=t(3.4)$$

where each column of $\left[\text{col}\left(X\right)\right]$ corresponds to a benchmark constraint, and each column of $\left[r\right]$ the corresponding indicator vector, and $t$ the vector of all the benchmark totals. Notice the similarity between (3.4) and (1.1). A minimum adjustment approach follows on specifying the adjustable and fixed values and the distance (or discrepancy) function.

Both the benchmark constraints and the micro-level constraints can be seen as linear constraints on the very long vector containing all elements of $X,\text{ vec}\left(X\right),$ say. Conceptually, all constraints together can therefore be expressed in the form (1.1). The restriction matrix of this formulation is, however, huge and very sparse. The rows corresponding to the micro-level constraints contain possibly non-zero values corresponding to the values in the record they apply to and zeros for all other values of $\text{vec}\left(X\right)$ and the rows corresponding to the benchmark constraints contain non-zero elements only corresponding to the values in $\text{vec}\left(X\right)$ that contribute to that benchmark total. In practice, the optimization problem generated by (3.4) in addition to the micro-level constraints can be handled using the SPA, i.e., one constraint at a time and operating only on the elements of $\text{vec}\left(X\right)$ corresponding to the non-zero elements in that constraint, without actually forming this huge and sparse constraint matrix. For the benchmark constraints we only need to process the columns of $\left[\text{col}\left(X\right)\right]$ one by one and for the micro-level constraints we process each unit-level record one at a time. These iterative minimum adjustments along the columns and rows of $X$ resemble the iterative proportional fitting (or raking) algorithm for fitting log-linear models to contingency table data and for adjusting (contingency) tables to new margins, which is formally identical to a SPA with the KL-divergence and equality constraints only.

Previous | Next