An assessment of accuracy improvement by adaptive survey design
Section 4. The group vector case

• Previous
• Next

4.1  The terms of the decomposition

We consider the important case of a group vector, $x_k={\left(0,\dots ,1,\dots ,0\right)}^{\prime }\text{}.$ In practice, its dimension $J$ is often considerable, 30 or more, as when several categorical $x$ -variables are fully crossed. The diagonal form of matrices requiring inversion in (2.5) brings considerable simplification. For $j=1,2,\dots ,J,$ denote by $s_j$ the subset of the sample $s$ that falls in group $j.$ Its proportion of the sample is $W_j={\displaystyle {\sum }_{s_j}{d}_k}/{\displaystyle {\sum }_s{d}_k}\text{}.$ The response set in $s_j$ is denoted $r_j;$ the nonresponse set is $n{r}_j=s_j-r_j.$ The group $j$ nonresponse rate is $Q_j={\displaystyle {\sum }_{n{r}_j}{d}_k}/{\displaystyle {\sum }_{s_j}{d}_k}=1-{\displaystyle {\sum }_{r_j}{d}_k}/{\displaystyle {\sum }_{s_j}{d}_k}=1-P_j;$ the overall rate is $Q={\displaystyle {\sum }_{j=1}^J{W}_j{Q}_j}=1-P.$ The imbalance becomes a variance of group response rates, see, for example, Särndal (2011a),

$$\text{IMB}={\displaystyle \sum _{j=1}^J{W}_j{\left(Q_j-Q\right)}^2}={\displaystyle \sum _{j=1}^J{W}_j{\left(P_j-P\right)}^2\text{}.}$$

The $y$ -means are ${\overline{y}}_{r_j}={\displaystyle {\sum }_{r_j}{d}_k{y}_k}/{\displaystyle {\sum }_{r_j}{d}_k;}$ ${\overline{y}}_{n{r}_j}={\displaystyle {\sum }_{n{r}_j}{d}_k{y}_k}/{\displaystyle {\sum }_{n{r}_j}{d}_k};$ ${\overline{y}}_{s_j}={\displaystyle {\sum }_{s_j}{d}_k{y}_k}/{\displaystyle {\sum }_{s_j}{d}_k}.$ The $J­$ dimensional column vector $b_r-b_{nr}$ has elements ${\delta }_j={\overline{y}}_{r_j}-{\overline{y}}_{n{r}_j},$ $j=1,2,\dots ,J.$ We call ${\delta }_j$ the group $j$  divergence, namely, between response $y$ -mean and nonresponse $y$ -mean. The calibration estimator is ${\widehat{Y}}_{\text{CAL}}={\displaystyle {\sum }_{j=1}^J{\widehat{N}}_j}{\overline{y}}_{r_j},$ where ${\widehat{N}}_j={\displaystyle {\sum }_{s_j}{d}_k},$ and the unbiased benchmark estimator is ${\widehat{Y}}_{\text{FUL}}={\displaystyle {\sum }_{j=1}^J{\widehat{N}}_j}{\overline{y}}_{s_j},$ so

$${\Delta }_{\text{CAL}}=\left({\widehat{Y}}_{\text{CAL}}-{\widehat{Y}}_{\text{FUL}}\right)/\widehat{N}={\displaystyle \sum _{j=1}^J{W}_j}\left({\overline{y}}_{r_j}-{\overline{y}}_{s_j}\right)={\displaystyle \sum _{j=1}^J{W}_j}{Q}_j{\delta }_j.$$

Straightforward application of Result 3.1 to the group vector case gives the following Result 4.1.

Result 4.1. When $x$ is a group vector of dimension $J,$ $x_k={\left(0,\dots ,1,\dots ,0\right)}^{\prime },$ the terms of the decomposition in Result 3.1 take the form

$$D_1=Q{\displaystyle \sum _{j=1}^J{W}_j{\delta }_j};D_2={\displaystyle \sum _{j=1}^J{W}_j\left(Q_j-Q\right){\delta }_j};{\Delta }_{\text{CAL}}=D_1+D_2={\displaystyle \sum _{j=1}^J{W}_j{Q}_j{\delta }_j},$$

where ${\delta }_j={\overline{y}}_{r_j}-{\overline{y}}_{n{r}_j}\text{},$ and $Q_j={\displaystyle {\sum }_{n{r}_j}{d}_k}/{\displaystyle {\sum }_{s_j}{d}_k}$ and $Q={\displaystyle {\sum }_{j=1}^J{W}_j{Q}_j}$ are, respectively, the group $j$ nonresponse rate and the overall nonresponse rate.

To what degree can adaptive data collection and better balanced response $r$ improve accuracy and reduce the deviation ${\Delta }_{\text{CAL}}=D_1+D_2?$ In the discussion that follows, “reduction” should be understood as “reduction in absolute value”, “smaller” as “smaller in absolute value”, because ${\Delta }_{\text{CAL}}$ and its components can have either sign.

In an adaptive data collection acting on an $x$ -vector that codes quite many groups, it is hard to arrive at the perfect balance where $Q_j=Q$ for every group $j,$ and IMB = 0. But it may be possible to bring all $Q_j$ in fairly close agreement with the overall nonresponse rate $Q;$ then both IMB and $D_2$ may come close to zero. We should think in terms of a scenario where an actual data collection, obeying a predefined protocol that remains unchanged to the end, is compared with alternative data collections that strive actively, through interventions, to end in a response with low imbalance. This is the format for the empirical validation (in Section 5), based on an important Statistics Sweden survey: There is a response set as actually recorded in the survey, and several alternative response sets are derived by experimental interventions in the actual response to get successively lower imbalance.

We address the following issues about the terms in Result 4.1:

1. When the nonresponse differentials $Q_j-Q$ move closer to zero, so that IMB is reduced, the term $D_2={\displaystyle {\sum }_{j=1}^J{W}_j\left(Q_j-Q\right){\delta }_j}$ tends to be reduced, possibly to near zero. The prospect is quite different for $D_1=Q{\displaystyle {\sum }_{j=1}^J{W}_j{\delta }_j}.$ A priori it is not excluded that the divergences ${\delta }_j$ will be somewhat reduced in the process. If this were to happen in a number of groups, then $D_1$ might also be reduced. But under certain conditions, a great change in $D_1$ is unlikely; we expect $D_1$ to be little affected by a reduced IMB. We justify this first by a model assisted theoretical argument in Section 4.6, then observe it empirically with actual survey data in Section 5.
2. It is not evident that $D_1$ and $D_2$ always have the same sign. When they do, an obtainable reduction of $D_2$ brings a reduced deviation ${\Delta }_{\text{CAL}}=D_1+D_2,$ thus an improved accuracy. Under what conditions do they have the same sign?
3. When $D_1$ and $D_2$ have the same sign and $D_1$ stays roughly constant when IMB is reduced, then any reduction of ${\Delta }_{\text{CAL}}=D_1+D_2$ comes from a reduction of $D_2;$ the relative size of the two terms then determines whether such gain by adaptive design is considerable or just modest or even trivial.

4.2  Within-group correlation between response and survey variable

Response and survey variable $y$ are correlated, in the whole sample $s,$ as within any sample group $s_j.$ This unavoidable consequence of “missing not at random” is a key to interpreting $D_1,D_2$ and their sum ${\Delta }_{\text{CAL}}.$ In group $j,$ let ${\rho }_j$ be the coefficient of correlation between response indicator $i(i_k=1$ for $k\in r_j,$ $i_k=0$ for $k\in n{r}_j)$ and survey variable $y.$ This ${\rho }_j$ has a simple relation, through a multiplicative factor, to the group divergence ${\delta }_j={\overline{y}}_{r_j}-{\overline{y}}_{n{r}_j}\text{}:$

$${\rho }_j=\sqrt{P_j\left(1-P_j\right)}{\delta }_j/S_{yj}\text{},(4.1)$$

where $S_{yj}^2={\left({\displaystyle {\sum }_{s_j}{d}_k}\right)}^{-1}{\displaystyle {\sum }_{s_j}{d}_k}{\left(y_k-{\overline{y}}_{s_j}\right)}^2$ is the $y$ -variance in group $j.$ The expression (4.1) follows (see Appendix, part 2) from the customary definition of correlation coefficient as “covariance divided by the product of the two standard deviations”, ${\rho }_j=S_{iyj}/\left(S_{ij}{S}_{yj}\right).$

In the special case where $y$ is dichotomous $0/1$ (as when $y_k=1$ if $k$ is “Employed”, $y_k=0$ otherwise), then ${\overline{y}}_{s_j}={\displaystyle {\sum }_{s_j}{d}_k}{y}_k/{\displaystyle {\sum }_{s_j}{d}_k}$ is a proportion, namely, the (design weighted) proportion of $“1”$ in group $j$ (the employment rate in group $j),$ so the within-group $y$ -variance is $S_{yj}^2={\overline{y}}_{s_j}\left(1-{\overline{y}}_{s_j}\right).$

4.3  Analysis of the resisting term

We now further examine the behavior of $D_1$ and $D_2$ for the group vector case in Result 4.1. Their (relative) sizes depend on several factors. The resisting term $D_1=Q{\displaystyle {\sum }_{j=1}^J{W}_j{\delta }_j}$ depends on the sizes and the distribution over the $J$ groups of the divergences ${\delta }_j={\overline{y}}_{r_j}-{\overline{y}}_{n{r}_j},$ or, in an alternative view, of the within-group correlations ${\rho }_j=\sqrt{P_j\left(1-P_j\right)}{\delta }_j/S_{yj}\text{}.$

In a typical survey setting, a look at the distribution of ${\delta }_j,$ $j=1,\dots ,J,$ where $J$ may be 30 or more, will typically reveal a mixture of positive and negative values. There may be “critical groups” with uncharacteristically large divergence ${\delta }_j,$ of the same sign, be it positive or negative. Such ${\delta }_j$ are influential for the (weighted) mean $\overline{\delta }={\displaystyle {\sum }_{j=1}^J{W}_j{\delta }_j}$ and therefore for $D_1=Q\overline{\delta }.$ The majority of the ${\delta }_j$ may be quite small, although non-zero.

Example 4.1. Consider $y=$ Employed $(y_k=1$ if unit $k$ is employed, $y_k=0$ if not). Positive within-group correlation ${\rho }_j$ between Employed and Response $(i_k=1$ for $k\in r_j,$ $i_k=0$ for $k\in n{r}_j)$ may be expected for groups of persons with one or more of characteristics such as male, young, low education and/or foreign origin. Such groups tend to be low (comparatively speaking) both on Response and on Employed. Alternatively expressed, employment rate is higher among respondents than among non-respondents, thus ${\delta }_j$ positive. On the other hand, groups with characteristics such as middle aged, well educated and/or home owner tend to be high on both Response and Employed, so ${\delta }_j$ is likely to be positive for those groups as well. Another $y$ -variable which may have a similar pattern is Income: In some sample groups, Income may be higher for respondents than for non-respondents, so ${\delta }_j$ again positive. Thus for both $y$-variables, several distinctly positive and influential ${\delta }_j$ may make $\overline{\delta }={\displaystyle {\sum }_{j=1}^J{W}_j{\delta }_j}$ and $D_1=Q\overline{\delta }$ positive. This is confirmed in Section 5 for Swedish Labour Force Survey data. However in other countries, a variable such as Income may behave differently: In some groups, income may instead be higher for non-respondents than for respondents, making ${\delta }_j$ negative, as when wealthy or high income persons tend to respond relatively less. This illustrates the difficulty in anticipating the sign of the group divergences for some survey variables.

4.4  Analysis of the reducible term

We can write the reducible term as $D_2={\displaystyle {\sum }_{j=1}^J{W}_j\left(Q_j-Q\right)\left({\delta }_j-\overline{\delta }\right)}.$ It is the covariance between nonresponse $Q_j=1-P_j$ and divergence ${\delta }_j={\overline{y}}_{r_j}-{\overline{y}}_{n{r}_j}.$ Here $D_2$ is a covariance across groups, $j=1,\dots ,J,$ whereas the divergence ${\delta }_j$ given in (4.1) reflects covariance within group $j$ between response indicator $i_k$ and survey variable $y_k.$ It is hard to draw general conclusions about the size of $D_2$ and whether or not it has the same sign as $D_1.$ It depends on a number of factors.

Example 4.1, continued. As argued in Example 4.1, a number of positive divergences ${\delta }_j$ are likely to occur for $y=$ Employed $(y_k=1$ if $k$ is Employed, = 0 if not), for example, for groups characterized by male, young and/or low education. Then $D_1=Q\overline{\delta }=Q{\displaystyle {\sum }_{j=1}^J{W}_j{\delta }_j}$ may be distinctly positive. But whether high nonresponse rate $Q_j$ tends to go together with high divergence ${\delta }_j,$ so as to make the covariance $D_2$ also positive, this is less evident. It may happen for variables such as Employed and Income; it is so for the Swedish data in Section 5.

4.5  Do the two terms have same sign?

General conclusions about the signs of $D_1$ and $D_2$ do not seem possible. Too many factors are involved, the population, the survey design, the particular survey variable $y,$ and others. When $D_1$ and $D_2$ have the same sign, a reduction of $D_2$ toward zero can give a (perhaps considerably) reduced deviation ${\Delta }_{\text{CAL}}=D_1+D_2.$ On the other hand, if the terms are of opposite sign, they counteract each other; striving for low imbalance and low $D_2$ may not reduce ${\Delta }_{\text{CAL}}=D_1+D_2.$ In this undesirable and perhaps rare situation, attempts at balancing the response $–$ reducing $D_2-$ would defeat the purpose of achieving a lower deviation ${\Delta }_{\text{CAL}};$ it may get larger, not smaller. Let us examine this “same sign issue”.

Example 4.1 and its continuation in Section 4.4 outlined a situation, for the $y$ -variables Employed and Income, where the mean $\overline{\delta }={\displaystyle {\sum }_{j=1}^J{W}_j{\delta }_j}$ and the covariance ${\displaystyle {\sum }_{j=1}^J{W}_j\left(Q_j-Q\right){\delta }_j}=D_2$ are likely to have the same sign, so that $D_1=Q\overline{\delta }$ and $D_2$ have the same sign. But more generally, opposite signs are not excluded. An example is when uncharacteristically low nonresponse, $Q_j-Q<0,$ happens in influential groups with distinctly positive divergence ${\delta }_j.$ This may cause $D_2$ to be negative, while $D_1$ is positive. Then the two terms counteract each other. A positive prospect remains, however, when the two terms have the same sign: We can reduce $D_2$ to low levels by making all group nonresponse rates nearly $Q_j$ equal. Then, supposing that $D_1$ changes little, as under conditions suggested below in Section 4.6, a considerable reduction of the deviation ${\Delta }_{\text{CAL}}$ can take place. These findings are summarized as follows.

Proposition 4.1. For the group vector case, suppose that the mean divergence $\overline{\delta }={\displaystyle {\sum }_{j=1}^J{W}_j{\delta }_j}$ and the covariance $D_2={\displaystyle {\sum }_{j=1}^J{W}_j\left(Q_j-Q\right){\delta }_j}$ have the same sign, say positive. Then $D_1=Q\overline{\delta }$ and $D_2$ are positive terms in the deviation ${\Delta }_{\text{CAL}}=D_1+D_2={\displaystyle {\sum }_{j=1}^J{W}_j{Q}_j{\delta }_j}.$ An adaptive data collection that reduces the imbalance $\text{IMB}={\displaystyle {\sum }_{j=1}^J{W}_j{\left(Q_j-Q\right)}^2}$ will reduce $D_2$ (to zero if IMB = 0), whereas the resisting term $D_1$ is likely to change little.

The choice of $x$ -vector, that is, the choice of variables entering into it and its dimension, can have important effects on $D_1$ and $D_2.$ Suppose we double the number of groups coded by the group vector, as when $J=8=2^3$ already existing groups (obtained by crossing three dichotomous $x$ -variables) are doubled in number to $J=16=2^4,$ by complete crossing on a fourth dichotomous $x$ -variable. A certain skewness in the distribution of the divergences ${\delta }_j={\overline{y}}_{r_j}-{\overline{y}}_{n{r}_j},$ $j=1,\dots ,J,$ is likely to persist. For a fixed response $r,$ $D_1$ and $D_2$ may be reduced by extending the $x$ -vector. The empirical evidence in Section 5 throws some light on this question.

4.6  Insensitivity of group divergences

For the group vector case in Result 4.1, the resisting term is $D_1=Q\overline{\delta },$ where $\overline{\delta }={\displaystyle {\sum }_{j=1}^J{W}_j{\delta }_j}$ is a weighted mean of the divergences ${\delta }_j,$ $j=1,\dots ,J.$ We would like to know more about how the ${\delta }_j$ evolve in a data collection that may extend, in a large survey, over a period of time.

An adaptive data collection is monitored during the period. Assessment and modification of protocol and emphasis may take place, at one or more intervention points. For example, units whose computed response propensity is seen to be low, at a given point, may become henceforth specially targeted, so as to boost their response propensity towards an overall mean propensity. Some so far non-responding units become converted into respondents; their response status is changing.

In the group vector case, the sample group $s_j$ consists, at a given point in the data collection period, of a certain response set and a corresponding nonresponse set. Contact attempts continue with units having not yet answered; a change in response status occurs for some units in $s_j.$ A few more become respondents, causing a certain change in the divergence ${\delta }_j.$ Under certain conditions, however, ${\delta }_j$ may be little affected. If this were to hold for most or all groups, $D_1$ would change little. Let us examine this possibility.

That ${\delta }_j$ would change little only cannot be observed in a real survey, because $y_k$ is missing for the nonresponse, but it can be made plausible by a model for response status change. We consider models where units within one and the same group are exchangeable or substitutable for one another, with respect to response status: At a given point in the data collection, all units are considered alike in regard to response status change; transfer from nonresponse into response, or vice versa, is equally probable for all units.

Such a model can be justified for a sufficiently detailed breakdown of the sample $s$ into small and homogenous groups $s_j,$ such that the units in one and the same group share important characteristics. It would be unrealistic in a large heterogeneous group with a mixture of units very different in regard to age, gender, income, education and other important aspects. In a heterogeneous group, some units have high conversion probability, others are unlikely to become converted. For example, if a group contains both older, well-educated persons and younger, less educated persons, conversion probability may be high for the former subgroup, considerably lower for the latter.

At a certain point in the data collection, let us consider two models for response status change in a sample group $s_j.$ We assume for simplicity that $s$ is a simple random sample from $U,$ of size $N,$ so the sampling weights are constant, $d_k=N/n$ for all $k.$

Model 4.1. Suppose transfer takes place, in the group $s_j,$ from the nonresponse set $n{r}_j=s_j-r_j$ to the response set $r_j$ (a conversion) in such a way that all transfer sets $t{r}_j\subset n{r}_j$ of fixed size $q_j$ are equally likely to occur.

In this model, $n{r}_j$ and $r_j$ are fixed sets with respective sizes $n_j-m_j$ and $m_j,$ while $t{r}_j$ is a simple random selection of $q_j$ non-respondents. Any particular transfer set $t{r}_j$ of $q_j$ units can get converted, and with equal probability. Every unit in the nonresponse set $n{r}_j$ has the same conversion probability, $q_j/\left(n_j-m_j\right).$

Before transfer, the divergence in group $j$ is ${\delta }_j={\overline{y}}_{r_j}-{\overline{y}}_{n{r}_j}.$ After transfer, indicated by a star $*$ in the indices, the new divergence is ${\delta }_{*j}={\overline{y}}_{*r_j}-{\overline{y}}_{*n{r}_j}.$ Its expected value under Model 1 satisfies

$$P_{*j}E\left({\delta }_{*j}\right)=P_j{\delta }_j\text{},(4.2)$$

where the new group response rate is $P_{*j}=\left(m_j+q_j\right)/n_j>m_j/n_j=P_j.$ The proof is given in the Appendix, part 3. If the group response rate change is small, as when $q_j$ is small compared with $n_j,$ then $P_{*j}\approx P_j,$ and the transfer may bring little change to the divergence. Its new value ${\delta }_{*j}$ is just marginally smaller, in expectation, than the before-transfer value ${\delta }_j.$

In the empirical Section 5, we experiment with response sets that get smaller instead of larger. Model 4.2 deals with a transfer in that direction, and leads to a similar conclusion.

Model 4.2. Suppose a transfer takes place within group $j$ from the response set $r_j$ to the nonresponse set $n{r}_j=s_j-r_j$ in such a way that all transfer sets $t{r}_j\subset r_j$ of fixed size $q_j$ are equally likely to occur.

A derivation analogous to the one that gave (4.2) shows that

$$Q_{*j}E\left({\delta }_{*j}\right)=Q_j{\delta }_j.(4.3)$$

The new (higher) nonresponse rate is $Q_{*j}=1-\left(m_j-q_j\right)/n_j>1-m_j/n_j=Q_j.$ The new divergence ${\delta }_{*j}$ is just slightly smaller in expected value, when $q_j$ is small compared with $n_j.$ The proof is analogous to that of (4.2).

The results (4.2) and (4.3) state that if it does not matter which particular units in group $j$ change their response status, then the divergence ${\delta }_j$ stays nearly the same, in expectation. But the assumption in Models 4.1 and 4.2 of equally probable transfer sets is hard or impossible to substantiate in practice. It would be hard to specify a vector $x_k={\left(0,\dots ,1,\dots ,0\right)}^{\prime }$ such that the groups coded by the vector will obey the models. We cannot claim “truth” of these models, only suggest that they may be plausible under a well-diversified grouping of the sample.

• Previous
• Article main page
• Next