Reducing the response imbalance: Is the accuracy of the survey estimates improved? Section 9. Empirical testingReducing the response imbalance: Is the accuracy of the survey estimates improved? Section 9. Empirical testing

• Previous
• Next

Results 1 and 2 give the basis for testing empirically in this section how mean and variance of the deviation ${\widehat{Y}}_{CAL}-{\widehat{Y}}_{FUL}=\widehat{N}{\Delta }_r=\widehat{N}{\left(b_r-b_s\right)}^{\prime }{\overline{x}}_s$ depend on the imbalance $IMB.$ Both results state that the variance of ${\Delta }_r$ increases in a roughly linear fashion as $IMB$ increases, without being small even if $IMB$  is near zero.

We use real data from an Estonian survey with 17,540 households. The following variables are available for every household: Household net income, used here as the study variable $y,$ and three categorical variables referring to the designated head of household, used here as auxiliary variables: (i) Gender (1 for male, 0 for female), (ii) Economic activity (1 for employed, 0 for not employed) and (iii) Education, with three exhaustive levels: low, medium, high.

We compute the mean $\overline{\Delta }$ of ${\Delta }_r$ and the variance $S_{\Delta }^2$ of ${\Delta }_r$ by averaging over the sets r with fixed mean ${\overline{x}}_r,$ given $s.$

9.1 Test situation 1

In line with Result 1, we want to consider the response sets $r$ with fixed size $m$ arising from a given sample $s$ of size $n.$ The computational volume is prohibitive even for rather small $n.$ We drew $s$ as a simple random sample of size $n=20$ from 17,540. The $d_k$ are then constant. The sample mean for the $y‑$variable (household income) was ${\overline{y}}_s=10,\mathrm{386.65.}$ We define $x_k$ as the group vector of dimension $J=3$ that identifies the three exhaustive levels of Education; low, medium, high. For the realized sample $s,$ we have $n{\overline{x}}_s={\left(5,8,7\right)}^{\prime }.$

We fixed the size of the response sets $r$ to be $m=12.$ The response rate is 60 per cent for every one of the $\left(\begin{array}{c}20\\ 12\end{array}\right)\approx 1.26\times {10}^5$ possible response sets $r.$ From these, we excluded all those for which the response count vector $m{\overline{x}}_r$ contained a zero, to avoid a singular ${\Sigma }_r.$ This left 31 configurations $\left(m_1,m_2,m_3\right)$ such that $m_1+m_2+m_3=12$ and all three counts $m_j\ge 1.$ For each of the 31 possibilities, we computed $\overline{\Delta }$ and $S_{\Delta }^2$ by averaging over the response sets $r$ satisfying the fixed configuration. For example, $\left(m_1,m_2,m_3\right)=\left(3,4,5\right)$ is satisfied by 14,700 response sets $r,$ so mean and variance of ${\Delta }_r$ are computed over those. Other configurations give much fewer response sets, for example, only 70 for the configuration (3, 8, 1); a few of those can then be very influential in the computations. For every one of the 31 cases, $\overline{\Delta }$ is theoretically zero, by Result 1. The computations confirmed this; a plot of $\overline{\Delta }$ against $IMB$ is unnecessary. Figure 9.1 shows the 31 point plot of $S_{\Delta }^2$ against $IMB.$ Because of the non-uniqueness of $IMB$ noted earlier, it happens several times that more than one $S_{\Delta }^2$ occurs at the same $IMB$ value. Figure 9.1 shows that $S_{\Delta }^2$ has a clear upward trend as $IMB$ increases. Figure 9.1 also shows the approximation $S_{\Delta }^{\text{2}}\approx S_{\Delta approx}^2=\left(S_y^2/m\right)\left(1-p+IMB/p^2\right)$ from Result 1. We have $p=0.6,m=12$ and $S_y^2=26.3\times {10}^6,$ so the computed approximation, linear in $IMB,$ is $S_{\Delta approx}^{\text{2}}=a+bIMB$ with $a=0.879\times {10}^6$ and $b=6.102\times {10}^6.$ For points with low $IMB,$ $S_{\Delta }^2$ agrees closely with the linearly increasing $S_{\Delta approx}^{\text{2}}.$ A contributing reason is that when $IMB$ is low, the group response rates $p_j$ vary little, and this is one of the conditions for close approximation, as the derivation of Result 1 in Appendix 1 explains. For higher $IMB$ values, the increasing trend in $S_{\Delta }^2$ is still evident, but the scatter around the theoretical line is more pronounced. Five outlying points in Figure 9.1 have very large $S_{\Delta }^2;$ three of them occur when one component of $\left(m_1,m_2,m_3\right)$ is equal to the maximal count (5 or 8 or 7). For those, less accurate linear approximation is expected, the $p_j$ being far from equal.

9.2 Test situation 2

The setup and the computational steps are similar to those in Test situation 1, but $x_k$ is no longer a group vector; some results change considerably, compared with Test situation 1.

A new simple random sample $s$ of size $n=20$ was drawn from the 17,540 households. For this sample, ${\overline{y}}_s=\text{9,618}\text{.4}.$ We let $x_k$ incorporate all three auxiliary variables (i), (ii) and (iii), but not completely crossed: Gender (univariate coded 0 or 1), Economic activity (univariate coded 0 or 1) and Education level (three exhaustive categories coded (1,0,0) or (0,1,0) or (0,0,1)). This $x_k$ is not a group vector; it has dimension $1+1+3=5$ and $2\times 2\times 3=12$ possible values; ${\Sigma }_r$ and ${\Sigma }_s$ are not diagonal. We have $n{\overline{x}}_s={\left(9,11,4,7,9\right)}^{\prime }.$ For this sample $s$ we considered the response sets $r$ of fixed size $m=12$ excepting those where one or more of the five components of the count vector $m{\overline{x}}_r$ are zero. This left 658 different vectors $m{\overline{x}}_r,$ each composed of five non-zero counts, and satisfied by a certain number of response sets $r$ over which we computed, by simple averaging, the mean $\overline{\Delta }$ and variance $S_{\Delta }^{\text{2}}.$ These are thus moments conditionally on  ${\overline{x}}_r.$

Figure 9.2 shows the 658 point plot of $\overline{\Delta }$  against $IMB.$  In Test situation 1, $\overline{\Delta }$  was zero for every point because $x_k$  was a group vector. This is not so in Figure 9.2, where the means $\overline{\Delta }$  fan out when $IMB$  increases. They are much more concentrated around zero for low $IMB$  than for large $IMB.$  Several points (that is several means ${\overline{x}}_r)$  can give the same or nearly the same $IMB.$  Figure 9.2 shows that in a small neighborhood of a fixed value $IM{B}_0$  on the $IMB$  axis, the mean of the means $\overline{\Delta }$  is roughly zero. With reference to Result 2, we can expect to see the average of $\overline{\Delta }$  for fixed $IMB$  to be near zero: Under model (8.1) for $y_k,$  Result 2 says that $E_{\xi }\left({\Delta }_r|X,r,s\right)=0.$  When $X$  and $r$  are fixed, so is $IMB.$  If the model is a reasonably good representation, the average of ${\Delta }_r$  for fixed $IMB$  should be close to zero, as Figure 9.2 indicates.

Figure 9.3 shows the plot of the conditional variance $S_{\Delta }^2$ against $IMB.$ The pattern with a variance $S_{\Delta }^2$ that increases linearly in $IMB$ prevails, even though $x_k$ is not a group vector here. Figure 9.3 shows the computed approximating line $S_{\Delta approx}^2=\left({\widehat{\sigma }}_{\epsilon }^2/m\right)\left(1-p+IMB/p^2\right)$ derived from Result 2, with ${\widehat{\sigma }}_{\epsilon }^2={\displaystyle {\sum }_s{\left(y_k-x_k^{\prime }{b}_s\right)}^2}/\left(n-J\right)$ used to estimate ${\sigma }_{\epsilon }^2.$ We have $J=5,p=0.6,m=12$ and ${\widehat{\sigma }}_{\epsilon }^2=33.6\times {10}^6,$ so the line in Figure 9.3 is $S_{\Delta approx}^{\text{2}}\text{}=a+bIMB$ with $a=1.12\times {10}^6$ and $b=7.78\times {10}^6.$ The linear approximation holds particularly well for small $IMB,$ say less than 0.1. For large $IMB,$ there is much scatter; $S_{\Delta }^2$ has some very large values, and some very low values as well. Figure 9.4 shows the joint behavior of $\overline{\Delta }$ and $S_{\Delta }^2$ for the 658 points. The size of a dot is proportional to $IM{B}^2;$ the reason for squaring is to better contrast larger and smaller $IMB$ values. Response sets $r$ with small $IMB$ are found to give small $\overline{\Delta }$ and $S_{\Delta }^2,$ a favourable sign because the CAL and FUL estimators are then close. To illustrate, for points satisfying $IMB\le 0.1,$ $\overline{\Delta }$ is in the interval (-1,390; 1,447) and $S_{\Delta }^2$ in (0.846×10⁶; 4.86×10⁶). These are narrow intervals; this is even more pronounced for $IMB\le \mathrm{0.05.}$ But when $IMB$ is large, this advantageous situation no longer holds. For example, $\overline{\Delta }$ can be very small and at the same time $S_{\Delta }^2$ very large (points in the middle and right side of the figure). On the other hand, $S_{\Delta }^2$ can be near zero while $\overline{\Delta }$ is very large in absolute value (points in the top and bottom left parts of the figure.) Test situation 2 illustrates that a non-group vector $x_k$ can give both a distinctly non-zero mean of $\Delta {}_r$ and a high variance of ${\Delta }_r,$ and that these tendencies are accentuated by large imbalance.

• Previous
• Next