Reducing the response imbalance: Is the accuracy of the survey estimates improved? Section 10. DiscussionReducing the response imbalance: Is the accuracy of the survey estimates improved? Section 10. Discussion

We comment on several issues arising and indicate limitations of our study.

1. Choice of variables for the auxiliary vector. The auxiliary variables for the vector $x$ is treated as a fixed choice in this article. That choice is important when a perhaps large supply of such variables is available. Which ones should be chosen to meet the ultimate objective, which is best possible accuracy in the estimates? Result 1 shows that in the group vector case two factors are important for $S_{Δ}^{2}$ (which determines the conditional variance of CAL): The response imbalance $I M B$ and the variance $S_{y}^{2}$ of the survey variable $y .$ The fact that $S_{Δ}^{2}$ is (approximately) linearly decreasing with $I M B$ gives incentive to try to reduce $I M B$ in data collection. But allowing more variables in $x$ increases $I M B$ (because agreement is sought on more $x ‑$ means). As for the $y ‑$ variance $S_{y}^{2},$ the trend is the opposite. By (7.1), $S_{y}^{2}$ is an averaged residual variance around group means; allowing additional variables in $x$ will, especially if they explain $y$ well, reduce $S_{y}^{2} .$ The two factors work in opposite directions: More auxiliary variables give greater $I M B$ but lower $y ‑$ variance. It suggests a possible trade-off, a question not examined in this article. A particularity of a group vector $x$ plays a role: When more categorical variables enter, the vector dimension grows in multiplicative bounds. The risk of small or empty cells restricts the expansion. To illustrate, if $x = (s e x \times e d u c a t i o n \times a g e)$ of dimension $J = 2 \times 3 \times 4 = 24$ is expanded to also include occupation with 4 categories, in completely crossed fashion, the new dimension (equal to the new number of groups) is $J = 24 \times 4 = 96.$ In principle, $S_{y}^{2}$ decreases, but risk of small cells is a good reason to abstain from completely crossing all the variables and instead involve them in a non-group $x ‑$ vector. That case is addressed in Result 2, which says that if $x$ explains $y$ well, then $σ_{ε}^{2}$ is small and will give a desired low variance for $Δ_{r} .$

2. Auxiliary information at different levels. In this article, the imbalance $I M B$ and the calibration estimator ${\hat{Y}}_{C A L}$ use the same $x ‑$ vector, and more particularly one that has auxiliary data for the sample units only. It is a realistic case. But in more general formulations, the data collection would use a monitoring vector $x_{M V}$ possibly different from the calibration vector $x_{C A L}$ used later in the estimation. The first is an instrument to get low imbalance $I M B$ in the response, the second serves to get good calibrated weights for ${\hat{Y}}_{C A L} .$ One reason why $x_{M V}$ and $x_{C A L}$ may differ in practice is that auxiliary variables for the estimation may be updated versions of the same variables available in the data collection. There may be other reasons to choose $x_{M V}$ and $x_{C A L}$ to be different. Also, they can contain information (if available) at the population level. Extensions of our approach to such situations are possible.

3. Uncertain benefit from reduced imbalance. Schouten et al. (2014) find evidence that balancing response reduces bias. We also find that there is incentive to strive, in data collection, for an ultimate response set with low imbalance $I M B .$ As Results 1 and 2 show theoretically, and as test situations 1 and 2 confirm empirically, low imbalance gives a deviation ${\hat{Y}}_{C A L} - {\hat{Y}}_{F U L} = \hat{N} Δ_{r}$ with zero or almost zero expected value and a small variance. This is our protection against large bias. If $I M B$ were to increase, the variance tends to increase. The zero expected value of the deviation ${\hat{Y}}_{C A L} - {\hat{Y}}_{F U L}$ is an average property. There is no guarantee that the deviation is small for any particular response $r$ with low $I M B .$

4. Perfect balance does not eliminate the bias. Zero imbalance, $I M B = 0,$ implies an equality of means for response and full sample, ${\bar{x}}_{r} = {\bar{x}}_{s} .$ If that perfect balance were achieved, the bias adjustment term in (5.2) would be zero; the calibration (CAL) estimator and the expansion (EXP) estimator are identically equal. One can say that if perfect balance is achieved, the power of the auxiliary vector is exhausted, not in its potential for explaining the study variable, but in its potential for distancing itself from the crude EXP estimator, which, although it uses no auxiliary information at all, is as good as the otherwise better choice CAL. However, $CAL \equiv EXP$ is still not ideal. As Result 1 shows, the variance of the CAL deviation is not near zero even if the imbalance $I M B$ is near zero. Perfect balance does not eliminate the deviation of CAL, but small $I M B$ protects against large deviation.

5. Practical implications. In this article we have primarily in mind surveys with a “substantial and non-eradicable nonresponse” that cannot realistically (under time and budget constraints for the survey) be brought to single-digit per cent levels even if large resources are spent. Surveys with 30 per cent or more nonresponse are common today. This is far from an ideal with near 100 per cent response, where imbalance and nonresponse would essentially cease to be issues; the EXP, CAL and FUL estimators would be close.

6. Directions for generalization. Results 1 and 2 show properties of the CAL deviation among response sets under a given formulation of the auxiliary vector. It would be desirable to generalize the results to other situations. Our proofs assume the existence of certain inverse matrices. Extensions to other cases would be possible with the aid of Moore-Penrose generalized inverse.

Acknowledgements

This work was supported by the Estonian Science Foundation grant 9127 and by the Institutional Research Funding IUT34-5 of Estonia. The authors gratefully acknowledge constructive comments from an Associate Editor and a Referee, both anonymous.

Appendix 1

Derivation of Result 1

We derive (7.2) to (7.4) under the conditions and notation in Section 7. The sample $s$ is self-weighting, of size $n,$ and $x$ is a group vector of dimension $J .$ We assume probability ${(\begin{matrix} n \\ m \end{matrix})}^{- 1}$ for every response set $r$ with fixed size $m .$ We derive the expected value and the variance of $Δ_{r} = {(b_{r} - b_{s})}^{'} {\bar{x}}_{s} = \sum_{j = 1}^{J} W_{j s} {\bar{y}}_{r_{j}} - {\bar{y}}_{s},$ where $W_{j s} = n_{j} / n,$ conditionally on fixed $m$ and mean ${\bar{x}}_{r} = (1 / m) (m_{1}, \dots, m_{j}, \dots, m_{J});$ $\sum_{j = 1}^{J} m_{j} = m .$ Under that conditioning, $R = \prod_{j = 1}^{J} (\begin{matrix} n_{j} \\ m_{j} \end{matrix})$ sets $r$ have the same probability, where $n_{j}$ is the size of sample group $s_{j};$ $\sum_{j = 1}^{J} n_{j} = n .$ This is identical to the probability structure for stratified simple random sampling of $m_{j}$ from $n_{j}$ in stratum $j; j = 1, \dots, J .$ Given $m$ and ${\bar{x}}_{r},$ the expected value and variance of ${\bar{y}}_{r_{j}}$ are, respectively, ${\bar{y}}_{s_{j}} = \sum_{s_{j}} y_{k} / n_{j}$ and $(1 / m_{j} - 1 / n_{j}) S_{y j}^{2}$ with $S_{y j}^{2}$ given in (7.1). Thus $\bar{Δ} = \sum_{j = 1}^{J} W_{j s} {\bar{y}}_{s_{j}} - {\bar{y}}_{s} = 0,$ which proves (7.2), and $S_{Δ}^{2} = \sum_{j = 1}^{J} W_{j s}^{2} (1 / m_{j} - 1 / n_{j}) S_{y j}^{2} .$ Substituting $p_{j} = m_{j} / n_{j}$ and $p = m / n,$ and using $S_{y}^{2} = \sum_{j = 1}^{J} W_{j s} S_{y j}^{2}$ given in (7.1), we get

$S_{Δ}^{2} = \frac{1}{n} \sum_{j = 1}^{J} W_{j s} (\frac{1}{p_{j}} - 1) S_{y j}^{2} = (\frac{1}{m} - \frac{1}{n}) S_{y}^{2} + \frac{1}{m} \sum_{j = 1}^{J} W_{j s} (\frac{p}{p_{j}} - 1) S_{y j}^{2} . (A .1)$

This proves (7.3). To analyze the penalty term (second term on right hand side) in (A.1), suppose that the $p_{j}$ vary little only around the overall rate $p .$ Then $δ_{j} = p_{j} / p - 1,$ $j = 1, \dots, J,$ are small quantities, and $1 / p_{j} = 1 / p (1 + δ_{j}) = (1 / p) (1 - δ_{j} + δ_{j}^{2} - δ_{j}^{3} + \dots) .$ Keeping terms to second order, $p / p_{j} - 1 \approx - δ_{j} + δ_{j}^{2} .$ The penalty term is then approximated as

$\frac{1}{m} \sum_{j = 1}^{J} W_{j s} (\frac{p}{p_{j}} - 1) S_{y j}^{2} \approx - \frac{1}{m} \sum_{j = 1}^{J} W_{j s} (\frac{p_{j}}{p} - 1) S_{y j}^{2} + \frac{1}{m} \sum_{j = 1}^{J} W_{j s} {(\frac{p_{j}}{p} - 1)}^{2} S_{y j}^{2} . (A .2)$

Let us further assume that the group variances $S_{y j}^{2},$ $j = 1, \dots, J,$ vary little only around their weighted mean $S_{y}^{2} .$ Approximating $S_{y j}^{2} \approx S_{y}^{2}$ in (A.2) we get

$\frac{S_{y}^{2}}{m} \sum_{j = 1}^{J} W_{j s} (\frac{p}{p_{j}} - 1) \approx - \frac{S_{y}^{2}}{m} \sum_{j = 1}^{J} W_{j s} (\frac{p_{j}}{p} - 1) + \frac{S_{y}^{2}}{m} \sum_{j = 1}^{J} W_{j s} {(\frac{p_{j}}{p} - 1)}^{2} .$

Here the first term on the right hand side is zero. The second term, equal to $(I M B / p^{2}) (S_{y}^{2} / m)$ with $I M B$ given in (3.3), becomes a second approximation for the penalty term in (A.1). Therefore, $S_{Δ}^{2} \approx (1 / m - 1 / n) S_{y}^{2} + (I M B / p^{2}) (S_{y}^{2} / m) .$ This gives the desired result (7.4).

Appendix 2

Comparing two quadratic forms

We compare the two quadratic forms in ${\bar{x}}_{r} - {\bar{x}}_{s},$ $Q_{r}$ and $Q_{s}$ defined in (3.1), and justify the approximation $Q_{r} \approx Q_{s}$ needed in the proof in Appendix 3 of Result 2. The respective weighting matrices, $Σ_{r}$ and $Σ_{s},$ are positive definite. Therefore $Q_{r}$ (or $Q_{s})$ can be equal to zero only under the perfect balance ${\bar{x}}_{r} = {\bar{x}}_{s} .$ Since $Q_{r} = Q_{s}$ for perfect balance, the continuity argument implies that $Q_{r} \approx Q_{s}$ for nearly balanced response sets. How close are they more generally?

The CAL estimator (5.1) uses the weight factors $g_{k} = {\bar{x}}_{s}^{'} Σ_{r}^{- 1} x_{k},$ defined for all $k \in s .$ Their link to $Q_{r}$ is shown in the second and third expressions in (A.3) below. Consider also the factors $f_{k} = {\bar{x}}_{r}^{'} Σ_{s}^{- 1} x_{k}$ for $k \in s .$ They are instrumental for $Q_{s},$ and for $I M B = P^{2} Q_{s},$ as the last two expressions in (A.3) show. The following moments of $g_{k}$ and $f_{k}$ are verified with the aid of the $x ‑$ vector condition (2.2):

${\bar{g}}_{r} = 1, {var}_{r} (g) = Q_{r}, {\bar{g}}_{s} = 1 + Q_{r}; {\bar{f}}_{s} = 1, {var}_{s} (f) = Q_{s}, {\bar{f}}_{r} = 1 + Q_{s} . (A .3)$

For $g_{k},$ the means are defined as ${\bar{g}}_{s} = \sum_{s} d_{k} g_{k} / \sum_{s} d_{k},$ ${\bar{g}}_{r} = \sum_{r} d_{k} g_{k} / \sum_{r} d_{k},$ and the variances are ${var}_{s} (g) = \sum_{s} d_{k} {(g_{k} - {\bar{g}}_{s})}^{2} / \sum_{s} d_{k},$ ${var}_{r} (g) = \sum_{r} d_{k} {(g_{k} - {\bar{g}}_{r})}^{2} / \sum_{r} d_{k} .$ For the corresponding moments of $f_{k},$ replace $g_{k}$ by $f_{k} .$ The variances ${var}_{s} (g)$ and ${var}_{r} (f)$ do not have an equally transparent form and will be approximated. Another important property following from (2.2) is $\sum_{s} d_{k} f_{k} g_{k} / \sum_{s} d_{k} = \sum_{r} d_{k} f_{k} g_{k} / \sum_{r} d_{k} = 1 .$ Those equations and appropriate expressions in (A.3) give

${cov}_{s} (f, g) = \sum_{s} d_{k} (f_{k} - {\bar{f}}_{s}) (g_{k} - {\bar{g}}_{s}) / \sum_{s} d_{k} = 1 - {\bar{f}}_{s} {\bar{g}}_{s} = - Q_{r},$

${cov}_{r} (f, g) = \sum_{r} d_{k} (f_{k} - {\bar{f}}_{r}) (g_{k} - {\bar{g}}_{r}) / \sum_{r} d_{k} = 1 - {\bar{f}}_{r} {\bar{g}}_{r} = - Q_{s} .$

Now use ${cov}_{s}^{2} (f, g) \leq {var}_{s} (f) {var}_{s} (g)$ and the analogous inequality where $r$ replaces $s .$ Using also ${var}_{s} (f) = Q_{s}$ and ${var}_{r} (g) = Q_{r}$ from (A.3), we get bounds for the ratio $Q_{r} / Q_{s} :$

$\frac{Q_{s}}{{var}_{r} (f)} \leq \frac{Q_{r}}{Q_{s}} \leq \frac{{var}_{s} (g)}{Q_{r}} . (A .4)$

For more transparent upper and lower bounds, approximate the two variances in (A.4) by assuming that the coefficient of variation (standard deviation divided by mean) is approximately the same for the response $r$ as for the sample $s,$ and this for both $f$ and $g .$ This assumes a certain stability of the coefficient of variation. Then ${var}_{s} (g) \approx {({\bar{g}}_{s})}^{2} {var}_{r} (g) / {({\bar{g}}_{r})}^{2} = {(1 + Q_{r})}^{2} Q_{r},$ so the upper bound in (A.4) is approximately ${(1 + Q_{r})}^{2} > 1.$ Similarly, ${var}_{r} (f) \approx {({\bar{f}}_{r})}^{2} {var}_{s} (f) / {({\bar{f}}_{s})}^{2} = {(1 + Q_{s})}^{2} Q_{s},$ which gives ${(1 + Q_{s})}^{- 2} < 1$ as an approximate lower bound in (A.4). The interval approximation for the ratio $Q_{r} / Q_{s}$ is therefore

$Q_{r} / Q_{s} \in ({(1 + Q_{s})}^{- 2}, {(1 + Q_{r})}^{2}) .$

This is to illustrate that the ratio is not far from 1, because for most data both $Q_{s}$ and $Q_{r}$ are small compared with 1, $Q_{r}$ usually the somewhat bigger. Empirical work suggests however that the approximate upper bound ${(1 + Q_{r})}^{2}$ can often be too low.

Appendix 3

Derivation of Result 2

We derive the expressions in (8.2) under the stated conditions. The sizes of $r$ and $s$ are $m$ and $n,$ respectively; the response rate is $p = m / n .$ The deviation of CAL from the unbiased FUL is ${\hat{Y}}_{C A L} - {\hat{Y}}_{F U L} = \hat{N} Δ_{r}$ where

$Δ_{r} = {(b_{r} - b_{s})}^{'} {\bar{x}}_{s} = Σ_{r} d_{k} g_{k} y_{k} / Σ_{r} d_{k} - Σ_{s} d_{k} y_{k} / Σ_{s} d_{k}$

with $b_{r}$ and $b_{s}$ given by (4.1), and $g_{k} = {\bar{x}}_{s}^{'} Σ_{r}^{- 1} x_{k} .$ Note that $b_{s}^{'} {\bar{x}}_{s} = {\bar{y}}_{s}$ by (2.2). Now $Σ_{r} d_{k} g_{k} x_{k}^{'} / Σ_{r} d_{k} = Σ_{s} d_{k} x_{k}^{'} / Σ_{s} d_{k} = {\bar{x}}_{s}^{'} .$ Post-multiply that equation by $β$ and use the result to get $Δ_{r} = Σ_{r} d_{k} g_{k} (y_{k} - x_{k}^{'} β) / Σ_{r} d_{k} - Σ_{s} d_{k} (y_{k} - x_{k}^{'} β) / Σ_{s} d_{k},$ which expresses $Δ_{r}$ in terms of the residuals $ε_{k} = y_{k} - x_{k}^{'} β$ of the model (8.1):

$Δ_{r} = \frac{\sum_{r} d_{k} g_{k} ε_{k}}{\sum_{r} d_{k}} - \frac{\sum_{s} d_{k} ε_{k}}{\sum_{s} d_{k}} .$

Then use the model properties of $ε_{k}$ in (8.1). From $E_{ξ} (ε_{k} | x_{k}) = 0$ for all $k$ it follows that $E_{ξ} (Δ_{r} | X, r, s) = 0.$ To evaluate the variance, use $E_{ξ} (ε_{k}^{2} | x_{k}) = σ_{ε}^{2},$ for all $k \in s,$ and $E_{ξ} (ε_{k} ε_{l} | x_{k}, x_{l}) = 0,$ all $k \neq l \in s .$ This gives

$E_{ξ} (Δ_{r}^{2} | X, r, s) = σ_{ε}^{2} \frac{\sum_{r} d_{k}^{2} g_{k}^{2}}{{(\sum_{r} d_{k})}^{2}} + σ_{ε}^{2} \frac{\sum_{s} d_{k}^{2}}{{(\sum_{s} d_{k})}^{2}} - 2 σ_{ε}^{2} \frac{\sum_{r} d_{k}^{2} g_{k}}{(\sum_{r} d_{k}) (\sum_{s} d_{k})} .$

Here the $d_{k}$ cancel out, because constant. The first and second expressions in (A.3) hold for any $d_{k},$ in particular constant $d_{k},$ so we get $\sum_{r} g_{k} / m = 1$ for the mean and $\sum_{r} g_{k}^{2} / m = Q_{r} + 1$ for variance plus squared mean. Therefore,

$E_{ξ} (Δ_{r}^{2} | X, r, s) = (\frac{1}{m} (1 + Q_{r}) + \frac{1}{n} - 2 \frac{1}{n}) σ_{ε}^{2} = (\frac{1}{m} - \frac{1}{n} + \frac{Q_{r}}{m}) σ_{ε}^{2} .$

As a final step, use the approximation $Q_{r} \approx Q_{s}$ justified in Appendix 2, and $I M B = p^{2} Q_{s} .$ Then, as claimed in Result 2, $E_{ξ} (Δ_{r}^{2} | X, r, s) \approx (1 - p + I M B / p^{2}) (σ_{ε}^{2} / m) .$

References

Beaumont, J.-F., Bocci, C. and Haziza, D. (2014). An adaptive data collection procedure for call prioritization. Journal of Official Statistics, 30, 607-622.

Bethlehem, J., Cobben, F. and Schouten, B. (2011). Handbook of nonresponse in households surveys. New York: John Wiley & Sons, Inc.

Brick, J.M. (2013). Unit nonresponse and weighting adjustments: A critical review. Journal of Official Statistics, 29, 329-353.

Couper, M.P., and Wagner, J. (2011). Using paradata and responsive design to manage survey nonresponse. Proceedings, 58^th World Statistics Congress, International Statistical Institute.

Deville, J.-C., and Tillé, Y. (2004). Efficient balanced sampling. The cube method. Biometrika, 91, 893-912.

Groves, R. (2006). Research synthesis: Nonresponse rates and nonresponse error in household surveys. Public Opinion Quarterly, 70, 646-675.

Groves, R.M., and Heeringa, S.G. (2006). Responsive design for household surveys: Tools for actively controlling survey errors and costs. Journal of the Royal Statistical Society, Series A, 169, 439-457.

Little, R.J.A., and Vartivarian, S. (2005). Does weighting for nonresponse increase the variance of survey means? Survey Methodology, 31, 2, 161-168. Paper available at http://www.statcan.gc.ca/pub/12-001-x/2005002/article/9046-eng.pdf.

Lundquist, P., and Särndal, C.-E. (2013). Aspects of responsive design. With applications to the Swedish Living Conditions Survey. Journal of Official Statistics, 29, 557-582.

Peytchev, A., Riley, S., Rosen, J., Murphy, J. and Lindblad, M. (2010). Reduction of nonresponse bias in surveys through case prioritization. Survey Research Methods, 4, 21-29.

R Core Team (2014). R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria.

Rao, R.S., Glickman, M.E. and Glynn, R.J. (2008). Stopping rules for surveys with multiple waves of nonrespondent follow-up. Statistics in Medicine, 27, 2196-2213.

Särndal, C.-E. (2011). The 2010 Morris Hansen lecture: Dealing with survey nonresponse in data collection, in estimation. Journal of Official Statistics, 27, 1-21.

Särndal, C.-E., and Lundström, S. (2005). Estimation in Surveys with Nonresponse. New York: John Wiley & Sons, Inc.

Särndal, C.-E., and Lundquist, P. (2014a). Accuracy in estimation with nonresponse: A function of degree of imbalance and degree of explanation. Journal of Survey Statistics and Methodology, 2, 361-387.

Särndal, C.-E., and Lundquist, P. (2014b). Balancing the response and adjusting estimates for nonresponse bias: Complementary activities. Journal de la Société Française de Statistique, 155(4), 28-50.

Schouten, B., Calinescu, M. and Luiten, A. (2013). Optimizing quality of response through adaptive survey designs. Survey Methodology, 39, 1, 29-58. Paper available at http://www.statcan.gc.ca/pub/12-001-x/12-001-x2013001-eng.pdf.

Schouten, B., Cobben, F. and Bethlehem, J. (2009). Indicators for the representativeness of survey response. Survey Methodology, 35, 1, 101-113. Paper available at http://www.statcan.gc.ca/pub/12-001-x/2009001/article/10887-eng.pdf.

Schouten, B., Cobben, F., Lundquist, P. and Wagner, J. (2016). Does more balanced survey response imply less non-response bias? Journal of the Royal Statistical Society, Series A, 179, 727-748.

Wagner, J. (2008). Adaptive survey design to reduce nonresponse bias. Ph. D. Thesis, University of Michigan, Ann Arbor.

Wagner, J., and Raghunathan, T.E. (2010). A new stopping rule for surveys. Statistics in Medicine, 29, 1014-1024.

ISSN : 1492-0921

Editorial policy

Survey Methodology publishes articles dealing with various aspects of statistical development relevant to a statistical agency, such as design issues in the context of practical constraints, use of different data sources and collection techniques, total survey error, survey evaluation, research in survey methodology, time series analysis, seasonal adjustment, demographic studies, data integration, estimation and data analysis methods, and general survey systems development. The emphasis is placed on the development and evaluation of specific methodologies as applied to data collection or the data themselves. All papers will be refereed. However, the authors retain full responsibility for the contents of their papers and opinions expressed are not necessarily those of the Editorial Board or of Statistics Canada.

Submission of Manuscripts

Survey Methodology is published twice a year in electronic format. Authors are invited to submit their articles in English or French in electronic form, preferably in Word to the Editor, (statcan.smj-rte.statcan@canada.ca, Statistics Canada, 150 Tunney’s Pasture Driveway, Ottawa, Ontario, Canada, K1A 0T6). For formatting instructions, please see the guidelines provided in the journal and on the web site (www.statcan.gc.ca/SurveyMethodology).

Note of appreciation

Canada owes the success of its statistical system to a long-standing partnership between Statistics Canada, the citizens of Canada, its businesses, governments and other institutions. Accurate and timely statistical information could not be produced without their continued co-operation and goodwill.

Standards of service to the public

Statistics Canada is committed to serving its clients in a prompt, reliable and courteous manner. To this end, the Agency has developed standards of service which its employees observe in serving its clients.

Copyright

Published by authority of the Minister responsible for Statistics Canada.

Use of this publication is governed by the Statistics Canada Open Licence Agreement.

Catalogue No. 12-001-X

Frequency: semi-annual

Ottawa

Date modified:: 2016-12-20

Language selection

Search and menus

Search