An assessment of accuracy improvement
by adaptive survey design
Section 1. Introduction
Many important probability sampling surveys have to close the data collection period by declaring a considerable rate of nonresponse. At the estimation stage that follows, the best possible estimates must nevertheless be produced, based on the respondents that a less-than-ideal data collection has delivered. Whatever the techniques used, nonresponse bias remains; it must be kept as low as possible. Responsive – or adaptive – data survey design for household and other surveys aims at an active control of survey errors and costs, in the planning and data collection phases. One objective is to get a final set of respondents likely to improve prospects for more accurate survey estimates. The concept responsive survey design is due to Groves and Heeringa (2006).
The data collection in a large survey typically extends over a period of time, days or weeks, during which contact attempts are made with the units in the selected probability sample. For some units, a fruitful result is obtained after a few attempts. For other units, despite repeated calls, refusal or non-contact will lead, finally, to declaring them nonrespondents. The background in this article is not that the nonresponse rate will ultimately be reduced to single-digit levels. More realistically, it is that high nonresponse, of the order 30% to 50%, will prevail when the data collection period must necessarily close. Best possible estimates must still be produced.
To improve data collection, a variety of techniques have been suggested and tried: case prioritization, stopping rules, balancing, and others. Case prioritization is considered, for example, in Peytchev, Riley, Rosen, Murphy and Lindblad (2010), Beaumont, Haziza and Bocci (2014). Stopping rules are discussed in Rao, Glickman and Glynn (2008), and in Wagner and Raghunathan (2010).
Variation between demographic subgroup response rates has been considered in seeking evidence of bias, as in Peytcheva and Groves (2009). Andridge and Little (2011) propose proxy pattern-mixture analysis as a method to assess non-response bias. The nonresponse bias cannot be quantified, but indicators of the risk of bias can be used, as discussed in Wagner (2012), Kreuter, Olson, Wagner, Yan, Ezzati-Rice, Casas-Cordero, Lemay, Peytchev, Groves and Raghunathan (2010), Lohr, Riddles and Morganstein (2016), Nishimura, Wagner and Elliott (2016).
Representativeness and balance are terms now frequently used in reference to the set of responding units. Indicators of representativeness, including the R-indicator based on estimated response probabilities, have been developed, see Schouten, Cobben and Bethlehem (2009), Bethlehem, Cobben and Schouten (2011), and Schouten, Shlomo and Skinner (2011), Bianchi, Shlomo, Schouten, da Silva and Skinner (2016).
Balancing is a procedure practiced in data collection, to get in the end a well representative set of respondents. The key is to make auxiliary variable means for the respondents agree closely with corresponding means computable for the probability sample, or known for the population. This lies behind the response imbalance statistic IMB, used in Särndal (2011a), Lundquist and Särndal (2013). Methods are available for reducing IMB in adaptive data collection, see for example Särndal and Lundquist (2014).
“Nonresponse adjustment” is an often used term suggesting that an ideal estimator that would perform to satisfaction under full response will need, under nonresponse, “repair” to be a credible surrogate. The goal of the adjustment is to keep the harm - the nonresponse bias - within limits. Current practice in statistical agencies is to use auxiliary variables in the estimation, in computing calibrated nonresponse adjustment weights. Reduced variance and reduced nonresponse bias can be realized. A number of recent contributions, some of them review articles, discuss weighting procedures for the estimation phase, for example, Brick (2013), Fattorini, Franceschi and Maffei (2013), Haziza and Lesage (2016), Little and Vartivarian (2005), Tourangeau, Brick, Lohr and Li (2017). The selection of auxiliary variables for weighting procedures is considered in Särndal and Lundström (2005, 2008, 2010), Särndal (2011b), but is not a topic in the present article.
It is evident that better balanced response could significantly improve accuracy if the estimator in use is rudimentary, such as the response mean expanded by the population size. But the starting point here is that auxiliary variables are used extensively at the estimation stage, and that their use also in adaptive data collection is an added feature that may pay off. Such combined role of auxiliary variables is potentially important.
The question whether balancing the response will significantly reduce nonresponse bias has been raised in the literature, see, for example, Schouten, Cobben, Lundquist and Wagner (2014), Lundquist and Särndal (2013), Särndal and Lundquist (2014), Särndal, Lumiste and Traat (2016). These references point to some, although not strongly pronounced, accuracy improvement from balancing. Gains have appeared modest rather than strong or convincing. Much of the evidence is empirical. The theoretical reasons behind a “limited success” of adaptive data collection need to be better understood. This article attempts to throw more light on the question: Can a use of auxiliary variables in an adaptive data collection contribute further to improving the estimation, given that such variables are in any case used in a calibrated weighting at the estimation stage? The question is important for research on adaptive survey design. These designs must sustain a clear promise for improved estimates. If little or no improvement is forthcoming, under fairly general conditions, a part of the motivation for adaptive design seems to be lost. We inquire into the theoretical reasons for a certain accuracy gain from better balanced response, more precisely why one might expect “a marginal advantage” from adaptive design, that is, a further improvement of estimates by putting auxiliary variables to work also at the data collection stage. It is perfectly legitimate to re-utilize auxiliary variables at the estimation stage.
The contents are arranged as follows: Notation is introduced (Section 2) for three important population subsets: the probability sample, the response set and its complement, the nonresponse set. The role of the auxiliary vector -vector) is emphasized, particularly its role in the response imbalance statistic denoted IMB. To estimate the population -total, we consider the estimator based on weights calibrated on the auxiliary vector. Its deviation from the unbiased estimator which is hypothetical because requiring full response – is an indicator of bias that we examine in depth.
We decompose the deviation of the calibration estimator (Section 3) into two terms, the reducible term and the resisting term. The former can be reduced by adaptive design; in fact, all the way to zero if perfect balance could be realized. The resisting term, by contrast, is little affected by adaptive data collection, hence suggesting that adaptive design is at best a partial remedy for getting rid of nonresponse bias. The special case where the -vector codes a set of mutually exhaustive and exhaustive sample groups is particularly important (Section 4). Mathematically more tractable, it gives a clearer understanding of the two terms of the decomposition. Empirical evidence using Swedish Labour Force Survey data (Section 5) illustrates and confirms the theoretical conclusions about the two terms. Final comments (Section 6) terminate the article. Proofs are presented in an Appendix.
- Date modified: