Nonresponse adjustments with misspecified models in stratified designs 1. Introduction
Adjusting the base weights for unit nonresponse using weighting classes is a standard approach to survey weighting, but the adjustments are not done in the same way by all researchers or survey organizations. Little and Vartivarian (2003), hereafter referred to as L&V, observed that using a nonresponse adjustment factor that is weighted by the inverse of the probability of selection appears to be the most common approach. They also pointed out that using design weights to compute a weighted nonresponse adjustment does not eliminate nonresponse bias in estimates of the mean of the population when the response mechanism is not specified correctly by the weighting adjustment model. L&V then conducted a simulation study using a simple stratified sample design to examine the effect of weighting the nonresponse adjustment factors. They concluded that weighting the nonresponse adjustment has little or no value.
Theoretical justifications for nonresponse adjustment require that either the response mechanism or the target variable must be modeled correctly to eliminate nonresponse bias; we are not aware of any theory that suggests that weighting by the inverse of the probability of selection completely eliminates bias when the model is misspecified (e.g., Kalton 1983; Little 1986; Little and Rubin 2002; Särndal and Lundström 2005). In this regard, the importance of modeling for nonresponse adjustment urged by L&V is essential for good statistical practice. However, correctly specifying a highly predictive model is an ideal that cannot be achieved in most surveys because of the complexity of the phenomenon and because powerful auxiliary variables rarely exist. The search for better auxiliary data for this modeling has fueled research into paradata, but the models using these data still have relatively poor correlations with response propensities (Kreuter, Olson, Wagner, Yan, Ezzati-Rice, Casas-Cordero, Lemay, Peytchev, Groves and Raghunathan 2010). In practice, imperfect models are used and nonresponse bias is never completely eliminated.
Consequently, understanding the effects of nonresponse adjustment methods and whether there is any value to weighting the nonresponse adjustment with an incorrectly specified response model is important. Even though a message of L&V was the need to include design variables in the nonresponse modeling, some researchers appear to have concluded that weighting the adjustment has no role (e.g., Chadborn, Baster, Delpech, Sabin, Sinka, Rice and Evans 2005; Haukoos and Newgard 2007). However, L&V’s conclusion that weighting the nonresponse adjustment factor is either incorrect or inefficient was based on comparisons to correctly specified models that always produce unbiased estimates. Their suggestion to condition on the design variables (in their setting the design variable was the stratum) resulted in identical weighted and unweighted estimators. Their simulations are also centered on a specific stratified sample design and they only consider estimating means. As discussed below, these are substantial limitations and the conclusions that some have drawn that weighting the adjustment is inappropriate need to be reconsidered.
Following L&V, researchers have examined the effects of weighting in other cases. Sukasih, Jang, Vartivarian, Cohen and Zhang (2009) compared nonresponse adjustments with and without weights by simulation within the context of a specific survey. West (2009) used simulation to study estimates of population means under more complex sample designs that featured clustering and differential sampling rates. Both of these studies concluded that weighting the nonresponse adjustments by the design weights was beneficial compared to using an unweighted approach, even though the differences due to weighting were not large. Kott (2012) assessed the robustness of the adjustments theoretically and described the conditions under which the various estimators for population means had greater protection against nonresponse bias; he recommended a weighted approach. Related research has been conducted on the need for weighting for estimating response propensity model coefficients (Wun, Ezzati-Rice, Diaz-Tena and Greenblatt 2007; Grau, Potter, Williams and Diaz-Tena 2006), but this line of research is sufficiently different that we do not discuss it here.
In this article, we explore the effect of weighting nonresponse adjustments when the nonresponse model is imperfect. In Section 2, we expand on the L&V results by looking at estimators for totals and domain means and totals; L&V only considered overall means. Using the same population and basic simulation setting of L&V, we also explore the effect of different sample allocation to the strata while L&V used one sample allocation. The results of the simulations presented in Section 3 show that there are important differences in the properties of the weighted and unweighted estimators and these vary by how the sample is allocated. We explain the behaviors of the estimators using simple approximations to show why they differ. Although weighting the adjustment factor does not always give estimates with lower bias and root mean square error when compared to estimates from the unweighted alternative, it has substantial benefits for estimates of totals and provides protection against large errors that may arise with an unweighted approach. As a result, we recommend a weighted approach when the true response mechanism is not fully known. Conclusions are presented in Section 4.
- Date modified: