Tests for evaluating nonresponse bias in surveys Section 1. Introduction
Nonresponse rates in probability samples are increasing worldwide. The U.S. Office of Management and Budget requires a nonresponse bias analysis when response rates are low or there are other indications that bias may be a problem (United States Office of Management and Budget 2006). Groves (2006) recommended using multiple approaches to assess potential nonresponse bias on key survey estimates.
Assessing potential nonresponse bias typically requires an external “gold standard” data source or rich sampling frame information. Common approaches for assessing nonresponse bias include: (1) comparing frame variables for respondents and nonrespondents, (2) comparing early and late respondents on frame variables and key survey variables, and (3) comparing estimates from the survey respondents (using nonresponse-adjusted weights) with estimates from an independent gold standard source. Differences in (1) and (2), however, do not necessarily imply that nonresponse bias remains after the weights are adjusted through calibration or propensity methods. If weight adjustments such as those described in Brick (2013) are successful in adjusting for nonresponse bias, the estimates from the survey using the nonresponse-adjusted weights may be approximately unbiased even if assessments (1) and (2) show differences.
In this paper we compare an estimate calculated using base weights from the selected sample with an estimate of the same quantity calculated using nonresponse-adjusted weights from the respondents only. An example might be comparing the estimated proportion of persons living in census tracts with more than 50% of housing units being owner occupied from (1) the selected sample, using the base weights, (2) the respondents, using the base weights, and (3) the respondents, using nonresponse-adjusted and/or poststratified weights. All three estimates of the proportion use the same characteristic, which is assumed to be known for everyone in the selected sample.
The requirement that be known for the selected sample restricts the set of variables that can be used to test for nonresponse bias. Typically, many of the key variables of interest are available only for the respondents, not for the entire selected sample. Other variables that are available for the entire selected sample may be used for poststratification or other nonresponse weighting adjustments. Poststratification forces the estimates of population totals for poststratification variables to equal the independent population counts for these variables, so these variables would not be expected to exhibit nonresponse bias after weight adjustments are performed. Variables that are available for the entire selected sample but are not used in the nonresponse weighting adjustments, and variables that are correlated with key survey variables, are the best choices for testing nonresponse bias. Examples of such variables include sample frame variables that are not used in poststratification (for example, an e-mail survey of university students may have information on academic performance that is not used in the nonresponse weighting), characteristics from a census (such as percent poverty in the block containing the sampled address), or information gathered by the interviewer (such as indications of children in the household that are visible from the street).
Eltinge (2002) and Harris-Kojetin (2012) recommended comparing estimates using different sets of weights to assess nonresponse bias and to choose among competing sets of nonresponse-adjusted weights. Such comparisons are common in nonresponse bias analyses: for example, Hamrick (2012) compared respondents with the full sample in the Eating and Health Module of the American Time Use Survey. To date, however, there has been no comprehensive examination of the statistical properties underlying these comparisons. In this paper, we derive the theoretical properties of variance estimators and hypothesis tests for the differences among estimated means that are calculated using the same outcome variable but with different weights and subsets of the data, and give conditions that will ensure consistency of the variance estimators.
Poststratification or inverse propensity weighting are commonly used to compensate for nonresponse bias. Yung and Rao (2000) derived linearization and jackknife estimators for the variance of a population mean estimated using poststratification, with and without nonresponse. They considered a uniform response mechanism in which each poststratum has the same response propensity, and considered the response indicator to be a fixed characteristic of the finite population. Kim and Kim (2007) studied asymptotic properties for inverse propensity weight adjustments, assuming that the response indicators of different units are independent. The previous work studied the variance of the poststratified or inverse-propensity-weighted statistic of interest. The problem we consider differs from the previous work because the estimated population total from the selected sample is often highly correlated with the estimate calculated using the respondents only, particularly when the response rate approaches one. The linearization and replication variance estimators in this paper account for that high correlation between the two sets of estimates, and thus can be used for testing the hypothesis that the poststratification or inverse propensity weighting removes the bias for the variables studied. We also extend previous research by allowing the response indicators to be correlated within primary sampling units, reflecting possible within-cluster homogeneity for responding to the survey.
Section 2 defines the parameter to be tested in the poststratification setting, derives the linearization and jackknife variance estimators, and gives sufficient conditions for the variance estimators to be consistent. In some circumstances the linearized variance of the test statistic may be zero under the null hypothesis, in which case higher-order terms of the variance are needed. The higher-order terms are derived for the special case of simple random sampling in Theorem 3. Section 3 provides the linearization and jackknife variance estimators for testing the hypothesis that the propensity weights remove the nonresponse bias. Section 4 presents simulation studies and Section 5 contains concluding remarks and discusses future work.
- Date modified: