Tests for evaluating nonresponse bias in surveys Section 4. Simulation resultsTests for evaluating nonresponse bias in surveys Section 4. Simulation results

• Previous
• Next

We examine the performance of the variance estimators in two simulation studies. The first study generates finite populations with response indicators $r_{hik}$ and then draws simple random samples from the population. The second simulation uses data from the 2009-2013 5-year American Community Survey Public Use Microdata Samples (ACS PUMS) as a population and then draws repeated cluster samples from this population under different nonresponse mechanisms.

For the simulation involving simple random sampling, we generated finite populations of 1,000,000 units. To study the poststratification estimator we used $C\mathrm{<mo>=</mo>\; 6}$ poststrata to generate nonresponse. The experimental factors were: 

• sample size, $n:$ 300 or 1,000.
• population proportion $\left(M_c/M\right)$ in each poststratum: (P1) (1/6, 1/6, 1/6, 1/6, 1/6, 1/6), (P2) (1/21, 2/21, 3/21, 4/21, 5/21, 6/21), and (P3) (6/21, 5/21, 4/21, 3/21, 2/21, 1/21).
• response rates in poststrata: (R1) (0.2, 0.3, 0.5, 0.6, 0.8, 0.9), (R2) (0.3, 0.7, 0.3, 0.7, 0.3, 0.7), and (R3) (1, 1, 1, 1, 1, 1). Level (R3), with full response, is included to explore the accuracy of the higher-order approximation to the variance when $V_1\left(\widehat{\theta }\right)\mathrm{<mo>=</mo>\; 0.}$
• poststratum means: (M1) (0, 0, 0, 0, 0, 0), (M2) (-2, -1, 0, 1, 2, 3) and (M3) (0, 1, 0, 1, 0, 1).
• number of poststrata used in nonresponse adjustment: 1, 3 (collapse adjacent pairs of poststrata), or 6. Only the settings with 6 poststrata are guaranteed to correct for the nonresponse bias.

Within each poststratum, population values $y_i$ were generated from a normal distribution with the specified poststratum mean and variance 1. The response indicators $r_i$ were generated as independent Bernoulli random variables with mean $R_i.$ The simple random sampling simulations were done in version 3.2.2 of R (R Core Team 2015), and 2,000 iterations were performed for each of the 162 simulation settings, which results in a standard error less than 0.005 for the Monte Carlo estimate of the rejection proportion when the null hypothesis of $\theta \mathrm{<mo>=</mo>\; 0}$ is true. Some of the generated samples had fewer than two respondents in one or more poststrata, which would result in some jackknife resamples having no respondents in those poststrata. For such samples, the two poststrata with the smallest number of respondents were combined iteratively until all poststrata had at least two respondents.

For each simulation setting, the Monte Carlo (MC) variance of $\widehat{\theta },$ ${\widehat{V}}_{MC}\left(\widehat{\theta }\right),$ was calculated as the sample variance of ${\widehat{\theta }}_b$ for $b\mathrm{<mo>=</mo>\; 1,}\dots \mathrm{,}2\text{,}000.$ The linearization and jackknife variance estimates were calculated for each simulated sample, and the means of those estimates over the 2,000 samples are denoted as ${\widehat{V}}_L\left(\widehat{\theta }\right)$ and ${\widehat{V}}_J\left(\widehat{\theta }\right),$ respectively.

Figures 4.1 and 4.2 display results for the simulation settings in which $V_1\left(\widehat{\theta }\right)\mathrm{<mo>></mo>\; 0.}$ Figure 4.1 displays histograms of the ratios of the mean linearization and jackknife variance estimates to ${\widehat{V}}_{MC}\left(\widehat{\theta }\right).$ The scatterplot in Figure 4.2 displays the percentage of the 2,000 iterations in which the null hypothesis $H_0\mathrm{<mo>:</mo>}\theta \mathrm{<mo>=</mo>\; 0}$ is rejected at the 5% significance level. Most of the variance estimates are close to the MC variance and the rejection rate for $H_0\mathrm{<mo>:</mo>}\theta \mathrm{<mo>=</mo>\; 0}$ is approximately 5% when $\theta \mathrm{<mo>=</mo>\; 0},$ with higher power for larger values of $\left|\theta \right|.$ Four of the simulation runs with $\theta \mathrm{<mo>=</mo>\; 0},$ however, have linearization and jackknife variances that are approximately twice the MC variance, and rejection rates that are between 0 and 1%. These results are from the simulations with poststratum means (M3), response rates (R3), population proportions (P2) or (P3), and three collapsed poststrata. Although the population means for the collapsed poststrata differ, they do not differ greatly and a sample size of 1,000 is too small for the first-order asymptotic approximation to be accurate. For these settings, a sample size of approximately 15,000 was needed to reduce the variance ratios ${\widehat{V}}_L\left(\widehat{\theta }\right)/{\widehat{V}}_{MC}\left(\widehat{\theta }\right)$ and ${\widehat{V}}_J\left(\widehat{\theta }\right)/{\widehat{V}}_{MC}\left(\widehat{\theta }\right)$ to 1.2.

Figure 4.3 shows the behavior of ${\widehat{V}}_L\left(\widehat{\theta }\right),$ ${\widehat{V}}_J\left(\widehat{\theta }\right),$ and ${\widehat{V}}_2\left(\widehat{\theta }\right)$ when the first-order term of the variance is $V_1\left(\widehat{\theta }\right)\mathrm{<mo>=</mo>0}$ but $V_2\left(\widehat{\theta }\right)\mathrm{<mo>></mo>0.}$ For all of those simulations, the true value of $\theta$ was 0 and the second-order term ${\widehat{V}}_2\left(\widehat{\theta }\right)$ was calculated using the SRS approximation in Theorem 3. Even though the true first-order variance $V_1\left(\widehat{\theta }\right)$ is zero for these settings, the estimated first-order variances from linearization and jackknife are nonzero. For the simulations with poststratum means (M1) and response rates (R3), for example, all poststrata have the same population mean. The sample means for the poststrata differ, however, and this causes the linearization and jackknife variance estimators to be positive and, on average, about twice as large as the MC variance. The same thing happens with poststratum means (M3), population proportions (P1), and response rates (R3) when three poststrata are used: the three collapsed poststrata each have population mean 1/2 but the sample means vary.

Only simulation settings with response rates (R3) required the use of higher-order terms or large sample sizes for the linearization and jackknife variance estimators to be accurate. It would be easy to identify these situations in practice from the absence of nonresponse.

To study the properties of the estimators in Section 3, we used a subset of the populations generated for the poststratification simulation as well as populations generated with continuous covariate $x,$ giving factors: 

• Sample size, $n:$ 300 or 1,000.
• Population values and nonresponse generation. 
  1. Nonresponse is generated in 6 poststrata with population proportions (P1) or (P2), and response rates (R1) or (R2). The variable of interest $y$ is generated with poststratum means (M1) and (M2) plus a $N\left(\mathrm{0,1}\right)$ error term.
  2. Covariate $x$ is generated from a $N\left(\mathrm{0,1}\right)$ distribution. Then $y$ is generated as (Y1) $0+N\left(\mathrm{0,1}\right)$ (independent of $x),$ (Y2) $x+N\left(\mathrm{0,1}\right),$ or (Y3) $x^2+N\left(\mathrm{0,1}\right).$ The response propensities are generated as (R1P) $R\mathrm{<mo>=</mo>\; 0.8}$ for all units, (R2P) logit $\left(R\right)\mathrm{<mo>=</mo>}1/\left(1+\exp \left(-x\right)\right),$ and (R3P) logit $\left(R\right)\mathrm{<mo>=</mo>}1/\left(1+\exp \left(-x^2/3\right)\right).$
• Response propensity model used.
  1. For poststratified populations, treat $x$ as a continuous variable with values 1 $–$ 6.
  2. For populations with generated covariate $x,$ use linear logistic regression with covariate $x.$ This model is correctly specified for response-generating mechanisms (R1P) and (R2P) but incorrectly specified for mechanism (R3P).

To reduce the instability of the estimators, estimated response propensities less than 0.05 were replaced by 0.05, corresponding to trimming weight adjustments larger than 20. Figures 4.4 and 4.5 display the variance ratios and empirical power for the propensity model simulations. All settings in this simulation had $V_1\left(\widehat{\theta }\right)\mathrm{<mo>></mo>\; 0.}$ As in the poststratification simulation, the linearization and jackknife variance estimators both perform well in general. There are a few settings, however, in which the linearization variance is substantially larger than the jackknife. This occurs because of the weight trimming: the jackknife automatically accounts for the effect of weight trimming on the variance because the jackknife replicates also trim the weights. The linearization variance used in this simulation was from Theorem 5, and the formula would need to be modified to include the effects of trimming. We also ran simulations using the jackknife in which the mean was estimated instead of the population total, and the jackknife performed well for that parameter as well.

The second simulation study used a population of 6,019,599 household-level records from the ACS PUMS studied in Lohr, Hsu and Montaquila (2015). There are 3,344 PSUs in the population defined by the public use microdata areas. Eight poststrata were formed based on the cross-classification of households by tenure (rent or own), presence of children in the household (yes or no), and number of income earners (0-1 or 2+). The primary outcome variable $y$ was household income. Additionally, a less skewed outcome variable $\log \left(y\right)$ was studied, where $\log \left(y\right)$ was set to 0 if $y\mathrm{<mo><</mo>\; 1.}$

A $2\times 2\times 3$ factorial design was used for this study with factors 

• overall response rate: 50% or 80%.
• number of PSUs for each sample: 25 or 100.
• nonresponse generating mechanism: (N1) missing completely at random (MCAR), with response propensity for all records equal to the response rate for all households; (N2) missing at random (MAR), where a linear logistic model with main effect terms for tenure, presence of children, and number of income earners generates the response propensities; and (N3) missing not at random (MNAR), where a linear logistic model with main effect terms for tenure, presence of children, and household income generates the response propensities.

For the first two nonresponse generating mechanisms, $\theta \mathrm{<mo>=</mo>\; 0.}$ For the first mechanism, there is no nonresponse bias. Poststratification corrects for the bias in the second mechanism because $R_{hik}\mathrm{<mo>=</mo>}{p}_c$ for units in poststratum $c.$ Poststratification does not correct for the bias in the third mechanism because the nonresponse depends on the $y$ variable, household income.

For each simulation setting, response indicators were generated independently for the population units using the calculated response propensities. One thousand samples were drawn for each setting, in which PSUs were selected with probability proportional to size and a simple random sample of 100 households was selected from each sampled PSU. The standard error for the rejection proportion when $\theta \mathrm{<mo>=</mo>\; 0}$ is less than 0.007.

Calculations for the ACS simulation were done in SAS^® software (SAS Institute, Inc. 2011). We first calculated the weights and jackknife weights for the selected sample, and then calculated the poststratified and jackknife poststratified weights for the respondents. The two sets of jackknife weights used the same replication structure, so that replicate weight $k$ for the respondents deleted the same PSU as replicate weight $k$ for the selected sample. To simplify computation of ${\widehat{\theta }}_M$ in (2.10), we concatenated the selected sample and respondents, with their respective weights, into one data set and set $u_i\mathrm{<mo>=</mo>\; 1}$ for records in the respondent data set and $u_i\mathrm{<mo>=</mo>\; 0}$ for records in the selected sample data set. The linear model $y_i\mathrm{<mo>=</mo>}{\beta }_0+{\beta }_1{u}_i$ was fit to the concatenated data using the SURVEYREG procedure, and ${\widehat{\theta }}_M\mathrm{<mo>=</mo>}{\widehat{\beta }}_1$ from the regression model.

Table 4.1 gives the results from the simulation. For all but one of the simulation settings, the mean of the jackknife variance estimates is larger than the Monte Carlo variance of ${\widehat{\theta }}_M,$ but the bias of the jackknife variance is reduced when more PSUs are sampled or the response rate is higher. The outcome variable $y,$ household income, is highly skewed, and the rejection rate when ${\theta }_M\mathrm{<mo>=</mo>\; 0}$ is closer to the nominal $\alpha$ of 0.05 when the log-transformed variable is used.

• Previous
• Next