Estimation of response propensities and indicators of representative response using population-level information
Section 4. Evaluation study

• Previous
• Next

In this section, we carry out an evaluation study on real census data from the 1995 Israel Census Sample to assess the sampling properties of the estimation procedures introduced in Section 3.

The aim of the evaluation is two-fold: a) to study the sampling properties of the unadjusted and bias adjusted population-based R-indicators, comparing them to those of their sample-based counterpart and assessing the effect of sample size, number of auxiliary variables in the model, and response rate; b) to investigate the performance of the bootstrap estimator for estimating the variance of the population-based R-indicator.

4.1  Data and design of evaluation study

The 1995 20% Israel Census Sample contains 753,711 individuals aged 15 and over in 322,411 households. The census sample design is a random systematic sample where every fifth household was delivered a long questionnaire covering a range of socio-economic questions. The sample units are households and all persons over the age of 15 in the sampled households are interviewed. Typically a proxy questionnaire is used and therefore there is no individual nonresponse within the household. In this study, we assume that every household has an equal probability to be included in the sample. This evaluation study uses data at the household level $\left(N=\text{322,411}\right).$

We carried out a two-step design to define response propensities in the population (census) data. This procedure ensures that we have a known model generating the response propensities. Moreover, in order to explore the effect of varying response rates and the number of auxiliary variables in the model on the performance of the estimators, we considered six scenarios defined by the level of response rates (3 categories) and the type of model (2 categories).

1. First, probabilities of response were defined according to variables: Type of locality (4 categories defined by rural/urban and type of population), number of persons in household grouped to 3 categories (1-2, 3-5, 6+), children in the household indicator (yes, no), region (7 categories dividing the country from north to south), and density (3 categories: less than 1.5, 1.5-3.0, greater than 3.0). These variables define groups that are known to have differential response rates for social surveys in practice. To study the effect of response rates on the performance of the estimators, probabilities of response $p$ were defined according to $p=p_1{p}_2{p}_3{p}_4{p}_5+\alpha$ with three choices $\alpha =\text{0}\text{.15}$ (RR1), $\alpha =\text{0}\text{.55}$ (RR2), and $\alpha =\text{0}\text{.75}$ (RR3), where the probabilities $p_1,p_2,p_3,p_4,p_5$ are given in Table 4.1. We generate three response indicator variables using the Bernoulli distribution for each of the response scenarios defined under RR1, RR2, and RR3.
2. For each of the response scenarios from step (A), we use the response indicator as the dependent variable and fit both a linear and a logistic regression model to the population to predict “true” response propensities for our evaluation study under both link functions. Two different models were considered for prediction of “true” response propensities. In Model 1, independent variables are exactly the explanatory variables used in step A for the definition of response probabilities (child indicator, number of persons in the household, region, type of locality, density). In Model 2, independent variables are type of locality, number of persons in household, child indicator. Notice that we use the same response indicator variables to fit the two models. This allows the effect of the model to be isolated, excluding differences due to random variability in the response indicator.

Response rates for the variables defining probabilities as well as the overall response rates and true population values of the R-indicator under the two models are shown in Table 4.1. For comparison purposes, we report population values of the R-indicator based on both linear and logistic regression models where the response rates range between 25.1% and 35.1% under RR1, between 64.7% and 75.4% under RR2 and between 84.7% and 94.6% under RR3. RR2 represents the type of response rate seen in large-scale national social surveys. As can be seen in Table 4.1, there is little difference in the population values of the R-indicators based on the linear and logistic link function for RR1 and RR2 and a slight difference for RR3 under both models where response rates are in the upper tail of the distribution. We also note that across the very different overall response rates, the population values of the R-indicator are generally high.

When using Model 2, the true R-indicator is always around 0.007 points greater than the corresponding value under Model 1. This is due to the fact that Model 2 for estimating the response propensities is mis-specified. There are fewer auxiliary variables and hence smaller variation in the estimated response propensities which leads to a higher R-indicator. As a consequence we obtain a slightly higher R-indicator for Model 2 as some of the variation is not captured. For this reason, it is always important to report R-indicators together with the auxiliary information used to calculate them since their values depend on the nonresponse model. In addition, we should use covariates that correlate to the survey variables (Schouten et al., 2012).

For each response scenario, five hundred samples were drawn from the population under simple random sampling (SRS) at three different sampling rates 1% $\left(n=\text{3,224}\right),$ 2% $\left(n=\text{6,448}\right)$ and 4% $\left(n=\text{12,896}\right).$ For each sample drawn, a sample response indicator was generated from the “true” population response probability based on the logistic link function. This determines the response set $r.$ Response propensities and R-indicators were then estimated from each sample for both sample and population-based auxiliary variables. Response propensities are estimated in the sample using the “true” model (either Model 1 or Model 2, depending on the scenario).

In order to estimate the variance of population-based estimators, we employ a non-parametric bootstrap algorithm. From each response set, we drew $B=500$ bootstrap samples using simple random sampling (SRS) with replacement. Subsequently, nonresponse was generated in the bootstrap sample by copying the 0-1 sample response indicator values. A replicate of the estimator was computed over each bootstrap sample.

4.2  Results

Table 4.2 presents results of the evaluation study for each response rate scenario, type of model and each sampling rate. We contrast the sample-based R-indicators (under both link functions to highlight any differences) with the population-based R-indicators. In the evaluation, we also investigate the performance of the population-based composite estimator (PC) as shown in (3.7).

For each estimator, Table 4.2 shows: a) the percentage Relative Bias (%RB) calculated as $100\left\{\left[{\displaystyle {\sum }_{j=1}^{500}\left({\widehat{R}}_{\widehat{\rho }j}-R_{\rho }\right)}/R_{\rho }\right]/500\right\},$ where ${\widehat{R}}_{\widehat{\rho }j}$ is the value of the estimator computed for the $j^{\text{th}}$ sample and $R_{\rho }$ is the true R-indicator based on the linear regression model (from Table 4.1), and similarly for ${\tilde{R}}_{{\tilde{\rho }}_{T1}}\text{},$ ${\tilde{R}}_{{\tilde{\rho }}_{T2}}\text{},$ and the composite estimator; b) the Relative Root Mean Square Error (RRMSE) calculated as

$$100\left\{R_{\rho }^{-1}\sqrt{{{\displaystyle {\sum }_{j=1}^{500}\left({\widehat{R}}_{\widehat{\rho }j}-R_{\rho }\right)}}^2/500}\right\}.$$

Table 4.2 shows that differences between the sample-based estimators computed using the linear and the logistic link functions are very small in general, except when the response rates get very close to 1 (RR3).

For sample-based and population-based Type 1 and Type 2 estimators there is a general downward bias in the unadjusted R-indicators and this tends to decrease as the sample size increases for both Models 1 and 2. This is as expected. Sampling error tends to lead to overestimation of the variability of the estimated response propensities and this leads to underestimation of the R-indicator. The degree of underestimation is generally larger for population-based estimators than for the sample-based estimators, especially for higher response rates. The variation of response propensities is larger in this case than the variation under sample-based auxiliary variables. In addition, the RRMSE of the estimators decreases as sample size increases and is generally larger for population-based estimators. Thus, the population-based R-indicators are in general less accurate than their sample-based counterparts and allow for weaker conclusions regarding the nature of response.

In general, the unadjusted population-based composite estimators have a better performance than the corresponding unadjusted population-based estimators, both in terms of %RB and RRMSE, especially for higher response rates. They still show some degree of overestimation under the correct Model 1 for low response rates and underestimation for high response rates. However, for Model 2 we see overestimation.

We now turn to the bias-adjusted estimated R-indicators in Table 4.2. For Type 1, the bias adjustment is able to remove the bias. The analytical bias adjustment for Type 1 population-based estimator works well and generally outperforms the analytical bias adjustment for Type 2 population-based estimates. It seems to pick up most of the bias and provides adjusted estimates that are closer to sample-based R-indicators. The RRMSE for the bias-adjusted estimator is generally similar to the corresponding RRMSE for the unadjusted estimator, meaning that the increase in variability is compensated by the bias reduction. For higher response rates, the adjusted population-based composite estimate reduces the bias and RRMSE of their corresponding population based R-indicators.

In unadjusted form, the Type 2 R-indicator behaves better than the Type 1 R-indicator. This is rather surprising as we seem to be able to have more accurate estimation of the true R-indicator when using less information. The reason for this is that for the Type 1 estimator we do not include any of the sampling variation when we “plug in” the population covariance matrix, whilst for the Type 2 estimator we use only the marginal information and “plug in” the response covariance matrix which accounts for more of the sampling variation. After the bias adjustment, the Type 2 estimators have higher %RB (especially for lower response rates) but similar RRMSE. Type 2 bias adjustment performs worse than the bias adjustment for Type 1 and overcompensates for the bias. This result was expected as the Type 2 bias adjustment is based on a linear approximation, while Type 1 bias adjustment is computed exactly.

Regarding increasing response rates, surprisingly, for the population-based unadjusted estimators, we observe a better performance for lower response rates, both in terms of percentage relative bias (%RB) and RRMSE. The RRMSE of RR3 are 2 to 3 times larger than for RR1. Analytical bias adjustments work very well under all response rates, although with higher RRMSEs for higher response rates. These RRMSEs are reduced by the use of the composite estimators.

Regarding the effect of the number of variables in the model, a lower %RB and RRMSE are observed under Model 2 for unadjusted population-based estimators compared to Model 1. The composite estimators show in general an opposite pattern. The bias-adjusted versions show similar performance under the two models.

Table 4.3 shows the mean of the estimated ${\lambda }_{\text{opt}}$ for the composite population-based Type 1 and Type 2 estimators compared to the true value obtained from the population under the two extreme response rate scenarios, RR1 and RR3. It can be seen that the mean estimated ${\lambda }_{\text{opt}}$ does not deviate greatly from their true values in the evaluation study.

Table 4.4 analyses the performance of the bootstrap estimators for estimating the variance of population-based R-indicators under the two extreme response rate scenarios, RR1 and RR3. Analytical expressions for the variance of sample-based R-indicators have been developed and used in the evaluation study (see Shlomo et al., 2012). Simulation means of the variance estimators are compared in Table 4.4 with the simulation variances (calculated across the replicated samples), using percentage relative bias. The table also includes the Coverage Rate defined as the percentage of times that the true $R_{\rho }$ is contained in the confidence interval $100\left\{\left[{\displaystyle {\sum }_{j=1}^{500}I\left(R_{\rho }\in {\widehat{R}}_{\widehat{\rho }j}\pm \text{1}\text{.96}\sqrt{{\widehat{V}}_j\left({\widehat{R}}_{\widehat{\rho }j}\right)}\right)}\right]/500\right\},$ where ${\widehat{V}}_j\left({\widehat{R}}_{\widehat{\rho }j}\right)$ is the estimated variance for the $j^{\text{th}}$ sample (linearization variance estimator for sample-based estimator and bootstrap variance estimator for population-based estimators) and $I$ is the indicator function. The bootstrap variance estimators for population-based estimators work well. The sample-based estimator show better coverage than the corresponding population-based versions. Type 1 and Type 2 population-based estimators have similar coverages. The coverage always improves as the sample size gets larger.

The behaviour under different response rates is mixed. There seems to be an interaction between sample size and response rate. The number of variables in the model does not have a large impact on coverage. However, we observe problems with coverage for the population-based estimators under the highest response rate (RR3), especially for the 1% sample rate.

Figures 4.1, 4.2 and 4.3 present box plots comparing the estimators and their bias adjusted versions when fitting Model 1, under different response rate scenarios RR1, RR2 and RR3 respectively. The gains from the bias adjustments are evident for Type 1 and Type 2 R-indicators. Standard errors for RR3 are much larger than for RR1 under the same sampling rates. The variability of the bias-adjusted estimator increases and it is larger for smaller sample sizes.

• Previous
• Article main page
• Next