Model-assisted calibration of non-probability sample survey data using adaptive LASSO
Section 4. Simulation study

• Previous
• Next

We design a simulation to evaluate the finite sample properties of ${\widehat{T}}_y^{\text{LASSO}}$ and the asymptotic variance estimates of ${\widehat{T}}_y^{\text{LASSO}}\text{},$ $v^{\text{LASSO}}$ and $v_g^{\text{LASSO}}\text{}.$ We also consider a naive bootstrap estimator $v_{\text{boot}}^{\text{LASSO}}\text{},$ obtained by drawing 500 samples with replacement from each simulation sample, as an alternative variance estimator of ${\widehat{T}}_y^{\text{LASSO}}\text{}.$

To simulate non-probability samples, we generate samples with unequal selection probabilities, but set design weights to $d^A\mathrm{<mo>=</mo>}N/n.$ We also consider ${\widehat{T}}_y^{\text{GREG}}$ (traditional calibration estimator) and ${\widehat{T}}_y^{\text{HT}}$ (pure design-based Horvitz-Thompson estimator). Because ${\widehat{T}}_y^{\text{LASSO}}$ performs both variable selection and estimation, we implement a backward stepwise selection to select the working model for GREG. Although there is no theoretical justification for using stepwise variable selection, Skinner and Silva (1997) have shown that given two auxiliary variables, a stepwise procedure can result in improved efficiency of GREG estimator. We are interested in knowing the performance of each estimator under (1) populations with different signal-to-noise-ratios (SNR), (2) independent, informative, and biased sampling schemes, and (3) small and large sample sizes. The signal-to-noise ratio is calculated according to definitions in Czanner, Sarma, Eden and Brown (2008). We set two levels of correlations (low/high) between covariates, crossed with two levels of effect sizes (low/high) of the covariates. We set the low/high and high/low populations to have the same SNR in order to understand the influence of correlation and effect size on estimator’s performance given the same SNR. Three sampling schemes are used to draw samples: simple-random-sampling without replacement, SRS, Poisson sampling with selection probabilities proportional to covariates, $\text{POI}\left(\text{X}\right),$ and Poisson sampling with selection probabilities proportional to covariates and the outcome, $\text{POI}\left(\text{X+Y}\right).$ $\text{POI}\left(\text{X+Y}\right)$ sampling simulates self-selection bias of non-probability samples, where the propensity of a respondent to participate in a study relates to the analysis variable. We consider two sample sizes: 250 and 1,000. Thus we have a total of $2\times 2\times 3\times \mathrm{2\; <mo>=</mo>\; 24}$ experimental groups.

4.1  Population

To create collinearity among covariates, we follow an auto-decay correlation structure commonly used in LASSO-related simulations (Tibshirani, 1996): $\text{cor}\left(X_i\mathrm{,}{X}_j\right)\mathrm{<mo>=</mo>}{\rho }^{\mathrm{<mo>|</mo>}i-j\mathrm{<mo>|</mo>}},i\mathrm{<mo>=</mo>\; 1,}\dots \mathrm{,}p.$ We generate a population of size $N\mathrm{<mo>=</mo>}$100,000 from a multivariate normal distribution with mean $0_{\left(p\times 1\right)}$ and covariance ${\Sigma }^{\rho }\text{},$ $p\mathrm{<mo>=</mo>}$  40. The continuous outcome variable is generated by the regression model:

$$y_i\mathrm{<mo>=</mo>}{\beta }_0+{\beta }_1{x}_{i1}+{\beta }_2{x}_{i2}+\dots +{\beta }_{40}{x}_{i40}+N\left(\mathrm{0,3}\right).$$

The binary outcome variable is generated by the logistic regression model:

$$\begin{array}{ll}{\varphi }_i\hfill & \mathrm{<mo>=</mo>}\text{expit}\left({\beta }_0+{\beta }_1{x}_{i1}+{\beta }_2{x}_{i2}+\dots +{\beta }_{40}{x}_{i40}\right)\mathrm{,}\text{expit}\left(u\right)\mathrm{<mo>=</mo>}{\left(1+\text{exp}\left(u\right)\right)}^{-1}\hfill \\ y_i\hfill & \mathrm{<mo>=</mo>}\text{bernoulli}\left({\varphi }_i\right).\hfill \end{array}$$

We set $\rho \mathrm{<mo>=</mo>}$0.15 for low correlation population, and $\rho \mathrm{<mo>=</mo>}$0.73 for high correlation population. For both continuous and binary outcome variables:

$$\begin{array}{lll}\text{Low}\text{effect-size}\hfill & {\beta }^{\left(1\right)}\hfill & \mathrm{<mo>:=</mo>}{\beta }_{12}\dots {\beta }_{19}\mathrm{,}{\beta }_{32}\dots {\beta }_{39}\mathrm{<mo>=</mo>}\text{0}\text{.45}\hfill \\ \text{High}\text{effect-size}\hfill & {\beta }^{\left(1\right)}\hfill & \mathrm{<mo>:=</mo>}{\beta }_{12}\dots {\beta }_{19}\mathrm{,}{\beta }_{32}\dots {\beta }_{39}\mathrm{<mo>=</mo>}\text{0}\text{.74}\text{.}\hfill \end{array}$$

For continuous $y\text{:}$ ${\beta }_0\mathrm{<mo>=</mo>\; 1},$ for binary $y\text{:}$ ${\beta }_0\mathrm{<mo>=</mo>}\text{0}\text{.4}.$ The rest of ${\beta }_i\mathrm{<mo>=</mo>\; 0.}$ Out of 41 regression parameters, 16 are non-zero and 25 are zero.

4.2  Sampling schemes

Three sampling schemes are used to generate the sample:

1. Simple-Random-Sampling (SRS): selection probabilities  $=n/N.$
2. Poisson sampling with probabilities proportional to $X,$ $\text{POI}\left(\text{X}\right)$

$$\{\begin{array}{ll}\text{continuous}y\mathrm{<mo>:</mo>}\hfill & {\pi }_i\propto \text{0}\text{.4}+\text{0}\text{.4}{x}_{i5}+\text{0}\text{.4}{x}_{i15}+\text{0}\text{.4}{x}_{i25}+\text{0}\text{.4}{x}_{i35}\hfill \\ \text{binary}y\mathrm{<mo>:</mo>}\hfill & \text{logit}\left({\pi }_i\right)=\text{0}\text{.4}+\text{0}\text{.4}{x}_{i5}+\text{0}\text{.4}{x}_{i15}+\text{0}\text{.4}{x}_{i25}+\text{0}\text{.4}{x}_{i35}.\hfill \end{array}$$

3. Poisson sampling with probabilities proportional to $X$ and $y,$ $\text{POI}\left(\text{X+Y}\right)$

$$\{\begin{array}{ll}\text{continuous}y\mathrm{<mo>:</mo>}\hfill & {\pi }_i\propto \text{0}\text{.4}+\text{0}\text{.4}{x}_{i5}+\text{0}\text{.4}{x}_{i15}+\text{0}\text{.4}{x}_{i25}+\text{0}\text{.4}{x}_{i35}+\text{0}\text{.5}{y}_i\hfill \\ \text{binary}y\mathrm{<mo>:</mo>}\hfill & \text{logit}\left({\pi }_i\right)\propto \text{0}\text{.4}+\text{0}\text{.4}{x}_{i5}+\text{0}\text{.4}{x}_{i15}+\text{0}\text{.4}{x}_{i25}+\text{0}\text{.4}{x}_{i35}+y_i.\hfill \end{array}$$

4.3  Evaluation metrics

We evaluate empirical bias, variance, and RMSE for each estimator of total. We evaluate the asymptotic variance estimates and bootstrap variance estimates by their 95% nominal coverage and %bias relative to empirical variance. We use the normal approximation to generate confidence intervals. We calculate %bias as $\mathrm{<mi>\%</mi>}\text{bias}=100\left[v-\text{var}\left({\widehat{T}}_y^{\text{LASSO}}\right)\right]/\text{var}\left({\widehat{T}}_y^{\text{LASSO}}\right)\mathrm{,}$ where $\text{var}\left({\widehat{T}}_y^{\text{LASSO}}\right)$ is the empirical variance obtained from the simulation samples.

4.4  Simulation results

The simulation results are based on $S\mathrm{<mo>=</mo>}$1,000 simulated samples per each experimental group. Table 4.1 lists the numerical results of bias, variance, and root-mean-square-error of each estimator under different experimental designs for estimating the total of a continuous outcome variable. Table 4.2 lists the numerical results for estimating the total of a binary outcome variable.

4.4.1  Root mean square error

Under SRS, all estimators are unbiased, and LASSO and GREG perform approximately equally well relative to HT. $\text{POI}\left(\text{X}\right)$ and $\text{POI}\left(\text{X+Y}\right)$ induce biased samples by selecting cases with larger covariate values with higher probabilities. Under $\text{POI}\left(\text{X+Y}\right),$ the selection also favors cases with larger outcome values. The absolute bias of LASSO decreases relative to GREG as SNR increases. This improvement is more dramatic in the binary case than the continuous case, especially for $\text{POI}\left(\text{X+Y}\right).$ In terms of RMSE, LASSO has marginal improvement over GREG for estimating totals of continuous outcome variables. The improvement is slightly noticeable, about 3%, when there are highly correlated predictors in the model. For the binary setting, there is substantial improvement in MSE for LASSO over GREG as SNR increases, with reductions of 20% for the $\text{POI}\left(\text{X}\right)$ and nearly 50% for the $\text{POI}\left(\text{X+Y}\right)$ setting when SNR is large. In particular, under Low/High and High/Low population types, the SNR is the same, thus the difference in performance between LASSO and GREG is attributed to correlation or effect size. LASSO performs better in both bias and RMSE in High/Low population type, suggesting that LASSO has stronger advantage over GREG when there are highly correlated predictors in the model. This suggests that LASSO has a better variable selection capability in the presence of multicollinearity relative to stepwise variable selection procedure used in GREG.

4.4.2  LASSO variance estimates

Tables 4.3 and 4.4 list the 95% nominal coverage and percent-bias for each of the two asymptotic closed-form variance estimators developed in this research, as well as the naive bootstrap variance estimate of the LASSO calibration estimator.

For continuous outcomes, bootstrap variances have coverages that are consistently close to 95% under SRS and $\text{POI}\left(\text{X}\right)$ sampling schemes for both sample sizes. Under $\text{POI}\left(\text{X+Y}\right)$ sampling scheme, there is very modest undercoverage in Table 4.3. The closed-form variances have coverages that are sensitive to both sample size and sampling scheme, with smaller samples tending to undercover, particularly for the $\text{POI}\left(\text{X+Y}\right)$ sampling scheme. The difference in coverage of variance estimates between small and large sample sizes is expected, since the variance estimates are asymptotic and improve over larger samples. In terms of bias of variance estimators, there is evidence that bias reduces as SNR increases. With the same SNR, both asymptotic closed-form and bootstrap variances have smaller bias given predictors with high correlations relative to predictors with high effect sizes. Closed-form variances tend to underestimate the empirical variance, especially when the sample size is small. Overall, there is very little difference between the two closed-form variance estimates. Bootstrap variance tends to overestimate the empirical variance, but the absolute bias is generally smaller than those of the closed-form variance estimates.

For binary outcomes, both asymptotic closed-form and bootstrap variance estimates are sensitive to sample size, sampling scheme, and SNR. Bootstrap variance coverages are consistently close to 95% under SRS and $\text{POI}\left(\text{X}\right)$ for both sample sizes and all population types, but coverages range from 75% to 94% under $\text{POI}\left(\text{X+Y}\right).$ Under $\text{POI}\left(\text{X+Y}\right),$ the bootstrap variance coverages are better with sample size 250 than with sample size 1,000 when the bias becomes a larger part of the RMSE, and better with high-correlation populations than with low-correlation populations. In terms of coverage, closed-form variances show a similar trend under $\text{POI}\left(\text{X+Y}\right)$ as bootstrap: better coverage with smaller samples than bigger samples, and better coverage with high-correlation populations than with low-correlation populations. Under SRS and $\text{POI}\left(\text{X}\right),$ closed-form variance coverage improves as sample size increases. In terms of bias, both bootstrap and closed-form variances have smaller bias with larger sample sizes. Holding sample size fixed, closed-form variance estimates have larger bias as SNR increases. The same trend is not observed in bootstrap variance estimates. Similar to continuous outcome results, closed-form variance tends to underestimate the empirical variance, especially when the sample size is small. Unlike continuous outcome results, there is evidence that the $g-$ weighted closed-form variance estimates have better bias-properties than unweighted closed-form variance estimates. The bootstrap variance tends to overestimate the empirical variance. However, the biases are much smaller than for the closed-form variance estimates.

• Previous
• Article main page
• Next