5. Simulations

Jianqiang C. Wang, Jean D. Opsomer and Haonan Wang

Previous | Next

To evaluate the practical behavior of bagging in the survey context, we generate a finite population of size $N=2,000$  with three strata. The size of each stratum is denoted as $N_h$  with $h=1,2,3,$  and the stratum proportions are fixed at $\left(N_1\mathrm{,}{N}_2\mathrm{,}{N}_3\right)/N=\left(\mathrm{0.5,0.3,0.2}\right).$  The distribution of the target variable $y_i$  within each stratum is $y_{1i}\sim \left|N\left(-\mathrm{1,1}\right)\right|\mathrm{,}{y}_{2i}\sim \Gamma \left(\mathrm{1,1}\right)$  and $y_{3i}\sim \left|N\left(\mathrm{3,2}\right)\right|.$  An auxiliary variable $x_i$  is generated via $x_i=A_0+A_1{y}_i+A_2\left(G_i-\alpha /\beta \right)$  where $A_0=A_1=2,A_2=1,\alpha =2,\beta =1$  and $G_i\stackrel{iid}{\sim }\Gamma \left(\mathrm{2,1}\right).$  We repeatedly draw samples of size $n$  using stratified simple random sampling from the population of interest and the sample size allocation is $\left(n_1\mathrm{,}{n}_2\mathrm{,}{n}_3\right)/n=\left(\mathrm{0.3,0.3,0.4}\right).$  In this set-up, the design is clearly informative, because the observations are not $iid$  in the overall population and are correlated with the inclusion probabilities.

We are interested in three population quantities: a population $\alpha$ -quantile, a population proportion below a given fraction of a population quantile (see Berger and Skinner 2003, for an example) and the Rao-Kovar-Mantel (RKM) estimator of the distribution function (Rao et al. 1990). The former is an example of a non-differentiable estimating equation-based estimator, while the latter two are explicitly defined non-differentiable estimators. The sample estimator of the quantile is found by inverting the estimated cumulative distribution function. The sample estimator of the proportion below a given fraction of a population quantile is the HT estimator of the proportion of observations below the sample median of a variable of interest times a constant $c,$

$${\widehat{\theta }}_{pr}=\frac{1}{N}{\displaystyle \sum _{i\in A}}\frac{1}{{\pi }_i}{\text{I}}_{\left(y_i\le c{\widehat{\theta }}_{\text{med}}\right)}\mathrm{,}$$

where ${\widehat{\theta }}_{\text{med}}$  denotes the sample median of the $y_i.$  The design-based RKM difference estimator based on a ratio model is

$${\widehat{\theta }}_{\text{RKM}}=\frac{1}{N}\left\{{\displaystyle \sum _{i\in A}}\frac{1}{{\pi }_i}{\text{I}}_{\left(y_i\le t\right)}+{\displaystyle \sum _{i=1}^N}{\text{I}}_{\left(\widehat{R}{x}_i\le t\right)}-{\displaystyle \sum _{i\in A}}\frac{1}{{\pi }_i}{\text{I}}_{\left(\widehat{R}{x}_i\le t\right)}\right\}\mathrm{,}(5.1)$$

where $\widehat{R}$  denotes the estimated ratio between $y$  and $x.$

The design variance of these non-differentiable estimators is somewhat cumbersome to estimate. For variance and interval calculations for sample quantiles, the readers are referred to Francisco and Fuller (1991), Sitter and Wu (2001), and references therein. For proportion below an estimated level, see Shao and Rao (1993) and Berger and Skinner (2003).

The design variances of the original estimators ${\widehat{\theta }}_{qr},{\widehat{\theta }}_{pr}$  and ${\widehat{\theta }}_{\text{RKM}},$  are estimated via the without-replacement bootstrap procedure described in the previous section. We employ a bootstrap sample size of $k_h=n_h/2.$  The so-constructed bagging estimators are often referred to as subagging estimators (Bühlmann and Yu 2002). It was established that without-replacement samples of size $n/2$  produces similar results to with replacement samples of size $n$  in bagging (Buja and Stuetzle 2006; Friedman and Hall 2007). We apply the two variance approaches for bagging estimators proposed in the previous section, i.e. one identical to that of unbagged estimator (Var. 1) and another one that multiplies the original variance estimate by a model-based adjustment factor (Var. 2). The factor is determined by double bootstrap on one particular sample. In principle, one should repeat the exercise for each sample, but this is precluded by the heavy computational burden. The confidence intervals of all three estimators are constructed by normal approximation. The confidence intervals for the proportion and the RKM estimator are constructed by normal approximation on $logit$  transformed scale, $\log \left[\widehat{\theta }/\left(1-\widehat{\theta }\right)\right]$  or $\log \left[{\widehat{\theta }}_{bag}/\left(1-{\widehat{\theta }}_{bag}\right)\right],$  and then back transformation (Agresti 2002; Korn and Graubard 1998).

Table 5.1 summarizes the bias, standard deviation and MSE ratio of the original and bagged sample quantiles and Table 5.2 examines the variance estimators and confidence intervals. The sample sizes are chosen to be $n=100$  and $200.$  From Table 5.1, we can see that the bagged quantile estimator is more efficient than the original estimator since the MSE ratio is less than one in this simulation experiment. The smoothing effects of bagging generally become more prominent as we decrease the sample size. In Table 5.2, we compare the two confidence intervals with bagging point estimator to that of original confidence intervals. As expected, the confidence interval constructed via method 1 has the same length and higher coverage than the original. In this example, the confidence intervals via method 2 are narrower but maintain coverage level close to nominal.

Table 5.1
Bias, standard deviation and MSE ratios of sample quantiles and bagged sample quantiles; population size $N=2,000,$ number of bootstraps $B=2,000,$ and results are from $2,000$ simulations
Table summary
This table displays the results of Bias. The information is grouped by $\alpha$ (appearing as row headers), $n=100,k=50$ and $n=200,k=100$ (appearing as column headers).

$\alpha$	$n=100,k=50$					$n=200,k=100$
	0.2	0.3	0.5	0.7	0.8	0.2	0.3	0.5	0.7	0.8
$\text{bias}\left({\widehat{\theta }}_{qt}\right)$	0.002	0.008	0.000	-0.005	-0.035	-0.008	0.005	0.006	0.007	-0.005
$\text{bias}\left({\widehat{\theta }}_{qt,bag}\right)$	0.018	0.019	-0.001	-0.007	-0.043	-0.006	0.009	0.005	0.006	-0.022
$\text{sd}\left({\widehat{\theta }}_{qt}\right)$	0.093	0.124	0.149	0.181	0.212	0.07	0.076	0.103	0.136	0.148
$\text{sd}\left({\widehat{\theta }}_{qt,bag}\right)$	0.089	0.112	0.138	0.167	0.197	0.065	0.073	0.099	0.127	0.139
$\frac{MS{E}_p\left({\widehat{\theta }}_{qt,bag}\right)}{MS{E}_p\left({\widehat{\theta }}_{qt}\right)}$	0.946	0.844	0.859	0.854	0.875	0.866	0.924	0.919	0.862	0.912

 

Table 5.2
Relative bias, coverage probability and confidence interval width of bootstrap variance estimators for sample quantiles and unadjusted $\left({\widehat{V}}_1\right)$ and adjusted $\left({\widehat{V}}_2\right)$ variance estimators for bagged sample quantiles; simulation setting is the same as in Table 5.1
Table summary
This table displays the results of Relative bias. The information is grouped by $\alpha$ (appearing as row headers), $n=100,k=50$ and $n=200,k=100$ (appearing as column headers).

$\alpha$	$n=100,k=50$					$n=200,k=100$
	0.2	0.3	0.5	0.7	0.8	0.2	0.3	0.5	0.7	0.8
$\frac{\text{E}\left[{\widehat{V}}_{boot}\left({\widehat{\theta }}_{qt}\right)\right]}{V\left({\widehat{\theta }}_{qt}\right)}$	1.208	1.091	1.099	1.135	1.205	1.067	1.117	1.093	1.098	1.18
$\frac{\text{E}\left[{\widehat{V}}_1\left({\widehat{\theta }}_{qt\mathrm{,}bag}\right)\right]}{V\left({\widehat{\theta }}_{qt\mathrm{,}bag}\right)}$	1.327	1.325	1.279	1.331	1.402	1.224	1.217	1.188	1.273	1.326
$\frac{\text{E}\left[{\widehat{V}}_2\left({\widehat{\theta }}_{qt\mathrm{,}bag}\right)\right]}{V\left({\widehat{\theta }}_{qt\mathrm{,}bag}\right)}$	1.307	1.217	1.196	1.184	1.383	1.245	1.249	1.392	1.107	1.104
$\text{C}\text{.P}\text{.(C}\text{.I}\mathrm{.\; <mo\; stretchy="false">)</mo>}$	0.944	0.934	0.924	0.928	0.922	0.938	0.951	0.942	0.935	0.95
$\text{C}\mathrm{.}\text{P}\text{.}\left(\text{C}\mathrm{.}\text{I}{\mathrm{.1.}}_{bag}\right)$	0.95	0.946	0.938	0.938	0.939	0.942	0.95	0.946	0.943	0.954
$\text{C}\text{.P}\text{.}\left(\text{C}\text{.I}{\mathrm{.2.}}_{bag}\right)$	0.949	0.934	0.932	0.929	0.938	0.944	0.952	0.958	0.927	0.936
$\text{Width(C}\text{.I}\text{.)}$
$\text{Width}\left(\text{C}\text{.I}{\mathrm{.1.}}_{bag}\right)$	0.386	0.492	0.597	0.729	0.88	0.277	0.309	0.414	0.544	0.612
$\text{Width}\left(\text{C}\text{.I}{\mathrm{.2.}}_{bag}\right)$	0.383	0.472	0.577	0.688	0.874	0.279	0.313	0.448	0.508	0.559

Tables 5.3 and 5.4 summarize design-based results on the low-income proportion estimator. Based on the MSE ratio, we can see that the bagging estimator is uniformly more efficient than the original estimator, and the MSE of bagging estimator is less than 50% of that of original estimator in a few cases (see $c=1.2$). The likely reason for this is that the estimator involves two “levels” of non-differentiability: the sample median being a non-differentiable estimator, whose efficiency gain was shown in Table 5.1, and the low-income proportion being a non-differentiable function of the sample median. The “jumps” in the estimators are smoothed out by bagging, resulting in a more stable estimator. The confidence interval comparison in Table 5.4 leads to results similar to the quantile case.

Table 5.3
Bias, standard deviation and MSE ratio of estimated proportion below a constant $c$ multiplied by estimated median and the bagged proportion estimator; population size $N=2,000,$ number of bootstraps $B=2,000,$ and results are from $2,000$ simulations
Table summary
This table displays the results of Bias. The information is grouped by $c$ (appearing as row headers), $n=100,k=50$ and $n=200,k=100$ (appearing as column headers).

$c$	$n=100,k=50$					$n=200,k=100$
	0.2	0.4	0.6	1.2	1.5	0.2	0.4	0.6	1.2	1.5
$\text{bias}\left({\widehat{\theta }}_{pr}\right)$	-0.002	-0.002	-0.003	0.011	0.006	0	-0.002	-0.005	-0.004	-0.004
$\text{bias}\left({\widehat{\theta }}_{pr,bag}\right)$	-0.004	-0.004	-0.007	0.017	0.009	-0.001	-0.005	-0.009	-0.001	-0.004
$\text{sd}\left({\widehat{\theta }}_{pr}\right)$	0.034	0.039	0.038	0.034	0.046	0.023	0.027	0.026	0.026	0.036
$\text{sd}\left({\widehat{\theta }}_{pr,bag}\right)$	0.031	0.035	0.031	0.02	0.034	0.022	0.025	0.022	0.017	0.029
$\frac{MS{E}_p\left({\widehat{\theta }}_{pr,bag}\right)}{MS{E}_p\left({\widehat{\theta }}_{pr}\right)}$	0.861	0.821	0.709	0.538	0.581	0.883	0.86	0.783	0.434	0.671

 

Table 5.4
Relative bias, coverage probability and confidence interval width of bootstrap variance estimators for sample proportions and unadjusted $\left({\widehat{V}}_1\right)$ and adjusted $\left({\widehat{V}}_2\right)$ variance estimators for bagged sample proportions; simulation setting is the same as in Table 5.3. We use “C.I.T.” to denote confidence intervals obtained with logit transformation
Table summary
This table displays the results of Relative bias. The information is grouped by $c$ (appearing as row headers), $n=100,k=50$ and $n=200,k=100$ (appearing as column headers).

$c$	$n=100,k=50$					$n=200,k=100$
	0.2	0.4	0.6	1.2	1.5	0.2	0.4	0.6	1.2	1.5
$\frac{\text{E}\left[{\widehat{V}}_{boot}\left({\widehat{\theta }}_{pr}\right)\right]}{V\left({\widehat{\theta }}_{pr}\right)}$	1.122	1.191	1.325	1.472	1.281	1.14	1.191	1.251	1.35	1.217
$\frac{\text{E}\left[{\widehat{V}}_1\left({\widehat{\theta }}_{pr\mathrm{,}bag}\right)\right]}{V\left({\widehat{\theta }}_{pr\mathrm{,}bag}\right)}$	1.323	1.471	1.959	4.095	2.307	1.293	1.428	1.766	3.064	1.821
$\frac{\text{E}\left[{\widehat{V}}_2\left({\widehat{\theta }}_{pr\mathrm{,}bag}\right)\right]}{V\left({\widehat{\theta }}_{pr\mathrm{,}bag}\right)}$	1.24	0.963	1.19	1.174	1.149	1.145	1.262	1.319	2.039	1.524
$\text{C}\text{.P}\text{.(C}\text{.I}\text{.T}\mathrm{.\; <mo\; stretchy="false">)</mo>}$	0.969	0.97	0.984	0.991	0.98	0.964	0.974	0.977	0.983	0.946
$\text{C}\mathrm{.}\text{P}\text{.}\left(\text{C}\mathrm{.}\text{I}\text{.T}{\mathrm{.1.}}_{bag}\right)$	0.979	0.983	0.995	0.998	0.995	0.974	0.98	0.988	0.998	0.976
$\text{C}\text{.P}\text{.}\left(\text{C}\text{.I}\text{.T}{\mathrm{.2.}}_{bag}\right)$	0.976	0.944	0.973	0.922	0.942	0.962	0.969	0.968	0.993	0.957
$\text{Width(C}\text{.I}\text{.T)}$
$\text{Width}\left(\text{C}\text{.I}\text{.T}{\mathrm{.1.}}_{bag}\right)$	0.144	0.166	0.168	0.157	0.197	0.098	0.115	0.114	0.113	0.149
$\text{Width}\left(\text{C}\text{.I}\text{.T}{\mathrm{.2.}}_{bag}\right)$	0.139	0.134	0.131	0.085	0.14	0.093	0.108	0.099	0.092	0.136

Tables 5.5 and 5.6 summarize the design-based results on the RKM estimator. Again, we observe the efficiency gain by applying the bagging method, and the gain is between 2% and 12%. Both variance estimators of the bagging quantity perform quite well. Both versions of confidence intervals for bagging estimators have actual coverage rates close to 95%, and the confidence intervals using the adjustment factor approach (Var. 2) are slightly shorter than method 1.

Table 5.5
Bias, standard deviation and MSE ratios of RKM estimator and bagging RKM estimator (5.1); population size $N=2,000,$ number of bootstraps $B=2,000,$ and results are from $2,000$ simulations
Table summary
This table displays the results of Bias. The information is grouped by $t$ (appearing as row headers), $n=100,k=50$ and $n=200,k=100$ (appearing as column headers).

$t$	$n=100,k=50$					$n=200,k=100$
	0.5	1.5	2.5	3.5	4.5	0.5	1.5	2.5	3.5	4.5
$\text{bias}\left({\widehat{\theta }}_{\text{RKM}}\right)$	0.000	0.000	0.000	0.000	0.000	-0.001	0.001	0.000	0.000	0.001
$\text{bias}\left({\widehat{\theta }}_{\text{RKM},bag}\right)$	-0.001	0.000	-0.001	0.000	0.000	-0.001	0.001	0.000	0.001	0.001
$\text{sd}\left({\widehat{\theta }}_{\text{RKM}}\right)$	0.043	0.044	0.03	0.015	0.012	0.03	0.03	0.02	0.011	0.009
$\text{sd}\left({\widehat{\theta }}_{\text{RKM},bag}\right)$	0.042	0.042	0.028	0.014	0.012	0.03	0.029	0.019	0.011	0.009
$\frac{MS{E}_p\left({\widehat{\theta }}_{\text{RKM},bag}\right)}{MS{E}_p\left({\widehat{\theta }}_{\text{RKM}}\right)}$	0.965	0.911	0.877	0.914	0.917	0.976	0.928	0.917	0.918	0.981

 

Table 5.6
Relative bias, coverage probability and confidence interval width of bootstrap variance estimators for the RKM estimator (5.1) and unadjusted $\left({\widehat{V}}_1\right)$ and adjusted $\left({\widehat{V}}_2\right)$ variance estimators for bagging RKM estimators; simulation setting is the same as in Table 5.5
Table summary
This table displays the results of Relative bias. The information is grouped by $t$ (appearing as row headers), $n=100,k=50$ and $n=200,k=100$ (appearing as column headers).

$t$	$n=100,k=50$					$n=200,k=100$
	0.5	1.5	2.5	3.5	4.5	0.5	1.5	2.5	3.5	4.5
$\frac{\text{E}\left[{\widehat{V}}_{boot}\left({\widehat{\theta }}_{\text{RKM}}\right)\right]}{V\left({\widehat{\theta }}_{\text{RKM}}\right)}$	1.081	1.192	1.078	1.082	1.078	1.016	1.045	1.138	1.121	1.016
$\frac{\text{E}\left[{\widehat{V}}_1\left({\widehat{\theta }}_{\text{RKM}\mathrm{,}bag}\right)\right]}{V\left({\widehat{\theta }}_{\text{RKM}\mathrm{,}bag}\right)}$	1.115	1.324	1.183	1.198	1.156	1.038	1.138	1.223	1.21	1.062
$\frac{\text{E}\left[{\widehat{V}}_2\left({\widehat{\theta }}_{\text{RKM}\mathrm{,}bag}\right)\right]}{V\left({\widehat{\theta }}_{\text{RKM}\mathrm{,}bag}\right)}$	1.087	1.117	0.962	1.042	1.019	1.009	1.083	1.106	1.118	1.002
$\text{C}\text{.P}\text{.(C}\text{.I}\mathrm{.\; <mo\; stretchy="false">)</mo>}$	0.958	0.963	0.955	0.956	0.959	0.954	0.956	0.966	0.964	0.948
$\text{C}\mathrm{.}\text{P}\text{.}\left(\text{C}\mathrm{.}\text{I}{\mathrm{.1.}}_{bag}\right)$	0.958	0.968	0.958	0.967	0.964	0.958	0.964	0.97	0.97	0.956
$\text{C}\text{.P}\text{.}\left(\text{C}\text{.I}{\mathrm{.2.}}_{bag}\right)$	0.957	0.954	0.937	0.951	0.95	0.955	0.958	0.959	0.96	0.948
$\text{Width(C}\text{.I}\text{.)}$
$\text{Width}\left(\text{C}\text{.I}{\mathrm{.1.}}_{bag}\right)$	0.171	0.183	0.116	0.074	0.052	0.122	0.122	0.083	0.049	0.034
$\text{Width}\left(\text{C}\text{.I}{\mathrm{.2.}}_{bag}\right)$	0.169	0.168	0.105	0.069	0.049	0.12	0.12	0.079	0.047	0.033

In the context of nonsmooth estimators such as those considered here, it is often recommended that one uses a smoothed bootstrap instead of the simple bootstrap in variance estimation. We considered perturbing each resampled observation $y_{hi}^{*}$  in the $h\text{-th}$  stratum to obtain,

$${\tilde{y}}_{hi}^{*}={\overline{y}}_h+{\left(1+{\sigma }_Z^2\right)}^{-1/2}\left(y_{hi}^{*}-{\overline{y}}_h+s_h{Z}^{*}\right)\mathrm{,}(5.2)$$

where ${\overline{y}}_h\mathrm{,}{s}_h$  denote the sample mean and standard deviation of the original sample stratum, $y_{hi}^{\ast }$  denotes the originally resampled value and $Z^{\ast }$  denotes random noise with $Z^{\ast }\stackrel{iid}{\sim }N\left(\mathrm{0,}{\sigma }_Z^2\right).$  The variance of $Z^{\ast }$  controls the amount of smoothing. We applied this method to quantile estimation and the proportion below an estimated level, but it did not appear to improve the performance of the estimation procedure. One possible explanation is that noise contamination “jitters” duplicated observations arising from with-replacement sample and stabilizes subsequent variance estimator to some extent. Since we used without-replacement sampling, this problem was already mostly avoided. More careful study is necessary to understand the effect of smoothing in the context.

Previous | Next