6. Conclusions

Jianqiang C. Wang, Jean D. Opsomer and Haonan Wang

Previous

In this article, we have explored the use of bagging procedures for nonlinear and non-differentiable survey estimators. We presented theoretical results on bagging estimator both under design-based and model-based framework. The bagging estimator can be treated as the expectation of a two-phase estimator conditioning on the first phase, and this expectation smoothes out “jumps” in the non-differentiable estimator. The empirical study has revealed the potential of bagging non-differentiable survey estimators, and while the relative performance of bagging varies from one scenario to another, the results are certainly promising.

How to estimate the variance of bagged survey estimators remains an open question when the sampling design is a general complex design. We have proposed two ideas for variance estimation for practical use, but further theoretical study of variance estimation under design-based framework is certainly warranted.

Appendix

A.1  Design-based theory

Assumptions D.1-D.6 are used to show the design-based results given below (Theorems 3 and 4). Assumption D.1 specifies moment conditions on the study variable $y_i,$  and Assumption D.2 specifies conditions on the second order inclusion probability of the sampling design. Assumption D.3 guarantees that the size of each resample converges to infinity in the limit. Assumption D.4 specifies smoothness conditions on $m(\cdot )$  in the differentiable estimator. Assumptions D.5-D.6 are used to show the design consistency of bagging non-differentiable survey estimators.

The following theorem gives several asymptotic approximations for the bagged estimator, depending on the rate of convergence of $k$  relative to $n.$  In all three cases, the bagged estimator is design consistent. Intuitively speaking, the bagging estimator behaves like the original estimator when the resample size $k$  is large (approaches infinity no slower than $n^{1/2}$  ), but converges at a different speed when the resample size is small.

Theorem 3 Under Assumptions D.1-D.4, the bagged differentiable estimator ${\widehat{\theta }}_{d\mathrm{,}bag}$  admits the following second-order expansion,

${\widehat{\theta }}_{d\mathrm{,}bag}-{\theta }_d=\{\begin{array}{ll}{\left\{m^{\prime }\left({\mu }_N\right)\right\}}^T\left(\widehat{\mu }-{\mu }_N\right)+o_p\left(n^{-1/2}\right)\mathrm{,}\hfill & for\kappa >1/2\hfill \\ {\left\{m^{\prime }\left({\mu }_N\right)\right\}}^T\left(\widehat{\mu }-{\mu }_N\right)+\hfill & \hfill \\ \frac{1}{2\left(\begin{array}{c}n\\ k\end{array}\right)}{\displaystyle \sum }{\left(\widehat{\mu }\left(Y_b^{*}\right)-{\mu }_N\right)}^T{m}^{″}\left({\mu }_N\right)\left(\widehat{\mu }\left(Y_b^{*}\right)-{\mu }_N\right)+o_p\left(n^{-1/2}\right),\hfill & for\kappa =1/2\hfill \\ \frac{1}{2\left(\begin{array}{c}n\\ k\end{array}\right)}{\displaystyle \sum }{\left(\widehat{\mu }\left(Y_b^{*}\right)-{\mu }_N\right)}^T{m}^{″}\left({\mu }_N\right)\left(\widehat{\mu }\left(Y_b^{*}\right)-{\mu }_N\right)+o_p\left(k^{-1}\right)\mathrm{,}\hfill & for\kappa <1/2\hfill \end{array}$

where $\kappa >0$  is such that the resample size $k=O\left(n^{\kappa }\right).$

Proof of Theorem 3:

The proof easily follows from a Taylor expansion of the individual resample-based estimator $m\left(\widehat{\mu }\left(Y_b^{\ast }\right)\right)$  around ${\mu }_N.$  The linear expansion term reduces to ${\left\{m^{\prime }\left({\mu }_N\right)\right\}}^T\left(\widehat{\mu }-{\mu }_N\right)$  based on an earlier argument. Under D.1 and D.3, the quadratic term has the same order as the SRSWOR variance of $\widehat{\mu }\left(Y_b^{\ast }\right)$  and hence is $o_p\left(1/k\right).$

Next, Theorem 4 gives the design consistency of the non-differentiable bagged estimator.

Theorem 4 Under Assumptions D.1-D.3 and D.5-D.6, the bagged non-differentiable estimator ${\widehat{\theta }}_{nd\mathrm{,}bag}$  is design consistent for its population target ${\theta }_{nd},$  i.e., ${\widehat{\theta }}_{nd\mathrm{,}bag}-{\theta }_{nd}=o_p\left(1\right).$

Proof of Theorem 4:

We can establish that $\left(1/N\right){\displaystyle {\sum }_{i\in A}\left(1/{\pi }_i\right)h\left(y_i-{\lambda }_N\right)}$  is design consistent for ${\theta }_{nd}$  as a result of D.2 and the fact that $h(\cdot )$  is bounded (D.6). Then it suffices to show that ${\widehat{\theta }}_{nd\mathrm{,}bag}-\left(1/N\right){\displaystyle {\sum }_{i\in A}\left(1/{\pi }_i\right)h\left(y_i-{\lambda }_N\right)}=o_p\left(1\right),$  or

$$\frac{1}{N}{\displaystyle \sum _{i\in A}}\frac{1}{{\pi }_i}\left\{\frac{1}{\left(\begin{array}{c}n-1\\ k-1\end{array}\right)}{\displaystyle \sum _{A_b∍i}h\left(y_i-\widehat{\lambda }\left(Y_b^{*}\right)\right)-h\left(y_i-{\lambda }_N\right)}\right\}=o_p\left(1\right)$$

following (2.6). We can establish that the collection of resample-based estimators $\widehat{\lambda }\left(Y_b^{\ast }\right)$  are uniformly contained in a neighborhood of ${\lambda }_N,$  or, ${\sup }_{A_b}\left|\widehat{\lambda }\left(Y_b^{\ast }\right)-{\lambda }_N\right|=O\left(N^{-\alpha }s\right)$  for some $\alpha >0.$  Then we can apply D.6 to conclude the design consistency of the bagging estimator.

A.2  Model-based theory

Assumptions M.1-M.4 are used to show the model-based results (Theorems 1 and 2). Assumption M.1 specifies superpopulation distribution of population characteristics $y_i.$  Assumptions M.2 and M.3 assume simple random without replacement sampling for both the design and the resampling process. Assumption M.5 is needed for showing the model-based asymptotic results for the bagging estimator defined by estimating equations.

Proof of Theorem 1:

The bagging estimator ${\widehat{\theta }}_{nd\mathrm{,}bag}$  is a symmetric statistic, provided that $\widehat{\lambda }$  is symmetric (Lee 1990). We can project it onto a single dimension, say, $y_1.$  But projections onto other observations are equivalent due to symmetry,

$$\begin{array}{l}\text{E}\left\{{\widehat{\theta }}_{nd\mathrm{,}bag}|y_1=y\right\}\\ \text{ =E}\left\{\frac{1}{n}\frac{1}{\left(\begin{array}{c}n-1\\ k-1\end{array}\right)}{\displaystyle \sum _{A_b∍1}}h\left(y_1-\widehat{\lambda }\left(Y_b^{*}\right)\right)|y_1=y\right\}+\text{E}\left\{\frac{n-1}{n}\frac{1}{\left(\begin{array}{c}n-1\\ k-1\end{array}\right)}{\displaystyle \sum _{A_b∍\{i,1\},i\ne 1}}h\left(y_1-\widehat{\lambda }\left(Y_b^{*}\right)\right)|y_1=y\right\}\\ \text{ =}\frac{1}{n}u\left(y\right)+\frac{k-1}{n}v\left(y\right).\end{array}$$

Then we can derive the following linearization of bagging estimator using the theory of symmetric statistics,

$${\widehat{\theta }}_{nd\mathrm{,}bag}-{\theta }_{nd\mathrm{,}\infty }=\frac{1}{n}{\displaystyle \sum _{i=1}^n}\left\{u\left(y_i\right)-{\theta }_{nd\mathrm{,}\infty }\right\}+\frac{k-1}{n}{\displaystyle \sum _{i=1}^n}\left\{v\left(y_i\right)-{\theta }_{nd\mathrm{,}\infty }\right\}+o_p\left(n^{-1/2}\right)\mathrm{,}$$

where $u(\cdot ),v(\cdot )$  and ${\theta }_{nd\mathrm{,}\infty }$  are defined in Theorem 1. The asymptotic variance (3.3) can be easily derived given the $iid$  sampling assumption.

Proof of Theorem 2:

The bagged estimator defined in (2.7) can be treated as a one-sample $k\text{-th}$  order U-statistic, with kernel function

$$h\left(y_1\mathrm{,}{y}_2\mathrm{,}\cdots \mathrm{,}{y}_k\right)=\inf \left\{\gamma :\frac{1}{k}{\displaystyle \sum _{i=1}^k}\psi \left(y_i-\gamma \right)\ge 0\right\}\mathrm{.}$$

We can directly apply a well-known formula for linearizing U-statistic (Serfling 1980 and van der Vaart 1998, p. 161) to obtain the linearization

$${\widehat{\theta }}_{ee\mathrm{,}bag}-{\theta }_{ee\mathrm{,}\infty }=\frac{k}{n}{\displaystyle \sum _{i=1}^n}\left\{u\left(y_i\right)-{\theta }_{ee\mathrm{,}\infty }\right\}+o_p\left(n^{-1/2}\right)\mathrm{,}$$

where

$$\begin{array}{cc}u\left(y\right)& =\text{E }h\left(y\mathrm{,}{y}_1\mathrm{,}{y}_2\mathrm{,}\cdots \mathrm{,}{y}_{k-1}\right)\hfill \\ & =\text{E}\inf \left\{\gamma :\frac{1}{k}{\displaystyle \sum _{i=1}^{k-1}}\psi \left(y_i-\gamma \right)+\frac{1}{k}\psi \left(y-\gamma \right)\ge 0\right\}.\hfill \end{array}$$

The bagged estimating equation estimator (2.7) can be linearized as

$${\widehat{\theta }}_{ee\mathrm{,}bag}-{\theta }_{ee\mathrm{,}\infty }=\frac{k}{n}{\displaystyle \sum _{i=1}^n}\left\{u\left(y_i\right)-{\theta }_{ee\mathrm{,}\infty }\right\}+o_p\left(n^{-1/2}\right).(\text{A}.1)$$

The asymptotic variance of ${\widehat{\theta }}_{ee\mathrm{,}bag}$  can be directly obtained from linearization (A.1).

References

Agresti, A. (2002). Categorical Data Analysis. Second Edition, New York: John Wiley and Sons.

Bahadur, R.R. (1966). A note on quantiles in large samples. The Annals of Mathematical Statistics, 37, 577-580.

Beran, R. and Srivastava, M. (1985). Bootstrap tests and confidence regions for functions of a covariance matrix. The Annals of Statistics, 13, 95-115.

Berger, Y.G. and Skinner, C.J. (2003). Variance estimation for a low income proportion. Journal of the Royal Statistical Society, Series C: Applied Statistics, 52 (4), 457-468.

Bickel, P., Götze, F. and van Zwet, W. (1997). Resampling fewer than  observations: gains, losses and remedies for losses. Statistica Sinica, 7, 1-31.

Breidt, F. and Opsomer, J. (2008). Endogenous post-stratification in surveys: classifying with a sample-fitted model. Annals of Statistics, 36, 403-427.

Breiman, L. (1996). Bagging predictors. Machine Learning, 24, 123-140.

Bühlmann, P. and Yu, B. (2002). Analyzing bagging. The Annals of Statistics, 30 (4), 927-961.

Buja, A. and Stuetzle, W. (2006). Observations on bagging. Statistica Sinica, 16 (2), 323-351.

Chen, S.X. and Hall, P. (2003). Effects of bagging and bias correction on estimators defined by estimating equations. Statistica Sinica, 13 (1), 97-109.

Davison, A. and Hinkley, D. (1997). Bootstrap Methods and Their Applications. Cambridge University Press.

Dümbgen, L. (1993). On nondifferentiable functions and the bootstrap. Probability Theory and Related Fields, 95, 125-140.

Dunstan, R. and Chambers, R.L. (1986). Model-based confidence intervals in multipurpose surveys. Applied Statistics, 35, 276-280.

Efron, B. (1979). Bootstrap methods: another look at the jackknife. The Annals of Statistics, 7, 1-26.

Francisco, C.A. and Fuller, W.A. (1991). Quantile estimation with a complex survey design. The Annals of Statistics, 19, 454-469.

Frees, E.W. (1989). Infinite order U-statistics. Scandinavian Journal of Statistics, 16, 29-45.

Friedman, J.H. and Hall, P. (2007). On bagging and nonlinear estimation. Journal of Statistical Planning and Inference, 137 (3), 669-683.

Fuller, W. (2009). Sampling Statistics. John Wiley and Sons.

Godambe, V. and Thompson, M. (2009). Estimating functions and survey sampling. In C. Rao and D. P. (editors) (Eds.), Handbook of Statistics, vol. 29: Sample Surveys: Inference and Analysis, 669-687. Elsevier/North-Holland.

Hall, P. and Robinson, A. (2009). Reducing variability of crossvalidation for smoothing parameter choice. Biometrika, 96, 175-186.

Heilig, C.M. (1997). An Empirical Process Approach to U-processes of Increasing Degree. Ph. D. thesis, University of California, Berkeley.

Heilig, C.M. and Nolan, D. (2001). Limit theorems for the infinite-degree U-process. Statistica Sinica, 11, 289-302.

Inoue, A. and Kilian, L. (2008). How useful is bagging in forecasting economic time series? A case study of U.S. consumer price inflation. Journal of the American Statistical Association, 103 (482), 511-522.

Knight, K. and Bassett, J.G. (2002). Second order improvements of sample quantiles using subsamples. Unpublished manuscript.

Korn, E.L. and Graubard, B.I. (1998). Confidence intervals for proportions with small expected number of positive counts estimated from survey data. Survey Methodology, 24 (2), 193-201.

Lee, A.J. (1990). U-statistics: Theory and Practice. Marcel Dekker Inc.

Lee, T.-H. and Yang, Y. (2006). Bagging binary and quantile predictors for time series. Journal of Econometrics, 135 (1-2), 465-497.

Rao, J., Kovar, J.G. and Mantel, H.J. (1990). On estimating distribution functions and quantiles from survey data using auxiliary information. Biometrika, 77, 365-375.

Rao, J. and Wu, C. (1988). Resampling inference with complex surveys. Journal of the American Statistical Association, 83 (401), 231-241.

Särndal, C.-E., Swensson, B. and Wretman, J.H. (1997). Model Assisted Survey Sampling. Springer-Verlag Inc (Berlin; New York).

Schick, A. and Wefelmeyer, W. (2004). Estimating invariant laws of linear processes by U-statistics. The Annals of Statistics, 32, 603-632.

Sering, R.J. (1980). Approximation Theorems of Mathematical Statistics. John Wiley and Sons.

Shao, J. and Rao, J. (1993). Standard errors for low income proportions estimated from stratified multistage samples. Sankhya B, 55, 393-414.

Sitter, R.R. and Wu, C. (2001). A note on Woodruff confidence intervals for quantiles. Statistics & Probability Letters, 52 (4), 353-358.

van der Vaart, A.W. (1998). Asymptotic Statistics. Cambridge University Press.

Wang, J.C. and Opsomer, J.D. (2011). On the asymptotic normality and variance estimation of nondifferentiable survey estimators. Biometrika, 98, 91-106.

Previous