A design effect measure for calibration weighting in single-stage samples 5. Discussion, limitations, and conclusionsA design effect measure for calibration weighting in single-stage samples 5. Discussion, limitations, and conclusions

• Previous

We propose a new design effect that gauges the impact of calibration weighting adjustments on an estimated total in single-stage sampling. Two existing design effects are the Kish (1965) “design effect due to weighting” and one due to Spencer (2000). Both of these are inadequate to reflect efficiency gains due to calibration. The Kish $\text{deff}$ is a reasonable measure if equal weighting is optimal or nearly so, but does not reveal efficiencies that may accrue from sampling with varying probabilities. The Spencer $\text{deff}$ does signal whether the HT (or $\text{pwr)}$ estimator in varying probability sampling is more efficient than $\text{srs}\text{.}$ But, the Spencer $\text{deff}$ does not reflect any gains from using calibration.

The proposed design effect measures the impact of both sampling with varying probabilities and of using a calibration estimator, like the GREG, that takes advantage of auxiliary information. As we demonstrate empirically, the proposed design effects do not penalize unequal weights when the relationship between the survey variable and calibration covariate is strong. We also demonstrated empirically that the correlation components in the Spencer measure and our proposed measure can be important in some situations. It is not overly difficult to calculate these components, and these should be incorporated when possible to avoid over estimates of the design effects. However, the high correlations between survey and auxiliary variables that we observed in our establishment pseudopopulation data may be unattainable for some surveys that lack auxiliary information. In cases where the auxiliary information is ineffective or is not used, the proposed measure approximates Kish’s $\text{deff}\text{.}$ The measure presented here is applicable to single-stage sampling but can be extended to more complex sample designs, like cluster sampling.

Our measure uses the model underlying the general regression estimator to extend the Spencer measure. The survey variable, covariates, and weights are required to produce the design effect estimate. Since the variance (3.2) is approximately correct in large samples for all calibration estimators, our design effect should reflect the effects of many forms of commonly used weighting adjustment methods, including poststratification, raking, and the GREG estimator. Although design effects that do account for these adjustments can be computed directly from estimated variances, it is important for practitioners to understand that the existing Kish and Spencer $\text{deff}’\text{s}$ do not reflect any gains from those adjustments. The $\text{deff}$ introduced in this paper, thus, serves as a corrective to that deficiency.

For practical consideration, the deff in (3.4) is available in the “deffH” function in the R “PracTools” package; see Valliant et al. (2015) for documentation and examples.

Acknowledgements

We thank the referees for their thorough reviews which improved the presentation. Any opinions expressed are those of the authors and do not reflect those of the Internal Revenue Service.

Appendix

Proposed design effect in single-stage sampling

The appendix sketches the derivation of the proposed $\text{deff}\text{.}$ Most notation was defined in the previous sections of the paper. The average population one-draw probability is $\overline{P}=N^{-1}{\displaystyle {\sum }_{i=1}^N{p}_i}.$ Assume that the design satisfies $\overline{P}=N^{-1}.$ Consider the model $y_i=\alpha +x_i^T\beta +{\epsilon }_i.$ If the full finite population were available, then the least-squares population regression line would be

$$y_i=A+x_i^{T}B+e_i,(\text{A}\text{.1})$$

where $A$ and $B$ are the values found by fitting an ordinary least squares regression line in the full finite population. That is, $A=\overline{Y}-B\overline{X},$ $B={\left(X^{T}X\right)}^{-1}{X}^{T}y,$ where $X$ is the $N\times p$ population matrix of auxiliary variables, $\overline{Y}=N^{-1}{\displaystyle {\sum }_{i=1}^N{y}_i}$ is the population mean, and $\overline{X}$ is the vector of population means of the $x’\text{s}\text{.}$ The $e_i’\text{s}$ are defined as the finite population residuals, $e_i=y_i-A-x_i^{T}B,$ and are not superpopulation model errors. Denote the population variance of the $y’\text{s},e’\text{s},e^2,$ and weights as ${\sigma }_y^2,{\sigma }_e^2,{\sigma }_{e^2}^2,{\sigma }_w^2,$ e.g., ${\sigma }_y^2=N^{-1}{\displaystyle {\sum }_{i=1}^N{\left(y_i-\overline{Y}\right)}^2},$ and the finite population correlations between the variables in the subscripts as ${\rho }_{yp},{\rho }_{ew},$ and ${\rho }_{e^{2}w}.$ The GREG theoretical design-variance in with-replacement sampling is

$$\begin{array}{ll}\text{Var}\left({\widehat{T}}_{\text{GREG}}\right)\hfill & =n^{-1}{\displaystyle {\sum }_{i=1}^N{p}_i{\left(e_i/p_i-E_U\right)}^2}\hfill \\ \hfill & =n^{-1}\left({\displaystyle {\sum }_{i=1}^N{e}_i^2/p_i-E_U^2}\right),\hfill \end{array}(\text{A}\text{.2})$$

where $E_U={\displaystyle {\sum }_{i=1}^N{e}_i}.$ Using the model in (A.1) produces a design effect with several complex terms, many of which contain correlations that cannot be dropped as in Spencer’s approximation. The design effect can be simplified using an alternative formulation: $u_i=A+e_i,$ where $u_i=y_i-x_i^{T}B.$ First, we rewrite the population total of the $e_i’\text{s}$ as $E_U={\displaystyle {\sum }_{i=1}^N{e}_i}=N\overline{U}-NA,$ where $\overline{U}=N^{-1}{\displaystyle {\sum }_{i=1}^N{u}_i}.$ From this, $E_U^2={\left(N\overline{U}\right)}^2+{\left(NA\right)}^2-2N^2\overline{U}A.$ Second, using $w_i={\left(n{p}_i\right)}^{-1},$ or $p_i={\left(n{w}_i\right)}^{-1},$ we rewrite the component ${\displaystyle {\sum }_{i=1}^N{e}_i^2/p_i}$ as

$$\begin{array}{ll}{\displaystyle {\sum }_{i=1}^N{e}_i^2/p_i}\hfill & ={\displaystyle {\sum }_{i=1}^N\frac{{\left(u_i-A\right)}^2}{{\left(n{w}_i\right)}^{-1}}}\hfill \\ \hfill & =n{\displaystyle {\sum }_{i=1}^N{w}_i{u}_i^2}+n{A}^2{\displaystyle {\sum }_{i=1}^N{w}_i}-2nA{\displaystyle {\sum }_{i=1}^N{w}_i{u}_i}.\hfill \end{array}(\text{A}\text{.3})$$

Subtracting $E_U^2$ from (A.3) and dividing by $n$ gives

$$\begin{array}{ll}{n}^{-1}\left({\displaystyle {\sum }_{i=1}^N{e}_i^2/p_i}-E_U^2\right)\hfill & ={\displaystyle {\sum }_{i=1}^N{w}_i{u}_i^2}-n^{-1}{\left(N\overline{U}\right)}^2\hfill \\ \hfill & +A^2\left({\displaystyle {\sum }_{i=1}^N{w}_i}-n^{-1}{N}^2\right)\hfill \\ \hfill & +n^{-1}2N^2\overline{U}A-2A{\displaystyle {\sum }_{i=1}^N{w}_i{u}_i}.\hfill \end{array}(\text{A}\text{.4})$$

Following Spencer’s approach using the covariance substitutions, the first and fifth terms in (A.4) can be rewritten as ${{\displaystyle {\sum }_{i=1}^N{w}_i{u}_i^2=N{\rho }_{u^{2}w}{\sigma }_{u^2}\sigma }}_w+N\overline{W}\left({\sigma }_u^2+{\overline{U}}^2\right)$ and ${\displaystyle {\sum }_{i=1}^N{w}_i{u}_i}=N{\rho }_{uw}{\sigma }_u{\sigma }_w+N\overline{W}\overline{U}.$ Plugging these back into the variance (A.4) gives

$$\begin{array}{ll}{n}^{-1}\left({\displaystyle {\sum }_{i=1}^N{e}_i^2/p_i}-E_U^2\right)\hfill & =N{\rho }_{u^{2}w}{\sigma }_{u^2}{\sigma }_w+N\overline{W}\left({\sigma }_u^2+{\overline{U}}^2\right)-n^{-1}{\left(N\overline{U}\right)}^2\hfill \\ \hfill & +N{A}^2\left(\overline{W}-n^{-1}N\right)\hfill \\ \hfill & +2n^{-1}{N}^2\overline{U}A-2A\left(N{\rho }_{uw}{\sigma }_u{\sigma }_w+N\overline{W}\overline{U}\right).\hfill \end{array}(\text{A}\text{.5})$$

The variance of the $\text{pwr}-$ estimator under simple random sampling with replacement, where $p_i=N^{-1},$ reduces to ${\text{Var}}_{\text{srswr}}\left({\widehat{T}}_{\text{pwr}}\right)=N^2{\sigma }_y^2/n.$ Taking the ratio of (A.5) to the $\text{pwr}-$ variance gives the following design effect:

$$\begin{array}{ll}{\text{Deff}}_H\hfill & ={\text{Var}}_{\text{GREG}}\left({\widehat{T}}_{\text{cal}}\right)/{\text{Var}}_{\text{srswr}}\left({\widehat{T}}_{\text{pwr}}\right)\hfill \\ \hfill & =\frac{n\overline{W}}{N}\left(\frac{{\sigma }_u^2}{{\sigma }_y^2}\right)+\frac{{\left(\overline{U}-A\right)}^2}{{\sigma }_y^2}\left(\frac{n\overline{W}}{N}-1\right)\hfill \\ \hfill & +\frac{n{\sigma }_w}{N{\sigma }_y^2}\left({\rho }_{u^{2}w}{\sigma }_{u^2}-2A{\rho }_{uw}{\sigma }_u\right).\hfill \end{array}(\text{A}\text{.6})$$

Since $u_i=A+e_i,\overline{U}=A,$ (A.6) becomes

$${\text{Deff}}_H=\frac{n\overline{W}}{N}\left(\frac{{\sigma }_u^2}{{\sigma }_y^2}\right)+\frac{n{\sigma }_w}{N{\sigma }_y^2}\left({\rho }_{u^{2}w}{\sigma }_{u^2}-2A{\rho }_{uw}{\sigma }_u\right).(\text{A}\text{.7})$$

We estimate measure (A.7) with

$${\text{deff}}_H\approx \left(1+{\left[\text{CV}\left(w\right)\right]}^2\right)\frac{{\widehat{\sigma }}_u^2}{{\widehat{\sigma }}_y^2}+\frac{n{\widehat{\sigma }}_w}{N{\widehat{\sigma }}_y^2}\left({\widehat{\rho }}_{u^{2}w}{\widehat{\sigma }}_{u^2}-2\widehat{\alpha }{\widehat{\rho }}_{uw}{\widehat{\sigma }}_u\right),(\text{A}\text{.8})$$

where the model parameter estimates are defined in Sections 2.3 and 3.

References

Brick, M., and Montaquila, J. (2009). Nonresponse. In Handbook of Statistics, Sample Surveys: Design, Methods and Application, (Eds., D. Pfeffermann and C.R. Rao), 29A, Amsterdam: Elsevier BV.

Chambers, J.M., Cleveland, W.S., Kleiner, B. and Tukey, P.A. (1983). Graphical Methods for Data Analysis. Pacific Grove CA: Wadsworth.

Deville, J.-C., and Särndal, C.-E. (1992). Calibration estimators in survey sampling. Journal of the American Statistical Association, 87, 376-382.

Deville, J.-C., Särndal, C.-E. and Sautory, O. (1993). Generalized raking procedures in survey sampling. Journal of the American Statistical Association, 88, 1013-1020.

Hansen, M.H., and Hurwitz, W.N. (1943). On the theory of sampling from a finite population. Annals of Mathematical Statistics, 14, 333-362.

Horvitz, D., and Thompson, D. (1952). A generalisation of sampling without replacement from a finite universe. Journal of the American Statistical Association, 47, 663-685.

Kalton, G., and Flores-Cervantes, A. (2003). Weighting methods. Journal of Official Statistics, 19 (2), 81-97.

Kish, L. (1965). Survey Sampling. New York: John Wiley & Sons, Inc.

Kish, L. (1990). Weighting: Why, when, and how? Proceedings of the Joint Statistical Meetings, Section on Survey Research Methods, American Statistical Association, 121-129.

Kish, L. (1992). Weighting for unequal Pi. Journal of Official Statistics, 8, 183-200.

Kott, P. (2009). Calibration weighting: Combining probability samples and linear prediction models. In Handbook of Statistics, Sample Surveys: Design, Methods and Application, (Eds., D. Pfeffermann and C.R. Rao), 29B, Amsterdam: Elsevier BV.

Lumley, T. (2012). Survey: Analysis of complex survey samples. R package version 3.28-2.

Park, I., and Lee, H. (2004). Design effects for the weighted mean and total estimators under complex survey sampling. Survey Methodology, 30, 2, 183-193.

Rao, J.N.K., and Scott, A.J. (1984). On chi-squared tests for multiway contingency tables with cell proportions estimated from survey data. Annals of Statistics, 12, 46-60.

Särndal, C.-E., and Lundström, S. (2005). Estimation in Surveys with Nonresponse. New York: John Wiley & Sons, Inc.

Särndal, C.-E., Swensson, B. and Wretman, J. (1992). Model Assisted Survey Sampling. New York: Springer: Berlin.

Spencer, B.D. (2000). An approximate design effect for unequal weighting when measurements may correlate with selection probabilities. Survey Methodology, 26, 2, 137-138.

Statistics of Income (2011). 2007 Charities & Tax-Exempt Microdata Files. Available at: http://www.irs.gov/uac/SOI-Tax-Stats-2007-Charities-&-Tax-Exempt-Microdata-Files.

Valliant, R., Dever, J.A. and Kreuter, F. (2013). Practical Tools for Designing and Weighting Survey Samples. New York: Springer.

Valliant, R., Dever, J.A. and Kreuter, F. (2015). PracTools: Tools for Designing and Weighting Survey Samples. R package version 0.2. http://CRAN.R-project.org/package=PracTools.

• Previous