A design effect measure for calibration weighting in single-stage samples 2. Existing methodsA design effect measure for calibration weighting in single-stage samples 2. Existing methods

• Previous
• Next

In this section, we specify notation and summarize the Kish and Spencer measures. The assumptions used to derive each of these are also presented.

2.1 GREG weight adjustments

Case weights resulting from calibration on benchmark auxiliary variables can be defined with a global regression model for the survey variables (see Kott 2009 for a review). Deville and Särndal (1992) proposed the calibration approach that involves minimizing a distance function between the base weights and final weights to obtain an optimal set of survey weights. Specifying alternative calibration distance functions produces alternative estimators. Suppose that a single-stage probability sample of $n$ units is selected with ${\pi }_i$ being the selection probability of unit $i$ and $x_i$ a vector of $p$ auxiliaries associated with unit $i.$ A least squares distance function produces the general regression estimator (GREG):

$${\widehat{T}}_{\text{GREG}}={\widehat{T}}_{\text{HT}y}+{\widehat{B}}^T\left(T_x-{\widehat{T}}_{\text{HT}x}\right)={\displaystyle {\sum }_{i\in s}{g}_i{y}_i}/{\pi }_i,(2.1)$$

where ${\widehat{T}}_{\text{HT}y}={\displaystyle {\sum }_{i\in s}{y}_i}/{\pi }_i$ is the Horvitz-Thompson (HT 1952) estimator of the population total of $y,{\widehat{T}}_{\text{HT}x}={\displaystyle {\sum }_{i\in s}{x}_i}/{\pi }_i$ is the vector of HT estimated totals for the auxiliary variables, $T_x={\displaystyle {\sum }_{i=1}^N{x}_i}$ is the corresponding vector of known totals, $\widehat{B}=A_s^{-1}{X}_s^T{V}_{ss}^{-1}{\Pi }_s^{-1}{y}_s$ is the regression coefficient, with $A_s=X_s^T{V}_{ss}^{-1}{\Pi }_s^{-1}{X}_s,$ $X_s^T$ is the matrix of $x_i$ values in the sample, $V_{ss}=\text{diag}\left(v_i\right)$ is the diagonal of the variance matrix specified under the working model $y_i=x_i^T\beta +{\epsilon }_i,{\epsilon }_i~\left(0,v_i\right),$ and ${\Pi }_s=\text{diag}\left({\pi }_i\right).$ In the second expression for the GREG estimator in (2.1), $g_i=1+{\left(T_x-{\widehat{T}}_{\text{HT}x}\right)}^T{A}_s^{-1}{x}_i{v}_i^{-1}$ is the $“g-$ weight,” such that the case weights are $w_i=g_i/{\pi }_i$ for each sample unit $i.$

The GREG estimator for a total is model-unbiased under the associated working model. The GREG is consistent and approximately design-unbiased when the sample size is large (Särndal, Swensson and Wretman 1992). When the model is correct, the GREG estimator achieves efficiency gains. If the model is incorrect, then the efficiency gains will be dampened (or nonexistent) but the GREG estimator is still approximately design-unbiased. Relevant to this work, the variance of the GREG estimator can be used to approximate the variance of any calibration estimator (Deville and Särndal 1992; Deville, Särndal and Sautory 1993) when the sample size is large. This allows us to produce one design effect measure applicable to all estimators in the family of calibration estimators.

2.2 The direct design-effect measures for single-stage samples

For a given non $-\text{epsem}$ sample $\pi$ and estimator $\widehat{T}$ for the finite population total $T,$ one definition for the direct design effect (Kish 1965) is

$$\text{Deff}\left(\widehat{T}\right)={\text{Var}}_{\pi }\left(\widehat{T}\right)/{\text{Var}}_{\text{srswr}}\left({\widehat{T}}_{\text{srswr}}\right)(2.2)$$

where ${\widehat{T}}_{\text{srswr}}$ is the estimator of a total based on a simple random sample selected with replacement $\left(\text{srswr}\right).$ We refer to this as a “direct” population quantity since it uses theoretical variances in the numerator and denominator. The design effect in (2.2) measures the size of the variance of the estimator $\widehat{T}$ under the design $\pi ,$ relative to the variance of the estimator of the same total if a $\text{srswr}$ of the same size had been used.

In large samples, we can approximate the variance of any calibration estimator ${\widehat{T}}_{\text{cal}}$ using the approximate variance of the GREG (GREG AV, Särndal et al. 1992; Deville et al. 1993), such that the design effect is

$$\text{Deff}\left({\widehat{T}}_{\text{cal}}\right)\doteq {\text{Var}}_{\pi }\left({\widehat{T}}_{\text{GREG}}\right)/{\text{Var}}_{\text{srswr}}\left({\widehat{T}}_{\text{srswr}}\right).(2.3)$$

To estimate these design-effects, we use the appropriate corresponding sample-based variance estimates. Estimates of both measures (2.2) and (2.3) can be produced using conventional survey estimation software. Our proposed design effect is a model-assisted approximation to (2.3).

2.3 Kish’s “Haphazard-sampling” design-effect measure for unequal weights

Kish (1965, 1990) proposed the “design effect due to weighting” as a measure to quantify the loss of precision due to using unequal and inefficient weights. For $w={\left(w_1,\dots ,w_n\right)}^T,$ this measure is

$$\begin{array}{ll}{\text{deff}}_K\left(w\right)\hfill & =1+{\left[\text{CV}\left(w\right)\right]}^2\hfill \\ \hfill & =\frac{n{\displaystyle {\sum }_{i\in s}{w}_i^2}}{{\left[{\displaystyle {\sum }_{i\in s}{w}_i}\right]}^2},\hfill \end{array}(2.4)$$

where $\text{CV}\left(w\right)=\sqrt{n^{-1}{\displaystyle {\sum }_{i\in s}{\left(w_i-\overline{w}\right)}^2}/{\overline{w}}^2}$ is the coefficient of variation of the weights with $\overline{w}=n^{-1}{\displaystyle {\sum }_{i\in s}{w}_i}.$ Expression (2.4) is derived from the ratio of the variance of the weighted survey mean under disproportionate stratified sampling to the variance under proportionate stratified sampling when all stratum unit variances are equal (Kish 1992). With equal stratum variances, sampling with a proportional allocation to strata is optimal, which leads to all units having the same weight.

Kish referred to (2.4) as a measure that is appropriate for “haphazard” weighting in which unequal weights are inefficient. Kish (1992) and Park and Lee (2004) give examples of informative sampling where this measure does not apply. Park and Lee (2004) also demonstrate this measure may not apply equally well to estimators of means and totals.

2.4 Spencer’s model-assisted measure for PPSWR sampling

Spencer (2000) derives a design-effect measure to more fully account for the effect on variances of weights that are correlated with the survey variable of interest. The sample is assumed to be selected with varying probabilities and with replacement (denoted as $\text{PPSWR}$ sampling here). A particular case of this would be $p_i\propto x_i,$ where $x_i$ is a measure of size associated with unit $i$ and $p_i$ is the one-draw probability of selecting unit $i.$ Suppose that $p_i$ is correlated with $y_i$ and that a linear model holds for $y_i:$ $y_i=\alpha +\beta p_i+{\epsilon }_i.$ If the entire finite population were available, then the ordinary least squares estimates of $\alpha$ and $\beta$ are $A=\overline{Y}-B\overline{P}$ and $B={\displaystyle {\sum }_{i\in U}\left(y_i-\overline{Y}\right)\left(p_i-\overline{P}\right)}/{\displaystyle {\sum }_{i\in U}{\left(p_i-\overline{P}\right)}^2},$ where $\overline{Y},\overline{P}$ are the finite population means for $y_i$ and $p_i.$ The finite population variance of the residuals, $e_i=y_i-\left(A+B{p}_i\right),$ is ${\sigma }_e^2=\left(1-{\rho }_{yp}^2\right)N^{-1}{\displaystyle {\sum }_{i\in U}{\left(y_i-\overline{Y}\right)}^2}=\left(1-{\rho }_{yp}^2\right){\sigma }_y^2,$ where ${\sigma }_y^2$ is the finite population variance of $y$ and ${\rho }_{yp}$ is the finite population correlation between $y_i$ and $p_i.$ The estimated total studied by Spencer is referred to as the $\text{pwr}-$ estimator or Hansen-Hurwitz (1943) estimator (Särndal et al. 1992, Section 2.9) and is defined as ${\widehat{T}}_{\text{pwr}}=n^{-1}{\displaystyle {\sum }_{i=1}^n{y}_i}/p_i,$ with design-variance $\text{Var}\left({\widehat{T}}_{\text{pwr}}\right)=n^{-1}{\displaystyle {\sum }_{i\in U}{p}_i{\left(y_i/p_i-T\right)}^2}$ in single-stage sampling. For use below, define $w_i={\left(n{p}_i\right)}^{-1}.$ Spencer substituted the model-based values for $y_i$ into the $\text{pwr}-$ estimator’s variance and took its ratio to the variance of the estimated total using $\text{srswr}$ to produce the following design effect for unequal weighting (see Appendix in Spencer 2000):

$${\text{Deff}}_S=\frac{A^2}{{\sigma }_y^2}\left(\frac{n\overline{W}}{N}-1\right)+\frac{n\overline{W}}{N}\left(1-{\rho }_{yp}^2\right)+\frac{n{\rho }_{e^{2}w}{\sigma }_{e^2}{\sigma }_w}{N{\sigma }_y^2}+\frac{2An{\rho }_{ew}{\sigma }_e{\sigma }_w}{N{\sigma }_y^2}(2.5)$$

where $\overline{W}=N^{-1}{\displaystyle {\sum }_{i\in U}{w}_i}={\left(nN\right)}^{-1}{\displaystyle {\sum }_{i\in U}1}/p_i$ is the average weight in the population, ${\rho }_{e^{2}w}$ and ${\rho }_{ew}$ are the finite population correlation of the $e_i^2’\text{s}$ with the $w_i’\text{s}$ and the $e_i’\text{s}$ with the $w_i’\text{s,}$ respectively; ${\sigma }_{e^2}^2$ and ${\sigma }_w^2$ are the finite population variances of the $e_i^2’\text{s}$ and $w_i’\text{s}\text{.}$ In skewed populations, the correlation ${\rho }_{ew}$ in (2.5) may be negligible but ${\rho }_{e^{2}w}$ can be large and negative if units with larger $x,y$ values have larger residuals but small weights. We found empirically in the simulations reported in Section 4 that ${\rho }_{e^{2}w}$ was generally negative and larger in relative size than ${\rho }_{ew}.$

Assuming that the correlations in the last two terms of (2.5) are negligible, Spencer approximates (2.5) with

$${\text{Deff}}_S\approx \left(1-{\rho }_{yp}^2\right)\frac{n\overline{W}}{N}+{\left(\frac{A}{{\sigma }_y}\right)}^2\left(\frac{n\overline{W}}{N}-1\right),(2.6)$$

A similar expression is given by Park and Lee (2004; expression 4.7). Spencer proposed estimating measure (2.6) with

$${\text{deff}}_S=\left(1-R_{yp}^2\right){\text{deff}}_K\left(w\right)+{\left(\widehat{\alpha }/{\widehat{\sigma }}_y\right)}^2\left({\text{deff}}_K\left(w\right)-1\right),(2.7)$$

where $R_{yp}^2\text{ and }\widehat{\alpha }$ are the $R-$ squared and estimated intercept from fitting the model $y_i=\alpha +\beta p_i+{\epsilon }_i$ with survey weighted least squares, ${\widehat{\sigma }}_y^2={\displaystyle {\sum }_{i\in s}{w}_i{\left(y_i-{\widehat{\overline{y}}}_w\right)}^2}/{\displaystyle {\sum }_{i\in s}{w}_i}$ with ${\widehat{\overline{y}}}_w={\displaystyle {\sum }_s{w}_i{y}_i}/{\displaystyle {\sum }_s{w}_i}$ is the estimated population unit variance. Spencer’s estimator (2.7) assumes that the population size $N$ is large.

When ${\rho }_{yp}$ is zero and ${\sigma }_y$ is large, measure (2.7) is approximately equivalent to Kish’s measure (2.4). However, Spencer’s method does incorporate the survey variable $y_i,$ unlike (2.4), and implicitly reflects the dependence of $y_i$ on the selection probabilities $p_i.$ We can explicitly see this by noting that when $N$ is large, $A=\overline{Y}-B{N}^{-1}\approx \overline{Y},$ and (2.6) can be written as

$${\text{Deff}}_S\approx \left(1-{\rho }_{yp}^2\right)\frac{n\overline{W}}{N}+\frac{1}{{\text{CV}}_Y^2}\left(\frac{n\overline{W}}{N}-1\right),(2.8)$$

where ${\text{CV}}_{\overline{Y}}^2={\sigma }_y^2/{\overline{Y}}^2$ is the population-level unit coefficient of variation (CV). We estimate (2.8) with

$${\text{deff}}_S=\left(1-R_{yp}^2\right){\text{deff}}_K\left(w\right)+\frac{1}{{\text{cv}}_y^2}\left({\text{deff}}_K\left(w\right)-1\right),(2.9)$$

where ${\text{cv}}_y^2={\widehat{\sigma }}_y^2/{\widehat{\overline{y}}}_w^2.$ Note that ${\text{cv}}_y$ is not the standard CV produced in conventional survey estimation software, since it estimates the population unit CV of $y.$

• Previous
• Next