A design effect measure for calibration weighting in single-stage samples 1. IntroductionA design effect measure for calibration weighting in single-stage samples 1. Introduction

• Previous
• Next

A design effect $\left(\text{deff}\right)$  in its general form measures the relative increase or decrease in the variance of an estimator due to departures from simple random sampling. Kish (1965) presented the $\text{deff}$  as a convenient way of gauging the effect of clustering on an estimator of a mean. Park and Lee (2004) review some of the history behind the formulation and use of $\text{deff}’\text{s}\text{.}$  Design effects are especially useful in approximating the total sample size needed in a cluster sample. Clustering usually causes some loss of efficiency and the variance from a simple random sample, which is easy to compute, can be multiplied by a $\text{deff}$  to approximate the variance that would be obtained from a cluster sample. This can, in turn, be used to determine the total sample size needed in a cluster sample to achieve a desired level of precision. Later work by Rao and Scott (1984) and others found that more complicated versions of $\text{deff}’\text{s}$ were useful to adjust inferential statistics calculated from complex survey data.

A specialized version of the $\text{deff}$ was proposed in Kish (1965) that addressed only the effect of using weights that are not all equal. Kish derived the “design effect due to weighting” for a case in which weights vary for reasons other than statistical efficiency. On the other hand, there are sample designs and estimators where having varying weights can be quite efficient. An establishment survey where population variances of analysis variables differ markedly among industries is one example. Calibrating to population counts can also produce different sized weights but is an essential tool in attempting to correct for coverage errors in some surveys, like ones done by telephone. Spencer (2000) proposed a simple model-assisted approach to estimate the impact on variance of using variable weights in a situation where an analysis variable depends on a single covariate.

The Kish and Spencer measures, reviewed in Section 2, do not provide a summary measure of the impact of the gains in precision that may accrue from sampling with varying probabilities and using a calibration estimator like the general regression (GREG) estimator. While the Kish design effects attempt to measure the impact of variable weights, they are informative only under special circumstances, do not account for alternative variables of interest, and can incorrectly measure the impact of differential weighting in some circumstances, facts noted in Kish (1992). Survey practitioners should be cautious when using this measure in informative sampling and estimation schemes in which there exists an intentional relationship between the weights and variables of interest. Spencer’s approach holds for with-replacement single-stage sampling for a very simple estimator of the total constructed with inverse-probability weights with no further adjustments. There are also few empirical examples comparing these measures in the literature.

Calibration adjustments are often applied to reduce variances and correct for undercoverage and/or nonresponse in surveys (e.g., Särndal and Lundström 2005; Kott 2009). When the calibration covariates are correlated with the coverage/response mechanism, calibration weights can improve the mean squared error (MSE) of an estimator. In many applications, since calibration involves unit-level adjustments, calibration weights can vary more than the base weights or category-based nonresponse or poststratification adjustments (Kalton and Flores-Cervantes 2003; Brick and Montaquila 2009). Thus, an ideal measure of the impact of calibration weights incorporates not only the correlation between the survey variable of interest $y$ and the weights, but also the correlation between $y$ and the calibration covariates $x$ to avoid “penalizing” weights for the mere sake that they vary.

In Section 3, we introduce a new design effect measure that accounts for the joint effect of a non- $\text{epsem}$ sample design and unequal weight adjustments in the larger class of calibration estimators. It is assumed that a probability sample design is used and that there are no missing data problems that would induce a dependence between sample inclusion and the values of the $y’\text{s}\text{.}$ Our summary measure incorporates the survey variable, using a generalized regression variance to reflect multiple calibration covariates. In Section 4, we apply the estimators in a simulation using variables similar to ones collected in establishment surveys and household surveys done by telephone and demonstrate empirically how the proposed estimator outperforms the existing methods in the presence of unequal calibration weights. Section 5 is a conclusion.

• Previous
• Next