A design effect measure for calibration weighting in single-stage samples 4. Empirical evaluationA design effect measure for calibration weighting in single-stage samples 4. Empirical evaluation

• Previous
• Next

We conducted two simulation studies using data that mimic single-stage sampling. The first utilizes publically-available data from tax returns and continuous variables of interest, while the second examines the performance of the alternative measures for a binary outcome measure in a single-stage survey.

4.1 Establishment data simulation study

Here a sample dataset of tax return data is used to mimic an establishment survey setup. The data come from the Tax Year 2007 Statistics of Income (SOI) Form 990 Exempt Organization (EO) sample. This is a stratified Bernoulli sample of 22,430 EO tax returns selected from 428,719 filed with and processed by the IRS between December 2007 and November 2009. This sample dataset, along with the population frame data, is free and electronically available online (Statistics of Income 2011). These data make a candidate “establishment-type” dataset for estimating design effects, in which Kish’s design effect may not apply.

The SOI EO sample dataset is used here as a pseudopopulation for illustration. Four variables of interest are used: Total Assets, Total Liabilities, Total Revenue, and Total Expenses. Returns that were sampled with certainty or that had “very small” assets (defined by having Total Assets less than $1,000,000, including zero) were removed, leaving 8,914 units. We then randomly replicated and perturbed the data to create a pseudopopulation of 50,000 units. We used simple random sampling with replacement to select more observations, then the additional data values were perturbed using the “jitter” (Chambers, Cleveland, Kleiner and Tukey 1983) function in R.

Figure 4.1 shows a pairwise plot of the pseudo-population, including plots of the variable values against each other in the lower left panels, histograms on the diagonal panels, and the correlations among the variables in the upper right panels. This plot mimics establishment-type data patterns. From the diagonal panels, we see that the variables of interest are all highly skewed. From the lower left panels, there exists a range of different relationships among them. The Total Assets variable is less related to Total Revenue and Total Expenses (with moderate correlations of $0.41-0.44);$ Total Revenue and Total Expenses are highly correlated.

Three sizes of samples were selected $\left(n=100;500;\text{1,000}\right)$ without replacement from the pseudopopulation using the square root of Total Assets as a measure of size. This type of sampling is referred to as $\pi \text{ps}$ sampling subsequently. The HT weights were then calibrated using the “linear” method in the “calibrate” function in the “survey” package for R (corresponding to a GREG estimator, Lumley 2012) to match the totals of an intercept, Total Assets and Total Revenue. The analysis variables are Total Liabilities and Total Expenses. (Note that we follow the common practice of developing procedures in the previous sections using formulas for with-replacement sampling but empirically evaluating them in without-replacement samples, which are the type used in applications.)

Eight design effects estimates are considered:

• Estimates of the design effect measures (2.2) and (2.3). Expression (2.2) reflects the efficiency of $\pi \text{ps}$ sampling and use of the $\text{HT}-$ estimator. Expression (2.3) reflects gains (if any) of $\pi \text{ps}$ sampling combined with GREG estimation;
• The Kish measure (2.4) computed using the GREG weights;
• Three Spencer measures computed using the GREG weights: (i) the exact measure that estimates (2.5), (ii) the approximation (2.7) assuming zero correlation terms, and (iii) the large-population approximation (2.9). The Spencer measures are designed to reflect gains due to $\text{PPSWR}$ sampling and use of the $\text{pwr}-$ estimator. It does not account for any gains due to calibration.
• Two proposed measures: (i) the exact proposed single-stage design effect (3.4) and (ii) the zero-correlation approximation (3.5). Both of these are meant to show the precision gains (if any) of $\text{PPSWR}$ sampling combined with GREG estimation.

Note that neither the Spencer nor the proposed measures account for any reduction in variances due to sampling a large fraction of the population.

We selected ten thousand samples to further understand the empirical behavior of the alternative design effect estimators. The empirical relbiases and ratio of the mean square errors (MSE’s) of the totals are

$$\begin{array}{ll}\text{relbias}\left(\widehat{T}\right)\hfill & =100\times {\displaystyle {\sum }_{s=1}^S\left({\widehat{T}}_s-T\right)}/T\hfill \\ \text{MSE ratio}\hfill & =\text{MSE}\left({\widehat{T}}_{\text{HT}}\right)/\text{MSE}\left({\widehat{T}}_{\text{GREG}}\right)\hfill \\ \hfill & ={{\displaystyle {\sum }_{s=1}^S\left({\widehat{T}}_{\text{HT},s}-T\right)}}^2/{\displaystyle {\sum }_{s=1}^S{\left({\widehat{T}}_{\text{GREG},s}-T\right)}^2}\hfill \end{array}$$

where ${\widehat{T}}_s$ is an estimated total from sample $s$ (either HT or GREG), $S=\text{10,000}$ is the number of samples selected, and ${\widehat{T}}_{\text{HT},s}$ and ${\widehat{T}}_{\text{GREG},s}$ are the estimated HT and GREG totals from sample $s.$ The empirical $\text{deff}$ of an estimated total is computed as $\text{empdeff}\left(\widehat{T}\right)=S^{-1}{\displaystyle {\sum }_{s=1}^S{\left({\widehat{T}}_s-\overline{\widehat{T}}\right)}^2}/{\text{Var}}_{\text{srswr}}\left({\widehat{T}}_{\text{srswr}}\right)$ where $\overline{\widehat{T}}=S^{-1}{\displaystyle {\sum }_{s=1}^S{\widehat{T}}_s}$ and ${\text{Var}}_{\text{srswr}}\left({\widehat{T}}_{\text{srswr}}\right)\doteq N^2{\sigma }_y^2/n.$

The results for relbiases and MSEs are shown in Table 4.1. Both estimators of totals are approximately unbiased. The GREG is also more precise than the HT estimator, especially for Total Expenses, as evidenced by the MSE ratios larger than one.

We also computed the biases of the various estimated design effects across the 10,000 samples. The relbiases of the Kish, Spencer, and proposed design effect estimates are computed as

$$\text{relbias}\left({\text{deff}}_K\right)=100\times \left({\overline{\text{deff}}}_K-\text{edeff}\left({\widehat{T}}_{\text{HT}y}\right)\right)/\text{edeff}\left({\widehat{T}}_{\text{HT}y}\right),$$

$$\text{relbias}\left({\text{deff}}_S\right)=100\times \left({\overline{\text{deff}}}_S-\text{edeff}\left({\widehat{T}}_{\text{HT}y}\right)\right)/\text{edeff}\left({\widehat{T}}_{\text{HT}y}\right),$$

and

$$\text{relbias}\left({\text{deff}}_H\right)=100\times \left({\overline{\text{deff}}}_H-\text{edeff}\left({\widehat{T}}_{\text{GREG}}\right)\right)/\text{edeff}\left({\widehat{T}}_{\text{GREG}}\right)$$

where ${\overline{\text{deff}}}_K,$ ${\overline{\text{deff}}}_S,$ and ${\overline{\text{deff}}}_H$ are the average Kish, Spencer, and proposed $\text{deff}’\text{s}$ over all samples. The terms $\text{edeff}\left({\widehat{T}}_{\text{HT}y}\right)$ and $\text{edeff}\left({\widehat{T}}_{\text{GREG}}\right)$ are computed in two ways: (1) as the simulation $\text{empdeff}$ of ${\widehat{T}}_{\text{HT}y}\left(\text{or }{\widehat{T}}_{\text{GREG}}\right),$ and (2) as the average over all samples of the $\text{deff}’\text{s}$ of ${\widehat{T}}_{\text{HT}y}$ computed from the “survey” package. The “survey” package’s default method of estimating the $\text{deff}$ from a particular sample uses a with-replacement variance estimate in the numerator. This corresponds to the sample design used to derive ${\text{deff}}_H.$ Results are displayed in Table 4.2.

For both variables of interest, we see large positive biases for the Kish design effect, and the design effects involving approximations. Thus, ignoring correlation components accounted for in the ‘exact’ Spencer and proposed design effects would lead to over-estimating the design effects.

The proposed estimator is closer to the “survey” package design effects than to the empirical simulation $\text{deff}’\text{s}$ of the GREG. Although the relbiases of ${\text{deff}}_H$ are fairly large for Total Expenses when computed with respect to $\text{edeff,}$ the empirical $\text{deff}’\text{s}$ themselves are small. We highlight the small magnitude of the Total Expenses $\left(y_2\right)$ variable $\text{deff}$ of 0.02 to put the relbiases into context. For example, the relbias of 12.9% for the exact version of our proposed estimator for $n=500$ for $y_2$ corresponds to a difference in the third decimal place. Specifically, in this scenario, on average we over-estimate the $\text{deff}$ by 0.003.

We can understand why calibration is more efficient for Expenses than for Liabilities by examining the distributions of $y_i$ and $u_i$ in one particular sample. Figures 4.2 and 4.3 show boxplots of $u_i$ and $y_i$ for each variable and sample size.

The $u_i-$ values in all of these samples have shorter ranges of values and less variation than $y_i,$ particularly for the Total Expenses variable. This occurs since the Total Expenses variable is highly correlated with the calibration variable Total Revenue (see Figure 4.1) and explains why the direct and proposed design effect measures are so much smaller for Total Expenses.

4.2 Simulation study with a binary variable

The second simulation study illustrates the performance of the proposed estimator when estimating the total of a binary variable in a single-stage survey that uses poststratification.

We use the “nhis.large” population, which has $N=\text{21,588}$ units, from the “PracTools” R package (Valliant, Dever and Kreuter 2015) to gauge the impact of poststratification weighting adjustments. The binary variable used is whether or not a person received Medicaid or not. Receipt of Medicaid, which is a social welfare program in the US, is an example of a variable that is collected in some telephone surveys. Missing values of Medicaid recipiency were recoded to be “no” responses. There is a fairly strong relationship between race-ethnicity, age, and whether Medicaid is received, as shown in Table 4.3 or Table 14.1 in Valliant, Dever and Kreuter (2013). The 15 age × race-ethnicity cells in the table will be used as poststrata, which is a typical procedure in telephone surveys.

In our simulation, we selected 10,000 simple random samples without replacement from the NHIS population. The HT estimator for the total number of persons receiving Medicaid is $N{\overline{y}}_s,$ where ${\overline{y}}_s$ is the proportion in sample $s$ that receives Medicaid. Due to the relatively large number of poststrata and varying number of persons receiving Medicaid by poststratum, we include results only for samples of size $n=500\text{ and 1,000}$ since no collapsing of poststrata within a given particular sample was needed for these sample sizes.

The base weights for the $\text{HT}-$ estimator are simply $w_i=N/n.$ The variance of the poststratified estimator is 91% of that of $N{\overline{y}}_s$ in samples of $n=500$ and 88% in samples of $n=\text{1,000}.$ Since the base weights are constant, Spencer’s design effects are not computable in this example. Therefore, only results for the Kish and proposed design effects are shown in Table 4.4.

The Kish design effect has positive biases of 17.5% and 17.6% when computed with respect to the empirical $\text{deff}’\text{s}\text{.}$ The exact proposed design effects are positively biased with respect to the “survey” $\text{deff}$ (3.2 and 5.0%), but much less so than the Kish estimator. In this example, the zero-correlation approximation is very similar to the exact version of the proposed estimator. The correlation components were negligible for these weighting adjustments within three decimal places.

• Previous
• Next