Strategies for subsampling nonrespondents for economic programs
Section 3. Case study

• Previous
• Next

This section presents the results of a simulation study that evaluates the considered allocation procedures on empirical sample data from the Annual Survey of Manufactures (ASM) from the 2010 and 2011 data collections. For more information on the ASM, see http://www.census.gov/manufacturing/asm.

The ASM is an establishment survey designed to produce “sample estimates of statistics for all manufacturing establishments with one or more paid employee(s)” (http://www.census.gov/manufacturing/ asm/); it is a Pareto-PPS sample of approximately 50,000 establishments selected from a universe of 328,500. Approximately 20,000 establishments are included with certainty, and the remaining establishments are selected with probability proportional to a composite measure of size. Selected units are in the sample for the four years between censuses. Sampling strata are defined by six-digit industry code using the North American Industry Classification System.

The ASM estimates totals with a difference estimator (Särndal et al., 1992). To reduce respondent burden, units below a certain threshold are dropped from the sampling frame entirely. Instead, their data are imputed using administrative data values for selected items and industry-level regression models for the remaining items. Similarly, the ASM imputes complete records for unit nonrespondents. See http://www.census.gov/manufacturing/asm/ for additional information on the ASM methodology.

Because the items collected by the ASM questionnaire are a subset of the EC’s manufacturing sector items, the ASM is often used to pretest new EC processing or data collection procedures. With the ASM and the EC, implementing a probability subsample of nonrespondents for NRFU represents a major procedural change. The ASM NRFU procedures are very similar to the EC procedures. Because a given company can comprise several establishments, the sets of multi-unit (MU) establishments corresponding to the company can be designated for phone follow-up as well as other company completeness checks. In contrast, the NRFU procedures for the single unit (SU) establishments $–$ establishments with one location and parent company $–$ differ. The largest SU establishments are included with certainty (sampled with probability = 1) and may receive a personal phone call in selected domains. The sampled SU establishments (“SU noncertainty establishments”) receive some reminders, but are very unlikely to receive a personal phone call.

Our simulation study examines one of the fourteen key ASM items and employs the double expansion estimate and the two ratio estimators described in the Appendix, not the difference estimator used in ASM production estimates. Consequently, our results should not be extrapolated to the ASM.

3.1  Simulation study design

Our simulation study compares the statistical properties of total shipment estimates obtained from the three considered nonrespondent subsampling designs over repeated samples, using three different estimators. Our sampling frame of nonrespondents is derived from the fully imputed 2011 ASM sample and is limited to the SU noncertainty establishments so that the overall ASM publication reliability requirements are maintained. The ratio estimators employ the sample-based values of annual payroll as an auxiliary variable. This variable is highly correlated with total shipments, but is subject to imputation. Note that we use the complete ASM sample (all MU and SU establishments) for the allocations but present the relative bias and MSE results for the subsampled domains (SU noncertainty establishments) only.

For the SU noncertainty establishments, the first NRFU attempt $–$ consisting of a reminder letter $–$ is historically very effective, so nonrespondent subsample selection occurs before the second NRFU attempt. The second NRFU attempt is generally more expensive (historically a package re-mail, although reminder letters via certified mail will be used in future collections). Nonrespondent subsampling of SU noncertainty establishments occurs after the second contact attempt (i.e., after the first NRFU attempt).

To perform the simulation, we removed all MU establishments and SU certainty establishments from the ASM sample data to create a frame, and then independently repeated the following procedure 5,000 times for each allocation procedure:

1. Using the estimated response propensities provided in Table 3.1, randomly induce nonresponse into the sample using a MAR response mechanism.
2. Sort the induced nonrespondents by sampling weight.
3. Select a stratified systematic sample using the nonrespondent domain subsampling rates for a given allocation strategy.
4. Simulate unit response for each round of NRFU. Table 3.1 provides the conditional response propensities used for each distinct NRFU contact phase. These statistics use paradata from the 2010 and 2011 ASM collections (Fink and Lineback, 2013). Hereafter, we refer to these conditional probabilities as “nonrespondent conversion rates”. If the unit responded, the mode of response is randomly assigned using historical frequencies provided by subject matter experts. After assigning response status/response mode to each unit, compute cumulative collection cost, URR, and estimates.
5. For each allocation, repeat Step 4 until either ten rounds of follow-up have been conducted or the total budget has been expended. If funds are exhausted within a round, then NRFU ceases. Given that the fixed budget assumes that $1/K$ of the original set of nonrespondents will receive NRFU, the budget can be exhausted under full follow-up. The total budget is never expended before ten rounds of NRFU with nonrespondent subsampling, as the cost-per-unit of mailing a reminder letter is quite low. Our choice of a maximum of ten rounds of NRFU in the simulation was subjective; the purpose was to demonstrate that subsampling would facilitate additional contact efforts at no additional cost.

The nonrespondent conversion rates in the majority of domains follow the same pattern: a decaying response probability followed by a slight increase in the fourth round due to a longer collection period. Domain 10 does not follow this pattern; it contained only four units that all responded before subsampling began. After the 4^th round of NRFU, the nonrespondent conversion rates are reduced by half until they achieve the minimum allowable value of 0.02. The pattern reflects the findings of Olson and Groves (2012). (Olson and Groves (2012) postulate that the response propensities change over the collection cycle, especially as data collection protocols are modified. With the ASM, the reminder letters become more stringent at each NRFU contact phase. Likewise, the authors demonstrate that response propensities decline over the collection phase when a stable data collection protocol is used, as reflected in nonrespondent conversion rates). Mail and phone response propensity estimates were provided by subject matter experts, as were approximate costs by mode and an overall budget figure.

To evaluate the statistical properties of each allocation method for each estimator, we computed the relative bias and the mean squared error. The relative bias (RBE) for each estimate of total shipments at NRFU phase $t$ for a given sampling overall interval $\left(K\right),$ allocation method $a$ $\text{(Constant}-K,$ $\text{Min}-K,$ $\text{Min}-\text{URR),}$ eventual response probability $q,$ and estimator $e$ (DE, SR, CR) is

$$\text{RBE}{\left(Y\right)}_{Kaqt}^e=100\text{*}\left[\left(\frac{{{\displaystyle \sum }}_{s=1}^{\text{5,000}}{\widehat{Y}}_{Kaqts}^e}{\text{5,000}}/Y\right)-1\right]$$

where ${\widehat{Y}}_{Kaqts}^e$ is the estimated total and $Y$ is the population total shipments value.

The mean squared error at NRFU phase $t$ for a given sampling interval, allocation method and estimator is

$$\text{MSE}{\left(Y\right)}_{Kaqt}^e=\left[{{\displaystyle \sum }}_{s=1}^{\text{5,000}}{\left({\widehat{Y}}_{Kaqts}^e-Y\right)}^2\right]/\text{5,000}.$$

Since our simulation induces MAR response, the DE estimates should be approximately unbiased over repeated samples, whereas the two ratio estimates should not be. However, the DE estimates are expected to have large variance; using ratio estimators with a positively correlated auxiliary variable is expected to reduce this variance (i.e., increase the precision). Thus, examining the MSE provides insight into the bias-variance tradeoff.

3.2  Allocation

The simulation study uses data from the 2011 ASM collection. Input parameters for allocation were estimated from 2010 ASM collection data. Recall that the target URR applies to the entire ASM program and is not restricted to the subsampling domains - in our case, SU noncertainty establishments. Consequently, the certainty SU and MU unit counts obtained from the 2010 ASM data are included in the allocation programs in the $r_{1h}$ as constants; the remainder of the $r_{1h}$ represents the estimated count of responding SU noncertainty establishments after the first round of NRFU is completed. To ensure that each nonrespondent sampling domain contained sufficient numbers of units to obtain a feasible solution, we used three-digit industry as NRFU sampling domain instead of the six-digit industry used for the ASM sample design [Note: the determination subsampling domain was determined collaborative with the ASM program managers and methodologists].

Both quadratic programs require an estimated probability of eventually responding to follow-up $\left(q_h\right)$ to compute the ${\text{URR}}^{\text{T}}$ (overall and by domain). To assess the sensitivity of the allocation procedure, we tested ten different constant values $\left(q_h=\text{0}\text{.10},\text{0}\text{.20},\dots ,1\right),$ keeping the value constant across all domains. A similar approach can be taken when historic paradata are not available. In addition, we estimate the $q_h$ directly from the 2010 ASM data. These estimates vary by 20-percent at three-digit industry level. However, the median of these is nearly 50-percent. Consequently, the allocation obtained using the estimated (historic-data) $q_h$ values are very similar to those obtained with $q=\text{0}\text{.50}.$

Approximately $21,000 was allotted for NRFU of SU noncertainty establishments after subsampling. With full follow-up, the expected final unit response rate was approximately 79%. Using data from the 2007 EC, Bechtel and Thompson (2013) found that the target industry unit response rates of 70% could only be achieved in a $1-\text{in}-3$ subsample if the average unit response rate in the majority of EC industries was 60% or larger before follow-up begins. With the ASM, the response rate prior to subsampling was approximately 57%. Instead, we select an overall $1-\text{in}-2$ subsample, which would save approximately 50-percent of the allotted budget after five completed rounds of NRFU at the cost of a decrease expected response rate (69%). The additional five rounds of NRFU added approximately $4,000 to the total cost without commensurate increases in response rate (70%). A larger subsample would be preferable in terms of quality, but is not cost effective.

For allocation, we obtain the ${\text{URR}}^{\text{T}}\text{},$ allowing the $q_h$ to vary by domain. The maximum URR is always achieved with the $\text{Min}-\text{URR}$ quadratic program. Table 3.2 presents the target URRs and the allocation subsampling rates obtained from the $\text{Min}-\text{URR}$ quadratic program. A dash (-) indicates no subsample is selected for NRFU (a sampling interval of $\infty ).$ If $K=1,$ all units in the domain are selected for NRFU (full follow-up). A label of $q=$ <value> indicates that the eventual probability of respondent is the same constant value in all domains; values estimated from historical data are labeled as $q_h=$  Est. Recall that ${\text{URR}}^{\text{T}}$ includes all respondent units in the ASM sample, not just the noncertainty single units that are eligible for subsampling. Consequently, selected domains have achieved their target URRs before subsampling and are not considered as subsampling candidates in the allocation programs.

As the probability of eventually responding increases, this allocation tends to select smaller subsamples in increasing numbers of domains. When the probability of an eventual response $\left(q_h\right)$ is small (20-percent or less), then the allocations sensibly tend towards no subsampling or full follow-up, focusing on obtaining sample from the few domains with the poorest response rates. As the probability of an eventual response increases, the amount of subsampling tends to increase as well. At 70-percent, almost half of the domains are allocated at least one sampled unit, thus spreading the allocated sample across several domains instead of concentrating in a few domains that have exceptionally poor response rates. Note that rates below 20-percent are (hopefully) unrealistic as are rates greater than 70-percent. Domain 10 has highly variable sampling rates regardless; because all four units responded before subsampling, the quadratic program selected any sampling rate because, in effect, it always subsamples zero cases.

Unlike the $\text{Min}-\text{URR}$ quadratic program, the $\text{Min}-K$ quadratic program did not always obtain a solution for a given target URR because of the domain-level constraints on the target URRs. When this occurred, we incrementally lowered the target response rate until a feasible solution could be obtained. Table 3.3 presents the target URRs and the allocations obtained from the $\text{Min}-K$ quadratic program.

Both the allocation methods tend to designate the same domains for either no subsampling or for full follow-up. However, the two methods produce very different allocations for the same $q_h$ in the subsampled domains. The $\text{Min}-K$ allocations avoid subsampling in domains that have already achieved their maximum estimated target URR, regardless of the probability of eventually obtaining a response, with 40- to 50-percent of the domains not being subsampled when $\text{0}\text{.30}\le q_h\le \text{0}\text{.50}.$ Otherwise, the subsampling tends to be split between full follow-up (all units selected) or subsampling at an approximately $1-\text{in}-2$ sampling rate. In short, the $\text{Min}-\text{URR}$ allocations yield domain subsampling intervals that can differ considerably from the overall interval, as the allocation seeks to equalize the target URR in each domain. The resultant variability in sampling intervals can lead to large increases in sampling variance. Because the $\text{Min}-K$ objective function seeks to equalize sampling intervals, the domain subsampling intervals tend to be less variable and are generally close to the overall sampling interval.

3.3  Results

Our baseline closely mimics the NRFU procedures used in the 2012 ASM NRFU $–$ four phases of full follow-up $\left(K=1\right)-$ but can include an additional incomplete fifth round when the planned budget was not depleted to retain programming consistency. For other values of $K,$ NRFU is concluded after ten rounds regardless of the remaining funds.

Table 3.4 presents the relative bias of the estimates (RBE) and the mean squared error (MSE) results obtained with full NRFU and the $\text{Constant}-K$ allocation for each considered estimator. In all cases, the unbiased double expansion (DE) estimator yields unbiased estimates, whereas the ratio estimators are slightly biased as expected. With subsampling, the relative bias of the ratio estimators increases, whereas the DE estimator remains unbiased. Regardless of estimator, the additional stage of subsampling increases the sampling variance and consequently the MSE; the bias tends to remain unaffected because the subsampled units are a representative subsample at each round of follow-up.

With equal probability subsampling $\text{(Constant}-K),$ a subsample may contain a few sampled cases in one or more domains. Although the subsampling weighting adjustment is not variable, the nonresponse adjustment factors can be quite large. The optimal allocations are designed to equalize response rates across domains, which can lead to occasionally “oversampling” in low-responding domains. Table 3.5 presents the RBE and the MSE for the $\text{Min}-\text{URR}$ optimal allocations, using three different constant values of $q\left(q=\text{0}\text{.30},\text{0}\text{.50},\text{0}\text{.70}\right)$ and the domain specific rates estimated from historical data $\text{(}{q}_h=$  Estimated). In all scenarios, the DE estimates are unbiased, the CR estimates are slightly biased, and the SR estimates are the most biased. This repeats the RBE pattern shown in the $\text{Constant}-K$ allocation results. Moreover, the RBE estimates do not appear to be overly sensitive to values of $q_h$ used in allocation. Again, even with the additional rounds of NRFU, the bias of the subsamples’ estimates is larger than that obtained with full follow-up of nonrespondents. In all cases, the MSE of the estimates obtained from the optimal allocations are smaller than those obtained with the $\text{Constant}-K$ allocations.

Regardless of estimator, the bias decreases when eventually probability of responding is low. This seems a bit counterintuitive but is in fact a direct consequence of the subsampling allocation procedure. When the probability of obtaining an eventual response is low, the $\text{Min}-\text{URR}$ allocation tends to subsample all or no units in a domain. With full follow-up, all responding units within the same domain have the same nonresponse adjustment. With a subsample, only the responding subsampled units’ weights are adjusted for nonresponse and subsampling, in turn occasionally creating extremely variable weights within domain. As the probability of an eventual response increases, then the optimal allocation has sample in more domains, and finer adjustments are possible. With that said, the CR estimators tend to produce the lowest MSEs, regardless of allocation.

The $\text{Min}-K$ allocation procedure is designed to reduce the variability in the subsampled units’ adjustment weights. Table 3.6 presents the relative bias of the estimate and MSE for the $\text{Min}-K$ optimal allocation method. The $\text{Min}-K$ estimators display the same pattern as before. The DE estimates are unbiased, the CR estimates are nearly unbiased and the SR estimates are slightly biased.

The MSE estimates for the $\text{Min}-K$ method follow a similar pattern as the $\text{Min}-\text{URR}$ method, as expected due to the similarities between corresponding $\text{Min}-\text{URR}$ and $\text{Min}-K$ allocations. These results appear to be relatively insensitive to assumed eventual probability of response $\left(q\right).$ The historical-data estimated conversion rates produce similar results to an assumed $q=$  0.50. In many cases, the $\text{Min}-\text{URR}$ method produces the least biased estimates. However, bias is only a single component of the MSE, and the $\text{Min}-\text{URR}$ allocations tend to have smaller expected number of respondents in several strata than their $\text{Min}-K$ counterparts. Moreover, the $\text{Min}-K$ allocations have smaller sampling variances by design, ultimately yielding estimates with lower MSEs than their $\text{Min}-\text{URR}$ counterparts.

Figures 3.1 and 3.2 plot the RBEs and MSEs obtained at each round of NRFU for the CR estimator (our “best” estimator) using the $q_h$ obtained from historical data for each of the considered optimal allocation methods along with the benchmark values (labeled as “Full Follow-up”). In Figure 3.1, the benchmark estimates are the least biased. However, this extremely low bias is in part a consequence of our nonresponse model, which is uniform within domain and NRFU phase. Neither of the optimal allocation estimates attained the benchmark estimate levels, but they become very close after seven rounds of NRFU and the RBEs of the $\text{Min}-\text{URR}$ and $\text{Min}-K$ CR estimates are less than one tenth of one percent (0.06% and 0.05% respectively). In summary, subsampling with either optimal allocation strategy yielded trivial biases increases over full follow-up.

Figure 3.2 plots MSE values by NRFU round using the CR estimator. The targeted nonresponse sampling strategy used for the $\text{Min}-K$ allocation appears to reduce the overall error. We believe that this is due to two factors. First, the $\text{Min}-K$ allocation procedure samples larger proportions of nonrespondents in low responding areas than obtained with the $\text{Min}-\text{URR}$ allocations. Second, the quadratic formula for the $\text{Min}-K$ allocation includes a constraint on the domain response rates, lowering the overall target response but reducing the variability in the proportion of respondents by domain. Ultimately, this approach yields similar response rates across sampling domains, indicative of a representative sample (Wagner, 2012; Schouten, Cobben and Bethlehem, 2009). Note that the increased MSE is not trivial with nonrespondent subsampling, even when using an adjustment procedure that benefits from a strong covariate in the ratio adjustment procedure. This is an acknowledged price paid for nonrespondent subsampling (Biemer, 2010). However, this additional variance component is measurable. If the measured component is too large, the program managers can subsample less (use a larger $K).$

3.4  Discussion

Given a sophisticated allocation method, a ratio estimator employing a highly correlated auxiliary variable, and a fairly large subsample, this case study shows that nonrespondent subsampling does not overly penalize quality to save cost. The additional stage of sampling increased the MSE for the studied variable, but the level was reduced by the judicious choice of estimator. Of course, we consider only one variable in our simulation, and this variable may or may not “behave” similarly to other survey items. One referee suggested the usage of an R-indicator (Schouten et al., 2009) or balance indicator (Särndal and Lundquist, 2014) to assess the overall representativeness of the respondent sets in a field survey setting. This might be useful at later stages of data collection (after nonrespondent subsampling and during NRFU), but would not provide any further insight into the degree of bias reduction on any collected item, as we can do in this simulation setting.

Of the three considered allocation methods, the $\text{Constant}-K$ method had the worst performance, often selecting a very small probability subsample when not needed and consequently increasing the sampling variance without reducing the bias. Of the three considered allocation methods, the $\text{Min}-K$ allocation was the most effective in realizing acceptable response rates and achieving reliable estimates; the larger bias caused by the varying domain sampling intervals is generally offset by the reduced sampling variance. However, implementation of the $\text{Min}-K$ allocation can be more challenging than the $\text{Min}-\text{URR}\text{.}$

For both optimal allocation procedures, we tested four different eventual probabilities of response to assess the sensitivity of the allocation procedures to these inputs. By comparing allocations obtained with a constant assumed input value to those obtained using the empirical estimates, we found that the realized allocations could over- or under- sample in selected domain, and the domain response rates could vary more than expected when the actual (survey) values are quite different from the input values. Consequently, we recommend using values estimated from historic paradata whenever possible.

If reducing cost is the overall goal, then we note that additional NRFU contact attempts beyond the fifth contact did not improve the bias or MSE of the subsampled estimates in our case study. Of course, if the achieved cost reduction for a $1-\text{in}-2$ subsample with up to ten NRFU contact attempts is acceptable, the funds allocated to these final contact attempts might be better expended earlier in the collection cycles using other contact strategies.

• Previous
• Article main page
• Next