Strategies for subsampling nonrespondents for economic programs
Section 1. Introduction

• Previous
• Next

Many federal programs are simultaneously experiencing declining response rates and reductions in funding. At the same time, these programs are required to maintain predetermined reliability levels and are encouraged to collect an increased number of data items and to publish more statistics. Of course, as nonresponse increases, the precision of the survey estimates will decrease from the original design levels and can be sensitive to nonresponse bias. Consequently, many federal agencies are investigating adaptive collection design strategies, where the term “collection design” refers to protocol(s) for collecting data.

With business surveys, the collection design may vary by type of unit. These populations are generally highly skewed; the majority of a tabulated total in a given industry is often provided by a small number of large businesses. Because publication statistics are generally industry totals or percentage change, missing data from the largest cases can induce substantive nonresponse bias in the totals, whereas missing data from the smaller cases (even those with large sampling weights) often have little apparent effect on the tabulated levels (Thompson and Washington, 2013). Thus, the contact strategies are designed to ensure that the largest cases provide valid response data. Figure 1.1 illustrates nonresponse follow-up (NRFU) procedures that differ by a survey-specific unit size classification, where both collection designs have fixed calendar schedules and a fixed NRFU budget.

For the large unit category, the NRFU procedures become progressively more costly (per unit) with the exception of the final contact attempt. In contrast, with the smaller units, the NRFU procedures do not include personal contact and are therefore less expensive.

Selecting a probability subsample of nonrespondents is a strategic feature of many responsive and adaptive collection designs (Tourangeau, Brick, Lohr and Li, 2016). Of course, this is not a new practice for surveys. Indeed, nonrespondent subsampling has been a survey practice since first discussed in Hansen and Hurwitz (1946). Actually, the setting of the two-phase sample approach presented in Hansen and Hurwitz (1946) paper is quite similar to the business survey setting discussed here: an “inexpensive” mailed questionnaire to all sampled units (c.f. the “21^st century design” that mails a letter containing a URL, user name, and password), followed by “expensive” personal interviews of subsampled nonrespondents (c.f. personal phone calls or certified reminder letters). Their proposed optimal allocation procedures are not entirely dissimilar either, with the final allocations being highly dependent on whether the response rates for each collection mode are known or estimated using auxiliary data rather than the previously collected responses.

Fitting nonrespondent subsampling into a responsive or adaptive design framework is straightforward. As originally proposed by Groves and Herringa (2006), responsive designs require a minimum of two distinct phases of collection, with the second phase often being a probability subsample of nonrespondents that occurs at the “phase capacity” when the survey estimates are no longer changing, providing evidence the existing collection protocol is no longer cost-effective. Schouten, Calinescu and Luiten (2013) characterize responsive designs as a special case of adaptive collection designs. With an adaptive collection design, the data collection procedures can change (adapt) during the collection period. Paradata and sample data are used to determine whether to change the current procedures. The overall budget is fixed, but the implementation of a given strategy depends on (1) the realized sample of respondents at a point in time, (2) informative data obtained during data collection about the respondents and nonrespondents, and (3) information known in advance about the survey unit from the sampling frame. Consequently, selecting a probability sample of nonrespondents for NRFU $–$ instead of attempting to contact all nonrespondents $–$ falls under the adaptive design umbrella, with paradata (specifically response status) used to determine the sampling frame and frame data (e.g., the unit’s size and industry classification) used as the basis of the sample design.

The U.S. Census Bureau is investigating nonrespondent subsampling strategies for the 2017 Economic Census (EC). Although a single program, the EC employs different sampling designs by sector (Probability proportional to size for the Construction sector, cut-off sampling for the Manufacturing and Mining sectors in collections prior to 2017, complete enumeration for the Wholesale Trade sector, and stratified simple random sampling without replacement (SRS-WOR) in the remaining sectors). Moreover, as is typical with many business programs, it is a multi-purpose collection, with the general statistics items collected from all surveyed units in a sector: examples include $–$ but are not limited to $–$ receipts/shipments, annual and first quarter payroll, and total employment. In addition, the EC collects information on product sales, types of which differ by sector and often industry. Imputation procedures differ by item, as do the estimators. Consequently, the subsampling design must be robust to sampling and estimator to the largest extent possible. We consider a systematic sample of nonrespondents sorted by a measure of size, a sampling design known to be as efficient as stratified simple random sampling without replacement (SRS-WOR) on average if the list is in random order and more efficient if the list is monotonic increasing or decreasing (Zhang, 2008; Lohr, 2010, Chapter 2, pages 50-51).

Ideally, the nonrespondent subsampling allocation procedure should be informed by properties of the respondent sample during the collection period. Of course, if the program is designed to collect one or two key items, then the allocation procedures should (at least attempt to) directly incorporate information on the survey design and estimation procedure, as well as detailed cost information, as proposed in Hansen and Hurwitz (1946) long-ago. In this case, one should use an optimal allocation procedure that minimizes costs subject to (estimable) reliability constraints. See Harter, Mach, Chaplin and Wolken (2007) and Beaumont, Bocci and Haziza (2014) for examples.

Such optimization is difficult to accomplish in the considered multi-purpose survey setting, especially when strongly correlated auxiliary variables are not available for all items. However, the OMB Statistical Standards for federal surveys require “survey (design) to achieve the highest practical rates of response, commensurate with the importance of survey uses, respondent burden, and data collection costs” and mandate nonresponse bias analyses for programs that fail to achieve these rates (Federal Register Notice, 2006). For nonrespondent subsampling occurring during the data collection cycle, imposing mandated lower bounds on the program-level response rate and in specified domains (examples include sampling strata or other post-strata such as industry code or type of government) is therefore a natural constraint to include in the allocation procedure.

In this paper, we explore allocation approaches that address such constraints, with an overall objective of selecting larger systematic subsamples in domains that have lower-than-targeted response rates. We introduce two optimized allocation procedures, both formulated as quadratic programs and solved with standard software packages: one that minimizes deviations between domain unit response rates and one that minimizes deviations between domain subsampling intervals. Our case study compares the statistical properties of subsamples obtained from each proposed allocation with three different estimators, considering two ratio estimators commonly used by business surveys along with the simple expansion (Horvitz-Thompson) estimator. The latter is not necessarily the most precise estimator when highly correlated auxiliary data are available, but gives an “upper bound” on the variance increase due to subsampling. The ratio estimators were selected to illustrate that the subsampling variance component can be reduced by incorporating correlated auxiliary data at the estimation stage.

Note that the presented allocation procedure is designed specifically for business surveys and implicitly assumes that largest units are excluded from the subsampling. In this case, the overall cost savings may not be substantial because the majority of a program’s NRFU budget will be likely allocated to obtaining responses from the designated larger cases. However, the estimate quality can be improved. By equalizing response rates in considered domains, we hope to reduce the bias of the estimates by obtaining a respondent set that resembles the parent sample. Moreover, equalizing the subsampling intervals should help avoid overly increasing the sampling variance due to the second phase of selection, an unpleasant side effect of the additional stage of sampling that can completely offset any bias reduction obtained via the probability subsample (Biemer, 2010). And, it may be possible to further reduce both nonresponse bias and subsampling variance via an improved ratio or regression estimation procedure, if related covariates are available.

Section 2 provides context, briefly introduces the studied estimators, and presents our allocation procedures. Section 3 presents a simulation study that compares the statistical properties of the considered estimators for each realized allocation. We conclude in Section 4 with recommendations and suggestions for future research.

• Previous
• Article main page
• Next