2 What are adaptive survey designs?

Barry Schouten, Melania Calinescu and Annemieke Luiten

Previous | Next

2.1  Adaptive survey designs in general

In this section, a mathematical framework is set out for adaptive survey designs. In subsequent sections, components of this framework are highlighted and elaborated.

Let the population consist of units $k=1,2,\dots ,N.$ The population of interest may consist of all units in a population but also of all recruited members of a panel. Each unit will be assigned a strategy $s$ from the set of candidate strategies $S=\{\phi ,s_1,s_2,\dots ,s_M\}.$ In the survey strategy set $S$ the empty strategy $\phi$ is explicitly included. The empty strategy means that no action is undertaken, i.e., the population unit is not sampled. This is the most general framework. In practice, one will often separate the sampling design from the strategy allocation and view the sample as given and fixed. However, one may include the decision to sample a unit explicitly in the overall allocation of resources.

In general a strategy $s$ is a specified set of design features and may involve a sequence of treatments where treatments are only followed when all previous treatments failed. Some of those features may be sequential such as the type of contact mode and the type of survey mode, but the features may also describe different aspects of a survey design. Examples of strategies are

• $s_1=$(advance letter 1, web questionnaire, one reminder);
• $s_2=$(advance letter 1, web questionnaire, no reminder);
• $s_3=$(advance letter 2, CATI administered, maximum of six call attempts);
• $s_4=$(advance letter 2, CATI administered, maximum of 15 call attempts).

In the literature many design features are suggested and evaluated, e.g., Groves and Couper (1998) and Groves, Dillman, Eltinge and Little (2002). We refer to De Leeuw (2008) for a discussion of survey modes, to Dillman (2007) and De Leeuw, Callegaro, Hox, Korendijk and Lensvelt-Mulders (2007) for an elaboration of advance letters and reminders, to Wagner (2008) for a discussion of contact protocol, to Barón, Breunig, Cobb-Clark, Gørgens and Sartbayeva (2009) for a review of incentives, to Kersten and Bethlehem (1984), Cobben (2009) and Lynn (2003) for research into condensed questionnaires, to Moore (1988) for a discussion of proxy reporting, and to Cobben (2009) for an example of interviewer assignment.

It is assumed in this paper that the set of strategies $S$ is known and fixed when strategy allocation is started. The set of strategies may be identified based on historical survey data, experience and pilot studies. We refer to Schouten, Luiten, Loosveldt, Beullens and Kleven (2010) and Schouten, Shlomo and Skinner (2011) for guidelines and examples on how to construct strategy sets.

With each population unit $k$ a vector of covariates $X_k={(X_{k1},X_{k2},\dots ,X_{kp})}^T$ is associated. $X_k$ contains characteristics that are known before data collection starts and before strategies are allocated. The covariates must, therefore, be available in registrations or administrative data that can be linked to the sampling frame or in the sampling frame itself. Next to these general characteristics a second vector of covariates ${\tilde{X}}_k={({\tilde{X}}_{k1},{\tilde{X}}_{k2},\dots ,{\tilde{X}}_{kq})}^T$ may exist for unit $k$ that reflects characteristics observed during data collection for sampled population units. These characteristics are termed paradata or process data because they are collected during the process of data collection by interviewers and data collection staff. However, other than the more traditional view on paradata as information about the process, in the adaptive survey design context ${\tilde{X}}_k$ contains observations about the sampled person or household. Examples of $X_k$ are gender, age, type of household or educational level. Examples of ${\tilde{X}}_k$ are the interviewer assessment of the propensity to respond or the propensity to be contacted, the state of the dwelling or the neighborhood, and the presence of an intercom. ${\tilde{X}}_k$ is deliberately restricted to observations about the sample that allow for differentiation of survey design features. It does not contain the values of the design features themselves such as the interviewer that was assigned to the address.

The important distinction between $X_k$ and ${\tilde{X}}_k$ is the level of availability. ${\tilde{X}}_k$ is known only for those units that are sampled and cannot be used in distinguishing subpopulations a priori. Let $q(x)$ represent the distribution of $X_k$ in the population and $q(\tilde{x},x)$ the joint distribution of $X_k$ and ${\tilde{X}}_k$ in the sample. Furthermore, $q(\tilde{x}|x)$ denotes the conditional sample distribution. It is assumed that $q(x)$ and $q(\tilde{x},x)$ are known in advance. In settings where no or little data can be linked, strategy allocation must be based fully on observations made during data collection.

Adaptive survey designs that allocate strategies based on population characteristics available in registry and frame data are termed static, while adaptive survey designs that allocate strategies that depend (also) on paradata are termed dynamic. It is important to remark that both static and dynamic designs have a strategy set that is fixed before data collection starts. However, for dynamic designs it is not known beforehand which strategies are going to be assigned to individual units because the choice of strategy depends on data that are observed during data collection.

Let $\rho (x,s)$ be the response propensity of a unit carrying characteristic $X=x$ and that is assigned strategy $s.$ It is assumed that $\rho (x,s)$ is available from historic data, i.e., from previous versions of the same survey, from surveys with similar topics and designs or from initial design phases. Obviously, the anticipated response propensity must be a close estimate of the true propensity. Section 2.4 returns to this essential component of adaptive survey designs.

The expected costs of the assignment of strategy $s$ to a unit with $X=x$ is denoted as $c(x,s).$ It is an individual cost component. Literature tells us that survey costs consist of many components of which some are overhead and others are individual, e.g., Groves (1989). Section 2.3 discusses cost functions.

Let $p(s|x)$ be the allocation probability of a population unit with characteristics $x$ for strategy $s,$ and let $p(s|x,\tilde{x})$ be the allocation probability to that strategy given that also paradata $\tilde{x}$ are observed. The following must hold

$0\le p(s|x)\le 1,0\le p(s|x,\tilde{x})\le 1$(2.1)

${\displaystyle {\sum }_{s}p(s|x)}=1,{\displaystyle {\sum }_{s}p(s|x,\tilde{x})}=1,$(2.2)

i.e., all units are assigned a strategy. In general, allocation probabilities may have values between 0 and 1. In other words subpopulations with the same scores on $x$ and $\tilde{x}$ may be (randomly) assigned to different strategies. For instance, only part of the non-respondents may be re-approached in a follow-up. Allowing for allocation probabilities between 0 and 1 increases the flexibility in meeting quality levels or cost constraints. In the following, $p$ denotes the matrix of allocation probabilities, i.e., $p={\{p(s_j|x,\tilde{x})\}}_{1\le j\le M,x,\tilde{x}}$ and contains the decision variables in the optimization.

The response propensities ${\rho }_X$ can be derived from the strategy response propensities and the allocation probabilities by

${\rho }_X(x)={\displaystyle {\sum }_{s\in S}{\displaystyle {\sum }_{\tilde{x}}q(\tilde{x}|x)p(s|x,\tilde{x})\rho (s,x,\tilde{x})}}.$(2.3)

The strategies, covariates, response propensities, cost functions and allocation probabilities form the ingredients to adaptive survey designs. With these building blocks the adaptive survey design optimization problem can be formulated. Two ingredients are still missing, however, a quality function and an overall cost function. Let $Q(p)$ be some indicator of quality and $C(p)$ be an evaluation of total costs. The dependence on the allocation probabilities in both functions is stressed as the probabilities are the decision variables in the optimization.

The optimization problem can now be formulated as

${\max }_{p}Q(p)$ given that $C(p)\le C_{\max }$(2.4)

or as

${\min }_{p}C(p)$ given that $Q(p)\ge Q_{\min },$(2.5)

where $C_{\max }$ represents the budget for a survey and $Q_{\min }$ minimum quality constraints. Problems (2.4) and (2.5) are called dual optimization problems, although the solutions to both problems may be different depending on the quality and cost constraints.

It is important to stress that the optimization of quality or costs is done only once, before survey data collection starts, and is not repeated during data collection. Hence, it is the strategy that is adapted to the population unit, and in case of a dynamic design to paradata about that unit, but it is not the optimization itself that is adapted. The optimization is based on historic survey data that includes the paradata that has become available in a survey. The joint density function $q(x,\tilde{x}),$ the response probabilities $\rho (x,\tilde{x},s)$ and the cost function $c(x,\tilde{x},s)$ are all estimated from historic survey data and are assumed to be given. Since in practice paradata becomes available only during data collection, the candidate strategies for units in the same stratum $x$ are the same up to the moment the paradata $\tilde{x}$ becomes available. For instance, there may be the following four strategies: 1) two telephone call attempts and no follow-up, 2) two telephone call attempts and a follow-up with incentives, 3) three telephone call attempts and no follow-up, and 4) three telephone call attempts and a follow-up with incentives. The decision to make two or three call attempts is based on $x,$ while the follow up is decided upon using a telephone paradata observation $\tilde{x}.$ Thus, beforehand, it is estimated how many units will fall in stratum $(x,\tilde{x})$ and how many will receive a follow up, but only when $\tilde{x}$ is measured, the full strategy is known for individual units.

2.2  Quality objective functions

Adaptive survey designs, as discussed by the literature, typically focus on nonresponse error. In this section, we start with a general classification of quality functions, and then move to quality functions for nonresponse error. In general, a focus on nonresponse error is too narrow a view, especially, when the survey mode is one of the candidate design features in the adaptive survey design. Here, we do, however, not explicitly discuss other survey errors, but we return to this issue in the discussion. We refer to Calinescu, Schouten and Bhulai (2012) for an extension of adaptive survey designs to measurement error and Beaumont and Haziza (2011) for a discussion on adaptive survey designs and nonresponse variance.

2.2.1  Covariate-based and item-based quality functions

When quality is optimized according to (2.4), then quality functions map the survey sample with linked data, paradata and answers to survey items to a single value which can be interpreted and optimized. When costs are minimized subject to constraints on quality as in (2.5), then quality may be multi-dimensional (but cost functions should be one-dimensional).

In general, two types of quality functions can be distinguished; quality functions that employ covariates from linked data and paradata only, and quality functions that also employ the answers to the survey target variables. We refer to them as covariate-based and item-based, respectively. An item-based quality function is a function of the response distribution of a survey item and the anticipated, estimated full population distribution given the available linked data and paradata. The main distinction between covariate-based and item-based quality functions is that item-based quality requires assumptions. Evidently, the answers of nonrespondents are missing. Hence, quality evaluation must be based on relations between target variables and covariates as observed in the response. As a consequence, there is a risk attached to item-based quality functions that originates directly from the phenomenon it attempts to measure. Relations between target variables and covariates may be different for nonrespondents and item-based quality may pose an incomplete image. Furthermore, in surveys with many survey target variables, different target variables may lead to different decisions and optimal survey designs. However, contrary to covariate-based quality functions, item-based quality functions tailor survey designs specifically to the topics of the survey. Covariate-based quality functions can only be related to the nonresponse bias of the covariates that are included.

2.2.2  Optimizing quality of response

We, first, describe briefly a number of quality functions that have appeared in recent literature. Next, we discuss the choice of a quality function.

The most well-known covariate-based quality function for nonresponse is the response rate. It is not a true covariate-based quality function in the sense that it depends on linked data or paradata. However, since the 0-1 response indicator may be viewed as the simplest form of paradata, it is termed a covariate-based quality function. The response rate is represented as the mean response propensity

Response rate:               $Q(p)=\overline{\rho }={\displaystyle {\sum }_{x,\tilde{x},s}q(x,\tilde{x})}p(s|x,\tilde{x})\rho (x,\tilde{x},s)$(2.6)

Schouten, Cobben and Bethlehem (2009) propose two covariate-based quality functions, the R-indicator and a measure they call the maximal or worst-case nonresponse bias. The label of the second indicator is misleading as it is only an estimator of the maximal bias of the unadjusted mean of respondents, not the true maximal bias. A better label is the coefficient of variation of response propensities, which we will use here. The measures can be written as

R-indicator:               $Q(p)=R({\rho }_Z)=1-2S({\rho }_Z)$(2.7)

Coefficient of variation:               $Q(p)=CV({\rho }_Z)=\frac{S({\rho }_Z)}{\overline{\rho }},$(2.8)

where the representativeness may be evaluated with respect to linked data only, $Z=X,$ or with respect to a vector containing both linked data and paradata, $Z={(X,\tilde{X})}^T.$ The standard deviations of the response propensities, $S({\rho }_X)$ and $S({\rho }_{X,\tilde{X}}),$ can be written in terms of the strategy allocation probabilities as

$S({\rho }_X)=\sqrt{{\displaystyle {\sum }_{x}q(x)}{\left({\displaystyle {\sum }_{\tilde{x},s}q(\tilde{x}|x)p(s|x,\tilde{x})\widehat{\rho }(x,\tilde{x},s)}-\overline{\rho }\right)}^2}$(2.9a)

$S({\rho }_{X,\tilde{X}})=\sqrt{{{\displaystyle {\sum }_{x,\tilde{x}}q(x,\tilde{x})\left({\displaystyle {\sum }_{s}p(s|x,\tilde{x})\widehat{\rho }(x,\tilde{x},s)}-\overline{\rho }\right)}}^2}.$(2.9b)

Särndal and Lundström (2010) and Särndal (2011a and b) propose indicators that are very similar in definition and nature to (2.7) and (2.8). These indicators were derived from the perspective of calibration and so-called balanced response, and may be used as alternatives to (2.7) or (2.8).

An example of an item-based quality function for nonresponse is presented by Groves and Heeringa (2006). For a specific target variable $Y,$ Groves and Heeringa (2006) suggest the nonresponse bias of the unadjusted mean of respondents

             

$$\text{Estimated nonresponse bias : }Q(p)=\frac{\mathrm{cov}(Y,{\rho }_X)}{\overline{\rho }},\text{ (2.10)}$$

with $\mathrm{cov}(Y,{\rho }_X)$ the response covariance between the target variable and the response propensities given covariates $X.$ It can be written as

$$\mathrm{cov}(Y,{\rho }_X)=\frac{{\displaystyle {\sum }_{x}q(x){\rho }_X(x)}({\rho }_X(x)-{\overline{\rho }}_X)(y(x)-{\overline{y}}_R)}{\overline{\rho }},\text{ (2.11)}$$

with ${\rho }_X(x)$ as in (2.3), $y(x)$ the mean value of $Y$ for $X=x$ and ${\overline{y}}_R$ the expected mean of respondents. Again, (2.11) can be extended to include paradata $\tilde{X}.$

All quality functions in this section are defined as population parameters. In practice, they need to be estimated from survey data. The true ${\rho }_X(x)$ need to be replaced by estimators ${\widehat{\rho }}_X(x),$ based on some form of regression, and the summations over the population will be replaced by design weighted summations over the sample. We return to the estimation of propensities in Section 2.4.

Now, how to choose a quality function? All quality functions mentioned here attempt to measure the impact of nonresponse beyond that of a mere reduction in sample size. They do this based on covariates from linked data and paradata. The rationale behind optimizing these quality functions is that stronger traces of nonresponse error on these covariates may imply larger nonresponse error on other variables as well; the quality functions are viewed as process quality indicators rather than product quality indicators. Although appealing, this conjecture clearly needs empirical support. The choice of a quality function for nonresponse should be based on the set of key survey variables, the population parameters of interest and the estimators that are going to be employed. The response rate and R-indicator do not aim at a specific population parameter or estimator. The coefficient of variation focuses on population means, but it is not specific to any survey variable or nonresponse adjustment, while the estimated nonresponse bias originates from the same perspective, but it is applied to a single survey variable. If a survey carries multiple key survey variables, then an item-based quality function for nonresponse is to be avoided, as it may lead to conflicting optimization problems. If a survey has a single key variable, then it is effective to either use an item-based quality function or to restrict to the most relevant covariates only in covariate-based quality functions. If a survey has multiple uses, then, in our view, it is too restrictive to focus on a specific population parameter and estimator, and we favor the R-indicator to any other quality function. If it can be expected that users will focus on population means or totals, then the coefficient of variation is to be preferred, in our opinion, as it does not assume a specific adjustment method.

However, even more important than the choice of the indicator is the set of linked data and paradata that are input to the indicator. If a survey has one or only a few key variables, then the selected linked data and paradata can and should relate to those variables. If, however, a survey has a wide range of survey variables, then one must restrict necessarily to auxiliary variables that generally distinguish persons or households.

So far, we have ignored the impact of nonresponse on precision, while requirements for the precision of the survey estimates may be given explicitly. There are two options to include precision in the optimization. First, one may add an additional constraint. The straightforward choice would be a constraint on the minimum number of respondents, possibly for a number of population subgroups; thereby avoiding to specify the population parameters and estimators. Second, the nonresponse variance may be included in the quality function itself, i.e., one would consider indicators for the mean square error. This option is proposed and elaborated by Beaumont and Haziza (2011). Under the second option, one again has to consider the set of key survey variables, the population parameter and the estimation strategy, as precision is specific to an estimator for some population parameter for a single survey variable.

2.3  Cost functions

Cost functions are the counterpart of the quality functions. There are several components to cost functions. It is important to restrict specification of costs relative to the design features that are varied in the adaptive survey design. For example, when it is the incentive that is differentiated with respect to different subpopulations, then costs need not be specified and detailed for interviewer traveling times. When it is the contact timing protocol that is the design feature that may be tailored, then, obviously, traveling times and traveling costs play a dominant role.

If a large number of design features is optional, then the cost functions have complex forms with many overhead and variable cost components. Generally, variable costs depend on the sample size while overhead costs do not. Overhead cost components may come from data collection staff, sampling and processing of samples. Variable costs arise for example from training and instruction of interviewers, mail and print of questionnaires and reminders, processing of paper questionnaires, interviewer hour rates and travel expenses, incentives and telephone number linkage, telephone usage and computer servers.

In the optimization two cost components may be identified: a fixed and a variable component. The variable component depends on the allocation of population units to strategies while the fixed component consists of all remaining costs. It must be stressed that the fixed component is different for adaptive survey designs that focus on different design features. The cost function $C(p)$ is the sum of two components

$C(p)=C_F+C_V(p),$               (2.12)

of which only the second, the variable component, depends on the allocation probabilities.

In general, with a survey strategy $s,$ costs $c(\tilde{x},x,s)$ are associated with population units from group $(\tilde{x},x).$ The individual cost function may be a function of response propensities, or even more specifically of contact and participation propensities. For instance, the interviewer costs in different contact timing protocols depend on the contact rates of the selected subpopulations. The cost function $c(\tilde{x},x,s)$ is a relative cost function as it describes only the contribution of the strategy to the variable cost component $C_V(p)$

$C_V(p)={\displaystyle {\sum }_{\tilde{x},x,s}q(\tilde{x},x)}p(s|x,\tilde{x})c(\tilde{x},x,s).$(2.13)

Three remarks are in order. First, the derivation of fixed and variable cost components is complicated when a survey organization runs many surveys in parallel. On the one hand, the interaction between surveys makes it hard to separate costs per survey, especially when strategies are tailored. On the other hand, when multiple surveys are conducted some of the variable costs components may be labeled as fixed. For example, when only a relatively small number of population units are assigned to the face-to-face survey mode, then traveling costs may be assumed to remain unchanged as the addresses are clustered with addresses from other surveys. The second remark concerns the multidimensional aspect of costs. Apart from the overall budget it may be requested that interviewer occupation rates are close to one throughout time or that none of the interviewers has to work overtime more than a fixed amount of time. As a consequence, the cost function becomes a vector and the constraint a vector of constraints. The third remark concerns the validity of the cost functions. Since cost functions are hard to construct in practice, it may turn out that the optimization was too optimistic. It is important to monitor data collection closely and to build in indicators for strategies.

2.4  Estimating response probabilities

Next to cost parameters and quality functions, the other important ingredient of adaptive survey designs is the set of response propensities for the various strategies. Such propensities need to be known from past surveys, preferably the same survey or otherwise a similar survey. Alternatively, as Groves and Heeringa (2006) propose, one may use earlier phases of the data collection to learn and derive propensities. This will be at the expense of efficiency since part of the survey is already conducted. Nonetheless, the gathered information directly feeds back to the current survey.

Literature on household surveys gives an extensive list of models for response that include design features. The common denominator in all models is that response propensities are estimated based on a number of assumptions about the true nature of the nonresponse missing-data mechanism. In general such models are simplifications. Consequently, anticipated response propensities $\rho (x,s)$ have a standard error, and may even be biased themselves when they are based on similar, but different surveys. In the optimization, this uncertainty can be accounted for by allowing response propensities to be random variables rather than fixed quantities. The randomness demands for sensitivity analyses and evaluations of the robustness of the optimization that provide insight into the variation of quality and costs when the survey is conducted multiple times (under the same circumstances).

2.5  The optimization problem

One may take two approaches to the optimization of (2.3) and (2.4): a trial-and-error approach or a mathematical optimization. In this paper, we concentrate on a mathematical framework and optimization, but one may be more modest and introduce adaptive survey designs gradually through pilot studies and field tests.

Quality functions (2.6), (2.7), (2.8) and (2.10) all are functions of the strategy allocation probabilities $p.$ The response rate is a linear function of the allocation probabilities, which makes it relatively easy to optimize using standard optimization software (e.g., the linprog package in R or any other software that can address linear programming problems). Still, as far as we know, due to the high dimensionality of $p$ there is no closed form solution to (2.4) even for linear problems. In general, however, the quality functions are nonlinear, nonconvex functions with respect to the allocation probabilities, and cannot be optimized without numerical or Monte Carlo methods. The complexity of the problem grows quickly as a function of the number of candidate strategies and the number of subgroups based on linked data and paradata.

Current statistical softwares contain procedures or packages that can handle nonlinear optimization problems, like nlm or nlminb in R or proc optmodel in SAS. However, nonlinear nonconvex problems may require long computational times or may converge to local optima. For this reason, specialized optimization softwares such as Xpress, Baron or AMPL are recommended.

In the examples of Section 3 and 4, we perform a number of optimizations. The optimization problem of Section 3 is relatively simple; the quality objective function is the R-indicator which is evaluated against two population subgroups. For two subgroups the optimization can be rewritten as a linear programming problem. For the example of Section 4, we were able to construct an algorithm that converges to the optimal solution in a small number of steps. All optimizations were programmed in R and the code is available upon request.

Previous | Next