Adaptive survey designs to minimize survey mode effects – a case study on the Dutch Labor Force Survey 2. The multi-mode optimization problemAdaptive survey designs to minimize survey mode effects – a case study on the Dutch Labor Force Survey 2. The multi-mode optimization problem

In this section, we construct the multi-mode optimization problem that accounts for mode effects on a single key survey variable. Apart from the survey mode, we also consider caps on the number of calls in telephone and face-to-face as design features in the optimization. In the optimization model, we allow different design features to be assigned to different subpopulations. Hence, the optimization may lead to an adaptive survey design; it does so when the optimal allocation probabilities differ over the subpopulations. In our case, the subpopulations are built on linked administrative data. Note that they could also be built based on paradata collected during the early stages of the survey. The last component to the optimization problem is given by a set of explicit quality and cost functions. In our case, the quality functions are derived from mode differences in selection and measurement bias and from requirements on the precision of statistics. As a cost function, we use the total variable costs of the survey design. In the following paragraphs, we discuss the components of the optimization problem.

We begin with the survey design features contained in the survey strategy set $\mathcal{S}.$ We consider single mode and sequential mixed-mode strategies, i.e., a strategies where nonrespondents in a mode are followed-up in another mode. A single mode would be labelled as $M$ and a sequential mixed-mode as ${M}_{1}\to {M}_{2}.$ We consider Web, telephone and face-to-face survey as the modes of interest and abbreviate them to $Web,$ $Tel$ and $F2F.$ Examples of single mode and sequential mixed mode are $Tel$ and $Web\to F2F,$ respectively. For interview modes, we additionally consider a cap $k$ on the number of calls, denoted as $Mk.$ For example, $F2F3$ denotes a single mode survey strategy that uses face-to-face with a maximum of three visits. We let $Mk+$ denote the counterpart strategy where there is no explicit cap. We do not consider concurrent mixed-mode strategies (two or more modes are offered simultaneously to sample units) in this paper. This restriction is without loss of generality. It would be straightforward to apply the methodology to any set of multi-mode strategies, including hybrid forms of sequential and concurrent mixed-mode strategies. A wide or diffuse set of strategies will, however, come at the cost of a larger number of input parameters that need to be estimated. The survey strategy set $\mathcal{S}$ explicitly includes the empty strategy, denoted by $\Phi ,$ which represents the case where a population unit is not sampled, i.e., no action is taken to get a response from the unit. We let ${\mathcal{S}}^{R}=\mathcal{S}\\left\{\Phi \right\}$ denote the set of real, non-empty strategies.

Population units are clustered into $\mathcal{G}=\left\{1,\dots ,G\right\}$ groups given a set of characteristics $X$ such as age, ethnicity, that can be extracted from external sources of data or from paradata. Let $p\left(s,g\right)$ be the allocation probability of strategy $s$ to group $g,$ i.e., a proportion $p\left(s,g\right)$ from subpopulation $g$ is sampled and approached through strategy $s.$ In general, it may hold that multiple strategies have non-zero allocation probabilities, so that the subpopulation is divided over multiple strategies. Define the allocation probability $p\left(\Phi ,g\right)$ as the probability that a unit from subpopulation $g$ is not included in the sample. The ratio $p\left(s,g\right)/\left(1-p\left(\Phi ,g\right)\right)$ is the probability that a unit is assigned strategy $s$ given that it has been sampled. For example, if only the allocation probabilities to the empty strategy $p\left(\Phi ,g\right)$ vary and the allocation probabilities $p\left(s,g\right),\forall s\in {\mathcal{S}}^{R}$ are equal conditional on being sampled, then the design is stratified but non-adaptive. The probabilities must satisfy

$∑ s∈ S R p( s,g ) +p( Φ,g ) = 1, ∀g∈G, 0≤p( s,g ) ≤ 1, ∀s∈S, g∈G. (2.1) MathType@MTEF@5@5@+= feaagKart1ev2aaatCvAUfeBSjuyZL2yd9gzLbvyNv2CaerbuLwBLn hiov2DGi1BTfMBaeXatLxBI9gBaerbd9wDYLwzYbItLDharqqtubsr 4rNCHbGeaGqiFu0Je9sqqrpepC0xbbL8F4rqqrpipeea0xe9Lq=Je9 vqaqFeFr0xbba9Fa0P0RWFb9fq0FXxbbf9=e0dfrpm0dXdirVu0=vr 0=vr0=fdbaqaaeGacaGaaiaabeqaamaabaabaaGcbaqbaeaabiWaaa qaamaaqafabaGaamiCamaabmaabaGaam4CaiaaiYcacaWGNbaacaGL OaGaayzkaaaaleaacaWGZbGaeyicI48efv3ySLgznfgDOfdaryqr1n gBPrginfgDObYtUvgaiuaacqWFse=udaahaaadbeqaaiaadkfaaaaa leqaniabggHiLdGccqGHRaWkcaWGWbWaaeWaaeaacqqHMoGrcaaISa Gaam4zaaGaayjkaiaawMcaaaqaaiabg2da9aqaaiaaigdacaaISaGa aGjbVlabgcGiIiaadEgacqGHiiIZcqWFge=rcaaISaaabaGaaGPaVl aaykW7caaMc8UaaGPaVlaaykW7caaMc8UaaGPaVlaaykW7caaMc8Ua aGPaVlaaykW7caaMc8UaaGPaVlaaykW7caaMc8UaaGjbVlaaicdacq GHKjYOcaWGWbWaaeWaaeaacaWGZbGaaGilaiaadEgaaiaawIcacaGL PaaaaeaacqGHKjYOaeaacaaIXaGaaGilaiaaysW7cqGHaiIicaWGZb GaeyicI4Sae8NeXpLaaGilaiaaysW7caWGNbGaeyicI4Sae8NbXFKa aGOlaaaacaaMf8UaaGzbVlaaywW7caaMf8UaaGzbVlaacIcacaaIYa GaaiOlaiaaigdacaGGPaaaaa@9A80@$

The allocation probabilities of survey strategies assigned to subpopulations $p\left(s,g\right)$ define the decision variables in the optimization model. More generally, and analogous to sampling designs, one could allow for dependencies between population units being sampled and/or being allocated to non-empty strategies $s\in {\mathcal{S}}^{R}.$ We will not add that complexity here, but assume independence.

We now discuss the quality and cost functions. We assume that the interest lies in estimating the population means of a survey variable $y.$ Given that we consider the survey mode as one of the design features, we view the nonresponse adjusted bias on $y$ between the proposed design and a specified benchmark design $\text{BM}$ as the main quality function. This bias may be viewed as the adjusted method effect with respect to $\text{BM},$ and it is a mix of mode-specific measurement biases and remaining mode-specific nonresponse biases after adjustment. If both the proposed design and the benchmark design are single mode, then the bias is a true (adjusted) mode effect. If one of the designs is multi-mode, then the bias represents a complex mixture of mode effects, see for instance Klausch, Hox and Schouten (2014).

Let ${N}_{g}$ be the population size of group $g,$ ${w}_{g}={N}_{g}/N$ be the proportion of group $g$ in the population of size $N,$ and $\rho \left(s,g\right)$ be the response propensity for group $g$ if strategy $s$ is assigned. For a specific group, we define the adjusted method effect as the nonresponse adjusted difference between the survey estimate ${\overline{y}}_{s,g}$ and a benchmark estimate ${\overline{y}}_{g}^{\text{BM}}$ of the population mean $\overline{Y},$ where the survey estimate ${\overline{y}}_{s,g}$ is obtained by allocating strategy $s\in {\mathcal{S}}^{R}$ to subpopulation $g\in \mathcal{G}.$ Let $D\left(s,g\right)$ denote this difference. The adjusted method effect is expressed as

$D( s,g )= y ¯ s,g − y ¯ g BM , ∀s∈ S R , g∈G. (2.2) MathType@MTEF@5@5@+= feaagKart1ev2aaatCvAUfeBSjuyZL2yd9gzLbvyNv2CaerbuLwBLn hiov2DGi1BTfMBaeXatLxBI9gBaerbd9wDYLwzYbItLDharqqtubsr 4rNCHbGeaGqiFu0Je9sqqrpepC0xbbL8F4rqqrpipeea0xe9Lq=Je9 vqaqFeFr0xbba9Fa0P0RWFb9fq0FXxbbf9=e0dfrpm0dXdirVu0=vr 0=vr0=fdbaqaaeGacaGaaiaabeqaamaabaabaaGcbaGaamiramaabm aabaGaam4CaiaaiYcacaWGNbaacaGLOaGaayzkaaGaeyypa0JabmyE ayaaraWaaSbaaSqaaiaadohacaaISaGaam4zaaqabaGccqGHsislce WG5bGbaebadaqhaaWcbaGaam4zaaqaaiaabkeacaqGnbaaaOGaaGil aiaaysW7cqGHaiIicaWGZbGaeyicI48efv3ySLgznfgDOfdaryqr1n gBPrginfgDObYtUvgaiuaacqWFse=udaahaaWcbeqaaiaadkfaaaGc caaISaGaaGjbVlaadEgacqGHiiIZcqWFge=rcaaIUaGaaGzbVlaayw W7caaMf8UaaGzbVlaaywW7caGGOaGaaGOmaiaac6cacaaIYaGaaiyk aaaa@6A82@$

For convenience, we omit the adjective “adjusted”' in the following and refer to $D\left(s,g\right)$ simply as the method effect.

In this paper, we seek to minimize the expected absolute overall method effect with respect to a given benchmark design $\text{BM},$ which is the weighted average of the method effects $D\left(s,g\right)$ per stratum and strategy to $\text{BM}.$ The expected absolute overall method effect with respect to $\text{BM}$ is equal to

$D ¯ BM =| ∑ g∈G w g ∑ s∈ S R p( s,g )ρ( s,g )D( s,g ) ∑ s∈ S R p( s,g )ρ( s,g ) |. (2.3) MathType@MTEF@5@5@+= feaagKart1ev2aaatCvAUfeBSjuyZL2yd9gzLbvyNv2CaerbuLwBLn hiov2DGi1BTfMBaeXatLxBI9gBaerbd9wDYLwzYbItLDharqqtubsr 4rNCHbGeaGqiFu0Je9sqqrpepC0xbbL8F4rqqrpipeea0xe9Lq=Je9 vqaqFeFr0xbba9Fa0P0RWFb9fq0FXxbbf9=e0dfrpm0dXdirVu0=vr 0=vr0=fdbaqaaeGacaGaaiaabeqaamaabaabaaGcbaGabmirayaara WaaWbaaSqabeaacaqGcbGaaeytaaaakiabg2da9maaemaabaGaaGPa VpaaqafabeWcbaGaam4zaiabgIGioprr1ngBPrwtHrhAXaqeguuDJX wAKbstHrhAG8KBLbacfaGae8NbXFeabeqdcqGHris5aOGaam4Damaa BaaaleaacaWGNbaabeaakmaalaaabaWaaabuaeqaleaacaWGZbGaey icI4Sae8NeXp1aaWbaaWqabeaacaWGsbaaaaWcbeqdcqGHris5aOGa amiCamaabmaabaGaam4CaiaaiYcacaWGNbaacaGLOaGaayzkaaGaeq yWdi3aaeWaaeaacaWGZbGaaGilaiaadEgaaiaawIcacaGLPaaacaWG ebWaaeWaaeaacaWGZbGaaGilaiaadEgaaiaawIcacaGLPaaaaeaada aeqbqabSqaaiaadohacqGHiiIZcqWFse=udaahaaadbeqaaiaadkfa aaaaleqaniabggHiLdGccaWGWbWaaeWaaeaacaWGZbGaaGilaiaadE gaaiaawIcacaGLPaaacqaHbpGCdaqadaqaaiaadohacaaISaGaam4z aaGaayjkaiaawMcaaaaacaaMc8oacaGLhWUaayjcSdGaaiOlaiaayw W7caaMf8UaaGzbVlaaywW7caaMf8UaaiikaiaaikdacaGGUaGaaG4m aiaacMcaaaa@8982@$

This objective function represents the expected shift in the time series of the key survey statistic when a redesign is implemented from the benchmark design to the adaptive design using allocation probabilities $p\left(s,g\right).$ If a survey is new or if the benchmark design was never actually fielded, the objective function represents the bias of the adaptive survey design to the benchmark design. It is, therefore, a very useful objective function. Note that ${\overline{y}}_{s,g}$ is a nonresponse adjusted estimate of $\overline{Y},$ while $\rho \left(s,g\right)$ is an unweighted estimate of the group $g$ response probability in strategy $s.$ We implicitly assume that the nonresponse adjustment does not influence the contribution of each group and strategy to the overall response. This allows us to write the objective function as in (2.4), while performing nonresponse adjustment within the optimization framework may lead to a very complex, perhaps even unsolvable, problem. We minimize the overall method effect ${\overline{D}}^{\text{BM}}$ by optimally assigning strategies $s\in {\mathcal{S}}^{R}$ to the groups $g\in \mathcal{G},$ i.e.,

$minimize p( s,g ) D ¯ BM . (2.4) MathType@MTEF@5@5@+= feaagKart1ev2aaatCvAUfeBSjuyZL2yd9gzLbvyNv2CaerbuLwBLn hiov2DGi1BTfMBaeXatLxBI9gBaerbd9wDYLwzYbItLDharqqtubsr 4rNCHbGeaGqiFu0Je9sqqrpepC0xbbL8F4rqqrpipeea0xe9Lq=Je9 vqaqFeFr0xbba9Fa0P0RWFb9fq0FXxbbf9=e0dfrpm0dXdirVu0=vr 0=vr0=fdbaqaaeGacaGaaiaabeqaamaabaabaaGcbaWaaCbeaeaaca qGTbGaaeyAaiaab6gacaqGPbGaaeyBaiaabMgacaqG6bGaaeyzaaWc baGaamiCamaabmaabaGaam4CaiaaiYcacaWGNbaacaGLOaGaayzkaa aabeaakiaaysW7ceWGebGbaebadaahaaWcbeqaaiaabkeacaqGnbaa aOGaaiOlaiaaywW7caaMf8UaaGzbVlaaywW7caaMf8Uaaiikaiaaik dacaGGUaGaaGinaiaacMcaaaa@5482@$

Ideally, ${\overline{D}}^{\text{BM}}=0.$ However, achieving this situation may have serious practical issues such as requiring unlimited resources. Therefore, various practical aspects such as scarcity in resources are reflected through a number of constraints in our model. A limited budget $B$ is available to setup and run the survey. Let $c\left(s,g\right)$ be the unit cost of applying strategy $s$ to one unit in group $g.$ The cost constraint is formulated as follows

$\sum _{s,g}{N}_{g}p\left(s,g\right)c\left(s,g\right)\le B.\text{ }\text{ }\text{ }\text{ }\text{ }\left(2.5\right)$

To ensure a minimal precision for the survey estimate of $\overline{Y},$ a minimum number ${R}_{g}$ of respondents per group is required. This translates to the following constraint

$∑ s∈ S R N g p( s,g )ρ( s,g )≥ R g , ∀g∈G. (2.6) MathType@MTEF@5@5@+= feaagKart1ev2aaatCvAUfeBSjuyZL2yd9gzLbvyNv2CaerbuLwBLn hiov2DGi1BTfMBaeXatLxBI9gBaerbd9wDYLwzYbItLDharqqtubsr 4rNCHbGeaGqiFu0Je9sqqrpepC0xbbL8F4rqqrpipeea0xe9Lq=Je9 vqaqFeFr0xbba9Fa0P0RWFb9fq0FXxbbf9=e0dfrpm0dXdirVu0=vr 0=vr0=fdbaqaaeGacaGaaiaabeqaamaabaabaaGcbaWaaabuaeaaca WGobWaaSbaaSqaaiaadEgaaeqaaaqaaiaadohacqGHiiIZtuuDJXwA K1uy0HwmaeHbfv3ySLgzG0uy0Hgip5wzaGqbaiab=jr8tnaaCaaame qabaGaamOuaaaaaSqab0GaeyyeIuoakiaadchadaqadaqaaiaadoha caaISaGaam4zaaGaayjkaiaawMcaaiabeg8aYnaabmaabaGaam4Cai aaiYcacaWGNbaacaGLOaGaayzkaaGaeyyzImRaamOuamaaBaaaleaa caWGNbaabeaakiaaiYcacaaMe8UaeyiaIiIaam4zaiabgIGiolab=z q8hjaai6cacaaMf8UaaGzbVlaaywW7caaMf8UaaGzbVlaacIcacaaI YaGaaiOlaiaaiAdacaGGPaaaaa@6C79@$

In addition to the objective function, the method effect between the proposed design and the benchmark design is also part of a constraint in the optimization problem: a constraint on comparability of population subgroups. The overall method effect as an objective function could lead to an unbalanced solution. For example, let a group $g$ be assigned a strategy $s$ such that the corresponding $D\left(s,g\right)$ is a large negative value and the other groups $h\in \mathcal{G}\\left\{g\right\}$ receive strategies that yield positive $D\left(s,h\right)$ values. The large negative $D\left(s,g\right)$ is canceled out but group $g$ will have a very different behavior compared to the other groups, and this complicates comparisons among groups. To prevent the occurrence of such designs, we limit the absolute difference in the method effect between two groups by the following constraint

$\underset{g,h\in \mathcal{G}}{\mathrm{max}}\left\{\frac{\sum _{s\in {\mathcal{S}}^{R}}p\left(s,g\right)\rho \left(s,g\right)D\left(s,g\right)}{\sum _{s\in {\mathcal{S}}^{R}}p\left(s,g\right)\rho \left(s,g\right)}-\frac{\sum _{s\in {\mathcal{S}}^{R}}p\left(s,h\right)\rho \left(s,h\right)D\left(s,h\right)}{\sum _{s\in {\mathcal{S}}^{R}}p\left(s,h\right)\rho \left(s,h\right)}\right\}\le M.\text{ }\text{ }\text{ }\text{ }\text{ }\left(2.7\right)$

However, when

$\frac{\sum _{s\in {\mathcal{S}}^{R}}p\left(s,g\right)\rho \left(s,g\right)D\left(s,g\right)}{\sum _{s\in {\mathcal{S}}^{R}}p\left(s,g\right)\rho \left(s,g\right)}-\frac{\sum _{s\in {\mathcal{S}}^{R}}p\left(s,h\right)\rho \left(s,h\right)D\left(s,h\right)}{\sum _{s\in {\mathcal{S}}^{R}}p\left(s,h\right)\rho \left(s,h\right)}\le M\text{ }\text{ }\text{ }\text{ }\text{ }\left(2.8\right)$

is included in the optimization problem for each pair $\left(g,h\right)\in \mathcal{G},$ then (2.7) is automatically satisfied. For practical reasons, i.e., a depletion of the sampling frame, we also introduce a constraint on the maximum sample size ${S}_{\text{max}},$ i.e.,

$\sum _{s,g}{N}_{g}p\left(s,g\right)\le {S}_{\text{max}}.\text{ }\text{ }\text{ }\text{ }\text{ }\left(2.9\right)$

Additionally, we require that at least one $p\left(s,g\right)$ be strictly positive,

$∑ s∈ S R p ( s,g ) > 0,∀g∈G, (2.10) MathType@MTEF@5@5@+= feaagKart1ev2aaatCvAUfeBSjuyZL2yd9gzLbvyNv2CaerbuLwBLn hiov2DGi1BTfMBaeXatLxBI9gBaerbd9wDYLwzYbItLDharqqtubsr 4rNCHbGeaGqiFu0Je9sqqrpepC0xbbL8F4rqqrpipeea0xe9Lq=Je9 vqaqFeFr0xbba9Fa0P0RWFb9fq0FXxbbf9=e0dfrpm0dXdirVu0=vr 0=vr0=fdbaqaaeGacaGaaiaabeqaamaabaabaaGcbaWaaabuaeaaca WGWbaaleaacaWGZbGaeyicI48efv3ySLgznfgDOfdaryqr1ngBPrgi nfgDObYtUvgaiuaacqWFse=udaahaaadbeqaaiaadkfaaaaaleqani abggHiLdGcdaqadaqaaiaadohacaaISaGaam4zaaGaayjkaiaawMca aiaaysW7caqG+aGaaGjbVlaaicdacaGGSaGaeyiaIiIaam4zaiabgI Giolab=zq8hjaaiYcacaaMf8UaaGzbVlaaywW7caaMf8UaaGzbVlaa cIcacaaIYaGaaiOlaiaaigdacaaIWaGaaiykaaaa@64AC@$

to avoid computational errors such as division by zero in (2.8).

Objective function (2.4) together with constraints (2.1), (2.5) $-$ (2.10) form the multi-mode optimization problem to minimize method effects against a benchmark through adaptive survey designs. This problem is a nonconvex nonlinear problem.

Date modified: