Considering interviewer and design effects when planning sample sizes
Section 2. Interviewer and design effects

• Previous
• Next

We define a sample as a set of $n$ distinct respondents, which we denote as $s=\left\{1,\dots ,n\right\},$ with $n\in \mathbb{N}.$ For the $k^{\text{th}}$ respondent our variable of interest $y$ is a real valued variable, where $y_k$ is the observation of this variable for the $k^{\text{th}}$ respondent in our sample $s.$ The observed data is given by $y={\left(y_1,\dots ,y_n\right)}^{\top }.$ We associate survey weights with every respondent in the sample, given by $w={\left(w_1,\dots ,w_n\right)}^{\top },$ where $w_k$ is the weight of the $k^{\text{th}}$ respondent and $w_k>0,$ for all $k\in s.$

We consider the weighted sample mean of $y$ as our estimator, given by

$$\overline{y}\left(w\right)=\frac{w^{\top }y}{w^{\top }{\mathbb{I}}_n}=\frac{{{\displaystyle \sum }}_{k\in s}{w}_k{y}_k}{{{\displaystyle \sum }}_{k\in s}{w}_k},(2.1)$$

where ${\mathbb{I}}_n$ is a column vector of ones of length $n.$ We focus on one estimator of interest, $\overline{y}\left(w\right),$ as it is the most common choice for describing interviewer and design effects (Kish, 1965, Section 8.1, Kish, 1962; Särndal, Swensson and Wretman, 1992, page 53). This choice enables us to use an established framework (Gabler et al., 1999) and produce formulas that are recognizable to readers that are already somewhat familiar with the topic. However, design effects of other estimators have been studied, notably, Lohr (2014), derives design effects for estimators of regression coefficients and Fischer, West, Elliott and Kreuter (2018), describe the impact of interviewer effects on the estimation of regression coefficients.

In the following, the variance of $\overline{y}\left(w\right)$ is derived under different measurement models for $y.$ The different models serve to distinguish between complex and simple sampling designs, as well as when there is and is not an interviewer effect. It should be noted that the model based variance of estimator $\overline{y}\left(w\right),$ which we use, is, in general, not the same as its design based variances, i.e., the variance of $\overline{y}\left(w\right)$ under a given sampling design (Särndal et al., 1992, page 492). Design based variances can be very complex and thus difficult to display in an accessible fashion, especially for multi-stage sampling. The model based approach reduces complexity while retaining the essential property of the complex sampling designs that we study, the cluster effect of multi-stage sampling. It also makes it possible to easily integrate cluster and interviewer effect into a common framework.

2.1  Simple random sampling without an interviewer effect

To model simple random sampling in the absence of an interviewer effect, i.e., without intra-PSU and intra-interviewer correlation, we assume the following measurement model $\left(M_0\right)$

$$y_k={\mu }_k+{\mathfrak{e}}_k,\left(M_0\right)$$

where ${\mu }_k$ is the value of $y$ for the $k^{\text{th}}$ respondent and ${\mathfrak{e}}_k$ is the measurement error. The measurement errors ${\mathfrak{e}}_k$ for all $k\in s$ are independent and identically distributed (iid) random variables with a variance-covariance structure of

$${\text{Cov}}_{M_0}\left(e_k,e_l\right)=\{\begin{array}{ll}{\sigma }^2,\hfill & \text{if }k=l\hfill \\ 0,\hfill & \text{else }\hfill \end{array},(2.2)$$

where $\sigma$ is a real value parameter greater than zero. Under model $\left(M_0\right),$ the variance of $\overline{y}\left({\mathbb{I}}_n\right)$ is given by $V_{M_0}\left(\overline{y}\left({\mathbb{I}}_n\right)\right)={\sigma }^2/n.$ This variance can be interpreted as the variance of the unweighted sample mean under simple random sampling with replacement (Särndal et al., 1992, page 73). Simple random sampling with estimator $\overline{y}\left({\mathbb{I}}_n\right)$ typically serves as a reference estimation strategy, which is compared with more complex sampling designs and estimators.

2.2  Simple random sampling with an interviewer effect

Next, we introduce interviewer variance into our measurement model for $y.$ Each respondent is interviewed by one and only one interviewer. There are $R\in {\mathbb{N}}_{>0},$ interviewers that conduct the interviews of all $n$ respondents. We denote $s_i\subset s$ as the set of all respondents that are interviewed by the $i^{\text{th}}$ interviewer and $\mathcal{R}=\left\{1,\dots ,R\right\}$ as the set of all interviewers. The workload of the $i^{\text{th}}$ interviewer is given by $n_i,$ $n_I={\left(n_1,\dots ,n_R\right)}^{\top }$ is the vector of interviewer workloads and ${\displaystyle {\sum }_{i=1}^R{n}_i=n}.$ Under measurement model $\left(M_1\right),$ which follows the explanations of Särndal et al. (1992), page 623, the observed values of $y$ for $k\in s_i$ are described as

$$y_{ik}={\mu }_k+{\Im }_i+{\mathfrak{e}}_{ik},\left(M_1\right)$$

with ${\Im }_i$ being the interviewer effect associated with all measurements conducted for respondents $k\in s_i.$ ${\mathfrak{e}}_{ik}$ represents the random error due to sources other than the interviewer. All ${\mathfrak{e}}_{ik}$ for $i\in \mathcal{R}$ and $k\in s$ are iid random variables with zero mean and variance ${\sigma }_e^2.$ ${\Im }_1,\dots ,{\Im }_R$ are iid random variables with zero mean and variance ${\sigma }_I^2,$ which we call interviewer variance, and they are independent of ${\mathfrak{e}}_{ik}$ for all $i\in \mathcal{R}$ and $k\in s.$ Särndal et al. (1992) interprets model $\left(M_1\right)$ as a random assignment of interviewers to a pre-defined partition of the sample $s$ into $R$ disjoint subsets $s_i,$ $i=1,\dots ,R.$ These subsets could correspond to different geographical areas where the survey is conducted and the interviewers are then randomly allocated to them. In practice, in many surveys fieldwork agencies assign interviewers to geographical areas based on experience and proximity. As this process is not necessarily observable by the researcher estimating the design effect, we assume a random allocation of interviewers to the PSUs. This can be seen as the recruitment of interviewers from an infinite, or very large, pool of possible interviewers.

If we define the random part in $y_{ik}$ as ${\epsilon }_{ik}={\Im }_i+{\mathfrak{e}}_{ik},$ then the variance-covariance structure of $y_{ik}$ under model $\left(M_1\right)$ is given by

$${\text{Cov}}_{M_1}\left({\epsilon }_{ik},{\epsilon }_{jl}\right)=\{\begin{array}{ll}{\sigma }^2,\hfill & \text{if }i=j,k=l\hfill \\ {\rho }_I{\sigma }^2,\hfill & \text{if }i=j,k\ne l\hfill \\ 0,\hfill & \text{else }\hfill \end{array},(2.3)$$

where ${\sigma }_I^2+{\sigma }_e^2={\sigma }^2$ and ${\rho }_I=\frac{{\sigma }_I^2}{{\sigma }^2}$ is the correlation between two different observations of $y$ made by the same interviewer. To derive the variance of $\overline{y}\left(w\right)$ under model $\left(M_1\right),$ we first determine the variance of ${\displaystyle {\sum }_{i\in \mathcal{R}}{\displaystyle {\sum }_{k\in s_i}{w}_{ik}{y}_{ik}}},$ where $w_{ik}$ and $y_{ik}$ are the survey weight and the observation for respondent $k\in s_i,$ respectively. Thus we have

$$\begin{array}{ll}{\text{Var}}_{M_1}\left({\displaystyle \sum _{i\in \mathcal{R}}{\displaystyle \sum _{k\in s_i}{w}_{ik}{y}_{ik}}}\right)\hfill & ={\sigma }^2\left({\displaystyle \sum _{i\in \mathcal{R}}{\displaystyle \sum _{k\in s_i}{w}_{ik}^2}}+{\rho }_I{\displaystyle \sum _{i\in \mathcal{R}}{\displaystyle \sum _{k\in s_i}{\displaystyle \sum _{\begin{array}{l}l\in s_i\\ l\ne k\end{array}}{w}_{ik}{w}_{il}}}}\right)\hfill \\ \hfill & ={\sigma }^2\left({\displaystyle \sum _{i\in \mathcal{R}}{\displaystyle \sum _{k\in s_i}{w}_{ik}^2}}+{\rho }_I\left[{\displaystyle \sum _{i\in \mathcal{R}}}{\left({\displaystyle \sum }_{k\in s_i}{w}_{ik}\right)}^2-{\displaystyle \sum _{i\in \mathcal{R}}}{\displaystyle \sum }_{k\in s_i}{w}_{ik}^2\right]\right)\hfill \\ \hfill & ={\sigma }^2{\displaystyle \sum _{i\in \mathcal{R}}{\displaystyle \sum _{k\in s_i}{w}_{ik}^2}}\left(1+{\rho }_I\left[\frac{{{\displaystyle \sum }}_{i\in \mathcal{R}}{\left({{\displaystyle \sum }}_{k\in s_i}{w}_{ik}\right)}^2}{{{\displaystyle \sum }}_{i\in \mathcal{R}}{{\displaystyle \sum }}_{k\in s_i}{w}_{ik}^2}-1\right]\right),\hfill \end{array}$$

from which follows

$${\text{Var}}_{M_1}\left(\overline{y}\left(w\right)\right)=\frac{{\sigma }^2{{\displaystyle \sum }}_{i\in \mathcal{R}}{{\displaystyle \sum }}_{k\in s_i}{w}_{ik}^2}{{\left({{\displaystyle \sum }}_{i\in \mathcal{R}}{{\displaystyle \sum }}_{k\in s_i}{w}_{ik}\right)}^2}\left(1+{\rho }_I\left[\frac{{{\displaystyle \sum }}_{i\in \mathcal{R}}{\left({{\displaystyle \sum }}_{k\in s_i}{w}_{ik}\right)}^2}{{{\displaystyle \sum }}_{i\in \mathcal{R}}{{\displaystyle \sum }}_{k\in s_i}{w}_{ik}^2}-1\right]\right).(2.4)$$

2.3  Multi-stage sampling with an interviewer effect

We consider a two-stage sampling design, where first PSUs are selected, and at the second stage respondents are selected from within the sampled PSUs. PSUs are the clustering units and we will treat the terms cluster and PSU as interchangeable. The sample of PSUs is denoted $\mathcal{K}=\left\{1,\dots ,K\right\},$ with $K>1.$ Each respondent belongs to one PSU and one PSU only. Let $s_q\subset s$ be the set of all respondents belonging to the $q^{\text{th}}$ PSU, $n_q$ be the number of respondents observed within the $q^{\text{th}}$ PSU, $n_C={\left(n_1,\dots ,n_K\right)}^{\top }$ the vector of cluster sizes, and ${\displaystyle {\sum }_{q\in \mathcal{K}}{n}_q=n}.$ Again, each respondent is interviewed by one interviewer and one interviewer only. Interviewers can work across PSUs and PSUs can be visited by multiple interviewers. Although interviewers might concentrate their work in a particular region, these regions are usually composed of multiple PSUs and interviewers do not work exclusively in one PSU only. This situation is frequently found in face-to-face surveys across Europe, e.g., in the ESS or EVS. Table 3.1 in Section 3.1 gives an overview on the level of interpenetration between PSUs and interviewer for countries that use a multi-stage sampling design in ESS6. Interpenetration between PSUs and interviewer can be observed across all ESS rounds for countries that use multi-stage sampling design.

We now introduce measurement model $\left(M_2\right),$ which incorporates both cluster and interviewer variance into the observed values of $y.$ For $k\in s_{qi}=s_q\cap s_i$ we model observations of $y$ as

$$y_{qik}={\mu }_k+{ℭ}_q+{\Im }_i+{\mathfrak{e}}_{qik},\left(M_2\right)$$

with ${ℭ}_q$ defined as a random variable with mean zero and variance ${\sigma }_C^2,$ which we call PSU variance, common to all respondents in PSU $q.$ ${ℭ}_1,\dots ,{ℭ}_K$ are iid random variables and are independent of ${\mathfrak{e}}_{qik}$ and ${\Im }_i$ for all $i\in \mathcal{R},$ $q\in \mathcal{K},$ and $k\in s_{qi}.$ ${ℭ}_q$ introduces a certain degree of similarity between respondents from the same PSU. It allows for a permanent random effect of the PSU on the measurement of $y,$ for the $k^{\text{th}}$ respondent, causing it to deviate from ${\mu }_k$ (Chambers and Skinner, 2003, page 201).

To establish the effect of sampling and interviewers on $\overline{y}\left(w\right),$ we define the random part of $y_{qik}$ as ${\epsilon }_{qik}={ℭ}_q+{\Im }_i+{\mathfrak{e}}_{qik},$ which has the following variance-covariance structure

$${\text{Cov}}_{M_2}\left({\epsilon }_{qik},{\epsilon }_{pjl}\right)=\{\begin{array}{ll}{\sigma }^2,\hfill & \text{if }q=p,i=j,k=l\hfill \\ {\rho }_C{\sigma }^2,\hfill & \text{if }q=p,i\ne j,k\ne l\hfill \\ {\rho }_I{\sigma }^2,\hfill & \text{if }q\ne p,i=j,k\ne l\hfill \\ \left({\rho }_I+{\rho }_C\right){\sigma }^2,\hfill & \text{if }q=p,i=j,k\ne l\hfill \\ 0,\hfill & \text{else }\hfill \end{array},(2.5)$$

where ${\sigma }_C^2+{\sigma }_I^2+{\sigma }_e^2={\sigma }^2$ and ${\rho }_C=\frac{{\sigma }_C^2}{{\sigma }^2}$ is the correlation between observation from the same PSU. The variance-covariance structure of ${\epsilon }_{qik}$ implies that the measurements of $y$ are correlated if they are made within the same PSU or the same interviewer. Further, measurements of $y$ are more homogeneous if they are made by the same interviewer within the same PSU. Model $\left(M_2\right)$ represents a generalization of model $M_4$ of Gabler and Lahiri (2009), by removing the restriction that no interviewer works in more than one PSU.

The variance of $\overline{y}\left(w\right)$ under model $\left(M_2\right)$ is given by

$$\begin{array}{ll}{\text{Var}}_{M_2}\left(\overline{y}\left(w\right)\right)\hfill & =\frac{{\sigma }^2{{\displaystyle \sum }}_{q\in \mathcal{K}}{{\displaystyle \sum }}_{i\in \mathcal{R}}{{\displaystyle \sum }}_{k\in s_{qi}}{w}_{qik}^2}{{\left({{\displaystyle \sum }}_{q\in \mathcal{K}}{{\displaystyle \sum }}_{i\in \mathcal{R}}{{\displaystyle \sum }}_{k\in s_{qi}}{w}_{qik}\right)}^2}\hfill \\ \hfill & \left(1+{\rho }_I\left[ {\overline{m}}_I\left(w\right)-1\right]+{\rho }_C\left[{\overline{m}}_C\left(w\right)-1\right]\right),(2.6)\hfill \end{array}$$

where $w_{qik}$ and $y_{qik}$ are the survey weight and the observation for respondent $k\in s_{qi},$ respectively, and

$${\overline{m}}_I\left(w\right)=\frac{{{\displaystyle \sum }}_{i\in \mathcal{R}}{\left({{\displaystyle \sum }}_{q\in \mathcal{K}}{{\displaystyle \sum }}_{k\in s_{qi}}{w}_{qik}\right)}^2}{{{\displaystyle \sum }}_{q\in \mathcal{K}}{{\displaystyle \sum }}_{i\in \mathcal{R}}{{\displaystyle \sum }}_{k\in s_{qi}}{w}_{qik}^2}\text{and}{\overline{m}}_C\left(w\right)=\frac{{{\displaystyle \sum }}_{q\in \mathcal{K}}{\left({{\displaystyle \sum }}_{i\in \mathcal{R}}{{\displaystyle \sum }}_{k\in s_{qi}}{w}_{qik}\right)}^2}{{{\displaystyle \sum }}_{q\in \mathcal{K}}{{\displaystyle \sum }}_{i\in \mathcal{R}}{{\displaystyle \sum }}_{k\in s_{qi}}{w}_{qik}^2}.$$

We can alter model $\left(M_2\right)$ to allow for a PSU interviewer interaction effect, meaning that the covariance between the observations made by the same interviewer within the same PSU is not equal to the sum of the intra-PSU and intra-interviewer covariance. We call this measurement model $\left(M_{2*}\right)$ and for $k\in s_{qi}$ the observation of $y$ is modeled as

$$y_{qik}={\mu }_k+{ℭ}_q+{\Im }_i+{\mathfrak{O}}_{qi}+{\mathfrak{e}}_{qik},\left(M_{2*}\right)$$

with ${\mathfrak{O}}_{qi}$ as a random variable with mean zero and variance ${\sigma }_{IC}^2$ common to all respondents in PSU $q$ that were interviewed by interviewer $i.$ All ${\mathfrak{O}}_{qi}$ for $q\in \mathcal{K}$ and $i\in \mathcal{R}$ are iid random variables and are independent of ${\mathfrak{e}}_{qik},$ ${\Im }_i,$ ${ℭ}_q$ for all $q\in \mathcal{K},$ $i\in \mathcal{R},$ and $k\in s_{qi}.$ Random effect ${\mathfrak{O}}_{qi}$ introduces some additional correlation between observations made by the same interviewer within the same PSU, which cannot be explained by the separate PSU and interviewer variances.

For $k\ne l$ and ${\epsilon }_{qik}={ℭ}_q+{\Im }_i+{\mathfrak{O}}_{qi}+{\mathfrak{e}}_{qik}$ we have under model $\left(M_{2*}\right)$ ${\text{Cov}}_{M_{2*}}\left({\epsilon }_{qik},{\epsilon }_{qil}\right)=\left({\rho }_I+{\rho }_C+{\rho }_{IC}\right){\sigma }^2.$ Thus, we can write the variance of $\overline{y}\left(w\right)$ under model $\left(M_{2*}\right)$ as

$$\begin{array}{ll}{\text{Var}}_{M_2}\left(\overline{y}\left(w\right)\right)\hfill & =\frac{{\sigma }^2{{\displaystyle \sum }}_{q\in \mathcal{K}}{{\displaystyle \sum }}_{i\in \mathcal{R}}{{\displaystyle \sum }}_{k\in s_{qi}}{w}_{qik}^2}{{\left({{\displaystyle \sum }}_{q\in \mathcal{K}}{{\displaystyle \sum }}_{i\in \mathcal{R}}{{\displaystyle \sum }}_{k\in s_{qi}}{w}_{qik}\right)}^2}\hfill \\ \hfill & \left(1+{\rho }_I\left[{\overline{m}}_I\left(w\right)-1\right]+{\rho }_C\left[{\overline{m}}_C\left(w\right)-1\right]+{\rho }_{IC}\left[{\overline{m}}_{IC}\left(w\right)-1\right]\right),(2.7)\hfill \end{array}$$

where

$${\overline{m}}_{IC}\left(w\right)=\frac{{{\displaystyle \sum }}_{q\in \mathcal{K}}{{\displaystyle \sum }}_{i\in \mathcal{R}}{\left({{\displaystyle \sum }}_{k\in s_{qi}}{w}_{qik}\right)}^2}{{{\displaystyle \sum }}_{q\in \mathcal{K}}{{\displaystyle \sum }}_{i\in \mathcal{R}}{{\displaystyle \sum }}_{k\in s_{qi}}{w}_{qik}^2}.$$

2.4  Survey effect

After we establish the variance of $\overline{y}\left(w\right)$ under the different measurement models, we can define the effect associated with complex sampling and interviewers. We will refer to this effect as the survey effect, which we define as

$${\text{eff}}_{ab}\left(w\right)=\frac{{\text{Var}}_{M_a}\left(\overline{y}\left(w\right)\right)}{{\text{Var}}_{M_b}\left(\overline{y}\left(w\right)\right)},(2.8)$$

where $M_a$ is the measurement model assumed for our survey of interest and $M_b$ is the reference model. We use the term survey effect to distinguish ${\text{eff}}_{ab}\left(w\right)$ from design and interviewer effect, as ${\text{eff}}_{ab}\left(w\right)$ incorporates both effects. Other sources of variance, as described in the TSE framework, are not considered. Consequently, we will use the term survey design for the combination of a sampling design and interviewer workplan.

The survey effect associated with measurement model $\left(M_2\right),$ is given by

$$\begin{array}{ll}{\text{eff}}_{20}\left(w\right)\hfill & =\frac{{\text{Var}}_{M_2}\left({\overline{y}}_w\right)}{{\text{Var}}_{M_0}\left(\overline{y}\right)}\hfill \\ \hfill & ={\text{eff}}_w\left(w\right)\left(1+{\rho }_I\left[{\overline{m}}_I\left(w\right)-1\right]+{\rho }_C\left[ {\overline{m}}_C\left(w\right)-1\right]\right),(2.9)\hfill \end{array}$$

where

$${\text{eff}}_w\left(w\right)=\frac{n{{\displaystyle \sum }}_{k\in s}{w}_k^2}{{\left({{\displaystyle \sum }}_{k\in s}{w}_k\right)}^2}\ge 1.$$

Factor ${\text{eff}}_w\left(w\right)$ does not depend on the measurement model and can be interpreted as a measure for the variance of the weights $w.$ If we write the variance of the weights as ${\sigma }_w^2=1/n{\displaystyle {\sum }_{k\in s}{w}_k^2-{\overline{w}}^2},$ with $\overline{w}=1/n{\displaystyle {\sum }_{k\in s}{w}_k},$ this relationship becomes more clear, as ${\text{eff}}_w\left(w\right)={\text{CV}}_w^2+1,$ with ${\text{CV}}_w={\sigma }_w/\overline{w}$ as the coefficient of variation of the survey weights. If the weights are all equal, then ${\text{CV}}_w=0$ and ${\text{eff}}_w\left(w\right)$ becomes 1. Terms ${\overline{m}}_I\left(w\right)$ and ${\overline{m}}_C\left(w\right)$ can be seen as measures for the average workload of the interviewers and the PSU size, respectively. If all weights are equal, ${\overline{m}}_I\left(w\right)$ has the value ${\overline{m}}_I\left({\mathbb{I}}_n\right)={\displaystyle {\sum }_{i\in \mathcal{R}}{n}_i^2}/n.$ Furthermore, if all interviewers have the exact same workload, i.e., $n_i=n/R$ for $i=1,\dots ,R,$ we have ${\overline{m}}_I\left({\mathbb{I}}_n\right)=n/R.$ ${\overline{m}}_C\left(w\right)$ has similar properties.

Following Gabler et al. (1999) and Gabler and Lahiri (2009) we can give the following upper bound for the survey effect.

Result 1.

$${\text{eff}}_{20}^{*}\left(w\right)\le {\text{eff}}_w\left(w\right){\text{eff}}_{20}^{*}\left({\mathbb{I}}_n\right),$$

where ${\text{eff}}_{20}^{*}$ is the survey effect under the condition that $n_i=n/R$ for all $i\in \mathcal{R}$ and $n_q=n/K$ for all $q\in \mathcal{K}.$ The upper bound of ${\text{eff}}_{20}^{*}\left(w\right),$ given in Result 1, follows from ${\overline{m}}_I\left(w\right)\le n/R$ if $n_i=n/R$ for all $i\in \mathcal{R}$ (Gabler et al., 1999). The proof is given in the Appendix. For ${\overline{m}}_C\left(w\right)$ an analogous result holds. It should be noted that, in general, we do not have

$${\text{eff}}_{20}\left(w\right)\ge {\text{eff}}_{20}\left({\mathbb{I}}_n\right)=1+{\rho }_I\left[{\displaystyle \sum }_{i\in \mathcal{R}}\frac{n_i^2}{n}-1\right]+{\rho }_C\left[{\displaystyle \sum }_{q\in \mathcal{K}}\frac{n_q^2}{n}-1\right].(2.10)$$

That is, we cannot say that the survey effect is greater or equal to the survey effect of an equally weighted design. If the weights have the same relative frequency distribution across all sets $s_{qi}$ inequality (2.10) holds (Gabler and Lahiri, 2009), i.e., if we have

$$n_{qig}=\frac{n_{qi}}{n}{n}_g,g=1,\dots ,G,(2.11)$$

where $G$ is the number of unique values in $w,$ $n_g$ the frequency of the $g^{\text{th}}$ weighting value, and $n_{qig}$ the frequency of the $g^{\text{th}}$ weighting value for respondents interviewed by the $i^{\text{th}}$ interviewer in the $q^{\text{th}}$ PSU.

We can, however, give a lower bound to ${\text{eff}}_{20}\left(w\right).$ Using the same argument that Gabler and Lahiri (2009) give in the proof of their Result 6, we get

$${\text{eff}}_{20}\left(w\right)\ge \left(1+{\rho }_I\left[\frac{n}{R}-1\right]+{\rho }_C\left[\frac{n}{K}-1\right]\right).(2.12)$$

With the right-hand side of inequality (2.12) an easy to calculate minimum of ${\text{eff}}_{20}\left(w\right)$ is given, which does not depend on the weights, the distribution of interviewer workloads, or the PSU sizes. This gives some valuable guidance at the planning stage of a survey design, as the planned survey effect of the survey should be at least as high as ${\text{eff}}_{20}^{*}\left({\mathbb{I}}_n\right).$ The practical utility of the upper bound in Result 1 is somewhat limited by strong assumptions about $n_I$ and $n_C.$ The further the values of $n_I$ and $n_C$ deviate from the one point distribution of interviewer workloads and PSU sizes, the less this bound should serve as a guide. To give survey planners a less complex statistic to plan the value of ${\overline{m}}_I\left(w\right),$ Lynn and Gabler (2004) proposed using

$${\overline{m}}_I^{\prime }\left(w\right)= \frac{H_{n_I}}{H_w},(2.13)$$

as a predictor for ${\overline{m}}_I\left(w\right),$ where $H_{n_I}={\displaystyle {\sum }_{i\in \mathcal{R}}{\left(n_i/n\right)}^2}$ is the Herfindahl index for the interviewer workload, a concentration measure, with $1/R\le H_{n_I}\le 1$ (Fahrmeir, Heumann, Künstler, Pigeot and Tutz, 1997, page 83). $H_{n_I}=1$ corresponds to $R=1$ and $H_{n_I}=1/R$ corresponds to $n_i=n/R$ for all $i\in \mathcal{R}.$ $H_w={\displaystyle {\sum }_{k\in s}{\left(w_k/{\displaystyle {\sum }_{k\in s}{w}_k}\right)}^2}$ is the Herfindahl index for the weights. If equation (2.11) holds, we have ${\overline{m}}_I\left(w\right)={\overline{m}}_I^{\prime }\left(w\right),$ but for most surveys this will not apply. For that reason, Lynn and Gabler (2004) suggested looking at $\text{Cov}\left(w_{qik},n_i\right),$ the covariance between the weights and interviewer workloads. The closer $\text{Cov}\left(w_{qik},n_i\right)$ is to zero the smaller the distance between ${\overline{m}}_I\left(w\right)$ and ${\overline{m}}_I^{\prime }\left(w\right).$ Planning a survey with assumed values for $H_{n_I}$ and $H_w$ should be easier than with exact values of $n_I$ and $w.$ Finding reasonable values for $H_{n_I}$ and $H_w$ could be guided by comparing these values from surveys with similar survey designs. Under equation (2.11) the findings are analogous for ${\overline{m}}_C\left(w\right).$

It should be noted that we can also write ${\text{eff}}_w\left(w\right)$ as

$${\text{eff}}_w\left(w\right)=H_{w}n.(2.14)$$

The expression of ${\text{eff}}_w\left(w\right)$ in equation (2.14) might also be useful at the planning stage of a survey, showing that it is possible to plan with a certain weight concentration, instead of specific values for $w.$

Giving a general close upper bound for ${\text{eff}}_{20}\left(w\right)$ is difficult if there are no restrictions on the values of $n_I,$ $n_C$ and $w.$ However, survey weights are usually scaled to either the sample or the population size and it is not uncommon for them to be bounded. For example, the ESS provides weights to its users that are greater than zero and smaller or equal to 4 and scales them to the sample size (ESS, 2014c, 2014b). If $a\le w_k\le b$ for all $k\in s$ with $b<\infty$ and $a>0,$ then with a given value for $n_I$ (or $n_C)$ upper limits of ${\overline{m}}_I\left(w\right),$ (or ${\overline{m}}_C\left(w\right))$ can be found, by solving a linear optimization problem. An upper limit for ${\text{eff}}_w\left(w\right)$ can be deduced for given values of $a$ and $b,$ as shown in equation (A.5) in the Appendix.

The obtained upper bound of ${\text{eff}}_{20}\left(w\right)$ will correspond to weight distributions with a very high concentration, i.e., a maximal number of the highest possible weights. However, adjusting the constraints of the linear optimization problem, based on the weight distribution of surveys with comparable sampling designs, can help to find bounds that are of higher practical relevance. (See Appendix for the formulation of this linear program.)

2.5  Corrected design effect

Now that we have established the survey effect of a survey design, we propose a new type of survey effect that we call corrected design effect. This statistic aims at quantifying the marginal effect of a complex survey design if an interviewer effect is present. We do this by defining the following effect

$$\begin{array}{ll}{\text{eff}}_{21}\left(w\right)\hfill & =\frac{{\text{Var}}_{M_2}\left({\overline{y}}_w\right)}{{\text{Var}}_{M_1}\left(\overline{y}\right)}\hfill \\ \hfill & ={\text{eff}}_w\left(w\right){\text{eff}}_I\left(1+{\rho }_I\left[{\overline{m}}_I\left(w\right)-1\right]+{\rho }_C\left[{\overline{m}}_C\left(w\right)-1\right]\right),(2.15)\hfill \end{array}$$

where

$${\text{eff}}_I=\frac{n}{n+{\rho }_I\left({{\displaystyle \sum }}_{i\in \mathcal{R}}{n}_i^2-n\right)}.$$

The reference model $\left(M_1\right)$ in ${\text{eff}}_{21}\left(w\right)$ models a simple random sample with an interviewer effect. Factor ${\text{eff}}_I,$ indicates how close the corrected design effect to the survey effect is. For ${\text{eff}}_I=1$ the corrected design and survey effect are equal and the closer ${\text{eff}}_I$ is to zero the further apart are both effects. Hence, we can use ${\text{eff}}_I$ to construct a measure for the contribution of the interviewer effect to the survey effect ${\text{eff}}_{20}.$ For this, we first establish the following bounds for ${\text{eff}}_I$ given in Result 2.

Result 2.

$$\begin{array}{c}\frac{1}{n}\le \frac{n}{{\rho }_I\left(n-R\right)\left(n-R+1\right)+n}\le {\text{eff}}_I\le \frac{R}{R+\left(n-R\right){\rho }_I}\le 1.\end{array}$$

The proof for Result 2 can be found in the Appendix.

Now we define a measure of the contribution of the interviewer effect to the survey effect ${\text{inv}}_I$ as

$$\begin{array}{ll}{\text{inv}}_I:\hfill & \left[\frac{1}{n},1\right]\mapsto \left[0,1\right],\hfill \\ {\text{inv}}_I\left(a\right):=\hfill & \frac{n\left(1-a\right)}{n-1}\text{for}\left[\frac{1}{n}\le a\le 1\right].\hfill \end{array}(2.16)$$

For any given value of interviewer workloads $n_I,$ measure ${\text{inv}}_I\left({\text{eff}}_I\right)$ is strictly increasing with decreasing ${\text{eff}}_I.$ The maximum of ${\text{inv}}_I\left({\text{eff}}_I\right)$ occurs at $R=1$ and ${\rho }_I=1,$ which occurs when there is only one interviewer that always produces the same measurement. The minimum of ${\text{inv}}_I\left({\text{eff}}_I\right)$ occurs at ${\rho }_I=0$ for any given value of $n_I.$ If the concentration of the distribution of the workload over the interviewers increases and ${\rho }_I$ stays fixed, ${\text{inv}}_I\left({\text{eff}}_I\right)$ also increases. This relation becomes clearer if we write

$${\text{eff}}_I=\frac{1}{1+{\rho }_I\left(H_{n_I}n-1\right)}.(2.17)$$

Alternatively, the coefficient of variation for the interviewer workloads ${\text{CV}}_{n_I}=R{\sigma }_{n_I}/n,$ with ${\sigma }_{n_I}^2=1/R{\displaystyle {\sum }_{i\in \mathcal{R}}{n}_i^2-{\left(n/R\right)}^2},$ could also be used to describe ${\text{eff}}_I,$ since $H_{n_I}=\left(1+{\text{CV}}_{n_I}^2\right)/R$ (Lynn and Gabler, 2004). Note that for ${\sigma }_{n_I}^2=0$ we have ${\text{eff}}_I=R/\left(R+\left(n-R\right){\rho }_I\right).$

Using Results 1 and 2, as well as inequality (2.12), we can give the following bounds for the corrected design effect.

Result 3.

$$\begin{array}{c}\frac{n}{{\rho }_I\left(n-R\right)\left(n-R+1\right)+n}{\text{eff}}_{20}^{*}\left({\mathbb{I}}_n\right)\le {\text{eff}}_{21}^{*}\left(w\right)\le {\text{eff}}_w\left(w\right)\frac{R}{R+\left(n-R\right){\rho }_I}{\text{eff}}_{20}^{*}\left({\mathbb{I}}_n\right),\end{array}$$

where ${\text{eff}}_{21}^{*}$ is the corrected design effect when there are equal interviewer workloads and equal PSU sizes. The bounds of ${\text{eff}}_{21}^{*},$ given in Result 3 , do not depend on $n_I,$ but it should be noted that in the lower bound of ${\text{eff}}_{21}^{*}\left(w\right),$ ${\text{eff}}_I$ takes on its value for the maximum concentration in $n_I,$ whereas ${\text{eff}}_{20}^{*}\left({\mathbb{I}}_n\right)$ corresponds to the minimal concentration of $n_I.$ Since ${\text{eff}}_I$ does not depend on $w,$ an upper (or lower) bound for ${\text{eff}}_{21}\left(w\right)$ can be found by obtaining the upper (or lower) bounds of ${\overline{m}}_I\left(w\right),$ ${\overline{m}}_C\left(w\right)$ and ${\text{eff}}_w\left(w\right)$ as described in the Appendix.

Finally, we introduce a corrected design effect that assumes the measurement model $\left(M_{2*}\right),$ given by

$$\begin{array}{ll}{\text{eff}}_{2*1}\left(w\right)\hfill & =\frac{{\text{Var}}_{M_{2*}}\left({\overline{y}}_w\right)}{{\text{Var}}_{M_1}\left(\overline{y}\right)}\hfill \\ \hfill & ={\text{eff}}_w\left(w\right){\text{eff}}_I\left(1+{\rho }_I\left[{\overline{m}}_I\left(w\right)-1\right]+{\rho }_C\left[{\overline{m}}_C\left(w\right)-1\right]+{\rho }_{IC}\left[{\overline{m}}_{IC}\left(w\right)-1\right]\right).(2.18)\hfill \end{array}$$

Similarly to Result 3 we can establish the following bounds for ${\text{eff}}_{2*1}\left(w\right).$

Result 4.

$$\begin{array}{c}\frac{n}{{\rho }_I\left(n-R\right)\left(n-R+1\right)+n}{\text{eff}}_{2*0}^{*}\left({\mathbb{I}}_n\right)\le {\text{eff}}_{2*1}^{*}\left(w\right)\le {\text{eff}}_w\left(w\right)\frac{R}{R+\left(n-R\right){\rho }_I}{\text{eff}}_{2*0}^{*}\left({\mathbb{I}}_n\right).\end{array}$$

Here ${\text{eff}}_{2*1}^{*}$ corresponds to the case where $n_{qi},$ the number of respondents that belong to the $q^{\text{th}}$ PSU and are interviewed by the $i^{\text{th}}$ interviewer, is a constant, i.e., $n_{qi}=n/\left(RK\right)$ for all $i\in \mathcal{R}$ and $q\in \mathcal{K}.$ This also implies that for ${\text{eff}}_{2*1}$ we have $n_i=n/R$ and $n_q=n/K.$ The proof of Result 4 can be found in the Appendix. Using model $\left(M_{2*}\right)$ instead of $\left(M_2\right)$ gives some additional flexibility in fitting the measurement model to the observed data. Whether this is required is a part of Section 3.2, where the different measurement models are tested against each other for ESS6 data.

• Previous
• Article main page
• Next