The anchoring method: Estimation of interviewer effects in the absence of interpenetrated sample assignment
Section 3. The anchoring method

• Previous
• Next

As noted in Section 2.3, existing methods adjust for possible interviewer effects introduced at the recruitment and measurement stages of data collection by including respondent- and area- or interviewer-level covariates in multilevel models (Hox, 1994), but such adjustment may be erroneous if part of the interviewer variance is simply arising due to non-interpenetrated sampling. As noted by Elliott and West (2015), if subjects with similar values on a variable of interest are assigned to interviewers in a non-random fashion $–$ for example, if a telephone interviewer working day shifts tends to interview older respondents, where age may be correlated with main variables of interest $–$ these variables will be correlated with specific interviewers. However, we are just re-ordering the random sample, not introducing measurement error in the manner described in Section 1, e.g., West and Blom (2017). Thus the actual data are not being altered, and there are no true interviewer effects: we term the resulting within-interviewer correlation “spurious” from a variance inflation perspective. Thus estimating interviewer effects while failing to account for differential sample assignment can lead to conservative inferences, resulting in misleadingly large estimates of interviewer variance, $p$ -values and confidence intervals that are too wide, and incorrect operational decisions based on predicted effects for individual interviewers.

To address this important gap in the literature, we describe an “anchoring” method that analysts can use to estimate the unique components of variance due to interviewer effects on selection and measurement. The method aims to leverage correlations between variables where interviewer measurement error is of concern and variables known ‒ or reasonably believed ‒ to be free of measurement error to remove the fraction of the within-interviewer correlation that is due to non-interpenetrated sample assignment. In the simplest case, if we have two variables, one $\left(Y_1\right)$ treated as measurement error-free (the “anchor”) and one $\left(Y_2\right)$ treated as possibly having interviewer-induced measurement error, and our objective is to estimate a mean of $Y_2,$ we fit a multilevel model to the observed data for the two variables that includes a random interviewer effect only for the variable subject to measurement error:

$$y_{ijk}={\mu }_k+I\left(k=2\right)b_i+{\epsilon }_{ijk}.(3.1)$$

In (3.1), $i=1,\dots ,I$ indexes interviewers, $j=1,\dots ,J_i$ indexes respondents within interviewers, $k=1,2$ indexes the variable (1 = anchor, 2 = variable of interest), $b_i~N\left(0,{\sigma }_b^2\right)$ is the interviewer effect, and

$$\left(\begin{array}{c}{\epsilon }_{ij1}\\ {\epsilon }_{ij2}\end{array}\right)~N\left(\left(\begin{array}{c}0\\ 0\end{array}\right),\left[\begin{array}{cc}{\sigma }_1^2& {\sigma }_{12}\\ {\sigma }_{12}& {\sigma }_2^2\end{array}\right]\right).$$

Our focus of inference in this manuscript is ${\mu }_2,$ although ${\sigma }_b^2$ or $b_i$ may also be of interest if the focus is on interviewer variance or determining individual interviewers who are contributing to that variance.

To provide a heuristic explanation of why this proposed “anchoring” approach works, assume that $y_{ij1}$ and $y_{ij2}$ net of $b_i$ are almost perfectly correlated. Since $y_{ij1}$ lacks measurement error, it can serve as a proxy for the non-measurement error component of $y_{ij2},$ absorbing artificial error in $y_{ij2}$ induced in the ordering of the data. Lack of interpenetration means that estimating a linear mixed model using $y_{ij2}$ only will yield an upwardly biased estimate of ${\sigma }_b^2.$ If ${\sigma }_{12}>0,$ information will be available to reduce the bias in ${\widehat{\sigma }}_b^2,$ with large samples and high correlations between ${\epsilon }_{ij1}$ and ${\epsilon }_{ij2}$ yielding increasingly accurate estimates of ${\sigma }_b^2$ and thus of the true impact of the interviewer-induced measurement error on the variance of ${\widehat{\mu }}_2.$

This approach easily extends to the setting where $K-1\ge 2$ “anchoring” variables free from measurement error are available:

$$y_{ijk}={\mu }_k+I\left(k=K\right)b_{iK0}+{\epsilon }_{ijk},i=1,\dots ,I,j=1,\dots ,J_i,k=1,\dots ,K.(3.2)$$

In this case, the first $K-1$ variables are assumed to be free of interviewer measurement error and the $K^{\text{th}}$ variable is the variable of interest, $b_{iK0}~N\left(0,{\tau }^2\right),$ and ${\left({\epsilon }_{ij1}\dots {\epsilon }_{ijK}\right)}^T~N_K\left(0,\Sigma \right),$ where $\Sigma$ is an unstructured $K\times K$ covariance matrix. Alternatively, instead of using (3.2) directly, we can reduce (3.2) back to the bivariate setting in (3.1) by replacing $Y_{1i}$ with the best linear predictor of $Y_{Ki}$ using the anchoring variables: ${\widehat{Y}}_{Ki}=E\left(Y_{Ki}|Y_{1i},\dots ,Y_{K-1,i}\right)={\widehat{\beta }}^T{X}_i$ where ${\widehat{X}}_i={\left(Y_{1i},\dots ,Y_{K-1,i}\right)}^T$ and $\widehat{\beta }={\left(X^{T}X\right)}^{-1}{X}^T{Y}_K.$

3.1   Estimation remarks

One can use standard linear mixed model software (e.g., SAS PROC MIXED) to fit the models in (3.1) or (3.2) and obtain a restricted maximum likelihood (REML) point estimate ${\widehat{\mu }}_2$ together with an associated variance estimate. We have provided an annotated example of such code in the supplemental materials. Weights used to account for unequal probabilities of selection, non-response adjustment, and calibration to known population values can be incorporated using pseudo-maximum likelihood estimation (PML; Pfeffermann, Skinner, Holmes, Goldstein and Rasbash, 1998; Rabe-Hesketh and Skrondal, 2006) when fitting the models in (3.1) or (3.2). We would generally recommend that interviewers be assigned a weight of 1 when fitting weighted multilevel models of these forms, to mimic the notion of simple random sampling of interviewers from a hypothetical population of interviewers. The weights for respondents should be rescaled to sum to the final respondent count for each interviewer (Carle, 2009), and extensions of the PML method outlined by Veiga, Smith and Brown (2014) and Heeringa, West and Berglund (2017, Chapter 11) can be used to incorporate the rescaled weights in estimation of the residual covariance structure in (3.1) or (3.2). In multistage samples where interviewers cross geographic areas, cross-classified random effects models (Rasbash and Goldstein, 1994) can also be utilized.

3.2   The Bayesian anchoring method

In the presence of prior information on the parameters of interest in this model (e.g., in a repeated cross-sectional survey using interviewer administration), the models in (3.1) or (3.2) can also be fitted using a Bayesian approach to incorporate the prior information. In repeated surveys that carefully monitor interviewer performance, good predictions of individual interviewer effects based on the estimated variance component are important. Given historical data from a survey with the same essential design conditions, one can estimate the parameters of interest in (3.1) using this historical data, and then define informative prior distributions for these parameters. (Examples of these types of surveys would include high-quality government sponsored surveys with repeated cross-sectional data collections, such as the National Health Interview Survey or, for the example considered in this paper, the Behavioral Risk Factor Surveillance System.) Specifically, we consider a prior distribution on the interview effect standard deviation ${\sigma }_b$ that follows a half $t$ distribution (Gelman, 2006) with degrees of freedom $\nu$ and standard deviation $s:$

$$p\left({\sigma }_b|\nu ,s\right)=\frac{2\Gamma \left(\frac{\nu +1}{2}\right)}{\Gamma \left(\frac{v}{2}\right)\sqrt{\nu \pi s^2}}{\left(1+\frac{{\sigma }_b^2}{\nu s^2}\right)}^{-\frac{\nu +1}{2}},\tau \ge 0.(3.3)$$

Following Gelman, we assume $\nu =3,$ and we estimate $s$ based on prior estimates of interviewer effects. We consider standard weak priors for the fixed effect means: $p\left({\mu }_k\right)\stackrel{\text{ind}}{{\displaystyle ~}}N\left(0,{10}^6\right)$ and for the residual variance:

$$p\left(\begin{array}{cc}{\sigma }_1^2& {\sigma }_{12}\\ {\sigma }_{12}& {\sigma }_2^2\end{array}\right)~\text{INV}-\text{WISHART}\left(2,I\right).$$

This approach offers advantages relative to likelihood ratio testing approaches that rely on asymptotic theory, particularly for smaller samples. By using prior information to constrain the resulting posterior distribution for the interviewer variance components, it generally prevents extremely large draws of the variance component while not constraining the means or residual variances. It also constrains posterior draws of variance components to be greater than zero, enabling inference based on small components of variance, while frequentist model-fitting procedures generally fix such estimates of variance components to be exactly zero (which equates to a rather unreasonable assumption that each interviewer produces exactly the same survey estimate; West and Elliott, 2014). In such cases, the effects of interviewers (even if they are small) would be ignored completely; the Bayesian approach would still enable small effects to be integrated into the inference. The Bayesian approach also yields credible intervals for the interviewer variance components based on posterior draws.

3.3   Choosing anchoring variables

A key assumption underlying both the standard regression-based approach and the “anchoring” method is that selected variables are free from interviewer-induced error. Like the “missing at random” assumption in the missing data literature, we do not expect that there will often be cases where we can be certain of this, but that approximations may be available based on simple demographic measures (e.g., age) or other factual questions with simple response options (e.g., current employment) and little room for the introduction of interviewer error. The identification of error-free covariates in advance of data collection is an important substantive and methodological component of this approach, and prior methodological literature on the variables most prone to interviewer effects (West and Blom, 2017) can be consulted for this component of the approach.

As we note above, if we have multiple error-free covariates measured on the respondents, we can preserve their predictive power (and thus the correlation of the anchor’s residuals with the residuals of the variable of interest) by computing a linear predictor of the variable of interest from a linear model that includes fixed effects of all of the error-free covariates. We consider such an approach in our simulation studies and applications, and compare it with the “standard” approach of simply adjusting for these covariates in a multilevel model in an effort to improve the estimate of the interviewer variance component (Hox, 1994).

Finally, the anchoring approach employs mixed-effects models that should yield correct estimates with a sufficient amount of data. However, these models may be more difficult to fit, especially for smaller samples, and we therefore also consider alternative Bayesian approaches when evaluating the anchoring approach.

• Previous
• Article main page
• Next