Considering interviewer and design effects when planning sample sizes
Section 3. Empirical findings from the ESS

• Previous
• Next

After we established the effects associated with interviewers and multi-stage or cluster sampling, we now estimate the survey effect and our proposed corrected design effect for ESS6 data (ESS, 2016).

There were 29 participating countries in ESS6 (ESS, 2018a), but not all have been considered in our analysis. We excluded all countries with a single-stage design (there were no single-stage cluster sampling designs in ESS6). In addition, we excluded those countries that had a multi-domain sampling design. These countries employed different sampling designs in different regions of the country, but they all refer to a certain level of the Nomenclature of Territorial Units for Statistics (NUTS), as established by Eurostat (ESS, 2013, pages 21-22). For example, Norway used a single stage sample for its more densely populated regions, which, combined, contained almost 75 percent of the target population, and a two-stage sampling design for the rest of the country.

First, in Section 3.1 we assess whether the estimation of the measurement models described in Section 2 is generally feasible, given the PSU-interviewer structure found in ESS6. To this end, we use a model-based simulation study. In Section 3.2, we test the different measurement models against each other in order to use the most appropriate ones for the estimation of the survey effect and the corrected design effect. Afterwards, we compare our results with the design effect that was used by the ESS to plan the sample size.

The PSU and interviewer identification variables needed for our simulation study and the estimation of the effects were obtained from the so-called Sampling Design Data Files (SDDFs) and the Interviewer Questionnaire, respectively (ESS, 2014a). The SDDFs contain information on the sampling design, including a PSU identifier. For ESS6, the SDDFs have to be downloaded individually for each country (ESS, 2018b).

3.1 Simulation for the stability assessment of effect estimates

Interviewers and sampling have long been recognized as principal sources of survey error. The way interviewers are deployed during fieldwork makes it difficult to separate the interviewer variance from the PSU variance. To make data collection more efficient, interviewers are usually assigned to work exclusively in certain regions (Von Sanden, 2004, Section 1.3). Correspondingly, interviewers in ESS6 seldom work across regions. For ESS6, we observe the following situation: In general, interviewers work in a number of PSUs within a certain area, but never in all PSUs. PSUs might be visited by more than one interviewer, but never by all of them. For 25% of all considered countries, the mean number of regions (variable region, ESS (2013), pages 21-22) an interviewer visited was 1.017 or lower. For 75% of all countries, the mean number of regions per interviewer was 1.256 or lower.

The non-hierarchical structure of PSUs and interviewers can be considered typical of large scale social surveys like the ESS. A so-called fully interpenetrated survey design, where all interviewers work in all PSUs, is in general unfeasible for country-wide surveys. This makes it difficult to decide what amount of observed similarity between observations made by an interviewer is due to intra-interviewer correlation or instead due to intra-PSU correlation. This problem has been addressed in a number of studies. For instance, by using a fully nested survey design, where multiple interviewers work in the same PSU but not across them (Schnell and Kreuter, 2005). But also so-called partially interpenetrated surveys, where different interviewers work in multiple PSUs and PSUs are visited by multiple interviewers, have been analyzed, (Davis and Scott, 1995; O’Muircheartaigh and Campanelli, 1998). These partially interpenetrated surveys resemble more the situation we observe for ESS6.

To test our measurement models and to disentangle the different variance components, we fit a multilevel model with crossed random effects. In another context, Raudenbush (1993) proposes to allow for so-called crossed effects in the random effects structure. These crossed effects allow for the situation of partially interpenetrated factors, and are able to estimate all three variance components of measurement model $\left(M_{2*}\right),$ ${\sigma }_I^2,$ ${\sigma }_C^2,$ and ${\sigma }_{IC}^2.$

Vassallo, Durrant and Smith (2017) show, using simulations on synthetic data, how well a multilevel model with crossed random effects for cluster and interviewer can estimate the variance-covariance structure of the data model under different patterns of interpenetration between cluster and interviewer. They identify the sample size, the number of interviewers and PSUs, and the level of interpenetration as the driving factor for the quality of the estimates of the variance components. The level of interpenetration plays a decisive role for the quality of the variance component estimates. Vassallo et al. (2017) found that already 2-3 interviewers per PSU lead to relatively stable estimates of the variance components. However, their survey designs were all balanced and symmetric, meaning that the interpenetration of PSUs by interviewers was constant for all PSUs and vice versa. This is not the case for countries in ESS6. Therefore, we perform a simulation to test whether under the partial interpenetrated survey designs of ESS6 the variance components of our measurement model $\left(M_2\right)$ can be estimated or not.

For the simulation, we generate samples from a $n$ -dimensional multi-variate normal distribution $\text{MVN}\left(\mu ,\Sigma \right).$ The vector of means $\mu$ contains, for each dimension, the same value. The covariance matrix $\Sigma$ follows the variance-covariance structure of measurement model $\left(M_2\right)$ and was constructed for each country based on the observed PSU-interviewer structure. The variance components were set to ${\sigma }_I^2=$ 0.2, ${\sigma }_C^2=$ 0.08, ${\sigma }^2=$ 2. We generated 1,000 samples from the superpopulation model $\text{MVN}\left(\mu ,\Sigma \right)$ for each country and estimated measurement model $\left(M_2\right)$ for each of these samples. The simulation was implemented in R (R Core Team, 2019). The samples for the simulation were generated with the help of the mvtnorm package (Genz, Bretz, Miwa, Mi and Hothorn, 2019) and the estimation of the model was done using the lme4 package (Bates, Mächler, Bolker and Walker, 2015, 2019).

Table 3.1 depicts the relative Monte Carlo bias of the estimators for the variance components of model $\left(M_2\right).$ For an estimator $\widehat{\theta }$ of $\theta$ we define this measure as

$$\text{MC-RBias}\widehat{\theta }=\frac{{\overline{\widehat{\theta }}}_{\text{MC}}}{\theta }-1,$$

where ${\overline{\widehat{\theta }}}_{\text{MC}}={\displaystyle {\sum }_{d=1}^D{\widehat{\theta }}_d/D},$ $\theta$ is the true value, ${\widehat{\theta }}_d$ the value of $\widehat{\theta }$ for the $d^{\text{th}}$ sample of the simulation and $D$ is the total number of samples generated, i.e., $D=$ 1,000, in our simulation. We see that ${\sigma }_I^2$ and ${\sigma }_C^2$ are estimated with a relative low bias for all considered countries in ESS6. In addition to the relative Monte Carlo bias, we have added the number of PSUs $K,$ the number of interviewers $R,$ the sample size $n,$ the average number of PSUs that an interviewer works in ${\overline{K}}_I,$ and the average number of interviewer that work in a PSU ${\overline{R}}_C$ to Table 3.1. ${\overline{K}}_I$ and ${\overline{R}}_C$ are used as measures for the level of interpenetration of PSUs by interviewers and interviewers by PSUs, respectively. For all countries, other than Germany, there are more PSUs than interviewers and ${\overline{K}}_I$ is greater than ${\overline{R}}_C.$ ${\overline{K}}_I$ reaches from 1.423 in Germany to 17.396 in Albania. The level of ${\overline{K}}_I$ observed for all countries seems to be high enough to disentangle the variance components of model $\left(M_2\right).$ We can observe a negative relationship between ${\overline{K}}_I$ and MC-RBias ${\widehat{\sigma }}_I^2,$ which can be mediated by $n$ and $K.$ Higher $n$ and $K$ correspond to a higher accuracy of ${\widehat{\sigma }}_I^2.$ An analogous observation can be made for MC-RBias ${\widehat{\sigma }}_C^2.$ A higher ${\overline{R}}_C$ also improves the precision of the estimates and can compensate for a low ${\overline{K}}_I.$ A high enough one-sided interpenetration, either of the PSUs by the interviewers or vice versa, is sufficient to accurately estimate ${\sigma }_I^2$ and ${\sigma }_C^2$ for model $\left(M_2\right).$ For example, the Czech Republic, which has the lowest ${\overline{R}}_C,$ but a ${\overline{K}}_I$ of round 1.848, enables relative precise estimates for the variance components.

It should be noted that for measurement model $\left(M_{2*}\right),$ both ${\overline{K}}_I$ and ${\overline{R}}_C$ are of importance. For example, ${\sigma }_C^2$ and ${\sigma }_{IC}^2$ cannot be estimated with precision if ${\overline{R}}_C$ is too low. For example, in a similar simulation for model $\left(M_{2*}\right),$ it was not possible to obtain accurate estimates of ${\sigma }_C^2$ and ${\sigma }_{IC}^2$ for the Czech Republic, although the relative bias of ${\widehat{\sigma }}_I^2$ was around 1 percent.

For Bulgaria and Czech Republic ${\overline{R}}_C=1,$ that is, their PSUs are nested within the interviewers. In this case, we do not have crossed random effects, but nested random effects, as we never have the case where respondents are within the same PSU but not interviewed by the same interviewer. For this special case, strictly speaking, ${\sigma }_C^2$ should be labeled ${\sigma }_{IC}^2.$ But, for simplicity, for both cases we use ${\sigma }_C^2$ as a label for the variances of the PSU random effect. This is not entirely unjustified, as ${\sigma }_{IC}^2$ defines the additional correlation between respondents that are in the same PSU, compared to those respondents that are interviewed by the same interviewer, but are in different PSUs.

Our simulation study confirms and extends the findings of Vassallo et al. (2017) for the unbalanced situation of the ESS6. We also saw that the PSU-interviewer structure observed for ESS6 does not prohibit the disentanglement of ${\sigma }_C^2$ and ${\sigma }_I^2$ for measurement model $\left(M_2\right).$

3.2 Survey effects in ESS round 6

As seen in our simulation study, the estimation of the interviewer and cluster variance is feasible in ESS6. Now we test, for a set of selected variables from the ESS main questionnaire (ESS, 2013), each variance component of model $\left(M_{2*}\right)$ on its significance. All used variables, except age and gender, have an ordinal scale, but are treated as metric variables for the purpose of this analysis. A list of all used variables can be found in the Appendix.

As a variance component has its minimum at zero, the test is performed on the boundary of the parameter space, which imposes classical problems from test theory. Scheipl, Greven, and Kuechenhoff (2008) proposed a restricted likelihood ratio test, designed to test for a zero random effects variance. We use their implementation of this test in the R-Package RLRsim and perform three test decisions.

First, we test on the significance of the interaction variance of interviewers and PSUs, when assuming relevant interviewer and PSU variances. Our null hypothesis is $H_0\text{:}{\sigma }_{IC}^2=0$ versus alternative hypothesis $H_A\text{:}{\sigma }_{IC}^2>0.$ The per country average of rejected null hypothesis over the different variables is displayed in Table 3.2. The first two columns correspond to two different type I error levels for the test of $H_0\text{:}{\sigma }_{IC}^2=0,$ indicated by $\alpha =$ 0.01 and 0.05. Israel is the country that has the highest number for significant interaction variance ${\sigma }_{IC}^2$ on all type I error levels. For all other countries the null hypothesis is not rejected for all variables at a significance level of 1%. Although not displayed in Table 3.2 it can be noted that at a 10% significance level two-thirds of the countries have at least some variables with a significant interaction variance. Therefore, the possibility of an interaction effect should be considered when estimating survey effects.

In our second test decision an interviewer variance but no interaction variance is assumed. The null hypothesis is that the PSU variance is not relevant, that is $H_0\text{:}{\sigma }_C^2=0$ versus the alternative hypothesis $H_A\text{:}{\sigma }_C^2>0.$ Average test results for the different type I error levels can be found in the columns 3 to 4 of Table 3.2. For some variables, the PSU variances are not significant as an addition to the interviewer variance. This result is especially strong for Belgium, where only 3% of the variables seem to have a PSU variance. However, also for France and Slovenia, the PSU variance is only significant at a level of 1% for a relative small number of the variables and for Albania for none of the variables. In contrast to that, Bulgaria, Ireland, Israel and Slovakia have significant PSU variance for the majority of variables. Overall, the PSU variance appears to be relevant in most countries and thus should be considered when estimating survey effects.

For the third test decision we perform, a PSU variance but no interaction variance is assumed. The null hypothesis is that the interviewer effect is not relevant $H_0\text{:}{\sigma }_I^2=0$ versus the alternative hypothesis $H_A\text{:}{\sigma }_I^2>0.$ Average test results can be found in columns 5 to 6 of Table 3.2. The lowest rejection rates are found in Germany and France, although 19% of the variables for Germany and 23% for France still have a significant interviewer variance at a 1% significance level. The other countries show a far higher proportion of variables with significant interviewer variance. On the 1% and 5% significance level, the interviewer variance has a higher rejection rate than the PSU variance for 13 out of the 15 countries. Thus, the interviewer variance appears to be of relevance for all countries in ESS6, indicating that possible interviewer effects should be taken into account when assessing the efficiency of survey designs.

Based on the selected models for the different variables, survey effects defined in equation (2.8) are estimated. Table 3.3 shows the country specific average of estimated survey effects over all considered variables. In addition Table 3.3 also contains the average of design effect deff, as it is used by the ESS to plan sample sizes. In our notation this design effect has the form

$$\text{deff}={\text{eff}}_w\left(1+{\rho }_C\left({\overline{m}}_C\left(w\right)-1\right)\right).$$

To estimate ${\rho }_C$ in deff we used an ANOVA estimator (The ESS Sampling Expert Panel, 2016; Ganninger, 2010, page 45) and do not test for the significance of the PSU variance. Measurement model $a$ used in ${\text{eff}}_{a0}$ can include interviewer, PSU and interaction variance, if the model selection identifies it as significant at a level of 0.05. The same applies to measurement model $a$ used in ${\text{eff}}_{a1},$ i.e., the corrected design effect. If interviewer variance is identified as not significant for a variable, then ${\text{eff}}_{a1}$ becomes ${\text{eff}}_{a0}.$ To measure the influence of the interviewer on the survey effect ${\text{inv}}_I$ is also shown.

By comparing deff and ${\text{eff}}_{a1}$ in Table 3.3 an interesting observation can be made: For Germany deff is clearly lower than for Ireland and the Czech Republic. From this we could deduce that Germany would need a much lower sample size to achieve the same average effective sample size as Ireland and the Czech Republic. However, if we look at ${\text{eff}}_{a1},$ this relation switches. Table 3.3 shows that the cluster effect of the complex sampling design is higher in Germany than it is in Ireland or the Czech Republic. Meaning that, if we are interested in equal average effective sample across countries, Germany would need a higher sample size than in Ireland or the Czech Republic. For example, for the Czech Republic to achieve an effective sample size of 1,500 with the standard design effect deff from Table 3.3 we would plan with a net sample of round 3,925 and for Germany with one of 3,115. If instead we use the corrected design effect ${\text{eff}}_{a1},$ to base the planning of the net sample size solely on the effect of the sampling design, we would select a net sample size of round 1,707 and 2,598, for the Czech Republic and Germany, respectively. This finding is also reflected in the values of ${\text{inv}}_I,$ which indicates that a large part of ${\text{eff}}_{a0}$ for Ireland and the Czech Republic can be attributed to an interviewer effect, whereas for Germany, the interviewer effect is smaller and ${\text{eff}}_{a0}$ seems to be dominated by the cluster effect. Apart from Israel, Slovakia, and Slovenia, all countries have different ranks for deff and ${\text{eff}}_{a1},$ indicating that the allocation of the sample size over all countries would be very different, if the corrected design was used to plan effective samples sizes, instead of the conventional design effect deff.

${\text{eff}}_{a1}$ is smaller than deff for all countries, and their distance, $\left|\text{deff}-{\text{eff}}_{a1}\right|,$ has a positive but non-linear relationship with ${\text{inv}}_I.$ The lowest values of $\left|\text{deff}-{\text{eff}}_{a1}\right|$ are observed for Spain, Belgium, France, and Germany, which are all countries whose ${\text{inv}}_I$ value is below the median of ${\text{inv}}_I.$ The opposite is observed for Slovakia, Kosovo, Ireland, and Ukraine, the countries with the highest distance between deff and ${\text{eff}}_{a1}.$ These countries all have a value of ${\text{inv}}_I$ that is higher than the median value of ${\text{inv}}_I.$ These patterns for countries with a relatively high distance between deff and ${\text{eff}}_{a1}$ are consistent with what we would expect if there is a high interviewer effect present in the data. The opposite can be said for countries when a relatively small distance between deff and ${\text{eff}}_{a1}$ is observed.

Interviewer effects depend on many different factors (West and Blom, 2017), including the type of the question asked and the used ESS6 data is mostly gathered from attitude questions. Hence, the presented results in this section cannot be extrapolated to other types of surveys in the same countries.

• Previous
• Article main page
• Next