1. Introduction
Natalja Menold
Biases in survey statistics are described by the total survey error models (Groves, Fowler, Couper, Lepkowski, Singer and Tourangeau 2004; Smith 2007). Total survey error results from two types of errors, which are referred to as observation errors and non-observation errors. This article focuses on cross-cultural comparability influenced by factors related to non-observation errors, that is to say the failure of survey statistics to adequately represent the target population. These types of errors—also called representation errors—result from differences between the obtained net sample (number of survey participants, Haeder and Lynn 2007) and the target population.
Previous research in cross-cultural contexts has revealed highly pronounced differences in response rates between countries (Billiet, Phillipsen, Fitzgerald and Stoop 2007; Couper and de Leeuw 2003; de Heer 1999; de Heer and Israis 1992; de Leeuw and de Heer 2002; Hox and de Leeuw 2002; Johnson, O’Rourke, Burris and Owens 2002; Stoop 2005; Symons, Matsuo, Beullens and Billiet 2008), differences in field procedures (Billiet et al. 2007; Kohler 2007; Kreuter and Kohler 2009; Smith 2007; Stoop 2005; Symons et al. 2008) and differences in sampling methods (Lynn, Haeder, Gabler and Laaksonen 2007). The latter refer to procedures for constructing sampling frames and selecting sample elements. All differences listed may impede cross-cultural comparability. In particular, cross-cultural comparability might be limited due to varying sampling methods to obtain a probability sample since standardising sampling methods is restricted by local availability of sampling frames, by their quality and usability, and by the survey budget (Lynn et al. 2007).
Lynn et al. (2007) addressed design effects and the sample sizes necessary to achieve comparability of net samples in the ESS. In doing so, they focused on sample selection prior to the field work stage. However, comparability of samples could also be influenced by interviewers during their work in the field. Interviewers’ degree of freedom in substituting sampled individuals with persons who are not sampled (individuals who are easy to contact and are cooperative) differs based on sampling methods (Hoffmeyer-Zlotnik 2006; Kohler 2007; Sodeur 2007). “Field substitution occurs when a non-responding unit is replaced by a substitute (reserve) unit during the field work stage of the survey process” (Vehovar 1999, page 335). The substitutions addressed by Vehovar are legitimate substitutions that are allowed by protocol. In contrast, this article addresses illegitimate substitutions (referred to below simply as “substitutions”) which occur without permission. According to the AAPOR (2003), deliberate substitutions made by interviewers represent a kind of falsification.
The aim of this article is to show whether the effect of interviewers, which is assumed to be associated with substitutions, varies across different sampling methods that are used to obtain probability samples in a cross-cultural context. In addition, it will be shown whether payment, control procedures, the data collector (institution that carries out data collection in the field) or time factors are associated with this interviewer effect. The results may help surveyors when deciding upon sampling methods—a highly relevant cost factor in surveys—and when deciding upon methods to foster interviewer motivation to not substitute. The results are also important for research on interviewer behaviour since they hint at errors associated with interviewer influence on cross-cultural comparability.
The next section Section 2 provides the theoretical and empirical background of the study presented in this article. The hypotheses are described in Section 3. Section 4 provides information on the procedure and the method used for analysing the influence of the interviewer. The results are presented in Section 5. Finally, Section 6 discusses the results and provides conclusions.
- Date modified: