Is undesirable answer behaviour consistent across surveys? An investigation into respondent characteristics
Section 1. Introduction
The relation between answer behaviour in surveys and measurement error has been studied extensively. Measurement error refers to the extent to which a response deviates from the true value that a survey question was intended to measure (De Leeuw, Hox and Dillman, 2008). The occurrence and size of measurement error and hence response data quality can be influenced by respondent characteristics (Olson and Smyth, 2015; Tourangeau, Rips and Rasinski, 2000). Respondent characteristics can be thought of as fixed tendencies of a respondent that may lead to undesirable answer behaviour (UAB), like satisficing (Holbrook, Green and Krosnick, 2003; Kaminska, McCutcheon and Billiet, 2010). When respondents satisfice, they take short-cuts in the question-answering process. Satisficing can be seen as the outcome of the interaction of question difficulty, motivation, and cognitive ability (Krosnick, 1991, 1999; Krosnick, Narayan and Smith, 1996). Cognitive ability may be considered a characteristic of the respondent that is relatively constant over time. A straightforward proxy for cognitive ability like age or educational level may be used as a background variable to investigate its relation to answer behaviour. Background variables may not be free of measurement errors themselves, but these errors are assumed not to relate to answer behaviour and to be relatively stable over time (Schouten and Calinescu, 2013).
Answer behaviour should be stable and typical for the respondent in order to investigate its relation to respondent characteristics. That is, the behaviour for a specific respondent must be shown consistently in order to be typical for that respondent. Here, the term “consistent” refers to a pattern of answer behaviour that is shown over several moments in time, across multiple surveys. When a respondent only incidentally shows a specific answer behaviour, it is not to say whether this is typical for that specific respondent. For instance, a respondent could fill out a single battery or set of five multiple choice items by choosing the very first answering option for each item. It is however not clear to what extent this may be a form of satisficing (Krosnick, 1991, 1999; Krosnick et al., 1996), as the answers may just as well be truly applicable to that respondent. In case of consistent answer behaviour, we may connect the behaviour to other stable characteristics of the same respondent. In this paper, we investigate the relation between cognitive ability and consistent undesirable answer behaviour. For this purpose, we use the respondent background variables age and educational level as proxies for cognitive ability. From here, we use the abbreviation “UAB” for the term “undesirable answer behaviour” throughout the paper.
Investigating the relation between cognitive ability and UAB is not new. However, this relation has not previously been investigated for a large sample of panel respondents across many surveys. To empower finding potential consistency for types of respondents in showing specific UAB, we use data from ten large population surveys administered by CentERdata in the LISS Panel. These surveys vary broadly in topic and contain many different kinds of items. By including many different surveys, variation will be present in survey topic and design. As a result of this variation, we assume that each survey has its own specific effect on the UABs. In our study, we want to distinguish respondent UAB that is survey-specific from UAB that occurs consistently across surveys. In order for respondent consistency to appear, UAB needs to occur across topics and survey designs. In other words, we need the full presence of topic and design variability to investigate UAB consistency across various surveys. We consider this topic and design variability as given and do not take into account survey and item characteristics for this study.
This study aims at linking cognitive ability to measurement error by using our method of constructing behaviour profiles. In case cognitive ability appears to have a consistent relation to specific UABs, surveys can be adapted according to the age or educational level of respondents in order to minimize measurement error. In case of such structural associations, the adaptation can be done globally, regardless of the survey. This also implies that our method could be used to predict measurement error. This means that time-consuming and expensive tests that examine the risk of measurement error could initially be omitted. If our method shows an increased risk of measurement error for specific respondents, setting up such tests could be valuable. If our method does not find such an increased risk, we could conclude that survey-independent adaptive survey design based on cognitive ability may not be useful.
For the purpose of our study, the specific survey topic or design would not even have to be taken into account. We realize that examining item characteristics and other respondent characteristics on their relation to measurement error across surveys is relevant as well. However, we consider our study a first step into investigating characteristics of respondents and items in their potentially consistent relation to UAB and measurement error across surveys. For this first step, we chose to examine the obvious respondent characteristics age and educational level in relation to eight relevant UABs (see Section 2).
Note that the undesirability of answer behaviour is potential by definition as we cannot validate its truthfulness (see Bais, Schouten and Toepoel, 2020 for an elaboration). Considering the aforementioned example, filling out the first answering option for all five items of a battery may refer to satisficing or to truthful responses. In the case of satisficing, we could say that this answer behaviour is undesirable. In the case of truthful responses, the behaviour is not undesirable. Our idea is that answer behaviour may refer to being undesirable as it is consistently shown across more surveys. The more consistent the behaviour, the more likely it becomes that the respondent is showing a personal pattern or style, and the more undesirable the behaviour may be considered. Therefore, the term “undesirable” is inherently potential when used throughout this paper. In summary, our foundation of ten large different surveys to detect potential behaviour consistency and to indicate the extent to which behaviour may be undesirable is solid and powerful.
This paper reads as follows: In Section 2 of this paper, we briefly elaborate on the theoretical framework on which our main research question is based. In Section 3, we describe the data, methods, and statistics that were used to compare the different age and educational categories for each UAB across surveys. As a method to detection of consistent UAB, we use so-called “respondent profiles”, as suggested and explored by Bais (2021). In Section 4, we show all statistical results and give answers to our main research question. In Section 5, we conclude with a discussion of these results and make suggestions on how to proceed.
- Date modified: