Is undesirable answer behaviour consistent across surveys? An investigation into respondent characteristics
Section 4. Results

• Previous
• Next

In this section, we first show the Cliff’s Delta’s for all surveys together as if they were one large survey. Second, we consider the Cliff’s Delta’s per survey to give an indication about UAB consistency across surveys to answer our research question. All Cliff’s Delta’s are obtained by comparing each category profile to the combined profile of the remaining categories. For instance, this means that the profile for respondents aged 15-24 are compared to the profile for the respondents from all other age categories. We chose for this type of comparison, as we are interested in whether a specific subgroup deviates from the complete sample of respondents, considered representative regarding age and education, minus that subgroup.

First, we need to note that respondents varied in the number of surveys they filled out. Some respondents filled out only one or two surveys, while others filled out all or almost all surveys. Behaviour data for every survey that the respondent filled out were used for the analyses. For instance, if a respondent filled out the surveys Health, Income, and Personality, this respondent is included in the data analyses for all these surveys. Second, respondents are classified in one category for both age and education. This means that a respondent can be older than 64 years and highly educated, and is included in the data analyses for both characteristics. Hence, respondents are included in each survey and characteristic analysis that is applicable to them. From this, it should be clear that we do not analyze individual respondents in this study, but that we focus on groups of respondents sharing the same characteristic. The reason is that we want to relate UAB to characteristics that are known from the literature to affect UAB, rather than to isolate individuals and explore potentially related characteristics.

We consider an individual respondent profile based on less than five items non-informative and too imprecise to take into account. Therefore, for each respondent group profile, we only include respondents who filled out at least five items. This means that part of the respondents may be excluded from several subgroups for the analyses. As a result, the occurrence of UAB for a subgroup after excluding respondents may differ from the initial occurrence of UAB for that subgroup. Thus, after excluding respondents from a subgroup, the remainder of the subgroup may not be representative for the original subgroup anymore in terms of the initial UAB occurrence. Therefore, we used two criteria to guarantee the representativeness of each original subgroup: 1) Each subgroup consists of more than 30% of the number of respondents in the original group, and; 2) the UAB occurrence in each subgroup does not differ more than 0.02 from the original group’s UAB occurrence.

4.1   Exploring survey participation and respondents aged 65 or older

Before elaborating on the main results, we give the outcomes of a few explorations. First, we investigated to what extent frequency of survey participation may have differed between the various age and educational subgroups. See Table 4.1. The average number of surveys that was filled out per respondent overall is 7.6. The average number of surveys per educational subgroup appeared to be relatively high and not to differ much between subgroups. For the age subgroups however, it is evident that younger respondents filled out a lower number and older respondents a higher number of surveys on average.

We used respondent profiles and Cliff’s Delta to explore whether the degree of participation made a difference in the occurrence of the specific UABs taking all surveys together. We split up the complete sample of panel respondents into a group who filled out at most eight surveys and a group who filled out at least nine surveys. See Table 4.2. It is clear that participation rate did not affect the occurrence of most UABs. Not surprisingly, respondents who participated in relatively few surveys showed relatively more “won’t tell”-answers. A second effect was relatively more straightlining in case of a lower participation rate.

Lastly, respondents aged 75 or older may be even more vulnerable to difficulty in cognitive processing and hence showing UAB than respondents aged 65-74. Therefore, we compared respondents aged 65-74 to respondents aged 75 or older on their group UAB proportion. See Table A.1 in Appendix A. Age subgroups did not or hardly differ for most UABs and surveys. Only regarding straightlining there were a few striking differences, but interestingly, these showed that respondents aged 75 or older expressed less straightlining than respondents aged 65-74. This means that we do not have a reason to split up the age subgroup of 65 years or older into two smaller subgroups.

4.2   Overall outcomes for Cliff’s Delta

The overall results for Cliff’s Delta concern the global picture for specific subgroups for all surveys taken together. We use the rules that $\left|\delta \right|<0.11$ indicates no effect, $0.11\le \left|\delta \right|<0.28$ a small effect, $0.28\le \left|\delta \right|<0.43$ a medium effect, and $\left|\delta \right|\ge 0.43$ a large effect, as investigated by Vargha and Delaney (2000), see also Goedhart (2016). A subgroup is always compared to the aggregated total of all remaining applicable subgroups regarding the specific characteristic. See Table 4.3 for the Cliff’s Delta’s for all surveys taken together.

From Table 4.3, it is clear that subgroups for age and education differ in various forms of specific satisficing behaviours overall. Younger and lower educated respondents showed more “don’t know”-answers than older and higher educated respondents. Higher educated respondents showed more acquiescent, but less neutral responses than lower educated respondents. Younger respondents showed less extreme responses than respondents from other age categories. Respondents from the middle age categories showed more primacy responses than both younger and older respondents (see Graph 1 in Figure 4.1), while higher educated respondents showed more primacy responses than lower educated respondents. Respondents from the middle age categories showed more straightlining than older respondents, while higher educated respondents showed more straightlining than lower educated respondents. From Table 4.3, it is also evident that some subgroups for age and education differ for sensitivity-based answer behaviour overall. Younger respondents showed more “won’t tell”-answers than older respondents. Higher educated respondents showed more socially desirable responses (see Graph 2 in Figure 4.1), but less “won’t tell”-answers than lower educated respondents. In summary, overall satisficing and sensitivity-based behaviours are clearly present, in most cases particularly for the youngest, oldest, lowest educated, or highest educated respondent groups.

A present overall effect size for a specific category and UAB does not by definition mean a present effect size for various surveys; an overall effect size may exist without effect sizes for any surveys. The opposite may be true as well; an overall effect size may be absent, as positive and negative effect sizes for various surveys cancel each other out. In the following section, we investigate to what extent either positive or negative effect sizes consistently exist across surveys and answer our main research question.

4.3   Consistency outcomes for Cliff’s Delta

These results for Cliff’s Delta concern the consistency of subgroups across surveys. To reveal consistency, we considered the number of surveys for which at least a small effect $\left(\left|\delta \right|\ge 0.11\right)$ was the result. Considering consistency conservatively, as an at least small effect for a specific UAB and category for all or almost all applicable surveys, we would draw the conclusion that there is no consistency to be found: There is no consistent satisficing or sensitivity-based behaviour evident across surveys. See Table 4.4 containing all results for the UABs and categories for which more than half of the applicable surveys showed either positive or negative effect sizes: There is no category that shows an effect for all or almost all surveys for any UAB.

Therefore, for each UAB and category, we considered the number of surveys for which at least a small either positive or negative effect was found. See Table 4.5. It is striking that relatively many cells or category-UAB pairs showed both positive and negative effects (marked by “2” in Table 4.5). This means that a category may show more of a specific UAB for some surveys, while less for other surveys. For instance, consider the category 15-24 years for the UAB answering “won’t tell” (WT) in Table 4.5. Here, this age category showed more “won’t tell”-answers than the other categories combined for one survey, while less “won’t tell”-answers for another survey. For a more liberal perspective on consistency, we elaborate on the cases for which more than half of the applicable surveys showed either positive or negative effect sizes (see Table 4.4). Strikingly, this is applicable to only seven out of the 96 possible cases (as we have results for eight UABs and twelve categories) and at a maximum of only 75% of the applicable surveys.

For the UAB answering “don’t know”, Table 4.4 shows that respondents 15-24 years of age gave more “don’t know”-answers and respondents of 65 years or older gave less “don’t know”-answers than other respondents for multiple surveys (see Graphs 1 through 4 in Figure 4.2). Respondents who finished only primary education gave more “don’t know”-answers than other respondents for various surveys. For primacy responding, we found that respondents 15-24 years of age or who finished only primary education chose less early response options than other respondents for multiple surveys. Respondents who finished the highest educational level chose more early response options and less neutral responses than other respondents for various surveys.

In summary, the results refer to an absence of UAB consistency across all or almost all surveys: Both satisficing and sensitivity-based UABs did not emerge consistently across surveys. We conclude that respondents’ UAB across surveys may be more influenced by the survey and its topic and items than solely by the age or educational level of the respondent. We close with a discussion in the following section.

• Previous
• Article main page
• Next