Is undesirable answer behaviour consistent across surveys? An investigation into respondent characteristics
Section 3. Method

• Previous
• Next

3.1   LISS panel and surveys

We selected ten Dutch general population surveys that were administered by CentERdata to respondents of the Longitudinal Internet studies for the Social Sciences (LISS) Panel. This was done in the time period between June 2012 and December 2013. The surveys were the first wave of the Dutch Labour Force Survey from Statistics Netherlands and nine of the core studies from CentERdata. The data for the background variables as presented in Section 2 were also provided by CentERdata. All surveys were administered in computer-assisted format. The ten surveys cover a broad range of topics in the field of general population statistics, see Table 3.1. Also note the relatively high response rates for all surveys, ensuring comparable samples across the surveys. Considering these high and comparable response rates, we do not expect them to have a substantial relation to the occurrence of UAB within the context of this study.

The LISS Panel consists of about 7,000 individuals from about 4,500 households and is based on a probability sample of households. This sample is drawn from the population registry by Statistics Netherlands. All panel members were invited for all surveys included in this study. The first administration period for each survey was approximately a month. In case of initial nonresponse, the respondent was sent one or two reminders within this period. To increase the response rate, a second administration period of about a month including one or two reminders was executed for each survey. The respondents were compensated for each survey that they completed. This whole procedure was standardized for each survey, ensuring the comparability of the response rates for the surveys. The number of respondents that filled out a specific survey differed per survey and the number of surveys that respondents filled out varied across respondents. The average number of surveys filled out by a respondent was almost eight. Altogether, the surveys contain 2,074 items that were used to cover the UABs as presented in Section 2.

3.2   Coding the undesirable answer behaviours

Each item (the total of the question and all answering options together) of all surveys was investigated on whether it was eligible for the selected UABs separately. The answering categories of the eligible items were coded for each UAB. In case a category was filled out for which the UAB occurred, the response was coded as 1; in case a category was filled out for which the UAB did not occur, the response was coded as 0. For all UABs, the coding was relatively straightforward. For neutral responding and answering “don’t know” and “won’t tell”, the neutral, don’t know- and won’t tell-options respectively were coded as 1, while all other options were coded as 0. For extreme responding, the most negative and most positive option were coded as 1, while all other options were coded as 0. For primacy responding, the first two options were coded as 1, while all other options were coded as 0. This coding method was based on Medway and Tourangeau (2015) for the UABs that matched our research. See Table 3.2 for an overview of the UABs and their eligible kind of items. See Table 3.3 for the proportions of items for which the UABs are applicable per survey and in total. From here, we discuss the coding process of the UABs that need more elaboration: Socially desirable responding, acquiescence, and straightlining.

Socially desirable responding

About 50% of all items of the involved surveys together were coded as potentially asking for sensitive information by at least one of three coders (see Bais, Schouten, Lugtig, Toepoel, Arends-Tóth, Douhou, Kieruj, Morren and Vis, 2019). Next, the answering categories of these items were coded by an independent fourth coder on whether they may refer to a socially desirable answer. Let us consider the following example:

“Can you indicate, on a scale from 0 to 10, how hard or how easy it is for you to live off your income?

0 means that it is very hard to live off your income, 10 means that it is very easy.

The idea is that it is socially desirable to state that it is relatively easy to live off one’s income. For our study, we only considered the answering options 8 through 10 as socially desirable options. In this way, we hoped to distinguish respondents who are clearly sensitive to responding in a socially desirable manner across surveys from those who are not.

Acquiescence: Responding agreeably/affirmatively to a question

The answering categories of all items were evaluated on whether they showed an extent of agreeableness or affirmativeness (see Medway and Tourangeau, 2015). Both positively and negatively worded items were present throughout the surveys to measure acquiescence. Both battery (a set of related items sharing the same answering options) and non-battery items were considered and also subjective variants of the typical answering option “agree”, like “satisfied”, “applicable”, and “yes”, were considered for acquiescence. We chose to include those variants as acquiescent options to capture a broad range of possible acquiescent behaviour across many items. Such a broad range may result into more variation between respondents in showing acquiescence, so that we may better distinguish acquiescent from non-acquiescent respondents. Let us consider the following example:

“I really enjoy responding to questionnaires through the mail or Internet.

For our study, we considered the answering options 5 through 7 as acquiescent options. We decided to consider the option “somewhat agree” (option 5 in the example) as an acquiescent response as well, as we hoped to distinguish respondents who acquiesce clearly or to only a certain extent from respondents who do not acquiesce.

We need to note that the coding of socially desirable responding and acquiescence is more or less arbitrary; the coding of both UABs may have been executed either more or less strictly. On the one hand, this means that a response option that was coded as socially desirable or acquiescent may be a socially desirable or acquiescent response for some respondents, but the intended response for others. On the other hand, a response option that was not coded as socially desirable or acquiescent may indeed be the intended response for some respondents, but should have been coded as socially desirable or acquiescent for others. However, in order to investigate socially desirable responding and acquiescence at all, a coding threshold that distinguishes the occurrence from the non-occurrence of these UABs simply needs to be placed at some point. By the current way of coding these UABs, enough variability between respondents is present in order to distinguish age and educational subgroups that may differ in the occurrence of UAB.

Straightlining: Choosing the same answering category for all items in a battery

Our idea is to consider straightlining for a battery only when the very same answering options were filled out for all its items (see Schonlau and Toepoel, 2015). When this is the case, the number of times that a “1” is coded is equal to the number of items that the battery consists of. For instance, the occurrence of straightlining for a battery of five items received the code “1” five times. This means that we took into account the length of the battery for this UAB. In other words, the more items a battery consists of, the stronger the UAB refers to straightlining in case a respondent filled out the same option for each item. See the following section for an elaboration on how the coding at the item level for all UABs is transformed into meaningful respondent behaviour summaries.

3.3   Respondent profiles

In order to compare respondents on consistent UAB across surveys, a few aspects need to be taken into account regarding the UAB. First, the number of items that is applicable to the UAB per survey can be relatively small. This means that uncertainty exists around the actual occurrence of UAB, since it is based on, by definition, a limited number of items per respondent. To give an example, suppose a respondent A fills out ten items and gives a “don’t know”-answer five times, while another respondent B fills out 100 items and gives a “don’t know”-answer 50 times. Although both respondents can be attributed a probability of 0.50 for answering “don’t know”, this probability is relatively more certain for respondent B since it is based on more response data. In other words, the actual occurrence of UAB for respondents may be more uncertain as respondents fill out a smaller number of items.

Second, when a survey contains filter questions that may or may not branch out into follow-up questions, each respondent is likely to fill out a different number of items for that survey. Therefore, the actual occurrence of UAB is indicated with varying uncertainty across different respondents within a survey. Hence, to compare respondents sharing the same characteristic on their UAB across surveys, simply using individual UAB proportions is insufficient: A method must be used that takes into account these uncertainties. For this purpose, we introduce the method of using respondent profiles. See Bais (2021) for an extensive statistical elaboration on this method.

The respondent profile

The respondent profile is a summary of UAB for a group of respondents. It represents the relative proportions of a specified population group (for instance lower educated respondents) in showing a specified UAB (for instance answering “don’t know”) at all possible probabilities from 0 to 1. In constructing a respondent profile, we make use of the binomial distribution to take into account the abovementioned uncertainties. Note that when we speak of a “respondent profile”, we refer to a group of respondents by definition. When we discuss a profile for a single respondent, we explicitly speak of an “individual respondent profile”.

Consider an individual respondent $r$ who fills out a survey consisting of 50 items of which each offers the answering option “don’t know”. Suppose that the respondent chooses the “don’t know”-option 10 times out of the 50 possible occasions. Then these numbers are used to construct a binomial distribution. This binomial distribution shows the occurrence of answering “don’t know” for respondent $r.$ The likelihood of the UAB occurrence is calculated for each probability along the probability range from 0 to 1. For practical calculation, we chose for a probability step size interval of 0.01 in order to construct the binomial distribution on the basis of 100 probabilities. We call the resulting binomial distribution for respondent $r$ an individual respondent profile. An individual respondent profile is the likelihood curve for the UAB occurrence and is calculated for each probability from 0 to 1. Hence, to construct the individual profile for respondent $r,$ the likelihood of the UAB occurrence is calculated on the basis of 10 actual “don’t know”-answers out of 50 possible occasions for all 100 probabilities:

$${\lambda }_{}\left(p\right)=\left(\begin{array}{c}{I}_r\\ G_r\end{array}\right)p^{G_r}{\left(1-p\right)}^{I_r-G_r},(3.1)$$

where ${\lambda }_{}$ is the likelihood curve or individual profile for respondent $r,$ $p$ is the probability between 0 and 1 with step size 0.01, $I_r$ is the number of items for which choosing the UAB is possible for respondent $r,$ and $G_r$ is the number of items for which the behaviour is actually shown by respondent $r$ . In order to make individual respondent profiles comparable, we normalize the resulting distribution to obtain an area below the curve of 1 regardless of step size. This is done by dividing each of the likelihoods that the profile consists of by the sum of all likelihoods:

$${\tilde{\lambda }}_r\left(p\right)=\frac{{\lambda }_r\left(p\right)}{{\displaystyle {\int }_{p=0}^1{\lambda }_r\left(p\right)dp}},(3.2)$$

where ${\tilde{\lambda }}_r$ is the normalized individual profile for respondent $r.$ For a single respondent $r,$ the average or expected value $E_r$ for the UAB occurrence can be estimated on the basis of the respondent’s profile and the integral over $p.$ This means that each probability from 0 to 1 is multiplied by its accompanying likelihood:

$$E_r={\displaystyle \underset{p=0}{\overset{1}{\int }}p{\tilde{\lambda }}_r\left(p\right)dp.}(3.3)$$

The likelihood curve resulting from formula’s (3.1) and (3.2) is an individual respondent profile. The profile delineates the expected UAB occurrence across the full potential probability range from 0 to 1 and gives consideration to the amount of occurrence uncertainty. To illustrate the uncertainty on the individual level, consider two respondents who may both have an expected UAB value of 0.50, but who filled out a different number of items for which the UAB was possible. For instance, respondents A and B showed UAB for 10 out of 20 items and for 30 out of 60 items respectively. See Graph 1 in Figure 3.1. Here, our method takes into account that the expected value of 0.50 is more precisely estimated for respondent B than for respondent A. This is visible by the relatively more narrow and peaked profile for respondent B, indicating that this respondent’s UAB occurrence is relatively more certain.

By considering all respondents who meet the condition of a specific category for a characteristic (for instance lower educated respondents for educational level), the average respondent group profile can be calculated by simply summing their comparable individual profiles and dividing the outcome by the number of respondents:

$$\overline{\lambda }\left(p\right)=\frac{1}{R}{\displaystyle \sum _{r=1}^R{\tilde{\lambda }}_r\left(p\right)},(3.4)$$

where $\overline{\lambda }$ is the respondent profile of the group UAB occurrence averaged over all respondents, and $R$ is the total number of respondents in the group. By means of this average respondent profile, the averaged expected value $\overline{E}$ for the UAB occurrence for this group of respondents can be calculated as follows:

$$\overline{E}={\displaystyle \underset{p=0}{\overset{1}{\int }}p\overline{\lambda }\left(p\right)dp}.(3.5)$$

The likelihood curve resulting from formula (3.4) is a group respondent profile. To illustrate the uncertainty on the group level, consider the two groups of lower and higher educated respondents showing a specific UAB. See Graph 2 in Figure 3.1. The expected values for the groups are 0.50 and almost 0.80 respectively. Our method shows that the expected UAB occurrence is more precisely estimated for higher than for lower educated respondents. It is also visible that for lower educated respondents, the UAB occurrence is not centered around the expected group value of 0.50, but around the values of 0.40 and 0.60. Although formula (3.4) refers to a profile for a group of respondents, it does give an indication of individual UAB. Consider the respondent profile in Figure 3.2 containing individuals on all educational levels. The majority of individuals does not show a specific UAB very often considering the large bump left of the center. On the right, a small peak is visible that refers to a subgroup of individuals showing the UAB very often. These respondents may be either lower or higher educated respondents, or they may share another characteristic that is associated with a high UAB occurrence. The point here is that the respondent profile takes into account the individual UAB and that subgroups of individuals showing a specific occurrence of UAB may be identified in the profile.

Note that by using this method of constructing respondent profiles, we assume that individual UAB is independent across items. This assumption may be partly unjustified, as there may be interdependence across items to some extent in practice. Elaborating on taking into account interdependence across items is beyond the scope of this paper. We refer to Bais (2021) for suggestions on how to cope with interdependence across items in future research using respondent profiles.

Also note that we choose not to use a more traditional model like multilevel analysis to analyze our data. We do not follow identified individual respondents across surveys, but we analyze subgroups of respondents sharing the same characteristic by our profile method for several reasons. Besides taking into account the uncertainty that comes along with the delimited and varying number of respondents and/or items, respondent profiles fully summarize and graphically visualize UAB for subgroups of respondents. And by means of full respondent profiles, relatively small subgroups that deviate from the main body of a larger group may be detected. Throughout this paper, note that a category of respondents refers to respondents in a specific single age or educational category (see Table 2.1), while a (sub)group of respondents may also refer to respondents from several age or educational categories.

In summary, the expected values of two groups with different characteristics indicate the average UAB occurrences for the groups as a whole. In this way, an idea is obtained about the difference of the occurrences of specific UAB (for instance answering don’t know) between two groups (for instance lower and higher educated respondents). The next step is to use a solid analysis to compare the UAB occurrences of two groups.

3.4   Cliff’s Delta for comparing groups of respondents

To compare two groups or categories of respondents meeting a specific characteristic, an adaptation of the effect size Cliff’s Delta (Cliff, 1993, 1996ab) is used. Cliff’s Delta $\delta$ can be used as a robust alternative to using two independent group means. Using Cliff’s Delta for our research asks for an adapted version of the statistic, as we are not considering data observations but density distributions.

The original Cliff’s Delta for data observations

Cliff’s Delta $\delta$ is a robust effect size that indicates to what extent two groups are different. It calculates the probability that a random data observation $X_a$ from a group A is larger than a random data observation $X_b$ from another group B, minus the reverse probability (Hess and Kromrey, 2004; Rousselet, Foxe and Bolam, 2016; Rousselet, Pernet and Wilcox, 2017). In practice, this means that each data observation in group A is compared to each data observation in group B. Then a value is assigned to each such comparison. If an observation from group A is larger than an observation in group B, this value is 1. If an observation in group A is smaller than an observation in group B, this value is -1. If the observations in group A and B are equal, this value is 0. Then the total sum of all these values is divided by the total number of comparisons, giving Cliff’s Delta. The smaller the overlap between the distributions of two groups, the more difference between the two groups. A Cliff’s Delta of -1 or 1 indicates absence of overlap between two groups and a Cliff’s Delta of 0 refers to group equivalence (Hess and Kromrey, 2004). The sample estimate of Cliff’s Delta $\widehat{\delta }$ is

$$\widehat{\delta }=\frac{{\displaystyle {\sum }_{a=1}^A{\displaystyle {\sum }_{b=1}^B\mathrm{sgn}\left(X_a-X_b\right)}}}{AB},(3.6)$$

where $\left(X_a-X_b\right)$ results in a positive or negative number or 0, the sign function “sgn” transforms each positive number into 1 and each negative number into -1, and preserves each 0, and A and B are the sizes of group A and group B respectively.

Adapting Cliff’s Delta for density distributions

We need to adapt the original Cliff’s Delta for our respondent profiles that consist of likelihood distributions. Consider Cliff’s Delta for which each specific observation from sample A is compared to each specific observation from sample B exactly once. This means that when an observation with a specific value from sample A occurs three times, this observation value is compared to all observations from sample B three times as well. Therefore, we may regard both observations for each such comparison on its own as having a “frequency” or “weight” of 1. When we transpose this idea to respondent profiles, we may consider the UAB probabilities from 0 to 1 (with a specific step size interval) our “observations” and the likelihoods for each probability their “frequencies” or “weights”.

$$\widehat{\delta }=\frac{{\displaystyle {\sum }_{a=1}^A{\displaystyle {\sum }_{b=1}^B\mathrm{sgn}\left(P_a-P_b\right)\overline{\lambda }\left(P_a\right)\overline{\lambda }\left(P_b\right)}}}{{\displaystyle {\sum }_{a=1}^A{\displaystyle {\sum }_{b=1}^B\overline{\lambda }\left(P_a\right)\overline{\lambda }\left(P_b\right)}}},(3.7)$$

where $P_a$ and $P_b$ are the probabilities from 0 to 1 from group A and group B respectively, $\overline{\lambda }(P_a)$ and $\overline{\lambda }(P_b)$ are the averaged likelihoods of the probabilities $P_a$ and $P_b$ respectively, and $A$ and $B$ are the same number of step size intervals for both groups.

As a brief illustration, we calculate the adapted Cliff’s Delta by means of formula (3.7) for the respondent profiles in Figure 3.1. Consider Graph 1. When comparing the profiles for respondent A to respondent B, Cliff’s Delta is 0. Although the two profiles slightly differ, their shapes are symmetrically formed around the shared expected value of 0.50. This means that the various values in the denominator of formula (3.7) cancel each other out. Consider Graph 2. When comparing the profiles for lower to higher educated respondents, Cliff’s Delta is -0.99. The profiles hardly overlap and the higher educated respondents clearly show more of some UAB than the lower educated respondents. The reason that Cliff’s Delta is not exactly 1 can be explained by the very small part of overlap around the probability of 0.70 (see Graph 2). Note that the sign would change and Cliff’s Delta would be 0.99 when we would compare higher to lower (instead of lower to higher) educated respondents.

For our study, we use the adaptation of Cliff’s Delta in order to compare respondent profiles. The respondent profiles and this adaptation take into account the fact that each respondent fills out a delimited and different number of items (see Section 3.3). Cliff’s Delta has many advantages with respect to answering our research question. Cliff’s Delta makes no assumption about the shape of the underlying distribution (Cliff, 1993, 1996ab; Goedhart, 2016; Vargha and Delaney, 2000) and is robust in case of outliers or skewed or otherwise non-normal distributions (Goedhart, 2016). Cliff’s Delta is easy to calculate, straightforward to interpret, and standardized, meaning different effect size categories can be distinguished (Goedhart, 2016; see Section 4.2 for these categories). For our adapted Cliff’s Delta, relatively small or unequal sample sizes are no issue.

3.5   Confidence intervals for Cliff’s Delta and statistics

For each Cliff’s Delta, we use confidence intervals to refer to its amount of uncertainty. For a respondent characteristic, each Cliff’s Delta is based on the comparison between the profile of a category and the overall profile of the remaining categories taken together. For a confidence interval, we bootstrap 10,000 category profiles and 10,000 overall profiles. We use the so-called empirical bootstrap method, as we cannot make assumptions about the profiles that are non-parametric by definition (see for instance Dekking, Kraaikamp, Lopuhaä and Meester, 2005 for more on this bootstrap method). For each profile, respondents are randomly sampled with replacement and their individual profiles are averaged by means of formula (3.4). The number of sampled respondents is equal to the number of respondents in the category or overall group respectively. By means of these averaged bootstrap profiles, we calculate 10,000 Cliff’s Delta’s and rank them from low to high. Because of the large number of Cliff’s Delta’s in our study, we choose to use 99% confidence intervals. This means that we use the 51^st and the 9,950^th Cliff’s Delta in the ranking to construct each confidence interval. In the results section, we show Cliff’s Delta outcomes for the respondent characteristics and their categories for all UABs. Each Cliff’s Delta is accompanied by its 99% confidence interval.

• Previous
• Article main page
• Next