Is undesirable answer behaviour consistent across surveys? An investigation into respondent characteristics
Section 5. Conclusion and discussion
In this study, we investigated to what extent cognitive ability is associated with a high occurrence of undesirable answer behaviour (UAB) consistently across different surveys. For cognitive ability, we used the respondent characteristics age and educational level. The occurrence of UAB is indicated by varying uncertainty, as every respondent filled out a different number of the items that were applicable to each behaviour. To take this varying uncertainty into account, we used an adaptation of the robust effect size statistic Cliff’s Delta to compare groups of respondents in the form of density distributions or respondent profiles. The UAB of respondents from a specific category (for instance “15-24 years” for the characteristic “age”) was compared to the UAB of respondents from the other categories of the characteristic together. For our study, we included the specific satisficing behaviours “answering don’t know”, “acquiescence”, “neutral responding”, “extreme responding”, “primacy responding”, and “straightlining”; the specific sensitivity-based behaviours “socially desirable responding” and “answering won’t tell”; and the respondent characteristics “age” and “education”.
Considering all surveys together overall, specific satisficing and sensitivity-based behaviours are evident for specific age and educational groups. However, there is no consistency across surveys present for the age and educational categories for any of the UABs. This study used response data from a panel consisting of the same respondents. In general, if UAB consistency was to be expected at all, this should particularly be found in such a panel. If respondents would have any predisposition to show a behaviour style or pattern, this should especially occur while getting familiar with filling out multiple panel surveys within a specific time span. The fact that we did not find such patterns means that cognitive ability is most likely not a predictor of consistent UAB across surveys.
Considering consistency from a more liberal perspective, specific forms of satisficing across surveys seem evident for specific respondents in particular. Young and lower educated respondents gave relatively more “don’t know”-answers; higher educated respondents chose relatively more answering options early in the list; young and lower educated respondents chose relatively less answering options early in the list; and higher educated respondents showed relatively less neutral responses for multiple surveys. However, there is no category for age or education that showed specific UAB consistently across all or almost all surveys.
Note that within a single survey, items are clustered around a central topic and may also be similar in their characteristics. This means that some item interdependency may occur within surveys. If we would have found consistent response patterns across surveys, these patterns may have been influenced by such item interdependency. Obviously, some respondents may be more sensitive to item interdependency in showing UAB across surveys than others. In our study, we did not find any consistent response patterns across surveys. This means that item interdependency was unlikely to exert a structurally different influence on the various categories of respondents across surveys.
Our results seem to go beyond the absence of UAB consistency across surveys. As the more surveys were applicable to an UAB, the more contrasting outcomes were found; many categories were associated with relatively more of an UAB for some surveys, while relatively less of that UAB for other surveys. Most contrasting results were found for giving socially desirable responses. More evidence was found for contrasting UAB than for consistent UAB across surveys. This evidence is not compatible to our idea that specific groups will show consistency for at least some of the specific UABs across most or all surveys. Overall, we conclude that the occurrence of UAB cannot unambiguously be attributed to the respondent’s cognitive ability, but may be substantially determined by the characteristics of the survey and its items instead.
Following this conclusion, we do not recommend survey-independent adaptive survey design for respondents based on their cognitive ability. The findings for age and educational level are not consistent and clearly differ depending on both survey and UAB. In essence, this means that our outcomes confirm the different associations and their different directions of the existing literature. The added value of our study is the overarching overview for age and educational level, systematically examined across a set of ten different surveys for a range of eight different UABs. We conclude that age and educational level may be taken into account for adaptive survey design only for specific surveys and survey topics.
In our study, we did not focus on UAB of identified individual or groups of respondents. For all age and educational categories, each respondent was considered for every applicable survey that the respondent participated in. Thus, for the consistency analysis of a category, some respondents were considered for only one or two surveys, while other respondents were considered for all or almost all surveys. Our purpose was neither to attribute UAB to individual or groups of identified respondents, nor to compare them between surveys for the same category and UAB. Considering respondents multiple times, for each applicable survey, was the strength of our study. Taking into account every respondent who fell into a category for every applicable survey resulted in large groups per survey. We compared respondent profiles of large groups for a single category to respondent profiles of large groups for the remaining categories. This means that we focussed on the association between the respondent’s characteristics and potentially consistent UAB across surveys. In other words, we did not attribute UAB to identified respondents, but to the specific category (for instance respondents aged 15-24) in which they were placed. Considered from this approach, we note that we deliberately did not use a more classic method like cross-classified multilevel analysis (see for instance Olson and Smyth, 2015; Olson, Smyth and Ganshert, 2019) that takes into account repeated measurements of individual respondents. The focus of our study was placed on visualizing summaries of UAB and comparing subgroups that share the same characteristic.
We used the comparisons between a category and the remaining categories together for age and education to answer our consistency research question. For this purpose, we used an adaptation of Cliff’s Delta; a robust effect size measure that was both useful because of its many advantages regarding our data, and sufficient for comparing two groups representing a specific category versus the remaining categories. In case of differences in expected group value or group shape, follow-up research may zoom in on these differences to reveal characteristics of subgroups showing relatively more of an UAB for specific surveys and their topics and items. Other relevant characteristics like respondent gender and origin may also be investigated. In particular, we would be interested in single groups with higher expected values than the other groups for a characteristic and in the respondents who are located to the right of the respondent profile.
Other follow-up research using the profile method may focus on the relation between item characteristics and UAB. Just as respondent characteristics, item characteristics have their influence on data quality and may be associated with measurement error. See Bais et al. (2019); Beukenhorst, Buelens, Engelen, Van der Laan, Meertens and Schouten (2014); Campanelli et al. (2011); Gallhofer, Scherpenzeel and Saris (2007), and Saris and Gallhofer (2007) for overviews of item characteristics and their relation to measurement error. Items can be coded on the presence or absence of characteristics like for instance question sensitivity. Hence, items that are coded as sensitive could be compared to items that are not coded as sensitive on the occurrence of UAB. In this way, the presence of item characteristics may be connected to UAB for the items of whole surveys specifically or across the items of multiple surveys more generally. Based on such associations, an overview of present item characteristics and their relation to UAB and measurement error may be obtained.
Acknowledgements
We would like to thank Joost van der Neut for contributing to the adaptation of Cliff’s Delta. We would like to thank CentERdata for the availability of LISS Panel data.
Appendix A
| TO | AS | FA | HE | HO | IN | PE | PO | RE | WO | LF | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| SD 65+ | 0.66 | 0.95 | 0.61 | 0.66 | Note *** | 0.79 | 0.77 | 0.59 | 0.27 | 0.77 | This is an empty cell |
| SD 75+ | 0.65 | 0.96 | 0.60 | 0.64 | This is an empty cell | 0.78 | 0.76 | 0.58 | 0.30 | 0.79 | This is an empty cell |
| PR 65+ | 0.33 | This is an empty cell | 0.49 | 0.65 | This is an empty cell | 0.36 | 0.25 | 0.18 | 0.68 | 0.17 | This is an empty cell |
| PR 75+ | 0.31 | This is an empty cell | 0.50 | 0.65 | This is an empty cell | 0.33 | 0.24 | 0.16 | 0.66 | 0.13 | This is an empty cell |
| DK 65+ | 0.06 | This is an empty cell | This is an empty cell | This is an empty cell | 0.07 | 0.16 | This is an empty cell | 0.06 | 0.00 | This is an empty cell | This is an empty cell |
| DK 75+ | 0.06 | This is an empty cell | This is an empty cell | This is an empty cell | 0.07 | 0.14 | This is an empty cell | 0.07 | 0.00 | This is an empty cell | This is an empty cell |
| ST 65+ | 0.10 | This is an empty cell | 0.05 | 0.36 | This is an empty cell | 0.32 | 0.02 | 0.07 | 0.24 | This is an empty cell | This is an empty cell |
| ST 75+ | 0.08 | This is an empty cell | 0.04 | 0.25 | This is an empty cell | 0.29 | 0.01 | 0.06 | 0.19 | This is an empty cell | This is an empty cell |
| WT 65+ | 0.05 | This is an empty cell | This is an empty cell | This is an empty cell | 0.02 | 0.04 | This is an empty cell | This is an empty cell | This is an empty cell | This is an empty cell | 0.03 |
| WT 75+ | 0.04 | This is an empty cell | This is an empty cell | This is an empty cell | 0.01 | 0.03 | This is an empty cell | This is an empty cell | This is an empty cell | This is an empty cell | 0.03 |
| AC 65+ | 0.47 | This is an empty cell | 0.44 | This is an empty cell | This is an empty cell | 0.50 | 0.45 | 0.19 | This is an empty cell | This is an empty cell | |
| AC 75+ | 0.49 | This is an empty cell | 0.42 | This is an empty cell | This is an empty cell | 0.51 | 0.48 | 0.21 | This is an empty cell | This is an empty cell | |
| NE 65+ | 0.22 | This is an empty cell | 0.28 | This is an empty cell | This is an empty cell | 0.25 | 0.21 | 0.22 | This is an empty cell | This is an empty cell | This is an empty cell |
| NE 75+ | 0.21 | This is an empty cell | 0.28 | This is an empty cell | This is an empty cell | 0.25 | 0.21 | 0.22 | This is an empty cell | This is an empty cell | This is an empty cell |
| EX 65+ | 0.19 | This is an empty cell | 0.37 | This is an empty cell | This is an empty cell | 0.11 | 0.23 | 0.11 | This is an empty cell | This is an empty cell | This is an empty cell |
| EX 75+ | 0.20 | This is an empty cell | 0.40 | This is an empty cell | This is an empty cell | 0.11 | 0.25 | 0.10 | This is an empty cell | This is an empty cell | This is an empty cell |
|
|||||||||||
References
Aichholzer, J. (2013). Intra-individual variation of extreme response style in mixed-mode panel studies. Social Science Research, 42, 957-970. doi: http://dx.doi.org/10.1016/j.ssresearch.2013.01.002.
Alwin, D.F., and Krosnick, J.A. (1991). The reliability of survey attitude measurement. The influence of question and respondent attributes. Sociological Methods and Research, 20,139-181.
Andersen, H., and Mayerl, J. (2019). Responding to socially desirable and undesirable topics: Different types of response behaviour? Methods, Data, Analyses: A Journal for Quantitative Methods and Survey Methodology (MDA), 13(1), 7-35. https://doi.org/10.12758/mda.2018.06.
Andrews, F.M., and Herzog, A.R. (1986). The quality of survey data as related to age of respondent. Journal of the American Statistical Association, 81(394), 403-410.
Antoni, M., Bela, D. and Vicari, B. (2019). Validating earnings in the German National Educational Panel Study: Determinants of measurement accuracy of survey questions on earnings. Methods, Data, Analyses: A Journal for Quantitative Methods and Survey Methodology (MDA), 13(1), 59-90. https://doi.org/10.12758/mda.2018.08.
Bachman, J.G., and O’Malley, P.M. (1984). Yea-saying, nay-saying, and going to extremes: Black-white differences in response styles. Public Opinion Quarterly, 48, 491-509.
Bais, F. (2021). Constructing Behaviour Profiles for Answer Behaviour Across Surveys. Dissertation, Utrecht University. https://doi.org/10.33540/538.
Bais, F., Schouten, B., Lugtig, P., Toepoel, V., Arends-Tóth, J., Douhou, S., Kieruj, N., Morren, M. and Vis, C. (2019). Can survey item characteristics relevant to measurement error be coded reliably? A case study on eleven Dutch General Population Surveys. Sociological Methods and Research, 48(2), 263-295. https://doi.org/10.1177/0049124117729692.
Bais, F., Schouten, B. and Toepoel, V. (2020). Investigating response patterns across surveys: Do respondents show consistency in undesirable answer behaviour over multiple surveys? Bulletin de Méthodologie Sociologique, 147-148(1-2), 150-168. https://doi.org/10.1177/0759106320939891.
Beatty, P., and Herrmann, D. (2002). To answer or not to answer: Decision processes related to survey item nonresponse. In Survey Nonresponse, (Eds., R.M. Groves, D.A. Dillman, J.L. Eltinge and R.J.A. Little), First Edition, New York: John Wiley & Sons, Inc., 71-86.
Beukenhorst, D., Buelens, B., Engelen, F.,Van der Laan, J., Meertens, V. and Schouten, B. (2014). The Impact of Survey Item Characteristics on Mode-Specific Measurement Bias in the Crime Victimisation Survey. CBS Discussion paper 2014-16. Statistics Netherlands, The Hague.
Billiet, J.B., and McClendon, J.M. (2000). Modeling acquiescence in measurement models for two balanced sets of items. Structural Equation Modeling, 7, 608-628. doi: http://dx.doi.org/10.1207/S15328007SEM0704_5.
Binswanger, J., Schunk, D. and Toepoel, V. (2013). Panel conditioning in difficult attitudinal questions. Public Opinion Quarterly, 77, 783-797.
Bishop, G.F., Tuchfarber, A.J. and Oldendick, R.W. (1986). Opinions on fictitious issues: The pressure to answer survey questions. Public Opinion Quarterly, 50, 240-250.
Bradburn, N., Sudman, S., Blair, E. and Stocking, C. (1978). Question threat and response bias. Public Opinion Quarterly, 42, 221-234.
Campanelli, P., Nicolaas, G., Jäckle, A., Lynn, P., Hope, S., Blake, M. and Gray, M. (2011). A Classification of Question Characteristics Relevant to Measurement (Error) and Consequently Important for Mixed Mode Questionnaire Design. Paper presented at the Royal Statistical Society, October 11, London, UK.
Cliff, N. (1993). Dominance statistics: Ordinal analyses to answer ordinal questions. Psychological Bulletin, 114, 494-509.
Cliff, N. (1996a). Answering ordinal questions with ordinal data using ordinal statistics. Multivariate Behavioral Research, 31, 331-350.
Cliff, N. (1996b). Ordinal Methods for Behavioral Data Analysis. New Jersey: Lawrence Erlbaum Associates.
Dekking, F.M., Kraaikamp, C., Lopuhaä, H.P. and Meester, L.E. (2005). A Modern Introduction to Probability and Statistics – Understanding Why and How. Springer. https://doi.org/10.1007/1-84628-168-7.
De Leeuw, E.D. (1992). Data Quality in Mail, Telephone, and Face-to-Face Surveys. Amsterdam: TT-Publicaties.
De Leeuw, E.D., Hox, J.J. and Dillman, D. (2008). International Handbook of Survey Methodology. Taylor & Francis Group.
DeMaio, T.J. (1984). Social desirability and survey measurement: A review. In Surveying Subjective Phenomena, (Eds., C.F. Turner and E. Martin). New York: Russell Sage Foundation, 2, 257-281.
Díaz de Rada, V., and Domínguez, J.A. (2015). The quality of responses to grid questions as used in web questionnaires (compared with paper questionnaires). International Journal of Social Research Methodology, 18, 337-348. doi: http://dx.doi.org/10.1080/13645579.2014.895289.
Fricker, S., Galesic, M., Tourangeau, R. and Yan, T. (2005). An experimental comparison of web and telephone surveys. Public Opinion Quarterly, 69, 370-392. http://dx.doi.org/10.1093/poq/nfi027.
Galesic, M., Tourangeau, R., Couper, M.P. and Conrad, F.G. (2008). Eye-tracking data new insights on response order effects and other cognitive shortcuts in survey responding. Public Opinion Quarterly, 72, 892-913.
Gallhofer, I.N., Scherpenzeel, A. and Saris, W.E. (2007). The Code-Book for the SQP Program, available at http://www.europeansocialsurvey.org/methodology/.
Goedhart, J. (2016). Calculation of a Distribution Free Estimate of Effect Size and Confidence Intervals Using VBA/Excel. doi: http://dx.doi.org/10.1101/073999.
Greenleaf, E.A. (1992). Measuring extreme response style. Public Opinion Quarterly, 56, 328-351. http://www.jstor.org/stable/2749156.
He, J., and Van de Vijver, F.J.R. (2013). A general response style factor: Evidence from amulti-ethnic study in the Netherlands. Personality and Individual Differences, 55, 794-800. http://dx.doi.org/10.1016/j.paid.2013.06.017.
He, J., Van de Vijver, F.J.R., Espinosa, A.D. and Mui, P.H. (2014). Toward a unification of acquiescent, extreme, and midpoint response styles: A multilevel study. International Journal of Cross Cultural Management, 14, 306-322. doi: http://dx.doi.org/10.1177/1470595814541424.
Heerwegh, D., and Loosveldt, G. (2011). Assessing mode effects in a national crime victimization survey using structural equation models: Social desirability bias and acquiescence. Journal of Official Statistics, 27, 49-63.
Hess, M.R., and Kromrey, J.D. (2004). Robust Confidence Intervals for Effect Sizes: A Comparative Study of Cohen’s d and Cliff’s Delta under Non-Normality and Heterogeneous Variances. Paper Presented at the Annual Meeting of the American Educational Research Association, San Diego, California.
Holbrook, A.L., Green, M.C. and Krosnick, J.A. (2003). Telephone versus face-to-face interviewing of national probability samples with long questionnaires: Comparisons of respondent satisficing and social desirability response bias. Public Opinion Quarterly, 67, 79-125.
Hox, J.J., De Leeuw, E. and Kreft, I.G. (1991). The effect of interviewer and respondent characteristics on the quality of survey data: A multilevel model. In Measurement Errors in Surveys, (Eds., P.P. Biemer, R.M. Groves, L.E. Lyberg, N.A. Mathiowetz and S. Sudman), New York: John Wiley & Sons, Inc., 439-461.
Jann, B., Krumpal, I. and Wolter, F. (2019). Editorial: Social desirability bias in surveys – Collecting and analyzing sensitive data. Methods, Data, Analyses: A Journal for Quantitative Methods and Survey Methodology (MDA), 13(1), 3-6.
Johnson, T., and Van de Vijver, F.J.R. (2003). Social desirability in cross cultural research. In Cross-Cultural Survey Methods, (Eds., J. Harness, F.J.R. van de Vijver and P. Mohler.), New York: John Wiley & Sons, Inc., 193-202.
Kalton, G., Roberts, J. and Holt, D. (1980). The effects of offering a middle response option with opinion questions. Statistician, 29, 65-78. http://www.jstor.org/stable/2987495.
Kaminska, O., McCutcheon, A. and Billiet, J. (2010). Satisficing among reluctant respondents in a cross-national context. Public Opinion Quarterly, 74, 880-906. doi: http://dx.doi.org/10.1093/poq/nfq062.
Kieruj, N.D., and Moors, G. (2013). Response style behavior: Question format dependent or personal Style? Quality and Quantity, 47, 193-211. doi: http://dx.doi.org/10.1007/s11135-011-9511-4.
Kreuter, F., Presser, S. and Tourangeau, R. (2008). Social desirability bias in CATI, IVR, and web surveys: The effects of mode and question sensitivity. Public Opinion Quarterly, 72, 847-865. doi: http://dx.doi.org/10.1093/poq/nfn063.
Krosnick, J.A. (1991). Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied Cognitive Psychology, 5, 213-236.
Krosnick, J.A. (1992). The impact of cognitive sophistication and attitude importance onresponse order effects and question order effects. In Order Effects in Social and Psychological Research, (Eds., N. Schwarz and S. Sudman), New York: Springer, 203-218.
Krosnick, J.A. (1999). Survey research. Annual Review of Psychology, 50, 537-567.
Krosnick, J.A., and Alwin, D.F. (1987). An evaluation of a cognitive theory of response order effects in survey measurement. Public Opinion Quarterly, 51, 201-219.
Krosnick, J.A., and Alwin, D.F. (1989). Aging and susceptibility to attitude change. Journal of Personality and Social Psychology, 57, 416-425.
Krosnick, J.A., and Fabrigar, L.R. (1997). Designing rating scales for effective measurement in surveys. In Survey Measurement and Process Quality, (Eds., L. Lyberg, P. Biemer, M. Collins, L. Decker, E. de Leeuw, C. Dippo, N. Schwarz and D. Trewin), New York: John Wiley & Sons, Inc., 141-164.
Krosnick, J.A., Holbrook, A.L., Berent, M.K., Carson, R.T., Hanemann, W.M., Kopp, R.J., Mitchell, R.C., Presser, S., Ruud, P.A., Smith, V.K., Moody, W.R., Green, M.C. and Conaway, M. (2002). The impact of “no opinion” response options on data quality: Non-attitude reduction or an invitation to satisfice? Public Opinion Quarterly, 66, 371-403.
Krosnick, J.A., Narayan, S. and Smith, W.R. (1996). Satisficing in surveys: Initial evidence. New Directions for Evaluation, 70, 29-44.
Leigh, J.H., and Martin, C.R. (1987). Don’t know item nonresponse in a telephone survey: Effects of question form and respondent characteristics. Journal of Marketing Research, 24, 418-424.
Lensvelt-Mulders, G.J.L.M. (2008). Surveying sensitive topics. In International Handbook of Survey Methodology, (Eds., E.D. de Leeuw, J.J. Hox and D.A. Dillman). New York: Taylor and Francis, Psychology Press, EAM series, 461-478.
Marín, G., Gamba, R.J. and Marín, B.V. (1992). Extreme response style and acquiescence among hispanics. Journal of Cross-Cultural Psychology, 23, 498-509.
McClendon, M.J. (1986). Response-order effects for dichotomous questions. Social Science Quarterly, 67, 205-211.
McClendon, M.J. (1991). Acquiescence and recency response-order effects in interview surveys. Sociological Methods and Research, 20, 60-103.
Medway, R., and Tourangeau, R. (2015). Response quality in telephone surveys. Do pre-paid cash incentives make a difference? Public Opinion Quarterly, 79, 524-543. doi: https://dx.doi.org/10.1093/poq/nfv011.
Meisenberg, G., and Williams, A. (2008). Are acquiescent and extreme response styles related to low intelligence and education? Personality and Individual Differences, 44, 1539-1550. https://dx.doi.org/10.1016/j.paid.2008.01.010.
Messick, S.J. (1966). The psychology of acquiescence: An interpretation of research evidence. In Response Set in Personality Assessment, (Ed., I.A. Berg), Chicago: Aldine, 115-145.
Narayan, S., and Krosnick, J.A. (1996). Education moderates some response effects in attitude measurement. Public Opinion Quarterly, 60, 58-88.
Olson, K., and Smyth, J.D. (2015). The effect of CATI questions, respondents, and interviewers on response time. Journal of Survey Statistics and Methodology, 3, 361-396. doi: https://dx.doi.org/10.1093/jssam/smv021.
Olson, K., Smyth, J.D., and Ganshert, A. (2019). The effects of respondent and question characteristics on respondent answering behaviors in telephone interviews. Journal of Survey Statistics and Methodology, 7(2), 275- 308. https://doi.org/10.1093/jssam/smy006.
O’Muircheartaigh, C., Krosnick, J.A. and Helic, A. (2000). Middle Alternatives, Acquiescence, and the Quality of Questionnaire Data. Retrieved, October 1, 2009, from http://harrisschool.uchicago.edu/About/publications.
Paulhus, D.L. (2002). Socially desirable responding: The evolution of a construct. In The Role of Constructs in Psychological and Educational Measurement, (Eds., H.I. Braun, D.N. Jackson and D.E. Wiley), Mahwah, NJ: Erlbaum, 49-69.
Pickery, J., and Loosveldt, G. (1998). The impact of respondent and interviewer characteristics on the number of “No opinion” answers. A multilevel model for count data. Quality and Quantity, 32, 31-45.
Roberts, C. (2007). Mixing modes of data collection in surveys: A methodological review. ESRC National Centre for Research Methods, NCRM Methods Review Paper 008, UK. Retrieved July 2019 from http://eprints.ncrm.ac.uk/418/1/MethodsReviewPaperNCRM-008.pdf.
Roberts, C., and Jäckle, A. (2012). Causes of Mode Effects: Separating out Interviewer and Stimulus Effects in Comparisons of Face-to-Face and Telephone Surveys. ISER Working Paper, 2012-27. Colchester: University of Essex.
Roßmann, J., Gummer, T. and Silber, H. (2017). Mitigating satisficing in cognitively demanding grid questions: Evidence from two web-based experiments. Journal of Survey Statistics and Methodology.
Rousselet, G.A., Foxe, J.J. and Bolam, J.P. (2016). A few simple steps to improve the description of group results in neuroscience. European Journal of Neuroscience, 44, 2647-2651. doi: http://dx.doi.org/10.1111/ejn.13400.
Rousselet, G.A., Pernet, C.R. and Wilcox, R.R. (2017). Beyond differences in means:robust graphical methods to compare two groups in neuroscience. European Journal of Neuroscience, 1-27. doi: http://dx.doi.org/10.1101/121079.
Saris, W.E., and Gallhofer, I.N. (2007). Estimation of the effects of measurement characteristics on the quality of survey questions. Survey Research Methods, 1(1), 29-43. http://dx.doi.org/10.18148/srm/2007.v1i1.49.
Saris, W.E., Revilla, M., Krosnick, J.A. and Shaeffer, E. (2010). Comparing questions with agree/disagree response options to questions with item-specific response options. Survey Research Methods, 4(1), 61-79. http://www.surveymethods.org.
Schaeffer, N.C., and Presser, S. (2003). The science of asking questions. Annual Review of Sociology, 29, 65-88. doi: http://dx.doi.org/10.1146/annurev.soc.29.110702.110112.
Schonlau, M., and Toepoel, V. (2015). Straightlining in web survey panels over time. Survey Research Methods, 9, 125-137. doi: http://dx.doi.org/10.18148/srm/2015.v9i2.6128.
Schouten, B., and Calinescu, M. (2013). Paradata as input to monitoring representativeness and measurement profiles: A case study of the Dutch Labour Force Survey. In Improving Surveys with Paradata: Analytic Uses of Process Information, (Ed., F. Kreuter), Hoboken, NJ: Wiley, 231-258.
Schuman, H., and Presser, S. (1981). Questions and Answers in Attitude Surveys: Experiments on Question Form, Wording and Context. New York: Academic Press.
Shoemaker, P.J., Eichholz, M. and Skewes, E.A. (2002). Item nonresponse: Distinguishing between don’t know and refuse. International Journal of Public Opinion Research, 14, 193-201.
Si, S.X., and Cullen, J.B. (1998). Response categories and potential cultural bias: Effects of an explicit middle point in cross-cultural surveys. International Journal of Organizational Analysis, 6, 218-230.
Stern, M.J., Dillman, D.A. and Smyth, J.D. (2007). Visual design, order effects, and respondent characteristics in a self-administered survey. Survey Research Methods, 1, 121-138. http://www.surveymethods.org.
Stricker, L.J. (1963). Acquiescence and social desirability response styles, item characteristics, and conformity. Psychological Reports, 12, 319-341.
Tarnai, J., and Dillman, D.A. (1992). Questionnaire context as a source of response differences in mail versus telephone surveys. In Context Effects in Social and Psychological Research, (Eds., N. Schwarz and S. Sudman), New York: Springer Verlag.
Tourangeau, R., Rips, L.J. and Rasinski, K. (2000). The Psychology of Survey Response. Cambridge: Cambridge University Press.
Tourangeau, R., and Yan, T. (2007). Sensitive questions in surveys. Psychological Bulletin, 133, 859-883. doi: http://dx.doi.org/10.1037/0033-2909.133.5.859.
Van Herk, H., Poortinga, Y.H. and Verhallen, T.M.M. (2004). Response styles in rating scales: Evidence of method bias in data from six EU countries. Journal of Cross-Cultural Psychology, 35, 346-360.
Van Rosmalen, J., Van Herk, H. and Groenen, P.J.F. (2010). Identifying response styles: A latent-class bilinear multinomial logit model. Journal of Marketing Research, 47(1), 157-172. doi: http://dx.doi.org/10.1509/jmkr.47.1.157.
Vargha, A., and Delaney, H.D. (2000). A critique and improvement of the CL Common Language effect size statistics of McGraw and Wong. Journal of Educational Behavioral Statistics, 25(2), 101-132. doi: http://dx.doi.org/10.2307/1165329.
Vis-Visschers, R., Arends-Tóth, J., Giesen, D. and Meertens, V. (2008). Het Aanbieden Van “Weet Niet” en Toelichtingen in Een Webvragenlijst. Report DMH-2008-02-21-RVCS, Statistics Netherlands, Methodology Department, Heerlen, The Netherlands.
Ye, C., Fulton, J. and Tourangeau, R. (2011). More positive or more extreme? A meta-analysis of mode differences in response choice. Public Opinion Quarterly, 75(2), 349-365. doi: http://dx.doi.org/10.1093/poq/nfr009.
Zhang, C. (2013). Satisficing in Web Surveys: Implications for Data Quality and Strategies for Reduction, (Ph.D.) Ann Arbor, MI: University of Michigan. Retrieved from http://deepblue.lib.umich.edu/handle/2027.42/97990.
Zhang, C., and Conrad, F.G. (2013). Speeding in Web surveys: The tendency to answer very fast and its association with straightlining. Survey Research Methods, 8, 127-135.
- Date modified: