The promise and challenge of pushing respondents to the Web in mixed-mode surveys
Section 7. Summary and Conclusion
Web-push data collection that begins with a postal mail request to respond over the Internet is one of the major survey design developments of the early 21st century, now offering promise of faster, less-expensive surveys. Many surveyors have been surprised by today’s reliance on an initial postal contact. Although mail surveys had often been used to collect survey data, it was expected by many to disappear with the rise of Internet.
The critical development that encouraged reconsideration of mail contact methods greater use was by Link et al. (2008) and Battaglia et al. (2008). This research showed that residential address lists available from the U.S. Postal Service provided the best sample coverage of U.S. residences and could be used to support effective mail surveys of the general public. This work was encouraged by the strong desire to find alternatives to RDD telephone surveys that faced continually declining response and other challenges.
A series of studies, beginning in 2007, looked for ways to use mail contacts to push householders to the web from these address-based lists. This work focused on combining both Internet and paper responses. It was supported by several years of earlier research on measurement differences across modes that showed responses to web and paper questionnaires were quite similar so long as similar question structures, wordings, and visual layouts were used for both data collection methods. Ten experimental comparisons made in these studies received a web-push response rate of 43% of households, with about 60% of the responses coming over the internet and the remainder being obtained by a mail follow-up (Dillman et al. 2014). Major surveys in several countries have researched and adopted the use of web-push methods that rely on not only web and mail, but now include telephone and/or in-person follow-up in their protocols. The goal is to achieve greater response rates and data quality, which a decade ago were thought to be no longer possible in household surveys.
We are now in an era of tailored design in which different survey designs are used for different survey topics, populations, and survey situations. However, it seems likely that web-push data collection methods will see increased use throughout the industrialized world, as survey sponsors seek to benefit from the low cost of internet data collection in order to lower the overall cost of current surveys.
However such methods face challenges that need attention. One is the risk to surveys and respondents from malware, phishing, and server attacks. Another is the increased reliance on smartphones that may require significant changes in how questions are structured and presented to respondents. In addition, the reluctance of organizations and individuals to accept and master the greater complexity associated with shifting from single mode to mixed-mode surveys is a significant challenge.
The history of surveying over the last 75 years has involved significant transitions from the dominance of in-person interviews, to heavy reliance on voice telephone methods, and now to online and mixed mode surveys. It remains to be seen whether web-push methods now growing in use as a replacement have a lasting presence, or will eventually be replaced with web-only data collection, or with other procedures that remain to be innovated or have not yet been conceived.
References
Anderson, M., and Perrin, A. (2016). 13% of Americans don’t use the Internet, Who are they? Available at http://www.pewresearch.org/fact-tank/2016/09/07/some-americans-dont-use-the-internet-who-are-they/. Accessed May 2, 2017.
Australian Bureau of Statistics (2016). Making Sense of the Census. Available at http://www.abs.gov.au/websitedbs/censushome.nsf/home/2016. Accessed May 2, 2017.
Barlas, F.M., and Thomas, R.K. (2016). Good questionnaire design: Best practices in the mobile era. American Association for Public Opinion Research, January 19th.
Battaglia, M.P., Link, M.W., Frankel, M.R., Osborn, L. and Mokdad, A.H. (2008). An evaluation of respondent selection methods for household mail surveys. Public Opinion Quarterly, 72(3), 459-469.
Biemer, P., Murphy, J., Zimmer, S., Berry, C., Deng, G. and Lewis, K. (2016). A test of Web/PAPI protocols and incentives for the residential energy consumption survey. Unpublished paper presented at Annual Conference of the American Association for Public Opinion Research. May 13th.
Blankenship, A.B. (1977). Professional Telephone Surveys. New York: McGraw-Hill Book Company.
Blumberg, S.J., and Luke, J.V. (2017). Wireless Substitution: Early Release of Estimates from the National Health Interview Survey, January-June 2016.
Brick, J.M., and Williams, D. (2013). Explaining rising nonresponse rates in cross-sectional surveys. Annals of the American Academy of Political and Social Science, 645(1), 36-59.
Brick, J.M., Williams, D. and Montaquila, J.M. (2011). Address-based sampling for subpopulation surveys. Public Opinion Quarterly, 75(3), 409-428.
Christian, L.M., and Dillman, D.A. (2004). The influence of symbolic and graphical language manipulations on answers to paper self-administered questionnaires. Public Opinion Quarterly, 68, 1, 57-80.
Christian, L.M., Dillman, D.A. and Smyth, J.D. (2008). The effects of mode and format on answers to scalar questions in telephone and Web surveys. In Advances in Telephone Survey Methodology, (Eds., J.M. Lepkowski, C. Tucker, J.M. Brick, E.D. de Leeuw, L. Japec, P.J. Lavrakas, M.W. Link and R.L. Sangster). New York: Wiley-Interscience, 250-275.
Christian, L.M., Parsons, N.L. and Dillman, D.A. (2009). Designing scalar questions for Web surveys. Sociological Methods and Research, 37(3), 393-425.
City of Sapporo (2015). The Japanese government is conducting a Population Census. Available at https://www.city.sapporo.jp/city/english/news/news201508_1e.html. Accessed October 1, 2016.
Clark, S., and Roberts, A. (2016). Evaluation of August 2015 ACS mail contact strategy modification. 2016 American Community Survey Research and Evaluation Report Memorandum Series ACS16-0RER-13.
Couper, M.P., Antoun, C. and Mavletova, A. (2017). Mobile Web surveys. In Total Survey Error in Practice, (Eds., P.P. Biemer, E. de Leeuw, S. Eckman, B. Edwards, F. Kreuter, L.E. Lyberg, N.C. Tucker and B.T. West). New Jersey: Hoboken.
de Leeuw, E.D. (1992). Data quality in mail, telephone and face-to-face surveys. TT-Publications Amsterdam.
de Leeuw, E.D. (2005). To mix or not to mix data collection modes in surveys. Journal of Official Statistics, 21(2), 233-255.
de Leeuw, E., Villar, A., Suzer-Gurtekin, T. and Hox, J. (2017). How to design and implement mixed-mode surveys in cross National Surveys: Overview and guideline. In Total Survey Error in Practice, (Eds., P.P. Biemer, E. de Leeuw, S. Eckman, B. Edwards, F. Kreuter, L.E. Lyberg, N.C. Tucker and B.T. West). New Jersey: Hoboken.
Dillman, D.A. (1978). Mail and Telephone Surveys: The Total Design Method. New York: John Wiley & Sons, Inc.
Dillman, D.A. (2000). Mail and Internet Surveys: The Tailored Design Method, 2nd Edition. New York: John Wiley & Sons, Inc.
Dillman, D.A. (2002). Navigating the rapids of change: Some observations on survey methodology in the early 21st century. Public Opinion Quarterly, 66(3), 473-494.
Dillman, D.A. (2005). Telephone surveys. In Encyclopedia of Social Measurement, (Ed., K. Kempf-Leonard), Volume 3. London, UK: Elsevier Press, 757-762.
Dillman, D.A. (2007). Mail and Internet Surveys: The Tailored Design Method. 2007 Update with New internet. Visual and Mixed-mode Guide. New Jersey: Hoboken.
Dillman, D.A., and Christian, L.M. (2005). Survey mode as a source of instability in responses across surveys. Field Methods, 17(1), 30-52.
Dillman, D.A., and Tarnai, J. (1988). Administrative issues in mixed-mode surveys. In Telephone Survey Methodology, (Eds., R.M. Groves, P.P. Biemer, L.E. Lyberg, J.T. Massey, W.L. Nicholls II and J. Waksberg), New York: John Wiley & Sons, Inc., 509-528.
Dillman, D.A., and Edwards, M.L. (2016). Designing a mixed-mode survey. In The SAGE Handbook of Survey Methodology, (Eds., C. Wolfe, D. Joye, T.W. Smith and Y.-c. Fu), Sage Publications, Thousand Oaks, CA, 255-268.
Dillman, D.A., Clark, J.R. and Sinclair, M.D. (1995). How prenotice letters, stamped return envelopes, and reminder postcards affect mailback response rates for census questionnaires. Survey Methodology, 21, 2, 159-165. Paper available at http://www.statcan.gc.ca/pub/12-001-x/1995002/article/14394-eng.pdf.
Dillman, D.A., Gertseva, A. and Mahon-Haft, T. (2005). Achieving usability in establishment surveys through the application of visual design principles. Journal of Official Statistics, 21(2), 183-214.
Dillman, D.A., Hao, F. and Millar, M.M. (2016). Improving the effectiveness of online data collection by mixing survey modes. In The Sage handbook of Online Research Methods, 2nd Edition, (Eds., N. Fielding, R.M. Lee and G. Blank). Sage Publications, London, 220-237.
Dillman, D.A., Smyth, J.D. and Christian, L.M. (2014). Internet, Phone, Mail and Mixed-Mode Surveys: The Tailored Design Method, 4th Edition. New Jersey: Hoboken.
Dillman, D.A., Phelps, G., Tortora, R., Swift, K., Kohrell, J., Berck, J. and Messer, B.L. (2009). Response rate and measurement differences in mixed-mode surveys using mail, telephone, interactive voice response (IVR) and the Internet. Social Science Research, 38(1), 1-18.
Dutwin, D., and Lavrakas, P. (2016). Trends in telephone outcomes, 2008-2015. Survey Practice, 9(3). Available at http://www.surveypractice.org/.
Edwards, M.L., Dillman, D.A. and Smyth, J.D. (2014). An experimental test of the effects of survey sponsorship on Internet and mail survey response. Public Opinion Quarterly, 78(3), 734-750.
Finamore, J., and Dillman, D.A. (2013). How mode sequence affects responses by internet, mail and telephone in the national survey of college graduates. Presentation to European Survey Research Association, Ljubljana, Slovenia, July 18.
Friedman, T.L. (2016). Thank you for Being Late: An Optimist’s Guide to Thriving in the Age of Accelerations. New York, Farrar: Straus and Giroux.
Groves, R.M., and Kahn, R.L. (1979). Surveys by Telephone. New York: John Wiley & Sons, Inc.
Groves, R.M., and Peytcheta, E. (2008). The impact of nonresponse rates on nonresponse bias: A meta-analysis. Public Opinion Quarterly, 72(2), 167-189.
Harter, R., Battaglia, M.P., Buskirk, T.D., Dillman, D.A., English, N., Mansour, F., Frankel, M.R., Kennel, T., McMichael, J.P., McPhee, C.B., Montaquila, J., Yancey, T. and Zukerberg, A.L. (2016). Address-base sampling. American Association for Public Opinion Research Task Force Report. Available at http://www.aapor.org/getattachment/Education-Resources/Reports/AAPOR_Report_1_7_16_CLEAN-COPY-FINAL-(2).pdf.aspx, 140 pages.
Hoffman, D.D. (2004). Visual Intelligence: How we Create What we See. New York: Norton.
Keeter, S., Miller, C., Kohut, A., Groves, R.M. and Presser, S. (2000). Consequences of reducing nonresponse in a national telephone survey. Public Opinion Quarterly, 64(2), 125-148.
Kerlinger, F.N. (1965). Foundations of Behavioral Research. New York: Holt, Rinehart and Winston.
Krosnick, J.A., and Alwin, D.F. (1987). An evaluation of a cognitive theory of response order effects in survey measurement. Public Opinion Quarterly, 51(2), 201-219.
Link, M.W., Battaglia, M.P., Frankel, M.R., Osborn, L. and Mokdad, A.H. (2008). A comparison of address-based sampling (ABS) versus random-digit dialing (RDD) for General Population Surveys. Public Opinion Quarterly, 72(1), 6-27.
Lozar Manfreda, K., Bosnjak, M., Berzelak, J., Haas, I. and Vehovar. V. (2008). Web surveys versus other survey modes: A meta-analysis comparing response rates. International Journal of Market Research, 50(1), 79-104.
McMaster, H.S., LeardMann, C.A., Speigle, S. and Dillman, D.A. (2016). An experimental comparison of web-push vs. paper-only survey procedures for conducting an in-depth health survey of military spouses. BMC Medical Research Methodology.
Messer, B.L. (2012). Pushing households to the web: Results from Web+mail experiments using address based samples of the general public and mail contact procedures. Ph.D. Dissertation. Washington State University, Pullman.
Messer, B.L., and Dillman, D.A. (2011). Surveying the general public over the Internet using address-based sampling and mail contact procedures. Public Opinion Quarterly, 75(3), 429-457.
Messer, B.L., Edwards, M.L. and Dillman, D.A. (2012). Determinants of item nonresponse to Web and mail respondents in three address-based mixed-mode surveys of the general public. Survey Practice, 5(2), 1-9. Paper available at http://www.surveypractice.org/.
Millar, M.M., and Dillman, D.A. (2011). Improving response to Web and mixed-mode surveys. Public Opinion Quarterly, 75(2), 249-269.
Mistichelli, J., Eanes, G. and Horwitz, R. (2015). Centurion: Internet Data Collection and Responsive Design. Presentation to Federal Economic Statistics Advisory Committee, June 12.
Mohorko, A., de Leeuw, E. and Hox, J. (2013). Coverage bias in European telephone surveys: Developments of landline and mobile phone coverage across countries and over time. Survey Methods: Insights from the Field. Retrieved from http://surveyinsights.org/?p=828.
Nathan, G. (2001). Telesurvey methodologies for household surveys – A review and some thoughts for the future? Survey Methodology, 27, 1, 7-31. Paper available at http://www.statcan.gc.ca/pub/12-001-x/2001001/article/5851-eng.pdf.
O’Muircheartaigh, C., English, N. and Eckman, S. (2007). Predicting the relative quality of alternative sampling frames. 2007 Proceedings of the Survey Research Methods Section, American Statistical Association, [CD ROM], Alexandria, VA: American Statistical Association.
Palmer, S.E. (1999). Vision Science: Photons to Phenomenology. London: Bradford Books.
Parten, M. (1950). Surveys, Polls and Samples. New York: Harper and Brothers.
Pew Research Center (2012). Assessing the representativeness of public opinion surveys. Available at http://www.people-press.org/2012/05/15/assessing-the-representativeness-of-public-opinion-surveys/. Accessed October 24, 2016.
Redline, C.D., and Dillman, D.A. (2002). The influence of alternative visual designs on respondents’ performance with branching instructions in self-administered questionnaires. In Survey Nonresponse, (Eds., R. Groves, D. Dillman, J. Eltinge and R. Little), New York: John Wiley & Sons, Inc.
Redline, C.D., Dillman, D.A., Dajani, A. and Scaggs, M.A. (2003). Improving navigational performance in U.S. census 2000 by altering the visual languages of branching instructions. Journal of Official Statistics, 19(4), 403-420.
Roberts, C., Joye, D. and Staehli, M.E. (2016). Mixing modes of data collection in Swiss social survey: Methodological report of the LIVES-FORS mixed mode experiment. Lives working paper 2016.48. Swiss National Centre of Competence in Research, a research instrument of the Swiss National Science Foundation.
Rookey, B.D., Hanway, S. and Dillman, D.A. (2008). Does a probability-based household panel benefit from assignment to postal response as an alternative to Internet-only? Public Opinion Quarterly, 72(5), 962-984.
Sarraf, S., Brooks, J., Cole, J. and Wang, X. (2015). What is the impact of smartphone optimization on long surveys? Presentation to American Association for Public Opinion Research Annual Conference, Hollywood, FL, May 16.
Smyth, J., Christian, L.M. and Dillman, D.A. (2008). Does ‘Yes or No’ on the telephone mean the same as check-all-that-apply on the Web? Public Opinion Quarterly, 72(1), 103-111.
Smyth, J.D., Dillman, D.A., Christian, L.M. and McBride, M. (2009). Open-ended questions in Web surveys: Can increasing the size of answer boxes and providing extra verbal instructions improve response quality? Public Opinion Quarterly, 73(2), 325-337.
Smyth, J.D., Dillman, D.A., Christian, L.M. and O’Neill, A.C. (2010). Using the Internet to survey small towns and communities: Limitations and possibilities in the early 21st century. American Behavioral Scientist, 53(9), 1423-1448.
Smyth, J.D., Dillman, D.A., Christian, L.M. and Stern, M.J. (2006). Comparing check-all and forced-choice question formats in Web surveys. Public Opinion Quarterly, 70(1), 66-77.
Statistics Canada (2016). 2016 Census of Population collection response rates. Available at http://www12.statcan.gc.ca/census-recensement/2016/ref/response-rates-eng.cfm. Accessed October 24, 2016.
Statistics Japan (2015). Almost 20 million households responded online in the 2015 Population Census of Japan. Available at http://www.stat.go.jp/english/info/news/20151019.htm. Accessed October 1, 2016.
Stern, M., Sterrett, D. and Bilgen, I. (2016). The effects of grids on Web surveys completed with mobile devices. Social Currents, 3(3), 217-233.
Sullivan, M., Leong, C., Churchwell, C. and Dillman, D.A. (2015). Measurement and Cost Effects of Pushing Household Survey Respondents to the Web for Surveys of Electricity and Gas Customers in the United States. Unpublished paper presented to the European Survey Research Association, Reykjavik, Iceland, July 16th.
Tancreto, J. (2012). 2011 American Community Survey Internet Tests: Results from First Test in April 2011. #ACS12-RER-13-R2. 2012 American Community Survey Research and Evaluation Report Memorandum Series, June 25th.
Tarnai, J., and Dillman, D.A. (1992). Questionnaire context as a source of response differences in mail and telephone surveys. In Context Effects in Social and Psychological Research, (Eds., N. Schwarz and S. Sudman), New York: Springer Verlag, Inc. 115-129.
The Economist (2017). Now we’re talking: Voice technology is making computers less daunting and more accessible. January 7th – 13th, 422 (No. 9022), 9.
Thomas, R., and Barlas, F. (2016). It’s a Small Screen After All: Improving Measurement in an Ever-changing Online Survey World. GFK Webinar, September 27th.
Tourangeau, R. (2017). Mixing modes: Tradeoffs among coverage, nonresponse and measurement error. In Total Survey Error in Practice, (Eds., P.P. Biemer, E. de Leeuw, S. Eckman, B. Edwards, F. Kreuter, L.E. Lyberg, N. Clyde Tucker and B.T. West), New Jersey, Hoboken: John Wiley & Sons, Inc.
Tourangeau, R., Couper, M.P. and Conrad, F. (2004). Spacing, position, and order: Interpretive heuristics for visual features of survey questions. Public Opinion Quarterly, 68(3), 368-393.
United Kingdom Cabinet Office (2016). Consultation Response: Community Life Survey: Development and implementation of online survey methodology for future survey years. Available at https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/539111/community_life_survey_consultation_response_final.pdf.
United States Census Bureau (2014). American Community Survey Design and Methodology, Version 2.0. Available at http://www.census.gov/programs-surveys/acs/methodology/design-and-methodology.html. Accessed October 15, 2016.
Ware, C. (2004). Information Visualization: Perception for Design, 2nd Edition. Karlsruhe, West German: Morgan Kaufman.
Williams, D., Brick, J.M., Montaquila, J.M. and Han, D. (2014). Effects of screening questionnaires on response in a two-phase postal survey. International Journal of Social Research Methodology, 19(1), 51-67.
- Date modified: