The promise and challenge of pushing respondents to the Web in mixed-mode surveys
Section 6. Challenges facing web-push data collection
Despite the potential of web-push data collection methods, there are also uncertainties regarding whether the use of Internet surveying will continue to expand in use. These concerns are the focus of the final section of this paper.
6.1 Fear of responding over the internet
When the push-to-web Australian Census began in 2016, a series of denial of service (DOS) attacks on the site prompted the Bureau of Statistics to turn off the system for fear of hackers. Such attacks are designed to overload a server with traffic, thus making it inaccessible to the intended users. This is only one kind of attack that might be made on a particular survey or computer user. Others include sending malware (e.g., spyware or ransomware) designed to gain access to or damage a computer that users unknowingly access by opening attachments or clicking on links. In addition, phishing emails are sometimes sent. They are designed to trick people into opening them, and providing personal information, for example, by appearing to be sent from a user well known to the recipient. The result of these various possibilities is to cause many people to worry about the security, or lack thereof, of the website, and information they provide in response to web survey requests. The lack of trust in web surveys and concerns that information could be kept and used for non-survey purposes are also potential barriers to response.
Large scale surveys, especially those that have a great deal of public visibility, such as a nationwide Census that involves widespread prior communication inviting a response, present an inviting target for those hoping to harm the response process. Thus, even though sponsorship is known the perception of risk may be substantial. In the case of the Australian Census, the marketing focus on inviting everyone to respond on a particular “census day,” made the situation worse than might otherwise have been the case. Thus, in addition to having to combat the potential of a cyber attack, survey sponsors face the challenge of restoring confidence in the data collection system.
Intentional attacks on individual computers and devices or on specific surveys are probably the largest peril facing surveying that involves the Internet. They are also a major justification for developing multiple response mode opportunities, and not relying entirely on the Internet. The reliance of web-push methodologies on multiple modes of responding provides some measure of protection for attacks on a particular survey, just as it now provides an alternative for those who now consider an internet response unacceptable. In especially large surveys, such as countrywide censuses, shifting away from asking everyone to respond on the same day may also lessen exposure as well as the impact of some of the potential Internet problems.
It is difficult to anticipate whether technological and social control advancements will negate risks associated with computer use. For now, this is an issue that threatens successfully surveying over the Internet that cannot be ignored.
6.2 Smartphones and the purse/pocket problem
A second, but quite distinct issue now challenging Internet data collection is the use of multiple devices for responding. Increasingly, people carry a computer device mainly smartphones with them. In most respects this is a very positive development. Because people carry the capability for providing a survey response with them throughout the day, survey requests can be responded to almost anytime from anywhere. This constant availability also brings to the fore what can be described as the pocket/purse problem. There are size preferences and probably limitations on the devices most people are willing to carry with them for use in cars, on public transportation, while working and when recreating.
Recent research has shown that while increasing portions of the population will respond to web requests on their smartphone, the small screen sizes present significant problems. Considerable research summarized elsewhere (Dillman, Hao and Millar 2016) has revealed that the proportion of smartphone responses has increased. In addition, it is difficult to ask many types of questions that seemed to work well in other survey modes. For example, Sarraf, Brooks, Cole and Wang (2015) have shown that the common question format of the item-on-the left with answer categories horizontally displayed to the right and the four-point scale placed below it, resulted in early abandonment of the response process and a dramatic increase in missing responses. In a later set of experiments, Barlas and Thomas (2016) have demonstrated the benefit of shortening scalar questions. These works bring into question the advisability of asking seven point fully labeled scales, often favored in the past as ideal for interview surveys. Work by Stern, Sterrett and Bilgen (2016) suggest that grids in which a general question establishing a set of response categories is followed by lists of items requiring an answer to each, a staple of paper and web questionnaires, are not an acceptable visual layout for smartphones.
An excellent review of the available research by Couper et al. (2017) concludes that questionnaires completed on mobile phones have lower response rates, higher breakoff rates, and longer completion times than do web surveys on personal computers. The authors note that part of the reason for these persistent problems may be that surveyors have not yet succeeded in optimizing design for mobile phones. Another factor that contributes to these problems may be competing demands for attention from smartphones and other activities as people are going about the daily rounds of life.
One of the challenges associated with designing for smartphones is maintaining unified question construction across all survey modes. This problem could be particularly acute when respondents in well-established surveys find that previously used question structures, wordings, and visual layouts are unilaterally changed for smartphone use. This challenge has been pointed out by Mistichelli, Eanes and Horwitz of the U.S. Census Bureau (2015). It’s not yet clear whether survey designers are willing to change long standing ways of asking questions, (e.g., attitude questions with fewer categories and asking items-in-a series as individual items rather than a list of items introduced with a question that applies to the entire group of items being rated). If unified mode construction is to be used on smartphones, the needs of such devices are likely to be the major determinant of how items are presented across all modes.
The challenge now facing surveyors with regard to smartphones and mobiles also goes much deeper than how to present questions effectively in less space without the need for horizontal and vertical scrolling. In the early days of surveying, in-person interviewers could by their presence to engage the respondent’s full attention. With mail, desktop, laptop, and tablets, it might be expected that respondents would often, if not normally, complete surveys at times they were not likely to be interrupted. Smartphones, by their nature are interruption devices, with the possibility of receiving texts, voice phone calls, emails at any moment, often while moving physically through one’s daily activities. Answering some surveys may require consultation of records one does not have access to when away from home, or consultation with another household member, which seems harder to achieve if one tries to complete a questionnaire while on the move. The competition for attention that occurs with such devices might lead a surveyor to encourage a respondent not to fill out the survey on a smartphone and instead ask them to do it on their laptop or from home. The problem with that approach is for significant numbers of people smartphones may be their only computer or the only one they attend to on a daily basis. Also, it seems likely that the more one introduces barriers to answering a questionnaire “now”, the less likely people are to answer at all.
Working through these issues is one of the largest challenges facing survey methodologists today. But, on a positive note, when multiple modes of contact are used and multiple ways of responding are offered, it seems easier to guide respondents to the most effective way for them to respond as well as for the success of the survey.
6.3 Sponsor Reluctance to undertake mixed-mode surveys and modify single-mode procedures
An additional peril facing web-push surveys is associated with the stress many organizations face in using multiple modes of survey contact and/or response. Each mode of contact and response requires specialized skills, equipment, and software. In order to be effective, it must also be effectively coordinated to deal with many issues at once, as described elsewhere (Dillman et al. 2014, Chapter 11).
Survey sponsors that have specialized in only one form of data collection, or who want to keep data collection activities simple, may be tempted to avoid the use of second or third modes of data collection. This is not likely to happen when high response rates are required (e.g., a national census) or when a substantial economic incentive exists for pushing early respondents to the web (e.g., Biemer et al. 2015). However, the development of do-it-yourself software has encouraged many surveyors to find ways of using only web data collection. Previous research has suggested that significantly biased results toward greater education and income will be produced if data collection stops with only web responses (Rookey, Hanway, and Dillman 2008; Messer and Dillman 2011). Over time this bias may be reduced, but appears not to have happened yet for general populations. Another source of low response rates and potential bias occurs when surveyors obtain only email addresses for a proposed survey, thus eliminating the potential for prior mail contact that allows inclusion of an incentive for encouraging respondents to respond over the Internet.
Making appropriate changes, even when the need is substantial, takes time. In the 1990’s the U.S. Census Bureau developed a pre-notice, paper questionnaire, follow-up postcard strategy for data collection (Dillman, Clark and Sinclair 1995), which was used in the 2000 and 2010 Censuses. After the ACS push-to-web strategy was introduced in 2013, it continued to use this approach. The problem it presented is that the follow-up postcard could not provide password information (visible to anyone who picked up the postcard), thus creating the expectation that they would need to return to the web-request letter for that information. Also, the impression of the sequence of sending a pre-notice informing people they would receive a request to respond (by internet), a second letter asking them to go to the web using the provided information, and then a postcard reminder to follow through seemed unnecessarily laborious. Thus, a new procedure of abandoning the pre-notice and using a letter follow-up was introduced. A test of this procedure by the Census Bureau led to its adoption in August 2015 (Clark and Roberts 2016) and a significant increase of 2.5 percentage points in internet responses and a slight reduction in overall costs.
There are many other issues involved in shifting from single-mode thinking to widespread adoption of web-push surveys that involve multiple modes of response. For example, how do researchers overcome the frustration of willing respondents, who are irritated by being told they will have to wait for that request to come in a few weeks? Also, when telephone numbers are available, a phone call could be used as a follow-up reminder with encouragement, rather than simply trying to interview people over the phone. Experimental testing of these alternatives needs to be done.
6.4 Impacts of new discoveries and innovations
Anticipating the future is difficult. When telephone interviewing was rising towards prominence in the 1970’s, personal computers were not yet available. And, virtually no one thought, or even imagined, that only two decades later the telephone that had been tethered to our homes and workplaces would be carried with us nearly everywhere we went using wireless connections. When Internet surveying began in the 1990’s few anticipated that not only would the large and clunky desktops that began occupying people’s homes would transition to laptops that people would carry with them from place to place. And, that device would later transition to tablets and smartphones with touch screens both with far more computing power than their original desktops and laptops.
A recent analysis by Friedman (2016) details the monumental changes in the capabilities and power of personal devices that are increasingly taken for granted by large portions of the worldwide population. He traces these capabilities to the exponential growth of each of five different components of today’s computers: 1) integrated circuits that do the computing, 2) memory units that store and retrieve information, 3) networking systems that provide communications within and across computers, 4) the software applications that enable different computers to perform various tasks individually and together, and 5) sensors that detect movement, language, light, sound and other features of the environment and turn it into digitized data. He traces the rapid acceleration in these aspects to the development of the iPhone and related innovations occurring since 2007, and their melding into what he describes as the supernova (or cloud).
These developments were only dimly anticipated, even by many of the innovators who created them. Trying to imagine the future is no easier now than it was in the past. For example, voice activation of computer searches is rapidly replacing the individual tapping and swiping of commands on smartphones. Twenty percent of Google searches on Android-powered handsets in the United States are now input by voice (The Economist 2017). In addition, people can also dictate emails and text messages with reasonable success. Will voice-activated answers be the next wave of development for survey designers? It is easy to imagine one being interviewed by his or her smartphone. And, is it possible that simultaneous translations from one language to another, which can now be done with reasonable success, become common on surveys? But, herein lies a fundamental challenge, described by Friedman the speed with which human beings and societies can adapt to those changes.
Many potential respondents of interest to surveyors still rely on feature phones, while others are racing madly to adopt the most advanced computing and communication device they find practical. And still others are reluctant to use any computer at all. The differences in people’s capabilities and preferences require surveyors to be neither too far ahead nor too far behind where most people are.
This raises the issue of whether web-push methods are simply another transitional phase of survey design that may fade out as quickly as it has risen in prominence. The mixed-mode and tailored design focus that now appears to dominate the thinking of survey designers is recognition of the heterogeneity that exists among populations, whose opinions and behaviors surveyors seek to describe.
For a time it appeared that some surveyors thought the value of mixed-mode surveying was in offering people a choice of which mode they would use to respond to a survey request. However, this is only partly true. The real response power of mixed-mode designs for improving response rates stems from making multiple contacts effectively. Each contact gives an opportunity to provide new information about one’s survey request and, in some cases, to reach people who cannot be contacted by other survey modes. When survey response requests are offered by different modes there is often an opportunity to improve coverage (reaching people who can’t be reached by another mode) and get people to attend to persuasive arguments for being a respondent. Also, the sequencing of those contacts may help with motivating people to respond (e.g., use of email augmentation of postal letters that makes it easier to respond).
- Date modified: