The promise and challenge of pushing respondents to the Web in mixed-mode surveys
Section 3. Overcoming barriers to the acceptance of mixed-mode designs
3.1 Historical barriers to mixing modes
Use of more than one survey mode, as a means of contacting and/or asking questions, was seldom done in the late 20th century. Gaining acceptance of mixed-mode designs for any purpose has been a slow process. The biggest barrier prior to the 1990’s was simply the lack of perceived benefit. Response rates to in-person, telephone and mail surveys were considered high enough that use of a second or third mode was considered unnecessary. A significant exception was those surveys in which less expensive survey methods were used early in the data collection process, but in-person methods were necessary to achieve response rates over 90%. The U.S. Decennial Census from 1970-1990, which followed a mail questionnaire start with in-person and in some cases telephone follow-up is an example.
Another barrier to the mixing of survey modes was that the data collection technology of the times made it difficult to simultaneously implement multiple modes of data collection in a single survey. The lack of networked computers and software meant that using a second mode of data collection required finishing up data collection for one mode before switching the effort to a separate data collection unit for implementation of a second mode (Dillman, Smyth and Christian 2009, Chapter 8). An earlier review of the use of telephone in mixed-mode surveys in the late 1980’s found that few mixed-mode surveys had been done, other than pre-letters to an anticipated telephone or in-person interview (Dillman and Tarnai 1988).
During the 1990’s it became apparent that new methods of surveying needed to be developed. Response rates, especially for personal interviewing and telephone, had begun to decline (Brick and Williams 2013). Coverage problems were also increasing, as locked multi-unit buildings and gated-residential communities made it impossible to reach many households in-person. The landline coverage of households also began its long inexorable decline that now leaves about half of U.S. households without such connections.
Interest in coordinating the use of multiple modes in some way to improve response rates brought attention to interview concerns that had previously been ignored because of the practical barriers to mixing modes. For example, interviewed respondents often gave socially desirable responses so that estimates of desirable behaviors of e.g., “having voted in the last election” were higher than the actual behavior. In addition, estimates of undesirable behaviors, e.g., smoking marijuana or having sex outside of marriage were lower (de Leeuw 1992). Differences were also being observed between mail answers to survey questions on the one hand vs. telephone and in-person surveys where respondents gave more extreme positive answers on opinion questions (de Leeuw 1992; Tarnai and Dillman 1992). Research had also suggested that respondents were more likely to choose the first response categories in mail surveys, a primacy effect, and the last categories in telephone surveys, described as a recency effect (Krosnick and Alwin 1987). As a consequence, it became increasingly difficult for survey sponsors to argue that telephone, or even in-person interviews, were superior survey modes.
Mixed-mode surveys were proposed as a potential, albeit imperfect, solution to the problems of individual survey modes. Five types of mixed-mode surveys were identified, ranging from collecting the same data from different members of a sample to using one mode only to prompt completion by another mode (Dillman 2000, page 219). The major advantage of combining modes appeared to be improvements that could be achieved in coverage and response rates. The major difficulty identified was the prospect of measurement differences when different response modes were used.
A critical article by de Leeuw (2005) influenced an important transition in thinking about mixed-mode surveys. She articulated a variety of accepted possibilities for combining survey modes and provided evidence that an increase in use of mixed mode surveys was occurring. She also noted the transition that occurred from debate on which survey mode was best for a particular study, to how modes could be used together and produce better results.
A contextual change was also underway as modern societies throughout the world began shifting from person-mediated activities (e.g., getting money from tellers in banks, making travel reservations through an agent, and purchasing goods in stores and from catalogues) to self-administration (Dillman 2000). But researchers had not yet answered the question of whether interviews by telephone could persist in the face of these self-administration trends.
3.2 Institutional barriers to the joint use of survey modes
Considerable reluctance, if not outright opposition, existed with regard to mixing survey modes and especially to giving up cherished ways of asking questions differently in different survey modes (Dillman 2000). One of the consequences of the emergence of new ways of collecting survey data in the last third of the 20th century is that data collection staff became quite specialized. Some organizations conducted surveys by only one mode. It was common for some data collection staff and their organizations to do only telephone surveys, and to a lesser extent mail surveys. A few large firms had in-person sampling and data collection units. A tendency existed to want to do surveys by the mode that a group knew best. This tendency was exacerbated in the late 90’s as Internet-only survey organizations began to emerge.
In addition different styles of wording questions by individual modes had emerged. Interviews tended to withhold “don’t know” categories, offering them only when respondents would object. Designers of paper and web surveys often used “mark all that apply” question formats to make responding easier, but the awkwardness of that format on telephone led to using only forced choice formats that obtained an answer after each individual item was presented on the telephone. The issue facing survey designers was whether to maximize question formats for the mode, or try to keep the same stimulus across all modes (Dillman and Christian 2005).
One of the factors supporting this tendency to stick with what surveyors knew best was the recognition that single mode surveys were best for many survey situations. That situation existed for each of the modes. In-person interviews were the only way of obtaining adequate coverage for certain national surveys such as the Current Population Survey that produces employment rate estimates. RDD telephone surveys were the best means of conducting election and other cross-sectional household surveys. Mail was the most adequate means when one was conducting regional and local surveys for which only residential addresses were available. Interactive Voice Response surveys were most practical for many customer satisfaction surveys when people contacted calling centers to obtain a particular service. And Internet surveys quickly became the go-to methodology for client surveys and other situations for which email addresses had been previously collected.
This situation marked the development of interest in “tailored design,” i.e., recognition that different modes of data collection fit better with surveys of particular populations, survey topics, and data collection situations. This trend in survey design now persists more powerfully than it did at the turn of the century. It is clear that picking the single best survey mode for a particular survey is increasingly inadequate because of negative effects on coverage, response rates and nonresponse error.
As the 20th century came to an end, there was much uncertainty with regard to where data collection methods might be headed. Prospects seemed dim for continuing sole reliance on either in-person or telephone surveys. The coverage challenges and costs were growing significantly, and it seemed unlikely that response rates were likely to improve for voice telephone surveys. Great interest existed in replacing interview methods with the Internet, but at the turn of the century only about half of the households in the U.S. as yet had computers, and even fewer had access to the internet (Dillman 2000).
- Date modified: