4. What can be done?
Constance F. Citro
Survey researchers have not been idle in the face of multiple and increasing threats to the survey paradigm. For at least the last 15 years, they have actively worked on ways to reduce or compensate for coverage error, unit and item nonresponse, measurement error, and, more recently, burden on respondents. Strategies have included: (1) spending more on case completion (although budget constraints limit the viability of this strategy); (2) using paradata and auxiliary information for more effective unit nonresponse bias identification and adjustment; (3) employing more sophisticated missing data adjustments that do not assume MAR; (4) using adaptive design methods to optimize the cost and quality of response; (5) using multiple frames to reduce coverage error (e.g. cell-phone and land-line frames for telephone surveys); (6) using multiple modes to facilitate more cost-effective response as in the ACS, which recently added an Internet response option to its mail, CATI and CAPI options; (7) reducing burden by optimizing follow-up calls and visits; and (8) describing the needs for the survey data. In the United States, data users are often recruited to make the case to Congress and other stakeholders. For example, the Association of Public Data Users, the Council of Professional Associations on Federal Statistics and the Population Association of America frequently mobilize data users on behalf of statistical agency programs.
My thesis is that these steps, while laudable and necessary, are not sufficient to restore the probability survey-based paradigm for official statistics on households or other types of respondents. I propose, instead, that statistical agencies consistently begin by determining policymakers’ and public needs and work backwards to identify appropriate data sources to serve those needs in the most cost-effective and least burdensome manner possible. This multiple sources paradigm should apply to all statistical programs, whether traditionally based on a survey, administrative records, or another source.
Some important statistical programs, such as the NIPAs and the Consumer Price Index (see Horrigan 2013) in the United States and other countries, have for decades used multiple data sources. One reason is that these programs are built around a widely accepted conceptual framework that determines required elements to constitute an acceptable set of estimates. It is not acceptable to omit one or more components of income from the NIPAs simply because data are not available from a single source. Moreover, because key NIPA estimates are periodically revised to add data, improve methodology and refine concepts, there is a built-in positive bias to search for new and improved data sources to fill gaps and improve accuracy. The U.S. economic censuses also use multiple sources, specifically, income tax records for sole proprietors and very small employers together with surveys for larger companies. U.S. household statistics programs, in contrast, have most closely adhered to the probability sample survey paradigm. Moreover, because long intervals typically occur between revisions to household survey concepts and design, the surveys too often fall behind in their ability to serve policymakers and the public, when the use of additional data sources could make possible significant improvements.
- Date modified: