Filter results by
Search HelpKeyword(s)
Subject
Author(s)
Results
All (3)
All (3) ((3 results))
- Articles and reports: 12-001-X202200100005Description:
Methodological studies of the effects that human interviewers have on the quality of survey data have long been limited by a critical assumption: that interviewers in a given survey are assigned random subsets of the larger overall sample (also known as interpenetrated assignment). Absent this type of study design, estimates of interviewer effects on survey measures of interest may reflect differences between interviewers in the characteristics of their assigned sample members, rather than recruitment or measurement effects specifically introduced by the interviewers. Previous attempts to approximate interpenetrated assignment have typically used regression models to condition on factors that might be related to interviewer assignment. We introduce a new approach for overcoming this lack of interpenetrated assignment when estimating interviewer effects. This approach, which we refer to as the “anchoring” method, leverages correlations between observed variables that are unlikely to be affected by interviewers (“anchors”) and variables that may be prone to interviewer effects to remove components of within-interviewer correlations that lack of interpenetrated assignment may introduce. We consider both frequentist and Bayesian approaches, where the latter can make use of information about interviewer effect variances in previous waves of a study, if available. We evaluate this new methodology empirically using a simulation study, and then illustrate its application using real survey data from the Behavioral Risk Factor Surveillance System (BRFSS), where interviewer IDs are provided on public-use data files. While our proposed method shares some of the limitations of the traditional approach – namely the need for variables associated with the outcome of interest that are also free of measurement error – it avoids the need for conditional inference and thus has improved inferential qualities when the focus is on marginal estimates, and it shows evidence of further reducing overestimation of larger interviewer effects relative to the traditional approach.
Release date: 2022-06-21 - Articles and reports: 11-522-X201700014744Description:
This presentation will begin with Dr. West providing a summary of research that has been conducted on the quality and utility of paradata collected as part of the United States National Survey of Family Growth (NSFG). The NSFG is the major national fertility survey in the U.S., and an important source of data on sexual activity, sexual behavior, and reproductive health for policy makers. For many years, the NSFG has been collecting various forms of paradata, including keystroke information (e.g., Couper and Kreuter 2013), call record information, detailed case disposition information, and interviewer observations related to key NSFG measures (e.g., West 2013). Dr. West will discuss some of the challenges of working with these data, in addition to evidence of their utility for nonresponse adjustment, interviewer evaluation, and/or responsive survey design purposes. Dr. Kreuter will then present research done using paradata collected as part of two panel surveys: the Medical Expenditure Panel Survey (MEPS) in the United States, and the Panel Labour Market and Social Security (PASS) in Germany. In both surveys, information from contacts in prior waves were experimentally used to improve contact and response rates in subsequent waves. In addition, research from PASS will be presented where interviewer observations on key outcome variables were collected to be used in nonresponse adjustment or responsive survey design decisions. Dr. Kreuter will not only present the research results but also the practical challenges in implementing the collection and use of both sets of paradata.
Release date: 2016-03-24 - Articles and reports: 12-001-X201400214092Description:
Survey methodologists have long studied the effects of interviewers on the variance of survey estimates. Statistical models including random interviewer effects are often fitted in such investigations, and research interest lies in the magnitude of the interviewer variance component. One question that might arise in a methodological investigation is whether or not different groups of interviewers (e.g., those with prior experience on a given survey vs. new hires, or CAPI interviewers vs. CATI interviewers) have significantly different variance components in these models. Significant differences may indicate a need for additional training in particular subgroups, or sub-optimal properties of different modes or interviewing styles for particular survey items (in terms of the overall mean squared error of survey estimates). Survey researchers seeking answers to these types of questions have different statistical tools available to them. This paper aims to provide an overview of alternative frequentist and Bayesian approaches to the comparison of variance components in different groups of survey interviewers, using a hierarchical generalized linear modeling framework that accommodates a variety of different types of survey variables. We first consider the benefits and limitations of each approach, contrasting the methods used for estimation and inference. We next present a simulation study, empirically evaluating the ability of each approach to efficiently estimate differences in variance components. We then apply the two approaches to an analysis of real survey data collected in the U.S. National Survey of Family Growth (NSFG). We conclude that the two approaches tend to result in very similar inferences, and we provide suggestions for practice given some of the subtle differences observed.
Release date: 2014-12-19
Stats in brief (0)
Stats in brief (0) (0 results)
No content available at this time.
Articles and reports (3)
Articles and reports (3) ((3 results))
- Articles and reports: 12-001-X202200100005Description:
Methodological studies of the effects that human interviewers have on the quality of survey data have long been limited by a critical assumption: that interviewers in a given survey are assigned random subsets of the larger overall sample (also known as interpenetrated assignment). Absent this type of study design, estimates of interviewer effects on survey measures of interest may reflect differences between interviewers in the characteristics of their assigned sample members, rather than recruitment or measurement effects specifically introduced by the interviewers. Previous attempts to approximate interpenetrated assignment have typically used regression models to condition on factors that might be related to interviewer assignment. We introduce a new approach for overcoming this lack of interpenetrated assignment when estimating interviewer effects. This approach, which we refer to as the “anchoring” method, leverages correlations between observed variables that are unlikely to be affected by interviewers (“anchors”) and variables that may be prone to interviewer effects to remove components of within-interviewer correlations that lack of interpenetrated assignment may introduce. We consider both frequentist and Bayesian approaches, where the latter can make use of information about interviewer effect variances in previous waves of a study, if available. We evaluate this new methodology empirically using a simulation study, and then illustrate its application using real survey data from the Behavioral Risk Factor Surveillance System (BRFSS), where interviewer IDs are provided on public-use data files. While our proposed method shares some of the limitations of the traditional approach – namely the need for variables associated with the outcome of interest that are also free of measurement error – it avoids the need for conditional inference and thus has improved inferential qualities when the focus is on marginal estimates, and it shows evidence of further reducing overestimation of larger interviewer effects relative to the traditional approach.
Release date: 2022-06-21 - Articles and reports: 11-522-X201700014744Description:
This presentation will begin with Dr. West providing a summary of research that has been conducted on the quality and utility of paradata collected as part of the United States National Survey of Family Growth (NSFG). The NSFG is the major national fertility survey in the U.S., and an important source of data on sexual activity, sexual behavior, and reproductive health for policy makers. For many years, the NSFG has been collecting various forms of paradata, including keystroke information (e.g., Couper and Kreuter 2013), call record information, detailed case disposition information, and interviewer observations related to key NSFG measures (e.g., West 2013). Dr. West will discuss some of the challenges of working with these data, in addition to evidence of their utility for nonresponse adjustment, interviewer evaluation, and/or responsive survey design purposes. Dr. Kreuter will then present research done using paradata collected as part of two panel surveys: the Medical Expenditure Panel Survey (MEPS) in the United States, and the Panel Labour Market and Social Security (PASS) in Germany. In both surveys, information from contacts in prior waves were experimentally used to improve contact and response rates in subsequent waves. In addition, research from PASS will be presented where interviewer observations on key outcome variables were collected to be used in nonresponse adjustment or responsive survey design decisions. Dr. Kreuter will not only present the research results but also the practical challenges in implementing the collection and use of both sets of paradata.
Release date: 2016-03-24 - Articles and reports: 12-001-X201400214092Description:
Survey methodologists have long studied the effects of interviewers on the variance of survey estimates. Statistical models including random interviewer effects are often fitted in such investigations, and research interest lies in the magnitude of the interviewer variance component. One question that might arise in a methodological investigation is whether or not different groups of interviewers (e.g., those with prior experience on a given survey vs. new hires, or CAPI interviewers vs. CATI interviewers) have significantly different variance components in these models. Significant differences may indicate a need for additional training in particular subgroups, or sub-optimal properties of different modes or interviewing styles for particular survey items (in terms of the overall mean squared error of survey estimates). Survey researchers seeking answers to these types of questions have different statistical tools available to them. This paper aims to provide an overview of alternative frequentist and Bayesian approaches to the comparison of variance components in different groups of survey interviewers, using a hierarchical generalized linear modeling framework that accommodates a variety of different types of survey variables. We first consider the benefits and limitations of each approach, contrasting the methods used for estimation and inference. We next present a simulation study, empirically evaluating the ability of each approach to efficiently estimate differences in variance components. We then apply the two approaches to an analysis of real survey data collected in the U.S. National Survey of Family Growth (NSFG). We conclude that the two approaches tend to result in very similar inferences, and we provide suggestions for practice given some of the subtle differences observed.
Release date: 2014-12-19
Journals and periodicals (0)
Journals and periodicals (0) (0 results)
No content available at this time.
- Date modified: