Coverage and frames

Scope and purpose

The target population is the set of elements about which information is wanted and estimates are required. Practical considerations may dictate that some units be deliberately excluded (e.g., institutionalized individuals, the homeless, or those that are not be possible to access without incurring excessive cost). This gives arise to the concept of the survey population, the set of units that the practical constraints force us to narrow down to, by excluding some units that are hard or expensive to access to. Differences between the target population and the survey population are the result of deliberate restrictions to coverage. If the two differ, valid statistical inference under probability sampling can be made about the survey population, not about the target population itself.

A frame is any list, material or device that delimits, identifies, and allows access to the elements of the survey population. Frames are generally of two types: area frames and list frames. Area frames are usually made up of a hierarchy of geographical units, that is, the frame units at one level can be subdivided to form the units at the next level. All the elements included in the frame constitute the frame population. The discrepancies between the survey population and the frame population are referred to as coverage errors.

The survey frame should conform to the survey population and contain minimal undercoverage and overcoverage. Frame information must be kept up-to-date. Coverage errors occur due to omissions, erroneous inclusions, duplications and/or misclassifications of the units in the survey frame.

Characteristics of the frame units (e.g., identification, contact, classification, address, size, maps in case of geographical units) should be of high quality because of their use in stratification, sample selection, collection, follow-up, data processing, imputation, estimation, record linkage, quality assessment and analysis. Frame imperfections such as coverage errors and out-of-date characteristics are likely to bias or diminish the reliability of the survey estimates and to increase data collection costs.

• Test possible frames at the planning stage of a survey for their suitability and quality.
  
  
• When several frames exist, some of which are incomplete but less expensive to use and others more complete but prohibitively expensive, consider the use of multiple frames (Hartley, 1962; Bankier, 1986; Sirken and Casady, 1988; Kott and Vogel, 1995). Random Digit Dialling (RDD) may also be used for some telephone surveys, by itself or in combination with other area or list frames.
  
  
• At Statistics Canada, for business and institutional surveys, the Business Register is the usual frame. For agricultural surveys, the Farm Register is the usual frame. For household surveys, the Address Register, the Labour Force Survey frame (which is an area frame), telephone files and the Census of Population geographic units are options to consider.
  
  
• Where possible, use the same frame for surveys with the same target population, to avoid inconsistencies, to facilitate combining estimates from the surveys and to reduce costs of frame maintenance and evaluation.
  
  
• Retain and store information about sampling, rotation and data collection so that coordination between surveys can be achieved and respondent relations and response burden can be better managed. For example, record how often each unit is selected by each survey that is using the frame.
  
  
• To improve and/or maintain the level of quality of the frame, incorporate procedures to eliminate duplication and to update for births, deaths, out-of-scope units and changes in characteristics.
  
  
• Monitor the frame quality by periodically assessing its coverage and the quality of the information on the characteristics of the units. Many techniques exist for this purpose:
  
  
  • matching the frame or a sample of the frame with comparable alternative sources, often provided by administrative records, for the survey population or subsets of it;
  • analyzing survey returns for duplicates, deaths, out-of-scope units, and changes in characteristics;
  • using specific questions on the questionnaire to aid in monitoring coverage and classification information; verifying with local authorities (e.g., regional offices, field survey staff, the survey units themselves);
  • verifying the frame or subsets of it in the field (which could include verification of out-of-scope units);
  • comparing the frame with a sample of units from a corresponding area frame;
  • updating the frame to determine changes to it;
  • checking the consistency of counts with other sources or with data from specially designed replicates;
  • using evaluative information obtained from other surveys with the same frame (Lessler and Kalsbeek, 1992).
    
    
• Monitor the frame between the time of sample selection and the survey reference period.
  
  
• Incorporate frame updates in the timeliest manner possible.
  
  
• Minimize frame errors through effective training of staff, an emphasis on the importance of coverage, and the implementation of quality assurance procedures of frame-related activities.
  
  
• For area frames, implement map checks to ensure clear and non-overlapping delineation of the geographic areas used in the sampling design (e.g., through field checks or the use of other map sources). When appropriate, use the Generalized Area Delineation System (GArDS) for creating the geographic frame units.
  
  
• For statistical activities from administrative sources or for derived statistical activities, where coverage changes may be outside the control of the immediate manager, determine and monitor coverage, and negotiate required changes with the source manager.
  
  
• Make adjustments to the data or use supplementary data from other sources to offset coverage error of the frame.
  
  
• Include descriptions of the target and survey populations, any differences between the target population and the survey population, as well as the description of the frame and its coverage errors in the survey documentation.

Top of Page

Archer, D. (1995). Maintenance of business registers. In Business Survey Methods, B.G. Cox et al (eds.), Wiley-Interscience, New York, 85-100.

Bankier, M.D. (1986). Estimators based on several stratified samples with applications to multiple frame surveys. Journal of the American Statistical Association, 81, 1074-1079.

Burgess, R.D. (1988). Evaluation of reverse record check estimates of undercoverage in the Canadian Census of Population. Survey Methodology, 14, 137-156.

Colledge, M.J. (1995). Frames and business registers: an overview. In Business Survey Methods, B.G. Cox et al (eds.), Wiley-Interscience, New York, 21-47.

Gambino, J.G., Singh, M.P., Dufour, J., Kennedy, B. and Lindeyer, J. (1998). Methodology of the Canadian Labour Force Survey. Statistics Canada Catalogue no. 71-526.

Gosselin, J.-F., Chinnappa, B.N., Ghangurde, P.D. and Tourigny, J. (1978). Coverage. Chapter 2 in A Compendium of Methods of Error Evaluation in Censuses and Surveys, Statistics Canada Catalogue no 13-564E, 7-9.

Hartley, H.O. (1962). Multiple frame surveys. Proceedings of the Social Statistics Section, American Statistical Association, 203-206.

Kott, P.S. and Vogel, F.A. (1995). Multiple-frame business surveys. In Business Survey Methods, B.G. Cox et al (eds.), Wiley-Interscience, New York, 185-203.

Laniel, N. and Finlay, H. (1991). Data quality concerns with sub-annual business survey frames. Proceedings of the Section on Survey Research Methods, American Statistical Association, 202-207.

Lessler, J.T. and Kalsbeek, W.D. (1992). Nonsampling Errors in Surveys. Wiley, New York.

Massey, J.T. (1988). An overview of telephone coverage. In Telephone Survey Methodology, R.M. Groves et al (eds.), Wiley, New York, 3-8.

Pollock, K.H., Turner, S.C. and Brown, C.A. (1994). Use of capture-recapture techniques to estimate population size and population totals when a complete frame is unavailable. Survey Methodology, 20, 117-124.

Satin, A. and Shastry, W. (1993). Elements of a sample plan. Chapter 2 in Survey Sampling: A Non-Mathematical Guide - Second Edition, Statistics Canada Catalogue no 12-602E, 7-10.

Sirken, M.G. and Casady, R.J. (1988). Sampling variance and nonresponse rates in dual frame, mixed mode surveys. Chapter 11 in Telephone Survey Methodology, P. Biemer, R.M. Groves, L. Lyberg, J. Massey, W. Nicholls and J. Waksberg (eds.), Wiley, New York, 175-188.

Swain, L., Drew, J.D., Lafrance, B and Lance, K. (1992). The creation of a residential address register for coverage improvement in the 1991 Canadian Census. Survey Methodology, 18, 127-141.