Chapter 3.3: Enhancing how surveys are conducted

Context

The mandate of a National Statistical Office (NSO) is to provide quality statistical information on, and analysis of the country's economy and society. While the use of administrative data is widely recognized as a cost-effective option to obtain quality information while minimizing response burden when appropriate (see Chapter 3.5: Acquisition, use and management of administrative data), NSOs still need to conduct an important number of surveys to provide information not available through administrative records.

In the last decades, NSOs have been facing considerable challenges, such as budget pressures, declining response rate, increased collection costs and evolving technology: factors that a statistical agency must consider to continually improve and modernize the way surveys are conducted.

In the context of improving and modernizing the corporate business architecture (see Chapter 3.1: Corporate Business Architecture) or Information Technology (see Chapter 3.2: Modernization of information technology and informatics services), NSOs also have to consider strategies and tools, and must implement those that will help enhance the way surveys are conducted.

As reflected in its organizational structure, Statistics Canada conducts surveys providing two types of outcomes:

  1. Economic statistics, which are the results of business surveys, for which respondents are businesses. The majority of these surveys are mandatory – see Chapter 4.5: Relations with survey respondents.
  2. Social statistics result from social surveys, which are, for the most part, not mandatory and intended for households. Currently, only the Census and the Labour Force Survey are mandatory survey.

It is crucial to mention that while considering enhancing the way surveys are conducted, NSOs need to adopt a visionary approach to that challenge. That means improving and enhancing the entire statistical infrastructure: NSOs need to avoid being "reactive" to specific data needs and thus allocate funds to conduct surveys only to respond to those needs with a short-term approach; but rather NSOs should use each new survey initiative to continually invest and build on their statistical infrastructure to make it more responsive to all data needs. By doing so, they would be able to conduct any economic or social survey with efficiency and quality.

While business and social surveys share the same objectives of producing quality data while keeping to a minimum response burden, using common tools, ensuring integration and harmonization in each step of the statistical process, different requirements in terms of frames, collection, processing, estimation and dissemination lead to the development of different sets of tools, responding to needs typically associated with social vs business surveys.

This chapter aims to provide insights on the strategies and tools for modernization that were used to enhance how surveys are conducted— both for businesses and for households.

Strategies, Mechanisms and Tools

1. Business survey strategies and tools

Over the last few decades, NSOs have faced growing demand for better and more detailed business statistics, together with the need to make production more efficient, to improve existing statistics and to develop new statistics.

To gain in efficiency and responsiveness, several NSOs have opted to create Statistical Business Registers (SBR) as the backbone for the production of business statistics. This offers the potential benefit of integrating the SBR with other statistics, by combining it with information from other administrative or statistical registers. Using an SBR as a unique survey frame for all business surveys also means offering an opportunity to streamline the statistical production process.Endnote 1

Indeed, some organizations have also moved towards an integrated business statistics program, allowing them to be more efficient and consistent in the way they collect and process business statistics. This section will focus on Statistics Canada's experience in moving its economic statistics surveys into an integrated business survey program.

1.1 Overall Business Surveys Strategy – Integrated Business Survey Program (IBSP)

In 2010, Statistics Canada launched the Corporate Business Architecture (CBA) initiative. For more information, refer to Chapter 3.1: Corporate Business Architecture.

This work resulted in numerous recommendations, including the development and mandatory use of shared and generic corporate services for collecting, processing, storing and disseminating statistical information, including a major transformation project for its economic statistics surveys, the Integrated Business Statistics Program (IBSP).

The IBSP provides a standardized framework for the majority of the economic surveys conducted at Statistics Canada. IBSP surveys use Statistics Canada's Business Register as a common frame. Questionnaires are based on harmonized concepts and content, and surveys share common sampling, collection and processing methodologies that are driven by metadata (the metadata-driven model is explained in section 1.1.3 of this chapter). In addition, common tools are in place to collect, edit, correct, and analyze data.

Although the IBSP was an ambitious undertaking, this new program was developed through a continuation of efforts to build a harmonized business surveying approach, which began in the late 1990s with the Unified Enterprise Statistics (UES) program. The UES program originally covered seven pilot surveys, and gradually expanded to include sixty annual business surveys in the agriculture, manufacturing, trade and services sectors. Nevertheless, over time, substantial resources were required for systems maintenance, and the UES model could not easily adapt to changing requirements. As a result, the UES systems infrastructure became antiquated. This provided an opportune time to redesign the model and implement CBA principles.

Under IBSP, the business surveying infrastructure has been completely redeveloped, and innovative methodologies and processes have been introduced to improve upon the UES model. Many of these innovations resolve longstanding shortcomings of the UES, including implementing a system that has flexibility to adapt to new requirements. By 2019, the majority of Statistics Canada's economic surveys will be incorporated into the IBSP model.Endnote 2

1.1.1 Objectives

In designing the IBSP model, six core objectives were identified:

  • improving data quality by applying standardized methods and processes, implementing harmonized content, and facilitating coherence analysis;
  • reducing response burden;
  • modernizing data processing infrastructure with the creation of common IT systems and tools (refer to Chapter 3.2: Modernization of information technology and informatics services);
  • integrating the majority of economic surveys into the new model;
  • simplifying and standardizing processes to shorten learning curves and improve timeliness; and
  • reducing ongoing costs associated with operational aspects of surveys to realize efficiencies.
1.1.2 Guiding Principles

Common features of IBSP surveys that promote operational efficiency in each step of the survey process include the following:

  • A full use of the Business Register;
  • The use of electronic questionnaires as the principal mode of collection;
  • A metadata-driven approach to questionnaire development, sampling, edit and imputation, allocation and estimation processes;
  • The implementation of rolling estimates and an active collection management based on quality indicators and;
  • An increased use of administrative data to reduce response burden.

IBSP surveys share these common features, but there are many variants to accommodate survey-specific requirements. For example, the model is designed with flexibility to process surveys with different frequencies, including monthly, quarterly and annual surveys; and, with different coverage, such as economy-wide surveys, industry-based surveys and activity-based surveys.

1.1.3 Strategies and tools for business surveys
1.1.3.1 Full use of the Business Register

Statistics Canada uses its Business Register (BR) as the common frame for all business surveys. The BR is a database that is updated through a number of sources, which include administrative data files, feedback on Statistics Canada business surveys, and profiling activities that involve direct contact with companies to obtain information about their operations and Internet research findings. Using the BR ensures quality, while providing an integrated tool that can measure overlap and minimize response burden to the greatest extent possible. The development of the BR underwent a series of improvements over the years.

More than twenty-five years ago, the BR included only businesses with employees and, thus, covered only a subset of the Canadian economy. At that time, the employer's account data was the only administrative data source available that was reliable for identifying and maintaining businesses on the register. In the late 1980's, it was supplemented with Tax Records but with no integration between the two sources. In the late 1990's, Canada Revenue Agency (CRA) introduced the Business Number (BN) to help administer its various programs, and this greatly increased the potential use of administrative data.

The BN allowed Statistics Canada to link between multiple administrative data sources, which significantly enriched information on the BR frame for the economic survey program. During the past ten years, administrative data has become a key component of the Canadian BR.

The role of the BR is to provide statisticians, who produce and analyze economic statistics, with the highest quality frame in terms of coverage and data elements. In addition to having updates from administrative files, the BR is connected to the collection tools, and as such, the BR is updated daily with information coming from interviewers in the field.

The BR encompasses some of the fundamental concepts of the System of National Accounts, and provides the infrastructure to store, browse, maintain, and retrieve frame information. It also has the capacity for storage of the contact name, address and telephone number and the survey questionnaire identifiers. The Register can also generate an accurate list of contacts, which is required for the survey-data collection process. It monitors the level of response burden imposed on individual businesses by Statistics Canada, and provides relevant information to effectively manage response issues. Finally, it provides statistical information regarding the composition of the population of businesses in Canada in terms of organizational structure, industrial activity, size and geography.

In Canada, the BR contains approximately 6 million active businesses, 99% of which are single-unit businesses. There are about 35,000 businesses that have more than one operating entity (complex businesses), and they account for more than 45% of the Canadian economy. Different types of organizations are defined as a business such as: a corporation, a self-employed individual, a government entity, a non-profit organization, a partnership, or a financial fund.

In addition to being used as a frame for survey sampling, the BR also serves to

  • browse the business structures;
  • analyse the population of business structures;
  • update the business and operational information;
  • manage, monitor and control respondent burden;
  • contact the businesses (support data collection);
  • run longitudinal studies; and
  • disseminate data from Canadian Business Counts.

In terms of maintenance, for simple business structures, the Business Number (BN) Registration file provides administrative data (tax data) to the BR, which updates automatically. For complex businesses, face-to-face or telephone interviews and coherence analysis are used to complete a full profile maintenance. The integrated nature of the IBSP also allows automated updates of the BR using a harmonized survey feedback process that applies to all business surveys.

In terms of response burden, a control module for excessive response burden was implemented on the Business Register system in December 2014. The objectives of this module are to reduce the excessive response burden for small and medium enterprises, as well as calculate the potential accumulated response burden for all active enterprises in the Business Register system. This is calculated over a three-year period. Depending on the class size of the enterprise and the total number of hours spent completing questionnaires, Statistics Canada determines if an enterprise has exceeded its response-burden limit. If an enterprise is overburdened for its class size, it will be excluded from all surveys for a duration of one calendar year.

1.1.3.2 The use of electronic questionnaires as the principal mode of collection

To gain on efficiency, but also to respond to businesses' desire to interact electronically with the Government of Canada, the use of electronic questionnaire (EQ) is now the preferred option for business survey data collection. In 2015, 46% of the business surveys converted to EQ with some form of follow-up, by either mail, fax or computer-assisted telephone interviews.

In addition, Statistics Canada has begun to develop and implement an Integrated Collection and Operation Systems Initiative (ICOS) and within it a Business Collection Portal to support business survey programs. The objective is to develop an integrated collection systems environment to achieve the targeted level of flexibility between modes and sites, and to fully exploit the use of the Internet for e-questionnaires. For more information about ICOS, refer to Chapter 3.4: Data Collection Planning and Management.

1.1.3.3 A metadata-driven approach to questionnaire development, sampling, edit and imputation, allocation and estimation processes

Statistics Canada has a long history of developing corporate metadata repositories for managing publications, services, and statistical holdings. However, relatively few survey programs have well developed metadata repositories for managing survey operations. The UES program did implement a metadata system that housed processing edits, along with variable cell numbers and cell descriptions. For IBSP, this metadata framework has been expanded to cover all aspects of survey processing. This approach increases efficiency, robustness, and responsiveness in delivering processing services for IBSP programs.

In the IBSP model, metadata are stored in easily modifiable tables that are used to drive systems programs. This has been a shift from the UES model where metadata were often hard-coded into programs. IBSP systems programs simply access information from metadata tables to direct their execution.

A key advantage of the IBSP metadata-driven system is that changes are required as program needs evolve, and can be accommodated by modifying metadata, rather than by rewriting system code. This provides more control for the processing team and more flexibility for users.

According to metadata management guidelines, metadata should be active, created for a purpose, and used in "downstream" processes. While the "no data without metadata" principle is often applied to final data output, in the form of descriptive metadata, this is also true for processing, especially in the IBSP. When a variable is created, it is tagged with descriptive elements, such as a name and an origin. However, metadata will also indicate how validation, editing and imputation must be done, and will track the variable's passage through the various processing steps.

Users have a single point of entry into the IBSP, since metadata will be integrated into every processing step, and managing metadata, along with the processes they direct, will naturally form components of the same seamless portal. This integration enables the system to instantly check whether run conditions are being met. For example, if a user chooses to execute a process, the interface can prompt the user to input the necessary metadata and to ensure that other prerequisites are in place. Inputs can then be validated automatically, and, as a result, the system would either stop to give a warning or allow the user to proceed with subsequent steps. The metadata interface approach does not require the user to deal with multiple applications, and does not require knowledge about the order of the various steps necessary to run processing.

While the efficiency goal of a metadata-driven system is to minimize rework and facilitate re-use, improving quality and coherence is an equally important outcome. The integration of metadata in processing operations facilitates automatic coherence checks. Data integrity rules are enforced through the system's database to ensure the quality of inputs. Metadata also generates invaluable management information to aid in monitoring progress, thereby improving the overall quality of survey processing.

Building an integrated metadata-driven infrastructure also facilitates the implementation of harmonized conceptual framework. For IBSP, this began with the mainstreaming of statistical units, populations, concepts, variables, classifications and sets of questions. All IBSP surveys are now required to apply statistical standards, including

  • the North American Industry Classification System (NAICS) to classify the target population by industry
  • the North American Product Classification System (NAPCS) to categorize and collect business input and output data
  • the Chart of Accounts (COA) as the reference taxonomy for organizing business financial information. (e.g., revenue, expenses, assets and liabilities).

There are a number of financial variables that are common across many economic surveys. By harmonizing the definitions of these variables, and systematically applying standards, common content has been developed and implemented across programs.

The IBSP content model is based on a series of generic modules that cover common variables, and are applied to surveys without modifications from one survey to the next. This approach plays a critical role in creating coherence across programs and in minimizing the effort required to build, test and implement survey content.

In essence, the standardized modules are a series of business survey questions used to collect information to meet stakeholder requirements. There are standardized modules for income-statement data (revenues and expenses), sales data by type of client, sales data by client location, and by purchased service inputs.

The objective of using tax information to its full potential guided the development of questionnaire content. Specifically, IBSP revenue and expense variables have been mapped directly with information available from tax files. This direct link eliminates the need for collecting financial information from small and medium enterprises, since data for these can be easily accessed from administrative sources.

One key issue that had to be resolved in developing financial data content was how to ensure that the conceptual needs of the Canadian System of National Accounts would be met through the use of administrative data. The COA bridges the two sets of concepts. As part of developing the IBSP content model, the COA was reviewed and revised to ensure that COA variables, which are directly linked to tax concepts, meet the information requirements of national accountants.

To add flexibility and meet specific survey requirements, subject-matter experts can customize certain modules that appear on their IBSP questionnaires. For example, products appearing on manufacturing questionnaires will be different from those appearing on service-industry questionnaires. And some of the standardized modules might not be required because they are not relevant for the industry. In constructing IBSP survey questionnaires, staff need to simply select relevant standardized content modules, and then focus efforts on developing industry-specific content, where required. This greatly reduces the time needed to develop, implement and test new questionnaires, while improving the quality and coherence of the data collected.

1.1.3.4 The implementation of rolling estimates and active collection-management based on quality indicators

Another feature added to IBSP is its capacity to use historical and partially collected data to produce key estimates (called rolling estimates) and quality indicators, while collection is still underway. These quality indicators are then compared to previously set quality targets to determine if more effort is required or if active collection can be terminated. If collection needs to continue, item scores are calculated in order to gauge a unit's impact on the quality indicator of each key estimate. These scores are then aggregated within each unit to create a global unit score. These scores enable decisions regarding follow-up activities to be madeEndnote 3.

1.1.3.5 An increased use of administrative data to reduce response burden

Statistics Canada's rich history of using administrative data for business statistics has helped to reduce response burden. In fact, under the UES business survey model, tax data have already been used as a direct substitute for a sub-sample of sampled units and for imputation of non-response records.

Over time, tax data imputation methods have improved through the use of administrative data, and the quality of information from surveys has improved. This has led to an even greater reliance on tax data as a primary information source. In the transition to the IBSP model, methods were adapted to take full advantage of tax data availability, and, in turn, will reduce additional response burden across survey programs.

Statistics Canada aims to reduce the time businesses spend responding to surveys—either by reducing the number of surveys and survey questions, by limiting the time that a business can be part of a sample, or by using more efficient data collection methods. The agency is also working to improve relations with respondents through its choice of communications tools. For more details about initiatives undertaken by the agency to better manage respondent burden, including the creation of an Ombudsman for businesses, refer to Chapter 4.5: Relations with survey respondents.

2. Household Surveys

Statistics Canada used to have two distinct infrastructures: for social surveys and for the census. Following the CBA principles, emphasis has been put in developing infrastructures that can be used by both sets of environment. This has started, particularly around infrastructures used for the creation of frames or for collection. As per the CBA principles, synergies continue to be explored for the other components of the Generic Statistical Business Process Model (GSBPM).

Over the years, while the Census of population has been conducted every five years (refer to Textbox 3.3.1 - Benefiting from the Census Program as a locomotive for innovation), Statistics Canada has developed a number of social surveys to allow the measurement of characteristics of all or some of the members of the household. These characteristics typically include a subset of variables, such as health, education, income, expenditure, employment status, and use of various types of services. Since these variables became common in the 1940s, a number of major trends in household surveys have become evident. Many of these trends are closely linked to technological advances, both in statistical agencies and in society, and have accelerated since the spread of personal computers in the early 1980s.Endnote 4

Over the past few decades, household surveys have faced a challenging environment: declining response rate, more specifically with respect to household surveys that are not mandatory; considerable increased use of smart phones, which makes it difficult to contact respondents; increased collection costs; and, the use of evolving technologies (e.g. Telephone call screening devices).

2.1 Household Surveys Strategy – Guiding Principles

In Canada, the pillars of the modernization process for household surveys are

  • the full use of a Household Survey frame service;
  • the introduction of e-questionnaires and the use of a multi-modal collection system (ICOS)
  • active collection management based on paradata;
  • the development, use and maintenance of common tools for harmonizing the business processes used across social statistics;
  • the increased use of administrative data to reduce response burden.

2.2 Strategies and tools

2.2.1 The full use of a Household Survey Frame Service

Ideally, statistical organizations would have access to an up-to-date, person-based database containing geographic localization, contact information and basic socio-demographic information to produce social statistics and conduct household surveys. Unfortunately, for most countries, this is unrealistic, and statistical agencies use an area frame as the sampling base for their household surveys that target the entire population.

In Canada, the size of the country and the need to reduce travel costs associated with personal interviews have driven the need for a reliable area frame to take advantage of multi-stage sampling. The increasing use of cell phones and the fact that cell phone numbers were excluded from the Canadian random-digit dialing methodologies, made it clear that a telephone survey would only be efficient if more complete telephone registers were available. In parallel, an Address Register (AR) has been created in 1986, initially as a post listing coverage check for the 1991 Census operations. Over the years, many administrative files were added to the AR and its use and coverage improved dramatically.

To rationalize the investments dedicated to developing and maintaining different household survey frames, Statistics Canada has decided, based on corporate business architecture principles, to create a Household Survey frame service responsible for

  • maintaining a series of three files that can be used by the Census and the social statistics programs for various survey processes (sampling frames, contact information, imputation methods)
  • providing support to household survey and population census managers with respect to using these files for their operations.

The first component maintained by the HSFS is the Dwelling Universe File (DUF). It links together the AR (a database of residential addresses across Canada) and the National Geographic Database, which provides geo-coded information as specific as the block level. Since the AR is updated quarterly using information from dozens of administrative files, the DUF is a reliable survey frame for the geographic and address-based household surveys and collection operations that involve mail-out activities, such as the Census of Population.

The second component is the Residential Telephone Files (RTF). It includes a list of telephone numbers and some related information (contact name, address). It is built from multiple sources (InfoDirect, telephone companies' billing files, tax files, census information), and is updated quarterly. The RTF can generate a list of up to 5 telephone numbers per dwelling. It can be used as a frame for telephone surveys. Since 88% of the RTF can be linked to the DUF, it can also provide a source of contact information for address-based surveys.

The last component supported by the HSFS is the Socioeconomic Indicators Files (SEF). It contains auxiliary socio-demographic information at both the household and the person levels. At the dwelling-level, it provides information of household composition and income level. At the person-level, it includes characteristics of each person in the household (age, sex, language, income).

Once the SEF is built, it is maintained annually using the quinquennial population census and annual T1 family files from the Canada Revenue Agency. The SEF can be used at the design stage for clustering, stratification, sample allocation or simulation; at the collection stage for responsive design; and at the estimation stage to facilitate non-response adjustment and imputation.

Creating the HSFS is a first step to developing integrated household survey frames. Although many challenges need to be overcome to allow for a perfect linkage of the three components, creating the HSFS will facilitate research in this area. Statistics Canada also plans to use the HSFS to better coordinate household survey samples and, in turn, to reduce response burden.

2.2.2 The introduction of e-questionnaires and the use of a multi-modal collection system (ICOS)

In terms of data collection, Statistics Canada's household survey enhancement strategy relies on two objectives. The first is to streamline of all collection activities under a unique multi-modal integrated collection and operations System (ICOS). This system would replace the eight collection platforms currently supported by the organization for the various modes of collection used for the Census and the household surveys. For more information about ICOS, refer to Chapter 3.4: Data Collection Planning and Management.

The second objective is to offer all household survey respondents the option of providing their information first via a self-response electronic questionnaire (EQ). If a response was not obtained through EQ, follow-up would then be done using more expensive collection modes, such as telephone interviewing and/or personal interviewing. Indeed, under the multi-modal ICOS initiative, it is expected that all surveys would use the same application for the self-response electronic questionnaire and for interviewers doing telephone or personal interviews. Allowing respondents to provide their information through the Internet, and without the assistance of an interviewer, could significantly reduce collection costs, but does not come without risks to quality. To achieve this objective, Statistics Canada is carrying out significant research to assess, and to find ways of overcoming, the impact on data quality—e.g., understanding concepts without the help of an interviewer, potential selection bias, mode effect, impact of response rates and costs—of introducing e-questionnaires and multi-modal data collection.

2.2.3 Active collection management based on paradata

Data collection for the Census and for household surveys has improved through the use of paradata to inform data-collection management decisions and strategies. For more information about active collection management, refer to the Chapter 3.4: Data Collection Planning and Management.

2.2.4 The development, use and maintenance of common tools for harmonizing the business processes used across social statistics

In the context of social statistics, Statistics Canada's initiative, known as the Common Tools Project, serves as its effort to rationalize business processes. The objective of this CBA sub-project is to develop, use and maintain generalized tools and systems to harmonize the business processes used across all social surveys (starting with programs under the social, health and labour statistics field) and eventually some administrative data files. The main components of this tool are the Social Survey Metadata Environment (SSME) and the Social Survey Processing Environment (SSPE).

The SSME provides an environment for creating, managing, developing and disseminating information that describes the data collected by surveys. In this environment, all metadata are saved in a central location, which enables each tool to access all common metadata throughout the life cycle of a survey. The SSME also includes three common tools:

  • A Questionnaire Development Tool (QDT) that allows subject-matter staff to develop and disseminate questionnaires in a timely fashion, and using a standard approach. Also, since QDT acts as the entry point to the Metadata Repository, it can generate collection specifications required for program collection application and can upload questionnaires in the Integrated Metadata base (IMDB)
  • A Processing and Specifications Tool (PST) that facilitates the creation and management of metadata related to variables within the Social Survey Metadata Environment (SSME).
  • A Data Dictionary Tool (DDT) that enables subject-matter staff to document variables for dissemination.

The SSPE consists of a set of generalized processes for the activities related to a survey life cycle. The principle behind the SSPE is that, even if each survey requires different processing steps and utilities, because of its unique processing requirements, it is still possible to create a processing template using a general flow of steps when setting up processing for any survey. A common processing environment for all household surveys will facilitate the development and use of a core set of generalized systems and, in turn, reduce their maintenance costs.

2.2.5 The increased use of administrative data to reduce response burden

In Canada, the use of administrative data files, which were not created for statistical purposes, has contributed to and increased the production of social statistics. Administrative data are now used to improve household survey frames, as a source of extra information for an existing frame, as a substitute for survey content to reduce response burden, and to improve the quality of statistical products. They are used by themselves or in combination with survey files or other administrative data files. For example, the Canada Revenue Agency maintains the Canadian Child Tax Benefit (CCTB) file, which is used by surveys focussing on sampling children (in fact, their parents). Sets of income questions are gradually being replaced by personal tax information received from the Canada Revenue Agency for the Canadian Census of Population and for several household surveys. An Immigration Database has been created linking information from various sources to better measure outcomes of immigrants to Canada.

The foundation of the household survey enhancement strategy is built on continuously seeking opportunities to increase the use of administrative or alternate data for improving the quality of social statistics products (e.g., reducing sampling or non-sampling errors), improving their relevance (e.g., producing more detailed or more frequent estimates), filling data gaps, reducing the costs of statistical production, and reducing response burden (see Chapter 3.5: Acquisition, use and management of administrative and alternative data). Strategies are currently being built to facilitate record linkage between various sources.

Key Success Factors

A key success factor in enhancing business and household surveys is to consolidate all frame services under the Statistical Registers and Geography Division. This Division is responsible for collecting, compiling, maintaining and disseminating the frames of businesses, dwellings and geographies required for Statistics Canada's surveys, censuses and data-integration activities. The role of these frames is to support the use of the business register under the IBSP and to provide the Household Survey frame service.

Investing in the accuracy and relevance of a business register and in the household survey frames, as well as in the efficiency of their maintenance process, can also be rewarding. Statistics Canada's experience shows that securing reliable administrative data (e.g., tax data), automating the updates as much as possible, standardizing the procedures and concepts, and using accepted standards (geographic, classification.) can greatly improve the quality of business and household survey frames and, consequently, the statistical products.

The management of a large-scale transformation strategy needed to enhance business and household surveys, such as the IBSP, ICOS, HFSF and the Common Tools project, really benefit from the Corporate Business Architecture Governance (see Chapter 3.1: Corporate Business Architecture) and Project Management framework (see Chapter 2.4). A combination of strong governance, transparency in decision making, and the involvement of partners in developing common solutions, has been a successful approach. Frequent communication among all levels, and through many different channels, has kept stakeholders informed and engaged in projects. The adoption of a metadata-driven approach also greatly contributes to realizing efficiencies, while improving the coherence of economic statistics.

Finally, modernizing the production of economic and social statistics must be supported by rigorous research to ensure the efficiency and scientific soundness of new methods. Innovating and conducting experiments to test ideas and improve practices is essential. Research should therefore be an identified and supported function in the organization.

At Statistics Canada, annual investments in methodology research are secured to develop, promote, monitor and guide the adoption of new and innovative techniques in statistical methodology to support Statistics Canada's statistical programs. The International Cooperation and Corporate Statistical Methods Division (ICCSMD) is responsible for methodology research and provides technical leadership, advice and guidance to employees elsewhere in the Methodology Branch. ICCSMD staff are also jointly involved with members of the other methodology divisions through research projects on specific topics, e.g. estimation methods, imputation methods, small area estimation methods, record linkage techniques, use of administrative data, measurement of non-sampling errors, which are sponsored by the Methodology Research and Development Committee.

The Advisory Committee on Statistical Methods also advises the Chief Statistician on matters related to the use of efficient statistical methods in the agency's program, as well as its research and development program in statistical methods. The committee's activities include the following:

  • Review and comment on Statistics Canada's priorities in methods research;
  • Review and comment on methods used in particular programs (e.g., survey design; census under coverage measurement; gross flow estimation; small area estimation techniques);
  • Review and comment on generic methods used widely in the agency's programs (e.g. seasonal adjustment methods, edit and imputation methods, quality control methods);
  • Suggest functions or programs within the agency that could benefit from the innovative application of statistical methods;
  • Comment on the agency's allocation of resources to provide methodological support to its programs, and to research methods;
  • Suggest means by which Statistics Canada can ensure its continuing leadership role in the development of statistical methods; and, finally,
  • Advice on the agency's quality assurance program and the actions that stem from it.

This research capacity and its governing body ensures that the statistical organization remain at the leading edge of the profession.

Challenges

To successfully achieve integration across many programs and processes, Statistics Canada's experience shows that the following considerations might reduce the risk of its potential pitfalls:

  • Continuous engagement with the administrative data suppliers for the purpose of maintaining the accuracy and relevancy of the statistical survey frames needs to be ensured;
  • Large-scale projects with significant impact on the organization should receive continual support and buy-in from senior managers and should benefit from strong and efficient governance;
  • Subject-matter areas should have the ability to negotiate and adapt. Generic solutions have limitations, and concerns should be addressed according to corporate priorities; and
  • Staff should be trained to use new tools early in the process.

Adopting a modular approach (i.e., building the integrated survey program in phases) is also strongly recommended to make a project more manageable.

The important challenge is the balance between actually delivering on common tools and infrastructures for surveys while at the same time planning and finding ways to enhance the use of administrative data in the future.

Looking ahead

For efficiency purposes, national statistical organizations will need to consider, more and more strategically, the use of administrative data. It should be systematic to, first, see whether any administrative data exist, or can be combined with existing data collection. If this is not the case, then one can consider a new survey project. For more information about strategies with regard to acquisition, use and management of administrative data, refer to Chapter 3.5.

Box 3.3.1: Benefiting from the Census Program as a locomotive for innovation

The foundation for the many of the transformations that need to be operated on household surveys will originate from the models used on the Census of Population Program. Statistics Canada has been using a multi-mode collection approach for the Census since 2006 when internet as a response option was first introduced. The collection methods were then adapted for the 2011 Census with the introduction of a wave approach to boost response by internet while minimizing the risks of non-response. The wave approach favors a more costs efficient set of initial contacts and reminders using letters inviting respondents to participate primarily by internet. Telephone and in person follow-ups are introduced in later stages of collection and targeted to non-respondents from the first waves. The approach used for the 2011 Census generated a high level of internet response (54%) and self-response (85%). A similar approach could be adapted to most household surveys as they move to corporate sampling and collection tools.

The Census Program has also been innovative in the development and use of automated processes. It developed the Address Register as its frame and has been leading in the use of administrative sources to maintain that frame as opposed to direct address listing. The focus for the program going forward is shifting to reducing under and over coverage of addresses in the frame.

Different tools were also developed first for the Census Program to support its multi-mode collection approach and support field personnel in the management of their workload. These include a Master Control System to keep track of the status of every dwelling in the frame in quasi-real-time and the Field Management System in 2011 which has now been converted into the Collection Management Portal under ICOS for 2016. The functionalities within these systems will form the basis of what is needed to convert household surveys to the new collection approaches.

The Census Program is so expanding the use of administrative data to reduce both burden and cost as well as improve quality. In 2016, the questions related to income have been replaced by administrative data provided by the Canada Revenue Agency. Going forward, the Census Program is conducting work on the creation of the virtual population register further expanding the use of administrative data sources to ultimately replace the enumerated head count. In addition, this register will be used directly as a frame for some household surveys or to supplement the current Household Survey Frame. Work is also being conducted on how frame information can be used in adaptive collection methodologies.

The process of innovation on the Census Program is aiming at making the program more efficient and reducing burden on respondent while maintaining a high level of data quality.

Endnotes:

Endnote 1

United Nations Economic Commission for Europe, 2015.

Return to endnote 1 referrer

Endnote 2

Statistics Canada, 2015a.

Return to endnote 2 referrer

Endnote 3

Turmelle and Al., 2014.

Return to endnote 3 referrer

Endnote 4

Gambino and Nascimento, 2009.

Return to endnote 4 referrer

Bibliography

Bosa, K, Godbout, S., Mills, F and Turmelle, C. (2014). A Quality Driven Approach to Managing Collection and Analysis. Consulted on the 11th of March 2016 and retrieved from http://www.q2014.at/fileadmin/user_upload/Q2014_Paper_QIMI_final.pdf

G. Gambino, Jack and Luis do Nascimento Silva, Pedro (2009). Sample Surveys: Design, Methods and Applications, Vol. 29a. Consulted on the 11th of March 2016 and retrieved from http://www.sciencedirect.com/science/handbooks/01697161/29/part/PA

Government of Canada (2003). Survey Methods and Practices, Statistics Canada. Consulted on 11th of March 2016. Retrieved from http://www5.statcan.gc.ca/olc-cel/olc.action?ObjId=12-587-X&ObjType=2&lang=en&limit=0

Statistics Canada (2015a). Integrated Business Statistics Program Overview, Statistic Canada. Consulted on 11th of March 2016. Retrieved from http://www.statcan.gc.ca/pub/68-515-x/68-515-x2015001-eng.htm

Statistics Canada (2015b). Response Burden Reduction Efforts, Statistic Canada. Consulted on 11th of March 2016 and retrieved from http://www.statcan.gc.ca/eng/about/rbre

St-Louis, Gaetan (2008). The Evolution of Administrative Data Use for the Canadian Business Register (BR), Statistics Canada.

United Nations Economic Commission for Europe (2015). Guidelines on Statistical Business Registers, Task Force on Guidelines on Statistical Business Registers, Consulted on the 11th of March 2016 and retrieved from http://www.unece.org:8080/index.php?id=40574&L=0

Report a problem on this page

Is something not working? Is there information outdated? Can't find what you're looking for?

Please contact us and let us know how we can help you.

Privacy notice

Date modified: