Quality Assurance Framework
Statistical Outputs

G. Relevance

Description

Statistics Canada aims to provide its data users and stakeholders with information on subjects that are important to them, and in a format and within a time frame that facilitates their research, analysis, decision-making and communications. The relevance of statistical information reflects the degree to which it meets the needs of data users and stakeholders.

Managing relevance requires ensuring that the Agency’s programs remain aligned with information needs as they evolve. Being aware of changing priorities and having the flexibility to respond to them are vital to ensuring continued relevance.

Assessment

Statistics Canada’s efforts to maintain relevance of its statistical programs are assessed by evaluating the extent to which the Agency:

  1. captures emerging trends and changes to the Canadian sociological and economic framework, and thus identifies information gaps in its current statistical programs
  2. communicates information about its mandate, values, priorities and management practices to data users and stakeholders
  3. ensures continued production of relevant, high-quality and timely information through long-term planning and priority-setting
  4. supports refreshment of current statistical programs and promotion of new initiatives by identifying and redirecting savings in existing programs and by procuring additional funding.

Implementation

Statistics Canada undertakes a broad range of specific initiatives to maintain relevance in its statistical programs. Below is a list of such initiatives, in groups that correspond to the items under Assessment.

G.1 Statistics Canada captures emerging trends and changes to the Canadian sociological and economic framework, and thus identifies information gaps in its current statistical programs

  • Seek input from data users and stakeholders so that Statistics Canada keeps abreast of changes in society, culture and commerce and the impact of these changes. This engagement is accomplished through such practices as meetings of the Chief Statistician and senior managers with counterparts internationally and in other federal departments, federal–provincial–territorial meetings, use of advisory groups of subject-matter and technical experts, and advocacy groups. Further, liaison with businesses, not-for-profit organizations, consortia of data users, the media and other secondary distributors of data are also used.
  • Regularly consult with Canadians to monitor data user and stakeholder satisfaction with the current statistical programs, using a variety of methods. For example, the Agency consults with: business and industry associations, small-business advocacy groups and labour unions, in particular to learn about information needs and reporting preferences; not-for-profit organizations including community groups, social organizations and volunteer groups; and interested groups on particular programs (e.g., Census of Population content). When possible, the results of consultation are made available to the public.
  • Supplement the current statistical program with additions funded on a cost-recovery basis. Develop and maintain the capacity to respond quickly to demands for information on matters of current interest. Allow additions funded on a cost-recovery basis to become part of the core statistical program, if appropriate.
  • Gather information on the use of Statistics Canada’s products and services. Statistics Canada’s division responsible for dissemination tracks data users and the uses to which they put statistical products. Usage and other statistics are compiled along with bibliometric and webometric analyses. Feedback from data analysts identifies gaps and weaknesses in data holdings.

G.2 Statistics Canada communicates information about its mandate, values, priorities and management practices to data users and stakeholders

G.3 Statistics Canada ensures continued production of relevant, high-quality and timely information through long-term planning and priority-setting

  • Manage priority-setting and planning to ensure continued relevance. Weigh competing needs objectively, based on usage statistics and other information, when elaborating Agency-level priorities, and set program-level priorities accordingly. Assess quality of existing statistical programs and adapt to changing needs efficiently and quickly through responsive allocation and flexible deployment of resources. An integrated strategic planning process includes a review of corporate priorities and integrates risk management, investment planning, and evaluation into the planning process.
    • Integrated Strategic Planning Process
    • Integrated Business and Human Resources Plan
    • Information Technology Plan
  • Forecast the regular and strategic investments necessary to preserve quality and continuity of statistical programs over time. This forecast covers an extended horizon (10 years) and considers the cyclical nature of specific statistical programs (e.g., five-year Census cycle, 10-year redesign cycle for certain surveys) when planning redesigns and other program modifications or enhancements. A separate fund exists for initiatives that cannot be covered by program area base budgets.
    • Continuity and Quality Maintenance Investment Plan
    • Continuity and Quality Management Fund
  • Continuously review programs from the perspective of evolving data user and stakeholder needs, and make necessary adjustments to program content. The annual strategic planning sessions provide a forum for addressing relevance issues, as does the annual risk assessment activity. The division responsible for dissemination identifies products for which demand has declined.
  • Conduct external audits of statistical programs and benchmarking to external indicators, as required.

G.4 Statistics Canada supports refreshment of current statistical programs and promotion of new initiatives by identifying and redirecting savings in existing programs and by procuring additional funding

  • Target efficiencies by regularly conducting Agency-wide reviews of systems and practices. Statistics Canada has in place the Corporate Business Architecture which is a long-term Agency-wide review and modernization that covers the business processes, business rules, computer systems and internal organizational and physical infrastructure that Statistics Canada uses to carry out its main business of collecting, analyzing and publishing statistical information. By consolidating processes and standardizing systems where necessary, cost savings are achieved while still maintaining high standards of quality and timeliness.
    • Corporate Business Architecture
  • Regularly review the cost-effectiveness of statistical programs, in particular those required to meet legislative or regulatory requirements. Manage survey collection, processing operations, dissemination practices and quality assurance procedures so that the resulting statistics will meet needs in a cost-effective fashion. Look for operational efficiencies, in particular by managing and exploiting interdependency among statistical programs.
  • Partner with stakeholders, especially other federal government departments, to finance valuable additions to the national statistical database, in particular to meet legislative or regulatory requirements or to provide information for policy development and evaluation.

H. Accuracy and reliability

Description

Statistics Canada aims to develop, produce and disseminate information that presents its data users and stakeholders with a true reflection of reality. The accuracy of statistical information refers to the degree to which it correctly describes the phenomena it was designed to measure. The accuracy of statistical estimates is usually quantified by the evaluation of different sources of error, where the magnitude of an error represents the degree of difference between the estimate and the true value. Common sources of error include coverage, non-response, measurement and processing. For estimates derived from survey data, an additional error source is sampling error, which reflects the fact that the estimates are computed from samples, rather than the entire population.

Related to accuracy is the concept of reliability, which reflects the degree to which statistical information, consistently over time, correctly describes the phenomena it was designed to measure. That is, reliability characterizes repeated observations of accuracy, over time. Reliability applies both to multiple measurements of the same phenomenon (e.g., preliminary, final and revised estimates) and to a series of measurements (e.g., monthly estimates of the employment rate).

A definitive assessment of accuracy and reliability is impeded by the fact that an estimate needs to be compared with a true value that possibly cannot be expressed quantitatively or that can be obtained only at great expense. Consequently, much evaluation of accuracy and reliability is qualitative, described by the steps taken to mitigate sources of error. One exception is sampling error, for which a sound mathematical framework exists and quantitative measures, estimates of variance, are typically calculated and made available. The variance is a measure of precision. In contrast to accuracy, precision expresses how close the estimates are to their average value, rather than to the true value. Other measures of precision, such as coefficients of variation and confidence intervals, can be derived from the variance. The term bias is used to describe average differences between estimates and the true value. Bias is generally due to systematic or non-random sources of error.

In managing accuracy, investment in a strong statistical methods research program increases the capacity to improve accuracy without increasing cost or impacting timeliness, through the incorporation of state-of-the-art statistical techniques.

Assessment

Statistics Canada’s efforts pertaining to accuracy and reliability are assessed by evaluating the extent to which the Agency:

  1. incorporates quality assurance measures into program and process design, implementation and execution
  2. manages and monitors accuracy during implementation and execution of its statistical programs and processes
  3. assesses accuracy and reliability, both pre-release and post-release, and communicates the results.

Implementation

Statistics Canada undertakes a broad range of specific initiatives to promote accuracy and reliability in its statistical programs. Below is a list of such initiatives, in groups that correspond to the items under Assessment.

H.1 Statistics Canada incorporates quality assurance measures into program and process design, implementation and execution

  • Design and implement a process quality management plan, which includes quality assurance actions and practices to ensure that the process reliably produces the desired results and products, and indicators reflecting the effectiveness of the quality assurance actions and practices. The process quality management plan: includes results from intermediate processing steps; uses information obtained to correct problems detected and reduce error incidence; and also includes process control of manual operations, as needed.
  • Establish accuracy targets for key estimates (variable and domain). Program areas define requirements for corporate performance indicators of accuracy, including the contribution of non-response adjustment.
    • Quality Guidelines
    • Guidelines on coefficients of variation
    • Corporate performance indicators (coefficient of variation)
  • Maintain or improve accuracy while minimizing response burden and cost by identifying and considering alternate sources of the same or similar data, including existing survey data and administrative data. The reduction in response burden is managed in part by the corporate committee responsible for administrative data management.
    • Administrative Data Management Committee
  • Study and evaluate new and alternative statistical methods. Consider their impact on accuracy, reliability and other dimensions of quality as part of the evaluation of design alternatives. The research is guided by an advisory committee of experts and supported by a research fund.
    • Advisory Committee on Statistical Methods
    • Methodology Research Block Fund
  • Develop expertise in statistical techniques and improve accuracy through incorporating such refinements. Examples of such techniques include calibration, benchmarking and seasonal adjustment.
    • Time Series Research and Analysis Centre
  • Establish, maintain and periodically evaluate survey frames. The use of centrally-maintained frames ensures up-to-date and consistent coverage and classification of target populations for survey programs. Program areas assess coverage of their target population by the proposed survey frame, for the entire population and significant sub-populations.
  • Objectively weigh trade-offs between accuracy and other factors, including relevance, timeliness, cost and response burden, both before deciding to undertake a survey and when planning each aspect of it.

H.2 Statistics Canada manages and monitors accuracy during implementation and execution of its statistical programs and processes

  • Execute a process quality management plan. Monitor and interpret indicators reflecting the effectiveness of quality assurance actions and practices. Process quality monitoring is integrated into regular on-going management functions.
  • Assess and validate administrative and other source data using standardized guidelines for detecting and correcting errors and assessing accuracy.
    • Administrative Data Management Committee
  • Ensure effective questionnaire design through adequate justification for each question asked, standardized terminology, banks of common questions, and appropriate pre-testing and review. A corporate resource centre is mandated to review and evaluate each questionnaire against these standards.
    • Questionnaire Design Resource Centre
    • Policy on the Review and Testing of Questionnaires
  • Implement adequate measures for encouraging response, following-up non-response and compensating for total and item non-response.

H.3 Statistics Canada assesses accuracy and reliability, both pre-release and post-release, and communicates the results

  • Evaluate the accuracy and reliability of estimates. Program areas conduct an evaluation after each survey cycle and document results. They track differences between target and realized values of measures of accuracy and analyze sampling errors and non-sampling errors in the post-release evaluation of each survey cycle. Reliability can then be studied by analyzing errors over time. Program areas identify best practices, lessons learned and insights gained, and then recommend improvements. This information is shared with other programs within Statistics Canada.
  • Address quickly and uniformly errors detected in published estimates and inform users transparently.
    • Directive on Corrections to Daily Releases and Statistical Products
  • Ensure that preliminary or revised data are clearly identified as such to users, and that users are provided with the accuracy indicators necessary to make effective use of them. Produce and make public information that quantifies differences between preliminary, final and revised estimates, and use this information to improve the survey process.
  • Report data quality and methodology information necessary for data users. Require discussion of sources of error and estimates of accuracy for key variables and domains as part of data quality and methodology information provided to users. Periodically compile and disseminate quality reports. Include both a qualitative and quantitative analysis of all types of errors.

I. Timeliness and punctuality

Description

Statistics Canada strives to make its data products available as quickly as possible and to respect its commitments to data availability. The timeliness of a data product is defined as the length of time between the end of the reference period (or the reference date) to which the data relate and the date the product is made available. Punctuality refers to the difference between planned and actual availability.

Alternate definitions may be used for other products and services. For example, timeliness of a custom data request may be defined as the length of time between the receipt and delivery of a request. For new initiatives, the definition of timeliness may be extended to include development time. Using a new survey as an example, timeliness may be defined as the length of time between the commitment to conduct the survey and the release of its data.

These definitions view timeliness and punctuality from the user perspective (i.e., relative to release or delivery), and will be the basis of discussion in this chapter. Despite efforts to introduce innovation and to effectively manage resources, gains in timeliness may require or lead to affecting other dimensions of quality. Consequently, the impact on other dimensions of quality must be considered when evaluating efforts to improve timeliness.

Assessment

Statistics Canada’s efforts pertaining to timeliness and punctuality are assessed by evaluating the extent to which the Agency:

  1. defines policies that reflect its commitment that Canadians have equitable and timely access to its data products and information releases
  2. effectively manages user expectations and other issues related to timeliness and punctuality
  3. communicates release dates transparently and ensures equitable and timely access to data
  4. implements methods to evaluate timeliness and punctuality and to improve outcomes.

Implementation

Statistics Canada undertakes a broad range of specific initiatives to promote timeliness and punctuality in its statistical programs. Below is a list of such initiatives, in groups that correspond to the items under Assessment.

I.1 Statistics Canada defines policies that reflect its commitment that Canadians have equitable and timely access to its data products and information releases

  • Address both scheduling of release dates and how users and stakeholders are informed about them. The policies outline release procedures and suggest timeliness targets, and they allow for flexibility (i.e., release procedures appropriate for the type of data product, and timeliness targets that reflect the length of the reference period). Timeliness targets are guided by international dissemination standards.
  • Outline the procedures for modifying release dates. Users and stakeholders are informed of the reasons for any changes and of new release dates. Further, the policies outline scheduled revisions to data products and inform users about unplanned corrections as quickly and transparently as possible.
  • Eliminate incentives for interested parties to influence or delay a particular release for their own benefit through transparency regarding release dates.
  • Implement practices to support the Agency’s commitment to monitor release dates and track divergence from them.

I.2 Statistics Canada effectively manages user expectations and other issues related to timeliness and punctuality

  • Consult data users and stakeholders on issues related to timeliness and punctuality. This is done by engaging with data users and stakeholders through such practices as meetings of the Chief Statistician and senior managers with counterparts internationally and in other federal departments, federal–provincial–territorial meetings, internal and external advisory committees and working groups.
  • Give explicit consideration to timeliness in design, planning and project management. A production schedule is created in consultation with the collection, processing and dissemination teams. The schedule is effectively managed through collaboration between the project team and Statistics Canada’s corporate collection service. Conflicts, lapses and other threats to punctuality are identified and evaluated based on their impact and likelihood. Contingency plans and other recourses are elaborated and employed as needed.
  • Include quality assurance, peer and professional review and other preventative measures in the timelines. Trade-offs between timeliness, other dimensions of quality and cost are managed effectively so that fitness for use, fiscal responsibility and other Statistics Canada priorities are not compromised.
  • When feasible, make preliminary data or other leading indicators available, with guidance on their quality and appropriate use. Differences in magnitude and direction between preliminary, revised and final estimates are tracked and used to assess the timeliness–accuracy balance. Persistent or predictable biases in preliminary data are identified and, when possible, addressed in estimation.
  • Effectively manage user expectations regarding timeliness and data availability through communication of Agency priorities and constraints. Feedback on the impact of timeliness on relevance to users and stakeholders is of particular interest to the Agency.

I.3 Statistics Canada communicates release dates transparently and ensures equitable and timely access to data

  • Publish release dates well in advance. A single release calendar includes all key economic indicators and major releases planned in the next 12 months. The release calendar is easily accessible with a link on themain page of The Daily and gives users and stakeholders sufficient advance notice of upcoming releases.
  • Require that all releases pass through a uniform dissemination service, where The Daily is Statistics Canada’s official release bulletin. It is made public at a standard time and port of entry (i.e., 8:30 a.m. Eastern time).
  • Correct errors in a structured, timely and transparent fashion.
    • Directive on Corrections to Daily Releases and Statistical Products
  • Work with the media and others to maximize exposure and awareness of Statistics Canada’s releases. Releases are scheduled so that they do not overshadow each other and do not coincide with other major (planned) news events. User and media anticipation of data releases is built through effective communication and outreach activities. Various social media (including, Facebook, Twitter, YouTube, Statistics Canada blog and “Chat with an expert”) are used to communicate with users. A media spokesperson is always available on release day. Further, many free products are available on the website and the re-dissemination of Statistics Canada information products is encouraged through an open-data licence agreement that requires no payment of fees for specific data products.

I.4 Statistics Canada implements methods to evaluate timeliness and punctuality and to improve outcomes

  • Develop and implement effective project management tools. A departmental project management office provides support in the development of common processes and tools to improve the timely delivery of projects in a cost-effective fashion while adhering to quality standards and meeting client needs. A supporting framework exists as a set of standard project management processes, templates and tools.
    • Departmental Project Management Office
    • Departmental Project Management Framework
  • Develop, calculate and publish quality indicators related to timeliness and punctuality and use them to improve outcomes.
    • Corporate performance indicators (timeliness and punctuality)

J. Accessibility and clarity

Description

Statistics Canada’s goal is to make its official statistics and data products available to meet the information needs of all Canadians. Accessibility refers to the ease with which users are able to identify, obtain and use statistical products and services. Clarity refers to the degree to which metadata and other information are provided so that users are able to locate and select products or services that correspond to their needs.

Accessibility and clarity apply to all Statistics Canada products and services: statistical products (i.e., aggregate tables, analytical articles), data access programs and other requests (i.e. custom surveys, specialized tabulations, services). They also apply to metadata and other support provided with a product or service. Many factors that influence data availability and access, such as dissemination policies and data access systems, are determined by Agency-wide initiatives. Individual program areas also play a vital role by designing statistical products and providing them in formats that meet user needs. User experience is an important aspect to consider when evaluating accessibility and clarity; i.e., from beginning a search, to identifying and obtaining a product or service and completing the related work.

The Agency must balance the increasing demand for data and the desire to minimize barriers to access with its obligation to protect the privacy and confidentiality of respondents and to ensure equitable access for all Canadians.

Assessment

Statistics Canada’s efforts pertaining to accessibility and clarity are assessed by evaluating the extent to which the Agency:

  1. publicizes products and services, including the availability of metadata and other supplementary resources
  2. delivers products and services in an impartial and open manner, and ensures equitable access by minimizing cost barriers
  3. facilitates the finding of information about products and services so that users can make informed decisions about which ones meet their needs
  4. enables the use of its products, by providing clear procedures on accessing them, providing products in formats compatible with user requirements, and ensuring barrier-free redistribution of its data
  5. fosters correct use of information products by using clear, audience-appropriate presentation, combined with informative and useful resource materials
  6. supports and promotes its data-access programs and services.

Implementation

Statistics Canada undertakes a broad range of specific initiatives to promote accessibility and clarity in its statistical programs. Below is a list of such initiatives, in groups that correspond to the items under Assessment.

J.1 Statistics Canada publicizes products and services, including the availability of metadata and other supplementary resources

  • Work with the media and others to maximize awareness, in particular by encouraging re-dissemination of Statistics Canada information products through use of modern information technology and an open-data licence agreement that requires no payment of fees for specific data products. This approach includes applications for smartphone and tablet access, a presence on social media (including Facebook, Twitter and YouTube), a regular blog and a facility to interact with Agency specialists.
  • Provide special and dedicated services for members of the media.
  • Publish release dates well in advance. A single release calendar includes all key economic indicators and major releases planned in the next 12 months. The release calendar is easily accessible with a link on themain page of The Daily and gives users sufficient advance notice of upcoming releases.

J.2 Statistics Canada delivers products and services in an impartial and open manner, and ensures equitable access by minimizing cost barriers

  • Require that all releases pass through a uniform dissemination service, where The Daily is Statistics Canada’s official release bulletin. It is released at a standard time and port of entry (i.e., 8:30 a.m. Eastern time) each business day.
  • Promote availability by specifying requirements for non-release or restricted release of data. Ensure decisions regarding release are independent of funding source or stakeholder.
    • Directive on Access to Information and Privacy
  • Minimize the barrier of cost by providing standard products, for each statistical program, free of charge on the Statistics Canada website.
  • Produce and promote a broad range of statistical products, and define a fair and balanced policy for funding these on a cost-recovery basis. For example, research papers, analytical products, specialized tabulations, as well as the design and implementation of customized surveys, are available.

J.3 Statistics Canada facilitates the finding of information about products and services so that users can make informed decisions about which ones meet their needs

  • Provide modern search capabilities on a single, well-known point of entry to Statistics Canada’s online presence, as well as a catalogue and customer service to help users find the desired products. Manage inquiries clearly and consistently.
  • Provide clear descriptions of products and services, as well as links to complementary information, such as methodological notes and quality measures through the Integrated Metadatabase.
    • Integrated Metadatabase
  • Ensure that Statistics Canada products and metadata are stored in centralized repositories in a logical and organized fashion. The dissemination framework and the socioeconomic database CANSIM are two such examples.
    • Statistics Canada Dissemination Model
    • CANSIM
  • Ensure that past products, including those produced before electronic release and storage, are properly archived.
    • Web Archiving Directive

J.4 Statistics Canada enables the use of its products, by providing clear procedures on accessing them, providing products in formats compatible with user requirements, and ensuring barrier-free redistribution of its data

  • Tailor outputs to conform to medium-specific content guidelines and standards.
  • Ensure that outputs are available in alternate formats to accommodate users with special needs (e.g., outputs in large print or Braille available free of charge, upon request).
  • Ensure that outputs are compatible with common user software.
  • Develop tools that automate data retrieval and display, and offer them free of charge.
  • Encourage citation, quotation, and transmission (re-dissemination) of Statistics Canada information products by the media and other users through the use of modern information technology and social media, and an open-data licence agreement that requires no payment of fees for specific data products.
  • Provide channels for users to interact with Statistics Canada, to communicate their needs, to get support, and to provide feedback.

J.5 Statistics Canada fosters correct use of information products by using clear, audience-appropriate presentation, combined with informative and useful resource materials

  • Ensure the consistency of messages by using a controlled media-spokesperson strategy.
    • Directive on Media Relations
  • Ensure clear and comprehensible presentation of information products by adhering to publication guidelines and other content standards.
    • Statistics Canada Publishing Guidelines
    • Policy on Highlights of Publications
  • Ensure consistency of information products over time by specifying standard concepts, definitions and classifications, and adopting best practices.
  • Minimize errors and inconsistencies in information products by implementing quality assurance procedures, including staff training and institutional and peer reviews.
  • Ensure users are able to obtain supporting materials, such as metadata, user guides, help lines, quality reports and supplemental software.
  • Share analytical tools developed by Statistics Canada to foster the consistent use of the data and to make typical data transformations easier for users.

J.6 Statistics Canada supports and promotes its data-access programs and services

  • Facilitate researcher access to microdata without compromising data security and confidentiality, by using secure remote-access tools and flexible staffing arrangements.
  • Facilitate access to Statistics Canada’s staff and expertise through consultation, participation in peer groups, and international and private-sector collaboration.
  • Define objective release or suppression criteria for custom tabulations and other special requests. Implement a vetting process that includes confidentiality measures.
    • Quality Guidelines 
    • Policy on Microdata Release
    • Guidelines for the Release of Microdata Files
  • Ensure that software and other non-data products, such as generalized systems, are marketed effectively to benefit Statistics Canada and users.
    • Policy on Software Dissemination

K. Coherence and comparability

Description

Statistics Canada aims to develop, produce and disseminate statistics that are consistent and comparable over time and across jurisdictions, and can be used in conjunction with information from other sources. Coherence of statistics refers to the extent to which they are logically consistent in terms of definition and measurement and thus can be reliably combined in different ways and for various uses. Comparability refers to the extent to which differences over time or among sources can be attributed to changes in the true values of the statistics, and not to changes in definition or measurement.

The use of standard concepts, definitions, classifications and target populations promotes coherence and comparability, as does the use of common methodology, statistical techniques and processes across surveys. Lack of coherence or comparability is generally attributed to two primary sources: differences in concepts and definitions; and differences in collection, processing and dissemination methods. In the first case, the target of measurement is not consistent; in the second, the process of measurement introduces inconsistency.

Assessment

Statistics Canada’s efforts pertaining to coherence and comparability are assessed by evaluating the extent to which the Agency:

  1. applies concepts, definitions, frameworks and protocols uniformly to ensure consistency across its statistical programs and to facilitate comparison over time and with alternate sources of related information
  2. designs and implements common methods for data collection, processing and dissemination to minimize inconsistency during operations and to manage change between cycles
  3. verifies outputs, including comparisons with estimates from other sources and related information
  4. provides documentation and other supplementary information and applies statistical techniques to help users make meaningful comparisons over time and with other sources.

Implementation

Statistics Canada undertakes a broad range of specific initiatives to promote coherence and comparability in its statistical programs. Below is a list of such initiatives, in groups that correspond to the items under Assessment.

K.1 Statistics Canada applies concepts, definitions, frameworks and protocols uniformly to ensure consistency across its statistical programs and to facilitate comparison over time and with alternate sources of related information

  • Develop and maintain a protocol for statistical standards. Statistical standards apply to concepts, definitions, frameworks, units, variables, classification systems and target populations. The Agency promotes and monitors the adoption of statistical standards uniformly across statistical programs. Exceptions to statistical standards are justified and documented.
  • Maintain and disseminate information on statistical standards in a corporate metadata repository. The elements in the repository are updated, changes are tracked and users are informed.
  • Provide structures and tools that support administration and governance of statistical standards. A corporate committee assists and advises on the development, approval and application of statistical standards and metadata within the Agency’s programs. A specific division is responsible for all the classifications and standards - including industry, product, occupation and education classifications, geography and economic accounts - used by Statistics Canada.
  • Participate in the development of national and international standards and other frameworks, and encourage consistency with them. Statistics Canada participates in numerous development groups, such as those of the United Nations Statistics Division and the United Nations Economic Commission for Europe. The Agency is also involved in groups developing standard classification systems such as the North American Industrial Classification System and the National Occupational Classification-Statistics.
  • Set up cross-program committees to ensure that quantities being estimated bear relation to each other through use of consistent terminology, commonly formulated questions and comparable variable definitions. In particular, maximize consistency both within (e.g., between quarterly and annual estimates, and between preliminary and final estimates) and across statistical programs.
    • Policy on Standards
    • Policy on the Review and Testing of Questionnaires
    • Questionnaire Design Resource Centre
  • Participate in various external fora, such as working groups and subject-matter expert groups, to engage with other national statistical offices in recognizing and promoting national and international best practices, arithmetic and accounting identities and other standards. Ensure communication of information so that program areas are acquainted with and guided by national and international best practices and cutting-edge methods.
  • Ensure program areas periodically assess compliance with standards and frameworks as well as consistency and comparability with related administrative data, other estimates from Statistics Canada and estimates from elsewhere.

K.2 Statistics Canada designs and implements common methods for data collection, processing and dissemination to minimize inconsistency during operations and to manage change between cycles

  • Use standardized frameworks and systems to support process management.
  • Develop and use common frames and processing environments. Examples of these are the Business Register, Household Survey Frame Service, Social Survey Processing Environment and the Integrated Business Statistics Program.
  • Optimize use of generalized systems and corporate services.
    • Generalized Systems Resource Centre
    • Resource and support centres
  • Ensure internal consistency of outputs during operations. In particular, ensure that changes to definitions or other inputs are incorporated, arithmetic or accounting operations do not lead to discrepancies (e.g., rounding)Note 1 and that outputs of complementary processes are integrated properly.
    • Quality Guidelines
    • Quality Assurance Framework
    • Directive and Guidelines for the Validation of Statistical Outputs
    • Quality Secretariat
  • Implement an appropriate structure for approval and testing when developing and implementing new software applications and other processing tools. Develop appropriate guidelines and other oversight tools and structures, as needed.
  • Ensure consistency of message and convenient user access through a corporate dissemination framework and access tools such as The Daily and tables from the Agency’s socioeconomic database CANSIM.
    • Statistics Canada Dissemination Model
    • Directive on Media Relations
    • Directive on Corrections to Daily Releases and Statistical Products
    • The Daily
    • CANSIM

K.3 Statistics Canada verifies outputs, including comparisons with estimates from other sources and related information

  • Reconcile estimates and outputs with other comparable statistical and administrative sources on similar subjects, including previous estimates from the same statistical program. Any differences are identified and substantiated.
    • Directive on the Management of Aggregate Statistics
  • Elaborate a strategy for detecting and correcting errors in previously released data. Retroactive corrections are implemented, if necessary.
    • Directive on Corrections to Daily Releases and Statistical Products
  • For redesigns and other significant changes in survey methodology or international standards, explain breaks in series and develop methods for reconciliation. Produce historical revisions when deemed appropriate. For example, a data series is updated when a classification framework is revised.
  • Solicit feedback from users, in particular their experience in producing historical series, comparing estimates between statistical programs, and comparing the Agency’s outputs with those of other sources. Identify inadequacies and issues.

K.4 Statistics Canada provides documentation and other supplementary information and applies statistical techniques to help users make meaningful comparisons over time and with other sources

  • Document changes to concepts, definitions, classifications and methods, including changes between preliminary and final estimates. Changes are expressed in quantitative terms when possible.
    • Directive on Documenting Statistical Metadata
  • Produce and distribute quality reports and other supporting methodological documentation that address internal consistency, comparability over time and comparability with other statistics. Changes to important parameters and methods are emphasized, as well as deviations from international standards and other practices.
  • Provide quantitative and qualitative information that allows users to compensate for differences in periodicity, survey population, variable definitions and processing methods when comparing data from different sources and comparing data over time.
    • Integrated Metadatabase
    • Directive on Media Relations
  • Develop expertise in statistical techniques and produce data products that incorporate such refinements. Examples of such techniques include calibration, benchmarking and seasonal adjustment.
    • Time Series Research and Analysis Centre
    • Advisory Committee on Statistical Methods
    • Methodology Research Block Fund

L. Interpretability and management of metadata

Description

Statistics Canada has a responsibility to provide both statistical information and the necessary tools and guidance to ensure its proper interpretation and appropriate use. Without such tools and guidance, statistical information is of limited use and its interpretation becomes subject to the bias of the user. There is also a risk of information products becoming a liability to the Agency if they are misunderstood or misinterpreted as supporting a particular agenda. Interpretability refers to the ease with which users are able to understand, properly analyze and draw correct inferences from statistical information. Statistical metadata are defined as “information about statistical data and the statistical business process”Note 2. Thus managing interpretability primarily involves providing metadataNote 3 so that the statistical information can be understood and used appropriately.

The information needed to understand statistical data falls under three broad headings. Concepts, variables and classifications that underlie the data define what is being measured or estimated in the statistical information product, and are required by users to assess the product’s relevance to their needs. Information on the methodology of data collection, processing and dissemination addresses how the concept is being measured, in particular whether methods were scientific, objective and carefully implemented. If users are aware of the methodology, they can choose the appropriate analytical tools. Data-quality measures quantify how well the concept was measured and help users associate the appropriate level of confidence with the results.

Assessment

Statistics Canada’s efforts pertaining to interpretability are assessed by evaluating the extent to which the Agency:

  1. enshrines metadata requirements in policies and practices
  2. provides users with the basic information they need to interpret data
  3. strives to ensure the correct interpretation of its data releases by the media and the public
  4. establishes metadata requirements for public-use microdata files and custom data products.

Implementation

Statistics Canada undertakes a broad range of specific initiatives to provide, manage and disseminate metadata and thus promote interpretability in its statistical programs. Below is a list of such initiatives, in groups that correspond to the items under Assessment.

L.1 Statistics Canada enshrines metadata requirements in policies and practices

  • Define policies that specify metadata requirements for every statistical product and data release, including the provision of information on data quality and methodology. Responsibility is directly assigned to ensure that these requirements are met.
  • Define a governance for metadata. To ensure that statistical metadata are managed as a corporate resource, governance for statistical metadata needs to be established at the Agency level. This will allow the management of statistical metadata to be coordinated across the Agency, in particular for statistical metadata that cross domains, organizational groups or systems.
    • Metadata Architecture Modernization
  • Define a metadata management structure. The Agency’s metadata architecture includes the establishment of operational governance as well as formalization of centres of responsibility for statistical metadata.
    • Strategy for Statistical Metadata Management
  • Ensure that metadata are aligned with statistical business processing standards.
  • Ensure that metadata are reviewed regularly and updated when needed, particularly prior to data releases.
  • Define processes to ensure that the exchange of metadata between projects is transparent and efficient.

L.2 Statistics Canada provides users with the basic information they need to interpret data

  • Establish an integrated metadata repository that contains the information needed to describe each of Statistics Canada’s data holdings.
    • Directive on Documenting Statistical Metadata
    • Integrated Metadatabase
  • Ensure that metadata are released concurrently with the data product. At release, standard documentation is produced and disseminated such as the Integrated Metadatabase (which publishes definitions, data sources and methods for each survey and statistical program), user guides (which accompanies a dataset release to provide background details on the data and methods), and technical reports (which explain procedures used in the creation of the statistics). Program areas also produce quality reports and other supporting methodological information needed for users to make meaningful comparisons over time and with other sources of information. The above reports provide information on changes to concepts, definitions, classifications and methods, as well as any deviations from international standards and other practices.
  • Provide metadata in formats that are convenient to users and in language that is understood by users and not obscured by statistical jargon.
  • Include the usefulness and adequacy of metadata in discussions with data users and stakeholders.

L.3 Statistics Canada strives to ensure the correct interpretation of its data releases by the media and the public

  • Implement a clear, timely and consistent communications strategy. Such a strategy ensures that the media and the public understand and draw the correct inferences from information in a data release, in particular when they first read or hear about it.
  • Publicly address misinterpretations of the Agency’s data.
    • Policy on Highlights of Publications
    • Directive on Media Relations

L.4 Statistics Canada establishes metadata requirements for public-use microdata files and custom data products

  • Provide a record layout and documentation of concepts and definitions.
  • Provide a “codebook”, data dictionary or other source of coding and classification details.
  • Provide metadata in a format compatible with the data file (i.e., in the same software).
    • Policy on Information Management
Date modified: