Chapter 1.5: Quality Management

1. Quality management governance

In order to champion quality management initiatives, statistical agencies or systems should consider establishing a neutral focal point or resource centre for quality management that is supported and empowered by senior management. Having such a quality unit ensures that quality remains a constant priority and that quality assurance mechanisms and tools are continuously improved upon to meet the demands resulting from a changing environment.

It is with this perspective that Statistics Canada has created a quality unit called the Quality Secretariat, whose mandate is to promote and support the use of sound quality management practices across the organization. The role of the Secretariat is to

1. support the development, revision and implementation of key quality management documents, such as a quality assurance framework, quality guidelines, and policies;
2. support the development and implementation of new quality improvement initiatives or procedures;
3. promote sound quality management practices;
4. provide advice and assist programs as regards compliance with good practices that support quality;
5. support corporate management in the preparation of performance reports on quality;
6. answer requests from other statistical agencies for information or assistance related to quality management.

The activities of the Quality Secretariat are overseen by the Methods and Standards Committee, whose role is to act as the focal point for the review and monitoring of corporate data quality practices and initiatives and to ensure that these activities are coherent with other management frameworks and policies. The Methods and Standards Committee reports to Statistics Canada's most senior management committee, the Executive Management Board. Through this governance structure, the Quality Secretariat is neutral with respect to statistical programs, and is supported by senior management. Neutrality ensures that quality assurance tools are uniformly relevant across statistical programs, and senior management support guarantees compliance.

It is also a recommended practice to have a quality unit or focal point within statistical programs. The objective is not to absolve all other team members from taking responsibility for quality assurance. On the contrary, the responsibilities of a program-level quality unit include the following:

• encouraging and monitoring quality assurance activities;
• monitoring and interpreting performance indicators;
• investigating quality issues when they arise;
• staying up-to-date on quality-related policy instruments and how they apply to the program;
• liaising with the Quality Secretariat; and
• tailoring quality management practice,s as necessary, to the subject-matter particularities of the program.

2. Quality Management Framework and Guidelines

Although various quality management approaches, models and frameworks exist within the statistical community, systematic quality management within a national statistical office or system usually takes the form of a national quality assurance framework (NQAF). A NQAF is typically an overarching framework that provides context for quality concerns, activities and initiatives, and explains the relationships between the various quality concerns. Recognizing the importance of such a document in quality management, the statistical community has asked an expert group to prepare a Template for a Generic National Quality Assurance Framework and some accompanying guidelines to assist national statistical systems in developing, implementing and enhancing their NQAF. While the template should not be approached as a one-size-fits-all framework, it provides examples of elements required to manage quality at four levels: statistical system, institutional environment, statistical processes and statistical outputs.

Statistics Canada's first Quality Assurance Framework was published in 1997. It was followed by a second version, in 2002, and a third one in 2016. As the statistical environment is not static, the subsequent versions of Statistics Canada's quality assurance framework have taken into account the evolution of the organization's programs, methodology, technology and policies, as well as the progress made internationally in quality management practices.

While this document was originally intended for reference and training purposes, it has become central to developing a quality culture within the organization and in providing a baseline to continuous quality practices improvement. It also contributes to greater transparency and, therefore, reinforces the image of the organization as a credible source of statistics. Finally, it allows Statistics Canada to share ideas and best practices on quality management nationally and internationally.

Another key document in managing quality at Statistics Canada is the Quality Guidelines.^{Endnote 2} The first version was published in 1998, followed by new versions in 2003 and 2009. These guidelines are expected to be revised periodically as methodologies and approaches evolve. They are highly valuable in terms of ensuring the use of sound and consistent methodologies throughout the various programs. The focus is on how to assure quality with respect to the effective and appropriate design or redesign of a statistical project or program from inception through to data evaluation, dissemination and documentation. The guidelines provide strategies and tools to minimize the risk of introducing sampling and non-sampling errors in the survey process and to address some of the quality concerns related to the use of administrative data.

In 2009, Statistics Canada developed and adopted a reference model describing the sub-processes in the production of statistical products based on the Generic Statistical Business Process Model (GSBPM) developed by the United Nations Economic Commission for Europe (UNECE), whose membership includes non-European countries, such as Canada and the United States. This process model, which was revised in 2013,^{Endnote 3} is now used to establish and communicate quality assurance practices for particular activities in the production of statistical products; it is also used for reference and training. The GSBPM will be applied to the development of future versions of Statistics Canada's quality guidelines.

It is important to mention that the array of possible quality measures proposed by the quality guidelines are not necessarily applied uniformly to every program: program managers have the responsibility to determine which measures should be applied and to make these choices very explicit. However, most programs should be subject to a periodic, independent and systematic quality review to ensure proper checks and balances.

Most recently, in September 2015, the Organisation for Economic Co-operation and Development (OECD) Council on Good Statistical Practice issued draft recommendations.^{Endnote 4}

From an OECD perspective, the quality of its statistics and analytical work depends largely on the quality of official statistics produced and transmitted by countries. This draft Recommendation would provide a common reference against which the quality of national statistical systems can be assessed. In addition, the good practices contained in this draft Recommendation would also serve as a reference for the statistical work done by the OECD. Further, the draft Recommendation and its good practices would be referred to in the context of applying the Organization's Quality Framework and Guidelines for OECD Statistical Activities.

3. Quality Assessment and Reporting Tools

Quality can be assessed through multiple means: self-assessments, audits, peer reviews or certification processes. This work can be performed by internal or external experts. Assessment processes can lead to the awarding of an official statistics or certification label. The breadth, depth and length of these evaluations can vary depending on the scope. However, the objective of all quality assessment is more or less identical: "the identification of improvement actions or opportunities in processes and products."^{Endnote 5}

Quality assessment at Statistics Canada has evolved. There has been a long tradition of assessing statistical quality at Statistics Canada. Through the 1990s and early 2000s, this was done by means of the Quality Assurance Framework. User feedback has also been an important feature in quality assessment at Statistics Canada, directly reflecting the relevance, coherence, accessibility, interpretability and timeliness of statistical products. In 2006, the agency started conducting quality assurance reviews, which were aimed at identifying risks to data quality, and sharing good practices to mitigate those risks. Independent program evaluations conducted by a specialized internal organizational unit that follows a formal protocol to evaluate relevance, quality and efficiency were reinitiated in 2011, following a period during which this function was carried out by program managers as part of their ongoing responsibilities (see Chapter 2.8: Program Evaluation). Recently, beginning in 2015, the focus has been on strengthening the quality assessment aspect of Program Evaluation, particularly with regard to how the implementation and execution of statistical programs is assessed.

Quality reporting is also an important aspect of quality management. First, product quality indicators shared with data users allow them to determine whether the statistical product fulfills the requirements of their intended use. Second, process and intermediate product indicators can be used by producers and managers to monitor data production and improve quality practices on an ongoing basis. Finally, at the corporate level, quality performance indicators allow an organization to assess how well it is performing on the quality front and to address any risks that might prevent it from successfully fulfilling its mandate.

Statistical agencies communicate about quality, using quality indicators and quality reports. At Statistics Canada, quality indicators and reports are produced by the various programs during data production. While the type of quality indicators produced internally might differ from one program to another, the adoption of the GSBPM and a corporate business architecture will serve to rationalize the development of a common set of quality indicators for data production in the near future.

Quality reporting for both internal and external audiences is standardized in both its structure and content through the Standard on Statistical Metadata. Quality of statistical products is communicated to users in a common format prescribed by the Policy on Informing Users of Data Quality and Methodology (see Chapter 4.3: Management of, and access to, metadata).

Finally, in order to know how well Statistics Canada performs in terms of quality management, performance measures have been developed. The corporate suite of performance indicators includes several indicators of product and process quality (for the detailed list of performance indicators, refer to Appendix A). A dashboard will allow program managers to view their performance indicators at any time. They can tailor their display to compare performance against targets and to monitor levels. Corporate-level indicators required by Treasury Board Secretariat—the  federal department responsible for overseeing corporate reporting and performance measurement— are produced on a quarterly and annual basis.

In the context of decentralized statistical systems, quality assessment processes can also be a tool to improve the quality of statistical products across the system. Box 1.5.2 describes how Colombia's National Administrative Department of Statistics implemented a quality certification process for all official statistics.

4. Other management frameworks contributing to quality enhancement

Most operations and functions of a statistical agency have an impact on the quality of the agency's information. The management of quality is therefore an integral part of the management of almost every statistical activity carried out by the agency and an important component of the agency's management as a whole. Frameworks are used to manage a statistical agency's quality, human resources, financial resources and overall performance towards achieving its objectives or mandate.

The extent to which a statistical agency can fulfill its mandate and related objectives depends on its ability to optimize its management and operations through organizational efficiency. A significant feature of the management of quality is the balancing of quality and quantity objectives against financial and human resources constraints. These trade-offs, while inevitable in the real world, must be made as explicit as possible so that users can understand the limitations of certain data and why they exist. Ethical standards, fundamental values and principles, and utmost transparency should guide the personnel of a statistical agency in fulfilling their official duties and responsibilities. These principles serve to maintain and enhance public and user confidence in the integrity of the agency. To achieve its mandate and objectives, a statistical agency needs an effective governance and management structure, one that integrates strategic priority setting and decision making and ensures accountability.

At Statistics Canada, quality-management strategies are coherent with the mandate and objectives of management committees for such things as information management, communications and dissemination, administrative data management, collection planning, corporate business architecture and human resources management. Ultimately, this coherence is achieved through strong governance mechanisms, including coordination of all management committees through the Executive Management Board and clear and consistent promotion of corporate priorities and strategies.