Principal risks

View the most recent version.

As noted in Section 2, one of the important conclusions that can be drawn from the Quality Assurance Review is the fact that all the managers interviewed know about the main risks that their program faces in the short and medium term.  The review also confirmed that the programs were well managed and that the program managers tried to produce and publish the best detailed information while minimizing the risks to their program. Nevertheless, some programs are more at risk than others.  The Consumer Price Index, the Balance of Payments, and the International Trade programs are overall more at risk than others and should receive more immediate attention. At the other end of the spectrum, the Labour Force Survey and the Quarterly Income and Expenditure Accounts have a low overall risk.  The remaining programs present moderate overall risks and could be addressed through the adoption of best practices and specific measures directed to the particular needs of the program.

Examining the programs in detail, the review teams categorized the risks in five categories:

1. risks related to people (e.g. shortage of staff, expertise, succession issues);
2. risks related to the timeframe (for certification, last minute changes, etc.);
3. risks related to the systems (e.g. complexity, insufficient testing or documentation, etc.);
4. risks related to the service areas (e.g. lack of control on turnover, reduction of services) and;
5. risks related to input data (e.g. administrative data).

Obviously not all risks have the same importance or the same impact for each program.  Some of the risks need more attention in the short term in order to reduce the potential of errors.  Others need to be addressed in the medium term as they do not immediately affect the program, but could potentially increase the risk of errors if nothing is done. Finally, there are those risks that can be managed and minimized on an ongoing basis. 

What follows is a summary of the main type of risks identified during the review.

Risks related to people

As mentioned in Section 3, the most frequently mentioned risks were related to human resources issues. Within this general category there are a number of sub-issues.

General staff shortage

One issue is simply an overall shortage of staff in the Agency. Program managers, with the single exception of IEA, all seem to be short of sufficient people to do the work.  As one example, the MSM section of Manufacturing, Construction and Energy Division nominally has ten employees: one ES-6 Chief, one ES-5 Head of analysis, two ES-4 analysts, three SI-03 or SI-04 subject matter specialists, an SI-5 Production Head, an SI-03 production officer, and a CR-04 operations clerk. In actual fact, however, the ES-05 is on full-time language training, one of the ES analysts is a recruit, who changes every eight months, one of the three SI subject matter specialists is loaned to the Business Condition Survey once every quarter, and the SI-05 Production Head position is vacant.

Recommendation 16: At the time of the drafting of this report, efforts were being made at the Agency level to increase the intake of new employees. Nevertheless, it will take some time to alleviate the pressure of staff shortage on programs. In the meantime, mission critical programs should receive priority in staffing and prioritization of work should be done in order to identify what activity can be expended in the short term.

Stress of monthly production programs

A second issue is the difficulty in finding people to work on monthly production programs. Most of the monthly programs mentioned that they had difficulty in attracting people to work on a monthly survey because of a perception that such positions are grueling and have a poor work-life balance.  This issue was particularly acute for the LFS, GDP, IT and MRTS programs.  In addition to the difficulty in attracting resources, these programs also have difficulty in retaining their staff as there is a perception that their effort is not recognized by the organization and that these employees do not perform well in generic competitions because they have little time to prepare.
 
Recommendation 17: More recognition and resources should be given to the monthly production programs. Recruits should be encouraged to spend some time in a monthly production program to change the perception. The benefits of working on a regular production survey should be better disseminated.

Shortage of specialized staff

A third issue is the shortage of specialized staff.  While for most programs there is a good balance in terms of the mix of people running the various steps of data production, programs such as LP, GDP, BOP, IT and the CPI have a greater need for highly specialized personnel. This is due in part to the complexity of the program and the type of research and analysis necessary to certify the data.  For these programs the certification and validation often rely heavily on the expertise and experience of the analyst. In the case of the BOP, there is a need to understand how transactions are made on the international capital markets and who are the main actors involved. The review found that there is relatively little documentation in BOP explaining how certification is done and therefore the program relies mainly on the expertise of its specialized staff.  This type of staff is difficult to replace and often has to be trained on the job. The CPI and IT programs are also confronted with similar issues as there are very few university programs that provide index theory as a discipline and recruits have to be trained in-house. In the case of the IT program, their prices program is dependent on one senior analyst.²

In addition to the shortage of specialized staff, their number in the some of these programs is simply not sufficient to fully research and keep abreast of what is happening in the economy.  In the case of the Labour Productivity program, the total number of people dedicated to the program has decreased from 10 to 7, leaving the program without any breathing space in case of illness or emergency.  In the case of the BOP, the program does not have the specialized resources to investigate increasingly complex transactions on the international market. The same remarks apply to the CPI.  The world has become more complex and the demands on many programs such as the CPI, IT, BOP and LP have increased substantially over time and there are not enough analysts to review the concepts, methodologies and data sources and at the same time certify the monthly or quarterly analysis. This situation could explain in part the delay in finding the error in the Traveler Accommodation Price Index.

Recommendation 18: We recommend the expanded use of communities of practice, asset qualifications and other mechanisms in our staffing processes in order to ensure an adequate supply of specialists for programs such as the CPI, BOP, LP and IT. 

Mix of people dedicated to production, analysis and certification

A fourth issue is related to the mix of people dedicated to production, analysis and certification.  The review teams found that the good quality assurance of a program depends on the existence of a research and analytical capacity independent of the production operations. People dedicated to production operations do not necessarily have the preparation or the time to question the data they are processing. That function is better served by an independent team of analysts with the mandate of challenging the numbers and comparing them with other series, keeping abreast of the market intelligence and understanding the research and policy questions related to the statistics produced.  The review has identified a variety of good models along these lines in IEA, LP and LFS.  However, it also found that there was little or no analytical capacity in the CPI and IT.  In the case of the CPI, the lack of comparable external indicators at the certification stage prior to data release limits the scope for reasonableness checks. This is a case where additional analytical capacity would allow more research on external sources and better use of its own data. The same comments apply to the IT program where there is a need to ensure that a thorough review of data for exports is done on a monthly basis.

This issue is addressed by recommendation 4.

Succession planning

Each time a key resource leaves, there is a loss of knowledge.  Given the shortage of resources mentioned previously, it has become difficult for most programs to prepare succession plans. In addition, some programs experience difficulties in finding experienced analysts, which puts them increasingly at risk. One example is the Chief position in the GDP program that is in charge of a highly visible monthly program that covers the whole economy. The experience that is needed for such a position can only be acquired after many years of training and experience in the division.

Even if the issue of succession is generalized throughout the Agency the risk remains moderate with the exception of three of the programs identified earlier (IT, BOP and CPI), where the risk is higher.

Recommendation 19: In order to reduce risks associated with the departure of experienced staff, each program should develop a specific succession plan for key functions in the program. The plan should include training, rotation of staff within the community of practices and some job shadowing when retirements can be identified.

Risks related to the timeframe of the program

Reduced time for certification and publication of the data

Another major issue encountered in all programs is the very short time frame available to the staff to certify the data to be published.   The time allocated to certify the data varies from a couple of days in some programs to 5 days in others.  In the case of the DCD operations, the window is often 24 hours, which leaves very little time to correct mistakes.  With the increasing difficulties of getting responses from the respondent, survey programs such as MSM and MRTS have sometimes extended their collection cut-off dates in order to increase the rate of response, but at the expense of the time that is allocated to the certification. This also has some repercussion on programs in the National Accounts which depend on the information from the survey areas.   In the case of the IT program the pressure comes from the fact that they must align their publication dates with their American counterpart.  Last year the Census Bureau improved the timeliness of its publication by six days. This has meant that there are six days less for certification for the IT program, as the data coming from the Canadian Border Services Agency (CBSA) does not come earlier.

There is a real trade-off between timeliness and the other dimensions of quality, particularly accuracy.  Statistics Canada’s main indicators are generally published a bit later than those of some other OECD countries, but it publishes more detail and its revisions are much smaller.  The review team is of the opinion that this trade-off is a real issue and that any further attempt to improve timeliness should give serious consideration to the effect on risks to other dimensions of the quality of the estimate.

Recommendation 20: Under current conditions, proposals to further increase the timeliness of the programs should be regarded with extreme caution, especially if there is a risk that the time would be taken from the certification or data release steps.

Last minute changes

Another issue related to the timeframe in which programs operate is the frequency with which programs have to make some last minute changes before publication.  These changes are often due to the late arrival of information from the survey or from external input data files, corrections to the data or system problems.  Most programs we reviewed experienced some last minute changes on a fairly regular basis. Although the risk is moderate for most programs, it is higher for the IT, CPI and MRTS programs. These last minute changes have a particularly acute effect on the Dissemination and Communications operations, due to the number of places in which changes are required (The Daily, CANSIM, IMDB, publications, etc.)

While some last minute changes are unavoidable for the reasons given above, and while programs are generally aware that last minute changes significantly increase the risk of errors, we believe that many programs could do a better job of having contingency plans in place to deal with the situation when it arises. This could include such things as contact lists, a list of cutoff times, alternative dissemination plans, and so on. We do note that Dissemination and Communications does produce a contact list for its various services as part of its Business Continuity Plan, but there was some concern that the subject matter divisions were not as well-trained as they should be on how to use it.

Recommendation 21: All programs should have contingency plans, developed in concert with Dissemination and Communications, to deal with unplanned last minute changes.

Risks related to systems

Risks due to the complexity of systems

One issue that has surfaced in some programs is the complexity of the interactions between systems.  For the BOP, CPI and IT programs there are often more than two or three systems that need to interact to produce the output of the program. This was not created by design, but because of the evolution of the program towards more complex operations as an answer to a more complex world.

Each time data moves from one platform to another there is a risk. For example, Disseminations and Communications have identified some 19 different interdependencies in their operations.  An example of this is the fact that the text of The Daily is produced by the team of the Communication Division while the graphics appearing in the same Daily article are prepared by the Dissemination Division. When a last minute change occurs, the only safeguard is the great care taken in the chain of communications. We do note that Dissemination and Communications are aware of this issue and have begun to document and control the interdependencies.

In addition to the multiple systems involved in the production of data, there is often only a very small technical team that understands how they operate. For instance, there is only a small group of staff in ITD with experience in using the FAME application. This limited expertise restricts the division’s ability to manage the monthly production process and to cope with unplanned events. The same comments apply to the MSM program which depends on one person for knowledge of their systems and to SMART publishing in DCD which depends on a handful of individuals.

Recommendation 22: Each program should review its interdependencies among systems and prepare a plan with its associated costs to reduce it. It should be noted that the present mainframe migration is also an occasion to reduce the interdependencies of systems.

Risks due to manual processing

A related issue is the fact that in many programs there are still significant manual operations. For instance, the CPI production requires considerable manual intervention and is fairly paper-based. For the US exports, one employee makes all the updates directly to the database using paper listings. Similarly, the main LFS numbers in CANSIM are verified manually. These situations augment the chances of errors and limit the ability to detect errors efficiently.  Programs should be encouraged to examine their manual operations and prepare proposals for converting these to automated operations where feasible (see recommendation 1).

Risks due to insufficient testing and documentation

Finally under the system related risks, the review team has identified cases where there is insufficient testing and documentation.  Proper testing of the Matrix Processing System (MPS) which was used to calculate the Traveler Accommodation Price Index would had avoided the incident with the CPI.  Lack of documentation is also a risk factor when programs are running their production on legacy systems. This is a higher risk for the IT and the MSM programs.

By contrast, the LFS demonstrated some excellent testing procedures, including monthly end-to-end testing, use of dummy data with known results, and deliberate redundancy in software development (e.g., having two programmers code the same application and compare the results).

Recommendation 23: Programs should ensure that they conduct thorough testing and prepare complete systems documentation.

Risks related to service areas

The programs expressed some concerns related to their dependencies on service areas. We do note that the review teams did not have sufficient time to explore these issues with the service areas themselves.

One issue identified during the review was the lack of control of the programs over the turnover that takes place in the service areas. There are two aspects to this issue. 

The first one has to do with the rotation of systems staff or of methodologists.  While subject matter programs recognized the benefit of rotating staff, it must be done in an orderly manner and should also take into account the particular HR situation of the program.  Given that there is always a learning curve for new systems people or methodologists assigned to a monthly program, transitions should be carefully planned.  Given the staff shortage this is even more important.

The other aspect of the issue has to do with the fact that monthly programs never have a large span of free time between production cycles and cannot go unattended. It is not rare that the use of leave with income averaging in service divisions has some ripple effects on the subject matter program.  While reviewing the LFS, it was noted that during the summer months it is difficult to obtain the same level of service required and this can put the program at risk.

Recommendation 24: Service areas should ensure that better planning and communication protocols are in place to avoid leaving programs short-staffed. Also it is suggested to rotate leave periods throughout the summer.

Risks related to provision of input data from other parties

During the review, some programs indicated that they have less time for certification and validation because they have to compensate for the reduction in service in OID and SOD.  For these programs and in particular for the MRTS and LFS, it meant that they had to spend more time on the collection step of the process and less time on certification. Similarly, the programs of the National Accounts have noticed that the survey programs seem to have less time or no time at all to examine the data from a macro perspective before sending it to the National Accounts.  The impact of this is that the programs in the Accounts have to spend more time certifying the input data and have less time to do the coherence analysis and the balancing.  None of this has been identified as a high risk, but if it is not managed it could generate errors. 

In the case where programs depend on external input data, there has also been some risk associated with the lack of control on what is sent to the program.  In the case of the IT program, the dependency on external data sources is significant and changes in policies and practices by the Canadian Border Services Agency (CBSA) could lead to data quality problems. The error in 2004 with the International Trade figures occurred because of the lack of control and monitoring of the input data provided by the CBSA. Similar situations exist with the BOP program with regards to information on international transactions and the MRTS program with regards to GST data from Revenue Canada. 

Recommendation 25: Formal protocols should exist at a senior enough level to ensure that data that are critical to programs are provided on a timely basis and that changes at the source of the data are communicated.

Footnotes

1. As of the drafting of this report, an additional resource has been re-allocated to the prices program of IT.