Main article

Warning View the most recent version.

Archived Content

Information identified as archived is provided for reference, research or recordkeeping purposes. It is not subject to the Government of Canada Web Standards and has not been altered or updated since it was archived. Please "contact us" to request a format other than those available.

  1. Introduction
  2. The higher education sector
  3. The workshop
  4. Recommendations
  5. Next steps
  6. References

1   Introduction

The higher education sector generates new knowledge through the performance of research and development, and it produces the highly qualified people needed to renew itself and to work in other sectors of the economy as they compete in a global market. Higher education is supported by programmes from all levels of government, by the private sector, private non-profit organizations, and by foreign institutions. Measuring the activities of the sector, the costs of their performance, and the sources of their funding, provides a means of monitoring the activities and informing policy decisions and public discourse. One of the most important activities of the higher education sector is the performance of research and development (R&D).

Higher Education Research and Development (HERD) is a third of Canadian Gross Domestic Expenditure on Research and Development (GERD), and the $10 billion of expenditure places Canada in the top ranks of countries in the Organisation for Economic Co-operation and Development (OECD), based on the HERD/GERD ratio.

This dominant position results from a series of government programmes introduced over the last decade to promote R&D in the higher education sector, with a view to increasing the production of new knowledge and the development, attraction and retention of world class researchers. These programmes include the Canada Research Chairs (CRCs), the Canada Foundation for Innovation (CFI) and the Indirect Costs Program (ICP). In addition, the budgets of the granting councils have increased over the last decade significantly and the Canadian Institutes of Health Research (CIHR) has replaced the Medical Research Council as the principal funding body for health research.

In view of the policy interest, HERD, and its funding sources, have become key statistics for monitoring the effects of the policy interventions and they are also able to support evaluation of the policies. HERD is an indicator of high policy relevance, a fact supported by the discussion of Canada's position in relation to other OECD countries which has raised questions about the methods used to estimate HERD. The response to those questions must be a clear statement of how the data quality of the indicator is to be maintained by Statistics Canada, taking account of the significant changes since the last revision a decade ago. This is not a matter of imposing burden upon the higher education sector through surveys, but of the intelligent use of administrative data and collaboration with the principal supplier of the that data, the Canadian Association of University Business Officers (CAUBO).

2   The higher education sector

The higher education sector is composed of all universities, colleges of technology and other institutes of post-secondary education, whatever their source of finance or legal status. It also includes all research institutes, experimental stations and clinics operating under the direct control of, or administered by, or associated with, the higher education establishments (Statistics Canada, 2009).

While the bulk of the R&D performance is found in the universities, research is also done in teaching hospitals and in colleges. Understanding research in hospitals is an important contribution to monitoring and evaluating policies on health research. The colleges contribute to economic competitiveness through their relations with industry as part their R&D activities. While HERD estimates are used for the monitoring of policies of governments, they do not provide impact measues of the consequences of funding or engaging in research and development. That is quite another issue.

There are various methods for estimating expenditure on research and development and the sources of funds, such as surveys of the budget allocation of R&D performing institutions, time use surveys of personnel engaged in research to determine the percentage of their time allocated to the task, and there is the use of administrative data. Statistics Canada, following the guidelines in the Frascati Manual of the Organisation for Economic Co-operation and Development (OECD, 2002), makes maximum use of administrative data in order to minimize the reporting burden on institutions in the higher education sector. The principal administrative data source of the data, as already indicated, is the CAUBO survey. Within Statistics Canada, some data collected for other purposes on universities by the Centre for Education Statistics and on hospitals by the Health Statistics Division are used in the HERD estimation.

Any statistical mechanism for producing estimates must be reviewed at regular intervals to ensure the quality of the estimates. In the case of the HERD methodology, the last review was a decade ago (Brochu, 2000) and there have been significant changes in funding and ways of performing R&D since then. As a result, the technical workshop on Estimates of Research and Development in the Higher Education Sector (HERD) was held in Ottawa on October 16, 2009 as a first step in assessing the magnitude of any needed revision to the estimating procedures and their implications for data quality. 1 

3   The workshop

The workshop gathered experts and data users to review and advise on the methods used to estimate HERD. The agenda is provided in Appendix 1 and the participants are listed in Appendix 2. The participants came from universities and colleges, from granting councils and from provincial and federal government departments. CAUBO was represented as were the universities by the Association of Universities and Colleges of Canada (AUCC) and the polytechnics by Polytechnics Canada.

The presentations included a review of the issues (Hamdani, 2009) and of the recommendations of the last working group to address the revision of the estimation procedures (Brochu, 2000). These presentation introduced the components of the problem such as the treatment of sponsored research and non-sponsored research, the latter using a 1982 CAUBO guideline (CAUBO, 1982) for the estimation. Current and capital expenditures for research were reviewed along with the operating expenditure related to capital investment and the overheads incurred by current or capital funding of R&D.

In round table, all participants were able offer comments and to make recommendations for future work which are presented in the next section.

As one of the changes in the last decade has been a decision by the United Nations (UN) Statistical Commission to treat R&D expenditure as a capital expenditure in the System of National Accounts, some time was spend on reviewing the implications of this and on broader methodological issues which led to additional recommendations.

The slides used by all of the presenters are in Appendix 3.

4   Recommendations

4.1  General recommendations

Workshop participants recommended that:

  1. The HERD estimation model be revised to maintain the quality of the estimate and that discussions be held with CAUBO on possible changes to survey questions;
  2. The estimation model be simplified, as part of the revision process, and that all assumptions used in the model be based on clearly stated principles and published in the documentation of the model;
  3. The terms used to describe the model, such as 'support and maintenance', 'overheads' and 'indirect costs', be defined;
  4. The model description, the terms and the definitions be published and, ideally, with sufficient detail to allow the reader to reproduce the estimates from data in the public domain; and,
  5. The model include estimates of sponsored and non-sponsored research and of indirect costs for all universities, hospitals, and colleges.

4.2  Detailed recommendations

At a more detailed level, participants recommended that:

Data sources for R&D performance and funding

  1. The estimation should take a performance approach to HERD, followed by an estimate of source of funds (It was noted that CAUBO takes a funding approach and that this would have to be resolved.);
  2. The responses to the CAUBO questionnaire be reviewed to probe consistency of response and to raise the possibility of working with CAUBO to address any inconsistencies;
  3. Data sources on R&D in hospitals, both those covered by CAUBO and those not, be reviewed;
  4. The annual reports to the Indirect Cost Program be reviewed to see if they could be used in their present form, or in a revised form, to provided information for the estimation process without adding additional response burden on the higher education sector;

Salaries of researchers

  1. The methods of estimating researcher time use be reviewed, including the possibility of a time use survey, of taking the estimates from collective agreements compiled by CAUBO, and of any other methods;
  2. The salaries of research participants, other than those of professorial rank, be considered for inclusion in the HERD estimates;
  3. Sources of salary, or additions to existing salary, such as the Canada Research Chairs program, be examined to determine their impact on HERD estimates;

Indirect costs

  1. The 5% estimate for indirect costs be reviewed;
  2. Capital expenditure, operating costs and indirect costs involving grants from the Canada Foundation for Innovation, and matching grants, be treated consistently;

New working methods and short term interventions

  1. The impact of the Networks of Centres of Excellence and, more generally of collaborative research, on HERD estimates be reviewed;
  2. The impact of the Knowledge Infrastructure Program (KIP) on HERD estimates be reviewed; and,

International comparisons

  1. The issues affecting international comparisons of HERD estimates and HERD/GDP ratios be reviewed and documented.

4.3  Establishment of a working group

Participants recognized the complexity of the recommendations and the need to proceed in a coherent manner to a revised estimation procedure. To support the process they recommended that:

  1. A working group be established to review the recommendations of the workshop participants and to recommend and cost the implementation of a revised method for estimating HERD and its related variables.

5   Next steps

The establishment of a working group to review all of the recommendations in Section 4 and to recommend the means for addressing them is not without cost as the Working Group would require a commitment of time from Statistics Canada personnel and from participants from outside of the agency. It would also require the support of a Secretariat to gather materials and to produce the report. However it need not be as comprehensive in its review as the previous working group (Brochu, 2000) as some parts of the estimation procedure have not been influenced by the changes over the last decade.

Two broad principles could guide the working group. The first is to address those parts of the estimation procedure that have been most affected by changes to funding of higher education and which are likely to have the most impact on HERD. The second is to follow recommendation 4.1.2 and to simplify the estimation procedure so that it, and the data supporting it, are more accessible to the higher education community.

The report of the working group should include cost estimates and options for revising the method of estimation. Implementing the recommendations of the working group could involve substantial costs and its terms of reference, were it to be established, should include a requirement to propose work plans of different time durations and with different cost implications for Statistics Canada. This exercise would have to be guided by the priorities of Statistics Canada.

The Chief Statistician of Canada (Sheikh, 2009) has made clear that the priorities of Statistics Canada for data are: relevance; quality; and, efficiency in their production. There is no question about the relevance of HERD statistics to the public discourse in Canada and internationally. The significant changes in the last decade to the means of support and performance of R&D in the higher education sector pose a threat to the quality of the estimates used in the public discourse. Finally, all of the recommendations assume that the estimates will continue to make extensive use of administrative data to ensure a minimum of burden upon institutions in the sector and efficiency in the use of administrative data in the estimation of the HERD components.

6   References

Brochu, Mireille (2000), A Framework for Enhanced Estimations of Higher Education and Health R&D Expenditures, Catalogue no. 88F0006XIB-00003, Ottawa: Statistics Canada.

CAUBO (1982), Report of the Study on the Costs of University Research, Ottawa: CAUBO.

Hamdani, Daood (2009), A Review of Estimates of Research and Development Expenditures in the Higher Education (HERD) Methodology, Ottawa: Statistics Canada.

OECD (2002), Frascati Manual: Proposed Standard Practice for Surveys on Research and Development, Paris: OECD.

Sheikh, Munir A. (2009), A Long-Term Vision for Statistics Canada, Ottawa: Statistics Canada.

Statistics Canada (2002), Statistics Canada's Quality Assurance Framework 2002, Catalogue no. 12-586-XIE, Ottawa: Statistics Canada.

Statistics Canada (2009), 'Estimates of Research and Development Expenditures in the Higher Education Sector, 2007/2008', in Science Statistics, Catalogue no. 88-001-X, vol. 33, no. 5, Ottawa: Statistics Canada.