Data quality, concepts and methodology: Survey methodology

View the most recent version.

Archived Content

Information identified as archived is provided for reference, research or recordkeeping purposes. It is not subject to the Government of Canada Web Standards and has not been altered or updated since it was archived. Please "contact us" to request a format other than those available.

Archived

This page has been archived on the Web.

The 2009 survey

The 2009 survey collected data on four years. The four years were:

1. 2008 for which the expenditures data are revised, if required, and are considered as final;
2. 2009 for which respondents report actual expenditures,
3. 2010 for which respondents report planned expenditures, and
4. 2011 for which respondents forecast spending intentions.

Estimates are not available from administrative data for 2010 and 2011, and for 2009 are subject to revision. Therefore, estimates are imputed for planned expenditures and spending intentions that are based on the administrative data.

The sample was made up of firms which had reported R&D in the previous surveys, firms claiming an R&D income tax incentive for 2009, firms reported by government respondents as R&D contractors or grantees for 2009 to 2010, firms reported by other firms as funding or performing of R&D, and firms indicated in some other way, such as newspaper or journal articles or provincial directories. These performing and/or funding firms received the Research and Development in Canadian Industry (RDCI) questionnaire, covering R&D performing expenditures for: 2008, 2009, 2010 and 2011. The largest performers by industry group were selected, along with a random sample of small and medium R&D performers.

The 2009 survey was mailed out in August 2010.

Upcoming and recent changes to survey methodology

The RDCI survey is continuing through an ongoing process of change. There have been a series of changes in methodology over the past few reference years and further changes are anticipated. These changes are itemized below by reference year in which they are being or were implemented.

Changes implemented for the 2009 reference year

Data users are advised that the RDCI was formally linked to the Business Register (BR) for reference year 2009. The BR is the survey frame for all industry-based surveys. As part of the linking process some statistical entities which were treated as enterprises on the RDCI frame were determined to be in fact companies on the BR (http://www.statcan.gc.ca/concepts/units-unites-eng.htm). Steps were taken to ensure consistency of the data at the industry level, but there were some impact in the distribution R&D expenditures and personnel at the industry level.

Changes implemented for the 2008 reference year

For reference year 2008 greater use was made of information available from the Business Register. This included information about the business structure of larger firms who performed R&D and who had business operations in multiple provinces. This change also had some impact on the count of R&D performers by province, as information about the structure of the enterprise has been used to allocate R&D expenditures as reported through administrative data across multiple provinces where applicable. Previously, the expenditures and personnel were reported in one province only, based on the province in the address from which the tax records were filed.

For reference years 2008, and those following, the survey sample methodology was revised to improve the quality of forecast estimates at the industry level. The entire population of all known R&D performing firms and firms which fund or purchase technologies were sorted by NAICS-based industrial categories and then divided into the following groups:

1. Special entities were included in a "must-take" stratum. These entities included industrial non-profit organizations, known R&D performers that do not file scientific research and experimental development (SR&ED) tax incentive applications, and technology purchasers or vendors.
2. The largest R&D performers in each industrial category compose a "take-all" stratum. These large R&D performers cover about two-thirds of R&D expenditures in each of the given industry groups.
3. Mid-size R&D performers in each industrial category were placed in a "take-some" stratum, which meant that these firms were randomly selected within each industrial category.
4. The smallest R&D performers in each industrial category were placed in a "take-none" stratum and excluded from the sample so as to reduce response burden. These firms continue to be included in our tabulations as their key R&D data are obtained from CRA administrative data from the SR&ED tax incentive program. Detailed data not directly available from administrative sources, such as level of education of R&D personnel, are imputed.

Changes implemented for the 2007 reference year

For reference year 2007, all firms believed to be performing or funding one and a half million dollars or more of R&D were sent a questionnaire. The sample was composed of firms which had reported R&D in the previous survey, of firms claiming an R&D tax incentive for 2007, of firms reported by government respondents as R&D contractors or grantees for 2007 to 2008, of firms reported by other firms as funding or performing of R&D, and of firms indicated in some other way, such as newspaper or journal articles or provincial directories. These larger performing and/or funding firms received the Research and Development in Canadian Industry questionnaire, covering R&D performing expenditures for: 2006, 2007, 2008 and 2009.

Changes implemented for the 2006 reference year

To relieve respondent burden, the survey threshold was raised from one million dollars to one and one half million dollars in the survey year 2006, thereby reducing the number of surveyed firms. These firms continue to be included in our tabulations as their R&D data are obtained from the Scientific Research and Experimental Development tax incentive program's administrative data.

Beginning reference year 2006, Canada Revenue Agency (CRA) Payroll Deductions total employment data (PD7) was used to improve the quality of missing or inconsistent total employment data for survey years 2001 through the current survey year. Payroll Deduction data are monthly data, therefore an annual average is calculated from CRA monthly Payroll Deduction data for all firms that reported having one or more employees in at least one of the twelve months of the tax year.

Changes implemented for the 2005 reference year

Beginning reference year 2005, revenue figures for the SR&ED tax filers were adjusted to reflect corporate income tax data for the corresponding filer. These tax data are from T2 corporate income tax data mapped to the Statistics Canada Chart of Accounts (COA) classification, by firm, from Tax Data Division. The variable COA4 comprises (Total) Revenue for firms. COA4 values were used to improve data quality for missing total revenues data from reference year 1997 through the current year. Inconsistent reported total revenue data were also examined by subject matter experts with reference to COA4 data. Within the publication, the revisions have impacted the revenue size groups. It is believed the revisions have substantially improved the quality of the revenue variable.

Canada Revenue Agency (CRA) changes to the Scientific Research and Experimental Development (SR&ED) tax forms (2008)

In 2008, the CRA introduced new tax forms for applicants to the Scientific Research and Experimental Development (SR&ED) tax incentive program. These changes have impacted the data produced from the Research and Development in Canadian Industry (RDCI) survey. The new forms went into effect in November 2008. SR&ED applicants have been given the opportunity to use either the new or the old forms for their financial years ending in 2008. Please see the CRA's web-site for copies of the new and old SR&ED tax forms (http://www.cra-arc.gc.ca/E/pbg/tf/t661/).

The CRA changes that impact data continuity include:

1. February 25, 2008, the federal budget provided for a change in the SR&ED tax qualified expenditures for wages and salaries of R&D activities performed outside of Canada that was directly performed by employee(s) of the applicant; "the employee who performed the SR&ED work was a resident of Canada at the time the expense was incurred; the SR&ED work carried on by the employees outside Canada was an integral part and solely in support of the SR&ED work for a project carried on in Canada; and salary or wages paid were not subject to income or profits tax from another country." (Guide to Form T661 – Scientific Research and Experimental Development (SR&ED) Expenditures Claim, http://www.cra-arc.gc.ca/E/pub/tg/t4088/, accessed December 09, 2008).
2. The nature of R&D data are no longer available.
3. The area of specialization of R&D activities (biotechnology, software development, and environmental protection) are no longer available.
4. R&D personnel are not clearly identified as required in full-time equivalent on the SR&ED form.

Other changes to the SR&ED forms which impacted data processing for 2008 reference year are:

1. R&D expenditures are by project rather than program.
2. Selected type of R&D activity by project is included.
3. Field of science has been added.
4. Type of location used for R&D has been added.

For the 2008 R&D expenditures, SR&ED tax data were processed from two forms, therefore, data availability for 2008 are limited when compared with data from previous years.

The forms for the Scientific Research and Experimental Development (SR&ED) tax incentive program were subject to minor changes for 2010 and 2011. One area of change is the "field of science" category assigned to software development. In the 2008 Guide software development was assigned to Computer and information systems, but was moved to Electrical engineering, electronic engineering and Information technology in 2010. For reference year 2010, the data will be mapped to the new category. For reference year 2011, the survey questionnaire will be modified for 2011 to match the tax form.

The survey's history

Data on R&D in the business enterprise sector have been collected since 1955. Until 1969, the survey was biennial. From 1970 to 1981, all known performing or funding firms of industrial R&D were surveyed for odd-numbered years and a sample, including the leading performers, were surveyed for even-numbered years. From 1982 to 1991, a full survey was conducted annually.

Only the top 100 R&D performers (accounting for 64% of all industrial R&D) were surveyed for the 1992 and 1994 reference years. However, as a result of a cost-sharing agreement with the province of Quebec, the 1992 and 1994 industrial R&D survey results also included small firms having R&D activities in the province of Quebec.

From 1991 to 1997, with the exceptions of 1992 and 1994 above, Statistics Canada surveyed all firms known to be performing or funding industrial R&D in Canada. Virtually all of these firms also provided information to CRA in order to claim tax benefits under the Scientific Research and Experimental Development (SR&ED) tax incentive program. In an effort to reduce respondent burden, Statistics Canada stopped surveying the small performing and funding firms (those with less than $1 million of R&D in Canada) and instead, obtains their R&D data using CRA administrative data from the SR&ED tax incentive program. In the 2006 survey year this threshold was raised to $1.5 million thereby further reducing respondent burden.

When first implemented, this administrative data initiative resulted in an understatement of the total value of intramural expenditure and of the total number of R&D personnel. Under the current tax regulations, firms must file their application to the SR&ED program within 18 months of expenditure. Once claims are submitted, they are processed and forwarded to Statistics Canada. As a result, data may not arrive for up to two years after the expenditures were incurred. To remedy the situation, an imputation system was subsequently put into place to impute values for outstanding administrative data. This imputation system confirms the firm is active using Statistics Canada's extensive Business Register, and then applies an imputation based on industry trends.

Recent developments in R&D spending are important economic signals, desired promptly by a variety of users. Because the small imputation of outstanding CRA data does not seriously influence overall trends, the R&D data are published as soon as possible after the survey is conducted, and revised in subsequent publications.

Other notes

Different interpretations of the definition of R&D may also result in discrepancies between federal government reporting of funds to industry (the business enterprise sector) for R&D and industry's reporting of such funds. For example, a federal government department may regard a contract to industry for the building of a prototype (e.g., communications satellite) as R&D. The contractors and subcontractors, however, may only use a portion of the R&D contract and even that portion may not be reported because the contract is considered as part of the firm's "routine" contract work. Differences may also arise for contracts awarded to industry for services or equipment required for a government in-house project which are reported by the federal sponsor as industrial R&D contracts. Therefore, the totals for R&D grants and contracts from the federal government to industry shown in this publication do not agree with those reported in Federal Science Activities, 2009/2010, (Catalogue no. 88-204-X).

The business enterprise sector is the only sector in which data are not collected on R&D in the social sciences and humanities.

In this survey, the sampling unit is the enterprise while the reporting unit may, in some cases, be the company. The survey is designed to reflect the structure of the enterprise as it appears on the Business Register and the structure of the enterprise as it reports its R&D activities (including reporting R&D expenditures for the SR&ED tax incentive program). This procedure creates a problem when classifying data by industry. An enterprise can only be assigned to one industry although that enterprise may have companies or establishments in several industries. The assignment is based on the activity from which the firm derived the greatest portion of its income. Thus, comparisons between R&D data collected at the enterprise or company level and other data collected at the establishment level, such as "census value added", may be misleading. Since industrial R&D is highly concentrated, the use of the company/enterprise as the main reporting unit also means that classification cannot be very detailed, to avoid disclosing individual firm data.

The survey response

For 2009, the response rate was 62%. Survey questionnaires were mailed to 1,985 firms: 1,114 were returned; 162 indicated no research and development activity; 7 were out of business and 10 were duplicates.

An additional 18,301 firms were added to the survey universe from the 2009 Scientific Research & Experimental Development tax incentive program data.

Interpretation of R&D

1. Research and experimental development (R&D) definition: Research and experimental development (R&D) comprise creative work undertaken on a systematic basis in order to increase the stock of knowledge, including knowledge of man, culture and society, and the use of this stock of knowledge to devise new applications (OECD, Frascati manual (2002) http://www.oecd.org/document/6/0,3746,en_2649_34451_33828550_1_1_1_1,00.html).

Therefore, industrial R&D is intended to result in an invention which may subsequently become a technological innovation. An essential requirement is that the outcome of the work is uncertain, i.e., that the possibility of obtaining a given technical objective cannot be known in advance on the basis of current knowledge or experience. Hence much of the work done by scientists and engineers is not R&D, since they are primarily engaged in "routine" production, engineering, quality control or testing. Although they apply scientific or engineering principles, their work is not directed towards the discovery of new knowledge or the development of new products and processes. However, work elements which are not considered R&D by themselves, but which directly support R&D projects, should be included with R&D in these cases. Examples of such work elements are design and engineering, shop work, computer programming, and secretarial work.

If the primary objective is to make further technical improvements to the product or process, then the work comes within the definition of R&D. If however, the product, process or approach is substantially set and the primary objective is to develop markets, to do pre-production planning or to get a production or control system working smoothly, then the activity can no longer be considered as part of R&D even though it could be regarded as an important part of the total innovation process. Thus, the design, construction and testing of prototypes, models and pilot plants are part of R&D. But, when necessary modifications have been made and testing has been satisfactorily completed, the boundary of R&D has been reached. Hence, the costs of tooling (design and try-out), construction drawings and manufacturing blueprints, and production start-up are not included in development costs.

Pilot plants may be included in development only if the main purpose is to acquire experience and compile data. As soon as they begin operating as normal production units, their costs can no longer be attributed to R&D. Similarly, once the original prototype has been found satisfactory, the cost of other "prototypes" built to meet a special need or fill a very small order are not to be considered as part of R&D.

Processes in place to address sources of errors

Coverage

1. Coverage errors which consist of omissions, erroneous inclusions, and duplications in the frame used to conduct the survey. Since they affect all survey estimates, they constitute one of the most important types of error. Coverage errors may translate into a negative or positive bias in the data, and the impact may vary depending on the survey universe subgroup. One should also be concerned about classification errors, notably industrial and geographical, among others. For example, badly defined limits or erroneous coding may lead to an omission of part of the territory. ¹ 

Survey questionnaires are sent to all known and suspected, large R&D performing and/or funding firms i.e., those believed to have the largest R&D expenditures within their industry group.

Administrative data are used for the remaining R&D performing or funding firms which are not included in the questionnaire coverage. Firms have up to 18 months after their fiscal year end to file a Scientific Research & Experimental Development tax incentive program claim for their R&D expenditures. Underreporting due to this time lag is estimated to be less than 8%, and is largely corrected by imputation based on industry trends for all known performers who have not yet submitted their claim.

Nonresponse

1. Nonresponse errors occur when there is no response to one or all of the survey questions. Nonresponse leads to an increase in variance as a result of a reduction in the actual size of the sample and the recourse to imputation, and produces a bias if the nonrespondents have characteristics of interest that are different from those of the respondents. Furthermore, there is a risk of significantly underestimating the sampling error, if imputed data are treated as though they were observed data. ² 

Non-response is a potential problem in three areas. One is the estimate of R&D expenditures two years past the base year. If no response is provided, editing rules are applied and a response is imputed based on the response of a similar firm in the same industry group.

The second involves the administrative data used for the smaller R&D performers. These represent 20% of all R&D performed by businesses. Certain information is not asked of them. However, the missing data are imputed from the replies of the sampled performers in the same industry.

Failure of surveyed firms to reply is the third type of non-response. We believe non-response error to be minor and may result in a minor under-estimation of R&D expenditures.

Nonresponse is addressed through imputation. Automatic imputations are made for the administrative SR&ED tax data portion of the universe as well as for non-response and invalid response within the questionnaire portion of the universe. Imputation of records is based on responses to other questions within the questionnaire, prior year's projections by that firm, tax data and ratios of expenditures or personnel and rates of change in expenditures or personnel reported by other firms in the same industry group.

Measurement

1. Measurement errors occur when the response provided differs from the real value; such errors may be attributable to the respondent, the interviewer, the questionnaire, the collection method or the respondent's record-keeping system. Such errors may be random or they may result in a systematic bias if they are not random. ³ 

As a result of a reconciliation of federal and industrial accounts of government grants and contracts, we think that industrial R&D performance estimates may be slightly low. This is caused by the non-reporting of industrial R&D funded by contract. Such work is sometimes not distinguishable from non-R&D contract work.

The accuracy of the firm's estimates of future expenditures has also been a problem in the past, particularly in the oil and gas extraction industries. Where applicable, adjustments are made to microdata records.

Processing and data capture

1. Processing errors occur at subsequent stages of the process, when checking, coding, entering, imputing, and tabulating data. Like measurement errors, processing errors may lead to variance and bias. It is also necessary to look into the potential impact of snags in the survey process: uneven staff training, unusually high staff turnover, procedural changes in mid-operation, etc. ⁴ 

Processing errors are monitored and controlled using quality control techniques. Significant uncorrected data capture errors are unlikely because of the examination of numerous tables and listings prepared for data validation and analysis before publication tables are created.

Sampling

1. Sampling errors occur when survey results were obtained from a sample rather than the population as a whole. In practice, these errors may also include estimation errors that may be attributable to the use of estimators which, deliberately or otherwise, create a bias (e.g., some small area estimators). ⁵ 

Although a complete enumeration is carried out of known and suspected R&D performing and/or funding firms, records received from the administrative data do not provide as much information as does the sampled universe. Certain data are imputed for records from the administrative file based on ratios in responses received from survey questionnaire respondents in the same industry group.

Technical notes

Data availability

Data for the reference year 2009 are available for all tables with the exception of counts of firms.

In the even years prior to 1982 and for 1992 and 1994, the estimation procedures did not permit the preparation of tables based on revenue size, employment size, sources of funds and country of control of firms.

Regional data on research and development (R&D) expenditures and personnel are only available for 1977, 1979 and 1981 to 2009.

Terminology

The following terminology is used within the publication:

Performing company: is the organization which carried out the R&D. In the case of a consolidated return, performing company could include several companies. It also includes divisions of an enterprise which send separate returns or organizations such as industrial non-profit organizations.

Related companies: Includes parent, subsidiary and other affiliated companies. In the case where a consolidated return is submitted, "related companies" would exclude companies included in the consolidation.

R&D contracts for other companies: R&D contract work performed by the reporting company for other companies.

Federal grants: Federal R&D grants and the R&D portion of any other federal grants; it excludes funds or tax credits from R&D tax incentives.

Federal contracts: Federal R&D contracts and the R&D portion of any other federal contracts.

Provincial sources: Provincial R&D grants and contracts, and the R&D portion of any provincial grants and contracts; it excludes funds or tax credits from R&D tax incentives.

Other Canadian sources: Includes funds from universities and from levels of government other than federal and provincial.

Intramural expenditures: Expenditures for R&D work performed within the reporting company, including work financed by others.

Current intramural expenditures: Labour costs, fringe benefits and other current costs for R&D, including non-capital purchases of materials, supplies and equipment but excluding capital depreciation. Current intramural expenditures also include contracts for services required to carry out R&D (e.g. contracts awarded for drilling needed for heavy oil R&D).

Capital expenditures: Expenditures on fixed assets used in the R&D program, classified into land, buildings, and equipment.

Revenues: Revenues resulting from the sale of products and services (after deducting sales and excise taxes), and other revenues such as those generated from investment and rentals.

Non-commercial firms: R&D performers without a directly affiliated Canadian commercial base. Included are industrial non-profit organizations and trade associations, R&D performed by consortia, and R&D performed by non-residents without associated commercial enterprises and funded principally from abroad.

Country of control: In most cases of foreign control, the country of control is the country of residence of the ultimate foreign controlling parent corporation, family, trust, estate or related group. Each subsidiary within the global enterprise is assigned the same country of control as its parent. A corporation whose voting rights are equally owned by Canadian-controlled and foreign- controlled corporations is Canadian-controlled. If two foreign-controlled corporations jointly own an equal amount of the voting rights of a Canadian resident corporation, the country of control is assigned according to an order of precedence based on their aggregate level of foreign control in Canada. For example, United States takes precedence over all other foreign countries because it has the highest level of aggregate foreign control in Canada.

R&D personnel: Calculated in full-time equivalent (FTE). R&D may be carried out by persons who work solely on R&D projects or by persons who devote only part of their time to R&D, and the balance to other activities such as testing, quality control and production engineering. To arrive at the total effort devoted to R&D in terms of person-years, it is necessary to estimate the full-time equivalent of these persons working only part-time in R&D.

Professional personnel: are researchers or R&D managers.They can be either scientists or engineers.Researchers are professionals engaged in the conception or creation of new knowledge, products, processes, methods and systems and also in the management of the projects concerned.Managers and administrators engaged in the planning and management of the scientific and technical aspects of a researcher's work also fall into this category.

Scientists and engineers: Are professionally trained to conduct investigations or enquiries to acquire a theoretical, abstract or practical knowledge of physical and natural phenomena, improve or develop concepts, theories and operational methods, or apply scientific and technological knowledge relating to fields such as physics, astronomy, meteorology, chemistry, geophysics, geology, mathematics, statistics, computing, architecture, engineering and technology.

Senior administrators and managers: Managers and administrators engaged in the planning and management of the scientific and technical aspects of a researcher's work also fall into this category.

Technical and administrative support personnel: Technicians and equivalent staff are persons whose main tasks require technical knowledge and experience in one or more fields of engineering, physical and life sciences.Other supporting staff includes skilled and unskilled craftsmen, secretarial and clerical staff participating in R&D projects or directly associated with such projects.Both technical and administrative support personnel work to directly support the activities of researchers.

Technicians and technologists: Technically trained personnel who assist scientists and engineers in R&D, e.g. chemical technicians, draftspersons.They may be certified by either provincial educational authorities or by provincial or national scientific or engineering associations.

Other administrative support: Personnel directly engaged in the R&D program, e.g. machinists and electricians in construction of prototypes, or clerks, typists, accountants and storekeepers engaged in the administration or clerical support of R&D firms.

Full-time equivalent (FTE) = number of persons who work solely on R&D projects + estimate of time of persons working only part of their time on R&D.

Example calculation:

If out of four scientists engaged in R&D work, one works solely on R&D projects and the remaining three devote only one quarter of their working time to R&D, then: FTE = 1 + 1/4 + 1/4 + 1/4 = 1.75 scientists.

Federal government funds for industrial R&D: Federal support consists of grants and contracts for R&D to be performed by business enterprises. Taxes foregone as a result of income tax incentives for R&D are not considered direct government support and are not attributed to the federal government.

Industrial classification

North American Industry Classification System (NAICS) is the standard industrial classification system used for presenting R&D expenditures data for the business enterprise sector. There are limitations to its use. One important limitation is due to firms with activities in more than one industry (e.g., firms which both refine petroleum and extract oil). Another is caused by the concentration of the R&D activity among a few firms. In order to prevent disclosure of individual respondents NAICS codes may be combined to provide sufficient observations for publication.

A third problem is that the classification, chosen to represent general industrial activity, may not be entirely suitable for identifying firms chosen only for their involvement in R&D.

There are some restrictions on the application of the NAICS, for example, large R&D performing firms that are classified as "holding companies" are assigned to the principle industrial activity of the firm.

The R&D activities of other sectors such as the federal government, provincial governments, higher education, and private non-profit organizations are covered in other reports.

Definitions

Research and development

For the purpose of this survey, research and development (R&D) is systematic investigation carried out in the natural and engineering sciences by means of experiment or analysis to achieve a scientific or technological advance.

Research is original investigation undertaken on a systematic basis to gain new knowledge.

Development is the application of research findings or other scientific knowledge for the creation of new or significantly improved products or processes. If successful, development will usually result in devices or processes which represent an improvement in the "state of the art" and are likely to be patentable.

Example:

The investigation of electrical conduction in crystals was research. The application of this knowledge to the creation of a new amplifying device - the transistor - was development. The application of the device to the construction of new electrical circuits for television receivers was development. The formulation of new plastic cases for a television receiver is design, not development.

Research and development may be carried out either by a permanent R&D unit (e.g., R&D division) or by a unit generally engaged in any non-R&D activity such as engineering or production. In the first case, the R&D unit may spend part of its time on routine testing or trouble shooting or on some other activities which should not be included in R&D. In the second, only the R&D portion of such units' total activity should be considered.

Research and development should be considered to be "scientific research and experimental development" as defined in Section 37, Regulation 2900 of the Income Tax Act; this section specifically excludes the following:

1. market research, sales promotion,
2. quality control or routine analysis and testing of materials, devices or products,
3. research in the social sciences or the humanities,
4. prospecting, exploring or drilling for or producing minerals, petroleum or natural gas,
5. the commercial production of a new or improved material, device or product or the commercial use of a new or improved process,
6. style changes, or routine data collection

Note:

Although the definition of "scientific research and experimental development" is considered to be the same as R&D, certain expenditures for scientific research cannot be claimed for income tax purposes (e.g., land, building). All expenditures attributable to R&D are included in this report.