Data quality Data quality

2016 Census of Agriculture — concepts, methodology and data quality

The following information will ensure a clear understanding of the basic concepts that define the data in this product, and of the underlying census methodology and key aspects of the data quality. The information will help clarify how the data can effectively be used and analyzed, taking its strengths and limitations into account. This information may be particularly important when making comparisons with data from other surveys or other sources of information, and in drawing conclusions regarding changes over time.

Data sources and methodology

The Census of Agriculture collects and disseminates a wide range of data on the agriculture industry, including the number and type of farms, farm operator characteristics, business operating arrangements, land management practices, crop areas, the number of livestock and poultry, farm capital, total operating expenses and receipts, and farm machinery and equipment. Census data provide a comprehensive picture of the agriculture industry across Canada every five years at the national, provincial and territorial levels, as well as at lower levels of geography.

General methodology

Target population

The target population for the Census of Agriculture comprises all ‘census farms’ in Canada. In 2016, a census farm was defined as an agricultural operation that produces at least one of the following products intended for sale: crops (hay, field crops, tree fruits or nuts, berries or grapes, vegetables, seed); livestock (cattle, pigs, sheep, horses, game animals, other livestock); poultry (hens, chickens, turkeys, chicks, game birds, other poultry); animal products (milk or cream, eggs, wool, furs, meat); or other agricultural products (Christmas trees, greenhouse or nursery products, mushrooms, sod, honey, maple syrup products). The definition of a census farm has, however, changed over time; for a summary of changes since 1921, please refer to Census farm.

Establishments on Statistics Canada’s Business Register that responded to the 2011 Census of Agriculture or had reported and confirmed agricultural activity in a recent Statistics Canada agriculture survey constituted the initial observed population who received a Census of Agriculture questionnaire. In addition, establishments that have an indicator of agricultural activity in their tax remittances or that have reported agriculture as their main business activity to the Canada Revenue Agency also received a census questionnaire. However because of operational and budgetary constraints, only those establishments with indicators showing a high likelihood of being a farm were included. During collection, potential farms identified through the Census of Population questionnaire were also added to the observed population.

The 2016 Census of Agriculture target population remained unchanged from 2011, but the observed population has changed. In 2011 the observed population only consisted of agricultural operations included in Statistics Canada’s Farm Register and of operations identified through the Census of Population questionnaire.

Collection

The 2016 Census of Agriculture followed a multi-mode wave collection methodology and put an emphasis on Internet reporting. To initiate collection, Canada Post delivered invitation letters to all agriculture operations that were included on the census frame. The letters asked respondents to fill out a Census of Agriculture questionnaire online. If an electronic questionnaire was not submitted to Statistics Canada within the required period, a paper questionnaire was mailed, giving respondents an opportunity to return their completed questionnaire by mail or electronically. Finally, if neither the electronic nor the paper version of the questionnaire was received, a telephone interview was conducted by Statistics Canada employees.

Throughout the entire collection period, there were many reminders sent to farm operators to encourage them to complete the Census of Agriculture, one of them being a voice broadcast campaign. In addition, the Census Help Line was available to all operators who had questions about the questionnaire or about sending in their responses.

A similar collection methodology was used for the new farms identified though the Census of Population responses.

For a more detailed description of the collection process, please refer to Data collection.

Data processing

Census of Agriculture questionnaires, whether completed and submitted on the Internet or on paper, were routed to the Data Operations Centre in the National Capital Region. There, the paper questionnaires were sorted, scanned and the data were automatically captured using Intelligent Character Recognition (ICR) software, a technology that reads data from images. Data from both Internet and paper questionnaires were then subjected to many rigorous quality controls and processing edits to identify and resolve problems related to inaccurate, missing or inconsistent data. Subject-matter analysts also reviewed the aggregated data and individual values so that any remaining errors due to coverage, misreporting, data capture or other reasons were identified and corrected. For a more thorough explanation, please refer to Data processing.

Reference period

The Census of Agriculture has been conducted concurrently with the Census of Population every five years since 1951. The 2016 Census of Agriculture reference date was May 10, 2016.

Revisions

Data from the Census of Agriculture are not subject to revision.

Adjustments

Data from the Census of Agriculture are not subject to seasonal adjustments or benchmarking to other data sources.

Data accuracy

An integral part of each Census of Agriculture is the implementation of new or enhanced methods, procedures and technologies that improve not only the collection, but also the processing, validation and dissemination of the data. For the first time in 2016, the Census of Agriculture used Statistics Canada’s Business Register—the central repository of information on businesses in Canada—as its primary source for identifying census farms and for building the frame. This register is frequently updated through administrative sources and includes potential farms that may have been missed in previous censuses.

Another important change to the Census of Agriculture was the increased promotion of online electronic data collection. In addition to cost savings, this ‘Internet first’ approach reduced response burden. Gains in data quality were observed through the use of edits triggered within the questionnaire when invalid data was entered, through skip patterns that helped the respondent fill in all applicable sections and through pre-filled information for some fields that simply required the respondent’s confirmation and reduced the chance of typing errors.

In addition, people who received a Census of Agriculture questionnaire in error could call the Census Help Line and explain that they would not be submitting a questionnaire because it was not relevant for them.

To help ensure that data from the 2016 Census of Agriculture would be of consistently high quality, quality assurance and control activities took place throughout the census processes, beginning prior to data collection and ending before dissemination.

The 2016 Census of Agriculture data are of very good quality, primarily as a result of adopting these methods, procedures and technologies. A response rate of 94.3% and an estimated 4.9% undercoverage rate of farms underline the overall success of the 2016 Census of Agriculture. It should be noted that farms that were missed by the 2016 Census of Agriculture are small in terms of farm area and of gross farm receipts. More details on the quality of the data are provided in the Data quality indicators section.

With projects as large and complex as the Census of Agriculture, the estimates produced are inevitably subject to a certain degree of error. Knowing the types of errors that can occur and how they affect specific variables can help users assess the usefulness of the data for their particular applications, as well as assess the risks involved in making conclusions or decisions based on these results.

Errors can arise at virtually every stage of the census process, from preparing materials, through collecting data, to processing and tabulating the final results. Moreover, errors may be more predominant in certain areas of the country or vary according to the characteristic being measured. Some errors occur at random, and when individual responses are aggregated for a sufficiently large group they tend to cancel each other out. For errors of this nature, the larger the group, the more accurate the corresponding estimate. For this reason, data users are advised to be cautious when using estimates based on a small number of responses. Some errors, however, might occur more systematically and result in biased estimates. Because the bias from such errors is persistent no matter how large the group for which responses are aggregated, and because such bias is particularly difficult to measure, systematic errors are a more serious problem for most data users than random errors.

The most common types of errors are described below.

Coverage errors

Coverage errors occur when farms are missed when creating the census frame, erroneously classified as farms or as non-farms during data processing procedures, or counted more than once.

The population used for Statistics Canada’s ongoing agriculture surveys consists of establishments on Statistics Canada’s Business Register that responded to the 2011 Census of Agriculture or that had provided a clear and recent indication of agricultural activity. To improve the coverage of the 2016 Census of Agriculture, additional establishments were added to the frame in an attempt to enumerate new or unknown agricultural operations. These included establishments that had indicators of agricultural activity in their tax remittances or had reported a main business activity of agriculture in the Business Register. Additionally, during collection, potential farms were identified through the Census of Population questionnaire and were then added to the census frame.

In spite of these efforts, some coverage errors exist. Some active agricultural operations may still not have been included in the census frame. Coverage error among the establishments in the frame population can occur when non-responding units are erroneously classified as farms or as non-farms during Census of Agriculture data processing. These errors are reduced by well-defined and thoroughly studied methods of determining whether a non-respondent establishment was an active agricultural operation or not. In addition, the farm operators who were part of the Census of Agriculture frame and who did not respond to the mail-out or who did not receive a letter due to an incorrect mailing address were contacted by phone to complete a questionnaire.

Also, to reduce overcoverage (over-counted farms), unduplication activities were undertaken before the creation of the frame and during Census of Agriculture data processing.

However errors still occur. Measurement of the coverage errors is discussed in the Data quality indicators section.

Non-response errors

Some Census of Agriculture questionnaires are only partially completed or not completed at all, usually because of the respondent's absence during the census period or unwillingness to complete the questionnaire. To encourage participation, Statistics Canada provided respondents with three ways to respond to the Census of Agriculture – by Internet, by paper or by telephone. The principal collection period for the Census extended from May to August in order for operators to respond at a time that was convenient for them. Non-respondents were sent several reminders through different media to encourage them to respond. Extra effort was put on getting responses from operations which were thought to be large in order to reduce the potential non-response bias.

If no response was received, then the data for the operation were imputed using statistical methods. The imputation methods use information that Statistics Canada already had on the non-responding operations to derive plausible responses to the Census questions.

Response errors

Respondents sometimes provide inaccurate responses on the questionnaire, perhaps as a result of misinterpretation of a question, incorrect placement of a response or approximation of a response. In the Census of Agriculture, implausible or inconsistent responses are confirmed or corrected by contacting the respondents, since they could have a significant impact on totals at either the provincial or the sub-provincial level.

Online responses have better quality data since there are edits that prompt a respondent when invalid data is entered or data is missing. The online census questionnaires also pre-filled certain fields based on business information from the Business Register (e.g., the legal name and address of the business) which reduced errors. Furthermore, automated skip patterns alleviated respondent burden by skipping questions that were not applicable. The online questionnaires also had helpful information available for respondents who wanted additional information about a census question, which reduced the likelihood of misinterpretation of terms and questions.

Processing errors

Errors can arise at any stage of data processing, including scanning or character recognition errors during data capture, coding and classification errors, and errors due to limitations in the imputation procedure (to correct missing or inconsistent responses, as described in the Non-response errors section). A detailed set of computerized checks at each processing stage identifies such errors for corrective action. In addition, quality assurance and control procedures were developed for the processing stages.

Sampling errors

Sampling errors apply only when answers to questions are obtained from a sample. This type of error does not apply to the Census of Agriculture. 

Comparability of data and related sources

For the Census of Agriculture, the final data validation process is the certification of the data. At this stage, the results are compared to estimates from previous censuses or estimates from other data sources to evaluate the coherence and accuracy of the Census of Agriculture data. During data certification, response rates, invalid or inconsistent responses, edit failure rates, coverage rates and a comparison of the data before and after imputation are among the measures used to evaluate the quality of the data and possibly explain differences with other sources. Detailed cross-tabulations are also checked for consistency and accuracy.

Some estimates are not comparable with those of previous censuses. This may be due to wording or conceptual changes in the questions in 2016 or the addition or removal of questions between 2011 and 2016. After thoroughly investigating each case, notes were developed to identify the affected questions and explain the reasons that users should use caution when comparing the results.

Following each Census of Agriculture, other agricultural surveys use the Census of Agriculture data as a basis, or benchmark, for the production of regularly published estimates of various agriculture industry characteristics and indicators.

Data quality indicators

Many data quality indicators are computed to measure the reliability and the overall quality of census data. The most important ones are the response rates and the coverage errors. 

Response rates

Response rates are one of the key data quality measures for the Census of Agriculture. Table 1 below shows the response rates at the national level and for each province. These rates were calculated after the data processing and certification steps.  

Overall, the response rate is similar to the 2011 Census of Agriculture response rate. However, the Internet reporting rate has surpassed expectations, having increased fivefold from 11% in 2011 to 55% in 2016 at the Canada level. This is explained by the push to adopt Internet as the primary mode of collection. Also, because of the focused wave collection methodology, the self-response in 2016 is higher by about 10 percentage points compared to 2011.

Coverage evaluation

A coverage evaluation was undertaken to estimate the coverage errors in the 2016 Census of Agriculture.

Coverage errors occur when there is difference between the target population and the survey population and they affect the quality of estimates of all censuses. For the Census of Agriculture, coverage errors occur when farms are missed, incorrectly included or double counted. Estimating these errors is one way to assess the quality of the Census of Agriculture estimates.

The processes involved in the creation of the frame, data collection, and data processing are not perfect and can contribute to these coverage errors. For example, when creating the frame, real farms might be missed because they were simply not included in one of the sources used to create the Census of Agriculture frame. Also, at the end of the collection period, non-responding units might be erroneously classified as farms or as non-farms during Census of Agriculture data processing.

The overall coverage of the Census of Agriculture was measured using two components. The first component measured the misclassification of non-respondents: both undercoverage (missed agricultural operations) and overcoverage (units erroneously counted as agricultural operations) resulting from the processing methods were estimated.

The second component measured undercoverage errors arising from missing operations on the frame. These were estimated using a post-census survey called the Agriculture Frame Update Survey. This survey targeted establishments from the Business Register that had some indication of being an agricultural operation, but were not included in the Census of Agriculture frame. Through this survey, additional operations that had been missed by the Census of Agriculture were identified.

The final net undercoverage estimates combine the estimates of these two components and is calculated using the following formula.

It is possible to have a negative net undercoverage rate in certain cases. This indicates an estimated overcoverage.

Please note that there are no estimates of undercoverage for Yukon, the Northwest Territories and Nunavut.