4. Content Determination Method
Statistics Canada recognizes that agricultural producers are the key to the success of the census. This success depends on the willingness of all Canadian producers to complete and return their census questionnaire and, as such, Statistics Canada is continually striving to minimize response burden while producing high-quality data that meets the needs of a wide group of data users. For this reason, Statistics Canada is especially sensitive about the need to strike a balance between incorporating new questions and retaining relevant questions while minimizing respondent reporting burden.
Data user and stakeholder consultations are a key element in determining the content for the next CEAG. Engaging with data users in developing the census questionnaire permits Statistics Canada to:
- better understand and respond to the social and economic priorities of the agriculture sector
- gauge reaction to proposed content changes
- formulate creative solutions inspired by data users and experts.
During the consultations for the 2016 CEAG, data users expressed a strong interest in maintaining content stability from census to census, so that major trends could be measured over time. However, data users also expressed a strong interest in incorporating new questions into the census to reflect significant changes in the industry and to maintain data relevance through time.
Consultations for the 2016 CEAG were conducted through several means:
- Several workshops were conducted across the country soliciting feedback and engaging participants in discussions on what content should be included in the CEAG questionnaire, at which geographic levels and at what frequency.Note 1
- Submissions and comments were solicited by contacting known data users and stakeholders of the CEAG using a Feedback and Submission Form (presented in Appendix C). Organizations, individuals and producers across the country were encouraged to share the submission form with others, so that submissions could be sent in by anyone interested.
- Submissions and comments were also solicited through the federal government’s Consulting Canadians Website.Note 2
Thirteen consultation workshops were held across the country in October 2012. A variety of stakeholders in the agricultural community participated. There were two groups of audiences for the workshops: one consisted of industry partners, represented by producer associations, farm organizations and advisory groups, and the other was made up of data users such as Agriculture and Agri-Food Canada (AAFC), provincial agricultural departments, university researchers, other departments and other divisions within Statistics Canada. Through conversations with this broad range of participants from the agricultural community, Statistics Canada was able to obtain a comprehensive picture of data requirements.
In broad terms these workshops were used to:
- present a discussion questionnaire, which proposed some content to be potentially removed thus generating discussion about the impact on stakeholder programs
- understand what data are absolutely needed
- establish contacts and solicit support to obtain producer lists
- discuss the possibilities of acquiring data from administrative sources that might have the potential to replace some census and survey data.
Over 460 invitations were sent out to direct contacts who were also encouraged to forward the invitations to additional data users and stakeholders of the CEAG. A total of 168 persons responded by attending one of the 13 consultation workshops held across the country. (A detailed list of the towns where the consultations were held is found in Appendix B.)
4.1.3 Consulting Canadians
In addition, through the Consulting Canadians Website, Statistics Canada’s received 176 hits. Those who did not participate in the consultation workshops were encouraged to submit written submissions during October 2012.
Following these three methods of consultation, Statistics Canada received more than 200 submissions for content change. Over one thousand comments and suggestions were evaluated and those that met the basic acceptance criteria moved on to testing with farm operators. Table 2 shows the types of organizations that participated at the different consultations, as well as the number of submissions that Statistics Canada received.
A discussion questionnaire was created specifically for the consultations to clearly identify the content proposed for removal. This discussion questionnaire consisted of a modified 2011 CEAG questionnaire with proposed content for removal or to be combined.Note 3 The discussion questionnaire was intended to be a catalyst for discussion on identifying what the data priorities are and what the real potential is for using alternative data sources to replace census questions.
During the workshops, each participant received a copy of guidelines on how to prepare a submission, a Feedback and Submission Form and a discussion questionnaire. Participants were informed that only written submissions would be retained for further consideration and analysis.
Data users and stakeholders who could not attend the workshops were provided with electronic copies of the same consultation documents. The Feedback and Submission Form had to be received by the end of October 2012, but this deadline was extended to mid-November to accommodate a few stakeholders.
The Feedback and Submission Form asked the following questions:
- Which existing question(s) in the 2011 census questionnaire do you use or intend to use? At what geographic level do you use the data? How do/will you use them? Do they address the needs or priorities of your organization and how?
- Which existing question(s) in the 2011 census questionnaire would you never use? Why? What are your alternate data sources for these questions?
- On the Discussion Questionnaire, what change, additional detail or new topic would you like to see on the 2016 Census of Agriculture?
- Provide a brief explanation of why these data are required.
- How would these data address the priorities or needs of your organization? What industry, program or policy issues are you attempting to address or answer through these data?
- What is the required geographic level of the data? (e.g., national, provincial, federal electoral districts or smaller geographic units)
- What would be the minimum reporting frequency required to make the data useful? (e.g., annual, quinquennial, or a one-time query)
- Provide your suggestions for the wording of the question(s) you would like to see asked in the 2016 questionnaire.
- What other data sources address this data need? What are the strengths or limitations of those data sources?
- Are there any other data within your organization, which can provide the same information as the census, or when combined with census data would be useful for policy purposes? If yes, please list and provide details.
4.2.1 Submitted Content
In total, more than 200 submissions were received, resulting in a total of 1,007 comments.Note 4 Examining the submissions revealed that several comments were similar and could be grouped by topic. Table 3 shows the 63 topics and the number of comments submitted for each category. A summary of the submissions received is presented in Table 4 and Table 5. The first table focuses on topics that were in the 2011 CEAG and the second table focuses on new topics.
Almost two thirds of the comments (648 comments) were related to topics that users wanted to see retained in the CEAG. Most submissions presented justifications on how the data were used, why the topic should be retained and why census data were needed. A larger number of comments focused on organic production, use of input (pesticides, manure or irrigation water) and farm practices (land management, tillage, summerfallow, etc.). Although some proposals suggested dropping content, other users suggested retaining many of these same topics; therefore there was no clear consensus on content to potentially eliminate. Many submissions asked for expanding these topics with more detailed questions. Overall, these suggestions were not feasible due to space limitations on the questionnaire, the commitment to reduce reporting burden and because the CEAG would not be the most efficient method to collect additional data with questions that are considered too specific for a national census.Note 5
The second largest group (158 comments) were related to removal of detailed categories for farm expenses, machinery and equipment, and uncommon crops or livestock. Overall, users wanted the detailed information to be retained. They did not want detailed items to be combined into sub-totals (e.g. total crop or livestock expenses) but overall they were satisfied if an alternative administrative data sources like taxation data could replace detailed expense items. They provided suggestions for which categories of farm machinery and equipment could be combined together. Suggestions for more detailed information about organic, genetically modified or bio-product crops were submitted. Also, more detailed information was requested for common and uncommon types of livestock. These suggestions were not feasible either due to the results of testing in previous censuses, space limitations on the questionnaire, the commitment to reduce reporting burden or because the CEAG would not be the most efficient method to collect additional data with questions that are considered too specific for a national census.
The third largest group (109 comments) were related to topics that were initially identified as core to the CEAG. Users expressed their support to retain all core questions by giving examples of how they use CEAG data. Some would have preferred that specific topics be expanded by asking for more detail. These suggestions were not feasible due to space limitations on the questionnaire, the commitment to reduce reporting burden and because the CEAG would not be an efficient method to collect additional data with questions that are too specific for a national census.
There were also suggestions to add new content (92 comments). The most frequently submitted new topics were direct marketing/sales, technology and succession planning. All suggestions for new topics were reviewed and researched. Depending on the results of the research and the testing experience with previous censuses, some were included in qualitative testing. Suggestions that tested well at this stage moved on to quantitative testing in May 2014 in the National Census Test.
4.2.2 Assessment of Submitted Content
All submissions were given a preliminary review by CEAG analysts. This review included a summary of submissions related to each specific topic, an assessment of what other countries are doing, the identification of alternative sources of data (from surveys or administrative programs), different versions of questions that could be tested (including those tested in previous censuses), a list of probing questions to ask to test participants, and a list of industry experts to contact if necessary.
During a group review, CEAG analysts responsible for a specific topic presented their findings and the topics were evaluated using an evaluation grid. The evaluation covered different points, such as issues encountered in past censuses with existing questions, information requirements by program and policy, data demand, the willingness, ability and ease of answering by producers, results from other countries, sensitivity of the topic, past testing results, CEAG objectives, the existence of alternative data, sustainability of the topic (one-time need versus long-term trend), historical comparability, national scope and the possibility of producing intra-provincial data.
All submissions were classified into one of three categories:
- Topics considered unsuitable for the CEAG: This category included topics that did not meet one or several criteria in the evaluation grid (see Appendix D). They were either not related to primary agriculture, limited to specific regions or not of national scope; they were opinion, subjective, forecast and prediction assessments; they would be difficult to answer because they were either sensitive questions or because they would require lengthy explanation; or they could not simply be answered with a “Yes” or “No” or the responses could not be quantified.
- Topics considered unsuitable for CEAG collection methods, but nonetheless representing a data need: This category included topics with questions that would be easier to answer with the assistance of an interviewer (questions are not appropriate for self-enumeration, the collection method used by the CEAG), topics for which alternative data source exist (from surveys or programs); and topics that required detailed questions, which could be asked in a post-census survey targeting a specific group of producers. In addition to avoiding a potential increase in response burden, space limitations in the 2016 CEAQ questionnaire were a major factor for screening highly detailed topics.Note 6
- Topics deemed suitable for testing: This category included topics that first pass all evaluation criteria.Note 7 Different versions of draft questions that could be tested (including those tested in previous censuses) were developed for the first phase of modular testing (described in the following section).
Drafts of the new or revised questions were subjected to an array of tests carried out in collaboration with questionnaire design specialists from Statistics Canada’s Questionnaire Design Resource Centre (QDRC).
These qualitative tests were used to:
- obtain feedback from respondents on their overall impressions of and reactions to the proposed content and questions
- test the respondents’ perception and thought processes in answering the questions
- assess respondents’ understanding of the concepts, terminology, questions and response categories
- assess the availability and ease of reporting of the information as requested
- test respondents’ ability and willingness to answer the questions
- test the respondent-friendliness of the questionnaire (i.e., that it is easily understood and can be accurately completed)
- ensure that respondents from all regions understand the questions and in the same way.
In total, four qualitative tests were conducted.Note 8 Participants were recruited from a list of agricultural operators in proximity to the selected test locations. While avoiding locations that were selected for the previous Census or for recent survey testing, test locations with a sufficient number of diversified farms were selected. At least one test location in each province needed to be selected during each phase of testing. A Statistics Canada employee recruited farm operators of various farm sizes and types, operating arrangements, age, mother tongue and level of schooling. Both French and English versions of the questionnaire were tested. Appendix E shows summary statistics about participation in the different tests.
4.3.1 The Modular Tests
The focus of the two qualitative tests (also called “modular” tests) was on specific new topics or substantially modified questions from the previous census.Note 9 These tests were used to improve the questions or to confirm that the questions performed well (ease of understanding and ability to complete, willingness to answer, and identify any sensitivities or issues with the topic).
The modular tests were conducted by means of one-on-one interviews with individual producers and by focus groups consisting of ten to twelve producers. Both of these means of testing have specific benefits.
The one-on-one interviews allow interviewers to observe the producer's reaction, perception and thought processes while answering each question. This gives valuable insight into the time it takes for producers to answer each question, which questions require producers to consult records and what calculations could be required.
Focus groups with farmers are suited for more general topics, questionnaire format preferences, insights into attitudes, opinions, concerns and suggestions. They are useful for evaluating farmers’ understanding of the language and wording for specific questions or instructions.
Modular Test - Phase 1
The first phase of the modular testing was conducted in February 2013 in British Columbia, Manitoba, Ontario and Nova Scotia.
The following new content modules were tested:
- succession planning
- direct marketing practices
- technology used on the farm
- drainage systems.
Other questions were revised to simplify or improve understanding and reporting:
- name and contact information of the person completing the questionnaire
- main farm location and information on farm operators
- farm work and other work
- nursery products
- greenhouse products
- farm practices and land features
- manure application
- irrigation systems
- farm machinery and equipment
- farm receipts and expenses
- paid labour.
If the results for a question were not entirely satisfactory during the first phase of modular testing, the question could be modified and evaluated again in a subsequent test. If the results for a question were sufficiently negative (for example, the questions were too onerous for respondents to answer), these questions were set aside.
Modular Test - Phase 2
The second phase of the modular testing was conducted in March 2013 in Saskatchewan, Quebec and Prince Edward Island. Phase 2 testing focused on addressing issues that arose during Phase 1 testing. Questions were either modified or the position within the questionnaire was altered.
If the results of the second phase of modular testing were deemed satisfactory, the modules were revised (if necessary). The new and revised questions were then combined with the unchanged questions from the 2011 CEAG into an integrated test questionnaire used for integrated testing.
4.3.2 The Integrated Tests
The integrated questionnaire was used to continue testing the new and revised questions and to evaluate the flow of the entire questionnaire. The new and modified questions continued to be improved.
Two rounds of integrated testing were conducted and locations across the country were selected for testing. The tests were conducted by means of one-on-one interviews with producers. The first test was conducted using the paper questionnaire while the second test used an electronic questionnaire developed for Internet data collection. The electronic test replicated the conditions similar to Census day and was done using the participants’ computers and Internet browsers.
Integrated Test using Paper Questionnaires
The first Integrated Test was held in May and June 2013 to replicate Census Day in 2016 as closely as possible (usually the second Tuesday of May). Census Day is during a busy period for most farms growing crops in Canada. It was held in Alberta, Saskatchewan, Ontario, Quebec, New Brunswick and Newfoundland and Labrador.
The focus of these tests was to obtain participants’ feedback and perception of the entire questionnaire and to test their ability to accurately respond to it. A paper questionnaire was mailed to the participants to be filled out before the scheduled one-on-one interview.
As with the previous phases of testing, questions that tested well in the paper questionnaire were either included in the electronic questionnaire unchanged or slightly modified. The modifications were based on comments received from participants and observations and recommendations from Statistics Canada questionnaire design specialists.
Integrated Test using the Electronic Questionnaire
One-on-one interviews were conducted with agricultural producers to test the electronic questionnaire. This was the final qualitative test. The objective was to evaluate the usability of the electronic questionnaire and to test minor content changes that were made as a result of the previous integrated test.
The entire questionnaire was tested with emphasis on comprehension and fine-tuning of the wording and format. The electronic questionnaire testing was conducted in British Columbia, Alberta, Manitoba, Ontario, Quebec, New Brunswick and Nova Scotia.
4.3.3 Census of Agriculture Test, May 2014
The National Census Test was conducted in May 2014. It included both Census of Population and CEAG test questionnaires. It was meant to test processes and systems, from beginning to end. The reference date for the Census Test, May 13, 2014, was chosen to simulate the timing of the approximate Census date used since 1996. A random sample of 9,000 farms (across Canada) was selected to participate voluntarily in this Census Test.
The test provided the opportunity to study how large numbers of respondents complete the census questions on either the paper questionnaire or the electronic questionnaire. In addition to final content testing, different aspects of collection were tested.For example, 2016 will be the first time that Internet will be the preferred mode of collection. In 2016, respondents will first receive a letter inviting them to complete their census forms online. If they prefer to fill out a paper questionnaire, they will be required to telephone a call centre and one will automatically be mailed to them. The Census of Agriculture Test also allowed for some fine tuning of the Census Help Line procedures and systems. It helped to estimate how many respondents in 2016 will complete their questionnaire online without a reminder letter or a call from interviewers, etc. This quantitative test helped to identify any systematic reporting problems with questionnaire content by comparing participants’ responses to the 2011 CEAG or subsequent agriculture survey information.