Statistics Canada
Symbol of the Government of Canada
Skip to content | Skip to institutional links

Common menu bar links

Home > Publications > 12-004-X > Main page >

Site navigation menu

Publications

Page content follows

Answers

Archived Content

Information identified as archived is provided for reference, research or recordkeeping purposes. It is not subject to the Government of Canada Web Standards and has not been altered or updated since it was archived. Please contact us to request a format other than those available.

On first glance, conducting a survey might appear to be simply asking questions and compiling the answers to obtain statistics. However, it’s important to follow precise steps so that the survey results will provide accurate and useful information.

To begin, the following questions should be addressed:

  • Why is this survey being conducted?
  • Whom will the collected information be about?
  • What do I need to know?
  • How will the information be used?
  • How accurate and timely does the information have to be?

To design a survey, many decisions have to be made that address the following issues:

Survey objectives

A survey plan begins with objectives that describe why and for whom the survey is being done. The survey objectives tell a lot about the data that need to be collected. The objectives also help determine the population to be targeted.

For example, imagine that Ridgemont High School’s student council wants to survey students to get information that would help in planning the graduation prom. From this general goal, you can make some more refined objectives. Let’s say that the survey objectives are:

  • To gather information from students in order to determine the factors that will make the prom a success. (The criteria of "success" are that the largest possible number of students will attend the prom and that it will fulfill their expectations.)
  • To obtain useful data that will help the prom organising committee.

The survey plan will show how the objectives will be reached by clearly describing the target population, the data requirements and the variables to be measured, as well as looking at the questions and possible answers and how the data will be processed and analysed.

Target population

If a survey’s objective is to collect information from students, for example, then asking the question "which students?" will help to define the target population.

In the example described previously, the prom organizing committee will probably want to question only students who will be graduating this year, that is, those in the last year of high school (Grade 12). If some of the Grade 12 students are studying part-time and don’t intend to graduate this year, they need not be consulted. The target population would therefore be defined as "the full-time Grade 12 graduating students of Ridgemont High School".

Sometimes the target population (the population for which information is required) and the survey population (the population actually completing the survey) differ for practical reasons, even though they should, in reality, be the same.

In our example, some of the full-time Grade 12 graduating students might be away from school at the time of the survey. Since it would be too difficult to reach them, they would not be part of the survey population, although they are part of the target population.

It is also possible that some of the survey concepts and methods that are used may be considered inappropriate for certain segments of the population. For example, consider a survey of post-secondary graduates where the objective is to determine if the graduates found jobs and, if so, what types of jobs. In this case, you might exclude graduates coming from specialized schools such as religious seminaries or military schools. These types of graduates would be reasonably assured of securing employment in their respective fields. The target population would therefore be those who graduated from universities, colleges and trade schools.

It may also be necessary to impose geographic limits that will exclude some members of the target population, as some regions may be too difficult or expensive to reach. For example, a business that is doing a survey using in-person interviews may wish to use a sample of the target population living in a densely populated area in order to minimise the travel involved.

Data requirements

To determine what kind of data to collect, ask "What exactly do we want to know?" and "How will the collected information be used"?

In our example, the organizing committee might consider the following questions:

  • Do we need to know the number of students who intend to go to the prom? (This number might also be established from ticket sales.)
  • If we ask students whether they intend to go to the prom, should we ask anything in particular to those who don’t intend to go? (By understanding better their reasons for not going, it might be possible to plan certain activities that are of interest thus influencing them to change their minds!)
  • When asking about student preferences concerning the prom, what aspects should we consider?
    • the cost of tickets
    • the music
    • the type of refreshments
    • the day of the week
    • the venue or location
  • Are there any other factors to consider? Would the students like to have a photographer available? Does everyone want to have a meal before the dance or do some students want just the dance?
  • Concerning security, are students interested in having security guards at the entrance of the venue? What type of transportation would students like to use to get to and from the prom? (The rental of a bus from a central location might be considered)

When planning a survey, it’s tempting to want to collect as much information as possible. However, the more questions that are asked, the longer the survey takes and the more it costs. It’s important to ask: « Do we really need this information? » while considering the time and resources needed to test the questionnaire, process the data and analyse the results.

Another aspect to take into account is the burden the survey imposes on the respondent, so that it’s not seen as a nuisance. Respondent burden is affected by

  • the number of questions asked
  • the intrusiveness of the questions
  • the number of times the respondent is contacted (for a same survey or for many surveys)
  • the detail of information requested (for example, if asked for a precise income figure, respondents need to consult their official documents, but if asked to choose between five different income ranges, they can answer more easily)
  • the time it takes to complete the survey.

Choosing the type of data collection

The level of accuracy pursued and the resources available will determine the choice among three main types of data collection.

  1. A census is a survey that collects information from all the people in a group or population
  2. A sample survey collects information from only a part (a sample) of a population. It is possible to estimate results for a total population using data that is collected from a sample.
  3. Administrative data is collected through an organisation and is used as an alternative to a survey.

Each has advantages and inconveniences and the choice of collection type will depend on various factors. See Types of data collection.

In our example, the organising committee may decide to do a census of all the graduating students or to survey only a sample of that group.

The type of collection chosen often depends on the budget available. Costs are one of the main justifications for choosing to conduct a sample survey instead of a census. With sample surveys, it is possible to obtain reasonable results with a relatively small sample of the target population. For example, if you need information on all Canadian citizens over 15 years of age, a survey of a small number of these (1,000 or 2,000 depending on the data requirements) might provide adequate results.

Another advantage of using a sample survey is that it permits investigators to produce information soon after they have identified the need for it, within a rapid turnaround time. For example, if an organization wants to measure the public awareness created through an advertising campaign, it should conduct a survey shortly after the campaign is undertaken. Since using a sample of the target population requires a smaller scale of operation, it reduces the data collection and processing time, while allowing more time for planning.

Minimizing error

When planning a survey, you must be aware of potential sources of error and try to reduce them as much as possible.

In a sample survey, the variation that exists between different samples causes a certain bias, called "sampling error". For example, let’s say you are estimating the average distance between home and school for students in your class of 25 from a sample of 5 persons. Your estimate will depend on which 5 students are sampled. If all 5 sampled students live very close to the school, the results will not be representative of the whole class. It’s the variation from one sample to another that causes the sampling error.

As a general rule, the more people surveyed (the larger the sample size), the smaller the sampling error will be. Also, it is possible to estimate the sampling error associated with a particular sampling plan, and try to minimize it. See Sampling error.

By choosing to do a census, you can avoid errors related to sample variation, but all surveys also risk having sources of "non-sampling error". For example, a question might be asked in a way that encourages a certain answer or an error might be made while processing the data or calculating a percentage for a table of results. These types of error can be avoided as much as possible by paying attention to quality control throughout every step of the survey process. See Non-sampling error.

Sample size

Since every sample survey is different, there are no hard and fast rules for determining sample size. The deciding factors are time, cost, operational constraints and the desired precision of the results. Evaluate and assess each of these issues and you will be in a better position to decide the sample size. Also, consider what should be the acceptable level of error in the sample. If there is a lot of variability in the population, the sample size will need to be bigger to obtain the specified level of reliability. See Sample size.

Analysis plan

After identifying all the elements (or variables) to be measured and preparing the sample design, the next step is the analysis plan—conceiving what the results tables will look like. In other words, you need to plan the tables that you will create for the survey variables. These tables will not yet contain any data, but will show any cross-tabulations you want to make.

In our example, the organizing committee might plan results tables showing the number and percentage for each survey variable (for example, the number and percentage of students who prefer location A to location B for the prom). Some tables could also present cross-tabulations such as "Preferred music by gender".

These "empty" tables help you verify whether the questions you are considering will allow you to reach your survey objectives. They illustrate concretely how the collected information will be used and whether it will adequately measure what you want to know.

Questionnaire design

The questionnaire’s design is based on the survey’s data requirements and analysis plan. As you formulate the questions, it can be helpful to consult the people who will be using the results. You can also consult subject matter experts or look at questions from other surveys on similar topics or themes.

It’s important to ensure that the questions relate to the survey objectives and that each question is relevant. See Questionnaire design.

Data collection methods

Planning the method of data collection is an important step: you will need to consider the costs, physical resources, and time required to conduct the survey.

Select the best method to gather the required data. Keep in mind that cost of the survey and data quality will be directly impacted by the method that you choose. There are several options available: the personal interview (face-to-face or by telephone, with or without computer assistance) and the self-completed questionnaire.

Personal interviews are administered by a trained interviewer and can have either a structured or unstructured line of questioning. When done by telephone, questions are structured in a formal interview schedule.

The self-completed questionnaire must be highly structured as the respondent will not have any help from an interviewer. It can be returned by mail or through a drop-off system or completed online. See Data collection methods.

In our example, the organizing committee may opt for a personal interview administered by interviewers who fill out an electronic questionnaire in a spreadsheet program. The interviewer would use a laptop computer to enter the students’ answers into the spreadsheet during the interviews. If some students are concerned about the confidentiality of their answers, the interviewer could give them the option of entering their answers themselves. Such an option, however, might cause more errors and compromise the quality of the collected data, which in turn could increase the time needed for data processing.

Data processing plan

This step deals with processing the questionnaire responses into output. The tasks involved in data processing include: coding, data capture, editing, dealing with invalid or missing data and, if necessary, creating derived variables. In short, the aim in this step is to produce a file of data that is as free of errors as possible. See Data processing.

Quality control

This process identifies errors and verifies results. No matter how much planning and testing goes into a survey, something unexpected will often happen. As a result, no survey is ever perfect. Quality control tasks are required to minimize non-sampling errors introduced during various stages of the survey. These tasks include: interviewer training, data editing, computer program testing, follow-up of non-respondents, and spot-checks of collected responses and output data. Statistical quality-control programs ensure that error levels are kept to a minimum.

Analysis and dissemination of results

After planning data collection and processing, look ahead to the final steps in analyzing and disseminating the results: 

In our example, members of the prom organizing committee might share the tasks of organizing and analyzing the data, then writing up the conclusions. Decisions about the prom venue, ticket price, type of music, etc. would then be based on these findings. By publishing highlights of the survey in the school newspaper, the student council might demonstrate that its decisions about the prom are based on what students told them.
Date Modified: 2021-03-31

Top of Page
Important Notices