Information identified as archived is provided for reference, research or recordkeeping purposes. It is not subject to the Government of Canada Web Standards and has not been altered or updated since it was archived. Please contact us to request a format other than those available.
On first glance, conducting a survey might appear to be simply asking questions and compiling the answers to obtain statistics. However, it’s important to follow precise steps so that the survey results will provide accurate and useful information.
To begin, the following questions should be addressed:
To design a survey, many decisions have to be made that address the following issues:
A survey plan begins with objectives that describe why and for whom the survey is being done. The survey objectives tell a lot about the data that need to be collected. The objectives also help determine the population to be targeted.
For example, imagine that Ridgemont High School’s student council wants to survey students to get information that would help in planning the graduation prom. From this general goal, you can make some more refined objectives. Let’s say that the survey objectives are:
The survey plan will show how the objectives will be reached by clearly describing the target population, the data requirements and the variables to be measured, as well as looking at the questions and possible answers and how the data will be processed and analysed.
If a survey’s objective is to collect information from students, for example, then asking the question "which students?" will help to define the target population.
Sometimes the target population (the population for which information is required) and the survey population (the population actually completing the survey) differ for practical reasons, even though they should, in reality, be the same.
It is also possible that some of the survey concepts and methods that are used may be considered inappropriate for certain segments of the population. For example, consider a survey of post-secondary graduates where the objective is to determine if the graduates found jobs and, if so, what types of jobs. In this case, you might exclude graduates coming from specialized schools such as religious seminaries or military schools. These types of graduates would be reasonably assured of securing employment in their respective fields. The target population would therefore be those who graduated from universities, colleges and trade schools.
It may also be necessary to impose geographic limits that will exclude some members of the target population, as some regions may be too difficult or expensive to reach. For example, a business that is doing a survey using in-person interviews may wish to use a sample of the target population living in a densely populated area in order to minimise the travel involved.
To determine what kind of data to collect, ask "What exactly do we want to know?" and "How will the collected information be used"?
In our example, the organizing committee might consider the following questions:
When planning a survey, it’s tempting to want to collect as much information as possible. However, the more questions that are asked, the longer the survey takes and the more it costs. It’s important to ask: « Do we really need this information? » while considering the time and resources needed to test the questionnaire, process the data and analyse the results.
Another aspect to take into account is the burden the survey imposes on the respondent, so that it’s not seen as a nuisance. Respondent burden is affected by
The level of accuracy pursued and the resources available will determine the choice among three main types of data collection.
Each has advantages and inconveniences and the choice of collection type will depend on various factors. See Types of data collection.
The type of collection chosen often depends on the budget available. Costs are one of the main justifications for choosing to conduct a sample survey instead of a census. With sample surveys, it is possible to obtain reasonable results with a relatively small sample of the target population. For example, if you need information on all Canadian citizens over 15 years of age, a survey of a small number of these (1,000 or 2,000 depending on the data requirements) might provide adequate results.
Another advantage of using a sample survey is that it permits investigators to produce information soon after they have identified the need for it, within a rapid turnaround time. For example, if an organization wants to measure the public awareness created through an advertising campaign, it should conduct a survey shortly after the campaign is undertaken. Since using a sample of the target population requires a smaller scale of operation, it reduces the data collection and processing time, while allowing more time for planning.
When planning a survey, you must be aware of potential sources of error and try to reduce them as much as possible.
In a sample survey, the variation that exists between different samples causes a certain bias, called "sampling error". For example, let’s say you are estimating the average distance between home and school for students in your class of 25 from a sample of 5 persons. Your estimate will depend on which 5 students are sampled. If all 5 sampled students live very close to the school, the results will not be representative of the whole class. It’s the variation from one sample to another that causes the sampling error.
As a general rule, the more people surveyed (the larger the sample size), the smaller the sampling error will be. Also, it is possible to estimate the sampling error associated with a particular sampling plan, and try to minimize it. See Sampling error.
By choosing to do a census, you can avoid errors related to sample variation, but all surveys also risk having sources of "non-sampling error". For example, a question might be asked in a way that encourages a certain answer or an error might be made while processing the data or calculating a percentage for a table of results. These types of error can be avoided as much as possible by paying attention to quality control throughout every step of the survey process. See Non-sampling error.
Since every sample survey is different, there are no hard and fast rules for determining sample size. The deciding factors are time, cost, operational constraints and the desired precision of the results. Evaluate and assess each of these issues and you will be in a better position to decide the sample size. Also, consider what should be the acceptable level of error in the sample. If there is a lot of variability in the population, the sample size will need to be bigger to obtain the specified level of reliability. See Sample size.
After identifying all the elements (or variables) to be measured and preparing the sample design, the next step is the analysis plan—conceiving what the results tables will look like. In other words, you need to plan the tables that you will create for the survey variables. These tables will not yet contain any data, but will show any cross-tabulations you want to make.
These "empty" tables help you verify whether the questions you are considering will allow you to reach your survey objectives. They illustrate concretely how the collected information will be used and whether it will adequately measure what you want to know.
The questionnaire’s design is based on the survey’s data requirements and analysis plan. As you formulate the questions, it can be helpful to consult the people who will be using the results. You can also consult subject matter experts or look at questions from other surveys on similar topics or themes.
It’s important to ensure that the questions relate to the survey objectives and that each question is relevant. See Questionnaire design.
Planning the method of data collection is an important step: you will need to consider the costs, physical resources, and time required to conduct the survey.
Select the best method to gather the required data. Keep in mind that cost of the survey and data quality will be directly impacted by the method that you choose. There are several options available: the personal interview (face-to-face or by telephone, with or without computer assistance) and the self-completed questionnaire.
Personal interviews are administered by a trained interviewer and can have either a structured or unstructured line of questioning. When done by telephone, questions are structured in a formal interview schedule.
The self-completed questionnaire must be highly structured as the respondent will not have any help from an interviewer. It can be returned by mail or through a drop-off system or completed online. See Data collection methods.
This step deals with processing the questionnaire responses into output. The tasks involved in data processing include: coding, data capture, editing, dealing with invalid or missing data and, if necessary, creating derived variables. In short, the aim in this step is to produce a file of data that is as free of errors as possible. See Data processing.
This process identifies errors and verifies results. No matter how much planning and testing goes into a survey, something unexpected will often happen. As a result, no survey is ever perfect. Quality control tasks are required to minimize non-sampling errors introduced during various stages of the survey. These tasks include: interviewer training, data editing, computer program testing, follow-up of non-respondents, and spot-checks of collected responses and output data. Statistical quality-control programs ensure that error levels are kept to a minimum.
After planning data collection and processing, look ahead to the final steps in analyzing and disseminating the results: