3 Data gathering and processing
3.1 Planning

At first glance, conducting a survey might appear to be simply asking questions and compiling the answers to obtain statistics. However, it’s important to follow precise steps so that the survey results will provide accurate and useful information.

To begin, the following questions should be addressed:

• Why is this survey being conducted?
• Whom will the collected information be about?
• What do I need to know?
• How will the information be used?
• How accurate and timely does the information have to be?

To design a survey, many decisions have to be made about the following issues, which will be covered in detail in the present section.

• Survey objectives
• Target population
• Data requirements
• Choosing the type of collection
• Minimizing error
• Sample size
• Analysis plan
• Questionnaire design
• Data collection methods
• Data processing plan
• Quality control
• Analysis and dissemination of results

Survey objectives

A survey plan begins with objectives that describe why the survey is being done and which population has to be reached to carry this survey. The survey objectives tell a lot about the data that needs to be collected. The objectives also help determine the target population.

Example

Imagine that Ridgemont High School’s student council wants to survey students to get information that would help in planning the graduation prom. From this general goal, you can refine objectives. Let’s say that the survey objectives are

• To gather information from students in order to determine the factors that will make the prom a success. (The criteria of “success” are that the largest possible number of students will attend the prom and that it will fulfill their expectations.)
• To obtain useful data that will help the prom organizing committee.

The survey plan shows how the objectives will be reached by clearly describing the target population, the data requirements and the variables to be measured, as well as looking at the questions and possible answers and how the data will be processed and analyzed.

Target population

If a survey’s objective is to collect information from students, for example, then asking the question “which students?” will help to define the target population.

Sometimes the target population, that is, the population for which information is required, and the survey population, the population actually being covered by the survey, differ for practical reasons. Ideally, the two populations should be very similar. It is important to note that conclusions based on the survey results will apply only to the survey population.

It is also possible that some of the survey concepts and methods that are used may be considered inappropriate for certain segments of the population. For example, consider a survey of post-secondary graduates where the objective is to determine if the graduates found jobs and, if so, what types of jobs. In this case, you might exclude graduates coming from specialized schools such as military schools. These types of graduates would be reasonably assured of securing employment in their field. The target population would therefore be those who graduated from universities, colleges and trade schools.

It may also be necessary to impose geographic limits that will exclude some members of the target population, as some regions may be too difficult or expensive to reach. For example, a business that is doing a survey using in-person interviews may wish to use a sample of the target population living in a densely populated area in order to minimize travel costs.

Data requirements

To determine what kinds of data to collect, ask, “What exactly do we want to know?” and “How will the collected information be used?”

When planning a survey, it’s tempting to want to collect as much information as possible. However, the more questions are asked, the longer the survey takes and the more it costs. It’s important to ask: “Do we really need this information?” while considering the time and resources needed to test the questionnaire, process the data and analyze the results.
Another aspect to take into account is the burden the survey imposes on the respondent, so that it’s not seen as a nuisance. Response burden is affected by

• the number of questions asked,
• the intrusiveness of the questions,
• the number of times the respondent is contacted (for the same survey or for many surveys),
• the detail of information requested (for example, if asked for a precise income figure, respondents need to consult their official documents, but if asked to choose between five different income ranges, they can answer more easily),
• the time it takes to complete the survey.

Choosing the type of data collection

The level of precision and details pursued and the resources available will determine the choice of the type of data collection. Advantages and disadvantages of different types of collection have been presented on the section on the type of data.

The type of collection chosen often depends on the budget available. Costs are one of the main justifications for choosing to conduct a sample survey instead of a census. With sample surveys, it is possible to obtain valuable results with a relatively small sample of the target population. For example, if you need information on all Canadian citizens over 15 years of age, a survey of a small number of these (1,000 or 2,000 depending on the data requirements) might provide adequate results.

Another advantage of using a sample survey is that it allows investigators to produce information soon after they have identified the need for it, within a rapid turnaround time. For example, if an organization wants to measure the public awareness created through an advertising campaign, it should conduct a survey shortly after the campaign is undertaken. Since using a sample of the target population requires a smaller scale of operations, it reduces the data collection and processing time, while allowing more time for planning and quality control.

Minimizing error

When planning a survey, you must be aware of potential sources of error and try to reduce them as much as possible.

In a sample survey, the variation that exists between different samples causes uncertainty called sampling error. For example, let’s say you are estimating the average distance between home and school for students in your class of 25 from a sample of 5 persons. Your estimate will depend on which 5 students are sampled. If all 5 sampled students live very close to the school, the results will not be representative of the whole class. It’s the variation from one sample to another that causes the sampling error.

As a general rule, the more people surveyed, the smaller the sampling error will be. It is often possible to estimate the sampling error associated with a particular sampling plan, and try to minimize it.

By choosing to do a census, you can avoid errors related to sample variation, but not the other sources of errors, called non-sampling errors. For example, a question might be asked in a way that encourages a certain answer or an error might be made while processing the data or calculating a percentage for a table of results. These types of error must be avoided as much as possible by paying attention to quality control throughout every step of the survey process.

These two types of errors will be discussed in greater details in the section devoted to estimation.

Sample size

Since every sample survey is different, there are no hard and fast rules for determining sample size. The deciding factors are time, cost, operational constraints and the desired precision of the results. Evaluate and assess each of these issues and you will be in a better position to decide the sample size. Also, consider what should be the acceptable level of error in the sample. If there is one characteristic that is central to fulfill the survey objectives and there is a lot of variability of this characteristic in the population, the sample size will need to be bigger to obtain the specified level of precision.

Analysis plan

After identifying all the elements or variables to be measured and preparing the sample design, the next step is the analysis plan—conceiving what the results tables will look like. In other words, you need to plan the tables that you will create for the survey variables. These tables will not yet contain any data, but will show any cross-tabulations you want to make.

These tables help you verify whether the questions you are considering will allow you to reach the survey objectives. They illustrate concretely how the collected information will be used and whether it will adequately measure what you want to know.

Questionnaire design

The questionnaire design is based on the survey’s data requirements and analysis plan. To formulate the questions, it can be helpful to consult the people who will be using the results. You can also consult subject matter experts or look at questions from other surveys on similar topics or themes. Good practices for questionnaire design and testing will be presented in the data collection subsection.

Data collection methods

The choice of the method to gather the required data will have a direct impact on cost, human and material resources, time needed to carry the survey and to assess the quality of data. A first option is the interview, which can be face-to-face or by telephone, with or without computer assistance. The second option is the self-completed questionnaire, which can be paper or online.

Personal interviews are administered by a trained interviewer and can have either a structured or unstructured line of questioning. When done by telephone, questions are structured in a formal interview schedule.

The self-completed questionnaire must be highly structured as the respondent will not have as much help as he would have with an interviewer. It can be returned by mail, through a drop-off system or completed online.

Data processing plan

This step deals with transforming the questionnaire responses into output. The tasks involved in data processing include coding, capture, editing, imputation and the creation of derived variables. In short, the aim of this step is to produce a file of data that is free of invalid and missing values and that can be used for estimation and data analysis.

Quality control

This process aims to identify errors and verify results. No matter how much planning and testing goes into a survey, something unexpected will happen. As a result, no survey is ever perfect. Quality control tasks are required to minimize non-sampling errors introduced during various stages of the survey. These tasks include interviewer training, computer program testing, follow-up of non-respondents, and spot-checks of collected responses and output data. Statistical quality-control programs ensure that error levels are kept to a minimum.

Analysis and dissemination of results

After planning data collection and processing, look ahead to the final steps in analyzing and disseminating the results:

• organizing the data using frequency distribution tables;
• summarizing the data using measures of central tendency and measures of dispersion;
• displaying the data through different graph types; and
• writing up the survey’s findings and then disseminating them to the public.

Sections on data exploration and data visualization will present some of these steps with more details.