Concepts and Methods Guide
7. Data quality

Skip to text

Text begins

7.1 Overview of data quality evaluation

The objective of the 2017 Aboriginal Peoples Survey (APS) is to produce quality estimates in the areas of employment, health, education and other core indicators for First Nations people living off reserve, Métis and Inuit aged 15 years and over.

Sections 7.2 and 7.3, below, explain the two types of errors that occur in surveys: sampling and non-sampling errors. Each type of error is evaluated in the context of the 2017 APS. Sampling error is the difference between the data obtained from the survey sample and the data that would have resulted from a complete census of the entire population taken under similar conditions. Thus, sampling error can be described as differences arising from sample-to-sample variability. Non-sampling errors are all other errors that are unrelated to sampling. Non-sampling errors can occur at any stage of the survey process, and include non-response for the survey as well as errors introduced during data collection or computer processing. Respondents may have made errors in their responses, trying to recall facts from the past, for example, or when a proxy stands in for a respondent. A response may have been incorrectly captured due to interviewer fatigue or a computer malfunctioning. An error may have been made in programming when the data were being processed or totaled. These are all examples of non-sampling errors.

This chapter describes the various measures adopted to prevent errors from occurring wherever possible and to adjust for any errors found throughout the different stages of the APS. Areas of caution for interpreting APS data are noted.

7.2 Sampling errors and bootstrap method

The estimates that can be derived from the 2017 APS are based on a sample of individuals. Somewhat different estimates might be obtained if a complete census had been taken using the same questionnaires, interviewers, supervisors, processing methods, etc. as those actually used. The difference between an estimate obtained from the sample and the one resulting from a complete count taken under similar conditions is called the “sampling error” of the estimate.

In order to provide estimates of sampling error for statistics produced in the APS, a particular type of bootstrap method (the bootstrap being itself a specific resampling method), was developed. Several bootstrap methods exist in the literature but none of them was appropriate for the APS sampling design. The particularities of the APS design that made the estimation of sampling errors difficult were the following:

• Two-phase sampling design in which households (or dwellings) were selected in the first phase and individuals in the second phase (section 3.2.3);
• The sampling fraction of the first phase sample (census long form sample) was non negligible (about one-fourth in the  2A-L regions) and the APS sampling fraction was generally relatively high in most strata;
• The APS strata (combinations of domains of estimation, 2A-L or 2A-R form type, census self-respondent vs. NRFU respondent, identity vs. ancestry-only) were not nested within the census strata (collection units);
• The method used had to be flexible enough to produce standard statistics such as proportions, totals, means and ratios but also more sophisticated statistics, including percentiles, logistic regression coefficients, etc.

Several bootstrap methods exist in the literature for single-phase sampling and for multi-stage sampling. The most common one is called the “with-replacement bootstrap” and consists of selecting M with-replacement subsamples from the main sample and producing estimates for each subsample. The bootstrap variance estimate is then derived as a function of the squared differences between estimates coming from each of the M bootstrap subsamples and the estimate coming from the survey sample.

Variance calculation is greatly simplified through the use of bootstrap weights. For each subsample (bootstrap replication), the initial sampling weight first has to be adjusted for bootstrap subsampling, which produces what is called “initial bootstrap weights”. Since each bootstrap sample is drawn by selecting the units with replacement, a unit can appear several times in a particular bootstrap sample. It can be shown that the bootstrap weights are a function of the initial sampling weight of the observation multiplied by what is called “the multiplicity” of the unit in the bootstrap sample, which is the number of times the unit is selected in the bootstrap sample. The multiplicity of a unit in the bootstrap sample is a random variable following what is called a “multinomial distribution”. Hence, the bootstrap weights can be seen as the product of the initial sampling weights by a random adjustment factor (in this case, a function of the multiplicity of the unit). Once initial bootstrap weights have been derived, all weight adjustments applied on the initial sampling weights are applied to the initial bootstrap weights to obtain the final bootstrap weights, which will capture the variance associated with not only the particular sampling design but also the variance associated to all weight adjustments applied to the full sample to derive the final weights.

For the 2006 APS, a general bootstrap method for two-phase sampling^Note was developed. In 2006, the first phase of sampling corresponded to the 2006 Census long-form sample while the 2006 APS corresponded to the second phase sample. As mentioned earlier, bootstrap weights can be seen as the product of the initial sampling weight by a random adjustment factor. This is the idea behind the general bootstrap methodology used in 2006. In the case of that two-phase sample, the variance was decomposed into two components, each one associated to a phase of sampling. The general two-phase bootstrap methodology produced a random adjustment factor for each phase of sampling. In the case of the 2006 APS, the initial bootstrap weight of a unit was the product of the initial sampling weight by these two random adjustment factors.

For the 2012 APS, this general bootstrap method was adapted to account for the National Household Survey (NHS) sampling design which itself included two phases: the initial sample of approximately 1 in 3 dwellings as the first phase and the sub-sample of non-respondents on which non-response follow-up (NRFU) was conducted as the second phase. A more detailed description of the NHS sampling design is found in chapter 3 of the National Household Survey User Guide. In the 2012 APS, for the purpose of calculating variances only, NRFU respondents were considered a third phase sample. These three phases of the NHS were then combined into a single phase and the general two-phase bootstrap methodology (one NHS phase and one APS phase) was applied. More details can be found in the Aboriginal Peoples Survey, 2012: Concepts and Methods Guide.

With the return of the 2016 Census long form, the 2017 APS could have used the general bootstrap method as was done in 2006. The first phase would have consisted of the census long-form sample while the second phase would have consisted of the 2017 APS sample. However, to increase the precision of the variance estimation, a modified version of the 2012 approach was utilized. For the purpose of calculating variances only, the 2016 Census was seen to have two phases: the initial sample of approximately 1 in 4 dwellings as the first phase and census respondents as the second phase. Although the final response rate was quite high for the 2016 Census (97.8% for the long form), this second phase ensures that the variance calculation takes into account the non-response that occurred.

For the 2017 APS, the two phases of the census were combined into a single phase using the same methodology as in 2012.  The general two-phase bootstrap methodology (one census phase and one APS phase) was then applied, which involved calculating two sets of random adjustment factors; therefore, one set for each phase.

The presence of these two sets of random adjustment factors had a major advantage. The first set could be used for estimates based on the first phase only, that is, estimates based on the census long-form sample. These estimates were used when the weights were adjusted based on the census totals at the time of post-stratification (section 6.5). This produced variable census totals for each bootstrap sample and reflected the fact that census totals were based on a sample and not on known, fixed totals.

For the APS, 1000 sets of bootstrap weights were generated using the method described above. The method used is slightly biased upward in the sense that it slightly overestimates the variance. However, the amount of overestimation was found to be negligible for the APS. The method can also lead to negative bootstrap weights. To overcome this problem, a transformation was done on the bootstrap weights that reduced their variability. Therefore, the variance calculated on these transformed bootstrap weights has to be multiplied by a factor that is a function of a certain parameter, called phi. The value of the parameter corresponds to the smallest integer that makes all bootstrap weights positive. For the APS, this parameter has a value of 4. The variances calculated on the transformed bootstrap weights have to be multiplied by 4² = 16. In addition, the CVs obtained (square root of the variance divided by the estimate itself) have to be multiplied by 4. However, most software which produce sampling error estimates from bootstrap weights have an option to specify this adjustment factor such that the correct variance estimate is obtained without the need of an extra step to multiply by the constant.

Note that if C is the variance multiplicative factor, some software use the parameter k instead where k = 1-1 ⁄ √C. In our case, since C=16, then k=0.75. For examples of procedures using the Fay adjustment factor, see the Aboriginal Peoples Survey, 2017: User’s Guide to the Analytical File.

The sampling error measure used for the APS is the CV of the estimate, which is the standard error of the estimate divided by the estimate itself. When the CV of an estimate is less than or equal to 16.6%, the estimate can be used without restriction. In this survey, when the CV of an estimate is greater than 16.6% but smaller or equal to 33.3%, the estimate will be accompanied by the letter “E” to indicate that the data should be used with caution. When the CV of an estimate is greater than 33.3%, or if an estimate is based on less than 10 units, the cell estimate will be replaced by the letter “F” to indicate that the data is suppressed for reasons of reliability.

7.3 Non-sampling errors

Besides sampling, a number of factors at almost every stage of a survey can cause errors in survey results. Non-sampling errors arise primarily from the following sources: non-response, coverage, measurement and processing. For each of these areas, the following sections discuss the various measures used to minimize and correct error. For example, measurement errors may be due to respondents misunderstanding the questions and answering them inaccurately; also responses may be entered incorrectly during data capture and errors may be introduced in the processing and tabulation of data. Using Computer Assisted Interviewing (CAI) in 2017 reduces the level of non-sampling error because CAI allows for the direct capture of responses, automated flows between questions, built in edits which eliminate inconsistencies and outliers, etc. (for more information on CAI, please refer to section 2.1).

Over a large number of observations, randomly occurring errors will have little effect on the estimates from the survey. However, errors occurring systematically will contribute to biases in the survey estimates. Thus, much time and effort was devoted to reduce non-sampling errors in the survey as described in the following sections.

7.3.1. Non-response errors

Non-response errors result from a failure to collect complete information on all units in the selected sample. Non-response produces errors in the survey estimates in two ways. First, non-respondents often have different characteristics from respondents, which can result in biased survey estimates if non-response is not corrected properly. The larger the non-response rate, the larger the risk of potential bias will be. Second, having a larger number of non-respondents reduces the effective size of the sample. As a result, the precision of the estimates decreases (the sampling error on the estimates will increase). This second aspect can be overcome by selecting a larger sample size initially. However, this will not reduce the potential bias in the estimates.

There are many types of non-response. One form of non-response is item non-response (or partial non-response), where the respondent does not respond to one or more questions, but has completed a significant portion of the overall questionnaire. Item non-response can be due to difficulty understanding a particular question.

Generally, the extent of item non-response was relatively small in the APS. Extensive qualitative reviews and testing of questionnaire was done prior to the survey, hence reducing the extent of item non-response. A response to key pre-defined questions was required before a case was classified as “respondent” as described in section 5.3.1. There were some cases, however, where a large proportion of responses to key questions were missing. These cases were eliminated from the database of respondents (did not satisfy definition of respondent) and were treated during weighting as a special case of total non-response (see section 6.4). Finally, there is total non-response when the person selected to participate in the survey could not be contacted or did not participate once contacted. Weights of respondents were inflated in order to compensate for those who did not respond as described in section 6.3.

To mitigate the number of non-response cases, many initiatives were undertaken. In the months leading up to the survey, a comprehensive communications strategy was implemented to encourage participation as described in section 4. In addition, in-depth interviewer training was conducted. Interviewer training in conjunction with detailed interviewer manuals was done by experienced Statistics Canada training staff, who oversaw activities in the field. Efforts to reach non-respondents through call-backs and follow-ups were also made by senior interviewers to encourage respondents to participate in the survey. When possible, additional telephone numbers were provided to maximize the chances of reaching a respondent during collection. These numbers were obtained using administrative files as well as the most recent version of the residential telephone file^Note at Statistics Canada. Field follow-up, using CAPI interviewers, was also conducted in many specific regions.

A detailed table of final response rates obtained for the 2017 APS is provided in section 3.3 of this guide (Table 4).

7.3.2. Coverage errors

As mentioned in section 3.1, the target population of the 2017 APS was the Aboriginal identity population of Canada, aged 15 years and over as of January 15, 2017, living in private dwellings, excluding persons living on Indian reserves or settlements and in certain First Nations communities in Yukon and the Northwest Territories. The population sampled or covered by the survey corresponded to 2016 Census long-form respondents reporting Aboriginal ancestry or identity (see section 3.1.1) with the same restrictions as those for the target population in terms of age and geography. For data on First Nations people living on reserve, researchers are directed to use the 2016 Census.

Coverage errors occur when there are differences between the target population and the sampled population (population covered by the frame). Over-coverage is generally not an issue since out of scope units in the sample are typically identified during data collection and can be estimated for the entire survey frame. However, under-coverage can exist. Because the APS sample was selected from those who had participated in the 2016 Census, individuals who did not participate in the Census could not be sampled for the APS. If this group of individuals is significantly different than the ones who participated in the Census with respect to the characteristics measured in the APS, a bias could be introduced. This bias is assumed to be relatively small given the very high response rate obtained in the census (97.8% response rate for the long form) and given the adjustments made on the initial census sampling weights. 

7.3.3. Measurement errors

Measurement errors occur when a provided response differs from the real value. Such errors may be attributable to the respondent, the interviewer, the questionnaire, the collection method or the respondent’s record-keeping system. Extensive efforts were made to develop questions for the 2017 APS which would be understood, relevant and culturally sensitive.

Following the release of data from the 2012 APS, an extensive content review was conducted of 2012 APS questions. The review brought together expertise from a diverse group of researchers and subject matter experts from within and outside of Statistics Canada. An analysis was conducted on which questions worked the best and which were most effective in producing valid indicators. This process also extended into an extensive search for relevant questions from other standardized survey questions at Statistics Canada.

Questions selected for potential inclusion on the 2017 questionnaire then underwent several rounds of qualitative testing using one-on-one interviews with respondents in ten different communities across various regions of Canada, including Iqaluit and Yellowknife. Testing was done among First Nations people, Métis and Inuit. Qualitative testing of the survey questionnaire was carried out by Statistics Canada’s Questionnaire Design Resource Centre (QDRC). To minimize measurement error, adjustments were made to question wording and flows based on those results.

Many other measures were also taken to specifically reduce measurement error, including the use of skilled interviewers, extensive training of interviewers with respect to the survey procedures and content, and observation and monitoring of interviewers to detect problems of questionnaire design or misunderstanding of instructions.

7.3.4. Processing errors

Processing errors may occur at various stages of the survey process including data capture, coding and editing. Quality control procedures were applied to every stage of APS data processing to minimize this type of error.

At the data processing stage, a detailed set of procedures and edit rules was used to identify and correct any inconsistencies between the responses provided. A set of thorough, systematized procedures was developed to assess the quality of every variable and to make corrections to any errors found. A snapshot of the output files was taken at each step and verification was done by comparing files at the current and previous step. The programming of all edit rules was exhaustively tested before being applied to the data. Some examples of the data processing verifications were:

• the review of all question flows, including very complex sequences, to ensure skip values were accurately assigned and distinguished from different types of missing values;
• quality control double-coding of “Other-specify” responses;
• experienced supervision of coding to standardized classifications; and
• the review of all derived variables against their component variables to ensure accurate programming of derivation logic, including very complex derivations.

See the data processing chapter (section 5) of this guide for more details.

• Previous
• Next