Section 2
Nonresponse

View the most recent version.

Archived Content

Information identified as archived is provided for reference, research or recordkeeping purposes. It is not subject to the Government of Canada Web Standards and has not been altered or updated since it was archived. Please "contact us" to request a format other than those available.

Archived

This page has been archived on the Web.

Response, nonresponse and vacancy rates
Nonresponse according to urbanization level
Nonresponse according to income strata
Adjustment for nonresponse

Errors due to nonresponse result from the fact that some potential respondents do not provide the necessary information or the provided information proves to be unusable. When the respondent has failed to respond to only some questions, this is referred to as partial nonresponse. In such a case, the missing data are imputed. Errors associated with imputation are described in Section 5, which deals with processing errors. In the present section, nonresponse includes collection nonresponse, which is mainly due to the inability to contact the household or to the refusal of the members of the household to participate partially or completely in the survey, as well as data collected from households that prove to be unusable.

The main impact of nonresponse on data quality is that it can introduce a bias in the estimates if the characteristics of respondents and nonrespondents differ and the difference has an impact on the characteristics studied. Nonresponse rates may easily be calculated, but they have only an indicative value with regard to data quality, since they do not allow estimation of the bias associated with the estimates. The scope of nonresponse may be considered as an indicator of the risks of bias in the estimates.

2.1 Response, nonresponse and vacancy rates

Since the units selected in the SHS are dwellings, interviewers must first identify ineligible dwellings, that is, dwellings occupied by persons who are not part of the target population, as well as dwellings that no longer exist (demolished, mobile home moved or dwelling converted to business) and vacant dwellings (unoccupied, seasonal or under construction).

Among eligible dwellings, the proportion of households that did not respond to the survey is evaluated next. This is called the collection nonresponse rate. Included are households that refused to participate in the survey and households where no contact could be made with the respondents, either because they were absent or because of special circumstances (language problem, illness, death).

Again among eligible dwellings, the rate of unusable data is determined. Unusable data refers to the number of households whose questionnaires were at least partially completed but which were rejected during data processing. When many questions on income or expenditures have been left unanswered, the questionnaire is classified as incomplete and is not used.

Note that in the years prior to the 2006 SHS survey, a data quality control measure called the balance edit check was used. This measure identified the records where the expenditure reported was more than 20% different from the sum of income and net change of assets for a household. The interviewer or senior interviewer was instructed to attempt to collect additional information to try to balance the expenses with income and changes in assets within 15%. Unbalanced questionnaires (more than 20%) at the processing stage were deemed unusable and were not included in estimates. In the 2006 SHS, with the introduction of computer assisted interviews, the balance edit was not used at the collection stage. Instead a number of automatic edits flagged entries that seemed unusual or inconsistent. However, when the balancing was applied at the processing stage, the number of unbalanced questionnaires in the 2006 SHS increased significantly, from 546 questionnaires for the 2005 reference year to 4,300 or 29.4% of the 14,635 completed questionnaires for 2006.

Discarding as unusable this number of questionnaires would seriously risk biasing the results, so a careful analysis was done comparing the balanced and the unbalanced questionnaires. There were few significant differences in the average and percentage reporting of expenses between the balanced and the unbalanced questionnaires. Most of the difference lay in the income and the change of assets reported on the unbalanced responses. We concluded that we could include the unbalanced questionnaires in the estimates of expenses, but users should note that the quality of the income and change of assets may be lower than in previous years.

For the 2007 SHS, the electronic questionnaire will be modified to re-introduce the balance edit feature and ensure identification and correction of out-of-balance records during the interview and collection, as in previous years. The rates of unusable data shown in the tables of this document are therefore lower than those of previous years, since they do not take out-of-balance questionnaires into account.

Note that all rates provided in this section are unweighted. For the 2006 Survey of Household Spending, the final response rate is 71.6%. Table 2.1-1 shows the final response rate as well as the sample size (eligible households) broken down by refusals, units not contacted, unusable data and usable data. This rate is provided at both the national and the provincial level.

Table2.1-1 Sample size and response rate (%) by province and at the national level

Table 2.1-2 shows the final nonresponse rate; the collection nonresponse rate, broken down by refusals and units not contacted; and the rate of unusable data owing to incomplete questionnaires. The vacancy rate is also included. These rates are provided at the national and provincial level.

Note that the vacancy rates shown in tables of Section 2 include vacant dwellings (unoccupied, seasonal or under construction) as well as dwellings that no longer exist (demolished, mobile home moved or dwelling converted to business).

Table 2.1-2 Nonresponse and vacancy rates (%) by province and at the national level

The final nonresponse rate in Canada is 28.4%. It is due to refusals (19.9%), to households that could not be contacted (6.5%), and finally to households for which the data were unusable (2.0%). For each province, refusals are the main cause of nonresponse, followed by units not contacted and by the households for which the data were unusable.

The final nonresponse rate varies from one province to another. Newfoundland and Labrador has the lowest nonresponse rate at 24.1%. Saskatchewan has the lowest rate of units not contacted (4.4%). The rate of unusable data is very low in Quebec at 1.2%. The nonresponse rates in Newfoundland and Labrador, Manitoba and Saskatchewan are all less than 25%, while rates in excess of 30% are observed in Ontario, Alberta and British Columbia. The latter provinces have an especially high refusal rate, combined with a rate of units not contacted that is above the national average.

Vacancy rates are shown in Table 2.1-2, but it should be kept in mind that vacant dwellings do not contribute to the bias of the sample if they are correctly identified. By analysing vacancy rates, we can detect dwelling identification problems associated with the collection process. The national vacancy rate for the 2006 SHS is 13.4%.

2.2 Nonresponse according to urbanization level

Nonresponse varies according to urbanization level. The various rates at the national scale are shown by urbanization level in Table 2.2¹.

Table 2.2 Nonresponse and vacancy rates (%) by urbanization level

The final nonresponse rate generally increases with urbanization level. The urbanization category "250,000 to 499,999" has the highest rates of units not contacted (9.7%), refusal (22.8%) and unusable data (2.5%).

The collection nonresponse rate also tends to increase with urbanization level. There is a difference of nearly 9% between the urbanization categories "6,000 to 30,000" and "1,000,000 or more." Refusals account for more than 50% of total nonresponse at each level of urbanization.

From an examination of the vacancy rate by urbanization level, it emerges that the vacancy rate in rural areas (25.6%) is at least twice that for low-population urban areas (11.9%). These areas also have a higher vacancy rate, on average, than higher-population urban areas. This phenomenon is also observed in the Labour Force Survey (LFS) and is probably attributable to a greater number of seasonal dwellings in rural areas. This explains, among other things, why the vacancy rate is higher in the Atlantic provinces, as illustrated in Table 2.1-2, and especially in Prince Edward Island, since that province has a higher proportion of rural dwellings in the sample.

2.3 Nonresponse according to income strata

Since income information is not available for nonrespondents, it is not possible to compare nonresponse rates according to income. However, the LFS sample design, used for the SHS, was constructed in such a way that in census metropolitan areas, there are strata consisting of geographic areas with a high concentration of high-income households. While the number of high-income strata remains relatively small (51 out of a total of 1,060 strata), the comparison of response rates in this group in relation to the other strata provides relevant information on the potential effect of nonresponse (see Table 2.3).

Note that in addition to regular strata, the "Other" strata category includes the following four types of strata: strata with a high vacancy rate, high-cost strata, strata with a concentration of immigrants, and strata with a concentration of Aboriginals. Since the portion of the SHS sample allocated to the latter four strata was smaller, the results for them are not broken out in Table 2.3.

Table 2.3 Comparison of nonresponse and vacancy rates (%) in high-income strata in relation to other strata

In high-income strata, the final nonresponse rate (37.8%) is approximately 38% higher than in the other strata. The refusal rate for high-income strata is 28.0%, which is higher than for the other strata. High-income strata and the other strata also have different rates of unusable data.

As may be seen, the vacancy rate is lower for high-income strata than for the other strata. This phenomenon was also observed for previous surveys.

2.4 Adjustment for nonresponse

To compensate for nonresponse, the weights in the SHS are inflated by the inverse of the weighted response rate within certain predefined groups. Following the overhaul of the LFS sample design, the nonresponse adjustment groups were redefined. As was the case for previous years, these groups are defined on the basis of the different urbanization levels in each province and of sub-provincial geographic areas for Quebec, Ontario and British Columbia. Also, specific nonresponse adjustment groups were created for high-income strata. As was seen in the previous section, the number of high-income strata was increased with the introduction of the new LFS sample design. Thus, it is now possible to form nonresponse adjustment groups for high-income strata in all provinces except Prince Edward Island. Such a group cannot be formed in Prince Edward Island because that province has no high-income strata.

The weighted rates differ from the rates presented in this section, since the former take the sampling weight of each household into account. An algebraic description of the adjustment for nonresponse is provided in Appendix A.

The adjustment of weights for nonresponse serves to take account of differences in nonresponse by urbanization level (as illustrated in Section 2.2) and geographic area or by groups of high-income strata. It will serve to reduce the bias insofar as the characteristics of respondents and non-respondents are similar for a given urbanization level and geographic area or for a given group of high-income strata.

It should be noted that a nonresponse adjustment group can be combined with another group if the number of households in the group is too small or the adjustment factor is too high.

---

Note

1. Tables on nonresponse rates by urbanization level and province are available on request from the Household Survey Methods Division.