Section 3
Coverage errors

View the most recent version.

Archived Content

Information identified as archived is provided for reference, research or recordkeeping purposes. It is not subject to the Government of Canada Web Standards and has not been altered or updated since it was archived. Please "contact us" to request a format other than those available.

Archived

This page has been archived on the Web.

Undercoverage and overcoverage: slippage rates
Adjustment at the population and household levels

The target population was defined in the design of the survey. It is useful to go over this definition, since a good understanding of the target population is necessary in order to properly interpret the survey data. It is important to note that the Survey of Household Spending (SHS) uses the sampling frame of the Labour Force Survey (LFS).

Target population

We did not collect data from persons temporarily living away from their families (for example, students at university) because the information would be obtained from their families if selected in the sample.

Coverage errors result from inadequate representation of the target population based on the units in the sampling frame. Some units of the target population may be omitted from the sampling frame, in which case there is undercoverage. Other units that are not in the target population may be included by error, or some units may be included more than once. These units are responsible for overcoverage.

3.1 Undercoverage and overcoverage: slippage rates

In the SHS, the sample is selected using a list of dwellings in each selected cluster. Factors contributing to undercoverage are: the omission of dwellings in the creation of the list, new dwellings that are added between the creation of the list and the interviewer's visit (mainly in developing areas), and the erroneous classification of vacant dwellings. The inclusion of dwellings that are not within the boundaries of the cluster is a source of overcoverage. Similarly, errors can occur during data collection, due to improper identification of persons as members of the selected household. These errors also contribute to undercoverage or overcoverage.

Also, as described in Section 2.4, reweighting methods are implemented to take account of nonresponse. However, when these adjustments are made, it is impossible to correct the survey weights to ensure that all subgroups within the population are well represented.

A good representation of the target population is essential to the production of realistic expenditure estimates. The sample must adequately represent the individuals in the target population as well as the distribution of households according to their size.

There is generally a net undercoverage of the number of persons and the number of households in the SHS. This undercoverage is corrected by an adjustment of weights using auxiliary or reference data based on post-censal demographic estimates. The slippage rate (see Appendix A) is a measure of the percentage of difference between the estimates from these auxiliary data and the survey estimates calculated using weights not adjusted with these data¹. Slippage therefore represents the combined effect of undercoverage and unbalance in certain subgroups of the population created by survey nonresponse that could not be corrected at the reweighting stage.

For the 2004 survey, several changes were made to the approach with a view to adjusting the weights by using auxiliary data. First, post-censal demographic estimates now come from the 2001 Census rather than the 1996 Census; this makes the 2004 slippage rates hard to compare with those from previous surveys. Also, changes were made to the weights adjustment strategy [6]. For example, in previous surveys, demographic adjustments were made in cross-tabulations of nine age groups by sex, whereas the new strategy is limited to eight age groups. Analyses showed that sex does not appear to be associated with household expenditures, since households are generally made up of persons of both sexes. Therefore, adjustments according to sex do not serve to improve the quality of the estimates. However, it is clear that sex has an effect on expenditures for one-person households. But since there are no annual demographic statistics on the number of one-person households, broken down by sex, it is not possible to make adjustments for this particular case.

Slippage rates by age group and sex at the national level

Slippage rates by age group and sex at the national level are shown in Table 3.1-1. A positive rate indicates overcoverage of the number of persons in the survey.

Table 3.1-1 National slippage rates by age-sex group, Canada

For the 2004 SHS, the national undercoverage rate was 9.3%. The slippage rates for children (aged 0 to 6 and 7 to 17) are quite different from those for other age groups. The undercoverage rate for all children combined is 0.1%, while it is 11.9% for adults (data not shown). Also, for girls aged 0 to 17, a slight overcoverage is observed. For girls aged 0 to 6, this is due to the overcoverage obtained in Ontario (4.3%), Alberta (13.0%) and British Columbia (14.9%) for this age-sex group (see Table 3.2-1). Similarly, the overcoverage for girls aged 7 to 17 is due to the overcoverage obtained in Quebec (10.9%) and Alberta (16.7%) for this age group.

The highest national rates occurred among men aged 18 to 24, 25 to 34 and 55 to 64. Note that the undercoverage rate for women is consistently lower than the corresponding rate for men.

As mentioned previously, the SHS uses the LFS sampling frame. Over the same period, the national LFS undercoverage rate was 9.8% (reference [5]). This is slightly lower than the 11.6% SHS rate for those aged 15 and over (data not shown).

Since sampling weights are no longer adjusted according to the sex of the persons in respondent households, net undercoverage or overcoverage by age-sex group persists. This net under- or overcoverage is measured by the residual slippage rate. The residual slippage rate is a measure of the percentage of difference between estimates from the auxiliary data and the survey estimates, which this time are calculated using the final weights. Residual slippage rates by age-sex group at the national level are shown in Table 3.1-2.

Table 3.1-2 National residual slippage rates by age-sex group, Canada

Residual slippage rates for males and females are necessarily of opposite signs, since the weights were adjusted to correspond to the benchmark demographic estimates for each age group. Although sex is no longer taken into account in adjusting the weights, the fact remains that controlling for age greatly reduces the size of the undercoverage by age-sex group observed in Table 3.1-1. After adjusting the weights by age group, the residual slippage rates are less than 1.2% for adults and less than 1.9% for children at the national level.

Provincial slippage rates by age group and sex

Slippage rates by age group and sex at the provincial level are shown in Table 3.2-1.

Table 3.2-1 Slippage rates for provinces by age-sex group

We observe net undercoverage for all provinces, with the rates varying from 4.1% to 14.2%.  Alberta has the lowest undercoverage rate at 4.1%. However, a low overall rate of undercoverage is not a guarantee of better coverage. For example, the overall slippage rate observed in Alberta (4.1%) conceals the worst case of overcoverage at the provincial level among age groups (18.2% for boys) and the worst cases of undercoverage for the 55-64 age group.

Among the provinces, the highest undercoverage rate occurred among 18 to 24 year-old men in Newfoundland and Labrador (34.4%). Also, 18 to 24 year-old men had the highest undercoverage rates in the other Atlantic provinces and in British Columbia. Although this age group was found to have the highest undercoverage rates, the highest rate at the national level instead occurred among men aged 25 to 34 (see Table 3.2-1). This is because in Quebec, among men aged 18 to 24, there was no coverage error (to be more precise, there was a slippage rate of -0.014%) prior to adjustment of the weights according to age groups. Prince Edward Island had the highest slippage rates for the 65 and over age group. Another point worth noting is that the pattern of slippage rate variation differs substantially for age-sex groups from one province to the next.

Residual slippage rates by age-sex group at the provincial level are shown in Table 3.2-2.

Table 3.2-2 Residual slippage rates for provinces by age-sex group

Just as for the national results, the adjustment of weights by age group tends to moderate the size of the under- or overcoverage by age group seen in Table 3.2-1. The highest residual slippage rates are on the order of 10% to 12%. These rates occurred among children aged 0 to 6, children aged 7 to 17 and adults aged 18 to 24. As noted above, analyses have shown that sex does not appear to be a factor in household expenditures except perhaps in the case of one-person households, and almost none of the persons in the latter three age groups constitute one-person households. Thus, while the residual slippage rate is higher for these age groups, there would likely be no bias in the estimates owing to poorer representativeness in these age groups.

It should also be noted that the overall slippage rates for females and males in each province are relatively low.

Slippage rates by household size

Table 3.3 shows the slippage rates by household size that were used in adjusting the weights. A negative rate corresponds to undercoverage of the number of households in the survey.

Table 3.3 Slippage rates for provinces by household size

Nationally, the number of households was underestimated by 8.3%. This underestimation is slightly lower than the 9.3% underestimation observed for the number of individuals. Both nationally and provincially, undercoverage is observed for all sizes of household. Nationally, there are few differences in undercoverage according to household size, with rates ranging from 7.9% to 9.0%. 

Provincially, there is greater variation in the slippage rate. However, the gap observed is nearly the same between household sizes. For one-person households, the rate varies from -2.2% for Nova Scotia to -16.7% for Newfoundland and Labrador For two-person households, the slippage rate ranges from -2.3% in Saskatchewan to -13.7% in Ontario. For households of three or more persons, the slippage rates vary from -2.2% for Alberta to -16.5% for Saskatchewan.

Except for Ontario, undercoverage was highest for one-person households and households of three or more persons, at nearly equal levels.

3.2 Adjustment at the population and household levels

To correct the problem of the sample's representativeness, shown in Table 3.1-1 and 3.2-1, and to reduce the resulting bias, the survey data are adjusted during weighting using demographic estimates for the age groups defined in these tables, for each province. For more details on the adjustment methodology, see references [1] and [6]. This adjustment reduces the bias but does not eliminate it entirely if the characteristics of the individuals omitted from the survey differ from those of individuals included for a given age group in a province.

It should also be noted that the effectiveness of the adjustment based on demographic estimates depends largely on the quality of those estimates and their accuracy in representing the target population of the survey. The demographic estimates are not error-free. They are post-censal estimates based on the population counts from the 2001 Census adjusted for net undercoverage, and they take into account recent statistics on migration, births, deaths, etc. These demographic estimates are adjusted to account for certain exclusions specific to household surveys, such as persons living in institutions. Conceptually, they differ slightly from the SHS target population in that they include persons living in non-institutional collective dwellings, such as members of groups living communally and individuals permanently residing in hotels or rooming houses. However, this difference is considered negligible, since such individuals represent less than 0.4% of the Canadian population.

To remedy the problem of the representativeness of the sample in terms of the number of households by size as illustrated in Table 3.3, the survey data are adjusted using supplementary data. By adjusting the SHS weights to reflect post-censal estimates of the number of households by size, the goal is to compensate for the bias resulting from inadequate representation of households. However, the bias will not necessarily be eliminated if characteristics of households not interviewed (i.e., omitted or non-responding households) differ from those of responding households for a given household size. As in the case of demographic estimates of population, the effectiveness of the adjustment will depend on the quality of the supplemental data on the number of households. 

In addition to demographic estimates of age groups by province, two other sets of supplementary data are used during weighting to adjust survey data and thereby improve their representativeness. The first set of data is used to control for the number of children and adults in certain major cities. For the second set, counts for major categories of income from wages and salaries are used when adjusting weights to ensure a degree of consistency between the income distributions from the SHS and those from outside sources.

---

Note

1. The subweight, which is the survey weight adjusted for nonresponse, is used (see Appendix A).