Income Research Paper Series
Improvements to the Canadian Income Survey Methodology for the 2022 Reference Year

Introduction

The Canadian Income Survey (CIS) has introduced improvements to the methods and data sources used to produce income and poverty estimates with the release of its 2022 reference year estimates. Foremost among these improvements is a significant increase in the sample size for a large subset of the CIS content. Estimates for 2022 and moving forward also include improvements to the weighting methodology and the extension of the target population for the CIS from persons aged 16 years and over to persons aged 15 years and over^Note  .

This paper begins by explaining the changes leading to the increased sample size for the CIS. Next, it describes the updates to the weighting methodology. Finally, it presents the approximate net result of these changes on income estimates for the 2021 reference year. The changes described in this paper highlight the ways in which data quality has been improved while producing small impacts on key CIS estimates and trends.

This paper is a follow up to Improvements to the Canadian Income Survey Methodology for the 2021 Reference Year which detailed key changes to the methodology of the CIS introduced for 2021. The goal of these updates over the past two years was to stabilize and improve current and future income estimates and generate more reliable estimates in smaller domains for the CIS.

Increasing to the full Labour Force Survey sample size

The CIS questionnaire is asked to a sub-sample of respondents to the Labour Force Survey (LFS), gathering information on labour market activity, school attendance, support payments, child care expenses, personal income, disabilities, unmet health care needs, food security, and characteristics and costs of housing.

The LFS uses a rotating panel sample design so that selected dwellings remain in the LFS sample for six consecutive months. Each month, about one-sixth of the LFS sampled dwellings are in their first month of the survey, one-sixth are in their second month of the survey, and so on. Those answering the LFS for the last time (sixth month in the survey) are referred to as the “rotate-out” group. In most years, the CIS questionnaire was asked to rotate-out groups from January to June of the year following the reference year. The final CIS sample included all persons in households that responded to the CIS questionnaire.

In 2022, a new quarterly supplement to the LFS was developed: the Labour Market and Socioeconomic Indicators (LMSI). It is conducted from July to December every year, and similarly to the CIS, the questionnaire is asked for the rotate-out groups in these months. In addition to labour-related content, it includes some of the CIS modules: support payments and child care expenses, which are used in the production of poverty estimates, as well as disability and unmet health care needs.

For the 2022 reference year, the CIS sample size, which would have traditionally only included CIS respondents from January to June 2023, was increased to include the LMSI respondents from July to December 2022, as well the LFS respondents who did not respond to either LMSI or CIS, but were part of the rotate-out groups from July 2022 to June 2023. This population, which corresponds to the LFS sample size for a year, is referred to as “CIS-Plus”, and its goal is to improve and stabilize CIS income and poverty estimates. The CIS-Plus sample is more than double the traditional CIS-only sample.

Description for Figure 1

The figure compares the traditional Canadian Income Survey (CIS) sample versus the Canadian Income Survey Plus (CIS-Plus) sample. The first image displays the traditional CIS sample showing that it includes only CIS respondents from January to June 2023, inclusive. The second image displays the CIS-Plus sample showing that it includes 1) LMSI respondents and non-respondents from July to December 2022, inclusive, and 2) CIS respondents and non-respondents from January to June 2023, inclusive.

The CIS-Plus sample is made possible by linking to income variables from the Administrative Personal Income Masterfile (APIM), which is a comprehensive, centrally processed source of personal income data generated from tax returns and associated tax slips from the Canada Revenue Agency (CRA). Additionally, the CIS content on the LMSI supplement allows poverty estimates to be produced for the whole CIS-Plus sample^Note  . As a result, CIS estimates or tables that only include income or poverty variables, as well as variables derived from the LFS such as demographic information, main job characteristics or family composition, can now be produced using the whole CIS-Plus sample.

Canadian Income Survey tables that include content that is only available from the traditional CIS questionnaire must still be produced using the CIS-only (subset of CIS-Plus) sample. For example, annual labour force information, food insecurity, and most housing-related variables are only available for the CIS-only sample.

Following the methodology of the CIS questionnaire, the disability and unmet health care need modules in the LMSI supplement are asked for one randomly selected person per household. The inclusion of the disability and unmet health care need modules in the LMSI supplement doubles the sample for these estimates (but only for content also available for the CIS-Plus sample, such as income and poverty).

Updates to the weighting strategy

The calibration classes were adjusted to include economic family type, ensuring less frequently surveyed family types such as lone-parent families are properly represented. In addition, the number of social assistance recipients was introduced into the calibration strategy as social assistance recipients appear to be otherwise underrepresented among respondents. Calibration is the process of modifying the weights so that the estimates from the survey for relevant population characteristics respect population totals from other reliable sources. Calibration is applied to correct for coverage and non-response issues.

The first set of population totals used by the CIS are estimates provided by Statistics Canada's Centre for Demography of population counts based on the 2016 Census of Population.  For each province, population counts by age/sex groups and economic family size are used. CIS also employs population counts for six Census Metropolitan Areas (Montreal, Toronto, Winnipeg, Calgary, Edmonton, and Vancouver).

The second set of totals is derived from the APIM and is intended to ensure that the weighted distribution of income (based on wages and salaries) in the data set matches that of the Canadian population.

For CIS 2020 and CIS 2021, two sets of totals were added to account for the changes in response patterns and new sources of income available due to the pandemic. The first set of totals, based on the 2021 Census results, adjusted the weights according to the number of households by housing tenure. The second set of totals, based on APIM estimates, adjusted the weights to reflect the number of Canada Emergency Response Benefit recipients and Canada Emergency Student Benefit recipients in 2020, and the number of Canada Recovery Benefit recipients in 2021.

For CIS 2022, the weight adjustment for the number of households by housing tenure remains. In addition, the weights are adjusted to reflect the number of families by economic family type and size, using estimates from Statistics Canada's Centre for Demography. The following categories are used:

1. Non-senior males not in an economic family
2. Non-senior females not in an economic family
3. Senior males not in an economic family 
4. Senior females not in an economic family
5. Couples without children or relatives
6. Couples with one or more children under 18 years
7. Couples with other relatives only
8. Male lone-parent families 
9. Female lone-parent families
10. Other

Where appropriate, the categories are further grouped by family size of two and three or more.

Further, the weights are adjusted to reflect the number of social assistance recipients. Control totals were created for the number of social assistance recipients by province from the APIM.

Inclusion of 15-year-olds in the CIS income universe

The in-scope age for the CIS income, demographic and labour variables was lowered from 16 years old to 15 years old. This change aligns with the LFS which collects labour information for individuals who are 15 years and older.

Impacts on the Canadian Income Survey estimates

This section will discuss some of the key changes resulting from implementing the methodological improvements described previously. For the tables presented, the published estimates for 2021 were produced before the aforementioned methodological changes, while the published estimates for 2022 incorporated all the improvements.

For analytical purposes, another set of estimates for 2021 (Experimental) were also produced incorporating the methodological changes for the purpose of assessing their impact. As the name suggests, the results of this analysis should be treated as experimental for 2021, as the changes weren’t fine-tuned as they were for the 2022 reference year. Therefore, the experimental estimates for 2021 should not be considered as replacements for the official (Published) estimates.

Note that although the following tables highlight the key impacts on the CIS estimates, other differences can be observed when looking at the data at more disaggregated levels.

Changes to the distribution of economic family types

Updating the weighting strategy to include economic family type in the calibration has an impact on the distribution of families by family type. Among economic families, senior families increased by 161,000 families (+7.5%) while non-senior families decreased by 160,000 families (-2.0%). Lone-parent families, which are traditionally underrepresented on the CIS increased by 161,000 families (+27.4%). Among persons not in an economic family, unattached seniors decreased by 242,000 (-11.8%) while unattached non-seniors increased by 242,000 (+5.7%).



Social assistance recipients increase

With the introduction of social assistance recipients into the calibration strategy, the number of families receiving social assistance benefits on the CIS increased by 34.9%. The increases are statistically significant across almost all family types, ranging from 30.8% for unattached individuals to 83.5% for lone-parent families. Both changes to the calibration strategy, i.e. the addition of the number of families by family type and the addition of the number of social assistance recipients, contributed to the increase in the number of lone-parent families receiving social assistance benefits.



Relatively small changes to income estimates

As seen previously, changes made to the calibration strategy have impacted the distribution of families by family type as well as the number of social assistance recipients. These changes, along with the large increase in the sample size, will in turn have impacts on income estimates. Comparing the official 2021 estimates to the experimental 2021 estimates that incorporate the methodological changes, the experimental estimate of median after-tax income for economic families and persons not in an economic family was 1.2% lower for Canada in 2021. Nova Scotia (-2.8%) and Ontario (-2.8%) experienced decreases in their median after-tax income, while all other provinces remained relatively unchanged.

The introduction of social assistance recipients into the calibration strategy contributes to the decrease in after-tax income. Increasing the number of social assistance recipients to match APIM control totals increases the survey weights for this population. Since social assistance recipients typically have lower incomes on average, this will tend to decrease the median after-tax income.

The impact of changes in the distribution of families by family type is more difficult to assess. For example, the increase in the number of lone-parent families would tend to lower the median after-tax income, but the growth in the number of couples with other relatives would tend to increase the median. Other changes in the distribution of families by family type would also have contrary effects on the median, making it difficult to conclude on the net results of the addition of the family type in the calibration strategy.



For the majority of family types, the median after-tax income remained relatively unchanged. Couples with children had a small increase of 1.7% in their median after-tax income, while other non-senior-families and unattached non-seniors had decreases of 8.6% and 3.5%, respectively.



As could be expected given the previous findings, the new methodological changes also led to a decrease in the median market income. Similar to after-tax income, only couples with children showed an increase in their median market income.



In terms of government transfers, although the overall median remained relatively unchanged, increases were observed for a few family types. Worth mentioning is the decrease in median government transfers for persons not in an economic family, even though the median rose for both seniors and non-seniors not in an economic family. This can be explained by the change in the distribution of family types. As seen in Table 1, the number of seniors not in an economic family decreased, while the number of non-seniors increased. Since non-seniors receive in general much lower government transfers than seniors, the median government transfers for all persons not in an economic family decreased.



A look at poverty and low-income estimates

In this section, we will look at the impact the previously detailed changes have on the poverty and low-income estimates.

It should be noted that since the LMSI survey was only implemented in 2022, it was not possible to derive poverty estimates that include the whole CIS-Plus sample for 2021. Therefore, poverty estimates presented below only include the traditional CIS sample.

Comparing the official 2021 poverty estimate to the experimental estimates produced after implementing the methodological changes, the experimental estimate was 0.3 percentage points higher nationally, and also higher for some provinces and family types. For example, the experimental estimates were higher in Newfoundland and Labrador, Nova Scotia, New Brunswick, Quebec and Ontario. The experimental estimate was lower for senior families (-0.2 percentage points) in 2021, while non-senior families (+0.2 p.p.) and unattached non-seniors (+0.9 p.p.) saw slightly higher experimental estimates. The tendency towards higher poverty estimates under the new methodology is consistent with changes in the methods which yield more lone parent families and families receiving social assistance.

Nevertheless, the methodological updates to include economic family type and social assistance recipients into the calibration strategy result in better estimations for the distribution of family types, as well as more accurate income estimates for historically underrepresented family types and sources of income. This in turn produces a better picture of individuals experiencing poverty on the CIS, while having only a minor impact on the incidence of poverty.

 

Using the low income measure after tax (LIM-AT), it is possible to look at the impact of both the change in the calibration strategy and the use of the whole CIS-Plus sample.



As can be seen in Table 9, the low-income rate is slightly higher when including the whole CIS-Plus sample, although not in all provinces. This could be a result of random sampling variance between the two groups. Comparing with the published estimates, the low-income rate when all methodological changes are taken into account is higher for New Brunswick, Ontario, Saskatchewan, and Canada as a whole.

Impact of including 15-year-olds in the income universe

Extending the target population for income data from persons aged 16 years and over to persons aged 15 years and over has a minimal impact on median incomes at the family level. However, at the individual level, the inclusion of 15-year-olds decreases median incomes, especially for the younger age group (15 to 24 years old). As can be seen in Table 10, the median market income and median total income for those 15 years and over decreased slightly with the addition of 15-year-olds. For the younger age group, the medians were reduced by 6.6% for those with market and/or total income. Accounting for those with zero income resulted in the median market and total income being 17.2% and 15.8% lower, respectively, as 15-year-olds are less likely to earn income.



Impact on data quality

The following tables show the coefficient of variations (CVs) for median after-tax income by province and selected family types. CVs are a widely used measure of sampling error which are the estimated standard error expressed as a percentage of the estimate.

The larger sample size resulting from the implementation of the CIS-Plus improves income estimates, as there is a reduction of the CVs for every province and family type in 2021 and 2022. The CVs tend to improve more for smaller domains, which is important for better quality estimates of smaller populations such as racialized groups or Indigenous people, and less common income sources.

 

Conclusion

The 2022 reference year Canadian Income Survey estimates featured key updates with the goal of improving data quality and accuracy. Notably, the survey sample size increased to utilize the full Labour Force Survey sample, and the calibration strategy was modified to include the family type and social assistance recipients. This paper described the main changes on CIS estimates resulting from these methodological updates.

The impact of these updates was measured using the 2021 reference year data. Overall, the median after-tax income was slightly lower in Canada when estimated using the new methods, while the poverty rate and the low-income rate were slightly higher. In both cases, differences were relatively small.

In conclusion, the introduction of methodological updates for the 2022 CIS had a small impact on key CIS estimates and trends, while making significant improvements in data quality.