An examination of the NAACCR method of assessing completeness of case ascertainment using the Canadian Cancer Registry

View the most recent version.

Archived Content

Information identified as archived is provided for reference, research or recordkeeping purposes. It is not subject to the Government of Canada Web Standards and has not been altered or updated since it was archived. Please "contact us" to request a format other than those available.

Archived

This page has been archived on the Web.

by Dianne Zakaria

Reliable cancer registry data are needed for planning, monitoring, and evaluating cancer control programs. An important aspect of data quality is case ascertainment, generally defined as the percentage of all incident tumours in a registry’s surveillance population that are captured in the registry’s database.^{Note 1} Incomplete case ascertainment can lead to underestimated incidence and prevalence, and biased socio-demographic and clinical characteristics (for example, stage at diagnosis, treatment provided, survival) if the cancers recorded by a registry differ substantially from those that are missed.

According to a recent survey of European cancer registries, 86% estimated their case ascertainment completeness.^{Note 2} The methods used most frequently were comparing current with historical incidence (73%) and comparisons with a presumably complete reference registry (65%). More complex procedures, such as the capture-recapture method (25%)^{Note 3-6} and flow method (21%),^{Note 4,Note 7,Note 8} were employed less often. The use of more than one method was also infrequent (29%).

The method used by the North American Association of Central Cancer Registries (NAACCR) to estimate case completeness is to express the observed number of cancers as a percentage of the expected number for a given population.^{Note 9} Age-standardized race-, sex- and cancer-site-specific incidence-to-mortality rate ratios are calculated, based on Surveillance Epidemiology and End Results (SEER) Program cancer incidence data and U.S. cancer mortality data. The products of these rate ratios and mortality rates for the region and year of interest provide expected age-standardized race-, sex- and cancer-site-specific incidence rates for that region and year. Summation of these estimates yields the overall expected age-standardized cancer incidence rate for the race, sex, region and year of interest. The observed cancer incidence rate is then expressed as a percentage of this expected incidence rate to estimate the completeness of case ascertainment.^{Note 9-11}

This method assumes that cancer death data are complete and that the ratio of age-standardized cancer incidence rates to age-standardized cancer mortality rates by race, sex and cancer site varies little by geographic area (within ± 20% attributed to differential case fatality).^{Note 9} But despite the latter assumption, NAACCR uses U.S. cancer mortality data, rather than SEER cancer mortality data, to produce ASRRs and then adjusts for differences between region-specific and U.S. mortality. If the age-standardized region-specific mortality rate is greater than the age-standardized U.S. mortality rate, the region-specific mortality rate is adjusted downward before calculating the expected incidence rate; if the region-specific rate is lower than the U.S. rate, the region-specific rate is adjusted upward.^{Note 12}

Completeness of case ascertainment can also be estimated with simpler indicators: the percentage of cancers registered by death certificate only (%DCO); the percentage microscopically confirmed (%MC); and the age-standardized incidence-to-mortality rate ratio (I:M). In fact, simpler case completeness indicators are routinely used for the publication, Cancer Incidence in Five Continents,^{Note 1} and by other international studies.^{Note 13,14}

A high %DCO suggests incomplete case ascertainment due to failure to capture cases while patients are alive. This means that missed non-fatal cases (cancer not indicated on the death certificate) will probably never be registered.^{Note 15} Conversely, %DCO=0% suggests that death certificates are not being used, or that linkage with a vital statistics registry to identify missed cases is not occurring, and thus, incomplete case ascertainment is likely.^{Note 1,Note 9,Note 16}

High and low %MC can also signal completeness issues. A high percentage may reflect over-reliance on hospital or pathology laboratory cases; a very low percentage may indicate a lack of adequate pathology laboratories or a lack of collaboration between a cancer registry and pathology laboratories.^{Note 1,Note 16-18} Based on the experience of the SEER Program, the %MC for all cancer cases combined is expected to range from 92% to 96%.^{Note 18}

Finally, the I:M should exceed 1.00. A ratio below 1.00 indicates under-reporting.^{Note 9}

Despite use of the NAACCR case completeness indicator, little has been published about its methodology, usefulness, and accuracy in Canada. The effect that limiting mortality data to the same geographic region that contributed the incidence data would have on calculations of the age-standardized incidence-to-mortality rate ratios is unknown. The indicator is based on the assumption that age-standardized incidence-to-mortality rate ratios by race, sex and cancer site are approximately constant across geographic areas. Therefore, better performance might be expected of the indicator if both incidence and mortality data were derived from the same geographic area. As well, the benefits of the NAACCR indicator over simpler methods have not been thoroughly explored.

Using data from the Canadian Cancer Registry (CCR), vital statistics, and population statistics, the first objective of the present study is to examine the impact of limiting mortality data to the same geographic regions that contribute incidence data when calculating age-standardized incidence-to-mortality rate ratios. This includes assessing the assumption that the age-standardized incidence-to-mortality rate ratios by sex and cancer site vary little by region. The second objective is to quantify relationships between simpler methods of estimating completeness and the NAACCR indicator. The final objective is to determine if the NAACCR indicator identifies known differences in difficulty of case ascertainment, and known case completeness issues in the CCR.

Data and methods

The NAACCR indicator was calculated for primary cancers diagnosed in Canada during 2007, because, at the time of analysis, this was the most recent year for which national data were available; it was the most recent year linked to national vital statistics data; and it minimized confounding of case completeness and timeliness.^{Note 15} The methodology for calculating the indicator has been described by NAACCR.^{Note 12} Statistics Canada’s CCR,^{Note 19} Vital Statistics Death Database^{Note 20} and Census of Population^{Note 21} furnished cancer incidence, cancer mortality, and population data, respectively, for all provinces and territories for the five-year period (2003 to 2007) ending in the evaluation year (2007). These five years of data were combined to calculate sex-, age-, and cancer-site-specific incidence and mortality rates, which were age-standardized using the July 1, 1991 population (Appendix A). The cancer sites included in the NAACCR indicator and the method of extraction from the CCR and Vital Statistics Death Database are presented in Appendix B.

The age-standardized sex- and cancer-site-specific incidence-to-mortality rate ratios (ASRR) used in the indicator were calculated two ways: ASRR₁ and ASRR₂. For ASRR₁, the cancer incidence and mortality data were limited to provinces attaining NAACCR gold or silver certification in each year from 2003 through 2007: Alberta, Saskatchewan, Manitoba, New Brunswick, and Prince Edward Island. For ASRR₂, consistent with the NAACCR approach, cancer incidence rates were derived from data for the best-performing provinces, but mortality rates were derived from data for all Canada.

The expected age-standardized sex- and cancer-site-specific incidence rate for a province or territory in 2007 was calculated using ASRR₁ or ASRR₂ (equation 1) (Formulas). To account for differences in cancer case fatality rates across regions, NAACCR incorporates a mortality adjustment term (equation 2), which is used to adjust the age-standardized sex- and cancer-site-specific mortality rate for the region of interest (equation 3). Completeness of case ascertainment for a specific sex and cancer site in a province/territory was calculated (equation 4), and overall completeness of case ascertainment for a specific sex in a province/territory was calculated (equation 5).

The case completeness indicators produced using these two methods are referred to as I₁ and I₂. Variances for age-standardized rates were calculated as per Fay and Feuer,^{Note 22} and confidence intervals for age-standardized rate ratios were calculated as per Armitage, Berry and Matthews.^{Note 23} Confidence intervals were not calculated for the completeness of case ascertainment indicators because of the lack of published methods, a previously identified limitation, particularly for estimates based on small counts.^{Note 24}

The Pearson product-moment correlation coefficient was used to examine associations between sex- and cancer-site-specific I₁, I₂ and I:M (quantified for 2007). Because associations between the sex- and cancer-site-specific estimates of case completeness (I₁ and I₂) and %DCO and %MC (both quantified for 2007) were not expected to be linear, they were assessed using the point biserial correlation coefficient. This statistic measures the degree of association between a dichotomous variable (%DCO or %MC) and an interval or ratio variable (I₁ and I₂). Its properties and interpretation are similar to the Pearson product-moment correlation coefficient in that it ranges from -1 to +1, with larger absolute values indicating a stronger relationship. It shows the degree to which %DCO or %MC discriminates between complete and incomplete case ascertainment: larger absolute values indicate better discrimination.^{Note 25} %DCO was dichotomized such that values of 0% or more than 5% (exceeding upper limit for NAACCR silver certification) were considered to suggest incomplete case ascertainment. %MC was dichotomized such that values less than 90% or greater than 98% were considered to suggest incomplete case ascertainment, a slightly wider range than the NAACCR guideline for all cases combined (92% to 96%). Pearson product-moment correlation coefficients were also used to examine the association between I₁ and I₂ and the continuous forms of the %MC and %DCO, but the findings were similar (data not shown).

To meet the confidentiality requirements of the Statistics Act, all estimates based on fewer than five cases, or comprised of other estimates based on fewer than five cases, were suppressed. Because suppression occurred frequently for Nunavut, Northwest Territories, Yukon Territory and Prince Edward Island, estimates are presented only for the remaining nine provinces. However, the results for the smaller provinces and territories are included in the estimates for Canada as a whole. All analyses were conducted using SAS 9.2^©.^{Note 26}

Results

ASRR₁ and ASRR₂  were generally similar (overlapping confidence intervals) (Table 1). When differences did exist (non-overlapping confidence intervals), ASRR₁ was generally greater, indicating that the mortality rate in the best-performing provinces was lower than that for Canada overall.

Comparisons of cancer-site-specific ASRR₁s across the best-performing provinces showed that only prostate cancer and female breast cancer had instances of non-overlapping confidence intervals (Table 2). Both cancers are excluded from NAACCR’s overall estimate of case completeness (Appendix B).

However, even when the smallest province (Prince Edward Island) was excluded, substantial differences in ASRR₁s were apparent across regions. For example, the female stomach cancer ASRR₁ in Saskatchewan was 2.00, compared with 1.35 in Manitoba, a relative difference larger than that for female breast cancer (4.37 versus 4.01, respectively). Examination of the underlying age-standardized rates revealed that differences in mortality, not incidence, created the disparity between the two provinces. However, the power to identify these differences as statistically significant was limited by the small case counts, compared with prostate cancer and female breast cancer.

The two sex- and cancer-site-specific case completeness indicators—I₁ and I₂—calculated for the nine provinces with adequate case counts were highly correlated (r=0.93, n=315, p<0.0001). Generally, they were either 90%+ (adequate for NAACCR silver case completeness certification) or less than 90%. However, in 11% of comparisons, differences emerged, with one indicator scoring 90%+, and the other, less than 90%. In the majority of these instances (67%), I₁ scored lower than I₂ because the ASRR₁ was larger than the ASRR₂ (Table 1). A larger ASRR means that the expected number of cases will be greater, which translates into lower case completeness (a lower observed-to-expected ratio). I₁ identified about 27% of the sex- and cancer-site-specific completeness indicators across the nine provinces as less than 90%; I₂ identified 23% as less than 90% (data not shown).

Of the simpler indicators, I:M was most strongly and consistently associated with I₁ and I₂; correlations of %MC and %DCO with I₁ and I₂ were rare (Table 3).

Table 4 presents I₁ and I₂ for selected cancers with varying degrees of difficulty of ascertainment.^{Note 10} If less than 90% is considered to represent potentially incomplete ascertainment, difficulty of ascertainment was not consistently associated with undercoverage. Aside from pancreatic cancer, the frequency of undercoverage based on I₂ did not increase for cancers of average difficulty compared with those of low difficulty. Undercoverage of breast cancer, which is considered to have average difficulty of ascertainment, was low based on both I₁ and I₂. As well, I₁ did not identify any instances of undercoverage of prostate cancer, which is considered to be one of the most difficult to ascertain.

Both I₁ and I₂ suggested undercoverage of bladder cancer in Ontario (Table 4),^{Note 27} a finding that was expected because Ontario does not report in situ bladder tumours to the CCR. For Quebec, both I₁ and I₂ suggested undercoverage of melanoma of the skin, but only I₂ suggested undercoverage of prostate cancer. In 1996, incomplete ascertainment of prostate and melanoma skin cancer had been documented for adults in Quebec.^{Note 28}

Cancer sites not presented in Table 4 were examined to identify other potential instances of substantial undercoverage (less than 80% complete); 10 emerged:

• esophageal cancer in Saskatchewan females (I₁=72%, I₂=75%) and Manitoba females (I₁=64%, I₂=66%);
• liver cancer in Manitoba females (I₁=55%, I₂=58%);
• ovarian cancer in Nova Scotia (I₁=79%, I₂=77%) and Newfoundland (I₁=75%, I₂=71%);
• kidney and renal pelvis cancer in Manitoba males (I₁=75%, I₂=73%);
• cancer of the brain and other parts of the central nervous system in Newfoundland females (I₁=67%, I₂=71%);
• Hodgkin lymphoma in Saskatchewan males (I₁=62%, I₂=65%) and Quebec females (I₁=47%, I₂=69%); and
• multiple myeloma in Nova Scotia males (I₁=58%, I₂=59%).

Caution is warranted, however, because of the small counts underlying some of these estimates (for example, esophageal and liver cancer).

The overall completeness-of-case­ascertainment indicator identified undercoverage among males in Newfoundland (Table 5), with or without inclusion of prostate cancer. Newfoundland’s I:M and %MC were at the lower and upper end, respectively, of the range of values for the nine provinces. These patterns also held for Newfoundland females, among whom one of the four completeness indicators dipped slightly below 90%.

Discussion

An assumption underlying the NAACCR indicator is that the ratio of age-standardized cancer incidence-to-mortality rates by race, sex and cancer site varies little by geographic region. That is, cancer incidence and mortality rates may vary across regions, but the ratio of the two will not. However, examination of this assumption across Canadian provinces used to develop the age-standardized incidence-to-mortality rate ratios revealed differences of practical importance, apart from prostate and female breast cancer, both of which are excluded from the NAACCR indicator.

In the present analysis, differences between ASRR₁ and ASRR₂ contributed to disparities between I₁ and I₂ in about 11% of comparisons. Users who want to identify potential undercoverage may prefer I₁ over I₂ because, in instances of disagreement, I₁ was more likely to score below 90%.

Of the simpler indicators, only I:M showed frequent, statistically significant associations with I₁ and I₂. This seems reasonable in light of the underlying assumptions of I₁ and I₂ (stability of the age-standardized cancer incidence-to-mortality rate ratios across regions and completeness of cancer death data). Given these assumptions, a relatively small I:M would signal missed cancer incidence. The I:M need not be less than 1.00, but merely low in relation to other regions. Thus, the I:M may offer a less complicated, more direct initial signal of case completeness issues, which can then be investigated by comparing age-standardized cancer incidence and mortality rates over time within a province or territory, and across provinces and territories. An advantage of the I:M is the ability to calculate confidence intervals for more meaningful comparisons with a standard I:M based on the best-performing provinces.

The lack of association of %DCO and %MC with I₁ and I₂ probably arises because of the lack of consistent cut-points across cancer sites,^{Note 13,Note 18} and because %DCO, by itself, is not an indicator of completeness of registration.^{Note 1,Note 4} A low %DCO could result from efficient registration of cancer cases while patients are alive, or from aggressive follow-back procedures for cases brought to a registry’s attention through death certificates. In the latter situation, missed cases are likely. De Angelis et al.^{Note 13} state that the extent of microscopic confirmation depends on the accessibility of the cancer to biopsy, whether surgery is performed, and the availability of pathology reports to cancer registries. For %DCO and %MC, examination of the range of values across provinces and territories, in conjunction with knowledge of registry procedures (use of death certificates, linkage to vital statistics database, follow-back procedures), would be of greater value in identifying undercoverage.

The importance of cancer-specific estimates for each province and territory was illustrated by how coverage issues were masked when sites with high and low completeness estimates were aggregated. Despite potential undercoverage of specific cancers in several provinces, the overall completeness-of-case-ascertainment indicator identified undercoverage only for Newfoundland males.

The primary limitations of this study are the assumptions underlying the completeness-of-case-ascertainment indicators and the lack of confidence intervals, particularly for estimates based on small counts. As Fulton and Howe^{Note 10} observed, even among SEER registries with superb case completeness, percentage case completeness varied, suggesting the existence of differences in age-standardized incidence-to-mortality rate ratios across states. Consistent with the findings of the present report, they concluded that percentage case completeness may be used to “cautiously” identify cancer sites for which undercoverage may be an issue and which require further exploration to rule out real differences in case fatality, incidence, and random variation before concluding that under-reporting is the cause.^{Note 10}

Because of the importance of case completeness, registries could pursue alternative evaluation methods, such as the capture-recapture method, flow method, Parkin’s death certificate notification method, and basic case-finding audits.^{Note 4} If registries captured and submitted information on all the distinct sources of notification for a case, the extent of case completeness could be explored by the CCR through capture-recapture methods.^{Note 3,Note 29-32} Similarly, submitting information on whether a case was death-certificate-notified and the date it was first registered would allow the CCR to estimate case completeness using the flow method^{Note 7} and Parkin’s death certificate notification method.^{Note 4}

Conclusion

The assumption of stable age-standardized sex- and cancer-site-specific incidence-to-mortality rate ratios across regions, which underlies NAACCR’s completeness of case ascertainment indicator, was not consistently supported by CCR data—substantial regional differences emerged. Although the frequency of undercoverage did not increase consistently with expected case finding difficulty, some known undercoverage issues in the CCR were identified. The importance of examining cancer-specific indicators was reinforced, as aggregation of cancer sites with high and low completeness estimates can obscure undercoverage in specific cancers. NAACCR’s indicator was associated with the basic incidence-to-mortality rate ratio for a region, but not with the %MC or the %DCO. Thus, the I:M and corresponding 95% confidence interval may offer a less complicated method of identifying undercoverage.

Acknowledgement

Statistics Canada maintains the Canadian Cancer Registry and Vital Statistics Death Database, which are comprised of data supplied by the provinces and territories, whose cooperation is gratefully acknowledged.