Economic and Social Reports
Quantitative impact analysis: A practical overview

Abstract

This spotlight article outlines practical methods for assessing the economic impacts of public programs delivered by federal agencies and Crown corporations. It summarizes key steps in conducting quantitative impact analysis, including data linkage, cohort construction and implementation of quasi-causal estimators. Drawing on recent findings from a Business Development Bank of Canada impact analysis, it illustrates how these tools generate evidence on firm-level outcomes. The paper concludes by highlighting advanced causal-inference approaches—such as the staggered difference-in-differences estimator—that address limitations of traditional two-way fixed effects difference-in-differences designs and strengthen future program evaluations.

Keywords: Difference-in-differences, event study, survival analysis, impact assessment, policy analysis, firm performance, public program effectiveness.

Acknowledgments

The authors would like to thank Ryan Macdonald, Marc Frenette, Amélie Lafrance-Cooke, Hassan Goreja, Meena Aier and Sylvie Ratté for their helpful comments and suggestions.

Introduction

Federal agencies and Crown corporations deliver a wide range of programs intended to support Canadian businesses—through financing, advisory services, export assistance and targeted interventions. As these programs expand, the demand for rigorous, transparent evaluation grows. Policy makers increasingly require evidence not only of whether programs work, but also of how, for whom and under what conditions they generate economic benefits.

Program evaluation in real-world settings is challenging. Randomized controlled trials are rarely feasible, administrative data are dispersed across systems, and program participation is seldom random (Frenette, Chan, & Handler, 2025). Firms differ widely in size, age, industry and managerial capacity. Furthermore, external shocks—such as the 2008/2009 recession or the COVID-19 pandemic—affect firms unevenly. These complexities make it difficult to isolate program impacts from broader economic forces.

Statistics Canada has developed a robust analytical framework to address these challenges. By linking program client data to the agency’s extensive microdata holdings, researchers can construct longitudinal cohorts of treated and comparable untreated firms. These data support quasi-experimental methods—such as difference-in-differences (DiD), event studies and survival analysis—that approximate causal inference in non-experimental settings (Angrist & Pischke, 2009; Khandker, Koolwal, & Samad, 2010).

Methods and data

The methodological approach for impact analysis begins with linking program client data to the Business Register, which provides foundational information on firm characteristics such as industry, size, revenue and location. High-quality linkage is essential—names and addresses alone do not necessarily identify a business uniquely, making the Business Number the preferred identifier. Once linked, the dataset is cleaned to remove duplicates, missing identifiers and low-quality matches.

The linked records are then merged with the Canadian Employer–Employee Dynamics Database, a comprehensive longitudinal dataset that includes the National Accounts Longitudinal Microdata File and the Diversity and Skills Database. These datasets provide annual information on sales, payroll, employment, and demographic characteristics of business owners and employees (Liu & Zhang, 2025). This integration enables analysts to construct treatment indicators, covariates and outcome variables with precision, and to examine outcomes for different types of firms and owners.

Because program participation is not random, treated firms must be compared with similar untreated firms. Matching reduces bias from pre-existing differences and strengthens the validity of causal inference. To do so, methods such as propensity score matching, nearest-neighbour matching and kernel matching are commonly used (Stuart, 2010). Recent impact studies increasingly rely on kernel matching, which uses all untreated firms and weights them based on similarity to treated firms. This approach preserves sample size and improves covariate balance, though it requires careful implementation.

Once matched cohorts are constructed, regression-based estimators are applied. First, DiD compares changes in outcomes for treated firms with changes for matched controls, under the assumption that both groups would have followed parallel trends without treatment. Second, event studies extend DiD by estimating effects for each period before and after treatment, providing insight into dynamic patterns and the plausibility of parallel trends. Finally, survival analysis complements these methods by assessing whether treated firms are more likely to remain active in subsequent years, offering a clear measure of business resilience.

Together, these steps form a coherent framework that allows analysts to isolate program effects from broader economic forces and firm-specific characteristics, while making full use of Statistics Canada’s linked administrative data.

Example using Business Development Bank of Canada impact study

The full research paper, “Applied Quantitative Impact Analysis: A Review of Contemporary Methods for Evaluating Programs Serving Businesses,” illustrates this framework using an evaluation of the Business Development Bank of Canada (BDC), which provides financing and advisory services to small and medium-sized enterprises. The analysis examines three types of support—loans, grants and advisory services—and evaluates their effects on labour productivity, employment, sales and survival. The example described in this spotlight article focuses only on the event study method, using labour productivity as the indicator of firm performance. Chart 1 below shows the average change in labour productivity for treated firms compared with the control group.

Data table for Chart 1

Data table for Chart 1
Table summary
The information is grouped by Period (appearing as row headers), , calculated using (appearing as column headers).

Period	Percent
Source: Authors calculation.
T-2	0
T-1	-2
T+0	0
T+1	22
T+2	32
T+3	42

The chart indicates that firms that received any products from BDC experienced positive and statistically significant impacts on their labour productivity relative to the comparison group. Furthermore, the findings show continuously increasing impacts up to three years after the initial treatment (T+0). In terms of interpretation, estimates are percentage changes and provide information on the magnitude of the difference in the effects between the treated firms and the control group. For example, the labour productivity of firms receiving any service or product from BDC was 22% higher on average than that of the matched control group in period T+1. Notably, the estimates obtained in the pre-treatment period support the required assumption of parallel trends, as differences are not statistically significant. This method thus allows for consistent and reliable estimates of average treatment effects on the treated.

Conclusion and discussion

Quantitative impact analysis plays a vital role in assessing the effectiveness of public programs that support Canadian businesses. By linking program client data to Statistics Canada’s extensive microdata holdings and applying quasi-experimental methods, analysts can generate credible evidence on how programs affect firm performance.

The framework outlined in this spotlight article provides a practical and scalable approach for evaluating a wide range of programs. The BDC case study demonstrates how these methods reveal meaningful differences in program impacts across financing and advisory services, and how dynamic patterns over time can be captured through event studies and survival analysis.

However, traditional two-way fixed effects DiD models can produce biased estimates in settings with staggered treatment adoption. Newer estimators, such as the Callaway and Sant’Anna (2021) approach, address these limitations by constructing cohort-specific comparisons that avoid negative weighting and contamination from already-treated units. These advances strengthen the credibility of future evaluations and support more nuanced policy insights.

As governments continue to invest in programs that shape the economic landscape, rigorous impact analysis will remain essential. The methods described here offer a robust foundation for evidence-based decision making and contribute to a broader culture of accountability and continuous improvement in public program delivery. Readers interested in the full set of results, robustness checks and methodological details are encouraged to consult the complete research paper.

Authors

Manassé Drabo, Landry Kuate and Michael Willox are with the Economic and Social Analysis and Modelling Division, Analytical Studies and Modelling Branch, at Statistics Canada.

References

Angrist, J. D., & Pischke, J.-S. (2009). Mostly harmless econometrics: An empiricist’s companion. Princeton University Press.

Callaway, B., & Sant’Anna, P. H. C. (2021). Difference-in-differences with multiple time periods. Journal of Econometrics, 225(2), 200–230.

Frenette, M., Chan, P., & Handler, T. (2025). Leveraging Statistics Canada data for impact analysis. Statistics Canada.

Khandker, S. R., Koolwal, G. B., & Samad, H. A. (2010). Handbook on impact evaluation: Quantitative methods and practices. World Bank.

Liu, H., & Zhang, X. (2025). The Canadian Employer–Employee Dynamics Database: Structure and applications. Statistics Canada.

Stuart, E. A. (2010). Matching methods for causal inference: A review and a look forward. Statistical Science, 25(1), 1–21.