Methodology

Introduction
Sample frame development and sampling strategy
Quantitative survey design
Data collection and processing
Data quality assurance and weighting
Satellite Account of Nonprofit Institutions and Volunteering

Introduction

The objective of the NSNVO was to produce statistical information on the size, scope, capacity needs and challenges of nonprofit and voluntary organizations in Canada. The result is a rich source of data for analytical research reports and for recommending focussed follow-up research on voluntary sector capacity. The major phases of the two-year-long development and implementation of the survey were:

• sample frame development and sampling strategy
• quantitative survey design
• data collection and processing
• data quality assurance and weighting.

Each of these phases is outlined below, with a description of the quality of the data obtained by the survey.

Top of Page

Sample frame development and sampling strategy

The frame for this survey was developed using a combination of Statistics Canada's Business Register (BR), files on charities and nonprofit organizations from the Canada Revenue Agency (CRA), and lists of incorporated nonprofit organizations maintained by individual provinces and territories and by Industry Canada . The BR contains a North American Industry Classification System (NAICS) code and gross business income for each record. CRA files also include an activity code and income figure for most records. The files from the provinces and territories contained no information on either the activities or the size of the organizations listed therein.

Statistics Canada merged the files to produce a survey frame of charitable and incorporated nonprofit organizations in Canada. The purposes of the matching process were to remove duplicate organizations that appeared on more than one file and to add stratification variables wherever possible. The activity codes that appeared on the BR and the CRA files were used to determine the primary activity of the organization, based on the International Classification of Nonprofit Organizations (ICNPO). Many CRA records matched with BR records, enabling the gross business income figure on the BR to be used as an indicator of size for CRA records that were missing an income value. The remaining CRA records were matched to the supplementary portion of the BR, which includes all organizations with gross business income of less than $30,000. Since no specific gross business income figure is stored for these records, they were defaulted to the lowest revenue grouping for the purposes of stratification. At the end of this process, there were two groups of organizations: those with and those without stratification information. Most records in the second group came from the provincial and territorial files. While all efforts were made to ensure as clean a matching process as possible, this is an imperfect science.

The survey was conducted in two phases. For Phase 1, a sample of 20,000 organizations with no classification or size information was selected for each province individually and for the territories combined. Provinces with fewer organizations were oversampled so that sufficient sample sizes could be obtained for each province. These organizations were contacted in order to gather some basic stratification information, such as their primary activity according to the ICNPO major groupings and their size by gross revenue.

For Phase 2, a second sample of approximately 20,000 organizations was selected, a portion of which had been previously surveyed as part of Phase 1. The sample was designed such that results of the survey would be representative of each province and the combined territories, each revenue size category, and each of the ICNPO major activity groupings. It was determined that the initial sample included many public sector entities (e.g., school boards, public libraries and public schools) that did not meet the ICNPO criteria for inclusion in the survey, and these entities were therefore removed from the sample.

Top of Page

Quantitative survey design

Two questionnaires were developed for use in the NSNVO: one for the purpose of profiling the subset of organizations without adequate stratification information (Phase 1), and the other for deployment with the final sample of nonprofit organizations selected (Phase 2).

The questionnaire administered to respondents selected for Phase 1 of the survey consisted of a limited number of questions that confirmed that the organization still existed and was still nonprofit, and that gathered some indicators as to the activities and size of the organization, along with up-to-date contact information.

The questionnaire administered to respondents selected for Phase 2 of the survey collected detailed financial, human resources, policy and structural information about nonprofit and voluntary organizations. Questionnaire content and design were developed by the consortium of organizations who conducted this research, under the leadership of the Canadian Centre for Philanthropy. Content related to the capacity needs and strengths of organizations was based on a nationwide series of focus group consultations and interviews with key stakeholders in the sector. Portions of the questionnaire were adapted from a survey developed as part of the Johns Hopkins Comparative Non-Profit Sector Project directed by Lester M. Salamon and Helmut K. Anheier (see United Nations (2003). Handbook on non-profit institutions in the system of national accounts (ST/ESA/STAT/SER.F/91). New York: United Nations).

The survey instrument for Phase 2 was pretested extensively by Statistics Canada and the Canadian Centre for Philanthropy, as well as by members of the research consortium. Testing was conducted through personal interviews, focus groups, and telephone interviews.

Top of Page

Data collection and processing

Phase 1 entailed a computer-assisted telephone interview (CATI) survey of 20,000 organizations. This phase engaged approximately 40 interviewers for two months. Statistics Canada confirmed that some 9,000 of these organizations were still in operation and were, in fact, nonprofit. However, a significant number of organizations could not be located at all during collection, meaning that their existence could not be confirmed. After collection had ended, a sample of these organizations was thoroughly researched in order to better determine how they should be weighted.

For Phase 2 of the survey, an introductory letter and notes on specific questions pertaining to the human resources and revenue questions were mailed to all respondents. Phase 2 was also conducted as a CATI survey. Organizations contacted for participation in the NSNVO were asked to identify the person in the organization who would be most likely to have the information required to complete the questionnaire. Those agreeing to participate were surveyed using CATI. The interview took approximately 30 minutes to complete. This phase of the survey engaged some 60 interviewers for three months. The response rate was about 77%—a strong one for business surveys, in the experience of Statistics Canada.

Top of Page

Data quality assurance and weighting

Various validity and consistency edits were built into the CATI system to allow corrections to be made during the collection phase. Edits were also applied again after collection to further ensure data quality. Univariate frequencies were produced to identify any unusual values or outliers. Yet, like all surveys, the NSNVO is subject to errors, despite all the precautions taken at each stage of the survey to control them. There are two main types of errors: sampling and non-sampling errors.

Sampling errors

Sampling errors can occur when inferences drawn from the survey about the population are based on information collected from a sample rather than from the entire population. In addition to the sample design and the estimation method, the sample size and the variability of each characteristic also determine sampling error. Characteristics that are rare or are distributed very unevenly in the population will have greater sampling error than characteristics that are observed more frequently or are more homogeneous in the population.

It is standard practice to indicate the magnitude of the sampling error for estimates from a sample survey. The standard error of the estimate is the degree of variation of the estimate, considering that a particular sample was chosen rather than another of many possible samples of the same size under the same sample design. The standard error is usually expressed relative to the estimate to which it pertains. This measure, expressed as a percentage, is known as the coefficient of variation (CV). It is obtained by dividing the standard error of the estimate by the estimate itself. For example, if a sample statistic of .78 has a standard error of 0.03, then the coefficient of variation of the estimate is calculated as

(0.03/0.78) * 100% = 3.8%

For this report, survey estimates are put into one of three categories:

• sample estimates with a CV of 33.3% or less, which are presented unqualified
• sample estimates with a CV greater than 33.3%, which are flagged with the symbol ^E, meaning 'use with caution'
• sample estimates constructed from fewer than seven observations, which are flagged with the symbol x, signifying that a data point has been suppressed to protect respondents' confidentiality.

A range with a known probability of containing the true value can be defined using the CV and the estimate. For example, with a 95% probability, the range around the sample estimate is found by adding and subtracting 1.96 multiplied by the sample estimate and by its CV.²

In this case, the true value is in the range between 72.2% (78% - 5.8%) and 83.8% (78% + 5.8%), 19 times out of 20. A lower CV is preferable: it indicates a statistically more precise estimate.

Non-sampling errors

Errors that are not related to sampling may occur at almost every step of a survey. For example, interviewers may misunderstand instructions, respondents may make errors in answering questions, responses may be entered incorrectly on the questionnaire, and errors may occur in the processing and tabulation of the data.

Over a large number of observations, errors occurring randomly will have little effect on survey estimates. Errors occurring systematically, however, will contribute to biased estimates. Considerable effort has been made to reduce non-sampling errors by implementing quality assurance measures at each step of data collection and processing. These measures include the use of skilled interviewers with extensive training on survey procedures and the questionnaire; evaluation of possible flaws in the survey design or instructions; implementation of procedures to minimize data response or coding errors; and performance of quality checks to verify data editing and coding. A major source of non-sampling error is the effect of non-response on the survey results. The extent of non-response varies from partial non-response (failure to answer just one or some questions) to total non-response. Total non-response occurs when the interviewer is unable to contact the respondent, no member of the organization is able to provide the information, or the respondent refuses to participate in the survey.

The weighting process is designed to account for total non-response. Each organization in the sample was assigned a weight to reflect the number of organizations it represented in the population. After collection, organizations were classified as respondents, non-respondents or otherwise-most notably out of scope.³ The process had to account for the fact that the non-respondent organizations might have been in or out of the scope of the survey, depending on the answer given to the first question. By way of adjustment, the weights of respondents and of organizations classified otherwise were increased. Weights were specific to each stratum, i.e., the combination of province, ICNPO and revenue category prior to Phase 2 collection.

For example, if there are 400 units of medium size in a particular province and classification category of the ICNPO, and if 50 are chosen for the sample, the weight of each sample unit is (400/50) = 8.

Imputation procedures

The purpose of imputation is to assign values to missing answers of records with partial non-response. If however, a particular answer were deemed to be invalid and needed replacement, a replacement value could be obtained using imputation. Imputation achieves two objectives: users of the microdata file need not filter records, depending on the variables to be tabulated; and the weight of a record is unchanged, regardless of the variable(s) under study.

Two approaches to imputation were used. Wherever possible, values of missing answers were deduced or derived from answers that had been provided. The advantage of this approach is that changes are consistently made according to a set of prescribed rules. However, for the NSNVO, most missing values were imputed using the approach of donor imputation, whereby missing answers of a partial non-respondent—the receiver—are replaced with the answers of a randomly chosen respondent who has supplied answers—the donor. The choice of donor for a receiver was random from among a set of potential donors, all of which shared some degree of commonality with the partial non-respondent. Typically, the receiver and the set of donors might have a similar magnitude of total revenue and were of the same ICNPO.

Estimation procedures

Estimates were generated using SAS software and produced in tables of results in Microsoft Excel. In order to verify the quality of the estimates, measures of quality were generated using the Statistics Canada Generalized Estimation System (GES). GES is a SAS-based application for producing estimates and quality indicators by domain (e.g., by ICNPO or by the six ranges for amount of revenue) for random stratified sample design. Using GES, CVs were generated for quantitative point estimates. Similarly, standard errors were provided for estimates of proportions and ratios. Letter grades based on the measures of quality were assigned to each estimate. A further check on the quality of the estimates ensured that at least seven answers contributed to every estimate.

For further information on the data quality of the NSNVO, please contact

Client Services
Small Business and Special Surveys Division
10^th Floor, Section C-7
Jean Talon Building
Tunney's Pasture
Ottawa, ON
K1A 0T6
sbss-info@statcan.gc.ca
1-877-679-2746

Top of Page

Satellite Account of Nonprofit Institutions and Volunteering

It is a frequent practice at Statistics Canada to conduct comparative analysis of complementary information originating from alternate data sources. Such analysis is one of the many measures undertaken as additional assurance that high standards of data quality are upheld.

The Satellite Account of Nonprofit Institutions and Volunteering is a complementary Statistics Canada project. It was designed to measure the contribution of the nonprofit sector to the Canadian economy, putting it on an equal footing with other sectors in official economic statistics. Estimates in the satellite account are built from a wide range of data sources, including administrative files (tax records), Statistics Canada surveys and public accounts information. They are compiled according to international standards within the Canadian System of National Accounts.

Prior to the release of the survey estimates, extensive reconciliation analysis was undertaken to ensure that financial information collected on the NSNVO was consistent with estimates compiled in the Satellite Account of Nonprofit Institutions and Volunteering. From the point of view of the NSNVO, this analysis was important in assessing the validity of entities included in the survey sample, and helped ensure that the universe covered by the NSNVO was in conformity with international concepts.

From the point of view of the satellite account, the reconciliation analysis enabled coverage to be supplemented for certain types of organizations not well represented in the data sources used to compile the estimates. Since organizations not covered were generally small, the impact in terms of financial weight was not substantial. Coverage adjustments were concentrated in ICNPO groups for sports and recreation, business and professional associations, and social services.

Satellite account and NSNVO estimates could not be reconciled entirely, and differences remain between the two datasets. These differences are due to a range of factors, including different reference periods, conceptual and methodological differences, and varying classification procedures. Universe and coverage differences also remain in selected areas. The reconciliation analysis shows, however, that the broad messages on the size and scope of the nonprofit sector in economic terms reported by the two programs are largely consistent.

Notes

1. L.M. Salamon and H.K. Anheier, 1997, Defining the Nonprofit Sector: A Cross-national Analysis,Manchester, N.Y. , Manchester University Press.

2. The calculation is 1.96 x 78% x 3.8% = 5.8%.

3. Organizations were considered to be outside the scope of this survey if they were not found to be a nonprofit organization (i.e., they were for-profit organizations or government bodies).