Statistical inference with non-probability survey samples
Section 5. Quota surveys and poststratification

• Previous
• Next

Quota surveys are one of the oldest non-probability survey sampling methods which are still used in practice in present days. For a pre-specified overall sample size $n_A,$ quotas of sample sizes are set for subpopulations which are defined by demographic variables and social-economic status indicators or other characteristic variables suitable for units of the target population. Data collection processes continue until quotas for each of the subpopulations are filled. Units from the population are typically approached using whatever convenient ways available and there are little or no controls on how units are selected for the final sample other than the pre-specified quotas.

The theory of the IPW estimators for non-probability survey samples provides an opportunity to examine scenarios where quota surveys may succeed or fail. For the convenience of notation without loss of generality, let $S_A$ be the quota survey sample and $x$ be the set of categorical variables used for defining the subpopulations and setting the quotas. The overall sample can be partitioned into $S_A\mathrm{<mo>=</mo>}{S}_{A1}\cup \cdots \cup S_{AK}$ corresponding to the cross-classification of sampled units using the combinations of levels of the $x$ variables. For instance, if $x\mathrm{<mo>=</mo>}{\left(x_1\mathrm{,}{x}_2\right)}^{\prime }$ with $x_1$ having two levels and $x_2$ having three levels, we have a total of $K\mathrm{<mo>=</mo>}2\times 3\mathrm{<mo>=</mo>}6$ subpopulations defined by $x.$ Let $n_k$ be the pre-specified size of $S_{Ak}$ and $N_k$ be the size of the corresponding subpopulation. Under the assumption A1, the propensity scores ${\pi }_i^A=\pi \left(x{}_i\right)$ become a constant for units in the same subpopulation and are given by ${\pi }_i^A=n_k/N_k$ for the $k^{\text{th}}$ subpopulation. The IPW estimator ${\widehat{\mu }}_{\text{IPW}2}$ given in (4.8) reduces to

$${\widehat{\mu }}_{\text{IPW}2}\mathrm{<mo>=</mo>}\frac{1}{{\widehat{N}}^A}{\displaystyle \sum _{k=1}^K{\displaystyle \sum _{i\in S_{Ak}}\frac{y_i}{{\widehat{\pi }}_i^A}}}\mathrm{<mo>=</mo>}{\displaystyle \sum _{k=1}^K{\widehat{W}}_k{\overline{y}}_k,}(5.1)$$

where ${\overline{y}}_k=n_k^{-1}{\displaystyle {\sum }_{i\in S_{Ak}}{y}_i},$ ${\widehat{W}}_k={\widehat{N}}_k/{\widehat{N}}^A,$ ${\widehat{N}}_k$ is the size of the $k^{\text{th}}$ subpopulation obtained or estimated from external sources, and ${\widehat{N}}^A={\displaystyle {\sum }_{k=1}^K{\widehat{N}}_k}.$ Under the current setting with the availability of a reference probability sample $S_B,$ we form the same partition as cross-classified by levels of $x$ and obtain $S_B=S_{B1}\cup \cdots \cup S_{BK}.$ We can then use ${\widehat{N}}_k\mathrm{<mo>=</mo>}{\displaystyle {\sum }_{i\in S_{Bk}}{d}_i^B}.$

The estimator given in (5.1) is the standard poststratified estimator of ${\mu }_y.$ It requires the information on the “stratum weights” ${\widehat{W}}_k,$ $k=\mathrm{1,}\dots \mathrm{,}K,$ which is not available from the sample data itself. Quota surveys, combined with the use of the poststratified estimator, can be successful in producing valid population estimates for the study variable $y$ if the following conditions hold:

(i)   The categorical variables $x$ used in defining the subpopulations and setting the quotas provide characterizations of the participation behavior of the units for voluntary surveys.

(ii)  The inclusion of units in the survey is relatively random within each subpopulation and no specific groups are intentionally excluded from the survey.

(iii) The information on the stratum weights corresponding to the cross-classifications in setting the quotas can be reliably obtained from external sources.

(iv) The hardcore nonrespondents in the population who never take any voluntary surveys possess similar features to respondents in terms of the study variable $y.$

The IPW estimators ${\widehat{\mu }}_{\text{IPW}1}$ and ${\widehat{\mu }}_{\text{IPW}2}$ given in (4.8) may be sensitive to small values of estimated propensity scores. The poststratified estimator in the form of (5.1) serves as a robust alternative under general scenarios where the dimension of $x$ is not low and/or some components of $x$ are continuous. The $K$ strata are formed based on homogeneous groups in terms of the propensity scores. Suppose that ${\widehat{\pi }}_i^A=\pi (x_i,\widehat{\alpha }),$ $i\in S_A$ are computed based on a parametric model, $q.$ Suppose also that $n_A=m_{A}K$ with the chosen $K$ where $m_A$ is an integer. Let ${\widehat{\pi }}_{(1)}^A\le \cdots \le {\widehat{\pi }}_{(n_A)}^A$ be the estimated propensity scores in ascending order. Let $S_{A1}$ be the set of the first $m_A$ units in the sequence, $S_{A2}$ be the second $m_A$ units in the sequence, and so on. The poststratified estimator of ${\mu }_y$ is computed as ${\widehat{\mu }}_{\text{PST}}={\displaystyle {\sum }_{k=1}^K{\widehat{W}}_k{\overline{y}}_k},$ which has the same form of the estimator given in (5.1). The estimates of the stratum weights, ${\widehat{W}}_k,$ $k=\mathrm{1,}\mathrm{2,}\dots \mathrm{,}K$ can be obtained by using the reference probability sample $S_B$ as follows. Let $b_k=\max \left\{{\widehat{\pi }}_i^A\text{}\mathrm{<mo>:</mo>}i\in S_{Ak}\right\},$ $k=\mathrm{1,}\mathrm{2,}\dots \mathrm{,}K-1.$ Let $b_0=0$ and $b_K=1.$

1. Compute ${\widehat{\pi }}_i=\pi (x_i\mathrm{,}\widehat{\alpha }),$ $i\in S_B.$
2. Define $S_{Bk}=\left\{i|i\in S_B,b_{k-1}<{\widehat{\pi }}_i\le b_k\right\},$ $k=\mathrm{1,}\mathrm{2,}\dots \mathrm{,}K.$
3. Calculate ${\widehat{N}}_k={\displaystyle {\sum }_{i\in S_{Bk}}{d}_i^B},$ $k=\mathrm{1,}\mathrm{2,}\dots \mathrm{,}K.$

It is apparent that $S_B=S_{B1}\cup \cdots \cup S_{BK}$ and ${\displaystyle {\sum }_{k=1}^K{\widehat{N}}_k}={\widehat{N}}^B={\displaystyle {\sum }_{i\in S_B}{d}_i^B}.$ The estimated stratum weights are given by ${\widehat{W}}_k={\widehat{N}}_k/{\widehat{N}}^B.$

The choice of $K$ needs to reflect the balance between homogeneity of the units within each post-stratum (in terms of the propensity scores) and the stability of the poststratified estimator (in terms of the stratum sample sizes). When the sample size $n_A$ is small or moderate, a small number such as $K=5$ should be used. For scenarios where $n_A$ is large, a larger $K$ should be used such that units within the same poststratified sample $S_{Ak}$ have similar estimated propensity scores. A practical guidance for the choice of $K$ is to ensure that $m_A\ge 30$ for the poststratified samples. For those who are old enough, do you remember the good old days when “the sample size is large” means $“n\ge 30”?$

• Previous
• Article main page
• Next