6. Empirical evaluations

Piero Demetrio Falorsi and Paolo Righi

Previous | Next

Several simulations were carried out on real and simulated data sets to investigate the empirical properties of the proposed sampling strategy. Here, we show the results obtained for a single real data exercise, referred to the 1999 population of enterprises having a number of employed persons between 1 and 99, and belonging to Computer and related economic activities (2-digits of the Statistical classification of economic activities in the European Community rev.1, abbreviated as NACE). Three experiments were performed. Experiment (a) checked whether the allocation obtained by the proposed algorithm converged towards the solution of the standard Chromy’s algorithm for the SSRSWOR design. Experiment (b) compared the sample sizes of the standard SSRSWOR design with the Incomplete Stratified Sampling (ISS) design, in which the cross-classified strata were unplanned subpopulations; this experiment studied the risk of statistical burden due to repeated selection on different survey occasions. Finally, Experiment (c) measured the discrepancies between the expected Coefficients of Variation (CV) computed by the algorithm and the empirical CV obtained by a Monte Carlo simulation.

The $c_k$  values were, in all three experiments, uniformly set equal to 1. The Anticipated Variance according to the approximation proposed in Remark 4.1 was also calculated.

The population chosen for the experiments had a size of $N=\text{10,392}$  enterprises. The domains of interest identify two partitions of the target population: the geographical region, with 20 marginal domains (DOM1), and the economic activity group (3-digits of the NACE with 6 different groups) by size class (defined in terms of number of employed persons: $1=1-4;2=5-9;3=10-19;$   $4=20-99),$  with 24 marginal domains (DOM2). The overall number of marginal domains was 44, while the number of cross-classified or multi-way strata with a not-zero population size was 360. The modal value of the population size distribution is 1, and 29.17% of the cross-classified strata have at most 2 units. This type of strata represents a critical issue in the context of standard stratified approaches. Indeed, for calculating unbiased variance estimates, these strata must be take-all strata (so that they do not contribute to the variance of the estimates), although the allocation rule would require fewer units and, in general, a non-integer number of sample units. The variables of interest were the labour cost and the value added, which are available for each population unit from an administrative data source. Typically both variables have highly skewed distributions.

The target estimates for all the empirical studies are the 88 totals at the domain level (2 variables by 44 marginal domains). In each experiment, the inclusion probabilities were determined by fixing the ${\overline{V}}_{\left(dr\right)}={\left(\text{0}\text{.1}{t}_{\left(dr\right)}\right)}^2$  in (5.1), which is equivalent to fixing the maximum accepted level of the percent CV of the domain level estimates at 10%.

Empirical study (a). The first experiment took into account the partition DOM1. These domains represented both planned domains and estimation domains. Since the planned domains defined a partition of the population of interest, they could also be considered as strata in the standard sampling designs. The predictive working model was given by

$$\{\begin{array}{l}{y}_{rk}={\alpha }_d+u_{rk}\forall k\in U_d\left(d=1,\dots ,20\right)\hfill \\ E_M\left(u_{rk}\right)=0,E_M\left(u_{rk}^2\right)={\sigma }_{rd}^2\forall k\in U_d;E_M\left(u_{rk},u_{rl}\right)=0\forall k\ne l\hfill \end{array},(6.1)$$

where ${\alpha }_d$  is a fixed effect and the superpopulation variances ${\sigma }_{rd}^2$  were estimated by means of the residual variance of the predictive model in each region. The algorithm proposed in Section 5 was performed using three different initial values of the inclusion probabilities $\overline{\pi },$  equal to 0.01, 0.50 and 0.99 respectively. The initial inclusion probability values had no impact on the final solution, although it was achieved with a different number of iterations. We note that the overall number of inner loops was 17 for $\overline{\pi }=\text{0}\text{.01}.$  The convergence was achieved with 13 inner loops for $\overline{\pi }=\text{0}\text{.50};$  14 inner loops were needed for $\overline{\pi }=\text{0}\text{.99}.$  However, after the ninth iteration, the three sampling sizes were quite similar (Figure 6.1). In the experiment, the overall sample sizes were 3,105 for the benchmark Chromy allocation and 3,110 for the method proposed here. However, the differences between the two sampling sizes at the domain level were fractional numbers that were always lower than 1, and with the absolute largest relative difference lower than 0.3%. This highlights that the proposed algorithm actually defines the same domain sampling sizes of those calculated by the benchmark allocation. With regards to convergence, the initial inclusion probability values have no impact on the final solution, although this is achieved with a different number of iterations.

Figure 6.1 Convergence of the algorithm with different initial inclusion probabilities in the empirical study (a)

Description for Figure 6.1

Similar results were obtained if the domains of interests were identified by the partition DOM2.

Empirical study (b). Let $U_{d_1}$  be a specific region $\left(d_1=1,\dots ,20\right)$  of DOM1, and let $U_{d_2}$  (with $d_2=1,\dots ,24)$  be a specific economic activity group by the enterprise size class of the partition DOM2. Two prediction models, $M_1$  and $M_2,$  were used. Referring to the notation of the ANOVA models, $M_1$  is the saturated model given by

$$\{\begin{array}{l}{y}_{rk}={\alpha }_{d_1}+{\lambda }_{d_2}+{\left(\alpha \lambda \right)}_{d_1{d}_2}+u_{rk}\forall k\in U_{d_1}\cap U_{d_2}\hfill \\ E_M\left(u_{rk}\right)=0,E_M\left(u_{rk}^2\right)={\sigma }_{r\left(d_1{d}_2\right)}^2\forall k\in U_{d_1}\cap U_{d_2};E_M\left(u_{rk},u_{rl}\right)=0\forall k\ne l\hfill \end{array},(6.2)$$

in which ${\alpha }_{d_1}$  and ${\lambda }_{d_2}$  are the main effects, related to the domains $U_{d_1}$  and $U_{d_2}$  respectively and with ${\left(\alpha \lambda \right)}_{d_1{d}_2}$  as the interaction effect. The model variances ${\sigma }_{r\left(d_1{d}_2\right)}^2$  were estimated by means of the ordinary least square method, by computing the variances of the residual terms at the $U_{d_1}\cap U_{d_2}$  level. Model $M_2$  is identical to model $M_1$  without the interaction factor. Table 6.1 shows the goodness of fit of the two models.

Table 6.1
Goodness of fit of the models used for the prediction
Table summary
This table displays the results of Goodness of fit of the models used for the prediction. The information is grouped by Model (appearing as row headers), Goodness of fit $R^2\%$ (appearing as column headers).

Model	Goodness of fit $R^2\%$
	Labour cost	Value added
Model $M_1$ (Expression 6.2)	68.1	64.1
Model $M_2$ (Expression 6.2 without interactions)	65.1	61.0

Three different allocations were considered for the SSRSWOR in the case of model $M_1:$  (i) no stratum sample size constraint is given; (ii) at least 1 sample unit per stratum is required (to obtain unbiased point estimates); (iii) at least 2 sample units per stratum are required (to achieve unbiased variance estimates) for all strata having a population size of 2 or more enterprises. The first two allocations were rather theoretical since in all the business surveys conducted by the Italian National Statistical Institute, the selection of at least two units per stratum is required. The results of the experiment are shown in Table 6.2 below. Only the results for the case in which the initial inclusion probabilities were equal to $\overline{\pi }=\text{0}\text{.50}$  are investigated herein; identical sample sizes were obtained with the other initial values of the inclusion probabilities, with a slightly slower convergence process. The three SSRSWOR designs have 716.6, 944 and 1,042 sample units respectively. The Incomplete stratified Sampling (ISS) design with model $M_1$  led to 936 units; while model $M_2$  led to 991 units. The better result obtained by model $M_1$  with respect to model $M_2$  was due to the fact that model $M_1$  had a better fit. Finally, the ISS designs helped tackling the statistical burden of respondent enterprises. Indeed, assuming that the inclusion probabilities remain fixed for the different survey occasions, their distributions may be used to assess the statistical burden in repeated surveys. Table 6.2 shows that the number of enterprises drawn with certainty in each survey occasion was 175 for the third SSRSWOR designs, while 30 and 40 enterprises were selected with certainty in the first and second ISS designs, respectively. Analysing the sizes (in terms of employed persons) of the enterprises included in the sample with certainty, the third SSRSWOR design had an average size equal to 20.6. In some cases, enterprises with 2 employed persons were included in the sample with certainty. Conversely, we observe that in the first and second ISS designs, the enterprises with minimum size had 17 and 16 employed persons respectively, and an average size larger than 40 units.

Table 6.2
Sample sizes and distribution of the enterprises included in the sample with certainty, for different sampling designs
Table summary
This table displays the results of Sample sizes and distribution of the enterprises included in the sample with certainty. The information is grouped by Sampling design (appearing as row headers), Sample size, Enterprises selected with certainty, Number and Number of employed (appearing as column headers).

Sampling design		Sample size	Enterprises selected with certainty
			Number	Number of employed
				Average	Minimum
Standard Stratified with $M_1$ model	No stratum sample size constraint	716.6	10	47.0	23.0
	At least 1 sample unit per stratum	944.0	119	24.0	2.0
	At least 2 sample units per stratum	1,042.0	175	20.6	2.0
Incomplete Stratified Sampling with $M_1$ model		936.0	30	50.1	17.0
Incomplete Stratified Sampling with $M_2$ model without interactions		991.0	40	42.9	16.0

Finally, to assess the solution’s sensitivity, the experiment was repeated artificially and the prediction values of ${\tilde{y}}_{rk}$  and ${\tilde{\sigma }}_{rk}^2$  in the optimization problem (5.1) were changed. In particular, we increased the prediction values of ${\tilde{\sigma }}_{rk}^2$  by 20% and 120% respectively, and decreased by 20% the ${\tilde{y}}_{rk}$  values predicted by model $M_1.$  As expected, the sample sizes increased, but the SSRSWOR design with at least 1 sample unit per stratum and the first ISS design roughly defined the same sample sizes (Table 6.3).

Table 6.3
Sample sizes with modified expected values of the predictions of model (4.1)
Table summary
This table displays the results of Sample sizes with modified expected values of the predictions of model (4.1). The information is grouped by Sampling design (appearing as row headers), Sample size (appearing as column headers).

Sampling design		Sample size
		${\tilde{\sigma }}_{rk}^2$ increased by 20%	${\tilde{\sigma }}_{rk}^2$ increased by 120%	${\tilde{y}}_{rk}$ decreased by 20%
SSRSWOR with $M_1$ model	No stratum sample size constraint	821.0	1,269.0	993.8
	At least 1 sample unit per stratum	1,035.0	1,472.0	1,206.0
	At least 2 sample units per stratum	1,125.0	1,536.0	1,283.0
ISS design with $M_1$ model		1,039.7	1,460.9	1,207.5

Empirical study (c). The heteroschedastic linear prediction model $M_3$  was used:

$$\{\begin{array}{l}{y}_{rk}={\alpha }_r+{\phi }_r{x}_k+u_{rk}\\ E_M(u_{rk})=0,E_M(u_{rk}^2)={\sigma }_{rk}^2={\sigma }_r^2{x}_k^{}\forall k\in U;E_M({\epsilon }_{rk},{\epsilon }_{rl})=0\forall k\ne l\end{array},(6.3)$$

where $x_k$  is the number of employed persons in the $k^{\text{th}}$  enterprise, and ${\alpha }_r$  and ${\phi }_r$  are the regression parameters. Note that the number of employed persons is available in the sampling frame in Italy.

Two different model variance estimates were carried out:

(a) ${\tilde{\sigma }}_{rk}^2=1/N_{\left(X=x_k\right)}{\displaystyle {\sum }_{k\in U_{\left(X=x_k\right)}}{\left(y_{rk}-{\text{A}}_r-{\text{F}}_r{x}_k\right)}^2}$  and (b) ${\tilde{\sigma }}_{rk}^2={\tilde{\sigma }}_r^2{x}_k,$  in which ${\tilde{\sigma }}_r^2=1/\left(N-2\right){\displaystyle {\sum }_{k\in U}{\left[\left(y_{rk}-{\text{A}}_r-{\text{F}}_r{x}_k\right)/x_k\right]}^2},$  where $U_{\left(X=x\right)}$  is the population of enterprises, of size $N_{\left(X=x\right)},$  for which the variable $X$  assumes the value $x;$   ${\text{A}}_r$  and ${\text{F}}_r$  are the weighted least square estimates for the complete enumerated population of ${\alpha }_r$  and ${\phi }_r$  respectively. The sum of the estimated model variances obtained with method (a) is smaller than that obtained with method (b). This was reflected in the computed sample sizes. The first allocation defined an overall sample size of 927 units, while the sample size of the second allocation was 951. Successively, 1,000 samples were drawn for both allocations and the ratios $\text{RCV}\left({\widehat{t}}_{\left(dr\right)}\right)=\text{ECV}\left({\widehat{t}}_{\left(dr\right)}\right)/\text{SCV}\left({\widehat{t}}_{\left(dr\right)}\right)$  were calculated, with $\text{ECV}\left({\widehat{t}}_{\left(dr\right)}\right)=\left[\sqrt{\text{AAV}\left({\widehat{t}}_{\left(dr\right)}\right)}/{\widehat{t}}_{\left(dr\right)}\right]100$  as the expected CV (%) and

$$\text{SCV}\left({\widehat{t}}_{\left(dr\right)}\right)=100\sqrt{\left(1/I\right){\left[{\displaystyle {\sum }_{i=1}^I{\widehat{t}}_{\left(dr\right)}^i}-\left(1/I\right){\displaystyle {\sum }_{i=1}^I{\widehat{t}}_{\left(dr\right)}^i}\right]}^2}/\left(1/I\right){\displaystyle {\sum }_{i=1}^I{\widehat{t}}_{\left(dr\right)}^i}$$

as the simulated (or empirical) CV, obtained as a result of the simulation, having denoted with ${\widehat{t}}_{\left(dr\right)}^i$  the HT estimate in the $i^{\text{th}}$  iteration and $I=\text{1,000}\text{.}$  For the sake of brevity, only the the main results of allocation (b) are shown in Figure 6.2, for DOM1 and DOM2 respectively, and both variables of interest. Examining the figure on the left, we emphasize that the simulation generally produces a simulated CV that is smaller than expected, with an RCV ratio larger than 1 for both variables. One exception occurs, for the value added in one domain of DOM1.

Figure 6.2 RCVs by population size for labour cost and value added

Description for Figure 6.2

RCV lower than 1 may be explained by the increase of the domain sample sizes, due to the calibration step. We note that in general, these discrepancies are observed in domains with a small population size; thus, the calibration step may have a non-negligible impact. The figure on the right shows more articulated and conflicting empirical evidence. First, we note that the RCV are often larger or very close to 1. Nevertheless, in three domains, the value added variable has simulated CV’s equal to 11.5%, 12.0% and 12.3%. In these rare cases, and in some others (labour cost in two domains), the discrepancies are coherent with the findings of Deville and Tillé (2005) on the empirical properties of variance approximation for balanced sampling.

Previous | Next