Estimation of domain discontinuities using Hierarchical Bayesian Fay-Herriot models
Section 4. Results

• Previous
• Next

4.1   Model selection

In Subsection 3.2, two different versions for the fixed effects (FE_uq and FE_eq) and three different covariance structures of the random effects (RE_f, RE_d and RE_s) are considered for the bivariate FH model. The step-forward selection procedure from Subsection 3.6 is applied to each of these six combinations separately to select covariates. Recall from Subsection 3.1 that for the bivariate FH model and the univariate FH model for the discontinuities, potential covariates are available from the Municipal Base Administration and the Police Register of Reported Offences. Names of these covariates start with $MBA\_$ and $PR\_$ respectively. For the univariate FH model for the alternative survey, direct estimates from the regular CVS are also considered as covariates (van den Brakel et al., 2016). Names of these covariates start with $CVSR\_.$ See the appendix for an overview of the covariates.

The finally selected models for the bivariate FH model are summarized in Table 4.1. The models presented in Table 4.1 are selected with the step-WAIC-se procedure. The step-WAIC procedure selects models with a substantially larger amount of covariates which improve the WAIC only marginally. For $offtot$ and $unsafe$ the step-WAIC result in a model with 4 covariates with unequal regression coefficients and diagonal covariance matrices for the random effects (RE_d and RE_s). For $satispol$ and $propvict$ the step-WAIC result in a model with respectively 3 and 2 covariates with unequal regression coefficients, also with diagonal covariance matrices with equal variances for the random effects (RE_s). With only 25 domains, there is a substantial risk that these models overfit the data. An exception is $nuisance,$ where both selection procedures result in the same model.

With the step-WAIC-se procedure more parsimonious models are obtained as follows from Table 4.1. For total offences, $offtot$ and $nuisance,$ a model with only one covariate with equal regression coefficients for both surveys (FE_eq) is obtained in combination with a full covariance matrix (RE_f) with large random domain effects with a strong positive correlation of 0.98 for $offtot$ and 0.81 for $nuisance.$ Also for $unsafe$ a more parsimonious model, with one covariate and equal regression coefficients (FE_eq) is obtained with the step-WAIC-se procedure. In this case a diagonal covariance matrix with equal variances (RE_s) is selected. For $propvict$ and $satispol$ the step-WAIC-se procedure avoids the selection of large amounts of covariates, found with the step-WAIC approach. The selected model for $propvict$ has a full covariance matrix with a weak positive correlation of 0.1 (RE_f), and one covariate with unequal regression coefficients (FE_uq). The model for $satispol$ has a diagonal covariance matrix with equal variances (RE_s) and one covariate with unequal regression coefficients (FE_uq). Since parsimonious models are preferred in this application, the models obtained with the step-WAIC-se approach are finally selected. See van den Brakel and Boonstra (2018) for a more detailed discussion of the model selection resulting in the finally selected models.

The models selected with the univeriate FH model for the direct estimates of the discontinuities, developed in Subsection 3.3, are summarized in Table 4.2. The models are selected with the step-WAIC-se procedure. For $unsafe$ the step-WAIC results in a model with four covariates. For the other variables the same models are selected as with the step-WAIC-se procedure. The univariate FH models developed in van den Brakel et al. (2016) for the alternative survey approach are summarized in Table 4.3.

Standard model diagnostics test the underlying assumptions that the random domain effects and the residuals are normally and independently distributed. Since the number of domains in this application is small, the power of the tests for normality are weak and do not indicate deviations from normality. Therefore the posterior predictive tests as summarised in Subsection 3.5 are used to evaluate the model adequacy. In addition, the domain predictions aggregated to the national level are compared with the direct estimates at the national level to evaluate the bias introduced with the small area estimation procedures in Subsection 4.2. The posterior predictive $p$ -values for the domain estimates of the target variables and the discontinuities are summarized in Table 4.4 for the bivariate FH model and Table 4.5 for the univariate FH model for the discontinuities. The general measure for goodness-of-fit $\left(T_1\right)$ indicates that the fit for the discontinuities of $offtot$ is of reduced quality (other models considered had similar high values). The values for the bivariate FH model are slightly better compared to the univariate FH model for the discontinuities. The posterior predictive $p$ -values for maximum $\left(T_2\right)$ and minimum $\left(T_3\right)$ values do not indicate problems with the tails of the distributions. For these posterior predictive $p$ -values there are no systematic differences between bivariate and univariate FH model. The values for $T_4,$ $T_5,$ and $T_6$ for the discontinuities of the bivariate model are comparable with the values for the univariate model. The posterior predictive values for the mean $\left(T_4\right)$ and asymmetry of the distribution $\left(T_6\right)$ indicate that the distributions are symmetrically concentrated around their mean. The posterior predictive $p$ -values for the variance $\left(T_5\right)$ indicate some undershrinkage for the discontinuities of $nuisance,$ $propvict,$ and $offtot$ under both the bivariate and univariate FH model.

4.2   Estimation results

In this Subsection estimation results for the three different modelling approaches are discussed. In Subsection 4.2.1 the HB predictions for the target variables under the regular and alternative survey obtained with the bivariate FH model are compared with the direct estimates and with the domain predictions obtained with the univariate FH model where the direct estimates of the regular approach are potential auxiliary variables in the model selection. Subsequently results for the domain discontinuities are discussed in Subsection 4.2.2. Here the results obtained with the univariate FH model for the discontinuities are also discussed.

With model-based small area estimation, the design variance of the direct estimators is reduced at the cost of accepting some amount of design bias. To evaluate differences in the direct point estimates and the small domain predictions, the following two measures are defined. The first one is the Mean Relative Difference (MRD), which summarizes the differences between the direct estimates and the domain predictions:

$$\text{MRD}\mathrm{<mo>=</mo>}\frac{\mathrm{100\; <mi>\%</mi>}}{m}{\displaystyle \sum _{i\mathrm{<mo>=</mo>\; 1}}^m\frac{{\widehat{y}}_i^q-{\tilde{y}}_i^q}{{\widehat{y}}_i^q}\mathrm{,}}q\mathrm{<mo>=</mo>}r\mathrm{,}a\mathrm{,}(4.1)$$

and ${\tilde{y}}_i^q$ is the domain prediction based on the bivariate FH model or the univariate FH model. The second measure is the Absolute Mean Relative Difference (AMRD) between the direct estimate and the domain prediction, which is defined as:

$$\text{AMRD}\mathrm{<mo>=</mo>}\frac{\mathrm{100\; <mi>\%</mi>}}{m}{\displaystyle \sum _{i\mathrm{<mo>=</mo>\; 1}}^m\left|\frac{{\widehat{y}}_i^q-{\tilde{y}}_i^q}{{\widehat{y}}_i^q}\right|\mathrm{,}}q\mathrm{<mo>=</mo>}r\mathrm{,}a\mathrm{,}(4.2)$$

the increased precision of the small domain predictions is measured with Mean Relative Difference of the Standard Errors (MRDSE) between the direct estimates and the domain predictions and is defined as

$$\text{MRDSE}\mathrm{<mo>=</mo>}\frac{\mathrm{100\; <mi>\%</mi>}}{m}{\displaystyle \sum _{i\mathrm{<mo>=</mo>\; 1}}^m\frac{\text{SE}\left({\widehat{y}}_i^q\right)-\text{SE}\left({\tilde{y}}_i^q\right)}{\text{SE}\left({\widehat{y}}_i^q\right)}\mathrm{,}}q\mathrm{<mo>=</mo>}r\mathrm{,}a\mathrm{,}(4.3)$$

these measures are defined in a similar way for the estimates and predictions of the domain discontinuities ${\widehat{\Delta }}_i$ and ${\tilde{\Delta }}_i.$

4.2.1     Results for variables under the regular and alternative survey

In Table 4.6 the domain predictions and their standard errors averaged over the domains as well as the MRD, AMRD and MRDSE are given for the alternative survey under the univariate FH model with the models presented in Table 4.3. Results under the bivariate FH model, based on the final models of Table 4.1, are presented in Table 4.7 for the variables under the alternative survey and in Table 4.8 for the variables under the regular survey. Comparing the standard errors (SE) and the MRDSE in Table 4.6 and Table 4.7 shows that the bivariate FH model results in stronger reductions of the standard errors for all variables with the exception of $nuisance.$ This comes at the cost of an increased bias. Comparing MRD and AMRD in both tables shows that the deviations between the direct estimates and the small area predictions are larger under the bivariate FH model. Comparing the SE and MRDSE in Tables 4.7 and 4.8 shows that the improvement in precision with the bivariate FH model for the regular survey is smaller, as expected since the sample size of the regular survey is larger. The bias in the bivariate FH model predictions for the regular survey are also smaller, which follows from a comparison of MRD and AMRD in Tables 4.7 and 4.8.

The domain predictions under the univariate and bivariate FH model are plotted against the GREG estimates in Figures 4.1 through 4.5. The graphs also contain the GREG estimate at the national level versus the domain predictions aggregated to the national level according to (4.4). Figures 4.1 and 4.3 show that there is only a small amount of shrinkage for $offtot$ and $nuisance.$ Figure 4.2 shows for $unsafe$ that the bivariate FH model shrinks the domain predictions for the alternative survey while the amount of shrinkage for the univariate FH model for the alternative CVS and the bivariate FH model for the regular survey is smaller. For $propvict,$ see Figure 4.4, there is a small difference between the amount of shrinkage of the alternative CVS under the bivariate and univariate model. From Figure 4.5 it follows that the bivariate FH model for $satispol$ cannot adequately model the observations under the alternative survey with the auxiliary information from the two registers (MBA and PRRO). In this case the domain predictions of $satispol$ under the alternative approach display extreme overshrinkage. The univariate FH model indeed selects the same auxiliary variable from the regular survey only, see Table 4.3 and results in more realistic domain predictions.

For variables related to opinions and views such as $unsafe$ and $satispol,$ the reduction in the standard errors is accompanied by a relatively strong increase in the bias. This is especially the case with the small area prediction of the bivariate FH model for the alternative survey. For these variables, there are no strongly correlated covariates in the MBA and PRRO. In these cases the univariate FH model performs better since related covariates from the regular survey are selected (see Table 4.3), while the bivariate model doesn’t detect correlation between the random effects (see Table 4.1).

The direct estimates at the national level are accurate estimates since they are based on sufficiently large sample sizes. Therefore the bias in model-based domain predictions is often assessed by comparing the direct estimates at the national level with the domain predictions aggregated to the national level. The target variables in this application are all defined as population means. Therefore the aggregated domain predictions are obtained as the average over the domains weighted with the relative domain sizes,

$${\tilde{y}}^q\mathrm{<mo>=</mo>}{\displaystyle \sum _{i\mathrm{<mo>=</mo>\; 1}}^m\frac{N_i}{N}{\tilde{y}}_i^q}(4.4)$$

with $N_i$ the population size of domain $i$ and $N$ the size of the total population.

Table 4.9 compares the weighted average of the domain predictions according to (4.4) with the national GREG estimates. For the univariate FH model for the alternative CVS, the aggregated domain predictions are almost exactly equal to the GREG estimates at the national level. For the bivariate FH model the differences are slightly larger but the aggregated domain predictions are still very close to the GREG estimates at the national level. The largest relative difference amounts to 3% and is observed for $offtot$ under the regular survey.

4.2.2     Results for discontinuity estimates

In the last four columns of Table 4.9, the GREG estimates for the discontinuities at the national level are compared with the domain predictions obtained with the univariate FH model for the alternative CVS, the bivariate FH model and the univariate FH model for the discontinuities, aggregated to the national level using (4.4). The differences between the GREG estimates for the discontinuities at the national level and the aggregated domain predictions are the largest for the bivariate model and the smallest for the univariate FH model for the alternative CVS. This can be expected since the bivariate FH model shrinks both the domain estimates for the regular and alternative survey. With the univariate FH model for the alternative CVS, only the estimates for the alternative survey are replaced by domain predictions, while the estimates for the regular survey are not adjusted. In addition the domain predictions for the alternative survey have larger MRD’s and AMRD’s under the bivariate FH model compared to the univariate FH model (compare Table 4.6 and 4.7). The differences for the univariate FH model for the discontinuities are smaller compared to the bivariate FH model but larger compared to the univariate FH model for the alternative CVS.

In Tables 4.10, 4.11, and 4.12 the domain predictions and their standard errors for the discontinuities averaged over the domains as well as the MRD and MRDSE are summarized for the univariate FH model for the alternative CVS, bivariate FH model and the univariate FH model for the discontinuities respectively. The MRD’s are large because the GREG estimates for the discontinuities in the denominator of (4.11) frequently take values close to zero, which make these indicators unstable. Therefor the AMRD is replaced by the median of the absolute relative differences, $\left|\left({\widehat{y}}_i^q-{\tilde{y}}_i^q\right)/{\widehat{y}}_i^q\right|,$ and is abbreviated as MARD. The latter are indeed more stable indicators for bias. The MARD is the smallest for the univariate FH model for the alternative CVS, since this approach only adjusts the domain predictions of the alternative CVS. The MARD values for the bivariate FH model on their turn are smaller than those for the univariate FH model for the discontinuities.

With the exception of $nuisance$ the standard errors for the domain predictions under the bivariate FH model are smaller compared to the univariate FH model for the alternative CVS. In the case of $propvict$ and $offtot,$ this is the result of slightly more precise domain predictions for the alternative survey with respect to the univariate FH model (compare Table 4.6 with 4.7), a clear improvement in precision of the domain predictions of the regular survey compared to the GREG estimators (Table 4.8 and 2.2) and the positive correlation between the random effects. In the case of $satispol$ and $unsafe$ this is mainly the result of a clear improvement of precision of the domain predictions with the bivariate FH model for the regular compared to the GREG estimators (Table 4.8 and 2.2) and also a clear improvement of the precision of the domain predictions with the bivariate FH model for the alternative survey compared to the univariate model (compare Table 4.6 with Table 4.7).

For all five variables, the smallest standard errors are obtained with the univariate FH model for the discontinuities. This comes at the cost of a larger bias, as illustrated with the MARD values. An exception is $satispol,$ for which the bias in terms of MARD for the bivariate FH model is clearly larger than the univariate FH model for the discontinuities. For this variable the bias is the lowest with the univariate FH model for the alternative CVS, but the reduction of the standard errors is also smaller.

The last columns of Tables 4.11 and 4.12 contain the shrinkage factors for the domain discontinuities averaged over the domains. For the univariate FH model for the discontinuities the shrinkage factors for the predictions of the domain discontinuities, i.e., the weights attached to the direct estimator for the discontinuities, are defined as ${\gamma }_i\mathrm{<mo>=</mo>}{\widehat{\sigma }}_v^2/\left({\widehat{\sigma }}_v^2+\left({\psi }_i^r+{\psi }_i^a\right)\right).$ For the bivariate model the shrinkage factors for the predictions of the domain discontinuities are defined as ${\gamma }_i\mathrm{<mo>=</mo>}\iota \widehat{\Sigma }{\iota }^t/\left(\iota \widehat{\Sigma }{\iota }^t+\left({\psi }_i^r+{\psi }_i^a\right)\right),$ with $\iota \mathrm{<mo>=</mo>}{\left(\mathrm{1,}-1\right)}^t.$ The average shrinkage factor is defined as $\overline{\gamma }\mathrm{<mo>=</mo>}1/M{\displaystyle {\sum }_{i\mathrm{<mo>=</mo>\; 1}}^M{\gamma }_i}.$ Note that this statistic is not available for the discontinuities obtained with the univariate FH model for the alternative CVS, since under this approach domain discontinuities are obtained as the contrast between the GREG estimate for the regular survey and the domain prediction for the alternative approach. With the exception of $satispol,$ the shrinkage factors under the univariate FH model for discontinuities are a factor 10 smaller compared to those of the bivariate FH model. The question rises whether the extremely small shrinkage factors of the univariate FH model for the discontinuities overshrink the direct estimates of the discontinuities.

Plots of discontinuities estimated with the GREG estimator, the univariate FH model for the alternative CVS, the bivariate FH model and the univariate FH model for the discontinuities are provided in Figures 4.6 through 4.10. The predictions for the domain discontinuities obtained with the three models are more stable compared to the GREG estimates. This is e.g., clearly illustrated with $unsafe$ (Figure 4.7), where the GREG estimates for the discontinuity are sometimes positive and sometimes negative. The predictions for the domain discontinuities under the bivariate FH model and the univariate FH model for the discontinuities are consistently positive, which appears more plausible since it is unlikely that the domain discontinuities have opposite signs. The predictions for the domain discontinuities under the univariate FH model for the alternative CVS are closer to the GREG estimates and consequently less stable. A similar pattern can be observed for the other variables.

These plots illustrate that for $propvict,$ $offtot$ and $unsafe$ the bivariate FH model results in a clear improvement of the predictions for the domain discontinuities compared to the univariate FH model for the alternative CVS. For $nuisance$ the standard errors for the discontinuities increase with the bivariate FH model compared to the univariate FH model for the alternative CVS. The bivariate FH model for $satispol$ cannot adequately model the observations under the alternative survey with the auxiliary information from the two registers (MBA and PRRO). In this case the domain predictions of $satispol$ under the alternative approach display overshrinkage. The univariate FH model indeed selects an auxiliary variable from the regular survey, see Table 4.3, and clearly performs better.

It was anticipated that it would be difficult to produce reasonable predictions for the domain discontinuities with the univariate FH model for the direct estimates of the discontinuities since it is hard to imagine that the available auxiliary variables from registers like the MBA and PRRO contain good predictors for systematic differences in survey errors. Nevertheless, reasonable results are obtained with this more pragmatic approach. A possible interpretation is that the discontinuities are to some extent proportional to the values of the target variable and therefore show some systematic pattern that can be explained partially with the selected covariates. A point of concern are the very small shrinkage factors under this model, which might be an indication that the model gives too much weight to the synthetic estimator.

• Previous
• Article main page
• Next