Two local diagnostics to evaluate the efficiency of the empirical best predictor under the Fay-Herriot model
Section 4. Two diagnostics to evaluate the local performance of the B estimator

• Previous
• Next

4.1 An approach conditional on ${\widehat{\theta }}_i$

From expression (2.5) in Section 2 and noting that ${\gamma }_i\left({\widehat{\theta }}_i-{\beta }^T{z}_i\right)\mathrm{<mo>=</mo>}{b}_i{\sigma }_v\sqrt{{\gamma }_i}{\epsilon }_i,$ we obtain the conditional distribution of $v_i$:

                                                    $$v_i|\text{Z}\mathrm{,}{\widehat{\theta }}_i~\text{N}\left({\sigma }_v\sqrt{{\gamma }_i}{\epsilon }_i\mathrm{,}(1-{\gamma }_i){\sigma }_v^2\right)\mathrm{.}$$

Conditioning on ${\widehat{\theta }}_i$ gives a better idea of the possible values $v_i$ can take. In particular, when the value of ${\gamma }_i$ is strictly greater than 0, the conditional distribution of $v_i$ may deviate significantly from its unconditional distribution: $v_i|\text{Z}~\text{N}(\mathrm{0,}{\sigma }_v^2).$

The first diagnostic is defined as the conditional probability:

                                        $$\begin{array}{ll}{D}_{1i}\hfill & \mathrm{<mo>=</mo>}\text{Prob}\left({\text{MSE}}_p({\widehat{\theta }}_i^B)\le {\text{MSE}}_p({\widehat{\theta }}_i)|\text{Z}\mathrm{,}{\widehat{\theta }}_i\right)\hfill \\ \hfill & \mathrm{<mo>=</mo>}\text{Prob}\left(-v_{L\mathrm{,}i}\le v_i\le v_{L\mathrm{,}i}|\text{Z}\mathrm{,}{\widehat{\theta }}_i\right)\mathrm{.}(4.1)\hfill \end{array}$$

This diagnostic can be written as a function of ${\gamma }_i$ and the standardized error (2.4):

                   $$\begin{array}{ll}{D}_{1i}\mathrm{<mo>=</mo>}{D}_{1i}\left({\gamma }_i\mathrm{,}\left|{\epsilon }_i\right|\right)\hfill & =\Phi \left\{\sqrt{\frac{{\gamma }_i}{1-{\gamma }_i}}\left(\left|{\epsilon }_i\right|+\frac{\sqrt{1+{\gamma }_i}}{{\gamma }_i}\right)\right\}\\ \hfill & -\Phi \left\{\sqrt{\frac{{\gamma }_i}{1-{\gamma }_i}}\left(\left|{\epsilon }_i\right|-\frac{\sqrt{1+{\gamma }_i}}{{\gamma }_i}\right)\right\}\mathrm{,}(4.2)\end{array}$$

where $\Phi (\cdot )$ is the distribution function of the standard normal distribution. The proof of result (4.2) is given in Appendix A.

When this diagnostic takes values close to 0, we may conclude that $\left|v_i\right|$ is most likely larger than $v_{L\mathrm{,}i}$ and that the direct estimator is preferable to the B estimator. To obtain a decision rule associated with this diagnostic, it is necessary to choose a threshold below which we decide to choose the direct estimator and above which the B estimator is chosen. A 50% threshold seems quite natural. Another idea is to apply an empirical approach and identify a break in the distribution of the values of diagnostic $D_{1i}$ for the $m$ domains.

This diagnostic is not entirely design-based because it involves the conditional distribution $v_i|\text{Z}\mathrm{,}{\widehat{\theta }}_i.$ It is therefore necessary to validate carefully the Fay-Herriot model before using it. Unfortunately, it is not possible to validate the assumptions on both $v_i$ and $e_i$ because the values of the parameters ${\theta }_i\mathrm{,}i\mathrm{<mo>=</mo>}\mathrm{1,}\dots ,m$ are not observed. However, the combined Fay-Herriot model (2.3) can be validated using model residuals (see, for example, Hidiroglou, Beaumont and Yung, 2019). These residuals are obtained by replacing the unknown quantities in the standardized error (2.4) with their estimates (see Section 5). A graph of residuals versus model predicted values is often suggested to validate the linearity assumption of the model. The normality assumption of the error $b_i{v}_i+e_i$ can be verified by a Q-Q plot of the residuals or normality tests such as the Shapiro-Wilk test. In case the model is not completely satisfactory, a conservative threshold of 75% may be appropriate.

The diagnostic in the following section is entirely design-based. It is therefore not dependent on the validity of the linking model. In this sense, it is considered more robust than the diagnostic (4.2). However, it relies on assumptions about the sampling errors $e_i,$ discussed in Section 2, including the normality assumption of $e_i.$

4.2 Use of a design-based hypothesis test on the parameter $v_i$

In the design-based approach to inference, $v_i$ is fixed and the standardized error (2.4) follows the distribution:

                                            $${\epsilon }_i|\Omega ~\text{N}\left(v_i\frac{\sqrt{{\gamma }_i}}{{\sigma }_v}\mathrm{,}(1-{\gamma }_i)\right)\mathrm{.}(4.3)$$

We have a unique observation of this random variable. We use it to test if $\left|v_i\right|$ is larger than $v_{L\mathrm{,}i}.$ We consider the test:

                                                   $$H_0\text{}\mathrm{<mo>:</mo>}\left|v_i\right|\mathrm{<mo>=</mo>}{v}_{L\mathrm{,}i}\text{versus}{H}_1\text{}\mathrm{<mo>:</mo>}\left|v_i\right|\mathrm{<mo>></mo>}{v}_{L\mathrm{,}i}\mathrm{.}$$

We use $\left|{\epsilon }_i\right|$ as our test statistic. We expect that $\left|{\epsilon }_i\right|$ will have smaller values under $H_0$ than under $H_1.$ Let ${\epsilon }_{\text{obs}\mathrm{,}i}$ be the observed value of the statistic ${\epsilon }_i$ and $P_i\left(v_i\right)\mathrm{<mo>=</mo>}\text{Prob}\left(\left|{\epsilon }_i\right|\mathrm{<mo>></mo>}\left|{\epsilon }_{\text{obs}\mathrm{,}i}|\right|\Omega \mathrm{<mo>;</mo>}{v}_i\right).$ The $p$ -value of the test is defined as the probability that the statistic $\left|{\epsilon }_i\right|$ is greater than the observed value $\left|{\epsilon }_{\text{obs}\mathrm{,}i}\right|$ under the null hypothesis. Appendix B shows that the $p$ -value is:

                                         $$P_i\left(v_{L\mathrm{,}i}\right)\mathrm{<mo>=</mo>}{P}_i\left(-v_{L\mathrm{,}i}\right)\mathrm{<mo>=</mo>}\Phi \left(-{\tau }_i\right)+\Phi \left(-{\tau }_i-2\frac{\sqrt{1+{\gamma }_i}}{\sqrt{1-{\gamma }_i}}\right)\mathrm{,}$$

where

$${\tau }_i\mathrm{<mo>=</mo>}\frac{\left|{\epsilon }_{\text{obs}\mathrm{,}i}\right|-\sqrt{1+{\gamma }_i}}{\sqrt{1-{\gamma }_i}}\mathrm{.}$$

Since the second term is often negligible compared to the first term, especially when ${\tau }_i\mathrm{<mo>></mo>}0$ or ${\gamma }_i$ is large, our second diagnostic is:

                                 $$D_{2i}\mathrm{<mo>=</mo>}{D}_{2i}\left({\gamma }_i\mathrm{,}\left|{\epsilon }_{\text{obs}\mathrm{,}i}\right|\right)\mathrm{<mo>=</mo>}\Phi \left(\frac{\sqrt{1+{\gamma }_i}-\left|{\epsilon }_{\text{obs}\mathrm{,}i}\right|}{\sqrt{1-{\gamma }_i}}\right)\mathrm{.}(4.4)$$

This second diagnostic can be interpreted as follows: When $D_{2i}$ is small, we can assume that $\left|v_i\right|$ is likely to be larger than $v_{L\mathrm{,}i}$ and the direct estimator is then preferred to the B estimator. For the choice of a decision threshold, values typically used as levels for hypothesis testing (e.g., 5% or 10%) can be used as a guide. With these small values, the B estimator is favoured. As with the previous diagnostic, the threshold can be determined by locating a break in the distribution of the values of diagnostic $D_{2i}$ for the $m$ domains.

4.3 Some properties of diagnostics 1 and 2

In this section, we study the behaviour of the functions $D_{1i}\left({\gamma }_i\mathrm{,}\left|{\epsilon }_i\right|\right)$ and $D_{2i}\left({\gamma }_i\mathrm{,}\left|{\epsilon }_i\right|\right)$ for limiting cases of ${\gamma }_i$ and $\left|{\epsilon }_i\right|$ and note their similarities and differences.

Case 1: $0\mathrm{<mo><</mo>}{\gamma }_i\mathrm{<mo><</mo>}1$ is fixed and $\left|{\epsilon }_i\right|\to \infty .$

From equations (4.2) and (4.4) it can be shown that, for $\left|{\epsilon }_i\right|\mathrm{<mo>></mo>}0,$ the two functions $D_{1i}\left({\gamma }_i\mathrm{,}\left|{\epsilon }_i\right|\right)$ and $D_{2i}\left({\gamma }_i\mathrm{,}\left|{\epsilon }_i\right|\right)$ decrease as $\left|{\epsilon }_i\right|$ increases. In other words, the derivative of these functions with respect to $\left|{\epsilon }_i\right|$ is negative. In addition, the limit when $\left|{\epsilon }_i\right|\to \infty$ of these two functions tends toward 0. For a sufficiently large value of $\left|{\epsilon }_i\right|,$ the two diagnostics will therefore favour the direct estimator.

Case 2: $0\mathrm{<mo><</mo>}{\gamma }_i\mathrm{<mo><</mo>}1$ is fixed and $\left|{\epsilon }_i\right|\mathrm{<mo>=</mo>}0.$

From equation (4.2), we observe that

                                          $$D_{1i}\left({\gamma }_i\mathrm{,}0\right)\mathrm{<mo>=</mo>}\Phi \left(\sqrt{\frac{1+{\gamma }_i}{{\gamma }_i\left(1-{\gamma }_i\right)}}\right)-\Phi \left(-\sqrt{\frac{1+{\gamma }_i}{{\gamma }_i\left(1-{\gamma }_i\right)}}\right)\mathrm{.}$$

We can show that $D_{1i}\left({\gamma }_i\mathrm{,}0\right)$ is minimized when ${\gamma }_i\mathrm{<mo>=</mo>}-1+\sqrt{2}.$ Therefore, $D_{1i}\left({\gamma }_i\mathrm{,}0\right)\ge D_{1i}\left(-1+\sqrt{2}\mathrm{,0}\right)\mathrm{<mo>=</mo>}$ 0.98. Since this value is close to 1, diagnostic 1 leads to choosing the B estimator in this case if a threshold of 0.50 or even 0.75 is chosen.

From equation (4.4) we obtain:

$$D_{2i}\left({\gamma }_i\mathrm{,}0\right)\mathrm{<mo>=</mo>}\Phi \left(\sqrt{\frac{1+{\gamma }_i}{1-{\gamma }_i}}\right)\mathrm{.}$$

We can show that, for $0\le {\gamma }_i\mathrm{<mo><</mo>}1,$ the function $D_{2i}\left({\gamma }_i\mathrm{,}0\right)$ is minimized when ${\gamma }_i\mathrm{<mo>=</mo>}0.$ Hence, $D_{2i}\left({\gamma }_i\mathrm{,}0\right)\ge D_{2i}\left(\mathrm{0,}0\right)\mathrm{<mo>=</mo>}$ 0.84. With a threshold smaller than 0.50, diagnostic 2 leads to the same decision as diagnostic 1 in this case, i.e. to choose the B estimator.

Case 3: $\left|{\epsilon }_i\right|\mathrm{<mo><</mo>}\sqrt{2}$ is fixed and ${\gamma }_i\to 1.$

The two functions $D_{1i}\left({\gamma }_i\mathrm{,}\left|{\epsilon }_i\right|\right)$ and $D_{2i}\left({\gamma }_i\mathrm{,}\left|{\epsilon }_i\right|\right)$ tend toward 1 in this case. Therefore, diagnostics 1 and 2 lead to choosing the B estimator.

Case 4: $\left|{\epsilon }_i\right|\mathrm{<mo>></mo>}\sqrt{2}$ is fixed and ${\gamma }_i\to 1.$

The two functions $D_{1i}\left({\gamma }_i\mathrm{,}\left|{\epsilon }_i\right|\right)$ and $D_{2i}\left({\gamma }_i\mathrm{,}\left|{\epsilon }_i\right|\right)$ tend toward 0 in this case. Diagnostics 1 and 2 lead here to choosing the direct estimator.

Case 5: $\left|{\epsilon }_i\right|$ is fixed and ${\gamma }_i\to 0.$

The function $D_{1i}\left({\gamma }_i\mathrm{,}\left|{\epsilon }_i\right|\right)$ tends toward 1 for any fixed value of $\left|{\epsilon }_i\right|.$ Therefore, diagnostic 1 favours the B estimator for small values of ${\gamma }_i.$

We note that $D_{2i}\left(\mathrm{0,}\left|{\epsilon }_i\right|\right)\mathrm{<mo>=</mo>}\Phi \left(1-\left|{\epsilon }_i\right|\right).$ Therefore, contrary to Diagnostic 1, Diagnostic 2 will lead to choosing the direct estimator if $\left|{\epsilon }_i\right|$ is sufficiently large even when ${\gamma }_i$ is infinitely close to 0. For example, with a decision threshold at 0.05 and ${\gamma }_i\mathrm{<mo>=</mo>}0,$ Diagnostic 2 favours the direct estimator when $\left|{\epsilon }_i\right|\mathrm{<mo>></mo>}1-{\Phi }^{-1}\left(\text{0}\text{.05}\right)\mathrm{<mo>=</mo>}$ 2.64.

In the first four cases above, both diagnostics lead to the same decision. There is a difference only in Case 5 where ${\gamma }_i\to 0.$ We therefore expect that Diagnostic 2 will choose the direct estimator more often than Diagnostic 1 for small values of ${\gamma }_i.$ Consider, for example, a threshold of 0.5 for Diagnostic 1 and of 0.05 for Diagnostic 2. For a threshold of 0.5, we can show that Diagnostic 1 leads to choosing the direct estimator as soon as $\left|{\epsilon }_i\right|$ is larger than a value approximately equal to $\frac{\sqrt{1+{\gamma }_i}}{{\gamma }_i},$ i.e. as soon as $\left|{\epsilon }_i\right|\underset{~}{>}\frac{\sqrt{1+{\gamma }_i}}{{\gamma }_i}.$ As for Diagnostic 2, for a threshold of 0.05, it leads to choosing the direct estimator as soon as $\left|{\epsilon }_i\right|>\sqrt{1+{\gamma }_i}-\sqrt{1-{\gamma }_i}{\Phi }^{-1}\left(\text{0}\text{.05}\right).$ For ${\gamma }_i\mathrm{<mo>=</mo>}$ 0.01, Diagnostic 1 thus leads to choosing the direct estimator when $\left|{\epsilon }_i\right|\underset{~}{>}$ 100.5, while Diagnostic 2 leads to choosing the direct estimator when $\left|{\epsilon }_i\right|\mathrm{<mo>></mo>}$ 2.64. The gap narrows as ${\gamma }_i$ increases. For example, for ${\gamma }_i\mathrm{<mo>=</mo>}$ 0.2, Diagnostic 1 chooses the direct estimator when $\left|{\epsilon }_i\right|\underset{~}{>}$ 5.48 and Diagnostic 2 chooses the direct estimator when $\left|{\epsilon }_i\right|\mathrm{<mo>></mo>}$ 2.57. The above discussion seems to suggest that Diagnostic 2 leads to choosing the direct estimator more often than Diagnostic 1. However, there are cases where Diagnostic 1 chooses the direct estimator contrary to Diagnostic 2. These cases generally occur for fairly large values of ${\gamma }_i.$ For example, for ${\gamma }_i\mathrm{<mo>=</mo>}$ 0.8, Diagnostic 1 chooses the direct estimator when $\left|{\epsilon }_i\right|\underset{~}{>}$ 1.68, while Diagnostic 2 chooses the direct estimator only when $\left|{\epsilon }_i\right|\mathrm{<mo>></mo>}$ 2.08.

• Previous
• Article main page
• Next