5. Robust estimation of domain totals

Cyril Favre Martinoz, David Haziza and Jean-François Beaumont

Previous | Next

In practice, we usually want to produce estimates for population domains as well as an estimate at the global level. Let $t_g={\displaystyle {\sum }_{i\in U_g}}{y}_i$ be the total of the $y-$ variable in domain $g.$ We assume that the domains form a partition of the population such that $t={\displaystyle {\sum }_{i\in U}{y}_i}={\displaystyle {\sum }_{g=1}^G{t}_g},$ where $G$ is the number of domains. Let $S_g$ be the set of sampled units in domain $g.$ The expansion estimator of $t_g$ is given by ${\widehat{t}}_g={\displaystyle {\sum }_{i\in S_g}{d}_i{y}_i}\mathrm{.}$ We have the consistency relation ${\displaystyle {\sum }_{g=1}^G{\widehat{t}}_g}=\widehat{t}\mathrm{.}$

In the presence of influential values, we can apply a robust procedure separately for each domain using the method described in Section 3, which leads to $G$ robust estimators, ${\widehat{t}}_{R\mathrm{,}g}\mathrm{.}$ A robust estimator of the total at the population level, ${\widehat{t}}_{R\left(\text{agg}\right)}\mathrm{,}$ is easily obtained by aggregating the robust estimators ${\widehat{t}}_{R\mathrm{,}g}.$ Thus, we have ${\widehat{t}}_{R\left(\text{agg}\right)}={\displaystyle {\sum }_{g=1}^G{\widehat{t}}_{R\mathrm{,}g}}\mathrm{.}$ The consistency relation between the domain-level estimates and the population-level estimate is therefore satisfied. However, aggregating $G$ robust estimators, each suffering from a potential bias, may produce a highly biased aggregate robust estimator, ${\widehat{t}}_{R\left(\text{agg}\right)}.$ In most cases, the bias of ${\widehat{t}}_{R\left(\text{agg}\right)}$ will be negative, since each of the ${\widehat{t}}_{R\mathrm{,}g}$ estimators has a negative bias.

To avoid having an estimator with an unacceptable bias, we first compute the robust estimator (4.8), ${\widehat{t}}_{R\mathrm{,}g}\mathrm{,}$ for each domain. Then, we independently compute a robust estimator of the total $t$ in the population, ${\widehat{t}}_{R\mathrm{,0}}\mathrm{,}$ given by (4.8). In this case, however, the consistency relation is no longer necessarily satisfied. In other words, we have ${\widehat{t}}_{R\mathrm{,0}}\ne {\displaystyle {\sum }_{g=1}^G{\widehat{t}}_{R\mathrm{,}g}}\mathrm{,}$ in general. It is therefore necessary to force consistency between the robust domain estimates and the aggregate robust estimate using a method similar to calibration. To do so, we compute final robust estimates ${\widehat{t}}_{R\mathrm{,}g}^{*},g=0,1,\mathrm{..,}G\mathrm{,}$ that are as close as possible to the initial robust estimates ${\widehat{t}}_{R\mathrm{,}g},$ based on a particular distance function, and that satisfy the calibration equation

$${\displaystyle \sum _{g=1}^G{\widehat{t}}_{R\mathrm{,}g}^{*}}={\widehat{t}}_{R\mathrm{,0}}^{*}\mathrm{.}(5.1)$$

In the case of the generalized chi-square distance function, we are seeking final robust estimates, ${\widehat{t}}_{R\mathrm{,}g}^{*}\mathrm{,}$ such that

$${\displaystyle \sum _{g=0}^G}\frac{{\left\{{\widehat{t}}_{R\mathrm{,}g}^{*}-{\widehat{t}}_{R\mathrm{,}g}\right\}}^2}{2q_g{\widehat{t}}_{R\mathrm{,}g}}(5.2)$$

is minimized subject to (5.1). The coefficient $q_g$ in the above expression is a weight assigned to the initial estimate in domain $g,{\widehat{t}}_{R\mathrm{,}g}\mathrm{,}$ and is interpreted as its importance in the minimization problem. Using the Lagrange multipliers method, we can easily obtain a solution to this minimization problem. The solution is given by

$${\widehat{t}}_{R\mathrm{,}g}^{*}={\widehat{t}}_{R\mathrm{,}g}-\frac{{\displaystyle \sum _{h=0}^G{\delta }_h{\widehat{t}}_{R\mathrm{,}h}}}{{\displaystyle \sum _{h=0}^G{q}_h{\widehat{t}}_{R\mathrm{,}h}}}{\delta }_g{q}_g{\widehat{t}}_{R\mathrm{,}g}\mathrm{,}(5.3)$$

where ${\delta }_0=-1$ and ${\delta }_g=1,$ for $g=1,\dots \mathrm{,}G.$

We make the following remarks: (i) If $q_g=0,$ then the final robust estimate ${\widehat{t}}_{R\mathrm{,}g}^{*}$ is identical to the initial robust estimate ${\widehat{t}}_{R\mathrm{,}g}.$ Thus, if we want to ensure that the initial estimate in domain $g$ is not modified excessively, we simply associate it with a small value of $q_g.$ This point is also illustrated empirically in Section 6.2. (ii) Note that like the initial robust estimates at the domain level, ${\widehat{t}}_{R\mathrm{,}g}\mathrm{,}$ for $g=1,\dots \mathrm{,}G,$ the initial robust estimate at the population level, ${\widehat{t}}_{R\mathrm{,0}}\mathrm{,}$ can also be modified. (iii) If $q_0=0$ (in other words, the initial robust estimate for the population level is not modified) and $q_g=q$ for $g=1,\dots \mathrm{,}G,$ where $q$ is a strictly positive constant, expression (5.3) simplifies to

$${\widehat{t}}_{R\mathrm{,}g}^{*}={\widehat{t}}_{R\mathrm{,}g}\left(\frac{{\widehat{t}}_{R\mathrm{,0}}}{{\widehat{t}}_{R\left(\text{agg}\right)}}\right)\mathrm{.}(5.4)$$

In this case, the initial estimates ${\widehat{t}}_{R\mathrm{,}g}$ are all modified by the same factor, ${\widehat{t}}_{R\mathrm{,0}}/{\widehat{t}}_{R\left(\text{agg}\right)}.$ (iv) How can we set the values of $q_g$ in practice? It seems natural to adopt the following choice:

$$q_g=\widehat{\text{CV}}\left({\widehat{t}}_g\right)/{\displaystyle \sum _{g=1}^G\widehat{\text{CV}}\left({\widehat{t}}_g\right)}\mathrm{,}$$

where $\widehat{\text{CV}}\left({\widehat{t}}_g\right)$ is the estimated coefficient of variation (CV) associated with domain $g.$ For example, in a repeated survey, the estimated CV observed in a previous iteration can be used. This choice of $q_g$ is based on the fact that we will not want to make a large change in the initial estimate associated with a domain that has a small estimated CV. In such a domain, the problem of influential values is clearly less serious, and the initial robust estimate ${\widehat{t}}_{R\mathrm{,}g}$ is expected to be relatively close to the actual total $t_g.$ In other words, the robust estimator ${\widehat{t}}_{R\mathrm{,}g}$ should have low bias and be relatively stable. It therefore makes sense not to attempt to change the initial robust estimate substantially. (v) In (5.2), we used the generalized chi-square distance, which leads to the linear method. In the literature on calibration (e.g., Deville and Särndal 1992), there are a number of other calibration methods. In particular, there is the Kullback-Leibler distance, which leads to the exponential method and the logit and truncated linear methods. Using the last two methods, we can specify positive bounds $C_1$ and $C_2$ such that $C_1\le {\widehat{t}}_{R\mathrm{,}g}^{*}/{\widehat{t}}_{R\mathrm{,}g}\le C_2.$ In other words, we ensure that the ratio ${\widehat{t}}_{R\mathrm{,}g}^{*}/{\widehat{t}}_{R\mathrm{,}g}$ falls within the interval between $C_1$ and $C_2.$ Note that the calibration procedure may lead to ${\widehat{t}}_{R\mathrm{,}g}^{*}-{\widehat{t}}_g\ge \mathrm{0,}$ for a certain $g,$ which is counterintuitive. In this case, we simply include the constraint ${\widehat{t}}_{R\mathrm{,}g}^{*}\le {\widehat{t}}_g$ for $g=1,\dots \mathrm{,}G\mathrm{,}$ in the calibration procedure. (vi) An alternative is to express ${\widehat{t}}_{R\mathrm{,}g}^{*}$ as a weighted sum of the initial values using modified weights:

$${\widehat{t}}_{R\mathrm{,}g}^{*}={\displaystyle \sum _{i\in S_g}{\tilde{d}}_i{}^{*}{y}_i}\mathrm{,}$$

where

$${\tilde{d}}_i{}^{*}={\tilde{d}}_i\left(1-{\delta }_g{q}_g\frac{{\displaystyle \sum _{h=0}^G{\delta }_h{\widehat{t}}_{R\mathrm{,}h}}}{{\displaystyle \sum _{h=0}^G{q}_h{\widehat{t}}_{R\mathrm{,}h}}}\right)$$

and ${\tilde{d}}_i$ is given by either (4.3) or (4.6). We can also write the estimator ${\widehat{t}}_{R\mathrm{,}g}^{*}$ as a weighted sum with the initial weights using modified values:

$${\widehat{t}}_{R\mathrm{,}g}^{*}={\displaystyle \sum _{i\in S_g}{d}_i{\tilde{y}}_i^{*}}\mathrm{,}$$

where

$${\tilde{y}}_i^{*}={\tilde{y}}_i\left(1-{\delta }_g{q}_g\frac{{\displaystyle \sum _{h=0}^G{\delta }_h{\widehat{t}}_{R\mathrm{,}h}}}{{\displaystyle \sum _{h=0}^G{q}_h{\widehat{t}}_{R\mathrm{,}h}}}\right)\mathrm{,}i\in g$$

and ${\tilde{y}}_i$ is given by either (4.1) or (4.4). (vii) We may want to find the winsorization thresholds $K_g\mathrm{,}g=1,\dots \mathrm{,}G,$ such that the standard winsorized estimator or the Dalén-Tambay winsorized estimator is equal to ${\widehat{t}}_{R\mathrm{,}g}^{*}.$ We can follow a procedure similar to the one in Section 4, and we can use an algorithm similar to the one in the Appendix. A necessary condition for the existence of a solution is that ${\widehat{t}}_g-{\widehat{t}}_{R\mathrm{,}g}^{*}\ge 0.$ (viii) With the proposed calibration procedure, more than one partition of the population can be dealt with jointly. For example, we may be interested in publishing both provincial estimates and industry estimates. If so, we simply insert the following calibration equations into the calibration procedure:

$${\displaystyle \sum _{g=1}^G{\widehat{t}}_{R\mathrm{,}g}^{*}}={\widehat{t}}_{R\mathrm{,0}}^{*}\mathrm{,}$$

$${\displaystyle \sum _{l=1}^L{\widehat{t}}_{R\mathrm{,}l}^{*}}={\widehat{t}}_{R\mathrm{,0}}^{*}\mathrm{,}$$

where $G$ and $L$ denote the number of provinces and the number of industries respectively. The method can also be applied to more than two partitions of the population.

Previous | Next