2. Review of given poverty indicators and their linearized variables

Eric Graf and Yves Tillé

Previous | Next

Let $U$  be a finite population consisting of $N$  identifiable units $u_1\mathrm{,...,}{u}_k\mathrm{,...,}{u}_N.$  To simplify the notation, let unit  $u_k$  be denoted simply by the index  $k.$ In practice the population  $U$ is a sampling frame with acceptable coverage of a given population for which we wish to make inferences. To each unit  $k$ is associated the value  $y_k$ for a given characteristic (in this case, income). Without loss of generality, to simplify the notation, assume that the values of $y_k$  are distinct and sorted by order of magnitude, so that  $y_k=y_{\left[k\right]}.$  Data from sample surveys often contain duplicates, that is, a number of units with the same value  $y$ , as a result of rounding or range questions. In these cases and for this study, we can simply increase the values by a small (negligible), randomly selected, uniformly distributed amount so that the data may be sorted unambiguously.

Let  $S$  be a random sample of size  $n$ obtained using a sample design $p(s)=P(S=s),$  for all $s\subset U.$  In addition, let ${\pi }_k=P(k\in s)\text{>}0$  be the inclusion probability of unit  $k$ of $U.$  As well, let $d_k=1/{\pi }_k$  be the sampling weight, and let $w_k=w_k(s)$  be the estimation weight, which may be equal to $d_k$  or may be more refined. For example, $w_k$  may have been obtained after calibration (Deville and Särndal 1992) and therefore also reflect a non-response adjustment.

The estimators of poverty and inequality indicators are non-linear statistics that can’t be expressed as regular functions of totals (that is, continuously differentiable up to the second order). In fact, they are rank statistics for the Gini coefficient and quantile statistics for the others. As Osier (2009) points out, their variance therefore can’t be estimated using a Taylor linearization; the generalized linearization method is required instead (Deville 2000, Demnati and Rao 2004, and Osier 2009). An alternative for estimating variance would be to use bootstrap-type re-sampling techniques but, for the EU-SILC survey data, preference was given to the linearization technique, at least for a certain number of participating countries. Indeed, re-sampling methods often require more human and machine resources. As well, since Eurostat collaborates with some 30 countries that have different sampling designs and that may perform non-response adjustments and calibration to external sources, it seemed more appropriate to select an analytical solution for estimating variance. In addition, some countries might be using the existing SAS software POULPE (Ardilly and Osier 2007) to generate the required estimates. That was the case for initial tests using Swiss EU-SILC data. Here we use a procedure that, as Antal, Langel and Tillé (2011) point out, reconciles the approach introduced by Deville (2000) with that of Demnati and Rao (2004). Both approaches use the concept of influence function initially developed in the field of robust statistics (Hampel 1974). Antal et al. (2011) also state that the same linearized variables can be found by applying the method proposed by Graf (2011 and 2013) that constructs a linearized variable on the basis of a Taylor expansion with respect to sample inclusion indicators. Note also the work by Kovačević and Binder (1997) in which a linearization approach using estimating equations is developed.

Deville (2000) states that the influence of a unit  $k$ on a population parameter of interest  $\theta$ is determined by an infinitesimal variation in the importance assigned to the unit. The parameter is expressed as a functional  $\theta =T(M),$ where $M$  is a measure allocating a mass of 1, $M(k)=M_k=1,$  only at points on the continuum corresponding to units $k\in U.$  The specialization of the general measure $M$  into a discrete measure turns the functional  $T,$ predefined on a continuum, into a discrete functional, in the same way as the total  $Y$ is defined as the sum of all  $y_k$  over the given finite population. The influence function of  $T,$  or the linearized variable, is defined as

$$I{\left[T\left(M\right)\right]}_k=z_k=\underset{t\to 0}{{\displaystyle \lim }}\frac{T\left(M+t{\delta }_k\right)-T\left(M\right)}{t}\mathrm{,}\text{ for all}k\in U\mathrm{,}$$

where ${\delta }_k$  is the Dirac measure for unit  $k\left({\delta }_k\left(i\right)=1\text{if}i=k\text{and}0\text{otherwise}\right).$ In practice, we have known data only from a sample  $S$ and Deville (2000) defines a linearized variable  ${\widehat{z}}_k$ , or empirical influence function, by (1) determining the limit above using differential calculus and (2) replacing the unknowns in the evaluation with the corresponding estimated quantities using the sample. Deville justifies this procedure by showing that

$$T\left(\widehat{M}\right)-T\left(M\right)\approx \left({\displaystyle \sum _{k\in S}}{w}_k{z}_k-{\displaystyle \sum _{k\in U}}{z}_k\right)\mathrm{.}$$

The key result is that, under asymptotic conditions described by Deville (2000), which are in theory satisfied when the sample is “sufficiently large”, the variance of the estimated total of the variable  ${\widehat{z}}_k$ is an approximation of the variance of the (complex) statistic $\widehat{\theta }:$

$$\text{var}\left[{\displaystyle \sum _{k\in s}}{\widehat{z}}_k{w}_k\right]\approx \text{var}\left(\widehat{\theta }\right)\mathrm{.}$$

The starting point of Deville’s approach is therefore the population parameter and not the estimator that is proposed to be used for the evaluation using the sample. When the estimator used follows naturally from the population parameter expression (for example, the total  $Y$ approached by the Horvitz-Thompson estimator), the procedure is unambiguous. However, imprecision arises if we estimate the same total  $Y$ using the ratio estimator with an auxiliary variable  $x.$ In that case, Deville’s approach, which does not specify the form of the total estimator to use, will yield a constant influence function equal to 1, instead of bringing the unknown ratio of interest into play.

An alternative that avoids these problems is the approach by Demnati-Rao, when used in Deville’s framework, as done in Antal et al. (2011). They present the Demnati-Rao approach as resulting from Deville’s framework when the measure  $M$ used is not the discrete measure defined for  $U$ described above, but rather the following measure defined for  $S,$ the sample:

$$\widehat{M}\left(k\right)=w_k\mathrm{,}k\in S$$

where $w_k$  is a weight. By defining the measure for  $S,$ the starting point becomes the estimator and not the parameter; it is the parameter that is initially expressed as a functional, and not the population parameter to be estimated. That is, the functional corresponds to the estimator for which we are seeking a variance estimate using generalized linearization. We then obtain the linearized variable based on that functional as follows:

$$I{\left[T\left(\widehat{M}\right)\right]}_k={\widehat{z}}_k=\underset{t\to 0}{{\displaystyle \lim }}\frac{T\left(\widehat{M}+t{\delta }_k\right)-T\left(\widehat{M}\right)}{t}\mathrm{,}\text{ for}\text{all}k\in S\mathrm{.}$$

Antal et al. (2011) note that, to the extent that the functional in this limit is expressed as an explicit function of the variables that are the weights assigned by the measure  $\widehat{M}$ to the observations, this linearized variable is in fact a function of the partial derivatives with respect to the weights:

$$I{\left[T\left(\widehat{M}\right)\right]}_k=\frac{\partial T\left(\widehat{M}\right)}{\partial w_k}\mathrm{.}$$

Antal et al. (2011) point out that the linearized variables that we will discuss below can be obtained using either approach. In fact, computing the limit using the Demnati-Rao approach does not necessarily result in the variance estimate suggested by Deville (2000). The practical approach used in this article might therefore be called the Deville-Demnati-Rao approach, in recognition of the theoretical framework provided by Deville (2000) and the practical algorithm for Deville’s framework provided by Demnati and Rao (2004).

Using this method, the variance of $\widehat{\theta }$  can be estimated for any sampling design, and a confidence interval can therefore be obtained by substituting the linearized variable in the variance formula for a total for the selected sampling design. If the sampling design is simple random sampling without replacement, the estimator of the variance of an inequality indicator  $\widehat{\theta }$ is defined as

$${\widehat{\mathrm{var}}}_{\text{lin}}\left[\widehat{\theta }\right]=\frac{N\left(N-n\right)}{n}\frac{1}{n-1}{\displaystyle \sum _{k\in S}}{\left({\widehat{z}}_k-\overline{z}\right)}^2\mathrm{,}\left(2.1\right)$$

where

$$\overline{z}=n^{-1}{\displaystyle \sum _{k\in S}}{\widehat{z}}_k\mathrm{.}$$

Below, we review the empirical definitions of the inequality indicators considered with respect to population income measurement, as well as the expressions for the linearized variables of the indicators as we have implemented them.

2.1 Gini coefficient

The Gini coefficient, $G,$  ranges from 0 (complete equality, that is, all individuals earn the same amount) to 1 (complete inequality, that is, one individual has all the income and the other individuals have no income). The coefficient  $G$ is expressed on the basis of the cumulative income of a given proportion of the poorest individuals. If $\mathcal{Y}$  is a random variable representing income, $f(y)$  its density function and $F(y)$  its distribution function, then the Lorenz curve (Lorenz 1905) can be written as

$$L\left(\alpha \right)=\frac{{\displaystyle {\int }_0^{F^{-1}(\alpha )}}yf\left(y\right)dy}{{\displaystyle {\int }_0^{\infty }}yf\left(y\right)dy}=\frac{1}{E\left(\mathcal{Y}\right)}{\displaystyle {\int }_0^{\alpha }}{F}^{-1}\left(u\right)du\mathrm{.}$$

The Gini coefficient represents twice the area between the Lorenz curve and the line of complete equality (the diagonal line $f_{eg}\left(x\right)=x),$ as shown in Figure 2.1. Therefore, the Gini coefficient can be defined as

$$G=2{\displaystyle {\int }_0^1}\left[\alpha -L\left(\alpha \right)\right]d\alpha \mathrm{.}$$

Description for figure 2.1

If a population  $U$ is finite, then the values of $y_k$  will not be random and the Gini coefficient can be calculated as

$$G=\frac{2{\displaystyle {\sum }_{k\in U}k{y}_k}}{N{\displaystyle {\sum }_{k\in U}{y}_k}}-\frac{N+1}{N}\mathrm{,}$$

where the values of $y_k$  are sorted by rank. For a sample, the Gini coefficient can be estimated as

$$\begin{array}{l}\widehat{G}=\frac{2}{\widehat{N}\widehat{Y}}{\displaystyle \sum _{k\in S}}{w}_k{\widehat{N}}_k{y}_k-\left(1+\frac{1}{\widehat{N}\widehat{Y}}{\displaystyle \sum _{k\in S}}{w}_k^2{y}_k\right)\\ \begin{array}{cc}& =\frac{{\displaystyle {\sum }_{k\in S}{\displaystyle {\sum }_{\ell \in S}{w}_k{w}_{\ell }\left|y_k-y_{\ell }\right|}}}{2\widehat{N}\widehat{Y}}\mathrm{,}\end{array}\end{array}$$

where ${\widehat{N}}_k={\displaystyle {\sum }_{\ell \in S}}{w}_{\ell }{1}_{\left[y_{\ell }\le y_k\right]}$  is the sum of the weights  $w_k,$ $\widehat{Y}={\displaystyle {\sum }_{k\in S}}{w}_k{y}_k$  is the estimated total income of the population, and $\widehat{N}={\displaystyle {\sum }_{k\in S}}{w}_k$  is the estimated size of the population. The expression can be simplified as follows if all the weights are equal to $N/n:$

$$\widehat{G}=\frac{2{\displaystyle {\sum }_{k\in S}k{y}_k}}{n{\displaystyle {\sum }_{k\in S}{y}_k}}-\frac{n+1}{n}\mathrm{.}$$

Note that the definition may vary by a factor of $n/\left(n-1\right)$  depending on the author (Osier 2009 and Eurostat 2004b); however, this subtlety becomes negligible if the sample is large enough.

Langel and Tillé (2012) combine the various approaches to obtain the same estimated linearized variable of the Gini coefficient for the sample:

$${\widehat{z}}_k^{\text{GINI}}=\frac{1}{\widehat{N}\widehat{Y}}\left[2{\widehat{N}}_k(y_k-{\widehat{\overline{Y}}}_k)+\widehat{Y}-\widehat{N}{y}_k-\widehat{G}\left(\widehat{Y}+y_k\widehat{N}\right)\right]\mathrm{,}$$

where ${\widehat{\overline{Y}}}_k={\displaystyle {\sum }_{\ell =1}^k}{w}_{\ell }{y}_{\ell }/{\widehat{N}}_k,$  and the values of $y_{\ell }$  are sorted and distinct.

2.2 Quintile Share Ratio (QSR or $S_{80}/S_{20}$ )

A good overview of this indicator is provided by Langel and Tillé (2012). Let $q_{80}$  and $q_{20}$  be the 80th and 20th percentiles of the distribution function $F\left(y\right).$  The QSR is the ratio of the total income of the 20% of the population with the highest income to the total income of the 20% of the population with the lowest income. In the continuous case, the QSR can be defined as

$$\text{QSR = }\frac{E\left(\mathcal{Y}|\mathcal{Y}\text{> }{q}_{80}\right)}{E\left(\mathcal{Y}|\mathcal{Y}<q_{20}\right)}=\frac{1-L\left(\text{0}\text{.8}\right)}{L\left(\text{0}\text{.2}\right)}$$

where $\mathcal{Y}$  is a random variable representing income. For finite populations, the QSR can be expressed and estimated for a sample on the basis of partial sums,

$$\widehat{\text{QSR}}=\frac{\widehat{Y}-{\widehat{Y}}_{\text{0}\text{.8}}}{{\widehat{Y}}_{\text{0}\text{.2}}}\mathrm{,}$$

where, given the results obtained by Langel and Tillé (2011), we will use the following definition of the partial sum, which differs slightly from the official definition of Eurostat (2004a):

$${\widehat{Y}}_{\alpha }={\displaystyle \sum _{k\in S}}{w}_k{y}_{k}H\left(\frac{\alpha \widehat{N}-{\widehat{N}}_{k-1}}{w_k}\right)\mathrm{,}\left(2.2\right)$$

with

$$H\left(x\right)=\{\begin{array}{ll}0\hfill & \text{if}x<0\hfill \\ x\hfill & \text{if}0\le x<\hfill \\ 1\hfill & \text{if}x\ge 1.\hfill \end{array}1$$

To obtain the linearized variable of the QSR, we must first calculate the linearized variable of the partial sum (2.2), which is

$$I{\left(Y_{\alpha }\right)}_k=y_{k}H\left(\alpha N-k+1\right)+\left[\alpha -1_{\left[y_k<Q_{\alpha }\right]}\right]Q_{\alpha }\mathrm{,}$$

where $Q_{\alpha }=y_i,$  with ${\widehat{N}}_{i-1}<\alpha \widehat{N}\le {\widehat{N}}_i,$  corresponds to the first definition of the quantile of a finite population in the article by Hyndman and Fan (1996). Osier (2009) obtains a linearized variable that is dependent on the density of the variable  $\mathcal{Y}.$ However, Langel and Tillé (2011) have shown that the problem of estimating this density for the QSR can be avoided through a simplification, so that it is not necessary to make a kernel approximation of income density as proposed by Osier (2009).

The influence function of the QSR is dependent on the influence functions of the partial sums:

$$I{\left(\text{QSR}\right)}_k=z_k^{\text{QSR}}=\frac{y_k-I\left(Y_{\text{0}\text{.8}}\right)}{Y_{\text{0}\text{.2}}}-\frac{\left(Y-Y_{\text{0}\text{.8}}\right)I\left(Y_{\text{0}\text{.2}}\right)}{Y_{\text{0}\text{.2}}^2}\mathrm{.}$$

By making the necessary substitutions, we can see that the estimated linearized variable for a sample is

$$\begin{array}{ccccc}{\widehat{z}}_k^{\text{QSR}}& =& y_k-\left\{y_{k}H\left(\frac{\text{0}\text{.8}\widehat{N}-{\widehat{N}}_{k-1}}{w_k}\right)+{\widehat{Q}}_{\text{0}\text{.8}}\left[\text{0}\text{.8}-1_{\left[y_k<{\widehat{Q}}_{\text{0}\text{.8}}\right]}\right]\right\}& & \left(2.3\right)\\ & -& \frac{\left(\widehat{Y}-{\widehat{Y}}_{\text{0}\text{.8}}\right)\left\{y_{k}H\left(\frac{\text{0}\text{.2}\widehat{N}-{\widehat{N}}_{k-1}}{w_k}\right)+{\widehat{Q}}_{\text{0}\text{.2}}\left[\text{0}\text{.2}-1_{\left[y_k<{\widehat{Q}}_{\text{0}\text{.2}}\right]}\right]\right\}}{{\widehat{Y}}_{\text{0}\text{.2}}^2}\mathrm{.}& & \end{array}$$

2.3 Linearized variable of a quantile

Before we discuss poverty indicators, we should give a few details on the linearized variable of an $\alpha$  -order quantile, which can be expressed as

$${\widehat{z}}_k^{Q_{\alpha }}=-\frac{1}{f\left({\widehat{Q}}_{\alpha }\right)}\frac{1}{\widehat{N}}\left[1_{\left[y_k\le {\widehat{Q}}_{\alpha }\right]}-\alpha \right]\mathrm{,}$$

where the weighted quantile can be defined in a manner similar to the partial sum (2.2), and $f(\cdot )$  is an income density function that will be discussed in details in Section 3. Note that Eurostat (2004a) recommends the second definition by Hyndman and Fan (1996). We could dispute the Eurostat definition and use another quantile definition, for example $Q_{\alpha }=y_{k-1}+\left(y_k-y_{k-1}\right)\left[\alpha N-\left(k-1\right)\right]$ , where $\alpha N<k\le \alpha N+1,$  which is the fourth definition according to Hyndman and Fan (1996). We then estimate the quantile for a sample as follows:

$${\widehat{Q}}_{\alpha }=y_{k-1}+\left(y_k-y_{k-1}\right)\left(\frac{\alpha \widehat{N}-{\widehat{N}}_{k-1}}{w_k}\right)\mathrm{.}$$

The linearized variable of a quantile is dependent on the value of the income density function in that quantile. However, the actual income density is unknown and therefore must also be estimated using the sample. Deville (2000) and Osier (2009) suggest the use of Gaussian kernel estimation. We will discuss the problem of estimating $f$  in more details in Section 3.

In addition to the problem of estimating the income density function, Croux (1998) shows that the empirical influence function of the median income is not a consistent estimator of the corresponding theoretical influence function. For a positive variable (such as income), the empirical influence function of the median income (the case discussed in Croux’s article) converges toward an exponential distribution, the expectation of which is the influence function. It is not robust to large proportions of extreme values. It can be said to lack robustness in that the value of the estimator for a sample can differ greatly from the actual value for the population as a result of outliers (that is, values that are relatively very large) in the sample (see Hampel (1974) for a basic idea of robustness for infinite populations, and Beaumont, Haziza and Ruiz-Gazen (2013) for recent thoughts on this topic for finite population sampling).

2.4 Median income and at-risk-of-poverty threshold

Let $\widehat{m}={\widehat{Q}}_{\text{0}\text{.5}}$  be the estimated median income of the sample. The At Risk of Poverty Threshold (ARPT) is defined as 60% of the median income:

$$\begin{array}{c}\text{ARPT}=\text{0}\text{.6}{F}^{-1}\left(\text{0}\text{.5}\right)\\ \widehat{\text{ARPT}}=\text{0}\text{.6}{\widehat{Q}}_{\text{0}\text{.5}}=\text{0}\text{.6}\widehat{m}\mathrm{.}\end{array}$$

This is an absolute measure that is scale-dependent. The linearized variable of the ARPT is proportional to that of the median income:

$${\widehat{z}}_k^{\text{ARPT}}=I{\left(\text{ARPT}\right)}_k=\text{0}\text{.6}I{\left(\text{MED}\right)}_k=-\frac{\text{0}\text{.6}}{f\left(\widehat{m}\right)}\frac{1}{\widehat{N}}\left[1_{\left[y_k\le \widehat{m}\right]}-\text{0}\text{.5}\right]\mathrm{.}$$

2.5 At Risk of Poverty Rate

The At Risk of Poverty Rate (ARPR), where $\text{ARPR}\in \left[0,1\right],$  is the share of the population with an income below the ARPT: $\text{ARPR}=F\left(\text{ARPT}\right).$  The ARPR is scale-independent, like the Gini coefficient, QSR and relative median poverty gap (see Section 2.7). The official Eurostat definition (Eurostat 2004a) of the estimated ARPR for a sample is

$$\widehat{\text{ARPR}}=\frac{{\displaystyle {\sum }_{y_k<\widehat{\text{ARPT}}}{w}_k}}{\widehat{N}}\mathrm{.}$$

The linearized variable of the ARPR is defined by Osier (2009) as

$$\begin{array}{lll}{\widehat{z}}_k^{\text{ARPR}}\hfill & =\hfill & \frac{1}{\widehat{N}}\left(1_{\left[y_k\le \widehat{\text{ARPT}}\right]}-\widehat{\text{ARPR}}\right)-\frac{f\left(\widehat{\text{ARPT}}\right)}{f\left(\widehat{m}\right)}\frac{\text{0}\text{.6}}{\widehat{N}}\left(1_{\left[y_k\le \widehat{m}\right]}-\text{0}\text{.5}\right)\hfill \\ \hfill & =\hfill & \frac{1}{\widehat{N}}\left(1_{\left[y_k\le \widehat{\text{ARPT}}\right]}-\widehat{\text{ARPR}}\right)+f\left(\widehat{\text{ARPT}}\right){\widehat{z}}_k^{\text{ARPT}}\mathrm{.}\hfill \end{array}$$

Here, the income density function must be estimated at two points, namely the median income and the ARPT.

2.6 Median income of individuals below the ARPT

The median income of individuals below the ARPT is $m_p=F^{-1}\left(1/2F\left(\text{ARPT}\right)\right).$  It is estimated in the same way as any other quantile, the exact definition of which may vary. The linearized variable of $m_p$  (Osier 2009) is dependent on that of the ARPR:

$${\widehat{z}}_k^{m_p}=\frac{1}{f\left({\widehat{m}}_p\right)}\frac{{\widehat{z}}_k^{\text{ARPR}}}{2}-\frac{1}{\widehat{N}}\left(1_{\left[y_k\le {\widehat{m}}_p\right]}-F\left({\widehat{m}}_p\right)\right)\mathrm{.}$$

The estimated income density therefore appears three times, namely in the median income and ARPT for  ${\widehat{z}}_k^{\text{ARPR}}$ , and in the median income of individuals below the ARPT, $m_p.$

2.7 Relative Median Poverty Gap

The relative median poverty gap (RMPG) is the relative difference between the ARPT and the median income of individuals below the ARPT. $RMPG=0$  if the income of all “poor” individuals is equal to the ARPT, and $\text{RMPG}=1$  if the income of all these individuals is zero. The RMPG is a measure of the extent to which the “poor” individuals are poor:

$$\text{RMPG}=\frac{\text{ARPT}-m_p}{\text{ARPT}}\mathrm{.}$$

The estimated RMPG for a sample has already been described. The influence of each observation on the RMPG is defined by Osier (2009):

$${\widehat{z}}_k^{\text{RMPG}}=\frac{{\widehat{m}}_p{\widehat{z}}_k^{\text{ARPT}}-\widehat{\text{ARPT}}{\widehat{z}}_k^{m_p}}{{\widehat{\text{ARPT}}}^2}\mathrm{.}$$

The estimated income distribution density appears four times: once in the calculation of  ${\widehat{z}}_k^{\text{ARPT}}$ and three times in the calculation of ${\widehat{z}}_k^{m_p}.$

Previous | Next