4 Bayes linear method for categorical data

Kelly Cristina M. Gonçalves, Fernando A. S. Moura et Helio S. Migon

Previous | Next

Often one may be interested in cases where the observed characteristic is whether or not the population unit possesses some attribute of interest. We can define a dichotomized variable $y_i=1,$ if the $i^{\text{th}}$ unit has that attribute, and refer to this as a success, and $y_i=0$ otherwise. For the binary case when the sample size is not large enough to rely on the Central Limit Theorem, the design-based approach could use the randomization introduced by the sampling design to justify the distribution of the binary random quantities. For instance, Cochran (1977), sections 3.4 and 3.5, shows how to apply hypergeometric and binomial distributions to obtain confidence intervals for population proportions when, respectively, simple random sampling with and without replacement designs are employed. On the other hand, model-dependent approaches have also been advanced and applied for predicting totals or means in the categories of interest. Malec, Sedransk, Moriarity and LeClere (1997) considered a logistic hierarchical model with two levels, where the clusters are the second one. They also compared the full hierarchical Bayes estimates with empirical Bayes estimates and standard methods. Moura and Migon (2002) presented a logistic hierarchical model approach for small area prediction of proportions, taking into account both possible spatial and unstructured heterogeneity effects. Nandram and Choi (2008) proposed a time-dependent multinomial-Dirichlet model to predict the results of an election under ignorable and non-ignorable non-response. They also used a Bayesian approach to allocate the undecided voters to the candidates.

Here again, we do not need to make any use of full model assumptions or a randomization approach, but we do need to make some assumptions about the first and the second moments of the random quantities involved. The BLE for binary data was briefly introduced by O'Hagan (1985), but here we develop it more generally for the case where we are interested in analyzing more than one attribute in a population. The purpose is to describe the estimation of the proportion of successes with categorical data. Let $y_{ij}$ be the variable that indicates that unit $i,$ $i=\mathrm{1,}\dots \mathrm{,}N$ is in category $j,$ $j=\mathrm{1,}\dots \mathrm{,}k$ given by

$$y_{ij}=\{\begin{array}{l}\mathrm{1,}\text{if}{i}^{\text{th}}\text{unit}\text{has}{j}^{\text{th}}\text{attribute}\mathrm{<mo>;</mo>}\hfill \\ \mathrm{0,}\text{otherwise}\mathrm{.}\hfill \end{array}$$

The main aim is to estimate a vector $p=\left(p_1\mathrm{,}\dots \mathrm{,}{p}_k\right)\prime ,$ where $p_j=N^{-1}{\displaystyle {\sum }_{i=1}^N}{y}_{ij},$ $j=\mathrm{1,}\dots \mathrm{,}k,$ is the proportion of units in category $j,$ given $y_s,$ a vector of dimension $nk,$ defined as $y_s=\left(y_{11}\mathrm{,}{y}_{21}\mathrm{,}\dots \mathrm{,}{y}_{n1}\mathrm{,}\dots \mathrm{,}{y}_{1k}\mathrm{,}{y}_{2k}\mathrm{,}\dots \mathrm{,}{y}_{nk}\right)\prime .$ As we are dealing with situations in which for each unit it is only possible to associate a unique attribute, we have ${\displaystyle {\sum }_{j=1}^k}{p}_j=1.$ Thus, we only need to estimate $k-1$ parameters, since it follows that ${\widehat{p}}_k=1-{\displaystyle {\sum }_{j=1}^{k-1}}{\widehat{p}}_j$ and the variance estimate is also analogously obtained by $\widehat{V}\left({\widehat{p}}_k\right)={\displaystyle {\sum }_{j=1}^{k-1}}\widehat{V}\left({\widehat{p}}_j\right)+{\displaystyle {\sum }_{l\ne j=1}^{k-1}}\widehat{C}ov\left({\widehat{p}}_j,{\widehat{p}}_l\right).$

In the absence of any other structural information, we suppose that the units in any given category are second-order exchangeable, but we do not assume any exchangeability between units of different categories. Our prior beliefs are expressed for $i=\mathrm{1,}\dots \mathrm{,}N,$ $j=\mathrm{1,}\dots \mathrm{,}k-\mathrm{1,}$ as follows:

$$\begin{array}{cc}\hfill m_j& =E\left(y_{ij}\right)=P\left(y_{ij}=1\right)\mathrm{,}{v}_j=V\left(y_{ij}\right)=m_j\left(1-m_j\right)\text{and}\hfill \\ \hfill \text{cov}\left(y_{ij}\mathrm{,}{y}_{i^{\prime }j}\right)& =P\left(y_{i^{\prime }j}=1|y_{ij}=1\right)P\left(y_{ij}=1\right)-P\left(y_{ij}=1\right)P\left(y_{i^{\prime }j}=1\right)\hfill \\ & =m_j\left(m_{jj}-m_j\right)=c_j\mathrm{,}\forall i\ne i^{\prime }\text{and}{\sigma }_j^2=v_j-c_j=m_j\left(1-m_{jj}\right)\mathrm{,}\hfill \end{array}$$

where $m_{jj}=P\left(y_{i^{\prime }j}=1|y_{ij}=1\right),$ for all $i\ne i^{\prime }.$

For $j\ne j^{\prime },$ we analogously obtain the covariance between these categories as

$$\text{cov}\left(y_{ij}\mathrm{,}{y}_{i^{\prime }{j}^{\prime }}\right)=\{\begin{array}{ll}{m}_j\left(m_{j^{\prime }j}-m_{j^{\prime }}\right)\mathrm{,}\hfill & \text{if}i\ne i^{\prime }\mathrm{,}\hfill \\ -m_j{m}_{j^{\prime }}\mathrm{,}\hfill & \text{if}i=i^{\prime }\mathrm{.}\hfill \end{array}$$

Often, we do not have all the data $y_s,$ but only a sufficient statistics, such as the sample proportion for each category, ${\overline{y}}_s.$ Let ${\overline{y}}_s$ be the $k-1­$ -vector whose $j^{\text{th}}$ position is given by the sample mean for category $j.$ Using the general model in (2.4), we obtain:

$$E\left({\overline{y}}_s\right)=E\left(E\left({\overline{y}}_s|\beta \right)\right)=a\text{and}\text{Var}\left({\overline{y}}_s\right)=E\left(V\left({\overline{y}}_s|\beta \right)\right)+V\left(E\left({\overline{y}}_s|\beta \right)\right)=V_s+R\mathrm{.}$$

Applying the general model in (2.4), where: the responde variable is given by ${\overline{y}}_s;$ the vector $\beta$ has dimension $k-1;$ $X_s=I_s$ and $V=\text{diag}\left(V_{\overline{s}}\mathrm{,}{V}_s\right),$ we obtain from (2.10):

$$\widehat{p}=\frac{n{\overline{y}}_s+\left(N-n\right)\widehat{\beta }}{N}\text{and}\widehat{V}\left(\widehat{p}\right)=\frac{{\left(N-n\right)}^2\left[V_{\overline{s}}+C\right]}{N^2}\mathrm{,}\text{ (4}\text{.1)}$$

where $C^{-1}=R^{-1}+V_s$ and $\widehat{\beta }=C\left(V_s^{-1}{\overline{y}}_s+R^{-1}a\right),$ as stated in (2.6).

Let $Q=V_s+R.$ The BLE of $p$ and its associate variance given in (4.1) can be written in terms of the prior quantities $m_j,$ $m_{j{j}^{\prime }}$ and $j=\mathrm{1,}\dots \mathrm{,}k-1$ by noting that $a=\left(m_1\mathrm{,}\dots \mathrm{,}{m}_{k-1}\right)\prime ,$ $Q_{jj}=c_j+{\sigma }_j^2/n$ and $Q_{j{j}^{\prime }}=m_j\left(m_{j^{\prime }j}-m_{j^{\prime }}\right)-m_j{m}_{j^{\prime }j}/n.$ Therefore, the matrix $R=\left\{r_{j{j}^{\prime }}\right\}\mathrm{,}j\mathrm{,}{j}^{\prime }=\mathrm{1,..,}k-1$ with $r_{jj}=c_j$ and $r_{j{j}^{\prime }}=m_j\left(m_{j^{\prime }j}-m_{j^{\prime }}\right)$ and $V_s=1/n\left\{v_{j{j}^{\prime }}\right\}\mathrm{,}j\mathrm{,}{j}^{\prime }=\mathrm{1,..,}k-1$ with $v_{jj}={\sigma }_j^2$ and $v_{j{j}^{\prime }}=-m_j{m}_{j^{\prime }j}.$ Analogously, we get $V_{\overline{s}}=n/\left(N-n\right)V_s.$

4.1  Prior elicitation

Elicitation is the process of formulating a person's knowledge and beliefs about one or more uncertain quantities into a probability distribution for those quantities. According to Garthwaite, Kadane and O'Hagan (2005), it is convenient to think of the elicitation task as involving a facilitator, who helps the expert formulate the expert's knowledge in probabilistic form. In the context of eliciting a prior distribution for a Bayesian analysis, it is the expert's prior knowledge that is being elicited, but in general the objective is to express the expert's current knowledge in probabilistic form. If the expert is a statistician, or is very familiar with statistical concepts, then there may be no formal need for a facilitator, but this is rare in practice. O'Hagan (1998) illustrated with a practical example how to elicitate first and second moments. In particular, he adopted the Bayes linear approach because it makes easy the application of the elicitation procedure by engineers.

In this section, some restrictions about the prior quantities and an alternative to facilitate the process of elicitation are presented to obtain the BLE for categorical data. Because $m_j$ and $m_{j{j}^{\prime }}$ are probabilities, and $R$ and $V_s$ are the covariance matrices in model (2.4), the following restrictions must be satisfied:

1. $0<m_j<1$ and $0\le m_{j{j}^{\prime }}\le 1,$ $j\mathrm{,}{j}^{\prime }=\mathrm{1,}\dots \mathrm{,}k-1;$

2. $R$ and $V_s$ are positive-definite symmetric matrices.

In order to verify if condition (2.2) is satisfied, the following steps may be carried out:

i. verify if $R$ and $V_s$ are symmetric by checking if $m_j{m}_{j{j}^{\prime }}=m_{j^{\prime }}{m}_{j^{\prime }j};$

ii. verify if $R$ and $V_s$ are positive-definite matrices by finding the eigenvalues of $R$ and $V_s.$ If the eigenvalues are positive, then the matrices are positive-definite.

It should be noted that the eigenvalues are the roots of the characteristic polynomial and if this polynomial is of degree $n,n\le 4,$ it is possible to analytically get its roots by using Bhaskara, Cardan or Ferrari; see Jacobson (2009), chapter 4, for formulas. However, if $n\ge 5,$ we usually need to apply an iterative method to get them. Nevertheless, for matrices higher than $2\times 2,$ it is not trivial to analytically obtain these restrictions based on eigenvalues. The next proposition presents the conditions that $m_j$ and $m_{j{j}^{\prime }},$ $j=\mathrm{1,}\dots \mathrm{,}k-1,$ must satisfy in order to obtain a suitable prior for a multinomial model with three categories using the Bayesian linear estimation approach.

Proposition 1 Suppose that we elicit $m_j,$ such that $0<m_j<1,$ $j=\mathrm{1,2.}$ Then, given ${\rho }_{11},$ ${\rho }_{12}$ and ${\rho }_{22},$ we obtain $m_{11},$ $m_{12},$ $m_{21}$ and $m_{22}$ by (4.2). The prior quantities $m_j$ and $m_{j{j}^{\prime }},$ for $j\mathrm{,}{j}^{\prime }=\mathrm{1,2}$ must satisfy the following constraints for the matrices $R$ and $V_s$ to be positive-definite:

$$\begin{array}{l}{m}_{11}>m_1\text{and}{m}_{22}>m_2\mathrm{,}{m}_{11}{m}_{22}-m_{11}-m_{22}+1>m_{12}{m}_{21}\text{and}\\ m_{11}{m}_{22}-m_{11}{m}_2-m_1{m}_{22}>m_{12}{m}_{21}-2m_2{m}_{12}\mathrm{.}\end{array}$$

The verification of the Proposition 1 requires some algebra. We check that the matrices $R$ and $V_s$ are positive-definite using (i) and (ii) above. We use the fact that the eigenvalues of a matrix with dimension $2\times 2$ are positive if and only if its determinant is positive and then we obtain $m_{j{j}^{\prime }},$ $j\mathrm{,}{j}^{\prime }=\mathrm{1,2}$ which satisfies this restriction for both matrices. For cases with more than three categories we must numerically verify if the matrices $R$ and $V_s$ are positive-definite when replacing the numerical values of $m_j$ and $m_{j{j}^{\prime }},$ $j=\mathrm{1,}\dots \mathrm{,}k-1$ into them.

On the other hand, if an expert has some difficulty in specifying some of these conditional probabilities $m_{j{j}^{\prime }},$ it may be simpler to assign a prior to the coefficient of correlation. Define ${\rho }_{j{j}^{\prime }}$ as the prior of the coefficient of correlation between two different units within categories $j$ and $j^{\prime },$ that is:

$${\rho }_{j{j}^{\prime }}=\text{corr}\left(y_{ij}\mathrm{,}{y}_{i^{\prime }{j}^{\prime }}\right)=\{\begin{array}{ll}\frac{m_{jj}-m_j}{1-m_j}\mathrm{,}\hfill & j=j^{\prime }\mathrm{,}\hfill \\ \frac{m_j\left(m_{j^{\prime }j}-m_{j^{\prime }}\right)}{\sqrt{m_j\left(1-m_j\right)m_{j^{\prime }}\left(1-m_{j^{\prime }}\right)}}\mathrm{,}\hfill & j\ne j^{\prime }\mathrm{,}\hfill \end{array}$$

for $i\mathrm{,}{i}^{\prime }=\mathrm{1,}\dots \mathrm{,}n,$ $i\ne i^{\prime },$ $j\mathrm{,}{j}^{\prime }=\mathrm{1,}\dots \mathrm{,}k-1.$

Therefore, given ${\rho }_{j{j}^{\prime }},$ $j\mathrm{,}{j}^{\prime }=\mathrm{1,}\dots \mathrm{,}k-1,$ we get

$$m_{j{j}^{\prime }}=\{\begin{array}{ll}{m}_j+{\rho }_{jj}\left(1-m_j\right)\hfill & j=j^{\prime }\mathrm{,}\hfill \\ \frac{m_j{m}_{j^{\prime }}+{\rho }_{j^{\prime }j}\sqrt{m_j\left(1-m_j\right)m_{j^{\prime }}\left(1-m_{j^{\prime }}\right)}}{m_{j^{\prime }}}\mathrm{,}\hfill & j\ne j^{\prime }\mathrm{.}\hfill \end{array}\text{ (4}\text{.2)}$$

It should be noted that if there is some past data obtained from a previous survey, it is possible for an expert to use this information. For instance, $m_j$ can be obtained by estimating the proportion of units in category $j,$ $j=\mathrm{1,}\dots \mathrm{,}k-1$ from the previous survey. Analogously, ${\rho }_{j{j}^{\prime }}$ can be obtained using previous survey data. As stated in restriction (2.1), $m_j$ cannot assume the values 0 and 1, otherwise the correlations would not be defined.

4.2  Prior sensitivity analysis

It is worth checking how the estimator and its associated variance depend on the priors assigned. We deal with the simple case of only two categories. It should be noted that in the case with more than 2 categories the number of prior quantities to be elicited increases fast, but the conclusions obtained under this illustration can be extended. On the other hand, if there is no prior information available we can use non-informative priors and, as described in Section 2.2, the estimators from the design-based approach are recovered.

The BLE for proportion for binary data can be obtained as a particular case of the estimator in (4.1),

$${\widehat{p}}_1=\frac{n{\overline{y}}_1+\left(N-n\right)\widehat{\mu }}{N}\mathrm{,}$$

where

$$\begin{array}{l}\widehat{\mu }=\omega {\overline{y}}_1+\left(1-\omega \right)m_1\text{is}\text{the}\text{expected}\text{value}\text{of}\text{the}\text{non-observed}\text{values}\text{in}\text{category}1\mathrm{,}\\ \omega =\frac{n{\sigma }_1^{-2}}{n{\sigma }_1{}^{-2}+c_1{}^{-1}}\mathrm{,}\end{array}$$

and ${\widehat{p}}_2=1-{\widehat{p}}_1.$ Note that ${\sigma }_1^2$ and $c_1$ depend on $m_{11}=m_1+{\rho }_{11}\left(1-m_1\right),$ see page 13. We analyze how the estimates are affected by ${\rho }_{11}.$

1. If ${\rho }_{11}\to 0,$ then $\omega \to 0$ and $\widehat{\mu }\to m_1.$ Thus, the estimator for the non-observed values largely depend on the value of the prior.

2. If ${\rho }_{11}\to 1,$ then $\omega \to 1$ and $\widehat{\mu }\to {\overline{y}}_1.$ Thus, the estimator for the non-observed values does not depend on the value of the prior.

Moreover, it is trivial to see that if $n/N\to 1,$ ${\widehat{p}}_1\to {\overline{y}}_1.$ To illustrate these results, we created some artificial dataset by fixing the true proportion at $p=\left(\mathrm{0.2380,0.7620}\right)\prime$ and the sample mean at ${\overline{y}}_s=\left(\mathrm{0.2614,0.7386}\right)\prime .$ These values were taken from Moura and Migon (2002). Then, we assessed how the values of $m_1,$ $N,$ $f=n/N$ and ${\rho }_{11}$ affect the estimator ${\widehat{p}}_1.$ Figure 4.1 presents the two-dimensional plots of the absolute error of ${\widehat{p}}_1$ versus ${\rho }_{11}$ for some particular cases. The grey line represents the absolute error between the sample proportion ${\overline{y}}_1$ and the true $p_1.$

It should be noted that, as $f$ or $N$ increases, the absolute error decreases for any prior values. Moreover, when ${\rho }_{11}\to 0$ the absolute error increases when $m_1$ considerably differs from the true proportion $p_1,$ but it decreases as the sample size increases. Finally, as ${\rho }_{11}\to 1$ we observe that the absolute error of ${\widehat{p}}_1$ tends to the absolute error of the sample proportion ${\overline{y}}_1$ . Thus, if we have good prior information, in terms of $m_1,$ the estimator proposed performs well for all the values of ${\rho }_{11}.$ But, if there is no prior information available, non-informative priors characterized by ${\rho }_{11}\to 1$ can be used and we obtain results similar to a design-based approach.

Figure 4.1 Two-dimensional plots of the absolute error of ${\widehat{p}}_1$ versus ${\rho }_{11}$ for some particular cases

Description for figure 4.1

Note: Absolute error for fixed $m_1\in \left\{\mathrm{0.1,0.4,0.7,0.9}\right\},$ $N\in \left\{\text{1,500};\text{15,288}\right\}$ and $f\in \left\{1\text{\%}\mathrm{,10}\text{\%}\right\}$ and varying ${\rho }_{11}\in \left\{\mathrm{0.01,0.25,0.5,0.75,0.9}\right\}.$ The grey line represents the absolute error of the sample proportion ${\overline{y}}_1$

Previous | Next