3 Pseudo-likelihood-based selection with BIC

Chen Xu, Jiahua Chen and Harold Mantel

Previous | Next

3.1 BIC in surveys

With the model settings described in Section 2, it is clear that, if the measurement $\left(y_i\mathrm{,}{x}_i\right)$ is observed for every unit in population $D,$ the randomness in the data introduced by the probability sampling design is completely gone. In this situation, the selection of the influential variables is based on the entire population and the classical selection criteria developed in non-survey settings (purely model-based) remain valid for model-design-based inference. In particular, let $s\subseteq \left\{\mathrm{1,}\dots \mathrm{,}p\right\}$ be an arbitrary set of $\tau \left(s\right)$ covariates, which corresponds to a candidate model in form of (2.1). The "census-based� BIC (Schwarz 1978) selects the model (covariates) that minimizes

$${\text{BIC}}_N\left(s\right)=-2l_N\left({\stackrel{⌣}{\beta }}_s\right)+\tau \left(s\right)\log N\mathrm{,}\left(3.1\right)$$

where $l_N\left(\beta \right)={\displaystyle {\sum }_{i=1}^N}\log f\left(y_i;x_i\beta \right)$ is the census log-likelihood function and ${\stackrel{⌣}{\beta }}_s$ is the maximizer of $l_N\left(\beta \right)$ based on $s.$ It can be seen that the BIC (3.1) is a decreasing function of the maximized log-likelihood and an increasing function of the number of variables included in the model. Hence, a lower BIC implies either a simpler model (fewer explanatory variables), a better fit (higher maximized likelihood), or both. A model with balanced complexity and goodness of fit is preferred.

We note that the census BIC (3.1) is conceptual, because observing $\left(y_i\mathrm{,}{x}_i\right)$ for all units in $D$ is usually not feasible in applications. Instead, a representative sample $d=\left\{i_1\mathrm{,}\dots \mathrm{,}{i}_n\right\}\subset \left\{\mathrm{1,}\dots \mathrm{,}N\right\}$ with $n$ units is often drawn from $D$ and the measurements are observed based on the sampled units. Due to the intrinsic dependence structure among the sampled units, a full likelihood on $d$ is prohibitive to compute in general. Alternatively, for the model-design-based inference, a pseudo-log-likelihood function is frequently used, which takes the form

$$l_n\left(\beta \right)={\displaystyle \sum _{i\in d}{w}_i\log f\left(y_i;\beta \right)}\left(3.2\right)$$

with $w_i=k/P\left(i\in d\right)$ denoting the survey weight for the $i^{\text{th}}$ unit. The scaling parameter $k$ in $w_i$ does not have analytical impacts on the pseudo-likelihood-based inference. For the simplicity of presentation, we choose $k=n/N$ such that $n^{-1}{l}_n\left(\beta \right)$ is design-unbiased to $N^{-1}{l}_N\left(\beta \right).$ Maximizing $l_n\left(\beta \right)$ over $\beta$ leads to a maximum pseudo-likelihood estimator (MPLE) $\widehat{\beta }$ for $\beta ,$ i.e.,

$$\widehat{\beta }=\text{arg}\underset{\beta }{\max }{l}_n\left(\beta \right)\mathrm{.}$$

Under the appropriate sampling designs, $\widehat{\beta }$ is often $n^{-1/2}$ consistent for $\beta$ under the joint randomization framework. The idea of using pseudo-likelihood for inference on model parameters has been widely adopted in the literature (see, e.g., Binder 1983; Godambe and Thompson 1986; Molina and Skinner 1992).

In this paper, we aim to develop an analogue of BIC criterion based on the pseudo-likelihood. Following the super-population formulation described in Section 2, let ${\beta }_s$ be the $\tau \left(s\right)­$dimensional coefficient of model $s$ and let ${\nu }_s$ be the prior density of ${\beta }_s.$ Then a pseudo-marginal density function of the data is given by

$$P_n\left(y|s\right)={\displaystyle \int }{L}_n\left(y;{\beta }_s\right){\nu }_s\left({\beta }_s\right)d{\beta }_s$$

with $L_n\left(y;{\beta }_s\right)=\exp \left\{l_n\left(y;{\beta }_s\right)\right\}.$ Consequently, we may regard the following expression as the pseudo-posterior probability of the model $s:$

$$P_n\left(s|y\right)=\frac{P_n\left(y|s\right)P\left(s\right)}{{\displaystyle \sum _{s\in S}P\left(s\right)P_n\left(y|s\right)}}\mathrm{,}\left(3.3\right)$$

where $S$ denotes the collection of all candidate models. In the spirit of Bayesian analysis, the model with the highest $P_n\left(s|y\right)$ is then considered to be the one that receives the most support from the data. Since ${\displaystyle {\sum }_{s\in S}P\left(s\right)P_n\left(y|s\right)}$ does not depend on any specific model, the highest $P_n\left(s|y\right)$ is achieved by the model that maximizes the corresponding $P_n\left(y|s\right)P\left(s\right)$. When the uniform prior $P\left(s\right)=\zeta$ is used and the weight scaling is chosen as $k=n/N,$ we obtain a Laplace approximation under some regularity conditions (see Xu and Chen 2012):

$$-2\log \left\{P_n\left(y|s\right)\right\}=-2l_n\left({\widehat{\beta }}_s\right)+\tau \left(s\right)\log n+O_p\left(1\right)\mathrm{.}$$

Accordingly, we choose the model $s$ that minimizes

$${\text{BIC}}_n\left(s\right)=-2l_n\left({\widehat{\beta }}_s\right)+\tau \left(s\right)\log n\mathrm{.}\left(3.4\right)$$

Compared with the census BIC (3.1), the first term in BIC (3.4) is the maximum survey-weighted pseudo-likelihood, which is potentially helpful to avoid sampling errors that might lead to biased inferences for the target population. We refer to (3.4) as a pseudo-likelihood-based version of BIC in the context of surveys. In the joint randomization framework, we establish the selection consistency of using BIC (3.4) through a PPL-based implementation procedure, as will be seen in Section 4.

3.2 Implementing BIC via penalized pseudo-likelihood

In applications, a straightforward way to implement BIC is best-subset selection, where BIC is evaluated and compared for each candidate model. However, this procedure can be computationally impractical when the number of covariates is large. Alternatively, penalized likelihood methods have recently been used as computationally efficient procedures for implementing a selection criterion. These methods exclude variables from the model by estimating their coefficients to be zero, and shrink the other coefficients accordingly. By varying the penalty on the likelihood, we can obtain a series of models with differing sparsity. To avoid an exhaustive search of the entire model space, the selection criterion is used to pick an optimal one among these sparse models. The effectiveness of this implementation strategy has been illustrated in the non-survey context for BIC (Wang, Li and Tsai 2007; Liu, Wang and Liang 2011) and GCV (Fan and Li 2001; Xie, Pan and Shen 2008) among others.

Sharing the same spirit, we proposed a penalized pseudo-likelihood (PPL) procedure for the implementation of BIC (3.4) for survey data. Specifically, following pseudo-likelihood (3.2) with $k=n/N,$ we define the survey-weighted penalized estimator ${\widehat{\beta }}_{\lambda }$ that maximizes the penalized pseudo-likelihood function

$$Q_n\left(\beta \right)=l_n\left(\beta \right)-n{\displaystyle \sum _{j=1}^p{\varphi }_{\lambda }}\left(\left|{\beta }_j\right|\right)\mathrm{,}\left(3.5\right)$$

where ${\varphi }_{\lambda }(\cdot )$ is a penalty function indexed by a tuning parameter $\lambda$ controlling the size of the penalty. With an appropriate choice of ${\varphi }_{\lambda }(\cdot ),{\widehat{\beta }}_{\lambda }$ contains zero estimates for some coefficients and thus automatically produces a sparse model. The desirable sparsity of ${\widehat{\beta }}_{\lambda }$ typically requires the singularity of the corresponding ${\varphi }_{\lambda }(\cdot )$ at the origin. Some popular choices of ${\varphi }_{\lambda }(\cdot )$ include the $L_{\gamma }$ penalty (Frank and Friedman 1993; Tibshirani 1996), i.e., ${\varphi }_{\lambda }\left(\left|\beta \right|\right)=\lambda {\left|\beta \right|}^{\gamma }$ with $\gamma \in \left(\mathrm{0,1}\right],$ and the SCAD penalty (Fan and Li 2001), which is defined by the following derivative:

$${{\varphi }^{\prime }}_{\lambda }\left(\left|\beta \right|\right)=\lambda \left\{I\left(\left|\beta \right|\le \lambda \right)+\frac{{\left(a\lambda -\left|\beta \right|\right)}_{+}}{\left(a-1\right)\lambda }I\left(\left|\beta \right|>\lambda \right)\right\}\left(3.6\right)$$

with $a=$ 3.7 being a common choice.

With different values of $\lambda$ for a properly specified ${\varphi }_{\lambda }(\cdot ),{\widehat{\beta }}_{\lambda }$ leads to models of differing sparsity. These sparse models (with respect to $\lambda$ ) naturally form a collection of candidate models. BIC (3.4) can then be used to select an optimal model within this collection. To be more specific, let $\Omega$ be the range of $\lambda$ and let $s_{\lambda }$ denote the model produced by ${\widehat{\beta }}_{\lambda }.$ We treat $S_{\Omega }=\left\{s_{\lambda }:\lambda \in \Omega \right\}$ as the collection of candidate models under consideration, and select the model $s^{*}\in S_{\Omega }$ such that ${\text{BIC}}_n\left(s^{*}\right)={\min }_{\lambda \in \Omega }\text{BIC}\left(s_{\lambda }\right).$ We refer to this selection procedure as the penalized pseudo-likelihood-based BIC method (PPL-BIC). Compared with traditional best-subset selection, the PPL-BIC procedure focuses on the models that are produced by the survey-weighted penalized estimators, and therefore it can be much less computationally expensive.

Previous | Next