Model-assisted calibration of non-probability sample survey data using adaptive LASSO
Section 3. Model selection and robust calibration using adaptive LASSO

• Previous
• Next

3.1  Adaptive LASSO background

3.1.1  Definition and parameters

The adaptive LASSO regression coefficients are obtained by solving a penalized regression equation. For linear adaptive LASSO regression (Zou, 2006):

$$\widehat{\beta }=\underset{\beta }{\text{argmin}}\left({\displaystyle \sum _{i\in s_A}}{\left(y_i-x_i^T\beta \right)}^2+{\lambda }_n{\displaystyle \sum _{j\mathrm{<mo>=</mo>\; 1}}^p}{\alpha }_j^{\gamma }\left|{\beta }_j\right|\right)(3.1)$$

where ${\alpha }_j^{\gamma }$ is an adjustable weight and ${\lambda }_n$ is a penalty used to optimize a model fit measure. Similarly for logistic adaptive LASSO:

$$\widehat{\beta }\mathrm{<mo>=</mo>}\underset{\beta }{\text{argmin}}\left({\displaystyle \sum _{i\in s_A}}\left[-y_i\left(x_i^T\beta \right)+\text{log}\left(1+\text{exp}\left(x_i^T\beta \right)\right)\right]+{\lambda }_n{\displaystyle \sum _{j\mathrm{<mo>=</mo>\; 1}}^p}{\alpha }_j^{\gamma }\left|{\beta }_j\right|\right).(3.2)$$

Given ${\lambda }_n$ and $\gamma ,$ we can calculate $\widehat{\beta }$ through iterative procedures. The R package $glmnet$ will compute both the linear and logistic adaptive LASSO (Friedman, Hastie and Tibshirani, 2010).

The role of the weight parameter, ${\alpha }_j,$ is to prevent LASSO from selecting covariates with large effect sizes in favor of lowering prediction error when the sample size is small. Thus the weights are inversely proportional to effect sizes of regression parameters: ${\alpha }_j\propto 1/\left|{\beta }_j\right|.$ A common choice of ${\alpha }_j$ is $1/\left|{\widehat{\beta }}_j^{\text{MLE}}\right|,$ where ${\widehat{\beta }}_j^{\text{MLE}}$ is the maximum likelihood estimate of ${\beta }_j.$ The power of the weight parameter, $\gamma ,$ is a constant greater than 0 that interacts with ${\alpha }_j$ to control LASSO from selecting or excluding parameters. For example, if we still want LASSO to favor large effect covariates when the sample size is small, we should set $\gamma$ small. If we want to de-emphasize effect sizes further, we should set $\gamma$ large.

3.1.2  Oracle property

An important concept in measuring the performance of a model selection and estimation method is called the “oracle property”. The optimal method selects the correct variables and provides unbiased estimates of selected parameters. Suppose the parameters in a full regression model have both zero and non-zero components. Without loss of generality, let the first $p$ be non-zero and the last $q$ zero:

$${\beta }^F\mathrm{<mo>=</mo>}\left(\begin{array}{c}{\beta }_{\left(p\times 1\right)}^{\left(1\right)}\\ {\beta }_{\left(q\times 1\right)}^{\left(2\right)}=0\end{array}\right).$$

A regression model has the oracle property if it satisfies the following conditions (Fan and Li, 2001):

• The probability of estimating 0 for zero-valued parameters tends to one: $\text{Pr}\left({\widehat{\beta }}^{\left(2\right)}=0\right)\to 1$ as $n\to \infty .$
• The estimates of non-zero parameters are as good as if the true sub-model is known: $\sqrt{n}\left({\widehat{\beta }}^{\left(1\right)}-{\beta }^{\left(1\right)}\right)\to N\left(0\mathrm{,}C\right)$ where $C\mathrm{<mo>=</mo>}\Sigma \left({\beta }^{\left(1\right)}\right)$ is the covariance matrix of ${\beta }^{\left(1\right)}$ under linear model, and $C\mathrm{<mo>=</mo>}{I}^{-1}\left({\beta }^{\left(1\right)}\right)$ is the inverse of the Fisher information matrix of ${\beta }^{\left(1\right)}$ under the generalized linear model.

For finite-population inference, suppose $\nu$ indexes a population with size $N_{\nu },$ let $B_{\nu }$ be the quasilikelihood estimates of $\beta$ in population $\nu ,$ and ${\widehat{B}}_{\nu }$ is the estimate of $B_{\nu }$ based on a sample with size $n_{\nu }\le N_{\nu }.$ We assume that $N_{\nu }\to \infty ,$ $n_{\nu }\to \infty ,$ and $n_{\nu }/N_{\nu }\to 0$ as $\nu \to \infty .$ The finite-population equivalent of the oracle property is then:

$$\begin{array}{ll}\text{Pr}\left({\widehat{B}}_{\nu }^{\left(2\right)}\mathrm{<mo>=</mo>}0\right)\hfill & \to 1\hfill \\ \sqrt{n_{\nu }}\left({\widehat{B}}_{\nu }^{\left(1\right)}-B_{\nu }^{\left(1\right)}\right)\hfill & \to N_{\nu }\left(0\mathrm{,}{C}_{\nu }\right)\hfill \\ B_{\nu }\hfill & \to \beta \hfill \\ \text{as}\hfill & \nu \to \infty \hfill \end{array}$$

where $C_{\nu }\mathrm{<mo>=</mo>}\Sigma \left(B_{\nu }^{\left(1\right)}\right)$ is the covariance matrix of $B_{\nu }^{\left(1\right)}$ if the model is linear, and $C_{\nu }\mathrm{<mo>=</mo>}{I}^{-1}\left(B_{\nu }^{\left(1\right)}\right)$ is the inverse of Fisher information matrix of $B_{\nu }^{\left(1\right)}$ under the generalized linear model.

Zou (2006) has shown that if ${\lambda }_n/\left(\sqrt{n}/{\left(\sqrt{n}\right)}^{\gamma }\right)\to \infty$ and ${\lambda }_n/\sqrt{n}\to 0,$ then the adaptive LASSO satisfies the oracle property. The conditions require that ${\lambda }_n$ grow at least at the rate of $\sqrt{n}/{\left(\sqrt{n}\right)}^{\gamma }\text{},$ but not faster than $\sqrt{n}.$ The choice of ${\lambda }_n$ and $\gamma ,$ and R code for implementing it, are discussed in the Appendix.

3.2  LASSO calibration

This section derives the analytical formula for a LASSO estimator of total, its model expectation, and estimators of the asymptotic design variance. We make the following assumptions:

1. The samples are drawn from a single-stage sample design $\mathcal{A},$ allowing for unequal probabilities of selection. The selection probability for unit $i$ is denoted by ${\pi }_i^A\text{},$ and the joint selection probability of units $i$ and $j$ is denoted by ${\pi }_{ij}^A\text{}.$ We denote the design weight for unit $i$ by $d_i^A\mathrm{<mo>=</mo>}1/{\pi }_i^A\text{},$ the vector of design weights by $d^A\text{},$ and the diagonal matrix of design weights by  $D^A\text{}.$
2. Population-level auxiliary data are known, denoted by $X\mathrm{<mo>=</mo>}\left(x_i^T\right)\mathrm{,}i\mathrm{<mo>=</mo>\; 1,}\dots ,N.$
3. A superpopulation model is assumed, as is described in Section 2.2:

$$\begin{array}{ll}{E}_{\xi }\left(y_i|x_i\right)\hfill & \mathrm{<mo>=</mo>}\mu \left(x_i\mathrm{,}\beta \right)\hfill \\ V_{\xi }\left(y_i|x_i\right)\hfill & \mathrm{<mo>=</mo>}{\nu }_i^2{\sigma }^2.\hfill \end{array}$$

4. The true superpopulation parameters are a subset of the full regression model for LASSO:

$${\beta }^F\mathrm{<mo>=</mo>}\left(\begin{array}{c}{\beta }_{\left(p\times 1\right)}\\ {\beta }_{\left(q\times 1\right)}^{\left(2\right)}\end{array}\right).$$

5. The full-range of $X$ in the population has non-zero probability of being observed in the analytical sample.

3.2.1  Point estimate: ${\widehat{T}}_y^{\text{LASSO}}$

The LASSO calibration estimate of total can be obtained following the steps: 

1. Obtain LASSO regression coefficients $\widehat{B}$ as described in the Appendix.
2. Use $\widehat{B}$ to calculate ${\widehat{\mu }}_i\mathrm{<mo>=</mo>}\mu \left(x_i\mathrm{,}\widehat{B}\right)$ in the population.
3. Define $T^M\mathrm{<mo>=</mo>}\left(N\mathrm{,}{\displaystyle {\sum }_i^N}{\widehat{\mu }}_i\right)$ and $M\mathrm{<mo>=</mo>}\left[d^A\text{}\mathrm{,}{\displaystyle {\sum }_{i\in s_A}}{\widehat{\mu }}_i\right],$ under chi-square distance measure with $q_i\mathrm{<mo>=</mo>\; 1}:$

$$w^{\text{LASSO}}\mathrm{<mo>=</mo>}{d}^A+D^{A}M{\left(M^T{D}^{A}M\right)}^{-1}{\left(T^M-{\left(d^A\right)}^{T}M\right)}^T.(3.3)$$

4. Determine the LASSO calibration estimator of total:

$$\begin{array}{ll}{\widehat{T}}_y^{\text{LASSO}}\hfill & \mathrm{<mo>=</mo>}{\left(w^{\text{LASSO}}\right)}^T\text{}y\hfill \\ \hfill & \mathrm{<mo>=</mo>}{\left(d^A\right)}^T\text{}y+\left(T^M-{\left(d^A\right)}^T\text{}M\right){\left(X^T{D}^{A}X\right)}^{-1}\text{}{X}^T{D}^{A}y\hfill \\ \hfill & \mathrm{<mo>=</mo>}{\left(d^A\right)}^T\text{}y+\left({\displaystyle \sum _i^N}{\widehat{\mu }}_i-{\displaystyle \sum _{i\in s_A}}{d}_i^A{\widehat{\mu }}_i\right){\widehat{B}}^{\text{MC}}(3.4)\hfill \end{array}$$

• where ${\widehat{B}}^{\text{MC}}$ is the calibration slope to satisfy the calibration constraints:

$${\widehat{B}}^{\text{MC}}\mathrm{<mo>=</mo>}\frac{{\displaystyle {\sum }_{i\in s_A}{d}_i^A\left({\widehat{\mu }}_i-\widehat{\overline{\mu }}\right)\left(y_i-\overline{y}\right)}}{{\displaystyle {\sum }_{i\in s_A}{d}_i^A{\left({\widehat{\mu }}_i-\widehat{\overline{\mu }}\right)}^2}}\mathrm{,}\widehat{\overline{\mu }}\mathrm{<mo>=</mo>}{\displaystyle \sum _{i\in s_A}}{d}_i^A{\widehat{\mu }}_i/{\displaystyle \sum _{i\in s_A}}{d}_i^A\text{}\mathrm{,}\overline{y}\mathrm{<mo>=</mo>}{\displaystyle \sum _{i\in s_A}}{d}_i^A{y}_i/{\displaystyle \sum _{i\in s_A}}{d}_i^A\text{}\mathrm{.}$$

3.2.2  Asymptotic behavior of ${\widehat{T}}_y^{\text{LASSO}}$

Wu and Sitter (2001) established the conditions to derive an asymptotic model-assisted calibration estimator. We state the conditions here with slight modification in notations to be consistent with the current research. Let $\beta$ be the true superpopulation parameter for the model defined in equation (2.5), and $B$ be the finite-population quasilikelihood estimator of $\beta \text{.}$ The following conditions are used for deriving LASSO calibration asymptotic estimator of total:

1. $\widehat{B}\mathrm{<mo>=</mo>}B+O_p\left(1/\sqrt{n}\right),$ $B$ is the finite-population regression slope of $\beta \text{,}$ $B\to \beta \text{.}$
2. For each $x_i,$ $\partial \mu \left(x_i\mathrm{,}t\right)/\partial t$ is continuous in $t,$ and ${\text{max}}_i\left|\partial \mu \left(x_i\mathrm{,}t\right)/\partial t\right|\le h\left(x_i\mathrm{,}\beta \right)$ for $t$ in a neighborhood of $\beta ,$ and $N^{-1}{\displaystyle {\sum }_{i\in U}}h\left(x_i\mathrm{,}\beta \right)\mathrm{<mo>=</mo>}O\left(1\right).$
3. For each $x_i,$ ${\partial }^2\mu \left(x_i\mathrm{,}t\right)/\partial t\partial t^T$ is continuous in $t,$ and ${\text{max}}_{j\text{}\mathrm{,}k}\left|{\partial }^2\mu \left(x_i\mathrm{,}t\right)/\partial t_j\partial t_k\right|\le k\left(x_i,\beta \right)$ for $t$ in a neighborhood of $\beta \text{,}$ and $N^{-1}{\displaystyle {\sum }_{i\in U}}k\left(x_i\mathrm{,}\beta \right)\mathrm{<mo>=</mo>}O\left(1\right).$
4. The Horvitz-Thompson (HT) estimators of certain population means are asymptotically normally distributed (Fuller, 2009; pages 47-57).
5. ${\lambda }_n/\left(\sqrt{n}/{\left(\sqrt{n}\right)}^{\gamma }\right)\to \infty \text{and}{\lambda }_n/\sqrt{n}\to 0.$

Lemma 1: Assume that superpopulation model (2.5) holds. Let $B$ be the finite-population quasilikelihood estimate of $\beta \text{,}$ $B\to \beta \text{.}$ Under conditions (1)-(5), the model-assisted asymptotic estimator of population total is:

$${\widehat{T}}_y^{\text{MC}}\mathrm{<mo>=</mo>}{\displaystyle \sum _{i\in s_A}}{d}_i^A\left(y_i-{\mu }_i{B}^{\text{MC}}\right)+{\displaystyle \sum _{i\mathrm{<mo>=</mo>\; 1}}^N}{\mu }_i{B}^{\text{MC}}+o_p\left(\frac{N}{\sqrt{n}}\right)(3.5)$$

where

$$\begin{array}{ll}{\mu }_i\hfill & \mathrm{<mo>=</mo>}\mu \left(x_i\mathrm{,}B\right)\hfill \\ B^{\text{MC}}\hfill & \mathrm{<mo>=</mo>}\frac{{\displaystyle {\sum }_{i\mathrm{<mo>=</mo>\; 1}}^N\left({\mu }_i-\overline{\mu }\right)\left(y_i-\overline{y}\right)}}{{\displaystyle {\sum }_{i\mathrm{<mo>=</mo>\; 1}}^N{\left({\mu }_i-\overline{\mu }\right)}^2}}.\hfill \end{array}$$

Proof. See Appendix.

Given Lemma 1, we derive ${\widehat{T}}_y^{\text{LASSO}}$ the asymptotic LASSO estimator of total in Theorem 1. We show ${\widehat{T}}_y^{\text{LASSO}}$ is model unbiased for the population total in Theorem 2. Finally, Theorem 3 determines variance estimates for the LASSO calibration estimator of a total.

Theorem 1: Suppose the parameters in a full regression model have both zero and non-zero components. Without loss of generality, let the first $p$ be non-zero and the last $q$ be zero:

$${\beta }^F\mathrm{<mo>=</mo>}\left(\begin{array}{c}{\beta }_{\left(p\times 1\right)}^{\left(1\right)}\\ {\beta }_{\left(q\times 1\right)}^{\left(2\right)}\end{array}\right),{\beta }^{\left(1\right)}\mathrm{<mo>=</mo>}\beta \text{and}{\beta }^{\left(2\right)}\mathrm{<mo>=</mo>}{0}_{\left(q\times 1\right)},$$

under conditions (1)-(5), the asymptotic LASSO calibration estimator of total is:

$${\widehat{T}}_y^{\text{LASSO}}\mathrm{<mo>=</mo>}{\displaystyle {\sum }_{i\in s_A}}{d}_i^A\left(y_i-{\mu }_i{B}^{\text{MC}}\right)+{\displaystyle {\sum }_{i\mathrm{<mo>=</mo>\; 1}}^N}{\mu }_i{B}^{\text{MC}}+o_p\left(\frac{N}{\sqrt{n}}\right).$$

Proof. See Appendix.

Theorem 2: ${\widehat{T}}_y^{\text{LASSO}}$ is model-unbiased, that is $E_{\xi }\left({\widehat{T}}_y^{\text{LASSO}}\right)\mathrm{<mo>=</mo>}T.$

Proof. See Appendix.

Thus, as long as LASSO regression parameters include the superpopulation parameters, ${\widehat{T}}_y^{\text{LASSO}}$ is model-unbiased regardless of design weights. (Note that this is a quality that ${\widehat{T}}_y^{\text{GREG}}$ shares with ${\widehat{T}}_y^{\text{LASSO}}\text{}.$ However, ${\widehat{T}}_y^{\text{LASSO}}$ can assume models with much larger numbers of covariates than ${\widehat{T}}_y^{\text{GREG}}.)$ This property is essential in non-probability samples, where there are no initial design weights to guarantee unbiasedness.

Theorem 3: The estimator of the asymptotic variance of ${\widehat{T}}_y^{\text{LASSO}}$ is given by

$$\begin{array}{ll}{v}_{\mathcal{A}}\left({\widehat{T}}_y^{\text{LASSO}}\right)\hfill & \mathrm{<mo>=</mo>}{\displaystyle \sum _{i\in s_A}}{\left(\frac{y_i-{\widehat{\mu }}_i{\widehat{B}}^{\text{MC}}}{{\pi }_i}\right)}^2\left(1-{\pi }_i\right)\hfill \\ \hfill & +{\displaystyle \sum _{i\in s_A}}{\displaystyle \sum _{j\ne i}}\frac{{\pi }_{ij}-{\pi }_i{\pi }_j}{{\pi }_{ij}}\frac{\left(y_i-{\widehat{\mu }}_i{\widehat{B}}^{\text{MC}}\right)}{{\pi }_i}\frac{\left(y_j-{\widehat{\mu }}_j{\widehat{B}}^{\text{MC}}\right)}{{\pi }_j}.(3.6)\hfill \end{array}$$

Proof. The theoretical design variance of the LASSO estimator is

$$\begin{array}{ll}{V}_{\mathcal{A}}\left({\widehat{T}}_y^{\text{LASSO}}\right)\hfill & \mathrm{<mo>=</mo>}{V}_{\mathcal{A}}\left({\displaystyle \sum _{i\in s_A}}{d}_i^A\left(y_i-{\mu }_i{B}^{\text{MC}}\right)+{\displaystyle \sum _{i\mathrm{<mo>=</mo>\; 1}}^N}{\mu }_i{B}^{\text{MC}}\right)\hfill \\ \hfill & \mathrm{<mo>=</mo>}{V}_{\mathcal{A}}\left({\displaystyle \sum _{i\in s_A}}{d}_i^A\left(y_i-{\mu }_i{B}^{\text{MC}}\right)\right)\hfill \\ \hfill & \mathrm{<mo>=</mo>}{\displaystyle \sum _{i\in U}}{\left(\frac{y_i-{\mu }_i{B}^{\text{MC}}}{{\pi }_i}\right)}^2{\pi }_i\left(1-{\pi }_i\right)\hfill \\ \hfill & +{\displaystyle \sum _{i\in U}}{\displaystyle \sum _{j\ne i}}\left({\pi }_{ij}-{\pi }_i{\pi }_j\right)\frac{\left(y_i-{\mu }_i{B}^{\text{MC}}\right)}{{\pi }_i}\frac{\left(y_j-{\mu }_j{B}^{\text{MC}}\right)}{{\pi }_j}(3.7)\hfill \end{array}$$

which follows from equation (3.30) derived for the variance of traditional LASSO calibration estimator of total in McConville (2011). Equation (3.6) then follows from replacing estimates for population quantities.

An alternative variance estimate, suggested by Särndal, Swensson and Wretman (1989), multiplies $\left(y_i-{\widehat{\mu }}_i{\widehat{B}}^{\text{MC}}\right)$ by $g-$ weights, which are the ratios of calibrated weights to the original design weights:

$$g\mathrm{<mo>=</mo>}{1}_{\left(n\times 1\right)}+M{\left(M^T{D}^{A}M\right)}^{-1}{\left(T^M-{\left(d^A\right)}^{T}M\right)}^T$$

$$\begin{array}{ll}v\mathrm{.}{g}_{\mathcal{A}}\left({\widehat{T}}_y^{\text{LASSO}}\right)\hfill & \mathrm{<mo>=</mo>}{\displaystyle \sum _{i\in s_A}}{\left(\frac{g_i\left(y_i-{\widehat{\mu }}_i{\widehat{B}}^{\text{MC}}\right)}{{\pi }_i}\right)}^2\left(1-{\pi }_i\right)\hfill \\ \hfill & +{\displaystyle \sum _{i\in s_A}}{\displaystyle \sum _{j\ne i}}\frac{{\pi }_{ij}-{\pi }_i{\pi }_j}{{\pi }_{ij}}\frac{g_i\left(y_i-{\widehat{\mu }}_i{\widehat{B}}^{\text{MC}}\right)}{{\pi }_i}\frac{g_j\left(y_j-{\widehat{\mu }}_j{\widehat{B}}^{\text{MC}}\right)}{{\pi }_j}.(3.8)\hfill \end{array}$$

To simplify notations, we refer to $v_{\mathcal{A}}\left({\widehat{T}}_y^{\text{LASSO}}\right)$ as $v^{\text{LASSO}}$ and $v\mathrm{.}{g}_{\mathcal{A}}\left({\widehat{T}}_y^{\text{LASSO}}\right)$ as $v_g^{\text{LASSO}}.$

• Previous
• Article main page
• Next