Conditional calibration and the sage statistician
Section 4. Calibration – A simulation perspective

• Previous
• Next

Consider how to evaluate a proposed procedure, generically called $P,$ which is to be applied to a data set, generically called $Y,$ yet to be collected from a population; $P$ is a specified function of the data $Y$ and will be used to estimate the estimand, here a scalar quantity, $Q,$ which describes some aspect of the population from which $Y$ is drawn. For descriptive simplicity, suppose $P$ is a purported 95% interval for $Q;$ for $P$ to be exactly calibrated means that $P$ includes $Q$ in exactly 95% of repeated samples. Further, suppose $Y$ is drawn from its population using design $D,$ which is known and fixed throughout this discussion; for concreteness, $D$ is simple random sampling. Although $D$ is known, at the design stage, the data set $Y$ is not yet known. Also suppose, again for simplicity, that all “experts” interested in this problem agree on a set of $K$ possible “Truths” for describing the unknown population to which $D$ will be applied to obtain data set $Y;$ call these possible Truths $T_1,T_2,\dots ,T_K,$ and the values of their associated local (local to each Truth) estimands $Q_1,Q_2,\dots ,Q_K,$ where $Q_k=\tilde{Q}\left(T_k\right),$ for $k=1,\dots ,K,$ for the function $\tilde{Q},$ common to all possible truths. The estimand $Q$ is the value of the function $\tilde{Q}$ evaluated at the actual Truth.

The $Q_k$ are here called local estimands, that is, local to the truths. As far as I can tell, Neyman never fomally considered such local estimands, but I see them as important bridges to the Bayesian perspective as well as to being a sage statistician. Only one of the possible truths is the actual truth. The inferential objective is the value of $\tilde{Q}$ for the Truth that generated the yet-to-be observed data $Y.$

The collection of $K$ possible Truths can often be compactly described mathematically, so that $K$ can be essentially infinite. One example of such Truths, and their associated local estimands, could be $K$ Gaussian univariate populations, with unknown local means, ${\mu }_k,$ and with the scalar estimand $Q$ equal to the mean of the one true population. Or the Truths could be all possible $N-$dimensional vectors of real numbers; this is the standard finite population set-up for survey sampling with $N$ units and one scalar variable, as in Cochran (1963) and Kish (1965), where the estimand $Q$ is typically the mean of the $N$ values for the true population.

We continue by defining simple calibration using a simulation to fix ideas; this simulation will be used to define concepts throughout this manuscript, including the key concept of conditional calibration. Suppose that for each possible truth, $T_k,k=1,\dots ,K,$ with local estimand $Q_k,$ we have drawn $J$ data sets, labeled $Y_{jk},j=1,\dots ,J,$ each drawn using design $D.$ To each of these data sets, we apply procedure $P$ to the data to create an interval estimate for $Q_k,$ where for each $k,Q_k$ is the same for all $Y_{jk}\left(j=1,\dots ,J\right)$ because all such $Y_{jk}$ arose from the same truth $T_k.$ We then assess whether when $P$ is applied to $Y_{jk},$ the resulting interval includes the local estimand $Q_k.$ The proportion of data sets, $\left\{Y_{jk},j=1,\dots ,J\right\},$ for which the interval $P$ includes $Q_k$ is called here the local calibration (or local coverage) of the procedure $P$ for the $k^{\text{th}}$ Truth, notationally written $C_k$ for $k=1,\dots ,K.$ For evaluating point estimators, rather than interval estimators, the calibration of $P$ for $Q_k$ could be replaced by the bias or mean squared error of the point estimate of $Q_k.$

This simulation is depicted in Table 4.1, where each column represents a possible truth, and the $J$ rows represent the $J$ data sets generated under each truth.

Now we define local calibration using 95% to represent any level of coverage. A 95% interval estimate of $Q,P,$ is called “locally (for truth $T_k)$ conservatively calibrated” if $C_k>=$  95%; we could say that $P$ is “approximately locally calibrated” (for Truth $T_k)$ if $C_k$ is close to 95%, but this idea was never formally defined by Neyman, although in Fisher’s (1934) discussion of Neyman (1934), we can see Fisher had something like this in mind with his criticism of Neyman’s formulation.

Next, following Neyman, the interval estimate $P$ is called “confidence calibrated” across the ensemble of possible truths, $\left\{T_k\text{,}k=\text{1,}\dots ,K\right\}\text{,}$ if all $C_k\text{ >=}$  95%, or returning to Neyman’s original definition, $P$ is then simply called a 95% confidence interval for $Q.$ The critical point here for calibration is that all that matters to a die-hard Neymanian frequentist, when evaluating a procedure, $P,$ for its validity, is whether the collection of $C_k$ values for procedure $P$ are all greater than the nominal level for $P.$ The word “confidence” arises because when confronted with the results of Table 4.1 for procedure $P$ and with a critic who selected one Truth from the collection of possible truths, you should be “confident” that the result of applying $P$ to $Y^{*}$ will be an interval that includes $Q.$

These assessments of 95% confidence calibration are well-defined no matter what the etiology of the procedure $P.$ BUT, are they statistically apposite for evaluating $P$ as a 95% interval estimate of the unknown $Q$ after seeing a specific data set, call it $Y^{\text{*}}?$ That is, after seeing a specific instance of $Y,$ now known to be $Y^{\text{*}},$ does the 95%-attached to $P$ necessarily reflect the judgment of a sage statistician? Maybe we should seek only procedures that are approximately calibrated for truths that plausibly could have generated the observed $Y^{\text{*}}?$

We now consider the formal Bayesian perspective because it sheds light on this concept of being sage after seeing $Y=Y^{*}.$

• Previous
• Article main page
• Next