Conditional calibration and the sage statistician
Section 7. The conditional calibration plot and its use for sagely selecting procedures to use with observed data $Y^{*}$

Table of contents

The conditionally calibrated (CC) statistician faced with estimating $Q$ using procedure $P$ from data set $Y^{*}$ cares about being approximately calibrated, i.e., close $C_{k}$ to 95% especially for Truths with large values of $M_{k}^{*},$ indicating that such Truths could have plausibly generated $Y^{*} .$ In other words, when comparing procedures for estimating $Q$ from $Y^{*},$ the sage statistician, in addition to conservative unconditional calibration (i.e., confidence coverage), especially cares about accurate calibration for Truths that are plausible, and therefore implicitly ignores the calibration of procedures for Truths that are implausible given $Y^{*} .$

Figure 7.1 Ck versus M*k Plots for a fixed data set, with K=9 Truths (columns)

Description for Figure 7.1

Figure presenting the conditional calibration plot. It contains hypothetical simulation results with a fixed data set $Y^{*}$ and a fixed set of nine possible Truths for three procedures: conditional calibration (CC), not conditional calibration (Not CC) and confidence interval (CI). The calibration $C_{k}$ is on the y-axis, ranging from 0% to 100%, where 0% to 95% correspond to invalid, 95% is nominal and 95% to 100% correspond to too inclusive. The axis is not linear in $C_{k}$ but expanded for values of $C_{k}$ closer to unity. The $M_{k}^{*}$ for the nine possible truths are on the x-axis, ranging from 0 to 1.

CC procedure is labeled “Smile” because it is approximately calibrated $(C_{k}$ close to 95% for $M_{k}^{*}$ close to 1), even if $C_{k}$ is well below 95% for $M_{k}^{*}$ much lower than 1. A second procedure, Not CC, is labeled “Frown” because it is not CC, i.e. $C_{k}$ is substantially less than 95% even if $M_{k}^{*}$ is close to 1. CI procedure is labeled “Neutral [CI]” because, although it is a valid confidence interval in Neyman’s sense of having its minimum local calibration at least 95%, it is not approximately calibrated for $M_{k}^{*}$ close to 1.

Figure 7.1 presents hypothetical simulation results with a fixed data set $Y^{*}$ and a fixed set of nine possible Truths (with nine associated local match rates to $Y^{*})$ for three procedures, indicated by faces. The vertical axis is not linear in $C_{k}$ but expanded for values of $C_{k}$ closer to unity, which is where our interest is focused. One procedure is labeled “Smile” because it is approximately calibrated $(C_{k}$ close to 95%) for possible Truths that could have generated $Y^{*}$ $(M_{k}^{*}$ close to 1), even though poorly calibrated $(C_{k}$ well below 95%) for a priori possible Truths that are implausible given the observed $Y^{*}$ $(M_{k}^{*}$ much lower than 1). A second procedure is labeled “Frown” because it is not CC, being invalid (meaning its local calibration is substantially less than 95%), including for truths that are plausible given $Y^{*} .$ The third procedure is labeled as “Neutral [CI]” because, although it is a valid confidence interval in Neyman’s sense of having its minimum local calibration at least 95%, it is not approximately calibrated for Truths that are plausible given the observed data set, $Y^{*} .$ This procedure could, for me, be described by a mild frown, but maybe not for Neyman, based on our 1970’s conversation.

That is, to repeat, Neymanian (conservative = confidence) calibration for each procedure formally just cares about the procedures’ minimum $C_{k}$ across the entire ensemble of a priori possible truths. Also, the rigid Bayesian just cares about the weighted average of the $M_{k}^{*}$ across the possible truths, weighted by the prior possibly unreliable distribution for the truths, $W_{k} .$ The sage CC statistician cares about approximate local calibration of procedures for those Truths that are plausible; if a confidence-valid 95% procedure $P$ displays $C_{k}$ values substantially bigger than 95% for plausible Truths, this suggests that there exist better CC procedures for this situation with data set $Y^{*};$ that is, calibrated procedures that are more efficient and so result in shorter intervals. Notice for example, that the confidence-valid procedure in Figure 7.1 (Neutral face) has worse CC than Smile, and thus although a plausible competitor to Smile at the design stage should be seen as inferior to Smile after seeing data $Y^{*}$ because it is too conservative for some of the relevant Truths.

ISSN : 1492-0921

Editorial policy

Survey Methodology publishes articles dealing with various aspects of statistical development relevant to a statistical agency, such as design issues in the context of practical constraints, use of different data sources and collection techniques, total survey error, survey evaluation, research in survey methodology, time series analysis, seasonal adjustment, demographic studies, data integration, estimation and data analysis methods, and general survey systems development. The emphasis is placed on the development and evaluation of specific methodologies as applied to data collection or the data themselves. All papers will be refereed. However, the authors retain full responsibility for the contents of their papers and opinions expressed are not necessarily those of the Editorial Board or of Statistics Canada.

Submission of Manuscripts

Survey Methodology is published twice a year in electronic format. Authors are invited to submit their articles in English or French in electronic form, preferably in Word to the Editor, (statcan.smj-rte.statcan@canada.ca, Statistics Canada, 150 Tunney’s Pasture Driveway, Ottawa, Ontario, Canada, K1A 0T6). For formatting instructions, please see the guidelines provided in the journal and on the web site (www.statcan.gc.ca/SurveyMethodology).

Note of appreciation

Canada owes the success of its statistical system to a long-standing partnership between Statistics Canada, the citizens of Canada, its businesses, governments and other institutions. Accurate and timely statistical information could not be produced without their continued co-operation and goodwill.

Standards of service to the public

Statistics Canada is committed to serving its clients in a prompt, reliable and courteous manner. To this end, the Agency has developed standards of service which its employees observe in serving its clients.

Copyright

Published by authority of the Minister responsible for Statistics Canada.

Use of this publication is governed by the Statistics Canada Open Licence Agreement.

Catalogue No. 12-001-X

Frequency: Semi-annual

Ottawa

Date modified:: 2019-09-10

Language selection

Search and menus

Search

Conditional calibration and the sage statistician
Section 7. The conditional calibration plot and its use for sagely selecting procedures to use with observed data $Y^{*}$