A cautionary note on Clark Winsorization Section 5. SummaryA cautionary note on Clark Winsorization Section 5. Summary

• Previous

The usage of Clark Winsorization is very appealing for the simplicity of its implementation and lack of parameters as long as one can build a viable robust regression model. However, as with many outlier detection procedures, the method has certain vulnerabilities that are not always obvious. This note demonstrates how the procedure can be effective at identifying and treating influential values, but is also highly sensitive to the number of influential values in the sample and their magnitude with respect to the regression line used to determine the detection region bounds. The properties of the detection region vary by whether an influential value is present and by the number and severity when one or more appear. If the sample contains no influential values, the procedure is anti-conservative in that it trims values not considered influential to minimize the MSE (by reducing the variance). In contrast, the procedure can become very conservative depending on the degree of difference of the weighted influential value from the others in the sample. When the sample contains two or more influential values, Clark Winsorization detects and adjusts only the influential values and does not trim any values that are not influential. However, our results demonstrate a potential for masking which should be considered when implementing the procedure.

If the occurrence of an influential value is truly a rare event and large influential values are of interest, then the small trimming of a handful of values that are not influential is a disadvantage. However, in applications where influential values are common or where historic data are not available for modeling, implementing Clark Winsorization definitely requires an assessment of the amount of trimming to determine if the aggregated small changes greatly affect the estimated total. If not, then this is an appealing approach. If yes, then other methods such as M-estimation $–$ which give more control over the detection region $–$ may be advantageous.

Acknowledgements

This report is released to inform interested parties and encourage discussion of work in progress. The views expressed on statistical, methodological, and operational issues are those of the authors and not necessarily those of the U.S. Census Bureau. The authors thank Lynn Weidman, Eric Slud, Scott Scheleur, William C. Davie Jr. and Carma Hogue for their helpful reviews of previous versions of the manuscript. The authors also thank Ray Chambers for his comments during presentations of our work in progress. The authors appreciate the comments from the Associate Editor and the anonymous Referees.

References

Barnett, V., and Lewis, T. (1994). Outliers in Statistical Data. 3^rd Edition. New York: John Wiley & Sons, Inc.

Beaumont, J.-F., and Alavi, A. (2004). Robust generalized regression estimation. Survey Methodology, 30, 2, 195-208. Paper available at http://www.statcan.gc.ca/pub/12-001-x/2004002/article/7752-eng.pdf.

Chambers, R.L. (1986). Outlier robust finite population estimation. Journal of the American Statistical Association, 81, 1063-1069.

Chambers, R., Kokic, P., Smith, P. and Cruddas, M. (2000). Winsorization for identifying and treating outliers in economic surveys. ICES II, The Second International Conference on Establishment Surveys, Survey Methods for Businesses, Farms, and Institutions, American Statistical Association, 717-726.

Clark, R. (1995). Winsorization Methods in Sample Surveys. Masters Thesis. Department of Statistics. Australia National University. http://hdl.handle.net/10440/1031 (accessed September 29, 2016.).

Kokic, P.N., and Bell, P.A. (1994). Optimal winsorising cut-offs for a stratified finite population estimator. Journal of Official Statistics, Stockholm, Sweden, 10, 419-435.

Martinoz, C.F., Haziza, D. and Beaumont, J.-F. (2015). A method of determining the winsorization threshold, with an application to domain estimation. Survey Methodology, 41, 1, 57-77. Paper available at http://www.statcan.gc.ca/pub/12-001-x/2015001/article/14199-eng.pdf.

Mulry, M.H., Oliver, B.E. and Kaputa, S.J. (2014). Detecting and treating verified influential values in a monthly retail trade survey. Journal of Official Statistics, 30(4), 1-28.

SAS (2014). Help and Documentation. SAS Institute, Inc. Cary, NC.

Smith, P. (2013). Sampling and estimation for business surveys. In Designing and Conducting Business Surveys, (Eds., G. Snijkers, G. Haraldsen, J. Jones and D. Willimack), Hoboken, NJ: John Wiley & Sons, Inc., 219-52.

U.S. Census Bureau (2014). Monthly Retail Trade Survey Methodology. U.S. Census Bureau, Washington, DC. http://www.census.gov/retail/mrts/how_surveys_are_collected.html (accessed September 29, 2016).

• Previous