Some Remarks on the Reliability of Categorical Probability Forecasts

Bröcker, Jochen
November 2008
Monthly Weather Review;Nov2008, Vol. 136 Issue 11, p4488
Academic Journal
Studies on forecast evaluation often rely on estimating limiting observed frequencies conditioned on specific forecast probabilities (the reliability diagram or calibration function). Obviously, statistical estimates of the calibration function are based on only limited amounts of data and therefore contain residual errors. Although errors and variations of calibration function estimates have been studied previously, either they are often assumed to be small or unimportant, or they are ignored altogether. It is demonstrated how these errors can be described in terms of bias and variance, two concepts well known in the statistics literature. Bias and variance adversely affect estimates of the reliability and sharpness terms of the Brier score, recalibration of forecasts, and the assessment of forecast reliability through reliability diagram plots. Ways to communicate and appreciate these errors are presented. It is argued that these errors can become quite substantial if individual sample points have too large influence on the estimate, which can be avoided by using regularization techniques. As an illustration, it is discussed how to choose an appropriate bin size in the binning and counting method, and an appropriate bandwidth parameter for kernel estimates.


Related Articles

  • Improving Probabilistic Ensemble Forecasts of Convection through the Application of QPF--POP Relationships. Schaffer, Christopher J.; Gallus, William A.; Segal, Moti // Weather & Forecasting;Jun2011, Vol. 26 Issue 3, p319 

    Four new approaches of postprocessing quantitative precipitation forecasts (QPFs) from model ensemble output were used to generate probability of precipitation (POP) tables in order to develop a forecasting method that could outperform a traditional method that relies upon calibration of POP...

  • An Evaluation of a Drop Distribution–Based Polarimetric Radar Rainfall Estimator. Brandes, Edward A.; Zhang, Guifu; Vivekanandan, J. // Journal of Applied Meteorology;May2003, Vol. 42 Issue 5, p652 

    A method for estimating the governing parameters of gamma drop size distributions (DSDs) and associated rainfall rates from polarimetric radar measurements at the S band is examined. The technique uses radar reflectivity at horizontal polarization, differential reflectivity, and an empirical...

  • Improved Seasonal Probability Forecasts. Kharin, Viatcheslav V.; Zwiers, Francis W. // Journal of Climate;Jun2003, Vol. 16 Issue 11, p1684 

    A simple statistical model of seasonal variability is used to explore the properties of probability forecasts and their accuracy measures. Two methods of estimating probabilistic information from an ensemble of deterministic forecasts are discussed. The estimators considered are the...

  • Erratum to: Estimating reliability and resolution of probability forecasts through decomposition of the empirical score. Bröcker, Jochen // Climate Dynamics;Dec2012, Vol. 39 Issue 12, p3123 

    No abstract available.

  • Harvest surprises. Schemper, Janel // High Plains Journal;6/17/2019, Vol. 137 Issue 24, p9B 

    No abstract available.

  • Verification of precipitation forecasts of MM5 model over Epirus, NW Greece, for various convective parameterization schemes. Sindosi, O. A.; Bartzokas, A.; Kotroni, V.; Lagouvardos, K. // Natural Hazards & Earth System Sciences;2012, Vol. 12 Issue 5, p1393 

    The mesoscale meteorological model MM5 is applied to 22 selected days with intense precipitation in the region of Epirus, NW Greece. At first, it was investigated whether and to what extend an increased horizontal resolution (from 8 to 2 km) improves the quantitative precipitation forecasts. The...

  • The Calibration Simplex: A Generalization of the Reliability Diagram for Three-Category Probability Forecasts. Wilks, Daniel S. // Weather & Forecasting;Oct2013, Vol. 28 Issue 5, p1210 

    Full exposition of the performance of a set of forecasts requires examination of the joint frequency distribution of those forecasts and their corresponding observations. In settings involving probability forecasts, this joint distribution has a high dimensionality, and communication of its...

  • TOWARD SEAMLESS PREDICTION. Palmer, T. N.; Doblas-Reyes, F. J.; Weisheimer, A.; Rodwell, M. J. // Bulletin of the American Meteorological Society;Apr2008, Vol. 89 Issue 4, p459 

    Trustworthy probabilistic projections of regional climate are essential for society to plan for future climate change, and yet, by the nonlinear nature of climate, finite computational models of climate are inherently deficient in their ability to simulate regional climatic variability with...

  • Improved Results for Probabilistic Quantitative Precipitation Forecasting. Gahrs, Gregory E.; Applequist, Scott; Pfeffer, Richard L.; Xu-Feng Niu // Weather & Forecasting;Oct2003, Vol. 18 Issue 5, p879 

    As a follow-up to a recent paper by the authors in which various methodologies for probabilistic quantitative precipitation forecasting were compared, it is shown here that the skill scores for linear regression and logistic regression can be improved by the use of alternative methods to obtain...


Read the Article


Sorry, but this item is not currently available from your library.

Try another library?
Sign out of this library

Other Topics