Inferential, Nonparametric Statistics to Assess the Quality of Probabilistic Forecast Systems

Maia, Aline de H. N.; Meinke, Holger; Lennox, Sarah; Stone, Roger
February 2007
Monthly Weather Review;Feb2007, Vol. 135 Issue 2, p351
Academic Journal
Many statistical forecast systems are available to interested users. To be useful for decision making, these systems must be based on evidence of underlying mechanisms. Once causal connections between the mechanism and its statistical manifestation have been firmly established, the forecasts must also provide some quantitative evidence of “quality.” However, the quality of statistical climate forecast systems (forecast quality) is an ill-defined and frequently misunderstood property. Often, providers and users of such forecast systems are unclear about what quality entails and how to measure it, leading to confusion and misinformation. A generic framework is presented that quantifies aspects of forecast quality using an inferential approach to calculate nominal significance levels (p values), which can be obtained either by directly applying nonparametric statistical tests such as Kruskal–Wallis (KW) or Kolmogorov–Smirnov (KS) or by using Monte Carlo methods (in the case of forecast skill scores). Once converted to p values, these forecast quality measures provide a means to objectively evaluate and compare temporal and spatial patterns of forecast quality across datasets and forecast systems. The analysis demonstrates the importance of providing p values rather than adopting some arbitrarily chosen significance levels such as 0.05 or 0.01, which is still common practice. This is illustrated by applying nonparametric tests (such as KW and KS) and skill scoring methods [linear error in the probability space (LEPS) and ranked probability skill score (RPSS)] to the five-phase Southern Oscillation index classification system using historical rainfall data from Australia, South Africa, and India. The selection of quality measures is solely based on their common use and does not constitute endorsement. It is found that nonparametric statistical tests can be adequate proxies for skill measures such as LEPS or RPSS. The framework can be implemented anywhere, regardless of dataset, forecast system, or quality measure. Eventually such inferential evidence should be complemented by descriptive statistical methods in order to fully assist in operational risk management.


Related Articles

  • K-modes Clustering. Chaturvedi, Anil; Green, Paul E.; Caroll, J. Douglas // Journal of Classification;2001, Vol. 18 Issue 1, p21 

    We present a nonparametric approach to deriving clusters from categorical (nominal scale) data using a new clustering procedure called K-modes, which is analogous to the traditional K-Means procedure (MacQueen 1967) for clustering interval scale data. Unlike most existing methods for clustering...

  • Measuring Input Substitution and Output Expansion Effects: A Nonparametric Approach with Application. Wan, Guang H. // Empirical Economics;1996, Vol. 21 Issue 3, p361 

    A simple framework is developed for measuring input substitution and output expansion effects. These measures are nonparametric in the sense that specification and/or estimation of any parametric functions are not resquired. Monte Carlo experiments performed in the paper demonstrate the...

  • Nonparametric goodness-of-fit tests for the rasch model. PONOCNY, IVO // Psychometrika;Sep2001, Vol. 66 Issue 3, p437 

    A Monte Carlo algorithm realizing a family of nonparametric tests for the Rasch model is introduced which are conditional on the item and subject marginals. The algorithm is based on random changes of elements of data matrices without changing the marginals; most powerful tests against all...

  • Nonparametric intensity bounds for the delineation of spatial clusters. Oliveira, Fernando L. P.; Duczmal, Luiz H.; Cançado, André L. F.; Tavares, Ricardo // International Journal of Health Geographics;2011, Vol. 10 Issue 1, p1 

    Background: There is considerable uncertainty in the disease rate estimation for aggregated area maps, especially for small population areas. As a consequence the delineation of local clustering is subject to substantial variation. Consider the most likely disease cluster produced by any given...

  • Confidence intervals for the difference of two normal population variances. Niwitpong, Suparat // World Academy of Science, Engineering & Technology;Aug2011, Issue 56, p602 

    Motivated by the recent work of Herbert, Hayen, Macaskill and Walter [Interval estimation for the difference of two independent variances. Communications in Statistics, Simulation and Computation, 40: 744-758, 2011.], we investigate, in this paper, new confidence intervals for the difference...

  • A MONTE CARLO SAMPLING PLAN FOR ESTIMATING NETWORK RELIABILITY. Fishman, George S. // Operations Research;Jul/Aug86, Vol. 34 Issue 4, p581 

    For an undirected network G = (V, E) whose arcs are subject to random failure, we present a relatively complete and comprehensive description of a general class of Monte Carlo sampling plans for estimating g = g(s. T). the probability that a specified node s is connected to all nodes in a node...

  • Research of Monte Carlo Simulation in Commercial Bank Risk Management. Xiao, Beiming // Journal of Systems Science & Information;Dec2004, Vol. 2 Issue 4, p685 

    Simulation method is an important tool in financial risk management. It can simulate financial variable or economic variable and deal with non-linear or non-nominal issue. This paper analyzes the usage of "Monte Carlo" approach in commercial bank risk management.

  • Quantum Monte Carlo Simulations. Troyer, Matthias; Werner, Philipp // AIP Conference Proceedings;8/20/2009, Vol. 1162 Issue 1, p98 

    In these lecture notes we present an introduction to modern quantum Monte Carlo methods for strongly correlated quantum lattice models. After an introduction to classical Monte Carlo methods we will present the loop algorithm, directed loop algorithm, worm algorithm, Wang-Landau sampling for...

  • Finite-Temperature Coarse-Graining of One-Dimensional Models: Mathematical Analysis and Computational Approaches. Blanc, X.; Le Bris, C.; Legoll, F.; Patz, C. // Journal of Nonlinear Science;Apr2010, Vol. 20 Issue 2, p241 

    The article presents a new method for the computation of free energies and ensemble canonical averages of one-dimensional coarse-grained models in materials science. The three dominant computational approaches used for the calculation are Markov chains methods, Monte Carlo methods and molecular...


Read the Article


Sorry, but this item is not currently available from your library.

Try another library?
Sign out of this library

Other Topics