Moving toward user-focused verification
Barbara Brown, NCAR
Traditionally, the three purposes of forecast verification defined by Brier and Allen in 1951 have been specified in verification and meteorological statistics textbooks; these purposes are administrative, scientific, and economic. However, most verification measures and approaches have only focused on the administrative goal. The techniques that satisfy the administrative goal are able to provide monitoring information about forecast performance, but they do not provide information that can be used to improve forecasts, or that can aid in decision-making by end users. In particular, these techniques (e.g., RMSE) are able to indicate that a forecast is incorrect, but they cannot specify why the forecast is wrong. Thus, these approaches, in general, do not meet the needs of forecast developers (or scientists), and they do not provide relevant information for economic/user purposes.
In recent years, some efforts have focused on development of verification approaches that provide more diagnostic information regarding forecast performance, indicating which aspects of a forecasting system are in need of improvement, and in what ways, and providing information about the uncertainty in the verification measures. These approaches including basic diagnostic information about forecast errors, object- and entity-based approaches, and approaches that characterize variations in performance with scale go part of the way toward user-focused verification. However, close interactions with particular sets of users (including forecast developers) will be required to develop measures that are truly user-focused.
Five levels of user-focused verification are defined. These levels range from traditional measures (Level 0), to approaches that require user interaction (Level 3), and finally to direct assessment of the forecasts economic value (Level 4). Each of these levels is discussed more deeply, with examples presented showing how they can be achieved.
SECURITY: If you are coming from outside the NOAA campus, please be advised that you will need an on-site sponsor. Please contact that person in advance of the seminar to be put on the list and allow 10 minutes extra on the day of the seminar. Please contact Joe Barsugli (303-497-6042) or Lucia Harrop (303-497-6188) at least a day before the seminar if you have any questions.