WCD-10. Initial development of GSL's interactive verification scorecard tool

Abstract
The Model Analysis Tool Suite (MATS) is a software product developed in-house by the National Oceanic and Atmospheric Administration (NOAA)’s Global Systems Laboratory (GSL) to assess the forecast skill of regional and global weather models. MATS consists of individual applications, which are each designed to verify a particular meteorological facet (e.g. ceiling heights, radar reflectivities, precipitation accumulations, etc.). Using MATS, model developers can produce interactive, publication-quality plots to quickly and easily compare the skill of individual models’ forecasts as they occur, as well as assess the performance of models under development. In the past year, GSL added a new feature to MATS: interactive verification scorecards that provide a high-level overview of the relative performance of two modeling systems (e.g., the operational HRRR vs. the experimental RRFS). In general, scorecards consist of a matrix of cells, with each cell showing which model performs better for a set of given parameters (e.g., time period, domain, variable, skill score, etc.), and whether that performance difference is statistically significant. MATS interactive scorecards are comprised of three components. The first is an interface that allows the user to select all parameters relevant to the model comparison. The second is a data processing service that allows for timely calculations of all the individual cells’ model data. The third is a visualization application that displays the results in the scorecard, allowing the user to click on any of its cells. When clicked, the data that was used to generate the score for that particular cell will be passed to the appropriate MATS application, thereby allowing the user to interactively plot the data using the range of MATS plot types, and understand the characteristics of the data. We envision that this interactive capability to visualize verification statistics in a number of different variables, forecast hours, and regions simultaneously will greatly speed up our understanding of model behavior and biases, which will improve the speed of model development.