Authors
Gerard S. Ketefian (CIRES,NOAA/GSL), Michael J. Kavulich, Jr. (NCAR/UCAR), Julia M. Simonson (CIRES,NOAA/GSL), Jeff Beck (NOAA/GSL), Michelle A. Harrold (NCAR/UCAR), Kathryn M. Newman (NCAR/UCAR), Christina R. Holt (CIRES,NOAA/GSL)
Abstract
Since 2019, projects at the Developmental Testbed Center (DTC) supporting development of the Rapid Refresh Forecast System (RRFS) have used the verification (vx) infrastructure in the Unified Forecast System (UFS) Short-Range Weather (SRW) App to perform METplus-based vx. In response to the needs of these projects, the capabilities of this infrastructure have been extended and generalized. Current capabilities include ingestion of forecasts from both regional and global operational models (e.g. RAP, HRRR, NAM, RRFS, GFS), retrieval of several observation types from multiple data stores (e.g. CCPA, NOHRSC, MRMS, NDAS, AERONET, AIRNOW), generation of deterministic and ensemble vx metrics, and verification of either a standard or custom set of forecast fields at user-specifiable output intervals. Application of the Unified Workflow tools (uwtools) package, increased use of more powerful and interoperable Python scripts, and consolidation of experiment settings into a minimal set of configuration files have resulted in a modular, highly configurable, and extensible system for generating vx workflows for a wide variety of forecast model outputs, observation types, and METplus verification tools and statistics.
As the capabilities of this vx system have grown beyond the original scope of regional weather prediction, additional groups within the DTC and the Earth Prediction Innovation Center (EPIC) have recognized its utility for improved cross-project coordination and reduced duplication of resources. To avoid unnecessary overhead and increase the speed of development, the DTC has created a new GitHub repository containing a stand-alone vx workflow generation system, dubbed the METplus Verification Framework, that can be shared by all DTC projects as well as the broader NWP community. This new vx workflow combines the SRWâs verification capabilities with the tropical-cyclone-focused verification capabilities of the Hurricane Analysis and Forecast System (HAFS) that have also been developed at the DTC. In addition, design inefficiencies resulting from close ties to the SRW App have been removed, and incremental tests have been added to ensure the integrity of results as new capabilities are added. Our goal is standardization of verification across DTC projects, independent of UFS application, making verification much more reliable and consistent, and providing project team members the ability to quickly replicate results by simply sharing workflow configuration files. In this presentation, we describe the implementation of this new METplus Verification Framework and show examples of its use in several DTC projects. Future efforts will integrate verification capabilities for both global and AI-based forecast model outputs.