Dr Beth Ebert , Bureau of Meteorology ( beth . ebert @ bom . gov . au ) Tennessee Leeuwenburg , Bureau of Meteorology ( tennessee . leeuwenburg @ bom . gov . au )

Measuring the accuracy of forecasts and predictions is a complex science , referred to as forecast verification . Verification results help forecast users make more informed decisions based on evidence from past performance . Verification results also guide forecasters and developers of forecasting systems on where to make improvements and provide managers with insights into trends in forecast performance .

The Bureau of Meteorology is a global centre of excellence for forecast verification . To support high standards for quality , the Bureau has produced some nifty tools , including the scores Python package recently released as open source . This is its story .

Back in the distant times of 2015 , the Bureau was getting serious about streamlining and automating the forecast procedures used by operational meteorologists . The Jive verification environment was built with this goal in mind . Jive introduced many things into the Bureau that are now taken for granted — Jupyter Notebooks , standardised metrics , schemas for the forecast and observation data sets — and supported the experimental infrastructure needed . Since then , the Jive Verification Hub has been used by almost a third of the Bureau staff .

As useful as this environment is , it is limited in other ways . There is an increasing need to perform consistent and sophisticated verification in other environments , such as working at the NCI supercomputing facility or sharing code and methodologies with staff from international partner organisations . For that , a reusable library rather than a platform is needed . Further , to engage and collaborate effectively with others , the code needs to be shared . And the easiest way to do that is to remove the barriers using an open-source approach .

Scores includes most of the traditional verification metrics that are widely used around the world to verify weather forecasts and some novel ( and really useful !) metrics developed by Bureau researchers such as threshold-weighted consistent scores for extreme weather verification . Every metric in the package has a companion tutorial Jupyter Notebook which makes it easy to get started for new and experienced users alike .

There is a beneficial network effect when multiple groups adopt common tools . It ' s easier to ensure that results can be directly compared when produced in a common fashion . This is why the research community often comes together around tools , particularly open ones . METplus , an open-source verification system developed by the Developmental Testbed Center in the United States , and other ecosystems also offer this benefit . The complexity of METplus and other systems is often much higher than scores , as they are complete ecosystems , meaning they do more things , but are accordingly harder to install and use .

Scores focuses on ease of adoption . It is available in PyPI and conda-forge . This ease of adoption helps both locally and when

An AI-generated image from Microsoft Copilot depicting the scores verification package . working with external collaborators . NCI has begun to include scores in its training materials for evaluation of machine learning models .

Since its first release in May 2024 , scores has started gaining interest from others . For example , when it was presented at the 9th International Verification Methods workshop in Cape Town , South Africa , some participants immediately jumped online and started using it . Artificial intelligence researchers at NCAR and ECMWF are also trying it out . Scores has been used in a research context at NIWA for verifying a machine learning prototype and CSIRO is exploring its adoption for hydrological verification .

You can find scores on GitHub at https :// scores . readthedocs . io /.

“ We hope scores is of value to others . It offers a useful verification library for many users and also encourages community contributions for new metrics . We hope it will grow organically , and of course , we plan to make lots of good use of it !”— Lead developer Tennessee Leeuwenburg

Contributors to scores include Tennessee Leeuwenburg , Nicholas Loveday , Beth Ebert , Harrison Cook , Mohammedreza Khanarmuei , Rob Taggart , Nikeeth Ramanathan , Maree Carroll , Stephanie Chong , Aidan Griffiths , John Sharples and Durga Lal Shreshti .