Advancements in Weather Prediction

Communications of the ACM
7m read

For most of human history, we have attempted to predict the weather—in one way or another, by means both scientific and superstitious. While tremendous strides have been made in the last century, the last decade has brought exponential progress.

The discipline of numerical weather prediction (NWP), the mathematical modeling of the atmosphere and oceans in order to predict the weather, is the predominant way we forecast the weather today. It’s relatively new (at least, compared to Mother Nature herself). It began to take off in the mid-20th century, when the first operational NWP forecasts were made on the earliest forms of computers.

From there, the story of NWP is inextricably interlinked with computation. Better computers have led to both better data and better ways of using that data to predict everything from the daily forecast to extreme weather events like tornadoes or tropical storms.

However, despite steady progress in NWP over the last half-century thanks to the computing revolution, dramatic progress—beyond the expectations of many experts—has happened only in the last 10 years, thanks to another epochal shift: major advances in artificial intelligence and machine learning, and in the amount and quality of data we can now feed these technologies.

An explosion of data

The first big shift we’ve seen in the last decade when it comes to weather prediction is the sheer amount and quality of data with which forecasters now can work.

According to Kyle Hilburn, a meteorological research scientist at Colorado State University, there has been a literal 10X improvement in the spatial resolution of NWP models in the last decade.

NWP models rely on “grid cells” to make their forecasts. Grid cells are segments of the atmosphere in any given location divided up into a three-dimensional grid. Weather conditions then are analyzed and forecasted at the grid cell level.

Just a decade ago, the standard used to be analyzing and forecasting grid cells that were 30 square kilometers (or 18 square miles) in size, said Hilburn. Today, operational models are now able to run on grid cells as granular as 3 square kilometers (1.8 square miles) in size.

More detailed resolution of the grid being modeled, including its topography, land use, land cover, and coastlines, results in better predictions. In other words, the closer you’re able to “zoom in” on weather phenomena, the better you’re able to predict what could happen next.

“This has improved our ability to forecast thunderstorms and other highly impactful ‘mesoscale’ severe weather phenomena,” Hilburn said.

We also have seen explosive growth in the number and frequency of observations now available from space and at the Earth’s surface, said Jerald Brotzge, a climatologist and director of the Kentucky Climate Center.

“Previously, we were limited to mostly in situ observations near populated land areas and some limited number of federally-operated satellite systems,” Brotzge said. “Now, data from commercial platforms are becoming a critical contributor to our national observing infrastructure.”

That includes data from commercial satellites, weather radar, upper-atmosphere balloons, and ocean buoys—data that can fill critical gaps in attempts to predict the weather. With more and better data, especially from previously inaccessible areas of the globe, we’ve been able to significantly improve analysis and prediction.

This has also led to a maturation and expansion of “ensemble” modeling. This refers to the practice, often seen in election polling, of taking an ensemble, or averaging, of individual forecasts to produce a more accurate composite. More advanced computer chips have permitted increasing numbers of model runs to be incorporated into forecasts, said Brotzge.

That, in turn, has given us greater forecast accuracy. For instance, the latest version of the U.S. National Weather Service’s Global Ensemble Forecast System (GEFS) has demonstrated improved forecasts of rainfall and tropical storm tracking thanks to this trend.

Together, these advancements have resulted in steady improvement over the years in the overall accuracy of weather predictions. For example, seven-day forecasts are approaching the level of accuracy that five-day forecasts had two decades ago, said Hilburn.

Artificially intelligent weather prediction

Even more progress has come from using artificial intelligence and machine learning in NWP.

“Arguably the greatest innovation in numerical weather prediction in the last half-century is AI and machine learning, and yet its full potential is still unknown,” said Brotzge.

AI and machine learning methods are being applied to nearly every step of the NWP process today, he said. That includes everything from aiding data quality control to replacing statistical model parameterization to speeding up the performance time of prediction algorithms. Some commercial firms are even operating pure AI-based NWP models that they claim either match or exceed the performance of traditional, physics-based models.

While Brotzge said AI-based models aren’t across the board as robust as physics-based models yet, he acknowledged they are demonstrating greater accuracy in downscaling NWP results, which leads to faster, more accurate decision-making.

Not to mention, AI and machine learning are making strides to actually simulate the physics that underlie weather phenomena, said Sue Ellen Haupt, a senior scientist at the National Center for Atmospheric Research.

Deep learning methods, in particular, are now being used to “train whole global models of the atmosphere,” she said. Today, many of these are currently configured by private sector scientists and machine learning experts, and most are trained with data from the European Centre for Medium-Range Weather Forecasts (ECMWF). ECMWF runs an AI-based model called the Artificial Intelligence/Integrated Forecasting System (AIFS), in parallel with their popular physics-based model.

“This has made for very rapid change within the field,” said Haupt.

In fact, the ECMWF’s models and data are so good that they’ve partnered with one of the leaders in AI to devise even better predictions. In July 2024, Google unveiled a hybrid weather prediction model called NeuralGCM in partnership with the ECMWF. The model combines traditional physics-based models with a cutting-edge neural network that can predict small-scale atmospheric processes.

NeuralGCM produces more accurate weather forecasts for two- to 15-day periods than the existing physics-based models, and outperforms state-of-the-art models for forecast accuracy. It’s also far more efficient computationally, running faster than traditional physics-based models.

And this is just at the global level. Many countries run even-higher-resolution models for more specific regional forecasts—and these often provide better local forecasts.

“That resolution is what is needed to resolve processes such as clouds, as well as to get the local effects driven by terrain, land cover and land use, and coastal boundaries,” explained Haupt.

Also popular at the regional level are “convection-allowing models,” which can simulate thunderstorms, said Hilburn. Examples in the U.S. include the High Resolution Rapid Refresh model and the North American Mesoscale Nested model.

The next decade of progress

Just because huge strides have been made in NWP thanks to AI and machine learning doesn’t mean there is not still major work to be done. Significant challenges remain that will define the next 10 years of weather prediction. And many of these challenges are actually direct products of NWP’s success over the last decade.

For one, we need more data. Today’s AI and machine learning systems make better predictions when they have more and higher-quality data. However, even when we have this data, we can’t always process it effectively.

“With an ever-increasingly diverse mixture of observations totaling over 100 terabytes daily, how can we quality control and assimilate these data in real time in a manner that improves NWP?” asked Brotzge. In fact, much of the satellite data collected today is discarded because it can’t be used it in models due to the sheer volume and speed at which it is being collected, he said.

Beyond that, we’re also trying to collect even more data beyond just what satellites can observe, in our quest to make even better weather predictions.

“The atmosphere is just one part of the Earth system,” said Hilburn. “And accurately predicting the weather across time scales requires modeling the other components that influence the atmosphere, namely the ocean, lakes, sea ice, land surface, soil, biosphere, and fire.”

Making even more granular predictions means better ways are needed to handle the processes that are smaller than a single grid cell, Hilburn said. We need to account for processes that include clouds and precipitation, radiation and heat fluctuations, soil and vegetation, and turbulence and topography—all at increasingly small scales. Even tiny errors at these scales can “cascade over time to produce bigger errors,” he said.

That inhibits the ability to make better predictions, including the ability to predict rare weather events, which may be occurring more frequently due to climate change.

“Much like politics, all weather is local because variations in surface characteristics have a strong influence on the spatial distribution of weather phenomena, and because relatively small variations in the track of weather features can have a huge influence on how many people are impacted and by what type of weather,” said Hilburn.

Further Reading

cacm.acm.org