Monday, August 24, 2015

Science, Models, and Verification

Lies, Damn Lies, Statistics, and Models

Some of the biggest tools used to warn of the impending perils of anthropogenic global warming (AGW) are computer models showing increasing global temperatures over the coming years. These models, it is said, are settled scientific proof that AGW is a threat to our very existence.
Having had some experience with computer models over the last few decades, I find them valuable tools... a good computer model can be used to anticipate all sorts of future events. Some examples are:
  • When the Space Shuttle flew for the very first time in 1981, it was the first spacecraft ever to fly with a crew on its very first flight. Engineers felt confident that their computer models of the Shuttle's performance in all flight domains adequately predicted how the spacecraft would actually fly. Turns out they were right.
  • Numerical weather prediction (NWP) models are usually quite accurate in predicting future weather a few days out (the occasional missed forecast notwithstanding -- but that's as much a failure of input data as it is of the model).
  • Engineers can build virtual prototypes of new products, testing them in cyberspace to find weaknesses before having to build real products -- saving cost in testing as well as reducing time from design to production.
There are more, but you get the idea. These examples cover a wide range of domains, but the models used in all of these areas have one thing in common -- Verification and Validation (V&V). The models, themselves, were put through a rigorous regime of tests. They were used to predict an outcome of some future event or system performance which was then compared to reality. If the predicted performance was different from reality, it was back to the drawing board. Thus, models used for operational decisions -- especially for critical things like the first Space Shuttle flight -- have been tested thoroughly enough to have a high degree of confidence their predictions will match reality.
No models are 100% accurate; very few are even mostly right... most require updates to include new knowledge or capabilities. Even our well-exercised weather models' predictions are compared to the actual weather occurrences on a daily basis -- any differences are used to regularly update or "tune" the models in hopes of improving their predictive skills.
This leads us to the models used to support AGW predictions. Certainly, a great deal of research and scientific knowledge has gone into their creation -- we can't deny that. However, there is one area where AGW models fall short -- Verification and Validation. The predicted AGW conditions and changes are unique to our time and are predicted to happen over decades (or more) -- thus, we can't truly compare model predictions to an outcome that hasn't happened, yet... and won't happen for a hundred years.
This, alone, would give me pause when considering the AGW predictions made by computer modeling. However, we have the ability to look at the model predictions made by the Intergovernmental Panel on Climate Change (IPPC) back in 2000. The dire predictions by the IPPC in their past and current reports are certainly frightening enough. But they are based on models whose performance has never been verified or validated.
In the 15 years since, there has been enough data collected so that we can assess the quality of these predictions in the short term -- our own V&V, if you will. The data of choice for this V&V is average global temperatures based on satellite-sensed atmospheric temperatures. These data come from microwave radiometers flying on DoD and NOAA weather satellites (and were available as far back as the late 70's). I have a good deal of experience with them, as one of my jobs in the Air Force was in charge of processing these sensor data from the spacecraft, and making them ready for inclusion in the Air Force NWP models. Thus, I can confirm that these are a good measure of temperature throughout the entire depth of the atmosphere.

So, what do we see when comparing satellite-sensed temperatures against prediction? Take a look at the this chart. 

The set of green lines represents the range of IPCC model predicted temperature change from 2000 to present. The black line represents the satellite-sensed average global temperature changes. While we clearly see a bump of about 0.25 degrees in the late 90s, the global temperatures have been, on average, remarkably steady since then. No rise in temperatures has been seen over the period of the IPCC forecasts, in complete contrast to the model predictions.
In any other business, we would say back to the drawing board -- clearly, something is amiss in the models that have skewed the predictions away from reality. Computer models are not "science" in and of their own. The scientific process isn't just research and prediction. Theories must be verified by actual measurements if they are to be accepted. If the predictions are wrong, then there is something wrong with the theory. In other words, the science isn't settled.
In the AGW industry, however, we charge full speed ahead, reality be damned. Excuses for the missed predictions are found (and disproved, BTW), and the cries of "The Science Is Settled" continue.
So, the next time AGW proponents try to scare you with dire predictions of a hellishly hot Earth, just remember the models used to make these predictions have either never been validated, or have failed the validation efforts made so far.

No comments:

Post a Comment