Forecasting the Future Wrong and Why

I just read a rare essay in which a futurist humbly explained why a forecast from his past proved to be very wrong—in this case, why breakthroughs in neuroscience have not contributed as much to our knowledge of learning and human behavior as he and others had hoped in the 1990s. His mea culpa prompted me to think through why forecasts can be wrong and what, if anything, we can do to become better forecasters.

The primary mistake in this case was failing to understand how complex the system of the brain is and how much more we have yet to learn about how brains work. Another good example is the optimistic forecasts tied to sequencing the human genome. At best we now have a map to explore the interactions of genes and environment in individual health. Therapies and interventions for most diseases are probably many years away. Forecasts about our ecosystem in all its glorious complexity also are likely to be unreliable. Where complex systems are concerned, our knowledge is often too limited for confident forecasts.  

Sometimes we get forecasts wrong because we did not anticipate intervening events. The 2008-09 global recession certainly threw a kink into many business projections and strategic plans. But the recession hardly qualifies as a wild card event. Many knowledgeable people had that sick sense of uncertainty about the underlying fundamentals of our economy. Every forecast rests on a set of assumptions and most presume some continuity of present conditions. The best defense against unanticipated events is testing the assumptions about what must be true for any forecast to happen.

Another forecasting mistake is misjudging the rate of change. Forecasts can be right in assessing the general direction of change, but get ahead of our technological capabilities.  Today we are many generations of IT development and standardization beyond where we were when futurists first forecast the electronic medical record. Now this idea is a key link in improving healthcare delivery in evolving policies and programs for healthcare reform in the US.  The forecast has been right for more than a decade and yet we may not experience the full benefits of an electronic medical record for still another generation.

Perhaps the most puzzling contributor to bad forecasting is misreading individual and public priorities. Although we puzzle about sudden shifts in public opinion or purchasing patterns, we should not be surprised. Just think how your own personal priorities shift in any given year. Maybe this shift happens because of unanticipated events, like postponing your plans to buy new furniture when your home heat pump suddenly dies. Or a new friend brings new interests into your life and you shift your time and purchases toward this new awareness. Now multiply this dynamic across a community, country or the globe and you can see how quickly today’s priority can quickly get lost in tomorrow’s decisions.

So is there any value in forecasting or hope for getting it right? Yes, if we think of forecasting as a form of strategic conversation in a community with the potential to influence the outcome of the forecast. Forecasts become a way to test our understanding of the world, how it is changing, and more importantly, how we want it to change. We can create better forecasts if we remain humble in our interpretation of complex systems, if we challenge our underlying assumptions, if we are patient about the rate of change, and if we accept that sometimes we are just going to change our minds.