What is the likelihood of nuclear war in Korea in the next 6 months? Who is going to win the next Nigerian presidential election?
In Superforecasting, Philip Tetlock & Dan Gardner reveal their research findings of a group of “Superforecasters” who are much better at making predictions than intelligence agencies, pundits, and talking heads, and most everyone else. The questions above are examples of the types of questions these superforecasters attempted to answer. The first portion of Superforecasting takes time to note the first major issue with current forecasts are how poor they are lacking clarity and timelines. It’s easy to say there will be a major recession soon vs. saying there is an 85% chance of a global recession where we will see a contraction in the credit market occurring in the next 6 months. The first is vague – I don’t define what I mean by major – leaving it up to interpretation nor did I put a specific timeline. It could happen tomorrow or in 500 years. #1 having clear time bound and measurable forecasts are the first step.
Additionally Tetlock & Gardner cite a study that showed forecasters are about as accurate as chimps throwing darts at a dartboard. They may hit the bull’s-eye with a prediction but are not very accurate most of the time. When it comes to pundits on TV and those with strong opinions or worldviews who scream the loudest often are considered forecasters, but that is false according to their research. They make poor immeasurable forecasts like the ones above, tend to utilize confirmation bias, and don’t go back to check their forecasts when they are wrong.
The book gets its title from a research project called the Good Judgment Project where individuals were asked to make forecasts on the questions above. They were then measured on the degree of accuracy to their forecasts which would yield a Brier score. This score represents how great of a forecaster you are – the Superforecasters were just that super and had the best Brier scores. They performed better than the average and continually updated their forecasts as new pieces of information arose. As events became more certain they updated their percentages of occurrence accordingly. Superforecasters also approached problems differently and the best of the best accepted criticism and challenges from others to their point of view.
The key takeaways for me are that you can be better at forecasting and it starts by having a great question that is measurable and time bound. Another common mistake was answering an easier question than the one asked. For example, a study took people and broke them into 3 groups – they were asked the same question about how big of a problem and how much they would donate to help migratory ducks whose environment had been contaminated with oil. The researchers changed the # of ducks from 2,000 to 20,000 to 200,000 for the 3 groups. They found that the average amount of money they were willing to donate to solve the problem for the 3 groups was about the same. People thought about a single duck in oil – measured their desire to help that duck and then gave an answer based on their feeling towards the example rather than the actual # of birds. The lesson learned is to ensure you are answering the actual question posed not an easier related question.
There are ways we can get better at forecasting in our daily lives and this applies to business as well. We often time make a prediction and either fail to update it or measure the success of that prediction. In business, you may have forecasted annual growth of 25% – but something happens in the market and a major competitor announces they are exiting the space or announces an acquisition of another large competitor. It would be wise to assess how that event may or may not affect your projected annual 25% growth. Should we shoot higher? Or maybe 15% is more realistic based on the new developments. The key is to update your forecasts regularly to ensure accuracy as you plan. Especially if you are a small business, it is important to be cognizant because you will likely operate and make decisions differently if you are expecting a 25% growth rate vs. a 5%. It can have a much more dramatic impact on your overall business.
The ability to improve forecasting and prediction is in contrast to another book I really enjoyed reading Nassim Taleb’s Black Swan. In the book, the authors actually address this as well noting that Taleb’s ideas about how it’s next to useless to make forecasts because the “Black Swan” events (unpredictable) have the largest impact. The idea is that these Black Swan events have the profound change yet there is no way to predict them or model for them rendering those forecasts that do not take them into account almost worthless. Tetlock & Gardner do a good job of noting while this may be true – some of the black swans may be more predictable if we are able to tackle smaller questions in bundles with accuracy that may in turn point to the likelihood of a Black Swan event.
I really enjoyed this book a lot and found value in how I think about my opinions and mental forecasts. As I build out my real estate portfolio and certain things happen – it helps to update my forecasts and timelines for acquiring property. If we experience a major crash – my timelines may move up as assets become less expensive or if I come across a new market opportunity. I need to be cognizant of how it affects my timelines and forecasts for the goals I have.
Other Book Reviews: