So don’t ditch the models, be humble with them; include wide error bars, and space within the model for unknown things to happen. And, of course, make sure you can afford to lose whatever you’re betting, even if the bet seems a good one (which Viniar, Goldman Sachs et al signally failed to do). I think this is roughly what it means to be “antifragile” and “robust” and “resilient”, terms K&K throw around a lot. Expect things to be unpredictable. Make room for what they call “radical uncertainty”.
But K&K’s prescription goes further than that. Instead of putting figures on things, they say, forecasters should instead be willing to ask “What is going on here?” and willing to say “I don’t know”. Obama and the Abbottabad decision is an example they return to again and again.
His advisers were giving him percentage likelihoods that the man in the house was bin Laden, ranging from 95% to about 30%. K&K praise him for, instead of combining all those probability judgments, simply making a decision. Obama is quoted as saying “it’s 50/50”, but K&K say he did not mean “probability = 0.5”. Either he’s in the house or he’s not; you don’t know; it’s radically uncertain, so there’s no point putting numbers on it. Instead, you say “I don’t know.”
But: of course you can put numbers on these things! We can say things like, there are about 32 million households in Pakistan. So if we are confident that bin Laden is in Pakistan and we were to pick one at random, we’d have about a one in 32 million chance of getting the right house. That is, if you like, the base rate of bin Ladens in Pakistani households. We can adjust that number up with new information; sightings, web chatter, or suspicious movements of goods, say. You use judgment and probabilistic reasoning to put numbers on your intuition.
I’m being simplistic, but that’s the basic idea of Philip Tetlock’s Good Judgment Project (GJP), which I’ve written about before, and which is kind of famous at the moment because Andrew Sabisky did well enough at it to earn the title “superforecaster”. The GJP superforecaster prediction teams systematically outperform CIA analysts and others at forecasting things like “Will Scott Morrison cease to be prime minister of Australia before 1 January 2021?” or “When will total cases of Wuhan coronavirus (COVID-19) in California reach 25 or more?”
They do it by putting numbers on things: say, 70% likely or 27% likely. And then they make loads of predictions, and they see if the 70% predictions come in 70% of the time, and so on. If they come in less often, they’re overconfident; if they come in more often, they’re underconfident.
Tetlock’s work is briefly mentioned in Radical Uncertainty, but only to be dismissed. They say that these predictions are only predicting well-defined, falsifiable events – “Who will win the 2020 Bolivian presidential election?” – and so cannot allow for radical uncertainty, for the possibility that some totally other thing will happen. These techniques, say K&K, cannot put useful numbers on big, difficult questions. ‘The sensible answer to the question ‘Will there be another global financial crisis in the next 10 years?’,” they say, “is ‘I don’t know.’”
But this seems straightforwardly wrong. For a start, GJP-style use of numbers can allow for radical uncertainty. The GJP’s Bolivian presidency predictions are: Jeanine Áñez, 27% probability; Luis Arce, 31%; Luis Camacho 16%; Carlos Mesa 23%; and “Someone else or another outcome, 3%”. They explicitly include in their model the possibility that something totally other will happen. Perhaps the space for uncertainty should be bigger; that’s an empirical question. But it’s there.
If, as K&K seem to be suggesting, the only rational response to radical uncertainty is to say “I don’t know,” then they should refuse to put a number on any outcome: maybe Jeanine Áñez will win; maybe Luis Arce will; or maybe a robot dinosaur and the ghost of Freddie Mercury will materialise in the senate building and expound a new theory of government via the medium of interpretive dance. We just don’t know.
And these sorts of predictive methods do work on big questions like “Will there be another financial crisis?” The methods are agnostic to the size of the question, although the greater the complexity the greater the uncertainty, of course, and you have to be able to define the question (but if you can’t define the question, then what are you even asking?). You might start by examining the base rate of global financial crises per decade – depending on your definition, there have probably been two in the past century, so you might start with a base rate of 20%. And from there, you look at the fundamentals relevant to the current situation, and adjust your estimate with that.
Tetlock’s work seems to show, empirically, that this method is more effective than alternatives. He was inspired to start the GJP partly because of the preponderance of “vague verbiage” among predictions by politicians and pundits; “a recession seems likely”, OK, but is “likely” a 10% chance or a 90% chance? It becomes impossible to say when someone is wrong if they can hide behind unclear terms. So it’s darkly amusing to see K&K explicitly recommending a return to vague verbiage: to say “it is likely” rather than “it’s 60% likely”, or “it seems probable” rather than “there is a 75% chance”.
I also struggled to determine what K&K suggested we should do instead. They give lots of examples of people using models and numbers badly, but if you want to determine how much money to save for a pension, or what percentage of GDP to spend on social care, you need numbers. They quote Keynes, approvingly, on how economics is not physics and how you need intuition and judgment and domain knowledge, not just rules. Which is all true, and I’m a sucker for anyone quoting Keynes. But if their prescription is “have really good intuition and judgment and domain knowledge” – basically, “be really good at this” – then I’m not sure how helpful that is.
I’d be very happy if their suggestion was simply that, empirically, people are too keen on spuriously precise numbers, and ought to be more humble with their forecasts. That would make sense. But the repeatedly stated message of this book is that we shouldn’t use numbers at all. Instead, they say again and again and again, we should ask “what is going on here?” and say “I don’t know”.
As I said, it is vanishingly unlikely that K&K are as stupid as this book seems to me. I tried to think about what the assumptions that we didn’t share might be, and the one that seemed likeliest was that we simply hang out in very different circles.
I see a lot of journalists and pundits making predictions about the future. And the problem is rarely that they are too willing to make robust, falsifiable predictions with explicit numbers. The problem is they talk a load of guff about X being likely, and then when X doesn’t happen, just quietly pretend they never said anything. If they were forced to give percentage estimates for things it might make them face up to how little they actually get right.
K&K, on the other hand, presumably hang around with economists and finance types, the sorts who collateralised all their subprime mortgage debt in the early 2000s and felt very pleased with themselves. Could it be that lots of them really do build statistical models of the world which say “lots of poor people in the Midwest defaulting on their mortgage at the same time is a once-in-several-trillion-lifetimes-of-the-universe event”, and think, “OK, sounds legit”? Could it be that if these masters of the universe were writing a forecast of the Bolivian presidential campaign, their percentages for Áñez, Arce, Camacho and Mesa would add up to exactly 100, with no room for the unforeseeable at all? Maybe K&K are speaking to those people, and they’re common enough in finance and economics to be the norm rather than the exception.
It would be terrifying to learn that doctors actually think that “drug trial finds positive result” means “this drug works for all patients in all circumstances”. Similarly, if K&K are right in thinking that economists and finance types need to be told that their models are not 100% precise representations of reality – if they think they never need to put a term for “maybe this model is wrong” into their model, and are just happy to bet all our money on them definitely not having forgotten to carry the 2 at some point in their big spreadsheet – then that, too, is completely terrifying. If K&K are right that this book needed to be written, then: holy shit.
It doesn’t make me want to stop using percentage probabilities. It does, however, make me think that, if the financial system really is full of maniacs like that, then the chance of a massive financial crisis in the next decade is a lot higher than 20%.
Join the discussion
Join like minded readers that support our journalism by becoming a paid subscriber
To join the discussion in the comments, become a paid subscriber.
Join like minded readers that support our journalism, read unlimited articles and enjoy other subscriber-only benefits.
Subscribe