Don't Cop Out on Knightian Uncertainty
I apologize for the posting lull. I actually spotted an issue than I wanted to address a few weeks ago, but I’ve been pondering how to approach it. It’s pretty complicated and subtle. I even ran a couple of drafts by Rafe to refine my thinking. So please bear with me.
As I’ve mentioned before, I am a fan of Dave Zetland. When I saw him propagate what I think is a fundamentally false dichotomy in this post, I knew I had to take on the concept of Knightian uncertainty. It crops up rather often in discussions of forecasting complex systems and I think a lot of people use it as a cop out.
Uncertainty is all in your mind. You don’t know what will happen in the future. If you have an important decision to make, you need an implicit or explicit model that projects your current state of knowledge onto the space of potential future outcomes. To make the best possible decision, you need the best possible model.
Knight wanted us to distinguish between risk, which is quantifiable, and uncertainty, which is not. If you prefer the characterization of Donald Rumsfeld, risk consists of “known unknowns” and uncertainty consists of “unknown unknowns”. This taxonomy turns two continuous, intersecting spectra into a binary categorization.
There are some random events where we feel very confident about our estimation of their likelihood. There are other random events where we have very little confidence. These points define the confidence spectrum. Moreover, there are some events that we can describe very precisely in terms of the set of conditions that constitute them and the resulting outcomes. Others, we can hardly describe at all. These points define the precision spectrum. There’s obviously some correlation between confident likelihood estimation and precise event definition, but it’s far from perfect. Unsurprisingly, trying to cram all this subtlety into two pigeon holes causes some serious analytic problems.
The biggest problem is that proponents of the Knightian taxonomy say that you can use probability when talking about risk but not when talking about uncertainty. Where exactly is this bright line? If we’re talking about a 2 dimensional plane of confidence vs precision, drawing a line and saying that you can’t use probability on one side is hard to defend.
Now, the Knightians do have a point. As we get closer to the origin of the confidence vs precision plane, we enter a region where confidence and precision both become very low. If we’re looking at a decision with potentially tremendous consequences, being in this region should make us very nervous.
But that doesn’t mean we quit! “Knightian uncertainty” is not a semantic stopsign. We don’t just throw up our hands and stop analyzing. As I was writing this post, Arnold Kling pointed to a new essay by Nassim Taleb of The Black Swan fame. Funnily enough, Taleb has a chart very much like the confidence vs precision plane I propose. His lower right quadrant is similar to my origin region. Taleb says this area represents the limits of statistics and he’s right. But he still applies “probabilistic reasoning” to it. In fact, he has a highly technical statistical appendix where he does just that.
Before I saw Taleb’s essay, a draft of this post included a demonstration that for any probabilistic model M that ignored Knightian uncertainty, I could create a probabilistic model M’ that incorporated it. M’ wasn’t a “good” model mind you, I merely wanted an existence proof to illustrate that we could apply probability to Knightian uncertainty. The problem of course was that the new random variables in M’ all reside in the danger zones of Taleb’s and my respective taxonomies. But Taleb’s a pro and he’s done a far better job than I ever could of showing how to apply probabilistic reasoning to Knightian uncertainty. So I won’t inflict my full M vs M’ discussion on you.
The key take home point here is that you can in fact apply probability to Knightian uncertainty. Of course, you have to be careful. As Taleb wisely notes in the essay, you shouldn’t put much faith in precise estimates of their probability distributions. But this is actually good advice for all distributions, even well behaved ones.
Back when I was in graduate school, my concentration was in Decision Analysis, which included both theoretical underpinnings and real-world projects trying to construct probabilistic models. I dutifully got my first job applying this knowledge to electrical power grid planning. What I learned was that you should never rely on the final forecast having a lot of precision. Even if you’re dealing with well behaved variables. Because if you put a dozen or so well behaved variables together, the system still often becomes extremely sensitive.
However, “doing the math” leads to a much deeper qualitative understanding of the problem. You can identify structural deficiencies in your model and get a feel for how assumptions flow through to the final result. Most importantly, you identify which variables are the most important and which you can probably ignore. Often, the variation in a couple will swamp everything else.
For example, one of insights Taleb identifies is that you should be long in startup investments (properly diversified, of course). That’s because the distribution of Knightian outcomes is asymmetric. Your losses are bounded by your investment but your gains are unbounded. Moreover, other people probably underestimate the gains because we don’t have enough data points to have seen the upper bounds on success. There’s a bunch of somewhat complicated math here having to do with the tendency to underestimate the exponential parameter in a power law distribution, but most numerate folks can understand the gist and the gist is what counts. The potential of very early startups is systematically underestimated. Now, this isn’t just some empty speculation. I’m actually taking this insight to heart and trying to create a financial vehicle that takes advantage of this.
I’ll give you an example from another of my favorite topics, climate change. I would love for someone to try and apply this sort of analysis to climate change outcomes. We have a power law distribution on our expectations of climate sensitivity for both CO2 warming and aerosol cooling. We also have a power law distribution on our expectations of natural temperature variability. If someone really good at the math could build a model and run the calculations, there are some very interesting qualitative questions we might be able to answer.
First, is the anthropogenic affect on future temperatures roughly symmetric, i.e., could make things colder or warmer? Second, and more importantly, is the anthropogenic contribution to variability significant compared to natural variability? If it isn’t, we should budget more for adaptation than mitigation. If it is, the reverse. But to get these answers, we need to be able to manipulate symbols and make calculations. Probability is the only way I know to do this. So saying you can’t use probability to tackle Knightian uncertainty seems like a cop out to me. How else are we suppose to make big decisions that allocate societal resources and affect billions of people?