Must Read Paper On Overconfidence
Via the indispensable Tyler Cowen, a new paper from Johnson and Fowler explores whether overconfidence is, in fact, adaptive. They show that it it is under some very reasonable assumptions. They model competition for resources as a two-player game and then analyze the evolutionary dynamics of populations playing this game.
The basic result is that overconfidence is beneficial in proportion to two factors: (1) the size of the payoff relative to the cost to play and (2) uncertainty about competitor capabilities. There are two optimal strategies for a population, overconfidence (which minimizes unclaimed resources) and underconfidence (which minimizes conflict costs). Unbiased self-perception is always dominated by these strategies. However, an overconfident person can successfully invade an underconfident population while the reverse is not true. So overconfidence is the stable solution.
The direct implication is that resources get destroyed. It is optimal for an individual to be overconfident, but then he ends up fighting with other overconfident individuals, which imposes costs. If you think about it for a minute, this is a pretty important fundamental problem. All of the big societal decisions we face have potentially big payoffs (or avoidance of costs), but it’s really unclear who has the best expertise to make a recommendation. So we get a bunch of “experts” telling us they are absolutely right.
Note that if it is public knowledge how “good” someone is, the “overconfidence premium” goes to zero. This is why forcing experts to make public predictions is so important. Then you can figure out how good they really are.
Possible Insight 
what a blogging coinkidink.
I think we only exacerbate the problem by handing out call options to decision makers. Isn’t part of the problem that it’s hard to distinguish a correct overconfident bet from a correct bet?
Alex
September 25, 2009 at 1:08 pm
I like point that Michael Sandel made in his Reith Lectures (BBC institution) and probably has made elsewhere.
Where expertise is available, politicians should bring experts into the public debate (and referee it).
Predictions are not what is required because they are often not the point. We cannot predict the short term responses of complex systems or motivated actors.
The debate should be intended to draw out those factors that are relevant and those the range of normal variability. When expertise is put on the public table, which is the internet these days, it will get reviewed by the cognitive surplus. And we may arrive at two conclusions.
1. Too complex too predict or decide. Therefore a huge range of outcomes is acceptable.
2. Too important to complicate beyond a simple value statement – eg. to say to a banker – thou shall not fail. So if you want to fail please do not become a banker. If you fail, the only question will the extent of the offense and the severity of our retribution. A moral decision in other words.
Jo Jordan
September 27, 2009 at 8:50 am