Moneyball for Tech Startups: Kevin’s Remix
Several people have pointed me to Dan Frommer’s post on Moneyball for Tech Startups, noting that “Moneyball” is actually a pretty good summary of our approach to seed-stage investing at RSCM. Steve Bennet, one of our advisors and investors, went so far as to kindly make this point publicly on his blog.
Regular readers already know that I’ve done a fair bit of Moneyball-type analysis using the available evidence for technology startups (see here, here, here, here, here, and here). But I thought I’d take this opportunity to make the analogy explicit.
I’d like to start by pointing out two specific elements of Moneyball, one that relates directly to technology startups and one that relates only indirectly:
- Don’t trust your gut feel, directly related. There’s a quote in the movie where Beane says, “Your gut makes mistakes and makes them all the time.” This is as true of tech startups as it is of baseball prospects. In fact, there’s been a lot of research on gut feel (known in academic circles as “expert clinical judgement”). I gave a fairly detailed account of the research in this post, but here’s the summary. Expert judgement never beats a statistical model built on a substantial data set. It rarely even beats a simple checklist, and then only in cases where the expert sees thousands of examples and gets feedback on most of the outcomes. Even when it comes to evaluating people, gut feel just doesn’t work. Unstructured interviews are the worst predictor of job performance.
- Use a “player” rating algorithm, indirectly related. In Moneyball, Beane advocates basing personnel decisions on statistical analyses of player performance. Of course, the typical baseball player has hundreds to thousands of plate appearances, each recorded in minute detail. A typical tech startup founder has 0-3 plate appearances, recorded at only the highest level. Moreover, with startups, the top 10% of the startups account for about 80% of the all the returns. I’m not a baseball stats guy, but I highly doubt the top 10% of players account for 80% of the offense in the Major Leagues. So you’ve got much less data and much more variance with startups. Any “player” rating system will therefore be much worse.
Despite the difficulty of constructing a founder rating algorithm, we can follow the general prescription of trying to find bargains. Don’t invest in “pedigreed” founders, with startups in hot sectors, that have lots of “social proof”, located in the Bay Area. Everyone wants to invest in those companies. So, as we saw in Angel Gate, valuations in these deals go way up. Instead, invest in a wide range of founders, in a wide range of sectors, before their startups have much social proof, across the entire US. Undoubtedly, these startups have a lower chance of succeeding. But the difference is more than made up for by lower valuations. Therefore, achieving better returns is simply a matter of adequate diversification, as I’ve demonstrated before.
Now, to balance out the disadvantage in rating “players”, startup investors have an advantage over baseball managers. The average return of pure seed stage angel deals is already plenty high, perhaps over 40% IRR in the US according to my calculation. You don’t need to beat the market. In fact, contrary to popular belief, you don’t even need to try and predict “homerun” startups. I’ve shown you’d still crush top quartile VC returns even if you don’t get anything approaching a homerun. Systematic base hits win the game.
But how do you pick seed stage startups? Well, the good news from the research on gut feel is that experts are actually pretty good at identifying important variables and predicting whether they positively or negatively affect the outcome. They just suck at combining lots of variables into an overall judgement. So we went out and talked to angels and VCs. Then, based on the the most commonly cited desirable characteristics, we built a simple checklist model for how to value seed-stage startups.
We’ve made the software that implements our model publicly available so anybody can try it out [Edit 3/16/2013: we took down the Web app in Jan 2013 because it wasn’t getting enough hits anymore to justify maintaining it. We continue to use the algorithm internally as a spreadsheet app]. We’ve calibrated it against a modest number of deals. I’ll be the first to admit that this model is currently fairly crude. But the great thing about an explicit model is that you can systematically measure results and refine it over time. The even better thing about an explicit model is you can automate it, so you can construct a big enough portfolio.
That’s how we’re doing Moneyball for tech startups.
Possible Insight 
Nice post, Kevin. I knew you’d be able to summarize your approach and relationship to quantitative analysis of entrepreneurs and opportunities better than I could. Can’t wait until you have some actual portfolio results to analyze and compare against Cambridge returns.
Steve Bennet
September 28, 2011 at 12:54 am
[…] at the moment as we look at the story behind Moneyball. Dick addresses this specifically in a recent blog post, and how it coincides with the methods of RSCM: “Despite the difficulty of constructing a […]
Right Side Capital skips the investment gut check, opting for an algorithm instead | Pro Blog Project
October 5, 2011 at 8:24 pm
[…] at the moment as we look at the story behind Moneyball. Dick addresses this specifically in a recent blog post, and how it coincides with the methods of RSCM: “Despite the difficulty of constructing a […]
OnlineMagazine » Blog Archive » Right Side Capital skips the investment gut check, opting for an algorithm instead
October 5, 2011 at 11:42 pm
I read Nick Shalek’s contributed article on Techcrunch titled “Thankfully, Software Is Eating The Personal Investing World.” Is Nick’s advice as appropriate for accounts worth millions of dollars, as it is for small accounts?…
We use a combination of founder resumes and actual progress in the current startup. It is a linear model, albeit with lots of variables. But even linear models with uniform weights almost always outperform gut feel (see Hastie and Dawes work on expert …
Quora
June 26, 2012 at 11:27 pm
Kevin, I really like your approach to investing. I am currently diving into ways to reduce gut-based decisions in innovation. Is it possible to get the excel-file to use as inspiration?
Pieter van der Boog
September 7, 2015 at 9:24 am
Thanks for the note Pieter. We don’t disclose our particular algorithm. But I describe the general procedure by which you could easily build your own in this post:
https://possibleinsight.com/2012/09/17/valuing-seed-stage-startups/
Kevin
September 7, 2015 at 5:06 pm
Thanks! If you’re interested in staying updated, let me know.
Pieter van der Boog
September 8, 2015 at 1:59 pm
Kevin, belatedly just read your article. After spending the last 20 years in Institutional Finance, I believe the aforementioned concept is known as an “edge”. I recently began work with an “outlier” software group which has created several models which enable one to evaluate a Founder or Team by reading their unstructured text. The ability to see whether a CEO is aligned with his position and his purpose and what motivates him, are some of the benefits of this software. In another words, the ability to objectively evaluate key members of a given start up is possible.
Andrew Fielding
October 13, 2015 at 9:35 pm
Interesting, but I am skeptical. I have a fair amount of background in statistical learning and human decision making. I would be very surprised if this software provided a return edge at the pre-seed stage based purely on automated text evaluation. I would need to see results from a well designed experiment to be convinced.
Kevin
October 13, 2015 at 9:50 pm
Investing is not a choice between data or gut, it is about our choice to be involved or not. When we choose to be involved, we are involved from our experience and our knowledge. This choice ultimately evolves us and the investments we fund.
Andrew Fielding
October 14, 2015 at 10:02 pm
I’m not sure how this is related to your comment about software that can purportedly evaluate founders from text samples.
In any case, I don’t see how being involved settles the question of data or gut as the input to the investment decision. At some point, you either put money in or you don’t. Either that action is random or it’s the outcome of a deliberate process. Such a process can take a variety of inputs. I’m not sure how one could ever escape having to chose among those inputs and the weight to give them.
Kevin
October 14, 2015 at 10:13 pm
The software that evaluates Founders from text samples (CV’s or Executive Summaries) assists in making the investment decision deliberate. It is analogous to having another set of eyes in the decision making process to probe the strong and weak points of the founders, their plans and alignment a quantifiable manner. It seems that such insights would be integral to the type of approach you have.
Andrew Fielding
October 14, 2015 at 11:24 pm
If there is some evidence demonstrating the predictive validity of this software for founder performance or even general job performance, I’d be happy to take a look.
Kevin
October 15, 2015 at 2:54 am
The software has been used in more than a dozen “studies”. Most of this was done as paid consulting work for HR departments, Forensic Analysis / Security, and e discovery. We have also used some of these “tools’ in vetting deals with a South Eastern Angel Network. Do you have interest in building some of this evidence?
Andrew Fielding
October 15, 2015 at 2:12 pm