When the model is right but the outcome is wrong

Nate Silver is on record (somewhere) as saying that he was greatly relieved when the most likely outcome of the 2012 election was the actual outcome. He had build a model of what was going to happen that represented the probabilities of certain occurrences. People incorrectly interpreted this as predictions that could be right or wrong. They likely did this because we’re in a period of punditry, where we value people on the basis of their predictions rather than their insights.  Nate’s model indicated likely and less likely outcomes, but any of those could have occurred. The problem was that if Romney had won the election, people would have incorrectly assumed that Nate was wrong.  Sometimes, the less likely occurrence occurs.

Last weekend I saw this first-hand.  Over the course of several weeks I had built a set of Monte Carlo simulations of the provincial NDP convention. Every few days I was generating 150,000 possible convention scenarios based on polling data, reported support, and actual events (such as who had already voted and who had not). Fundamentally, each of these simulations was a possible outcome based on a certain selection on the probability distribution of convention outcomes. Each ballot was simulated, with simulated voters moving between candidates as candidates were removed or dropped off the ballot. The movement of these voters reflected what we knew about them.

In the end, after we had results from the first ballot, 70% of my scenarios predicted a Meili win.  Of course, that meant that in 30% of them, Broten won.  The less likely outcome is what actually came to pass.  There are two frustrations here. The first is that it’s difficult to verify if the models were actually correct.  Intuitively (and in most conventions), downballot support follows first ballot support. It’s a rule of thumb in conventions.  I ignored that rule of thumb, however, and allowed my simulations to do what was possible, not what was predicted.  I believe that was probably the right way to go, but without doing this 100 times in the next year or two, I can’t verify that my less-pundit-based method was a better model than the simple rule-of-thumb applied by other members of the campaign team.

The second problem is that many people will assume that I was wrong. I may not have been.  Just because the less likely outcome actually came to pass doesn’t mean that the model was incorrect. In fact, it’s agnostic as to the correctness of the model by itself. It’s just one data point. Fortunately, I worked with a highly educated campaign team open to understanding statistics (or at least trusting me when they didn’t). So this criticism hasn’t come from the campaign. I feel like I keep having to justify myself, though. Perhaps I don’t.

By the way, the first ballot results were within 2% points of what we considered the most likely outcome. That was within the error margin of most of the data that we had collected to that point.  Of course, that’s just one data point as well.


One thought on “When the model is right but the outcome is wrong

  1. Pingback: Head Tale - Some Thoughts On The Role of Social Media in the Sask NDP Leadership Race, Written on the Occasion of Twitter’s 7th Anniversary in 144 Chars

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s