By James Kwak
I think Nate Silver does a good job aggregating polls to make meaningful quantitative predictions about upcoming elections. But as he said himself shortly before the election, if the polls he relies on are systematically biased, then his forecasts are going to be off.* Many people have noted that Silver (and other quantitative poll aggregators like Sam Wang) correctly predicted an Obama victory and the outcomes in most if not all states.
But the fact remains that Obama did modestly better than the polls, and hence the poll aggregators, expected (not to mention than the Romney campaign expected). We shouldn’t read too much into this, as even where Obama significantly overperformed—like in Iowa, where Silver forecast a 3.2 percentage point victory and the actual came in at 5.7 points—the results were within the confidence intervals. But it’s also possible that the polls really were systematically biased, only they were biased against Obama—not against Romney, as conservative pundits were claiming in the last days.
Why would that be? One possibility is turnout. Many polls incorporate a likely voter model, which weights the sample to try to approximate the expected composition of the electorate. Gallup’s problem, I believe I read somewhere, was that they expected the electorate to be whiter than it turned out to be. In retrospect, we have anecdotal evidence that the electorate was younger and less white than many people (such as Paul Ryan) expected. And one common explanation is that this was due to the strength of the Obama campaign’s get-out-the-vote operation.
One piece of evidence for this theory is that Obama’s performance relative to expectations was especially good in the swing states, where you would expect him to have devoted most of his GOTV efforts.** Of the nine major swing states (in order of competitiveness, according to Silver, Florida, North Carolina, Virginia, Colorado, Iowa, New Hampshire, Ohio, Nevada, and Wisconsin), Obama beat Silver’s poll-based forecast in seven; on average, including the two states where he underperformed, he beat the polls by 1.1 percentage points. Of the other forty-one states and the District of Columbia, by contrast, Obama overperformed in only twenty-three, or just over half, and on average he beat the polls by only 0.4 percentage points.
Now, this is not something that Nate Silver was supposed to predict. Just before the election, his forecast is based almost entirely on the polls. And he freezes his model several months before the election, precisely because he doesn’t want it to be influenced by subjective judgments.
But this is exactly the kind of thing that journalists (and their subspecies known as pundits) are supposed to predict. That Obama would have the best turnout operation ever is not something that Nate Silver could predict in January. But all those people who don’t believe in polls, who think that old-fashioned beat reporting and gut instinct are the way to predict elections, could have done the work to figure out that Obama had the best turnout operation ever. Based on that research, the pundits and the political experts could then have said, “I expect Obama will do better than the polls, because the current generation of likely voter models does not take into account the strength of Obama’s turnout operation.” And that would have added value to what Silver was doing.
In other words, this was an election where old-fashioned reporting and punditry could have provided some insight into the outcome. But they didn’t, because the pundits were too busy spinning false stories about momentum (which were provably false, since momentum does show up in polls) instead of looking for relevant facts.
In theory, poll aggregation should not be the last word in election forecasting; there should be a place for political expertise. But with the “experts” we’ve got now, it is the last word.
* Does anyone know why all posts from the first six days of November have vanished from Silver’s blog?
** You would also expect him to have devoted most of his other efforts in those states, but those activities, such as TV advertising, should have shown up in the polls before Election Day.
The Baseline Scenario 
Isn’t fraud the simplest explanation?
@Nemo – I think free will is the simplest explanation. Elections are like a football game – the final score was achieved by the actions in that time frame. When Nate Silver can predict the score for any and all games in advance, well, then the Gaming Commission will come after him :-)
It’s OVER after the fat lady sings….ie, after FREE WILL BEINGS vote….
Fraud IS the simplest explanation for simpletons and nitwits, perhaps.
“Romney pumping gas disheveled, priceless.”
This sounds like a Paul Krugman article. Come up with something truly original next time. And Happy Thanksgiving!
@James: I can see all the posts from early November in my rss feed, with links that open the full articles, even though the articles aren’t listed on the nytimes page you linked to. Here’s the post from November 6. You can get to the rest with repeated use of the “previous” links at the bottom.
http://fivethirtyeight.blogs.nytimes.com/2012/11/06/live-blog-the-2012-presidential-election/
> Does anyone know why all posts from the first six days of November have vanished from Silver’s blog?
The first post that doesn’t show up is the election liveblog, so my guess is the listing code chokes on that entry and stops. You can still see earlier posts if you visit the url for each individual day:
http://fivethirtyeight.blogs.nytimes.com/2012/11/1
etc.
In other countries, when polls diverge from election results, we call that a priori evidence of fraud.
But only for other countries.
Nemo: you really don’t understand how this stuff works very well, do you? I mean, what part of ‘within the confidence interval’ is unclear?
The posts are still there. Your link filters by month, yes, but there’s still a result limit per page. At the bottom there’s a link to “older posts” that takes you to http://fivethirtyeight.blogs.nytimes.com/2012/11/page/2/ and the rest of the November posts.
Nate Silver.
No one did it better.
I suspect I understand this stuff a lot better than you think. (Dr. Kwak is not the only person who attended the Math Olympiad Program.)
If everything is “within the confidence interval”, then the entire post is pointless. On the other hand, if there is something to explain — which I suppose there is, since Dr. Kwak spent half a dozen paragraphs here on hypothetical explanations — then the simplest explanation is the old-fashioned one.
Nate Silver is a guy I have a great deal of admiration for. He has outstanding skills with numbers and math. And the fact he’s starting with good H*brew genetics and puts those good H*brew genetics to good use doesn’t hurt either. Just his last name would make me pop my ears up on what he has to say.
That being said, as much as I admire the guy and how accurate his methods seemed to turn out (better than anyone I’m aware of), I still have a hard time respecting his method. Why?? After saying all the above and then I say I don’t respect his method?? How can you really respect any stats method that isn’t making first hand/eye-witness efforts to assure the quality of the sampling data (I mean the samples themselves) ??? The answer is, you cannot.
Norah O’Donnell won an Emmy for her Election Night coverage in 2008. Which is the best reason I could come up with to rationalize putting this video up again. I’m not sure if Gwen Ifill has done any follow-ups on this frightening glimpse of Professional Seemer’s past:
“In theory, poll aggregation should not be the last word in election forecasting; there should be a place for political expertise.”
It really is amusing how closely the Nate Silver Versus the Pundits discussion mirrors the analogous discussion that was going on about baseball six or seven years ago.
FIRE JOE MORGAN LIVES.
But Obama’s GOTV effort was not as effective as 2008. John McCain got more votes than Romney. From my memory, turnout was above expectation in 2008 also. It’s funny that polls used pretty much assumptions about turnout rates, when they could have been polling specifically about turnout. I don’t see any other way to analytically incorporate turnout – either pro-Obama or pro-Romney, the pre-election turnout arguments were just stories. I donated to Obama and was polled by phone at least 4 times – I imagine by different poll organizations because of the nature of the questions. I thought there had to be a problem with sampling if I was being targeted so frequently.
Is this perhaps a problem similar to doing better than the market indices? While some people can, it is incredibly hard to keep biases from creeping in.
Quite right, Nemo. Fraud is the simplest explanation.
Here’s a good example of the merits of simplicity: A fellow by the name of Lamark had a very simple explanation for the evolution of species. For example, Lamark theorized that giraffes evolved to have long necks by stretching higher into the trees to reach fruit. This lengthened their necks, an acquired trait they passed on to their offspring who did the same over many generations until we have the animal we know today. Very simple and a bit off la Marc, if you will. Most of us know that Darwin’s more complex theory is correct.
Per Nemo: “In other countries, when polls diverge from election results, we call that a priori evidence of fraud.”
Nemo, you left out the word “exit”, as in “exit polls”, which are normally very reliable. I have not found Nate Silver’s analysis of exit polls yet.
“But only for other countries.”
True, that. I remember an election where one territory – which happened to have one candidate’s brother holding executive office in that region – had exit polls that said one thing and official results that said another. Then some activist judges took sides so the scandal would become moot. If we had seen that in a third-world country, we would have known the election results were fraudulent.
…
“I suspect I understand this stuff a lot better than you think. (Dr. Kwak is not the only person who attended the Math Olympiad Program.)”
It should be simple to point out any mistakes in the arithmetic. Go ahead.
Fraud is NOT the simplest explanation for inaccurate predictions. The simplest explanation for inaccurate predictions is “The predictions were inaccurate.” It’s then the predictor’s job to figure out why, and refine his predictions for next time.
Fraud WOULD BE the simplest explanation when (for example). the total number of votes in a precinct is larger than the number of adults in the precinct.
As pointed out by Ogden Wernsgrom, unexpected divergence between results and polling implying potential fraud only applies only to exit polling. I am not sure why they didn’t teach this in whatever Math Olympiad Nemo is talking about, but that is because a good exit poll is essentially a random sample of the votes already cast, not a sample of people who may or may not vote for reasons that are difficult to model. As Kwak noted, just how much better Obama’s GOTV effort was than Romney’s might not have been clear to many observers, especially since the Romney had plowed a lot of resources into a very sophisticated (but new and less well tested) computerized GOTV system called ORCA. Multiple accounts in the press pointed out that the system performed very poorly on election day, so in swing states you had the Obama GOTV effort versus a poorly functioning and apparently buggy new system. That had to be worth something.
There were two other factors at work that may have had the effect of raising turnout above expectations based on the models:
1) intense media coverage of voter suppression efforts in some of the swing states for weeks and months leading up to the election angered African-American and Latino voters enough to mobilize them in greater numbers than predicted,
2) many of the models are based on surveys that under poll cell-only voters who tend to be young and minority voters
Interesting to hear the endless sobs and crying from Republicans over their complete and absolute failure of Nov. 6, 2012. There is one very good thing about this blindness caused by excessive John Boehner like tears (although more sincere than Boehner’s). The deeper they bury themselves into their pathetic nonsensical rationalizations, the less likely they are to find the solution to election night impotence.
And on that note, I give you Jon Chait’s breakdown on the commentary of John Podhoretz. Podhoretz deserves “The David Brooks Tribute Award for Empty Skirt Commentary” for this one:
http://nymag.com/daily/intel/2012/11/podhoretz-has-awkward-postelection-epiphany.html
Followed by the empty skirt commentary itself. I strongly discourage you from clicking on such rubbish, but if curiosity (or the need for laughter) compels you:
http://www.commentarymagazine.com/article/the-way-forward/
I’m currently reading Nate’s book and I admire that he realizes he has to continually assess the effectiveness of his methods, he’s not dogmatic. It’s easy to blame Obama’s turnout operation as the reason he won after the fact, but maybe the news organizations that the old fashioned beat reporters work for were so drunk on all the Republican ad money, they wouldn’t have been interested in old fashioned get out the vote drives. Don’t kill the golden goose of the media (particularly, television) industry.
I used to avoided watching MSNBC because I thought they’d just reinforce my opinion, like Fox does on the Right. But election coverage on the other cable and network channels was (mostly) so poor that I turned to MSNBC. Those “kids” get it. They followed Nate and Sam Wang closely and they did their homework. They love politics but aren’t such junkies that they ignore issues and governance. Add heavyweights like Andrea Mitchell, Chris Matthews, Steve Schmidt among others, and it’s a thinking person’s news source. The Right will never match this. This take is nuanced and complex. The Right must have a simplified version of the world spoon-fed to them. That’s why it’s so easy to keep them corralled in the bubble.
I am sorry to say you might be right about the value of ordinary punditry. I am still hoping that we can smash the life out of mindless punditry (99%) through a greater understanding of polling and statistics.
The criticism of the media failing to do its job seems to miss or not pay enough attention to the political economy of the mainstream media in the U.S., where (political, economics, and international affairs) reporters are responding principally to two incentives: 1) Broadcast revenues 2) Access to policymakers. This causes them to make less accurate statements than their membership in The Fourth Estate presumably obliges. With all mainstream reporters more or less operating within this environment, it seems unreasonable for you to expect them to augment Nate Silver’s analysis (I doubt reporters care half as much about Nate Silver as the academically/mathematically inclined like myself do) in the fairly technical manner you describe. Even then, maybe a reporter did do exactly what you’re wishing they would have. But would Nate Silver have responded? Would the NYT have wanted him to? Perhaps if Nate Silver cared about accurately predicting elections more than anything else, he would join academia.
Folks. NEMO is on to something here. This type of statistical outcome (the high end of the confidence interval in so many states) is the equivalent of a statistical anomaly. Would the outcome of the election be changed if some of the less then reputable actions of GOTV not occurred. No. The real story is that the proposed fraud (discovered by the same statisticians that called the election) was not necessary. Obama won this straight out and did not need to bend the rules. The Republicans are in some serious trouble here from an electoral college perspective. They do not want to admit that we are living through the modern version of the late 1920’s and 1930’s. Can you say 16 straight years of Democratic presidencies. That is where this is headed. Only to end because of the high probability of a negative outcome (would be the same of either party).
Despite the incentives of ad revenue and access to policymakers, there are some reporters and analysys who manage to swim against the tide of conventional wisdom and maintain a significant degree of independence from the groupspeak of Washington DC. Matt Taibbi and James Fallows come immediately to mind.
Nonsense. The problem is with the survey methodology: the under-sampling of Obama voters due to the use of landline-only and robo calling, the likely voter models, and the fact that more partisan GOP polls were run than partisan Dem polls skewing the averages in some swing states.
If you believe there was significant fraud favoring Obama, cough up the evidence. This is a well-worn GOP canard. Accuse the Dems of the very practices you’re using.
The evidence is overwhelming. it was the GOP that attempted to rig the election through voter suppression tactics of many types in key swing states. In PA we have it on tape. In Ohio, Colorado and Florida, it’s plain as day. Even in my deep red Texas there were attempts to purge polls of allegedly dead but very-much-living voters in heavily Dem precincts.
Sorry, Charlie. The election just wasn’t close enough to steal.
@publiustex – you and Nate SIlver et al might be interested in reading the comments the hoi poloi posted in response to the news article on Huff and Post about Texas wanting to leave the USA…I would be truly interested in those comments being modeled into some kind of *math* prediction of future events.
LOL. Now we know where late night comics are getting their material from most of the time these days…
Annie,
Yes, if we were not the country’s laughingstock before Rick Perry’s madcap adventure last spring, we damn sure are now. Us Austinites have filed a counter-petition to the Texan-secession petition on the White House website, calling for Austin to secede from Texas if Texas secedes.
I say let ’em (us) go. No, insist they go. Good riddance to the treasonous SOBs. Without the gerrymandered Texas congressional delegation, the two GOP senators, and the GOP EVs, the US will turn solidly blue. The next time oil and gas prices drop or the builders create another real estate bubble, the Texas economy will tank, and by 2022 Hispanics, blacks, and moderate whites will become an electoral majority. Then we’ll petition for readmittance.
It’ll be painful for those of us living here, but one has to make sacrifices.
@ Can you say 16 straight years of Democratic presidencies. That is where this is headed. Only to end because of the high probability of a negative outcome (would be the same of either party).
Is that a 2 term outlook or a one term outlook. I mean, the stability of today’s executive branch has already been proven, not since the founders has the executive branch had 3 two term presidents in a row. I say we are headed back to the future, where we belong.
Well, except for FDR-Truman.
@publiustex – Unfortunately, the “underground” economy of illegal weapons sales and drugs has already become the dominant *commerce* that operates via tax free cash. That why the people who get to DC from Texas are BOLD FACED liars, thieves and murderers – with *holy book* in hand, no less. Anytime I make the lucid and factual observation that Jesus did NOT write the bible and even went to great lengths NOT to write anything down, that 360 head spin winds up and I end up being judged as a devil – or something :-))
Which is yet another reason I say that things are too far gone – no one addresses the “black market” economy in DC – which was SERIOUSLY involved in fraudclosure! – when it comes to tax reform or economic justice, do they?
No taxation without representation. How much more proof does anyone need that DC is AGAINST the normal minded, producing Middle Class?!
Lots of people, not just Texans, are wondering where to go to live in a rational and sane environment – that’s why all the comments were basically, “…don’t let the door hit your ass on the way out…” to the Texans.
Agree with you about the changing demographics of USA bringing about permanent change to USA standing in the world community. But that change won’t be the only one happening. EU is going to go *radical* about immigration issues and who can challenge their RIGHT to want to hang on to their EARNED culture?
Prime Directive is a very *wise* policy :-)