There’s been a lot of talk in the blogosphere today about why the polls underestimated Obama’s lead in the 2012 election. Obama won by four points while most polls showed him with a smaller lead. While those studying the polls in depth might find other reasons, I was not surprised as I had discussed possible reasons why the polls might be underestimating Obama’s lead prior to the election. These include under-counting Latinos and exclusive cell phone users. I also felt during the campaign that Obama had greater up-side potential than Romney if he could get his coalition out to vote. Since the election we have heard a lot about how good Obama’s ground game was, and how poor Romney’s was.
Pollsters adjust the raw numbers based upon their model of the actual voters. While conservatives accused the pollsters of being biased against Romney and predicting a higher percentage of Democrats in the electorate, the electorate in 2012 did turn out to be more favorable demographically to Obama than most predicted. Having a larger number of young and minority voters overlaps with the points I previously made about polls potentially underestimating Latino voters and cell phone users (who tend to be younger.) It certainly helped Obama that blacks turned out in a higher percentage than other minority groups. and possibly at a higher percentage than white voters.
Polls are also just a snapshot of a given point in time. The fact that Obama had a far better final week of the campaign might have given him a greater margin of victory than if the election was a week earlier. During the final week Obama benefited from favorable jobs numbers and favorable response to his handling of the storm hitting the east coast, including the pictures of him with Chris Christie.
Poll aggregation presented another issue when polls with different degrees of accuracy were lumped together:
Real Clear Politics began the practice of averaging polls before the 2002 midterm elections. RCP was joined by Pollster.com–which is now part of The Huffington Post–four years later. “Pollster started in 2006, and we were really building on what Real Clear Politics did,” founding Coeditor Mark Blumenthal said. The statistician Nate Silver began a similar practice in 2008, and his site, FiveThirtyEight, was acquired by The New York Times shortly thereafter. More recently, the left-leaning website Talking Points Memo started its PollTracker website before the 2012 election.
Each of these organizations differ in their approaches. Real Clear Politics does a more straightforward averaging of the most recent polls. TPM‘s PollTracker is an aggregation involving regression analysis that uses the most recent polls to project a trajectory for the race. FiveThirtyEight and HuffPost Pollster use polls, adjusting them for house effects–the degree to which a survey house’s polls lean consistently in one direction or another. FiveThirtyEight also uses non-survey data to project the election results.
All four of these outlets underestimated Obama’s margin of victory. Both Real Clear Politics and PollTracker had Obama ahead by only 0.7 percentage points in their final measurements. HuffPost Pollster had Obama leading by 1.5 points, while FiveThirtyEight was closest, showing Obama 2.5 points ahead of Romney in the last estimate. The aggregators that came closest to Obama’s overall winning margin were the ones that attempted to account for pollsters’ house effects.
Rasmussen threw off the average with its Republican bias, typically having a two percent house effect. It was easy to throw out Rasmussen. It was harder to feel comfortable ignoring Gallup, which also was more favorable to Republicans. Here is a Fordham University study ranked the polls.
1. PPP (D)
1. Daily Kos/SEIU/PPP
3. YouGov
4. Ipsos/Reuters
5. Purple Strategies
6. NBC/WSJ
6. CBS/NYT
6. YouGov/Economist
9. UPI/CVOTER
10. IBD/TIPP
11. Angus-Reid
12. ABC/WP
13. Pew Research
13. Hartford Courant/UConn
15. CNN/ORC
15. Monmouth/SurveyUSA
15. Politico/GWU/Battleground
15. FOX News
15. Washington Times/JZ Analytics
15. Newsmax/JZ Analytics
15. American Research Group
15. Gravis Marketing
23. Democracy Corps (D)
24. Rasmussen
24. Gallup
26. NPR
27. National Journal
28. AP/GfK
New York Magazine spoke to Public Policy Polling’s director about how he got it right:
When I talked to Tom Jensen, PPP’s director, this morning, he was understandably in the mood to gloat. “These supposed polling experts on the conservative side are morons,” Jensen crowed. “Jay Cost” — the Weekly Standard’s polling expert who’d waged a number-crunching war against PPP — “is an idiot.” But Jensen conceded that the secret to PPP’s success was what boiled down to a well informed but still not entirely empirical hunch. “We just projected that African-American, Hispanic, and young voter turnout would be as high in 2012 as it was in 2008, and we weighted our polls accordingly,” he explained. “When you look at polls that succeeded and those that failed that was the difference.” Given the methodological challenges currently confronting pollsters, those hunches are only going to prove more important. “The art part of polling, as opposed to the science part,” Jensen said, “is becoming a bigger and bigger part of the equation in having accurate polls.”