First off, even if one were to consider RCP biased towards the right, the have Obama with a huge advantage. But there are man who think they cherry pick the best polls for the right:
From fivethirteight.com
Last week, I discussed Real Clear Politics' decision to exclude the
Research 2000 daily tracking poll from their national averages. After a lengthy discussion with RCP founder John McIntyre, I decided to defend them, pointing out that while RCP may have a debatable framework for deciding which polls they do or do not include, they at least appeared to have applied this framework consistently.
My gut-level takeaway from my conversation with McIntyre was somewhat different from what I represented, however. My gut-level feeling was that RCP was in fact cherry-picking those results that were to its liking, and then coming up with post-facto rationalizations to justify its decisions.
RCP made a decision yesterday that convinces me I should have trusted my gut. Specifically, it was their decision to re-include polls from
American Research Group (ARG) in their state-by-state averages.
ARG had been effectively "banned" from RCP for several months now, going back to the Democratic primaries. Sometimes RCP listed ARG polls with an asterisk and did not include them in their averages; more often they didn't list them at all.
Why didn't RCP include ARG's polls in its averages? Presumably because they had an exceptionally erratic performance during this year's primaries, a performance that -- when coupled with ARG's relative lack of disclosure about its methodology -- had led RCP to conclude that their polls were not credible.
As you may know, I am not much of a fan of ARG myself. However, while ARG
ranks toward the back end of our pollster ratings, so do a lot of other pollsters that RCP has no problems with.
What I do know, however, is that whatever RCP's reasons were for excluding ARG from its averages, none of those reasons should have changed over the past 48 hours. Did ARG suddenly improve its level of disclosure? No. Did their polls suddenly become more accurate? How could they, when we haven't had any elections to evaluate them.
No. The only thing that changed is that ARG
released a set of three polls yesterday that had considerably more favorable numbers for John McCain than other contemporaneous polls of those states. In each of those states --
North Carolina,
Virginia,
Nevada -- ARG is presently the outlier on the John McCain side (tied with Mason-Dixon in the case of Virginia). RCP did not feel any compulsion to include ARG's numbers when ARG cycled through all 50 states a couple of weeks ago (including many where there has been very little polling). Only when ARG released this gravity-defying set of polling in Virginia and North Carolina and Nevada did RCP suddenly have a change of heart.
This is not RCP's sole problem with consistency. Take a look, for example, at which polls they presently include in their averages for
Virginia:
You'll see three polls -- SurveyUSA, Mason-Dixon, and ABC/Post -- that were conducted at essentially the same time. The ABC/Post poll was in the field from the 18th to the 21st, Mason-Dixon from the 17th to the 22nd, and SurveyUSA from the 19th to the 21st. And yet, the Mason-Dixon and ABC/Post polls are included in its average whereas SurveyUSA is not. Guess which one had the strongest numbers for Obama?
A similar example from
Minnesota. The Star-Tribune conducted polling from September 10th through September 12th; SurveyUSA conducted polling from September 10th to September 11th. The Star-Tribune poll, which shows a tie, is included; the SurveyUSA poll, which showed Obama ahead by 2, is not.
As long as we're having fun with this, how about
Alaska? Anchorage-based pollster
Ivan Moore had released a poll in July, then showing just a 3-point race between McCain and Obama. We included it, Pollster.com included it, and RCP did not. But then last month, when the same firm released a
fresh poll showing the profound effect that Sarah Palin had on the race (McCain +19), RCP decided they were a worthy pollster.
Or how about
North Carolina Senate? RCP presently includes two weeks-old polls from SurveyUSA and Research 2000, whch had shown Elizabeth Dole with a solid lead in her race against Kay Hagan. But they didn't bother to include more recent polls from
Elon and
Civitas, which show the race essentially tied.
Why is one of
Mitch McConnell's internal polls included in their
Kentucky Senate averages?
Why, a week or so ago, did RCP
include a poll from the Marketing Resource Group of Lansing, Michigan, which the
Detroit News describes as a "Republican consulting firm", while excluding a Nevada poll that came out on the same day from
Project New West, a Democratic strategy firm? Might it have had anything to do with the fact that the Michigan poll showed McCain ahead -- and the Nevada poll showed him behind?
Look -- I'm not going to tell you that my site is completely devoid of spin. I am a Democrat, and I see the world through a Democratic lens. But what I can promise you is that we'll keep the spin separate from our metrics. The spin is a side dish, which you can choose to consume or ignore.
Unfortunately, that is not a choice you have at RCP. Their partisan leaning is infused into their numbers. If RCP disclosed their methodology -- articulated their rationale for excluding or including certain polls -- I would give them the benefit of the doubt. But they do not, so I do not.
You do, however, have a number of good alternatives in the marketplace. So here is a recipe for RCP detox.
For polling averages, you can of course look here. But otherwise, I would recommend
Pollster.com. Pollster.com has the most unbiased standard imaginable: they simply include every poll.
For content aggregation, I would recommend
Memeorandum or
PoliticsHome, each of whom refresh their material on a 24/7 basis.
We're not asking that you go cold turkey. Do bookmark and continue to read
Jay Cost at RCP, who is an outstanding analyst.
Full disclosure: RCP is a competitor, so I have every incentive to tell you this. (I'm pretty sure they see me as a competitor too, as I don't think they've linked to me in about a month). But the fact is that there's not an exceptionally high degree of difficulty in simply collecting a bunch of polls and averaging them together. There's no excuse not to do it right.
-- Nate Silver at
8:08 AM
hno:
