A SoapBlox Politics Blog
[Mobile Edition]
- About Us
- Email Us (news/tips)
- Editorial Policy
- Posting Guidelines
- Advertise Here

Subscribe to Michlib daily email summary. (Preview)
Enter address:

Michigan Political Blog Ad Network

Advertise Liberally

50 State Ad Network

Possibility of spurious polling on Kos

by: Grebner

Wed Jun 30, 2010 at 01:13:04 AM EDT

As more and more polls appear, the question arises:  exactly who is checking their legitimacy?  The answer, unfortunately, is that nobody is. Last year, a firm called Strategic Vision, LLC was discovered to have essentially fabricated a huge number of supposed samples in dozens of critical races around the country.

Now, Research 2000,  until recently the house polling firm for DailyKos, has been called into question.  The paper speaks for itself:


In brief, we looked at some 75 published polls, and examined how much statistical variation they contained compared to the minimum expected by statistical theory given their sample sizes.  The Research 2000 polls showed too little fluctuation, were too consistent to one another, and generally showed signs that they were not properly calculated or were not based on actual interview data.

Grebner :: Possibility of spurious polling on Kos

Michael Weissman, a retired physics prof who was largely responsible for assembling definitive evidence against Strategic Vision, LLC, was responsible for doing all the heavy lifting and wrote most of the text.

His son, Jon Weissman, also a physicist, assembled the data from a large number of polls with varying formats and questions.

My role was largely limited to starting this ball rolling, making a few statistical suggestions, and generally getting in the way of progress. 

Tags: , , (All Tags)
Print Friendly View Send As Email

This story has really grown legs
Nate Silver at fivethirtyeight.com has covered it, to the point where Research 2000's lawyer sent him a cease and desist letter -- which Nate promptly posted so folks could point and laugh at the absurdity of it.

Taegan Goddard's also covering it at Political Wire: http://politicalwire.com/archi...

"The two most common elements in the universe are hydrogen and stupidity." ~ Harlan Ellison

Doug Rivers
offers a possible explanation for what may be causing the "funky" graph results.

If you read all the way to the bottom....
... you'll discover Doug Rivers is finally brought to entirely concede to my co-author that our findings were not affected, because that we had properly considered the issue.

For interested readers, our point was that when you plot all the changes from one week to the next, instead of a typical bell-shaped curve, Research 2000's polls look more like Mt. St. Helen: a mountain with a deep central crater, where the zero-percent changes should be piled up.  We were unable to propose a mechanism that would result in such a deficiency of zero-percent changes in comparison to the large number of minus-one and plus-one changes.  After a day of hot discussion, we still haven't heard one.

Nate Silver, in his follow-up on FiveThirtyEight.com, plotted the same curves, and showed they were narrower than expected for the sample sizes - that is, there weren't as many with drops of minus-three or more, or increases of plus-three or more, in comparison of changes between minus two and plus-two.  As Rivers pointed out, Nate failed to account for the fact that the partisan composition of each week's survey was effectively frozen by a quota, which substantially reduced the variance.  Nate has updated his posting, showing that even after the correction, his point remains valid.

We were quite careful.  Aside from two (now corrected) very minor typos, nobody has brought anything to our attention that warrants fixing.

[ Parent ]
Who said
anything needed to be corrected?

I read it to and was aware of the points raised. I was just pointing out a piece of alternative discussion that wasn't widely circulated in the avalanche of discussion going on in the blogosphere.

[ Parent ]
Great work !
Mark, I think it's great that your work is nationally known on this topic.  This has been picked up in a lot of places and puts forward many questions you have raised about the validity of certain polls and the need to really look into  how they are written, how the sample is selected, etc...

How about doing a similar study of some MI based polls ?  Also, MSNBC in particular is using the opportunity to bash IVR polls...thoughts ?

Great idea!
I'd love to do it.  Here's what I imagine:  some wealthy do-good operation sets aside $100,000, and pays me $200 per hour to sit and read polls while sipping iced latte with my dog in a park.  I'd issue certificates, on very nice paper, to each one that seems okay.  

Until that happens, unfortunately, I've got to keep working.

[ Parent ]
Good going, Mark!
Political folks around here were pleased that our home-grown practical consultant was involved. Any chance you'd be willing to come and talk to a U-M campaigns and elections class this Fall?

The story was picked up in quite a few places (I saw it via TalkingPointsMemo on Tuesday), although at first only the actual Kos site named Mark Grebner. Today's NYTimes does, too.

You didn't mention it here, but wasn't the impetus the pairs of matching evens and odds? That's the kind of thing that any reviewer should have noticed.

Although the miscreants probably won't make that mistake in the future....

Actually, spotting anomalies in tables is very hard.
As Thomas Bayes teaches us, the effect of receiving new data is - at best - to strengthen some of our preconceptions and weaken others.  If you start without any suspicion, there's nothing to strengthen.  We eventually discovered that the parity (odd-even status) of the number reported for men always matched that for women (meaning that if 31% of men said they thought Pelosi was doing a good job, among women the number might have been 57% or 53%, but never 54%).  But even when we put together a chart with all the appropriate numbers in columns, I couldn't see it - until somebody pointed it out.

The person who first found the pattern (Michael Weissman) immediately assumed he had screwed up the data collection process, and spend considerable effort trying to find and fix his own error.  It was the last thing on his mind that it could be a real finding.

[ Parent ]
More interesting data
on the costs of polling and the number of polls conducted.

You might be able to help tighten this up for them here Mark...

R2K responds...
R2K's Del Ali described you as  "Kos bloggers posing as statisticians" in this TPM article: http://tpmmuckraker.talkingpoi...

Care to comment on your own existence?

I never thought of myself as a "Kos Blogger".
Del's posting on TPM was largely incomprehensible to me.  One of the few lines I think I understand was:

"Yes we weight heavily and I will, using the margin of error adjust the top line and when adjusted under my discretion as both a pollster and social scientist, therefore all sub groups must be adjusted as well."

That was exactly as punctuated in the original, by the way.  I believe it says that he feels free to exercise his "discretion" to "adjust" the reported results of his polling by any amount up to "the margin of error".  If that's all he did - which would be a firing offense at every polling firm in the country - it would explain several of the anomalies we reported.

If there was any doubt the political world would be improved by R2K's departure, Del's behavior and comments have dispelled it.

[ Parent ]

Progressive Blogroll
For MI Bloggers:
- MI Bloggers Facebook
- MI Bloggers Myspace
- MI Bloggers PartyBuilder
- MI Bloggers Wiki

- Blogging for Michigan
- Call of the Senate Dems
- [Con]serving Michigan (Michigan LCV)
- DailyKos (Michigan tag)
- Enviro-Mich List Serve archives
- Democratic Underground, Michigan Forum
- Jack Lessenberry
- JenniferGranholm.com
- LeftyBlogs (Michigan)
- MI Eye on Bishop
- Michigan Coalition for Progress
- Michigan Messenger
- MI Idea (Michigan Equality)
- Planned Parenthood Advocates of Michigan
- Rainbow Mittens
- The Upper Hand (Progress Michigan)

Upper Peninsula:
- Keweenaw Now
- Lift Bridges and Mine Shafts
- Save the Wild UP

Western Michigan:
- Great Lakes Guy
- Great Lakes, Great Times, Great Scott
- Mostly Sunny with a Chance of Gay
- Public Pulse
- West Michigan Politics
- West Michigan Rising
- Windmillin'

- Among the Trees
- Blue Chips (CMU College Democrats Blog)
- Christine Barry
- Conservative Media
- Far Left Field
- Graham Davis
- Honest Errors
- ICDP:Dispatch (Isabella County Democratic Party Blog)
- Liberal, Loud and Proud
- Livingston County Democratic Party Blog
- MI Blog
- Mid-Michigan DFA
- Pohlitics
- Random Ramblings of a Somewhat Common Man
- Waffles of Compromise
- YAF Watch

Flint/Bay Area/Thumb:
- Bay County Democratic Party
- Blue November
- East Michigan Blue
- Genesee County Young Democrats
- Greed, Eggs, and Ham
- Jim Stamas Watch
- Meddling Outsider
- Saginaw County Democratic Party Blog
- Stone Soup Musings
- Voice of Mordor

Southeast Michigan:
- A2Politico
- arblogger
- Arbor Update
- Congressman John Conyers (CD14)
- Mayor Craig Covey
- Councilman Ron Suarez
- Democracy for Metro Detroit
- Detroit Skeptic
- Detroit Uncovered (formerly "Fire Jerry Oliver")
- Grosse Pointe Democrats
- I Wish This Blog Was Louder
- Kicking Ass Ann Arbor (UM College Democrats Blog)
- LJ's Blogorific
- Mark Maynard
- Michigan Progress
- Motor City Liberal
- North Oakland Dems
- Oakland Democratic Politics
- Our Michigan
- Peters for Congress (CD09)
- PhiKapBlog
- Polygon, the Dancing Bear
- Rust Belt Blues
- Third City
- Thunder Down Country
- Trusty Getto
- Unhinged

MI Congressional
District Watch Blogs:
- Mr. Rogers' Neighborhood (CD08)

MI Campaigns:
MI Democratic Orgs:
MI Progressive Orgs:
MI Misc.:
National Alternative Media:
National Blogs:
Powered by: SoapBlox