Tonight we've got the third and final installment of Neural Gourmet's interview with Dr. Mark Lindeman. As you may recall, Dr. Lindeman, along with Dr. Elizabeth Liddle (the ink on the PhD is still wet! Congrats Lizzie!), contributed significant statistical analysis to the 2004 Presidential election exit poll debate. Dr. Lindeman was agnostic toward the possibility of a "stolen election" while Dr. Liddle was initially a believer. Both however came to the conclusion that the evidence just didn't support the case of widespread fraud. If you're looking for Part 1 of this interview it can be found here, while you'll find Part 2 here. Read on for Part 3.

tng: I'd like to spend some time talking about the exit poll process. I know some of these questions might sound naive to someone as familiar with polling and statistical analysis as yourself, but when I told friends and acquaintances that I'm interviewing a expert on exit polls and the 2004 election, these are pretty much the questions that got bounced off me. And for the most part, I couldn't really answer them. Additionally, I think there is a good deal of mystery of how exit polls are done (polling in general really) and people just want to generally understand the process.

So with that in mind, you've mentioned Warren Mitofsky a few times. Warren Mitofsky of course is head of Mitofsky International which was contracted by the National Election Pool to provide analysis of the interview data and projections based on that analysis. As I understand it, Edison Media Research was the company that actually conducted the exit poll interviews. Who designs the interviews though? Is that the responsibility of Mitofsky's company?

Mark: Yes, you're right, the logistics of interviewing and data collection are in Edison's purview.  I alluded to Mitofsky because it is Warren Mitofsky's reputation for accuracy that people invoke (and often misunderstand). He has been doing exit polls for almost 40 years now.  As far as I can tell, the two companies work together closely and well; I don't see much point in singling out one or the other. (Sometimes I think I should mention Joe Lenski at Edison more often, but considering the abuse that Mitofsky has taken, I'm not sure I would be doing anyone any favors.)

The interviews are designed to specification for the National Election Pool, which consists of the major-network sponsors. Interestingly, the exit polls would probably be more accurate if the questionnaires were shorter -- but the networks really want to be able to present all those details about how various groups voted, and so forth. (I'm not faulting the networks for that. Any curious researcher wants to ask many more questions than most respondents want to answer.) 

tng: And is it really just a matter of exit poll takers asking people who leave the polling place, "Who'd ya vote for?" Is it all done via in-person interviews or do they use phone polling as well?

Mark:  Literally, they don't ask that -- experience shows that a "secret ballot" format works better.  Also, the questionnaires have up to 30 questions, so it's more practical to hand people clipboards!  Exit polls in other countries generally have much simpler questionnaires.

Anyway, to answer your question: most of the interviews are conducted at polling places.  However, as early and absentee voting increases in many states, the exit pollsters have supplemented the polling-place interviews with increasingly large telephone surveys targeted at those early voters in particular states. In 2004 Edison/Mitofsky did about 5,000 telephone interviews in 13 states (plus a 500-call supplement to their national sample).  Mostly it's an election day operation.

tng: How big of a sample population do the exit polls use? Is it comparable to other national surveys?

Mark:  All told, on election day, interviewers collected about 110,000 interviews, which is huge; typical political surveys might have 1000 or 1500 respondents. But bear in mind that the exit polls are covering 51 different elections; each state has a different questionnaire. The exit pollsters sample anywhere from 15 to 55 precincts per state, and they try to complete about 100 interviews from each precinct.

There is also a special national sample; even that sample uses four different questionnaires. The national sample had about 12,000 interviews.

tng: How do they pick their sample? For instance, I live in upstate NY and the joke is that we never see exit pollsters up here because they'd be hard pressed to find most of the polling places. Do rural areas get accurately reflected and does it matter if they don't?

Mark:  The short answer is that rural areas should get represented in proportion to their voters (with some arcane caveats). It does matter, because rural voters tend to be more Republican. Edison/Mitofsky found that overall, their precinct sample matched the overall returns pretty closely. So the exit poll discrepancy wasn't caused by failing to get to rural areas. The bigger the precinct, the more likely it is to be included -- so each voter's probability of inclusion should be roughly equal.

tng: Perhaps the single most important contributor to the notion that the exit polls indicated fraud was Steve Freeman's paper, an early version of which came out within a week of the election. He initially estimated that chance of the exit poll discrepancies in Ohio, Florida, and Pennsylvania occurring at the same time as being 250 million to one against. Later he downgraded that to 662,000 to 1. Even assuming he's correct, is that a valid measurement to make? To me that seems similar to calculating the odds against a person who likes (for example) ice cream and bicycling and anime and 1920s pop music and hockey and art and 18th C. English literature altogether as being so high that no such person could exist.

Mark:  You make a good point.  A somewhat closer analogy to his specific calculation would be: what is the chance of New York City, my home city of Kingston (90 miles up the Hudson River), and Hartford reaching 95 degrees on the same day? As I write, it is very good (UPDATE: 100%!).  Freeman's eye-popping odds assume that the observations are independent -- like rolling three fair dice three times in a row and getting nothing but snake eyes (actually, that would be about one chance in ten million).  But places fairly near each other do often have similar weather.  Having hot weather in all three cities isn't shocking.  If it happened in February, that would be shocking.  Similarly, Freeman's calculation conjures the impression of a bizarre event, but it doesn't actually assess whether the event was bizarre.

OK, fine, immediately after the election, Freeman apparently felt he had to prove (or dramatize?) that the exit poll discrepancy couldn't be explained by random sampling error.  That is true. It was probably even a point worth making, immediately after the election.  Unfortunately, he seems to think that random sampling error is the only innocent source of exit poll error -- which is just plain wrong. And worse, whether he intended to do it or not, he convinced a bunch of people that the official results were "statistically impossible."

tng: What else does Freeman get wrong?

Mark: Well, Freeman makes wild claims about the accuracy of exit polls, both in practice and in theory. I think the theoretical claims bother me the most, although it's a close call.  For instance, Freeman writes in his book, "A well-designed exit poll will closely approximate the underlying reality of how many votes each candidate got within a defined range of certainty" (page 104).  Not coincidentally, he doesn't cite a reference for that categorical claim.  The premise that a "well-designed" survey Will, Just Has To, be accurate within sampling error -- well, my goodness, who taught this man survey methods, anyway?  Is there really some natural law that compels Bush voters and Kerry voters to participate at the same rate? Hell, no. That's not methodology, it's mythology.

Freeman's cherry-picking of sources to bolster the accuracy myth can be painful to read.  For instance, in one paragraph (page 96), he argues that we know exit polls are accurate because two sources say that they are analytically useful (yes, they are), and a third source says that they allow the press to predict the outcome before the election is concluded (yes, they help, especially when the election is not very close!).  Ergo, "Exit-poll results were considered money in the bank." It's a pastiche of non sequiturs.  If you read closely, he doesn't cite a single example of a U.S. national exit poll that matched the official count within sampling error -- yet he argues that no one has ever questioned that the exit polls are accurate.  The mind reels.

Let me make say this again... The eventual calls have been accurate, but that doesn't imply that the interview results have been spot on. The exit pollsters aspire to cautious judgment, not 'infallibility in the original data.'

Maybe worst of all, he feeds a toxic urban legend about the great exit poll cover-up: "Had it not been for leaks from the media and a technical glitch on the CNN site that caused the unadjusted data to be aired, the unadjusted exit-poll data would never have been collected and preserved, and we might never have known about the exit-poll discrepancy at all. These data were not intended for public release." (That's also on page 104 -- that page is pretty well marked up in my copy!)  Actually, the CNN.com site announced in advance that it would post exit poll tabulations soon after the polls closed in each state.  I think the Ohio results appeared within about 5 minutes of the official closing time; I must have been one among many hitting the "refresh" button waiting for them to show up.  I don't know whether these tables would have been "collected and preserved," but it's just ridiculous to claim that they weren't intended for public release.  It's also wrong to call them "unadjusted data," because the tabulations actually factor in pre-election polls. Does Freeman still not know that, or is he trying to convey the impression that some webmaster posted The Truth by mistake?

I could go on. It's a 250-page book. In my AAPOR paper [American Association for Public Opinion Research. Mark is referring to his paper Beyond Exit Poll Fundamentalism linked below, but may be found on his website along with much more on the 2004 election -- tng], I attempt to respond to all of Freeman's "neglected correlations" that supposedly point to fraud, his completion-rate argument, his "who did you vote for in 2000?" argument, and so on -- it starts to feel like whack-a-mole. His book has a few new twists. But those points make me especially angry, because they play into a narrative in which the exit pollsters and networks 'must have known' that the exit polls pointed to a stolen election, and willfully covered up the evidence. That is wrong from soup to nuts. Even if one thinks the election was stolen, there is no good reason to think that the exit pollsters and networks thought so.  It's fine to be skeptical, it's fine to be critical, but factually challenged insinuations of complicity in a coup -- well, it's just too much.  The reality-based community shouldn't go there.

tng: It's clear you have put in quite a bit of time responding to Steve Freeman's claims and I've got to recommend people read Beyond Exit Poll Fundamentalism for a reasoned and insightful examination of the debate around the 2004 election.

You've also spent much time interacting and responding to amateur and non-academic investigators of election fraud. One of your primary critiques of amateur approaches to understanding the discrepancies between the exit polls and the recorded vote in 2004 is that amateurs are fixated on only looking at one explanation -- that of vote miscount -- and fail to consider that the discrepancies might be caused by vote miscount, non-response bias (people who are reluctant to tell pollsters whom they voted for), other polling errors or a combination of those three. The case of non-response bias has been termed the "reluctant Bush responder" (rBr) hypothesis since the exit polls so dramatically favored Kerry and generally decried as preposterous among people believing in a stolen election. What are they getting wrong here?

Mark:  First of all, I think they've been misled into thinking that non-response bias is a bizarro-world excuse that Mitofsky made up on the spur of the moment in November 2004.  Survey researchers worry about non-response bias all the time.  It is a major research subject in its own right.  And we pretty much know there was non-response bias in the 2004 exit poll, because interviewers recorded a 43% completion rate among voters aged 60+, compared to over 55% for everyone else.  We also know that many other exit polls have evinced precinct-level bias on vote choice.  It's a crazy leap to assume that there should be no such bias in 2004.

I should point out that the exit polls probably evince a combination of non-response bias among voters and sampling bias among interviewers.  Imagine that a Bush voter and a Kerry voter are coming out of a polling place at about the same time, and they spot a young interviewer who looks like (and maybe is) one of those "Kerry youth voters" they have been hearing about.  The Kerry voter smiles; the Bush voter adjusts course to miss the interviewer by a crucial ten feet.  Which voter probably gets interviewed?  Which of the three people decides the outcome?  Am I describing reluctant Bush voters, eager Kerry voters, biased interviewers, or what? non-response bias, or sampling bias?  Reality is messy.

Maybe that is the main point that gets missed: reality is messy. Science in general, and certainly "social science," proceeds incrementally and cautiously on the basis of radically incomplete information.  Some folks have argued that the exit pollsters bear the "burden of proof" of demonstrating that non-response bias explains the exit poll results.  But they rarely attempt to offer a coherent fraud hypothesis that does any better.  It strikes me as an Intelligent Design argument applied to an election rather than speciation. In the election context, the intelligence is assumed to be malevolent, but in both cases, it is somewhat inscrutable.  The largest exit poll discrepancies were in Vermont and Delaware, but no one seems very interested in explaining how and why those states became epicenters of vote fraud.  I think of it as a "fraud of the gaps" argument: Fraud is invoked as an "explanation" of residual variance.  (Or, as long as I'm offering strained puns, the argument asserts irreducible complicity.)

Of course, since the exit polls could evince vote miscount as well as survey error, we need to look more closely.

tng: Another commonly held belief is that Edison/Mitofsky and the National Election Pool are participating in a coverup by refusing to release all of the raw exit polling data (the precinct level data). While this notion strikes me as approaching a systemic conspiracy theory, why is this data being withheld?

Mark:  Actually, it depends on what one means by "raw" data.  As in past years, the exit pollsters released their compilation of every question on every questionnaire for which they entered the individual answers.  (The interviewers phone in tallies of all the reported vote choices, and the complete responses for a subsample of them -- overall in 2004 it was about two-thirds. It's a tradeoff: the more time they spend phoning in interviews, the fewer interviews they can conduct.)  Unlike past years, they made the datasets available on the Internet.  Whatever else one says, those data are pretty raw!  But they haven't released precinct identifiers that would allow people to match the interviews to particular precincts in the real world.

So, why don't they release precinct identifiers?  They probably have more than one reason, but I think a good one is to protect the confidentiality of the respondents.  The respondents answer quite a few demographic questions, which in small precincts would probably suffice to identify some of them uniquely.  Every questionnaire says in capital letters, "YOUR ANSWERS ARE CONFIDENTIAL." Decent researchers do take that seriously -- both out of principle, and because hanging respondents out to dry doesn't exactly encourage future trust and cooperation.  If the exit poll data really held the key to fraud, obviously ways could be found to get out the crucial results without betraying individual respondents. But bluntly, they don't hold the key. Even if the data were perfect, I don't see how 50 exit poll precincts in Ohio could hold the key to understanding the 11,000 others -- and the data are far from perfect

tng: I'd love to explore everything you covered in your Beyond Exit Poll Fundamentalism [PDF] paper, but we'd still be having this conversation next month if we did. Still, could you touch on the evidence that you believe points away from fraud? Maybe you could start by elaborating on something you said in Part 1 of our interview about New York and DREs that intrigued me:

Mark: "If one really believes that the exit polls were accurate, then one should be looking for massive numbers of stolen votes in New York and probably California. Or, setting aside the exit polls, if one really believes that DREs were massively hacked, then presumably one should expect strikingly anomalous election returns in DRE jurisdictions, and I don't see them."

That's how we left things hanging at the end of Part 1 so I thought it'd be interesting if you could expand upon that point...

Mark:  The day after the election, I talked with a colleague about whether the election could have been stolen through hackery. He suggested an obvious test: see whether the exit poll discrepancies were focused in precincts that used electronic voting machines.  Apparently the exit pollsters actually ran that test very early on, and they found that actually, the largest discrepancies were in precincts with lever machines.  But we don't have to limit ourselves to the exit poll results: we can look at official returns all around the country and see whether Bush did anomalously well in counties that use DREs or other suspect equipment -- perhaps, especially, in counties that just switched over to suspect equipment.  Well, I just don't see it.  I certainly don't see it in Ohio.  Some people are convinced that Ohio was "Diebolded," but Ohio hardly used Diebold equipment in 2004.  (I could go on, but I won't.)

So, thinking about the lever machines brings us back to New York, which had the third largest exit poll discrepancy in the country.  (As you know, almost the entire state still votes on lever machines.)  New York happens to have had four pre-election polls in the week before the election, all of which gave John Kerry a lead of between 15 and 18 points.  The exit poll in New York put Kerry ahead by 31.3 percentage points.  The offficial returns say he won by 18.3.  I figure, if someone really believes that the exit polls are accurate, he or she can explain how Kerry "really" did 13 to 16 points better than the pre-election polls indicated -- and will have a theory and maybe some evidence as to how Bush then stole a net 13 percent of the vote. This is not a small number of votes -- in fact, it's 960,000, almost one-third of Bush's margin in the popular vote.  Where did those votes come from?  Why does nobody seem to be looking for them?

New York is one example of a general tendency.  Although no one thinks that the pre-election polls are perfect, one would expect that if the polls are useful at all, and if fraud varied from state to state, and if fraud weren't somehow invisible to exit polls, probably Bush would tend to do better (compared to the pre-election polls) in the states with the biggest exit poll "red shift."  Actually, if anything, Kerry tended to do better than expected in the states with big red-shift.  To put it another way, some people talk as if it was miraculous for the exit polls to be wrong, but it would have been more miraculous for the exit polls to be right.  I hoped that Kerry would pull out a win in Ohio, but the exit poll estimate had him up by 6.5 points. It seems very unlikely.

I think the best exit poll evidence against massive, widespread fraud is the "shift/swing analysis." Swing, in this context, refers to the change in Bush's vote share between 2000 and 2004, calculated for each exit poll precinct.  (It can be figured a few different ways -- I'm trying not to get too wonky.)  Swing is usually fairly small, in the range of plus-or-minus 10 percentage points.  "Shift" refers to exit poll discrepancy, which also can be measured in various ways.  I'm told that it doesn't matter much what measures one uses: there is basically zero correlation between swing and shift.  Colloquially, Bush didn't do better (compared to 2000) in the precincts where the exit polls supposedly indicated vote miscount in his favor.

It turns out to be really hard to reconcile that result with massive fraud -- certainly with a reversal in the popular vote. The easiest way to do it is to assume that Bush controlled (or could have controlled) the vote count in practically every precinct in the country.  The fraud has to be either uniform, or targeted in precincts where Bush would otherwise have done substantially worse.  Actually it's a bit tricky to make it work even if Bush controlled the vote count in almost every precinct.  I guess an easier approach is to assume that Bush controlled the exit polls and rigged them to achieve maximum possible confusion!

All in all, the exit polls paradoxically don't seem consistent with massive vote miscount, although it's hard to say just how much vote miscount could have happened nationwide or in any particular state.  I think the vote count was generally pretty clean.

tng: Even though you don't believe that the case for wide spread fraud in 2004 is supported, how do you feel about the integrity of U.S. elections? Are you relatively sanguine or do you feel there's cause for concern? If the latter, then how concerned are you and what do you think voting integrity activists should be concentrating on?

Mark: Well, my basic outlook is more of SNAFU than apocalypse -- although "SNAFU" has been devalued through overuse.  We've seen legitimate voters disenfranchised, we've seen them stuck in the rain, we've seen huge sums spent on voting machines with eye-rubbing security and reliability flaws, we've seen random recounts that weren't random, we've seen public servants who don't serve.  But we've seen worse.  So, compared to many activists, I think I really am "relatively sanguine," but I'm also concerned.  These are all distinct, real issues; I won't tell people what to focus on.

The rush to paperless DREs is bad news. I don't endorse the "Republican stealer machine" analysis, but I haven't seen a good answer to the obvious problems of security and transparent auditability. I support paper trails and tough audits.  Voting technology is being debated and acted on at every level of government, so there is something for everyone to do.

I do suspect that the miscount issue attracts a bit more passion than all the issues of voter suppression -- and I think that voter suppression mattered more in 2004. Weirdly, some people who profess to see no real difference between the two major parties are also quick to trumpet their conviction that the election was despicably stolen.  Voter suppression tends to thrive when potential voters aren't plugged into the political process -- so activists who are mostly indifferent to electoral politics will be predisposed to miss the point. People need to look for the big picture.

I won't venture a five-point program to counter voter suppression.  The Democratic Party probably needs to put some serious effort into party-building, as in, actually knowing who its likely voters are, and bringing some new ones into the fold. Some non-partisan voting groups did a great job of trying to follow up to ensure that the people they "registered" actually became registered and knew where and when to vote; it entails close work with voters and with election officials.  We seem to need a new generation of pollworkers.  Again, there is something for everyone to do.

tng: That's very similar to how I've come to view it, though I might be more inclined to use the phrase emergent behavior. Well, finally... What should I have asked you that I didn't?

Mark: Heck if I know!

tng: OK then! I think after 10 days and 27 e-mails that's a wrap! Again, thank you so much for agreeing to do this. This was my first time interviewing anybody on a complex topic and I'm sure it could have been done a lot better, but you've been great. Can we call on you in the future with questions or seeking brief comments relative to your fields of expertise?

Mark: Sure. It might be safest to give me a word limit!

tng: Hah! We don't need no stinkin' word limits on this blog! Once more, thanks. I've really enjoyed this.

And so concludes our three part interview with Mark Lindeman on the subjects of public opinion, exit polls and the case against widespread fraud in the 2004 Presidential election. This is a touchy subject with us lefties. We know that Al Gore won both the popular and electoral votes in 2000, but the Supreme Court stopped the recount of the votes in Florida and selected George W. Bush instead. And after nearly 4 years of Republican policy, after 9-11 and Afganistan, and Iraq it seemed impossible to us that anyone could want George W. Bush as President again. And Kerry was, even despite the deplorable Swiftboat ads, really close in the pre-election polls. But I think when one fairly and dispassionately examines the evidence that the only conclusion you can come to is that widespread fraud just didn't take place.

But we know there were many problems that took place on the local level. The disproportionment of voting machines in Ohio for example. And the DREs did malfunction in some places. There were the voter roll purges in 2000 in Florida. These are all documented cases of voter suppression and vote miscount. So just because the last election wasn't stolen, doesn't mean the potential isn't there and it doesn't mean there isn't work to be done. And that's where we should be concentrating our efforts. Not on proving a stolen election that isn't supported by the data.

At least those are my thoughts. Yours might differ. In any case, I do encourage you to read Mark's Beyond Exit Poll Fundamentalism paper as well as the other material covering the 2004 election debate on Mark's website.

I hope you've enjoyed this series and maybe even learned something. I know I did. 

Bookmark/Search this post with:

| | | | | | | | | |

tng's blog | printer friendly version