This is the second part of our interview with Dr. Mark Lindeman. If you're looking for the first part of the interview you can find it here [0] and the third part is located over here [0]. Before we get to the actual interview though, Mark wanted me to make sure that everyone knew that Elizabeth Liddle deserves most of the credit for novel analysis. Personally, I think both deserve much credit for tirelessly explaining their work on various internet discussion boards to less than receptive audiences. So, I'm sorry that I unintentionally slighted Elizabeth Liddle. That certainly wasn't my intent. Well, maybe Mark can find some more things I totally screwed up in time for tomorrow. In the meantime, read on for the second part of the interview. I think you'll enjoy it and, maybe, come away marginally more informed than you were before.
tng: One of the things I'm curious about is this notion that just seems to be a lefty axiom that the exit polls are solid indicators of the actual vote. Robert Kennedy Jr. in his recent Rolling Stone article [1] reflects this sentiment well I think:
Over the past decades, exit polling has evolved into an exact science. Indeed, among pollsters and statisticians, such surveys are thought to be the most reliable. Unlike pre-election polls, in which voters are asked to predict their own behavior at some point in the future, exit polls ask voters leaving the voting booth to report an action they just executed. The results are exquisitely accurate: Exit polls in Germany, for example, have never missed the mark by more than three-tenths of one percent.(17) ''Exit polls are almost never wrong,'' Dick Morris, a political consultant who has worked for both Republicans and Democrats, noted after the 2004 vote. Such surveys are ''so reliable,'' he added, ''that they are used as guides to the relative honesty of elections in Third World countries.''(18) In 2003, vote tampering revealed by exit polling in the Republic of Georgia forced Eduard Shevardnadze to step down.(19) And in November 2004, exit polling in the Ukraine -- paid for by the Bush administration -- exposed election fraud that denied Viktor Yushchenko the presidency.(20)
And this DailyKos commenter [2] notes:
But exit polls have been reliable indicators of voter sentiment for decades - until recently.
When I read stuff like this I'm struck by the fact that there seems to be several factors at play, all based on broad assumptions and which bring to mind several questions. The answers don't seem to be readily available either. At least when I go looking I usually find repetitions of the conventional wisdom that I really have no way of knowing if it's true or not since it depends on rather technical knowledge about polling and historical data. So, if you don't mind I'd like to take these questions one by one since it would seem that if we don't know what to expect from exit polls then we can't really know if what they're telling us is correct or not. Are exit polls really such reliable predictors of the actual vote?
Mark: I'm going to change "reliable" to "accurate." Technically, statistical "reliability" means consistency -- but (simply put) a measure could be consistently wrong. So, are the exit polls really such accurate predictors of the actual vote?
Short answer: no, they aren't. Actually, I had to go back all the way to 1980 to find a U.S. presidential exit poll that seems to have been accurate within sampling error. (There could be one or two others; partly it could depend on how one assesses accuracy.) The 1992 exit poll was almost as discrepant as the 2004 poll, and I still haven't encountered a serious, sustained argument that it evinces fraud. That's probably why Kennedy resorted to Germany to make his point -- which is actually flat-out wrong anyway. I'll say a bit more on that below. When (for instance) Warren Mitofsky talks about his record of accuracy, what he means is that he has rarely made wrong projections. Basically that is because the "exit pollsters" don't rely on interviews alone to project close races; they wait for quick count and/or county returns. So the interview data don't have to be entirely "accurate." To answer your original question, the U.S. exit polls don't seem to be very reliable either. The discrepancies vary from state to state and from year to year.
tng: Good call on the mismatch between every day usage and technical usage of terms like reliable and accurate. Some of the confusion out there might actually spring from these types of language problems. I wonder though, if exit polls in the U.S. have never been particularly accurate then where did this notion that they are come from?
Mark: Well, in one sense it comes from the fact that some smart-sounding people confidently said it over and over again. I think it also comes from confusing the accuracy of the projections with the accuracy of the interview data. Mitofsky's accuracy partly depends on being suspicious of the interview data.
Also there is the idea that the exit polls ought to be accurate because they ask real voters how they actually voted -- unlike pre-election polls, which ask potential voters how they might vote. The premise there is valid, and a real advantage. But it obviously doesn't guarantee that the poll will be accurate. Exit polls also have some potential disadvantages: for instance, they involve face-to-face interaction, in often chaotic settings, with only one chance to complete an interview.
tng: Has the accuracy of exit polls changed recently?
Mark: Maybe at the margins. But lots of people are convinced the U.S. exit polls were spot on until Florida 2000, and that just isn't true.
U.S. exit polls may be increasingly subject to bias because the response rates are falling, but that's one of those iffy arguments that doesn't really have much to do with the fraud debate. A survey can be erroneous pretty much no matter what the response rate is.
However, Kennedy is just wrong that German exit polls "have never missed the mark by more than three-tenths of one percent." I don't know the whole history of German exit polls -- and I suspect that he doesn't, either -- but in 2005, the German network ZDF's initial projection of the margin between the top two parties was off by 3 percentage points. ZDF put the margin at 4 points, but the official margin was about 1 point. It appears that we just barely avoided a whole new exit poll controversy. But the real riddle is: why didn't Kennedy, or someone at Rolling Stone, look this up?
tng: Is it fair to compare the accuracy of exit polling in other countries to those in the U.S.?
Mark: That's the least of the problems here. Let me say this: there were two exit polls in Ukraine for the first presidential runoff (the one that the court threw out), and they differed from each other by 8 points on the margin. That's bigger than the difference between the U.S. exit poll and the official returns. So the notion that Ukraine proves the reliability of exit polls is just jaw-dropping. (In fairness, the U.S. polls are probably more "reliable" than that.)
Kennedy's rhetoric here has to be parsed carefully. One can say that the Ukraine exit polls "exposed election fraud," but there was ample evidence of fraud there even without the exit polls. The key statements by international observers don't even allude to the exit polls. Richard Lugar's statement on behalf of the U.S. doesn't mention the exit polls. Colin Powell doesn't mention the exit polls when he says that the U.S. can't accept the result. As far as I can tell, one U.S. official mentions the exit poll results, in congressional testimony after the result has been overturned. The U.S. media did sometimes mention the exit polls, but they weren't the center of the story. It's hard to tell what Ukrainians thought, but protests began even before the first results were reported. Georgia in 2003 seems pretty similar: yes, exit polls indicated fraud, but they weren't crucial. And even if they had been crucial, it still wouldn't prove that they were accurate, or that the U.S. polls were accurate. It's just a heaping pile of non sequiturs.
Of course the object isn't to settle a bogus dichotomous question, "are exit polls accurate or inaccurate?" If we accept that the exit polls could be wrong, and that the official results also could be wrong -- that, in fact, both could be wrong simultaneously -- then we have to do some more work.
tng: Why do you say that asking if exit polls are accurate or inaccurate is a bogus dichotomy? If people are in large part basing their opinions on whether or not the last election was stolen based upon the idea that exit polls are accurate, then isn't it sort of important to know whether that assumption about accuracy is valid? Am I drawing a false dichotomy here? If so that wasn't my intent.
Mark: Oops, I didn't mean to imply that you were forcing that dichotomy. I happened to encounter it elsewhere in the last hour or so, not for the first time. It can be framed like this:
"Well, if the exit polls aren't Accurate, then they must be Useless, and if they are Useless, then why do the sponsors pay millions of dollars for them?"
That's the false dichotomy I was trying to get past.
"Why are you trying to convince us to ignore the exit polls?"
I'm not.
Certainly there is an analytical dichotomy between "assuredly unbiased" and "possibly biased," but most survey researchers would never consider "assuredly unbiased" as a plausible assumption. Survey research is largely about anticipating, minimizing, and compensating for bias -- not assuming it away. So, yes, the first step is to realize that the method isn't guaranteed to be accurate, and in fact, that single-digit discrepancies seem to be pretty common. (Some people have referred to the "uncanny accuracy" of the exit polls. I think it's a very bad sign when the adjective "uncanny" drifts into nominally scientific discourse.) That doesn't mean that we scrap the data; it means that we try to interpret it judiciously. It's a pretty banal point, but some folks can't seem to get past it.
tng: Ah, I see... Sadly that's a typical attitude -- that questioning is equivalent to dissent and dissuasion.
Well, that's where I'm going to wrap up this part of our interview with Mark Lindeman. I'll be posting the third and final part of the interview tomorrow where we'll be asking Mark about the exit polling process, as well as going into a little more detail about why Mark did not see evidence for fraud in 2004.
[3] | 
[4] | 
[5] | 
[6] | 
[7] | 
[8] | 
[9] | 
[10] | 
[11] | 
[12] | 
[13]