Putting that Ph.D. to good use
Lest you think I’m simply lazing about all summer in Maine, I’m doing far more than that. I’m writing letters to the editor.
Here’s my letter addressing a problem with a poll in the Portland (Me.) Press Herald:
In Tuesday’s front-page article about a recent Rasmussen poll, the newspaper states that Paul LePage “leads the race” with support from 39 percent of respondents compared to 31 percent for Elizabeth Mitchell. The poll’s margin of error is plus or minus 4.5 percent.Because of the intricacies of statistical modeling, the margin of error means that LePage’s true support could be as low as 34.5 percent and Mitchell’s could be as high as 35.5 percent. Therefore, LePage and Mitchell are actually in a statistical dead heat.
According to the article, Rasmussen surveyed 500 people for the poll. In order to achieve a lower margin of error (in the 3 percent range), approximately 1,000 people would need to be interviewed.
Of course, interviewing twice as many respondents would cost a lot more money — which is why we see a lot of polls with large margins of error.
The Press Herald is surely not alone in this oversight. Research has found that many media outlets tend to ignore or downplay the details of polling margins.
In the future, reporters should be careful how they phrase poll results, and editors may want to be more selective in which polls they choose to highlight.
Matt J. Duffy, Ph.D.
Westport Island
That’s my first official use of the Ph.D. They took out the following coda: “Duffy teaches journalism at Zayed University in Abu Dhabi, UAE. He summers in Maine.”
No related posts.
Related posts brought to you by Yet Another Related Posts Plugin.
Facebook comments:
Matt,
Actually, your analysis of the poll results is incorrect and Paul LePage does indeed “lead the race” at the 96.8% confidence level.
The margin of error (MOE) is a statistic about the confidence interval of an individual candidate’s true poll result. It does not directly convey the confidence interval associated with the *difference* between two candidate’s poll results.
You claim that the candidates are in a statistical dead heat. A statistical tie, or statistical dead heat, is a polling result for which the difference between two (or more) candidates is such that we expect sampling error alone to reasonably explain the difference. In lay terms, we would say that the our confidence is < 95% that one candidate truly leads the other.
Fortunately, with a simple statistical calculator and the poll information you provide in your blog — p1=0.39, p2=0.31, n=500 — we can calculate the actual significance level of the hypothesis "LePage leads Mitchell". It is 0.0317, which loosely translates to "we are 96.8% confident that LePage leads." The newspaper article is correct.
Here is a link to a simplified discussion about the fallacy of directly applying MOE to the difference of poll percentages: http://www.washingtonmonthly.com/archives/individual/2004_08/004536.php. You can also email me directly and I can share the math.
Best of luck in Abu Dhabi!
Andrew B. Gardner
PhD EE
Well, I agree that the issue is more complicated than I could state in a few paragraphs. Public opinion experts don’t agree on exactly how margin of error should be calculated and reported. For instance, Harris Polls doesn’t even release a margin of error figure – they insist that the term itself is misleading. I should not have presented my perspective about reporting poll results as though I had the final say on the matter – I don’t.
That said, I feel comfortable with my position. It agrees with the conclusions of several peer-reviewed journal articles that have studied newspaper poll reporting. One such article is: “USA Today Reports of Tracking Polls Sometime Ignore Sampling Error,” by Matthew Reavy (Newspaper Research Journal; Spring 2004, Vol. 25 Issue 2). That author follows the formula used by the National Council of Public Polls: 1.96 x sqrt (P x (1-P)/N) where P is the population proportions for a given candidate and N is the “effective sample size.” “In order for the difference between the two candidates to be deemed significant,” Reavy writes, “that difference must be greater than the cumulative sampling error (the sum of the two candidates’ sampling error.)” Plugging this poll’s numbers into the NCPP formula produces a margin of 4.38 – twice that margin is 8.76. Since one candidate is ahead by 8 points, the poll numbers fall within the margin of error – at least according to NCPP method.
I understand that such a broad definition may not agree with other methodologies. However, the NCPP definition serves the greater good since journalists aren’t likely to have statistical calculators lying around to help evaluate a poll’s result.
Regardless of these specifics, my two major complaints about poll reporting should not be missed: 1) Polls are too often constructed with sample sizes that are too small to offer valid results. 2) Journalists too often engage in “horserace journalism” – where fluctuations in poll results are analyzed in news stories, rather than concentrating on the policy differences between candidates.
Matt,
I appreciate the blog and your active participation with your followers — keep up the good work! Now, back to the thread….
I agree that there are many important, debatable and open questions in journalism on the topic of poll reporting. However, if one accepts that a poll was conducted in a fair, unbiased manner, then the calculation and interpretation of basic poll statistics is not one of these questions. In response to your recent comments, here are four points for you to consider:
(1) I have contacted Matt Reavy and the National Council of Public Polls (NCPP), and both have pointed out that the organization actually addresses the very issue of “does A lead B”: http://www.ncpp.org/node/4/#12. The NCPP recommendation to journalists is: “When the gap between the two candidates is more than the error margin but less than twice the error margin, you should say that Candidate A “is ahead,” “has an advantage” or “holds an edge.” The story should mention that there is a small possibility that Candidate B is ahead of Candidate A.” Clearly, from the NCPP recommendation, the poll did not indicate a statistical dead heat, and it would be proper, by convention, for the paper to state that “LePage is ahead in the race.”
(2) More precisely, however, if one works out the math, they will find that it is statistically accurate to state that “LePage *leads* the race.” There is no need to soften the claim at all! We can use the stronger statement because our calculations for *this poll instance* justifies it. Why? According to commonly accepted practice, to state that “LePage leads the race” requires that our statistical confidence in that result is at least 95%. “The result” refers to the potential difference in true poll proportions between the two candidates. In this poll instance, using the numbers cited for proportion and sample size, we confirm our requirement, that our 95% confidence interval for the true difference between LePage and Mitchell’s proportions is > 0. There is a single interpretation for our result under the “fair poll” assumption: if the election were held today, we have 95% or greater confidence that LePage would win. Now, I wouldn’t necessarily expect a journalist to report using the stronger claim, but it is not difficult to confirm, either by online calculators, statistical software, or tutorial examples available in books and journal articles.
(3) The exercise in #2 is common in statistics, and the technique that applies is called “estimating the difference between proportions for dependent samples.” Polling — in particular the “A leads B” question– is the classic example used to teach it at the AP high school, undergraduate and graduate levels of study. For one succinct and accessible reference, please visit http://bit.ly/aXqexf. In addition to a numerically worked example, they give the precise formula for estimating the confidence interval associated with the difference of proportions between two candidates in a poll. As expected, it differs from the formula you cited above (which is applicable only to the margin of error associated with estimating a single candidate’s proportion). The formula that applies to our example is correctly stated as: error = 1.96 * sqrt((p1 + p2 – (p1 – p2)^2) / n), where p1, p2 are the candidate proportions, and n=500 is the sample size.
(4) The point that seems to have been lost in our thread was the original error in reasoning: the margin of error in a poll does not directly address the “A leads B” question. Hopefully, points 1 – 3 have convinced you that, at a minimum, the polling and professional journalism communities have rules-of-thumb which indicate this, and the paper did not make an error.
On a more personal note, let me say that this has been an interesting discussion. I’m sure that as a newly-minted doctoral graduate, and junior faculty member, you strongly support quantitative journalism. Perhaps we can explore this topic more formally as co-authors on a tutorial paper?
Thanks and good luck with your classes!
Andy
I appreciate this discussion as well. It forced me to check with a former colleague (my partner in crime regarding this issue) at the Nashua Telegraph to verify my point. I’ve found this process to be very enlightening, and I do see a journal article in the future. We’ll have to discuss…
I’ve been aware that both the NCPP organization and the AP stylebook insist on the “apparently leading” phrasing when dealing with an MOE that’s greater than one but less than two times the difference between candidates. However, I’ve been told by several statisticians that there’s no statistical difference between the two. Statistically speaking, there’s no reason to think that an 8-point lead is more meaningful than a 1-point lead if the margin of error is 4.5.
Ignoring this point for a moment, at the very least, the Portland newspaper was guilty of not following the NCPP and AP guidelines. Given the gap between the candidates, the correct phrase should have read that “Page apparently leads the race.” That much, I think, is indisputable.
My former colleague, David Brooks, the science correspondent of the Nashua Telegraph, re-checked with a statistician as well. He got an interesting response — one that will likely make me modify my position slightly.
Rather than try to summarize the point, I’m going to wait for David to write his column about the issue, and then I’ll post a link. He said he’d publish it in the next month or so.
Again, I do appreciate the peer-review.