I have been “mistaken,” “misled,” “misrepresented,” and been “unaccountably in error,”
and am sorry if you have been offended
Saturday, January 05, 2008
Good Example of the Need to Look behind the Numbers
National Journal debunks an estimate of Iraqi deaths since the U.S. went to war there in 2003. It highlights the dangers of failing to look behind the numbers. I found it through the Real Clear Politics website.
Perhaps a look behind the look behind the numbers is also in order. The National Journal article, like some other critiques of the Lancet studies, partially misrepresents the studies' claims and oversimplifies the incredibly complicated business of making such an estimate. While this might seem an odd source, this 2006 episode of This American Life and this one from 2005, both have one of the studies' authors on explaining how they got their numbers and what the numbers mean.
Thanks, Garret. Unfortunately, it is hardly an exaggeration to say that the 2005 and 2006 podcast interviews do not take up any of the specific points raised in the NJ compilation of the critiques listed under “Design and Implementation” and “Black-Box Data”. The 2005 interview basically takes on bloggers in general terms, and the message appears to be, look, we did the best we could, given the circumstances. Statistical concerns are addressed in very general terms as a confidence interval issue. The 2006 interview does take up a couple of issues raised by Iraq Body Count (and does bring up death certificates) but does not go into the related specific points raised in the NJ. In fact, the main objective in bringing up the IBC argument appears to be to lend credibility to the claim at the end of the interview that Dr. Roberts and his colleagues would not have produced such incredible numbers if they had intended to fake the data.
The problem with the interviews from a journalistic standpoint is that they don’t ask any hard questions, nor even try to draw the interviewee out Inside the Actors Studio style.
Perhaps a look behind the look behind the numbers is also in order. The National Journal article, like some other critiques of the Lancet studies, partially misrepresents the studies' claims and oversimplifies the incredibly complicated business of making such an estimate.
ReplyDeleteWhile this might seem an odd source, this 2006 episode of This American Life and this one from 2005, both have one of the studies' authors on explaining how they got their numbers and what the numbers mean.
Thanks, Garret. Unfortunately, it is hardly an exaggeration to say that the 2005 and 2006 podcast interviews do not take up any of the specific points raised in the NJ compilation of the critiques listed under “Design and Implementation” and “Black-Box Data”. The 2005 interview basically takes on bloggers in general terms, and the message appears to be, look, we did the best we could, given the circumstances. Statistical concerns are addressed in very general terms as a confidence interval issue. The 2006 interview does take up a couple of issues raised by Iraq Body Count (and does bring up death certificates) but does not go into the related specific points raised in the NJ. In fact, the main objective in bringing up the IBC argument appears to be to lend credibility to the claim at the end of the interview that Dr. Roberts and his colleagues would not have produced such incredible numbers if they had intended to fake the data.
ReplyDeleteThe problem with the interviews from a journalistic standpoint is that they don’t ask any hard questions, nor even try to draw the interviewee out Inside the Actors Studio style.
I mean, Garrett. Beg your pardon.
ReplyDelete