Forum

joe emersberger

Mr. Pederson:

Fred Kaplan quoted you in Slate days ago in a response to an article
published by Les Roberts and Gilbert Burnham. According to Kaplan you of the second study the Lancet published on mortality in Iraq:

""I simply do not believe that it is possible to obtain accurate mortality
estimate[s] with the methods they are using as far back in time as they
purport to do." He adds (and I agree here wholeheartedly), "That being
said-a lot of people have died in Iraq."

The Washington Post quoted you as [saying] that the Lancet study's findings [were]

"high, and probably way too high. I would accept something in the vicinity of 100,000 but 600,000 is too much.'

I have emailed you twice asking you if the Washington Post quote was
accurate - as did Les Roberts himself. You have not replied to me or (to my knowledge) Les Roberts.

Were you quoted accurately by Kaplan and by the Washington Post?
Why would you respond to Fred Kaplan but not to Les Roberts?

Regards,

Joe Emersberger

Dear Joe Emersberger,
I am sorry, but I have neither seen your nor, Les' comments to me. I would of course have responded if I had seen them.
Jon

Thank you for acknowledging my email. I was hoping you would reply to the questions

1) Were you quoted accurately by the Washington post?
2) Were you quoted accurately by Fred Kaplan in Slate?
3) What do you estimate as the pre-ivasion mortality rate in Iraq based on the data you gathered?

Thanks for any help.

Best wishes

Joe Emersberger

Dear Joe Emersberger,
1) yes, more or less, would perhaps say 150,000
2) yes
3) As should be clear from what I have been saying generally, not really, but there is nothing in the ILCS data, that are inconsistent with what I have been saying, i.e. that I believe the pre-invasion mortality to be higher than 5.
All the best,
jon

Mr. Pedersen

Thanks for confirming that you were quoted accurately.

You are therefore saying that the Lancet study's estimate for mortality due to the war is too high by a factor of about 4. Moreover from your remarks to Kaplan it appears that you believe the main cause of this drastic overestimate is recall bias. Is that correct?

Why not explain your reasoning for this assessment and have it published - say in a letter to the editor of the Lancet, or in some other publication so that the substance of your critique is clear?

For example, from your remarks it is not clear to me, (and I sure many others who have followed the debate over mortality due to the war) how the study you did suggests a pre-invasion death rate greater than 5. Why not calculate the pre-inasvion death rate from your data, or if this is not possible for a time period which is covered by the data? Is that possible? Was it done?

Thanks again for replying.

Joe Emersberger

joe emersberger

Dear Joe Emersberger,
You are wrong in your inference from my remarks to Kaplan. I did not say that the error was due to recall bias. I said that underestimation of the baseline rate might be due to that. I would think that part of what has happened with the Lancet study is that deaths were transferred from non-violent to violent. My initial comments on the baseline rate was mainly built on the UNPD rates (and considerable experience in Middle East demography).
yes, and no - it is possible in the way that the Lancet study did, probably not possible using proper demographic assumptions.
And yes, I may publish something (If I can find the time to write).
regards,
Jon

Mr. Pedersen:
I urge you to publish your critique of the study as soon as possible. It is particularly important that people know how you arrived at your estimate of 150,000 dead. As I'm sure you are aware the Iraqi Minister of health made a similar estimate recently.

As for recall bias skewing pre-invasion rate, what do you make of the fact that the 2004 study found essentially the same pre-invasion death rate? Doesn't that significantly weaken the argument that recall bias was significant in 2006?

Your comments that death rates were "transferred from violent to non-violent" are interesting but, even if true, wouldn't that effect the sub totals, not the overall total death estimate?

If you calculated a pre-invasion death rate "the way Lancet did" what would the results be?
Could you elaborate on what you mean by "proper demographic assumptions" for the calculation.Are you referring to the UN estimate of 10 as a pre-invasion death rate? Wasn't that based on census data about a decade old?

I think you can appreciate from these questions why it would be extremely helpful if you could publish your criticism of the study. This couldn't be more important or deserving of your time.
Regards,
Joe Emersberger

joe emersberger

Mr. Pedersen:

Couldn't a study with a much smaller sample size (say the type used for all the Iraqi opinion polls we see) confirm if the Lancet study findings are reasonable? If not, what type of field work would you suggest to do a quick check on the findings?

Thanks again.

Joe Emersberger

joe emersberger

BELOW FROM STEPHEN SOLDZ ON HIS CONVERSATIONS WITH PEDERSDEN

I have had a number of emails with Jon Pedersen going back to last spring, if I remember correctly. I also had the privilege of having lunch with him and discussing these issues for almost a couple of hours.
He struck me as extremely thoughtful and as having no ax to grind. As I relate his thoughts, [I am responsible for this account, which could be incorrect on details. Comments in brackets are my thoughts.] he believes the Lancet rates are too high. He thinks two things are involved, and had two other comments on the methodology.

1. The prewar mortality is too low. This would be due to recall issues. He asserted that demographers are well aware that there are problems with assessing mortality by survey more than 1 year in the past. [See, for example, the SMART methodology which states that one should never assess mortality beyond one year: "Accuracy: The shorter the recall period, the more accurate the estimate of mortality. This is because more distant events are more likely to be forgotten by respondents and there are likely to be more mistakes in the time of death. (Recall periods longer than one year should not be used.)" (p. 31-32)]

[As a research psychologist, this is not surprising. We routinely find that memory, even for major events, gets transformed relatively quickly. As one minor example, one half of those who, at age 18 reported major depression did not report ever having had major depression ever, when reinterviewed at age 21. Especially in the context of a brief interview, it is entirely possible that people could briefly "forget" deaths a ways back, or "choose" not to talk about them as they may be painful. I, too, was concerned about this recall issue.] Pedersen said it was possible that this effect may be larger for nonviolent than for violent deaths.

[Note that, if the UN prewar mortality rate of 9.03 is correct, this will not, by itself, change the qualitative message of the study. My calculations are that the excess mortality would then be about 350,000. But other problems would arise, such as the mortality rates for the two years postwar, which would then be low. These might support the Pedersen/SMART position that 1 year prior is the outer limit for good results with this methodology.]

2. Pedersen thought that people were likely reporting nonviolent deaths as violent ones. [One argument for this is the surprisingly low postwar mortality rates in the Lancet 2006 study, which Pederesen thought were unlikely to be correct.] We did not discuss the precise mechanisms for this misreporting, or Pedersen's other reasons for believing it.

[We did discuss the issue of the death certificates, to which Pedersen said "who knows what's on them." I have since found two accounts of death certificates which indicated that they have details of the death. More details from the Lancet authors as to the details of how and to what degree interviewers examined the certificates would be helpful.]

3.A third issue concerns the inability, because of security, of the US Lancet team to supervise the interviewers. Pedersen raised the possibility of his FAFO team repeating the survey, but was extremely hesitant as they would not be able to closely supervise the interviewers and,therefore, could not vouch for the protocol being followed exactly.

4. Pedersen believes the confidence intervals (CIs) for the Lancet studies are too small. This involves two issues. First, the calculated CIs only include sampling error, but the study also includes nonsampling error, as in when a different household was chosen for security reasons or the protocol wasn't followed for other reasons. Recall issues and other forms of inaccurate reporting would also increase nonsampling bias.

A second reason the CIs are too small is a technical one. Death is a relatively rare event. The statistical methods used to calculate CIs [linearized variance estimators] are based on an assumption that the events aren't too rare. Thus, the CIs are likely too small.

[While the Lancet 2006 authors say they also calculated bootstrap standard errors, they give no detail on how this is done, and are not clear which CIs they report. Bootstrap CIs with survey data are very complicated and can be done many different ways with, potentially, different results.]

Two additional points: Pedersen acknowledges that the ILCS cannot be the gold standard for mortality as asking one brief set of questions at the end of a long survey would lead to underreporting of mortality. Thus, he assumes that the ILCS figures are low. But he thinks they are in the ballpark.

Another point: He thinks that the IBC argument about how the number of bombings reported in the Lancet 2006 study would result in such a large number of wounded that it seems too large. If I understood correctly, he does not buy the IBC argument about the inconsistency between the estimated wounded and those reported as treated in hospitals, as he agreed [I thought] with a point I made that there were likely serious problems with Iraq's health surveillance systems, so that these figures cannot be taken very seriously. But, he felt that 800,000 or more wounded would swamp any public heath system in the world. [An additional point occurred to be after the conversation. As one who has read the press on Iraq, including Iraqi bloggers, very carefully, I can't remember reading articles about the walking maimed of Iraq. It does seem that, if the Lancet mortality figures were correct, that the magnitude of wounded would be such as to be reported frequently.]

---------------
I want to reiterate that Jon Pedersen, as far as I could tell, was extremely thoughtful. Several times he raised an argument only to raise a counterargument. He did NOT appear to be argumentative or to be involved in proving a point. I was extremely impressed. Therefore, I take his arguments and thoughts seriously. He may know more about middle east demography and survey methodology than anyone, so his judgments and intuitions cannot be simply dismissed.

I would also remind people that Les Roberts refers to Pedersen as "he highly revered Jon Pederson." Pedersen also spoke highly of Roberts. These are scientific debates, and scientific debates do not always have bad guys. They also are often unresolvable with the current state of knowledge.

By the way, Pedersen did NOT think that there was anything to the "Main Street Bias" issue. He agreed, I thought, that, if there was a bias, it might be away from main streets [by picking streets which intersect with main streets]. In any case, he thought such a "bias", if it had existed, would affect results only 10% or so.

As to his not communicating with Les Roberts, both he and I have sent emails to Les [I also sent one to Gilbert Burnham] which have not been responded to. There have been hundreds of emails on this issue, and people are very busy. Emails have been missed by many participants. Please cut out the snide comments about why people didn't send emails to someone.

As to Pedersen's writing a letter on this, I would hope he would. On the other hand, remember that he's got a lot of other things going on. As far as I'm aware, he hasn't sought out the role of critic but simply responded to those who asked his opinion. While such a communication would be very helpful, he is not obligated to undertake it.

So, the bottom line is that two experts differ. The rest of us are left trying to assess the differences. But we are also left not knowing.

That said, I'm not sure that the true mortality figure is as low as the 150,000 that Pedersen cites. I suspect the true figure is somehat higher, but below the 650,000 of Lancet 2006. But I can't provide any strong arguments for my instincts here. I now take 150,000 that as the rock bottom number for Iraqi excess deaths. There may be more, even many more, but we can't be sure. It simply is very difficult to get accurate figures in a time of chaos.

I would say that the moral case regarding the war is well made if the true excess mortality is "only" 150,000. This war and occupation has resulted in horrifying death and destruction. It is morally unjustifiable on any grounds of reducing harm.

joe emersberger

REPLY TO SOLDZ

Very interesting and reasonable comments throughout but I'm afraid we part company when you say

"As to Pedersen's writing a letter on this, I would hope he would. On the other hand, remember that he's got a lot of other things going on. As far as I'm aware, he hasn't sought out the role of critic but simply responded to those who asked his opinion. While such a communication would be very helpful, he is not obligated to undertake it."

On the contrary all of us who live relatively free and privledged countries should feel very obligated - much more if our privledges have allowed us to aquire high degree of technical competence in this field.

No one has suggested he put his life at risk. He should detail the substance of his criticism clear and thereby reduce the opportunities for war mongers to distort it.

For someone like him to say he's busy too wade into this (beyond making very general and brief remarks to journalists) is not a morally defensible position.