Discussion with authors of Lancet critique

[AlyK]

Dear Prof. Johnson,

I just came across your recent piece in Science and I was wondering whether I might ask a couple of questions. In the report, there is the following statement:

"[They] also question whether the sample is representative. The paper indicates that the survey team avoided small back alleys for safety reasons. But this could bias the data because deaths from car bombs, street-market explosions, and shootings from vehicles should be more likely on larger streets.."

I have a couple of questions about this,

1. Is this quantifiable? Are you able to estimate the level of uncertainty that such a bias (assuming that the interviewers did, in fact, avoid these areas) would produce? Are we talking a ballpark of 10% or 100%? Does this effectively invalidate the 400,000-900,000 confidence interval?

2. Is there not a risk of overestimating the bias? For example, street markets would likely be in major streets, but populated equally by people from different (nearby, but non main) streets. That is the market would not be populated exclusively by people living in that street. In fact main streets would be bustling with people from a wide mix of areas - they are *main* streets after all. A bomb in a market place or a shooting in a main street would not necessarily kill only people who also live in main streets. Would this not tend to equalise the death rate among people in the cluster, regardless of where they lived?

3. Has this bias been observed before in similar studies? That is, is this a standard criticism of such studies, or is this an essentially new observation?

Many thanks for your time,

Aly Kassam

[AlyK]

Dear Aly
Actually the bias is even worse than suggested by the Science
article. It seems the researchers focused around main roads,
therefore missing whole blocks of residential streets which
- are not back-alleys, in the sense they are heavily populated
- do not lie on a road that convoys, patrols etc. would regularly
take -- and may not even be formal roads (i.e. unpaved)

So yes there are people in main streets who come from nearby places,
but their chances of being there presumably fall off the further away
their house is. Just imagine a housing area off the main streets in
Oxford (see attached). The roads in red are ones that would not be
covered by the survey.

This approach is OK if you are doing a survey of incidence of a
disease, or vaccinations, or favourite TV program, since these don't
have such a main-street dependence. But we are now looking into where
this technique may have been (mis)used in other conflicts in the past.

We are also now starting to simulate this effect based on maps, to
quantify the overestimate.
Best
Neil

[AlyK]

Hi Aly,

I'll take a few of the points if you don't mind

On 20 Oct 2006, at 10:29, Aly Kassam wrote:

> Dear Prof. Johnson,
>
> I just came across your recent piece in Science and I was wondering whether I might ask a couple of questions. In the report, there is the following statement:
>
> "[They] also question whether the sample is representative. The paper indicates that the survey team avoided small back alleys for safety reasons. But this could bias the data because deaths from car bombs, street-market explosions, and shootings from vehicles should be more likely on larger streets.."
>
> I have a couple of questions about this,
>
> 1. Is this quantifiable? Are you able to estimate the level of uncertainty that such a bias (assuming that the interviewers did, in fact, avoid these areas) would produce? Are we talking a ballpark of 10% or 100%? Does this effectively invalidate the 400,000-900,000 confidence interval?

This is more difficult to say, as the Science article noted the Research team destroyed their data and are not able to say what neighbourhoods were used and did not even provide information about a specific list of main roads except to say that a main road constituted a "major avenue or commercial road".

-> if they take a standard definition of major avenue + commercial road then they could be sampling just 20% of the population for somewhere like Baghdad.

-> the best comparison as far as the number of death go is to look at the UNDP survey done for the first year of the war 2003-04. This survey consisted of 21,668+ households each sampled from a random list of houses within the particular area. i.e. there was no main street bais to speak of - each house had a chance of being selected in the study.

-> the UNDP survey for this period of time estimated between 18,000 and 29,000 dead which if extended

(2) There are a couple of key points that make this survey relevant.
(a) the survey did not include any "main street bias", meaning each household had a roughly equal chance of being selected
(b) without the 'main street bias' the survey for the same period of time was ~4 times lower than the Lancet report**


** see http://www.iraqbodycount.org/editorial/defended/3.6.2.php for details of the scaling up of the UNDP survey to 14 month period over which the first set of figures in the L2 paper refer to. Essentially 28,000 for the UNDP paper scaled up and 112,000 for the L2 paper (all from violent causes since the non-violent death rate drops to below the pre-war estimate during this period). Then divide the two (112/28)=4

- this shows that with the 'main street bias' removed you might get a number 4 times less for the first year - my feeling is that this would only increase in the following years as the percentage of attacks from car bombs and IED's start to increase and we see an increase in death from violent causes.

>
> 2. Is there not a risk of overestimating the bias? For example, street markets would likely be in major streets, but populated equally by people from different (nearby, but non main) streets. That is the market would not be populated exclusively by people living in that street. In fact main streets would be bustling with people from a wide mix of areas - they are *main* streets after all. A bomb in a market place or a shooting in a main street would not necessarily kill only people who also live in main streets. Would this not tend to equalise the death rate among people in the cluster, regardless of where they lived?
>

If there was unlimited movement of people between all locations all of the time i.e. heterogeneous mixing of the population - then some of the bias would be reduced. But this is definitely not the case travel times and distances are severely reduced for somewhere like Baghdad where road blocks and the general fear of violence stop you from leaving your own neighbourhood.

> 3. Has this bias been observed before in similar studies? That is, is this a standard criticism of such studies, or is this an essentially new observation?

one interesting example is that in the first Lancet study for the controversial Falluja

>>>one final point, we went back through their original Lancet paper and looked at the sampling method there - they used a GPS device to make a more representative sample. Except for the case of Falluja where they employed the 'main street bias' method described in the the second lancet paper.

This means that they found a main street in falluja and sampled houses located near this main street and claimed that this was somehow representative of the number of dead in the whole city.

(2) we are also looking at other studies including Congo - and there is evidence there for main street bias too. Though it's impact is will not be as great due to the reduced percentage of violent deaths out of the death total.

Sean

>

[AlyK]

Dear Prof Johnson,

many thanks for the speedy and interesting response. I just want to clarify something, though.

As far as main street dependence goes, aren't you making a number of implicit assumptions about the target, victims and type of attacks? A bomb or shooting at a market place, for example, doesn't seem to suggest that the victims will predominantly be "main-streeters". Especially if this is the main market for the neighbourhood. Similar conclusions could be drawn with regard to attacks on office blocks, police stations etc. The victims are likely to be a random cross section of people, or have I misunderstood something?

I think your point is very valid with regard to attacks on US convoys that might be moving exclusively on such roads, but again, I wonder whether the make-up of the victims will be heavily biased. There might be a time dependancy which determines to some extent the type of person to be found in the neighbourhood, i.e. main-streeter or not.

I would be very interested in the results of your simulations. Are you planning on publishing them?

Finally, do you have any objection with me these messages with a discussion board that I subscribe to?

Thanks again

Aly

[AlyK]

Dear Aly

No problem about the message board.

The way I see it is this: this Lancet article was/is hugely
influential, but the authors cannot rule out (or offer a coherent
rejection of) the idea that there is a main street bias.
Yet they are the authors who make the claims about the large numbers
-- so it is their job in the first instance to remove this potential
problem with the bias. They have to show that their results do not
depend on it. But when we were discussing this with Science, and the
authors replied, one of them even admitted to not knowing exactly how
the survey was done. So this, in my mind, undermines completely the
study's methods and hence conclusions. I am not saying the actual
numbers aren't large or the war is wrong/right -- just an academic
looking at an academic study and finding fundamental problems. I
think if this were a thesis, it wouldn't pass the viva!

The next (independent) step is that we finish our simulation and
publish it.

Many thanks for your continued interest in this. I don't myself make
judgements on wars since I am not an expert in that -- however I am
happy to make judgements on what I see as flawed scientific
research ;-)

Neil

[AlyK]

Hi Sean

thanks for the reply. Interesting stuff. I just sent in another email to Prof Johnson, can't remember if I copied you.

The other thing I meant to ask was it was mentioned that you guys are doing some other study on the mortality figures, do you mind me asking what that study is?

Cheers

Aly

[AlyK]

Dear Prof Johnson

Many thanks.

I think yours is the first serious critique of the methodology, and is probably the most interesting, certainly the most important, for future studies. The reason that I'm interested in your results is that they actually deal directly with the science involved.

It is an important area, but that just makes it more important that the numbers being produced by everyone are as "watertight" as possible.

I'm still interested in this last bit.

"As far as main street dependence goes, aren't you making a number of implicit assumptions about the target, victims and type of attacks? A bomb or shooting at a market place, for example, doesn't seem to suggest that the victims will predominantly be "main-streeters". Especially if this is the main market for the neighbourhood. Similar conclusions could be drawn with regard to attacks on office blocks, police stations etc. The victims are likely to be a random cross section of people, or have I misunderstood something?"

Anyway. I appreciate your responses so far!

Cheers

[AlyK]

Dear Aly

I think the points are:
- we have identified a potential cause of bias, it is now the
authors' job to show that this bias was avoided or unimportant. They
have been challenged to do this, but have failed to do so. We have
back-tracked to look at all the WHO guidelines for conducting health
surveys -- and while fine for health, they would also have such a
potential bias for surveys related to political/economic issues (as
opposed to health or human tastes etc.).
- we cannot quantify the effect of the bias yet, hence the
simulation. But it is clearly there.
- such biases are not unknown in epidemiology studies applied to non-
health issues: see Wiki for story of US election survey bias..
http://en.wikipedia.org/wiki/Bias_%28statistics%29
Best
Neil

[AlyK]

Dear Aly

> As far as main street dependence goes, aren't you making a number of implicit assumptions about the target, victims and type of attacks? A bomb or shooting at a market place, for example, doesn't seem to suggest that the victims will predominantly be "main-streeters". Especially if this is the main market for the neighbourhood. Similar conclusions could be drawn with regard to attacks on office blocks, police stations etc. The victims are likely to be a random cross section of people, or have I misunderstood something?
>
Again this would be true if you have an unlimited mixing of the population - if anyone could be anywhere in the city with equal probability. But in somewhere like Iraq you tend not to travel far from home - travel is difficult and roads are a dangerous place to be . Your location is biased towards the region of your house - you don't have to be at your house 100% of the time you just need to have a preference - this bias is then reflected in the increased death rate.


> I think your point is very valid with regard to attacks on US convoys that might be moving exclusively on such roads, but again, I wonder whether the make-up of the victims will be heavily biased. There might be a time dependancy which determines to some extent the type of person to be found in the neighbourhood, i.e. main-streeter or not.

if a US convoy is rolling up your street and is being attacked by insurgents are you going to be there unless you live there and have no where else to go?

Sean

[AlyK]

> Hi Sean
>
> thanks for the reply. Interesting stuff. I just sent in another email to Prof Johnson, can't remember if I copied you.
>
> The other thing I meant to ask was it was mentioned that you guys are doing some other study on the mortality figures, do you mind me asking what that study is?

we are looking at patterns within data in conflict zones around the world - focussing on three questions, when do attacks happen, where do they happen and how big are they.

for this we are not interested in absolute values, i.e. how many people died in a war - rather we look for trends and patterns which we then model in order to understand the conflicts. The paper we are writing on this is in the final stages of being written.

Sean

[Felix]

Dear Dr Gourley,
I was interested to read about your criticism of the Johns Hopkins Iraq mortality survey. I wondered how serious you thought that 'main street bias' was in potentially altering the large projected mortality results recently published in the Lancet?
I also wondered how you and your colleague, Prof Neil Johnson and Professor Michael Spagat came to be involved in a matter of medical epidemiology?
I would be interested to hear your reply,
Felix

[Felix]

Hi (Felix),

thanks for you questions


> Dear Dr Gourley,
> I was interested to read about your criticism of the Johns Hopkins Iraq mortality survey. I wondered how serious you thought that 'main street bias' was in potentially altering the large projected mortality results recently published in the Lancet?

>>1. Is this quantifiable? Are you able to estimate the level of uncertainty that such a bias (assuming that the interviewers did, in fact, avoid these areas) would produce? Are we talking a ballpark of 10% or 100%? Does this effectively invalidate the 400,000-900,000 confidence interval?

This is more difficult to say, as the Science article noted the Research team destroyed their data and are not able to say what neighbourhoods were used and did not even provide information about a specific list of main roads except to say that a main road constituted a "major avenue or commercial road".

-> if they take a standard definition of major avenue + commercial road then they could be sampling just 20% of the population for somewhere like Baghdad.

-> the best comparison as far as the number of death go is to look at the UNDP survey done for the first year of the war 2003-04. This survey consisted of 21,668+ households each sampled from a random list of houses within the particular area. i.e. there was no main street bais to speak of - each house had a chance of being selected in the study.

-> the UNDP survey for this period of time estimated between 18,000 and 29,000 dead which if extended

(2) There are a couple of key points that make this survey relevant.
(a) the survey did not include any "main street bias", meaning each household had a roughly equal chance of being selected
(b) without the 'main street bias' the survey for the same period of time was ~4 times lower than the Lancet report**


** see http://www.iraqbodycount.org/editorial/defended/3.6.2.php for details of the scaling up of the UNDP survey to 14 month period over which the first set of figures in the L2 paper refer to. Essentially 28,000 for the UNDP paper scaled up and 112,000 for the L2 paper (all from violent causes since the non-violent death rate drops to below the pre-war estimate during this period). Then divide the two (112/2=4

- this shows that with the 'main street bias' removed you might get a number 4 times less for the first year - my feeling is that this would only increase in the following years as the percentage of attacks from car bombs and IED's start to increase and we see an increase in death from violent causes.
________

also see attached image of the 'main street bias' for Oxford where I live - we can see that many (if not most) of the roads cannot be sampled using this technique. We would like to do the same for Iraq but the authors of the study 'destroyed all the material' identifying neighbourhoods and roads that were used.




> I also wondered how you and your colleague, Prof Neil Johnson and Professor Michael Spagat came to be involved in a matter of medical epidemiology?

We have been investigating the conflict in Iraq as part of a different study and were asked to comment on the Lancet figures. We then found the 'main street bias' and determined that this would lead to a significant over-estimation of the numbers. We are not medical epidemiologists - but this question is one of sampling bias and in order to see this you have to take a step back from what the authors call 'standard procedure' to see that the standard procedure is actually flawed - it may be fine for studies of immunisation for which the test was designed, but it is entirely unsuitable for estimating violent deaths


Sean

[Felix]

Dear Sean,
Thanks for your quick reply on a day where you must be snowed under by emails!

I've a couple of extra questions, if that's OK.

Is the 'main street bias' that you describe, a novel one postulated by yourselves?

Do you think that such a bias accounts for all the differennce between the Burnham et al study and the UNDP Iraq Living Conditions Survey (ICLS)?

There could be other reasons to account for the difference in mortality figures for 2004 between the 2 studies, the ICLS was not designed to detect mortality and it's surveying technique was criticised by it's lead researcher Jon Pederson. (http://online.wsj.com/public/article/SB112309371679604061.html).

If this main street bias that you descibe is based solely on the discrepancy between Burnham et al and the ICLS figures, how can you be confident that other factors (for example mentioned above) are not relevant?

Thanks again (in advance!) for taking the time to reply to me.

Yours,

(Felix)

[Felix]

> Dear Sean,
> Thanks for your quick reply on a day where you must be snowed under by emails!
>
> I've a couple of extra questions, if that's OK.
>
> Is the 'main street bias' that you describe, a novel one postulated by yourselves?

as far as we know we were the first to pick up on this

>
> Do you think that such a bias accounts for all the differennce between the Burnham et al study and the UNDP Iraq Living Conditions Survey (ICLS)?
>
there are other elements of the surveys that differ, but as far as we can tell the main one for the

> There could be other reasons to account for the difference in mortality figures for 2004 between the 2 studies, the ICLS was not designed to detect mortality and it's surveying technique was criticised by it's lead researcher Jon Pederson. (http://online.wsj.com/public/article/SB112309371679604061.html).

it wasn't exactly critiqued by the lead researcher in this article - but he did express reservations. That said they had issues with infant mortality being low and so went back and re-sampled the data for 500 Baghdad homes and had the same set of results. This should provide a good measure of the accuracy of their interview methods. There were no such re-checks done in the Lancet paper.

It's true that the specific goal of the paper was not to determine mortality - however each interview lasted ~ 80 minutes and so there was more than enough time to investigate the issues of death/mortality with each family in question.


> If this main street bias that you descibe is based solely on the discrepancy between Burnham et al and the ICLS figures, how can you be confident that other factors (for example mentioned above) are not relevant?

of course other factor will be relevant - they are different surveys - but it does give us an estimate of potential differences.

-> We know that a bias existed in the authors methodology

-> We know that this will cause an increase in deaths

-> The authors have destroyed all reference to streets and neighbourhoods and do not know the extent of the bias themselves

-> we are forced to estimate the impact - we are currently conducting simulations to get another estimate

-> the biggest survey of its type conducted in Iraq without 'main street bias' estimated ~4 times lower death rate
>
> Thanks again (in advance!) for taking the time to reply to me.

this main street bias argument also questions the authors assertions that media have not picked up the overwhelming majority of deaths in Iraq. The authors claim that the media is only reporting 1 out of every 20 car bombs (email me for the maths) across the country. Their argument is that media never venture far from main roads.

However their sampling procedure never ventures far from main roads either

Sean

[Felix]

Dear Sean

Again thanks for your prompt replies. This is very interesting, given the huge numbers reported by Burnham et al.

I'm sorry to take your time, and I will go away (!), but a couple of further questions have occured to me;

How marked do you think the 'main street bias' is in the non-urban areas surveyed?

Is it as relevant in rural or less densley populated areas?

Do you have any evidence that car bombs and IEDs are more frequent on main streets rather than side streets?

Yours,

(Felix)

[Felix]

> Dear Sean
>
> Again thanks for your prompt replies. This is very interesting, given the huge numbers reported by Burnham et al.
>
> I'm sorry to take your time, and I will go away (!), but a couple of further questions have occured to me;
>
> How marked do you think the 'main street bias' is in the non-urban areas surveyed?

it should be even more marked since there is likely to be only one major road/commercial avenue so all the houses surveyed must then reside in streets directly crossing this. i.e. exactly where the violence will occur if any does

>
> Is it as relevant in rural or less densley populated areas?

assuming that major avenues even exist in rural areas (again the authors have said that they do not have access to this information any more) it applies equally well to both areas.

>
> Do you have any evidence that car bombs and IEDs are more frequent on main streets rather than side streets?
>

Coming from the IBC dataset we know some information about the preferred location of car bombs and IED attacks. They majority that are located are located near markets and 'main streets', road blocks and security check-points - all of which are on or near main roads.

those that are not located on a specific street are listed as attacks on 'coalition convoys', 'US military supply', 'police patrols' - again all of these have a strong bias towards main streets and the streets connected to main streets. A lot of these houses that cannot be surveyed using the methodology are located two or three turns off a main road - there is no reason to go there unless you are going specifically to that house.

If you look at it from a pragmatic point of view as an insurgent you have two goals - kill people in a large event and maximise your publicity

with this in mind it doesn't make sense to place a car bomb anywhere but a public place and by definition a car bomb has to be in a car and thus on a street.

-> the main point here is that there is certainly evidence - both empirical and theoretical - that car bombs and other attacks have a bias towards areas near main streets. It is not our job to prove this connection - it is the job of the authors to say that there is no bias. This is because they have designed a survey that only samples on or near a main road and as such they have to prove that the death rate here is representative of the country as a whole.

hope this helps

Sean

[Felix]

Dear Sean,

Thanks for the amount of time you've given me, just a couple of points;

Firstly, is there a more comphrehensive location detailed on the IBC database? I can only find a very vague location detailed on the webpage database (Baghdad, Basra, etc.) Is it normal for street details to be carried in the media reports that IBC uses? If there is, do you know how i can access it?

Secondly, in an earlier email you wrote;"That said they had issues with infant mortality being low and so went back and re-sampled the data for 500 Baghdad homes and had the same set of results"

I've just checked back with the ILCS (sorry for being so slow!) and, unless I'm mistaken-on page 50, second column, third para it states that omissions were found after rechecking the 500 households in Baghdad, and that (infant mortality) part of the survey had to be redone.
if I have that right wouldn't that provide a rather poorer view of the measure of the accuracy of their interview methods?

Thanks,

(Felix)

[Felix]

he IBC reports that they publish are pretty general in nature - focussing more on the how many than the where. That said, they do have more information available (though not to the public). I have just done a quick survey of the results for car bombs from anti-coalition forces (rough guide if you will)

US convoys/convoys 28%

Police/Army patrol 25%

Police stations 14%

Road Blocks/check points 13%

government offices/bases 12%

commercial 4%

undetermined 4%

(these breakdowns should be available on the IBC website - but I am working off a data set from IBC that they sent to me, it splits the data down into different attack types, groups etc)

so whilst these don't give locations you can see that convoy's will by definition have to travel on main/paved roads

the police stations will not be located on back streets - likewise for government offices and commercial institutions

the road blocks/check points will be on main roads - why would you road block a small back street with no traffic?

also find attached the results from the UNDP survey that looks at the percentage of houses that have access via a paved road


> Secondly, in an earlier email you wrote;"That said they had issues with infant mortality being low and so went back and re-sampled the data for 500 Baghdad homes and had the same set of results"
>
> I've just checked back with the ILCS (sorry for being so slow!) and, unless I'm mistaken-on page 50, second column, third para it states that omissions were found after rechecking the 500 households in Baghdad, and that (infant mortality) part of the survey had to be redone.
> if I have that right wouldn't that provide a rather poorer view of the measure of the accuracy of their interview methods?
>

Nice spot on the UNDP paper - you are right, I had read this part of the paper wrong....trying to do things too quickly - What it does show though is that checks were carried out and the sample was re-done with the improved interview technique.

I'm not trying to defend the UNDP survey here, I have no affiliation to it either way.

But what I will say is that in the Lancet paper the authors have shown a clear bias in their sampling methodology - this is because of location/proximity to main roads.

The burden of proof falls squarely on the authors to show explicitly that by interviewing only people living in close proximity to a main road - you will get a representative sample of the entire population.

Sean

[Felix]

Sean,

Thanks for all your time-do you mind if I share these emails and replies on a message board that I post to?

Either way, it's been very interesting corresponding with you. When are you planning to publish your independant street simulation paper?

Hope it all goes well, I've just spent most of a day trying to get a paper past manuscript central..

Yours,

(Felix)

[Felix]

Hi (Felix),

it's been an interesting day too for me

feel free to post the info on your message board

we have been looking at the temporal nature of attacks and the patterns in their intensity - this paper should be out soon. The paper on spatial correlation will be out after this - I would say 6 weeks to get it to print....hopefully

cheers

Sean

[AlyK]

Sean

I know, I know - I'm turning into a pain! Kind of serves you right for
doing interesting research though!

I've just come across something interesting and was wondering what
your opinion(s) are about it. In the new Lancet paper (L2) the authors
re-calculate the mortality rate for the same time period as the
original (L1), and get numbers that are very close (112,000
[69,000-155,000] vs 98,000 [9,000-194,000]).

From what I can see of the methodology in L1, there would have been no
main-street bias to the numbers. Except in Falluja (as you pointed out
to me) which was discounted. Given we can more or less safely say
there was no main-street bias in L1, and the numbers from L1 agree
with L2, does that not tell us that main-street bias was unlikely to
have played much of a role in L2?

Thinking about this, and the points that I made in the last email to
you, it does seem pretty likely that the local populations around
centers of activity, like markets etc., are more heterogeneous than
first expected, thus making the impact of main street bias in these
kind of attacks small. That is not to say it doesnt exist at all,
simply that it seems to me that it is easily over-inflated.

Related to this, I have come across a paradox that seems pretty
interesting. The most likely scenario in which main-street bias would
be important is if there is very little movement of the population,
either within or outside of the local neighbourhood. This would mean
that the L2 numbers (assuming they only interviewed people on main
roads) would be far far too high, and other smaller estimates (US
Govt., or IBC) are likely more correct. That means that the level of
violence is much less than we think. Surely this means that the
likelihood of being killed is smaller, and therefore people are more
likely to move around, thereby reducing the impact of main street
bias?

i.e.
GIVEN main-street bias is very high, AND L2 suffers from main-street bias
==> L2 vastly over-estimates mortality,
==> Iraq is a lot less violent than we thought (30,-50,000 dead)
==> people move around more,
==> main-street bias is close to zero.

Something there does not add up...

Anyway, I hope that these points are reasonably clear, and I would be
interested in your thoughts on them.

Cheers

Aly

[AlyK]

[From Sean]

Remember Aly the main street bias says nothing about the non-violent
death rates, it is only concerned with the death rates from violence.
Thus we need to compare L1 and L2 from this point of view. It is not
right to say that the two surveys agree on total deaths, we need to
know if they agree on violent deaths.

Here's the latest version of the violent death rate comparison
between L1 and L2

from L1 - (excluding Falluja due to main street bias)
_____________

total violent deaths/1000

pre – 1/7438 or 0.134/1000 per 14.6 months
post – 21/7868 or 2.67/1000 per 17.8 months

normalising the pre-invasion rate to 17.8 months we have

pre = 0.164

or a difference of 2.50/1000 per 17.8 months (or 1.68/1000/year)

thus with a population of 24.4 million as estimated in the L1 paper
we get 61,000

from L2
_____________

violent death rate per 1000 per year for the first year = 3.2 (or is
this for 13 months? Doesn’t make a large difference as it is a rate/
year that we are measuring)

violent death rate per 1000 per year for the second year = 6.6

this is an average rate for the whole year, so at month six -
assuming a linear rate of increase (reasonable given the data) - we
have a death rate of 6.6/1000/year.

At the start of the second period you can measure the death rate
(from the graph in the Lancet paper) to be 4.95/1000/year.

This corresponds to an increase over the 5.8 month period of ~1.6

Thus the average death rate for the 5.8 month period is 4.95+0.8=5.75
per 1000/year

Correct for the 5.8 month period, gives us 2.77/1000/5.8 months

total violent deaths = year 1 deaths + second 5.8 months deaths
= 78,080 + 67,588
= 145,668

where we assume the population to be 24.4 million as we did for L1.

this difference is higher than the excess deaths figure – but that is
because the non-violent count drops over the first two years of the
war when compared with the pre-war calculation.

So for the same survey, but one measured using gps and the other
using the main street + cross street algorithm we have a 61/145.6=2.4
fold increase in measured violent deaths.

____________

This is a pretty significant result I think, strongly points towards
the main street bias being a significant effect.

Sean

[joe emersberger]

I think the calculation above is flawed for two reasons

1) They overestimated the violent death rate in L2 over the 17.8 months that it overlapped with L1. There were 63 deaths by violence during the occupation in L2 up to the end of August 2004, 70 by the end of September, 2004.

That means 63 over 17.3 months, or 70 over 18.3 months.

I got this data from the Lancet team. Gourley estimated death rates by looking at the figure 2 in the 2006 study . The problem is he was looking at a graph with 3 points on it showing very general trends. It's very hard to be accurarte getting data from a graph like that and certainly no substitute for looking at data.

The violent death rate for 17.3 moncths of occupation is (63/12801)*(12/17.3)
= 3.4 deathst per 1000 per year

The violent death rate for 18.3 moncths off occupation is (70/12801)*(12/18.3)
= 3.6 deathst per 1000 per year

Therefore over the 17.8 months we get 3.5 violent deaths per 1000 per year

Calculating "excess deaths" from violence we use 3.5 -.0.1 to account for the pre-invasion period

We get 3.4*24.4 million*(17.8/12)

which is 123,000 deaths (not 145,668)

With this we get violent deaths in L2 / L1 = 2.0 (not 2.4)

The survey populations changed from 11956 in January 2002 to 12801 by the time the survey was done. If you assumed a population of 12400 should have been used above then the ratio becomes 2.06

2) The comparison that is relavant to what the MSB guys are arguing is violent death rates determined by GPS (in L1) and those determined by the "street-off-main-street" method (L2) over the 17.8 months that they overlap.

But L1 did not use GPS for Falluja (and therefore not for any of Anbar province).
They should excluded Anbar from both L1 and L2 to get a true comparison between the GPS and "Street-off-main-street" methodologies.

As I understand things L1's conservative 98,000 estimate took Falluja out completely. It was therefore an estimate for 97% of Iraq. The rest of Anbar was assumed to be have average death rate (same as in all of Iraq). To take Anbar out of L1 completely its death toll should be reduced by about 2% (in addition to the 3% that represented Falluja). If the rest of Anbar was assumed to have an average death rate, and if it accounted for 2% of the Iraqi population (all of Anbar is 5%) then the share of violent deathst it would contribute would be equal to its share of the population

Hence, instead of 61,000 violent deaths over 17.8 months we should estimate 59,780 - not a big correction.

However, for L2 Anbar contributed about 25% of violent deahts over the first 17.8 months. Again I got this from the Lancet Team.

Taking Anbar out of L2 reduces the violent death toll from 123000 to 92,250

The ratio of violent deaths (that truly compares the different methodologies in L1 and L2)

is 92,250 / 59,780

That is a ratio of 1.54

When you consider the wide CI's involved in both studies that is hardly compelling evidence that the L2 methodology was unsound.

[AlyK]

Thanks Joe. Interesting numbers... I'm also gonna look through them. In the meantime, these are some values from the formula in the MSB paper, evaluated for dfiferent parameter values.

If we assume the lack of movement that the paper assumes, then for various n,q we get the following table: (top row is %age of population unreachable, first column is how much more violent SoM streets are than the unreachable ones):

fi=fo=13/14

[AlyK]

===
Correspondence
Nature 446, 611 (5 April 2007) | doi:10.1038/446611b; Published online 4 April 2007


Authors defend study that shows high Iraqi death toll
Les Roberts* & Gilbert Burnham**

* Program on Forced Migration and Health, Mailman School of Public Health, Columbia University, New York, New York 10032, USA
** Center for Refugee and Disaster Response, Bloomberg School of Public Health, Johns Hopkins University, Baltimore, Maryland 21205, USA

Sir:
In our opinion, your News story about our Lancet paper "Death toll in Iraq: survey team takes on new critics" (Nature 446, 6–7; 2007) has confused the matter rather than clarified it. You outline three criticisms of our work: that there was not enough time to have conducted the survey; that the sampling method suffered from a 'main-street bias'; and that the study team fabricated the data (the last being attributed to anonymous "researchers"). These criticisms have been previously addressed, and have little merit.

On the first point, the 1,849 interviews in 49 days described in our study suggest that 38 interviews had to be conducted each day by our eight interviewers. Although introducing themselves and explaining the confidentiality agreement might have taken interviewers several minutes, the five-question interview would take only a couple of minutes for most households that reported no deaths. The idea that eight interviewers could not conduct a total of 38 interviews in a day is not credible.

Second, we dismiss the suggestion that our sampling over-represented main streets, where car bombs are more likely. As stated in our paper (G. Burnham, R. Lafta, S. Doocy and L. Roberts Lancet 368, 1421–1428; 2006), when excluding the statistically outlying cluster of Falluja from the first report, we estimated 98,000 (95% c.i.: 8,000–194,000) excess deaths versus 112,000 (95% c.i.: 69,000–155,000) over the same period with the second survey. The first survey was done selecting random starting points with a Global Positioning System unit. The second used the random street-selection process, which is being criticized as biased. It rarely occurs in the field that two sampling methods are used allowing for comparison, and here the results are nearly identical. Moreover, there is no plausible mechanism for a significant main-street bias to operate, because only 15% of all deaths are from car bombs and other ordnance, and because most violent deaths are believed to occur away from the home.

Third, as for the accusation that researchers fabricated the data, we are ready, willing and eager to have an established international authority take a sample of the cluster forms and go to the field with our interviewers to verify the findings. Until that time, the Coalition and Iraqi governments' statements that during the first three years of occupation, Iraq's violent-death rate was lower than those of Russia, Estonia, Latvia, South Africa and Kazakhstan remain an implausible contrast with our findings.

When Nature called one of our study members in Iraq and asked if local officials joined them during the survey, that individual later clarified to Nature by e-mail that 'local officials' did not mean local clinicians and colleagues. This was inaccurately reported in the Nature summary along with a statement by our co-author that interviewers often worked alone. These points were wrongly cited as contradictions between the study team members in your News story.

All reports will eventually have "criticisms that dogged the study", if previously addressed criticisms with so little merit are given a voice in the press.

Nature stands by its version of the events described in the penultimate paragraph of the above Correspondence — Editor, Nature.