Discussion with authors of Lancet critique

[Danny]

Aly,

I have nothing to add to your excellent thread, except to call it a great resource and to congratulate you for your tenacity. It's maybe hard to see sometimes but you are saving lives, keep it up. I know too much self-congratulation doesn't read well, but people like Aly deserve a pat on the back every now and then.

all good things,

a passer-by

[dav toirtap]

conflictanalysis@rhul.ac.uk


Dear Neil F. Johnson/Michael Spagat/Sean Gourley,

I read with interest your paper dealing with potential bias in the methodology used by the John's Hopkins team's study of conflict mortality in Iraq (as published in the Lancet). I had read about your initial theory, but have to admit that the paper that followed posed, for me, more questions than it answered.

I hope you don't mind if I take a few moments of your time in order to outline some of my observations, and perhaps you might be able answer a few questions?

With direct reference to your paper 'Bias in epidemiological studies of conflict mortality' as published here: http://www.rhul.ac.uk/economics/Research/conflict-analysis/iraq-mortality/index.html
1) I would like to make an observation with regard to the section title 'Visual Summary of "Main-Street Bias"'.

While the essence of the diagram is obvious, it is slightly misleading. The size of the area that 'will never be surveyed' is not quantifiable. A cursory reading of the paper might lead the reader to believe that almost all areas off 'residential streets' 'will never be surveyed'. This is incorrect, as I presume you have attempted to show by drawing small side streets off the 'residential street'. This is though, in my opinion, insufficient. The diagram should come with a qualifying statement, such as:

"Where there exists "residential streets" separated by more than 80 dwellings, there will be - based on the best interpretation of published methodology - dwellings not included in the survey."

Following this, a number of other questions arise, the first of which is: "Ask yourself which streets you think are the main streets?" A question you yourself asked.

However, no where in the paper were you confidentially able to define what it is, and therefore identifying this type of street becomes, ultimately, 'guess work'. Is this a crucial hinge upon which this theory rests?

2) With regards to the section titled 'Iraq Maps':

Based on the initial diagram, the first image you have shaded in order to identify areas that 'will never be surveyed' appears to be unrepresentative of the paper's main thesis. For instance, the first map, appears to show blue areas (at a minimum 7 separate areas) - those where dwellings 'will not be surveyed' - between "residential streets" that do not seem (perhaps a more defined image might show otherwise) to be separated by more that 80 dwellings. Of course from the quality of the images shown, it is almost impossible to be certain of this, but this also makes it impossible for you to confidentially state otherwise.

With the second image you have identified areas shaded in pink as areas that 'will never be surveyed'. However this shading appears to show that almost the entire city will not be surveyed and you go on to say:

"The picture is not quite complete. There should be a bit more pink shading"

How did you come to this conclusion - it's certainly not clear from these images/maps? And it does not conform to the diagram which illustrates the theory your paper intends to establish.

3) Some further, more general but no less pertinent questions arise from my reading of the paper, with reference to the section titled 'Urban Homicide Rates Around the World':

You write: "Below we provide links to literature showing how dramatically urban violence varies from neighborhood to neighborhood in a wide variety of cities. Iraqi cities are likely to follow this pattern. Therefore, any sampling scheme for measuring Iraq violence that cannot reach many neighborhoods is vulnerable to distortion."

Is urban violence in peace time cities a workable comparison? Would the present situation in Iraq, and the situation since 2003, not be more accurately comparable to patterns of violence in other war time cities? There were I presume no guerrilla or occupation wars being fought in any of the cities referenced at the time of the mapping of homicide rates. I will discuss this further in my next observation.
4) In the section titled 'Danger to Interview Teams versus Danger to Neighborhood Residents':

You make reference, in your final comments, to 'the Stathis Kalyvas theory', which you interpret again to suggest the survey results may be skewed as a result of the survey method.

Having no knowledge of the book, I can only go on the parts used in your paper: "The Kalyvas theory predicts that violence will be lowest in areas of strong control by one side or the other."

Here again questions are raised as to where violence as a result of war, insurgency, criminal activity, sectarian conflict and terrorism happens.

A logical assumption, one of literally hundreds, may be that in order to control certain areas, it is first necessary to control the main thoroughfares in a particular block, therefore these areas would according to the theory be the least violent - obviously there is a corollary to this, in that in order to gain control of these areas, one must win these areas, presumably by violence.

It is also probably fair to assume that in coalition or insurgent strongholds the insurgents/occupation forces patrol the main streets of their particular stronghold zones as a show of force - you would need to study the relevant military and insurgent tactics with this in mind. Do insurgents target convoys etc on open streets where they are more likely to be caught, but perhaps more likely to inflict damage, or do they attack occupation forces in less open areas? Do occupation forces predominantly fight insurgents or counter-insurgents, depending on your political persuasion, on main streets (wait for them to attack presumably) or do they conduct house to house raids (as in Ireland circa 1900)?

To expand further on the tactics of insurgents and occupation forces we should take into account the fact comparatively few occupation forces have been killed, although according to the United States Government Accountability Office [commonly called the investigative arm of Congress or the congressional watchdog http://www.gao.gov/about/what.html], while attacks on coalition forces are steadily high, actual fatalities are not comparable. Therefore I think focusing on the placement of mines etc is a dead end.

Guerilla attacks in Iraq since 2003 through April 2007 via Juan Cole:

http://www.juancole.com/2007/05/char...q-here-is.html

and the original document:

http://www.gao.gov/new.items/d07677.pdf

Another assumption - In order to control particular zones the occupying force must first control all entry and exit points – main streets no doubt, then they must control strategic locations – government buildings, high ground, bunkers etc, then they must control all parts of the zone – therefore each and every building and residence.

For the time being I can offer only anecdotal evidence from WW2 Stalingrad: "The Germans, calling this unseen urban warfare Rattenkrieg ("rat war"), bitterly joked about capturing the kitchen but still fighting for the living-room."

http://en.wikipedia.org/wiki/Battle_of_Stalingrad

5) To summarise…

In my opinion, it is of grave detriment to the premise of this paper for you to have focused on urban violence from peace time cities, as opposed to guerilla and military tactics from war time cities - occupation France perhaps, WW2 Stalingrad, Ireland during British occupation, Mogadishu in 1993, or even '70s Northern Ireland (for the purposes of studying sectarian conflict).

Just one instance from Mogagishu, resulting in the deaths of 50 people:

"One of the most unfortunate results of this occurred on July 12, 1993, when a United States-led operation was launched on what was believed to be a safe house in Mogadishu where members of Aidid's Habar Gedir clan were supposedly meeting to plan more violence against U.S. and U.N. forces. In reality, elders of the clan, not gunmen, were meeting in the house. According to U.N. officials, the agenda (which was advertised in the local newspaper) was to discuss ways to peacefully resolve the conflict between Aidid and the multinational task force in Somalia, and perhaps even to remove Aidid as leader of the clan.

During the 17 minute combat operation in which U.S. Cobra attack helicopters fired 16 TOW missiles and thousands of 20-millimeter cannon rounds into the compound, killing more than 50 of the clan elders. Some in Mogadishu believe that this was a turning point in unifying Somalis against the U.S. and U.N. efforts in Somalia. It would also lead to the deaths of four journalists."

http://en.wikipedia.org/wiki/Battle_of_Mogadishu_%281993%29

I have not even attempted to discuss your theory as it relates to the initial invasion – both in terms of air strikes and ground 'operations'.

And I have not gone into any detail with regard to the assumption that people +residing+ in dwellings on side streets between residential streets separated from other residential streets by more than 80 dwellings are less likely to experience violence, whether in their home or outside of it, as I understand you are inundated with questions on this issue; though I still feel there are many questions unanswered here too.

Thank you for your time and I hope you get a chance to respond to my observations and questions.

Yours sincerely,

David Manning

[AlyK]

Hi David,

interesting questions and I hope you get some kind of response. I would be interested to see what they say.

And Danny, many thanks for your comment.

Regards

Aly

[dav toirtap]

Received today:



Dear Mr. Manning

Perhaps it will interest you that our working paper has been updated and the new version should be posted fairly soon on the website of the Households in Conflict Network where the early version was posted.

We would also like to emphasize to you, that our paper does not try to prove one particular value of the bias factor, but instead shows that there is good reason to believe that such a bias factor exists. The responsibility lies with the researchers of the Lancet paper to account for this bias -- specifically, to quantify its effect -- for their particular dataset and methodology.

Respectfully

Neil Johnson, Michael Spagat, Sean Gourley, Jukka-Pekka Onnela, Gesine Reinhert

[dav toirtap]

Dear Neil Johnson, Michael Spagat, Sean Gourley, Jukka-Pekka Onnela, Gesine Reinhert,

Thank you for taking the time to reply.

While I understand you must be very busy and inundated with questions given the highly politicised nature of studies and criticisms of studies of conflict mortality, it seems to me you have either ducked or disregarded the questions and observations I have made.

I would put to you that your paper has not shown " there is good reason to believe that such a bias factor exists", and that parts of the paper appear unintentionally misleading.

Yours sincerely,

David Manning

[dav toirtap]

Main Street Bias?

"the US military on Tuesday launched one of its biggest operations since the invasion of Iraq in 2003.

"It is house to house, block to block, street to street, sewer to sewer,""


US in 'sewer to sewer' fighting


IRAQ: Thousands of US soldiers on the offensive north of Baghdad are facing fierce resistance from hundreds of al-Qaeda militants who are ready to fight to the death, an American general said yesterday.

The militants are making their stand in and around the Iraqi city of Baquba, 65km north of Baghdad, where the US military on Tuesday launched one of its biggest operations since the invasion of Iraq in 2003.

http://members.boardhost.com/mediabite/msg/1182609875.html

[dav toirtap]

Jul 1, 2007 3:41 PM

Dear Neil Johnson, Michael Spagat, Sean Gourley, Jukka-Pekka Onnela, Gesine Reinhert,

Given the importance of both the John's Hopkins study and your criticisms of it, I would very much appreciate a response to the questions I posed.

Yours sincerely,

David Manning

Jul 4, 2007 5:39 PM

Dear Mr. Manning

The responsibility lies with the researchers of the Lancet paper to show that their study does not suffer from the bias discussed in our paper.

Respectfully

Neil Johnson, Michael Spagat, Sean Gourley, Jukka-Pekka Onnela,

Gesine Reinert

Jul 4, 2007 6:18 PM

Dear Neil Johnson, Michael Spagat, Sean Gourley, Jukka-Pekka Onnela, Gesine Reinert

Thank you again for considering my request.

And please again excuse my persistence, but the issue is of such importance as to excuse the wasting of a few more words and moments.

You are right in saying 'the responsibility lies with the researchers of the Lancet paper to show that their study does not suffer from bias', yet I have put it to you, and laid out a number of quite logical reasons why, your paper does not evidence an effective bias. That is not to say a bias does not exist, it may well do, and your paper does go some small way towards finding one.

I could also put it to the Lancet paper authors that their paper suffers from a 'Gravitational Bias', in that their researchers were constrained by gravity and may therefore have favoured ground floor dwellings as opposed to third floor dwellings – but I would have to provide sufficient evidence to show that Iraqi dwellings follow this criteria such that the study method could not have reached them, and more importantly - that people who live in high rising building are more or less likely to experience violence.

This contention offers reason enough for those that wish to bury the figures.

Yours sincerely,

David Manning