ISAF says “Don’t quote me, bro!”

In case you missed it, a couple of days ago the Afghan Analysts Network dropped a report on NATO night raids in Afghanistan that was picked up by a number of media sources, including The Guardian, who saw fit to turn it into pretty and informative graphics.

ISAF aren’t happy about that.

Among a number of sentences that read like the soundtrack to rattles being thrown out of prams, this one really took the biscuit:

“ISAF was never consulted in the preparation of this study, nor asked for data regarding operations.”

For the record, the study was looking at ISAF press releases because:

“ISAF actually releases a large amount of information about its activities in the form of press releases. These press releases range from two to fifteen per day. Although this data is not the complete picture (not all operations are written up as press releases), it offers insight into how ISAF sees its contribution to the war and presents a far more differentiated picture of the capture‐or‐kill raids than the released aggregate data on its own.”

The point, therefore, was not to present a statistical analysis of the efficacy of NATO raids, which would probably be impossible due to operational security, but an analysis of what ISAF is saying, why they appear to be saying it and what that means about the conflict. They go on to make a few points about the apparent interchangeability of terms like “insurgent leader” and “insurgent facilitator”.

What is really interesting is that in the midst of their knee-jerk defence, ISAF says this:

“The published ISAF press releases used in preparing the AAN report were never intended to be an authoritative database of all ISAF operations conducted in Afghanistan, nor even a representative sample from which to draw scholarly conclusions.  Any analysis of complex combat operations based on press releases alone, which by definition are written to provide basic, factual information, inevitably will produce an overly simplistic, flawed and inaccurate product.”

Which begs the question: What, exactly, is the purpose of ISAF reports in the first place? Is ISAF seriously going on record to say that the data that they are making public cannot be relied upon? If analysts cannot build arguments out of the “basic, factual information” that ISAF is providing, why provide it at all? Personally, I thought the research methodology of “believing what ISAF says” to be rather erring on the side of NATO. Apparently not, according to ISAF. We should instead disbelieve in their own chain of public reporting as a method of gaining any understanding of the conflict whatsoever, since we couldn’t possibly understand the conflict from what they say. At least until the next positive press release, I imagine.

Standard

15 thoughts on “ISAF says “Don’t quote me, bro!”

  1. Hi Jack,

    Thanks for the writeup. You hit the nail on the head here. There’ll be a response coming up on the AAN blog at some point soon, I think.

    Also, you know I’m a PhD student at King’s College in the War Studies Dept, right?

    Thanks,

    Alex

  2. Pingback: @ISAFMedia Hates My Face, or We’re ISAF, and We’re Here to Kill You | Hamsters on the Titanic

  3. I think what ISAF is saying is not that individual press releases or reports can’t be relied upon, but that all of the press releases, taken as an aggregate, were “never intended to be an authoritative database of all ISAF operations conducted in Afghanistan, nor even a representative sample from which to draw scholarly conclusions”.

    • That’s rather my point. Is ISAF saying that their own reporting of the Afghan conflict is so biased as to be useless in analysing the conflict itself? Since, well, this same reporting is the best-case scenario for ISAF for people to be basing their opinion of the conflict upon (since it is the least critical of ISAF policy). I think that this is important for two reasons, firstly, that NATO’s PR machinery would rather call its own accuracy into question rather than take a vaguely critical report lying down (it is not as if the report itself is excessively critical of ISAF/NATO), and it also sounds to me like the age old “You don’t know everything, you can’t possibly comment” line which is used to silence any and all criticism of quasi-secret activities. Furthermore, the point being that it seems rather excessive to be so defensive over public statements that ISAF asserts as fact. I could imagine them being angry if their reporting was called into question, but as stated above, the report includes believing ISAF as part of its research methodology.

      Given the sample sizes in the report, I’d think that ISAF would be hard pushed to say that the sample is unrepresentative of the raids themselves, particularly in the latter phases when the volume of daily raids increases. One could say “Oh, more people died/were captured than reported”, but unless ISAF has a nasty sideline in unreported execution squads that are killing a statistically significant number of people a day instead of capturing them, then it won’t affect the distribution of kills/captures. Not saying that this is an impossible example, but it would seem a strange possibility to rely upon in order to score points against a report that… isn’t too critical in the first place. Similarly, the converse would be that there are ISAF squads that are capturing statistically significant numbers of Afghans and disappearing them. Again, I’m pretty sure there are people disappearing “off camera” but the volume reported means that there would need to be thousands of Afghans under lock and key somewhere in order to alter the balance of the reported kill/capture ratio.

    • Gunrunner says:

      “Is ISAF saying that their own reporting of the Afghan conflict is so biased as to be useless in analysing the conflict itself? ”

      No.

      Press releases are open source data of a non-sensitive nature that do not compromise operations, on-going or otherwise.

      Basically, they are not a comprehensive listing of what is going on, and to take these releases and then try to produce some sort of “scholarly” document is nonsense as they are acting on incomplete information, vastly incomplete.

      ISAF Bias? Hardly. ISAF honest and fair assessment of the AAN report? Most assuredly so.

    • I think you’re misconstruing what I’m saying about bias.

      In statistical terms, as a thought experiment. Let’s say you are given a data set, A, which is presented to you as a roughly accurate data set, though it is incomplete (being a sample of data set A*). There are lots of variables in this data set, but for some reason, you’re only interested in two: K and C. The data set is collected from a number of different geological locations, let’s say 30 in total.

      Now, when analysing that data set, you see a couple of things. Firstly the amount of K’s and C’s (the volume of readings associated with this variable) increases over time. Secondly, there are some substantial differences between that ratios of K’s to C’s depending on the sample location. Thirdly, in roughly half the sample location, the volume of K’s and C’s is a small fraction of the volume in the other half.

      Now, given the incompleteness of the data set, you can’t say for certain what the total number of K’s and C’s are. But you can observe the ratio, particularly given a statistically significant volume of them (Seriously, if you disagree with this, then you’re arguing against statistics as a profession). You can also account for outliers, like when there are three times as many K’s as C’s in a given sampling location, but that location has a small sample volume, so maybe there was a bad reading, or there’s K’s and C’s that you don’t know about that will bring it back in line with the other results.

      After all this, you publish your findings, but then your mean PhD supervisor has a go at you. “First,” he says, “why didn’t you ask me for more complete data?” And you say “Well, maybe we were actually interested in the A data set, and maybe we assumed since you told us that A reflected A* that we didn’t need A*.” This does not, however, calm your supervisor down, “Well,” he says, “Why didn’t you account for variables A to ZZZZZ?” and, slightly stunned, you say “Well, there’s only two of us, and that’s not what we were interested in, nor was it what we were analysing. Besides which, when someone does a paper on micro-economics, they tend to exclude 99.99999% of possible variables in some manner or other.” Hearing this, your supervisor gets angrier still and says “You’re wrong, your data set is totally biased and you can’t draw any conclusions from in, not that you ever should.” and storms off.

      Sitting down with your erstwhile research partner, you start thinking it through: If data set A is so unrepresentative of A*, why did your supervisor say it was what was going on? Secondly, if your findings re: the K/C ratio are so out of whack, then there have to either be a whole lot more K’s floating around out there, or a whole lot more C’s, either of which are not represented in data set A.

    • olaf says:

      Agree

      What is the problem? First (!) thing a student of history (in Germany) learns is “kritische Quellenalanyse” (source analysis), wich means to enquire: Who wrote a piece at question, who where the addresseees, and what could have been the intent of the author? To answer these questions one has to dig a bit into the subject matter.

      Press releases earn their name from their target group. Why should press releases reflect anything but what the publisher wants the press to be published?

      It is the work of the analyst to read these press releases critically. Thats why he/she is called analyst.

      Cheers

    • Gunrunner says:

      “Press releases earn their name from their target group”

      Not exactly.

      They are written for the general public, with the aim of informing on the basics of what is going on, therefore, not designed for the “press,” and because the reports are basic, lacking operational security information and other considered limitations, using them as a basis for analysis is flawed from the beginning.

  4. Pingback: We’re ISAF, and We’re Here to Kill You | It's Always Sunny in Kabul

  5. Pingback: Sunny in Kabul | We’re ISAF, and We’re Here to Kill You

  6. Pingback: People's Republic of Snarkistan | We’re ISAF, and We’re Here to Kill You

  7. Pingback: We’re ISAF, and We’re Here to Kill You | Sunny in Kabul

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>