Statistical Gymnastics Come Under Close Scrutiny, Outed as Bullshit

Recent assertions by the U.S. military that violence in Iraq has markedly decreased in recent months has come under severe scrutiny from experts both inside and outside the government. Consensus is that that statistics offered by the military are questionable at best, as the pattern emerged that the military simply ignores negative trends. (Before anyone screams "Outliers are tossed all the time!" let me save you the humiliation of a lecture - I actually know what an outlier is. For the record, an outlier is a data point that is more than three standard deviations from the mean. You can't just call any data point you don't like an outlier and omit it to get the result you want and expect your findings to be taken seriously.)

The administration has pinned everything on the reduction of violence metric to prove that finally they have a strategy that is working! That is what Gen. David Petraeus is going to say in front of congress next week when he comes to give his aWol's report on the state of security in Iraq. He is expected to try to convince the congress that there has been a 75% decrease in sectarian attacks. (Problem there: I do not think that word actually means what they want us to believe it means.)

When others look at the full range of data, they find that the military cherry-picks the data for positive indicators and ignores the negative. "Let's just say that there are several different sources within the administration on violence, and those sources do not agree," Comptroller General David Walker told Congress on Tuesday in releasing a new Government Accountability Office report on Iraq.

Meanwhile, military officials would have us believe that the CIA, DIA and GAO are all using flawed methodologies that give an inaccurate picture.

The intelligence community has its own problems with military calculations. Intelligence analysts computing aggregate levels of violence against civilians for the NIE puzzled over how the military designated attacks as combat, sectarian or criminal, according to one senior intelligence official in Washington. "If a bullet went through the back of the head, it's sectarian," the official said. "If it went through the front, it's criminal."

"Depending on which numbers you pick," he said, "you get a different outcome." Analysts found "trend lines . . . going in different directions" compared with previous years, when numbers in different categories varied widely but trended in the same direction. "It began to look like spaghetti."

One of the troubling trends is the ommission of violence attributed to Shia militias fighting other Shia militias, which has gripped Basra and last month resulted in the assassination of two southern provincial governors.

According to a spokesman for the Baghdad headquarters of the Multi-National Force-Iraq (MNF-I), those attacks are not included in the military's statistics. "Given a lack of capability to accurately track Shiite-on-Shiite and Sunni-on-Sunni violence, except in certain instances," the spokesman said, "we do not track this data to any significant degree."

Attacks by U.S.-allied Sunni tribesmen -- recruited to battle Iraqis allied with al-Qaeda -- are also excluded from the U.S. military's calculation of violence levels.

Here it might be beneficial to remember that the Iraq Study Group report that was released in December identified "significant underreporting of violence," noting that "a murder of an Iraqi is not necessarily counted as an attack. If we cannot determine the sources of a sectarian attack, that assault does not make it into the data base." The report concluded that "good policy is difficult to make when information is systematically collected in a way that minimizes its discrepancy with policy goals."

They are bringing their wares to the Capitol next week and the hucksters, pitchmen and carnival barkers are hocking the cherry-picked data in a most full-throated manner.

I just have two words for my countrymen, especially the 535 of 'em that control the pursestrings: Caveat Emptor.