In a recent article, social scientist Patrick Ball revisited his and Kristian Lum’s 2015 study, which made a compelling argument for the underreporting of lethal police shootings by the Bureau of Justice Statistics (BJS). Lum and Ball’s study may be old, but it bears revisiting amid debates over the American police system — debates that have featured plenty of data on the excessive use of police force. It is a useful reminder that many of the facts and figures we rely on require further verification.

The 2015 study considered the number of police-committed homicides on record, how different records, or “lists,” overlap, and what this overlap said about the total number of homicides. Lum and Ball noted then that “the more overlaps among the lists, the more plausible it is that the population they are drawing from is small.” The BJS, a major source of law enforcement data, once used the Arrest-Related Deaths program (ARD), a database that drew from media sources to compile “an annual national census of persons who died either during the process of arrest or while in the custody of state or local law enforcement personnel.”

It turns out that that database drastically undercounted police shootings. In 2015, the BJS conducted a report on the completeness of the ARD. They checked it against the FBI’s Supplementary Homicide Report (SHR), which recorded “homicides committed by police that in the judgment of the police department or the local FBI have been . . . considered legal.” There was substantial overlap between the ARD and SHR: The ARD reported 1,939 homicides, the SHR reported 1,704, and the overlap between the two amounted to 1,681. The BJS therefore estimated a total of 7,427 police homicides between 2003 and 2009 as well as in 2011. This statistical effort was a triumph, but it also illuminated the feeble record-keeping system in use to track police shootings.

You Might Like

In their report, Lum and Ball called this “a striking admission of the weakness of the federal bureaucracy with respect to recalcitrant local law enforcement officials who refuse to publicly share the most basic facts about potential abuses.” In their estimation, the BJS’s updated estimate fell short, failing to account for putative flaws with police homicide reporting. Lum and Ball asserted that the overlap between the ARD and SHR is probably larger than it should be, meaning that the eventual figure was also likely an underestimation.

Why would the overlap be too large? Lum and Ball assumed two things: one, that victims of police force fear government retaliation, and two, that police who commit crimes strive to hide their actions. The result of these incentives, they argued, is that the reported incidents are often the most public:

Consequently, lists of homicides tend to be partial, and they tend to emphasize victims with high social visibility: victims who are relatively well-known, and whose killing occurs in daylight, in urban areas, and in view of bystanders motivated to report the crime. Other killings without these aspects more frequently remain hidden from public knowledge.

Ball argues that in the absence of witnesses and media, a police shooting is more likely to go unreported or filled as an accident. Accordingly, Lum and Ball turned to list correlations that they had found in other countries, including Colombia and Kosovo, to estimate the total number of U.S. police homicides as close to 10,000 over the tested period (about 1,250 a year).

Does that estimate hold up? Thankfully, the ARD no longer exists given its abundant flaws, but other sources have stepped in to provide a more comprehensive look at police shootings. In 2015, the Washington Post started its own police-shooting database, which has found about 1,000 police shootings per year. According to the Post, the data is compiled “by culling local news reports, law enforcement websites and social media, and by monitoring independent databases such as Killed by Police and Fatal Encounters.” Interestingly, these data conflict with Lum and Ball’s findings; the Post is simply counting each incident as it occurs, rather than making statistical inferences. The Post’s reliance on media sources for data also conflicts with Ball and Lum’s assertion that only certain police homicides make it to the news.

This is a promising step forward. The Post’s efforts represent the most comprehensive attempt to track all police shootings. But it would still be wise to heed Lum and Ball’s warnings. Data collection, like all information, can be affected by incentives.