How late and partial information impedes the prevention of gun crimes
Imagine a world in which the Super Bowl winner was not announced until nine months after the season ended. Imagine if we found out that the Kansas City Chiefs beat the Philadelphia Eagles around Thanksgiving, well into the following season. Now imagine trying to reconstruct the NFL standings for the 2022 season, but only 20 of 32 teams self-reported their records and statistics. One could go to an individual team’s website and download the box scores each week, but not every team publicized its statistics, and every team that did so, did so slightly differently. To say the least, it would be tough to compile league-wide stats.
This is an absurd hypothetical for the relatively low-stakes world of sports, but it effectively describes how life-and-death crime data are collected and reported nationally in the United States.
Major crime categories were introduced in 1930 as part of the FBI’s Uniform Crime Report. There are seven categories of major crime that law enforcement agencies throughout the country collect data on and report up to the FBI: violent crimes, murder, rape, robbery and aggravated assault, property crimes theft, burglary and auto theft. The FBI collates all this information, audits it and produces an annual report that estimates the number of such crimes that occurred each year.
This entire process takes a long time, so the first formal reporting on national crime trends typically comes out in September or October of the following year.
Adding to the problems with national crime data, the FBI switched reporting systems in 2021 to the National Incident-Based Reporting System (NIBRS). NIBRS is a more advanced reporting system that was first introduced in the 1980s. Agencies can report dozens of types of crimes rather than the more limited set of options under the old system.
The first formal reporting on national crime trends typically comes out in September or October of the following year.
The problem is that far fewer agencies report data via NIBRS than they did with the old system, despite NIBRS being more than 30 years old. Less than a third of the U.S. population was covered by an agency that participated in NIBRS when the FBI announced in 2015 that agencies could only start reporting via NIBRS starting in 2021, and fewer than two-thirds of the population was covered through NIBRS in 2021. Many of the nation’s largest police departments, such as New York, Los Angeles and Chicago, failed to submit a full year of data to the FBI in 2021 via NIBRS, and New York and Los Angeles are still not yet NIBRS compliant.
The result was that the FBI’s estimates of national crime figures for 2021 came with some large margins of error.
The FBI estimated that there were 22,900 murders nationally in 2021 with a margin of error putting the actual total between 21,300 and 24,600 murders. Violent crime was estimated to have fallen 1% from 2020 to 2021, with the margin of error putting the actual trendline between a 12% decline and a 12% increase. Property crime was estimated to have fallen 4% from 2020 to 2021, with the margin of error putting the actual change anywhere between a 38% decrease and a 50% increase.
One problem with relying solely on FBI data is that we want to know the nation’s murder trend in as close to real time as possible. Those who follow crime trends often marvel at the speed with which information is collected and reported in the world of economics. The Department of Labor produces a jobs report for a month on the first Friday after that month ends, giving interested parties a first glance at a critical economic indicator and revising reports from previous months to reflect further data collection. Improving the speed with which crime and policing data reach the hands of researchers and policymakers is the only way to ensure that this information can be useful, but there are many challenges to collecting and reporting information from 18,000 law enforcement agencies in anything approaching real time.
The FBI’s estimates of national crime figures for 2021 came with some large margins of error.
The FBI did try to work around the slowness issue by publishing quarterly reporting with a three-month lag starting in mid-2020. These data were important for seeing the national spike in murder as it occurred, but the switch to NIBRS has meant that there have not been enough reporting agencies to make quarterly national estimates for every quarter since the end of 2020. The FBI has been publishing quarterly data from several hundred agencies serving more than 100,000 people, but these data are not audited by the FBI and is often incomplete or inaccurate.
The way around the FBI data weaknesses is to collect data from the dozens of agencies that publish data online. These are mostly big agencies, so this sample will likely overstate the national trend, but collecting data from 50 to 100 large agencies will accurately mimic the national trend. The year-on-year change in murder from a sample of 100 big cities will usually be within two to three percentage points of the trend published by the FBI many months later.
There are certainly challenges with this methodology. There is no standardized procedure for what data is reported, how frequently and in what format. Some agencies publish daily murder counts on their websites, others publish weekly, monthly or quarterly reports of all crimes. Some agencies report data with virtually no delay while others report data weeks or months after the reporting period has concluded.
For most law enforcement agencies, there are no publicly available data on shootings in their jurisdiction.
Still, this methodology can be regularly updated and provides unique insights into U.S. crime trends in as close to real time as possible. AH Datalytics’ murder dashboard tracks year-to-date murder counts in more than 100 cities nationwide, and — as of mid-August — it is showing a 13% decline in murder this year relative to last year. This provides strong evidence that murder is falling at a strong clip nationally, although time will tell if the current trend persists for the rest of the year. We won’t know just how far murder fell in 2023 until the FBI publishes its estimates in October 2024, but the real-time sample highlights a major change that is underway as it is occurring.
Another challenge — specifically as it relates to tracking gun violence — is that there are vast differences in how agencies report shooting data, if they do so at all.
The root of the problem is that the FBI does not require that agencies report data specifically on shootings.
Nonfatal shootings are treated as aggravated assaults for reporting purposes, so only a handful of agencies publish victim-level shooting data. Cincinnati, Philadelphia and Rochester police departments all publish detailed information on every shooting victim going back several years. Other agencies, like Chicago and Los Angeles, publish aggregated counts of shooting incidents or victims in their weekly update reports. For most law enforcement agencies, however, there are no publicly available data on shootings in their jurisdiction.
The National Crime Victimization Survey regularly points to crime being underreported, including dramatic underreporting of certain types of crime, so understanding crime trends is already beginning at a reporting disadvantage. On top of that, the slapdash system for collecting and reporting crime data makes it exceptionally difficult to identify trends or understand whether policies are effectively addressing problems.
Creating a world where crime data are reported with reasonable accuracy in something approximating real time is the only way to maximize the efficiency of these data and empower policymakers to respond to problems as they occur.