Politics, evidence and sheer dumb luck (not necessarily in that order)
Why isn’t the strength of evidence supporting an idea the very first thing a policymaker looks at when deciding what direction to take? There are so many reasons, but here are a few.
Knowledge is one: People making decisions often don’t know what the latest ideas, or even the most established ideas, from the academy are. Lights reduce crime, summer youth employment reduces mortality, for example.
Timing is another: Policy moves fast, research moves slowly. Policy is often formed in the flash of a crisis: What do we do right now? How can we show that we are addressing a problem in the most visible and action-oriented way possible? Research tends to be about finding ideas that are road- (or at least lab-) tested for the long run, and the search for knowledge unspools at a different pace.
Data and money, both hard to get, are other perennial obstacles to getting evidence to drive policy.
And of course, the great beast of politics and ideology overshadows much decision-making where evidence can be an inconvenience.
All of this was in the back of my mind at the beginning of the summer of 2014 when I was the director of the New York City Mayor’s Office of Criminal Justice. There had been a shooting spike in the city, concentrated in public housing. Of all the things that absolutely could not be permitted to happen in the first year of Bill de Blasio’s mayoralty, a rise in violent crime was at the top of the list. De Blasio had run an impassioned campaign, animated by his opposition to “stop and frisk,” a policy that had been implemented in the previous administration in so wanton a way that a federal court had eventually declared it unconstitutional. De Blasio’s use of this issue led many in the city’s establishment to worry that the streets would run with blood under his administration. Aware of this sentiment, and perhaps concerned about the reality too, de Blasio made a Nixon-to-China move, appointing Bill Bratton, the progenitor of broken windows policing under Rudy Giuliani, as his first police commissioner.
Each week I participated in a meeting in the mayor’s office with the mayor, the first deputy mayor, the police commissioner and the city’s corporation counsel. The office is a cozy one, tucked in a corner of New York City’s magnificent Federal-style City Hall, with signature arched windows looking out on City Hall Park and the Tweed Courthouse. Taking up one half of the office is Mayor Fiorello La Guardia’s desk, so big that it is hard to imagine how the Little Flower commanded it. But during this weekly meeting, we squashed companionably on the other side of the room in the Miss Marple-type chairs and sofa, overstuffed, floral-patterned Victoriana.
Even armed with good data, it was a fight — because of the usual agency politics, but also because looking at a study with Greek letters in it presented a different way of making decisions than public servants are accustomed to.
The meeting, in truth, followed the police commissioner’s agenda. But from time to time I would murmur a thought or two, once raising the idea that we could reduce crime substantially by combining both civic and police power. The previous few decades had taught a lot about what, beyond police power, makes a neighborhood safe. I ventured that we could use some of the accountability mechanisms so effectively pioneered by the Police Department, but this time in service of this combined approach. Just as the Police Department had a “CompStat” system that brought the entire department together regularly to look at crime numbers and strategize about what resources to deploy, so too could residents, community groups and various city agencies, from sanitation to youth services, come together to identify neighborhood conditions that a resident led plan could ameliorate. Essentially “NeighborhoodStat” was a way to rally resources and drive results.
As we discussed the shooting spike during that early summer evening, the mayor turned to me — I’m summarizing here — and said: “What about doing that whole vision thing you mentioned?” My job in the mayor’s office was not my first rodeo. I had just finished a stint working for a demanding governor, where I oversaw the state’s public safety agencies. If there was one thing I knew, it was that once a mayor or a governor says something like this, the door to funding is ajar just enough to permit a solid plan to blow it wide open.
The talented team at the Mayor’s Office for Criminal Justice worked furiously to assemble the pieces of a viable plan. We scrutinized crime rates in the city’s public housing developments and identified the small number of places that bore the brunt of violence. We put together a budget both to convene the NeighborhoodStat meetings, and also, critically, to have immediately visible assets and programs materialize in the key neighborhoods. At the top of our list were two things: keeping community centers open late for the first time in 30 years and lighting the dark places on the grounds of the housing developments where felony crimes clustered. These ideas did not come out of deep research or a thorough literature review. It was more in line with the “we’ve heard” and “everyone knows” school of thought, as in “we’ve heard that if kids have nowhere to go at night, they’ll get into trouble” and “everyone knows you get knifed in a dark alley.”
Of course, the plan needed to be shaped — by budget officials, by agencies’ needs and desires, by small- and big-p politics. For example, the plan needed to include all five boroughs, not just the top 15 places that bore the greatest brunt of the violence, even if that meant going pretty far down the list to ensure that Staten Island was included. It was also important that this effort feel substantial and have impact in every housing development, not just the 15, so the idea of opening community centers late, even though expensive, was extended throughout the city.
Oftentimes it is sheer serendipity whether evidence carries the day.
But we had almost no idea how to answer a fundamental question: Would all or any of this reduce crime? Who would do such an evaluation? These were not just questions of academic curiosity. The answers could have a real effect on whether, when next year’s budget rolled around, the programs implemented to meet the crisis of the moment would be sustained and become part of the way the City did business. Would the community centers continue the longer hours, would lighting be provided in a methodical way to all the places that needed it? To evaluate any of this would take money and time, and it turned out, data, of which we had precious little and no way to collect.
I had naively thought that academic institutions would not only jump at the chance to perform this kind of research but that the work would simply be part of their salaried day jobs. I soon found out that was not the case. Researchers, of course, need to be paid too. But the City’s procurement rules are so byzantine that it could be a year or more before someone was hired, even assuming we had the money. Bureaucracy is the price of integrity.
In a stroke of luck, another potential source of money materialized that we could tap. As part of a civil settlement, the Manhattan District Attorney’s Office had received an enormous sum of money that was to be dedicated to efforts to reduce crime. DA Cyrus Vance Jr. became a close partner in supporting the work in the housing developments, among other efforts.
We needed money for all kinds of things that you might not associate with a pilot program or with research: The City had to rent temporary lights, staff had to be hired or to work overtime because the temporary lights required fuel that needed to be refilled in the night, the sites had to be selected carefully, in line with the wishes of residents. Neighborhood residents bristled at the notion of being an “experiment,” as well they might have. So for many reasons, not least of which was that people are the experts in the conditions of their own lives, these teams were an essential part of site selection.
In the first round of results, Chalfin and his team showed that lighting resulted in nighttime major felony crime dropping by 36%. Would those kinds of results be good enough to persuade the City to invest in permanent lights? Yes!
Another crucial partner was the Arnold Foundation. Arnold provided seed money that allowed the brilliant Aaron Chalfin, then at Crime Lab New York, to conduct a randomized controlled study on whether lights reduce crime. One day, Aaron informed me that the study didn’t have enough “power.” It was the first time I had heard this concept. He wasn’t talking about wattage. No, it turned out we needed more locations to get a result reliable enough to prove or disprove the idea that lighting reduces crime. We were lucky that we had the money and backing to provide the necessary power.
In the first round of results, Chalfin and his team showed that lighting resulted in nighttime major felony crime dropping by 36%. A subsequent follow-up showed that those results were sustained over time. Would those kinds of results be good enough to persuade the City to invest in permanent lights? Yes!
With the results in and the City budget season starting up, the first deputy mayor convened a meeting with the budget director, police research and operations staff and me. We met at City Hall, an important signal to the group that this was no simple agency decision but one where the mayor himself had a stake. In City Hall, many of the rooms are designated by color, and we met in the Blue Room, an elegant hall with an Adam-style ceiling, hung with oversized 18th- and 19th-century portraits of famous New Yorkers. It is large enough to be the site of mayoral press conferences but gets reconfigured for big meetings as well. The room is magnificent but the lighting is bad and it is freezing in winter and boiling in summer.
Everyone had received a copy of the results, as well as a single page Cliffs Notes I had penned to try to summarize essential but puzzling (to a layman) technical language like this:
The optimal penalty term, λ, is selected via k-fold cross-validation by randomly partitioning the data into k different training sets and associated test sets. For each training set, a series of models are estimated for varying values of λ and predictions are computed on the associated test set. The optimal λ is chosen by taking the mean of the errors across the k test sets and choosing the value which minimizes this quantity.
The first deputy mayor kicked off a lively discussion during which it became clear the police were not happy. Why? It was not the Poisson regression, although there were brave attempts to dive in there. It appeared more that the concern was that the police might not control where the lights were placed and how the money was spent.
It is not so hard to imagine that bit by bit, or even all at once, cities could get to the point where every city agency has the time and capacity to look at research before it makes decisions.
The first deputy mayor was a patient man, a person both cerebral and practical who had headed up many complicated agencies and was familiar with budget, operations and politics. Eventually, he signaled he was wrapping up and said: “We have the money, we have strong evidence this is a cost-effective investment, we have a regular contract for city street lighting. Why wouldn’t we do this?” And that was that.
Or at least, that was that for that budget cycle. Even armed with good data, it was a fight — because of the usual agency politics, but also because looking at a study with Greek letters in it presented a different way of making decisions than public servants are accustomed to. And the decision for that year was just that: What would happen the year after, and the year after that? Who would devise and, importantly, execute a plan for the long term that meshed the Department of Transportation’s lighting contract with the City’s public safety needs and the desires of its residents?
What is there to learn from this effort to launch NeighborhoodStat and improve lighting in public housing? Sometimes the action has to come first — let’s light the dark places — and the evidence comes later. And evidence by itself does not speak clearly and plainly; it requires translation so that it communicates in a language that is comprehensible to the people making the decisions. Even then, the power of evidence is often overwhelmed by the power of politics — interagency, citywide, among public servants, elected officials and residents. Oftentimes it is sheer serendipity whether evidence carries the day.
But perhaps there is a way to tame the serendipity and to hard-wire evidence into the daily practice of governance in New York City and other places while respecting the important democratic role that politics can play in the mix. It is not so hard to imagine that bit by bit, or even all at once, cities could get to the point where every city agency — or perhaps just the city’s budget shop — has the time and capacity to look at research before it makes decisions. Perhaps an “evidence impact statement”?
The lighting story does not offer an easy template to replicate. A lot of dominoes had to fall just so. We were lucky that they did. But it shows where the opportunities are to begin to regularize what is otherwise a Wild West of decision-making. Perhaps it shines a light on how, as a simple way of doing business, government can make decisions, powered by evidence, to improve the quality of life for city dwellers.