Using AI to measure — and improve — city government performance
Former New York City Mayor Ed Koch was famous for asking New Yorkers, “How’m I doin’?” It is a crucial question for any incumbent mayor seeking reelection: Voters tend to punish their mayors for increases in the homicide rate, increases in local inflation, increases in local unemployment, local unemployment that is higher than national unemployment, and decreases in student test scores.
Of course, crime rates, local economic conditions, student test scores and other outcomes voters care about are surely affected by factors that lie outside of a mayor’s control. But a mayor who cares about reelection will try to improve outcomes as much as possible using the levers at his disposal. And in the face of binding budget constraints, a mayor will want to find the programs that most improve residents’ lives per dollar spent.
A critical challenge, however, is that a mayor and his deputies simply do not know what effect their city’s policies and programs are having on outcomes. In the trade, we call this the fundamental problem of causal inference. A mayor needs to know not only whether outcomes are trending in the direction of making his constituents better or worse off. He needs to know how those outcomes would have changed had his administration done something differently. And the mayor and his team need to be able to make dollar-for-dollar comparisons across programs: How much would outcomes have improved per dollar spent, had the City invested in program B instead of program A?
The reason this is a fundamental problem is that it requires knowing what happens to outcomes in two alternative worlds: a world in which we implement some program, and a world in which we don’t. But we only ever observe outcomes in the “factual” world (the one in which we implement the program), not the “counterfactual” world (the one in which we do not).
This fundamental problem of inference confounds efforts to improve city government performance. Consider New York City’s Mayor’s Management Report (MMR), required by the City Charter since 1977, which reports more than 2,000 “performance indicators.” As others have pointed out, about 2/3 of the MMR’s indicators actually measure process, not outcomes. For example, the MMR’s “critical indicator” for measuring how well the NYPD is doing in reducing quality-of-life violations is the number of quality-of-life summonses that have been issued. But even those MMR indicators that measure outcomes don’t tell us anything about government performance. For example, the MMR reports that domestic violence assaults in NYC have been increasing over time. Does that indicate poor performance by the NYPD? Not necessarily! Perhaps domestic violence assaults would have increased even more steeply, had the NYPD not taken the actions it did. But perhaps domestic violence assaults would have increased less steeply, or even declined, had the NYPD done something differently. Because the MMR reports only trends in outcomes, not the impacts of programs on outcomes, it provides little guidance to mayors seeking to improve their management of the city’s tax dollars.
So, how can mayors and their deputies ever know how programs and policies are impacting outcomes voters care about? Until very recently, the only reliable way for a mayoral administration to learn about the causal impacts of its programs was to work with researchers trained in causal inference. Those research teams might run randomized controlled trials (RCTs) of government programs. Or they might look for “natural experiments” that approximate the conditions of RCTs. These methods work because they leverage the power of random assignment to alternative “worlds” (e.g., a constituent is randomly or as-if-randomly assigned to participate or not participate in a program) to learn about the average outcomes in both of those worlds, approximating both the factual and the counterfactual outcomes we care about.
But partnering with the scientific community on program evaluation has often been a lengthy, expensive and unappealing process for incumbents. It has required recruiting a research team with the right expertise and experience, coming up with funding sufficient to secure that team’s services, negotiating legal agreements providing access to restricted data, providing staff time to pull, clean and transfer the requisite data, and waiting for the results of the evaluation (which in any case would likely only have assessed the effects of a single program in isolation). As anybody who has worked on these kinds of projects can attest, securing data access agreements alone can take several years. By the time a mayor has the results of an evaluation in hand, his next election has in all likelihood come and gone.
This is probably why some jurisdictions have opted for government performance assessments that are quicker and easier but substantially less reliable. The California Assembly, for example, has implemented a process for “Outcomes Reviews” whereby a legislator can hold hearings on how outcomes changed after passage of a bill of their choice. But hearings aren’t going to credibly identify the difference between factual and counterfactual outcomes that we need in order to rigorously assess and improve government performance.
The good news is that AI has substantially reduced the time and expense of program evaluation. AI applications like Claude for Government provide access to powerful large language models within high-security environments that protect agency data. And the scientific community has learned how to deploy these models to execute sophisticated program evaluations at minimal time and expense. With appropriate scaffolding and human oversight, AI models can now capably write and implement code that retrieves, cleans and analyzes data using credible causal inference designs. This new AI-assisted evaluation capacity could significantly reduce the need for lengthy negotiations over data-sharing agreements and the transfer of data off of agency servers. Instead, evaluation experts would simply need to be able to interact with an agency’s AI model; agency data could remain secure behind a firewall.
By the time a mayor has the results of an evaluation in hand, his next election has in all likelihood come and gone.
How can Mayor Zohran Mamdani and his team best take advantage of this new capacity to improve outcomes for New Yorkers? There are at least three clear use cases. Together, they create the potential for a new kind of performance tool for the city — a Mayor’s Performance Report, if you will — that could help agencies redirect their budgets to support the programs that most improve outcomes per dollar spent.
First, AI models that sit behind agency data firewalls could be asked to assess the performance of agencies’ programs, policies and contracts using natural experiments and leveraging agencies’ own administrative data. For example, the City spent $32.4 billion on contracts in FY26, or approximately 25% of its budget. Which of those contracts delivered the most value for money, relative to alternatives? Having reliable estimates of changes in outcomes per dollar spent would help mayors and their commissioners find the contracts that are overperforming (and those that are underperforming).
Second, AI models could be asked to design efficient and inexpensive RCTs that could be implemented internally, without the need for time-consuming and expensive contracts with external research organizations. For example, would it be more cost-effective to bring some externally contracted functions in-house? AI-designed RCTs could test the cost-effectiveness of service delivery under a contract model versus an in-house model, enabling agencies to implement the continual test-and-learn cycles advocated by the government reform community.
Third, AI models could be asked to help agencies design and test new efficiency-enhancing tools, including those that enable service delivery to be brought in-house instead of contracted out. For example, labor-intensive contracts to count people or cars on the street are expensive. Blanketing the city with car- and people-counting sensors is also expensive. But calibrating less reliable but inexpensive data (e.g., anonymized cellphone location data) to more reliable but expensive sensor data, thereby enabling citywide use of the inexpensive calibrated data, is the kind of project an AI agent can develop and test efficiently.
At least for now, the AI models still need scaffolding and oversight from scientific experts in order to produce credible policy and program evaluations, RCT designs and decision assistance tools. But those experts no longer need direct access to agency data, and certainly don’t need to export data off of agency servers. Research experts just have to be able to interact with an agency’s AI model to ensure that models produce reliable and relevant outputs. Local research universities could easily provide agencies with this kind of expert research guidance across policy domains.
Voters reasonably ask that mayors use their taxpayer dollars to make their lives better. A critical obstacle has been that mayors and their deputies did not and, in fact, simply could not know how to cost-effectively improve outcomes for constituents. Now, with the help of AI models (and a little help from the scientific community), mayors have a valuable new capacity to deliver improved government performance.






