AI to Enhance Transparency and Accountability
David Tan

Open data notwithstanding, public records are not yet public knowledge.

Recent pushes for more open government have put an enormous volume of official documents and information in the public domain. Across federal, state and local governments, official records, including bills, hearing transcripts, court decisions and other data, are now regularly published, many of them free for everyone to read. But making sense of these documents takes expertise and resources that few people have, limiting the public’s ability to scrutinize what public agencies and servants are doing and to hold them accountable. Artificial intelligence can change this. New York’s highest court shows us how.

The New York Court of Appeals is the final authority on questions about New York state law, including the state’s constitution. Its decisions resolve everything from everyday matters, like divorce and custody disputes, to matters of national significance, like redistricting. The court will soon have a vacancy for one of its seven seats, with at least three more vacancies following over the next few years. Each term lasts 14 years, so there are high stakes for New Yorkers in each appointment: The decisions made about these vacancies will shape the law in the state for more than a decade — and in turn, they will shape millions of New Yorkers’ lives. 

If you have not heard about the Court of Appeals, or do not know how its decisions affect your life, you are hardly alone. U.S. Supreme Court decisions are analyzed by countless journalists, academics and policy wonks, but the decisions of state high courts attract little public attention — even within their own states. That is why many New Yorkers are familiar with the names and records of U.S. Supreme Court justices like Antonin Scalia and Amy Coney Barrett, but not those of Court of Appeals judges like Jenny Rivera, whose expiring term will create the upcoming vacancy.

In 2025, the Court of Appeals dismissed a lawsuit claiming that New York City’s public school system, including its admissions policies, denies Black and Latino students the sound basic education and equal protection the state constitution guarantees. Judge Rivera was one of two judges who would have allowed at least part of the case to move forward. This is one of hundreds of decisions that Rivera has participated in that have had direct relevance to millions of New Yorkers, but very few New Yorkers have heard of it — or even of Rivera herself. 

This information gap is not due to the court system’s opaqueness — a consequential and real problem in New York in its own right — but rather to the difficulty of turning public records into public knowledge. Court of Appeals decisions are publicly and freely available for everyone to read. But to understand even one decision and its impact, a person often needs to read dozens of pages of dense “legalese” and have specialized knowledge: a deep understanding of that area of the law and a good mental model of who will be impacted by any change, and how. That task becomes more complex and resource-intensive if you want to understand a judge’s record over years, and more so still if you want to compare judges’ records. Many people have worked to generate collective public knowledge and understanding of the decisions of the U.S. Supreme Court, where immense resources are brought to bear on every aspect and angle of every ruling, as well as the entirety of decisions over multiple years. So far, no one has done this for the decisions of any state’s highest court.

U.S. Supreme Court decisions are analyzed by countless journalists, academics and policy wonks, but the decisions of state high courts attract little public attention — even within their own states.

Lack of public knowledge and understanding of the Court of Appeals, like that of other state high courts, is an especially acute problem because accountability for its work is possible. Scrutiny of U.S. Supreme Court decisions is widespread, but because the justices serve lifelong appointments, meaningful accountability for their decisions and behavior is limited to impeachment. In contrast, New York’s state court judges, like the state court judges of almost every other state, do not receive life tenure. Instead, all New York state court judges must seek new terms, through either reelection or reappointment. Court of Appeals judges who seek reappointment must be vetted by a screening committee, nominated by the governor and confirmed by a vote of the state Senate. 

For this process to create the accountability it was designed to provide, New Yorkers must have knowledge and understanding of the real-world effects of the Court of Appeals’ decisions. Without this public knowledge, judicial selection decision-makers — the governor, senators and the screening committee — must rely on anecdotal evidence, their personal impressions and political considerations. Even when these stakeholders read and weigh decisions, they are incredibly unlikely to be able to accurately assess a record containing upwards of a thousand decisions. The same is true for even the most involved New Yorkers, advocates and journalists, who have access to far less information than insiders. Therefore, while evaluation of judges is built into our Court of Appeals’ appointment system, we often fail to take meaningful advantage of the opportunity. 

The organization I founded, Scrutinize, is using artificial intelligence to close this information gap and help all New Yorkers learn about and better understand the Court of Appeals, its decisions, and how each judge has ruled. Our data program runs the text of each Court of Appeals decision through a series of prompts to large language models (LLMs). Some steps ask the LLMs for simple information, like the names of the judges who took part in the decision. Other steps ask for information that requires some more analysis, like identifying the issue areas that the decision affects, such as criminal law, tenant protections or voting rights. More complex steps call on the LLM to substantively analyze the law, such as by extracting the rules that each opinion in the decision articulates, or comparing the effects of such rules on an issue area. Our program does this for over 1,700 decisions, every Court of Appeals decision since Judge Rivera, the longest-serving judge, joined the bench. The result, which we recently made publicly and freely available on our website, mimics what has been produced for the U.S. Supreme Court: it is a trove of public knowledge about the positions that each judge has taken across a wide range of issue areas, and it allows people to compare judges to each other.

Our work demonstrates how large language models can change the economics of producing public knowledge from public records. Instead of relying on an army of lawyers and policy wonks with specialized knowledge and expertise, we were able to analyze more than 1,700 dense, difficult legal texts, and do so quickly and at scale. Though developing and testing this process took time and money, those resources were vastly less than what was previously required to undertake a project like this. 

The potential to turn government transparency into public knowledge does not stop with the Court of Appeals, or even with the courts. Our data program can easily be adapted to analyze court decisions in other states, but the approach to converting public records into public knowledge that we have developed is even more widely applicable. It can be used to analyze changes in the language of legislative bills — and to understand who stands to benefit from such changes — or to analyze hearing transcripts and assess the positions of legislators, officials and people who submit testimony. For all the dangers that artificial intelligence poses, it offers vast new possibilities for making the inner workings of the government available and comprehensible to many more people. 

Using LLMs to create public knowledge comes with its own set of concerns. LLMs make mistakes, fail at simple tasks and “hallucinate” information. We should worry about the quality and accuracy of LLM-generated analysis and data, especially for fact- and meaning-sensitive work like legal analysis. Without care, we may end up producing misinformation, not public knowledge. 

But the process of developing this program has led me to believe that the solution is to create and follow a system of protocols, best practices and quality assurance procedures that give analysts and users confidence in the accuracy of the analysis. The process by which we generated our analysis is the first version of such a framework. For example, after the LLM extracted a legal rule, we subjected the rule to two different verification steps and restarted the extraction if it failed either of them. Similarly, we ran the analysis of each decision 10 independent times and marked the analytical judgment unresolved unless at least eight of the 10 runs agreed. Most importantly, every case links to the model’s underlying output, we have published a detailed methodology, and we provide users with the ability to flag any errors they find. 

The tool we’ve built offers no judgments on how the judges have ruled. It is a comprehensive, rigorous, quantitative database that analyzes a large number of decisions and presents the analysis in plain language so that everyone can understand how each judge has interpreted the law and what those interpretations mean for New Yorkers’ everyday lives. New York policymakers chose renewable terms over life tenure for our judges, rejecting the life appointments in the federal judiciary. That choice can only benefit New Yorkers if the people deciding whether to give judges additional terms know what those judges have decided. Democracy works best with an informed and engaged citizenry; some might say that’s the only way it works at all. 

Gov. Kathy Hochul and the state Senate will decide whether to reappoint Judge Rivera or appoint someone to replace her in a few months, and they will fill at least three more seats in the next few years. For the first time, the people making those choices, and the New Yorkers they answer to, can read what each judge has actually decided on hundreds of questions, before each of these consequential appointment decisions is made.


Great! You’ve successfully signed up.

Welcome back! You've successfully signed in.

You've successfully subscribed to Vital City.

Success! Check your email for magic link to sign-in.

Success! Your billing info has been updated.

Your billing was not updated.