We need to change the way the City buys software.
You don’t have to look very far in New York to find failed government technology projects. A City Comptroller’s report released in August found that veterans still struggle to use the VetConnectNYC online portal to access services, despite a $2 million investment and eight years of work. The City’s Office of Technology and Innovation requested an additional $81 million in the 2026 budget to shore up another failed digital service, the “one-stop-shop” MyCity app, after already spending four years and $100 million on development. The Metropolitan Transportation Authority is reportedly planning to pilot “virtual OMNY cards,” part of a contract signed in 2017 that has suffered extensive delays, scope reduction, and millions of dollars in change orders.
Failures of this magnitude are not unusual in government tech. Across a database of over 2,700 projects, researchers found that the median U.S. government IT project costs 310% of the estimated price.
Mayor Zohran Mamdani’s lauded Public Interest Technology (PIT) Crews — five teams of technologists designed to swoop in to agencies and deliver the mayor’s top digital priorities — are a positive start to addressing the overwhelming amount of technical debt in city agencies. There is no question that city government needs more professionals with skills that meet the moment. However, delivering digital services on time, on budget and in ways that actually solve user needs requires addressing government functions beyond IT. Most of the tools for transforming government technology are (in the words of Matthew Burton, who wrote about the problem recently for Vital City) “starkly non-technological.”
Fixing tech procurement is the most important piece to get right. Despite recent successes by in-house development teams, digital government transformation at scale requires a hybrid staffing model, where the talent of in-house teams is complemented by that of external vendors. Contracting out offers temporary surge capacity of software developers with niche skills. It also responds to the present realities that government tech staff are hindered by: a lengthy and cumbersome civil service hiring process, comparatively low salaries and operating budget constraints.
Which is to say, though it may make a lot of sense to bulk up government tech, contracting out will still be necessary. The key is to do it right.
Although some tech can be purchased off-the-shelf, the scale and complexity of New York City government obligations, regulations and public-sector union rules often require custom software. And here, we need a new model.
New York City and State agencies procure custom software using a linear, step-by-step model known as “waterfall.” Requirements are fully specified at the start of a project, and modifications typically require costly contract change orders, much like a vacation package with a fully-booked itinerary that cannot be altered without major fees. Vendors are paid based on predefined project milestones, instead of being paid for the delivery of functional code. Software is delivered at the end of the development cycle, meaning that government agencies only have the chance to provide feedback at the end of a project, months or even years after requirements were drafted. Sadly, only 13% of “waterfall” projects succeed.
Government oversight bodies would have you believe that exhaustive requirements and more vigilant contract monitoring are the only ways to improve government technology, but they are not. The “Agile” approach to software procurement uses minimal requirements, clear objectives and iterative cycles of user research, development, testing and deployment that allow for continuous improvement. This is the standard for private sector software development and what actually works, if done correctly, for government agencies, too. Initiatives built using Agile succeed at three times the rate of waterfall projects, where “successful” means on time and on budget, with a satisfactory result.
This is clearly a wiser way to develop software that will make government work better.
The good news is that New York City doesn’t have to start from scratch. Last year, the New York State Higher Education Services Corporation (HESC) issued the first “Agile RFP” in the state. Built on the foundation of successful examples from across the federal government, like Direct File, that RFP brought best practice to the State and did the hard part of adjusting contract terms and conditions to the New York State context. The HESC RFP, for example, shifts contract performance measurement away from checkbox fulfillment of pre-defined project requirements (the waterfall way) to assessment of vendor work based on code quality and performance standards.
But if the City is serious about developing software more intelligently, it needs to change some practices. Software development using Agile implies new ways of working. Vendors work in teams of four to nine people (called “Scrum teams” in Agile lingo) that are tightly coupled with a product owner and technical lead from a government agency. For early stage product development initiatives, only a single Scrum team may be required. Its work is nimble and iterative; fluid communication and integrated teamwork are key.
The City’s current approach to procurement makes this way of working difficult, if not impossible. Local Law 2019/174 specifies percentage goals of total annual agency expenditure for six classes of minority-owned businesses, adding up to 73.52% of total spend for professional service contracts. In my experience, procurement officers apply all or most of these percentage goals across all contracts. Requiring blanket target participation rates on a single Scrum team would imply a minimum of six distinct companies supplying staff for a four-person team — a certain (and nonsensical) recipe for failure.
To get the most from Agile procurement, the City Council, the City’s chief procurement officer, and the City’s Procurement Policy Board should unite to eliminate participation goals for these contracts. This change to procurement policy has nothing to do with whether you believe in supporting minorities and everything to do with whether procurement supports the ways of working necessary for modern software development using Agile.
Beyond procurement, enabling a leap forward in government technology requires changes to the city budget itself. Modern software development is not a one-time capital expense, like buying Microsoft Office was in 2010. It is an ongoing effort that requires a team (and a budget) over time to provide an experience analogous to routine updates of a Google Chrome browser. Even when vendors leave after a large digital product build, agency staff continue to respond to evolving user needs through continuous development. This implies a small but consistent budget allocation for technologists. This “product model,” which combines Agile procurement and development with continuous improvement and funding, stands in sharp contrast to government’s traditional “project model” of one-time funding, rigid scope and time-bound outputs.
New York’s legacy of fiscal austerity, however, haunts any increase (albeit small) to the operating budget, despite the lower net cost and higher rate of success of the product model. The City’s Office of Management and Budget should get on board with a shift toward the product model and approve operating budget funding streams for technologists to support software products. In practice, this may look like budgeting for “capabilities” that require multiple software products or systems, rather than budgeting one product at a time. According to policy researchers at the Niskanen Center, “a ‘capability’ might be eligibility and enrollment, claims processing or benefits delivery.” A block of funds would be designated for each capability, including all the software necessary for that capability, giving product teams the responsibility for end-to-end delivery of the capability. This is analogous to setting aside a budget to remodel your entire kitchen, rather than paying for the cabinets and drawers this year and leaving a gaping hole where the countertops should be until some unknown future date.
Enabling technological transformation in City agencies also requires a shift in the way legal teams attempt to mitigate risk. General counsel and auditors often want “one throat to choke,” namely a vendor’s, when something goes wrong. With the traditional waterfall approach to software procurement, agencies try to shift all risk back to the vendor. They rely on meeting specific contractual requirements and adhering to Gantt charts to determine whether a vendor has delivered a product, regardless of whether the software meets users’ needs. Risk is mitigated through close contract monitoring and fines for late or incomplete delivery. The extraordinarily high failure rate of software projects indicates that these risk mitigation measures simply don’t work.
In contrast, software procurement using Agile begins with a statement of objectives. Government staff have significantly more control over software development because they are intimately involved in every step of the process (e.g., identifying tasks, user research, testing, quality assurance, deployment, system integration). The “test-and-learn” approach inherent to Agile means faster identification and correction of problems, and smaller, controlled failures. These failures can be rectified through the iterative development cycle, rather than waiting until the end of the traditional “project” model for user testing. Risk can be further reduced by committing vendor-developed source code to the public domain, a practice encouraged by the General Services Administration’s former technology branch and the Department of Defense.
Despite the successful Agile track record, it feels more risky than the status quo. It will be uncomfortable, but necessary, for the Mayor’s Office of Contract Services, agency attorneys and the City Comptroller to accept that risk mitigation under the “product” model looks different from the “project” model. What this means, in practice, is that contract oversight must shift from output (did the vendor deliver the requirement) to outcomes (did the vendor team, under staff guidance, build something that meets user needs). The recommended short duration (less than three years) and low cost ($2 million or less, annually per contract) of an Agile contract allows for a quick pivot if a vendor doesn’t perform.
Lastly, technologists cannot fix processes that agency staff struggle to articulate. It is no small task to explain “how things work,” when such knowledge is tacit, still on paper or housed in a legacy system. Agency leaders must get their own houses in order, documenting existing workflows and requirements, information that will accelerate the work of the PIT Crews, an agency’s own technologists or vendors.
In New York, the area most ripe for stalling tech projects is labor relations. Benefits obtained through the collective bargaining process are least likely to have short-term flexibility for modifications that might facilitate the adoption of new technology or permit the use of either commercial off-the-shelf software or software as a service. Rigid constraints, whether explicitly written into contracts or ossified through longstanding practice, drive custom software design and can easily increase costs. For example, unlike the majority of transit operators in the United States, subway conductors and operators in New York City select their weekly days off independently from the shifts and specific operational jobs they work (“cafeteria style picking”). This unusual procedure impedes automation and increases costs, relative to “roster-based” job selection (where operators select bundled days off and vehicle runs); it also precludes the use of most existing software.
Knowing what is bendable is crucial for tech system design. Layers of memos that modify a decades-old agreement, rather than an updated and comprehensive document detailing all current aspects of a labor agreement, present major barriers to digitalization and software development. Labor relations teams must be able to communicate the restrictions of current labor agreements with precision and in a way that technologists can translate into code.
We need better government tech, but the problems preventing us from getting it are not technological. Transforming government tech depends on an overhaul of non-technological government functions. That must not wait.
