Almost overnight, AI became the technology every boardroom wanted to discuss. Executives saw a way to cut costs, speed up operations, and open new revenue streams, so they pushed their business units to integrate AI quickly. Far fewer paused to ask how AI ROI would be delivered or measured once the pilots were over.
Adoption followed. According to McKinsey, nearly 90% of enterprises now use AI in at least one business function, and 56% use it across three or more functions. No enterprise technology has spread this fast. The returns, however, have been a different story.
Against a backdrop of $30 to $40 billion in enterprise investment, MIT’s NANDA initiative examined how companies were using generative AI and found that about 95% of pilots had no measurable impact on the P&L. Most executives have seen that number by now. Fewer have asked the follow-up question that matters: if the models improve noticeably every quarter, why don’t returns improve?
The 95% figure is debated (the report was preliminary, and every study defines “pilot” and “success” slightly differently), but the larger evidence offers little comfort. A 2024 RAND Corporation study notes that, by some estimates, more than 80% of AI projects fail, twice the failure rate of IT projects that don’t involve AI. IBM’s 2026 report, “Is your AI paying off?”, puts the average return on AI investment for large enterprises at 17%.
In most of those projects, the AI worked. The model summarized, classified, predicted, and drafted exactly as advertised. What failed was everything surrounding it: the applications, the data structures, and the business processes the AI was supposed to improve.
The Pilot Was Never the Hard Part
Consider how a typical pilot gets built. A small team pulls an extract from the system of record, cleans it up in a spreadsheet or a cloud data store, and points a model at it. Six weeks later, the demo is impressive, the steering committee approves the next phase, and everyone expects production to be a matter of scaling up.
It isn’t. The pilot only had to read data. Production has to act on it, which means writing results back into a 20-year-old order management system, respecting its validation rules, fitting its field lengths, and running inside its batch windows. That is a different project entirely, and it is usually where the timeline doubles, and the business case quietly dies.
Cost follows the same pattern. A pilot runs on a sample of data and a handful of users, so its inference, storage, and cloud bills look modest. At full scale, those costs grow in ways the current business case never modeled, and legacy systems make the problem worse, because every extract, copy, and translation layer the AI depends on carries its own compute and maintenance cost. IBM’s 2026 Tech Leader Study found that 85% of technology leaders lack real-time visibility into AI spending, and that organizations report cloud costs running nearly 50% above original projections. A business case built on pilot economics can be wrong before the first production invoice arrives.
In short, most pilots succeed by going around the legacy application. Production can’t go around it, because the legacy application is where the work actually happens.
The Rules Live in the Code, Not the Data
The industry has spent the last three years telling executives that AI success depends on data readiness, and that is true as far as it goes. But organizations that spend heavily on data platforms usually discover that their AI still makes recommendations that the business won’t accept, and the reason is rarely the data itself.
The reason is that the rules governing how the business operates aren’t stored in the data. They’re stored in the code. A credit hold routine written in 1998 might contain a dozen exceptions for specific customer classes, regions, and contract types, none of which appear in any requirements document and all of which the business relies on every day. A pricing engine might apply a discount sequence that three people understand (two of whom have retired).
When an AI model recommends an action without knowing those rules, it contradicts the system of record. Users notice, start overriding the recommendations, and within a few months the tool becomes something people work around rather than with. Nobody files a failure report. Adoption flattens, and the projected savings never show up.
You can’t feed undocumented logic to a model, and you can’t safely automate a process when nobody can explain what the current process does.
The AI Can’t Learn What Nobody Wrote Down
MIT’s researchers offered their own explanation for the 95%, and it received far less coverage than the number itself. They called it a learning gap: most enterprise generative AI tools don’t retain feedback, adapt to context, or improve over time. One user interviewed for the study put it plainly: “the tool repeats the same mistakes and requires extensive context input for each session.” It’s no surprise that the research found that, for mission-critical work, 90% of users still prefer a human.
Part of that gap is a product limitation, and the tools are getting better. A larger part is a context problem. A model can only adapt to the business if the business can describe itself, and in most enterprises the most important context (the exceptions, the sequencing, the reason a field gets populated a certain way) lives in code nobody has documented in years. Retrieval, fine-tuning, and agent memory all depend on source material. When that source material is a million lines of undocumented COBOL or RPG, there is very little reliable context to utilize.
That is why current, as-built documentation isn’t a side task in an AI program. It is the context layer the AI needs before it can learn anything useful about how the business actually runs.
Faster Tasks, Same Process
Another pattern is subtler and shows up in how value gets measured. Most AI business cases count minutes saved at the task level: an adjuster reviews a claim in four minutes instead of twenty, or an analyst finishes a report before lunch instead of the next day.
Those savings are real, but they often never reach the P&L because the overall process is constrained elsewhere – think Theory of Constraints. If an AI drafts a claims decision in 30 seconds and the mainframe still posts it in an overnight batch, cycle time doesn’t change. The task got faster. The process didn’t.
This helps explain one of the more telling findings in IBM’s 2026 research: only 25% of executives say their organizations consistently measure business value and AI ROI. Some of that is governance discipline. Some of it, I suspect, is that teams sense the numbers won’t hold up once you measure the whole process instead of the step the AI touched.
Agents Raise the Stakes
Agentic AI makes all of this more urgent. An AI tool that only makes suggestions can live with a messy back end, because a person reviews each suggestion before anything reaches the system. An agent that takes action can’t. It needs stable APIs to call, transactional integrity so a partially completed action doesn’t corrupt a record, and automated tests so changes can be verified before they reach production.
Most legacy applications offer none of those things. There are no REST or GraphQL APIs, only screen scraping or flat-file transfers. There’s no test suite, so nobody will approve an agent changing records without a person checking every step, and once a person checks every step, the labor savings disappear. Gartner expects more than 40% of agentic AI projects to be canceled by the end of 2027, and its analysts are direct about one reason: integrating agents into legacy systems is “technically complex, often disrupting workflows and requiring costly modifications.”
Where AI ROI Actually Comes From
The MIT research contained a finding that received far less attention than the 95% headline. More than half of generative AI budgets went to sales and marketing tools, while the biggest returns came from back-office automation: replacing outsourced process work, cutting external agency costs, and improving operations. That is no accident. Back-office processes are repetitive, rules-based, and high-volume, which makes them a good fit for AI.
They are also the processes most likely to run on the oldest code in the portfolio. The best AI opportunities and the worst legacy constraints sit in the same place, which is exactly why so many promising projects stall.
IBM’s research found the same misalignment from another angle: only 6% of organizations direct their five largest AI investments toward the five business functions generating their highest AI returns. In other words, most companies aren’t spending their AI money where AI is actually paying off.
RAND’s research adds a caution worth heeding here. In interviews with 65 experienced data scientists and engineers, the most common root cause of AI project failure had nothing to do with technology: stakeholders misunderstood, or miscommunicated, what problem the AI was supposed to solve. Choosing the right problem and the right process matters as much as choosing the right model, and it’s the one decision no amount of engineering can fix after the fact.
Before funding the next AI initiative, it’s worth asking a short set of questions about the applications in its path:
- Can anyone explain, in writing, what the current process does, including its exceptions?
- Does the system expose APIs the AI can call, or will integration depend on extracts and screen scraping?
- Is there an automated test suite that can confirm a change didn’t break something downstream?
- Where does the process actually wait (batch jobs, queues, manual approvals), and will AI change that constraint at all?
- Who will measure the outcome at the process level six months after go-live?
If the answer to the first three is no, the project isn’t an AI project yet. It’s a modernization project with an AI goal attached, and it should be scoped, funded, and sequenced that way.
IBM’s research points in the same direction. Organizations that explicitly include modernization and technical-debt remediation in their AI business cases project almost 30% higher AI ROI than those that treat modernization as a separate investment. The figure shows expectations rather than measured results, but it’s telling.
Fix the Foundation in the Path
None of this argues for delaying AI until every legacy system is replaced. The practical approach is narrower: identify the applications sitting directly in the path of your highest-value AI use cases, and modernize those first.
It’s better to start with an honest picture of what the code does today. Aspen Codex™ produces current, as-built documentation from existing application assets, surfacing the business rules, dependencies, and data relationships that an AI initiative will encounter. Aspen’s Warp Drive Engine™ then modernizes those applications with generated test suites and modern interfaces, typically in three to four months for an application of about 600K lines of code, rather than the 18 months a traditional approach requires. The organizations that see real returns from AI over the next few years won’t necessarily be the ones with the best models. They’ll be the ones whose applications can actually put those models to work. You know what I know: the model is the easy part.
Book a Call with Aspen ESS
If your AI roadmap is taking shape and you’re weighing which initiatives to fund, book a call with Aspen to discuss the applications sitting in the path of your highest-value use cases and what it would take to make them ready. Fix the foundation once, and every AI project that follows gets easier.
Schedule a discovery chat: https://calendly.com/aspen-ess/aspen-ess-discovery-chat
Sources
MIT NANDA, “The GenAI Divide: State of AI in Business 2025” (July 2025), as reported by Fortune: https://fortune.com/2025/08/18/mit-report-95-percent-generative-ai-pilots-at-companies-failing-cfo/
RAND Corporation, “The Root Causes of Failure for Artificial Intelligence Projects and How They Can Succeed” (August 2024): https://www.rand.org/pubs/research_reports/RRA2680-1.html
IBM Institute for Business Value, “Is your AI paying off?” and “2026 Tech Leader Study,” as reported by TechInformed (September 2026): https://techinformed.com/ibm-says-cloud-costs-and-tech-debt-erode-ai-returns/
Gartner, “Gartner Predicts Over 40% of Agentic AI Projects Will Be Canceled by End of 2027” (June 2025): https://www.gartner.com/en/newsroom/press-releases/2025-06-25-gartner-predicts-over-40-percent-of-agentic-ai-projects-will-be-canceled-by-end-of-2027