AI Governance Starts Below the Model: Why Legacy Code Matters

Since 1999, courts in England and Wales have presumed that a computer was working correctly unless someone could prove otherwise.  Between 1999 and 2015, that presumption helped the Post Office prosecute more than 700 sub-postmasters for theft and false accounting, based on shortfalls reported by its Horizon accounting system (software that the software vendor knew contained bugs).  Hundreds received criminal records, some went to prison, and many were bankrupted.  The defendants couldn’t rebut it because they had no way to see inside the software, and the institution that could had little reason to look.

Horizon wasn’t an AI system.  It was ordinary accounting software with ordinary bugs.  But it remains the clearest case on record of what happens when an organization trusts the output of a system nobody can fully explain, and regulators haven’t forgotten the lesson.  When the UK Ministry of Justice opened a call for evidence on the presumption in January 2025, it said any reform would need to be future-proofed against the increasing use of AI, a lesson that applies directly to AI governance today.

Most AI governance writing today focuses on models, policies, and committees: model cards, acceptable use policies, risk tiers, and review boards.  That work matters.  In practice, though, an AI system is only as governable as the applications and data underneath it, and in most enterprises those applications are the least understood part of the stack.

Governance Comes Down to Three Questions

Strip away the frameworks, and every AI governance regime asks an organization to answer three questions.  What was the AI acting on?  Where did that data come from?  What changed, and when?

The EU AI Act makes those questions explicit for high-risk systems. Article 10 requires data governance, Article 11 requires technical documentation, and Article 12 requires that high-risk systems support the automatic recording of events over their lifetime.  The Digital Omnibus, which was promulgated in July 2026, pushed those high-risk deadlines to December 2027 and August 2028.  That bought time, but it didn’t reduce the work.

None of the three questions can be answered by examining the model alone.  The model reads from a system of record and acts through one.  If that system can’t explain itself, neither can the AI.

Legacy Applications Are the Least-Documented Part of the Stack

There’s an irony here that most governance programs miss.  A modern foundation model ships with a model card, published evaluations, and a usage policy.  The 20-year-old claims, billing, or eligibility application it reads from usually ships with nothing (the original developers left years ago, and the requirements document describes a system that no longer exists).

Consider what happens to a single field before an AI ever sees it.  A customer risk score might be written by a nightly batch job, adjusted by a COBOL routine that applies regional exceptions, truncated to fit a fixed-length field, and defaulted to zero when an upstream file arrives late.  Each step is a business rule.  None of them appears in the data dictionary.  When an auditor asks why the AI flagged an account or denied a claim, the honest answer may be that a program nobody has opened since 2009 quietly changed the input. That isn’t a model problem, and model explainability tooling won’t solve it.  Explainability tools describe how a model weighted its inputs.  They can’t tell you whether those inputs meant what everyone assumed they meant, or whether the system that received the AI’s output did what the AI intended.

Regulators Are Leaving More of This to You

Many executives assume regulators will eventually hand them a checklist.  In U.S. banking, the trend has moved the other way.  In April 2026, the Federal Reserve, OCC, and FDIC replaced SR 11-7, the risk guidance model banks had relied on for fifteen years, with SR 26-2.  The revised guidance explicitly places generative and agentic AI outside its scope, describing those models as “novel and rapidly evolving.”  Banks still own the risk.  They now must govern it under their own practices, and auditors will expect those practices to stand up to questioning.

Outside banking, the picture is similar.  The United States still has no federal AI law, and states continue to fill the gap with a record number of AI bills.  For an enterprise operating across several states or countries, governance must rest on something more stable than any single regulation, and the most stable foundation available is an accurate, verifiable understanding of what its own systems do.

AI Is Writing Code Faster Than Anyone Can Review It

A second blind spot gets even less attention.  The same organizations trying to govern AI are using AI to write the software it depends on, and early evidence on code quality should give governance leaders significant pause.

GitClear’s 2024 analysis of 211 million changed lines of code found an eightfold increase in duplicated code blocks of five or more lines, while the share of code being refactored fell sharply.  Veracode’s 2025 GenAI Code Security Report found that AI-generated code introduced security vulnerabilities in 45% of the coding tasks it tested, and that security performance wasn’t improving even as models got better at writing code that runs.

Taken together, those findings describe a governance problem hiding inside the engineering organization.  Codebases are growing and changing faster, with less consolidation and less human understanding of each change.  If you can’t prove what changed in an application, you can’t prove what changed in the behavior of the AI that relies on it.  Governance that stops at the model boundary will miss that drift entirely.

What an AI Governance Framework Needs Beneath It

A governable foundation isn’t another policy document.  It has specific, testable properties, and it’s worth asking whether the applications sitting in the path of your AI initiatives have them today.

  1. Current, as-built documentation of the business rules (including the exceptions), generated from the code rather than from memory.
  2. Data lineage that traces every field the AI consumes back through each calculation and handoff to its source.
  3. An automated test suite that captures current behavior, so any change, whether a person or an AI wrote it, can be verified before it reaches production.
  4. Modern REST or GraphQL APIs that record what the AI requested and what the system returned, which is the practical substance of Article 12.
  5. Ongoing monitoring that flags architectural drift in the codebase before it becomes a change nobody approved.

If the answer to most of those is no, the governance program rests on assumptions rather than evidence.  That’s the position the Post Office was in for years, and it isn’t one any board should accept for systems that make decisions about customers, patients, or citizens.

Start Below the Model

None of this argues for pausing AI.  It argues for sequencing.  Identify the applications that feed data to your highest-risk AI use cases, or receive actions from them, and make those applications explainable first.

It’s better to start with an honest picture of what the code does today.  Aspen Codex™ produces current, as-built documentation from existing application assets, surfacing the business rules, dependencies, and data relationships an auditor will ask about.  Aspen’s Warp Drive Engine™ then modernizes those applications with verifiable test suites and modern interfaces, without viewing or extracting any of the raw sensitive data, typically in 3 to 4 months for an application of about 600K lines of code.  Code Drift Monitoring keeps watch afterward, so the foundation stays governable as AI-assisted development speeds up.

The lesson of Horizon isn’t that software fails.  All software has bugs.  The lesson is that an organization must be able to demonstrate its work when the output is questioned.  And we should expect that someone will question that output in every audit.

Book a Call with Aspen

If your AI governance program is taking shape and you aren’t certain what sits underneath your highest-risk use cases, book a call with Aspen to discuss which applications need to become explainable first and what that would take.  Govern the foundation, and every model you place on top of it becomes easier to defend.

Schedule a discovery chat: https://calendly.com/aspen-ess/aspen-ess-discovery-chat

Sources

BCS, The Chartered Institute for IT, “Computer always right law must be revisited to avoid another Post Office IT scandal”: https://www.bcs.org/articles-opinion-and-research/computer-always-right-law-must-be-revisited-to-avoid-another-post-office-it-scandal/

Mayer Brown, “Technology in Disputes: The Double-Edged Sword of Justice” (April 2025), on the Ministry of Justice call for evidence: https://www.mayerbrown.com/en/insights/publications/2025/04/technology-in-disputes-the-double-edged-sword-of-justice

European Commission AI Act Service Desk, “Article 12: Record-keeping”: https://ai-act-service-desk.ec.europa.eu/en/ai-act/article-12

CASRAI, “AI Governance & Regulation” (Digital Omnibus, Regulation (EU) 2026/1744): https://casrai.org/news/topic/ai-and-research/ai-governance-and-regulation

Federal Reserve, SR 26-2, “Revised Guidance on Model Risk Management” (April 2026): https://www.federalreserve.gov/supervisionreg/srletters/SR2602.htm

American Banker, “Regulators’ guidance on model risk leaves many questions unanswered”: https://www.americanbanker.com/opinion/regulators-guidance-on-model-risk-leaves-many-questions-unanswered

TLT, “AI Brief: November 2026”: https://www.tlt.com/insights-and-events/insight/tlts-ai-brief-november-2026

GitClear, “AI Copilot Code Quality” (2025), as reported by LeadDev: https://leaddev.com/technical-direction/how-ai-generated-code-accelerates-technical-debt

Veracode, “2025 GenAI Code Security Report” (July 2025): https://www.veracode.com/press-release/ai-generated-code-poses-major-security-risks-in-nearly-half-of-all-development-tasks-veracode-research-reveals/