From agentic maturity to deterministic, self-sustaining systems
Part 3: Golden master testing for AI-assisted software
AI can now write a working integration in an afternoon.
Keeping it working is a different problem.
This article stands on its own. It’s for any team using AI to write or maintain ordinary, deterministic software, at any level on the maturity scale from Part 1. It’s also what makes L5 (Part 2) possible: moving AI work into code only pays off if that code doesn’t turn into a maintenance problem.
|
The short version • Capture what the code does today for a set of real inputs, have a person confirm it’s right, and lock it in. That’s a golden master. • Run every golden case on every change. Any difference stops the release until a person says it’s intended. • Wrap that in a loop where the AI finds and fixes problems and a person makes three decisions. Every fix adds a case, so fixed problems stay fixed. |
AI writes code faster than teams can maintain it
AI coding agents have changed the cost of writing software. A feature that used to take a sprint can be generated in an afternoon. A plain-code replacement for an agent workflow can be produced on demand.
Maintenance hasn’t kept pace. Generated code still breaks when:
- An upstream system changes its data or its API.
- A field arrives in a format nobody saw in testing.
- An edge case shows up in production.
- The business changes the rule.
Each of those failures has to be diagnosed, fixed and kept from coming back. If people do all of that by hand, the speed of generation becomes a liability. The codebase grows faster than anyone can understand it.
Generating less isn’t the answer. Making maintenance as automated as generation is, with people kept where judgment matters.
How a golden master works
How a golden master test works. Today’s output is compared with an approved output, and only a person can accept a difference.
Golden master testing (also called characterization or snapshot testing) has been used on legacy code for years. You don’t write down what the code should do. You capture what it does do for a representative set of real inputs, confirm that behavior is right, and lock it in as the baseline. From then on, any change that alters one of those outputs fails until a person confirms the change is intended.
|
GM Why golden masters fit AI-written code And where they stop. |
|
|
NO SPEC NEEDED Generated code rarely comes with a complete specification. A golden master captures behavior from real data, so you can protect code nobody specified line by line. |
IT SCALES WITH GENERATION The same agent that writes the code can capture the golden cases for it. Test coverage grows as fast as the code does. |
|
IT CHECKS THE OUTSIDE Behavior is checked at the boundary, whatever the code looks like inside. That matters when an agent wrote most of it and a person reviewed it lightly. |
WHERE IT STOPS It needs the same output for the same input. Parts of a system where AI still runs at runtime vary from run to run, and need evals instead (Part 4). |
The golden test harness is the full set of these cases: curated, versioned inputs paired with approved outputs, run on every change.
A worked example: an invoice-matching flow
Say an AI agent built a flow that reads supplier invoices, matches each line to the purchase order, and decides what happens next. Here’s one golden case for it.
|
What |
Input |
Approved output |
|
Invoice |
Supplier A, invoice 10442, 3 lines, total $12,480.00 |
|
|
Purchase order |
PO 7781, 3 lines, total $12,300.00 |
|
|
Line 2 |
Invoice price $1,240 vs PO price $1,180 |
Mismatch: unit price over PO by 5.1% |
|
Decision |
|
Route to a person: price variance over the 2% tolerance |
|
Lines 1 and 3 |
Match the PO |
Approved for payment |
Three months later the supplier starts sending prices with four decimal places. The agent updates the parser. The harness runs, and this case now shows line 2 as a match.
That’s the moment golden masters exist for. The difference stops the release. The AI traces it to a rounding change in the new parser, a person confirms it’s a bug, the fix goes in, and the four-decimal invoice becomes golden case number 31. That failure can’t come back without someone noticing.
|
An unreviewed snapshot is a recording of today’s bugs A golden output is only golden once a person has confirmed it’s right. Capturing outputs is easy. The value is in the approval. |
The closed-loop remediation cycle
The seven-step loop. Dark steps are where a person decides.
The harness gets powerful when it sits in the middle of a continuous maintenance loop that the AI runs and a person steers.
|
Step |
What happens |
Who |
|
1. Monitor |
User reports from chat, forms and tickets, production errors, failed runs and odd outputs come in |
The AI |
|
2. Diagnose |
Reproduces the problem, traces it through the code and data, and shows what failed, why, where and the evidence |
The AI, then a person confirms |
|
3. Negotiate |
Agree on the fix. The person can challenge the cause, weigh options, set limits, or decide it isn’t a defect |
A person decides, with the AI |
|
4. Implement |
Code, configuration and documentation, to the agreed plan |
The AI |
|
5. Regress |
Every golden case runs. Nothing ships with a difference nobody explained |
The AI, and a person signs off intended changes |
|
6. Grow the harness |
The scenario that caused the issue, with its corrected output, becomes a new golden case |
The AI, with a person approving the output |
|
7. Release and learn |
The fix deploys, and the pattern feeds back into monitoring |
The AI |
Who does what
Across the loop, the AI does the work and a person makes three decisions.
The loop is light-touch, but it isn’t hands-off. A person owns three decisions:
- Is the diagnosis right? The AI proposes. A person confirms.
- Is this the right fix? The plan is agreed, not handed down.
- Is a changed output intended? When a golden master changes, a person signs off the new baseline.
Everything else is the AI’s work: reproducing, tracing, coding, running the tests and growing the harness. The effort per issue moves from hours of debugging to minutes of review.
Building the harness
|
H Six habits for a harness that holds up Start small, keep it honest. |
|
|
SEED Start with real inputs that cover the main paths, known edge cases and past failures. A few dozen well-chosen cases per component is a good start. Distinct behaviors matter more than volume. |
CURATE Every case earns its place. Remove duplicates, tag each case with the behavior it protects, and record why it exists, ideally with a link to the issue that created it. |
|
APPROVE A person confirms every golden output before it counts. |
MASK THE NOISE Timestamps, IDs, ordering and rounding noise get normalized or masked, so the harness compares what matters. |
|
VERSION Cases, inputs and approved outputs live in source control next to the code. Changes to a baseline go through review like any other change. |
START WHERE IT HURTS Seed first around the code that breaks most often, or costs the most when it does. Coverage elsewhere can follow. |
Why this turns debt into an asset
The three sources of technical debt, and how the loop removes each one.
Technical debt is the future cost of changing code. It comes from not knowing what the code does, from fear of breaking it, and from the effort of fixing it. The loop takes away all three.
With the loop in place, generated code stops being something to apologize for. It’s behavior that’s written down, protected and cheap to change. That’s what lets a team, and an L5 organization in particular, generate code with confidence.
What good looks like over time
Illustrative: the harness grows with every fix, and repeat issues fall toward zero.
The harness is a ratchet. It only moves toward more coverage. Track these to see it working:
|
Measure |
What you want to see |
|
Time from issue to diagnosis, and from diagnosis to fix |
Falling |
|
Human minutes per resolved issue |
Falling |
|
Golden cases in the harness, and growth per month |
Rising steadily |
|
Regression escapes (issues that get past the harness) |
Rare, and each one becomes a case |
|
Repeat issues (previously fixed problems coming back) |
Close to zero |
Next in the series
Golden masters work because ordinary code gives the same output for the same input. Parts of a system where AI still runs at runtime don’t.
Part 4, eval-driven development, adapts the same loop to AI behavior, using golden datasets, scored evaluations and continuous measurement in place of exact-match baselines.
Is your AI-written code protected like this today? We’re happy to help you set up the harness and the loop.
