Golden Master Testing for AI-Assisted Software (Part 3)

From agentic maturity to deterministic, self-sustaining systems

Part 3: Golden master testing for AI-assisted software

AI can now write a working integration in an afternoon.

Keeping it working is a different problem.

This article stands on its own. It’s for any team using AI to write or maintain ordinary, deterministic software, at any level on the maturity scale from Part 1. It’s also what makes L5 (Part 2) possible: moving AI work into code only pays off if that code doesn’t turn into a maintenance problem.

The short version

• Capture what the code does today for a set of real inputs, have a person confirm it’s right, and lock it in. That’s a golden master.

• Run every golden case on every change. Any difference stops the release until a person says it’s intended.

• Wrap that in a loop where the AI finds and fixes problems and a person makes three decisions. Every fix adds a case, so fixed problems stay fixed.

 

AI writes code faster than teams can maintain it

AI coding agents have changed the cost of writing software. A feature that used to take a sprint can be generated in an afternoon. A plain-code replacement for an agent workflow can be produced on demand.

Maintenance hasn’t kept pace. Generated code still breaks when:

  • An upstream system changes its data or its API.
  • A field arrives in a format nobody saw in testing.
  • An edge case shows up in production.
  • The business changes the rule.

Each of those failures has to be diagnosed, fixed and kept from coming back. If people do all of that by hand, the speed of generation becomes a liability. The codebase grows faster than anyone can understand it.

Generating less isn’t the answer. Making maintenance as automated as generation is, with people kept where judgment matters.

How a golden master works

How a golden master test works. Today’s output is compared with an approved output, and only a person can accept a difference.

Golden master testing (also called characterization or snapshot testing) has been used on legacy code for years. You don’t write down what the code should do. You capture what it does do for a representative set of real inputs, confirm that behavior is right, and lock it in as the baseline. From then on, any change that alters one of those outputs fails until a person confirms the change is intended.

GM  Why golden masters fit AI-written code

And where they stop.

NO SPEC NEEDED

Generated code rarely comes with a complete specification. A golden master captures behavior from real data, so you can protect code nobody specified line by line.

IT SCALES WITH GENERATION

The same agent that writes the code can capture the golden cases for it. Test coverage grows as fast as the code does.

IT CHECKS THE OUTSIDE

Behavior is checked at the boundary, whatever the code looks like inside. That matters when an agent wrote most of it and a person reviewed it lightly.

WHERE IT STOPS

It needs the same output for the same input. Parts of a system where AI still runs at runtime vary from run to run, and need evals instead (Part 4).

 

The golden test harness is the full set of these cases: curated, versioned inputs paired with approved outputs, run on every change.

A worked example: an invoice-matching flow

Say an AI agent built a flow that reads supplier invoices, matches each line to the purchase order, and decides what happens next. Here’s one golden case for it.

What

Input

Approved output

Invoice

Supplier A, invoice 10442, 3 lines, total $12,480.00

 

Purchase order

PO 7781, 3 lines, total $12,300.00

 

Line 2

Invoice price $1,240 vs PO price $1,180

Mismatch: unit price over PO by 5.1%

Decision

 

Route to a person: price variance over the 2% tolerance

Lines 1 and 3

Match the PO

Approved for payment

 

Three months later the supplier starts sending prices with four decimal places. The agent updates the parser. The harness runs, and this case now shows line 2 as a match.

That’s the moment golden masters exist for. The difference stops the release. The AI traces it to a rounding change in the new parser, a person confirms it’s a bug, the fix goes in, and the four-decimal invoice becomes golden case number 31. That failure can’t come back without someone noticing.

An unreviewed snapshot is a recording of today’s bugs

A golden output is only golden once a person has confirmed it’s right. Capturing outputs is easy. The value is in the approval.

 

The closed-loop remediation cycle

The seven-step loop. Dark steps are where a person decides.

The harness gets powerful when it sits in the middle of a continuous maintenance loop that the AI runs and a person steers.

Step

What happens

Who

1. Monitor

User reports from chat, forms and tickets, production errors, failed runs and odd outputs come in

The AI

2. Diagnose

Reproduces the problem, traces it through the code and data, and shows what failed, why, where and the evidence

The AI, then a person confirms

3. Negotiate

Agree on the fix. The person can challenge the cause, weigh options, set limits, or decide it isn’t a defect

A person decides, with the AI

4. Implement

Code, configuration and documentation, to the agreed plan

The AI

5. Regress

Every golden case runs. Nothing ships with a difference nobody explained

The AI, and a person signs off intended changes

6. Grow the harness

The scenario that caused the issue, with its corrected output, becomes a new golden case

The AI, with a person approving the output

7. Release and learn

The fix deploys, and the pattern feeds back into monitoring

The AI

 

Who does what

Across the loop, the AI does the work and a person makes three decisions.

The loop is light-touch, but it isn’t hands-off. A person owns three decisions:

  1. Is the diagnosis right? The AI proposes. A person confirms.
  2. Is this the right fix? The plan is agreed, not handed down.
  3. Is a changed output intended? When a golden master changes, a person signs off the new baseline.

Everything else is the AI’s work: reproducing, tracing, coding, running the tests and growing the harness. The effort per issue moves from hours of debugging to minutes of review.

Building the harness

H  Six habits for a harness that holds up

Start small, keep it honest.

SEED

Start with real inputs that cover the main paths, known edge cases and past failures. A few dozen well-chosen cases per component is a good start. Distinct behaviors matter more than volume.

CURATE

Every case earns its place. Remove duplicates, tag each case with the behavior it protects, and record why it exists, ideally with a link to the issue that created it.

APPROVE

A person confirms every golden output before it counts.

MASK THE NOISE

Timestamps, IDs, ordering and rounding noise get normalized or masked, so the harness compares what matters.

VERSION

Cases, inputs and approved outputs live in source control next to the code. Changes to a baseline go through review like any other change.

START WHERE IT HURTS

Seed first around the code that breaks most often, or costs the most when it does. Coverage elsewhere can follow.

 

Why this turns debt into an asset

The three sources of technical debt, and how the loop removes each one.

Technical debt is the future cost of changing code. It comes from not knowing what the code does, from fear of breaking it, and from the effort of fixing it. The loop takes away all three.

With the loop in place, generated code stops being something to apologize for. It’s behavior that’s written down, protected and cheap to change. That’s what lets a team, and an L5 organization in particular, generate code with confidence.

What good looks like over time

Illustrative: the harness grows with every fix, and repeat issues fall toward zero.

The harness is a ratchet. It only moves toward more coverage. Track these to see it working:

Measure

What you want to see

Time from issue to diagnosis, and from diagnosis to fix

Falling

Human minutes per resolved issue

Falling

Golden cases in the harness, and growth per month

Rising steadily

Regression escapes (issues that get past the harness)

Rare, and each one becomes a case

Repeat issues (previously fixed problems coming back)

Close to zero

 

Next in the series

Golden masters work because ordinary code gives the same output for the same input. Parts of a system where AI still runs at runtime don’t.

Part 4, eval-driven development, adapts the same loop to AI behavior, using golden datasets, scored evaluations and continuous measurement in place of exact-match baselines.

Is your AI-written code protected like this today? We’re happy to help you set up the harness and the loop.

Leave a Reply

© Copyright 2021 Distributed Logic Corp.

contact via web form  |  info@distributedlogic.com   |    (866) 229-8621