Making LLM features testable

The uncomfortable property of LLM output

A traditional function is deterministic enough that a unit test with a fixed input and expected output is meaningful: same input, same output, every time. An LLM call breaks that assumption — the same prompt can produce different phrasing, different structure, occasionally a different answer, across calls and definitely across model versions. Teams that treat an LLM feature like a normal function to test end up either not testing it at all, or writing brittle string-equality tests that fail on harmless rephrasing and get deleted the first time someone’s annoyed by them.

The fix isn’t to give up on testing. It’s to test the parts of the system that are actually deterministic — the shape of the output, whether specific facts appear, whether the system behaves the same on a fixed input across model versions — and treat “is this a good response” as an evaluation problem with its own tooling, not a pass/fail unit test.

Force structure first

Before anything else, an LLM feature used for something that downstream code will act on should never return free text that gets parsed with regex. It should return a structure the model is constrained to produce and your code validates on the way in:

const TriageDraftSchema = z.object({
  patternDetected: z.string(),
  supportingFactors: z.array(z.string()),
  contradictingFactors: z.array(z.string()),
  suggestedAction: z.enum(["clear", "request_info", "escalate"]),
});

async function draftTriage(packet: RedactedCase): Promise<TriageDraft> {
  const raw = await model.complete({ prompt: buildPrompt(packet), responseFormat: "json" });
  return TriageDraftSchema.parse(JSON.parse(raw));
}

This alone makes a whole category of failure testable: does the model reliably produce valid JSON matching the schema, or does it fail validation on some fraction of inputs? That’s a metric you can track and regress against, independent of whether any individual response is “good.”

An evaluation set is the actual test suite

The real test suite for an LLM feature is a curated set of representative inputs with known-acceptable properties for their outputs — not exact strings, but checkable properties: “this case should suggest escalate,” “this response must not include the customer’s name,” “this summary must mention the specific rule that triggered the flag.”

const evalCases: EvalCase[] = [
  {
    id: "eval-014",
    input: loadFixture("structuring-pattern-with-history"),
    expect: {
      suggestedAction: "escalate",
      mustMention: ["prior flag", "structuring"],
      mustNotContain: [/\b\d{4}\s?\d{4}\s?\d{4}\s?\d{4}\b/], // no card-shaped numbers
    },
  },
  // dozens more, covering routine clears, ambiguous cases, edge patterns
];

async function runEval(cases: EvalCase[]) {
  const results = await Promise.all(cases.map(async (c) => {
    const output = await draftTriage(c.input);
    return { id: c.id, pass: checkExpectations(output, c.expect) };
  }));
  return results;
}

This runs in CI against every change to the prompt, the model version, or the surrounding code. A drop in pass rate is a regression, exactly the way a failing unit test is, even though no individual case is being compared for exact string equality.

Regression tests for prompt and model changes

Every time the prompt is edited or the model is upgraded, the full evaluation set runs before the change ships, and the pass rate is compared against the baseline from the previous version. A prompt tweak that improves one case type while quietly regressing another is exactly the failure mode this catches, and it’s also the reason the evaluation set needs to be large and varied enough to cover the actual distribution of inputs the feature sees in production — a set of five happy-path cases will not catch a regression in how ambiguous cases get handled.

Cost ceilings, tested like any other budget

An LLM feature has a cost per call that a bug can turn into a runaway bill — a retry loop with no backoff, a prompt that balloons in size as conversation history grows unbounded. These get tested the same way you’d test any other resource budget:

test("triage draft stays within token budget", async () => {
  const packet = loadFixture("largest-realistic-case");
  const usage = await draftTriageWithUsageTracking(packet);
  expect(usage.promptTokens).toBeLessThan(4000);
  expect(usage.completionTokens).toBeLessThan(500);
});

And in production, a hard ceiling — a per-request token cap, a circuit breaker on the calling service if cost per hour crosses a threshold — is the operational backstop for the case where a test didn’t catch what production traffic did.

What doesn’t need testing this way

Not every LLM-touched code path needs an evaluation set. The schema validation, the redaction step before data reaches the model, the retry and timeout logic around the API call — all of that is ordinary deterministic code and gets ordinary unit tests. Reserve the evaluation-set approach for the part that’s actually non-deterministic: the model’s judgment given a well-formed input.

What this means for you

Don’t test an LLM feature by asserting on its prose. Constrain it to a validated structure, build an evaluation set of representative cases with checkable properties instead of exact outputs, run that set as a regression gate on every prompt or model change, and put a tested ceiling on cost the same way you’d cap any other resource. The parts of the feature that are deterministic get unit tests like everything else; the part that isn’t gets evaluated, not asserted.