Testing an AI agent requires replacing pass/fail assertions with statistical evaluation: you build an eval harness — a versioned golden dataset of representative inputs, a scoring method appropriate to each output type, and a regression gate that compares aggregate scores across model and prompt versions. You do not assert that the agent returned exactly "Your refund was processed"; you assert that across 200 refund cases, the correct tool was called 98%+ of the time and no response scored below rubric threshold on policy compliance. Traditional unit and integration tests still apply to the deterministic scaffolding around the agent (tool implementations, retries, auth, parsing) — they just cannot evaluate the agent's judgment.

That distinction is where most teams get stuck. LangChain's State of Agent Engineering report (1,340 practitioners, surveyed late 2025) found quality is the single largest barrier to deploying agents, cited by 32% of respondents — ahead of latency, security, and cost. Yet in the same survey, 29.5% of organizations run no evaluation practices at all, and while roughly 89% of teams with production agents have adopted observability, only about 52% run offline evals. Teams are watching their agents in production without any ability to tell whether a change made them better or worse.

Why does a traditional test suite break against an AI agent?

It breaks in four specific ways, and recognizing them is faster than debugging them one at a time.

1. Non-determinism reads as flakiness. CI turns red intermittently, so someone adds a retry or sets temperature=0 and calls it solved. Neither works — batching, GPU non-determinism, and provider-side routing all produce drift even at temperature 0, and retries just hide variance instead of measuring it.

2. There is no ground truth string. For "explain why this invoice was rejected," a dozen phrasings are correct and a dozen more are subtly wrong. Exact-match and regex assertions force tests against superficial output features, which is how a suite passes while the agent quietly hallucinates policy details you never asserted on.

3. CI inputs are not user inputs. Test fixtures get written in the phrasing the engineer had in mind. Real users type fragments, misspellings, and requests that sit just outside scope. An agent scoring 95% on developer-written fixtures routinely drops 20+ points on transcripts pulled from real traffic.

4. Every test is a full integration test. Agents call tools, retrieve documents, and loop over multiple turns. A single "failure" could originate in retrieval, the tool schema, the system prompt, or the model — without per-step trace assertions, you get a red build and no localization.

The practical consequence: teams stop trusting the suite, disable it, and ship on vibes. Then the pilot cannot demonstrate reliability to a risk owner, and it stalls. That is the mechanism behind the widely cited industry estimate — attributed to IDC and echoed by Forrester and Anaconda research — that roughly 88% of AI agent pilots never reach production. Treat the number as directional rather than precise, but the pattern is real: pilots die at the reliability-evidence stage, not the demo stage.

What is an eval harness, and how do you build one?

An eval harness is a repeatable pipeline that runs a fixed dataset of inputs through your agent, scores each output, and produces aggregate metrics you can compare across versions. Three components, in build order.

Golden dataset

Start with real historical cases, not synthetic ones. Pull 100–300 actual transcripts, tickets, or queries from your logs (or from the human-run process the agent replaces), and label the expected outcome — which tool should fire, what facts must appear, what must never appear. Then deliberately add edge cases: ambiguous requests, out-of-scope requests the agent must decline, prompt-injection attempts, multi-intent inputs, and inputs in the languages you actually support.

A useful split is roughly 70% representative traffic, 30% adversarial and edge. Version the dataset in git alongside the agent code. Freeze a holdout slice that never informs prompt engineering — otherwise you will overfit prompts to your own test set and mistake memorization for improvement.

Scoring methodology

Match the method to the output type, and use the cheapest one that works:

• Deterministic checks for structured behavior: was the right tool called, with the right arguments, in the right order? Did the JSON validate? Was the citation ID present in the retrieved set? These are fast, free, and cover more agent behavior than most teams expect — often 50–70% of what matters.
• Rubric-based human review for the high-stakes slice. Slow and expensive, but it is your source of truth. LangChain's survey found human review remains the most common method at 59.8%.
• LLM-as-judge for scaled qualitative scoring (helpfulness, tone, policy adherence), used by 53.3% of respondents. It only earns trust if you calibrate it: score a sample with humans, measure agreement with the judge, and iterate the judge prompt until agreement is acceptable. An uncalibrated judge is a random number generator with good manners.

Regression gates

Store every run's results with the model version, prompt version, and dataset version. Gate merges on aggregate thresholds and on category-level thresholds — a change that lifts the overall score by 2 points while collapsing refusal behavior on adversarial inputs should fail. Because scores are statistical, define your gate in terms of a meaningful delta (for example, more than a 3-point drop on any category, or any regression on the safety subset), not exact equality.

Run the harness on every prompt change, every model upgrade, and every retrieval-index rebuild. Model provider version bumps are the single most common source of silent regression, and they happen on the provider's schedule, not yours.

What does it actually cost to build and maintain an eval harness?

Budget three to six engineer-weeks for a first credible harness on a single agent workflow, assuming you already have logs to mine. The rough shape:

Ongoing, plan on 0.2–0.5 FTE per production agent. Evals decay: usage patterns shift, new features add tool paths, and a dataset built at launch is measurably stale within a quarter. Schedule a monthly refresh — sample recent production traces, label new failure modes, retire cases the agent has trivially solved for months. Also budget the inference cost of running evals, which is why most teams run the fast deterministic subset per-commit and the full suite nightly.

Buying a platform compresses the runner, storage, and dashboard work — genuinely useful. It does not compress dataset curation or judge calibration, which is where most of the cost lives. Start with a harness you own in your repo; adopt tooling once you know what you need it to do.

Who should own agent evaluation — QA, ML engineering, or a dedicated role?

The pragmatic answer for most teams: a QA lead owns the harness and the dataset; an ML or backend engineer owns the scorers and the runner; the product owner signs off on the rubric. Do not create a dedicated eval engineer until you have three or more agents in production — before that, the role becomes a bottleneck and a single point of knowledge failure.

The reasoning behind that split:

• QA engineers already own the two hardest parts of this work — thinking adversarially about inputs and maintaining test data over time. Their gap is statistical scoring and prompt mechanics, which is a few weeks of upskilling, not a career change.
• ML engineers understand scoring and non-determinism but tend to optimize aggregate benchmark numbers and under-invest in the messy real-world edge cases that actually break production agents.
• Product must own the rubric, because "is this answer acceptable?" is a business-policy question. If engineers write the rubric alone, you will ship an agent that is technically correct and commercially wrong.

The failure mode to watch for is evals living inside the ML team's notebooks. If the harness is not in CI and not runnable by anyone on the team, it is a research artifact, not a quality gate.

How does a delivery partner shorten this path?

The value is process, not tooling. A team that has instrumented several agent projects arrives with a labeling schema, a calibrated judge template, a trace-capture pattern, and — most importantly — an opinion about which 30% of your agent's behavior actually needs qualitative scoring versus which 70% can be covered by cheap deterministic assertions. That last judgment call is typically what separates a three-week harness from a three-month one.

At Synchronized Codelab, our agent engagements start by building the eval harness before feature work expands, because the harness is what makes every later decision — model swap, prompt rewrite, retrieval change — measurable instead of speculative. If you are building this in-house, the sequence in this article is the same one we follow; the harness matters far more than who builds it.

FAQ

Can you use temperature 0 to make AI agent tests deterministic?

No. Temperature 0 reduces output variance but does not eliminate it — request batching, hardware-level non-determinism, and provider-side model routing all introduce drift. More importantly, an agent that only behaves correctly at temperature 0 has not been proven reliable at the settings you run in production. Measure variance instead of suppressing it: run each eval case several times and track score distribution, not a single result.

How large should a golden dataset be for an AI agent?

Start with 100–300 labeled cases per distinct workflow. Below about 50, aggregate scores are too noisy to detect real regressions; above a few hundred, curation and inference costs grow faster than the signal. Prioritize coverage of distinct failure modes over raw volume — 150 well-chosen cases beat 1,000 near-duplicates.

Is LLM-as-judge reliable enough for CI gates?

Only after calibration. Score 50–100 cases with human reviewers, measure the judge's agreement with those labels, and refine the judge prompt and rubric until agreement is high enough for your risk level. Use a different model family for judging than the one under test where possible, and keep safety-critical categories under human or deterministic review rather than judge-only scoring.

Do traditional unit and integration tests still matter for agent systems?

Yes, and they cover more surface than teams assume. Tool implementations, API clients, auth, retry logic, output parsers, retrieval plumbing, and cost/rate limits are all deterministic and should be tested conventionally. The eval harness only needs to cover the model's judgment layer — keeping that boundary clean makes both suites cheaper to maintain.

How often do AI agent evals need to be updated?

Plan a monthly refresh and a mandatory refresh whenever you add a tool, change retrieval sources, or upgrade the underlying model. Usage patterns shift and datasets built at launch typically go stale within a quarter. Sample recent production traces, label new failure modes, and retire cases that have passed consistently for months.

Should we buy an eval platform or build our own harness?

Build a minimal harness first, then buy. A platform compresses the runner, storage, and dashboards — roughly a week of engineering — but not dataset curation or judge calibration, which is where most of the effort lives. Building first means you evaluate vendors against requirements you actually understand.