AI Agent Regression Testing: Production Traces to CI
AI Agent Regression Testing: How to Turn Production Traces into CI Gates
TL;DR: Trace-based regression testing turns a failing production agent trace into a CI test case. The trace's recorded tool and model outputs become fixtures, so the test replays offline and deterministically on every pull request. Tracely is an MIT-licensed open-source project that packages this loop: OTLP ingest, graders, failure clustering, promotion of a trace to a case, and a GitHub Action gate. Langfuse, LangSmith and Promptfoo cover parts of the same loop. None of them removes the main engineering cost, which is routing every tool and model call through a seam you can record and replay.
Key Takeaways
- An agent regression test is a test built from a real failure. A trace that went wrong in production becomes a case your CI must pass forever after.
- Hermetic replay is what makes the test usable in CI. Recorded tool and LLM responses stand in for live calls, so the suite needs no API keys, costs nothing in tokens, and gives the same result every time.
- A fail-to-pass contract is the check that the test is trustworthy. The case must fail on the old code and pass on the fix, or the test proves nothing.
- Tracely (about 1.2k GitHub stars, MIT, Python/FastAPI with ClickHouse) bundles ingest, grading, clustering, promotion and a tracely-gate GitHub Action in one self-hosted stack.
- Langfuse and LangSmith let you add traces to datasets and run experiments over them. Promptfoo runs assertion suites in CI. All three re-call the model live unless you add your own fixture layer.
- Replay only works if the agent sends tool and model calls through a recordable seam. Plan that refactor before you pick a tool.
Why do AI agents need regression tests built from traces?
Agent failures rarely show up in the test inputs a team writes on day one. They come from real conversations: a user phrases a refund request the planner did not expect, a tool returns an empty list instead of an error, or a model upgrade changes how often the agent calls a search tool. By the time someone sees the failure, it lives in a trace (a nested record of every prompt, model response, tool call and result in one run).
Teams usually handle this in one of two ways, and both leak:
1. Hand-written eval sets. Someone reads the bad trace, writes a new test input and an expected output, and adds it to a YAML file or dataset. The new input is a paraphrase of what happened, not a copy. The tool responses that caused the failure are usually lost, and the test then calls live tools that now return different data.
2. Dashboards with no gate. The observability platform records the failure, an online evaluator scores it low, and the score sits on a chart. Nothing stops the next pull request from bringing the same failure back.
Trace-based regression testing closes that gap. The trace is the test: its exact inputs, the exact tool outputs the agent saw, and the model responses in recorded order. A grader decides pass or fail. The suite runs on every pull request and blocks merges that bring a known failure back.
This is the same discipline as "write a failing test before you fix the bug" in ordinary software. The difference is that agents are non-deterministic and depend heavily on external services, so the test needs recorded fixtures before it can be repeated.
What does a trace-to-test pipeline look like?
The loop has five stages. Every implementation, whether bought or built, needs all five.
| Stage | What happens | What breaks if it is missing |
|---|---|---|
| Observe | Every production run emits a trace (OpenTelemetry gen_ai.* or OpenInference attributes) indexed by agent, conversation, turn and step | You cannot reproduce a failure you never recorded |
| Grade | Structural checks (schema, tool-call order, no model needed) and LLM judges score runs online | Failures are found by users, not by the system |
| Triage | Failing runs are clustered by structural signature and semantic similarity | 31 broken runs become 31 tickets instead of one issue with a count |
| Promote | A representative failing trace becomes a hermetic case with its fixtures recorded | The "test" re-calls live services and drifts |
| Gate | CI replays the suite against the change and fails the build on regressions | The fix ships, and the next refactor quietly reverts it |
The Promote stage is where most homegrown setups stop. Copying a trace's input into a dataset is easy. Copying the world the agent saw (tool results, retrieval hits, intermediate model outputs) and replaying it in order is the actual work.
How does Tracely implement trace-native CI/CD?
Tracely is an open-source project by Julien Wuthrich that describes itself as "trace-native CI/CD for AI agents." It is MIT-licensed, has about 1.2k stars at the time of writing, and is still early (a few dozen commits). Its stack is a Python backend (FastAPI, Celery, uv) and a Next.js 15 UI. It stores traces in ClickHouse, the case registry in Postgres with pgvector, the queue in Redis and blobs in MinIO/S3. A local demo starts with docker compose --profile demo up -d --build --wait.
Ingest: OTLP first, SDK second
Any OpenTelemetry exporter can post to POST {endpoint}/v1/traces with a Bearer ingest key. Tracely reads both gen_ai.* semantic-convention attributes and OpenInference attributes, so existing instrumentation keeps working. The Python SDK (tracelyai-sdk) adds auto-instrumentation for OpenAI, Anthropic, Google, Mistral and LangChain:
import tracely_sdk as tracely
tracely.init(
endpoint="http://localhost:8000",
api_key="tracely_dev_key",
service_name="support-agent",
env="prod", # prod | staging | ci | dev — the gating axis
instrument="auto",
)
with tracely.trace(agent="support-agent", conversation="conv-1", user="u_42"):
client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": "Where is order ORD-4471?"}],
)
The env field matters more than it looks. It separates production traces (the source of failures) from CI traces (the ones the gate grades), so the same pipeline can serve both.
Grade and cluster
Evaluators come in two kinds. Structural checks need no model. LLM judges run at conversation, run or span level and can output a score, boolean, number, text or JSON-schema result. Failing runs are then clustered "structurally and semantically" with a structural signature plus embedding similarity. In the project's words, "31 broken runs become one issue with a count."
Promote with a fail-to-pass contract
This is the most important idea in the project. Promoting a failing trace bundles "recorded input, tool and LLM outputs" as fixtures. The case is trusted only if it "must fail on the old code and pass on the fix." A regression test that passes on the buggy code is a test that cannot fail, which is the most common way evaluation suites lie to their owners. The contract makes that check part of the workflow instead of leaving it to discipline.
Multi-turn behaviour uses scenarios instead of single cases: either a scripted conversation, or an adversarial goal that a red-team model improvises against.
Gate in GitHub Actions
Replay runs offline against the recorded fixtures, with no API keys and no model spend. tracely gate exits non-zero, posts a commit status and upserts a PR comment. The bundled action has two modes. Tracely can call your agent through an entrypoint (tracely replay planner --entrypoint my_pkg.agent:run, or --cmd "node run.js" for non-Python agents), or it can grade the CI traces your own test job emitted:
- uses: jwuthri-cloud/Tracely-ai/.github/actions/tracely-gate@master
with:
mode: gate
agent: planner
api: https://tracely.your-co.dev
key: ${{ secrets.TRACELY_KEY }}
Pin that @master reference to a commit SHA before you rely on it. A gate that can change underneath you is not a gate.
The constraint you cannot skip
The README states it plainly: hermetic replay "requires your agent to route tool/model calls through the SDK's call_tool / call_llm seam." Auto-instrumentation is enough to record a trace. It is not enough to replay one, because replay means intercepting the call and returning the recorded value instead of making it. If your agent calls requests.get from inside a tool, or a framework makes model calls you cannot reach, those calls go live during replay.
Other caveats from the project itself: judges and failure analysis degrade without an LLM key, the default AUTH_MODE=dev with tracely_dev_key is open access and only suitable for local use, and the Celery worker does not hot-reload.
How do Langfuse, LangSmith and Promptfoo handle the same loop?
None of the established tools is called "trace-native CI," but each covers some of the stages. The question is which stages you get for free and which you build.
| Stage | Tracely | Langfuse | LangSmith | Promptfoo |
|---|---|---|---|---|
| Production tracing | Yes (OTLP + SDK) | Yes, core product | Yes, core product | No (test runs only) |
| Online grading | Structural + LLM judges | Online evaluators on sampled traffic | Online evaluators | No |
| Failure clustering | Built in | Manual filtering and annotation | Manual filtering and annotation | No |
| Trace → test | Promote to hermetic case | Add trace to dataset | Add run to dataset | Copy input into YAML test case |
| Recorded tool fixtures | Yes, bundled on promote | Inputs/expected outputs only | Inputs/reference outputs only | No, you write mocks |
| CI gate | tracely-gate action, PR comment | Experiments via SDK in your CI job | Evaluations via SDK in your CI job | promptfoo eval + GitHub Action |
| Offline / no model spend | Yes, by design | No, experiments call the model | No, evaluations call the model | No, unless you mock providers |
| Maturity | Early, small community | Mature, MIT core | Mature, commercial | Mature, MIT |
A few clarifications on that table.
Langfuse and LangSmith datasets store inputs and expected outputs, not the world. Adding a trace to a Langfuse dataset or a LangSmith dataset captures what went in and, optionally, what should come out. When you run an experiment, your application runs again, live. That is the right design for comparing a new prompt or model against a fixed set of inputs: you want the new model's real behaviour. It is the wrong design for a regression test of agent logic, because the tools return today's data instead of the data that triggered the bug. For the trade-offs between these two platforms, see our [LangSmith vs Langfuse vs Phoenix comparison](/articles/LangSmith-vs-Langfuse-vs-Phoenix-LLM-Agent-Observability).
Promptfoo is the strongest CI gate and the weakest trace source. Its GitHub Action and assertion language are mature, and it runs red-team suites no trace tool matches. But it never sees production. Every case is something a human typed, so the "trace to test" step is copy-paste. Our [Langfuse vs Promptfoo comparison](/articles/langfuse-vs-promptfoo) covers why many teams pair the two.
Tracely's real differentiator is the fixture bundle plus the fail-to-pass contract. The tracing and judging overlap heavily with Langfuse. Indeed, the project describes parts of its design as mirroring Langfuse's. What is new is treating promotion as a test-engineering operation with a validity check, not as a "save to dataset" button.
How do you build hermetic replay without adopting a new platform?
If you already run Langfuse or LangSmith and do not want another stack, you can build the missing stages yourself. The core is a single seam that every tool and model call goes through. This is a generic sketch of the pattern, not any vendor's API:
import json, hashlib
from contextvars import ContextVar
_mode = ContextVar("mode", default="live") # live | record | replay
_tape = ContextVar("tape", default=None) # list of recorded calls
def _key(kind, name, args):
blob = json.dumps([kind, name, args], sort_keys=True, default=str)
return hashlib.sha256(blob.encode()).hexdigest()[:16]
def seam(kind, name, fn, **args):
"""Every tool call and LLM call goes through here."""
mode, tape = _mode.get(), _tape.get()
if mode == "replay":
rec = tape.pop(0) # strict order, not lookup
if rec["key"] != _key(kind, name, args):
raise AssertionError(f"divergence at {kind}:{name}")
return rec["output"]
out = fn(**args)
if mode == "record":
tape.append({"key": _key(kind, name, args), "kind": kind,
"name": name, "output": out})
return out
Three design choices in that sketch decide whether the tests are useful.
1. Replay in strict order, and fail on divergence. If the fixed agent makes a different call than the recorded one, a lookup-by-key replay silently returns nothing or the wrong record. Raising on divergence turns "the agent took a different path" into an explicit result. Sometimes that is the fix working, and you re-record. Sometimes it is a new bug.
2. Decide which calls are fixtures and which are under test. In a test of planner logic, the model calls are the system under test, so replaying them proves only that the code around them is deterministic. A common split: replay tools and retrieval always, and replay the model only for the steps before the step that failed, then call the model live (or a pinned cheap model) for the step being tested. Tracely's offline mode replays everything. That is cheaper and deterministic, but it catches regressions in orchestration code, not in model behaviour.
3. Grade the outcome, not the transcript. Exact-match on model text breaks on every model upgrade. Grade structural facts ("called refund with the original order ID", "did not call cancel_subscription") and use an LLM judge only where structure cannot express the requirement.
Then write the fail-to-pass check as an actual test step: run the case against the parent commit and require a failure, run it against the fix and require a pass. Without that step you will end up with cases that were promoted from a trace but never reproduced the bug, and they pass forever.
When is trace-based regression testing the wrong tool?
It is not a replacement for every kind of evaluation.
- Model or prompt comparisons need live calls by definition. Use dataset experiments (Langfuse, LangSmith) or a Promptfoo matrix, and accept the token cost.
- Security and adversarial testing need inputs no user has sent yet. A trace suite only knows past failures. Red-team generation (Promptfoo plugins, or Tracely's adversarial scenarios) fills that gap.
- Agents whose tools have side effects you cannot record. Examples are a browser agent, a coding agent running shell commands, or anything that depends on wall-clock time. These need a sandbox or a simulated environment, not a tape. Replaying a recorded DOM is possible but brittle.
- Very low traffic. If you have a handful of production runs a week, hand-written cases are faster than running an ingest stack.
The approach pays off when you have steady production traffic, recurring failure modes, and orchestration code that changes often. That is the profile of most customer-facing tool-calling agents.
How should a team adopt trace-to-test in practice?
A sequence that keeps the investment proportional:
1. Standardise tracing on OpenTelemetry gen_ai.* attributes (see the OpenTelemetry GenAI semantic conventions). Every tool in this article can ingest them, so the choice of backend stays reversible.
2. Introduce the call seam in the agent code. Do this first, because it is the step every option needs and the one that takes longest.
3. Promote by hand for a month. Take the five worst production failures, record them, write structural graders, and run them in CI. You will learn which calls leak past the seam.
4. Then decide on a platform. If clustering and promotion are now the bottleneck, a tool like Tracely earns its operational cost (ClickHouse, Postgres, Redis and object storage). If your observability platform's datasets plus your own fixture layer are enough, stay there.
5. Treat the gate as production infrastructure. Pin action versions, alert on gate failures you did not expect, and review the suite whenever you upgrade a model. A failure across the whole suite after a model change is a signal, not noise.
FAQ
What is AI agent regression testing?
AI agent regression testing checks that a change to an agent's code, prompts or model does not bring back a failure that was already fixed. The strongest form builds each test from a real failing trace and replays it on every pull request.
What is hermetic replay for LLM agents?
Hermetic replay runs an agent test with recorded tool and model responses instead of live calls. The test needs no network access, no API keys and no token spend, and it gives the same result on every run, which makes it usable as a CI gate.
What is a fail-to-pass contract?
A fail-to-pass contract is a rule that a regression case must fail on the code that had the bug and pass on the code that fixes it. It proves the test actually reproduces the failure, instead of being a test that cannot fail.
Is Tracely a replacement for Langfuse or LangSmith?
Not yet for most teams. Tracely overlaps with their tracing and judging but is an early project with a small community. Its distinctive parts are fixture-bundled promotion, failure clustering and an offline CI gate, which you can also build on top of an existing platform.
Can Promptfoo test agents built from production traces?
Promptfoo can gate CI on agent outputs and run red-team suites, but it does not ingest production traces. You copy the trace's input into a test case by hand and write your own mocks for tool responses.
Do I need OpenTelemetry to do trace-based testing?
No, but it keeps your options open. Tracely, Langfuse, LangSmith and Phoenix all accept OpenTelemetry traces, so instrumenting once with gen_ai.* attributes lets you change backends without re-instrumenting the agent.
Sources
- Tracely: trace-native CI/CD for AI agents (GitHub)
- Langfuse datasets documentation
- LangSmith evaluation documentation
- Promptfoo GitHub Action integration
- OpenTelemetry semantic conventions for generative AI
- OpenInference instrumentation conventions (Arize)
---
Originally published at fp8.co. Subscribe for weekly AI engineering analysis at fp8.co.
I shipped a linter for checks that cannot fail. It had four of them.
> Where this landed, 9 October 2026. This post opens with 40%. That number is not
> wrong so much as it is not the quantity you would take it for, and I only found that out
> by being argued with in the comments. It has taken a week of measuring to say what it
> should have been, and the tool has changed more than the number has. This block replaces
> the two corrections that were stacked here, because three corrections in a row is not
> honesty, it is sediment.
>
> The sampling was measuring the wrong unit. The samples behind every figure were
> stratified with a cap of two findings per repository, to stop one repo dominating. But
> the tool emits one finding per (job, upstream) pair, and the false positives cluster in
> jobs with many dependencies: one reporting job produced eleven by itself. The cap deletes
> exactly that clustering. Simulated against a population I later labelled completely, the
> capped sampler is an unbiased estimator of the per-job rate and runs about 16 points
> low for the per-finding rate. So every percentage in this post understates, all in the
> same direction. @arhancanli called this in the thread before
> I measured it.
>
> What the numbers are now, all hand-labelled, no estimate of an estimate. Four
> detectors are two. D4 went on 0 defensible of 18 sampled, D2 on 0 true of 27 censused.
> Corpus output went 1986 findings to 1026.
>
> | | basis | false |
> |---|---|---|
> | D3, all 653 | complete census, every finding read | 36.1% [32.5, 39.9] |
> | D1 | gated on --repo, and 0 findings with it | see below |
>
> The severity tiers are gone, and that is the result I did not expect. Censusing all
> three strata gave HIGH 30.9%, LOW 31.6%, MEDIUM 52.8%. HIGH's interval overlaps the
> merged rest. The only real separation is MEDIUM being worse, and it came from the
> severity function short-circuiting on whether the workflow runs on pull_request before
> it looked at the finding at all. The axis measured the trigger, not correctness. It
> hid the better findings behind --all and shouted the worse ones, so collapsing it
> raised the default output from 238 findings to 653 and lowered the false rate a reader
> sees, from 44.1% to 36.1%.
>
> D1 is gated rather than removed, and the gate reports nothing. Its text says a
> skipped upstream leaves the dependent running without it; [GitHub documents that the
> dependent is skipped too](docs.github.com).
> 68% of its findings are that case. So it now says nothing
> unless --repo confirms the job is a required check. Then I ran that against live branch
> protection for all 373 findings: 10 resolved to REQUIRED, all in one repository, and all
> 10 vanish on a complete checkout where the tool can read the script that job calls. On
> 275 repositories, --repo has never reported a finding. It is kept because the gate is
> correct, not because it is productive, and that distinction is the whole post in one line.
>
> Three numbers I published and had to take back, every one caught by checking my own
> claim rather than by anyone complaining. The "65% false" I quoted for D1 came from a
> census covering 12% of the population it was quoted against. The "37.0%, 245 of 662" in
> my own 0.3.0 release notes added a stratum censused at 153 to a population taken
> after a suppression had already removed 9 of them; every commit since emits 653. And the
> 54.9% in the table that used to sit right here kept 3 findings labelled true from a
> sample taken before the insight that refutes them, which is holding both positions at
> once: I shipped a suppression on the grounds that cmd | grep -q X is not a defect
> while still counting three of those lines as defects. The arithmetic carries its own
> check, and I had not run it. The census was 68 true and 85 false of 153, so removing 9
> false findings has to leave true at 68, and 68 + 76 = 144. A correction is a new claim
> and needs verifying as hard as the original.
>
> The part of this post I would still defend is the part about taste: the corpus average
> hid the problem, and pointing the tool at two repositories I respected is what found it.
> The part I would not defend is any specific percentage in it. The working is in the
> comment thread, including the fixes measurement killed before they shipped. The latest
> was a filter that genuinely improved precision, 31% false down to 17%, and I deleted it
> anyway: it bought those 14 points by discarding two thirds of the true findings, and
> on that population it turned out to be a shape classifier wearing a semantic test.
Yesterday I published a tool called deadgate to PyPI. It reads GitHub Actions workflows and finds checks that cannot fail: a required gate satisfied by a skipped job, an exit status swallowed by a pipe, a fan-in job that never reads whether its upstream passed.
Then I pointed it at Arize-ai/openinference, 1.2k stars, 22 workflows. It reported 13 HIGH findings.
Twelve of them were false.
Not borderline. Not "technically true but low impact." Their CI was correct and my tool could not see it. The version doing the reporting had been on PyPI for about an hour.
What it could not see
GitHub's documented way to ask "did anything upstream fail" is the wildcard form:
if: always()
run: |
if [[ "${{ contains(needs.*.result, 'failure') }}" == "true" ]]; then exit 1; fi
My detector tested for that with needs\.[A-Za-z0-9_-]+\.(result|outputs), which cannot match *. So the one spelling the documentation recommends was the one spelling the tool was blind to, and every correctly written fan-in gate came back as a gate that cannot fail.
The finding's own message read "never reads needs.*.result" while failing to match that exact string.
I fixed it, shipped 0.1.1, and pointed the fixed version at astral-sh/ruff. Twenty-one HIGH findings. Eighteen false. Ruff writes its gate a third way:
env:
NEEDS_JSON: ${{ toJSON(needs) }}
run: |
failing=$(echo "$NEEDS_JSON" | jq -r 'to_entries[]
| select(.value.result != "success" and .value.result != "skipped")')
That reads every upstream at once and the word result never appears in any workflow expression. There is nothing for my pattern to match.
The fourth one was not a spelling at all. scikit-learn has a job that calls a reusable workflow, so it has uses: and no steps:, and it passes needs.check-sdist.result through job-level with:. That expression was one I already understood perfectly. It just lived somewhere I was not looking, because I scanned step-level with and not job-level with.
Four classes. The first two shipped to PyPI. I found the third and fourth only because I kept pointing the "fixed" version at new repositories instead of declaring victory.
The part I think is actually generalizable
Here is the number that matters, measured on a 275 repository, 4543 file corpus with both versions reading identical bytes:
| version | findings | HIGH |
| --- | --- | --- |
| before | 4562 | 1104 |
| after | 2917 | 658 |
Forty percent of HIGH findings were false. That sounds bad, and it undersells it, because 40% is an average and the errors were not spread evenly.
Only 15 of those 275 repositories use the wildcard idiom at all. The false positives piled onto the repositories that write their gates carefully. On openinference the false rate was 92%. On ruff, 86%. On a repository with no fan-in gate at all, it was zero, because there was nothing to misread.
My linter was least accurate on the best code.
If a tool is noisy on bad code, the noise and the signal arrive together and you read both. If it is noisy specifically on good code, the people who did the work correctly are the ones punished, and the obvious response is to stop running it. A corpus average cannot surface that, because the corpus is mostly ordinary repositories where the tool happens to be right.
I had the corpus. I ran it. It said 40%, and I would have shipped that as the headline. What actually found this was pointing the tool at two repositories whose CI I respected.
The mutation that survived
To stop reporting jobs whose failure a gate already catches, I walked the needs graph transitively: if a gate covers ci, and ci needs changes, then changes is covered too.
I mutation tested the fix, which for me means planting the defect each test claims to catch and checking that something goes red. One mutation survived: the one that deleted my transitive walk.
My first instinct was a missing test. It was not. The mutation survived because the walk was wrong: needs.*.result reports only direct needs, and a failed grandparent makes the parent skip, and skipped is not failure, so the gate never sees it. Walking transitively would have suppressed real findings.
A surviving mutant has two meanings and they are easy to confuse. Either a test is missing, or the code does nothing. If I had written a test to kill it, I would have pinned a bug in place and called it coverage.
The small one that is somehow the worst
While fixing all this I noticed five of my repositories advertised a stale test count in the README. I corrected them by hand and added a CI check so they could not drift again.
The check went red immediately. On a count I had just "corrected."
I had changed agent-graph's README from 42 to 58 and written a public commit message saying the previous commit overstated the count at 67. CI collects 67. My local venv was missing an optional extra, and tests/test_rag.py opens with pytest.importorskip, so nine tests vanished at collection with no error, no warning, and no mention of the file. pytest said "58 tests collected" and meant it.
The original number was right. I corrected a correct claim, publicly, using a measurement from an environment that was not the real one.
The check I shipped an hour earlier had the same blind spot and agreed with 58. It now fails when any test file on disk contributes zero collected tests, because that is the only evidence an environment is incomplete.
What I am not showing
The sweeps above are driven by my own harness, and the agent scaffolding, gate implementations and prompts behind it are not in the repository and are not going in it. deadgate itself is small and MIT and you can read all of it. The thing that pointed it at 275 repositories and classified the survivors is the part I am keeping.
The question I actually have
I do not know how to test a linter for this class of error without a sample of good code, and "good code" is not a property I can detect automatically. A corpus gives you volume and volume gives you the average, which is exactly the statistic that hid this.
The only thing that worked was taste: picking two repositories I already believed were well engineered and reading the output by hand. That does not scale and it is not reproducible, and I would like to be talked out of it.
If you maintain a static analysis tool, how do you measure your false positive rate on the code that is already correct? I would rather hear that I am missing something obvious than keep doing it by hand.
deadgate is at pypi.org and egnaro9/deadgate, MIT, no model in the loop.
I work on AI evaluation and testing, open to relocation above $120K, portfolio at erikhill.dev.
4 reactions · 22 comments
Mentions
AI Agent Regression Testing: Production Traces to CI AI Agent Regression Testing: How to Turn Production Traces into CI Gates TL;DR: Trace-based regression testing turns a failing production agent trace into a CI test case. The trace's recorded tool and model outputs become fixtures, so the test replays offline and deterministically on every pull request. Tracely is an MIT-licensed open-source project that packages this loop: OTLP ingest, graders, failure clustering, promotion of a trace to a case, and a GitHub Action gate. Langfuse, LangSmith and Promptfoo cover parts of the same loop. None of them removes the main engineering cost, which is routing every tool and model call through a seam you can record and replay. Key Takeaways - An agent regression test is a test built from a real failure. A trace that went wrong in production becomes a case your CI must pass forever after. - Hermetic replay is what makes the test usable in CI. Recorded tool and LLM responses stand in for live calls, so the suite needs no API keys, costs nothing in tokens, and gives the same result every time. - A fail-to-pass contract is the check that the test is trustworthy. The case must fail on the old code and pass on the fix, or the test proves nothing. - Tracely (about 1.2k GitHub stars, MIT, Python/FastAPI with ClickHouse) bundles ingest, grading, clustering, promotion and a tracely-gate GitHub Action in one self-hosted stack. - Langfuse and LangSmith let you add traces to datasets and run experiments over them. Promptfoo runs assertion suites in CI. All three re-call the model live unless you add your own fixture layer. - Replay only works if the agent sends tool and model calls through a recordable seam. Plan that refactor before you pick a tool. Why do AI agents need regression tests built from traces? Agent failures rarely show up in the test inputs a team writes on day one. They come from real conversations: a user phrases a refund request the planner did not expect, a tool returns an empty list instead of an error, or a model upgrade changes how often the agent calls a search tool. By the time someone sees the failure, it lives in a trace (a nested record of every prompt, model response, tool call and result in one run). Teams usually handle this in one of two ways, and both leak: 1. Hand-written eval sets. Someone reads the bad trace, writes a new test input and an expected output, and adds it to a YAML file or dataset. The new input is a paraphrase of what happened, not a copy. The tool responses that caused the failure are usually lost, and the test then calls live tools that now return different data. 2. Dashboards with no gate. The observability platform records the failure, an online evaluator scores it low, and the score sits on a chart. Nothing stops the next pull request from bringing the same failure back. Trace-based regression testing closes that gap. The trace is the test: its exact inputs, the exact tool outputs the agent saw, and the model responses in recorded order. A grader decides pass or fail. The suite runs on every pull request and blocks merges that bring a known failure back. This is the same discipline as "write a failing test before you fix the bug" in ordinary software. The difference is that agents are non-deterministic and depend heavily on external services, so the test needs recorded fixtures before it can be repeated. What does a trace-to-test pipeline look like? The loop has five stages. Every implementation, whether bought or built, needs all five. | Stage | What happens | What breaks if it is missing | |---|---|---| | Observe | Every production run emits a trace (OpenTelemetry gen_ai.* or OpenInference attributes) indexed by agent, conversation, turn and step | You cannot reproduce a failure you never recorded | | Grade | Structural checks (schema, tool-call order, no model needed) and LLM judges score runs online | Failures are found by users, not by the system | | Triage | Failing runs are clustered by structural signature and semantic similarity | 31 broken runs become 31 tickets instead of one issue with a count | | Promote | A representative failing trace becomes a hermetic case with its fixtures recorded | The "test" re-calls live services and drifts | | Gate | CI replays the suite against the change and fails the build on regressions | The fix ships, and the next refactor quietly reverts it | The Promote stage is where most homegrown setups stop. Copying a trace's input into a dataset is easy. Copying the world the agent saw (tool results, retrieval hits, intermediate model outputs) and replaying it in order is the actual work. How does Tracely implement trace-native CI/CD? Tracely is an open-source project by Julien Wuthrich that describes itself as "trace-native CI/CD for AI agents." It is MIT-licensed, has about 1.2k stars at the time of writing, and is still early (a few dozen commits). Its stack is a Python backend (FastAPI, Celery, uv) and a Next.js 15 UI. It stores traces in ClickHouse, the case registry in Postgres with pgvector, the queue in Redis and blobs in MinIO/S3. A local demo starts with docker compose --profile demo up -d --build --wait. Ingest: OTLP first, SDK second Any OpenTelemetry exporter can post to POST {endpoint}/v1/traces with a Bearer ingest key. Tracely reads both gen_ai.* semantic-convention attributes and OpenInference attributes, so existing instrumentation keeps working. The Python SDK (tracelyai-sdk) adds auto-instrumentation for OpenAI, Anthropic, Google, Mistral and LangChain: import tracely_sdk as tracely tracely.init( endpoint="http://localhost:8000", api_key="tracely_dev_key", service_name="support-agent", env="prod", # prod | staging | ci | dev — the gating axis instrument="auto", ) with tracely.trace(agent="support-agent", conversation="conv-1", user="u_42"): client.chat.completions.create( model="gpt-4o", messages=[{"role": "user", "content": "Where is order ORD-4471?"}], ) The env field matters more than it looks. It separates production traces (the source of failures) from CI traces (the ones the gate grades), so the same pipeline can serve both. Grade and cluster Evaluators come in two kinds. Structural checks need no model. LLM judges run at conversation, run or span level and can output a score, boolean, number, text or JSON-schema result. Failing runs are then clustered "structurally and semantically" with a structural signature plus embedding similarity. In the project's words, "31 broken runs become one issue with a count." Promote with a fail-to-pass contract This is the most important idea in the project. Promoting a failing trace bundles "recorded input, tool and LLM outputs" as fixtures. The case is trusted only if it "must fail on the old code and pass on the fix." A regression test that passes on the buggy code is a test that cannot fail, which is the most common way evaluation suites lie to their owners. The contract makes that check part of the workflow instead of leaving it to discipline. Multi-turn behaviour uses scenarios instead of single cases: either a scripted conversation, or an adversarial goal that a red-team model improvises against. Gate in GitHub Actions Replay runs offline against the recorded fixtures, with no API keys and no model spend. tracely gate exits non-zero, posts a commit status and upserts a PR comment. The bundled action has two modes. Tracely can call your agent through an entrypoint (tracely replay planner --entrypoint my_pkg.agent:run, or --cmd "node run.js" for non-Python agents), or it can grade the CI traces your own test job emitted: - uses: jwuthri-cloud/Tracely-ai/.github/actions/tracely-gate@master with: mode: gate agent: planner api: https://tracely.your-co.dev key: ${{ secrets.TRACELY_KEY }} Pin that @master reference to a commit SHA before you rely on it. A gate that can change underneath you is not a gate. The constraint you cannot skip The README states it plainly: hermetic replay "requires your agent to route tool/model calls through the SDK's call_tool / call_llm seam." Auto-instrumentation is enough to record a trace. It is not enough to replay one, because replay means intercepting the call and returning the recorded value instead of making it. If your agent calls requests.get from inside a tool, or a framework makes model calls you cannot reach, those calls go live during replay. Other caveats from the project itself: judges and failure analysis degrade without an LLM key, the default AUTH_MODE=dev with tracely_dev_key is open access and only suitable for local use, and the Celery worker does not hot-reload. How do Langfuse, LangSmith and Promptfoo handle the same loop? None of the established tools is called "trace-native CI," but each covers some of the stages. The question is which stages you get for free and which you build. | Stage | Tracely | Langfuse | LangSmith | Promptfoo | |---|---|---|---|---| | Production tracing | Yes (OTLP + SDK) | Yes, core product | Yes, core product | No (test runs only) | | Online grading | Structural + LLM judges | Online evaluators on sampled traffic | Online evaluators | No | | Failure clustering | Built in | Manual filtering and annotation | Manual filtering and annotation | No | | Trace → test | Promote to hermetic case | Add trace to dataset | Add run to dataset | Copy input into YAML test case | | Recorded tool fixtures | Yes, bundled on promote | Inputs/expected outputs only | Inputs/reference outputs only | No, you write mocks | | CI gate | tracely-gate action, PR comment | Experiments via SDK in your CI job | Evaluations via SDK in your CI job | promptfoo eval + GitHub Action | | Offline / no model spend | Yes, by design | No, experiments call the model | No, evaluations call the model | No, unless you mock providers | | Maturity | Early, small community | Mature, MIT core | Mature, commercial | Mature, MIT | A few clarifications on that table. Langfuse and LangSmith datasets store inputs and expected outputs, not the world. Adding a trace to a Langfuse dataset or a LangSmith dataset captures what went in and, optionally, what should come out. When you run an experiment, your application runs again, live. That is the right design for comparing a new prompt or model against a fixed set of inputs: you want the new model's real behaviour. It is the wrong design for a regression test of agent logic, because the tools return today's data instead of the data that triggered the bug. For the trade-offs between these two platforms, see our [LangSmith vs Langfuse vs Phoenix comparison](/articles/LangSmith-vs-Langfuse-vs-Phoenix-LLM-Agent-Observability). Promptfoo is the strongest CI gate and the weakest trace source. Its GitHub Action and assertion language are mature, and it runs red-team suites no trace tool matches. But it never sees production. Every case is something a human typed, so the "trace to test" step is copy-paste. Our [Langfuse vs Promptfoo comparison](/articles/langfuse-vs-promptfoo) covers why many teams pair the two. Tracely's real differentiator is the fixture bundle plus the fail-to-pass contract. The tracing and judging overlap heavily with Langfuse. Indeed, the project describes parts of its design as mirroring Langfuse's. What is new is treating promotion as a test-engineering operation with a validity check, not as a "save to dataset" button. How do you build hermetic replay without adopting a new platform? If you already run Langfuse or LangSmith and do not want another stack, you can build the missing stages yourself. The core is a single seam that every tool and model call goes through. This is a generic sketch of the pattern, not any vendor's API: import json, hashlib from contextvars import ContextVar _mode = ContextVar("mode", default="live") # live | record | replay _tape = ContextVar("tape", default=None) # list of recorded calls def _key(kind, name, args): blob = json.dumps([kind, name, args], sort_keys=True, default=str) return hashlib.sha256(blob.encode()).hexdigest()[:16] def seam(kind, name, fn, **args): """Every tool call and LLM call goes through here.""" mode, tape = _mode.get(), _tape.get() if mode == "replay": rec = tape.pop(0) # strict order, not lookup if rec["key"] != _key(kind, name, args): raise AssertionError(f"divergence at {kind}:{name}") return rec["output"] out = fn(**args) if mode == "record": tape.append({"key": _key(kind, name, args), "kind": kind, "name": name, "output": out}) return out Three design choices in that sketch decide whether the tests are useful. 1. Replay in strict order, and fail on divergence. If the fixed agent makes a different call than the recorded one, a lookup-by-key replay silently returns nothing or the wrong record. Raising on divergence turns "the agent took a different path" into an explicit result. Sometimes that is the fix working, and you re-record. Sometimes it is a new bug. 2. Decide which calls are fixtures and which are under test. In a test of planner logic, the model calls are the system under test, so replaying them proves only that the code around them is deterministic. A common split: replay tools and retrieval always, and replay the model only for the steps before the step that failed, then call the model live (or a pinned cheap model) for the step being tested. Tracely's offline mode replays everything. That is cheaper and deterministic, but it catches regressions in orchestration code, not in model behaviour. 3. Grade the outcome, not the transcript. Exact-match on model text breaks on every model upgrade. Grade structural facts ("called refund with the original order ID", "did not call cancel_subscription") and use an LLM judge only where structure cannot express the requirement. Then write the fail-to-pass check as an actual test step: run the case against the parent commit and require a failure, run it against the fix and require a pass. Without that step you will end up with cases that were promoted from a trace but never reproduced the bug, and they pass forever. When is trace-based regression testing the wrong tool? It is not a replacement for every kind of evaluation. - Model or prompt comparisons need live calls by definition. Use dataset experiments (Langfuse, LangSmith) or a Promptfoo matrix, and accept the token cost. - Security and adversarial testing need inputs no user has sent yet. A trace suite only knows past failures. Red-team generation (Promptfoo plugins, or Tracely's adversarial scenarios) fills that gap. - Agents whose tools have side effects you cannot record. Examples are a browser agent, a coding agent running shell commands, or anything that depends on wall-clock time. These need a sandbox or a simulated environment, not a tape. Replaying a recorded DOM is possible but brittle. - Very low traffic. If you have a handful of production runs a week, hand-written cases are faster than running an ingest stack. The approach pays off when you have steady production traffic, recurring failure modes, and orchestration code that changes often. That is the profile of most customer-facing tool-calling agents. How should a team adopt trace-to-test in practice? A sequence that keeps the investment proportional: 1. Standardise tracing on OpenTelemetry gen_ai.* attributes (see the OpenTelemetry GenAI semantic conventions). Every tool in this article can ingest them, so the choice of backend stays reversible. 2. Introduce the call seam in the agent code. Do this first, because it is the step every option needs and the one that takes longest. 3. Promote by hand for a month. Take the five worst production failures, record them, write structural graders, and run them in CI. You will learn which calls leak past the seam. 4. Then decide on a platform. If clustering and promotion are now the bottleneck, a tool like Tracely earns its operational cost (ClickHouse, Postgres, Redis and object storage). If your observability platform's datasets plus your own fixture layer are enough, stay there. 5. Treat the gate as production infrastructure. Pin action versions, alert on gate failures you did not expect, and review the suite whenever you upgrade a model. A failure across the whole suite after a model change is a signal, not noise. FAQ What is AI agent regression testing? AI agent regression testing checks that a change to an agent's code, prompts or model does not bring back a failure that was already fixed. The strongest form builds each test from a real failing trace and replays it on every pull request. What is hermetic replay for LLM agents? Hermetic replay runs an agent test with recorded tool and model responses instead of live calls. The test needs no network access, no API keys and no token spend, and it gives the same result on every run, which makes it usable as a CI gate. What is a fail-to-pass contract? A fail-to-pass contract is a rule that a regression case must fail on the code that had the bug and pass on the code that fixes it. It proves the test actually reproduces the failure, instead of being a test that cannot fail. Is Tracely a replacement for Langfuse or LangSmith? Not yet for most teams. Tracely overlaps with their tracing and judging but is an early project with a small community. Its distinctive parts are fixture-bundled promotion, failure clustering and an offline CI gate, which you can also build on top of an existing platform. Can Promptfoo test agents built from production traces? Promptfoo can gate CI on agent outputs and run red-team suites, but it does not ingest production traces. You copy the trace's input into a test case by hand and write your own mocks for tool responses. Do I need OpenTelemetry to do trace-based testing? No, but it keeps your options open. Tracely, Langfuse, LangSmith and Phoenix all accept OpenTelemetry traces, so instrumenting once with gen_ai.* attributes lets you change backends without re-instrumenting the agent. Sources - Tracely: trace-native CI/CD for AI agents (GitHub) - Langfuse datasets documentation - LangSmith evaluation documentation - Promptfoo GitHub Action integration - OpenTelemetry semantic conventions for generative AI - OpenInference instrumentation conventions (Arize) --- Originally published at fp8.co. Subscribe for weekly AI engineering analysis at fp8.co.
Updates on Trueup, building with @neatlogs Arize's head of open source reviewed the fix Trueup sent them: "the right goal." The bug: an AI agent run crashes, the record says OK. We applied his changes. 300 tests pass. openinference#4024 #neatHack @EntireHQ @arithecfo t.co
If you use OpenInference with Anthropic streaming, your traces are a little more honest because of it. openinference#3902 Here's to the next one.
I shipped a linter for checks that cannot fail. It had four of them. > Where this landed, 9 October 2026. This post opens with 40%. That number is not > wrong so much as it is not the quantity you would take it for, and I only found that out > by being argued with in the comments. It has taken a week of measuring to say what it > should have been, and the tool has changed more than the number has. This block replaces > the two corrections that were stacked here, because three corrections in a row is not > honesty, it is sediment. > > The sampling was measuring the wrong unit. The samples behind every figure were > stratified with a cap of two findings per repository, to stop one repo dominating. But > the tool emits one finding per (job, upstream) pair, and the false positives cluster in > jobs with many dependencies: one reporting job produced eleven by itself. The cap deletes > exactly that clustering. Simulated against a population I later labelled completely, the > capped sampler is an unbiased estimator of the per-job rate and runs about 16 points > low for the per-finding rate. So every percentage in this post understates, all in the > same direction. @arhancanli called this in the thread before > I measured it. > > What the numbers are now, all hand-labelled, no estimate of an estimate. Four > detectors are two. D4 went on 0 defensible of 18 sampled, D2 on 0 true of 27 censused. > Corpus output went 1986 findings to 1026. > > | | basis | false | > |---|---|---| > | D3, all 653 | complete census, every finding read | 36.1% [32.5, 39.9] | > | D1 | gated on --repo, and 0 findings with it | see below | > > The severity tiers are gone, and that is the result I did not expect. Censusing all > three strata gave HIGH 30.9%, LOW 31.6%, MEDIUM 52.8%. HIGH's interval overlaps the > merged rest. The only real separation is MEDIUM being worse, and it came from the > severity function short-circuiting on whether the workflow runs on pull_request before > it looked at the finding at all. The axis measured the trigger, not correctness. It > hid the better findings behind --all and shouted the worse ones, so collapsing it > raised the default output from 238 findings to 653 and lowered the false rate a reader > sees, from 44.1% to 36.1%. > > D1 is gated rather than removed, and the gate reports nothing. Its text says a > skipped upstream leaves the dependent running without it; [GitHub documents that the > dependent is skipped too](docs.github.com). > 68% of its findings are that case. So it now says nothing > unless --repo confirms the job is a required check. Then I ran that against live branch > protection for all 373 findings: 10 resolved to REQUIRED, all in one repository, and all > 10 vanish on a complete checkout where the tool can read the script that job calls. On > 275 repositories, --repo has never reported a finding. It is kept because the gate is > correct, not because it is productive, and that distinction is the whole post in one line. > > Three numbers I published and had to take back, every one caught by checking my own > claim rather than by anyone complaining. The "65% false" I quoted for D1 came from a > census covering 12% of the population it was quoted against. The "37.0%, 245 of 662" in > my own 0.3.0 release notes added a stratum censused at 153 to a population taken > after a suppression had already removed 9 of them; every commit since emits 653. And the > 54.9% in the table that used to sit right here kept 3 findings labelled true from a > sample taken before the insight that refutes them, which is holding both positions at > once: I shipped a suppression on the grounds that cmd | grep -q X is not a defect > while still counting three of those lines as defects. The arithmetic carries its own > check, and I had not run it. The census was 68 true and 85 false of 153, so removing 9 > false findings has to leave true at 68, and 68 + 76 = 144. A correction is a new claim > and needs verifying as hard as the original. > > The part of this post I would still defend is the part about taste: the corpus average > hid the problem, and pointing the tool at two repositories I respected is what found it. > The part I would not defend is any specific percentage in it. The working is in the > comment thread, including the fixes measurement killed before they shipped. The latest > was a filter that genuinely improved precision, 31% false down to 17%, and I deleted it > anyway: it bought those 14 points by discarding two thirds of the true findings, and > on that population it turned out to be a shape classifier wearing a semantic test. Yesterday I published a tool called deadgate to PyPI. It reads GitHub Actions workflows and finds checks that cannot fail: a required gate satisfied by a skipped job, an exit status swallowed by a pipe, a fan-in job that never reads whether its upstream passed. Then I pointed it at Arize-ai/openinference, 1.2k stars, 22 workflows. It reported 13 HIGH findings. Twelve of them were false. Not borderline. Not "technically true but low impact." Their CI was correct and my tool could not see it. The version doing the reporting had been on PyPI for about an hour. What it could not see GitHub's documented way to ask "did anything upstream fail" is the wildcard form: if: always() run: | if [[ "${{ contains(needs.*.result, 'failure') }}" == "true" ]]; then exit 1; fi My detector tested for that with needs\.[A-Za-z0-9_-]+\.(result|outputs), which cannot match *. So the one spelling the documentation recommends was the one spelling the tool was blind to, and every correctly written fan-in gate came back as a gate that cannot fail. The finding's own message read "never reads needs.*.result" while failing to match that exact string. I fixed it, shipped 0.1.1, and pointed the fixed version at astral-sh/ruff. Twenty-one HIGH findings. Eighteen false. Ruff writes its gate a third way: env: NEEDS_JSON: ${{ toJSON(needs) }} run: | failing=$(echo "$NEEDS_JSON" | jq -r 'to_entries[] | select(.value.result != "success" and .value.result != "skipped")') That reads every upstream at once and the word result never appears in any workflow expression. There is nothing for my pattern to match. The fourth one was not a spelling at all. scikit-learn has a job that calls a reusable workflow, so it has uses: and no steps:, and it passes needs.check-sdist.result through job-level with:. That expression was one I already understood perfectly. It just lived somewhere I was not looking, because I scanned step-level with and not job-level with. Four classes. The first two shipped to PyPI. I found the third and fourth only because I kept pointing the "fixed" version at new repositories instead of declaring victory. The part I think is actually generalizable Here is the number that matters, measured on a 275 repository, 4543 file corpus with both versions reading identical bytes: | version | findings | HIGH | | --- | --- | --- | | before | 4562 | 1104 | | after | 2917 | 658 | Forty percent of HIGH findings were false. That sounds bad, and it undersells it, because 40% is an average and the errors were not spread evenly. Only 15 of those 275 repositories use the wildcard idiom at all. The false positives piled onto the repositories that write their gates carefully. On openinference the false rate was 92%. On ruff, 86%. On a repository with no fan-in gate at all, it was zero, because there was nothing to misread. My linter was least accurate on the best code. If a tool is noisy on bad code, the noise and the signal arrive together and you read both. If it is noisy specifically on good code, the people who did the work correctly are the ones punished, and the obvious response is to stop running it. A corpus average cannot surface that, because the corpus is mostly ordinary repositories where the tool happens to be right. I had the corpus. I ran it. It said 40%, and I would have shipped that as the headline. What actually found this was pointing the tool at two repositories whose CI I respected. The mutation that survived To stop reporting jobs whose failure a gate already catches, I walked the needs graph transitively: if a gate covers ci, and ci needs changes, then changes is covered too. I mutation tested the fix, which for me means planting the defect each test claims to catch and checking that something goes red. One mutation survived: the one that deleted my transitive walk. My first instinct was a missing test. It was not. The mutation survived because the walk was wrong: needs.*.result reports only direct needs, and a failed grandparent makes the parent skip, and skipped is not failure, so the gate never sees it. Walking transitively would have suppressed real findings. A surviving mutant has two meanings and they are easy to confuse. Either a test is missing, or the code does nothing. If I had written a test to kill it, I would have pinned a bug in place and called it coverage. The small one that is somehow the worst While fixing all this I noticed five of my repositories advertised a stale test count in the README. I corrected them by hand and added a CI check so they could not drift again. The check went red immediately. On a count I had just "corrected." I had changed agent-graph's README from 42 to 58 and written a public commit message saying the previous commit overstated the count at 67. CI collects 67. My local venv was missing an optional extra, and tests/test_rag.py opens with pytest.importorskip, so nine tests vanished at collection with no error, no warning, and no mention of the file. pytest said "58 tests collected" and meant it. The original number was right. I corrected a correct claim, publicly, using a measurement from an environment that was not the real one. The check I shipped an hour earlier had the same blind spot and agreed with 58. It now fails when any test file on disk contributes zero collected tests, because that is the only evidence an environment is incomplete. What I am not showing The sweeps above are driven by my own harness, and the agent scaffolding, gate implementations and prompts behind it are not in the repository and are not going in it. deadgate itself is small and MIT and you can read all of it. The thing that pointed it at 275 repositories and classified the survivors is the part I am keeping. The question I actually have I do not know how to test a linter for this class of error without a sample of good code, and "good code" is not a property I can detect automatically. A corpus gives you volume and volume gives you the average, which is exactly the statistic that hid this. The only thing that worked was taste: picking two repositories I already believed were well engineered and reading the output by hand. That does not scale and it is not reproducible, and I would like to be talked out of it. If you maintain a static analysis tool, how do you measure your false positive rate on the code that is already correct? I would rather hear that I am missing something obvious than keep doing it by hand. deadgate is at pypi.org and egnaro9/deadgate, MIT, no model in the loop. I work on AI evaluation and testing, open to relocation above $120K, portfolio at erikhill.dev.