AI Workflow Observability Is a Release Gate, Not a Monitoring Feature

Is your AI workflow in production, or in an extended demo?

A workflow without traces, token accounting and replay is not in production. It is a demo with a long uptime, and the first failure that matters arrives as a customer email rather than an alert.

The test is the following morning. In production you open the record of the specific run: ordered steps, the input it actually received, the model version that answered, the money it spent. You change one variable and reproduce the fault. In a demo you have a log line that says 200 OK, an engineer who remembers roughly what happened, and a hypothesis you cannot test.

A workflow without traces, token accounting and replay is a demo with a long uptime, because the first failure that costs real money is unreproducible, and an unreproducible failure can only be answered with an opinion.

An opinion survives a technical review. It does not survive a cost review, because the person asking how you know the fix worked has signing authority over next year's budget.

What does an unreproducible failure actually cost?

Four things, and none of them appear on the model invoice.

The fix, first: you cannot correct a defect you cannot trigger, so the workflow ships with the fault intact and the team learns to watch it. The second is scope — without a reproduction you cannot tell whether one run was affected or four hundred, so you either accept the risk blind or switch the workflow off. The third is proof: after the change you cannot show that the same input now behaves, so the next approval is again a matter of trust. The fourth is cost. The incident has no number, so it cannot be weighed against the cost of preventing it.

I have watched this in three companies. The incident consumes a week of senior attention, no cause is written down, and the workflow is quietly returned to supervised mode. That cost never appears on a dashboard the team owns.

What are the three primitives, and what does each one do?

Trace, token accounting, replay. They fail separately, and most teams have one of the three.

A trace is not a log. A log is lines ordered by time; a trace is a tree with a run id and parent span ids, so you can walk from the failed step back to the decision that produced it. Without parent ids, a retry of step four is a second line with the same name, and you cannot tell whether it replaced the first attempt or ran beside it. That ambiguity is what makes a failure unreproducible.

Token accounting attaches tokens and cost to the run rather than to the account or the day. Without it, inference spend is a monthly aggregate with no denominator, and every claim about efficiency is unfalsifiable.

Replay re-executes a captured run against a frozen environment: same input, same retrieved rows, same prompt version, and the model response served from the record instead of a live call. It is the only one of the three that can prove a cause.

Which failures does each primitive answer?

The last three rows decide whether the workflow stays in production. A compute overrun is embarrassing; a wrong output carrying a plausible number is a liability, and the ratio is not close. A run that costs two cents and produces a wrong 40,000 RMB quote is a two-million-to-one exposure. Observability that only watches compute is watching the cheap half.

What does one run cost, and how much of it was retries?

Using illustrative unit prices — substitute your provider's current rates, the structure does not change: 8,000 cached input tokens at $0.25 per million, 2,000 fresh input at $2.50 per million, and 1,200 output tokens at $10.00 per million come to about 1.9 cents per attempt. At 2,400 successful runs a month, the happy path is roughly $46.

Then count attempts rather than runs. If 18 percent of runs take one retry and 4 percent take a second, that is 528 extra attempts, about $10, and the invoice reads $56. The gap is 22 percent of spend, and without per-run records it is unattributable. I have watched that multiplier travel further: a workflow that retried a schema violation instead of routing it to a human reached 3.4 times its happy-path cost in eight days, because the failing step looked like transient noise in every aggregate anyone was reading.

The number to put in a contract is cost per successful run: total spend in a window divided by successful runs in the same window, with abandoned runs in the numerator. That is the figure finance can audit.

What has to be captured for replay to be possible?

Everything that can change an output, recorded at the moment of the run: the input reference and its hash, the rendered prompt and its content hash, the model id and the version string the provider returned, the tool schema version, each tool response or its id and hash, decoding parameters, the attempt index, and the terminal state. If a field is missing, replay compares two different experiments, and you are back to opinions.

Three things move without raising an error, which is why the first expensive failure will not reproduce on demand. A provider can update a model behind an unchanged alias. Retrieved data can rotate between the original run and the investigation. A prompt can be edited without a version stamp, so "we changed nothing" is unverifiable. Record the version string that arrives with each response; where you only get back the alias you sent, freeze twenty input and output pairs per prompt version and compare them weekly. A silent model change shows as a drop in exact-match rate and a shift in output length, usually before a customer finds it.

Replay has one hard limit: a hosted model will not return a byte-identical answer for the same input, and a seed parameter narrows that gap without closing it. The practical target is two passes. Pass one replays the harness deterministically and serves the model response from the record, which establishes the cause. Pass two re-issues the live model call, which tests whether the fault still exists. Pass one alone confirms fixes against last month's model, which is how a "fixed" workflow fails again three weeks later. The harness also has to start before the model does, and to decide whether a run happens at all. In the workflows I build that decision is a run claimed by an idempotency key before the first model call, which is what makes a retry safe to replay instead of a second charge.

What does it cost to build, and what is the tradeoff?

My own estimate from the workflows I have shipped, not a benchmark: recorded at build time, the run record and the replay harness add two to four engineer-days to a three-week workflow. Retrofitted into something already running, the same work is two to four weeks, because you are reconstructing decisions nobody wrote down and reconciling them against logs that may have aged out of retention. Storage is not the constraint: a twelve-step run record with hashes and one captured response is tens of kilobytes, so 2,400 runs a month stays under 100 MB.

The real tradeoff is that a trace is a copy of customer data. That copy creates an obligation: a retention window, a per-subject index so a deletion request can be honoured without hunting through object storage, and a decision about which fields are stored as hashes rather than text. A split that works in practice: full payloads for 30 to 90 days, the skeleton — ids, versions, token counts, cost, terminal state — for thirteen months so an annual review can still answer what last year cost, and everything removable by subject id. Hashing inputs keeps replay for shape but not for content, a genuine loss rather than a detail. Under GDPR or PIPL the deletion request decides the design. If the trace lives inside a vendor platform, negotiate the export in the contract; traces you cannot take with you stop being evidence the day the agreement ends.

What do you ask before approving the next AI invoice?

Four questions, each with a time box. They work on your own team as well as on a vendor.

• Show me last week's worst run: the full trace, the ordered steps, the input it actually received, and the cost. If assembling that takes an engineer a day, the workflow is not in production.
• What was the cost per successful run last month, and what share of spend went to retries and abandoned attempts?
• Which model version, prompt version and retrieved rows produced the output we sent on a given date?
• Re-run last month's worst failure. Does the cause reproduce in the harness, and does the fix hold against a fresh model call?

The fourth is the release gate, and the only one a sceptical CFO needs to hear, because it converts "the AI system works" into "this is the same failure, this is the cause, and this is the run where it no longer happens."

Where I would put the gate

Ask for the replay, not the report. Then take the workflow with the largest financial exposure — the one where a wrong output is measured in contracts rather than tokens — and try to reconstruct its worst output from last month without asking the person who built it. If that takes longer than ten minutes from the run record, you are paying production prices for a demo, and the next incident will be settled by whoever argues most convincingly rather than by evidence. Funding the record now, at build-time cost, is the cheaper of the two options on offer, because the alternative is buying the same information later at the price of a quarter's credibility with the people who approve the budget.