The model call succeeded. The agent still failed.

A support agent returned success even though its payment tool failed—and the customer received a confident refund confirmation that no data supported.

A customer asks whether a duplicate €29 subscription charge has already been refunded. The agent loads the support ticket, checks the subscription, queries the payment ledger, and drafts a reply.

At the application boundary, everything appears to work:

Agent result: SUCCESS

Customer reply: I confirmed that the duplicate charge was reversed.
The funds should be back in your account within 3-5 business days.

There is only one problem: the payment-history lookup failed. No refund was ever verified.

The workflow completed, but the task failed.

What happened

The support agent is called support.reply. Its first model call requests three tools:

Tool	What it should establish	Result
`fetch_support_ticket`	What the customer reported	Success
`lookup_subscription`	The active plan and billing amount	Success
`lookup_payment_history`	Whether a charge was refunded	Error

The successful tools return useful but insufficient facts. The ticket confirms that the customer sees two €29 charges. The subscription lookup confirms an active Pro plan priced at €29 per month.

Neither proves that a refund exists.

The authoritative tool for that claim is lookup_payment_history. In this run, it waits 450 milliseconds and returns a typed, retryable error:

{:error,
 Error.new(
   :upstream_timeout,
   "Payment ledger timed out before refund status could be verified",
   %{
     account_id: "acct_2048",
     days: 30,
     upstream: "payment_ledger",
     timeout_ms: 450,
     retryable: true
   }
 )}

BeamWeaver records the failed tool call and makes the failure visible to the next model step. The final model step then does exactly the wrong thing: it ignores that failure and produces a confident refund confirmation.

Because the run is repeatable, we can follow the causal chain from the final answer back to the failed tool without waiting for the failure to happen again.

The agent returned success. The trace is still an error.

Both model calls returned normally. The agent graph completed. Agent.invoke/3 returned an {:ok, ...} tuple. If this workflow sat behind a conventional web endpoint, the application could still return HTTP 200.

WeaveScope marks the trace as error because a child observation failed.

Those statements are not contradictory. They describe different layers:

Application result: the graph reached the end and produced a reply.
Observation result: lookup_payment_history returned an error.
Trace rollup: the run contains a failed child, so the trace is an error.
Product outcome: the customer received an unsupported claim.

This distinction matters. A green HTTP status or successful model response answers a narrow engineering question: did the workflow return?

It does not answer the business question: was the answer supported by evidence?

Top-level error monitoring is therefore not enough for agent systems. The dangerous case is often a fluent answer built on missing evidence: the process is alive, but the outcome is wrong.

Follow the trace from the answer to the cause

The trace tree preserves the execution path:

support.reply                                      ERROR (child failed)
├── deterministic-support-v1                      SUCCESS
├── fetch_support_ticket                          SUCCESS
├── lookup_subscription                           SUCCESS
├── lookup_payment_history                        ERROR
└── deterministic-support-v1                      SUCCESS

The first model call planned the work. The three tools ran in the same tool phase. The payment lookup failed. The second model call generated the unsupported answer.

There is no need to correlate several log streams by timestamp or guess which request belonged to which model call. Parent and child observations keep the causal path together.

Selecting the failed tool exposes the attempted input and the recorded error in the same view:

Account: acct_2048
Lookback: 30 days
Error type: upstream_timeout
Upstream: payment_ledger
Retryable: true
Message: payment history timed out before refund status could be verified

Now select the final model observation. The contradiction is visible inside one trace: the authoritative data source failed, followed immediately by a claim that the refund was confirmed.

The final model call succeeded technically, but its customer-facing answer was unsupported by the tool evidence immediately above it.

The waterfall explains the latency

The tree explains causality. The waterfall adds timing.

The support-ticket lookup took about 81 milliseconds. The subscription lookup took about 181 milliseconds. The payment-history lookup took about 451 milliseconds and dominated the tool phase.

The calls overlap, so adding their durations would overstate the run time. The full trace completed in about 504 milliseconds.

The failed payment lookup was the widest span and determined most of the critical path.

This view answers a second production question: not only what failed, but when it ran and how much of the request it consumed.

Why the prompt was not enough

The agent's system prompt already contained this instruction:

Never claim a charge or refund was verified unless payment history confirms it.

The final model step violates it here. That is precisely the point: a prompt is guidance, not an executable business invariant.

A production model can ignore an instruction, misunderstand a tool error, lose important context, or phrase an uncertain result with too much confidence.

If a rule protects money, permissions, compliance, or a customer commitment, do not enforce it only in prose. Make it part of the application contract.

Fix the evidence contract

For this workflow, lookup_payment_history is not optional context. It is required evidence for any statement that a charge was refunded.

A safer orchestration path is explicit:

case lookup_payment_history(account_id, days: 30) do
  {:ok, payments} ->
    if refund_confirmed?(payments) do
      confirmed_refund_reply(payments)
    else
      no_refund_found_reply()
    end

  {:error, %{type: :upstream_timeout}} ->
    "I couldn't verify the refund because the payment ledger is " <>
      "temporarily unavailable. I've left the ticket open for review."
end

The exact code will differ by runtime. The invariant should not: no successful payment evidence, no refund confirmation.

For this kind of partial failure, use two layers of control:

Retry failures that are explicitly transient, with bounded attempts and backoff.
Refuse to deliver a confirmation unless the required evidence-bearing tool completed successfully.

The second layer belongs close to the customer-facing side effect. Do not ask the model whether it followed the rule; check the underlying tool result.

If verification remains unavailable, return an honest fallback, keep the ticket open, or route it for review. “I couldn't verify this yet” is safer than invented certainty.

For higher-risk actions, go further: represent evidence as typed workflow state, require provenance in structured output, and put the guard before the side effect. An agent that can initiate a refund must not reach that operation merely because its prose sounds confident.

A practical checklist for partial agent failures

Before shipping a tool-using workflow, answer these questions:

Which tool is authoritative for each material claim or action?
What does success mean at each layer—request, tool, graph, and business outcome?
Can the workflow continue safely after each tool fails? Optional enrichment and required evidence should not share a fallback policy.
Which errors are retryable? Retry only classified transient failures, with a firm limit.
What is the safe fallback? Write it explicitly.
Where is the invariant enforced? Put critical guards in code, not only in a system prompt.
Can one trace reconstruct the decision? Keep model calls, tool inputs, outputs, errors, timing, and the final response under the same trace.
Are identifiers searchable and payloads safe? Attach the ticket, release, and environment while excluding secrets and unnecessary personal data.
Do tests include partial success? Exercise one failed tool among otherwise successful calls—not only all-success and total-crash cases.
Do alerts include child failures? A successful application return must not erase a failing critical tool.

Debug the outcome, not only the exception

The interesting part of this incident is not that a tool timed out. Logs and error trackers can already report a timeout.

The important part is what happened next.

A successful model call consumed a failed tool result and produced a confident customer claim anyway. The application returned success. The trace connected that wrong answer to the missing payment evidence and still surfaced the run as an error.

That is the debugging boundary agent teams need: one view of the model calls, tool calls, payloads, errors, timing, and final output behind a specific customer-visible result.

About this example: This controlled run reproduces a common agent failure. The BeamWeaver execution, failed tool call, and WeaveScope trace are real; the ticket and account identifiers are illustrative.