36 · Seeing what happened

When a run does something you did not expect, you want to see it step by step: what the router read, what each check decided and why, which call ran, what was undone. stalwart trace shows a run as a timeline, from the storage your agent already keeps or from the spans a tracer recorded, and every decision on it names the rule that made it.

Language support

The selector reads turns written in English and in Brazilian Portuguese (pt-BR). The language is chosen, never guessed from the turn: the console's Settings page (Language the selector reads), language on a plan request, or the settings.language of your vocabulary file; with none of them, English. A Portuguese turn may be typed with its accents or without them. European Portuguese is not a grammar the selector has, and a turn in any other language is unread.

A run as a timeline

stalwart trace has three subcommands:

• stalwart trace show: the timeline in the terminal, one block per step, each check with its answer and its reasons, each pause for a person, each undo and the step it undoes.
• stalwart trace html ... -o run.html: the same timeline as one self-contained HTML file, with no script and no request to anywhere, for a ticket or an audit.
• stalwart trace diff a.jsonl b.jsonl: where two runs of the same input went different ways: the first step whose decisions differ, and what only each run did there.

show and html read either a spans file or, with --thread and --store, your agent's storage.

The same from storage alone

A run your agent kept in storage: can be read back with nothing recorded beforehand. E6's failed order (chapter 34), with the shop's storage on disk:

desk = Agent.from_file("agent.yaml", context={"card_on_file": True})   # storage: sqlite:///runs.db

reply = desk.chat("charge order 4411 40.00, then update the crm for order 4411, "
                  "then provision the account for order 4411", user="u-17")
while reply.waiting:
    reply = desk.confirm("u-17", "approve")   # each write approved as it comes
print(reply.text)
desk.close()

This run stopped: provision_account failed (RuntimeError). Undone, latest first: update_crm (by restore_crm), charge_card (by refund_card).

The conversation is desk/u-17: the agent's name and the person's. Read back from the storage:

$ stalwart trace show --thread desk/u-17 --store sqlite:///runs.db
Run: thread desk/u-17
step -1
  node      input
step 0
  node      run
step 1
  node      run
step 2
  node      run
  check     charge_card -> confirm
  interrupt charge_card [write] (confirm)
step 3
  node      run
  check     charge_card -> allow
step 4
  node      run
  call      charge_card
step 5
  node      run
step 6
  node      run
step 7
  node      run
  check     update_crm -> confirm
  interrupt update_crm [write] (confirm)
step 8
  node      run
  check     update_crm -> allow
step 9
  node      run
  call      update_crm
step 10
  node      run
step 11
  node      run
step 12
  node      run
  check     provision_account -> confirm
  interrupt provision_account [write] (confirm)
step 13
  node      run
  check     provision_account -> allow
step 14
  node      run
  call      provision_account
step 15
  node      run
  undo      restore_crm -> allow undoes step 3
step 16
  node      run
step 17
  node      run
  undo      refund_card -> allow undoes step 1
step 18
  node      run
step 19
  node      run
step 20
  node      run

Each write was checked and asked about (confirm, [write]), approved (allow), and run; provision_account failed at step 14, and the two undos follow, each checked, latest first: restore_crm undoes the CRM update (history step 3) and refund_card the charge (history step 1). A step with nothing under it is one that only moved the run on.

Spans for any OpenTelemetry tool

A tracer records one span around each of these, generated from the package's tracer:

Span	Opened around
stalwart.node	each step of a run
stalwart.plan	the router reading a turn
stalwart.tool_check	the tool check deciding a call
stalwart.tool	a tool running
stalwart.compensate	an undo running
stalwart.rollback	a failed run being undone
stalwart.rewind	a run rewound
stalwart.interrupt	a run pausing for a person
stalwart.resume	a run resuming on the person's answer

A tool check's span carries its decision, as these attributes and as a stalwart.decision event:

Attribute	Holds
gen_ai.operation.name	execute_tool, as OpenTelemetry's GenAI conventions name a tool call
gen_ai.tool.name	the tool
gen_ai.tool.call.id	the call's id
stalwart.outcome	the check's answer: allow, confirm or refuse
stalwart.decided_by	which step of the check decided
stalwart.rule	the rule, flow or reason behind the answer
stalwart.reasons	every reason the check gave
stalwart.labels	the marks the call carried
stalwart.args_digest	the first 16 hex characters of the arguments' digest, never their values
stalwart.history_token	a short digest of the session's history token, never the token

gen_ai.* are OpenTelemetry's GenAI conventions, so any tool that reads them shows the calls; stalwart.* are ours. With pip install "stalwart-agents[otel]", the spans go to OpenTelemetry's tracer provider, and from it to whichever exporter you configured. With no exporter at all, a JSON-lines tracer writes them to a file, which stalwart trace show and diff read.

Content is off by default

No argument value, message, result or turn is put on a span unless the tracer is made with capture_content=True. Where content is off, a 16-character digest stands in for each, so two spans can be compared without either holding the value. Turning it on adds exactly these attributes:

Attribute	Content off (the default)	Content on (capture_content=True)
stalwart.arguments	stalwart.arguments_digest	the call's arguments, as values
stalwart.result	stalwart.result_digest	the tool's result
stalwart.message	stalwart.message_digest	a message of the conversation
stalwart.turn	stalwart.turn_digest	the person's turn, as typed

They go where every other attribute goes: to the exporters of the tracer provider you gave it, or to the JSON-lines file you named. A tracer sends nothing to us, either way: a trace is yours.

The state validator

The state a run keeps -- its messages, its history, what is waiting for a person, what to undo -- must stay consistent, and a step your team writes can break it. The validator names each break:

Violation	Means
call_without_result	a call the model asked for has no result, and the conversation went on past it
result_without_call	a result answers no call in the conversation
results_out_of_order	a step's results do not follow it in the order of its calls
duplicate_message_id	two messages share an id
message_after_interrupt	a message was added while a pause for a person was still open
stack_step_not_in_history	the list of what to undo names a step the history does not have
history_broken	the history's steps are not 0, 1, 2, ..., or it has entries and no token
history_token_invalid	the history's token does not verify: an entry was edited, dropped or reordered outside the run
label_on_removed_message	a mark names a message the conversation no longer holds

stalwart trace show and html mark the first break in a run read from storage with <<< FIRST VIOLATION, at the step that caused it, so the timeline shows where the state went wrong rather than where it failed later.

Following a decision to its rule

Each check on a timeline names its answer and who decided it, and its span names the rule:

• stalwart.decided_by is the step of the tool check that decided (chapter 32);
• stalwart.rule is the rule, flow or reason: a rulebook's refusal names its flow (chapter 45), an order rule names the binding it broke;
• stalwart.reasons lists every reason a confirmation gave.

If you already use LangGraph

The spans are opened by the run a tracer is given: desk.graph(tracer=JsonLinesTracer("run.jsonl")) compiles the agent's graph with stalwart_agents.langchain.core.tracing.JsonLinesTracer, and stalwart_agents.langchain.otel.OtelTracer opens the same spans in OpenTelemetry, with NodeSpans as the callback that opens one per node. validate_state(state) and the debug wrapper validating(graph), which stops at the first violation with the node and step, are in stalwart_agents.langchain.graph. Chapter 27 has them in LangGraph's own terms.

The limits

• A timeline read from storage shows what the storage kept: the checks, the calls, the pauses and the undos, not the spans' timings, and not what happened inside your tools.
• Captured content goes wherever your exporter sends it. Turn it on only where that destination may hold your users' words and your tools' results.