Tracing, Structured Logging & Cost Dashboards

An agent run is a tree of nested calls — the top-level task, its LLM turns, the tools each turn fires, the sub-calls those tools make. When something goes wrong, 'the agent gave a bad answer' is useless; you need to see which turn chose which tool with which arguments, how many tokens it burned, and how long it took. That structured, nested record is a trace, and instrumenting for it is not optional at production scale.

The vocabulary: traces and spans

• Trace: one complete run of your agent, start to finish, under a single trace_id. Everything about handling one user request lives here.
• Span: one unit of work inside a trace — an LLM call, a tool call, a retrieval step. Spans nest to form the call tree, and each records start/end time, inputs, outputs, and metadata.
• Metadata worth attaching to every span: input and output tokens, dollar cost, latency, model name and version, and on tool spans the tool name and whether it errored.

Tracing tools built for LLM apps — Langfuse is a common, well-documented open-source choice — give you this call tree, per-span token and cost accounting, and a UI to click through a failing run, nearly for free once wired in. The concepts below are portable; the SDK specifics change, so always confirm against current Langfuse docs before you rely on an exact signature.

# Reference skeleton — needs 'pip install langfuse', Langfuse credentials,
# and a running model client (client, SCHEMAS, run_tools). It's here to
# read and adapt, not to run standalone in Colab. Patterns shown are the
# stable, documented shape; confirm signatures against current Langfuse docs.
from langfuse import observe, get_client

langfuse = get_client()

@observe()                                   # wraps the whole run in a trace
def handle_request(user_id: str, prompt: str) -> str:
    # Attach identifiers so you can slice traces by user/session later.
    langfuse.update_current_trace(user_id=user_id, tags=["support-agent"])
    messages = [{"role": "user", "content": prompt}]
    return agent_loop(messages)

@observe()                                   # each loop turn nests as a span
def agent_loop(messages) -> str:
    while True:
        resp = call_model(messages)          # itself observed (below)
        if resp.stop_reason != "tool_use":
            return resp.text
        messages.append({"role": "assistant", "content": resp.content})
        messages.append({"role": "user", "content": run_tools(resp.content)})

@observe(as_type="generation")               # mark LLM calls as generations
def call_model(messages):
    resp = client.messages.create(
        model="claude-sonnet-5", max_tokens=1024,
        tools=SCHEMAS, messages=messages,
    )
    # Report usage so cost/token dashboards populate per generation.
    langfuse.update_current_generation(
        model="claude-sonnet-5",
        usage_details={
            "input": resp.usage.input_tokens,
            "output": resp.usage.output_tokens,
        },
    )
    return resp

Structured logging underneath

Even with a tracing UI, keep structured logs — machine-parseable key/value records, not free-text prints. When your tracing backend is down or you need to reconstruct an incident from raw logs, a line of JSON with trace_id, span, tool, tokens, and cost_usd is queryable; a sentence of English is not. The two are complementary: traces for interactive debugging, structured logs for durable, greppable history.

# Colab cell — pure Python, no key needed; run it as-is.
import json, logging, sys, time

logging.basicConfig(stream=sys.stdout, level=logging.INFO, format="%(message)s")
log = logging.getLogger("agent")

def log_event(trace_id: str, span: str, **fields):
    # One JSON object per line — ships cleanly to any log aggregator.
    record = {"ts": time.time(), "trace_id": trace_id, "span": span, **fields}
    log.info(json.dumps(record))

# Emit at every tool call so cost/latency are reconstructable from logs alone.
def run_tool(trace_id, name, args, fn):
    start = time.time()
    try:
        out = fn(**args)
        log_event(trace_id, "tool_call", tool=name, ok=True,
                  latency_ms=round((time.time() - start) * 1000))
        return out
    except Exception as e:
        log_event(trace_id, "tool_call", tool=name, ok=False,
                  error=type(e).__name__, latency_ms=round((time.time() - start) * 1000))
        raise

# demo: one successful tool call and one failing one, each a JSON log line
run_tool("trace-abc123", "search_orders", {"q": "late"}, lambda q: "3 results")
def boom():
    raise RuntimeError("upstream 500")
try:
    run_tool("trace-abc123", "charge_card", {}, boom)
except RuntimeError:
    pass   # re-raised after logging, as it should be

Diagnosing a cost regression from traces

Here is the payoff scenario from the checklist: your agent's cost doubled week-over-week with flat traffic. Without traces this is a panicked afternoon of guessing. With them it's a sequence of filters: is cost-per-run up, or run-count up (traffic is flat, so cost-per-run)? Then slice per-span: which span's token count grew? Common culprits surface immediately — a retrieval step now stuffing far more context, prompt caching silently breaking so the whole prefix re-bills every turn, or the loop taking more turns because a tool started erroring and the agent keeps retrying.

Online monitoring vs. offline evals

Everything in Lessons 1–3 is an offline eval: a fixed, curated suite you run against a candidate change before it ships, to answer 'does this look safe on the cases we thought to check?' Online monitoring is a different discipline that runs after shipping, against live, unfiltered traffic, to answer a question offline evals structurally cannot: 'is anything drifting right now, on inputs we never thought to check?' You need both — offline evals catch known failure modes cheaply before rollout; online monitoring catches the unknown-unknowns that only surface at scale or over time: a model provider silently updating behavior behind a pinned version string, a dependency's API response shape changing, or the population of real user requests drifting away from what your suite was built to represent.

• Canary sets: a small, fixed sample of production-like inputs, replayed against the live system on a schedule (hourly or daily) independent of real user traffic — the same idea as a regression suite, but run continuously to catch drift from anything other than your own code change: a provider-side model update, silent config drift, an expired credential quietly degrading a tool. Alert when the canary score moves, not only when it hits zero.
• Drift metrics: track the distribution of operational signals over time — input token length, tool-call frequency, average iterations per run, cost-per-run — even without an obvious quality drop, a shift in these is worth investigating, because it usually means the input distribution or a dependency changed underneath you before task success visibly suffers.
• User-feedback signals as weak labels: thumbs up/down, edit distance on a drafted response, regeneration requests, and abandonment are free and high-volume, but they are weak labels, not ground truth — they carry heavy selection bias (only unhappy or unusually engaged users click anything), they're individually noisy (a thumbs-down can mean 'wrong,' 'I changed my mind,' or a misclick), and their meaning drifts with UI placement. Treat them as a cheap signal for what to investigate next, never as a metric you gate a release on, and periodically validate them against a properly sampled human-labeled set the same way you'd validate a judge.

Whiteboard drills