recallA retrieval
field guide

Tracing and Observability

Build this skill

Trace every step of a RAG request so you can see exactly where a bad answer went wrong.

Seeing inside every request

When a student reports "the assistant gave me the wrong hostel fee", what can you do? Without records, nothing: you cannot see what was retrieved, what prompt was built, or what the model saw. You can only try the question again and hope to reproduce the problem. In production, where thousands of different questions arrive and documents keep changing, that is not good enough.

Observability means being able to understand what a system did, and why, from the data it records. For RAG systems, the core tool is the trace: a detailed record of everything that happened while answering one request.

Traces and spans

A trace is made of spans. Each span records one step: its name, start and end time, inputs, outputs and any errors. Spans are nested, so a single request's trace looks like a tree: the whole request at the top, and inside it spans for rewriting the query, embedding it, searching, reranking, building the prompt and calling the model.

requestanswer one questionrewrite queryLLMembed + searchrerankrerankbuild promptgenerateLLM answer0s0.5s1s1.5s2s

Each span shows when a step ran and how long it took. Here, generation takes most of the time.

A useful RAG trace records, at minimum:

Span What to record
Request User (or role), question, final answer, total time, feedback
Query rewrite Original and rewritten query
Retrieval Filters, top results with IDs, sources and scores
Rerank Scores before and after, what was kept
Prompt The exact prompt sent, its token count
Model call Model name and version, settings, output, tokens, cost, time
Errors Any failure, with its message

Explore a recorded trace below. Click each span to see its details. One step in this trace hides the reason the answer was wrong.

The trace shows the root cause in seconds: the search had no filter for current documents, and the prompt labels did not show dates, so the model used the first source. Without a trace, the team might have spent a day rewriting the prompt.

Tools: LangSmith and OpenTelemetry

LangSmith, from the makers of LangChain, is a platform for tracing, inspecting and evaluating LLM applications. Applications built with LangChain can send traces to it with a few settings, and others can use its SDK. It shows each trace as a tree, lets you search and filter traces, attach user feedback, and turn interesting traces into evaluation test cases.

OpenTelemetry is an open, vendor-neutral standard for collecting traces, metrics and logs from any software, widely used across the industry. It includes conventions for describing generative AI operations (model name, token counts and so on), so LLM traces can flow into the same monitoring tools a company already uses. Several LLM observability tools, including open-source ones such as Langfuse and Arize Phoenix, accept OpenTelemetry data.

Recording a trace yourself

The idea behind every tracing tool is simple: wrap each step, record when it started and ended and what it produced. The program below builds a tiny tracer with a Python context manager.

python · runs in your browser

Expected output:

text
{"name": "vector_search", "filters": {"access": "students", "status": "current"}, "top": ["fees-2026#1", "hostel-rules#3"]} | time recorded
{"name": "llm_generate", "model": "example-model", "output": "The hostel fee is Rs 96,000 a year [1]."} | time recorded
{"name": "answer_question", "question": "What is the hostel fee?", "answer": "The hostel fee is Rs 96,000 a year [1]."} | time recorded
Part of the code What it does
@contextmanager Turns span into something usable with with. Code before yield runs on entering the block, and code in finally runs on leaving it, even after an error.
record = {"name": name, **attrs} One span: its name and whatever details were passed in.
yield record Hands the record to the block, so the code inside can add results such as s["top"].
record["ms"] = ... On the way out, works out how long the block took and stores the finished span in trace.

The inner spans finish first, so they appear in the trace before the outer one that contains them.

What to be careful about

Traces contain questions, documents and answers, which can include personal information. Apply the same rules as for the knowledge base: restrict who can view traces, mask sensitive fields, and delete traces after a set period. Also decide how much to record: storing every full prompt for millions of requests costs money, so many teams record everything for a sample of requests and summaries for the rest.

Follow your curiosity.

↑ ↓ to explore · Enter to open · Escape to close