recallA retrieval
field guide

Monitoring in Production

Build this skill

Choose the quality, speed and cost signals to watch in production and set alerts that matter.

Knowing when something goes wrong

Traces let you investigate one request. Monitoring watches all requests together, over time, so that problems are noticed quickly, ideally before many users are affected. A monitoring setup collects a few key metrics, shows them on dashboards, and raises alerts when something crosses a threshold.

For a RAG system, it helps to monitor four kinds of signal: how fast it is, whether it is working, what it costs, and whether its answers are good.

What to measure

Area Metric Why it matters
Speed Latency: median (p50) and slow tail (p95, p99) Users feel the slow requests, not the average
Working Error rate; timeouts; empty retrievals Outages and broken dependencies
Cost Tokens per request; cost per day Budgets, and spotting runaway prompts
Retrieval Top similarity score; share of "no relevant result" Retrieval drifting or a collection going missing
Answers Share of "I couldn't find this"; faithfulness on a sample Quality slipping
Users Thumbs up/down; follow-up rephrasing rate What people actually experience
Volume Requests per minute; top topics Load, and what people need

Percentiles deserve a short explanation. The p95 latency is the time within which 95 percent of requests finish; the slowest 5 percent take longer. Averages hide slow requests: an average of 1.5 seconds can include many 8-second waits. Watching p95 and p99 tells you how the worst experiences are changing.

Speedp50 1.4 sp95 3.1 sp99 6.8 sHealtherrors 0.3%timeouts 0.1%empty retrievals 2%Quality“not found” 7%faithfulness 0.92 (sample)👍 81%Cost2,300 tokens / request₹4,100 / daytop cost: generation

A small set of numbers covering speed, health, quality and cost tells most of the story.

Measuring answer quality in production

Speed, errors and cost are easy to measure. Answer quality is harder, because live questions have no reference answers. Common approaches:

  • Automated checks on a sample: run reference-free metrics such as faithfulness on, say, 2 percent of live answers each day.
  • Proxy signals: the share of "I could not find this" answers, very low top retrieval scores, and users rephrasing the same question straight away all suggest trouble.
  • User feedback: thumbs up or down, with an optional comment. Few users give feedback, so treat it as a strong signal on the cases it covers, not a full measure.
  • Regular human review of a random sample and of everything users flagged.

Alerts that people trust

An alert notifies someone when a metric crosses a threshold, for example "p95 latency above 5 seconds for 10 minutes" or "error rate above 2 percent". Good alerts are actionable (someone knows what to do), rare (an alert that fires daily for no reason gets ignored), and sustained (based on a few minutes of data, not one spike).

The program below computes latency percentiles from a batch of request times and checks two alert rules.

python · runs in your browser

Expected output:

text
mean 2.30 s | p50 1.4 s | p95 7.9 s
ALERT  p95 latency > 5 s
ALERT  error rate > 2%

percentile sorts the values and picks the one a given percentage of the way up the list: p50 is the middle value, and p95 is the value that 95 percent of requests were faster than. Each alert rule is a name paired with a true-or-false test, so adding a rule means adding one line to rules.

The average of about 2.4 seconds hides the real problem: one request in ten is taking 6 to 8 seconds. The p95 alert catches it.

Drift: slow changes that hurt

Not every problem is sudden. Drift is gradual change: users start asking about a new topic the documents don't cover, a source system stops syncing so the index slowly goes stale, or a model provider updates its model and answers change style. Watching trends over weeks (the share of "not found" answers by topic, document freshness, feedback by topic) catches drift that day-to-day alerts miss.

Follow your curiosity.

↑ ↓ to explore · Enter to open · Escape to close