Monitoring in Production
Choose the quality, speed and cost signals to watch in production and set alerts that matter.
Knowing when something goes wrong
Traces let you investigate one request. Monitoring watches all requests together, over time, so that problems are noticed quickly, ideally before many users are affected. A monitoring setup collects a few key metrics, shows them on dashboards, and raises alerts when something crosses a threshold.
For a RAG system, it helps to monitor four kinds of signal: how fast it is, whether it is working, what it costs, and whether its answers are good.
What to measure
| Area | Metric | Why it matters |
|---|---|---|
| Speed | Latency: median (p50) and slow tail (p95, p99) | Users feel the slow requests, not the average |
| Working | Error rate; timeouts; empty retrievals | Outages and broken dependencies |
| Cost | Tokens per request; cost per day | Budgets, and spotting runaway prompts |
| Retrieval | Top similarity score; share of "no relevant result" | Retrieval drifting or a collection going missing |
| Answers | Share of "I couldn't find this"; faithfulness on a sample | Quality slipping |
| Users | Thumbs up/down; follow-up rephrasing rate | What people actually experience |
| Volume | Requests per minute; top topics | Load, and what people need |
Percentiles deserve a short explanation. The p95 latency is the time within which 95 percent of requests finish; the slowest 5 percent take longer. Averages hide slow requests: an average of 1.5 seconds can include many 8-second waits. Watching p95 and p99 tells you how the worst experiences are changing.
A small set of numbers covering speed, health, quality and cost tells most of the story.
Measuring answer quality in production
Speed, errors and cost are easy to measure. Answer quality is harder, because live questions have no reference answers. Common approaches:
- Automated checks on a sample: run reference-free metrics such as faithfulness on, say, 2 percent of live answers each day.
- Proxy signals: the share of "I could not find this" answers, very low top retrieval scores, and users rephrasing the same question straight away all suggest trouble.
- User feedback: thumbs up or down, with an optional comment. Few users give feedback, so treat it as a strong signal on the cases it covers, not a full measure.
- Regular human review of a random sample and of everything users flagged.
Alerts that people trust
An alert notifies someone when a metric crosses a threshold, for example "p95 latency above 5 seconds for 10 minutes" or "error rate above 2 percent". Good alerts are actionable (someone knows what to do), rare (an alert that fires daily for no reason gets ignored), and sustained (based on a few minutes of data, not one spike).
The program below computes latency percentiles from a batch of request times and checks two alert rules.
Expected output:
mean 2.30 s | p50 1.4 s | p95 7.9 s
ALERT p95 latency > 5 s
ALERT error rate > 2%
percentile sorts the values and picks the one a given percentage of the way up the list: p50 is the middle value, and p95 is the value that 95 percent of requests were faster than. Each alert rule is a name paired with a true-or-false test, so adding a rule means adding one line to rules.
The average of about 2.4 seconds hides the real problem: one request in ten is taking 6 to 8 seconds. The p95 alert catches it.
Drift: slow changes that hurt
Not every problem is sudden. Drift is gradual change: users start asking about a new topic the documents don't cover, a source system stops syncing so the index slowly goes stale, or a model provider updates its model and answers change style. Watching trends over weeks (the share of "not found" answers by topic, document freshness, feedback by topic) catches drift that day-to-day alerts miss.