recallA retrieval
field guide

Cost Engineering

Build this skill

Estimate the cost per question, find the biggest cost driver and cut it without losing quality.

Where the money goes

A RAG system has running costs that grow with use, and a pilot that costs a few hundred rupees a day can cost lakhs a month once a whole university uses it. Cost engineering means knowing where the money goes, estimating it before launch, and reducing it without hurting answer quality.

Costs fall into a few categories:

Cost Paid when Depends on
Embedding documents Indexing, and every re-index Total tokens in the collection; model price
Embedding queries Every question Question length; volume
Vector storage and search Continuously Number of vectors, dimensions, replicas, memory
Reranking Every question Number of candidates; model
Generation input tokens Every question Prompt size: instructions + context + history
Generation output tokens Every question Answer length
Extra model calls Every question that uses them Query rewriting, routing, evaluation, agents
Infrastructure and people Continuously Servers, monitoring, maintenance

For most systems that use a paid language model API, generation dominates, and within generation, the input tokens (mostly retrieved context) are often the largest part, because prompts are much longer than answers. Output tokens are usually priced higher per token than input tokens, so long answers add up too.

A cost per question

The most useful single number is the cost per question, broken down by stage. The program below computes it for an example configuration, using illustrative prices.

python · runs in your browser

Expected output:

text
generation input     Rs 0.2400  (68.8%)
generation output    Rs 0.0720  (20.6%)
query rewrite (LLM)  Rs 0.0370  (10.6%)
query embedding      Rs 0.0001  ( 0.0%)
TOTAL per question   Rs 0.3491
per month at 20,000 questions/day: Rs 209,436

Each stage's cost is its token count multiplied by the price per million tokens, divided by a million. Input and output tokens are priced separately, because output usually costs several times more. Sorting by cost puts the most expensive stage at the top, which is the first place to look for savings.

Generation input is about two-thirds of the cost here. Cutting the retrieved context from 2,400 to 1,200 tokens would save more than removing every other step combined.

Levers for reducing cost

In rough order of how much they usually save:

  1. Send less context. Fewer, better chunks (reranking, compression, token budgets) cut the largest cost directly.
  2. Cache. Repeated questions answered from a response cache cost almost nothing.
  3. Use the right model for each job. A small, cheap model is often good enough for query rewriting, routing and simple questions; send only hard questions to a large model. This is called model routing or model cascading.
  4. Keep answers concise. Ask for short answers where appropriate; output tokens are usually the most expensive per token.
  5. Use prompt caching. Providers often charge much less for a repeated prompt prefix, such as long fixed instructions.
  6. Avoid unnecessary steps. Query rewriting, multi-query and agents each add model calls; use them only for the question types they actually help.
  7. Embed once. Cache embeddings and re-embed only changed chunks.
  8. Right-size storage. Smaller dimensions and quantization reduce memory and server costs at scale.
Baseline100%+ token budget (half context)≈ 66%+ 30% cache hits≈ 46%+ small model for easy questions≈ 30%Illustrative monthly cost after each change, as a share of the baseline

A few well-chosen changes can cut costs by more than half, if quality is checked after each one.

Estimating cost before you launch

Use the calculator to estimate monthly cost for your expected traffic. Prices are illustrative; replace them with your provider's current prices.

Cost and quality together

Every cost cut should be checked against quality. Halving the context might also halve the faithfulness on multi-part questions; a cheaper model might misread tables. Track cost per question and quality metrics side by side in the same dashboard and regression tests, and treat a cost saving that drops quality below your thresholds as a regression, not a win.

Follow your curiosity.

↑ ↓ to explore · Enter to open · Escape to close