recallA retrieval
field guide

Context Compression and Token Budgeting

Build this skill

Fit the most useful context into a fixed token budget by trimming, compressing and ordering chunks.

More context is not always better

It is tempting to give the model plenty of context "just in case": retrieve ten chunks instead of three, include whole sections instead of paragraphs. But every extra token costs money and adds delay, and extra text that is only loosely related can make answers worse: the model may pick up a wrong detail, mix two rules, or bury the key fact in a long answer. In production, where every request is paid for and users are waiting, sending only what the question needs is one of the best optimisations available.

Two techniques do this: token budgeting, which caps how much context is sent, and context compression, which removes the parts of retrieved text that the question does not need.

Token budgeting

A token budget sets a maximum number of tokens for each part of the prompt, for example:

Part Budget
System instructions 300 tokens
Conversation history (summarised) 400 tokens
Retrieved context 1,500 tokens
Question 100 tokens
Room for the answer 500 tokens

The context builder then fills the context budget with the highest-ranked passages until it is full, skipping any passage that would overflow it. Budgets can also be dynamic: a simple factual question gets a small budget; a broad "explain all the rules about…" question gets a larger one. A router or a quick classification of the question can choose the budget.

Context compression

Context compression shrinks retrieved passages before they go into the prompt. There are three common levels:

  • Filtering: drop whole retrieved chunks that a quick check (a reranker score, or a small model) judges irrelevant.
  • Extraction: keep only the sentences from each chunk that relate to the question.
  • Rewriting: ask a small, cheap model to summarise each chunk with respect to the question.

LangChain, for example, offers a "contextual compression" retriever that wraps a normal retriever and applies such filters and extractors. Research tools such as LLMLingua compress prompts by removing low-information tokens. Each level saves more tokens, and each carries more risk of dropping something important, so compression must be tested on your own questions.

Retrievedchunks≈ 3,000 tokensFilterdrop irrelevantchunksExtractkeep relevantsentencesBudgetfit within thelimitPrompt≈ 800 tokens

Filtering and extraction remove what the question does not need, then the budget caps the rest.

The program below extracts only the sentences related to the question from three retrieved chunks, and reports the saving.

python · runs in your browser

Expected output:

text
- Students may take up to 10 days of medical leave in a semester.
- Duty leave for college events is approved by the HoD.
- More than 10 days of medical leave needs the Dean of Academics.
- Approved medical leave is added back to attendance.

words: 75 -> 43 (43% smaller)

relevant splits a sentence into words and counts how many of the keywords at least one word starts with (so "approv" matches both "approves" and "approval"). A sentence is kept when it matches two or more. The list comprehension applies that test to every sentence of every chunk, and the last lines compare the word counts before and after.

The library chunk disappeared entirely, and the attendance chunk shrank to its one relevant sentence. The answer's evidence is intact in less than half the space.

Try the trade-off

Pick which chunks to send for a question under a tight budget. Notice that sending everything is not only expensive but also adds misleading material.

Measure before and after

Compression and budgets should be judged on three numbers together: tokens per request (cost), latency, and answer quality on your test set (faithfulness and completeness). A change that saves 40 percent of tokens with no drop in quality is an easy win. One that saves 60 percent but drops completeness on multi-part questions is not. Run your regression tests every time you tighten a budget or add a compression step.

Follow your curiosity.

↑ ↓ to explore · Enter to open · Escape to close