Context Compression and Token Budgeting
Fit the most useful context into a fixed token budget by trimming, compressing and ordering chunks.
More context is not always better
It is tempting to give the model plenty of context "just in case": retrieve ten chunks instead of three, include whole sections instead of paragraphs. But every extra token costs money and adds delay, and extra text that is only loosely related can make answers worse: the model may pick up a wrong detail, mix two rules, or bury the key fact in a long answer. In production, where every request is paid for and users are waiting, sending only what the question needs is one of the best optimisations available.
Two techniques do this: token budgeting, which caps how much context is sent, and context compression, which removes the parts of retrieved text that the question does not need.
Token budgeting
A token budget sets a maximum number of tokens for each part of the prompt, for example:
| Part | Budget |
|---|---|
| System instructions | 300 tokens |
| Conversation history (summarised) | 400 tokens |
| Retrieved context | 1,500 tokens |
| Question | 100 tokens |
| Room for the answer | 500 tokens |
The context builder then fills the context budget with the highest-ranked passages until it is full, skipping any passage that would overflow it. Budgets can also be dynamic: a simple factual question gets a small budget; a broad "explain all the rules about…" question gets a larger one. A router or a quick classification of the question can choose the budget.
Context compression
Context compression shrinks retrieved passages before they go into the prompt. There are three common levels:
- Filtering: drop whole retrieved chunks that a quick check (a reranker score, or a small model) judges irrelevant.
- Extraction: keep only the sentences from each chunk that relate to the question.
- Rewriting: ask a small, cheap model to summarise each chunk with respect to the question.
LangChain, for example, offers a "contextual compression" retriever that wraps a normal retriever and applies such filters and extractors. Research tools such as LLMLingua compress prompts by removing low-information tokens. Each level saves more tokens, and each carries more risk of dropping something important, so compression must be tested on your own questions.
Filtering and extraction remove what the question does not need, then the budget caps the rest.
The program below extracts only the sentences related to the question from three retrieved chunks, and reports the saving.
Expected output:
- Students may take up to 10 days of medical leave in a semester.
- Duty leave for college events is approved by the HoD.
- More than 10 days of medical leave needs the Dean of Academics.
- Approved medical leave is added back to attendance.
words: 75 -> 43 (43% smaller)
relevant splits a sentence into words and counts how many of the keywords at least one word starts with (so "approv" matches both "approves" and "approval"). A sentence is kept when it matches two or more. The list comprehension applies that test to every sentence of every chunk, and the last lines compare the word counts before and after.
The library chunk disappeared entirely, and the attendance chunk shrank to its one relevant sentence. The answer's evidence is intact in less than half the space.
Try the trade-off
Pick which chunks to send for a question under a tight budget. Notice that sending everything is not only expensive but also adds misleading material.
Measure before and after
Compression and budgets should be judged on three numbers together: tokens per request (cost), latency, and answer quality on your test set (faithfulness and completeness). A change that saves 40 percent of tokens with no drop in quality is an easy win. One that saves 60 percent but drops completeness on multi-part questions is not. Run your regression tests every time you tighten a budget or add a compression step.