Performance Tuning for Vector Search
Tune index settings to trade recall against speed and memory, and measure the effect.
Fast enough, accurate enough
In production, vector search must be both quick and accurate, for many users at once. Most vector indexes offer settings that trade one for the other, and tuning them is a matter of measuring, not guessing. The goal is a configuration that meets your latency target (for example, p95 search time under 50 milliseconds) with recall close to exact search (for example, Recall@10 of at least 0.95), using memory you can afford.
The settings that matter
For HNSW indexes, three settings dominate:
| Setting | When it applies | Higher value means |
|---|---|---|
M (links per node) |
Build time | Better recall, more memory, slower build |
ef_construction |
Build time | Better-quality graph, slower build |
ef_search |
Every query | Better recall, slower queries |
For IVF indexes:
| Setting | When it applies | Higher value means |
|---|---|---|
lists / nlist (number of clusters) |
Build time | Smaller clusters; need more probes for the same recall |
probes / nprobe |
Every query | Better recall, slower queries |
The query-time settings (ef_search, probes) are the easiest to tune, because they can be changed without rebuilding the index, even per query. The build-time settings require a rebuild, so they are tuned less often.
Recall rises quickly and then flattens, while latency keeps growing. The best setting is usually at the bend.
How to tune
- Build a benchmark set of a few hundred real or realistic query embeddings, and compute their true top-K with exact (brute-force) search. This is the reference.
- Sweep the setting: run the queries at several values (say
ef_search= 16, 32, 64, 128, 256), recording Recall@K against the reference and the latency for each. - Plot recall against latency, and pick the smallest value that meets your recall target. Beyond that point you pay in speed for almost no gain.
- Check under load. Latency at one query at a time is not the same as latency when 200 queries arrive per second. Run a load test at your expected peak.
- Re-check after the data grows. A setting tuned on 100,000 vectors may not hold at 5 million.
The program below performs step 2 and 3 on a recorded sweep, picking the cheapest setting that meets a recall target.
Expected output:
ef_search 16: recall 0.860, p95 0.9 ms
ef_search 32: recall 0.920, p95 1.4 ms
ef_search 64: recall 0.960, p95 2.3 ms <- choose
ef_search 128: recall 0.980, p95 4.1 ms
ef_search 256: recall 0.990, p95 7.8 ms
ef_search 512: recall 0.995, p95 15.2 ms
ok keeps only the settings that meet both limits: recall of at least 0.95 and p95 latency of at most 10 ms. Among those, min(ok, key=lambda s: s[2]) picks the fastest. Raise target_recall to 0.99 and run it again to see how much latency that last bit of recall costs.
Other levers
When tuning index settings is not enough, several other changes help:
- Fewer dimensions: a smaller or Matryoshka-truncated embedding makes every comparison cheaper.
- Quantization: int8 or binary vectors use less memory and compare faster, with rescoring to keep quality.
- Filter design: very selective filters can slow HNSW searches; partial indexes or filter-aware indexes help.
- Enough memory: indexes that spill from memory to disk slow down dramatically. Keep the index (or at least its hot part) in RAM.
- Batching and connection pooling: reuse database connections and batch embedding calls, which often matters more than the index itself.
- Measure the whole request: vector search is often a few milliseconds out of a two-second answer. Make sure you are tuning the part that is actually slow.