recallA retrieval
field guide

Performance Tuning for Vector Search

Build this skill

Tune index settings to trade recall against speed and memory, and measure the effect.

Fast enough, accurate enough

In production, vector search must be both quick and accurate, for many users at once. Most vector indexes offer settings that trade one for the other, and tuning them is a matter of measuring, not guessing. The goal is a configuration that meets your latency target (for example, p95 search time under 50 milliseconds) with recall close to exact search (for example, Recall@10 of at least 0.95), using memory you can afford.

The settings that matter

For HNSW indexes, three settings dominate:

Setting When it applies Higher value means
M (links per node) Build time Better recall, more memory, slower build
ef_construction Build time Better-quality graph, slower build
ef_search Every query Better recall, slower queries

For IVF indexes:

Setting When it applies Higher value means
lists / nlist (number of clusters) Build time Smaller clusters; need more probes for the same recall
probes / nprobe Every query Better recall, slower queries

The query-time settings (ef_search, probes) are the easiest to tune, because they can be changed without rebuilding the index, even per query. The build-time settings require a rebuild, so they are tuned less often.

ef_search 16Recall@10 0.86 · 0.6 msef_search 400.94 · 1.1 msef_search 1000.98 · 2.4 msef_search 4000.995 · 8.9 msAn illustrative HNSW tuning curve on one dataset

Recall rises quickly and then flattens, while latency keeps growing. The best setting is usually at the bend.

How to tune

  1. Build a benchmark set of a few hundred real or realistic query embeddings, and compute their true top-K with exact (brute-force) search. This is the reference.
  2. Sweep the setting: run the queries at several values (say ef_search = 16, 32, 64, 128, 256), recording Recall@K against the reference and the latency for each.
  3. Plot recall against latency, and pick the smallest value that meets your recall target. Beyond that point you pay in speed for almost no gain.
  4. Check under load. Latency at one query at a time is not the same as latency when 200 queries arrive per second. Run a load test at your expected peak.
  5. Re-check after the data grows. A setting tuned on 100,000 vectors may not hold at 5 million.

The program below performs step 2 and 3 on a recorded sweep, picking the cheapest setting that meets a recall target.

python · runs in your browser

Expected output:

text
ef_search   16: recall 0.860, p95   0.9 ms 
ef_search   32: recall 0.920, p95   1.4 ms 
ef_search   64: recall 0.960, p95   2.3 ms <- choose
ef_search  128: recall 0.980, p95   4.1 ms 
ef_search  256: recall 0.990, p95   7.8 ms 
ef_search  512: recall 0.995, p95  15.2 ms 

ok keeps only the settings that meet both limits: recall of at least 0.95 and p95 latency of at most 10 ms. Among those, min(ok, key=lambda s: s[2]) picks the fastest. Raise target_recall to 0.99 and run it again to see how much latency that last bit of recall costs.

Other levers

When tuning index settings is not enough, several other changes help:

  • Fewer dimensions: a smaller or Matryoshka-truncated embedding makes every comparison cheaper.
  • Quantization: int8 or binary vectors use less memory and compare faster, with rescoring to keep quality.
  • Filter design: very selective filters can slow HNSW searches; partial indexes or filter-aware indexes help.
  • Enough memory: indexes that spill from memory to disk slow down dramatically. Keep the index (or at least its hot part) in RAM.
  • Batching and connection pooling: reuse database connections and batch embedding calls, which often matters more than the index itself.
  • Measure the whole request: vector search is often a few milliseconds out of a two-second answer. Make sure you are tuning the part that is actually slow.

Follow your curiosity.

↑ ↓ to explore · Enter to open · Escape to close