recallA retrieval
field guide

Multi-vector Retrieval

Build this skill

Explain late interaction and MaxSim scoring, and weigh multi-vector retrieval's precision against its storage.

One vector per passage loses detail

A standard embedding model squeezes an entire passage, however long, into a single vector. That vector is a kind of average of everything the passage says. For a focused passage this works well. For a passage that covers several details, such as "Hostel fee ₹96,000 a year; mess charges included; late fee ₹500 per week; refunds only before 1 August", the single vector is a blur, and a precise question like "is there a refund after August?" may not match it strongly.

Multi-vector retrieval keeps more detail by representing each passage with many vectors instead of one, typically one vector per token (word piece). Matching then happens between the parts of the query and the parts of the passage, rather than between two averages.

ColBERT and late interaction

The best-known multi-vector method is ColBERT, introduced by Omar Khattab and Matei Zaharia at Stanford in 2020. ColBERT encodes the query into one vector per query token, and each passage into one vector per passage token, all ahead of time for the passages, just like a bi-encoder.

The interesting part is how a query and a passage are compared, called late interaction and scored with MaxSim:

  1. For each query token vector, find the most similar token vector in the passage (its best match).
  2. Add up those best-match scores across all query tokens.

So "refund" in the query looks for its best partner in the passage ("refunds"), "August" looks for its best partner ("August"), and so on. A passage scores highly if every important part of the query finds a strong match somewhere in it.

Single-vector (bi-encoder)Passage → 1 vectorQuery → 1 vectorCompare two averagesSmall storageFine details blur togetherMulti-vector (ColBERT)Passage → 1 vector per tokenQuery → 1 vector per tokenEach query token finds its best matchMuch more storageFine details kept

Late interaction compares the parts of the query with the parts of the passage, rather than two averages.

MaxSim in code

The program below computes both kinds of score for a question and two passages, using tiny hand-made token vectors. Notice how the averaged vector hides the detail that the token-level comparison catches.

python · runs in your browser

Expected output:

text
Fee page (mentions refunds once)   single-vector 0.53   MaxSim 2.00
Mess menu notice                   single-vector 0.09   MaxSim 0.31
Function What it does
maxsim For each query token, finds the passage token it is most similar to (max(...)), then adds those best scores together.
single_vector Adds all the token vectors of the query into one vector, does the same for the passage, and compares the two averages.

With a single averaged vector, the fee page's refund detail is diluted by everything else on the page. With MaxSim, "refund" and "august" each find their exact partner, and the fee page scores clearly higher.

The storage cost

The price of all this detail is storage. A 300-token passage stored as one 1,024-dimension vector takes about 4 KB. Stored as 300 token vectors of 128 dimensions each (ColBERT uses smaller vectors per token), it takes about 150 KB at full precision, many times more. Later versions such as ColBERTv2 reduce this a lot with compression, and engines such as PLAID make searching these indexes fast.

Where multi-vector retrieval fits

Multi-vector retrieval tends to shine on questions that hinge on specific details inside longer passages, and it often transfers well to new domains. It is also used as a reranker: retrieve candidates cheaply with a single-vector index, then rescore them with late interaction, which avoids storing token vectors for the whole collection. Some embedding models, such as BGE-M3, can produce multi-vector (ColBERT-style) outputs alongside their dense vectors, and some vector databases support multi-vector fields directly. The same late-interaction idea has also been applied to images of document pages.

As always, the question is whether the gain on your own test questions justifies the extra storage and complexity. For many collections of short, focused chunks, a single vector plus a reranker is enough.

Follow your curiosity.

↑ ↓ to explore · Enter to open · Escape to close