Vector Compression
Shrink vector memory with scalar, binary and product quantization, recovering quality with rescoring.
Embeddings take a lot of memory
Each number in an embedding is normally stored as a 32-bit floating-point number, which takes 4 bytes. An embedding of 1,024 numbers therefore takes about 4 KB. That sounds tiny until you multiply. Ten million chunks need about 41 GB for the vectors alone, before any index is built on top. Fast indexes such as HNSW work best when the vectors sit in memory (RAM), and memory is expensive.
Vector compression, usually called quantization, stores each vector in fewer bytes by using less precise numbers. You get much lower memory use and faster comparisons, and you give up a little accuracy in the similarity scores.
Marks and grades work the same way. A marks sheet records 87.46; a grade card records "A". The grade takes far less space and still ranks most students correctly, although 87.46 and 86.90 can no longer be told apart.
Three ways to compress
Scalar quantization stores each number with fewer bits. The most common form turns each 32-bit float into an 8-bit integer, a whole number from −128 to 127, much like rounding marks to whole numbers. Memory falls by 4×, usually with very little loss in quality.
Binary quantization goes to the extreme of pass or fail. Each number becomes a single bit: 1 if it is positive, 0 if not. Memory falls by 32×, and comparing two binary vectors is extremely fast, because a processor can count the positions where two bit strings differ (the Hamming distance) in one instruction. A lot of detail is lost, so binary vectors are normally followed by a re-check step.
Product quantization (PQ) works on pieces of the vector. It cuts each vector into, say, 64 pieces, learns a codebook of 256 typical patterns for each piece position, and stores each piece as the number of the pattern it most resembles, which fits in one byte. A police sketch artist works similarly, building a face from the closest eyes, nose and mouth in a catalogue. A 1,024-number vector then takes 64 bytes instead of 4,096, a 64× saving. PQ is used heavily in FAISS for very large collections.
Compression shrinks each vector dramatically. The smaller formats need a re-check of the top results to keep quality high.
Seeing what is stored
The program below takes one small vector and shows what each format keeps.
Expected output:
float32: [ 0.42 -0.13 0.07 -0.56 0.31 0.02 -0.28 0.49] -> 32 bytes
int8: [ 95 -29 16 -127 70 5 -64 111] -> 8 bytes
binary: [1 0 1 0 1 1 0 1] -> 1 byte (8 bits)
int8 turned back: [ 0.42 -0.13 0.07 -0.56 0.31 0.02 -0.28 0.49]
largest error: 0.0022
v / scale * 127 stretches the numbers so the largest becomes 127, and np.round then rounds each to a whole number, which is where precision is lost. (v > 0) keeps only the sign. The back line reverses the int8 step: the result is within 0.003 of the original on every number, while the binary version remembers only which numbers were positive.
Rescoring: compress for speed, check with the original
Small errors in scores can change the order of results. A two-step pattern keeps the saving and repairs most of the damage. First, search the compressed vectors for a generous list of candidates, say the top 50. Then rescore only those 50 with the full-precision vectors, which can sit on cheaper disk storage, and return the best 5. The true top 5 are almost always somewhere in the compressed top 50.
Compare the three formats in the lab, then switch rescoring on.
Measuring the loss
The program below repeats the test on 5,000 vectors and reports recall@10: of the true 10 nearest vectors, the share each format finds.
Expected output:
int8 recall@10: 0.98
binary recall@10: 0.51
binary, then rescore the top 50: 1.00
| Part of the code | What it does |
|---|---|
bit_scores(q) |
Turns the query into bits and counts how many of the 256 bits agree with each stored vector. |
[:50 if rescore else 10] |
Without rescoring, the compressed search must get the top 10 right by itself. With rescoring, it only needs them in its top 50. |
data[candidates] @ q |
The rescoring step: the 50 candidates are scored again with the original vectors. |
The int8 vectors lose almost nothing. Binary vectors alone find only about half of the true top 10, and with rescoring they find all of them, while still storing 32 times less in memory.
Choosing a level of compression
Use the calculator to see how much memory each option needs for a collection.
Start uncompressed for small collections. When memory gets tight, try int8 first, because it rarely costs noticeable quality. Move to binary or product quantization, with rescoring, only for very large collections, and measure recall on your own data before and after. Qdrant, Milvus and Weaviate support these options directly, and pgvector offers halfvec (16-bit floats) and bit (binary) column types.