Building Semantic Search from Scratch
Build exact semantic search with NumPy and estimate when brute-force search becomes too slow.
Semantic search in four steps
Every semantic search engine, from a college helpdesk to a search engine with billions of pages, does the same four things:
- Embed the collection. Turn every passage into an embedding, once, ahead of time.
- Store the embeddings. Keep them next to the passage text so results can be shown.
- Embed the query. When a question arrives, turn it into an embedding with the same model.
- Find the closest. Compare the query embedding with every stored embedding and return the top K.
Building this once yourself, with nothing more than Python and NumPy, removes the mystery from the tools you will use later. Vector databases, indexes and frameworks all add speed, scale and convenience on top of these same four steps.
The simplest semantic search compares the query with every stored embedding. It is exact, and easy to build.
Embeddings as a table of numbers
If you have 5 passages and each embedding has 4 numbers, you can store all of them as a table with 5 rows and 4 columns, which NumPy calls a matrix. Each row is one passage's embedding. With normalised embeddings, the similarity between the query and every passage is just the dot product of the query with each row, and NumPy can do all of those at once with a single matrix multiplication.
This "compare with everything" approach is called brute-force or exact search. It always finds the true closest passages, because it checks every one.
A complete example
The program below is a full semantic search engine for five campus passages. Real embedding models cannot run inside this page, so the embeddings here are small, hand-made vectors whose four positions loosely stand for leave, hostel, library and exams. The search code is exactly what you would use with real embeddings: only the embed function would change.
Expected output:
How many sick days do I get?
1.0 Students may take up to 10 days of medical leave in a semester.
0.32 Apply for re-evaluation within 5 days of the results.
What time must I be back by?
1.0 Hostel gates close at 10 PM for first-year students.
0.93 The hostel fee is Rs 96,000 a year, paid in two instalments.
Can I get my paper rechecked?
1.0 Apply for re-evaluation within 5 days of the results.
0.36 The hostel fee is Rs 96,000 a year, paid in two instalments.
| Line | What it does |
|---|---|
matrix = np.array([embed(p) for p in passages]) |
Embeds every passage once and stacks the results, one row per passage. |
scores = matrix @ q |
Multiplies every row with the query in one step, giving five similarity scores. |
np.argsort(-scores)[:k] |
Sorts the scores from highest to lowest and keeps the row numbers of the best k. |
passages[i] for i in best |
Uses those row numbers to look up the text to show. |
Notice that "sick days" finds the medical leave passage, and "paper rechecked" finds the re-evaluation passage, even though they share almost no words. With a real embedding model, you would delete the toy_vectors and the embed function, and replace it with a call to your model. Everything else stays the same.
Using a real embedding model on your own computer
On your own machine, with a real model, the same program is even shorter. The example below uses the sentence-transformers library and a small open model. It needs an internet connection the first time, to download the model, so it is shown here as code to copy rather than a cell to run in this page.
# pip install sentence-transformers
from sentence_transformers import SentenceTransformer
import numpy as np
model = SentenceTransformer("BAAI/bge-small-en-v1.5") # a small open English model
matrix = model.encode(passages, normalize_embeddings=True) # one row per passage
def search(question, k=3):
q = model.encode(question, normalize_embeddings=True)
scores = matrix @ q
best = np.argsort(-scores)[:k]
return [(float(scores[i]), passages[i]) for i in best]
How far does brute force go?
Brute force is exact and simple, and for small collections it is all you need. Its cost grows in a straight line with the size of the collection: twice the passages means twice the work for every query. Use the calculator below to see roughly how much work one query needs as the collection grows.
For a college with 10,000 passages, a search takes about ten milliseconds and fits easily in memory. For 50 million passages, every query would compare against all 50 million rows, which is too slow when many people are searching at once. That is the problem that vector indexes solve: they find nearly the closest passages while checking only a small fraction of them.
What a real system adds
Your four-step engine already does semantic search correctly. A production system adds a few things around it, and it is useful to know what they are for:
- A vector database or index, so searches stay fast when the collection grows large.
- Metadata stored with each passage, such as source document, section and date, so answers can cite sources and old versions can be filtered out.
- Updates, so new or changed documents are embedded and stored without rebuilding everything.
- A keyword search alongside, for course codes, names and exact phrases that embeddings handle poorly.
Each of these is an addition to the four steps, not a replacement. Whatever tools you use later, the core is still: embed ahead of time, embed the query, score, and take the top K.