recallA retrieval
field guide

How Embedding Models Learn Meaning

Build this skill

Describe how bi-encoders are trained with contrastive learning, and why input limits and query prefixes matter.

How does a model learn what “similar” means?

An embedding model places "sick days" near "medical leave". Nobody typed in a rule saying these phrases are related. The model learned it from examples, the same way a child learns that "puppy" and "dog" are close by hearing them used in similar situations. Understanding how this learning works explains both why embeddings are so useful and why they sometimes fail.

The core idea is old and simple: words and sentences that appear in similar contexts tend to have similar meanings. A linguist, J. R. Firth, put it in 1957 as "you shall know a word by the company it keeps". "Medical leave" and "sick days" both appear near words like doctor, certificate, absent and days off. A model that learns from millions of such examples ends up placing them close together.

The bi-encoder: one text in, one embedding out

Most embedding models used for search today are bi-encoders. The name comes from how they are used: the same model encodes the query on its own and each document on its own, producing one embedding for each. Search then only compares the finished embeddings.

Inside, the model is a transformer, the same family of neural network used in large language models. It reads the whole text at once, lets every word look at every other word to work out its meaning in context, and then squeezes the result into a single fixed-length list of numbers. This last step is often called pooling, and a common version simply averages the information from all the words.

Text“sick dayspolicy”Split intotokenssick · days ·policyTransformereach word readsthe othersPoolcombine intooneEmbedding1,024 numbers

The same model turns queries and documents into embeddings separately, so documents can be embedded once, ahead of time.

Encoding each text separately is what makes bi-encoders practical for search. All the documents can be embedded once, ahead of time, and stored. When a query arrives, only the query needs to be embedded, which takes a few milliseconds, and then it is compared against the stored embeddings.

Contrastive learning: pull together, push apart

How does the model learn where to place things? Modern embedding models are trained with contrastive learning. The training data is a huge collection of pairs that belong together: a question and the passage that answers it, a title and its article, a search query and the page people clicked on. For each pair, the model is rewarded for placing the two embeddings close together, and for placing them far from other, unrelated texts in the same batch.

It is like a teacher handing out cards to a class, half with questions and half with answers, and asking students to find their partner. Each round, the pairs that found each other are encouraged to stand closer, and everyone else is gently pushed away. After millions of rounds, every question stands near its answers.

Pulled together (positive pairs)“sick days?” ↔ medical leave rule“hostel curfew” ↔ gates close at 10 PM“rechecking” ↔ re-evaluation rulePushed apart (negatives)“sick days?” ↔ library fine“hostel curfew” ↔ scholarship rules“rechecking” ↔ Wi-Fi help

Training pulls matching pairs closer and pushes unrelated texts apart. Repeated millions of times, this shapes the whole embedding space.

The most useful negatives are the tricky ones, called hard negatives: texts that look similar but are wrong, such as the attendance rule for a question about medical leave limits. Training on hard negatives teaches the model fine distinctions instead of just broad topics.

This also explains a weakness. A model can only learn the relationships present in its training data. If it was trained mostly on general English web pages, it may never have seen that "re-evaluation" is what Indian universities call rechecking a paper, or that "backlog" means a failed subject rather than a pile of work.

Queries and documents are not the same kind of text

A question like "how many sick days?" is short and informal. The passage that answers it is long and formal. Many modern embedding models therefore expect you to label which is which, by adding a short instruction or prefix before the text. Some models expect "query: " and "passage: ", others expect an instruction such as "Represent this question for retrieving relevant documents". Each model's documentation says what it expects, and skipping the prefix can noticeably reduce search quality.

Try matching the training ideas to what they achieve.

Limits worth knowing

Every embedding model has a maximum input length, measured in tokens (small pieces of words). Older models stop reading after 512 tokens, roughly a page of text; newer ones accept 8,192 tokens or more. Text beyond the limit is usually cut off silently, so the end of a long document simply does not count. This is one reason documents are split into smaller passages before embedding.

A single embedding also has to summarise everything in its text. A page about hostel fees, hostel timings and mess menus gets one embedding that is a blend of all three, and may not be very close to any one specific question. Shorter, focused passages tend to get sharper embeddings.

Finally, embeddings from different models are not compatible. Each model builds its own space, so an embedding from one model cannot be compared with an embedding from another. If you switch models, every stored document has to be embedded again.

You can see the effect of this "blending" with a small calculation. Below, a page that covers three topics is represented as the average of three topic embeddings, and compared with a focused page.

python · runs in your browser

Expected output:

text
focused page: 1.0
mixed page:   0.61

(fees + timings + menu) / 3 averages the three topic directions, which is roughly what happens when one embedding has to stand for a page about all three. The query points almost entirely along fees, so it lines up with the focused page and only partly with the blend.

The focused page scores much higher, even though the mixed page contains the same fee information. That is the main reason passages are kept focused before they are embedded.

Follow your curiosity.

↑ ↓ to explore · Enter to open · Escape to close