Multimodal Embeddings
Explain how CLIP-style models learn a shared image–text space and where they work well or poorly.
One space for words and pictures
Text embeddings place sentences with similar meanings close together. Multimodal embeddings extend the same idea across kinds of data: a photo of the library, the phrase "central library building" and the sentence "the library is open until 10 PM" can all become vectors in one shared space, where things that mean the same are close together whatever form they take. Once images and text share a space, a text question can find an image, and an image can find related text, using exactly the same nearest-neighbour search as before.
How a shared space is learned: CLIP
The best-known model of this kind is CLIP (Contrastive Language–Image Pre-training), released by OpenAI in 2021. CLIP has two encoders: one turns images into vectors, the other turns text into vectors of the same length. It was trained on about 400 million pairs of images and their captions collected from the internet.
Training used contrastive learning, the same idea used for text embedding models. In each batch, the model sees many image–caption pairs. It is rewarded for making each image's vector close to its own caption's vector, and far from the other captions in the batch. After enough examples, a photo of a library sits near "a library with bookshelves", and far from "a hostel room".
Each encoder maps its own kind of input into the same vector space, so matching images and texts end up close together.
SigLIP, from Google (2023), is a later model in the same family. It changes the training loss so that each image–text pair is scored independently (using a sigmoid function), which trains efficiently and performs strongly. OpenCLIP is an open-source project that re-trained CLIP-style models on large public datasets, making many sizes freely available.
Seeing a shared space
The toy plot below places a few images and texts in a hand-made two-dimensional space. Pick a text query and see which images and texts are its nearest neighbours.
Where multimodal embeddings work well, and where they don't
CLIP-style models are excellent at matching photos and natural scenes with descriptions: "a crowded canteen", "a laptop on a desk", "students in a lab". They are weaker on text-heavy images such as scanned pages, tables and charts, because their training data was mostly ordinary photos with short captions. Reading the numbers on a chart or the words on a dense page is a different skill. For document pages, models trained specifically on document images, or vision-language parsing into text, usually do better.
Two more practical points: multimodal embeddings are often shorter or less precise than the best text-only embeddings for text-to-text search, so many systems keep a strong text embedding model for text and use a multimodal model only for images. And, as always, embeddings from different models live in different spaces and cannot be compared.
Measuring the shared space
The program below builds a tiny "similarity matrix" between three images and three captions, using hand-made vectors, and checks that each image's best match is its own caption, which is exactly what contrastive training aims for.
Expected output:
a library wi a hostel roo bar chart of
library photo 0.99 0.28 0.23 best: a library with bookshelves
hostel room photo 0.35 0.99 0.42 best: a hostel room with two beds
placements chart 0.34 0.25 0.99 best: bar chart of students placed
Each row of the output is one image. iv @ cv is the similarity between that image's vector and one caption's vector, so the row holds three scores. np.argmax(sims) finds the position of the largest one, which is used to look up the best-matching caption.
The highest number in each row is on the diagonal: each image is closest to its own caption.