recallA retrieval
field guide

Cross-modal Retrieval

Build this skill

Search from text to images and back, using matching encoders and rank-based merging of mixed results.

Searching across kinds of content

Cross-modal retrieval means the query is one kind of content and the results are another. The two common directions are:

  • Text → image: a student types "photos of the new library" and gets photographs; a staff member types "campus map showing the exam cell" and gets the map.
  • Image → text: a student uploads a photo of a notice board, or a screenshot of an error on the portal, and gets the related policy or help article.

Image → image search (find similar photos) and text → text search (ordinary RAG) are the same machinery in a single modality. What makes cross-modal retrieval possible is a shared embedding space, where an image and the text that describes it end up close together.

How it works

Embed imagesimage encoder,ahead of timeStore vectorswith captionsand linksEmbed thequerytext encoder(or imageencoder)Nearestneighbourssame search astext RAGShow resultsimages orpassages

Images are embedded ahead of time with the image encoder; text queries are embedded with the matching text encoder.

The important rule is to use the matching pair of encoders from the same model: images go through that model's image encoder, text through its text encoder. Vectors from different models, or from a text-only embedding model, do not share a space.

With the open-source open_clip library, text-to-image search takes only a few lines. This code runs on your own computer after installing the library and downloading a model:

python
# pip install open_clip_torch torch pillow
import torch, open_clip
from PIL import Image

model, _, preprocess = open_clip.create_model_and_transforms("ViT-B-32", pretrained="laion2b_s34b_b79k")
tokenizer = open_clip.get_tokenizer("ViT-B-32")

paths = ["library.jpg", "hostel_room.jpg", "campus_map.png"]
with torch.no_grad():
    images = torch.stack([preprocess(Image.open(p)) for p in paths])
    image_vecs = model.encode_image(images)
    image_vecs /= image_vecs.norm(dim=-1, keepdim=True)

    query = model.encode_text(tokenizer(["where is the exam cell on campus"]))
    query /= query.norm(dim=-1, keepdim=True)

scores = (image_vecs @ query.T).squeeze()
print(paths[int(scores.argmax())])        # most likely: campus_map.png

Two practical problems

Scores from different modalities are not directly comparable

In a mixed index, a text query is usually much closer to text passages than to any image, simply because text-to-text similarities in these models tend to run higher than text-to-image similarities. If you rank images and passages together by raw score, images may never appear. Common fixes are to search each modality separately and show results in separate groups, or to merge the lists by rank (for example with Reciprocal Rank Fusion) rather than by raw score.

Images need text too

Embedding only the pixels throws away useful information. Store each image with its caption, the surrounding text, a generated description and metadata (where it came from, which page, what date). Then images can also be found by ordinary text and keyword search, and an answer can explain what the image is.

Compare three ways of searching a collection of campus images below.

Image-to-text search in practice

Image queries are increasingly common because phones make them easy. A student photographs a notice and asks "what does this mean for me?", or uploads a screenshot of a portal error. Two approaches work:

  • Embed the image with a multimodal model and search for similar images or related text.
  • Describe or read the image first with a vision-language model ("a notice from the exam cell about re-evaluation deadlines"), then run ordinary text retrieval with that description.

The second is often more reliable for text-heavy images such as notices and screenshots, because reading the text in the image gives precise keywords.

The program below shows the rank-merging fix for mixed results, so that images are not drowned out by text passages with higher raw scores.

python · runs in your browser

Expected output:

text
merged by raw score: ['Exam cell timings (text)', 'Block A directions (text)', 'Exam rules (text)']
merged by rank (RRF): ['Exam cell timings (text)', 'Campus map (image)', 'Block A directions (text)']

The first merge puts everything in one list and sorts by raw score. Text-to-text scores run higher than text-to-image scores, so the text results fill the top. The loop then does Reciprocal Rank Fusion: each result earns 1 / (60 + rank) according to its position within its own list, which puts the best image on the same footing as the best text passage.

By raw score, no image reaches the top three. Merged by rank, the campus map appears alongside the best text result.

Follow your curiosity.

↑ ↓ to explore · Enter to open · Escape to close