Audio and Video Retrieval
Make recordings searchable with timestamped transcripts, speaker labels and keyframes, and cite moments precisely.
Knowledge that was spoken, not written
A surprising amount of an organisation's knowledge is never written down. The principal's announcements at convocation, the academic council's discussion of new attendance rules, recorded lectures, orientation sessions for first-years, placement preparation webinars, support calls: all of it is audio or video. Students search for "what did the dean say about the exam schedule?" or "the lecture where the professor explained deadlock", and a text-only knowledge base has nothing to offer.
Audio and video retrieval makes spoken content searchable. The usual approach is simple in outline: turn speech into text with timestamps, split that text into chunks, and index it like any other text, keeping a link back to the exact moment in the recording.
From speech to searchable text
Speech becomes timestamped text; each chunk remembers where it is in the recording.
Transcription (also called automatic speech recognition, or ASR) converts speech to text. Whisper, released by OpenAI in 2022 as an open model, is widely used: it was trained on a very large amount of multilingual audio, handles many languages including Hindi, Tamil and Telugu, and outputs timestamps for each segment of speech. Cloud providers and Indian language initiatives also offer speech recognition services, some tuned specifically for Indian languages and accents.
Speaker diarization answers "who spoke when": it labels segments as Speaker 1, Speaker 2 and so on, which can then be mapped to names (the dean, a student). This matters for questions like "what did the dean say about attendance?".
Segmentation splits the transcript into chunks. Spoken language has no paragraphs or headings, so the usual approaches are fixed time windows (say 60 seconds, with some overlap), splitting at pauses or speaker changes, or semantic chunking that detects topic shifts.
Timestamps make answers useful
Every chunk keeps its start and end time. When a search finds a chunk, the answer can cite "Academic council meeting, 14 October, 23:40–24:35", and the interface can jump straight to that moment in the recording. Without timestamps, a user would have to scrub through a two-hour video to check the answer.
The program below takes timestamped transcript segments, groups them into chunks of about one minute, and searches them, returning the time range to jump to.
Expected output:
[00:47–01:35] Registrar: Medical leave will continue to be added back to attendance. Dean: Second item: the end-semester exams will start on 2 December.
[01:35–02:00] HoD CSE: Lab exams for CSE will be held one week earlier.
In to_chunks, a new chunk starts whenever adding the next segment would stretch the current chunk past the window (seg[1] - current[0][0] > window). Each chunk therefore knows the start time of its first segment and the end time of its last, and mmss turns those seconds into minutes and seconds for display.
The search returns the two stretches of the meeting that mention exams, each with its exact time range, so a student can jump straight to 00:47 and hear the dean announce the exam date. Notice that the first chunk also contains the registrar's remark about medical leave, because the one-minute window happened to include it. The lab lets you change that window on a longer recording.
Video adds pictures
Video carries more than its soundtrack: slides shown in a lecture, diagrams on a whiteboard, text on screen. Two common additions handle this:
- Keyframes: sample frames at regular intervals or at scene changes, and either describe them with a vision-language model or read their text with OCR (very useful for lecture slides).
- Multimodal embeddings of keyframes, so a text query like "slide with the deadlock diagram" can find the right moment visually.
Each keyframe description is stored with its timestamp, alongside the transcript chunks for the same stretch of time.
Quality and care
Transcription has its own error patterns. Names, technical terms and code-mixed speech ("next semester se attendance 75 percent hi rahega") are where speech recognition struggles most. Providing a list of expected terms (course codes, names of people and places) where the tool supports it, choosing a model that handles the languages spoken, and spot-checking transcripts all help.
Recordings also raise privacy questions. A recorded meeting may contain personal discussion that was never meant to be searchable by everyone. The same rules apply as for documents: decide what should be indexed at all, mask or restrict sensitive parts, and carry access labels with every chunk.