Scaling to Millions of Documents
Plan how a RAG system grows to millions of documents with sharding, replicas and incremental indexing.
What changes as the collection grows
A college helpdesk might start with 300 documents. A university system with many affiliated colleges, or a company knowledge base, can reach millions of documents and tens of millions of chunks. Nothing about RAG's basic design changes at that scale, but several practical limits appear: the index no longer fits on one machine, ingesting everything takes days, searches slow down under heavy traffic, and a single server going down takes the whole system with it.
Scaling a retrieval system means handling four kinds of growth: more data (storage and memory), more queries (throughput), more updates (ingestion speed) and higher expectations of reliability.
Splitting the data: sharding
When an index is too big for one machine, it is split into pieces called shards, each holding part of the data on its own machine. A search is sent to every shard in parallel; each returns its local top K; and the results are merged into one global top K. This pattern is sometimes called scatter-gather.
Each shard searches its own part of the data in parallel; the results are merged.
Data can be split in different ways. Random or hash-based sharding spreads chunks evenly, so every shard does similar work. Sharding by tenant or collection (one college or department per shard) keeps related data together and makes permissions and deletions simpler, but shards can become uneven.
Copying the data: replication
Replication keeps several copies of each shard on different machines. It serves two purposes: reliability (if one machine fails, a copy takes over) and throughput (queries can be spread across copies, so more searches run at once). Purpose-built vector databases such as Milvus, Qdrant and Weaviate support sharding and replication across a cluster; with PostgreSQL, read replicas and partitioning provide similar building blocks.
Ingestion at scale
Indexing millions of documents is often a bigger challenge than searching them. The slow steps are parsing (especially OCR) and embedding. Both can be run in parallel across many workers, with a queue feeding them work, so that adding workers speeds things up roughly in proportion. Embedding in batches (many chunks per call) on GPUs is far faster than one at a time. Incremental indexing, which only processes changed documents, keeps daily work small once the initial load is done.
Use the calculator to see how long an initial load takes, and how much parallel workers help.
Search at scale
Search speed at scale depends on the same levers as in a single index (index type and settings, dimensions, quantization) plus the cluster: enough shards that each fits in memory, enough replicas to serve the query rate, and fast merging of shard results. The program below simulates scatter-gather: three shards each return a local top 3, and the merge produces the global top 3.
Expected output:
global top 3: [(0.91, 'regs4.1'), (0.88, 'regs4.2'), (0.83, 'regs4.3')]
Every shard has already sorted its own results, so the merge only has to pool the nine candidates and keep the best three. heapq.nlargest(3, ...) does that without sorting the whole list. Tuples are compared by their first element, which is why the similarity comes first in each pair.
One subtle point: each shard must return at least K results for the merged top K to be correct. If shards return fewer (for example, because of heavy filtering), the global result can miss good matches.
Reliability
At scale, failures are normal: some machine is always slow, restarting or broken. Systems stay reliable by having replicas for every shard, timeouts so one slow shard cannot stall every search, retries for brief failures, graceful degradation (if a shard is down, answer from the others and say results may be incomplete, rather than failing completely), and backups of both the index and the source data.