recallA retrieval
field guide

Deployment Architectures

Build this skill

Choose between local, self-hosted, cloud and hybrid deployments and expose the pipeline as an API.

Where should the system run?

A RAG system has several moving parts: an embedding model, a vector store, possibly a reranker, a language model, and the application that ties them together. Each can run in different places: on your own computers, on servers you rent in the cloud, or as a service you call over the internet. The choice of deployment architecture decides cost, speed, how much the team must maintain, and, often most importantly, where your data goes.

Four common patterns

Local

Everything runs on one machine, such as a laptop or a single office server. Tools such as Ollama make it easy to download and run open language models and embedding models locally with one command. Local deployment is ideal for development, demos, and small private tools. Its limits are the machine's hardware: large models need powerful GPUs, and one machine serves only a limited number of users.

Self-hosted

The organisation runs the models and databases on its own servers or rented cloud machines, typically with GPUs. Open models are served with inference servers such as vLLM, built to serve many requests efficiently. Data never leaves the organisation's control, and there are no per-token fees, but the team must manage hardware, scaling, updates and security.

Cloud APIs

The language model (and often the embedding model and vector database) are managed services called over the internet and paid for per use. There is nothing to run, the strongest models are available, and capacity scales automatically. The costs are usage-based pricing, network latency, dependence on the provider, and sending data to a third party.

Hybrid

A mix: for example, keep documents, embeddings and the vector store in-house, and call a cloud language model only with the few retrieved passages needed for each question; or use a local model for sensitive questions and a cloud model for general ones.

Localone machine (Ollama)dev, demos, small toolslimited by hardwareSelf-hostedown servers + GPUs (vLLM)data stays in-houseteam runs everythingCloud APIsmanaged models + DBsnothing to run, best modelsper-use cost, data leavesHybriddata in-housemodel calls to the cloudor split by sensitivity

The right pattern depends on data rules, scale, budget and the team's skills.

Serving the system as an API

Whatever runs underneath, the RAG application is usually exposed as a small web API, so a website, mobile app or chat tool can call it. Python frameworks such as FastAPI make this straightforward. The sketch below shows the shape of such a service; it runs on your own computer once your own retrieval, prompt and model functions are plugged in.

python
# pip install fastapi uvicorn
from fastapi import FastAPI, Depends
from pydantic import BaseModel

app = FastAPI()

class Question(BaseModel):
    text: str

@app.post("/ask")
def ask(q: Question, user=Depends(current_user)):          # who is asking decides what they may see
    chunks = retrieve(q.text, allowed=user.roles, k=5)      # vector store, filtered by permission
    prompt = build_prompt(q.text, chunks)
    answer = llm(prompt)                                     # local (Ollama/vLLM) or cloud API
    return {"answer": answer, "sources": [c.citation for c in chunks]}

# run with: uvicorn app:app --host 0.0.0.0 --port 8000

Because the language model is called through one function (llm), switching between a local model and a cloud API is a configuration change, not a rewrite. Ollama and vLLM both offer APIs compatible with the widely used OpenAI request format, which makes such swaps easier.

Comparing the options

The program below scores the four patterns against a set of requirements, to show how the decision depends on what matters most.

python · runs in your browser

Expected output:

text
Hybrid       39
Cloud APIs   38
Self-hosted  37
Local        32

Each pattern has a score from 1 to 5 on every requirement, and weights says how much each requirement matters to this organisation. A pattern's total is the sum of score × weight. Change the weights to describe a start-up with no operations team (raise low_ops, lower privacy) and the order changes.

With privacy weighted heavily, the hybrid design comes out narrowly ahead, and all three server-based options score within two points of each other. Such close scores mean the decision really rests on the weights, so change them (for example, a start-up with no infrastructure team and no data restrictions) and cloud APIs win instead.

Practical considerations

  • Latency. Local and self-hosted models avoid network round trips but may run slower models; cloud APIs add network time but run on fast hardware. Measure end to end.
  • Indian data rules. Under the DPDP Act, organisations must think about where personal data is processed and by whom. Keeping personal data in-house and sending only non-personal passages to a cloud model is a common compromise.
  • Availability. Cloud providers have outages and rate limits; a fallback (another provider or a local model) keeps the system answering.
  • Cost curve. Cloud APIs are cheap at low volume and expensive at high volume; self-hosting has high fixed costs and low marginal costs. The crossover point depends on traffic.

Follow your curiosity.

↑ ↓ to explore · Enter to open · Escape to close