Building a Self-Hosted AI Stack: Ollama, Open WebUI, and Local RAG

A self-hosted AI stack is the practical end state of running models locally: instead of a terminal and ollama run, you get a chat UI, document retrieval, per-user access, and an API surface for the rest of your tools — all on hardware you control, with zero data leaving your network. This guide builds that stack with Ollama, Open WebUI, local embeddings, and a reverse proxy, and covers the operational decisions (storage, auth, upgrades) that make it run for months instead of days.

Architecture

┌─────────────┐      ┌─────────────────┐      ┌──────────────┐
│  Browser    │─────►│  Open WebUI     │─────►│  Ollama      │
│  (LAN/VPN)  │      │  (chat, auth)   │      │  :11434      │
└─────────────┘      └────────┬────────┘      └──────────────┘
                              │
                        ┌─────▼──────┐      ┌──────────────┐
                        │ Embeddings │─────►│ Vector DB    │
                        │ (local)    │      │ (Chroma)     │
                        └────────────┘      └──────────────┘

The stack has four moving parts: Ollama serves the models, Open WebUI is the user-facing chat and document layer, an embedding model plus vector store powers RAG, and a reverse proxy (Caddy or nginx) handles TLS and routing. Everything runs on one machine to start; each piece detaches cleanly when you outgrow it.

Step 1: Ollama as the Inference Backend

Install Ollama and serve it on the network interface (default is localhost-only):

curl -fsSL https://ollama.com/install.sh | sh
ollama pull qwen3-coder:30b
ollama pull nomic-embed-text    # 274MB, the RAG embedding default

By default Ollama binds to 127.0.0.1, which is fine because only Open WebUI and your tools on the same host need it. Do not expose :11434 directly to your LAN; put it behind the reverse proxy with auth if remote access is required. Persist models across upgrades with:

export OLLAMA_MODELS=/mnt/ai/ollama   # big disk, not the boot volume

Ollama keeps the full OpenAI-compatible surface at /v1, so Open WebUI and any other client speak a protocol they already know.

Step 2: Open WebUI as the Interface

Open WebUI is a self-hosted, extensible chat interface that connects to any OpenAI-compatible backend. The Docker install is the supported path:

# docker-compose.yml
services:
  ollama:
    image: ollama/ollama:latest
    volumes:
      - ollama:/root/.ollama
    restart: unless-stopped

  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    ports:
      - "3000:8080"
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
    volumes:
      - open-webui:/app/backend/data
    extra_hosts:
      - "host.docker.internal:host-gateway"
    restart: unless-stopped

volumes:
  ollama:
  open-webui:

Open at http://localhost:3000, create the admin account on first run, then set the model of choice in the model selector. Open WebUI also supports user accounts with roles — start with admin and add users as needed rather than sharing one login.

Step 3: Local RAG with Embeddings

RAG turns your docs into answers grounded in your own data. With Ollama's embedding model and a local vector store, nothing leaves the machine:

import chromadb
from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
chroma = chromadb.PersistentClient(path="./vecdb")

def embed(text: str) -> list[float]:
    resp = client.embeddings.create(model="nomic-embed-text", input=text)
    return resp.data[0].embedding

def ingest(collection_name: str, documents: list[str]):
    collection = chroma.get_or_create_collection(collection_name)
    collection.add(
        ids=[f"doc-{i}" for i in range(len(documents))],
        documents=documents,
        embeddings=[embed(d) for d in documents],
    )

def answer(collection_name: str, question: str) -> str:
    collection = chroma.get_collection(collection_name)
    hits = collection.query(query_embeddings=[embed(question)], n_results=5)
    context = "\n\n".join(hits["documents"][0])
    response = client.chat.completions.create(
        model="qwen3-coder:30b",
        messages=[
            {"role": "system", "content": "Answer using only the context. If the context lacks the answer, say so."},
            {"role": "user", "content": f"Context:\n{context}\n\nQuestion: {question}"},
        ],
    )
    return response.choices[0].message.content

Open WebUI has document upload built in, but for programmatic RAG (docs synced from a repo, internal wikis), run ingestion as a scheduled job: chunk documents by ~512 tokens with overlap, store the chunk source with each vector, and re-ingest only what changed. Chunking strategy matters more than the vector store; fixed-size chunks with 10-15% overlap is the sane default (see our chunking strategies guide).

Embedding Model Choice

nomic-embed-text (274MB, 8,192 token chunking) is the default for code and technical docs. It is a real model, not a toy, and it keeps the whole stack on one machine. If you later move RAG to a production server, swap to a hosted embedding API — the OpenAI-compatible contract means only the base_url changes.

Step 4: Secure Remote Access

Do not port-forward Open WebUI to the internet without a plan. Two sane options:

Option A: Tailscale (or any mesh VPN) — clients join your tailnet and reach the stack at a private IP. No exposed ports, no TLS configuration, and identity is your Tailscale account:

tailscale up
# Access at http://<machine-ip>:3000 from any device on the tailnet

Option B: Caddy reverse proxy with TLS and auth — for a public-facing instance, terminate TLS and require authentication in front of Open WebUI:

chat.example.com {
    reverse_proxy localhost:3000
    basic_auth {
        family $2a$14$...   # bcrypt hash, not plaintext
    }
}

Caddy issues and renews Let's Encrypt certificates automatically. Either way: Open WebUI already enforces its own user accounts, so the proxy auth is defense in depth, not the only gate.

Operational Notes

• Storage: models and the vector DB are the expensive bits. Keep OLLAMA_MODELS and the Chroma path on a fast SSD with headroom; a 30B model plus docs will pass 100GB quickly.
• Upgrades: pin the Docker image tag you test against. :main tracks bleeding edge; move to a tagged release when you want stability.
• Backups: the Open WebUI data volume holds users, chats, and settings. Back it up; re-creating it from scratch is not fun.
• Monitoring: ollama ps shows what is loaded and its memory split. If a model is evicted between requests, either keep-alive it (--keep-alive 30m) or reduce the number of concurrently loaded models.
• Multi-model: load one chat model plus the embedding model. Ollama swaps models on demand, but every swap reloads weights; --keep-alive avoids thrash for your primary model.

When to Move Off the Single Box

The single-machine stack is a great product; it is a fragile service. When multiple people depend on it daily, move inference to a machine with a real GPU or a GPU provider running vLLM (see the Ollama vs vLLM vs llama.cpp guide), keep Open WebUI pointing at it, and move the vector store to Postgres + pgvector. The stack survives the migration because every boundary speaks the same OpenAI-compatible protocol.

Implementation Checklist

• Install Ollama with models on a large, fast disk
• Deploy Open WebUI via Docker with persistent volumes
• Create per-user accounts in Open WebUI, not shared logins
• Add nomic-embed-text for local embeddings and a persistent vector store
• Run RAG ingestion as a scheduled chunking job, not ad hoc
• Reach the stack via Tailscale or Caddy + auth; never expose Ollama directly
• Pin image tags, back up the Open WebUI data volume, and set keep-alive on the primary model
• Plan the upgrade path to vLLM + pgvector before the stack becomes a dependency

---

MatterAI builds frontier AI infrastructure for engineering teams — from inference-optimized models to autonomous coding agents and agentic code reviews.

Explore what we're building:

• Orbital IDE — Autonomous AI coding agent with background agents and deep codebase memory
• AI Code Reviews — Agentic pre-commit reviews across GitHub, GitLab, and Bitbucket
• Axon Models — Frontier-grade reasoning models at 70% lower inference cost

Get started free - https://app.matterai.so

• Read the docs
• View pricing

---

Follow us on X · LinkedIn · GitHub