Ollama Stack Docker: Run AI Models Locally with a ChatGPT-Like Interface

Run powerful AI models locally with a ChatGPT-like interface. Private, offline, and portable.

One command to deploy Ollama + Open WebUI with GPU support, RAG (chat with your documents), and 12 pre-configured models covering general chat, coding, medical, vision, multilingual, and more.

Features

• Private & Offline — Everything runs locally. No data leaves your machine.
• Chat with Documents (RAG) — Upload PDFs, DOCX, TXT, CSV files and ask questions about them.
• 12 Pre-configured Models — General, coding, medical, reasoning, vision, multilingual, and math.
• GPU Accelerated — Automatic NVIDIA GPU detection. Falls back to CPU seamlessly.
• Portable — Copy the folder to any machine with Docker. One command to start.
• Web Interface — Beautiful ChatGPT-like UI accessible at http://localhost:3000.
• Multi-user — Create accounts, share models, manage access.
• Arena Mode — Compare multiple models side-by-side on the same prompt.
• Knowledge Collections — Organize documents into domain-specific collections for focused AI assistants.

Quick Start

Prerequisites

• Docker installed and running
• At least 16 GB RAM (32 GB recommended)
• NVIDIA GPU optional but recommended for faster inference

1. Clone and Start

git clone https://github.com/robomixes/ollama-stack-docker.git
cd ollama-stack-docker
docker compose up -d

2. Pull Models

bash setup-models.sh

This pulls all models listed in models.txt (~40 GB total). Takes 30-60 minutes depending on your internet speed.

3. Open the UI

Go to http://localhost:3000 and create your admin account (first registration = admin).

Included Models

Edit models.txt and re-run bash setup-models.sh to add or remove models. Browse all available models at ollama.com/library.

Chat with Documents (RAG)

Open WebUI includes built-in RAG (Retrieval-Augmented Generation) for chatting with your own documents.

Quick Upload

1. Open any chat
2. Click the + button in the message input
3. Upload a PDF, TXT, DOCX, CSV, or Markdown file
4. Ask questions about the document

Knowledge Collections

For organizing multiple documents into a domain-specific knowledge base:

1. Go to Workspace > Knowledge
2. Click + Create Knowledge
3. Name it (e.g., "Medical Research", "Legal Docs", "Company Wiki")
4. Upload multiple files
5. In any chat, type # to attach a knowledge collection

RAG Settings

Default settings work well. To customize, go to Admin Panel > Settings > Documents:

Create Domain-Specific Bots

Build specialized AI assistants by combining a model with a knowledge collection:

1. Go to Workspace > Models
2. Click + Create a Model
3. Configure: Name, Base Model, Knowledge, and System Prompt

Example: Medical Bot

Base Model: meditron:7b
Knowledge: Your medical textbooks and guidelines
System Prompt: You are a medical knowledge assistant. Answer based on the
provided documents. Always note when information should be verified by a
healthcare professional.

Example: Coding Assistant

Base Model: qwen2.5-coder:7b
Knowledge: Your codebase documentation and API docs
System Prompt: You are a senior software engineer. Help with code reviews,
debugging, and implementation. Reference the provided documentation when relevant.

Example: Geopolitical Analyst

Base Model: deepseek-r1:8b
Knowledge: Think tank reports, foreign policy journals, country profiles
System Prompt: You are a geopolitical analyst. Analyze questions considering
multiple perspectives, historical context, and power dynamics. Distinguish
facts from assessments.

Architecture

Browser --> Open WebUI (port 3000) --> Ollama (port 11434)
                |                           |
                |                      GPU/CPU inference
                |
            RAG Pipeline
            (ChromaDB + nomic-embed-text)

Both services run as Docker containers on a shared network. Data persists in Docker volumes.

GPU Support

GPU VRAM Guide

Models that don't fit in VRAM automatically offload layers to CPU (hybrid mode).

Hardware Requirements

Deploy to Cloud

This stack runs anywhere Docker runs. Copy the folder and start.

Azure VM (GPU)

# Create NC-series VM (GPU), SSH in, then:
sudo apt update && sudo apt install -y docker.io nvidia-container-toolkit
sudo systemctl restart docker
git clone https://github.com/robomixes/ollama-stack-docker.git
cd ollama-stack-docker
docker compose up -d
bash setup-models.sh
# Access at http://<VM-IP>:3000

Any Linux Server

git clone https://github.com/robomixes/ollama-stack-docker.git
cd ollama-stack-docker
docker compose up -d
bash setup-models.sh

Commands Reference

# Start the stack
docker compose up -d

# Stop the stack
docker compose down

# View running containers
docker compose ps

# Pull/update models
bash setup-models.sh

# Pull a specific model manually
docker exec ollama ollama pull <model-name>

# List installed models
docker exec ollama ollama list

# Test a model
docker exec ollama ollama run llama3.2:3b "Hello!"

# View Ollama logs
docker logs ollama

# View Open WebUI logs
docker logs open-webui

# Check GPU usage
nvidia-smi

# Update images to latest
docker compose pull && docker compose up -d

Troubleshooting