Written by Jakub Rusinowski · Last updated 2026-07-12 · Hardware figures computed by our VRAM engine
A production on-prem LLM deployment is four boxes: a serving engine (vLLM for multi-user throughput, Ollama for simplicity), an OpenAI-compatible gateway that adds SSO and per-user audit logging, an optional RAG layer over internal documents, and monitoring wired into what you already run. Total effort is about an engineer-week on hardware you've sized correctly — the six steps below take you from bare server to a logged, authenticated internal endpoint your existing tools can call.
Every workable business deployment converges on the same shape:
Users and internal apps → gateway (auth + logging + routing) → serving engine (vLLM/Ollama) → GPU. Beside the serving engine sits the optional RAG stack: an embedding model plus a vector database that the gateway consults before prompting. Everything speaks the OpenAI API dialect, which is the quiet superpower of the whole exercise: internal tools, IDE plugins, and off-the-shelf UIs (Open WebUI is the common choice) work unmodified against your endpoint, and if you ever change serving engines — or burst an approved workload to a cloud API — nothing downstream notices.
The design decision that matters most is putting the gateway in from day one. Teams that expose the serving engine directly always retrofit authentication and logging later, under audit pressure, badly. The gateway is where compliance lives: who asked what, when, and what came back. Get it in place before the first real user, and the GDPR story or Security-Rule audit answer writes itself.
| Ollama | vLLM | |
|---|---|---|
| Sweet spot | Tier-1 workstation, ≤ ~10 users | Tier-2/3 servers, real concurrency |
| Concurrency model | Sequential-ish; fine for light interleaving | Continuous batching — designed for parallel users |
| Model format | GGUF (quantized, easy) | safetensors (GPU-native, faster at scale) |
| Ops burden | Near zero — one binary, model pull built in | A service to run properly (container, pinned versions) |
| Quantization | Q4/Q5/Q8 GGUF out of the box | FP8/AWQ/GPTQ — better throughput on pro cards |
Decision rule: Ollama until concurrent users make people wait; vLLM the moment they do. A department workstation serving five people is a perfect Ollama deployment and gains nothing from vLLM's complexity. A 50-seat company on one RTX PRO 6000 wants vLLM's batching — it is the difference between one card serving the company and a queue forming at 9 AM. Because both sit behind the same gateway API, migrating later is a config change, not a project — so start simple without fear of lock-in.
Confirm the model/GPU pairing with the compatibility checker and the hardware tiers guide — VRAM for weights plus KV-cache headroom for your context lengths. Provision Linux (Ubuntu LTS is the well-trodden path), current NVIDIA drivers, and NVMe space for models (20–120 GB each; keep two generations). Block outbound egress at the firewall now — the deployment should be born with the network posture it will keep.
Tier 1: install Ollama, pull your model, confirm /v1/chat/completions answers locally. Tier 2/3: run vLLM in a container with pinned versions, serving your model in safetensors with an appropriate quantization for your card (FP8 on Hopper/Blackwell-class hardware). Either way the deliverable is identical: an OpenAI-compatible endpoint on localhost, benchmarked with a quick load test at your expected concurrency so you know your real tokens/sec before users define it for you.
Deploy a reverse-proxy gateway (LiteLLM proxy is the common open-source choice; a plain nginx + OIDC layer also works) that: authenticates against your identity provider (SAML/OIDC — the SSO your company already has), issues per-team API keys for service accounts, and writes an audit record per request — user, timestamp, model, request/response references — to your SIEM or log store. Set log retention deliberately (prompts may contain personal data; your retention policy applies to them). This step is what makes the deployment defensible; do not defer it.
Run an embedding model beside the LLM and a vector database (Qdrant or pgvector — both self-hosted, both boring in the good way). Ingest the document corpus with access-control metadata so retrieval respects existing permissions — the assistant must not become the tool that shows anyone anything. Our local RAG guide covers the stack in detail; the business-grade addition is that permission filter.
Point the interfaces at the gateway: Open WebUI for chat, IDE plugins for developers (the coding hub covers that stack), and internal apps via the OpenAI-compatible API with service-account keys. Because everything upstream speaks one dialect, each new consumer is a base-URL configuration, not an integration project.
Export the numbers that predict trouble — GPU memory utilization, queue depth, tokens/sec, per-team request volume — into the monitoring you already run (Prometheus exporters exist for both engines). Then write the runbook: how to restart serving, how to roll a model version forward and back, where logs live, who owns quarterly updates. Two pages, tested once by someone who didn't write it. This is the difference between a deployment and a science project.
Jakub Rusinowski, the founder of LLM Configurator, runs corporate workshops and lectures on deploying local LLMs — hardware sizing, model selection, compliance-friendly architectures, and hands-on setup for your team. Direct, vendor-neutral, practitioner-level.