Enterprise & Sovereign AI: Run AI on Infrastructure You Control
Written by Jakub Rusinowski · Last updated 2026-07-12 · Hardware figures computed by our VRAM engine
Running AI on your own infrastructure means open-weight models (Qwen, Llama, Mistral, GPT-oss) served from hardware you control — a workstation for a team, a GPU server for a company. Businesses do it for three reasons: data that must not leave the building, cost at sustained volume, and independence from any single vendor or jurisdiction. This hub covers the why (sovereignty, compliance, economics) and the how (hardware, models, deployment, vendor selection) — with every hardware figure computed by the same engine as our GPU checker.
Two shifts made this page necessary. First, production AI inference is moving out of the public cloud: in Broadcom's *Private Cloud Outlook 2026* — a survey of 1,800 senior IT decision-makers at enterprises across eight countries — the share using public cloud as the primary environment for production AI inference fell from 56% to 41% in a single year, while 56% now run or plan production inference on private cloud. Second, AI has become a sovereignty question: 77% of organizations now factor an AI vendor's country of origin into selection decisions (Deloitte, *State of AI in the Enterprise* — survey of 3,235 business and IT leaders across 24 countries, late 2025).
Neither shift is ideological. Inference at business volume is a predictable, steady-state workload — exactly the kind that owned hardware prices well against metered APIs — and the data flowing through internal AI assistants is exactly the data legal teams least want crossing borders or third-party processors. The result is that "can we run this ourselves?" has become a normal procurement question, asked by IT managers and compliance leads rather than enthusiasts.
This hub is written for that question. It is vendor-neutral: we don't sell hardware, platforms, or models — the recommendations come from the same open compute engine that powers this site's GPU compatibility checker and cost calculator, and where a page touches regulation it describes mechanics, not legal advice. Start with the guide that matches where you are: understanding the landscape, pricing a deployment, or choosing between vendors.
What business hardware actually runs
| Model | VRAM (Q4) | Runs on | Context | License |
|---|
Qwen 3 32B
Department tier — one 24–48 GB workstation — The proven internal-assistant class: summarization, drafting, RAG over internal documents. Apache 2.0.
ollama pull qwen3:32b | 20 GB | 24 GB GPU (RTX 3090/4090)
Mac: 32 GB unified | 128,000 | Apache 2.0 |
Llama 3.3 70B Instruct
Company tier — one 48–80 GB card — The reference open 70B for knowledge work; runs fully on a single RTX 6000 Ada / A100-class card.
ollama pull llama3.3:70b | 43.1 GB | 2×24 GB GPUs or 48 GB card
Mac: 64 GB unified | 128K | Llama Community |
GPT-oss 120B
Server tier — one 80–96 GB card — GPT-4-class benchmark scores from an open-weight model on a single H100 or RTX PRO 6000.
ollama pull gpt-oss:120b | 73.3 GB | 2×48 GB GPUs / big unified memory
Mac: 128 GB unified | 125K | Apache-2.0 |
Qwen 3 235B-A22B (MoE)
Multi-GPU tier — 2–4 server cards — Frontier-adjacent MoE quality for organizations that need the ceiling in-house.
ollama pull qwen3:235b-a22b | 80 GB | 2×48 GB GPUs / big unified memory
Mac: 128 GB unified | 128,000 | Apache 2.0 |
The guides
- What Is Sovereign AI — and Why Country of Origin Now Matters — The four layers of AI sovereignty, why 77% of firms now vet vendor origin (Deloitte), and the practical control spectrum.
- On-Prem & Private AI: The 2026 Data — Every credible 2026 number on the inference shift, sovereignty screening, and cost — sourced, methodology included, one myth debunked.
- On-Prem LLM Total Cost of Ownership vs Cloud APIs at Company Scale — The four cost lines, two worked examples (1M and 20M tokens/day), and the honest column where cloud stays cheaper.
- On-Prem AI Hardware for Business: 10, 50, and 200-Seat Tiers — Sizing by headcount: one workstation for 10 seats, one pro card for 50, a GPU server for 200+ — with verifiable VRAM math.
- Choosing an On-Prem AI Vendor: The 30-Point Evaluation Checklist — 30 RFP-ready questions across six dimensions — data path, model rights, sizing honesty, exit terms — with the disqualifying answers.
- On-Prem LLM Deployment Architecture: vLLM, SSO, and Logging — The production architecture in six steps: vLLM or Ollama behind an SSO gateway with audit logging — an engineer-week, not a program.
- Air-Gapped LLM Deployment for Healthcare and Legal — Who actually needs the air gap (healthcare and legal, specifically), why LLMs air-gap unusually well, and the six-step enclave build.
- Open-Model Licenses for Commercial Use: What Legal Will Ask — Apache/MIT vs Llama/Gemma community licenses — the five questions legal will ask, answered before they ask them.
- The EU AI Act and Self-Hosted AI: What Changes in August 2026 — The August 2026 timeline, deployer vs provider roles, and what self-hosting changes under the Act (evidence) vs what it doesn't (obligations).
Rolling this out in your organization?
Jakub Rusinowski, the founder of LLM Configurator, runs corporate workshops and lectures on deploying local LLMs — hardware sizing, model selection, compliance-friendly architectures, and hands-on setup for your team. Direct, vendor-neutral, practitioner-level.
Ask about a workshop →
Frequently asked questions
What does "sovereign AI" actually mean for a business?
Keeping the four layers of an AI system — the data, the models, the compute, and day-to-day operational control — inside infrastructure and jurisdictions you choose. For a company that usually means open-weight models on owned or EU-hosted hardware instead of a foreign cloud API. It became a mainstream procurement criterion in 2025–2026: 77% of organizations now factor an AI vendor's country of origin into selection decisions, per Deloitte's State of AI in the Enterprise survey of 3,235 business and IT leaders.
Is on-premise AI actually cheaper than cloud APIs?
At sustained volume, usually yes. Inference at business scale is a steady-state workload: a one-time hardware cost amortized over 2–3 years plus electricity, versus a metered per-token bill forever. At roughly 1M tokens/day a single-workstation deployment already undercuts frontier API pricing; at company scale the gap widens. Below a few hundred thousand tokens a day, cloud APIs stay cheaper — run your own volumes in our break-even calculator before deciding.
What hardware does a company need to run AI on-premise?
By team size: a 24–48 GB GPU workstation (RTX 4090 / RTX 6000 Ada class) serves a department of ~10 with a 32B-class assistant; a single 48–96 GB pro card (RTX PRO 6000, A100/H100) serves ~50 with a 70B-class model; a 2–8 GPU server serves hundreds with frontier-adjacent MoE models. Every figure on this hub comes from the same VRAM engine as our free compatibility checker, so you can verify any configuration yourself.
Are open-weight models good enough for business use in 2026?
For the workloads businesses actually deploy — internal assistants, document summarization, RAG over company knowledge, drafting, coding help — yes. Qwen 3 32B and Llama 3.3 70B handle these well on single-card hardware, and GPT-oss 120B matches GPT-4o on standard benchmarks from one server GPU. Frontier cloud models keep an edge on the hardest reasoning tasks; most organizations pilot with their real documents before committing either way.
Does running AI on-premise automatically solve GDPR and compliance?
No — but it removes the hardest parts. With inference on your own hardware there is no third-party model processor, no cross-border transfer analysis for the inference itself, and no dependence on a vendor's retention terms. You still need lawful basis, access controls, and sensible logging — internal controls you already operate for other systems. Our GDPR comparison page walks through the mechanics; involve your DPO for specifics.
Keep going