Written by Jakub Rusinowski · Last updated July 10, 2026
Last Updated: May 2026 — Businesses face a difficult choice: use powerful cloud AI services that process data on external servers, or run AI locally and maintain full data sovereignty. This guide expl
Last Updated: May 2026 — Businesses face a difficult choice: use powerful cloud AI services that process data on external servers, or run AI locally and maintain full data sovereignty. This guide explains how to set up private, compliant AI for business use.
The compliance case for local AI has never been stronger:
Under GDPR, you are the data controller. Using a cloud AI API makes that provider a data processor — requiring a Data Processing Agreement (DPA). Key requirements:
| Requirement | Cloud AI Risk | Local AI Solution |
|---|---|---|
| Data minimization | Hard to guarantee with opaque APIs | Full control — process only what you need |
| Right to erasure | Provider may retain training data | Data never leaves your premises |
| Data transfer restrictions | US providers may not comply | No transfer, no restriction |
| Processing records | API logs may be incomplete | Full audit logs on your own systems |
| Breach notification | You depend on provider's detection | You control all monitoring |
For healthcare organizations, local AI avoids the complex BAA process:
HIGH RISK (always use local AI):
- Patient records, diagnoses, treatment plans
- Employee personal information, HR files, salary data
- Client/customer PII (names, addresses, IDs)
- Legal documents, contracts, privileged communications
- Financial statements, unpublished earnings, M&A plans
- Source code with proprietary algorithms
- Authentication credentials, API keys
MEDIUM RISK (evaluate case by case):
- Anonymized business data
- Internal process documentation
- Non-sensitive customer support templates
- Public-facing marketing content
LOW RISK (cloud AI generally acceptable):
- Editing publicly available content
- General research on public topics
- Grammar/style checking of non-sensitive textA dedicated server in your office or data center running Ollama with an authentication layer:
Hardware recommendations:
Software stack:
# Install Ollama
curl -fsSL https://ollama.com/install.sh | sh
# Set Ollama to listen on LAN only (not public internet)
export OLLAMA_HOST=192.168.1.100:11434
# Start Ollama
ollama serve
# Add authentication layer (nginx + basic auth or OAuth)
# See security configuration belowFor teams using cloud infrastructure but needing data sovereignty:
This is the recommended path for larger enterprises that already have cloud infrastructure.
The most flexible approach:
# nginx reverse proxy with authentication
server {
listen 443 ssl;
server_name ai.company.internal;
ssl_certificate /etc/ssl/company.crt;
ssl_certificate_key /etc/ssl/company.key;
# Restrict to internal network only
allow 192.168.0.0/16;
deny all;
location / {
auth_request /auth;
proxy_pass http://localhost:11434;
proxy_set_header Host $host;
proxy_read_timeout 300s;
}
location /auth {
proxy_pass http://localhost:8080/validate;
proxy_pass_request_body off;
}
}For enterprise environments, integrate with your existing identity provider (Okta, Azure AD, Google Workspace):
# Use Open WebUI with SSO support
docker run -d \
-p 3000:8080 \
-e WEBUI_AUTH=true \
-e OAUTH_PROVIDER_NAME="Azure AD" \
-e OAUTH_CLIENT_ID="your-client-id" \
-e OAUTH_CLIENT_SECRET="your-client-secret" \
-e OPENID_PROVIDER_URL="https://login.microsoftonline.com/tenant-id/v2.0" \
-v open-webui:/app/backend/data \
ghcr.io/open-webui/open-webui:mainFor compliance, log all AI interactions:
import logging
import json
from datetime import datetime
# Audit log every request
def log_ai_interaction(user_id: str, model: str, prompt_hash: str, response_length: int):
audit_entry = {
"timestamp": datetime.utcnow().isoformat(),
"user_id": user_id,
"model": model,
"prompt_hash": prompt_hash, # hash, not the actual prompt (privacy)
"response_tokens": response_length,
"ip_address": request.remote_addr # log source IP
}
logging.getLogger("audit").info(json.dumps(audit_entry))For a team of 20 developers using AI assistance 4 hours/day:
| Approach | Monthly Cost | Annual Cost | Data Risk |
|---|---|---|---|
| ChatGPT Enterprise | $25/user × 20 = $500 | $6,000 | Data sent to OpenAI |
| Claude Team | $25/user × 20 = $500 | $6,000 | Data sent to Anthropic |
| Local: RTX 4090 server | Hardware: $1,600 + $30 electricity | ~$360/yr after year 1 | Zero — stays on-premise |
| Local: Dual 4090 server | Hardware: $3,200 + $60 electricity | ~$720/yr after year 1 | Zero — stays on-premise |
Break-even: A single RTX 4090 server pays for itself in under 3 months vs a 20-person ChatGPT Enterprise subscription.
| Use Case | Recommended Model | Why |
|---|---|---|
| General business assistant | Llama 3.3 70B Q4 | Best overall quality for business tasks |
| Code review & debugging | DeepSeek R1 32B or Qwen 2.5-Coder 32B | Specialized for code |
| Document summarization | Gemma 3 27B Q4 | Long context, efficient |
| Customer support drafts | Qwen 2.5 14B | Fast, cost-efficient |
| Legal document analysis | Llama 3.3 70B Q4 | Best reasoning, stays on-premise |
| Medical record summaries | Llama 3.3 70B Q4 (with HIPAA setup) | Quality + full data control |
LEGAL / COMPLIANCE
□ Document your AI use cases and data flows
□ Update Privacy Policy to reflect local AI use
□ Create AI Acceptable Use Policy for employees
□ If EU users: confirm no personal data leaves EU network
□ If HIPAA: confirm PHI stays within HIPAA-compliant environment
TECHNICAL
□ Deploy Ollama on dedicated hardware or private cloud
□ Configure network to block external AI API calls
□ Set up authentication (SSO/OAuth preferred)
□ Enable audit logging (who used AI, when, what model)
□ Configure backups for model storage
□ Document incident response plan for AI-related breaches
OPERATIONAL
□ Train employees on approved AI tools and data classification
□ Establish model update process (when to upgrade models)
□ Set up monitoring for hardware health (GPU temps, VRAM usage)
□ Create vendor assessment process if using cloud AI for any use caseDo I need a DPA with Ollama? No. Ollama is open-source software you run yourself. There's no external data processor involved.
Can we use open-weight models commercially? Most popular models (Llama 3, Gemma 3, Mistral, Qwen 2.5) allow commercial use. Check the specific license: Llama 3 has a community license allowing commercial use for companies under 700M monthly active users.
How do we handle model updates? Pull new model versions with ollama pull model:version. Test in staging before deploying to production. Document each model version used for auditability.
Is local AI good enough to replace cloud AI for business? For most business tasks (writing, summarization, code review, data analysis), a local 70B model is competitive with GPT-4-class performance. For complex reasoning or very long context (>100k tokens), cloud APIs still have an edge — but the gap is closing with each new model release.