HEALTHCARE · HIPAA

HIPAA-Compliant AI: Clinical LLMs Where PHI Never Leaves Your Network

Your privacy officer will not sign off on pasting discharge summaries into a public chatbot — and they are right to refuse.

Written by Jakub Rusinowski · Last updated 2026-08-01 · Hardware figures computed by our VRAM engine

This page is practitioner guidance on deployment mechanics, not legal advice — involve your counsel or DPO for decisions about your specific obligations.

HIPAA-compliant AI means protected health information stays under the covered entity's control for the entire time a model processes it. The dependable way to get there is to serve an open-weight model from hardware you administer: with no third-party inference provider in the path there is no disclosure to a business associate, so no BAA is needed for the model itself, and the access-control, audit, integrity, and transmission-security standards of 45 CFR §164.312 are met by infrastructure your team already operates.

Why this is hard

What the deployment gives you

PHI never leaves the covered entity

Inference runs on GPUs inside your existing network boundary, on hosts with no outbound route. "PHI cannot leave" stops being a policy statement and becomes a routing fact you can demonstrate to an auditor with a firewall rule.

Audit controls that actually satisfy §164.312(b)

Every prompt, retrieval, and completion is written to your SIEM with the user identity from your directory — not a shared service account. Reconstructing who asked what about which patient becomes a query, not an investigation.

De-identification as a pipeline stage

Safe Harbor's eighteen identifiers stripped in a deterministic pre-processing step, with expert determination where the use case needs richer data. A model prompt is not the place to ask politely for de-identification.

Minimum necessary, enforced by retrieval scope

The assistant can only retrieve from the records the requesting clinician already has a treatment relationship with. Scope is enforced in the retrieval layer, where it is verifiable, rather than trusted to the model.

What a HIPAA-safe deployment actually looks like

Four components, all inside the boundary you already defend. The only unusual element is the gateway, which exists so that authorization, redaction, and audit happen before a prompt ever reaches the model — the same reason you put an API gateway in front of a clinical system.

What a HIPAA-safe deployment actually looks likeCovered entity — no PHI crosses this line. 1. Clinician workstation: SSO session from your IdP; role and treatment relationship attached to every request. 2. Inference gateway: Enforces minimum necessary, redacts on egress, writes the audit record before the prompt runs. 3. Model server: Open-weight model on your GPUs. Weights from local storage, no outbound network route. 4. Chart & document index: Vector store on encrypted volumes, scoped per patient relationship, same trust boundary.COVERED ENTITY — NO PHI CROSSES THIS LINEClinician workstationSSO session from your IdP;role and treatmentrelationship attached toevery request.Inference gatewayEnforces minimum necessary,redacts on egress, writesthe audit record before theprompt runs.Model serverOpen-weight model on yourGPUs. Weights from localstorage, no outboundnetwork route.Chart & document indexVector store on encryptedvolumes, scoped per patientrelationship, same trustboundary.

What you have to satisfy

Models that fit this deployment

模型VRAM (Q4)可运行于上下文许可
Mistral Small 3.1 24B
Clinic tier — one 16 GB card — Ambient documentation and summarization for a single site, on hardware that fits in a closet.
ollama pull mistral-small3.1		15 GB24 GB GPU (RTX 3090/4090)
Mac: 24 GB 统一内存		125KApache 2.0
Qwen 3 32B
Department tier — one 24 GB card — The workhorse for chart summarization and RAG over internal protocols and formulary documents.
ollama pull qwen3:32b		20.6 GB24 GB GPU (RTX 3090/4090)
Mac: 32 GB 统一内存		125KApache 2.0
Llama 3.3 70B Instruct
Health-system tier — 48 GB class — Where clinical nuance matters most; still a single-card deployment on a pro GPU.
ollama pull llama3.3:70b		43.1 GB2×24 GB GPUs or 48 GB card
Mac: 64 GB 统一内存		128KLlama Community

Deploying clinical AI without a disclosure?

We help healthcare organizations size, architect, and stand up on-premise AI that keeps PHI inside the covered entity — hardware sizing, model selection, audit and retrieval design, and the walkthrough your privacy officer will ask for. Vendor-neutral: we sell no hardware, no platform, and no model.

Ask about a workshop →

常见问题

Do I need a BAA to use AI with PHI?
Only if a third party is handling the PHI for you. A BAA is the instrument that makes disclosure to a business associate permissible — so it is required for any hosted AI API that processes patient data, and it is the first thing to demand from a clinical AI vendor. When the model runs on hardware your organization administers, no outside party creates, receives, maintains, or transmits the PHI, so there is no business associate for the inference step and no BAA to negotiate for it. You still need agreements with anyone else in the chain, such as a managed hosting provider.
Is ChatGPT HIPAA compliant?
No product is "HIPAA compliant" on its own — compliance is a property of how a covered entity deploys and governs a system. In practice the question is whether the vendor will sign a BAA covering your use, what their subprocessor chain looks like, whether inputs are retained or used for training, and whether their audit logging is granular enough for §164.312(b). Consumer chat tiers generally fail that test; some enterprise tiers pass it contractually. Self-hosting sidesteps the entire question by removing the disclosure.
Does on-premise AI make us automatically HIPAA compliant?
No. It removes the hardest problem — third-party disclosure — but the technical safeguards still apply to your own system. You need access control tied to real identities, audit controls that capture AI activity, encryption at rest and in transit, integrity controls, and a retention decision for prompt logs, which are themselves PHI. The advantage is that these are controls you already operate for other clinical systems, applied to one more system, rather than a novel vendor risk assessment.
What hardware does a hospital need for a HIPAA-safe AI deployment?
Less than most people assume. A single 24 GB GPU runs a 32B-class model that handles summarization, drafting, and retrieval over internal documents for a department. A 48 GB card moves you to a 70B-class model for a health system. Sizing follows concurrent users rather than bed count: a handful of GPUs typically covers hundreds of intermittent clinical users, because inference demand is bursty. Every VRAM figure on this page comes from the same engine as our compatibility checker, so you can verify a configuration yourself before buying anything.
Can we use patient data to fine-tune a model?
Technically yes, and it stays inside your boundary if the training runs on your own hardware — but treat it as a use of PHI requiring the same lawful basis, minimum-necessary analysis, and governance as any other secondary use. Two practical cautions: models can memorize and reproduce rare training examples, which is a real re-identification risk for unusual cases, and a fine-tuned model is very hard to un-train if a patient later exercises a rights request. Retrieval over a curated index is usually the better default, because you can delete from an index.

Go deeper