This page is practitioner guidance on deployment mechanics, not legal advice — involve your counsel or DPO for decisions about your specific obligations.
HIPAA-compliant AI means protected health information stays under the covered entity's control for the entire time a model processes it. The dependable way to get there is to serve an open-weight model from hardware you administer: with no third-party inference provider in the path there is no disclosure to a business associate, so no BAA is needed for the model itself, and the access-control, audit, integrity, and transmission-security standards of 45 CFR §164.312 are met by infrastructure your team already operates.
Why this is hard
One pasted discharge summary is a disclosure to a third party — and you carry the burden of showing it was permitted.
A vendor BAA binds the vendor. It rarely enumerates their subprocessors, and almost never the model host underneath them.
Clinicians are already using AI on their phones. Shadow usage without a sanctioned alternative is the live breach risk, not a hypothetical one.
What the deployment gives you
PHI never leaves the covered entity
Inference runs on GPUs inside your existing network boundary, on hosts with no outbound route. "PHI cannot leave" stops being a policy statement and becomes a routing fact you can demonstrate to an auditor with a firewall rule.
Audit controls that actually satisfy §164.312(b)
Every prompt, retrieval, and completion is written to your SIEM with the user identity from your directory — not a shared service account. Reconstructing who asked what about which patient becomes a query, not an investigation.
De-identification as a pipeline stage
Safe Harbor's eighteen identifiers stripped in a deterministic pre-processing step, with expert determination where the use case needs richer data. A model prompt is not the place to ask politely for de-identification.
Minimum necessary, enforced by retrieval scope
The assistant can only retrieve from the records the requesting clinician already has a treatment relationship with. Scope is enforced in the retrieval layer, where it is verifiable, rather than trusted to the model.
What a HIPAA-safe deployment actually looks like
Four components, all inside the boundary you already defend. The only unusual element is the gateway, which exists so that authorization, redaction, and audit happen before a prompt ever reaches the model — the same reason you put an API gateway in front of a clinical system.
Encryption at rest on both the model host and the vector index — addressable under §164.312(a)(2)(iv), and the only defensible answer after a stolen-drive incident.
Unique user identification through your existing identity provider, so every audit record names a person rather than the application.
A deliberate retention decision for prompt logs: they contain PHI, so they inherit your record-retention schedule and your disposal process.
No outbound egress from the model host, enforced at the network layer — this is the control that turns the whole claim from policy into fact.
Human review before anything reaches the chart. Model output is a draft for a clinician to accept, never an entry that writes itself.
What you have to satisfy
45 CFR §164.312 — Technical safeguards: Access control, audit controls, integrity, and transmission security — the four standards any system touching ePHI has to implement, AI included.
45 CFR §164.502(b) — Minimum necessary: Each use is limited to the PHI reasonably needed. In an AI system that is a retrieval-scoping decision, not a paragraph in a policy binder.
45 CFR §164.308(b) — Business associate contracts: Required whenever a third party creates, receives, maintains, or transmits PHI on your behalf. Self-hosted inference removes the party, and with it the contract.
42 CFR Part 2 — Substance use disorder records: Stricter and consent-based, above HIPAA. If these records are in scope, segregate the index — do not rely on prompt-level filtering to hold the line.
Models that fit this deployment
Model
VRAM (Q4)
Runs on
Context
License
Mistral Small 3.1 24B Clinic tier — one 16 GB card — Ambient documentation and summarization for a single site, on hardware that fits in a closet. ollama pull mistral-small3.1
Qwen 3 32B Department tier — one 24 GB card — The workhorse for chart summarization and RAG over internal protocols and formulary documents. ollama pull qwen3:32b
20 GB
24 GB GPU (RTX 3090/4090) Mac: 32 GB unified
128,000
Apache 2.0
Llama 3.3 70B Instruct Health-system tier — 48 GB class — Where clinical nuance matters most; still a single-card deployment on a pro GPU. ollama pull llama3.3:70b
43.1 GB
2×24 GB GPUs or 48 GB card Mac: 64 GB unified
128K
Llama Community
Deploying clinical AI without a disclosure?
We help healthcare organizations size, architect, and stand up on-premise AI that keeps PHI inside the covered entity — hardware sizing, model selection, audit and retrieval design, and the walkthrough your privacy officer will ask for. Vendor-neutral: we sell no hardware, no platform, and no model.
Only if a third party is handling the PHI for you. A BAA is the instrument that makes disclosure to a business associate permissible — so it is required for any hosted AI API that processes patient data, and it is the first thing to demand from a clinical AI vendor. When the model runs on hardware your organization administers, no outside party creates, receives, maintains, or transmits the PHI, so there is no business associate for the inference step and no BAA to negotiate for it. You still need agreements with anyone else in the chain, such as a managed hosting provider.
Is ChatGPT HIPAA compliant?
No product is "HIPAA compliant" on its own — compliance is a property of how a covered entity deploys and governs a system. In practice the question is whether the vendor will sign a BAA covering your use, what their subprocessor chain looks like, whether inputs are retained or used for training, and whether their audit logging is granular enough for §164.312(b). Consumer chat tiers generally fail that test; some enterprise tiers pass it contractually. Self-hosting sidesteps the entire question by removing the disclosure.
Does on-premise AI make us automatically HIPAA compliant?
No. It removes the hardest problem — third-party disclosure — but the technical safeguards still apply to your own system. You need access control tied to real identities, audit controls that capture AI activity, encryption at rest and in transit, integrity controls, and a retention decision for prompt logs, which are themselves PHI. The advantage is that these are controls you already operate for other clinical systems, applied to one more system, rather than a novel vendor risk assessment.
What hardware does a hospital need for a HIPAA-safe AI deployment?
Less than most people assume. A single 24 GB GPU runs a 32B-class model that handles summarization, drafting, and retrieval over internal documents for a department. A 48 GB card moves you to a 70B-class model for a health system. Sizing follows concurrent users rather than bed count: a handful of GPUs typically covers hundreds of intermittent clinical users, because inference demand is bursty. Every VRAM figure on this page comes from the same engine as our compatibility checker, so you can verify a configuration yourself before buying anything.
Can we use patient data to fine-tune a model?
Technically yes, and it stays inside your boundary if the training runs on your own hardware — but treat it as a use of PHI requiring the same lawful basis, minimum-necessary analysis, and governance as any other secondary use. Two practical cautions: models can memorize and reproduce rare training examples, which is a real re-identification risk for unusual cases, and a fine-tuned model is very hard to un-train if a patient later exercises a rights request. Retrieval over a curated index is usually the better default, because you can delete from an index.