Written by Jakub Rusinowski · Last updated 2026-07-12 · Hardware figures computed by our VRAM engine
This page is practitioner guidance on deployment mechanics, not legal advice — involve your counsel or DPO for decisions about your specific obligations.
An air-gapped LLM deployment runs open-weight models on hardware with no network path to the outside world — models arrive on verified offline media, updates are deliberate events, and every prompt stays inside the security perimeter by construction rather than by policy. It is entirely practical in 2026 because open models need no phone-home: the honest questions are whether your data class actually warrants the operational cost, and how you handle updates and audit logging once the cable is out. This guide covers both, with healthcare and legal as the worked verticals.
Air-gapping gets prescribed far more often than it's indicated, so start with what the rules actually say.
HIPAA does not require air-gapping. Many healthcare organizations process PHI with cloud AI under a Business Associate Agreement, lawfully. What the Security Rule requires is access control, audit trails, and integrity protections — achievable in several architectures. The air gap earns its cost for specific triggers: PHI classes with re-disclosure restrictions (psychiatric records, substance-abuse treatment records under 42 CFR Part 2, genomic data), research datasets under consent terms that forbid third-party processing, and institutions whose audit posture treats "no external path exists" as materially cheaper to demonstrate than "external paths are controlled."
Legal privilege is a confidentiality duty, not an architecture mandate. Bar guidance across jurisdictions requires reasonable measures to protect client confidences and competence with the technology used. Most firms satisfy that with contractual and technical controls. The air gap becomes attractive for the extreme cases: M&A data rooms, government-classified adjacent matters, and disputes where the opposing party is a technology provider — situations where a firm wants to answer "could this have left the building?" with a topology, not a contract stack.
The general rule: air-gap when the *demonstration* of impossibility is worth more than the convenience of connectivity. Most organizations on this hub are better served by the standard on-prem deployment — networked internally, egress-blocked at the firewall. That gets 90% of the assurance at 20% of the operational friction. This page is for the 10% who need the last mile.
Air-gapping most enterprise software is painful because it assumes SaaS licensing, telemetry, and update servers. Open-weight LLM serving assumes none of that:
The genuinely hard parts are organizational, not technical: the update cadence, the audit logging, and the temptation to "just briefly" connect the machine. The walkthrough below is structured around those three.
Healthcare — the clinical-notes assistant. A hospital deploys a summarization and Q&A assistant over clinical documentation in a restricted enclave: a GPU server in the existing secure zone, 70B-class model, RAG over the document store, access via the clinical VLAN only. HIPAA's audit-control requirement shapes the design more than the model does — the gateway logs accessor identity, timestamp, action, and a reference to the PHI touched, into the same SIEM the organization already audits. The air gap's contribution: the DPIA/risk-analysis section on external disclosure becomes one sentence.
Legal — the matter enclave. A firm stands up a per-matter machine for a document-heavy case: workstation-class hardware (a 48–96 GB card is plenty — see the hardware tiers), the production set ingested into a local index, attorneys query it through the matter's access list. At matter close, the machine is wiped per the retention schedule — the deployment's lifecycle matches the engagement's, which is something no shared cloud service reproduces cleanly. Privilege review of AI outputs stays inside the firm by construction.
Decide what is inside: the GPU server, the document store, the access network segment, and nothing else. Document the boundary and the approved-media procedure *before* buying hardware — in regulated environments the paperwork is the deployment. Deliverable: a one-page topology your auditor can initial.
On a separate, connected staging box: download model weights (GGUF for llama.cpp/Ollama, safetensors for vLLM), the serving stack installers, and OS packages. Record SHA-256 checksums for everything, from two sources where the publisher provides them (Hugging Face checksums + publisher release notes). Media crossing the gap should carry artifacts and checksums separately.
Install OS, drivers, and the serving stack from the staged media with the network physically disconnected — build it dark from day one rather than disconnecting later, so no residual credentials, agents, or update daemons ever existed on it. Confirm the machine has zero configured external routes and that the serving stack starts clean with no DNS resolution available.
Run the model behind an internal gateway that (a) authenticates against your directory, (b) logs user, timestamp, request reference, and response reference for every call, and (c) exports those logs to your existing SIEM over the *internal* path. In healthcare this is your Security Rule audit control; in legal it is your privilege log's best friend. If the gateway doesn't log it, it didn't happen — build this before inviting users, not after.
Load the document set into the local RAG index (our local RAG guide covers the stack). Then validate with the actual professionals on actual documents for two weeks — clinicians or attorneys, not IT. The quality verdict on an air-gapped system must come from inside, because there is deliberately no telemetry to tell you.
Set a cadence — quarterly works for most — where a maintainer stages security patches and (optionally) a newer model on the connected staging box, re-verifies checksums, and walks them across. Log each crossing: date, media ID, artifact hashes, who carried it. An air gap without an update discipline decays into either an insecure system or a quietly reconnected one; the cadence is what keeps the gap honest.
Jakub Rusinowski, the founder of LLM Configurator, runs corporate workshops and lectures on deploying local LLMs — hardware sizing, model selection, compliance-friendly architectures, and hands-on setup for your team. Direct, vendor-neutral, practitioner-level.