Air-gapped LLM deployment: a practitioner's reference

What "air-gapped LLM deployment" means

"Air-gapped" gets used loosely, and the loose uses cover deployments that would fail a real gap review. A working definition:

Three adjacent postures are frequently conflated with air-gapped and should not be:

• Private-cloud LLM — inference is in a customer-tenant VPC on a public cloud, so the physical infrastructure is shared and the deployment retains network reachability to the cloud provider's control plane. Useful for many workloads; not air-gapped.
• VPC-isolated LLM — the LLM runs in an on-prem cluster that has an outbound-only path for pulling images, models, telemetry, or license checks. Sensitive data does not leave, but the deployment still relies on egress. Not air-gapped.
• Egress-blocked LLM — the LLM runs on-prem with a firewall rule that denies outbound flows. Better than nothing, but if the tooling assumes internet, the deployment breaks the first time an update is needed. Air-gapped deployments are engineered so nothing needs egress in the first place.

The important test: can this deployment operate indefinitely, including bringing on new models and applying security patches, with zero outbound flows? If yes, it is air-gapped. If updates require the enclave to reach the internet — even briefly, even through a proxy — it is not, and reviewers who care about the distinction will treat it as a compensating-control problem, not an air-gap posture.

Getting the first model across the air gap

The mechanics that surprise most teams on the second week of a project are the ones around initial intake: how the very first model, runtime, and configuration land inside an enclave that, by policy, has no way for them to get there on their own.

The pattern that works in practice:

• Bundle production outside the enclave. The vendor assembles the artifact — model weights, tokenizer, inference runtime, application code, and any OS or driver patches the deployment needs — in a build environment the customer's security team can inspect. The output is a single tar or squashfs image plus a signed manifest.
• Cryptographic signing against a customer-controlled trust root. The manifest is signed with a key the customer either controls directly or trusts through an established chain. This is the lever the customer keeps: rotating or revoking that key stops future bundles from being accepted, regardless of what the vendor does.
• Physical or cross-domain transfer. The signed bundle moves across the gap on read-only removable media, a one-way data diode, or an approved cross-domain solution. The transfer method is a policy decision for the enclave, not the LLM vendor.
• Verification inside the enclave. On the transfer host, the customer verifies the signature against the trust root and the bundle's per-file hashes against the manifest before anything is extracted. A failed verification stops the process.
• Staging on the change window. The verified bundle is installed to the inference nodes during the customer's change window. The customer owns the schedule.

Signed offline update bundles: what they contain, how they're verified

Ongoing updates use the same intake pattern as the initial install. This is the part that has to keep working every quarter, for years, without anyone quietly opening a firewall rule.

A well-formed offline update bundle is a single archive — typically a signed .tar.zst — with three things beside the payload itself: a manifest, a detached signature, and human-readable release notes the operator reads before scheduling the change window. The payload varies by what changed. A weights-only bundle carries the model artifact and any tokenizer diff. A runtime bundle carries a pinned vLLM version and its transitive dependencies. A full bundle carries the model, the runtime, the application layer (Open WebUI, the RAG service, embedding and reranker models), and any OS or driver patches the deployment includes. Version alignment across those layers is the manifest's job.

A minimal manifest for a full bundle looks like this:

{
  "bundle_id":  "onsite-2026.09.1",
  "supersedes": "onsite-2026.06.3",
  "signed_by":  "customer-trust-root:key-04",
  "signature":  "<detached SHA-256 signature over this manifest>",
  "files": [
    { "path": "models/qwen3-70b-bf16.safetensors",
      "sha256": "8f3c...9a4b", "size": 140213444608 },
    { "path": "runtime/vllm-0.7.4-cu124.tar.zst",
      "sha256": "c210...44e1", "size":   1048291337 },
    { "path": "app/open-webui-0.6.2.tar.zst",
      "sha256": "de70...11f0", "size":    184772512 }
  ],
  "notes": "release-notes.md"
}

Every file in the payload is hashed with SHA-256; the manifest itself carries a detached signature against the customer's trust root. That is the shape the industry already uses for disconnected OS updates and controlled software drops, applied to model artifacts.

Verification inside the enclave runs three checks. Signature against the customer's trust root. Per-file SHA-256 against the manifest, on every node the bundle will land on. A dry-run install against a staging host if the deployment includes one, or a walk-through against release notes if it doesn't. Only after all three pass does the bundle become eligible for the change window.

Rollback is a first-class part of the design. Each bundle installs to its own versioned directory under the deployment root; the previous one stays in place for at least one change cycle; a single symlink points at whichever is currently live:

/opt/onsite-ai/
  bundles/
    onsite-2026.06.3/    # previous, retained for rollback
    onsite-2026.09.1/    # current
  current -> bundles/onsite-2026.09.1

# apply next bundle after signature + hash verification
onsite-bundle apply    --manifest onsite-2026.12.1.manifest.json

# revert during the same change window
onsite-bundle rollback --to onsite-2026.06.3

A rollback is a symlink flip and a systemd service restart. In practice this catches the case where a new model regresses on the customer's specific corpus in a way the vendor's evaluation didn't hit; the operator reverts inside the same change window and files a support bundle without the deployment ever being down.

The commands above are the shape, not the spec. Real deployments substitute their own bundle-tool binary and directory layout. What has to hold across implementations is what the shape encodes: versioned bundles kept side-by-side, a verified manifest as the source of truth, and a documented one-command revert.

The operational contract that matters: no forced upgrades, and no way for the vendor to change deployment behavior without the customer's operator running a command.

Retrieval-augmented generation without internet

The primary use case for enterprise LLMs is retrieval: cited answers over the customer's own document corpus — SOPs, equipment manuals, PLC documentation, maintenance logs, drawings, legal filings, clinical records. Practitioners want to know that retrieval works inside the gap without secretly reaching for a hosted embedding endpoint or a hosted reranker.

The pieces of a RAG pipeline, and where each of them runs in an air-gapped deployment:

• Document ingestion — file watchers or scheduled pulls from the customer's document systems (SharePoint, network shares, CMMS, PLM) that already live on the same isolated network. No calls out; ingestion is entirely a local file-system and internal-API activity.
• Parsing and chunking — libraries run in-process on the inference host. Vision-language models handle schematics and drawings the same way, on the same GPUs.
• Embedding — a locally-hosted embedding model, shipped inside the same signed bundle as the language model. This is the single most important integrity property; a hosted embedding endpoint would silently move every document out of the enclave. Local embedding models keep the corpus in place.
• Vector store — an in-enclave vector index (for example, a pgvector-backed Postgres or a purpose-built vector store) that scales to millions of documents. Storage is on the same NVMe tier as the inference host.
• Keyword index — a lexical index, also local, running alongside the vector store so hybrid retrieval (vector + keyword) can outperform either alone on real corpora.
• Reranker — a cross-encoder rerank pass over the hybrid top-k. Local model, same signed bundle as everything else.
• Answer generation — the language model reads the retrieved chunks and generates a cited answer. Prompts and retrieved content never leave the enclave.

The invariant a practitioner should confirm during a pilot: run the deployment with strict egress monitoring and see zero outbound flows during indexing and inference. Any flow at all — a health check to a model registry, a license verification, a tokenizer download — indicates a component that has not been fully offlined.

SSO, RBAC, MFA, and audit logging inside the enclave

The compliance review that decides an LLM procurement runs on identity, access, and audit. Practitioners planning air-gapped installs need to know these surfaces behave the same way inside the gap as they do on the corporate network.

SSO binds to the enclave-local directory — Active Directory, FreeIPA, or the program's own identity provider — rather than a cloud IdP the enclave cannot reach. SAML and OIDC both work as long as the issuer lives inside the enclave. RBAC is enforced at the application layer over the same directory groups the rest of the enclave already uses: a maintenance-engineer group sees maintenance content, a legal group sees legal content, a program-controlled group sees program-controlled content. The LLM inherits the enclave's access model rather than carving out its own.

Three surfaces below identity finish the picture:

• MFA uses enclave-local tokens — hardware keys, PIV/CAC cards on federal deployments, or an on-prem TOTP issuer.
• Full-disk encryption on the inference and storage hosts covers the physical risk: a drive leaving the enclave for RMA or decommissioning.
• Audit logging captures every authentication, chat message, retrieved document, and administrative action. Logs stay inside the enclave and are one-way forwarded to an in-enclave collector; a scheduled cross-domain transfer moves them to the SIEM during the change window.

What a real deployment needs

The sizing rule inside the gap is the same one that applies anywhere else: size for concurrent users at the target context length, not for peak single-user latency. The reference entry tier is a dual-GPU, air-cooled 4U server with dual redundant power that racks next to the rest of the isolated compute, scaling to a dual-node HA cluster where uptime demands it. The full sizing math — VRAM budgets, KV-cache arithmetic, GPU selection, and the tier ladder — lives on the LLM hardware requirements page.

Four ways air-gapped LLM deployments go wrong

These are the failure modes that recur across disconnected-deployment practice. Each is easy to avoid once it has a name.

1. The "air-gapped except for updates" trap

   A supposedly air-gapped enclave keeps an outbound-only egress path — model-registry pulls, telemetry, license checks — because the tooling assumes internet. Teams then discover the deployment is not compliant with the policy that put it in the enclave in the first place.

   Mitigation: verify that the deployment operates with zero outbound flows in normal operation. Treat update bundles as physically-media-transferred artifacts, not as network pulls from a "trusted" registry.

2. Embedding model that "phones home"

   RAG indexing quietly calls a hosted embedding endpoint, hosted tokenizer, or model-registry health check. The corpus stays in-perimeter, but the model still crosses the gap in the wrong direction, and every indexed document is exposed as a result. This is the failure mode that shows up most often on second-week egress audits, because "the embedding model is the same one we use in dev" is a sentence that hides an outbound call.

   Mitigation: run the embedding and reranker models locally on the same inference plane as the language model. Verify with strict egress monitoring during the pilot.

3. No rollback plan for a bad bundle

   Deployment overwrites the previous model or vLLM version in place. When the new bundle regresses on the customer's workload — retrieval quality drops, a tokenizer change breaks prompt templates, an inference-runtime change reduces throughput — there is no path back except restore-from-backup, and restore-from-backup during a production change window is a bad Tuesday.

   Mitigation: keep the previous bundle installed alongside the new one for at least one change cycle. Treat a bundle apply as an A/B swap with a documented, one-command rollback (see the shape above).

4. Identity plumbed to a cloud IdP

   SSO is wired to a cloud identity provider that the enclave cannot reach. The deployment either can't authenticate anyone or holds an out-of-band trust to a network the enclave was designed to exclude.

   Mitigation: bind SSO/LDAP to the enclave-local directory — Active Directory, FreeIPA, or the program's own IdP. MFA via enclave-local tokens.

Observability and support without egress

The last question every operations team asks: how do you keep the deployment healthy, and how does the vendor help troubleshoot, when nothing can call home?

Observability lives entirely inside the segment. GPU utilization, request queue depth, KV-cache pressure, retrieval latency, and answer-quality signals are captured and rendered on a local dashboard, and the same structured audit and application logs the SIEM eventually receives are readable locally by the operator, so root-cause analysis of an incident does not require cross-domain access.

Support runs through the same cross-domain path used for audit logs. When a customer opens a case, the vendor asks for a specific support bundle — sanitized logs, config snapshots, GPU telemetry — which the customer produces inside the perimeter and delivers on the same schedule as the SIEM export. The vendor never gets remote access; the customer controls what leaves.

This is the contract on-premises makes with the customer generally, and it is especially strict inside the gap: you own the box and the schedule; we ship the software. The relationship is engineered so the vendor never needs the enclave to reach anywhere, because if the vendor needed egress, the deployment wouldn't be air-gapped in the first place.

Frequently asked questions

What counts as an air-gapped LLM deployment?

An LLM deployment where every runtime component sits inside a network that has no path to the public internet during normal operation: the inference plane, retrieval pipeline, embedding models, vector store, identity backend, and audit log all run in the enclave. Model weights and software updates arrive as signed offline bundles moved across the gap by physical media or an approved cross-domain transfer. VPC-isolated, private-cloud, and egress-blocked deployments are related but distinct: they typically retain an outbound path for telemetry, updates, or identity, which the air-gap version does not.

How do model and software updates cross the air gap?

As a signed offline bundle. One archive (typically a .tar.zst) holds the model weights, tokenizer config, inference-runtime binaries, a version-pinned vLLM build, and any patches. Outside the enclave the archive is cryptographically signed against a customer-controlled trust root and its contents hashed with SHA-256 into a manifest. Media crosses the gap by hand-carried disk or an approved cross-domain transfer. Inside, the operator verifies the detached signature, checks every file hash against the manifest, and only then extracts. The previous version stays on disk under a versioned directory with a current symlink, so an onsite-bundle rollback command reverts in one operator step. The bundle is applied on the customer's change window.

How do SSO, RBAC, and audit logging work without cloud identity?

SSO binds to whatever identity provider already lives inside the enclave: Active Directory, FreeIPA, or the program's own IdP. The Onsite AI application inherits the same directory groups the rest of the enclave uses, and RBAC is enforced at the application layer against those groups. MFA runs on enclave-local tokens: PIV/CAC where the customer uses them, or a local MFA service where they do not. Audit records go to an append-only in-enclave collector as they are written. A scheduled cross-domain transfer moves them to the customer's SIEM on the same change window used for updates. Nothing in the identity or audit path calls out to a cloud IdP the enclave cannot reach.