LLM hardware requirements: a buyer's guide for on-premises deployment
How to size on-premises LLM hardware for the workload you actually have: what drives VRAM, why the reference architecture picks NVIDIA RTX PRO 6000 Blackwell over H100, how sizing scales from a 2-GPU 192 GB entry server up to a 4-GPU 384 GB dual-node HA cluster, and where quantization pays off versus quietly costs.
Last reviewed: July 2026
What drives VRAM: weights, KV cache, and activation memory
Most under-sized on-prem LLM deployments trace to a single mistake: treating "VRAM budget" as a synonym for "model weights." It is not. The three consumers of GPU memory during production inference are the model weights, the KV cache, and activation memory — and only the first is fixed. The other two scale with load, and both are what a real deployment runs out of first.
The three consumers
Model weights — a fixed cost, roughly 2 bytes per parameter in BF16. A 70B open-weight model consumes ~140 GB before anything else. Quantization to INT8/FP8 halves this. Quantization to INT4 halves it again.
KV cache — a per-request cost that grows linearly with the length of each user's context. For a 70B model, plan roughly a couple of hundred kilobytes per token per user; at 8k-token contexts across dozens of concurrent users, the KV cache easily runs into the tens of gigabytes.
Activation memory — the working set the transformer needs during a forward pass. Small next to the KV cache at production concurrency, but non-zero, and the CUDA allocator wants headroom.
The practical implication is that the "how much VRAM do I need" question has two axes, not one: model size sets the floor, concurrency at your target context length sets the ceiling. Buyers who quote only the floor end up with servers that meet a benchmark and fail the deployment.
Sizing for concurrent users at your target context length
The heuristic that holds up in practice, and that the Onsite AI reference architecture is engineered against:
Working heuristic
Size against concurrent users at the target context length. Peak single-user latency benchmarks well and predicts almost nothing about production; the KV cache exhausts on the second axis, not the first.
For a 70B open-weight model in BF16 serving a plant-floor or engineering user population at production context lengths, the math works out approximately as follows. About 140 GB of the VRAM budget is spoken for by the weights. That leaves the remaining VRAM to divide between the KV cache for every in-flight request and the activation memory the runtime holds. On a 192 GB VRAM entry-tier server, that residual serves roughly 40 concurrent engineering users at realistic context lengths on a mixed chat + RAG workload — the concurrency target the reference architecture ships against. RAG-heavy workloads with long retrieved contexts settle nearer 30; chat-only sessions with short prompts stretch past 50. The 40 figure is the mixed-workload floor, not a benchmark ceiling.
The arithmetic behind that target is worth making explicit, because it is the number a buyer has to defend internally. Walk it start to finish with a representative 70B open-weight architecture (80 transformer layers, grouped-query attention with 8 KV heads at head-dim 128, BF16):
Worked example: 192 GB entry tier, 70B GQA in BF16
Step 1 — weights. 70 × 10⁹ parameters × 2 bytes in BF16 ≈ 140 GB. Fixed cost. Loaded once at startup, sharded across the two 96 GB cards over PCIe.
Step 2 — residual VRAM. 192 GB total − 140 GB weights = 52 GB. From that, the runtime reserves perhaps 8–12 GB for activation memory, the CUDA allocator's working headroom, and vLLM's own scheduler state. Round to ~40 GB of usable KV-cache budget.
Step 3 — per-token KV size. Each token costs 2 (K plus V) × 2 bytes (BF16) × 8 KV heads × 128 head-dim = 4 KB per layer per token. Across 80 layers that is ~320 KB per token — the "couple of hundred kilobytes per token" figure quoted informally.
Step 4 — worst-case concurrency. A single user at a fully-extended 8k-token context uses 8192 × 320 KB ≈ 2.5 GB. Divide the 40 GB budget: 40 GB / 2.5 GB ≈ 16 simultaneous worst-case contexts. That's the naïve floor.
Step 5 — what PagedAttention buys back. Real sessions rarely fill 8k tokens; a chat turn is typically 1–2k of prompt plus a short answer before the next turn resets working context. vLLM's paged attention allocates KV in small pages against what each request actually uses, not the worst-case slab. Across a mixed chat + RAG workload, the effective concurrency lands 2.5–3× the worst-case figure — the ~40-user target the reference architecture ships against.
The arithmetic is doing three things at once. It shows why 192 GB is the entry-tier line: dropping to 160 GB leaves 20 GB of KV residual, which halves the concurrency budget before PagedAttention gets a chance to help. It shows why context length is a first-class input to sizing: doubling the target context length halves the concurrency budget. And it shows where quantization changes the math: cutting weights to INT8 (~70 GB) frees another ~70 GB of residual, which — at the same context length — roughly doubles the usable concurrency headroom before the KV cache is the bottleneck.
Two levers move that concurrency budget without buying more GPUs: quantization of the weights (which enlarges the residual budget available to the KV cache), and the vLLM serving stack (which uses paged attention to allocate KV memory in small pages rather than reserving worst-case slabs per request). Both are discussed below. Neither invalidates the heuristic; they change the constants, not the shape.
Two engineers running one request each do not stress a 192 GB VRAM server. Forty engineers running one request each do. The buying decision has to be sized on the second scenario. A buyer who sees "70B model, 140 GB in BF16" and rounds up to a 160 GB configuration gets a server that boots, serves a single user fine, and then pre-empts requests when two dozen concurrent users each open an 8k-token context. Sizing total VRAM as weights + concurrency × per-user KV-cache budget + headroom is what points at the 192 GB configuration — and why the reference entry tier is two 96 GB cards rather than a smaller pairing.
The three-tier ladder: 192 GB entry → 288 GB single-server RAG → 384 GB HA cluster
The reference architecture ships three tiers, one for each of the offerings on the engagement model. Each tier is engineered for a distinct concurrency and workload shape. The full per-tier spec sheet is on the homepage; the tiers below explain the shape of the sizing decision, not just the numbers.
2× NVIDIA RTX PRO 6000 Blackwell · 192 GB VRAM · single server
512 GB ECC RAM · dual EPYC 9355 · 8 TB NVMe + 32 TB SAS · ~40 concurrent users (mixed load)
An air-gapped chatbot on a local open-weight model, with the familiar chat UI. One inference server handles a plant-floor deployment of dozens of concurrent engineering users on the customer's change window. The entry tier is where most first deployments land; it does not require a facility upgrade.
3× NVIDIA RTX PRO 6000 Blackwell · 288 GB VRAM · single server, multi-model
768 GB ECC RAM · dual EPYC 9355 · 16 TB NVMe + 48 TB SAS · same chat concurrency, plus retrieval pipeline
Adds parallel multi-model serving on top of the chat model — a dedicated embedding model, a cross-encoder reranker, and a vision-language model for schematics and drawings, all resident in VRAM at once. The extra GPU is not for more chat concurrency; it is for the retrieval pipeline that reasons over the customer's document corpus of millions of documents with cited answers.
4× NVIDIA RTX PRO 6000 Blackwell · 384 GB VRAM · dual-node HA cluster
1 TB ECC RAM across nodes · dual EPYC 9355 per node · 16 TB NVMe + 64 TB SAS · InfiniBand HDR 200 Gb/s
Two nodes, four GPUs total, InfiniBand HDR 200 Gb/s between them. Runs a model fine-tuned to the customer's industry with agentic capabilities, zero-downtime failover, and a contractual SLA. The jump from middle to top is a topology change, not just a GPU count — it is where the deployment stops being a single-server deployment and starts being a small cluster with the operational properties a critical-uptime workload needs.
All three tiers run the same software platform — the capabilities grid lists the full stack — under the same 3-year parts & labor warranty.
Why RTX PRO 6000 Blackwell, not H100
The reference architecture is not built on NVIDIA H100 because H100 is engineered for a different workload shape than the one an on-prem enterprise deployment actually has. The comparison across the GPUs a technical buyer will realistically weigh:
| Property | RTX PRO 6000 Blackwell | H100 (SXM5) | H200 | L40S |
|---|---|---|---|---|
| VRAM per card | 96 GB GDDR7 | 80 GB HBM3 | 141 GB HBM3e | 48 GB GDDR6 |
| Form factor | PCIe, air-cooled, 4U server | SXM5 on HGX baseboard | SXM5 on HGX baseboard | PCIe, air-cooled |
| Facility fit | Standard enterprise rack & PDU | Hyperscaler-grade, often liquid | Hyperscaler-grade, often liquid | Standard enterprise rack |
| Best fit | On-prem enterprise inference, tens of users | Hyperscale training & inference | Long-context hyperscale inference | Graphics-heavy inference |
| Cost profile | Workstation-tier per GB VRAM | Highest per card | Highest per card | Requires ~2× cards for 192 GB |
| Serviceability | Field-swappable PCIe · 3-year warranty | HGX baseboard, vendor service | HGX baseboard, vendor service | Field-swappable PCIe |
H100 and H200 win on raw FLOP/s and memory bandwidth for hyperscale training and hyperscale inference at thousands of concurrent requests. For a deployment sized for tens of concurrent engineering users the bottleneck is per-card VRAM and KV-cache headroom, not sustained FLOP/s, so the H100/H200 lead does not convert to a proportional user-experience improvement. Both are still HGX-baseboard SXM5 parts requiring hyperscaler-grade cooling and vendor service, which is the harder constraint at an enterprise plant or program facility. H200's 141 GB HBM3e helps most for long-context hyperscale inference — not the workload here. L40S is PCIe and air-cooled but only 48 GB per card; reaching a 192 GB budget takes four cards, which doubles slot pressure and power draw for the same VRAM footprint.
RTX PRO 6000 Blackwell lands in a narrower spot: 96 GB of VRAM per card at a workstation-tier price, PCIe air-cooled in a standard 4U server that fits an enterprise rack, and field-swappable under a 3-year parts & labor warranty. Two cards give a 192 GB entry tier at the concurrency target above; four cards in a dual-node cluster give a 384 GB HA tier with InfiniBand between nodes. For the on-prem enterprise workload — tens of concurrent users, 70B-class model, existing racks and PDUs — it fits the constraint set the other three cards were not built for.
CPU, system RAM, storage, and networking around the GPUs
A 2-GPU RTX PRO 6000 Blackwell server does not perform like a 2-GPU server if the CPU can't keep the queue full, the system RAM starves the vector index working set, or the storage can't page model artifacts fast enough. The rest of the server matters. Sizing it wrong is the second-most-common way an on-prem deployment falls short of the concurrency it was sized for.
The CPU choice is dual AMD EPYC 9355 across every tier. It handles the retrieval pipeline, the embedding calls that route to GPU, tokenization on hot paths, and the request queue. Two sockets are what keep PCIe lanes plentiful for the GPUs and the NVMe drives, and they give the runtime enough cores that the vLLM scheduler is not competing with retrieval work for CPU time.
System RAM scales with the tier — 512 GB ECC at the entry tier, 768 GB at the RAG tier, 1 TB across the HA cluster's two nodes — and it's ECC because this is production infrastructure. The unglamorous truth is that most of it exists to hold the vector index working set and the file-system page cache the RAG pipeline reads through. Storage sits above that in two tiers: NVMe hot for the model artifacts (memory-mapped, paged in on first use), the vector index, and recent documents; SAS cold for the long tail of documents, audit logs, and support-bundle archives. The 8 TB or 16 TB NVMe budget is sized to keep at least two full model bundles side by side — which is what the signed-bundle rollback story on the air-gapped deployment page depends on. Under-spec any of these and the queue, not the cards, becomes the bottleneck.
Networking splits at the HA tier. Single-server deployments run 25 GbE to the enclave LAN (100 GbE if the site's spine supports it). The HA cluster adds InfiniBand HDR 200 Gb/s between the two nodes for the tensor-parallel and KV-cache-sync paths inside vLLM, which are the latency-sensitive ones. InfiniBand is a cluster-tier requirement, not a single-server one. Power runs off dual redundant PSUs at every tier and fits inside standard 20A/30A rack circuits at entry and middle — part of what makes the PCIe air-cooled form factor easier to place than an 8-way HGX chassis in an existing enterprise facility.
Quantization tradeoffs: where INT8/FP8 pays off, where INT4 costs you
Every buyer has heard "just run it in 4-bit and it will fit on one GPU." Sometimes it will. Whether that's the right answer for a production deployment is a different question. The tradeoffs, in the order they matter for sizing:
BF16 is the reference: 2 bytes per parameter, 140 GB for a 70B model, no quality variable to argue with. Used to size the architecture. INT8/FP8 halves the weight footprint with typically negligible impact on chat-shaped workloads and roughly doubles the KV-cache residual at the same concurrency — a fair default when the workload is covered by well-tested INT8/FP8 recipes. INT4 halves it again but introduces measurable quality regressions on retrieval-heavy and code-shaped workloads; whether the regression is acceptable is a workload-specific question, answerable only against the customer's own corpus.
The trap that costs deployments the most VRAM: the KV cache is not quantized in the same step. INT4 weights do not shrink the KV cache in proportion. The KV cache is a per-request working set that scales with concurrency and context length; FP8 KV cache and KV cache compression are separate techniques that compose with vLLM's paged attention. Sizing by "the 4-bit weights fit in 40 GB" produces a rack that under-serves adoption on day one.
Sequence the decision the way the memory actually flows: workload quality requirements pick the precision, precision sets the KV-cache residual, the residual sets the concurrency budget, and the concurrency budget picks the tier. Starting from the tier and working backward is how deployments end up over-quantized to hit a hardware line that never needed hitting.
vLLM: PagedAttention, continuous batching, and what changes about the sizing math
Every Onsite AI deployment ships vLLM as the inference
runtime. Two of vLLM's design choices change how a buyer should think
about the sizing math, and skipping them causes over-provisioning.
- PagedAttention. vLLM allocates KV-cache memory in small pages (like an OS virtual-memory system) rather than reserving worst-case slabs per request. In practice this means a server with a given VRAM budget serves substantially more concurrent requests than a naïve "one KV cache per user at max context" model predicts. The 40+ concurrent-user figure on the entry tier assumes PagedAttention is doing its job.
- Continuous batching. vLLM batches requests at every decoding step, not per-request. Users whose requests complete free up their batch slot immediately for the next request in the queue. GPU utilization stays high across a spiky arrival pattern — which is what plant-floor and engineering use looks like — without the classic "batch of one" latency penalty.
- Tensor-parallel sharding across GPUs. The two-GPU entry tier and three-GPU middle tier both shard the model across the local PCIe fabric. The four-GPU HA cluster shards across GPUs within a node and replicates for failover across nodes over InfiniBand. Sharding is what allows a 70B model in BF16 to serve production concurrency without saturating a single card's memory bandwidth.
The sizing numbers on this page assume a vLLM-class serving stack. A deployment that skips PagedAttention and continuous batching — for example, an off-the-shelf single-GPU demo runtime — would need more VRAM and more GPUs to hit the same user experience at the same concurrency. Sandbox benchmarks against a demo runtime will over-predict what production sees.
Facility fit: rack, power, cooling, and network
The entry and middle tiers fit inside a standard 4U server with dual redundant power supplies and normal data-center airflow. That is a deliberate choice. The alternative — an 8-way HGX H100 chassis — is available, but it also asks for cage capacity, liquid cooling, and PDU review that most enterprise plant, hospital, and program facilities do not have without a project. The air-cooled PCIe Blackwell form factor is why the reference deployment can go into the customer's existing rack next to the rest of the enclave's compute, without a facility upgrade being on the critical path.
The specifics buyers ask about:
- 4U per single-server tier; two 4U slots plus interconnect for the HA cluster. Top-of-rack switchable.
- Entry and middle tiers land on standard 20A/30A rack circuits with dual redundant PSU. HA warrants a PDU review up front.
- Air-cooled at every tier; what matters is rack CFM at the chassis intake.
- 25 GbE to the enclave, 100 GbE optional. InfiniBand only within the HA cluster.
Frequently asked questions
How much VRAM does an on-premises LLM need for 40 concurrent users?
For a 70B open-weight model in BF16 serving roughly 40 concurrent engineering users under a mixed chat + RAG load, plan on 192 GB of total VRAM. About 140 GB is spoken for by the weights. The residual (~40 GB after activation memory and allocator headroom) becomes the KV-cache budget shared across in-flight requests. At a couple hundred kilobytes per token per user, an 8K-context conversation costs each session under 2 GB, and vLLM's PagedAttention allocates only what requests actually consume. That is where the 40-user landing comes from, and why a 192 GB budget delivered as two 96 GB cards is the entry-tier reference. Under-sizing usually happens when the buyer sizes VRAM against weights alone and forgets the KV cache scales with concurrency and context length.
Why use RTX PRO 6000 Blackwell instead of H100 for on-premises LLM inference?
H100 wins on raw FLOP/s and memory bandwidth for hyperscale training and multi-thousand-request inference. For an on-prem enterprise workload sized for tens of concurrent engineering users, the bottleneck is per-card VRAM and KV-cache headroom, and Blackwell delivers 96 GB per card at a workstation-class price. It drops into a 4U air-cooled PCIe server that fits standard enterprise racks and power circuits, and is field-serviceable under a 3-year parts-and-labor warranty. H100 SXM5 assumes an HGX baseboard, hyperscaler-grade cooling, and vendor service — the operational model of a data-center build. When the workload is enterprise inference on customer-owned hardware, Blackwell's per-card VRAM, standard-facility fit, and warranty story land better than raw compute against a stretched thermal envelope.
Does quantization change the VRAM math for on-premises LLM deployments?
Yes, though the savings are smaller than buyers expect. INT8 and FP8 halve the weight footprint and are near-lossless for chat-shaped workloads, which makes them a fair default when VRAM headroom is tight. INT4 halves it again but introduces measurable quality regressions on retrieval-heavy and code-shaped work — a tradeoff worth validating on the customer's own corpus before committing at scale. What matters most is that quantization compresses the weights and leaves the KV cache alone. Concurrency headroom is a KV-cache problem, and dropping to 4-bit weights does not shrink the per-token, per-user allocation that scales with active sessions and context length. Sizing entirely against 'the 4-bit weights fit in 40 GB' produces a server that seats one user comfortably and struggles at plant-shift concurrency.