Technical & Reference

GPU & Workload Guide

NVIDIA GPU reference for ML inference — hardware specs, workload patterns, and the interconnects that tie them together. Updated July 2026.

Start here

You're servingStart withWhy
Up to 8BOne L4 or A10G at FP88B at FP8 is ~10GB, so a 24GB card leaves room for KV cache. T4 still fine for embeddings.
13–34BL40S at FP8, or A100 80GB34B at FP8 is ~41GB. GDDR6 bandwidth is the tradeoff you accept for the price.
70B classH200 at FP8, or 2× H10070GB of weights makes an 80GB card tight once context grows; 141GB is comfortable.
100–405B4–8× H100/H200, or 2–4× B200 at FP4Tensor parallel over NVLink. FP4 halves the footprint if you're on Blackwell.
600B+ frontier MoE8× B300, or NVL72 territoryR1-class MoEs load total parameters, not active ones. This is rack-scale serving.

Will it fit?

— Estimated VRAM
GPUMemorySetup
params × bytes/weight × 1.2 overhead · KV cache is extra and grows with context and batch · MoEs count total params, not active

GPU types

B300 Blackwell Ultra
Released late 2025
Memory288GB HBM3eHBM3e (enhanced)Current-generation high bandwidth memory. 288GB per GPU on B300 — half again more than B200. Stacked on the GPU package for maximum speed.
Bandwidth8 TB/s
FP4FP4 (4-bit float)Ultra-low precision for extreme throughput. Blackwell-exclusive. 4x smaller than FP16. The default serving precision for frontier-scale inference. TFLOPS15,000

Current flagship. 1.5x B200 memory and FP4 compute, built for reasoning-era inference: long contexts and fat KV caches. Ships in GB300 NVL72 racks.

B200 Blackwell
Released 2024
Memory192GB HBM3eHBM3e (enhanced)Current-generation high bandwidth memory. 8 TB/s on B200. Stacked on GPU die for maximum speed. Most expensive memory type.
Bandwidth8 TB/s
FP4FP4 (4-bit float)Ultra-low precision for extreme performance. Blackwell-exclusive. 4x smaller than FP16. Enables massive throughput for quantized inference with specialized model formats. TFLOPS9,000+

The volume Blackwell. 2.5x H100 performance; what most GB200 NVL72 racks are made of.

B100 Blackwell
Released 2024
Memory192GB HBM3eHBM3e (enhanced)Current-generation high bandwidth memory. 8 TB/s on B100. Stacked on GPU die for maximum speed. Most expensive memory type.
Bandwidth8 TB/s
FP4FP4 (4-bit float)Ultra-low precision for extreme performance. Blackwell-exclusive. 4x smaller than FP16. Enables massive throughput for quantized inference with specialized model formats. TFLOPS7,000+

Blackwell mainstream on paper. In practice the market skipped straight to B200 — you will rarely see one.

H200 Hopper
Released late 2023
Memory141GB HBM3eHBM3e (enhanced)Current-generation high bandwidth memory. 4.8 TB/s on H200. 1.4x faster than H100's HBM3. Stacked on GPU die.
Bandwidth4.8 TB/s
FP8FP8 (8-bit float)Low precision for fast inference. Hopper-exclusive. 2x smaller than FP16. Ideal for quantized LLM inference with minimal quality loss. Most common for modern inference. TFLOPS3,958

H100 with more memory. Better for 70B-405B models. 1.4x H100 bandwidth.

H100 Hopper
Released Sep 2022
Memory80GB HBM3HBM3High bandwidth memory generation 3. 3.35 TB/s on H100. Stacked directly on GPU package for ultra-high speeds. 2x faster than A100's HBM2e.
Bandwidth3.35 TB/s
FP8FP8 (8-bit float)Low precision for fast inference. Hopper-exclusive. 2x smaller than FP16. Ideal for quantized LLM inference with minimal quality loss. Most common for modern inference. TFLOPS3,958

The fleet workhorse. Still what clouds have in volume, and the price/performance default for 70B-class serving.

A100 Ampere
Released May 2020
Memory40GB / 80GB HBM2eHBM2eHigh bandwidth memory 2 enhanced. 1.6-2.0 TB/s depending on variant. Previous gen high-end memory before HBM3. Still excellent for most workloads.
Bandwidth1.56 / 2.04 TB/s
FP16FP16 (16-bit float)Half precision. Standard for modern inference. Good balance of speed and quality. 2x smaller than FP32. Widely supported across all frameworks. TFLOPS312

Previous-generation workhorse. Increasingly the value pick as fleets turn over to Hopper and Blackwell.

L40S Ada Lovelace
Released Oct 2023
Memory48GB GDDR6GDDR6Graphics DDR6. Standard GPU memory. 300-864 GB/s depending on configuration. Much cheaper than HBM but slower. Good balance of cost and performance for most workloads.
Bandwidth864 GB/s
FP8FP8 (8-bit float)Low precision for fast inference. Ada Lovelace support via Tensor Cores. 2x smaller than FP16. Good for quantized models with minimal accuracy loss. TFLOPS733

High-memory inference. Llama 70B single GPU, large context windows, vision-language models.

L40 Ada Lovelace
Released Oct 2022
Memory48GB GDDR6GDDR6Graphics DDR6. Standard GPU memory. 300-864 GB/s depending on configuration. Much cheaper than HBM but slower. Good balance of cost and performance for most workloads.
Bandwidth864 GB/s
FP8FP8 (8-bit float)Low precision for fast inference. Ada Lovelace support via Tensor Cores. 2x smaller than FP16. Good for quantized models with minimal accuracy loss. TFLOPS362

Large memory capacity. Multi-modal models, large batch inference, video generation.

L4 Ada Lovelace
Released Mar 2023
Memory24GB GDDR6GDDR6Graphics DDR6. Standard GPU memory. 300-864 GB/s depending on configuration. Much cheaper than HBM but slower. Good balance of cost and performance for most workloads.
Bandwidth300 GB/s
INT8INT8 (8-bit integer)Integer quantization. Very efficient for inference. 4x smaller than FP32. Common for cost-effective deployments. Some quality trade-offs vs floating point. TOPS242

Cost-effective inference. Llama 7B/13B, Whisper, smaller vision models, embeddings.

A10G Ampere
Released 2021
Memory24GB GDDR6GDDR6Graphics DDR6. Standard GPU memory. 300-864 GB/s depending on configuration. Much cheaper than HBM but slower. Good balance of cost and performance for most workloads.
Bandwidth600 GB/s
FP16FP16 (16-bit float)Half precision. Standard for modern inference. Good balance of speed and quality. 2x smaller than FP32. Widely supported across all frameworks. TFLOPS125

Graphics-heavy workloads. ControlNet, image generation, video processing.

T4 Turing
Released Sep 2018
Memory16GB GDDR6GDDR6Graphics DDR6. Standard GPU memory. 300-864 GB/s depending on configuration. Much cheaper than HBM but slower. Good balance of cost and performance for most workloads.
Bandwidth300 GB/s
INT8INT8 (8-bit integer)Integer quantization. Very efficient for inference. 4x smaller than FP32. Common for cost-effective deployments. Some quality trade-offs vs floating point. TOPS130

Entry-level inference. Small language models, embeddings, classification, lightweight tasks.

V100 Volta
Released May 2017
Memory16GB / 32GB HBM2HBM2High bandwidth memory generation 2. 900 GB/s on V100. Older generation of stacked memory. Slower than HBM2e/HBM3 but still faster than GDDR6.
Bandwidth900 GB/s
FP16FP16 (16-bit float)Half precision. Standard for modern inference. Good balance of speed and quality. 2x smaller than FP32. Widely supported across all frameworks. TFLOPS125

Older generation. Still viable for many inference workloads at lower cost.

On the horizon

Rubin Announced
Vera Rubin systems expected late 2026
Memory288GB HBM4HBM4Next-generation stacked memory, successor to HBM3e. Wider interface per stack for another step in bandwidth.
Bandwidth~13 TB/s (claimed)
FP4 TFLOPS~50,000 (claimed)

Next architecture, paired with the Vera CPU. Numbers are NVIDIA's keynote claims, not benchmarks — treat as direction, not spec.

Workload patterns

Single GPU

Most common inference pattern. Model fits entirely in one GPU's memory. Simple, fast, cost-effective.

GPU
Examples

Llama 3.1 8B, Qwen2.5 7B, Whisper, SDXL, embedding and reranker models

Multi-GPU (2x)

Tensor parallelism splits model across 2 GPUs. Needed when model exceeds single GPU memory. Requires fast interconnect.

GPU 1
GPU 2
Examples

Llama 3.3 70B, Qwen2.5 72B, large vision-language models

Multi-GPU (4x+)

Large-scale tensor parallelism for massive models. 4-8 GPUs with NVLink. High complexity, high cost, highest capability.

GPU 1
GPU 2
GPU 3
GPU 4
Examples

Llama 3.1 405B, DeepSeek-R1 671B, frontier-scale MoEs

GPU connectivity

NVLink

Bandwidth (A100)600 GB/s
Bandwidth (H100)900 GB/s
Bandwidth (B200/B300)1.8 TB/s
Latency~1 μs
ConfigurationGPU-to-GPU direct

When it matters: multi-GPU inference for 70B+ models. NVLink enables efficient tensor parallelism by allowing fast weight sharing across GPUs. Critical for maintaining low latency at scale.

PCIe Gen5

Bandwidth (Gen5 x16)64 GB/s
Bandwidth (Gen4 x16)32 GB/s
Latency~10 μs
ConfigurationThrough CPU/chipset

When it matters: fine for single-GPU inference and model loading (Hopper and Blackwell are Gen5 parts). Becomes the bottleneck for multi-GPU tensor parallelism — that is what NVLink is for.

NVSwitch

TopologyAll-to-all mesh
Bandwidth per GPU900 GB/s (H100) / 1.8 TB/s (B200)
Configuration8-GPU DGX up to 72-GPU NVL72

When it matters: full bisection bandwidth between any GPU pair. The GB200/GB300 NVL72 puts 72 Blackwells in one NVLink domain — the unit of account for frontier serving. Expensive but necessary at that scale.