Walkthrough: Design a Realtime LLM Inference Fleet (10K QPS, sub-100ms p95, 7B-70B Models)
A production LLM inference fleet at 10K QPS with sub-100 ms p95 time-to-first-token (TTFT) on 7-70B models is a serious systems engineering problem. The naive “spin up vLLM on H100s behind a load balancer” works at 100 QPS and falls over at 1K. At 10K QPS the system needs aggressive batching, prefix caching, quantization, speculative decoding, disaggregated prefill/decode, and a routing layer that knows which model lives where, plus an autoscaler that doesn’t melt the cluster on traffic spikes.
This walkthrough designs the reference architecture: silicon choice across NVIDIA + AMD + AWS + Google + the wave of new inference-specific accelerators (Groq, Cerebras, SambaNova, Etched, Tenstorrent, Rebellions); serving engines and their optimizations; batching and KV-cache strategies; quantization tradeoffs; routing + autoscaling; observability with the right token-level metrics; and the cost model from $0.10 to $10 per million tokens depending on choices. CAPEX $15-60M for self-hosted; $0/CAPEX with managed inference platforms (Together AI, Fireworks, Cerebras Inference, Modal, RunPod, Lambda Labs, CoreWeave, Replicate, Inferless, Cerebrium, Beam Cloud, Fly.io GPU, Vast.ai).
1. Workload + SLO targets
| Parameter | Target |
|---|---|
| Sustained QPS | 10,000 (peak 25-40K) |
| Models served | 3-8 distinct (mix 7B / 13B / 32B / 70B dense + MoE) |
| Average input length | 500-2,000 tokens |
| Average output length | 50-500 tokens |
| TTFT p50 / p95 | 50 / 100 ms |
| TPOT (time per output token) p50 / p95 | 15 / 30 ms |
| ITL (inter-token latency) p99 | 50 ms |
| End-to-end p95 (200-token gen) | 6.0 s |
| Availability | 99.9% (43 min/mo downtime budget) |
| Cost / 1M output tokens (target) | $0.20-2.00 self-host; $0.30-15 managed |
TTFT < 100 ms is the threshold below which streaming feels “instant”; users tolerate higher TPOT once tokens are flowing. TPOT < 30 ms = ~33 tok/s = faster-than-reading-speed for English.
2. Silicon selection
| Accelerator | FP8/INT8 peak | HBM | Per-GPU $ | $/token strength | Notes |
|---|---|---|---|---|---|
| NVIDIA H100 SXM 80 GB | 1979 TF FP8 | 80 GB | $25-30K | strong baseline | dominant 2024 |
| NVIDIA H100 NVL (PCIe pair) | 1979 TF FP8 | 188 GB pair | $30-35K | 70B fits in pair | inference-tuned |
| NVIDIA H200 SXM 141 GB | 1979 TF FP8 | 141 GB | $32-40K | 70B in single GPU FP8 | sweet spot 2025 |
| NVIDIA L40S 48 GB | 733 TF FP8 | 48 GB | $8-11K | best perf/$ for 7-13B | air-cooled, PCIe |
| NVIDIA L4 24 GB | 121 TF FP8 | 24 GB | $2.5-3.5K | edge / small models | 72W PCIe |
| NVIDIA B200 SXM 192 GB | 4500 TF FP8 / 9000 TF FP4 | 192 GB | $40-55K | frontier inference + training | Blackwell |
| NVIDIA GB200 NVL72 | 720 PF FP8/rack | 13.4 TB | $3-3.5M/rack | rack-scale 70B+ | NVLink mesh |
| AMD MI300X 192 GB | 2610 TF FP8 | 192 GB | $15-20K | 70B FP8 single-GPU, ~25-35% cheaper than H100 | vLLM 0.6+ mature on ROCm |
| AMD MI325X 256 GB | 2610 TF FP8 | 256 GB | $22-28K | 70B + long context single-GPU | inference leader on memory |
| AWS Inferentia2 | 380 TF BF16 | 32 GB | $0.76/chip-hr | best AWS-native $/token | Neuron SDK |
| AWS Trainium2 (also inference) | 2.6 PF FP8 | 96 GB | $1.30/chip-hr | strong frontier-Inf option | Anthropic Rainier |
| Google TPU v5e | 197 TF BF16 / 393 TF INT8 | 16 GB | $1.20/chip-hr | strong serving TCO | GCP |
| Google TPU v6e Trillium | 918 TF BF16 / 1836 TF INT8 | 32 GB | $2.70/chip-hr | inference + small training | GCP |
| Groq LPU | ~750 TF INT8 | 230 MB SRAM only | proprietary | unmatched single-stream TPOT (~250-500 tok/s on 70B), poor batching | ”Groq Inference” API |
| Cerebras WSE-3 | wafer-scale | 44 GB SRAM on-die | $2-3M per CS-3 system | 2200+ tok/s on Llama 70B single-stream | ”Cerebras Inference” API |
| SambaNova SN40L | dataflow + 1.5 TB DRAM/node | 3-tier memory | proprietary | very long context, many concurrent models | enterprise on-prem + API |
| Etched Sohu (transformer ASIC) | 500K tok/s aggregate on Llama 70B (claimed) | 144 GB HBM3e | not yet shipping (2025-26) | extreme $/token if claims hold | first transformer-only ASIC |
| Tenstorrent Wormhole p150 / Blackhole p100 | open arch, RISC-V + Tensix | 12-32 GB GDDR6 | $1.4-1.5K (Wormhole), $10-15K (Blackhole) | hobbyist + commercial low-end | open SDK, growing |
| Rebellions ATOM | NPU 32 TFLOPS | 16 GB | proprietary | Korean fab, Samsung Cloud | enterprise APAC |
| AWS Graviton4 (CPU) | for small models | up to 768 GB DDR5 | $3-4/vCPU-hr | sub-3B int4 models reasonable | edge / cost-extreme |
Recommendation for 7-70B mix at 10K QPS: a heterogeneous fleet — L40S for 7B (cheapest per-token), H100 NVL or MI300X for 32B-class, H200 or MI325X single-GPU for 70B FP8, GB200 NVL72 for 200B+ MoE when added later. Send latency-critical low-volume traffic to Groq or Cerebras Inference via API as overflow. Pure managed (Together AI, Fireworks) for Year 1 until utilization > 50% on owned silicon — crossover is usually 30-60K$/mo of API spend per replicated model.
3. Serving engines
| Engine | Strengths | Weaknesses | Best for |
|---|---|---|---|
| vLLM (0.6+) | PagedAttention, continuous batching, prefix cache, speculative decoding, FP8 + INT4 + GPTQ + AWQ + FP4 (Blackwell); multi-LoRA; broad model support; CUDA + ROCm; TPU + Neuron in progress | Python overhead, occasional perf regressions per release | open-weight serving, multi-model fleets |
| SGLang | RadixAttention shared-prefix cache, grammar-constrained decoding (JSON, regex), structured output, fastest on multi-call / agent workloads | younger ecosystem | agentic / structured outputs |
| TensorRT-LLM | absolute peak throughput on NVIDIA, FP8/FP4 first-class, in-flight batching | NVIDIA-only, complex build pipeline, harder to swap models | when last 20% of perf matters |
| TGI (HuggingFace) | mature, easy HF model loading | slightly behind vLLM on throughput 2024-25 | HF ecosystem alignment |
| LMDeploy (InternLM) | top throughput on H100 for some models, INT4 first-class | smaller community | Internlm + Qwen-heavy workloads |
| DeepSpeed-MII | Microsoft-aligned | declining vs vLLM | legacy |
| Triton Inference Server | multi-framework orchestrator (wraps TRT-LLM, vLLM, TGI) | extra layer | multi-model production |
| Ray Serve | scales out serving, K8s-native | LLM-specific perf < dedicated engines | multi-model + multi-framework |
| BentoML | model packaging + serve, Python-friendly | thin perf layer | rapid productionization |
| LitServe (Lightning) | fast Python serving, batteries-included | newer | small teams |
| Replicate Cog | container packaging + Replicate cloud | Replicate-tied | quick deploy |
| Modal | serverless GPU, batteries-included | Modal-tied | serverless workloads |
Recommendation: vLLM as primary; SGLang for agentic + JSON-grammar workloads; TensorRT-LLM only for the highest-volume single-model endpoint where engineering cost is justified. Run via Triton or Ray Serve when serving 3+ models behind one autoscaling pool.
4. Throughput optimizations
These are the levers that move a single H100 from 200 to 2,000+ requests per minute on 70B.
4.1 Attention + KV-cache
| Technique | What it does | Impact |
|---|---|---|
| PagedAttention (vLLM) | non-contiguous KV-cache blocks (like OS paging) | 2-4× throughput vs naive batching |
| RadixAttention (SGLang) | radix-tree of shared prefixes, deduped KV | 3-10× on agentic + few-shot |
| Chunked prefill | break long prompts into chunks, overlap with decode | smooths TTFT under load |
| Prefix caching | reuse KV for common system prompt across requests | 2-30× TTFT improvement |
| KV-cache sharing across replicas (LMCache) | offload to CPU / disk / remote | longer effective context, lower memory pressure |
| FlashAttention 3 | optimized GPU attention kernel | baseline; ~1.5-2× over FA2 on H100 |
| PagedKV with FP8 | quantize KV to FP8 | ~2× KV memory savings |
4.2 Quantization
| Format | Bits | Quality loss (vs BF16) | Speedup | Notes |
|---|---|---|---|---|
| FP8 (E4M3 / E5M2) | 8 | ~0.1-0.3% | ~1.5-2× | native H100/H200/B200 |
| INT8 | 8 | ~0.3-0.8% | ~1.5-2× | older GPUs |
| GPTQ / AWQ | 4 | ~0.5-2% | ~2.5-3.5× | weight-only quant, broad support |
| GGUF (k-quants) | 2-8 | varies | varies | llama.cpp ecosystem, CPU + GPU |
| EXL2 (ExLlamaV2) | 2.5-8 | varies | ~3× | community favorite |
| HQQ | 4 | ~0.5-1.5% | ~2.5× | fast calibration-free |
| AQLM | 2 | ~3-5% | ~4× | extreme compression |
| QAT (quant-aware train) | 4/8 | ~0.1-0.5% | as above | post-train recovery |
| FP4 (Blackwell-only) | 4 | ~0.3-1% | ~2× over FP8 | Microscaling MXFP4/MXFP6 |
Recommendation: FP8 on H100/H200, AWQ-INT4 on L40S and older, FP4 on B200/GB200. Calibrate on a representative eval set; always measure downstream quality, not just perplexity.
4.3 Speculative decoding + multi-token prediction
| Technique | Speedup | Notes |
|---|---|---|
| Speculative decoding (draft model + verify) | 1.5-3× | classic — Llama 70B verified by Llama 7B draft |
| Medusa (multi-head draft) | 2-2.5× | added prediction heads, no separate draft model |
| EAGLE / EAGLE-2 | 2.5-3.5× | feature-level draft, top performer 2024 |
| Lookahead decoding / Lookahead-2 | 1.5-2× | n-gram-based, no extra model |
| REST (retrieval-based speculative) | 1.5-2× | retrieves candidate continuations |
| Multi-Token Prediction (MTP) | 1.5-2× | DeepSeek-V3 native; train-time |
| Speculative streaming (Apple) | ~2× | small dedicated speculator head |
| n-gram speculative (vLLM 0.6+) | 1.2-1.5× | no model needed, hot prefix-based |
Stacking: FP8 + speculative + continuous batching + prefix cache often compounds to 5-8× over baseline; rarely additive past 8×.
4.4 Batching strategies
- Static batching — fill batch, run, return all. Obsolete for chat.
- Dynamic batching (Triton classic) — group within window. Acceptable for fixed-length tasks.
- Continuous batching (vLLM, TGI default) — add/remove sequences each step. Required for serving chat at >100 QPS.
- Disaggregated Prefill/Decode (DistServe / Splitwise) — separate GPU pool for prefill (compute-bound) vs decode (memory-bound). 1.5-3× throughput at same SLO.
- Mooncake (Kimi-K2) — disaggregated PD with KV-cache transfer over RDMA; production at Moonshot AI scale.
- LMCache — KV-cache offload + reuse across model instances.
For 10K QPS mixed workload, disaggregated PD is now table-stakes — separate prefill replicas (B200 or H200, optimized for compute) and decode replicas (cheaper L40S / A100 / MI300X, optimized for memory bandwidth) connected by RDMA.
5. Routing + autoscaling
A 10K-QPS fleet has 5-50 model replicas across 2-8 distinct models in 2-3 geo regions. Routing layer responsibilities:
| Concern | Tool / pattern |
|---|---|
| Model selection (auto-route easy queries to cheap models) | RouteLLM (Anyscale, 2024), ATM (Adaptive Token Management), conditional compute, model cascades |
| Replica selection (load-balance among replicas of same model) | weighted round-robin with TTFT-aware bias, KV-cache-affinity routing (route to replica with hottest prefix cache) |
| Region selection | geo-DNS + latency-based routing (Cloudflare LB, AWS Global Accelerator) |
| Tenant/role-aware routing | tenant-tier → model assignment (free → Haiku, paid → Sonnet, enterprise → Opus) |
| Failover | circuit breaker → next region → managed API overflow |
Autoscaling:
| Tool | Best for |
|---|---|
| Knative Serving | scale-to-zero serverless functions |
| KServe | K8s-native model serving with auto-scaling on QPS or queue depth |
| KEDA | event-driven scale on Prometheus metrics (queue depth, TTFT p95) |
| Run:ai (NVIDIA) | GPU sharing + fractional allocation + queue management |
| Anyscale | Ray-based autoscaling |
| SkyPilot | multi-cloud GPU autoscale |
| nyx / kueue | K8s gang scheduling for LLM serving |
Managed serverless GPU platforms (when not owning silicon):
| Platform | Strength | $/H100-hr (typical 2025) |
|---|---|---|
| Modal | dev-friendly, fast cold start | $1.95-3.50 |
| Replicate | quick prototype + Cog packaging | $5+ via Cog |
| Inferless | sub-30s cold start | $2-4 |
| Cerebrium | serverless + agent workflows | $2-3.50 |
| Beam Cloud | serverless Python | $2-3 |
| Fly.io GPU | edge GPU with Fly platform | $2.50-3.50 |
| Vast.ai | spot marketplace (cheapest) | $1.20-2.50 |
| RunPod | community + secure cloud | $1.99-2.49 secure, $1.10-1.80 community |
| Lambda Labs | reserved + on-demand | $2.49 H100 SXM |
| CoreWeave | dedicated cluster + IB | $2.30-3.50 H100 SXM |
| Together AI | hosted vLLM-style serving API | per-token pricing |
| Fireworks AI | hosted optimized serving | per-token pricing |
| Cerebras Inference | extreme speed on supported models | per-token pricing |
Autoscaling rule of thumb: scale at queue-depth > N (where N = batch-size × 2) OR TTFT p95 > 80% of SLO. Cold start on a fresh GPU replica with model loaded is 30-90 s — keep N+2 warm replicas, never scale to zero on production.
6. Observability + KPIs
Inference observability is fundamentally different from web service observability. Key metrics:
| Metric | Why it matters | Alert at |
|---|---|---|
| TTFT (p50/p95/p99) | user-perceived latency | p95 > SLO |
| TPOT (p50/p95) | streaming smoothness | p95 > SLO |
| ITL (p99) | jitter perception | p99 > 2× p50 |
| Tokens/sec/replica | throughput | drop > 10% week-over-week |
| Queue depth | saturation early signal | > 2× batch-size sustained |
| KV-cache utilization | scaling signal | > 90% sustained |
| Prefix cache hit rate | cost + latency | < 30% on prompt-heavy workload |
| GPU SM utilization | are we GPU-bound? | < 60% means batching wrong-sized |
| GPU HBM memory pressure | OOM risk | > 92% |
| Cost per million tokens | unit economics | trend |
| Cost per request | per-tenant attribution | trend |
| Error rate by class | reliability | > 0.5% |
| Cache hit rate (response cache) | cost + latency | < 20% on bursty workload |
Tools: NVIDIA DCGM-exporter + Prometheus + Grafana for GPU; OpenTelemetry GenAI semantic conventions for token-level traces; Langfuse or Phoenix or Helicone for trace + cost. Datadog and New Relic added LLM Observability views 2024-25.
7. Cost model
7.1 Managed API pricing 2025-26 (per 1M tokens)
| Provider | Model | In | Out |
|---|---|---|---|
| Anthropic | Claude Sonnet 4.7 | $3.00 | $15.00 |
| Anthropic | Claude Haiku 4.5 | $1.00 | $5.00 |
| Anthropic | Claude Opus 4.7 | $15.00 | $75.00 |
| OpenAI | GPT-5 | $5.00 | $20.00 |
| OpenAI | GPT-5 mini | $0.40 | $1.60 |
| Gemini 2.5 Pro | $1.25 | $10.00 | |
| Gemini 2.5 Flash | $0.30 | $2.50 | |
| DeepSeek | DeepSeek-V3.1 | $0.27 | $1.10 |
| Together AI | Llama 4 Maverick | $0.27 | $0.85 |
| Together AI | Llama 4 Scout | $0.18 | $0.59 |
| Fireworks | Llama 3.3 70B | $0.20 | $0.60 |
| Fireworks | DeepSeek-V3 | $0.90 | $0.90 |
| Cerebras Inference | Llama 3.3 70B (2000+ tok/s) | $0.85 | $1.20 |
| Groq | Llama 3.3 70B (250+ tok/s) | $0.59 | $0.79 |
7.2 Self-host break-even
For Llama 3.3 70B at 10K QPS sustained, 200 tok output average:
| Path | $/1M output tokens |
|---|---|
| Anthropic Sonnet (frontier quality, not Llama) | $15.00 |
| OpenAI GPT-5 (frontier) | $20.00 |
| Together AI Llama 4 Maverick | $0.85 |
| Self-hosted vLLM on 16× H200 (FP8 + speculative) | $0.20-0.40 |
| Self-hosted vLLM on GB200 NVL72 (FP4 + speculative) | $0.08-0.20 |
Crossover for self-hosting an open-weight model: ~$50-100K/mo of API spend on the same model. Below that, managed wins on TCO + ops simplicity.
7.3 Reference CAPEX (16× H200 self-host)
| Item | Cost |
|---|---|
| 16× H200 SXM in 2× HGX 8-GPU nodes | $640,000 |
| Storage + networking + rack | $180,000 |
| Power + cooling fitout | $140,000 |
| Engineering setup + integration | $140,000 |
| CAPEX total | ~$1.1M |
Year-1 OpEx: $280K power + $180K SRE + $120K support + $60K observability + $40K spares = ~$680K. At sustained $150K/mo API equivalent, payback ~6-9 months.
8. Multi-tenancy + safety
- Per-tenant quota — RPS + tokens/min + monthly spend cap, enforced at routing layer with Redis token-bucket.
- Tenant-scoped prompts + LoRA — vLLM multi-LoRA serves 100+ adapters on one base model; route by tenant.
- PII redaction pre-inference — Microsoft Presidio inline filter for regulated tenants.
- Output safety — Llama Guard 3 or Lakera Guard on output for jailbreak / unsafe content; see design-agent-platform-deployment.
- Audit log — every request → trace store with tenant-id, model-id, input hash, output hash, tokens, cost, latency; 90-day retention default, configurable per tenant.
9. Risk register
- Cold start tax — fresh GPU replica with 70B weights load is 30-90 s; never autoscale-to-zero in production; warm pool minimum 2 replicas per model.
- KV-cache OOM under bursty traffic — sudden long-prompt burst exhausts paged KV blocks; admit-control with queue-then-degrade beats crash-then-restart.
- Model drift on quantization — FP8/INT4 calibration on dev data fails on production tail; eval calibration on shadow-traffic before flip.
- Vendor concentration — pure single-cloud GPU sourcing is fragile; keep a managed-API overflow path (Together / Fireworks / Cerebras) always-warm.
- GPU supply shock — H100/H200/B200 spot prices doubled then halved 2023-2025; reservation contracts at 30-50% of on-demand are the play.
- Token-cost runaway from agentic loops — per-tenant + per-conversation token budget hard-cap; circuit-break on cost-per-conversation > N×p95.
- Silent quality regression on serving engine upgrade — keep two-replica canary + golden-eval CI gating every vLLM/TGI/SGLang version bump.
10. Reference deployments
- Anthropic — Claude served on AWS Trainium2 (Project Rainier) + NVIDIA H100/H200 (Bedrock + first-party); prompt-caching API with 90% discount on cached prefix.
- OpenAI — GPT-5 served on Azure H100/H200 + custom silicon roadmap; prompt cache automatic.
- Google — Gemini served on TPU v5p/v6e; context caching API with TTL.
- Together AI — open-weight serving fleet on H100; ~$0.20-0.90 per 1M tokens.
- Fireworks AI — open-weight serving + fine-tuning + function calling; FireOptimizer auto-tunes batching.
- Cerebras Inference — Llama 3.3 70B at 2200+ tok/s single-stream on CS-3.
- Groq — Llama / Mixtral at 250-500 tok/s on LPU; weak on batching but unmatched single-stream.
- Mistral / La Plateforme — Mistral models served on French sovereign infra.
- DeepSeek Chat — DeepSeek-V3 served at ~$0.27/1M in / $1.10/1M out, with off-peak discount.
The pattern: every serious inference operator runs an internal heterogeneous fleet + a managed-API overflow + a routing layer that hides the heterogeneity from product code.
11. Build plan (8-week zero-to-production)
| Week | Milestone |
|---|---|
| 1 | Stand up vLLM on dev GPUs; serve 7B + 70B; baseline TTFT/TPOT |
| 2 | Add FP8 + AWQ + prefix cache; rerun benchmarks |
| 3 | Speculative decoding (EAGLE-2 or n-gram); disaggregated PD prototype |
| 4 | Routing layer + per-tenant quota + multi-LoRA |
| 5 | Observability — DCGM + OTel + Langfuse; SLO dashboards |
| 6 | Autoscaling (KEDA + KServe); warm-pool sizing; chaos test |
| 7 | Managed-API overflow (Together / Fireworks); failover drills |
| 8 | Load test to 2× target QPS; eval gate on golden set; ship |
12. Adjacent
- design-llm-training-cluster — where the model weights come from
- design-rag-at-scale-system — the retrieval feeding inference
- design-agent-platform-deployment — the agent layer above inference
- design-data-center-cooling-system — physical layer for self-hosted GPUs
- distributed-systems-fundamentals — replicas, queues, backpressure
- observability-stack — token-level metrics + tracing
- design-saas-platform-launch — the surrounding multi-tenant SaaS