Walkthrough: Design a 1024-GPU Foundation-Model Pretraining Cluster
A 1024-GPU cluster is the modern minimum for serious foundation-model pretraining — enough to train a 70B-parameter dense model on 5-15T tokens in 30-90 days, or a Mixture-of-Experts (MoE) model with 400B+ total / 30-50B active in similar wall-clock. Below 512 GPUs you cannot afford the data-parallel batch size to reach the Chinchilla-optimal token ratio in a reasonable schedule; above 8192 GPUs you are doing what xAI Colossus (200K H100, expanding to 1M H200), Meta’s 24K H100 Llama 4 training cluster, Anthropic’s Project Rainier (multi-hundred-thousand Trainium2 with AWS), and Microsoft Stargate (the announced $100B+ joint OpenAI-Microsoft buildout) are doing — a different cost class and a different operations problem.
This walkthrough builds the 1024-GPU reference: hardware selection across NVIDIA / AMD / Google / AWS silicon, the fabric, the storage, the orchestrator, the training framework, the observability surface, the cooling and power envelope, and the financial model. CAPEX lands at $70-110M depending on accelerator choice; OpEx $10-25M/yr. Reference builds: Meta Grand Teton + RSC, Microsoft Mt Hood + Azure Cobalt100, Tesla Dojo D1, Google Trillium TPU v6e and Ironwood TPU v7, AWS Trainium2/3 UltraServers.
1. What 1024 GPUs gets you
| Workload | Wall-clock at 1024× H100 | Wall-clock at 1024× B200 |
|---|---|---|
| 70B dense, 2T tokens (Chinchilla 2x) | ~55 days | ~22 days |
| 70B dense, 15T tokens (Llama 3.1 70B class) | ~410 days (impractical) | ~165 days |
| 405B dense, 15T tokens (Llama 3.1 405B class) | infeasible at this scale | ~950 days (infeasible) |
| MoE 400B-total / 50B-active, 8T tokens | ~95 days | ~38 days |
| 8B dense, 3T tokens (Llama 3 8B class) | ~7 days | ~3 days |
A 1024-GPU pod is the right size for: pretraining 7-70B dense and 50-150B-active MoE; high-throughput finetuning of frontier-scale checkpoints; full data-parallel ablation sweeps on architecture choices. Bigger frontier models (200B+ dense, 1T+ MoE) need a multi-pod build.
2. Accelerator selection: NVIDIA vs AMD vs Google TPU vs AWS Trainium
| Silicon | Per-GPU peak (BF16/FP8) | HBM | Interconnect | List price (2025-26) | Per-GPU $/TFLOPS·yr (3yr amort) |
|---|---|---|---|---|---|
| NVIDIA H100 SXM5 80 GB | 989 TF BF16 / 1979 TF FP8 | 80 GB HBM3 | NVLink 4 (900 GB/s) + IB NDR | $25-30K | best documented |
| NVIDIA H200 SXM 141 GB | 989 TF BF16 / 1979 TF FP8 | 141 GB HBM3e | NVLink 4 + IB NDR | $32-40K | +memory for inference |
| NVIDIA B200 SXM 192 GB | 2.25 PF BF16 / 4.5 PF FP8 / 9 PF FP4 | 192 GB HBM3e | NVLink 5 (1.8 TB/s) + IB XDR or Spectrum-X | $40-55K | best 2025 dense perf/$ |
| NVIDIA B300 / Blackwell Ultra | 2.5 PF BF16 / 5 PF FP8 / 15 PF FP4 | 288 GB HBM3e | NVLink 5 + Quantum-X800 | $60-75K (early 2026) | best 2026 perf/$ |
| NVIDIA GB200 NVL72 (rack-scale) | 720 PF FP8 per rack (36×B200 pairs) | 13.4 TB HBM3e/rack | NVLink 5 fully meshed in rack | $3.0-3.5M per rack | rack-scale TCO leader |
| NVIDIA GB300 NVL72 / NVL576 (2026) | 1.4 EF FP4 per NVL72 | 20 TB HBM3e/rack | NVLink 5 + Quantum-X800 800G | $4.5-5.5M per rack | frontier 2026 platform |
| AMD MI300X 192 GB | 1.31 PF BF16 / 2.61 PF FP8 | 192 GB HBM3 | Infinity Fabric (896 GB/s) + RoCE | $15-20K | ~30-40% cheaper than H100, lags FP8 maturity |
| AMD MI325X 256 GB | 1.31 PF BF16 / 2.61 PF FP8 | 256 GB HBM3e | Infinity Fabric + UALink (roadmap) | $22-28K | inference-leaning |
| AMD MI355X (CDNA 4, mid-2025) | 2.3 PF FP8 / 4.6 PF FP4 | 288 GB HBM3e | Infinity Fabric + UALink | $30-40K | first credible AMD training challenger |
| Google TPU v5p | 459 TF BF16 / 918 TF INT8 | 95 GB HBM | ICI 4800 Gb/s in 8960-chip pod | not sold — $2-4/chip-hr GCP | best TCO at scale, GCP-locked |
| Google TPU v6e Trillium | 918 TF BF16 / 1836 TF INT8 | 32 GB HBM | ICI in 256-chip pod | $2.70-4.20/chip-hr GCP | inference + small training |
| Google TPU v7 Ironwood (2026) | 4.6 PF FP8 | 192 GB HBM3e | ICI in 9216-chip pod | not yet retail | Google’s frontier-class TPU |
| AWS Trainium2 | 1.3 PF BF16 / 2.6 PF FP8 | 96 GB HBM3 | NeuronLink in UltraServer 64-chip | ~$1.30/chip-hr us-east-1 | Anthropic Project Rainier silicon |
| AWS Trainium3 (2026) | ~2x Trainium2, FP4 | 128 GB HBM3e | NeuronLink-v2 + EFA | not yet retail | follows Anthropic offtake |
Recommendation for 1024-GPU greenfield Q3 2025+ build: NVIDIA B200 SXM in 8-GPU HGX baseboards, 128 nodes — $50-65M silicon; or 14× GB200 NVL72 racks (1008 GPUs) at $45-55M for the rack-scale variant. GB200 NVL72 wins on training perf/$ because the 72-GPU NVLink domain replaces 9 InfiniBand hops with NVLink for the tensor-parallel and expert-parallel dimensions — see distributed-systems-fundamentals.
For organizations not GPU-locked, Google TPU v5p / v6e via GCP commitments offers ~20-35% lower $/token-trained at frontier scale but ships you fully onto JAX + MaxText + GCP — a different sovereignty story.
3. Network fabric: the cluster’s other half
A 1024-GPU pod has three logical networks layered on the physical fabric:
| Network | Bandwidth target | Protocol | Vendor stack |
|---|---|---|---|
| Compute backplane (collective ops, AllReduce / AllGather / AlltoAll) | 400-800 Gb/s per GPU | InfiniBand NDR / XDR, or RoCEv2, or NVLink within scale-up domain | NVIDIA Quantum-2 QM9700, Quantum-X800, NVLink Switch (NVSwitch) |
| Storage + management | 100-200 Gb/s per node | RoCEv2 over Ethernet, or NFS over IB | Spectrum-4 SN5600 (Ethernet), Quantum-2 (IB) |
| Out-of-band BMC / IPMI | 1-10 Gb/s | Ethernet | any 1G/10G ToR |
Compute backplane choices:
- InfiniBand NDR 400 Gb/s (NVIDIA Quantum-2 QM9700, 64 ports of 400G): the default for NVIDIA training clusters. Lossless, RDMA-native, ~700 ns switch latency. 8 NICs per H100/H200 node (one ConnectX-7 per GPU) on a non-blocking rails-optimized fat tree. For 1024 GPUs: 2 layers (128 ToR + 16 spine) of QM9700 — ~144 switches $40-60K each = $6-9M just in fabric.
- InfiniBand XDR 800 Gb/s (Quantum-X800 Q3400): 144 ports of 800G in a single chassis; collapses 2 layers into 1 at this scale. 2025 product; ~$120-180K per chassis. Required for B200/GB200 to feed full NVLink-5 egress.
- Spectrum-X (NVIDIA Ethernet, 800 GbE) + ConnectX-8 NICs: lossless RoCEv2 with NVIDIA’s adaptive routing + congestion control. Comparable to IB on AllReduce within ~5%, easier to integrate with existing Ethernet ops. Hyperscalers (Meta, Microsoft) are leaning toward Spectrum-X or in-house Ethernet for new builds.
- AWS EFA (Elastic Fabric Adapter) — only relevant in AWS-hosted clusters; SRD transport, comparable to RoCEv2.
- UALink (Ultra Accelerator Link) + Ultra Ethernet consortium (UEC) — emerging open standards for 2026-27, AMD/Intel/Broadcom-led; not yet deployable.
NVLink + NVSwitch (scale-up domain): within a GB200 NVL72 rack, NVLink-5 connects all 72 GPUs in a single coherent memory domain via 9 NVSwitch trays — 1.8 TB/s per GPU, 130 TB/s aggregate. This collapses what used to be cross-node tensor parallelism into in-rack NVLink, which is ~10× faster than IB.
Rails-optimized topology: for 8-NIC servers, each rail connects to a separate leaf switch, so a single AllReduce uses 8 parallel network paths and avoids hot-spot collision. See realtime-embedded for network-determinism context.
4. Storage tier
Foundation-model training reads training data (~1-10 PB tokenized) and checkpoints every 30-120 min (each checkpoint 1-3 TB for a 70B model, 8-20 TB for a 400B MoE). Three tiers:
| Tier | Capacity | Throughput | Vendors | Cost |
|---|---|---|---|---|
| Hot scratch (parallel filesystem) | 1-5 PB NVMe | 1-5 TB/s aggregate | VAST Data, WekaIO, DDN ES400NVX2 / EXA6, Pure FlashBlade//S, IBM Storage Scale (GPFS), Lustre on Cray ClusterStor | $1.50-3.50/GB raw |
| Warm dataset (object) | 5-50 PB | 100-500 GB/s | MinIO, Cloudian HyperStore, Ceph (RADOS), VAST (unified) | $0.30-0.80/GB raw |
| Cold archive | 50-500 PB | 1-10 GB/s | AWS S3 Glacier Deep Archive ($0.00099/GB-mo), Azure Archive ($0.00099/GB-mo), tape (LTO-9 $15/TB media) | $0.001-0.10/GB-mo |
Recommended baseline: VAST Data Universal Storage (3 PB usable, 2 TB/s) for hot — $6-8M; Cloudian or Ceph for warm bulk dataset; S3 Glacier for archive. VAST and WekaIO are the two architectures actually shipping at >2 TB/s sustained for AI training in 2025; DDN’s EXA6 ships in the same class for the HPC-native customers. Pure FlashBlade//S is the easiest to operate but tops out lower in raw bandwidth.
Checkpoint write strategy: distributed checkpointing (PyTorch DCP, MosaicML Streaming, Megatron checkpoint sharding) so each rank writes only its shard — 1024 ranks each writing 1-3 GB completes in 20-60 s versus 10-30 min for a serialized rank-0 dump.
5. Cluster orchestration + scheduling
Two camps: HPC-native (Slurm) and Kubernetes-native (Volcano + KubeRay).
| Stack | Strength | Weakness | Used by |
|---|---|---|---|
| Slurm 24.05 + Pyxis (NVIDIA container plugin) + Enroot | dense per-job GPU scheduling, gang scheduling first-class, decades of HPC ergonomics | weak multi-tenancy, no native autoscaling, awkward for serving | OpenAI training, Meta RSC, Anthropic, most national labs |
| Kubernetes + Volcano + KubeRay | unified train+inference platform, integrates with cluster autoscaler, MLOps-friendly | gang scheduling bolt-on, GPU topology awareness immature | CoreWeave, Lambda Labs, internal hyperscaler training-as-a-service |
| Determined.ai (HPE) | research-team UX, hyperparameter sweeps, integrated experiment tracking | smaller community, vendor risk | enterprise R&D teams |
| MosaicML Composer + MCLI (now Databricks) | turn-key recipes, optimized DDP/FSDP, Streaming dataset | Databricks-tied | Databricks-hosted training |
| Ray + KubeRay | flexible, Python-native, good for RL + agent pipelines | not ideal for pure synchronous pretrain | Anyscale, RL pipelines |
Recommendation: Slurm 24.05 + Pyxis for the pretraining workload itself (gang scheduling, MPI integration, predictable performance), with a separate small Kubernetes pool (KubeRay) for ablation jobs, eval harnesses, and inference smoke tests. This is the architecture Meta, Anthropic, Mistral, and most research labs converge on.
Containers: Apptainer (formerly Singularity) for HPC sites; Enroot for NVIDIA-native sites; both are rootless and avoid Docker daemon. NVIDIA NGC containers (PyTorch + CUDA + NCCL + cuDNN preinstalled) are the lingua franca.
6. Training framework + parallelism
For a 70B+ dense or 400B+ MoE on 1024 GPUs, raw PyTorch DDP is not viable — communication overhead dominates. Pick one:
| Framework | Parallelism dims | Best for | Notes |
|---|---|---|---|
| Megatron-LM / Megatron-Core (NVIDIA) | TP + PP + DP + EP + CP | dense + MoE, NVIDIA-tuned | Reference for NVIDIA-stack pretraining; Llama 4 trained on Megatron-derived stack |
| DeepSpeed (ZeRO 1/2/3) + DeepSpeed-MII | ZeRO sharding + 3D | dense, memory-constrained | Microsoft; integrates with HuggingFace Trainer |
| PyTorch FSDP2 (2024 GA) | fully-sharded data parallel + TP | flexible, native PyTorch | Replaces FSDP1; better composability with TP via DeviceMesh |
| ColossalAI | 3D + ZeRO + Gemini offload | open research, China-popular | Less production-hardened |
| Levanter (Stanford CRFM, JAX) | FSDP + TP via jax.sharding | TPU + GPU, reproducibility-first | Bit-exact resumption, used for Marin |
| MaxText (Google) | JAX-native, TPU-optimized | TPU pods | Reference for TPU training |
| GPT-NeoX (EleutherAI) | 3D parallel via Megatron-DeepSpeed | open-source pretraining | Pythia, OpenLLaMA derivative work |
Recommendation for B200/GB200 + 1024 GPUs: Megatron-Core with TP=8 (within NVL72 NVLink domain or within HGX node), PP=8 (across racks), DP=16, with EP=8 for MoE. FSDP2 for 7-13B dense where TP overhead exceeds memory benefit. See distributed-systems-fundamentals.
Mixed precision: FP8 (Transformer Engine on Hopper/Blackwell) for forward + backward; BF16 master weights; FP4 (Blackwell-only) for inference. ~1.5-2× throughput vs BF16 with ~0.1-0.3% downstream eval delta when calibrated.
7. Data pipeline
Tokenized training data lives upstream of the cluster. Pipeline:
- Crawl + collection: Common Crawl snapshots (250-400 TB/dump), curated repos (Wikipedia, ArXiv, StackExchange), licensed data (books, news, code repos under license), synthetic data (model-generated).
- Filtering + dedup: AI2 Dolma pipeline, RefinedWeb-style heuristics, FineWeb-Edu (educational filter), MinHash + LSH dedup at document + paragraph + line level. Datology AI, Cleanlab, and Snorkel offer commercial pipelines.
- Quality classification: educational-value classifier (Llama 3 ed classifier, FineWeb classifier), toxicity + PII redaction (Microsoft Presidio).
- Labeling / annotation (post-train data): Argilla, Scale AI, Surge AI, Lilac (Databricks) for human-feedback labels.
- Tokenization: SentencePiece or HuggingFace tokenizers; vocabulary 100K-200K (Llama 3 = 128K, GPT-4o = 200K).
- Sharding + streaming: MosaicML Streaming (MDS format), WebDataset (tar shards), Megatron’s
IndexedDataset. Stream from object store (Cloudian / S3) with prefetch — avoids loading 10 TB onto every node’s scratch.
A clean 5-10T-token mix takes 2-4 PB of raw crawl input and 3-6 weeks of CPU preprocessing on a 500-1000-core cluster. Plan for it as a parallel workstream, not an afterthought.
8. Cluster software baseline
| Component | Version (2025-26) | Role |
|---|---|---|
| OS | Ubuntu 22.04 LTS or RHEL 9.4 | NVIDIA-certified |
| Kernel | 6.5+ (with NVIDIA Open GPU Kernel Modules) | required for B200 |
| NVIDIA driver | 560+ | B200/GB200 support |
| CUDA | 12.6+ | FP8 + Transformer Engine |
| cuDNN | 9.x | conv + attention kernels |
| NCCL | 2.22+ | collectives; SHARP offload on Quantum-2 |
| Mellanox OFED | 24.07+ | InfiniBand drivers + libraries |
| UCX | 1.17+ | unified communication transport |
| Slurm | 24.05 | scheduler |
| Pyxis + Enroot | latest | container runtime |
| Apptainer | 1.3+ | alternative container runtime |
| Python | 3.11 | training runtime |
| PyTorch | 2.5+ (with Triton 3.1) | framework |
| Transformer Engine | 1.10+ | FP8 kernels |
| Megatron-Core | 0.9+ | parallelism |
A cluster-config-as-code repo (Ansible + Terraform) is the only sane way to manage 128-256 nodes. Updates are blue/green or rolling-with-drain.
9. Observability
GPUs fail. NCCL hangs. Power supplies trip. Without per-GPU + per-rank + per-job observability, a 30-day training run loses 15-40% wall-clock to silent stalls.
| Layer | Tool | What it captures |
|---|---|---|
| GPU metrics | NVIDIA DCGM + dcgm-exporter → Prometheus | utilization, SM occupancy, HBM ECC errors, NVLink errors, thermal throttle, power |
| Fabric | UFM (Unified Fabric Manager, NVIDIA) | IB link health, congestion, BER, port flap |
| Cluster | Prometheus + Grafana + Loki | metrics + logs aggregation |
| Distributed trace | Jaeger + OpenTelemetry | rank-to-rank dependencies |
| Job profiling | NVIDIA Nsight Systems, PyTorch Profiler, Pixie | per-step kernel timing |
| Cloud Native Stack | NVIDIA Cloud Native Stack 12+ | bundled reference deployment |
| Commercial | Datadog (DCGM integration), New Relic | optional |
Loss curves + gradient norms + token/sec throughput shipped to Weights & Biases or MLflow. SRE-style alerting on: throughput drop >10% from baseline (likely a stuck rank), HBM ECC error rate spike, NCCL timeout, loss-spike detection (NaN trap + autorollback). See observability-stack.
10. Power, cooling, and physical layout
A 1024-GPU B200 cluster draws ~1.5 MW IT load (1.2 kW/GPU × 1024 + CPU/NIC/storage). A GB200 NVL72 deployment at 14 racks × 132 kW = 1.85 MW IT — plus 0.3-0.4 MW for storage, network, and cluster mgmt. Plan 2.2-2.5 MW total facility load including PUE.
Rack-scale options (OCP-aligned and proprietary):
| Vendor | Reference platform | Form factor |
|---|---|---|
| NVIDIA-MGX | open spec, multiple ODMs | modular |
| GB200 NVL72 (NVIDIA + ODMs) | 72-GPU rack, NVLink-meshed | 132 kW liquid |
| Supermicro SYS-A21GE-NBRT | GB200 NVL72 implementation | rack-scale |
| Foxconn / Hon Hai | GB200 NVL72 ODM | rack-scale |
| Inspur NF5688M7 | HGX H200 8-GPU | 10 kW/node air |
| Gigabyte G593-SD0 | HGX B200 | 11 kW/node liquid |
| Quanta / QCT | HGX B200 + GB200 | rack + node |
| Wiwynn ES200G2 | GB200 NVL72 | rack-scale |
| Hyve Solutions | hyperscaler custom | rack-scale |
| Penguin Solutions | turn-key HPC clusters | hall-scale |
| Aivres (formerly Inspur USA) | HGX + GB200 | rack-scale |
| Pegatron | GB200 NVL72 ODM | rack-scale |
Cooling: direct-to-chip cold plates (CoolIT OMNI, Asetek RackCDU, Motivair MCDU, JetCool SmartPlate two-phase) on every GPU and CPU; rear-door heat exchangers (Motivair ChilledDoor, Vertiv Liebert XDR, STULZ CyberRow) for residual ~15-20% air load. CDU per rack pair (Motivair MCDU-1500 at 1.5 MW, ~$120-180K each). Primary loop W32-W45 ASHRAE class to dry-cooler array (Güntner, Modine) or chilled water from Schneider EcoStruxure ETM. See design-data-center-cooling-system.
50-100 kW/rack is the new normal for AI; GB200 NVL72 pushes 132 kW; GB300 NVL576 will push 200+ kW/rack via Vertiv + Schneider integrated solutions. Air-only is not on the menu.
11. CAPEX + OpEx
11.1 CAPEX (greenfield 1024-GPU B200 build, 14 racks GB200 NVL72 variant)
| Line | Cost |
|---|---|
| 14 × GB200 NVL72 racks (~1008 GPUs) @ $3.2M | $44,800,000 |
| Or: 128 × HGX B200 nodes @ $420K | $53,760,000 (alternative path) |
| Quantum-X800 IB fabric (compute backplane) | $6,500,000 |
| Spectrum-4 storage + mgmt network | $1,800,000 |
| VAST Data hot scratch (3 PB usable) | $7,200,000 |
| Cloudian warm bulk (15 PB usable) | $4,500,000 |
| CDUs (14 × Motivair MCDU-1500 + spares) | $2,100,000 |
| Power distribution (busways, PDUs, transformers) | $3,500,000 |
| Site build / fitout (white space, dry coolers, fire suppression) | $8,000,000 |
| Cluster head nodes + login + slurmctld + storage mgmt | $600,000 |
| Software (NVIDIA AI Enterprise sub for 1024 GPUs, ~$4,500/GPU/yr) | $4,608,000/yr (Year 1 prepay possible) |
| Integration + commissioning + Cx | $3,500,000 |
| Contingency 10% | $8,400,000 |
| Total CAPEX (GB200 NVL72 path) | ~$95M |
For the lower-end H200 path: $70-80M total. For frontier GB300 NVL72/NVL576 buildout (2026): $120-160M. The hyperscaler builds (xAI Colossus $2-3B for 100K H100, Stargate $100B+) scale linearly until you hit grid + cooling constraints.
11.2 OpEx (annual, fully operational)
| Line | Annual |
|---|---|
| Power (2.2 MW × 8760 h × $0.07/kWh blended) | $1,350,000 |
| NVIDIA AI Enterprise subscription (renewal) | $4,600,000 |
| InfiniBand support + spares | $650,000 |
| Storage software support (VAST + Cloudian) | $1,200,000 |
| Hardware spares + RMA pool (1.5% of CAPEX/yr) | $1,400,000 |
| SRE + cluster engineering (4 FTE × $280K loaded) | $1,120,000 |
| ML platform engineering (3 FTE × $300K loaded) | $900,000 |
| Data team (preprocessing + curation, 4 FTE × $240K) | $960,000 |
| Compliance + security audits | $200,000 |
| Facility lease (if not owned, ~$200/kW/mo) | $5,280,000 |
| Total OpEx | ~$17-18M/yr |
Owned facility + own-grid power can cut OpEx to $10-12M/yr; full colo at retail rates pushes $22-25M/yr.
For comparison: training a 70B-class model on a public cloud (AWS Trainium2, GCP TPU v5p, Azure ND H100 v5, Lambda Labs, CoreWeave, Together AI) lists at $2-8M per pretrain run at this scale — owning the cluster pays back at 3-6 full pretrains, plus retains all post-train + experimentation capacity.
12. Reference builds + what they actually look like
- Meta Grand Teton + Open Rack v3 + RSC: Meta’s training platform; 24,576 H100 cluster trained Llama 3.1 405B; two clusters reported. Grand Teton chassis fits 8 H100 in liquid-assisted air; RSC is the prior 16K A100 platform. Meta announced two 24K H100 clusters and is building toward 350K H100-equivalent by end-2024. MTIA (Meta Training and Inference Accelerator) v1/v2 silicon for inference workloads, complementing NVIDIA training.
- Microsoft Mt Hood + Azure Cobalt 100 + Stargate: Mt Hood is the GB200 NVL72 reference; Cobalt 100 is the ARM CPU in Azure for general-purpose; Stargate is the announced $100B+ joint OpenAI-Microsoft buildout reportedly targeting multi-GW frontier-training capacity through 2028.
- xAI Memphis Colossus: 100K H100 brought online in 122 days (mid-2024), expanding to 200K including H200, with public stated ambition of 1M H200-equivalent. Direct-to-chip liquid on Supermicro reference platform; controversial on-site natural-gas turbines for supplemental power.
- Anthropic Project Rainier: multi-hundred-thousand-chip AWS Trainium2 cluster announced 2024 as the substrate for Claude pretraining; first-party AWS silicon collaboration.
- Tesla Dojo D1: in-house 5nm training silicon, 354 TF BF16/FP16 per D1 die, 25-die “training tile”; AI training focused on Tesla FSD video corpus.
- Google Trillium TPU v6e + Ironwood TPU v7: Trillium v6e ramping through 2024-25; Ironwood v7 announced for 2026 as Google’s frontier-class TPU, ~4.6 PF FP8 per chip in 9216-chip pods.
- AWS Trainium2 + Trainium3 UltraServers: 64-chip Trainium2 UltraServer, NeuronLink coherence; Trainium3 mid-2026 with FP4 and ~2× throughput per chip.
The pattern: every frontier-lab build separates training (custom silicon or NVIDIA frontier) from inference (lower-cost custom or older-gen NVIDIA), with a custom fabric stack tuned to the model parallelism shape. A 1024-GPU pod is a small instance of the same pattern.
13. Risk register
- Silicon supply — H200/B200/GB200 allocation queues remain gated by NVIDIA OEM relationships; 6-12 month lead times in 2025. AMD MI300X/MI325X has shorter lead time but immature FP8 software story.
- Fabric mismatch — IB NDR feeds H100 fine; B200/GB200 want XDR 800G or Spectrum-X to keep collective ops from bottlenecking. Don’t deploy B200 on QM9700 NDR fabric without modeling the AllReduce loss.
- Power + cooling — 2.2 MW IT in a single hall is a substation conversation in most markets; greenfield sites need 18-30 month grid lead times. Northern Virginia, Ireland, and Singapore moratoria force new builds to Texas, Phoenix, the upper Midwest, or Sweden/Finland.
- Software drift — CUDA + driver + NCCL + framework versions interact subtly; a working stack today rarely upgrades cleanly. Pin everything; track NVIDIA NGC release notes weekly.
- Silent corruption — silent data corruption (SDC) in HBM and NVLink under sustained load is real (Google, Meta both published 2023-2024 papers). Run periodic NCCL bandwidth + DCGM diagnostics in-line with training; auto-rollback on NaN.
- Vendor concentration — NVIDIA + a single ODM + a single storage vendor is fragile. Dual-source where possible.
- Compliance / export controls — H100, H200, B200, GB200 are export-controlled to a long list of countries (US BIS rules amended 2022, 2023, Oct 2023, Dec 2024). Confirm shipment destination compliance.
14. Adjacent
- design-rag-at-scale-system — the retrieval system the trained model serves
- design-realtime-inference-fleet — the inference fleet that monetizes pretraining
- design-agent-platform-deployment — the agent layer above inference
- design-data-center-cooling-system — the 50 MW physical substrate
- distributed-systems-fundamentals — parallelism, collectives, fault models
- observability-stack — DCGM + Prometheus + tracing
- realtime-embedded — network determinism context