Question

Design the training infrastructure for a large language model trained across 1024 GPUs for several weeks on a multi-petabyte tokenized corpus. Hardware failures are routine at this scale (expect a GPU/node to die every few hours). Design the system to make a multi-week run survivable: cover the data pipeline feeding 1024 workers without becoming the bottleneck, the parallelism strategy, and how checkpointing + recovery keeps a single node failure from wasting the whole run.