fastgen-train

name: fastgen-traindescription: >-  Run, resume or troubleshoot NVIDIA FastGen training: SFT or distillation with train.py, bounded smoke tests, distributed launches, checkpoints, validation and W&B evidence. Use it whenever training is executed or a training run misbehaves.license: Apache-2.0metadata:   author: "Julius Berner <jberner@nvidia.com>"  domain: ml

Train or distill with FastGen

Apply fastgen-runtime first. The README and the configuration README own launch commands, run directories and checkpoint storage; the troubleshooting guide owns failure diagnosis. Read run contracts to bound a run, read its logs or continue a checkpoint.

1. Goal and inputs. Decide whether the deliverable is a smoke test, a bounded experiment, a quality target or a resumed run. Use fastgen-configure for the recipe and fastgen-dataset for data; check every enabled loss, and the validation loader, against the real-data versus prompt-only contract.
2. Weights versus state. Distinguish external weights, warm starts and full resume; the checkpoint table lists the settings and their order. Resume is on by default, so independent runs need new run identities, and FSDP runs resume only from sharded checkpoints (checkpoint storage).
3. Bound and observe. Before launching set trainer.max_iter, trainer.save_ckpt_iter, trainer.validation_iter, trainer.logging_iter and the W&B destination; keep the algorithmic callbacks. For a smoke test log scalars and media every iteration through the W&B callback and run long enough to cover a student update and every alternating optimizer phase.
4. Launch. Enable exactly one of trainer.ddp/trainer.fsdp for multi-GPU runs; a launcher alone distributes nothing. Budget memory for teacher, student, auxiliary networks, optimizer state and preprocessors (CUDA out of memory), then confirm the runtime startup checks and, when requested, online tracker uploads.
5. Monitor. Check finite losses and gradients, parameter updates and samples with fastgen-validate; compare fixed held-out inputs before and after training with fastgen-inference. Stop on non-finite updates, broken conditioning or degrading samples before spending the budget. Verify the run's saved checkpoint and readable media before reporting success; include these artifact checks in a review-only run plan. Scheduler requeue alone saves nothing; rely on periodic checkpoints or a trainer.auto_resume integration (default: no-op).

Report run URLs, the resolved config, executed iterations and optimizer phases, checkpoint location, compute used and validation evidence. A submitted job is not a successful run and a smoke test is not convergence. For stalls distinguish queueing, downloads, empty loaders and distributed startup; for resume mismatches inspect run identity and checkpoint contents first.