fastgen-dataset

name: fastgen-datasetdescription: >-  Prepare, register or repair datasets for NVIDIA FastGen loaders: class-conditional ZIPs, WebDataset shards of images, prompts, latents, videos, audio or camera data, and KD pairs or trajectories; then verify decoded batches. For experiment configuration over already prepared data, use fastgen-configure.license: Apache-2.0metadata:   author: "Julius Berner <jberner@nvidia.com>"  domain: ml

Prepare a FastGen dataset

Apply fastgen-runtime. The dataset README owns formats, preparation commands, loader presets and key parameters; the loaders in wds_dataloaders.py and wds_utils.py document decoding, partitioning and resume in their docstrings. Read loader contracts for shard acceptance, coverage and resume accounting.

1. Scope. Resolve source, permitted use, destination, task, the method's supervision needs (fastgen-configure if unsettled), the network's preprocessing and the available sample count. Inspect existing inputs before downloading; keep provenance and counts.
2. Loader. Choose the existing class-conditional or WebDataset loader. Shards need unique sample keys with the extensions named in key_map; constants come from files_map and presets_map. Raw media and prompts use the *_raw keys and cached latents and embeddings the plain keys; never label placeholders as model-space data, and record the model revision and preprocessing of every cache.
3. Pairing and geometry. Keep task pairing (source and target for editing, reference frames for image-conditioned video, camera data, aligned audio). Derive frame counts, FPS, aspect ratio and temporal compression from the network; the representable frame count is not the nominal clip length.
4. Splits and size. Separate training from held-out evaluation inputs before deriving labels or trajectories. Estimate the unique examples needed from throughput, effective batch, updates and acceptable repetition; a tiny dataset validates plumbing only.
5. Register. Import and copy a preset from data.py rather than reproducing its loader fields; add a shared preset there when needed. Supply paths through the config or environment roots; keep data and preprocessing artifacts outside the source tree. For KD data use the selected teacher and the documented KD loader format.
6. Verify. Iterate the real loader and a prepared batch (prepare_batch) in the proper runtime; check keys, shapes, dtypes, ranges, captions, pairing, negative conditions and modality lengths. Use enough batches and ranks to expose empty worker or shard splits, and keep a manifest of usable counts. Before prompt-only training, sample the teacher on a few real training prompts to confirm they are usable.

Report provenance, the split/count manifest, dataset location, data-config path and the observed batch contract. For empty or short batches inspect tar members, key_map, filters, shard/worker partitioning and files_map before touching the network; report corrupt or excluded samples instead of counting them.