cdf-video-data-augmentation

name: cdf-video-data-augmentationdescription: Run the CDF Video Data Augmentation Airflow workflow on Kubernetes or OSMO to create Cosmos video-to-video variants and auto-label both source and generated videos. Use for VDA payload authoring, preflight, execution, monitoring, output verification, or result retrieval. OSMO requires external endpoints. Standalone augmentation or auto-labeling configuration belongs to the respective component skill. Not for retrieval-driven CDF pipelines or image-to-video event generation.license: CC-BY-4.0 AND Apache-2.0metadata:   owner: NVIDIA  service: cosmos-data-factory  version: "1.0.0"

Run CDF Video Data Augmentation

Run the existing Video Data Augmentation workflow end to end. Read the getting-started guide and payload guide before authoring a payload. Treat the checked-in DAG, payload model, and platform manifest as the source of truth if the docs and implementation differ.

Use the bundled references progressively:

• Read setup-and-preflight.md when preparing input or checking a controller/compute environment.
• Read payload-contract.md before rendering or changing a payload.
• Read airflow-direct-api.md before triggering or monitoring a run through the API.
• Read outputs.md before downloading or interpreting artifacts.
• Read troubleshooting.md when a check or run fails, or when the user asks to stop a run.

The workflow prepares source videos, runs the full auto-labeling and training-export chain on the source branch, generates Cosmos Transfer variants, runs the same chain on passing generated videos, validates storage outputs, and shuts down run-owned services. It does not produce one combined final dataset manifest.

Component details

Distinguish the configuration layers before authoring a payload:

Consult the component skill before authoring or changing its container configuration, even when that configuration is used inside a full VDA run. Component YAML and the Airflow trigger payload are different contracts; do not put an entire container config directly into conf.payload. Changing supported payload values alone does not require a new DAG. Route structural changes to the DAG-authoring skill only when requested; setup/deployment remains separately authorized.

When the request needs component configuration or diagnosis, consult that component's documentation, and do not invent keys or values. The links below are pinned to the latest main commit of each component repository as of 2026-10-20; a component skill installed separately may describe a different version.

Use only documentation at these commits; do not substitute a moving branch or another version. If the documentation is inaccessible, report the exact reference and continue only work supported by the local workflow guidance; do not invent component settings.

The orchestration routes are sibling skills in this repository: cdf-orchestration-setup for environment preparation and Airflow controller setup, and cdf-orchestration-write-dag for custom DAG composition and specification parity review.

The pinned component skills own container schemas, model options, prompt authoring, and evaluator internals; use their referenced documentation rather than maintaining local copies. This skill owns the VDA trigger contract and the constraints imposed by its checked-in DAG. Preserve those overrides in payload-contract.md; component capabilities do not automatically become supported DAG options. Keep execution through Airflow for a workflow request. If a pinned component contract differs from the deployed image or DAG integration, report the version mismatch before proposing settings; do not silently update an image or bypass an override. Routine payload rendering, status checks, and GPU sizing do not require loading both components.

Select the execution target

List the DAGs Airflow actually loaded and check import errors before selecting a DAG. A platform DAG is registered only when its manifest exists. Never invent a DAG ID or reinstall Airflow merely because the requested DAG is absent; report that the existing installation does not contain it.

Use Kubernetes unless the user explicitly requests OSMO or supplies an OSMO execution context. For OSMO, require external_services: true, keep every service_lifecycle.*.enabled value false, and supply all VLM, LLM, Cosmos Transfer, and required Cosmos Evaluator endpoints. The OSMO manifest cannot deploy any of these services.

Minimum compute

For Kubernetes, let C be the number of enabled managed evaluator checkers, from zero to three. Each managed checker—hallucination, attribute verification, and physical plausibility—requires one GPU. Use C=0 when the evaluator is external or all checks are disabled.

The absolute floor reserves one GPU for a runnable batch task; the practical minimum reserves two because source auto-labeling and Cosmos/output processing can overlap. Internal Kubernetes adds one GPU for the shared language endpoint and two for Cosmos3-Nano Transfer. Thus the default internal payload with all three bundled checks enabled needs 7 GPUs to make forward progress and 8 GPUs for the default parallel graph. With physical plausibility disabled but managed hallucination and attribute verification enabled, the corresponding floors are 6 and 7 GPUs. OSMO assigns each active task 1 GPU, 11 CPUs, 100 GiB RAM, and 100 GiB storage. Read setup-and-preflight.md for assumptions, scaling, and the external-service examples.

Manual payload entry in the Airflow UI

If the user explicitly wants to enter their own payload in the Airflow UI, confirm controller readiness, make the UI reachable, and report the URL. Do not render, confirm, or submit a second payload on their behalf. Once they say they triggered a run, locate it through the API and resume monitoring from the recorded DagRun configuration.

Collect the run contract

Before building a payload, confirm:

• input_path: an accessible single video or directory containing .mp4, .avi, .mov, .mkv, or .webm files.
• output_directory: a writable, run-appropriate storage prefix.
• Target platform and external_services mode.
• max_videos: use a positive integer for a bounded run. Use null, zero, or a negative integer for all videos; validation normalizes every unlimited form to null before input discovery.
• cosmos.num_augmentation and a non-empty, video-appropriate cosmos.variable_distribution.
• Enabled evaluator checks and whether the evaluator is managed or external.
• For external mode, the real service URLs used by cosmos, captioning, visual_qa, and reasoning. cosmos.cosmos_transfer_service_url is operationally required even though the current schema does not reject its absence.

Never invent a path or endpoint — use only values the user supplied, including ones they supplied earlier for a previous run. Do not ask the user to paste credentials; inherit them from the configured Airflow connections, Kubernetes secrets, OSMO configuration, or shell environment.

For a local file or flat video directory, validate before upload:

python3 scripts/upload_videos.py --path /path/to/videos --validate-only
python3 scripts/upload_videos.py \
  --path /path/to/videos \
  --destination-path video-data-augmentation/my-run

The upload command requires a configured UPLOAD_DESTINATION and the standard AWS credential chain.

Bundled tools

• scripts/upload_videos.py: validate and optionally upload one local video or a flat directory.
• scripts/payload.py: render or validate standalone Kubernetes and OSMO payloads.
• scripts/summarize_results.py: summarize a downloaded run's candidate, evaluator, labeling, export, and report files without loading media.
• assets/variable-distribution.json: editable weather/time-of-day distribution for the bundled VDA prompt.

Run commands from this skill directory. The scripts never need credentials in command arguments.

Author and validate the payload

Start from the smallest payload that expresses the requested run, then validate it with scripts/payload.py. When the repository is available, also validate with VideoDataAugmentationDagConfig from airflow/dags/workflows/video_data_augmentation_dag/models/payload.py; it is authoritative for the deployed code.

Preserve these invariants:

• Top-level external_services controls the VLM, LLM, and Cosmos Transfer lifecycle. Nested service modes and output directories must agree with the top level.
• Internal Kubernetes mode enables the VLM, LLM, and Cosmos Transfer lifecycle entries. The checked-in Kubernetes manifest uses one shared language-model endpoint and a two-GPU Cosmos3-Nano Transfer endpoint.
• External mode supplies every endpoint used by each enabled stage. If caption summaries are disabled, the captioning LLM endpoint is not required.
• Whenever a cosmos object is present, include an explicit non-empty variable_distribution; otherwise the shared model can fall back to an unrelated clothing distribution.
• The bundled transfer config fixes the generation model to nvidia/Cosmos3-Nano; cosmos.cosmos_transfer_model does not override it.
• The DAG replaces visual_qa.question_bank_file with its bundled question bank. Do not claim that a payload value changes VDA questions.
• Unknown fields may be ignored by the schema. Review field names rather than treating successful parsing as proof that every requested option took effect.
• On Kubernetes, enabled evaluator checks may use either an explicit cosmos.cosmos_evaluator_service_url or managed evaluator components. On OSMO, every enabled check requires an external evaluator URL reachable from OSMO tasks; alternatively disable all checks explicitly.

Show the resolved, validated payload to the user without secrets and obtain confirmation before submitting a run. Submission authorization covers one run only.

Typical internal Kubernetes rendering:

python3 scripts/payload.py render \
  --platform k8s --service-mode internal \
  --input-path s3://bucket/vda/input/ \
  --output-directory s3://bucket/vda/output \
  --max-videos 1 --num-augmentation 1 \
  --variable-distribution assets/variable-distribution.json \
  --output /tmp/vda-payload.json

Preflight the existing installation

Do not deploy or reinstall unless the user explicitly asks. Before triggering:

1. Inspect the active repository branch/worktree and the live controller state. Confirm controller pods are healthy, the target DAG is loaded and unpaused, and no relevant import error exists.

2. Verify the required Airflow pools and open slots. Kubernetes uses k8s_rtx6000_1, default_pool (internal-mode augmentation also runs here), and cosmos_transfer_3_service_pool for external mode; OSMO uses osmo_rtx6000_1, default_pool, and cosmos_transfer_3_service_pool.

3. Verify the selected task platform, object-storage access, endpoint reachability, image/model compatibility, cache/storage headroom, and required Airflow connections or secrets without printing credential values.

4. Check actual free GPUs as well as advertised capacity and running consumers. Kubernetes batch stages use GPU task profiles in both service modes. Internal mode additionally deploys the shared language endpoint and two-GPU Cosmos Transfer endpoint. Each enabled managed checker consumes one GPU. Enforce the minimum-compute floor above, then size for the live execution graph, mapped inputs, replicas, and pool concurrency.

5. Report stale failed pods or prior run-owned resources. Clean up only resources belonging to a run the user owns and only with authorization.

6. From the repository root, confirm the deployment maps every bucket the payload reads or writes:

   python3 skills/cdf-orchestration-preflight/scripts/storage_check.py --payload <payload.json>
   

   It must print Storage mapping: ready. An UNMAPPED line means the run would fail at runtime; report the unmapped field and the configured prefixes to the user. Using that bucket requires changing the controller's storage configuration and redeploying it; with the user's authorization, route that change to orchestration setup, then rerun the check.

If the existing installation is not ready, report the failed prerequisite. Deployment, controller reinstallation, pool resizing, and destructive cleanup require separate explicit authorization. Use airflow-direct-api.md for the concrete checks; do not infer readiness only from a healthy Airflow UI.

Submit and monitor

Submit exactly one run with the confirmed payload under conf.payload, an explicit unique run ID when the interface supports it, and the required logical date for the Airflow REST API. Record the DAG ID, DagRun ID, platform, service mode, input, output prefix, requested input limit, augmentation count, and enabled evaluator checks. If the submission response is uncertain, look up the same run ID before retrying; do not create a duplicate run.

Monitor until the DagRun reaches success or failed. Inspect task instances and logs when state stalls or fails, especially:

• auto_labeling_input for the source-video branch.
• cosmos_augmentation and evaluator tasks for generated candidates.
• auto_labeling_output for passing generated videos.
• validated_output for final artifact checks.
• Service shutdown tasks for cleanup, without confusing successful cleanup with workflow success.

For a requested stop, follow Stopping a run. Use the rest of that reference for evidence-driven failure routing.

Verify and report the result

For <output_directory>/<dag_run_id>/, verify:

• cosmos/<video_key>/<augmentation_index>/ contains output.mp4, caption.txt, and metadata.json for each successful candidate. Account separately for evaluator-rejected or incomplete candidates.
• auto_labeling_input/ and training_export/auto_labeling_input/ contain source-branch output.
• auto_labeling_output/ and training_export/auto_labeling_output/ contain generated-branch output for passing candidates.
• Generated media fully decodes and the observed video/candidate counts agree with the confirmed payload and evaluator outcomes.
• Run-owned managed services were removed while external services remained untouched.

Return the terminal state, failing task and concise root cause when applicable, candidate and evaluator counts, artifact prefixes, service-cleanup state, and any validation gap. State clearly that the two training-export branches are separate. Download or analyze artifacts only when the user requests it. When they do, read outputs.md, download only the named run prefix, and use scripts/summarize_results.py before deeper media inspection.

Boundaries

• Route standalone generation to cdf-augmentation and standalone labeling to cdf-auto-labeling using the pinned references above. The existing VDA DAG has no generation-only switch; do not trigger the full graph for a subset request without acceptance.
• Use the autonomous_vehicle_dataset_generation_dag_k8s workflow for retrieval-driven CDF-AV processing and saved dataset or retrieval manifests.
• Use the event-video-generation workflow for image-to-video anomaly generation from seed images.
• Do not silently turn a run request into a deployment, DAG edit, custom workflow, endpoint replacement, or quality-policy change.