video_toolkit

name: video_toolkitdescription: Make short explainer videos from a brief with claude-code-video-toolkit — submit a scene spec, poll a job, get a QC-checked MP4. Also covers the individual tools (AI voiceover, images, clips, music, talking heads) for custom work.metadata:   openclaw:     emoji: "🎬"    skillKey: "video-toolkit"    os: ["darwin", "linux", "win32"]    requires:       bins: ["uv", "ffmpeg"]

Video Toolkit

Turn a brief into a finished MP4 with claude-code-video-toolkit. This skill is written for any agent runtime: Claude Code, OpenClaw, Codex, Hermes, a bare API loop with a run_tool call, or an MCP client. Nothing here depends on how your runtime backgrounds work, because the long work runs in a detached job you poll.

Who this is for

You are an agent: someone asks you for a video through a channel (a chat message, a Discord or Slack thread, an MCP client) and you make it without a person at your terminal. You report back in that channel. The person asking sees your messages and the finished video, never the toolkit.

What the toolkit can make: pick a route first

Every route has one of two support levels. catalog gives it per template as agent_support.

• job: one prompt. video_job.py runs it unattended from one spec. You write the scenes; it voices, illustrates, captions, builds and QCs them; you post the MP4. Any runtime can do this, through the CLI or MCP.
• session: a working session. Copy a template or example into projects/<name>/, edit its code with the skill it names, render with Node, look at the result, fix it, render again. That needs a Claude Code (or Codex-style) session in the toolkit, which is what the toolkit is built around. The session can be a person's, or yours, if your runtime can run one headless with a shell and Node. Either way, the skill's review points (script, a contact sheet, the render) go back to the person who asked, in their channel, before you spend more. Tell them up front what the route needs from them (for a synthetic UI video: the app's source, the click path, the narration).

A route moves from session to job once its inputs fit in a message, a spec drives the build with no code written, QC can judge the result, and there's a cost cap. Offer a job when one would do.

ds-release-notes is built for one product's screens, but the method isn't: keep its Stage and Cursor wiring and rebuild your own app's screens the way .claude/skills/synthetic-ui-demo/ describes.

The catalogue lives on disk. uv run tools/video_job.py catalog (MCP: video_catalog) lists every template with its demo videos and the skill that drives it, the example builds, components, brands, design sources and voice presets, read from _internal/toolkit-registry.json. When a user asks "what can you make?", answer from it, with the demo links, rather than from memory: the list grows each release. Skills under .claude/skills/ are written for Claude Code, but they are plain markdown: open SKILL.md in the folder and follow it.

When asked "what can you make?", answer with the job routes first (what you can do from this message), then the session routes and what each needs from them, each with its demo link from catalog.

How it looks: brand, design source, transitions

• Brand (brands/<name>/brand.json colours and fonts, voice.json a voice): pass "brand": "<name>" in a job spec, or --brand to the image tools. catalog lists the brands on this host.
• Design direction for a session: .claude/skills/video-design/ turns a direction into scenes (colour by act, type scale, one data accent) and critiques the render. The direction comes from a design source: ours (the default, video-design's own palettes, nothing to install), taste-skill (recommended, installed separately) or frontend-design. A project records its choice in design.md. After a render, contact_sheet.py tiles one frame per scene for the critique.
• Turning a scene in place (the same thing shown again in a new way): PageTurns. Between scenes, a hard wipe(); the effect transitions in lib/transitions/ are for brands that call for them.
• Other components (lib/components/): AnimatedBackground, SlideTransition, Label, Vignette, LogoWatermark, SplitScreen, Envelope, PointingHand, MazeDecoration, FilmGrain, Captions. Remotion itself is documented in .claude/skills/remotion-official/ (synced from upstream) and the toolkit's patterns in .claude/skills/remotion/.

The rest of this file is the job route, then the individual tools.

Where the toolkit is

Every command below runs from the toolkit root. Find it, in this order:

1. $VIDEO_TOOLKIT_ROOT if set (also honoured by the toolkit's own scripts);
2. $VIDEO_TOOLKIT_DIR (an alias some runtimes set);
3. a claude-code-video-toolkit/ directory under your workspace or home.

cd "${VIDEO_TOOLKIT_ROOT:-${VIDEO_TOOLKIT_DIR:-$HOME/claude-code-video-toolkit}}"
uv run tools/verify_setup.py --json     # prerequisites, keys and endpoints; [x] means ready

Never run a tool from inside a project directory: tools resolve paths relative to the toolkit root. If your runtime can't cd, pass --directory: uv run --directory /path/to/toolkit tools/video_job.py ....

No shell at all? Use the MCP server instead: uv run --directory /path/to/toolkit --extra mcp tools/video_mcp.py exposes video_estimate, video_submit, video_status, video_result and video_list_presets over stdio. Same flow as below, same JSON.

The quick path: estimate → submit → poll → result

tools/video_job.py is one async entrypoint. submit returns in seconds with a job id; a detached worker runs the whole pipeline (voiceover → visuals → captions → build → QC); every later call is short and reads state from disk. That is how it fits a per-call time limit, and it survives you going away between turns.

1. Write the spec (you are the LLM)

A spec is a list of scenes, each with narration text and one visual. Budget about 2.4 words a second of narration; a 45 s video is ~100 words in 3 to 5 scenes.

{
  "title": "Why do we have leap years?",
  "format": "9:16",
  "voice": {"provider": "qwen3", "speaker": "Ryan", "tone": "warm", "maxWpm": 165},
  "scenes": [
    {"slug": "hook",
     "text": "Every four years, February gets an extra day. Why? Because a year isn't 365 days long.",
     "visual": {"kind": "ltx", "prompt": "Slow orbital shot of Earth turning in space, sunlight sweeping the oceans, cinematic, no text"}},
    {"slug": "quarter",
     "text": "Earth takes about 365 and a quarter days to orbit the sun. Ignore the quarter and the calendar drifts.",
     "visual": {"kind": "ideogram", "caption": { "...filled from the caption template below..." }}},
    {"slug": "cta",
     "text": "So every fourth year we add one day back. Follow for one concept a week.",
     "visual": {"kind": "ideogram", "caption": { "...filled from the caption template below..." }}}
  ]
}

• format: 9:16 (Shorts, Reels, TikTok) or 16:9 (YouTube, decks).
• visual.kind: ltx (5 s motion b-roll, prompt the motion and atmosphere, never on-screen text), text (constructed: a generated background that drifts, with headline and optional subline animated in; about a cent), ideogram (a designed card with legible text: always a caption object, filled from the template below), flux (text-free generated background), file (the person's own photo or clip: "path" in their kit's assets or your workspace, optional headline; free), none (placeholder, drafts only). A scene with no visual gets a background. Alternate motion and constructed scenes so there is a pattern interrupt every ~15 s.
• backgrounds: where text scenes and scenes with no visual get their background. auto (default) generates one with FLUX when this host has it, else draws a flat brand-colour field with grain here (free, no GPU); flux or local to choose.
• kit: a kit name. A kit is set up once in a working session for one person or product (kits/README.md): a brand, a voice, a format and a folder of their own photos and clips. The spec fills those in from it, and file paths resolve in its assets. catalog lists the kits on this host, each with its assets and what each one shows; for someone with a kit, a request in a chat becomes a few file and text scenes, each file chosen from that list to fit its sentence, with no code written. The kit's focus keeps an off-centre subject in a crop (a landscape clip in 9:16 keeps only its middle third); a scene can set "focus": 0-1 itself. Prefer assets whose usedRecently is 0, so two videos for one person don't look the same.
• Copy is checked before any spend. Dashes and stock AI phrasing ("it's worth noting", "delve", "in today's world") in the narration, title, headlines or sublines are refused with the line named, and so is narration copied word for word from an earlier finished video on this host. Rewrite the named lines and submit again; "allowCopy": true only when the person asked for the same words. The same rules (tools/copy_lint.py) apply to every other word you send out: the message with the video, a title, a description. Run uv run tools/copy_lint.py --text "..." on them first.
• voice: an object as above, or a preset name: assets/voices/<name>.json (a cloned voice; its reference audio is gitignored, so the host needs its own copy) or a brand with a voice.json. catalog lists them. The title animates over the first scene and a soft music bed is mixed in by default; "titleCard": false and "music": false turn them off.
• No JSON at all: --scene "narration :: ltx: motion prompt" per scene, --scene "narration :: text: HEADLINE ; subline", --scene "narration :: file: candle.mp4 ; HEADLINE", --title, --voice <preset>, --kit <name> (MCP: kit; it works with --brief too, and the expander is given the kit's assets). See the docs page for the full flat form.
• Scene 1 must hook in one sentence. Scene N is the call to action. Captions are burned from the script text, so write every scene exactly as it should appear.
• Optional: "music": {"enabled": true, "prompt": "..."} (off by default), "brand": "digital-samba", "ideogram": {"speed": "TURBO"} to halve image cost, "captions": {"enabled": false}.
• uv run tools/video_job.py presets prints the full price table, formats and an example spec.

The caption template (every ideogram scene)

Ideogram renders text reliably only from a structured JSON caption. A plain prompt works but adds small filler text to the card ("A REMINDER", a fake tagline), so fill this in instead. Replace the <...> placeholders and leave everything else. The headline is under 6 words, all caps. bbox is [top, left, bottom, right] on a 0 to 1000 grid, so the same numbers work for 9:16 and 16:9. uv run tools/video_job.py presets --json prints it as ideogram_caption_template.

{
  "high_level_description": "A <dark-navy poster> with the headline <HEADLINE>, showing <one simple diagram or icon>, with clear space in the bottom fifth of the frame.",
  "style_description": {
    "aesthetics": "clean, diagrammatic, modern explainer",
    "lighting": "flat even lighting, soft glow on the main object",
    "medium": "flat vector graphic design",
    "art_style": "minimalist infographic poster, generous negative space",
    "color_palette": ["#0A1428", "#4DA8FF", "#FFB347", "#FFFFFF"]
  },
  "compositional_deconstruction": {
    "background": "Deep navy gradient, faint tiny stars.",
    "elements": [
      {"type": "text", "bbox": [80, 80, 200, 920], "text": "<HEADLINE>",
       "desc": "Bold condensed sans-serif headline, all caps, white, centered at the top.",
       "color_palette": ["#FFFFFF"]},
      {"type": "obj", "bbox": [280, 150, 760, 850],
       "desc": "<One object: what it is, its shape, its colour. No text on it.>",
       "color_palette": ["#4DA8FF", "#FFB347"]}
    ]
  }
}

Rules of thumb: one headline and one object per card; describe the object as shapes and colours, not as a concept (for a diagram, write the object as a chain, "object": "prompt -> agent -> video", and the toolkit lays out labelled boxes and arrows); keep the bottom fifth empty because the burned captions sit there; change color_palette only when the brand asks for it. The full caption format, for cards with more elements, is in the toolkit's .claude/skills/ideogram4/.

Have only a brief and want the toolkit to write the spec? Set VIDEO_JOB_EXPANDER to any command that reads {"brief","format","duration_s",...} on stdin and prints a spec JSON (claude -p, a curl to any model, your own script). Then --brief "..." works everywhere --spec does. Without it, estimate --brief still answers with a skeleton estimate; submit --brief is refused.

Runtimes that pass an argv list and can't write files (an allowlisted run_tool, most MCP bridges) take the spec inline: --spec-json '<json>', with the short caption form {"headline": "...", "object": "...", "description": "..."} per ideogram scene so it fits in one argument. Add --output files/<dated-name>.mp4 when your runtime serves a files/ directory at a public URL; the file lands there once the job is done.

What this host can make

Not every host has every provider. catalog and presets carry host: the visual kinds, voices and music providers this host has keys or endpoints for (host.available_visual_kinds), the local tools it has (host.local: ffmpeg, Node, the Python packages), and which route levels it can run (host.routes.job, host.routes.session: a session also needs Node 18+). Offer only those. estimate lists anything the spec uses that's missing as host_missing, and submit refuses it (exit 2, error_code: unavailable, with missing: what, which env vars, and an alternative) before anything is written or spent. An entry marked "soft": true (a music bed with no music provider) isn't refused: the video is made without it.

With no GPU provider at all, a job still runs: text scenes and scenes with no visual get local backgrounds, file scenes use the person's own photos, and ideogram cards need only an Ideogram key. A voice needs ElevenLabs or 60db (a key), or qwen3 on Modal or RunPod. ltx needs Modal; a scene that names flux needs FLUX. Music is optional: without a provider the job runs without a bed and says so. Setting a host up: uv run tools/verify_setup.py, then modal_setup.py --all.

2. Estimate, then submit under a cap

uv run tools/video_job.py estimate --spec spec.json --json
# {"estimate_usd": 0.35, "breakdown": [...], "scenes": 3, "video_s_approx": 31.0, ...}

uv run tools/video_job.py submit --spec spec.json --max-usd 1.00 --json
# {"id": "20261031-230304-f34a", "phase": "queued", "estimate_usd": 0.353, "poll": "..."}

submit is refused (exit 2, error_code: budget_exceeded) when the estimate is over --max-usd, before any spend. Only one job runs at a time: a second submit while one is running exits 3 with error_code: lock_held and the running job's id. The worker also stops itself before the next paid step if actual spend would pass the cap.

3. Poll status

uv run tools/video_job.py status JOB_ID --json

Poll every 20 to 30 s. phase walks queued → voiceover → visuals → captions → build → qc → publish → done, or ends in failed (with error and error_code) or cancelled. progress is 0 to 100, eta_s is a rough seconds-left figure (say "about 3 minutes to go", not a percentage), actual_usd is spend so far, step_detail names the asset being generated, events is a timestamped list of what started and finished (each scene voiced with its wpm, each clip or image made with its seconds and cost, QC, the render size), and log_tail is the last few log lines. Use the last two or three events to say what is happening ("two images done, the clip for scene 3 is rendering") rather than naming a phase. eta_s is calibrated from this host's recent jobs. A typical 30 s video takes 5 to 10 minutes; the LTX clip is the slow step (2 to 4 minutes each, more on a cold start).

Tell the user the id and the estimate as soon as submit returns. Do not promise to keep watching from inside one turn: the job carries on without you, and status works from any fresh process. If your runtime has a scheduler or a poller, point it at status; otherwise check when you are next invoked.

4. Collect the result

uv run tools/video_job.py result JOB_ID --json
# {"ready": true, "mp4": "/.../jobs/20261031-230304-f34a/why-do-we-have-leap-years-9x16.mp4",
#  "url": "https://...r2...", "duration_s": 31.4, "qc": {"passed": true, "errors": 0, "warnings": 1}, "actual_usd": 0.39}

url is a 2 h presigned download link when Cloudflare R2 is configured, otherwise null and you deliver the file. qc is tools/qc.py --render: loudness was normalised to -14 LUFS, and it reports still stretches, black gaps and a silent track. Pass qc.warnings on to the user in a line; qc.errors > 0 means look before you ship. warnings (a list of sentences) is for the user too: captions or music skipped, a scene voiced too fast, a file over the size cap. Exit 2 means not ready yet. The MP4 is encoded under 9.5 MB so it attaches in chat (Discord's bot limit is 10 MB); maxMB: 0 in the spec turns that off for a full-quality YouTube upload.

If a job ends failed or cancelled with resumable: true, run resume JOB_ID (add --max-usd after a budget_exceeded stop) rather than submitting again: it keeps the voice, visuals, captions and music already paid for and redoes only what is missing.

Also: list --json (recent jobs), cancel JOB_ID, prune (old jobs' project trees; submit does this itself), and every job keeps jobs/<id>/job.log, receipts.jsonl (one JSON line per paid call: ts, tool, cost_usd, elapsed_s, exit, args, job_id) and the project copy it rendered from, so a human can open jobs/<id>/project/ and re-render after edits.

Spend

Receipts also append to MEDIA_SPEND_LOG (default logs/media-spend.log in the toolkit) as JSON lines, one per paid call, so a runtime that keeps a spend ledger reads single assets and jobs from one file. If your runtime sets a workspace, export VIDEO_JOB_WORKSPACE to it: the ledger goes to its logs/, relative spec paths resolve there, and submit books the whole estimate at once so a daily cap counts it before the work starts. Prices (presets, checked against receipts on 2026-11-01): LTX clip ~$0.10, Ideogram image $0.03 to $0.10, FLUX image under a cent, Qwen3 voice ~$0.01 a scene, captions ~$0.0001 a second. A 7-scene, 45 s video with two clips and two cards lands around $0.40 to $0.55; the first clip after idle pays a Modal cold start on top. Under --progress json every call prints one cost event, failures included ($0).

---

Deeper: the individual tools

Everything below is what video_job.py drives for you. Reach for it directly when the user wants one asset (an image, a clip, a song), a different template, or a talking head. Add --progress json to any cloud tool for JSON-lines progress on stderr, and --json for a machine-readable result on stdout.

Defaults that are current (2026-11-06)

Voiceover

uv run tools/voiceover.py --provider qwen3 --cloud modal --scene-dir public/audio/scenes --max-wpm 165 --json
uv run tools/qwen3_tts.py --text "Hello" --speaker Ryan --tone warm --output hello.mp3 --cloud modal
uv run tools/qwen3_tts.py --text "Hello" --ref-audio ref.wav --ref-text "exact transcript" --output clone.mp3 --cloud modal

One .txt per scene in --scene-dir, one MP3 back per scene with duration_seconds, wpm and a pacing label. A brand's voice: --brand <name> reads brands/<name>/voice.json. --max-wpm 165 slows rushed takes with pitch-preserving atempo; a cloned voice copies its reference clip's pace, so a flat or fast clone needs a better reference (voice_prosody.py expand), not a higher temperature. For a cloned voice, generate one .txt per sentence three times and let pick_takes.py choose each sentence by pace and pitch spread (the toolkit's CLAUDE.md, "Clone Prosody", has the loop). Speakers: Ryan, Aiden, Vivian, Serena, Uncle_Fu, Dylan, Eric, Ono_Anna, Sohee. Tones: neutral, warm, professional, excited, calm, serious, storyteller, tutorial.

Images

# Text in the image: author the caption JSON yourself (see .claude/skills/ideogram4/), 9:16 = 1440x2560, 16:9 = 1280x720
uv run tools/ideogram4.py --json caption.json --resolution 1440x2560 --output card.png
# Text-free background
uv run tools/flux2.py --preset title-bg --brand digital-samba --output bg.png --cloud modal --progress json
# Edit a photo, upscale
uv run tools/image_edit.py --input photo.jpg --prompt "Add sunglasses" --cloud modal
uv run tools/upscale.py --input small.png --scale 4 --cloud modal

Video clips

uv run tools/ltx2.py --prompt "Aerial over a city at golden hour, cinematic" --width 1024 --height 576 \
    --num-frames 121 --output broll.mp4 --no-open --progress json --json
uv run tools/ltx2.py --prompt "Gentle camera drift, soft light shifts" --input still.png --output animated.mp4 --no-open

Sides must be multiples of 64; frames satisfy (n-1) % 8 == 0 (121 ≈ 5 s, 161 ≈ 7 s). About $0.07 to $0.20 a clip and 2 to 4 minutes; the first clip after idle adds a cold start. Prompt motion and atmosphere, never text. Add --negative-prompt "anime, cartoon, watermark, text, logo" when style drifts. chain_video.py chains clips for continuity (one command, per-scene prompts in a JSON file).

Talking heads

uv run tools/soulx.py --image presenter_16x9.png --audio scene01.mp3 --size 768 --output narrator01.mp4
uv run tools/soulx.py --image presenter_16x9.png --audio scene01.mp3 --model lite --width 512 --height 288 --output draft.mp4

SoulX follows the image's aspect ratio and holds identity over long takes (97% of frame-zero sharpness at 70 s). Pick one resolution per project: torch.compile recompiles on every size change (~10 min cold). Pro sides are multiples of 16, Lite multiples of 32. SadTalker (--preprocess full --still --expression-scale 0.8) is the RunPod fallback and square by default. Don't grade talking heads with a similarity metric; look at a contact sheet.

Music

uv run tools/music_gen.py --preset corporate-bg --duration 60 --output bg.mp3 --progress json
uv run tools/music_gen.py --prompt "lofi, warm keys, no vocals" --duration 45 --variations 3 --output bg.mp3
uv run tools/pick_music.py bg_*.mp3 --target 45 --under voice --pick bg.mp3 --json   # takes the best, says why

Presets: corporate-bg, upbeat-tech, ambient, dramatic, tension, hopeful, cta, lofi. Default provider is the acemusic API (free key); --cloud modal is the self-hosted fallback. Prompting and lyrics: .claude/skills/acestep/.

pick_music.py measures and can't listen, and good takes often tie. With a person in the loop, render the video once without music and let them watch each take under it:

uv run tools/audition.py --video out/video.nomusic.mp4 --music bg_*.mp3 --volume 0.15 --fade-out 1.5 \
    --recommend --under voice    # -> out/audition/video.bg_1.mp4 ..., each normalised to -14 LUFS

Sound effects: uv run tools/sfx.py --prompt "soft whoosh" --output whoosh.mp3.

QC, captions, cut-downs

uv run tools/qc.py --project projects/my-video                  # assets, timing, pacing, loudness
uv run tools/qc.py --render out/video.mp4 --fix                  # -14 LUFS, writes video.loudnorm.mp4; frame checks
uv run tools/qc.py --project projects/my-video --layout         # Remotion: text off canvas, too small, low contrast
uv run tools/contact_sheet.py out/video.mp4                      # one frame per scene, to look at before delivering
uv run tools/captions.py projects/my-video/public/audio/scenes  # word captions aligned to the script .txt
uv run tools/cutdown.py --input out/video.mp4 --platform shorts  # 16:9 -> 9:16, blur-fit

Run qc.py --render before every delivery: the templates mix voice and music without normalising, and YouTube turns quiet audio down, never up. It exits 1 on errors (cut-off narration, silent track); a video that is silent on purpose passes with --allow-silent. In sprint-review configs, align_captions.py re-times hand-written cap() phrases to the audio.

Fixing existing videos: redub.py --input v.mp4 --voice-id ID --sync (new voice, word-level re-timing), addmusic.py (a bed under a finished video), locate_watermark.py then dewatermark.py (ProPainter inpainting), notebooklm_brand.py (a NotebookLM video with your logo and outro). Publishing: youtube_upload.py --video out.mp4 --title "..." --json-out (one-time --auth, private unless told otherwise; docs/youtube-upload.md).

Remotion templates (when a job's compositor isn't enough)

The routes table at the top says which template fits. Copy to projects/<name>, npm install, write the config, generate per-scene audio, update each scene's durationSeconds to ceil(audio + 2) (or run uv run tools/sync_timing.py --apply), npm run render from the project directory, then qc.py --render. Always <OffthreadVideo> and <Audio> from remotion, never a raw <video> tag. The toolkit's CLAUDE.md holds the full workflow. Cloud GPU setup and endpoint trouble: .claude/skills/runpod/, docs/modal-setup.md. Image edit prompting: .claude/skills/qwen-edit/.

Progress JSON (stderr, one object per line)

{"ts":"14:23:15","stage":"submit","msg":"Sending to Modal...","pct":null,"elapsed":0.0}
{"ts":"14:24:02","stage":"complete","msg":"Saved: clip.mp4","pct":100,"elapsed":47.3}
{"ts":"14:24:02","stage":"cost","msg":"Est. cost: $0.0712 (103s on modal)","pct":null,"elapsed":103.0}

Stages: submit, queue, processing, waiting, item (n of m, pct set), cost, complete, error. video_job.py reads the cost events for its receipts; a runtime that wraps single tool runs can do the same.

Error recovery