SKILL.md

ComfyUI Workflow Generator

Description

Generate ready-to-use ComfyUI workflow JSON files through natural language conversation. Users describe what they want to create, and this skill produces valid ComfyUI API-format JSON that can be directly imported and executed.

Activation

Trigger when the user asks to:

• Create/generate a ComfyUI workflow
• Make an image generation workflow
• Build a video generation pipeline
• Set up txt2img, img2img, img2vid, txt2vid workflows
• Generate a ComfyUI JSON file

Trigger keywords: comfyui, workflow, 工作流, 文生图, 图生图, 文生视频, 图生视频, txt2img, img2img, txt2vid, img2vid, upscale, 放大, inpaint, 重绘, controlnet, lora, sd3, flux, ltxv, mochi, cosmos, stable audio, 音频生成, hunyuan3d, 3d生成, stable cascade, 图生3d

Instructions

Step 1: Understand User Intent

Ask the user what they want to create. Gather:

1. Task type: txt2img / img2img / txt2vid / img2vid / upscale / inpaint / audio / 3d
2. Model: SD1.5 / SDXL / FLUX / SD3 / Wan2.2 / HunyuanVideo / LTXV / Mochi / Cosmos / StableAudio / Hunyuan3D / StableCascade
3. Key parameters: resolution, steps, CFG, sampler
4. Optional modules: ControlNet, LoRA, IPAdapter, upscale model
5. Prompt content: positive and negative prompts

If the user is vague, suggest the most common configuration for their task type.

Step 2: Select and Compose Template

Based on user intent, load the appropriate base template from templates/ directory and customize it:

1. Read the matching template JSON file
2. Modify parameters according to user requirements
3. Add optional modules (ControlNet, LoRA, etc.) by inserting additional nodes and links
4. Validate the workflow DAG structure

Fallback — Official Template Repository: If no matching template is found in the local templates/ directory, search the ComfyUI official template repository: https://github.com/Comfy-Org/workflow_templates/tree/main/templates . The official repo contains 443 templates across 9 categories (Image / Video / Audio / 3D / LLM / Utility / Use Cases / Getting Started / Node Basics). First read templates/index.json to find a matching template name, then download the corresponding JSON file to use as the base template. The official templates use the same format as this project (LiteGraph UI format, version 0.4).

Step 3: Generate and Deliver

1. Output the complete workflow JSON (Litegraph UI format)
2. Save it to a .json file if the user requests
3. MANDATORY: Add models array to the JSON top level — extract ALL model filenames from nodes (ckpt_name, unet_name, vae_name, clip_name, lora_name, etc.), look up their download URLs from the Model Download Guide below, and include them. This makes ComfyUI auto-prompt download for missing models on import.
4. Provide a brief explanation of the workflow structure
5. List any required custom nodes the user needs to install

Workflow JSON Format Rules

CRITICAL — follow these rules exactly:

• Use Litegraph UI format (NOT API format) — with nodes array, links array, version: 0.4
• Top-level MUST have: id (UUID string), revision (0), last_node_id, last_link_id, nodes, links, groups ([]), config ({}), extra ({}), version (0.4)
• Each node has: id (int), type, pos ([x,y]), size ([w,h]), flags ({}), order (int), mode (0), inputs, outputs, properties, widgets_values
• Node inputs include BOTH connection inputs ({"name": "model", "type": "MODEL", "link": 4}) AND widget inputs ({"name": "seed", "type": "INT", "widget": {"name": "seed"}, "link": null})
• All inputs/outputs should have label and localized_name fields matching name
• Links format: [link_id, source_node_id, source_slot, target_node_id, target_slot, "TYPE"]
• widgets_values array order must match the widget inputs order
• Every workflow MUST have at least one output node (SaveImage, PreviewImage, SaveVideo, etc.)
• All required inputs must be provided (check references/nodes/index.md and the category files under references/nodes/, plus references/node-registry-additions.md)
• Values must satisfy min/max constraints
• For model/file selection inputs (ckpt_name, lora_name, etc.), use the real model filenames from the Model Download Guide below
• MANDATORY: Every workflow JSON MUST include a top-level models array. Scan all nodes for model file references (ckpt_name, unet_name, vae_name, clip_name, clip_name1, clip_name2, clip_name3, control_net_name, lora_name, model_name) and add each referenced model to the models array with name, url (direct HuggingFace resolve link), and directory. This enables ComfyUI to auto-detect missing models and prompt the user to download them on import. Example:

"models": [
  {"name": "flux1-dev.safetensors", "url": "https://huggingface.co/black-forest-labs/FLUX.1-dev/resolve/main/flux1-dev.safetensors", "directory": "diffusion_models"},
  {"name": "clip_l.safetensors", "url": "https://huggingface.co/comfyanonymous/flux_text_encoders/resolve/main/clip_l.safetensors", "directory": "text_encoders"}
]

Common Output Slot Mappings

CheckpointLoaderSimple: 0=MODEL, 1=CLIP, 2=VAE
UNETLoader:             0=MODEL
CLIPLoader:             0=CLIP
DualCLIPLoader:         0=CLIP
VAELoader:              0=VAE
LoraLoader:             0=MODEL, 1=CLIP
CLIPTextEncode:         0=CONDITIONING
EmptyLatentImage:       0=LATENT
KSampler:               0=LATENT
VAEDecode:              0=IMAGE
LoadImage:              0=IMAGE, 1=MASK
ControlNetLoader:       0=CONTROL_NET
UpscaleModelLoader:     0=UPSCALE_MODEL
ImageUpscaleWithModel:  0=IMAGE
CLIPVisionLoader:       0=CLIP_VISION
CLIPVisionEncode:       0=CLIP_VISION_OUTPUT

LLM / Text Generation Nodes (comfyui_LLM_party)

ComfyUI integrates LLM via comfyui_LLM_party custom node package (需安装: https://github.com/heshengtao/comfyui_LLM_party).

IMPORTANT: When user requests a workflow with LLM (script generation, character parsing, storyboard breakdown, prompt enhancement), you MUST:

1. Ask the user which LLM provider they want to use (OpenAI / Claude / Gemini / Ollama / DeepSeek / other)
2. Ask for their model preference or recommend the latest best model for their use case
3. Use the corresponding loader node and configure accordingly

Supported Providers & Loader Nodes

CRITICAL: Exact class_type Names (use these EXACTLY in workflow JSON)

DO NOT invent node names. Only use these exact class_type strings:

Core LLM Nodes — Detailed Specs

LLM_api_loader (class_type: LLM_api_loader)

• Inputs: model_name (STRING), base_url (STRING, optional), api_key (STRING, optional), is_ollama (BOOLEAN, optional)
• Outputs: model (CUSTOM) [slot 0]
• Widget values order: [model_name, base_url, api_key, is_ollama]

LLM (class_type: LLM)

• Inputs (required): system_prompt (STRING, widget), user_prompt (STRING, widget), model (CUSTOM, connection from loader), temperature (FLOAT, widget, default=0.7), is_memory (COMBO "enable"/"disable", widget), is_tools_in_sys_prompt (COMBO, widget), is_locked (COMBO, widget), main_brain (COMBO, widget), max_length (INT, widget, default=1920)
• Inputs (optional): system_prompt_input (STRING, connection), user_prompt_input (STRING, connection), tools (STRING), file_content (STRING), images (IMAGE)
• Outputs: assistant_response (STRING) [slot 0], history (STRING) [slot 1], tool (STRING) [slot 2], image (IMAGE) [slot 3]

mini_party (class_type: mini_party)

• Inputs: input_str (STRING, connection), prompt (STRING, widget), model_name (STRING, widget), base_url (STRING, widget), api_key (STRING, widget), is_enable (BOOLEAN, widget)
• Outputs: STRING [slot 0]

LLM Output → Image/Video Bridge Nodes

CLIPTextEncode_party (class_type: CLIPTextEncode_party)

• Inputs: clip (CLIP, connection), text (STRING, connection from LLM output)
• Outputs: CONDITIONING [slot 0]
• Usage: Bridge LLM text → image generation. Connect LLM.assistant_response → CLIPTextEncode_party.text

json_extractor (class_type: json_extractor)

• Inputs: input (STRING, connection)
• Outputs: STRING [slot 0] (cleaned/repaired JSON)

json_get_value (class_type: json_get_value)

• Inputs: text (STRING, connection), key (STRING, widget)
• Outputs: ANY [slot 0] (extracted value)

LLM Workflow Patterns

Pattern: Multi-stage video production pipeline

LLM_api_loader(model, base_url, api_key) → model
  ↓
LLM(model, system="You are a screenwriter", user="Write a script about: {input}") → script
  ↓
LLM(model, system="Extract all characters as JSON", user=script) → characters_json
  ↓
json_extractor(characters_json) → clean JSON → json_get_value(key="character_1.appearance") → prompt
  ↓
CLIPTextEncode_party(prompt) or CLIPTextEncodeFlux(prompt) → CONDITIONING → FLUX → character_ref_image
  ↓
LLM(model, system="Break into 10 shots with video prompts", user=script) → shots_json
  ↓
For each shot:
  json_get_value(shots_json, "shot_N.description") → scene_prompt
  FLUX(scene_prompt + IP-Adapter(character_ref)) → shot_ref_image
  WanImageToVideo(shot_ref_image + video_prompt) → 10s video clip

Pattern: Simple prompt enhancement

mini_party(input="a cat", prompt="Enhance this into a detailed image prompt") → enhanced_prompt
  ↓
CLIPTextEncode_party(enhanced_prompt) → CONDITIONING → KSampler → image

Node Reference

IMPORTANT: Do NOT read the entire node registry at once. Read only the category you need.

1. First read the index: references/nodes/index.md — lists all 42 categories with file pointers
2. Then read only the specific category file you need, e.g.:
   • references/nodes/01-loaders.md — CheckpointLoaderSimple, UNETLoader, VAELoader, CLIPLoader, LoraLoader
   • references/nodes/02-conditioning.md — CLIPTextEncode, CLIPTextEncodeFlux, CLIPTextEncodeSD3
   • references/nodes/07-video-wan.md — WanImageToVideo, WanCameraImageToVideo, WanFirstLastFrameToVideo
   • references/nodes/09-flux.md — CLIPTextEncodeFlux, BasicGuider, SamplerCustomAdvanced
   • references/nodes/15-custom-samplers.md — BasicScheduler, KSamplerSelect, RandomNoise
   • references/nodes/21-audio.md — StableAudio, EmptyLatentAudio, VAEDecodeAudio
   • references/nodes/22-ltxv.md — EmptyLTXVLatentVideo, LTXVImgToVideo
   • references/nodes/25-cosmos.md — EmptyCosmosLatentVideo, CosmosImageToVideoLatent
   • references/nodes/31-hunyuan-3d.md — Hunyuan3Dv2Conditioning, VAEDecodeHunyuan3D, SaveGLB
3. Also available:
   • references/workflow-format.md — JSON format specification
   • references/common-workflows.md — common workflow patterns and best practices

Available Templates

Text-to-Image

Image-to-Image

LoRA

ControlNet

Inpainting

Video Generation

Upscale

Audio

3D Generation

Special Architectures

LLM Integration (requires comfyui_LLM_party)

CRITICAL RULE: ComfyUI auto-adds a hidden control_after_generate widget after every seed INT that has control_after_generate: True. You MUST include "randomize" in widgets_values right after every seed value. This applies to KSampler, RandomNoise, and any node with a seed INT.

Quality Checklist

Before delivering a workflow, verify:

• Top-level has id, revision, last_node_id, last_link_id, nodes, links, groups, config, extra, version
• Every node has id, type, pos, size, flags, order, mode, inputs, outputs, properties
• Widget inputs are included in inputs array with "widget": {"name": ...}, "link": null
• widgets_values order matches widget inputs order
• All links are [link_id, src_node_id, src_slot, tgt_node_id, tgt_slot, "TYPE"]
• At least one output node exists (SaveImage/PreviewImage/SaveVideo)
• No circular dependencies
• Parameter values are within valid ranges

Step 4: Output Model Download Guide

After generating the workflow JSON, ALWAYS output a Model Download Guide section listing every model file referenced in the workflow. For each model, provide:

1. Model name — the exact filename used in widgets_values
2. What it is — checkpoint / VAE / CLIP / LoRA / ControlNet / Upscale model
3. Where to put it — the ComfyUI subdirectory (models/checkpoints/, models/vae/, etc.)
4. Download source — HuggingFace or CivitAI URL

Use this reference table for common models:

SD 1.5

SDXL

FLUX

Wan 2.2

HunyuanVideo

ControlNet (SD 1.5)

Upscale Models

Output Format Example

After the workflow JSON, output like this:

## 模型下载指南

本工作流需要以下模型文件：

1. **wan2.2_t2v_14b.safetensors** (扩散模型)
   - 放置目录: `ComfyUI/models/diffusion_models/`
   - 下载地址: https://huggingface.co/Comfy-Org/Wan_2.2_ComfyUI_Repackaged

2. **umt5_xxl_fp16.safetensors** (文本编码器)
   - 放置目录: `ComfyUI/models/text_encoders/`
   - 下载地址: https://huggingface.co/Comfy-Org/Wan_2.2_ComfyUI_Repackaged

...

💡 提示: 可使用 `huggingface-cli download` 命令批量下载