ControlNet Getting Started: Controlling Composition with Line Art, Depth, and Pose

Goal in one sentence: Add ComfyUI ControlNet to a text-to-image workflow you can already run, so that the composition, depth of field, and character poses of the image follow your reference image instead of relying entirely on luck with prompts.

Prerequisite: Your First Generation: The Default Workflow Explained Node by Node.

Preparation

1. ComfyUI can already generate images normally (see the prerequisite lesson).
2. Download a ControlNet model file, put it in the ComfyUI/models/controlnet/ directory, and restart ComfyUI.
3. Prepare a reference image: a line drawing on a white background, an ordinary photo, or a full-body photo of a person all work.
4. Search the node list for the Canny, Depth, and OpenPose preprocessors; the Comfy Core nodes are not complete, so if you can't find them you're still missing the preprocessor extension — the commonly used one is ComfyUI ControlNet aux (github.com/Fannovel16/comfyui_controlnet_aux); install it first before continuing.

Follow Along

Step 1: Wire ComfyUI ControlNet into the Workflow

How-to: Double-click the canvas to add a "Load ControlNet Model" node and select the model you placed; then add an "Apply ControlNet" node. Note that ControlNet sits on the conditioning path and does not pass through MODEL — the Checkpoint's MODEL still connects directly to the KSampler's model, so don't touch it. The lines you need to change are these:

• Load ControlNet Model's CONTROL_NET → Apply ControlNet's control_net
• Positive conditioning → Apply ControlNet's positive, negative conditioning → negative
• Checkpoint's VAE → Apply ControlNet's vae
• Apply ControlNet's output positive / negative → the corresponding KSampler inputs

Expected result: the conditioning data flow becomes "CLIP Text Encode → Apply ControlNet → KSampler", MODEL is still supplied directly from the Checkpoint to KSampler, and running no longer errors about conditioning not being connected.

Step 2: Convert the Reference Image into a Control Map

How-to: Choose a preprocessor based on your control target — Canny extracts edge line art, Depth generates near/far depth, OpenPose extracts the human skeleton. Load the reference image with Load Image, connect it to the preprocessor, then connect the preprocessor's output control map to the image input of Apply ControlNet.

Expected result: the preprocessor outputs a control map: Canny gives black-and-white edge lines, Depth gives a grayscale map that is bright near and dark far, OpenPose gives a colored skeleton of points and lines.

Step 3: Tune strength and Generate

How-to: First set Apply ControlNet's strength to 0.8, run the queue once with a fixed seed; raise it if the composition doesn't match, lower it if the image is locked down by the reference image and details turn mushy, then run again to compare.

Expected result: the overall composition, pose, or depth of field of the generated image matches the reference image, while style, color palette, and details are still driven by the prompt and the model.

Verify

• With the same seed and the same prompt, run once with Apply ControlNet disconnected and once with it connected: after connecting, the composition should clearly follow the control map.
• When switching control types, the preprocessor and the ControlNet model must be swapped as a pair (a Canny map with a Canny model), otherwise the control effect becomes chaotic or even ineffective.

Common Pitfalls

• Cranking strength straight to max 1.0: the composition becomes rigid and the image gets polluted by reference-image details. Start testing from the 0.6–0.9 range.
• Reference image differs too much in size from the target: scale the reference image before preprocessing, otherwise the lines or skeleton on the control map will be misaligned.
• Stacking multiple ControlNets: chain multiple Apply ControlNet nodes (the previous one's conditioning output connects to the next one's conditioning input) and set strength for each separately — for example, line art handles outlines while pose handles actions.
• Photo background is too cluttered: Canny will pick up background edges too; you can crop the reference image first or switch to Depth.

Next Lesson

ControlNet usage differs across models; the tutorials for Z-Image and Qwen-Image models have corresponding sections (see Related Reading). Next lesson: GGUF Quantized Models: Running Large Models on Low VRAM.

Related Reading: Z-Image-Turbo Text-to-Image · Qwen-Image Text-to-Image · LTX-2/2.3 Video Generation