Director 2CS v0.21
"WhatDreamsCost" Director node modded for Ref sheets
This commit is contained in:
CGlide
184
README.md
184
README.md
@@ -1,48 +1,25 @@
|
|||||||
# Overview
|
# Overview
|
||||||
A Complete Timeline Editor For LTX 2.3. This is the sucessor of my previous nodes, and has loads of features in it. It was originally based off of [Kijai's Prompt Relay node](https://github.com/kijai/ComfyUI-PromptRelay) and LTX Sequencer/Multi Image Loader nodes.
|
|
||||||
|
|
||||||
|
|
||||||
This will be a collection of free resources for ComfyUI.
|
This will be a collection of free resources for ComfyUI.
|
||||||
|
|
||||||
Hopefully it will make creating cool stuff easier.
|
Hopefully it will make creating cool stuff easier!
|
||||||
|
|
||||||
All of my nodes are created with the help of AI, so there may or may not be redundant, messy code.
|
Also if you want to support this project or my channel I did make a Ko-Fi due to popular demand lol (anything helps!)
|
||||||
|
|
||||||
**❗❗IMPORTANT❗❗**
|
[](https://ko-fi.com/J5N221K0D5)
|
||||||
|
|
||||||
|
|
||||||
This is a Modded LTX director node with some extra options to help you create videos with references sheets
|
|
||||||
|
|
||||||
This node uses Ollama Locally : you will need to install it (very lightweight) and
|
|
||||||
|
|
||||||
- In your Ollama model folder run this command to install qwen 3.5 2b q4 (1.9gb) "ollama run huihui_ai/qwen3.5-abliterated:2B"
|
|
||||||
|
|
||||||
- The model won't eat memory while generating since there is an auto clear VRAM when you hit Run or after 5min.
|
|
||||||
|
|
||||||
- You can still enter your description manually if you don't want to install it but it works very well!
|
|
||||||
|
|
||||||
- Both references mode work (fixed), Licon MSR and Ghost Mask.
|
|
||||||
|
|
||||||
- Use their Lora, that is an important step : https://huggingface.co/LiconStudio/LTX-2.3-Multiple-Subject-Reference/tree/main
|
|
||||||
|
|
||||||
- Enjoy!
|
|
||||||
|
|
||||||
## ▶️ YouTube Tutorial Videos
|
## ▶️ YouTube Tutorial Videos
|
||||||
|
|
||||||
<a href="https://youtu.be/j28z5PZXkKk?si=TPeWKm2BW-uA3BAx">
|
|
||||||
<img width="642" height="718" alt="Capture d'écran 2026-06-17 234136" src="https://github.com/user-attachments/assets/89433110-8b78-4971-8098-3a9e9226f460" />
|
|
||||||
|
|
||||||
|
|
||||||
<table>
|
<table>
|
||||||
<tr>
|
<tr>
|
||||||
<td>
|
<td>
|
||||||
<p align="center">LTX Director Trailer</p>
|
<p align="center">LTX Director 2.0 Trailer</p>
|
||||||
<a href="https://www.youtube.com/watch?v=fZgtkRcu4_k">
|
<a href="https://www.youtube.com/watch?v=o0l6Ikvn5Q0">
|
||||||
<img src="https://img.youtube.com/vi/fZgtkRcu4_k/0.jpg" alt="LTX Director Trailer" width="400">
|
<img src="https://img.youtube.com/vi/o0l6Ikvn5Q0/0.jpg" alt="LTX Director Trailer" width="400">
|
||||||
</a>
|
</a>
|
||||||
</td>
|
</td>
|
||||||
<td>
|
<td>
|
||||||
<p align="center">LTX Director Tutorial</p>
|
<p align="center">LTX Director 1.0 Tutorial</p>
|
||||||
<a href="https://www.youtube.com/watch?v=vM60pJJqqEI">
|
<a href="https://www.youtube.com/watch?v=vM60pJJqqEI">
|
||||||
<img src="https://img.youtube.com/vi/vM60pJJqqEI/0.jpg" alt="LTX Director Tutorial" width="400">
|
<img src="https://img.youtube.com/vi/vM60pJJqqEI/0.jpg" alt="LTX Director Tutorial" width="400">
|
||||||
</a>
|
</a>
|
||||||
@@ -53,17 +30,39 @@ This node uses Ollama Locally : you will need to install it (very lightweight) a
|
|||||||
## ❓ How to install nodes
|
## ❓ How to install nodes
|
||||||
|
|
||||||
- Navigate to your `/ComfyUI/custom_nodes/ folder`
|
- Navigate to your `/ComfyUI/custom_nodes/ folder`
|
||||||
|
- Run `git clone https://github.com/WhatDreamscost/WhatDreamsCost-ComfyUI`
|
||||||
|
- Or download through the ComfyUI Manager.
|
||||||
|
|
||||||
- Run `git clone -b main_cs https://github.com/CGlideWhatDreamsCost-CSGlide.git`
|
**❗❗IMPORTANT❗❗**
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
<img width="596" height="668" alt="Capture d'écran 2026-06-13 234036" src="https://github.com/user-attachments/assets/2dfe7992-a72a-4eda-a103-41b3a8f77714" />
|
|
||||||
|
|
||||||
|
If you don't see the latest version (v1.3.9) yet in the manager then just download the nightly version (or fetch the updates to update the list to see the latest version).
|
||||||
|
Also you will need to update ComfyUI-LTXVideo and ComfyUI-KJNodes to the latest version as well. You cannot use this node without updating ComfyUI-LTXVideo!
|
||||||
|
|
||||||
# 🔄 Recent Updates
|
# 🔄 Recent Updates
|
||||||
|
|
||||||
|
**v2.0.0**
|
||||||
|
* **Massive Update to LTX Director. I will add the full list of changes later.**
|
||||||
|
|
||||||
|
- **Complete Video Support:** Edit Videos with AI all inside the node. Videos can be extended using a combination of prompts, keyframes, and audio. Trim, Split, and combine videos all within the timeline.
|
||||||
|
|
||||||
|
- **IC-LoRA Support:** Take full advantage of IC-LoRA's to take your generations to the next level. Simply drag and drop videos onto the IC-LoRA track to quickly setup IC-LoRA videos. Compatible with prompt relay, keyframe, and custom audio features within the node.
|
||||||
|
|
||||||
|
- **Audio Inpainting:** Seamlessly blend imported audio with generated audio. Not only can audio be extended, but can also be prompted alongside your imprted audio to really bring your generations to life.
|
||||||
|
|
||||||
|
- **Retake Mode (Beta):** Redirect what happens within a shot. Allows you to select a segment within a video, and re-generate what happens in that segment. An early working experiment.
|
||||||
|
|
||||||
|
- **Timeline Saving/Loading:** You can now save your timeline and settings to a json file. It will keep any videos/audio/images you have imported into the node and every setting you have changed.
|
||||||
|
|
||||||
|
- **UI Overhaul:** Huge update to the UI, dozens of big changes such as a new side bar, redesigned prompt boxes, a bunch of new settings and redesigned menus, and more.
|
||||||
|
|
||||||
|
- **Quality of Life Improvements:** Snapping, in/out points, multi-select, mark selection, workspace folder, more HUD options, resizable prompt boxes, new hotkeys, labels, filename preview options, "split at playhead" functionality, end frames (convert any keyframe into a end/last frame), toggleable tracks, NAG Support, tons of bug fixes and more!
|
||||||
|
|
||||||
|
|
||||||
|
**v1.3.9**
|
||||||
|
* **Fixed recent updates not showing in the manager**
|
||||||
|
|
||||||
|
It took like 5 tries but I finally got it working 🤦♂️
|
||||||
|
|
||||||
**v1.3.3**
|
**v1.3.3**
|
||||||
* **LTX Director Hotfix 2**
|
* **LTX Director Hotfix 2**
|
||||||
- Fixed duration_seconds input issue.
|
- Fixed duration_seconds input issue.
|
||||||
@@ -82,6 +81,10 @@ If anyone finds anymore bugs or has idea for improvements please let me know!
|
|||||||
**v1.3.1**
|
**v1.3.1**
|
||||||
* **LTX Director Example Workflow Fix**
|
* **LTX Director Example Workflow Fix**
|
||||||
- Minor fix to the example workflow (i forgot to set the clip loader type to ltxv lol)
|
- Minor fix to the example workflow (i forgot to set the clip loader type to ltxv lol)
|
||||||
|
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary>Click to view older Updates</summary>
|
||||||
|
|
||||||
**v1.3.0**
|
**v1.3.0**
|
||||||
* **New nodes: LTX Director and LTX Director Guide**
|
* **New nodes: LTX Director and LTX Director Guide**
|
||||||
@@ -97,10 +100,7 @@ If anyone finds anymore bugs or has idea for improvements please let me know!
|
|||||||
- Fixed the "invisible hitbox" underneath node issue (actually this time).
|
- Fixed the "invisible hitbox" underneath node issue (actually this time).
|
||||||
|
|
||||||
Also added a text output to the Speech Length Calculator node (can't believe i didn't do this initially)
|
Also added a text output to the Speech Length Calculator node (can't believe i didn't do this initially)
|
||||||
|
|
||||||
<details>
|
|
||||||
<summary>Click to view older Updates</summary>
|
|
||||||
|
|
||||||
**v1.2.8**
|
**v1.2.8**
|
||||||
* **Updated Load Video UI and Color Conversion**
|
* **Updated Load Video UI and Color Conversion**
|
||||||
* Added crop mode, a simple interface to crop videos. It also include various aspect ratio presets.
|
* Added crop mode, a simple interface to crop videos. It also include various aspect ratio presets.
|
||||||
@@ -154,10 +154,10 @@ Overhaul of the load audio node. Features a simple interface to easily trim audi
|
|||||||
# ⚙️ Custom Nodes
|
# ⚙️ Custom Nodes
|
||||||
|
|
||||||
|
|
||||||
## LTX Director
|
## LTX Director 2.0
|
||||||
<img width="1481" height="833" alt="Clipboard Image (2)" src="https://github.com/user-attachments/assets/08f3fe53-9393-4f5d-9de5-58b229fbed47" />
|
<img width="1562" height="870" alt="LTX_Director_Wide" src="https://github.com/user-attachments/assets/e2f9edec-c492-443e-84de-0ad1c0db04b3" />
|
||||||
|
|
||||||
|
|
||||||
|
A Complete Timeline Editor For LTX 2.3. This is the sucessor of my previous nodes, and has loads of features in it. It was originally based off of [Kijai's Prompt Relay node](https://github.com/kijai/ComfyUI-PromptRelay) and my LTX Sequencer/Multi Image Loader nodes.
|
||||||
|
|
||||||
**Main Features:**
|
**Main Features:**
|
||||||
- **Fully Functional Timeline Editor:** I spent hours studying various video editors and ended up with this design. If anyone has ideas for improvements let me know! I will adding documentation on all the functions soon.
|
- **Fully Functional Timeline Editor:** I spent hours studying various video editors and ended up with this design. If anyone has ideas for improvements let me know! I will adding documentation on all the functions soon.
|
||||||
@@ -167,11 +167,107 @@ Overhaul of the load audio node. Features a simple interface to easily trim audi
|
|||||||
- **Image to Video:** Part of the goal of this node was to make it easier to do everything, including Image to Video. It has built in resize functionality, and of course all the benifits of the prompt relay and custom audio integration.
|
- **Image to Video:** Part of the goal of this node was to make it easier to do everything, including Image to Video. It has built in resize functionality, and of course all the benifits of the prompt relay and custom audio integration.
|
||||||
- **Text to Video:** Use text segments to create T2V videos. Compatible with all other features of the node.
|
- **Text to Video:** Use text segments to create T2V videos. Compatible with all other features of the node.
|
||||||
|
|
||||||
|
**LTX Director 2.0 Update Main Features**
|
||||||
|
- **Complete Video Support:** Edit Videos with AI all inside the node. Videos can be extended using a combination of prompts, keyframes, and audio. Trim, Split, and combine videos all within the timeline.
|
||||||
|
|
||||||
|
- **IC-LoRA Support:** Take full advantage of IC-LoRA's to take your generations to the next level. Simply drag and drop videos onto the IC-LoRA track to quickly setup IC-LoRA videos. Compatible with prompt relay, keyframe, and custom audio features within the node.
|
||||||
|
|
||||||
|
Special Thanks to https://nghtdrp.com for vibe coding the inital implementation of IC-LoRA support.
|
||||||
|
|
||||||
|
- **Audio Inpainting:** Seamlessly blend imported audio with generated audio. Not only can audio be extended, but can also be prompted alongside your imprted audio to really bring your generations to life.
|
||||||
|
|
||||||
|
- **Retake Mode (Beta):** Redirect what happens within a shot. Allows you to select a segment within a video, and re-generate what happens in that segment. An early working experiment.
|
||||||
|
|
||||||
|
- **Timeline Saving/Loading:** You can now save your timeline and settings to a json file. It will keep any videos/audio/images you have imported into the node and every setting you have changed.
|
||||||
|
|
||||||
|
- **UI Overhaul:** Huge update to the UI, dozens of big changes such as a new side bar, redesigned prompt boxes, a bunch of new settings and redesigned menus, and more.
|
||||||
|
|
||||||
|
- **Quality of Life Improvements:** Snapping, in/out points, multi-select, mark selection, workspace folder, more HUD options, resizable prompt boxes, new hotkeys, labels, filename preview options, "split at playhead" functionality, end frames (convert any keyframe into a end/last frame), toggleable tracks, NAG Support, tons of bug fixes and more!
|
||||||
|
|
||||||
Download workflows here: https://github.com/WhatDreamsCost/WhatDreamsCost-ComfyUI/tree/main/example_workflows
|
Download workflows here: https://github.com/WhatDreamsCost/WhatDreamsCost-ComfyUI/tree/main/example_workflows
|
||||||
|
|
||||||
**Tutorial videos and documentation coming soon**
|
**Tutorial videos and documentation coming soon**
|
||||||
|
|
||||||
|
|
||||||
|
## Multi Image Loader
|
||||||
|
<img width="1280" height="720" alt="Multi_Image_Loader_Wide_Gif" src="https://github.com/user-attachments/assets/99b6afd8-5197-4e6c-81da-a7bd156c42c7" />
|
||||||
|
|
||||||
|
An Image loader that features a built in gallery, allowing your to easily rearrange images and output them seperately or batched together. It also combines the image resize node and LTXVPreprocess node to reduce clutter in LTX workflows.
|
||||||
|
|
||||||
|
## LTX Sequencer
|
||||||
|
|
||||||
|
|
||||||
|
An overhaul of the LTXVAddGuideMulti node. It allows you to quickly create FFLF (First Frame Last Frame) videos, shot sequences, supports any number of middle frames.
|
||||||
|
|
||||||
|
Connect the Multi Image Loader node's multi_output to automatically update the node's widgets.
|
||||||
|
|
||||||
|
It also has a sync feature that syncs all LTX Sequencer nodes together in realtime, removing the need to edit every single node manually every time you want to make a change to something.
|
||||||
|
|
||||||
|
|
||||||
For those asking, I coded all of this myself (LOL).
|
## LTX Keyframer
|
||||||
|
<img width="1082" height="608" alt="LTX Keyframer Wide" src="https://github.com/user-attachments/assets/850ba4a2-dbca-4e5a-a580-1c271e9f0c41" />
|
||||||
|
|
||||||
|
An overhaul of the LTXVImgToVideoInplaceKJ node. It allows you to quickly create FFLF (First Frame Last Frame) videos and shot sequences. Also upports any number of middle frames.
|
||||||
|
|
||||||
|
Connect the Multi Image Loader node's multi_output to automatically update the node's widgets.
|
||||||
|
|
||||||
|
It also has a sync feature that syncs all LTX Keyframer nodes together in realtime, removing the need to edit every single node manually every time you want to make a change to something.
|
||||||
|
|
||||||
|
**I would recommend using the LTX Sequencer Node over this node, after further testing it seems superior in at pretty much everything. I'll leave it in just in case more people want to test it**
|
||||||
|
|
||||||
|
## Speech Length Calculator
|
||||||
|
<img width="1280" height="720" alt="Speech Length Calculator v2 Gif" src="https://github.com/user-attachments/assets/04b9a1cf-20e4-4b7b-a9c6-4a5a0825995b" />
|
||||||
|
<br>
|
||||||
|
<br>
|
||||||
|
This node calculates in realtime how long a video should be based on the dialogue. Any words in quotations will be considered as speech. The node updates in realtime without having to run the workflow, and outputs the length depending on how fast the speech is.
|
||||||
|
|
||||||
|
If you connect another string/text node to the text_input, it will still update in the length in realtime.
|
||||||
|
|
||||||
|
I kept having to play the guessing game on my own generations so I made this node to make it easier :man_shrugging:
|
||||||
|
|
||||||
|
## Load Video UI
|
||||||
|
<table width="100%">
|
||||||
|
<tr>
|
||||||
|
<td width="50%" align="center">
|
||||||
|
<p>Simple Controls</p>
|
||||||
|
<img src="https://github.com/user-attachments/assets/fb76ff03-a6ff-4837-bd63-7e429f5f3d37" width="100%" />
|
||||||
|
</td>
|
||||||
|
<td width="50%" align="center">
|
||||||
|
<p>New Crop Mode!</p>
|
||||||
|
<img src="https://github.com/user-attachments/assets/28cfb4ca-e42a-44da-9afb-f20cb01b9722" width="100%" />
|
||||||
|
</td>
|
||||||
|
</tr>
|
||||||
|
</table>
|
||||||
|
|
||||||
|
<br>
|
||||||
|
<br>
|
||||||
|
An upgraded Load Video node. It has the following features:
|
||||||
|
|
||||||
|
* Simple interface to quickly trim videos and preview them in realtime.
|
||||||
|
* Ability to load any length of video into the node (the default load video node was limited to 100MB files)
|
||||||
|
* Easily switch between showing seconds and frames with a toggle button. This will change the widgets as well as the interface.
|
||||||
|
* Multiple options for resizing the video (maintain aspect ratio, crop, stretch to fit, pad)
|
||||||
|
* Allows dragging and dropping files into the node
|
||||||
|
* Progress bar
|
||||||
|
* Optimized to use less RAM (still very limited due to ComfyUI limitations, but at least a little more efficient)
|
||||||
|
|
||||||
|
Please note that due to ComfyUI limitations (and the fact that this node doesn't use any addtional libraries), this node will not work well for outputting large videos. You can trim any length of video without a problem, but if the output is still large it will end up using a lot of RAM. I have implemented various optimizations though to make it use less memory.
|
||||||
|
|
||||||
|
## Load Audio UI
|
||||||
|
<img width="1280" height="720" alt="Load_Audio_UI_V2" src="https://github.com/user-attachments/assets/e3dc5c8d-d0b9-4336-8196-944204719239" />
|
||||||
|
<br>
|
||||||
|
<br>
|
||||||
|
An upgraded Load Audio node. Features a simple interface to easily trim audio. Also allows dragging and dropping files (fixes the original node that doesn't allow dropping in videos). Also compatible with nodes 2.0.
|
||||||
|
|
||||||
|
# 💡 Workflows
|
||||||
|
Download workflows here: https://github.com/WhatDreamsCost/WhatDreamsCost-ComfyUI/tree/main/example_workflows
|
||||||
|
|
||||||
|
# ❗ Known Issues
|
||||||
|
|
||||||
|
Fixed everything so far. If there are any other issue or bugs you find please let me know!
|
||||||
|
|
||||||
|
# 💡 Additional Info
|
||||||
|
|
||||||
|
Feel free to suggest improvements, and if you run into any bugs let me know!
|
||||||
|
|
||||||
|
For those asking, I mainly used gemini to create these nodes.
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
from .ltx_director import LTXDirector
|
from .ltx_director import LTXDirector
|
||||||
from .ltx_director_guide import LTXDirectorGuide
|
from .ltx_director_guide import LTXDirectorGuide, LTXDirectorCropGuides
|
||||||
from comfy_api.latest import ComfyExtension, io
|
from comfy_api.latest import ComfyExtension, io
|
||||||
from typing_extensions import override
|
from typing_extensions import override
|
||||||
from .latent_slice import CleanLatentSlice
|
from .latent_slice import CleanLatentSlice
|
||||||
@@ -9,7 +9,6 @@ class PromptRelay(ComfyExtension):
|
|||||||
async def get_node_list(self) -> list[type[io.ComfyNode]]:
|
async def get_node_list(self) -> list[type[io.ComfyNode]]:
|
||||||
return [
|
return [
|
||||||
LTXDirector,
|
LTXDirector,
|
||||||
LTXDirectorGuide
|
|
||||||
]
|
]
|
||||||
|
|
||||||
async def comfy_entrypoint() -> PromptRelay:
|
async def comfy_entrypoint() -> PromptRelay:
|
||||||
@@ -18,12 +17,14 @@ async def comfy_entrypoint() -> PromptRelay:
|
|||||||
NODE_CLASS_MAPPINGS = {
|
NODE_CLASS_MAPPINGS = {
|
||||||
"LTXDirectorCS": LTXDirector,
|
"LTXDirectorCS": LTXDirector,
|
||||||
"LTXDirectorGuideCS": LTXDirectorGuide,
|
"LTXDirectorGuideCS": LTXDirectorGuide,
|
||||||
|
"LTXDirectorCropGuidesCS": LTXDirectorCropGuides,
|
||||||
"CleanLatentSliceCS": CleanLatentSlice,
|
"CleanLatentSliceCS": CleanLatentSlice,
|
||||||
}
|
}
|
||||||
|
|
||||||
NODE_DISPLAY_NAME_MAPPINGS = {
|
NODE_DISPLAY_NAME_MAPPINGS = {
|
||||||
"LTXDirectorCS": "LTX Director CS",
|
"LTXDirectorCS": "LTX Director CS",
|
||||||
"LTXDirectorGuideCS": "LTX Director Guide CS",
|
"LTXDirectorGuideCS": "LTX Director Guide CS",
|
||||||
|
"LTXDirectorCropGuidesCS": "LTX Director Crop Guides CS",
|
||||||
"CleanLatentSliceCS": "Clean Latent Slice CS",
|
"CleanLatentSliceCS": "Clean Latent Slice CS",
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
379053
debug_prompt_relay.log
Normal file
379053
debug_prompt_relay.log
Normal file
File diff suppressed because it is too large
Load Diff
1
example_workflows/LTX Director CS.json
Normal file
1
example_workflows/LTX Director CS.json
Normal file
File diff suppressed because one or more lines are too long
11290
js/ltx_director.js
11290
js/ltx_director.js
File diff suppressed because it is too large
Load Diff
@@ -1,12 +1,24 @@
|
|||||||
import { app } from "../../scripts/app.js";
|
import { app } from "../../scripts/app.js";
|
||||||
|
|
||||||
// LTX Director Guide CS is a pure pass-through processor node.
|
// LTX Director Guide is a pure pass-through processor node.
|
||||||
// All configuration (images, insert frames, strengths) comes from
|
// All configuration (images, insert frames, strengths) comes from
|
||||||
// the guide_data output of the LTX Director CS (Timeline) node.
|
// the guide_data output of Prompt Relay Encode (Timeline).
|
||||||
app.registerExtension({
|
app.registerExtension({
|
||||||
name: "Comfy.LTXDirectorGuideCS",
|
name: "Comfy.LTXDirectorGuideCS",
|
||||||
async nodeCreated(node) {
|
async nodeCreated(node) {
|
||||||
if (node.comfyClass !== "LTXDirectorGuideCS") return;
|
if (node.comfyClass !== "LTXDirectorGuideCS") return;
|
||||||
// Nothing to initialize — the node has no configurable widgets.
|
|
||||||
},
|
// Hide retake_mode widget on LiteGraph as it is dynamically auto-detected from the timeline data.
|
||||||
});
|
const w = node.widgets?.find(x => x.name === "retake_mode");
|
||||||
|
if (w) {
|
||||||
|
w.hidden = true;
|
||||||
|
if (!w.options) w.options = {};
|
||||||
|
w.options.hidden = true;
|
||||||
|
if (!window.LiteGraph || !window.LiteGraph.vueNodesMode) {
|
||||||
|
w.computeSize = () => [0, -4];
|
||||||
|
w.draw = () => { };
|
||||||
|
}
|
||||||
|
if (w.element) w.element.style.display = "none";
|
||||||
|
}
|
||||||
|
},
|
||||||
|
});
|
||||||
2295
ltx_director.py
2295
ltx_director.py
File diff suppressed because it is too large
Load Diff
@@ -1,106 +1,637 @@
|
|||||||
from comfy_extras.nodes_lt import LTXVAddGuide
|
import logging
|
||||||
|
import math
|
||||||
|
import os
|
||||||
|
import av
|
||||||
|
import numpy as np
|
||||||
import torch
|
import torch
|
||||||
|
import comfy
|
||||||
|
import comfy.sd
|
||||||
import comfy.utils
|
import comfy.utils
|
||||||
|
import folder_paths
|
||||||
|
import node_helpers
|
||||||
|
from comfy_extras import nodes_lt
|
||||||
from comfy_api.latest import io
|
from comfy_api.latest import io
|
||||||
from .ltx_director import GuideData
|
from .ltx_director import GuideData, MotionGuideData, _resize_image, _execute_comfy_node, _unpack
|
||||||
|
|
||||||
|
log = logging.getLogger(__name__)
|
||||||
|
|
||||||
class LTXDirectorGuide(LTXVAddGuide):
|
# --- Helper Functions from Nghtdrp ---
|
||||||
@classmethod
|
|
||||||
def define_schema(cls):
|
def _get_guide_attention_entries(conditioning):
|
||||||
return io.Schema(
|
for item in conditioning:
|
||||||
node_id="LTXDirectorGuideCS",
|
entries = item[1].get("guide_attention_entries", None)
|
||||||
display_name="LTX Director Guide CS",
|
if entries is not None:
|
||||||
category="WhatDreamsCost CS",
|
return entries
|
||||||
description=(
|
return []
|
||||||
"Applies guide images from a Prompt Relay Timeline node at the frame positions "
|
|
||||||
"and strengths defined on the timeline. Connect guide_data from the timeline node."
|
def _set_guide_attention_entries(conditioning, entries):
|
||||||
),
|
return node_helpers.conditioning_set_values(
|
||||||
inputs=[
|
conditioning, {"guide_attention_entries": entries}
|
||||||
io.Conditioning.Input("positive", tooltip="Positive conditioning to add guide keyframe info to."),
|
)
|
||||||
io.Conditioning.Input("negative", tooltip="Negative conditioning to add guide keyframe info to."),
|
|
||||||
io.Vae.Input("vae", tooltip="Video VAE used to encode the guide images."),
|
def _append_guide_attention_entry(
|
||||||
io.Latent.Input("latent", tooltip="Video latent — guides are inserted into this latent."),
|
conditioning,
|
||||||
GuideData.Input("guide_data", tooltip="Guide data produced by Prompt Relay Encode (Timeline)."),
|
pre_filter_count,
|
||||||
io.Float.Input("scale_by", default=1.0, min=0.01, max=8.0, step=0.01, tooltip="Scale the latent by this factor."),
|
latent_shape,
|
||||||
io.Combo.Input("upscale_method", options=["nearest-exact", "bilinear", "area", "bicubic", "bislerp"], default="bicubic", tooltip="Method used to upscale/downscale the latent."),
|
attention_strength=1.0,
|
||||||
io.Float.Input("msr_strength", default=0.0, min=0.0, max=1.0, step=0.05, tooltip="Licon MSR only: per-stage reference strength override. 0 = use the Director's value (full pull, right for stage 1). On a refinement/upscale stage set ~0.4 to hold detail without the references repainting the opening (fixes stage-2 mist/ghosting)."),
|
attention_mask=None,
|
||||||
],
|
):
|
||||||
outputs=[
|
entries = [*_get_guide_attention_entries(conditioning)]
|
||||||
io.Conditioning.Output(display_name="positive"),
|
entries.append(
|
||||||
io.Conditioning.Output(display_name="negative"),
|
{
|
||||||
io.Latent.Output(display_name="latent", tooltip="Video latent with guide frames applied."),
|
"pre_filter_count": int(pre_filter_count),
|
||||||
],
|
"strength": float(attention_strength),
|
||||||
|
"pixel_mask": attention_mask,
|
||||||
|
"latent_shape": list(latent_shape),
|
||||||
|
}
|
||||||
|
)
|
||||||
|
return _set_guide_attention_entries(conditioning, entries)
|
||||||
|
|
||||||
|
def _clone_noise_mask(latent, latent_image):
|
||||||
|
if "noise_mask" in latent and latent["noise_mask"] is not None:
|
||||||
|
return latent["noise_mask"].clone()
|
||||||
|
batch, _, frames, _, _ = latent_image.shape
|
||||||
|
return torch.ones(
|
||||||
|
(batch, 1, frames, 1, 1),
|
||||||
|
dtype=torch.float32,
|
||||||
|
device=latent_image.device,
|
||||||
|
)
|
||||||
|
|
||||||
|
def _resize_latent_spatial(latent_image, noise_mask, width, height, method):
|
||||||
|
b, c, f, h, w = latent_image.shape
|
||||||
|
if width == w and height == h:
|
||||||
|
return latent_image, noise_mask
|
||||||
|
|
||||||
|
latent_4d = latent_image.permute(0, 2, 1, 3, 4).reshape(b * f, c, h, w)
|
||||||
|
latent_4d = comfy.utils.common_upscale(latent_4d, width, height, method, "disabled")
|
||||||
|
latent_image = latent_4d.reshape(b, f, c, height, width).permute(0, 2, 1, 3, 4)
|
||||||
|
|
||||||
|
if noise_mask is not None and (noise_mask.shape[-1] > 1 or noise_mask.shape[-2] > 1):
|
||||||
|
mb, mc, mf, mh, mw = noise_mask.shape
|
||||||
|
mask_4d = noise_mask.permute(0, 2, 1, 3, 4).reshape(mb * mf, mc, mh, mw)
|
||||||
|
mask_4d = comfy.utils.common_upscale(mask_4d, width, height, method, "disabled")
|
||||||
|
noise_mask = mask_4d.reshape(mb, mf, mc, height, width).permute(0, 2, 1, 3, 4)
|
||||||
|
|
||||||
|
return latent_image, noise_mask
|
||||||
|
|
||||||
|
def _ceil_to_multiple(value, multiple):
|
||||||
|
multiple = max(1, int(multiple))
|
||||||
|
return int(math.ceil(value / multiple) * multiple)
|
||||||
|
|
||||||
|
def _snap_latent_to_downscale(latent_image, noise_mask, downscale_factor, method):
|
||||||
|
factor = int(max(1, round(float(downscale_factor))))
|
||||||
|
if factor <= 1:
|
||||||
|
return latent_image, noise_mask
|
||||||
|
|
||||||
|
_, _, _, h, w = latent_image.shape
|
||||||
|
new_w = _ceil_to_multiple(w, factor)
|
||||||
|
new_h = _ceil_to_multiple(h, factor)
|
||||||
|
if new_w == w and new_h == h:
|
||||||
|
return latent_image, noise_mask
|
||||||
|
|
||||||
|
log.warning(
|
||||||
|
"[LTXDirectorGuide] Auto-snapping latent grid from %sx%s to %sx%s for IC-LoRA downscale factor %s.",
|
||||||
|
w, h, new_w, new_h, factor,
|
||||||
|
)
|
||||||
|
return _resize_latent_spatial(latent_image, noise_mask, new_w, new_h, method)
|
||||||
|
|
||||||
|
def _load_lora_model_only(model, ic_lora_name, strength_model):
|
||||||
|
lora_path = folder_paths.get_full_path_or_raise("loras", ic_lora_name)
|
||||||
|
lora, metadata = comfy.utils.load_torch_file(
|
||||||
|
lora_path, safe_load=True, return_metadata=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
try:
|
||||||
|
latent_downscale_factor = float(metadata["reference_downscale_factor"])
|
||||||
|
except Exception:
|
||||||
|
latent_downscale_factor = 1.0
|
||||||
|
log.warning(
|
||||||
|
"[LTXDirectorGuide] Could not read reference_downscale_factor from %s, using 1.0.",
|
||||||
|
ic_lora_name,
|
||||||
)
|
)
|
||||||
|
|
||||||
@classmethod
|
if strength_model == 0:
|
||||||
def execute(cls, positive, negative, vae, latent, guide_data, scale_by=1.0, upscale_method="bicubic", msr_strength=0.0) -> io.NodeOutput:
|
return model, latent_downscale_factor
|
||||||
scale_factors = vae.downscale_index_formula
|
|
||||||
|
|
||||||
# Clone latents to avoid mutating upstream nodes
|
model_lora, _ = comfy.sd.load_lora_for_models(
|
||||||
latent_image = latent["samples"].clone()
|
model, None, lora, strength_model, 0,
|
||||||
|
)
|
||||||
|
return model_lora, latent_downscale_factor
|
||||||
|
|
||||||
if "noise_mask" in latent:
|
def _encode_video_iclora_guide(vae, latent_width, latent_height, images, scale_factors, latent_downscale_factor, crop, use_tiled_encode, tile_size, tile_overlap, resize_method="crop"):
|
||||||
noise_mask = latent["noise_mask"].clone()
|
time_scale_factor, width_scale_factor, height_scale_factor = scale_factors
|
||||||
|
|
||||||
|
num_frames_to_keep = ((images.shape[0] - 1) // time_scale_factor) * time_scale_factor + 1
|
||||||
|
images = images[:num_frames_to_keep]
|
||||||
|
|
||||||
|
target_width = int(latent_width * width_scale_factor / latent_downscale_factor)
|
||||||
|
target_height = int(latent_height * height_scale_factor / latent_downscale_factor)
|
||||||
|
target_width = max(8, target_width)
|
||||||
|
target_height = max(8, target_height)
|
||||||
|
|
||||||
|
# For guides, we must match the exact target latent shape.
|
||||||
|
# "maintain aspect ratio" doesn't pad, which causes shape mismatch in VAE encode / concatenation.
|
||||||
|
# So we fallback "maintain aspect ratio" to "pad".
|
||||||
|
if resize_method == "maintain aspect ratio":
|
||||||
|
resize_method = "pad"
|
||||||
|
|
||||||
|
pixels = _resize_image(images, target_width, target_height, resize_method, divisible_by=1)
|
||||||
|
|
||||||
|
encode_pixels = pixels[:, :, :, :3]
|
||||||
|
if use_tiled_encode:
|
||||||
|
guide_latent = vae.encode_tiled(encode_pixels, tile_x=tile_size, tile_y=tile_size, overlap=tile_overlap)
|
||||||
|
else:
|
||||||
|
guide_latent = vae.encode(encode_pixels)
|
||||||
|
|
||||||
|
return pixels, guide_latent
|
||||||
|
|
||||||
|
def _dilate_latent(latent: dict, horizontal_scale: int, vertical_scale: int) -> dict:
|
||||||
|
if horizontal_scale == 1 and vertical_scale == 1:
|
||||||
|
return latent
|
||||||
|
|
||||||
|
samples = latent["samples"]
|
||||||
|
mask = latent.get("noise_mask", None)
|
||||||
|
|
||||||
|
dilated_shape = samples.shape[:3] + (
|
||||||
|
samples.shape[3] * vertical_scale,
|
||||||
|
samples.shape[4] * horizontal_scale,
|
||||||
|
)
|
||||||
|
dilated_samples = torch.zeros(dilated_shape, device=samples.device, dtype=samples.dtype, requires_grad=False)
|
||||||
|
dilated_samples[..., ::vertical_scale, ::horizontal_scale] = samples
|
||||||
|
|
||||||
|
dilated_mask_shape = (
|
||||||
|
dilated_samples.shape[0], 1, dilated_samples.shape[2],
|
||||||
|
dilated_samples.shape[3], dilated_samples.shape[4],
|
||||||
|
)
|
||||||
|
dilated_mask = torch.full(dilated_mask_shape, -1.0, device=samples.device, dtype=samples.dtype, requires_grad=False)
|
||||||
|
dilated_mask[..., ::vertical_scale, ::horizontal_scale] = (mask if mask is not None else 1.0)
|
||||||
|
|
||||||
|
return {"samples": dilated_samples, "noise_mask": dilated_mask}
|
||||||
|
|
||||||
|
def _resolve_input_video_path(video_file):
|
||||||
|
if os.path.isabs(str(video_file)) and os.path.exists(str(video_file)):
|
||||||
|
return str(video_file)
|
||||||
|
input_dir = folder_paths.get_input_directory()
|
||||||
|
candidate = os.path.join(input_dir, str(video_file))
|
||||||
|
if os.path.exists(candidate):
|
||||||
|
return candidate
|
||||||
|
try:
|
||||||
|
annotated = folder_paths.get_annotated_filepath(str(video_file))
|
||||||
|
if annotated and os.path.exists(annotated):
|
||||||
|
return annotated
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
raise FileNotFoundError(f"Could not find motion guide video: {video_file}")
|
||||||
|
|
||||||
|
class ResampleGuideFrames:
|
||||||
|
def execute(self, images, source_fps, target_fps, target_num_frames, mode):
|
||||||
|
if images is None: return images
|
||||||
|
frames = images
|
||||||
|
n = int(frames.shape[0])
|
||||||
|
target_num_frames = int(target_num_frames)
|
||||||
|
if n <= 1:
|
||||||
|
if target_num_frames > 1 and n == 1:
|
||||||
|
return frames.repeat(target_num_frames, 1, 1, 1)
|
||||||
|
return frames
|
||||||
|
source_fps = float(max(0.001, source_fps))
|
||||||
|
target_fps = float(max(0.001, target_fps))
|
||||||
|
if target_num_frames <= 0:
|
||||||
|
duration = (n - 1) / source_fps
|
||||||
|
target_num_frames = max(1, int(round(duration * target_fps)) + 1)
|
||||||
|
if target_num_frames == n and abs(target_fps - source_fps) < 1e-6:
|
||||||
|
return frames
|
||||||
|
positions = torch.linspace(0, n - 1, target_num_frames, device=frames.device, dtype=torch.float32)
|
||||||
|
if mode == "nearest":
|
||||||
|
idx = torch.round(positions).long().clamp(0, n - 1)
|
||||||
|
return frames.index_select(0, idx)
|
||||||
|
idx0 = torch.floor(positions).long().clamp(0, n - 1)
|
||||||
|
idx1 = torch.ceil(positions).long().clamp(0, n - 1)
|
||||||
|
alpha = (positions - idx0.to(positions.dtype)).view(-1, 1, 1, 1)
|
||||||
|
f0 = frames.index_select(0, idx0).to(torch.float32)
|
||||||
|
f1 = frames.index_select(0, idx1).to(torch.float32)
|
||||||
|
return (f0 * (1.0 - alpha) + f1 * alpha).to(frames.dtype)
|
||||||
|
|
||||||
|
def _load_motion_video_frames(video_file, trim_start_frames, length_frames, director_fps, resample_mode="nearest"):
|
||||||
|
path = _resolve_input_video_path(video_file)
|
||||||
|
target_fps = max(1.0, float(director_fps))
|
||||||
|
start_s = max(0.0, float(trim_start_frames) / target_fps)
|
||||||
|
dur_s = max(0.0, float(length_frames) / target_fps)
|
||||||
|
end_s = start_s + dur_s if dur_s > 0 else None
|
||||||
|
|
||||||
|
log.info(f"[LTXDirectorGuide] Loading video frames from {path}. start_s: {start_s:.3f}, dur_s: {dur_s:.3f}, end_s: {end_s}")
|
||||||
|
|
||||||
|
container = av.open(path)
|
||||||
|
stream = container.streams.video[0]
|
||||||
|
stream.thread_type = "AUTO" # Enable multi-threaded decoding
|
||||||
|
|
||||||
|
try:
|
||||||
|
source_fps = float(stream.average_rate) if stream.average_rate else float(stream.base_rate)
|
||||||
|
except Exception:
|
||||||
|
source_fps = target_fps
|
||||||
|
if source_fps <= 0: source_fps = target_fps
|
||||||
|
|
||||||
|
# Seek to keyframe slightly before target start_s to optimize loading speed and memory
|
||||||
|
if start_s > 0:
|
||||||
|
try:
|
||||||
|
if stream.time_base:
|
||||||
|
seek_pts = int(max(0, start_s - 0.5) / float(stream.time_base))
|
||||||
|
else:
|
||||||
|
seek_pts = int(max(0, start_s - 0.5) * av.time_base)
|
||||||
|
container.seek(seek_pts, stream=stream, backward=True)
|
||||||
|
log.info(f"[LTXDirectorGuide] PyAV seeked stream to pts={seek_pts} (approx {max(0, start_s - 0.5):.3f}s)")
|
||||||
|
except Exception as seek_err:
|
||||||
|
log.warning(f"[LTXDirectorGuide] Seek failed: {seek_err}, decoding from beginning.")
|
||||||
|
|
||||||
|
frames = []
|
||||||
|
decoded_count = 0
|
||||||
|
for frame in container.decode(stream):
|
||||||
|
if frame.time is not None:
|
||||||
|
t = float(frame.time)
|
||||||
|
elif frame.pts is not None and stream.time_base is not None:
|
||||||
|
t = float(frame.pts * stream.time_base)
|
||||||
else:
|
else:
|
||||||
batch, _, latent_frames, latent_height, latent_width = latent_image.shape
|
t = float(decoded_count / source_fps)
|
||||||
noise_mask = torch.ones(
|
decoded_count += 1
|
||||||
(batch, 1, latent_frames, 1, 1),
|
|
||||||
dtype=torch.float32,
|
if t < start_s - 0.01: continue
|
||||||
device=latent_image.device,
|
if end_s is not None and t >= end_s: break
|
||||||
)
|
|
||||||
|
# Append raw uint8 numpy arrays to minimize CPU allocation overhead
|
||||||
|
frames.append(frame.to_ndarray(format="rgb24"))
|
||||||
|
container.close()
|
||||||
|
|
||||||
|
if not frames: raise ValueError(f"No frames decoded for motion guide segment: {video_file}")
|
||||||
|
|
||||||
|
# Convert all frames to float32 at once to optimize memory allocation
|
||||||
|
frames_np = np.array(frames, dtype=np.float32) / 255.0
|
||||||
|
images = torch.from_numpy(frames_np)
|
||||||
|
|
||||||
|
target_count = max(1, int(round(float(length_frames))))
|
||||||
|
images = ResampleGuideFrames().execute(images, source_fps, target_fps, target_count, resample_mode)
|
||||||
|
return images
|
||||||
|
|
||||||
|
# --- Main Class ---
|
||||||
|
|
||||||
|
class LTXDirectorGuide:
|
||||||
|
@classmethod
|
||||||
|
def INPUT_TYPES(cls):
|
||||||
|
loras = folder_paths.get_filename_list("loras")
|
||||||
|
if not loras: loras = ["put_ic_lora_in_ComfyUI_models_loras"]
|
||||||
|
return {
|
||||||
|
"required": {
|
||||||
|
"positive": ("CONDITIONING", {"tooltip": "Positive conditioning to add guide keyframe info to."}),
|
||||||
|
"negative": ("CONDITIONING", {"tooltip": "Negative conditioning to add guide keyframe info to."}),
|
||||||
|
"vae": ("VAE", {"tooltip": "Video VAE used to encode the guide images."}),
|
||||||
|
"latent": ("LATENT", {"tooltip": "Video latent — guides are inserted into this latent."}),
|
||||||
|
"guide_data": ("GUIDE_DATA", {"tooltip": "Guide data produced by Prompt Relay Encode (Timeline)."}),
|
||||||
|
},
|
||||||
|
"optional": {
|
||||||
|
"motion_guide_data": ("MOTION_GUIDE_DATA", {"tooltip": "Connect motion guide data from the timeline node to use IC-LoRA video guidance."}),
|
||||||
|
"model": ("MODEL", {"tooltip": "Connect model if using IC-LoRA for motion guidance."}),
|
||||||
|
"ic_lora_name": (["None"] + loras, {"default": "None", "tooltip": "Select the IC-LoRA model to use for motion guidance."}),
|
||||||
|
"ic_lora_strength": ("FLOAT", {"default": 1.0, "min": -100.0, "max": 100.0, "step": 0.01}),
|
||||||
|
"scale_by": ("FLOAT", {"default": 0.5, "min": 0.01, "max": 8.0, "step": 0.01, "tooltip": "Scale the latent by this factor. 0.5 is recommended for IC-LoRA."}),
|
||||||
|
"upscale_method": (["nearest-exact", "bilinear", "area", "bicubic", "bislerp"], {"default": "bicubic", "tooltip": "Method used to upscale/downscale the latent."}),
|
||||||
|
"image_attention_strength": ("FLOAT", {"default": 1.0, "min": 0.0, "max": 1.0, "step": 0.01}),
|
||||||
|
"crop": (["disabled", "center"], {"default": "center"}),
|
||||||
|
"auto_snap_ic_grid": ("BOOLEAN", {"default": True}),
|
||||||
|
"use_tiled_encode": ("BOOLEAN", {"default": False}),
|
||||||
|
"tile_size": ("INT", {"default": 256, "min": 64, "max": 512, "step": 32}),
|
||||||
|
"tile_overlap": ("INT", {"default": 64, "min": 16, "max": 256, "step": 16}),
|
||||||
|
"retake_mode": ("BOOLEAN", {"default": False, "tooltip": "Force Retake Mode. If false, it will still auto-detect Retake Mode from the timeline data."}),
|
||||||
|
"msr_strength": ("FLOAT", {"default": 0.0, "min": 0.0, "max": 1.0, "step": 0.05, "tooltip": "Licon MSR only: per-stage reference strength override. 0 = use the Director's value (full pull, right for stage 1). On a refinement/upscale stage set ~0.4 to hold detail without the references repainting the opening (fixes stage-2 mist/ghosting)."}),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
RETURN_TYPES = ("CONDITIONING", "CONDITIONING", "LATENT", "MODEL", "FLOAT")
|
||||||
|
RETURN_NAMES = ("positive", "negative", "latent", "model", "latent_downscale_factor")
|
||||||
|
FUNCTION = "execute"
|
||||||
|
|
||||||
|
@classmethod
|
||||||
|
def execute(cls, positive, negative, vae, latent, guide_data, motion_guide_data=None, model=None, ic_lora_name="None", ic_lora_strength=1.0, scale_by=1.0, upscale_method="bicubic", image_attention_strength=1.0, crop="center", auto_snap_ic_grid=True, use_tiled_encode=False, tile_size=256, tile_overlap=64, retake_mode=False, msr_strength=0.0):
|
||||||
|
motion_segments = (motion_guide_data or {}).get("segments", []) if motion_guide_data else []
|
||||||
|
image_guides_count = len(guide_data.get("images", [])) if guide_data else 0
|
||||||
|
print(f"[LTXDirectorGuide] execute started. motion_segments: {len(motion_segments)}, image_guides: {image_guides_count}, ic_lora_name: {ic_lora_name}, model connected: {model is not None}, retake_mode: {retake_mode}")
|
||||||
|
|
||||||
|
active_resize_method = guide_data.get("resize_method") if guide_data else None
|
||||||
|
if not active_resize_method:
|
||||||
|
active_resize_method = motion_guide_data.get("resize_method") if motion_guide_data else None
|
||||||
|
if not active_resize_method:
|
||||||
|
active_resize_method = "crop" if crop == "center" else "stretch to fit"
|
||||||
|
|
||||||
|
latent_downscale_factor = 1.0
|
||||||
|
if model is not None and ic_lora_name != "None":
|
||||||
|
model, latent_downscale_factor = _load_lora_model_only(model, ic_lora_name, ic_lora_strength)
|
||||||
|
|
||||||
|
scale_factors = vae.downscale_index_formula
|
||||||
|
latent_image = latent["samples"].clone()
|
||||||
|
noise_mask = _clone_noise_mask(latent, latent_image)
|
||||||
|
|
||||||
# Apply scale factor if not 1.0
|
|
||||||
if scale_by != 1.0:
|
if scale_by != 1.0:
|
||||||
B, C, F, H, W = latent_image.shape
|
_, _, _, h, w = latent_image.shape
|
||||||
width = round(W * scale_by)
|
target_w = max(1, round(w * scale_by))
|
||||||
height = round(H * scale_by)
|
target_h = max(1, round(h * scale_by))
|
||||||
|
latent_image, noise_mask = _resize_latent_spatial(latent_image, noise_mask, target_w, target_h, upscale_method)
|
||||||
# Reshape to 4D for common_upscale
|
|
||||||
latent_4d = latent_image.permute(0, 2, 1, 3, 4).reshape(B * F, C, H, W)
|
|
||||||
latent_resized_4d = comfy.utils.common_upscale(latent_4d, width, height, upscale_method, "disabled")
|
|
||||||
latent_image = latent_resized_4d.reshape(B, F, C, height, width).permute(0, 2, 1, 3, 4)
|
|
||||||
|
|
||||||
# Also resize noise mask if it's not a broadcasted mask
|
if auto_snap_ic_grid and model is not None and ic_lora_name != "None":
|
||||||
if noise_mask.shape[-1] > 1 or noise_mask.shape[-2] > 1:
|
latent_image, noise_mask = _snap_latent_to_downscale(latent_image, noise_mask, latent_downscale_factor, upscale_method)
|
||||||
mask_4d = noise_mask.permute(0, 2, 1, 3, 4).reshape(B * F, 1, H, W)
|
|
||||||
mask_resized_4d = comfy.utils.common_upscale(mask_4d, width, height, upscale_method, "disabled")
|
# Ghost Mask: the Director kept the latent clean (so it stays length-matched with the audio
|
||||||
noise_mask = mask_resized_4d.reshape(B, F, 1, height, width).permute(0, 2, 1, 3, 4)
|
# latent) and asked us to pre-extend it here by one hidden frame per reference. The tail
|
||||||
|
# sits past the clean region; the reference images (in guide_data["images"], anchored at
|
||||||
|
# (clean + i)) are written into it by the loop below, then dropped downstream by Clean
|
||||||
|
# Latent Slice (length = clean_latent_frames).
|
||||||
|
_ghost_pre_extend = int(guide_data.get("ghost_pre_extend", 0)) if guide_data else 0
|
||||||
|
if _ghost_pre_extend > 0:
|
||||||
|
gb, gc, gf, gh, gw = latent_image.shape
|
||||||
|
latent_image = torch.cat(
|
||||||
|
[latent_image, torch.zeros((gb, gc, _ghost_pre_extend, gh, gw), dtype=latent_image.dtype, device=latent_image.device)],
|
||||||
|
dim=2,
|
||||||
|
)
|
||||||
|
mb, mc, mf, mh, mw = noise_mask.shape
|
||||||
|
noise_mask = torch.cat(
|
||||||
|
[noise_mask, torch.ones((mb, mc, _ghost_pre_extend, mh, mw), dtype=noise_mask.dtype, device=noise_mask.device)],
|
||||||
|
dim=2,
|
||||||
|
)
|
||||||
|
|
||||||
_, _, latent_length, latent_height, latent_width = latent_image.shape
|
_, _, latent_length, latent_height, latent_width = latent_image.shape
|
||||||
|
initial_latent_length = int(latent_length)
|
||||||
|
|
||||||
# MSR mode: the Director handed us the raw references; inject them at THIS node's
|
# Licon MSR (Prefix) mode: the Director handed us raw references via guide_data["msr"].
|
||||||
# resolution (so a half-res Stage 1 and a full-res Stage 2 each stay self-consistent).
|
# Inject them at THIS node's resolution (so a half-res Stage 1 and full-res Stage 2 each
|
||||||
msr = guide_data.get("msr")
|
# stay self-consistent), then return early — the standard guide paths don't apply.
|
||||||
|
msr = guide_data.get("msr") if guide_data else None
|
||||||
if msr is not None:
|
if msr is not None:
|
||||||
return cls._inject_msr(positive, negative, vae, latent_image, noise_mask, msr, msr_strength)
|
return cls._inject_msr(positive, negative, vae, latent_image, noise_mask, msr, model, latent_downscale_factor, msr_strength)
|
||||||
|
|
||||||
images = guide_data.get("images", [])
|
# Parse timeline JSON to see if retake mode is active in UI
|
||||||
insert_frames = guide_data.get("insert_frames", [])
|
import json
|
||||||
strengths = guide_data.get("strengths", [])
|
timeline_data_str = guide_data.get("timeline_data", "{}") if guide_data else "{}"
|
||||||
|
try:
|
||||||
|
tdata = json.loads(timeline_data_str)
|
||||||
|
except Exception:
|
||||||
|
tdata = {}
|
||||||
|
|
||||||
|
is_retake_active = bool(retake_mode) or tdata.get("retakeMode", False)
|
||||||
|
is_empty_latent = (latent_image.abs().max().item() < 1e-5)
|
||||||
|
|
||||||
for idx, img_tensor in enumerate(images):
|
# Director image guides and motion video segments
|
||||||
f_idx = insert_frames[idx] if idx < len(insert_frames) else 0
|
images = guide_data.get("images", []) if guide_data else []
|
||||||
strength = strengths[idx] if idx < len(strengths) else 1.0
|
insert_frames = guide_data.get("insert_frames", []) if guide_data else []
|
||||||
|
strengths = guide_data.get("strengths", []) if guide_data else []
|
||||||
|
|
||||||
|
director_fps = float((motion_guide_data or {}).get("frame_rate", guide_data.get("frame_rate", 24) if guide_data else 24))
|
||||||
|
segments = (motion_guide_data or {}).get("segments", [])
|
||||||
|
|
||||||
|
is_lora_active = (model is not None) and (ic_lora_name != "None")
|
||||||
|
time_scale_factor = scale_factors[0]
|
||||||
|
ltxv_length = (latent_length - 1) * time_scale_factor + 1
|
||||||
|
|
||||||
image_1, t = cls.encode(vae, latent_width, latent_height, img_tensor, scale_factors)
|
# -----------------------------------------------------------------------
|
||||||
frame_idx, latent_idx = cls.get_latent_index(positive, latent_length, len(image_1), f_idx, scale_factors)
|
# Retake Mode Branch:
|
||||||
|
# Load single base video, encode continuously, apply temporal mask.
|
||||||
|
# -----------------------------------------------------------------------
|
||||||
|
if is_retake_active:
|
||||||
|
print(f"[LTXDirectorGuide] Retake Mode active. Preserving base latent, masking selected regions. is_empty_latent: {is_empty_latent}")
|
||||||
|
target_width = latent_width * 32
|
||||||
|
target_height = latent_height * 32
|
||||||
|
|
||||||
assert latent_idx + t.shape[2] <= latent_length, (
|
# Calculate retake region latent indices first so we know what to copy/paste
|
||||||
f"Guide image {idx + 1}: conditioning frames exceed the length of the latent sequence."
|
retake_start = int(tdata.get("retakeStart", 0))
|
||||||
)
|
retake_len = int(tdata.get("retakeLength", 0))
|
||||||
|
retake_strength = float(tdata.get("retakeStrength", 1.0))
|
||||||
|
|
||||||
positive, negative, latent_image, noise_mask = cls.append_keyframe(
|
start_frame = int(guide_data.get("start_frame", 0))
|
||||||
positive, negative, frame_idx, latent_image, noise_mask, t, strength, scale_factors,
|
relative_start = max(0, retake_start - start_frame)
|
||||||
)
|
|
||||||
|
l_start = relative_start // time_scale_factor
|
||||||
|
l_end = int(math.ceil((relative_start + retake_len) / time_scale_factor))
|
||||||
|
|
||||||
|
l_start = min(l_start, latent_length)
|
||||||
|
l_end = min(l_end, latent_length)
|
||||||
|
|
||||||
return io.NodeOutput(positive, negative, {"samples": latent_image, "noise_mask": noise_mask})
|
# Stage 2 optimization: If the retake region covers the entire generation area,
|
||||||
|
# there are no preserved regions to copy over in Stage 2. We can bypass video loading/encoding.
|
||||||
|
need_base_video = True
|
||||||
|
if not is_empty_latent and l_start == 0 and l_end >= latent_length:
|
||||||
|
need_base_video = False
|
||||||
|
print("[LTXDirectorGuide] Stage 2: Retake region covers the entire generated range. Skipping base video loading and VAE encoding.")
|
||||||
|
|
||||||
|
# 1. Try to load and encode base video from timeline data
|
||||||
|
retake_vid_info = tdata.get("retakeVideo") or {}
|
||||||
|
video_file = retake_vid_info.get("imageFile", "") if isinstance(retake_vid_info, dict) else ""
|
||||||
|
|
||||||
|
# Fallback to first segment only if it exists and we have no retake video info at all (old workflows)
|
||||||
|
if not video_file and not retake_vid_info and len(segments) > 0:
|
||||||
|
video_file = segments[0].get("videoFile", "")
|
||||||
|
|
||||||
|
if need_base_video:
|
||||||
|
if not video_file:
|
||||||
|
if retake_vid_info and not retake_vid_info.get("imageFile"):
|
||||||
|
raise ValueError(
|
||||||
|
"Retake Mode is active, but the base video file upload is still in progress (or failed). "
|
||||||
|
"Please wait for the 'Uploading base video...' overlay on the timeline to disappear before queuing the prompt."
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
raise ValueError(
|
||||||
|
"Retake Mode is active, but no base video has been selected on the timeline. "
|
||||||
|
"Please drag and drop or upload a base video on the timeline first."
|
||||||
|
)
|
||||||
|
|
||||||
|
if video_file and need_base_video:
|
||||||
|
try:
|
||||||
|
print(f"[LTXDirectorGuide] Loading and encoding base video file: {video_file} starting at frame {start_frame} for length {ltxv_length} at resolution {target_width}x{target_height}")
|
||||||
|
video_frames = _load_motion_video_frames(
|
||||||
|
video_file, trim_start_frames=start_frame, length_frames=ltxv_length, director_fps=director_fps, resample_mode="nearest"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Retake base video must match the exact target latent shape.
|
||||||
|
# "maintain aspect ratio" doesn't pad, which causes shape mismatch in VAE encode.
|
||||||
|
# So we fallback "maintain aspect ratio" to "pad" for retake base video.
|
||||||
|
retake_resize_method = active_resize_method
|
||||||
|
if retake_resize_method == "maintain aspect ratio":
|
||||||
|
retake_resize_method = "pad"
|
||||||
|
|
||||||
|
pixels = _resize_image(video_frames, target_width, target_height, retake_resize_method, divisible_by=1)
|
||||||
|
|
||||||
|
num_clip_frames = pixels.shape[0]
|
||||||
|
num_frames_to_keep = ((num_clip_frames - 1) // time_scale_factor) * time_scale_factor + 1
|
||||||
|
encode_src = pixels[:num_frames_to_keep, :, :, :3]
|
||||||
|
|
||||||
|
if use_tiled_encode:
|
||||||
|
base_latent = vae.encode_tiled(encode_src, tile_x=tile_size, tile_y=tile_size, overlap=tile_overlap)
|
||||||
|
else:
|
||||||
|
base_latent = vae.encode(encode_src)
|
||||||
|
|
||||||
|
base_latent = base_latent.to(device=latent_image.device, dtype=latent_image.dtype)
|
||||||
|
|
||||||
|
# Copy to latent_image
|
||||||
|
paste_len = min(base_latent.shape[2], latent_length)
|
||||||
|
if is_empty_latent:
|
||||||
|
# Stage 1: Overwrite entire latent with base video VAE encode
|
||||||
|
latent_image[:, :, :paste_len] = base_latent[:, :, :paste_len]
|
||||||
|
else:
|
||||||
|
# Stage 2: Overwrite only preserved regions (before l_start and after l_end)
|
||||||
|
# leaving the generated retake region from Stage 1 untouched
|
||||||
|
print(f"[LTXDirectorGuide] Stage 2: Copying high-resolution base latent for preserved regions (0-{l_start} and {l_end}-{paste_len})")
|
||||||
|
if l_start > 0:
|
||||||
|
latent_image[:, :, :l_start] = base_latent[:, :, :l_start]
|
||||||
|
if l_end < paste_len:
|
||||||
|
latent_image[:, :, l_end:paste_len] = base_latent[:, :, l_end:paste_len]
|
||||||
|
except Exception as e:
|
||||||
|
print(f"[LTXDirectorGuide] Failed to load/encode base video: {e}. Falling back to input latent.")
|
||||||
|
|
||||||
|
# 2. Build the temporal noise mask (0.0 = frozen, 1.0 = regenerate)
|
||||||
|
noise_mask = torch.zeros_like(noise_mask) # Initialize fully frozen
|
||||||
|
|
||||||
|
l_start = min(l_start, latent_length)
|
||||||
|
l_end = min(l_end, latent_length)
|
||||||
|
|
||||||
|
if l_end > l_start:
|
||||||
|
noise_mask[:, :, l_start:l_end] = retake_strength
|
||||||
|
|
||||||
|
print(f"[LTXDirectorGuide] noise_mask slice: {noise_mask[0, 0, :, 0, 0].tolist()}")
|
||||||
|
|
||||||
|
# In retake mode, skip normal mode processing entirely and return immediately!
|
||||||
|
exact_crop_frames = max(0, int(latent_image.shape[2]) - initial_latent_length)
|
||||||
|
positive = node_helpers.conditioning_set_values(positive, {"nghtdrp_guide_crop_latent_frames": exact_crop_frames})
|
||||||
|
negative = node_helpers.conditioning_set_values(negative, {"nghtdrp_guide_crop_latent_frames": exact_crop_frames})
|
||||||
|
return (positive, negative, {"samples": latent_image, "noise_mask": noise_mask}, model, float(latent_downscale_factor))
|
||||||
|
|
||||||
|
# -----------------------------------------------------------------------
|
||||||
|
# Standard Timeline Keyframe Guidance:
|
||||||
|
# Appends image segments as keyframes (and motion segments if present)
|
||||||
|
# to the latent stream using standard LTX-Video cross-attention conditioning.
|
||||||
|
# Registers guide attention entries if IC-LoRA is active.
|
||||||
|
# -----------------------------------------------------------------------
|
||||||
|
if len(images) > 0 or len(segments) > 0:
|
||||||
|
print(f"[LTXDirectorGuide] Using Appended Keyframe Guidance. is_lora_active: {is_lora_active}")
|
||||||
|
|
||||||
|
# A. Process Image Guides
|
||||||
|
for idx, img_tensor in enumerate(images):
|
||||||
|
f_idx = insert_frames[idx] if idx < len(insert_frames) else 0
|
||||||
|
strength = float(strengths[idx] if idx < len(strengths) else 1.0)
|
||||||
|
if strength <= 0.0:
|
||||||
|
continue
|
||||||
|
|
||||||
|
B_img, H_img, W_img, C_img = img_tensor.shape
|
||||||
|
target_pix_w = int(latent_width * 32)
|
||||||
|
target_pix_h = int(latent_height * 32)
|
||||||
|
if target_pix_w != W_img or target_pix_h != H_img:
|
||||||
|
img_nchw = img_tensor.permute(0, 3, 1, 2)
|
||||||
|
img_resized = comfy.utils.common_upscale(img_nchw, target_pix_w, target_pix_h, upscale_method, "disabled")
|
||||||
|
img_tensor = img_resized.permute(0, 2, 3, 1)
|
||||||
|
|
||||||
|
image_pixels, guide_latent = nodes_lt.LTXVAddGuide.encode(vae, latent_width, latent_height, img_tensor, scale_factors)
|
||||||
|
frame_idx, latent_idx = nodes_lt.LTXVAddGuide.get_latent_index(positive, latent_length, len(image_pixels), int(f_idx), scale_factors)
|
||||||
|
|
||||||
|
if latent_idx >= latent_length:
|
||||||
|
continue
|
||||||
|
|
||||||
|
max_frames = latent_length - latent_idx
|
||||||
|
if guide_latent.shape[2] > max_frames:
|
||||||
|
guide_latent = guide_latent[:, :, :max_frames]
|
||||||
|
|
||||||
|
tokens_added = guide_latent.shape[2] * guide_latent.shape[3] * guide_latent.shape[4]
|
||||||
|
guide_orig_shape = list(guide_latent.shape[2:])
|
||||||
|
|
||||||
|
positive, negative, latent_image, noise_mask = nodes_lt.LTXVAddGuide.append_keyframe(
|
||||||
|
positive, negative, frame_idx, latent_image, noise_mask, guide_latent, strength, scale_factors
|
||||||
|
)
|
||||||
|
if is_lora_active:
|
||||||
|
positive = _append_guide_attention_entry(positive, tokens_added, guide_orig_shape, attention_strength=image_attention_strength)
|
||||||
|
negative = _append_guide_attention_entry(negative, tokens_added, guide_orig_shape, attention_strength=image_attention_strength)
|
||||||
|
|
||||||
|
# B. Process Motion Video Segments
|
||||||
|
for seg in segments:
|
||||||
|
try:
|
||||||
|
video_file = seg.get("videoFile")
|
||||||
|
if not video_file:
|
||||||
|
continue
|
||||||
|
|
||||||
|
start_frame = int(seg.get("start", 0))
|
||||||
|
length_frames = int(seg.get("length", 1))
|
||||||
|
trim_start = int(seg.get("trimStart", 0))
|
||||||
|
video_strength = float(seg.get("videoStrength", 1.0))
|
||||||
|
video_attention_strength = float(seg.get("videoAttentionStrength", 0.65))
|
||||||
|
|
||||||
|
if length_frames <= 0 or video_strength <= 0.0:
|
||||||
|
continue
|
||||||
|
|
||||||
|
start_frame_aligned = start_frame
|
||||||
|
video_frames = _load_motion_video_frames(video_file, trim_start, length_frames, director_fps, seg.get("resampleMode", "nearest"))
|
||||||
|
|
||||||
|
num_frames_to_keep = ((video_frames.shape[0] - 1) // time_scale_factor) * time_scale_factor + 1
|
||||||
|
video_frames = video_frames[:num_frames_to_keep]
|
||||||
|
causal_fix = int(start_frame_aligned) == 0 or num_frames_to_keep == 1
|
||||||
|
encode_frames = video_frames if causal_fix else torch.cat([video_frames[:1], video_frames], dim=0)
|
||||||
|
|
||||||
|
_, guide_latent = _encode_video_iclora_guide(vae, latent_width, latent_height, encode_frames, scale_factors, latent_downscale_factor, crop, use_tiled_encode, tile_size, tile_overlap, resize_method=active_resize_method)
|
||||||
|
|
||||||
|
if not causal_fix:
|
||||||
|
guide_latent = guide_latent[:, :, 1:, :, :]
|
||||||
|
|
||||||
|
frame_idx = start_frame_aligned
|
||||||
|
latent_idx = (frame_idx + time_scale_factor - 1) // time_scale_factor if frame_idx > 0 else 0
|
||||||
|
|
||||||
|
if latent_idx >= latent_length:
|
||||||
|
continue
|
||||||
|
|
||||||
|
if start_frame > 0 and guide_latent.shape[2] > 1:
|
||||||
|
guide_latent = guide_latent[:, :, 1:, :, :]
|
||||||
|
frame_idx += time_scale_factor
|
||||||
|
latent_idx += 1
|
||||||
|
if latent_idx >= latent_length:
|
||||||
|
continue
|
||||||
|
|
||||||
|
max_frames = latent_length - latent_idx
|
||||||
|
if guide_latent.shape[2] > max_frames:
|
||||||
|
guide_latent = guide_latent[:, :, :max_frames]
|
||||||
|
|
||||||
|
guide_orig_shape = list(guide_latent.shape[2:])
|
||||||
|
|
||||||
|
B_g, C_g, F_g, H_g, W_g = guide_latent.shape
|
||||||
|
guide_mask = torch.ones((B_g, 1, F_g, H_g, W_g), device=guide_latent.device, dtype=guide_latent.dtype)
|
||||||
|
|
||||||
|
if start_frame > 0:
|
||||||
|
ramp_steps = [0.25, 0.65]
|
||||||
|
for i, s in enumerate(ramp_steps):
|
||||||
|
if i < F_g:
|
||||||
|
guide_mask_val = 1.0 + video_strength * (1.0 - s)
|
||||||
|
guide_mask[:, :, i, :, :] = guide_mask_val
|
||||||
|
|
||||||
|
ldf = int(max(1, round(float(latent_downscale_factor))))
|
||||||
|
if ldf > 1:
|
||||||
|
dilated = _dilate_latent({"samples": guide_latent, "noise_mask": guide_mask}, horizontal_scale=ldf, vertical_scale=ldf)
|
||||||
|
guide_mask = dilated["noise_mask"]
|
||||||
|
guide_latent = dilated["samples"]
|
||||||
|
|
||||||
|
tokens_added = guide_latent.shape[2] * guide_latent.shape[3] * guide_latent.shape[4]
|
||||||
|
positive, negative, latent_image, noise_mask = nodes_lt.LTXVAddGuide.append_keyframe(
|
||||||
|
positive, negative, frame_idx, latent_image, noise_mask, guide_latent, video_strength, scale_factors, guide_mask=guide_mask, latent_downscale_factor=float(latent_downscale_factor), causal_fix=causal_fix
|
||||||
|
)
|
||||||
|
if is_lora_active:
|
||||||
|
positive = _append_guide_attention_entry(positive, tokens_added, guide_orig_shape, attention_strength=video_attention_strength)
|
||||||
|
negative = _append_guide_attention_entry(negative, tokens_added, guide_orig_shape, attention_strength=video_attention_strength)
|
||||||
|
except Exception as e:
|
||||||
|
raise RuntimeError(f"LTX Director Guide motion segment failed for {seg}: {e}") from e
|
||||||
|
|
||||||
|
else:
|
||||||
|
print("[LTXDirectorGuide] No timeline guides present. Passing through.")
|
||||||
|
|
||||||
|
exact_crop_frames = max(0, int(latent_image.shape[2]) - initial_latent_length)
|
||||||
|
positive = node_helpers.conditioning_set_values(positive, {"nghtdrp_guide_crop_latent_frames": exact_crop_frames})
|
||||||
|
negative = node_helpers.conditioning_set_values(negative, {"nghtdrp_guide_crop_latent_frames": exact_crop_frames})
|
||||||
|
|
||||||
|
return (positive, negative, {"samples": latent_image, "noise_mask": noise_mask}, model, float(latent_downscale_factor))
|
||||||
|
|
||||||
@classmethod
|
@classmethod
|
||||||
def _inject_msr(cls, positive, negative, vae, latent_image, noise_mask, msr, msr_strength=0.0):
|
def _inject_msr(cls, positive, negative, vae, latent_image, noise_mask, msr, model, latent_downscale_factor, msr_strength=0.0):
|
||||||
"""Inject the Licon MSR references at this node's current resolution."""
|
"""Inject the Licon MSR references at this node's current resolution.
|
||||||
|
|
||||||
|
Returns this node's 5-tuple (positive, negative, latent, model, latent_downscale_factor).
|
||||||
|
"""
|
||||||
from nodes import NODE_CLASS_MAPPINGS
|
from nodes import NODE_CLASS_MAPPINGS
|
||||||
from .ltx_director import _execute_comfy_node, _unpack
|
|
||||||
|
|
||||||
IC = NODE_CLASS_MAPPINGS.get("LTXAddVideoICLoRAGuide")
|
IC = NODE_CLASS_MAPPINGS.get("LTXAddVideoICLoRAGuide")
|
||||||
AG = NODE_CLASS_MAPPINGS.get("LTXVAddGuide")
|
AG = NODE_CLASS_MAPPINGS.get("LTXVAddGuide")
|
||||||
@@ -118,6 +649,8 @@ class LTXDirectorGuide(LTXVAddGuide):
|
|||||||
strength = clamp01(msr_strength) if float(msr_strength) > 0.0 else clamp01(msr["strength"])
|
strength = clamp01(msr_strength) if float(msr_strength) > 0.0 else clamp01(msr["strength"])
|
||||||
downscale = float(msr["downscale"])
|
downscale = float(msr["downscale"])
|
||||||
|
|
||||||
|
initial_latent_length = int(latent_image.shape[2])
|
||||||
|
|
||||||
# Pad the generated region so (pad + keyframes) == prefix_latents, so the downstream crop
|
# Pad the generated region so (pad + keyframes) == prefix_latents, so the downstream crop
|
||||||
# (which trims the slideshow's temporal footprint) lands on the true clean length.
|
# (which trims the slideshow's temporal footprint) lands on the true clean length.
|
||||||
pad_latents = max(0, prefix_latents - len(keyframes))
|
pad_latents = max(0, prefix_latents - len(keyframes))
|
||||||
@@ -140,15 +673,95 @@ class LTXDirectorGuide(LTXVAddGuide):
|
|||||||
IC, positive=positive, negative=negative, vae=vae, latent=latent, image=slideshow,
|
IC, positive=positive, negative=negative, vae=vae, latent=latent, image=slideshow,
|
||||||
frame_idx=0, strength=strength, latent_downscale_factor=downscale,
|
frame_idx=0, strength=strength, latent_downscale_factor=downscale,
|
||||||
crop="center", use_tiled_encode=False, tile_size=256, tile_overlap=64,
|
crop="center", use_tiled_encode=False, tile_size=256, tile_overlap=64,
|
||||||
))
|
))[:3]
|
||||||
|
|
||||||
positive, negative = cp, cn
|
positive, negative = cp, cn
|
||||||
|
|
||||||
# LATENT PATH: one frozen keyframe per reference at frame 0 -> keep latent, drop cond.
|
# LATENT PATH: one frozen keyframe per reference at frame 0 -> keep latent, drop cond.
|
||||||
for kf in keyframes:
|
for kf in keyframes:
|
||||||
_p, _n, latent = _unpack(_execute_comfy_node(
|
unpacked = _unpack(_execute_comfy_node(
|
||||||
AG, positive=positive, negative=negative, vae=vae, latent=latent,
|
AG, positive=positive, negative=negative, vae=vae, latent=latent,
|
||||||
image=kf, frame_idx=0, strength=strength,
|
image=kf, frame_idx=0, strength=strength,
|
||||||
))
|
))
|
||||||
|
latent = unpacked[2]
|
||||||
|
|
||||||
return io.NodeOutput(positive, negative, latent)
|
# NOTE: MSR is a PREFIX. We deliberately do NOT set nghtdrp_guide_crop_latent_frames
|
||||||
|
# (the new LTXDirectorCropGuides trims from the END, which would cut clean frames).
|
||||||
|
# The IC/AG nodes write position-aware keyframe metadata into the conditioning, so the
|
||||||
|
# MSR branch of the workflow should crop with the STOCK position-aware LTXVCropGuides
|
||||||
|
# node (exactly as the old MSR pipeline did).
|
||||||
|
return (positive, negative, latent, model, float(latent_downscale_factor))
|
||||||
|
|
||||||
|
|
||||||
|
def _conditioning_get_any_value(conditioning, key, default=None):
|
||||||
|
for item in conditioning:
|
||||||
|
meta = item[1]
|
||||||
|
if key in meta and meta.get(key) is not None:
|
||||||
|
return meta.get(key)
|
||||||
|
return default
|
||||||
|
|
||||||
|
def _get_exact_crop_count_from_conditioning(conditioning):
|
||||||
|
value = _conditioning_get_any_value(conditioning, "nghtdrp_guide_crop_latent_frames", None)
|
||||||
|
if value is not None:
|
||||||
|
try:
|
||||||
|
return max(0, int(value))
|
||||||
|
except Exception:
|
||||||
|
return 0
|
||||||
|
keyframe_idxs = _conditioning_get_any_value(conditioning, "keyframe_idxs", None)
|
||||||
|
if keyframe_idxs is None:
|
||||||
|
return 0
|
||||||
|
try:
|
||||||
|
return int(torch.unique(keyframe_idxs[:, 0, :, 0]).shape[0])
|
||||||
|
except Exception:
|
||||||
|
return 0
|
||||||
|
|
||||||
|
def _get_noise_mask_for_crop(latent):
|
||||||
|
latent_image = latent["samples"]
|
||||||
|
noise_mask = latent.get("noise_mask", None)
|
||||||
|
if noise_mask is None:
|
||||||
|
batch, _, frames, _, _ = latent_image.shape
|
||||||
|
return torch.ones((batch, 1, frames, 1, 1), dtype=torch.float32, device=latent_image.device)
|
||||||
|
return noise_mask.clone()
|
||||||
|
|
||||||
|
|
||||||
|
class LTXDirectorCropGuides:
|
||||||
|
@classmethod
|
||||||
|
def INPUT_TYPES(cls):
|
||||||
|
return {
|
||||||
|
"required": {
|
||||||
|
"positive": ("CONDITIONING",),
|
||||||
|
"negative": ("CONDITIONING",),
|
||||||
|
"latent": ("LATENT",),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
RETURN_TYPES = ("CONDITIONING", "CONDITIONING", "LATENT")
|
||||||
|
RETURN_NAMES = ("positive", "negative", "latent")
|
||||||
|
FUNCTION = "execute"
|
||||||
|
CATEGORY = "WhatDreamsCost CS"
|
||||||
|
|
||||||
|
def execute(self, positive, negative, latent):
|
||||||
|
latent_image = latent["samples"].clone()
|
||||||
|
noise_mask = _get_noise_mask_for_crop(latent)
|
||||||
|
|
||||||
|
crop_frames = _get_exact_crop_count_from_conditioning(positive)
|
||||||
|
if crop_frames <= 0:
|
||||||
|
return (positive, negative, {"samples": latent_image, "noise_mask": noise_mask})
|
||||||
|
|
||||||
|
crop_frames = min(crop_frames, max(0, latent_image.shape[2] - 1))
|
||||||
|
if crop_frames > 0:
|
||||||
|
latent_image = latent_image[:, :, :-crop_frames]
|
||||||
|
noise_mask = noise_mask[:, :, :-crop_frames]
|
||||||
|
|
||||||
|
clear_values = {
|
||||||
|
"keyframe_idxs": None,
|
||||||
|
"guide_attention_entries": None,
|
||||||
|
"nghtdrp_guide_crop_latent_frames": None,
|
||||||
|
}
|
||||||
|
positive = node_helpers.conditioning_set_values(positive, clear_values)
|
||||||
|
negative = node_helpers.conditioning_set_values(negative, clear_values)
|
||||||
|
return (positive, negative, {"samples": latent_image, "noise_mask": noise_mask})
|
||||||
|
|
||||||
|
NODE_CLASS_MAPPINGS = {
|
||||||
|
"LTXDirectorGuideCS": LTXDirectorGuide,
|
||||||
|
"LTXDirectorCropGuidesCS": LTXDirectorCropGuides,
|
||||||
|
}
|
||||||
117
ltx_keyframer.py
Normal file
117
ltx_keyframer.py
Normal file
@@ -0,0 +1,117 @@
|
|||||||
|
import torch
|
||||||
|
import comfy.utils
|
||||||
|
from comfy_api.latest import io
|
||||||
|
|
||||||
|
class LTXKeyframer(io.ComfyNode):
|
||||||
|
@classmethod
|
||||||
|
def define_schema(cls):
|
||||||
|
inputs = [
|
||||||
|
io.Vae.Input("vae", tooltip="Video VAE used to encode the images"),
|
||||||
|
io.Latent.Input("latent", tooltip="Video latent to insert images into"),
|
||||||
|
io.Image.Input("multi_input", tooltip="Batched images from MultiImageLoader"),
|
||||||
|
]
|
||||||
|
|
||||||
|
inputs.append(io.Int.Input("num_images", default=1, min=0, max=50, step=1, display_name="images_loaded", tooltip="Select how many index/strength widgets to configure."))
|
||||||
|
|
||||||
|
for i in range(1, 51): # 1 to 50 images
|
||||||
|
inputs.extend([
|
||||||
|
io.Int.Input(
|
||||||
|
f"insert_frame_{i}",
|
||||||
|
default=0,
|
||||||
|
min=-9999,
|
||||||
|
max=9999,
|
||||||
|
step=1,
|
||||||
|
tooltip=f"Frame insert_frame for image {i} (in pixel space).",
|
||||||
|
optional=True,
|
||||||
|
),
|
||||||
|
io.Float.Input(
|
||||||
|
f"strength_{i}",
|
||||||
|
default=1.0,
|
||||||
|
min=0.0,
|
||||||
|
max=1.0,
|
||||||
|
step=0.01,
|
||||||
|
tooltip=f"Strength for image {i}.",
|
||||||
|
optional=True,
|
||||||
|
),
|
||||||
|
])
|
||||||
|
|
||||||
|
return io.Schema(
|
||||||
|
node_id="LTXKeyframer",
|
||||||
|
display_name="LTX Keyframer",
|
||||||
|
category="WhatDreamsCost",
|
||||||
|
description="Replaces video latent frames with the encoded input images. Number of widgets is dynamically configured.",
|
||||||
|
inputs=inputs,
|
||||||
|
outputs=[
|
||||||
|
io.Latent.Output(display_name="latent", tooltip="The video latent with the images inserted and latent noise mask updated."),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
|
||||||
|
@classmethod
|
||||||
|
def execute(cls, vae, latent, multi_input, num_images, **kwargs) -> io.NodeOutput:
|
||||||
|
|
||||||
|
samples = latent["samples"].clone()
|
||||||
|
scale_factors = vae.downscale_index_formula
|
||||||
|
_, height_scale_factor, width_scale_factor = scale_factors
|
||||||
|
|
||||||
|
batch, _, latent_frames, latent_height, latent_width = samples.shape
|
||||||
|
width = latent_width * width_scale_factor
|
||||||
|
height = latent_height * height_scale_factor
|
||||||
|
|
||||||
|
# Get existing noise mask if present, otherwise create new one
|
||||||
|
if "noise_mask" in latent:
|
||||||
|
conditioning_latent_frames_mask = latent["noise_mask"].clone()
|
||||||
|
else:
|
||||||
|
conditioning_latent_frames_mask = torch.ones(
|
||||||
|
(batch, 1, latent_frames, 1, 1),
|
||||||
|
dtype=torch.float32,
|
||||||
|
device=samples.device,
|
||||||
|
)
|
||||||
|
|
||||||
|
batch_size = multi_input.shape[0] if multi_input is not None else 0
|
||||||
|
|
||||||
|
# We process inputs up to num_images, extracting values from kwargs
|
||||||
|
for i in range(1, num_images + 1):
|
||||||
|
# Skip if this image index exceeds the batch
|
||||||
|
if i > batch_size:
|
||||||
|
continue
|
||||||
|
|
||||||
|
image = multi_input[i-1:i] # Extract the single image frame from the batch
|
||||||
|
if image is None:
|
||||||
|
continue
|
||||||
|
|
||||||
|
insert_frame = kwargs.get(f"insert_frame_{i}")
|
||||||
|
if insert_frame is None:
|
||||||
|
continue
|
||||||
|
strength = kwargs.get(f"strength_{i}", 1.0)
|
||||||
|
|
||||||
|
if image.shape[1] != height or image.shape[2] != width:
|
||||||
|
pixels = comfy.utils.common_upscale(image.movedim(-1, 1), width, height, "bilinear", "center").movedim(1, -1)
|
||||||
|
else:
|
||||||
|
pixels = image
|
||||||
|
encode_pixels = pixels[:, :, :, :3]
|
||||||
|
t = vae.encode(encode_pixels)
|
||||||
|
|
||||||
|
# Convert pixel frame insert_frame to latent insert_frame
|
||||||
|
time_scale_factor = scale_factors[0]
|
||||||
|
|
||||||
|
# Handle negative indexing in pixel space
|
||||||
|
pixel_frame_count = (latent_frames - 1) * time_scale_factor + 1
|
||||||
|
if insert_frame < 0:
|
||||||
|
insert_frame = pixel_frame_count + insert_frame
|
||||||
|
|
||||||
|
# Convert to latent insert_frame
|
||||||
|
latent_idx = insert_frame // time_scale_factor
|
||||||
|
|
||||||
|
# Clamp to valid range
|
||||||
|
latent_idx = max(0, min(latent_idx, latent_frames - 1))
|
||||||
|
|
||||||
|
# Calculate end insert_frame, ensuring we don't exceed latent_frames
|
||||||
|
end_index = min(latent_idx + t.shape[2], latent_frames)
|
||||||
|
|
||||||
|
# Replace samples at the specified insert_frame range
|
||||||
|
samples[:, :, latent_idx:end_index] = t[:, :, :end_index - latent_idx]
|
||||||
|
|
||||||
|
# Update mask at the specified insert_frame range
|
||||||
|
conditioning_latent_frames_mask[:, :, latent_idx:end_index] = 1.0 - strength
|
||||||
|
|
||||||
|
return io.NodeOutput({"samples": samples, "noise_mask": conditioning_latent_frames_mask})
|
||||||
@@ -1,4 +1,3 @@
|
|||||||
import json
|
|
||||||
from comfy_extras.nodes_lt import get_noise_mask, LTXVAddGuide
|
from comfy_extras.nodes_lt import get_noise_mask, LTXVAddGuide
|
||||||
import torch
|
import torch
|
||||||
import comfy.utils
|
import comfy.utils
|
||||||
@@ -56,14 +55,12 @@ class LTXSequencer(LTXVAddGuide):
|
|||||||
node_id="LTXSequencer",
|
node_id="LTXSequencer",
|
||||||
display_name="LTX Sequencer",
|
display_name="LTX Sequencer",
|
||||||
category="WhatDreamsCost",
|
category="WhatDreamsCost",
|
||||||
description="Add multiple guide images at specified frame indices or seconds. Auto-syncs to Prompt Relay invisibly via the positive wire.",
|
description="Add multiple guide images at specified frame indices or seconds with strengths. Number of widgets is dynamically configured.",
|
||||||
inputs=inputs,
|
inputs=inputs,
|
||||||
outputs=[
|
outputs=[
|
||||||
io.Conditioning.Output(display_name="positive"),
|
io.Conditioning.Output(display_name="positive"),
|
||||||
io.Conditioning.Output(display_name="negative"),
|
io.Conditioning.Output(display_name="negative"),
|
||||||
io.Latent.Output(display_name="latent", tooltip="Video latent with added guides"),
|
io.Latent.Output(display_name="latent", tooltip="Video latent with added guides"),
|
||||||
# ---> ADDED SYNC LOG OUTPUT <---
|
|
||||||
io.String.Output(display_name="sync_log", tooltip="Outputs a readable text block showing the actual times used for each image"),
|
|
||||||
],
|
],
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -92,63 +89,29 @@ class LTXSequencer(LTXVAddGuide):
|
|||||||
insert_mode = kwargs.get("insert_mode", "frames")
|
insert_mode = kwargs.get("insert_mode", "frames")
|
||||||
frame_rate = kwargs.get("frame_rate", 24)
|
frame_rate = kwargs.get("frame_rate", 24)
|
||||||
|
|
||||||
# ---> EXTRACT GHOST SYNC DATA <---
|
|
||||||
timeline_data = None
|
|
||||||
if positive is not None and len(positive) > 0:
|
|
||||||
if "prompt_relay_timeline" in positive[0][1]:
|
|
||||||
try:
|
|
||||||
timeline_data = json.loads(positive[0][1]["prompt_relay_timeline"])
|
|
||||||
except json.JSONDecodeError:
|
|
||||||
pass
|
|
||||||
|
|
||||||
# Prepare the log text
|
|
||||||
sync_log_lines = []
|
|
||||||
if timeline_data:
|
|
||||||
sync_log_lines.append("=== TIMELINE SYNC ENABLED ===")
|
|
||||||
sync_log_lines.append(f"Invisibly Synced via Positive Wire! Using {insert_mode.upper()}:")
|
|
||||||
else:
|
|
||||||
sync_log_lines.append("=== MANUAL MODE ===")
|
|
||||||
sync_log_lines.append(f"No timeline detected, or Sync turned OFF. Using manual UI inputs ({insert_mode}):")
|
|
||||||
|
|
||||||
# Process inputs up to num_images, extracting dynamic frame/strength values from kwargs
|
# Process inputs up to num_images, extracting dynamic frame/strength values from kwargs
|
||||||
for i in range(1, num_images + 1):
|
for i in range(1, num_images + 1):
|
||||||
# Skip if this image index exceeds the batch
|
# Skip if this image index exceeds the batch
|
||||||
if i > batch_size:
|
if i > batch_size:
|
||||||
sync_log_lines.append(f"Image #{i}: Skipped (No image loaded in batch)")
|
|
||||||
continue
|
continue
|
||||||
|
|
||||||
img = multi_input[i-1:i] # Extract the single image frame from the batch
|
img = multi_input[i-1:i] # Extract the single image frame from the batch
|
||||||
if img is None:
|
if img is None:
|
||||||
continue
|
continue
|
||||||
|
|
||||||
|
# Calculate the final frame index based on the chosen mode
|
||||||
f_idx = None
|
f_idx = None
|
||||||
strength = kwargs.get(f"strength_{i}", 1.0)
|
if insert_mode == "frames":
|
||||||
|
f_idx = kwargs.get(f"insert_frame_{i}")
|
||||||
# 1. AUTO-SYNC
|
elif insert_mode == "seconds":
|
||||||
if timeline_data and "starts_frames" in timeline_data and (i - 1) < len(timeline_data["starts_frames"]):
|
sec = kwargs.get(f"insert_second_{i}")
|
||||||
display_frame = timeline_data["starts_frames"][i - 1]
|
if sec is not None:
|
||||||
display_sec = timeline_data["starts_seconds"][i - 1]
|
f_idx = int(sec * frame_rate)
|
||||||
|
|
||||||
if insert_mode == "frames":
|
|
||||||
f_idx = display_frame
|
|
||||||
sync_log_lines.append(f"-> Image #{i} (Strength: {strength}): Synced to Segment #{i} start @ {display_frame} frames")
|
|
||||||
elif insert_mode == "seconds":
|
|
||||||
f_idx = int(display_sec * frame_rate)
|
|
||||||
sync_log_lines.append(f"-> Image #{i} (Strength: {strength}): Synced to Segment #{i} start @ {display_sec:.2f} seconds (Frame {f_idx})")
|
|
||||||
|
|
||||||
# 2. MANUAL FALLBACK
|
|
||||||
else:
|
|
||||||
if insert_mode == "frames":
|
|
||||||
f_idx = kwargs.get(f"insert_frame_{i}")
|
|
||||||
sync_log_lines.append(f"-> Image #{i} (Strength: {strength}): Manual input @ {f_idx} frames")
|
|
||||||
elif insert_mode == "seconds":
|
|
||||||
sec = kwargs.get(f"insert_second_{i}")
|
|
||||||
if sec is not None:
|
|
||||||
f_idx = int(sec * frame_rate)
|
|
||||||
sync_log_lines.append(f"-> Image #{i} (Strength: {strength}): Manual input @ {sec:.2f} seconds (Frame {f_idx})")
|
|
||||||
|
|
||||||
if f_idx is None:
|
if f_idx is None:
|
||||||
continue
|
continue
|
||||||
|
|
||||||
|
strength = kwargs.get(f"strength_{i}", 1.0)
|
||||||
|
|
||||||
# Execution logic mirrored from LTXVAddGuideMulti
|
# Execution logic mirrored from LTXVAddGuideMulti
|
||||||
image_1, t = cls.encode(vae, latent_width, latent_height, img, scale_factors)
|
image_1, t = cls.encode(vae, latent_width, latent_height, img, scale_factors)
|
||||||
@@ -167,8 +130,4 @@ class LTXSequencer(LTXVAddGuide):
|
|||||||
scale_factors,
|
scale_factors,
|
||||||
)
|
)
|
||||||
|
|
||||||
sync_log_str = "\n".join(sync_log_lines)
|
return io.NodeOutput(positive, negative, {"samples": latent_image, "noise_mask": noise_mask})
|
||||||
print(f"\n[LTX Sequencer]\n{sync_log_str}\n")
|
|
||||||
|
|
||||||
# Returned the newly formatted sync log string at the end of NodeOutput
|
|
||||||
return io.NodeOutput(positive, negative, {"samples": latent_image, "noise_mask": noise_mask}, sync_log_str)
|
|
||||||
100
patches.py
100
patches.py
@@ -1,7 +1,10 @@
|
|||||||
|
import logging
|
||||||
import types
|
import types
|
||||||
import torch
|
|
||||||
import comfy.ldm.modules.attention
|
import comfy.ldm.modules.attention
|
||||||
|
|
||||||
|
log = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
|
||||||
def _masked_attention(q, k, v, heads, mask, transformer_options={}, **kwargs):
|
def _masked_attention(q, k, v, heads, mask, transformer_options={}, **kwargs):
|
||||||
# Bypass wrap_attn (sage/etc may ignore masks) by calling attention_pytorch directly.
|
# Bypass wrap_attn (sage/etc may ignore masks) by calling attention_pytorch directly.
|
||||||
@@ -18,7 +21,7 @@ def _wan_t2v_forward(self, mask_fn, x, context, transformer_options={}, **kwargs
|
|||||||
k = self.norm_k(self.k(context))
|
k = self.norm_k(self.k(context))
|
||||||
v = self.v(context)
|
v = self.v(context)
|
||||||
|
|
||||||
mask = mask_fn(q, k, transformer_options)
|
mask = mask_fn(q.shape[1], k.shape[1], q.dtype, q.device, transformer_options)
|
||||||
if mask is not None:
|
if mask is not None:
|
||||||
x = _masked_attention(q, k, v, heads=self.num_heads, mask=mask,
|
x = _masked_attention(q, k, v, heads=self.num_heads, mask=mask,
|
||||||
transformer_options=transformer_options)
|
transformer_options=transformer_options)
|
||||||
@@ -44,7 +47,7 @@ def _wan_i2v_forward(self, mask_fn, x, context, context_img_len, transformer_opt
|
|||||||
k = self.norm_k(self.k(context_text))
|
k = self.norm_k(self.k(context_text))
|
||||||
v = self.v(context_text)
|
v = self.v(context_text)
|
||||||
|
|
||||||
mask = mask_fn(q, k, transformer_options)
|
mask = mask_fn(q.shape[1], k.shape[1], q.dtype, q.device, transformer_options)
|
||||||
if mask is not None:
|
if mask is not None:
|
||||||
x = _masked_attention(q, k, v, heads=self.num_heads, mask=mask,
|
x = _masked_attention(q, k, v, heads=self.num_heads, mask=mask,
|
||||||
transformer_options=transformer_options)
|
transformer_options=transformer_options)
|
||||||
@@ -56,47 +59,65 @@ def _wan_i2v_forward(self, mask_fn, x, context, context_img_len, transformer_opt
|
|||||||
return self.o(x + img_x)
|
return self.o(x + img_x)
|
||||||
|
|
||||||
|
|
||||||
def _ltx_forward(self, mask_fn, x, context=None, mask=None, pe=None, k_pe=None, transformer_options={}):
|
def _make_masked_override(prev_override):
|
||||||
from comfy.ldm.lightricks.model import apply_rotary_emb
|
"""transformer_options override that routes mask-bearing attention calls through
|
||||||
|
attention_pytorch (sage/etc. drop arbitrary masks). Chains to a prior override
|
||||||
|
when no mask is present so we don't clobber other backends."""
|
||||||
|
def override(func, *args, **kwargs):
|
||||||
|
if kwargs.get("mask") is not None:
|
||||||
|
return comfy.ldm.modules.attention.attention_pytorch(*args, **kwargs)
|
||||||
|
if prev_override is not None:
|
||||||
|
return prev_override(func, *args, **kwargs)
|
||||||
|
return func(*args, **kwargs)
|
||||||
|
return override
|
||||||
|
|
||||||
is_self_attn = context is None
|
|
||||||
context = x if is_self_attn else context
|
|
||||||
|
|
||||||
q = self.q_norm(self.to_q(x))
|
def debug_log(msg):
|
||||||
k = self.k_norm(self.to_k(context))
|
try:
|
||||||
v = self.to_v(context)
|
import os
|
||||||
|
log_path = os.path.join(os.path.dirname(__file__), "debug_prompt_relay.log")
|
||||||
|
with open(log_path, "a", encoding="utf-8") as f:
|
||||||
|
f.write(msg + "\n")
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
if pe is not None:
|
|
||||||
q = apply_rotary_emb(q, pe)
|
|
||||||
k = apply_rotary_emb(k, pe if k_pe is None else k_pe)
|
|
||||||
|
|
||||||
if not is_self_attn:
|
def _make_ltx_mask_wrapper(underlying, mask_fn, attr):
|
||||||
temporal_mask = mask_fn(q, k, transformer_options)
|
"""Wrap an existing LTX cross-attn forward (the default `CrossAttention.forward`
|
||||||
if temporal_mask is not None:
|
or another node's patch — e.g. KJNodes NAG), injecting PromptRelay's additive
|
||||||
mask = temporal_mask if mask is None else mask + temporal_mask
|
mask via the `mask` kwarg the upstream signature already accepts.
|
||||||
|
|
||||||
if mask is None:
|
`underlying` must already be bound to its module — callable as
|
||||||
out = comfy.ldm.modules.attention.optimized_attention(
|
`underlying(x, context=..., mask=..., ...)`.
|
||||||
q, k, v, self.heads, attn_precision=self.attn_precision,
|
"""
|
||||||
|
def wrapped(_self, x, context=None, mask=None, pe=None, k_pe=None, transformer_options={}):
|
||||||
|
debug_log(f"wrapped called: x.shape={list(x.shape)} context.shape={list(context.shape) if context is not None else None} mask_is_none={mask is None}")
|
||||||
|
if context is not None:
|
||||||
|
opts = {**transformer_options, "promptrelay_attn_type": attr}
|
||||||
|
pr_mask = mask_fn(x.shape[1], context.shape[1], x.dtype, x.device, opts)
|
||||||
|
debug_log(f"mask_fn returned pr_mask_is_none={pr_mask is None}")
|
||||||
|
if pr_mask is not None:
|
||||||
|
debug_log(f"pr_mask info: shape={list(pr_mask.shape)} min={pr_mask.min().item()} max={pr_mask.max().item()} sum={pr_mask.sum().item()}")
|
||||||
|
mask = pr_mask if mask is None else mask + pr_mask
|
||||||
|
|
||||||
|
if mask is not None:
|
||||||
|
prev = transformer_options.get("optimized_attention_override")
|
||||||
|
transformer_options = {
|
||||||
|
**transformer_options,
|
||||||
|
"optimized_attention_override": _make_masked_override(prev),
|
||||||
|
}
|
||||||
|
|
||||||
|
return underlying(
|
||||||
|
x, context=context, mask=mask, pe=pe, k_pe=k_pe,
|
||||||
transformer_options=transformer_options,
|
transformer_options=transformer_options,
|
||||||
)
|
)
|
||||||
else:
|
|
||||||
out = _masked_attention(q, k, v, self.heads, mask=mask,
|
|
||||||
attn_precision=self.attn_precision,
|
|
||||||
transformer_options=transformer_options)
|
|
||||||
|
|
||||||
if self.to_gate_logits is not None:
|
wrapped._promptrelay_wrapper = True
|
||||||
gate_logits = self.to_gate_logits(x)
|
return wrapped
|
||||||
b, t, _ = out.shape
|
|
||||||
out = out.view(b, t, self.heads, self.dim_head)
|
|
||||||
out = out * (2.0 * torch.sigmoid(gate_logits)).unsqueeze(-1)
|
|
||||||
out = out.view(b, t, self.heads * self.dim_head)
|
|
||||||
|
|
||||||
return self.to_out(out)
|
|
||||||
|
|
||||||
|
|
||||||
class _CrossAttnPatch:
|
class _CrossAttnPatch:
|
||||||
"""Descriptor that binds (impl, mask_fn) as a method onto a cross-attn module."""
|
"""Descriptor that binds (impl, mask_fn) as a method onto a Wan cross-attn module."""
|
||||||
|
|
||||||
def __init__(self, impl, mask_fn):
|
def __init__(self, impl, mask_fn):
|
||||||
self.impl = impl
|
self.impl = impl
|
||||||
@@ -135,8 +156,8 @@ def _check_unpatched(model_clone, key):
|
|||||||
if key in getattr(model_clone, "object_patches", {}):
|
if key in getattr(model_clone, "object_patches", {}):
|
||||||
raise RuntimeError(
|
raise RuntimeError(
|
||||||
f"PromptRelay: cross-attention forward at '{key}' is already patched by "
|
f"PromptRelay: cross-attention forward at '{key}' is already patched by "
|
||||||
"another node (e.g. KJNodes NAG). Stacking is not supported — remove the "
|
"another node. Stacking is not supported for this architecture — remove "
|
||||||
"conflicting node."
|
"the conflicting node."
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
@@ -154,14 +175,19 @@ def apply_patches(model_clone, arch, mask_fn):
|
|||||||
return
|
return
|
||||||
|
|
||||||
if arch == "ltx":
|
if arch == "ltx":
|
||||||
|
to = model_clone.model_options["transformer_options"]
|
||||||
|
to["promptrelay_mask_fn"] = mask_fn
|
||||||
|
|
||||||
for idx, block in enumerate(diffusion_model.transformer_blocks):
|
for idx, block in enumerate(diffusion_model.transformer_blocks):
|
||||||
for attr in ("attn2", "audio_attn2"):
|
for attr in ("attn2", "audio_attn2"):
|
||||||
module = getattr(block, attr, None)
|
module = getattr(block, attr, None)
|
||||||
if module is None:
|
if module is None:
|
||||||
continue
|
continue
|
||||||
key = f"diffusion_model.transformer_blocks.{idx}.{attr}.forward"
|
key = f"diffusion_model.transformer_blocks.{idx}.{attr}.forward"
|
||||||
_check_unpatched(model_clone, key)
|
# get_model_object returns the prior patch if present, else the default bound forward.
|
||||||
model_clone.add_object_patch(key, _CrossAttnPatch(_ltx_forward, mask_fn).__get__(module, module.__class__))
|
underlying = model_clone.get_model_object(key)
|
||||||
|
wrapper = _make_ltx_mask_wrapper(underlying, mask_fn, attr)
|
||||||
|
model_clone.add_object_patch(key, types.MethodType(wrapper, module))
|
||||||
return
|
return
|
||||||
|
|
||||||
raise ValueError(f"Unknown model arch: {arch}")
|
raise ValueError(f"Unknown model arch: {arch}")
|
||||||
|
|||||||
@@ -20,7 +20,7 @@ def build_temporal_cost(q_token_idx, Lq, Lk, device, dtype, tokens_per_frame):
|
|||||||
return offset
|
return offset
|
||||||
|
|
||||||
|
|
||||||
def build_temporal_cost_scaled(q_token_idx, Lq, Lk, device, dtype, latent_frames):
|
def build_temporal_cost_scaled(q_token_idx, Lq, Lk, device, dtype, latent_frames, is_audio=False):
|
||||||
"""Penalty matrix for queries that don't map to integer frames (e.g. LTXAV audio tokens)."""
|
"""Penalty matrix for queries that don't map to integer frames (e.g. LTXAV audio tokens)."""
|
||||||
offset = torch.zeros(Lq, Lk, device=device, dtype=dtype)
|
offset = torch.zeros(Lq, Lk, device=device, dtype=dtype)
|
||||||
query_frames = torch.arange(Lq, device=device, dtype=torch.float32) * latent_frames / Lq
|
query_frames = torch.arange(Lq, device=device, dtype=torch.float32) * latent_frames / Lq
|
||||||
@@ -28,54 +28,90 @@ def build_temporal_cost_scaled(q_token_idx, Lq, Lk, device, dtype, latent_frames
|
|||||||
for seg in q_token_idx:
|
for seg in q_token_idx:
|
||||||
local = seg["local_token_idx"].to(device=device)
|
local = seg["local_token_idx"].to(device=device)
|
||||||
d = (query_frames[:, None] - seg["midpoint"]).abs()
|
d = (query_frames[:, None] - seg["midpoint"]).abs()
|
||||||
sigma_a = seg.get("sigma_audio", seg["sigma"])
|
if is_audio:
|
||||||
window_a = seg.get("window_audio", seg["window"])
|
sigma_val = seg.get("sigma_audio", seg["sigma"])
|
||||||
strength_a = seg.get("strength_audio", 1.0)
|
window_val = seg.get("window_audio", seg["window"])
|
||||||
cost = strength_a * (torch.relu(d - window_a) ** 2) / (2 * sigma_a ** 2)
|
strength_val = seg.get("strength_audio", 1.0)
|
||||||
|
else:
|
||||||
|
sigma_val = seg["sigma"]
|
||||||
|
window_val = seg["window"]
|
||||||
|
strength_val = seg.get("strength", 1.0)
|
||||||
|
cost = strength_val * (torch.relu(d - window_val) ** 2) / (2 * sigma_val ** 2)
|
||||||
offset[:, local] = cost.to(offset.dtype)
|
offset[:, local] = cost.to(offset.dtype)
|
||||||
|
|
||||||
return offset
|
return offset
|
||||||
|
|
||||||
|
|
||||||
|
def debug_log(msg):
|
||||||
|
try:
|
||||||
|
import os
|
||||||
|
log_path = os.path.join(os.path.dirname(__file__), "debug_prompt_relay.log")
|
||||||
|
with open(log_path, "a", encoding="utf-8") as f:
|
||||||
|
f.write(msg + "\n")
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
def create_mask_fn(q_token_idx, fallback_tokens_per_frame, latent_frames):
|
def create_mask_fn(q_token_idx, fallback_tokens_per_frame, latent_frames):
|
||||||
"""Closure: mask_fn(q, k, transformer_options) -> additive mask or None."""
|
"""Closure: mask_fn(Lq, Lk, dtype, device, transformer_options) -> additive mask or None.
|
||||||
|
|
||||||
|
Takes shapes/dtype/device instead of tensors so callers can compute the mask
|
||||||
|
without first materializing q/k projections — required so PromptRelay can
|
||||||
|
wrap an existing cross-attn forward (e.g. KJNodes NAG) instead of replacing it.
|
||||||
|
"""
|
||||||
cache = {}
|
cache = {}
|
||||||
max_token_idx = max(int(seg["local_token_idx"].max().item()) for seg in q_token_idx) + 1
|
max_token_idx = max(int(seg["local_token_idx"].max().item()) for seg in q_token_idx) + 1
|
||||||
|
|
||||||
def mask_fn(q, k, transformer_options):
|
def mask_fn(Lq, Lk, dtype, device, transformer_options):
|
||||||
Lq, Lk = q.shape[1], k.shape[1]
|
debug_log(f"mask_fn check: Lq={Lq} Lk={Lk} max_token_idx={max_token_idx} cond_or_uncond={transformer_options.get('cond_or_uncond', [])}")
|
||||||
|
|
||||||
if Lq == Lk:
|
if Lq == Lk:
|
||||||
|
debug_log("mask_fn: Lq == Lk, returning None")
|
||||||
return None
|
return None
|
||||||
|
|
||||||
# Only apply on conditional pass — not unconditional (negative prompt)
|
# Only apply on conditional pass — not unconditional (negative prompt)
|
||||||
cond_or_uncond = transformer_options.get("cond_or_uncond", [])
|
cond_or_uncond = transformer_options.get("cond_or_uncond", [])
|
||||||
if 1 in cond_or_uncond and 0 not in cond_or_uncond:
|
if 1 in cond_or_uncond and 0 not in cond_or_uncond:
|
||||||
|
debug_log("mask_fn: unconditional pass, returning None")
|
||||||
return None
|
return None
|
||||||
|
|
||||||
grid_sizes = transformer_options.get("grid_sizes", None)
|
grid_sizes = transformer_options.get("grid_sizes", None)
|
||||||
video_tpf = int(grid_sizes[1]) * int(grid_sizes[2]) if grid_sizes is not None else fallback_tokens_per_frame
|
attn_type = transformer_options.get("promptrelay_attn_type", "attn2")
|
||||||
video_lq = latent_frames * video_tpf
|
is_audio = (attn_type == "audio_attn2")
|
||||||
|
|
||||||
# Skip cross-modal attention — text keys are padded to a fixed length ≥ max_token_idx and != video_lq
|
if is_audio:
|
||||||
if Lk == video_lq or Lk < max_token_idx:
|
mode = "scaled"
|
||||||
return None
|
video_lq = -1
|
||||||
|
else:
|
||||||
|
if grid_sizes is not None:
|
||||||
|
video_tpf = int(grid_sizes[1]) * int(grid_sizes[2])
|
||||||
|
else:
|
||||||
|
if Lq % latent_frames == 0:
|
||||||
|
video_tpf = Lq // latent_frames
|
||||||
|
else:
|
||||||
|
video_tpf = fallback_tokens_per_frame
|
||||||
|
video_lq = latent_frames * video_tpf
|
||||||
|
|
||||||
mode = "video" if Lq == video_lq else "scaled"
|
# Skip cross-modal attention — text keys are padded to a fixed length ≥ max_token_idx and != video_lq
|
||||||
|
if Lk == video_lq or Lk < max_token_idx:
|
||||||
|
debug_log(f"mask_fn: Lk == video_lq ({Lk == video_lq}) or Lk < max_token_idx ({Lk < max_token_idx}), returning None")
|
||||||
|
return None
|
||||||
|
|
||||||
key = (Lq, Lk, mode, q.device)
|
mode = "video" if Lq == video_lq else "scaled"
|
||||||
|
|
||||||
|
key = (Lq, Lk, mode, device)
|
||||||
if key not in cache:
|
if key not in cache:
|
||||||
if mode == "video":
|
if mode == "video":
|
||||||
cost = build_temporal_cost(q_token_idx, Lq, Lk, q.device, q.dtype, video_tpf)
|
cost = build_temporal_cost(q_token_idx, Lq, Lk, device, dtype, video_tpf)
|
||||||
else:
|
else:
|
||||||
cost = build_temporal_cost_scaled(q_token_idx, Lq, Lk, q.device, q.dtype, latent_frames)
|
cost = build_temporal_cost_scaled(q_token_idx, Lq, Lk, device, dtype, latent_frames, is_audio=is_audio)
|
||||||
log.info(
|
log.info(
|
||||||
"[PromptRelay] Built penalty matrix (%s): Lq=%d, Lk=%d, nonzero=%d/%d",
|
"[PromptRelay] Built penalty matrix (%s): Lq=%d, Lk=%d, nonzero=%d/%d",
|
||||||
mode, Lq, Lk, (cost > 0).sum().item(), cost.numel(),
|
mode, Lq, Lk, (cost > 0).sum().item(), cost.numel(),
|
||||||
)
|
)
|
||||||
|
debug_log(f"Built penalty matrix ({mode}): Lq={Lq}, Lk={Lk}, key={key}")
|
||||||
cache[key] = -cost
|
cache[key] = -cost
|
||||||
|
|
||||||
return cache[key].to(q.dtype)
|
return cache[key].to(dtype)
|
||||||
|
|
||||||
return mask_fn
|
return mask_fn
|
||||||
|
|
||||||
@@ -160,7 +196,30 @@ def map_token_indices(raw_tokenizer, global_prompt, local_prompts):
|
|||||||
"""
|
"""
|
||||||
prefixed_locals = [" " + lp for lp in local_prompts]
|
prefixed_locals = [" " + lp for lp in local_prompts]
|
||||||
full_prompt = global_prompt + "".join(prefixed_locals)
|
full_prompt = global_prompt + "".join(prefixed_locals)
|
||||||
|
|
||||||
|
# Detect if the tokenizer appends EOS dynamically
|
||||||
has_eos = getattr(raw_tokenizer, "add_eos", False)
|
has_eos = getattr(raw_tokenizer, "add_eos", False)
|
||||||
|
if not has_eos:
|
||||||
|
try:
|
||||||
|
test_res = raw_tokenizer("test")
|
||||||
|
if isinstance(test_res, dict) and "input_ids" in test_res:
|
||||||
|
ids = test_res["input_ids"]
|
||||||
|
elif hasattr(test_res, "input_ids"):
|
||||||
|
ids = test_res.input_ids
|
||||||
|
elif isinstance(test_res, list):
|
||||||
|
ids = test_res
|
||||||
|
else:
|
||||||
|
ids = []
|
||||||
|
|
||||||
|
if ids:
|
||||||
|
eos_id = getattr(raw_tokenizer, "eos_token_id", None)
|
||||||
|
if eos_id is not None and ids[-1] == eos_id:
|
||||||
|
has_eos = True
|
||||||
|
elif ids[-1] == 1:
|
||||||
|
has_eos = True
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
eos_adj = 1 if has_eos else 0
|
eos_adj = 1 if has_eos else 0
|
||||||
|
|
||||||
prev_len = len(raw_tokenizer(global_prompt)["input_ids"]) - eos_adj
|
prev_len = len(raw_tokenizer(global_prompt)["input_ids"]) - eos_adj
|
||||||
|
|||||||
@@ -1,7 +1,7 @@
|
|||||||
[project]
|
[project]
|
||||||
name = "whatdreamscost-csglide"
|
name = "whatdreamscost-csglide"
|
||||||
description = "LTX Director (CS) — modded LTX Director node with reference-sheet / @char / MSR support. Coexists with the original WhatDreamsCost-ComfyUI without collisions."
|
description = "LTX Director (CS) — modded LTX Director with reference-sheet / @char / MSR + provider Analyze backend. Coexists with the original WhatDreamsCost-ComfyUI without collisions."
|
||||||
version = "1.0.0"
|
version = "2.0.0"
|
||||||
license = {file = "LICENSE"}
|
license = {file = "LICENSE"}
|
||||||
|
|
||||||
[project.urls]
|
[project.urls]
|
||||||
|
|||||||
Reference in New Issue
Block a user