Squashed 'vendor/dumas_h3_refmod_picker/' content from commit c732f8b
git-subtree-dir: vendor/dumas_h3_refmod_picker git-subtree-split: c732f8b8893bf4f00c6974c9b9941c763496fe48
@@ -0,0 +1,22 @@
|
||||
# Python bytecode / caches
|
||||
__pycache__/
|
||||
*.py[cod]
|
||||
*$py.class
|
||||
|
||||
# Virtual environments
|
||||
.venv/
|
||||
venv/
|
||||
env/
|
||||
|
||||
# Editor / OS clutter
|
||||
*.swp
|
||||
*.swo
|
||||
*~
|
||||
.DS_Store
|
||||
.directory
|
||||
Thumbs.db
|
||||
|
||||
# Tooling
|
||||
.history/
|
||||
*.code-workspace
|
||||
test/
|
||||
@@ -0,0 +1,21 @@
|
||||
MIT License
|
||||
|
||||
Copyright (c) 2026 Luisa (luisacaotica)
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
@@ -0,0 +1,89 @@
|
||||
# ComfyUI-H3RefModPicker
|
||||
|
||||
Companion add-on for [ComfyUI-MiniMaxH3Mod](https://github.com/Luisacaotica/ComfyUI-MiniMaxH3Mod)
|
||||
|
||||
This add-on is meant to work alongside **MiniMaxH3Mod**. It provides a small, focused subset of RefMod features: a Visual Picker, a simple loader, an apply axis, a batch Create From Folder node, and grounded RefMods-to-Video generation with an optional reference collector. It keeps compatibility with both the original RefMod format and the newer bundle format.
|
||||
|
||||
If you want the fuller MiniMax H3 RefMod toolset and more creation/apply options, the recommended install is [ComfyUI-MiniMaxH3Mod](https://github.com/Luisacaotica/ComfyUI-MiniMaxH3Mod).
|
||||
|
||||
Please note, the `Create from Input` node was removed from this add-on. The same functionality is available in the official MiniMaxH3Mod add-on, using the `Create H3 RefMod Master` node.
|
||||
|
||||
## Latest Updates
|
||||
|
||||
Video-file support has been removed from `Create H3 RefMod From Folder`. Video clips are not ideal for the simple, lightweight RefMods this add-on aims to create: around **six to ten well-chosen reference images** are usually sufficient. Keep their aspect ratios consistent within each dataset folder. This is a recommendation, not an image limit; optional audio remains supported, and video files in the folder are ignored.
|
||||
|
||||
## Preview image naming
|
||||
|
||||
To have the Visual Picker pick up a preview image, place the image beside the RefMod using the same base name as the RefMod file.
|
||||
|
||||
- For the newer bundle format, the image and the RefMod just need to share the exact same name. Example: `character_refMod.safetensors` with `character_refMod.png`.
|
||||
- For older split RefMods, do not name the preview image `_audio` or `_visual`. Use only the shared base name. Example: `character_refMod_visual.safetensors` and `character_refMod_audio.safetensors` should use `character_refMod.png`.
|
||||
- When a matching `_visual` and `_audio` pair exists, plus that shared preview image, the picker sees them as one logical item.
|
||||
|
||||
|
||||
<p align="center">
|
||||
<img src="docs/examples/browser_example.png" alt="Extract H3 RefMod in use" />
|
||||
<br />
|
||||
<em>Example using refMods from <a href="https://huggingface.co/spaces/malcolmrey/browser">Malcolm Reynolds</a>.</em>
|
||||
</p>
|
||||
|
||||
<p align="center">
|
||||
<img src="docs/examples/workflow_example.png" alt="Workflow example" />
|
||||
<br />
|
||||
<em>Use Left and Right arrows when over Picker to switch quickly.</em>
|
||||
</p>
|
||||
|
||||
<p align="center">
|
||||
<img src="docs/examples/create_from_folder_example.png" alt="Create from folder example" />
|
||||
<br />
|
||||
<em>Create refMods from Folder example.</em>
|
||||
</p>
|
||||
|
||||
## Main additions
|
||||
|
||||
- `Visual RefMod Picker` lets you browse using a **RefMods** browser. It supports legacy split RefMods and the newer single-file bundle format.
|
||||
- For legacy split RefMods, found pairs are grouped as one item in the picker. They are shown as one entry and loaded together, with separate video/audio weight controls. This uses a weight behavior: `0..1` is regular strength, values above 1 expand into repeated copies. For example, a weight of 2.7 would be the same as strength: 1.0, copies: 2.7.
|
||||
- `Create H3 RefMod From Folder` scans a folder of reference images and optional audio, and saves in the bundle format by default. Around six to ten images can be sufficient. It can also batch-create RefMods for all subfolders found. The optional `include_images` switch adds reconstructed encoder pictures to visual RefMods while keeping the existing latent format compatible; it is off by default.
|
||||
- `Load RefMod Simple` is a singular loader. It supports standalone visual/audio RefMods and the new bundle format, while still using the same weight behavior as the Picker.
|
||||
- `MiniMax H3 RefMods to Video` grounds prompt labels with RefMod pictures and applies their visual/audio latents, returning positive conditioning, an empty AV latent, and a reference map. Optional ordinary media can be added through `Collect H3 References`. It replaced minimax's Reference to Video node. It takes in mods as an input, bypassing the need to apply the RefMods using an apply node.
|
||||
- `Apply H3 RefMod Simple` is a streamlined version of the Apply H3 RefMod, without the extra controls.
|
||||
|
||||
## RefMods to Video Node
|
||||
|
||||
The grounded RefMods-to-Video node builds on work by [Adudeguyman's ComfyUI-Fantastic-MiniMaxH3-PromptBuilder](https://github.com/Adudeguyman/ComfyUI-Fantastic-MiniMaxH3-PromptBuilder). This also revisits earlier visual-grounding experiments this add-on had initially, which were perhaps set aside a little too quickly :)
|
||||
|
||||
Unlike plain prompt encoding followed by RefMod Apply, this node gives the text encoder visual content behind labels such as `<Picture 1>` and `<Video 1>`, while supplying the corresponding latents to diffusion. This lets you refer to specific references in your prompt, but does not guarantee exclusive character or voice assignment.
|
||||
|
||||
Connect an H3 CLIP and the `mods` output from the Picker, Prompt Manager's `Prompt Composer`, a RefMod loader, or MiniMaxH3Mod's standard stack to `MiniMax H3 RefMods to Video`. Prompt Composer preserves embedded encoder images and returns weights above 1 as repeated stack rows, matching the Picker. For RefMod-only generation, leave `references` disconnected. To add ordinary media, connect `Collect H3 References`; connect video frame batches to `ref_video_N` and their soundtracks to the same-numbered `ref_video_audio_N`. The reference inputs grow as connections are added.
|
||||
|
||||
`video_vae` reconstructs visual RefMods without stored encoder pictures, and encodes raw images/videos. Encoder pictures stored by this creator or Fantastic can be read without decoding. `audio_vae` is needed to encode raw audio, not audio RefMods. As in the native reference node, raw media without its VAE only enters the encoder presentation. Latent-only visual RefMods require `video_vae`. Video playback is assumed to be 24 fps.
|
||||
|
||||
The `reference_map` string output contains only prompt labels and reference names, such as `<Video 1> = Alice_visual`. Connect it to a text display to see the mapping. Raw references are presented first (pictures, videos with soundtracks, standalone audio), followed by active RefMods in stack order. Counters are separate per media type. For example, if the map lists Alice as `<Video 1>` and Bob as `<Video 2>`:
|
||||
|
||||
```text
|
||||
<Subject 1> is based on <Video 1>.
|
||||
<Subject 2> is based on <Video 2>.
|
||||
```
|
||||
|
||||
`stack_pictures` controls which reconstructed pictures from an image-stack RefMod the encoder sees. It never removes latent frames from the diffusion reference. Grounding helps the model associate labels with visual content; it does not enforce exclusive character assignments. Legacy combined visual/audio RefMods are supported. Do not Apply the same RefMods again after this node, as their latent references are already included.
|
||||
|
||||
### Include Encoder Images
|
||||
|
||||
`Create H3 RefMod From Folder` has an `include_images` switch, off by default. When enabled, it stores JPEG pictures reconstructed from the final visual latent, like Fantastic's creator, retaining every stored latent frame for image stacks. These are reconstructions, not copies of the original source images. Latents and audio are unchanged.
|
||||
|
||||
With the switch off, creation retains the existing latent-only behavior. Older RefMods without pictures remain supported by RefMods-to-Video through VAE reconstruction; adding pictures does not require converting existing files or change how their latents are applied.
|
||||
|
||||
The pictures are retained in both bundle and separate-file layouts and can be used by this add-on's RefMods-to-Video node without decoding again. Both layouts use Fantastic's `enc_N` tensor keys and `enc_times` / `enc_fps` metadata. In bundles, the picture metadata stays on the visual member; the optional audio member has no pictures. The version-5 bundle container is unchanged, so other loaders can read its standard latents, but reuse of its pictures depends on their bundle support. Enabling the switch adds a VAE decode during creation and increases file size. It also applies in subfolder mode and to unsaved mods passed directly to another node.
|
||||
|
||||
## Notes
|
||||
|
||||
- Supported RefMod files include older split `_visual` / `_audio` saves, single-file MiniMax bundles, and RefMods with stored encoder images, including the image-inclusive bundles added by this add-on. Existing latent-only files remain supported.
|
||||
- For more options and the complete MiniMax H3 RefMod feature set, use [ComfyUI-MiniMaxH3Mod](https://github.com/Luisacaotica/ComfyUI-MiniMaxH3Mod).
|
||||
|
||||
## Installation
|
||||
|
||||
### Manual
|
||||
```bash
|
||||
cd ComfyUI/custom_nodes
|
||||
git clone https://github.com/FranckyB/ComfyUI-H3RefModPicker
|
||||
```
|
||||
@@ -0,0 +1,36 @@
|
||||
"""
|
||||
ComfyUI-H3RefModPicker — Addons to MiniMaxH3Mod, adds a visual picker and a batch creation tool.
|
||||
"""
|
||||
|
||||
__author__ = "FranckB"
|
||||
__version__ = "0.2.0"
|
||||
|
||||
from .nodes import refmod_apply
|
||||
from .nodes import refmod_loader
|
||||
from .nodes import refmod_visual_picker
|
||||
from .nodes import refmod_create
|
||||
from .nodes import refmod_axis
|
||||
from .nodes import refmods_to_video
|
||||
# from .nodes import refmod_decode
|
||||
|
||||
NODE_CLASS_MAPPINGS = {
|
||||
**refmod_apply.NODE_CLASS_MAPPINGS,
|
||||
**refmod_loader.NODE_CLASS_MAPPINGS,
|
||||
**refmod_visual_picker.NODE_CLASS_MAPPINGS,
|
||||
**refmod_create.NODE_CLASS_MAPPINGS,
|
||||
**refmod_axis.NODE_CLASS_MAPPINGS,
|
||||
**refmods_to_video.NODE_CLASS_MAPPINGS,
|
||||
# **refmod_decode.NODE_CLASS_MAPPINGS,
|
||||
}
|
||||
NODE_DISPLAY_NAME_MAPPINGS = {
|
||||
**refmod_apply.NODE_DISPLAY_NAME_MAPPINGS,
|
||||
**refmod_loader.NODE_DISPLAY_NAME_MAPPINGS,
|
||||
**refmod_visual_picker.NODE_DISPLAY_NAME_MAPPINGS,
|
||||
**refmod_create.NODE_DISPLAY_NAME_MAPPINGS,
|
||||
**refmod_axis.NODE_DISPLAY_NAME_MAPPINGS,
|
||||
**refmods_to_video.NODE_DISPLAY_NAME_MAPPINGS,
|
||||
# **refmod_decode.NODE_DISPLAY_NAME_MAPPINGS,
|
||||
}
|
||||
WEB_DIRECTORY = "./js"
|
||||
|
||||
__all__ = ["NODE_CLASS_MAPPINGS", "NODE_DISPLAY_NAME_MAPPINGS", "WEB_DIRECTORY"]
|
||||
|
After Width: | Height: | Size: 1.6 MiB |
|
After Width: | Height: | Size: 75 KiB |
|
After Width: | Height: | Size: 1.3 MiB |
|
After Width: | Height: | Size: 43 KiB |
|
After Width: | Height: | Size: 112 KiB |
|
After Width: | Height: | Size: 1.0 MiB |
|
After Width: | Height: | Size: 29 KiB |
@@ -0,0 +1,392 @@
|
||||
import { app } from "../../scripts/app.js";
|
||||
|
||||
const TARGET_NODES = new Set([
|
||||
"H3RefModCreateFromFolder",
|
||||
]);
|
||||
|
||||
const MIN_NODE_WIDTH = 320;
|
||||
const CREATE_FOLDER_INSTRUCTIONS = [
|
||||
"Keep image count to a minimum; around six to ten well-chosen",
|
||||
"images can suffice. Add audio if wanted. Videos are ignored.",
|
||||
"",
|
||||
"Folder: absolute path or folder inside ComfyUI's input/.",
|
||||
"Connect the H3 video VAE; audio VAE is optional.",
|
||||
"use_subfolders creates one RefMod per immediate subfolder.",
|
||||
"include_images stores grounding pictures; off by default.",
|
||||
"Saves to models/refmods/ when save is enabled.",
|
||||
"Or use directly as a mod, by connecting to mods output.",
|
||||
].join("\n");
|
||||
|
||||
function addAspectRatioReminder(node) {
|
||||
let tooltip = null;
|
||||
let tooltipStyle = null;
|
||||
function hideTooltip() {
|
||||
tooltip?.remove();
|
||||
tooltip = null;
|
||||
tooltipStyle?.remove();
|
||||
tooltipStyle = null;
|
||||
}
|
||||
const onDrawForeground = node.onDrawForeground;
|
||||
node.onDrawForeground = function (context) {
|
||||
const result = onDrawForeground?.apply(this, arguments);
|
||||
if (!this.flags?.collapsed) {
|
||||
context.save();
|
||||
context.fillStyle = "#facc15";
|
||||
context.font = "bold 18px sans-serif";
|
||||
context.textAlign = "center";
|
||||
context.textBaseline = "middle";
|
||||
context.fillText("?", this.size[0] - 18, -15);
|
||||
context.restore();
|
||||
}
|
||||
return result;
|
||||
};
|
||||
const onMouseMove = node.onMouseMove;
|
||||
node.onMouseMove = function (event, position) {
|
||||
const result = onMouseMove?.apply(this, arguments);
|
||||
const overReminder = !this.flags?.collapsed && position &&
|
||||
Math.abs(position[0] - (this.size[0] - 18)) <= 12 &&
|
||||
Math.abs(position[1] + 15) <= 12;
|
||||
if (!overReminder) {
|
||||
hideTooltip();
|
||||
return result;
|
||||
}
|
||||
if (!tooltip) {
|
||||
tooltip = document.createElement("div");
|
||||
tooltip.setAttribute("role", "tooltip");
|
||||
const warning = document.createElement("strong");
|
||||
warning.textContent = "Warning: images should all have the same aspect ratio within each dataset folder.";
|
||||
warning.style.cssText = "display:block;margin-bottom:8px;color:#facc15;";
|
||||
tooltip.append(warning, document.createTextNode(CREATE_FOLDER_INSTRUCTIONS));
|
||||
tooltip.style.cssText = "position:fixed;z-index:10000;pointer-events:none;box-sizing:border-box;max-width:min(480px,calc(100vw - 16px));padding:12px 14px;border:1px solid #facc15;border-radius:4px;background:#252525;color:#fff;font-size:15px;line-height:1.45;white-space:pre-line;overflow-wrap:anywhere;";
|
||||
tooltipStyle = document.createElement("style");
|
||||
tooltipStyle.textContent = ".node-tooltip { display: none !important; }";
|
||||
document.body.append(tooltip, tooltipStyle);
|
||||
}
|
||||
const bounds = tooltip.getBoundingClientRect();
|
||||
tooltip.style.left = `${Math.max(8, Math.min(event.clientX + 12, window.innerWidth - bounds.width - 8))}px`;
|
||||
tooltip.style.top = `${Math.max(8, Math.min(event.clientY + 12, window.innerHeight - bounds.height - 8))}px`;
|
||||
return result;
|
||||
};
|
||||
const onMouseLeave = node.onMouseLeave;
|
||||
node.onMouseLeave = function () {
|
||||
hideTooltip();
|
||||
return onMouseLeave?.apply(this, arguments);
|
||||
};
|
||||
const onRemoved = node.onRemoved;
|
||||
node.onRemoved = function () {
|
||||
hideTooltip();
|
||||
return onRemoved?.apply(this, arguments);
|
||||
};
|
||||
}
|
||||
|
||||
const COMMON_PRESET_FIELDS = [
|
||||
"concept_type",
|
||||
"audio_concept_type",
|
||||
"mode",
|
||||
"ref_resolution",
|
||||
"pool_h",
|
||||
"pool_w",
|
||||
"max_tokens",
|
||||
"identity",
|
||||
"merge",
|
||||
"multiplier",
|
||||
"audio_max_seconds",
|
||||
"audio_max_tokens",
|
||||
"audio_budget_policy",
|
||||
"max_total_tokens",
|
||||
"budget_policy",
|
||||
];
|
||||
|
||||
const PRESET_CONFIG = {
|
||||
H3RefModCreateFromFolder: {
|
||||
manual: {
|
||||
values: {
|
||||
concept_type: "identity",
|
||||
audio_concept_type: "voice",
|
||||
mode: "Full Reference",
|
||||
ref_resolution: 1024,
|
||||
pool_h: 16,
|
||||
pool_w: 16,
|
||||
max_tokens: 8192,
|
||||
identity: 500,
|
||||
merge: false,
|
||||
multiplier: 1,
|
||||
audio_max_seconds: 30.0,
|
||||
audio_max_tokens: 5120,
|
||||
audio_budget_policy: "error",
|
||||
max_total_tokens: 0,
|
||||
budget_policy: "truncate",
|
||||
},
|
||||
fields: [],
|
||||
compactFields: [],
|
||||
},
|
||||
identity_encode: {
|
||||
values: {
|
||||
concept_type: "identity",
|
||||
audio_concept_type: "voice",
|
||||
mode: "Full Reference",
|
||||
ref_resolution: 1024,
|
||||
pool_h: 16,
|
||||
pool_w: 16,
|
||||
max_tokens: 8192,
|
||||
identity: 0,
|
||||
merge: false,
|
||||
multiplier: 1,
|
||||
audio_max_seconds: 30.0,
|
||||
audio_max_tokens: 5120,
|
||||
audio_budget_policy: "error",
|
||||
max_total_tokens: 0,
|
||||
budget_policy: "truncate",
|
||||
},
|
||||
fields: [
|
||||
"concept_type",
|
||||
"audio_concept_type",
|
||||
"mode",
|
||||
"ref_resolution",
|
||||
"identity",
|
||||
"merge",
|
||||
],
|
||||
compactFields: COMMON_PRESET_FIELDS,
|
||||
},
|
||||
style_experimental: {
|
||||
values: {
|
||||
concept_type: "style",
|
||||
audio_concept_type: "voice",
|
||||
mode: "Compressed Reference",
|
||||
ref_resolution: 1024,
|
||||
pool_h: 8,
|
||||
pool_w: 8,
|
||||
max_tokens: 8192,
|
||||
identity: 150,
|
||||
merge: false,
|
||||
multiplier: 1,
|
||||
audio_max_seconds: 30.0,
|
||||
audio_max_tokens: 5120,
|
||||
audio_budget_policy: "error",
|
||||
max_total_tokens: 0,
|
||||
budget_policy: "truncate",
|
||||
},
|
||||
fields: [
|
||||
"concept_type",
|
||||
"audio_concept_type",
|
||||
"mode",
|
||||
"pool_h",
|
||||
"pool_w",
|
||||
"identity",
|
||||
"merge",
|
||||
],
|
||||
compactFields: COMMON_PRESET_FIELDS,
|
||||
},
|
||||
motion_sequence: {
|
||||
values: {
|
||||
concept_type: "pose_motion",
|
||||
audio_concept_type: "voice",
|
||||
mode: "Compressed Reference",
|
||||
ref_resolution: 1024,
|
||||
pool_h: 16,
|
||||
pool_w: 16,
|
||||
max_tokens: 8192,
|
||||
identity: 500,
|
||||
merge: false,
|
||||
multiplier: 1,
|
||||
audio_max_seconds: 30.0,
|
||||
audio_max_tokens: 5120,
|
||||
audio_budget_policy: "error",
|
||||
max_total_tokens: 0,
|
||||
budget_policy: "truncate",
|
||||
},
|
||||
fields: [
|
||||
"concept_type",
|
||||
"audio_concept_type",
|
||||
"mode",
|
||||
"pool_h",
|
||||
"pool_w",
|
||||
"merge",
|
||||
],
|
||||
compactFields: COMMON_PRESET_FIELDS,
|
||||
},
|
||||
},
|
||||
};
|
||||
|
||||
function findWidget(node, name) {
|
||||
return node.widgets?.find((widget) => widget?.name === name) || null;
|
||||
}
|
||||
|
||||
function moveWidgetAfter(node, widgetName, afterName) {
|
||||
if (!node.widgets) return;
|
||||
const widget = findWidget(node, widgetName);
|
||||
const after = findWidget(node, afterName);
|
||||
if (!widget || !after) return;
|
||||
const from = node.widgets.indexOf(widget);
|
||||
const afterIndex = node.widgets.indexOf(after);
|
||||
if (from < 0 || afterIndex < 0) return;
|
||||
node.widgets.splice(from, 1);
|
||||
const insertAt = node.widgets.indexOf(after) + 1;
|
||||
node.widgets.splice(insertAt, 0, widget);
|
||||
}
|
||||
|
||||
function setWidgetVisible(widget, visible) {
|
||||
if (!widget) return;
|
||||
if (!widget.__h3VisibilityWrapped) {
|
||||
widget.__h3VisibilityWrapped = true;
|
||||
widget.__h3BaseComputeSize = widget.computeSize;
|
||||
widget.computeSize = function (...args) {
|
||||
if (widget.__h3Hidden) {
|
||||
return [0, -4];
|
||||
}
|
||||
if (typeof widget.__h3BaseComputeSize === "function") {
|
||||
return widget.__h3BaseComputeSize.apply(this, args);
|
||||
}
|
||||
return [MIN_NODE_WIDTH, Number(LiteGraph?.NODE_WIDGET_HEIGHT || 24)];
|
||||
};
|
||||
}
|
||||
if (typeof widget.__h3OriginalDisplay === "undefined" && widget.inputEl) {
|
||||
widget.__h3OriginalDisplay = widget.inputEl.style.display;
|
||||
}
|
||||
widget.__h3Hidden = !visible;
|
||||
widget.hidden = !visible;
|
||||
if (widget.inputEl) {
|
||||
widget.inputEl.style.display = visible ? (widget.__h3OriginalDisplay || "") : "none";
|
||||
}
|
||||
}
|
||||
|
||||
function refreshNodeLayout(node) {
|
||||
if (Array.isArray(node.size)) {
|
||||
node.size[0] = Math.max(Number(node.size[0]) || 0, MIN_NODE_WIDTH);
|
||||
}
|
||||
const nextWidth = Math.max(node.size?.[0] || 0, MIN_NODE_WIDTH);
|
||||
const nextHeight = node.computeSize?.()?.[1] || node.size?.[1] || 0;
|
||||
node.setSize?.([nextWidth, nextHeight]);
|
||||
node.onResize?.(node.size);
|
||||
node.setDirtyCanvas(true, true);
|
||||
app.graph?.setDirtyCanvas(true, true);
|
||||
}
|
||||
|
||||
function syncPresetValues(node) {
|
||||
const presetWidget = findWidget(node, "extraction_preset");
|
||||
const preset = String(presetWidget?.value || "manual");
|
||||
const nodeType = String(node.comfyClass || "");
|
||||
const nodePresets = PRESET_CONFIG[nodeType];
|
||||
if (!nodePresets) return;
|
||||
const config = nodePresets[preset] || nodePresets.manual;
|
||||
for (const [name, value] of Object.entries(config.values)) {
|
||||
const widget = findWidget(node, name);
|
||||
if (!widget) continue;
|
||||
widget.value = value;
|
||||
if (widget.inputEl) {
|
||||
widget.inputEl.value = String(value);
|
||||
if (typeof widget.inputEl.checked === "boolean" && typeof value === "boolean") {
|
||||
widget.inputEl.checked = value;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function applyFolderNamingVisibility(node) {
|
||||
if (String(node.comfyClass || "") !== "H3RefModCreateFromFolder") return;
|
||||
const useSubfoldersWidget = findWidget(node, "use_subfolders");
|
||||
const useFolderAsNameWidget = findWidget(node, "use_folder_as_name");
|
||||
const nameWidget = findWidget(node, "name");
|
||||
const useSubfolders = Boolean(useSubfoldersWidget?.value);
|
||||
const useFolderAsName = Boolean(useFolderAsNameWidget?.value);
|
||||
|
||||
setWidgetVisible(useFolderAsNameWidget, !useSubfolders);
|
||||
setWidgetVisible(nameWidget, !useSubfolders && !useFolderAsName);
|
||||
}
|
||||
|
||||
function applyPresetVisibility(node) {
|
||||
const presetWidget = findWidget(node, "extraction_preset");
|
||||
const advancedWidget = findWidget(node, "advanced");
|
||||
const preset = String(presetWidget?.value || "manual");
|
||||
const advanced = Boolean(advancedWidget?.value);
|
||||
const nodeType = String(node.comfyClass || "");
|
||||
const nodePresets = PRESET_CONFIG[nodeType];
|
||||
if (!nodePresets) return;
|
||||
const config = nodePresets[preset] || nodePresets.manual;
|
||||
const active = new Set(advanced ? config.fields : (config.compactFields || config.fields));
|
||||
const known = new Set(Object.values(nodePresets).flatMap((configEntry) => [
|
||||
...(configEntry.fields || []),
|
||||
...(configEntry.compactFields || []),
|
||||
]));
|
||||
for (const name of known) {
|
||||
setWidgetVisible(findWidget(node, name), !(preset !== "manual" && active.has(name)));
|
||||
}
|
||||
setWidgetVisible(advancedWidget, preset !== "manual");
|
||||
applyFolderNamingVisibility(node);
|
||||
refreshNodeLayout(node);
|
||||
}
|
||||
|
||||
function attachWidgetRefresh(node, widgetName, onAfterChange) {
|
||||
const widget = findWidget(node, widgetName);
|
||||
if (!widget || widget.__h3RefreshAttached) return;
|
||||
widget.__h3RefreshAttached = true;
|
||||
const originalCallback = widget.callback;
|
||||
widget.callback = function () {
|
||||
const result = originalCallback?.apply(this, arguments);
|
||||
onAfterChange();
|
||||
return result;
|
||||
};
|
||||
}
|
||||
|
||||
function enhanceCreateNode(node) {
|
||||
if (node.__h3CreateUiEnhanced) return;
|
||||
node.__h3CreateUiEnhanced = true;
|
||||
|
||||
const originalOnResize = node.onResize;
|
||||
node.onResize = function (size) {
|
||||
if (size) {
|
||||
size[0] = Math.max(size[0] || 0, MIN_NODE_WIDTH);
|
||||
}
|
||||
return originalOnResize?.apply(this, arguments);
|
||||
};
|
||||
|
||||
addAspectRatioReminder(node);
|
||||
moveWidgetAfter(node, "extraction_preset", "name");
|
||||
moveWidgetAfter(node, "advanced", "extraction_preset");
|
||||
moveWidgetAfter(node, "budget_policy", "max_total_tokens");
|
||||
syncPresetValues(node);
|
||||
applyPresetVisibility(node);
|
||||
|
||||
attachWidgetRefresh(node, "extraction_preset", () => {
|
||||
syncPresetValues(node);
|
||||
applyPresetVisibility(node);
|
||||
});
|
||||
attachWidgetRefresh(node, "advanced", () => applyPresetVisibility(node));
|
||||
attachWidgetRefresh(node, "use_subfolders", () => applyPresetVisibility(node));
|
||||
attachWidgetRefresh(node, "use_folder_as_name", () => applyPresetVisibility(node));
|
||||
|
||||
const onConfigure = node.onConfigure;
|
||||
node.onConfigure = function () {
|
||||
const result = onConfigure?.apply(this, arguments);
|
||||
moveWidgetAfter(this, "extraction_preset", "name");
|
||||
moveWidgetAfter(this, "advanced", "extraction_preset");
|
||||
moveWidgetAfter(this, "budget_policy", "max_total_tokens");
|
||||
syncPresetValues(this);
|
||||
applyPresetVisibility(this);
|
||||
attachWidgetRefresh(this, "extraction_preset", () => {
|
||||
syncPresetValues(this);
|
||||
applyPresetVisibility(this);
|
||||
});
|
||||
attachWidgetRefresh(this, "advanced", () => applyPresetVisibility(this));
|
||||
attachWidgetRefresh(this, "use_subfolders", () => applyPresetVisibility(this));
|
||||
attachWidgetRefresh(this, "use_folder_as_name", () => applyPresetVisibility(this));
|
||||
return result;
|
||||
};
|
||||
}
|
||||
|
||||
app.registerExtension({
|
||||
name: "H3RefModPicker.RefModCreate",
|
||||
|
||||
async beforeRegisterNodeDef(nodeType, nodeData) {
|
||||
if (!TARGET_NODES.has(nodeData?.name)) return;
|
||||
|
||||
const onNodeCreated = nodeType.prototype.onNodeCreated;
|
||||
nodeType.prototype.onNodeCreated = function () {
|
||||
const result = onNodeCreated?.apply(this, arguments);
|
||||
enhanceCreateNode(this);
|
||||
return result;
|
||||
};
|
||||
},
|
||||
});
|
||||
|
||||
console.log("[H3RefModPicker] RefModCreate extension loaded");
|
||||
@@ -0,0 +1,116 @@
|
||||
import { app } from "../../scripts/app.js";
|
||||
import { api } from "../../scripts/api.js";
|
||||
|
||||
function findWidget(node, name) {
|
||||
return node.widgets?.find((widget) => widget?.name === name) || null;
|
||||
}
|
||||
|
||||
function setWidgetVisible(widget, visible) {
|
||||
if (!widget) return;
|
||||
if (!widget.__h3LoaderVisibilityWrapped) {
|
||||
widget.__h3LoaderVisibilityWrapped = true;
|
||||
widget.__h3LoaderBaseComputeSize = widget.computeSize;
|
||||
widget.computeSize = function (...args) {
|
||||
if (widget.__h3LoaderHidden) {
|
||||
return [0, -4];
|
||||
}
|
||||
if (typeof widget.__h3LoaderBaseComputeSize === "function") {
|
||||
return widget.__h3LoaderBaseComputeSize.apply(this, args);
|
||||
}
|
||||
return [200, Number(LiteGraph?.NODE_WIDGET_HEIGHT || 24)];
|
||||
};
|
||||
}
|
||||
if (typeof widget.__h3LoaderOriginalDisplay === "undefined" && widget.inputEl) {
|
||||
widget.__h3LoaderOriginalDisplay = widget.inputEl.style.display;
|
||||
}
|
||||
widget.__h3LoaderHidden = !visible;
|
||||
widget.hidden = !visible;
|
||||
if (widget.inputEl) {
|
||||
widget.inputEl.style.display = visible ? (widget.__h3LoaderOriginalDisplay || "") : "none";
|
||||
}
|
||||
}
|
||||
|
||||
function refreshNodeLayout(node) {
|
||||
node.setDirtyCanvas(true, true);
|
||||
app.graph?.setDirtyCanvas(true, true);
|
||||
}
|
||||
|
||||
async function fetchLoaderCapabilities(mod) {
|
||||
const query = mod ? `?mod=${encodeURIComponent(mod)}` : "";
|
||||
const resp = await api.fetchApi(`/h3refmods/refmod-loader/capabilities${query}`);
|
||||
if (!resp.ok) {
|
||||
throw new Error(`Request failed (${resp.status})`);
|
||||
}
|
||||
return await resp.json();
|
||||
}
|
||||
|
||||
function applyModalityVisibility(node, caps) {
|
||||
const videoWidget = findWidget(node, "video_weight");
|
||||
const audioWidget = findWidget(node, "audio_weight");
|
||||
const hasVisual = typeof caps?.has_visual === "boolean" ? caps.has_visual : true;
|
||||
const hasAudio = typeof caps?.has_audio === "boolean" ? caps.has_audio : true;
|
||||
setWidgetVisible(videoWidget, hasVisual);
|
||||
setWidgetVisible(audioWidget, hasAudio);
|
||||
refreshNodeLayout(node);
|
||||
}
|
||||
|
||||
async function refreshLoaderVisibility(node) {
|
||||
const modWidget = findWidget(node, "mod");
|
||||
const mod = String(modWidget?.value || "").trim();
|
||||
if (!mod) {
|
||||
applyModalityVisibility(node, { has_visual: true, has_audio: true });
|
||||
return;
|
||||
}
|
||||
try {
|
||||
const data = await fetchLoaderCapabilities(mod);
|
||||
applyModalityVisibility(node, data.ok ? data : { has_visual: true, has_audio: true });
|
||||
} catch (error) {
|
||||
console.warn("[H3RefModLoader] Could not refresh capabilities:", error);
|
||||
applyModalityVisibility(node, { has_visual: true, has_audio: true });
|
||||
}
|
||||
}
|
||||
|
||||
app.registerExtension({
|
||||
name: "H3RefModPicker.RefModLoader",
|
||||
|
||||
async beforeRegisterNodeDef(nodeType, nodeData) {
|
||||
if (nodeData?.name !== "H3RefModLoader") return;
|
||||
|
||||
const onNodeCreated = nodeType.prototype.onNodeCreated;
|
||||
nodeType.prototype.onNodeCreated = function () {
|
||||
const result = onNodeCreated?.apply(this, arguments);
|
||||
const node = this;
|
||||
|
||||
const legacyWeightWidget = findWidget(node, "weight");
|
||||
const modWidget = findWidget(node, "mod");
|
||||
if (legacyWeightWidget) {
|
||||
setWidgetVisible(legacyWeightWidget, false);
|
||||
}
|
||||
|
||||
if (modWidget && !modWidget.__h3LoaderRefreshAttached) {
|
||||
modWidget.__h3LoaderRefreshAttached = true;
|
||||
const originalCallback = modWidget.callback;
|
||||
modWidget.callback = function () {
|
||||
const callbackResult = originalCallback?.apply(this, arguments);
|
||||
void refreshLoaderVisibility(node);
|
||||
return callbackResult;
|
||||
};
|
||||
}
|
||||
|
||||
const onConfigure = node.onConfigure;
|
||||
node.onConfigure = function () {
|
||||
const configureResult = onConfigure?.apply(this, arguments);
|
||||
if (legacyWeightWidget) {
|
||||
setWidgetVisible(legacyWeightWidget, false);
|
||||
}
|
||||
void refreshLoaderVisibility(this);
|
||||
return configureResult;
|
||||
};
|
||||
|
||||
void refreshLoaderVisibility(node);
|
||||
return result;
|
||||
};
|
||||
},
|
||||
});
|
||||
|
||||
console.log("[H3RefModPicker] RefModLoader extension loaded");
|
||||
@@ -0,0 +1,183 @@
|
||||
"""
|
||||
H3RefModApplySimple — identity-oriented RefMod apply node with one strength dial
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import random
|
||||
from dataclasses import replace
|
||||
from typing import Dict, List, Optional
|
||||
|
||||
from comfy_api.latest import io
|
||||
|
||||
# reference retention presets (master strength multiplier on Apply)
|
||||
RETENTION = {
|
||||
"fully_preserved": 1.0,
|
||||
"partially_preserved": 0.7,
|
||||
"attribute_transfer": 0.4,
|
||||
"weak_reference": 0.15,
|
||||
}
|
||||
|
||||
|
||||
def _filter_legacy_ref_block(block: Optional[Dict],
|
||||
use_video: bool,
|
||||
use_audio: bool) -> Optional[Dict]:
|
||||
"""Apply media toggles to ref blocks produced by older RefMod APIs."""
|
||||
if block is None or (not use_video and not use_audio):
|
||||
return None
|
||||
|
||||
kind = str(block.get("kind") or "").strip().lower()
|
||||
has_audio = block.get("audio_latent") is not None and int(block.get("ref_audio_t") or 0) > 0
|
||||
|
||||
if kind == "audio":
|
||||
return block if use_audio else None
|
||||
|
||||
if not use_video:
|
||||
if use_audio and has_audio:
|
||||
return {
|
||||
"kind": "audio",
|
||||
"ref_audio_t": int(block.get("ref_audio_t") or 0),
|
||||
"audio_latent": block.get("audio_latent"),
|
||||
}
|
||||
return None
|
||||
|
||||
if not use_audio and has_audio:
|
||||
filtered = dict(block)
|
||||
filtered["kind"] = "video"
|
||||
filtered["ref_audio_t"] = 0
|
||||
filtered["audio_latent"] = None
|
||||
return filtered
|
||||
|
||||
return block
|
||||
|
||||
|
||||
def _build_ref_block(mod, eff: float, curve, use_video: bool, use_audio: bool) -> Optional[Dict]:
|
||||
try:
|
||||
return mod.ref_block(eff, curve=curve, use_video=use_video, use_audio=use_audio)
|
||||
except TypeError as exc:
|
||||
message = str(exc)
|
||||
if "unexpected keyword argument 'use_video'" not in message and "unexpected keyword argument 'use_audio'" not in message:
|
||||
raise
|
||||
legacy_block = mod.ref_block(eff, curve=curve)
|
||||
return _filter_legacy_ref_block(legacy_block, use_video=use_video, use_audio=use_audio)
|
||||
|
||||
|
||||
def _ref_blocks(mods, retention, curve=None, seed=-1,
|
||||
use_video: bool = True, use_audio: bool = True) -> List[Dict]:
|
||||
"""Ref blocks for a loader bundle, scaled by row strength x retention.
|
||||
|
||||
``retention`` is a master strength multiplier: a float 0-1 (1.0 =
|
||||
fully_preserved, 0.7 = partially_preserved, 0.4 = attribute_transfer,
|
||||
0.15 = weak_reference), or one of those preset names for legacy
|
||||
workflows saved with the old combo widget.
|
||||
|
||||
``curve`` (optional) is a per-frame strength spec — a ``(direction,
|
||||
shape, value)`` tuple, a legacy preset name, per-frame values, control
|
||||
points (see ``core.curve_strengths``) — applied on top of the row
|
||||
strength. A flat/no curve keeps today's behavior.
|
||||
|
||||
``seed`` (default -1 = off) enables ref scrambling: with 2+ refs in the
|
||||
bundle, the order is shuffled and a random subset kept, so a different
|
||||
ref leads each run instead of the same one always "popping". Same seed
|
||||
-> same scramble; connect/randomize the seed for per-run variation.
|
||||
"""
|
||||
if isinstance(retention, str):
|
||||
factor = RETENTION.get(retention, 1.0)
|
||||
else:
|
||||
factor = float(retention)
|
||||
items = [] if mods is None else list(mods)
|
||||
if int(seed) >= 0 and len(items) > 1:
|
||||
rng = random.Random(int(seed))
|
||||
rng.shuffle(items)
|
||||
keep = rng.randint(max(1, len(items) // 2), len(items))
|
||||
items = items[:keep]
|
||||
print(f"[H3RefModApply] scramble seed={int(seed)}: "
|
||||
f"{len(mods)} refs -> kept {len(items)} (order shuffled)")
|
||||
blocks = []
|
||||
debug_rows = []
|
||||
for mod, strength in items:
|
||||
eff = min(1.0, max(0.0, strength * factor))
|
||||
block = _build_ref_block(mod, eff, curve=curve, use_video=use_video, use_audio=use_audio)
|
||||
if block is not None:
|
||||
blocks.append(block)
|
||||
debug_rows.append(f"{mod.name}@{eff:.2f}")
|
||||
else:
|
||||
debug_rows.append(f"{mod.name}@{eff:.2f} (skipped)")
|
||||
if debug_rows:
|
||||
print("[H3RefModApply] effective refs: " + ", ".join(debug_rows))
|
||||
return blocks
|
||||
|
||||
|
||||
class H3RefModApplySimple(io.ComfyNode):
|
||||
"""Identity-oriented RefMod apply node with one strength dial."""
|
||||
|
||||
@classmethod
|
||||
def define_schema(cls):
|
||||
template = io.MatchType.Template(
|
||||
"cond",
|
||||
allowed_types=[io.Custom("MINIMAX_H3_COND"), io.Conditioning])
|
||||
return io.Schema(
|
||||
node_id="H3RefModApplySimple",
|
||||
display_name="Apply H3 RefMod Simple",
|
||||
description=(
|
||||
"Apply one or more RefMods to a MiniMax H3 conditioning with a single "
|
||||
"Strength control. This identity-oriented version uses a flat full-length "
|
||||
"reference curve under the hood."
|
||||
),
|
||||
category="H3RefModPicker",
|
||||
inputs=[
|
||||
io.MatchType.Input("conditioning", template=template,
|
||||
tooltip="MINIMAX_H3_COND (ComfyUI-MiniMaxH3 pack) or CONDITIONING "
|
||||
"(core MiniMaxH3ReferenceToVideo)."),
|
||||
io.Custom("H3_REF_MODS").Input("mods",
|
||||
optional=True,
|
||||
tooltip="Bundle of RefMods to inject. Leave unconnected to bypass unchanged."),
|
||||
io.Boolean.Input("use_video", default=True,
|
||||
tooltip="Apply the visual latent from each RefMod. Turn this off to use only embedded audio."),
|
||||
io.Boolean.Input("use_audio", default=True,
|
||||
tooltip="Apply the embedded audio latent from each RefMod when present. Turn this off for visual-only application."),
|
||||
io.Float.Input("strength", default=1.0, min=0.0, max=1.0, step=0.01,
|
||||
tooltip="Master reference strength. For identity work this is the main "
|
||||
"dial: higher = tighter identity lock, lower = more freedom but "
|
||||
"more drift."),
|
||||
],
|
||||
outputs=[
|
||||
io.MatchType.Output(template=template, display_name="conditioning",
|
||||
tooltip="The conditioning with the ref blocks injected, same type as the input."),
|
||||
],
|
||||
)
|
||||
|
||||
@classmethod
|
||||
def execute(cls, conditioning, mods, use_video=True, use_audio=True, strength=1.0):
|
||||
if mods is None:
|
||||
print("[H3RefModApplySimple] no mods connected — bypassing conditioning unchanged")
|
||||
return io.NodeOutput(conditioning)
|
||||
if not use_video and not use_audio:
|
||||
print("[H3RefModApplySimple] use_video=False and use_audio=False — bypassing conditioning unchanged")
|
||||
return io.NodeOutput(conditioning)
|
||||
blocks = _ref_blocks(mods, strength, ("constant", "linear", 1.0), seed=-1,
|
||||
use_video=bool(use_video), use_audio=bool(use_audio))
|
||||
if isinstance(conditioning, list):
|
||||
out = []
|
||||
for t in conditioning:
|
||||
d = dict(t[1])
|
||||
d["minimax_refs"] = list(d.get("minimax_refs", [])) + blocks
|
||||
out.append([t[0], d])
|
||||
print(f"[H3RefModApplySimple] strength={strength} "
|
||||
f"({len(blocks)} ref block(s) injected)")
|
||||
else:
|
||||
out = replace(conditioning, refs=list(conditioning.refs) + blocks)
|
||||
print(f"[H3RefModApplySimple] strength={strength} "
|
||||
f"({len(blocks)} ref block(s) injected, {len(out.refs)} total)")
|
||||
return io.NodeOutput(out)
|
||||
|
||||
|
||||
NODE_CLASS_MAPPINGS = {
|
||||
"H3RefModApplySimple": H3RefModApplySimple,
|
||||
}
|
||||
|
||||
NODE_DISPLAY_NAME_MAPPINGS = {
|
||||
"H3RefModApplySimple": "Apply H3 RefMod Simple",
|
||||
}
|
||||
|
||||
__all__ = ["NODE_CLASS_MAPPINGS", "NODE_DISPLAY_NAME_MAPPINGS"]
|
||||
@@ -0,0 +1,186 @@
|
||||
"""
|
||||
H3RefModApplyAxis — choose between two picker bundles with separate video/audio signed values
|
||||
Think of it as creating a Slider from two RefMods
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
from typing import List, Optional, Tuple
|
||||
|
||||
from ..py.refmod_common import (
|
||||
_prompt_hint,
|
||||
)
|
||||
|
||||
from ..py.refmod_core import (
|
||||
H3RefMod,
|
||||
)
|
||||
_PACK_DIR = os.path.dirname(os.path.abspath(__file__))
|
||||
LEGACY_MODS_DIR = os.path.join(_PACK_DIR, "mods") # pre-models/refmods storage, still read
|
||||
|
||||
|
||||
# reference retention presets (master strength multiplier on Apply)
|
||||
RETENTION = {
|
||||
"fully_preserved": 1.0,
|
||||
"partially_preserved": 0.7,
|
||||
"attribute_transfer": 0.4,
|
||||
"weak_reference": 0.15,
|
||||
}
|
||||
|
||||
def _info_lines(mod):
|
||||
opt = mod.optimize_steps
|
||||
if mod.mode == "encode":
|
||||
opt = f"n/a ({mod.optimize_steps} — encode mode stores the actual encode)"
|
||||
return [
|
||||
"=" * 52,
|
||||
f" MiniMax H3 RefMod: {mod.name}",
|
||||
f" {'concept_type':<18} {mod.concept_type}",
|
||||
f" {'mode':<18} {mod.mode}",
|
||||
f" {'kind':<18} {mod.kind}",
|
||||
f" {'latent':<18} {tuple(mod.latent.shape)}",
|
||||
f" {'tokens injected':<18} {mod.token_count}",
|
||||
f" {'source':<18} {mod.source} ({mod.source_shape})",
|
||||
f" {'pool':<18} {mod.pool}",
|
||||
f" {'identity':<18} {opt}",
|
||||
f" {'tags':<18} {', '.join(mod.tags) if mod.tags else '-'}",
|
||||
f" {'description':<18} {mod.description or '-'}",
|
||||
"=" * 52,
|
||||
]
|
||||
|
||||
def _mod_group_key(mod: H3RefMod) -> str:
|
||||
name = str(getattr(mod, "name", "") or "")
|
||||
lower_name = name.lower()
|
||||
for suffix in ("_Video", "_Audio"):
|
||||
if lower_name.endswith(suffix.lower()):
|
||||
return name[:-len(suffix)]
|
||||
return name
|
||||
|
||||
|
||||
def _split_trailing_selection(rows, label: str) -> Tuple[List[Tuple[H3RefMod, float]], Optional[Tuple[H3RefMod, float]], Optional[Tuple[H3RefMod, float]]]:
|
||||
"""Return the last logical RefMod selection from a bundle."""
|
||||
items = [] if rows is None else list(rows)
|
||||
if not items:
|
||||
return [], None, None
|
||||
last_key = _mod_group_key(items[-1][0])
|
||||
start = len(items) - 1
|
||||
while start > 0 and _mod_group_key(items[start - 1][0]) == last_key:
|
||||
start -= 1
|
||||
group = items[start:]
|
||||
if start > 0:
|
||||
print(f"[H3RefModApplyAxis] warning: {label} input contains {start} earlier row(s); using only the last RefMod selection '{last_key}'.")
|
||||
video_row = None
|
||||
audio_row = None
|
||||
for row in group:
|
||||
mod, _strength = row
|
||||
if mod.kind == "audio":
|
||||
audio_row = row
|
||||
else:
|
||||
video_row = row
|
||||
return group, video_row, audio_row
|
||||
|
||||
|
||||
class H3RefModApplyAxis:
|
||||
"""Choose between two picker bundles with separate video/audio signed values."""
|
||||
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
return {
|
||||
"required": {
|
||||
"mods_a": ("H3_REF_MODS", {
|
||||
"tooltip": "RefMod bundle from one Visual RefMod Picker. If older chained selections are present too, only the last logical RefMod selection is used.",
|
||||
}),
|
||||
"mods_b": ("H3_REF_MODS", {
|
||||
"tooltip": "RefMod bundle from another Visual RefMod Picker. If older chained selections are present too, only the last logical RefMod selection is used.",
|
||||
}),
|
||||
"video_value": ("FLOAT", {
|
||||
"default": 0.0, "min": -1.0, "max": 1.0, "step": 0.01,
|
||||
"display": "number",
|
||||
"tooltip": "Signed video selector: negative uses A video, positive uses B video. Magnitude becomes the output video strength. 0 skips video.",
|
||||
}),
|
||||
"audio_value": ("FLOAT", {
|
||||
"default": 0.0, "min": -1.0, "max": 1.0, "step": 0.01,
|
||||
"display": "number",
|
||||
"tooltip": "Signed audio selector: negative uses A audio, positive uses B audio. If only one side has audio, that audio is used whenever this is non-zero.",
|
||||
}),
|
||||
"show_info": ("BOOLEAN", {
|
||||
"default": False,
|
||||
"tooltip": "Print the selected output RefMods and strengths to the console.",
|
||||
}),
|
||||
},
|
||||
}
|
||||
|
||||
RETURN_TYPES = ("H3_REF_MODS", "STRING")
|
||||
RETURN_NAMES = ("mods", "prompt_hint")
|
||||
FUNCTION = "select"
|
||||
CATEGORY = "H3RefModPicker"
|
||||
|
||||
def select(self, mods_a, mods_b, video_value=0.0, audio_value=0.0, show_info=False):
|
||||
_group_a, video_a, audio_a = _split_trailing_selection(mods_a, "A")
|
||||
_group_b, video_b, audio_b = _split_trailing_selection(mods_b, "B")
|
||||
|
||||
loads: List[Tuple[H3RefMod, float]] = []
|
||||
|
||||
def pick_signed(value: float, row_a, row_b, kind: str):
|
||||
mag = min(1.0, abs(float(value)))
|
||||
if mag < 1e-6:
|
||||
return None
|
||||
chosen = row_b if value > 0 else row_a
|
||||
chosen_side = "B" if value > 0 else "A"
|
||||
fallback = row_a if value > 0 else row_b
|
||||
fallback_side = "A" if value > 0 else "B"
|
||||
if chosen is None:
|
||||
if fallback is not None:
|
||||
print(f"[H3RefModApplyAxis] warning: {kind} {chosen_side} is missing; using {fallback_side} {kind} at {mag:.2f}.")
|
||||
return fallback[0], mag
|
||||
print(f"[H3RefModApplyAxis] warning: no {kind} RefMod available on either side.")
|
||||
return None
|
||||
return chosen[0], mag
|
||||
|
||||
video_pick = pick_signed(video_value, video_a, video_b, "video")
|
||||
if video_pick is not None:
|
||||
loads.append(video_pick)
|
||||
|
||||
if audio_a is not None and audio_b is not None:
|
||||
audio_pick = pick_signed(audio_value, audio_a, audio_b, "audio")
|
||||
elif audio_a is not None or audio_b is not None:
|
||||
mag = min(1.0, abs(float(audio_value)))
|
||||
if mag < 1e-6:
|
||||
audio_pick = None
|
||||
else:
|
||||
only = audio_a or audio_b
|
||||
side = "A" if audio_a is not None else "B"
|
||||
print(f"[H3RefModApplyAxis] audio exists only on {side}; using that audio at {mag:.2f}.")
|
||||
audio_pick = (only[0], mag)
|
||||
else:
|
||||
audio_pick = None
|
||||
if audio_pick is not None:
|
||||
loads.append(audio_pick)
|
||||
|
||||
if loads:
|
||||
print("[H3RefModApplyAxis] " + ", ".join(
|
||||
f"{mod.name}@{strength:.2f}" for mod, strength in loads
|
||||
))
|
||||
else:
|
||||
print("[H3RefModApplyAxis] no output refs selected")
|
||||
if show_info:
|
||||
for mod, strength in loads:
|
||||
print("\n".join(_info_lines(mod)))
|
||||
print(f" {'strength':<18} {strength:.2f}")
|
||||
hint = _prompt_hint(loads)
|
||||
if hint:
|
||||
print(f"[H3RefModApplyAxis] prompt_hint: {hint}")
|
||||
return (loads, hint)
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════════════
|
||||
# Registration
|
||||
# ═══════════════════════════════════════════════════════════════════════════
|
||||
|
||||
NODE_CLASS_MAPPINGS = {
|
||||
"H3RefModApplyAxis": H3RefModApplyAxis,
|
||||
}
|
||||
|
||||
NODE_DISPLAY_NAME_MAPPINGS = {
|
||||
"H3RefModApplyAxis": "Apply H3 RefMod Axis",
|
||||
}
|
||||
|
||||
__all__ = ["NODE_CLASS_MAPPINGS", "NODE_DISPLAY_NAME_MAPPINGS"]
|
||||
@@ -0,0 +1,116 @@
|
||||
"""
|
||||
decode_refmod.py — H3RefModDecode node.
|
||||
|
||||
Decode a saved RefMod's latent back into images, so you can see what the mod
|
||||
actually captured. Purely a debugging / inspection tool — it plays no part in
|
||||
generation.
|
||||
|
||||
A RefMod stores a video-VAE latent [1, 24, T, h, w] (the reference). This node
|
||||
runs it back through the H3 video VAE decoder and returns the frames as an IMAGE
|
||||
batch [T, H, W, 3], so you can preview a mod, sanity-check an extraction, or
|
||||
eyeball what a pooled (training-mode) mod actually kept vs an encode-mode one.
|
||||
|
||||
It decodes the full latent timeline exactly as stored in the RefMod; it does
|
||||
not resample or guess a frame subset.
|
||||
|
||||
Note: a training-mode (pooled) mod decodes to a tiny, blurry grid — that's the
|
||||
point of the mode, it only stores a concept thumbnail. An encode-mode mod
|
||||
decodes to recognizable full-res frames. Audio (if the mod has it) is not
|
||||
decoded here — this is the visual reference only.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import torch
|
||||
import torch.nn.functional as F
|
||||
|
||||
import comfy.utils
|
||||
from comfy_api.latest import io
|
||||
|
||||
from .refmod_loader import _list_mod_names, _load_mod
|
||||
|
||||
|
||||
def _reshape_decoded_frames(frames: torch.Tensor) -> torch.Tensor:
|
||||
if frames.dim() == 5: # [B, T, H, W, C] -> [B*T, H, W, C]
|
||||
frames = frames.reshape(-1, frames.shape[-3], frames.shape[-2], frames.shape[-1])
|
||||
return frames
|
||||
|
||||
|
||||
def _upscale_for_inspection(frames: torch.Tensor, min_edge: int, mode: str) -> torch.Tensor:
|
||||
if min_edge <= 0 or frames.dim() != 4 or frames.shape[0] <= 0:
|
||||
return frames
|
||||
height, width = int(frames.shape[1]), int(frames.shape[2])
|
||||
shortest = min(height, width)
|
||||
if shortest <= 0 or shortest >= min_edge:
|
||||
return frames
|
||||
scale = float(min_edge) / float(shortest)
|
||||
target_h = max(1, int(round(height * scale)))
|
||||
target_w = max(1, int(round(width * scale)))
|
||||
nchw = frames.movedim(-1, 1)
|
||||
if mode == "training":
|
||||
upscaled = F.interpolate(nchw, size=(target_h, target_w), mode="nearest")
|
||||
else:
|
||||
upscaled = comfy.utils.common_upscale(nchw, target_w, target_h, "lanczos", "disabled")
|
||||
return upscaled.movedim(1, -1)
|
||||
|
||||
|
||||
class H3RefModDecode(io.ComfyNode):
|
||||
"""Decode a RefMod's latent back to images for inspection."""
|
||||
|
||||
@classmethod
|
||||
def define_schema(cls):
|
||||
return io.Schema(
|
||||
node_id="H3RefModDecode",
|
||||
display_name="Decode H3 RefMod",
|
||||
category="H3RefMod",
|
||||
description="Decode a saved RefMod's latent back into images so you can see "
|
||||
"what it captured. Inspection only — not part of generation. "
|
||||
"The full stored latent timeline is decoded as-is; no frame "
|
||||
"sampling is applied. "
|
||||
"An encode-mode mod decodes to recognizable frames; a "
|
||||
"training-mode (pooled) mod decodes to a tiny blurry grid "
|
||||
"(that's all it stores). Audio is not decoded.",
|
||||
inputs=[
|
||||
io.Combo.Input("mod", options=_list_mod_names(),
|
||||
tooltip="The RefMod to decode (from models/refmods/)."),
|
||||
io.Vae.Input("vae",
|
||||
tooltip="The MiniMax H3 video VAE."),
|
||||
io.Int.Input("inspect_min_edge", default=320, min=0, max=2048, step=32,
|
||||
tooltip="Upscale small decoded previews so pooled/training mods are easier "
|
||||
"to inspect. 0 disables inspection upscaling. Training-mode uses "
|
||||
"nearest-neighbor to keep the stored cell structure visible."),
|
||||
],
|
||||
outputs=[
|
||||
io.Image.Output("images", display_name="images",
|
||||
tooltip="The decoded reference frames [T, H, W, 3]."),
|
||||
],
|
||||
)
|
||||
|
||||
@classmethod
|
||||
def execute(cls, mod, vae, inspect_min_edge=320) -> io.NodeOutput:
|
||||
m = _load_mod(mod)
|
||||
if m.kind == "audio":
|
||||
raise ValueError("H3RefModDecode only decodes visual RefMods; audio-only mods are not supported.")
|
||||
z = m.latent # [1, 24, T, h, w]
|
||||
total_t = z.shape[2]
|
||||
|
||||
with torch.no_grad():
|
||||
frames = vae.decode(z)
|
||||
# video latent decodes channel-last; frame count rides the batch dim, so a
|
||||
# 5-dim result is reshaped like the core VAEDecode node does
|
||||
frames = _reshape_decoded_frames(frames)
|
||||
frames = frames.clamp(0.0, 1.0).float().cpu()
|
||||
frames = _upscale_for_inspection(frames, int(inspect_min_edge), m.mode)
|
||||
|
||||
print(f"[H3RefModDecode] '{m.name}' ({m.mode}, {m.kind}, "
|
||||
f"latent_t={total_t}, inspect_min_edge={inspect_min_edge}) -> {tuple(frames.shape)}")
|
||||
return io.NodeOutput(frames)
|
||||
|
||||
|
||||
NODE_CLASS_MAPPINGS = {
|
||||
"H3RefModDecode": H3RefModDecode,
|
||||
}
|
||||
|
||||
NODE_DISPLAY_NAME_MAPPINGS = {
|
||||
"H3RefModDecode": "Decode H3 RefMod",
|
||||
}
|
||||
@@ -0,0 +1,283 @@
|
||||
"""
|
||||
H3RefModLoader — load one RefMod and optionally append it to another
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any, Dict, List, Tuple
|
||||
import os
|
||||
import folder_paths
|
||||
import server
|
||||
|
||||
from ..py.refmod_core import H3RefMod, load_refmods_from_file, read_refmod_meta, refmod_capabilities
|
||||
|
||||
from ..py.refmod_common import (
|
||||
_prompt_hint,
|
||||
)
|
||||
|
||||
|
||||
from ..py.refmod_common import (
|
||||
refmods_dir,
|
||||
refmods_dirs,
|
||||
)
|
||||
|
||||
_MOD_CACHE: Dict[str, Tuple[H3RefMod, ...]] = {}
|
||||
_MOD_CACHE_MAX = 24 # cap: never pin more mods in RAM than this (FIFO eviction)
|
||||
_MOD_LIST_CACHE_KEY = None # (dirs, mtimes, sizes) signature of the last _list_mod_names() scan
|
||||
_MOD_LIST_CACHE_VAL = None
|
||||
_MOD_PATH_CACHE_KEY = None
|
||||
_MOD_PATH_CACHE_VAL: Dict[str, str] = {}
|
||||
_MOD_SKIP_DIRS = {"graph_presets", ".git", "__pycache__"}
|
||||
_MAX_WEIGHT = 10.0
|
||||
|
||||
# Mod storage lives in ComfyUI's models/ tree (created on first run) and is
|
||||
# registered as a first-class folder type so it shows up next to loras/unet.
|
||||
try:
|
||||
folder_paths.add_model_folder_path("refmods", refmods_dir())
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════════════
|
||||
# Mods folder helpers
|
||||
# ═══════════════════════════════════════════════════════════════════════════
|
||||
|
||||
|
||||
def _iter_mod_paths(base_dir: str):
|
||||
"""Yield ``(relative_stem, absolute_stem)`` for RefMods under ``base_dir``."""
|
||||
for root, dirnames, filenames in os.walk(base_dir):
|
||||
dirnames[:] = sorted(d for d in dirnames if d not in _MOD_SKIP_DIRS)
|
||||
for fn in sorted(filenames):
|
||||
if not fn.endswith(".safetensors"):
|
||||
continue
|
||||
abs_stem = os.path.join(root, fn[:-len(".safetensors")])
|
||||
rel_stem = os.path.relpath(abs_stem, base_dir).replace(os.sep, "/")
|
||||
yield rel_stem, abs_stem
|
||||
|
||||
|
||||
def _mod_scan_signature() -> tuple[str, List[str]]:
|
||||
mods_dirs = refmods_dirs()
|
||||
sig = []
|
||||
for mods_dir in mods_dirs:
|
||||
for rel_stem, abs_stem in _iter_mod_paths(mods_dir):
|
||||
try:
|
||||
st = os.stat(abs_stem + ".safetensors")
|
||||
sig.append(f"{mods_dir}:{rel_stem}:{st.st_size}:{int(st.st_mtime)}")
|
||||
except OSError:
|
||||
pass
|
||||
return "\n".join(sig), mods_dirs
|
||||
|
||||
|
||||
def _root_display_labels(roots: List[str]) -> List[str]:
|
||||
labels: List[str] = []
|
||||
counts: Dict[str, int] = {}
|
||||
for index, root in enumerate(roots, start=1):
|
||||
base = os.path.basename(os.path.normpath(root)) or f"root{index}"
|
||||
count = counts.get(base, 0) + 1
|
||||
counts[base] = count
|
||||
labels.append(base if count == 1 else f"{base}#{count}")
|
||||
return labels
|
||||
|
||||
|
||||
def _refresh_mod_lookup() -> Dict[str, str]:
|
||||
global _MOD_LIST_CACHE_KEY, _MOD_LIST_CACHE_VAL, _MOD_PATH_CACHE_KEY, _MOD_PATH_CACHE_VAL
|
||||
|
||||
key, mods_dirs = _mod_scan_signature()
|
||||
if key == _MOD_LIST_CACHE_KEY and key == _MOD_PATH_CACHE_KEY and _MOD_LIST_CACHE_VAL is not None:
|
||||
return _MOD_PATH_CACHE_VAL
|
||||
|
||||
names_by_rel: Dict[str, List[Tuple[int, str]]] = {}
|
||||
for index, mods_dir in enumerate(mods_dirs):
|
||||
for rel_stem, abs_stem in _iter_mod_paths(mods_dir):
|
||||
meta = read_refmod_meta(abs_stem)
|
||||
if meta is None or meta.get("kind") not in ("image", "video", "audio", "bundle"):
|
||||
continue
|
||||
names_by_rel.setdefault(rel_stem, []).append((index, abs_stem))
|
||||
|
||||
labels = _root_display_labels(mods_dirs)
|
||||
lookup: Dict[str, str] = {}
|
||||
names: List[str] = []
|
||||
for rel_stem in sorted(names_by_rel, key=str.lower):
|
||||
matches = names_by_rel[rel_stem]
|
||||
first_index, first_path = matches[0]
|
||||
lookup[rel_stem] = first_path
|
||||
names.append(rel_stem)
|
||||
for dup_index, dup_path in matches[1:]:
|
||||
display_name = f"[{labels[dup_index]}] {rel_stem}"
|
||||
suffix = 2
|
||||
while display_name in lookup:
|
||||
display_name = f"[{labels[dup_index]} #{suffix}] {rel_stem}"
|
||||
suffix += 1
|
||||
lookup[display_name] = dup_path
|
||||
names.append(display_name)
|
||||
|
||||
_MOD_LIST_CACHE_KEY = key
|
||||
_MOD_LIST_CACHE_VAL = names
|
||||
_MOD_PATH_CACHE_KEY = key
|
||||
_MOD_PATH_CACHE_VAL = lookup
|
||||
return lookup
|
||||
|
||||
|
||||
def _list_mod_names() -> List[str]:
|
||||
"""Available RefMod names across the search dirs (for the loader dropdown).
|
||||
|
||||
Only entries with valid RefMod metadata (embedded in the safetensors header
|
||||
or a legacy sidecar .json) are listed, so other mod formats in
|
||||
models/mods/ (e.g. LTXMod files) don't show up.
|
||||
|
||||
Called by INPUT_TYPES/VALIDATE_INPUTS on every prompt validation, so the
|
||||
result is cached until any mod file appears/disappears/changes (checked
|
||||
via cheap os.stat, not by re-reading every safetensors header).
|
||||
"""
|
||||
_refresh_mod_lookup()
|
||||
return _MOD_LIST_CACHE_VAL
|
||||
|
||||
|
||||
def _find_mod_path(name: str) -> str:
|
||||
lookup = _refresh_mod_lookup()
|
||||
if name in lookup:
|
||||
return lookup[name]
|
||||
mods_dirs = refmods_dirs()
|
||||
raise FileNotFoundError(
|
||||
f"RefMod '{name}' not found. Searched:\n" +
|
||||
"\n".join(f" - {d}/{name}.safetensors" for d in mods_dirs))
|
||||
|
||||
|
||||
def _load_mods(name: str) -> List[H3RefMod]:
|
||||
if name in _MOD_CACHE:
|
||||
return list(_MOD_CACHE[name])
|
||||
path = _find_mod_path(name)
|
||||
mods = tuple(load_refmods_from_file(path, device="cpu"))
|
||||
_MOD_CACHE[name] = mods
|
||||
if len(_MOD_CACHE) > _MOD_CACHE_MAX:
|
||||
_MOD_CACHE.pop(next(iter(_MOD_CACHE)))
|
||||
return list(mods)
|
||||
|
||||
|
||||
def _load_mod(name: str) -> H3RefMod:
|
||||
mods = _load_mods(name)
|
||||
for mod in mods:
|
||||
if mod.kind != "audio":
|
||||
return mod
|
||||
return mods[0]
|
||||
|
||||
|
||||
def _mod_capabilities(name: str) -> Dict[str, bool]:
|
||||
path = _find_mod_path(name)
|
||||
meta = read_refmod_meta(path)
|
||||
if meta is None:
|
||||
return {"has_visual": True, "has_audio": False}
|
||||
caps = refmod_capabilities(meta)
|
||||
return {
|
||||
"has_visual": bool(caps.get("has_visual", False)),
|
||||
"has_audio": bool(caps.get("has_audio", False)),
|
||||
}
|
||||
|
||||
|
||||
def _normalize_weight(weight: float) -> float:
|
||||
return min(_MAX_WEIGHT, max(0.0, float(weight)))
|
||||
|
||||
|
||||
def _expand_weight(weight: float) -> List[float]:
|
||||
clipped = _normalize_weight(weight)
|
||||
whole = int(clipped)
|
||||
remainder = clipped - whole
|
||||
strengths = [1.0] * whole
|
||||
if remainder > 1e-6:
|
||||
strengths.append(remainder)
|
||||
return strengths
|
||||
|
||||
|
||||
def _append_weighted_mod(rows: List[tuple[H3RefMod, float]], mod: H3RefMod, weight: float) -> float:
|
||||
clipped = _normalize_weight(weight)
|
||||
rows.extend((mod, strength) for strength in _expand_weight(clipped))
|
||||
return clipped
|
||||
|
||||
|
||||
def _weight_display(weight: float) -> str:
|
||||
clipped = _normalize_weight(weight)
|
||||
whole = int(clipped)
|
||||
remainder = clipped - whole
|
||||
if whole <= 0:
|
||||
return f"{clipped:.2f}"
|
||||
if remainder <= 1e-6:
|
||||
return f"x{whole}"
|
||||
return f"x{whole} + {remainder:.2f}"
|
||||
|
||||
class H3RefModLoader:
|
||||
"""Load one RefMod and append it to an existing bundle."""
|
||||
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
return {
|
||||
"required": {
|
||||
"mod": (_list_mod_names(), {"tooltip": "The RefMod to load."}),
|
||||
"weight": ("FLOAT", {"default": 1.0, "min": 0.0, "max": _MAX_WEIGHT, "step": 0.01,
|
||||
"display": "number",
|
||||
"tooltip": "Legacy unified weight control kept for workflow compatibility. New graphs should use video_weight and audio_weight."}),
|
||||
"video_weight": ("FLOAT", {"default": 1.0, "min": 0.0, "max": _MAX_WEIGHT, "step": 0.01,
|
||||
"display": "number",
|
||||
"tooltip": "Unified video weight control. 0 skips video. 0..1 behaves like the old strength control. Values above 1 repeat the same visual RefMod as extra copies."}),
|
||||
"audio_weight": ("FLOAT", {"default": 1.0, "min": 0.0, "max": _MAX_WEIGHT, "step": 0.01,
|
||||
"display": "number",
|
||||
"tooltip": "Unified audio weight control. 0 skips audio. 0..1 behaves like the old strength control. Values above 1 repeat the same audio RefMod as extra copies."}),
|
||||
},
|
||||
"optional": {
|
||||
"mods": ("H3_REF_MODS",),
|
||||
},
|
||||
}
|
||||
|
||||
RETURN_TYPES = ("H3_REF_MODS", "STRING")
|
||||
RETURN_NAMES = ("mods", "prompt_hint")
|
||||
FUNCTION = "load"
|
||||
CATEGORY = "H3RefModPicker"
|
||||
|
||||
@classmethod
|
||||
def VALIDATE_INPUTS(cls, mod, **kwargs):
|
||||
if mod not in set(_list_mod_names()):
|
||||
return f"RefMod '{mod}' not found in any configured refmods folder. Run Create H3 RefMod first."
|
||||
return True
|
||||
|
||||
def load(self, mod, weight=1.0, video_weight=1.0, audio_weight=1.0, mods=None, strength=None, audio_strength=None):
|
||||
rows = list(mods) if mods is not None else []
|
||||
if strength is not None:
|
||||
weight = strength
|
||||
if audio_strength is not None:
|
||||
audio_weight = audio_strength
|
||||
if strength is not None or (weight != 1.0 and video_weight == 1.0 and audio_weight == 1.0):
|
||||
video_weight = weight
|
||||
audio_weight = weight
|
||||
loaded = _load_mods(mod)
|
||||
summaries = []
|
||||
hint_rows = list(mods) if mods is not None else []
|
||||
for item in loaded:
|
||||
current_weight = audio_weight if item.kind == "audio" else video_weight
|
||||
clipped = _append_weighted_mod(rows, item, current_weight)
|
||||
summaries.append(f"{item.name}({_weight_display(clipped)})")
|
||||
if clipped > 0.0:
|
||||
hint_rows.append((item, min(1.0, clipped)))
|
||||
print(f"[H3RefModLoader] {mod}: " + ", ".join(summaries))
|
||||
hint = _prompt_hint(hint_rows)
|
||||
if hint:
|
||||
print(f"[H3RefModLoader] prompt_hint: {hint}")
|
||||
return (rows, hint)
|
||||
|
||||
|
||||
NODE_CLASS_MAPPINGS = {
|
||||
"H3RefModLoader": H3RefModLoader
|
||||
}
|
||||
|
||||
NODE_DISPLAY_NAME_MAPPINGS = {
|
||||
"H3RefModLoader": "Load H3 RefMod Simple"
|
||||
}
|
||||
|
||||
|
||||
@server.PromptServer.instance.routes.get("/h3refmods/refmod-loader/capabilities")
|
||||
async def refmod_loader_capabilities(request):
|
||||
try:
|
||||
name = str(request.query.get("mod", "") or "").strip()
|
||||
if not name:
|
||||
return server.web.json_response({"ok": True, "has_visual": True, "has_audio": True})
|
||||
caps = _mod_capabilities(name)
|
||||
return server.web.json_response({"ok": True, **caps})
|
||||
except Exception as exc:
|
||||
return server.web.json_response({"ok": False, "error": str(exc)}, status=500)
|
||||
@@ -0,0 +1,165 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
from typing import Dict, List, Tuple
|
||||
|
||||
import server
|
||||
|
||||
from ..py.refmod_browser import (
|
||||
PREVIEW_EXTS,
|
||||
browser_root,
|
||||
list_browser_dir,
|
||||
paired_mod_path,
|
||||
safe_file_path,
|
||||
)
|
||||
from ..py.refmod_core import H3RefMod, load_refmods_from_file
|
||||
from ..py.refmod_common import _prompt_hint
|
||||
from .refmod_loader import _MAX_WEIGHT, _append_weighted_mod, _weight_display
|
||||
|
||||
_VISUAL_MOD_CACHE: Dict[str, Tuple[H3RefMod, ...]] = {}
|
||||
_VISUAL_MOD_CACHE_MAX = 24
|
||||
|
||||
|
||||
def _load_mods_from_path(mod_path: str) -> List[H3RefMod]:
|
||||
path = safe_file_path(mod_path)
|
||||
if not path.lower().endswith(".safetensors"):
|
||||
raise ValueError("Selected file is not a RefMod safetensors file.")
|
||||
if path in _VISUAL_MOD_CACHE:
|
||||
return list(_VISUAL_MOD_CACHE[path])
|
||||
mods = list(load_refmods_from_file(path[:-len(".safetensors")], device="cpu"))
|
||||
if not any(mod.kind == "audio" for mod in mods):
|
||||
paired_audio = paired_mod_path(path[:-len(".safetensors")], "audio")
|
||||
if paired_audio is not None:
|
||||
mods.extend(load_refmods_from_file(paired_audio[:-len(".safetensors")], device="cpu"))
|
||||
_VISUAL_MOD_CACHE[path] = tuple(mods)
|
||||
if len(_VISUAL_MOD_CACHE) > _VISUAL_MOD_CACHE_MAX:
|
||||
_VISUAL_MOD_CACHE.pop(next(iter(_VISUAL_MOD_CACHE)))
|
||||
return mods
|
||||
|
||||
|
||||
class H3RefModVisualPicker:
|
||||
"""Pick a RefMod by browsing models/refmods thumbnails and append it to a bundle."""
|
||||
|
||||
NAME = "Visual RefMod Picker"
|
||||
CATEGORY = "H3RefModPicker"
|
||||
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
return {
|
||||
"required": {
|
||||
"mod_path": (
|
||||
"STRING",
|
||||
{
|
||||
"default": "",
|
||||
"tooltip": "Selected RefMod safetensors path from the thumbnail browser.",
|
||||
},
|
||||
),
|
||||
"video_weight": (
|
||||
"FLOAT",
|
||||
{
|
||||
"default": 1.0,
|
||||
"min": 0.0,
|
||||
"max": _MAX_WEIGHT,
|
||||
"step": 0.01,
|
||||
"tooltip": "Unified video weight control. 0 skips video. 0..1 behaves like the old strength control. Values above 1 repeat the same video RefMod as extra copies.",
|
||||
},
|
||||
),
|
||||
"audio_weight": (
|
||||
"FLOAT",
|
||||
{
|
||||
"default": 1.0,
|
||||
"min": 0.0,
|
||||
"max": _MAX_WEIGHT,
|
||||
"step": 0.01,
|
||||
"tooltip": "Unified audio weight control. 0 skips audio. 0..1 behaves like the old audio strength control. Values above 1 repeat the same audio RefMod as extra copies.",
|
||||
},
|
||||
),
|
||||
},
|
||||
"optional": {
|
||||
"mods": ("H3_REF_MODS",),
|
||||
},
|
||||
}
|
||||
|
||||
RETURN_TYPES = ("H3_REF_MODS", "STRING")
|
||||
RETURN_NAMES = ("mods", "prompt_hint")
|
||||
FUNCTION = "pick"
|
||||
DESCRIPTION = (
|
||||
"Pick a RefMod by browsing models/refmods and its subfolders. "
|
||||
"A matching preview image with the same base name is shown when present; "
|
||||
"otherwise a placeholder is used. When a matching *_Audio file exists "
|
||||
"beside a *_Video or *_Visual file, both are loaded together and appended to the bundle, "
|
||||
"with separate video/audio weights. A weight in 0..1 behaves like the old strength control; "
|
||||
"a weight above 1 repeats the same RefMod as extra copies. Legacy combined RefMods still use one shared video weight."
|
||||
)
|
||||
|
||||
@classmethod
|
||||
def VALIDATE_INPUTS(cls, **kwargs):
|
||||
return True
|
||||
|
||||
def pick(self, mod_path: str, video_weight: float = 1.0, audio_weight: float = 1.0,
|
||||
mods=None, strength=None, audio_strength=None):
|
||||
rows: List[Tuple[H3RefMod, float]] = []
|
||||
if mods is not None and isinstance(mods, (list, tuple)):
|
||||
rows.extend(list(mods))
|
||||
|
||||
selected = str(mod_path or "").strip()
|
||||
if not selected or selected.lower() == "(none)":
|
||||
hint = _prompt_hint(rows)
|
||||
return (rows, hint)
|
||||
|
||||
if strength is not None:
|
||||
video_weight = strength
|
||||
if audio_strength is not None:
|
||||
audio_weight = audio_strength
|
||||
selected_mods = _load_mods_from_path(selected)
|
||||
hint_rows = list(mods) if mods is not None and isinstance(mods, (list, tuple)) else []
|
||||
summaries = []
|
||||
for mod in selected_mods:
|
||||
weight = audio_weight if mod.kind == "audio" else video_weight
|
||||
clipped = _append_weighted_mod(rows, mod, weight)
|
||||
if clipped > 0.0:
|
||||
hint_rows.append((mod, min(1.0, clipped)))
|
||||
summaries.append(f"{mod.name}({_weight_display(clipped)})")
|
||||
print("[H3RefModVisualPicker] " + ", ".join(summaries))
|
||||
hint = _prompt_hint(hint_rows)
|
||||
if hint:
|
||||
print(f"[H3RefModVisualPicker] prompt_hint: {hint}")
|
||||
return (rows, hint)
|
||||
|
||||
|
||||
NODE_CLASS_MAPPINGS = {"H3RefModVisualPicker": H3RefModVisualPicker}
|
||||
NODE_DISPLAY_NAME_MAPPINGS = {"H3RefModVisualPicker": "Visual H3 RefMod Picker"}
|
||||
|
||||
|
||||
@server.PromptServer.instance.routes.get("/h3refmods/refmod-browser/root")
|
||||
async def refmod_browser_root(request):
|
||||
try:
|
||||
return server.web.json_response({"ok": True, "root": browser_root()})
|
||||
except Exception as exc:
|
||||
return server.web.json_response({"ok": False, "error": str(exc)}, status=500)
|
||||
|
||||
|
||||
@server.PromptServer.instance.routes.get("/h3refmods/refmod-browser/list")
|
||||
async def refmod_browser_list(request):
|
||||
try:
|
||||
data = list_browser_dir(request.query.get("path", "").strip())
|
||||
return server.web.json_response({"ok": True, **data})
|
||||
except ValueError as exc:
|
||||
return server.web.json_response({"ok": False, "error": str(exc)}, status=404)
|
||||
except Exception as exc:
|
||||
return server.web.json_response({"ok": False, "error": str(exc)}, status=500)
|
||||
|
||||
|
||||
@server.PromptServer.instance.routes.get("/h3refmods/refmod-browser/file")
|
||||
async def refmod_browser_file(request):
|
||||
try:
|
||||
path = safe_file_path(request.query.get("path", "").strip())
|
||||
if os.path.splitext(path)[1].lower() not in PREVIEW_EXTS:
|
||||
return server.web.json_response(
|
||||
{"ok": False, "error": "Unsupported file type"}, status=403
|
||||
)
|
||||
return server.web.FileResponse(path, headers={"Cache-Control": "no-cache"})
|
||||
except ValueError as exc:
|
||||
return server.web.json_response({"ok": False, "error": str(exc)}, status=404)
|
||||
except Exception as exc:
|
||||
return server.web.json_response({"ok": False, "error": str(exc)}, status=500)
|
||||
@@ -0,0 +1,293 @@
|
||||
"""Native H3 reference conditioning with grounded RefMod stacks."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import io as bytes_io
|
||||
import math
|
||||
|
||||
import torch
|
||||
import torch.nn.functional as functional
|
||||
from comfy_api.latest import io
|
||||
|
||||
REFERENCE_TYPE = "H3_REFMOD_REFERENCES"
|
||||
|
||||
|
||||
def _mod_rows(mods):
|
||||
if mods is None:
|
||||
return []
|
||||
if hasattr(mods, "ref_block"):
|
||||
return [(mods, 1.0)]
|
||||
if isinstance(mods, (tuple, list)) and mods and hasattr(mods[0], "ref_block"):
|
||||
mods = [mods]
|
||||
rows = []
|
||||
for row in mods:
|
||||
if not isinstance(row, (tuple, list)) or len(row) not in (2, 3):
|
||||
raise ValueError("mods must contain (RefMod, strength) rows.")
|
||||
mod, strength = row[:2]
|
||||
if mod is None:
|
||||
continue
|
||||
if not callable(getattr(mod, "ref_block", None)):
|
||||
raise ValueError("mods contains an unsupported RefMod object.")
|
||||
strength = float(strength)
|
||||
if not math.isfinite(strength):
|
||||
raise ValueError("RefMod strengths must be finite.")
|
||||
if strength > 0:
|
||||
rows.append((mod, min(1.0, strength)))
|
||||
return rows
|
||||
|
||||
|
||||
def _decoded_frames(video_vae, latent):
|
||||
with torch.no_grad():
|
||||
frames = video_vae.decode(latent)
|
||||
if frames.ndim == 5 and frames.shape[0] == 1:
|
||||
frames = frames[0]
|
||||
if frames.ndim != 4 or frames.shape[-1] != 3 or frames.shape[0] < 1:
|
||||
raise ValueError(f"H3 video VAE returned invalid images: {tuple(frames.shape)}")
|
||||
return frames.detach().float().clamp(0, 1).cpu()
|
||||
|
||||
|
||||
def _encoder_frames(mod, video_vae, reference_fps):
|
||||
times = getattr(mod, "enc_times", None)
|
||||
path = getattr(mod, "path", "")
|
||||
packed_frames = getattr(mod, "enc_frames", None)
|
||||
is_stack = getattr(mod, "source", "") == "stack"
|
||||
if times and (packed_frames or path) and (is_stack or abs(getattr(mod, "enc_fps", 0) - reference_fps) < 1e-6):
|
||||
import numpy as np
|
||||
from PIL import Image
|
||||
def unpack(get_tensor):
|
||||
frames = []
|
||||
for index in range(len(times)):
|
||||
packed = get_tensor(f"enc_{index}").numpy().tobytes()
|
||||
with Image.open(bytes_io.BytesIO(packed)) as image:
|
||||
frames.append(torch.from_numpy(np.array(image.convert("RGB"))).float() / 255)
|
||||
return torch.stack(frames), list(times), "stored pictures"
|
||||
|
||||
if packed_frames:
|
||||
return unpack(packed_frames.__getitem__)
|
||||
from safetensors import safe_open
|
||||
filename = path if path.endswith(".safetensors") else path + ".safetensors"
|
||||
with safe_open(filename, framework="pt", device="cpu") as handle:
|
||||
return unpack(handle.get_tensor)
|
||||
if video_vae is None:
|
||||
raise ValueError(f"Connect video_vae to reconstruct visual RefMod '{mod.name}'.")
|
||||
if is_stack:
|
||||
frames = [_decoded_frames(video_vae, mod.latent[:, :, index:index + 1])[:1]
|
||||
for index in range(mod.latent.shape[2])]
|
||||
return torch.cat(frames), [float(index) for index in range(len(frames))], "decoded stack"
|
||||
frames = _decoded_frames(video_vae, mod.latent)
|
||||
if mod.kind == "image":
|
||||
return frames[:1], [0.0], "decoded image"
|
||||
times = [index / 2 for index in range(math.ceil(frames.shape[0] * 2 / reference_fps))]
|
||||
indices = [min(round(time * reference_fps), frames.shape[0] - 1) for time in times]
|
||||
return frames[indices], times, "decoded video"
|
||||
|
||||
|
||||
def _visual_item(mod, frames, times, strength, stack_pictures, visual_kind=None):
|
||||
if strength < 1:
|
||||
samples = frames.movedim(-1, 1)
|
||||
blurred = functional.adaptive_avg_pool2d(samples, (max(1, mod.latent_h // 8),
|
||||
max(1, mod.latent_w // 8)))
|
||||
blurred = functional.interpolate(blurred, size=samples.shape[-2:], mode="bilinear", align_corners=False)
|
||||
frames = (strength * samples + (1 - strength) * blurred).movedim(1, -1)
|
||||
if (visual_kind or mod.kind) == "image":
|
||||
return {"type": "image", "data": frames[:1]}, 1
|
||||
if getattr(mod, "source", "") == "stack":
|
||||
count = frames.shape[0]
|
||||
if stack_pictures == "every 4th":
|
||||
indices = list(range(0, count, 4))
|
||||
frames = frames[indices]
|
||||
times = [index / 2 for index in range(len(indices))]
|
||||
else:
|
||||
indices = ([round(index * (count - 1) / 7) for index in range(8)]
|
||||
if stack_pictures == "up to 8" and count > 8 else list(range(count)))
|
||||
frames = frames[indices].repeat_interleave(2, dim=0)
|
||||
times = [index + offset for index in range(len(indices)) for offset in (0.0, 0.5)]
|
||||
shown = len(indices)
|
||||
else:
|
||||
shown = frames.shape[0]
|
||||
return {"type": "video", "data": frames, "timestamps": times}, shown
|
||||
|
||||
|
||||
class H3RefModReferences(io.ComfyNode):
|
||||
@classmethod
|
||||
def define_schema(cls):
|
||||
return io.Schema(
|
||||
node_id="H3RefModReferences",
|
||||
display_name="Collect H3 References",
|
||||
category="H3RefModPicker",
|
||||
inputs=[
|
||||
io.Custom(REFERENCE_TYPE).Input("references", optional=True,
|
||||
tooltip="Optional references from another collector."),
|
||||
io.Autogrow.Input("ref_images", optional=True, template=io.Autogrow.TemplatePrefix(
|
||||
input=io.Image.Input("ref_image"), prefix="ref_image_", min=0, max=9)),
|
||||
io.Autogrow.Input("ref_videos", optional=True, template=io.Autogrow.TemplatePrefix(
|
||||
input=io.Image.Input("ref_video", tooltip="Video frame batch at 24 fps."),
|
||||
prefix="ref_video_", min=0, max=3)),
|
||||
io.Autogrow.Input("ref_video_audios", optional=True, template=io.Autogrow.TemplatePrefix(
|
||||
input=io.Audio.Input("ref_video_audio", tooltip="Soundtrack for the same-numbered video input."),
|
||||
prefix="ref_video_audio_", min=0, max=3)),
|
||||
io.Autogrow.Input("ref_audios", optional=True, template=io.Autogrow.TemplatePrefix(
|
||||
input=io.Audio.Input("ref_audio"), prefix="ref_audio_", min=0, max=3)),
|
||||
],
|
||||
outputs=[io.Custom(REFERENCE_TYPE).Output("references")],
|
||||
)
|
||||
|
||||
@classmethod
|
||||
def execute(cls, references=None, ref_images=None, ref_videos=None,
|
||||
ref_video_audios=None, ref_audios=None):
|
||||
entries = list(references or [])
|
||||
for name, image in (ref_images or {}).items():
|
||||
if image is not None:
|
||||
entries.append({"kind": "image", "data": image, "name": name})
|
||||
videos = ref_videos or {}
|
||||
soundtracks = ref_video_audios or {}
|
||||
for name, soundtrack in soundtracks.items():
|
||||
if soundtrack is not None and videos.get("ref_video_" + name.rsplit("_", 1)[-1]) is None:
|
||||
raise ValueError(f"{name} needs its same-numbered ref_video input; use ref_audio for standalone audio.")
|
||||
for name, frames in videos.items():
|
||||
if frames is not None:
|
||||
entries.append({"kind": "video", "data": frames, "name": name,
|
||||
"audio": soundtracks.get("ref_video_audio_" + name.rsplit("_", 1)[-1])})
|
||||
for name, audio in (ref_audios or {}).items():
|
||||
if audio is not None:
|
||||
entries.append({"kind": "audio", "data": audio, "name": name})
|
||||
return io.NodeOutput(entries)
|
||||
|
||||
|
||||
class _ReferencePresentation:
|
||||
def __init__(self):
|
||||
self.items = []
|
||||
|
||||
def tokenize(self, prompt, minimax_ref_items):
|
||||
self.items = minimax_ref_items
|
||||
return None
|
||||
|
||||
def encode_from_tokens_scheduled(self, tokens):
|
||||
return [[None, {}]]
|
||||
|
||||
|
||||
def _native_references(references, video_vae, audio_vae, width, height, length, ref_image_size):
|
||||
from comfy_extras.nodes_minimax_h3 import MiniMaxH3ReferenceToVideo
|
||||
|
||||
images, videos, soundtracks, audios = {}, {}, {}, {}
|
||||
names = []
|
||||
for kind in ("image", "video", "audio"):
|
||||
for index, entry in enumerate(references or []):
|
||||
if entry.get("kind") != kind:
|
||||
continue
|
||||
name = entry["name"]
|
||||
if kind == "image":
|
||||
images[f"ref_image_{index}"] = entry["data"]
|
||||
elif kind == "video":
|
||||
videos[f"ref_video_{index}"] = entry["data"]
|
||||
if entry.get("audio") is not None:
|
||||
soundtracks[f"ref_video_audio_{index}"] = entry["audio"]
|
||||
names.append(name + " soundtrack")
|
||||
else:
|
||||
audios[f"ref_audio_{index}"] = entry["data"]
|
||||
names.append(name)
|
||||
presentation = _ReferencePresentation()
|
||||
output = MiniMaxH3ReferenceToVideo.execute(
|
||||
presentation, "", width, height, length, ref_image_size=ref_image_size,
|
||||
vae=video_vae, audio_vae=audio_vae, ref_images=images, ref_videos=videos,
|
||||
ref_video_audios=soundtracks, ref_audios=audios)
|
||||
conditioning, latent = output.result
|
||||
return presentation.items, list(conditioning[0][1].get("minimax_refs", [])), names, latent
|
||||
|
||||
|
||||
class H3RefModsToVideo(io.ComfyNode):
|
||||
@classmethod
|
||||
def define_schema(cls):
|
||||
return io.Schema(
|
||||
node_id="H3RefModsToVideo",
|
||||
display_name="MiniMax H3 RefMods to Video",
|
||||
category="model/conditioning/minimax",
|
||||
description="Reference-to-Video with grounded RefMods and optional collected media references.",
|
||||
inputs=[
|
||||
io.Clip.Input("clip"),
|
||||
io.Vae.Input("video_vae", optional=True,
|
||||
tooltip="Encodes raw pictures/videos and reconstructs RefMods without stored encoder pictures."),
|
||||
io.Vae.Input("audio_vae", optional=True,
|
||||
tooltip="Encodes raw reference audio. Audio RefMods are already encoded."),
|
||||
io.Custom(REFERENCE_TYPE).Input("references", optional=True),
|
||||
io.MultiType.Input("mods", types=[io.Custom("H3_REFMOD"), io.Custom("H3_REF_MODS")], optional=True),
|
||||
io.String.Input("prompt", multiline=True, dynamic_prompts=True),
|
||||
io.Int.Input("width", default=1344, min=32, max=8192, step=32),
|
||||
io.Int.Input("height", default=768, min=32, max=8192, step=32),
|
||||
io.Int.Input("length", default=124, min=5, max=3600, step=17),
|
||||
io.Combo.Input("ref_image_size", options=["match", "max"], default="match"),
|
||||
io.Combo.Input("stack_pictures", options=["every 4th", "up to 8", "all"], default="every 4th",
|
||||
tooltip="Pictures shown to the text encoder for a stacked RefMod. All latent frames remain applied."),
|
||||
],
|
||||
outputs=[io.Conditioning.Output(display_name="positive"), io.Latent.Output(),
|
||||
io.String.Output("reference_map")],
|
||||
)
|
||||
|
||||
@classmethod
|
||||
def execute(cls, clip, prompt, width, height, length, video_vae=None, audio_vae=None,
|
||||
references=None, mods=None, ref_image_size="match", stack_pictures="every 4th"):
|
||||
if stack_pictures not in ("every 4th", "up to 8", "all"):
|
||||
raise ValueError("Unknown stack_pictures mode.")
|
||||
items, blocks, names, latent = _native_references(
|
||||
references, video_vae, audio_vae, width, height, length, ref_image_size)
|
||||
counters = {"image": 0, "video": 0, "audio": 0}
|
||||
labels = {"image": "Picture", "video": "Video", "audio": "Audio"}
|
||||
mapping = []
|
||||
|
||||
def append_label(item, name):
|
||||
kind = item["type"]
|
||||
counters[kind] += 1
|
||||
mapping.append(f"<{labels[kind]} {counters[kind]}> = {name}")
|
||||
|
||||
for item, name in zip(items, names):
|
||||
append_label(item, name)
|
||||
decoded = {}
|
||||
for mod, strength in _mod_rows(mods):
|
||||
block = mod.ref_block(strength)
|
||||
if block is None:
|
||||
continue
|
||||
block = dict(block)
|
||||
kind = block["kind"]
|
||||
name = mod.name
|
||||
if kind not in ("image", "video", "video_audio", "audio"):
|
||||
raise ValueError(f"Unsupported RefMod kind: {kind}")
|
||||
if kind == "audio" or (block.get("audio_latent") is not None and block.get("ref_audio_t", 0) > 0):
|
||||
item = {"type": "audio"}
|
||||
items.append(item)
|
||||
append_label(item, name + (" soundtrack" if kind != "audio" else ""))
|
||||
if kind != "audio":
|
||||
if id(mod) not in decoded:
|
||||
decoded[id(mod)] = _encoder_frames(mod, video_vae, 24.0)
|
||||
frames, times, _source = decoded[id(mod)]
|
||||
visual_kind = "video" if kind == "video_audio" else kind
|
||||
item, _shown = _visual_item(mod, frames, times, strength, stack_pictures, visual_kind)
|
||||
items.append(item)
|
||||
append_label(item, name)
|
||||
block["refmod"] = True
|
||||
blocks.append(block)
|
||||
if video_vae is not None or audio_vae is not None:
|
||||
import comfy.model_management
|
||||
comfy.model_management.soft_empty_cache()
|
||||
tokens = clip.tokenize(prompt, minimax_ref_items=items)
|
||||
conditioning = clip.encode_from_tokens_scheduled(tokens)
|
||||
output = []
|
||||
for embedding, metadata in conditioning:
|
||||
if items and "minimax_token_tags" not in metadata:
|
||||
raise ValueError("Connect a MiniMax H3 CLIP that supports reference token tags.")
|
||||
metadata = dict(metadata)
|
||||
if blocks:
|
||||
metadata["minimax_refs"] = list(metadata.get("minimax_refs", [])) + blocks
|
||||
output.append([embedding, metadata])
|
||||
reference_map = "\n".join(mapping) or "No references."
|
||||
return io.NodeOutput(output, latent, reference_map)
|
||||
|
||||
|
||||
NODE_CLASS_MAPPINGS = {
|
||||
"H3RefModsToVideo": H3RefModsToVideo,
|
||||
"H3RefModReferences": H3RefModReferences,
|
||||
}
|
||||
NODE_DISPLAY_NAME_MAPPINGS = {
|
||||
"H3RefModsToVideo": "MiniMax H3 RefMods to Video",
|
||||
"H3RefModReferences": "Collect H3 References",
|
||||
}
|
||||
|
||||
@@ -0,0 +1,253 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
from typing import Dict, List, Optional
|
||||
|
||||
from .refmod_common import refmods_dir, refmods_dirs
|
||||
from .refmod_core import read_refmod_meta, refmod_capabilities
|
||||
|
||||
|
||||
PREVIEW_EXTS = (".png", ".jpg", ".jpeg", ".webp")
|
||||
HIDDEN_BROWSER_DIRS = {"graph_presets"}
|
||||
VISUAL_SUFFIXES = ("_visual", "_video")
|
||||
AUDIO_SUFFIXES = ("_audio",)
|
||||
VISUAL_FILE_SUFFIXES = ("_visual", "_Visual", "_video", "_Video")
|
||||
AUDIO_FILE_SUFFIXES = ("_audio", "_Audio")
|
||||
|
||||
|
||||
def browser_root() -> str:
|
||||
root = os.path.abspath(refmods_dir())
|
||||
os.makedirs(root, exist_ok=True)
|
||||
return root
|
||||
|
||||
|
||||
def browser_roots() -> List[str]:
|
||||
"""Every declared RefMod folder; the first one is browser_root()."""
|
||||
return refmods_dirs()
|
||||
|
||||
|
||||
def _browser_root_labels() -> Dict[str, str]:
|
||||
labels: Dict[str, str] = {}
|
||||
counts: Dict[str, int] = {}
|
||||
for index, root in enumerate(browser_roots(), start=1):
|
||||
base = os.path.basename(os.path.normpath(root)) or f"root{index}"
|
||||
count = counts.get(base, 0) + 1
|
||||
counts[base] = count
|
||||
labels[os.path.realpath(root)] = base if count == 1 else f"{base}#{count}"
|
||||
return labels
|
||||
|
||||
|
||||
def _is_root(path: str) -> bool:
|
||||
real = os.path.realpath(_safe_abspath(path))
|
||||
return any(real == os.path.realpath(root) for root in browser_roots())
|
||||
|
||||
|
||||
def _root_label_for_path(path: str) -> str:
|
||||
current = os.path.realpath(_safe_abspath(path))
|
||||
for root_real, label in _browser_root_labels().items():
|
||||
if current == root_real or current.startswith(root_real + os.sep):
|
||||
return label
|
||||
return "refmods"
|
||||
|
||||
|
||||
def _safe_abspath(path: str) -> str:
|
||||
return os.path.abspath(os.path.expanduser(path))
|
||||
|
||||
|
||||
def _resolve_browser_path(path: str) -> str:
|
||||
raw = str(path or "").strip()
|
||||
if not raw:
|
||||
return browser_root()
|
||||
expanded = os.path.expanduser(raw)
|
||||
if os.path.isabs(expanded):
|
||||
return os.path.abspath(expanded)
|
||||
normalized = raw.replace("\\", os.sep).replace("/", os.sep)
|
||||
for root in browser_roots():
|
||||
candidate = os.path.abspath(os.path.join(root, normalized))
|
||||
if os.path.exists(candidate):
|
||||
return candidate
|
||||
return os.path.abspath(os.path.join(browser_root(), normalized))
|
||||
|
||||
|
||||
def _is_under_root(path: str) -> bool:
|
||||
current = os.path.realpath(_safe_abspath(path))
|
||||
for root_dir in browser_roots():
|
||||
root = os.path.realpath(root_dir)
|
||||
if current == root or current.startswith(root + os.sep):
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def safe_dir_path(path: str = "") -> str:
|
||||
if not path:
|
||||
return browser_root()
|
||||
current = _resolve_browser_path(path)
|
||||
if not _is_under_root(current):
|
||||
raise ValueError("Path is outside models/refmods")
|
||||
if not os.path.isdir(current):
|
||||
raise ValueError("Folder not found")
|
||||
return current
|
||||
|
||||
|
||||
def safe_file_path(path: str) -> str:
|
||||
if not path:
|
||||
raise ValueError("Missing path")
|
||||
current = _resolve_browser_path(path)
|
||||
if not _is_under_root(current):
|
||||
raise ValueError("Path is outside models/refmods")
|
||||
if not os.path.isfile(current):
|
||||
raise ValueError("File not found")
|
||||
return current
|
||||
|
||||
|
||||
def _strip_known_suffix(path_no_ext: str) -> tuple[str, str | None, str | None]:
|
||||
name = os.path.basename(path_no_ext)
|
||||
lower_name = name.lower()
|
||||
for suffix in VISUAL_SUFFIXES:
|
||||
if lower_name.endswith(suffix):
|
||||
return (path_no_ext[:-len(suffix)], "visual", suffix)
|
||||
for suffix in AUDIO_SUFFIXES:
|
||||
if lower_name.endswith(suffix):
|
||||
return (path_no_ext[:-len(suffix)], "audio", suffix)
|
||||
return (path_no_ext, None, None)
|
||||
|
||||
|
||||
def paired_mod_path(path_no_ext: str, target_kind: str) -> Optional[str]:
|
||||
base, current_kind, _suffix = _strip_known_suffix(path_no_ext)
|
||||
if current_kind is None:
|
||||
return None
|
||||
current_path = path_no_ext + ".safetensors"
|
||||
suffixes = VISUAL_FILE_SUFFIXES if target_kind == "visual" else AUDIO_FILE_SUFFIXES
|
||||
for suffix in suffixes:
|
||||
candidate = base + suffix + ".safetensors"
|
||||
if candidate != current_path and os.path.isfile(candidate):
|
||||
return candidate
|
||||
return None
|
||||
|
||||
|
||||
def _find_preview(path_no_ext: str) -> Optional[str]:
|
||||
candidates = [path_no_ext]
|
||||
base, kind, _suffix = _strip_known_suffix(path_no_ext)
|
||||
if kind is not None:
|
||||
candidates.insert(0, base)
|
||||
for stem in candidates:
|
||||
for ext in PREVIEW_EXTS:
|
||||
candidate = stem + ext
|
||||
if os.path.isfile(candidate):
|
||||
return candidate
|
||||
return None
|
||||
|
||||
|
||||
def _display_mod_name(path_no_ext: str) -> str:
|
||||
base, kind, _suffix = _strip_known_suffix(path_no_ext)
|
||||
if kind is not None:
|
||||
name = os.path.basename(base)
|
||||
return name or os.path.basename(path_no_ext)
|
||||
return os.path.basename(path_no_ext)
|
||||
|
||||
|
||||
def _mod_entry(path: str) -> Optional[Dict]:
|
||||
if not path.lower().endswith(".safetensors"):
|
||||
return None
|
||||
path_no_ext = path[:-len(".safetensors")]
|
||||
meta = read_refmod_meta(path_no_ext)
|
||||
if meta is None:
|
||||
return None
|
||||
caps = refmod_capabilities(meta)
|
||||
meta_kind = str(caps.get("kind", "") or "")
|
||||
if meta_kind not in ("image", "video", "audio", "bundle"):
|
||||
return None
|
||||
_base, paired_kind, _suffix = _strip_known_suffix(path_no_ext)
|
||||
paired_visual = paired_mod_path(path_no_ext, "visual")
|
||||
paired_audio = paired_mod_path(path_no_ext, "audio")
|
||||
paired_split = bool(paired_visual or paired_audio)
|
||||
if paired_kind == "audio" and paired_visual:
|
||||
return None
|
||||
has_visual = bool(caps.get("has_visual", False))
|
||||
has_audio = bool(caps.get("has_audio", False))
|
||||
if paired_split:
|
||||
has_visual = True
|
||||
has_audio = True
|
||||
preview = _find_preview(path_no_ext) if has_visual else None
|
||||
return {
|
||||
"name": _display_mod_name(path_no_ext) + ".safetensors",
|
||||
"path": path,
|
||||
"preview_path": preview,
|
||||
"kind": meta_kind or "unknown",
|
||||
"has_visual": has_visual,
|
||||
"has_audio": has_audio,
|
||||
"paired_split": paired_split,
|
||||
"concept_type": str(meta.get("concept_type", "generic") or "generic"),
|
||||
"description": str(meta.get("description", "") or ""),
|
||||
}
|
||||
|
||||
|
||||
def _scan_dir(current: str) -> tuple[List[Dict], List[Dict]]:
|
||||
dirs: List[Dict] = []
|
||||
mods: List[Dict] = []
|
||||
|
||||
try:
|
||||
entries = list(os.scandir(current))
|
||||
except PermissionError as exc:
|
||||
raise ValueError("Access denied") from exc
|
||||
|
||||
for entry in entries:
|
||||
name = entry.name
|
||||
if name.startswith(".") or name in HIDDEN_BROWSER_DIRS:
|
||||
continue
|
||||
try:
|
||||
if entry.is_dir(follow_symlinks=False):
|
||||
dirs.append({"name": name, "path": os.path.abspath(entry.path)})
|
||||
elif entry.is_file(follow_symlinks=False) and name.lower().endswith(".safetensors"):
|
||||
item = _mod_entry(os.path.abspath(entry.path))
|
||||
if item is not None:
|
||||
mods.append(item)
|
||||
except OSError:
|
||||
continue
|
||||
return dirs, mods
|
||||
|
||||
|
||||
def _annotate_merged_duplicates(items: List[Dict]) -> None:
|
||||
counts: Dict[str, int] = {}
|
||||
for item in items:
|
||||
key = str(item.get("name", "")).lower()
|
||||
counts[key] = counts.get(key, 0) + 1
|
||||
for item in items:
|
||||
key = str(item.get("name", "")).lower()
|
||||
if counts.get(key, 0) > 1:
|
||||
item["name"] = f"{item['name']} [{_root_label_for_path(str(item.get('path', '')))}]"
|
||||
|
||||
|
||||
def list_browser_dir(path: str = "") -> Dict:
|
||||
current = safe_dir_path(path)
|
||||
|
||||
if _is_root(current):
|
||||
dirs: List[Dict] = []
|
||||
mods: List[Dict] = []
|
||||
for root in browser_roots():
|
||||
root_dirs, root_mods = _scan_dir(root)
|
||||
dirs.extend(root_dirs)
|
||||
mods.extend(root_mods)
|
||||
_annotate_merged_duplicates(dirs)
|
||||
_annotate_merged_duplicates(mods)
|
||||
current = browser_root()
|
||||
parent_path = None
|
||||
else:
|
||||
dirs, mods = _scan_dir(current)
|
||||
parent = os.path.dirname(current.rstrip("\\/"))
|
||||
if parent and _is_root(parent):
|
||||
parent_path = browser_root()
|
||||
elif parent and _is_under_root(parent) and parent != current:
|
||||
parent_path = parent
|
||||
else:
|
||||
parent_path = None
|
||||
|
||||
dirs.sort(key=lambda item: item["name"].lower())
|
||||
mods.sort(key=lambda item: item["name"].lower())
|
||||
return {
|
||||
"root": browser_root(),
|
||||
"current_path": current,
|
||||
"parent_path": parent_path,
|
||||
"dirs": dirs,
|
||||
"mods": mods,
|
||||
}
|
||||
@@ -0,0 +1,96 @@
|
||||
"""Shared helpers for the RefMod pack: media loading and the refmods folder."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
from typing import List, Optional
|
||||
|
||||
import torch
|
||||
|
||||
IMAGE_EXTS = {".png", ".jpg", ".jpeg", ".webp", ".bmp", ".gif"}
|
||||
|
||||
|
||||
def refmods_dir() -> str:
|
||||
"""ComfyUI models/refmods — the mod storage folder, created on first use.
|
||||
|
||||
Sits next to loras/, unet/ etc. instead of inside the pack folder, so mods
|
||||
live in the standard model tree.
|
||||
"""
|
||||
import folder_paths
|
||||
d = os.path.join(folder_paths.models_dir, "refmods")
|
||||
os.makedirs(d, exist_ok=True)
|
||||
return d
|
||||
|
||||
|
||||
def refmods_dirs() -> List[str]:
|
||||
"""All RefMod folders ComfyUI knows about, write folder first.
|
||||
|
||||
Includes folders declared under ``refmods:`` in extra_model_paths.yaml
|
||||
(or --extra-model-paths-config). Duplicates are removed by real path,
|
||||
since a root may be a symlink to another one.
|
||||
"""
|
||||
import folder_paths
|
||||
|
||||
candidates = [refmods_dir()]
|
||||
try:
|
||||
candidates += list(folder_paths.get_folder_paths("refmods"))
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
roots: List[str] = []
|
||||
seen = set()
|
||||
for directory in candidates:
|
||||
if not os.path.isdir(directory):
|
||||
continue
|
||||
real = os.path.realpath(directory)
|
||||
if real in seen:
|
||||
continue
|
||||
seen.add(real)
|
||||
roots.append(os.path.abspath(directory))
|
||||
return roots
|
||||
|
||||
|
||||
def list_image_files(folder: str) -> List[str]:
|
||||
"""Images directly under ``folder`` (top level only), sorted by name."""
|
||||
images = []
|
||||
if os.path.isdir(folder):
|
||||
for fn in sorted(os.listdir(folder)):
|
||||
ext = os.path.splitext(fn)[1].lower()
|
||||
p = os.path.join(folder, fn)
|
||||
if os.path.isfile(p):
|
||||
if ext in IMAGE_EXTS:
|
||||
images.append(p)
|
||||
return images
|
||||
|
||||
|
||||
def load_image_file(path: str, max_edge: Optional[int] = None) -> torch.Tensor:
|
||||
"""Load one image file -> [1, H, W, 3] float32 in [0, 1]."""
|
||||
import numpy as np
|
||||
from PIL import Image, ImageOps
|
||||
with Image.open(path) as img:
|
||||
img = ImageOps.exif_transpose(img)
|
||||
img = img.convert("RGB")
|
||||
w, h = img.size
|
||||
if max_edge is not None:
|
||||
scale = min(1.0, max_edge / max(w, h))
|
||||
if scale < 1.0:
|
||||
img = img.resize((max(1, round(w * scale)), max(1, round(h * scale))),
|
||||
Image.LANCZOS)
|
||||
arr = torch.from_numpy(np.asarray(img).copy()).float() / 255.0
|
||||
return arr.unsqueeze(0) # [1, H, W, 3]
|
||||
|
||||
|
||||
def _prompt_hint(loads) -> str:
|
||||
"""Merge loaded mods' concept_type + description into one prompt-ready string.
|
||||
|
||||
e.g. "identity: ginger woman, tattooed neck, black lipstick; pose_motion:
|
||||
slow twirl into camera, hair whipping". Concat this onto your positive
|
||||
prompt (a string-concat node ahead of CLIP Text Encode) instead of
|
||||
retyping each mod's description by hand. Mods with no description are
|
||||
skipped — a bare concept_type with nothing to say isn't a useful clue.
|
||||
"""
|
||||
parts = []
|
||||
for mod, _strength in loads:
|
||||
if mod.description:
|
||||
parts.append(f"{mod.concept_type}: {mod.description}")
|
||||
return "; ".join(parts)
|
||||
@@ -0,0 +1,958 @@
|
||||
"""
|
||||
core.py — "RefMod" adapter for MiniMax H3 (no training required)
|
||||
|
||||
The reference-video path of MiniMax H3 works by injecting *reference tokens*
|
||||
into the packed sequence: the ref is VAE-encoded, patchified, projected and
|
||||
placed on the 3D RoPE grid, then every DiT block attends to it. A video ref
|
||||
is expensive because it contributes thousands of tokens.
|
||||
|
||||
Mode naming: ``encode`` (was ``full``) = straight full-res VAE encode;
|
||||
``training`` (was ``pooled``) = compressed grid refined by gradient steps
|
||||
(the only mode that "trains" — the refinement loop). The old names are
|
||||
accepted everywhere as legacy aliases so saved workflows and mods keep
|
||||
working.
|
||||
|
||||
A RefMod is the same reference, compressed to a handful of tokens:
|
||||
|
||||
* the ref is VAE-encoded to its full latent [1, 24, T, H, W],
|
||||
* the latent is average-pooled to a tiny grid (default 4x4) and a few
|
||||
latent frames, so the patchified token count drops to ~4-16,
|
||||
* optionally the small latent is refined with a few gradient steps that
|
||||
reconstruct the full latent (still no model weights involved).
|
||||
|
||||
At generation time the pooled latent is handed back to the model through the
|
||||
native ``refs`` payload, so it flows through the exact same per-block
|
||||
attention machinery as a full reference — the only thing that changes is the
|
||||
token budget.
|
||||
|
||||
This is the MiniMax H3 analog of the LTX "Mod" concept tokens + per-block
|
||||
injection: instead of a trained hypernetwork predicting AdaLN deltas, the
|
||||
model's own cross-attention over the compressed ref tokens does the work.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import tempfile
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Dict, List, Optional, Tuple
|
||||
|
||||
import torch
|
||||
import torch.nn as nn
|
||||
import torch.nn.functional as F
|
||||
from safetensors import safe_open
|
||||
from safetensors.torch import load_file, save_file
|
||||
|
||||
# Metadata key embedded in the safetensors header (keeps a mod in one file,
|
||||
# so it can be shared/uploaded as a single artifact).
|
||||
META_KEY = "refmod_meta"
|
||||
|
||||
# What a mod is standing in for. This is documentation + a routing hint, not
|
||||
# a hard constraint on the tensors — but it drives two things: (1) the
|
||||
# "identity in training mode" warning in nodes.py, and (2) the loader's
|
||||
# prompt_hint output, which merges each loaded mod's concept_type +
|
||||
# description into a string you can concat onto your CLIP Text Encode prompt.
|
||||
# The model only reads these labels through the text prompt, so prompt_hint is
|
||||
# the only place they are surfaced.
|
||||
CONCEPT_TYPES = (
|
||||
"generic", # unspecified / mixed
|
||||
"identity", # a specific person/character — use mode="encode" for this
|
||||
"pose_motion", # a pose, dance, gesture, camera move
|
||||
"clothing", # an outfit / garment, decoupled from who's wearing it
|
||||
"background", # environment / set / location plate
|
||||
"voice",
|
||||
"singing",
|
||||
"music_style",
|
||||
"sound_fx",
|
||||
"ambience",
|
||||
"style", # look/grade/animation style, not a concrete subject
|
||||
)
|
||||
|
||||
AUDIO_CONCEPT_TYPES = ("voice", "singing", "music_style", "sound_fx", "ambience")
|
||||
|
||||
|
||||
def _blur_latent(z: torch.Tensor, factor: int = 8) -> torch.Tensor:
|
||||
"""Heavy spatial low-pass (downsample then upsample) used as the
|
||||
weakening target for ``ref_block``'s ``strength`` below ``1.0``.
|
||||
|
||||
Originally this mixed toward ``torch.randn()`` noise, mirroring what the
|
||||
docstring below describes as the model's own ``visual_cond_noise_aug``
|
||||
mechanism. In practice, at the retention range people actually use this
|
||||
dial for (0.15-0.7, i.e. large noise fractions), it produces a visible
|
||||
woven/static texture instead of a gracefully weaker reference: a VAE
|
||||
latent's channels are correlated, so raw iid noise isn't read as "less
|
||||
reference," it's read as real-but-garbled content and gets rendered as
|
||||
an actual (wrong) texture. ``visual_cond_noise_aug`` in the model's own
|
||||
training is likely a small-magnitude robustness augmentation, not
|
||||
something swept this far — so copying its math doesn't reproduce its
|
||||
behavior at these strengths. A blurred copy stays on the latent
|
||||
manifold (smooth, plausible) while still discarding detail as strength
|
||||
drops, which is what "weaker reference" should actually look like.
|
||||
"""
|
||||
if z.dim() == 4:
|
||||
b, c, stereo, t = z.shape
|
||||
if t <= 1:
|
||||
return z
|
||||
flat = z.reshape(b * c * stereo, 1, t).float()
|
||||
down = F.adaptive_avg_pool1d(flat, max(1, t // factor))
|
||||
up = F.interpolate(down, size=t, mode="linear", align_corners=False)
|
||||
return up.reshape_as(z).to(z.dtype)
|
||||
if z.dim() != 5:
|
||||
raise ValueError(f"Unsupported RefMod latent shape: {tuple(z.shape)}")
|
||||
t, h, w = z.shape[2], z.shape[3], z.shape[4]
|
||||
sh, sw = max(1, h // factor), max(1, w // factor)
|
||||
down = F.adaptive_avg_pool3d(z.float(), (t, sh, sw))
|
||||
up = F.interpolate(down, size=(t, h, w), mode="trilinear", align_corners=False)
|
||||
return up.to(z.dtype)
|
||||
|
||||
|
||||
def read_refmod_meta(path_no_ext: str) -> Optional[Dict]:
|
||||
"""
|
||||
Read the metadata block from a mod/preset file without loading tensors.
|
||||
|
||||
Prefers the metadata embedded in the safetensors header; falls back to
|
||||
the legacy sidecar ``{path}.json`` written by older versions.
|
||||
"""
|
||||
try:
|
||||
with safe_open(path_no_ext + ".safetensors", framework="pt") as f:
|
||||
meta = f.metadata()
|
||||
if meta:
|
||||
for key in (META_KEY, "audio_refmod_meta"):
|
||||
if key in meta:
|
||||
return json.loads(meta[key])
|
||||
except Exception:
|
||||
pass
|
||||
jpath = path_no_ext + ".json"
|
||||
if os.path.isfile(jpath):
|
||||
try:
|
||||
with open(jpath) as f:
|
||||
return json.load(f)
|
||||
except Exception:
|
||||
return None
|
||||
return None
|
||||
|
||||
|
||||
def refmod_bundle_members(meta: Dict) -> List[Dict]:
|
||||
if not isinstance(meta, dict) or meta.get("_format_version") != 5 or meta.get("kind") != "bundle":
|
||||
raise ValueError("Unsupported RefMod bundle format.")
|
||||
refs = meta.get("members")
|
||||
if not isinstance(refs, list) or not refs or len(refs) > 256:
|
||||
raise ValueError("A RefMod bundle must contain 1-256 members.")
|
||||
for ref in refs:
|
||||
if not isinstance(ref, dict) or ref.get("kind") not in ("image", "video", "audio"):
|
||||
raise ValueError("Invalid RefMod bundle member.")
|
||||
return refs
|
||||
|
||||
|
||||
def refmod_capabilities(meta: Optional[Dict]) -> Dict[str, object]:
|
||||
caps: Dict[str, object] = {
|
||||
"kind": "unknown",
|
||||
"has_visual": False,
|
||||
"has_audio": False,
|
||||
"member_count": 0,
|
||||
}
|
||||
if not isinstance(meta, dict):
|
||||
return caps
|
||||
kind = str(meta.get("kind", "") or "unknown")
|
||||
caps["kind"] = kind
|
||||
if kind == "bundle":
|
||||
refs = refmod_bundle_members(meta)
|
||||
caps["member_count"] = len(refs)
|
||||
caps["has_visual"] = any(ref.get("kind") in ("image", "video") for ref in refs)
|
||||
caps["has_audio"] = any(ref.get("kind") == "audio" for ref in refs)
|
||||
elif kind in ("image", "video"):
|
||||
caps["has_visual"] = True
|
||||
caps["has_audio"] = int(meta.get("ref_audio_t", 0) or 0) > 0
|
||||
caps["member_count"] = 1 + (1 if caps["has_audio"] else 0)
|
||||
elif kind == "audio":
|
||||
caps["has_audio"] = True
|
||||
caps["member_count"] = 1
|
||||
return caps
|
||||
|
||||
|
||||
def _standalone_audio_member_meta(meta: Dict) -> Dict:
|
||||
name = str(meta.get("name", "refmod") or "refmod")
|
||||
if not name.lower().endswith("_audio"):
|
||||
name = f"{name}_Audio"
|
||||
return {
|
||||
**meta,
|
||||
"name": name,
|
||||
"kind": "audio",
|
||||
"latent_h": 0,
|
||||
"latent_w": 0,
|
||||
"latent_t": int(meta.get("ref_audio_t", 0) or 0),
|
||||
"concept_type": str(meta.get("audio_concept_type", meta.get("concept_type", "voice")) or "voice"),
|
||||
"audio_concept_type": str(meta.get("audio_concept_type", "") or ""),
|
||||
"ref_audio_t": int(meta.get("ref_audio_t", 0) or 0),
|
||||
"sample_rate": int(meta.get("sample_rate", 32000) or 32000),
|
||||
}
|
||||
|
||||
|
||||
def load_refmods_from_file(path_no_ext: str, device: str = "cpu") -> List["H3RefMod"]:
|
||||
meta = read_refmod_meta(path_no_ext)
|
||||
if not isinstance(meta, dict):
|
||||
raise ValueError(
|
||||
f"{path_no_ext}.safetensors has no RefMod metadata "
|
||||
f"(header key '{META_KEY}' or sidecar .json missing)."
|
||||
)
|
||||
if meta.get("kind") == "bundle":
|
||||
loaded: List[H3RefMod] = []
|
||||
refs = refmod_bundle_members(meta)
|
||||
with safe_open(path_no_ext + ".safetensors", framework="pt", device=device) as f:
|
||||
for idx, member_meta in enumerate(refs):
|
||||
latent = f.get_tensor(f"ref_{idx}").clone()
|
||||
enc_frames = _read_encoder_images(member_meta, f.get_tensor)
|
||||
loaded.append(H3RefMod.from_metadata(member_meta, latent, enc_frames=enc_frames))
|
||||
return loaded
|
||||
|
||||
tensors = load_file(path_no_ext + ".safetensors", device=device)
|
||||
if "latent" not in tensors:
|
||||
raise ValueError(f"{path_no_ext}.safetensors is missing its latent tensor.")
|
||||
latent = tensors["latent"].clone()
|
||||
audio_latent = tensors["audio_latent"].clone() if "audio_latent" in tensors else None
|
||||
if meta.get("kind") == "audio":
|
||||
return [H3RefMod.from_metadata(meta, latent)]
|
||||
|
||||
visual_meta = dict(meta)
|
||||
visual_meta["ref_audio_t"] = 0
|
||||
visual_meta["audio_concept_type"] = ""
|
||||
enc_frames = _read_encoder_images(meta, tensors.__getitem__)
|
||||
loaded = [H3RefMod.from_metadata(visual_meta, latent, enc_frames=enc_frames)]
|
||||
if audio_latent is not None and int(meta.get("ref_audio_t", 0) or 0) > 0:
|
||||
loaded.append(H3RefMod.from_metadata(_standalone_audio_member_meta(meta), audio_latent))
|
||||
return loaded
|
||||
|
||||
|
||||
def save_refmod_bundle(path_no_ext: str, name: str, mods: List["H3RefMod"]) -> str:
|
||||
unique = list({id(mod): mod for mod in mods}.values())
|
||||
if not unique:
|
||||
raise ValueError("RefMod bundle is empty.")
|
||||
if sum(bool(mod.enc_frames) for mod in unique) > 1:
|
||||
raise ValueError("Save RefMods with encoder pictures in separate files when more than one member has pictures.")
|
||||
metadata = {
|
||||
"_format_version": 5,
|
||||
"kind": "bundle",
|
||||
"name": str(name or "refmod"),
|
||||
"members": [mod.metadata() for mod in unique],
|
||||
}
|
||||
tensors = {
|
||||
f"ref_{idx}": mod.latent.detach().cpu().contiguous().clone()
|
||||
for idx, mod in enumerate(unique)
|
||||
}
|
||||
for mod in unique:
|
||||
tensors.update({key: value.contiguous() for key, value in mod.enc_frames.items()})
|
||||
destination = path_no_ext + ".safetensors"
|
||||
directory = os.path.dirname(destination) or "."
|
||||
os.makedirs(directory, exist_ok=True)
|
||||
fd, temporary = tempfile.mkstemp(prefix=".refmod-", suffix=".tmp", dir=directory)
|
||||
os.close(fd)
|
||||
try:
|
||||
save_file(tensors, temporary, metadata={META_KEY: json.dumps(metadata)})
|
||||
os.replace(temporary, destination)
|
||||
finally:
|
||||
if os.path.exists(temporary):
|
||||
os.unlink(temporary)
|
||||
return destination
|
||||
|
||||
|
||||
def _read_encoder_images(meta, get_tensor):
|
||||
record = meta.get("encoder_images") or {}
|
||||
times = record.get("times", meta.get("enc_times", []))
|
||||
prefix = record.get("prefix", "enc_")
|
||||
return {f"enc_{index}": get_tensor(f"{prefix}{index}").detach().cpu().clone()
|
||||
for index in range(len(times))}
|
||||
|
||||
|
||||
def pack_encoder_images(mod, frames, times, fps=24.0):
|
||||
import io as bytes_io
|
||||
from PIL import Image
|
||||
|
||||
if frames.shape[0] != len(times):
|
||||
raise ValueError("Encoder picture count does not match its timestamps.")
|
||||
packed = {}
|
||||
for index, frame in enumerate(frames):
|
||||
buffer = bytes_io.BytesIO()
|
||||
pixels = (frame.detach().cpu().float().clamp(0, 1) * 255).round().to(torch.uint8).numpy()
|
||||
Image.fromarray(pixels).save(buffer, "JPEG", quality=95)
|
||||
packed[f"enc_{index}"] = torch.frombuffer(bytearray(buffer.getvalue()), dtype=torch.uint8)
|
||||
mod.enc_frames = packed
|
||||
mod.enc_times = list(times)
|
||||
mod.enc_fps = float(fps)
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════════════
|
||||
# Latent compression
|
||||
# ═══════════════════════════════════════════════════════════════════════════
|
||||
|
||||
MODE_ALIASES = {
|
||||
"full": "encode",
|
||||
"pooled": "training",
|
||||
"Full Reference": "encode",
|
||||
"Compressed Reference": "training",
|
||||
}
|
||||
|
||||
|
||||
def normalize_mode(mode: str) -> str:
|
||||
"""Map legacy mode names (``full``/``pooled``) to the current ones."""
|
||||
return MODE_ALIASES.get(mode, mode)
|
||||
|
||||
|
||||
def aspect_grid(pool_h: int, pool_w: int, aspect: float):
|
||||
"""Even pool grid dims that match ``aspect`` (h/w) within the dial caps.
|
||||
|
||||
The DiT patches each 2x2 latent cell into one token and the rope grid is
|
||||
area-normalized per axis, so a pooled grid that ignores the source aspect
|
||||
squishes the subject: a portrait latent (tall person) forced into a square
|
||||
16x16 grid is compressed 5x in height but only 3x in width, and comes out
|
||||
"fat". The official ref2video node never pools — it feeds the model
|
||||
aspect-correct latent dims, which is what the DiT was trained on.
|
||||
|
||||
The long edge follows the user's dial (``max(pool_h, pool_w)``) and the
|
||||
short edge is derived from the source aspect, both rounded to even for the
|
||||
2x2 patch. Square sources keep the exact dial value (16x16 stays 16x16).
|
||||
"""
|
||||
long_edge = max(pool_h, pool_w)
|
||||
if aspect >= 1.0:
|
||||
h, w = long_edge, long_edge / aspect
|
||||
else:
|
||||
w, h = long_edge, long_edge * aspect
|
||||
h = max(2, round(h / 2) * 2)
|
||||
w = max(2, round(w / 2) * 2)
|
||||
return h, w
|
||||
|
||||
|
||||
def pool_latent(
|
||||
z: torch.Tensor,
|
||||
latent_t: int,
|
||||
latent_h: int,
|
||||
latent_w: int,
|
||||
) -> torch.Tensor:
|
||||
"""
|
||||
Average-pool a VAE latent ``[1, 24, T, H, W]`` down to a tiny grid.
|
||||
|
||||
``latent_h/latent_w`` are the *latent* grid dims (the DiT patches each 2x2
|
||||
latent cell into one token), so a 4x4 latent = 2x2 = 4 tokens per frame.
|
||||
Dims must be even (the DiT's 2x2 patch). Pooling runs in fp32; the result
|
||||
keeps the input dtype.
|
||||
"""
|
||||
if z.shape[2] == latent_t and z.shape[3] == latent_h and z.shape[4] == latent_w:
|
||||
return z
|
||||
if latent_h % 2 != 0 or latent_w % 2 != 0:
|
||||
raise ValueError(f"pool_h/pool_w must be even (got {latent_h}x{latent_w})")
|
||||
pooled = F.adaptive_avg_pool3d(
|
||||
z.float(), (latent_t, latent_h, latent_w)
|
||||
)
|
||||
return pooled.to(z.dtype)
|
||||
|
||||
|
||||
def optimize_latent(
|
||||
z_small: torch.Tensor,
|
||||
z_full: torch.Tensor,
|
||||
steps: int = 150,
|
||||
lr: float = 00.02,
|
||||
device: Optional[torch.device] = None,
|
||||
progress_every: int = 0,
|
||||
) -> torch.Tensor:
|
||||
"""
|
||||
Model-free refinement of the compressed latent.
|
||||
|
||||
Optimizes the small latent so its trilinearly upsampled reconstruction
|
||||
matches the full reference latent. This pulls the pooled representation
|
||||
closer to what the DiT would see from the full ref, with nothing but the
|
||||
tiny latent trainable (~1-2K params) and no diffusion model loaded.
|
||||
|
||||
``progress_every`` > 0 prints a ``[RefMod] identity <step>/<steps>`` line
|
||||
every N steps so long pooled-mode refinement isn't silent (the default
|
||||
``identity=500`` takes a while per ref on CPU/VRAM-bound setups).
|
||||
|
||||
Returns the refined latent detached, same shape/dtype as ``z_small``.
|
||||
"""
|
||||
if steps <= 0:
|
||||
return z_small
|
||||
device = device or z_full.device
|
||||
# Node execution runs under ComfyUI's global inference mode, which would
|
||||
# disable the autograd this refine loop needs. Re-enable it for this
|
||||
# scope and clone the inputs to drop the inference flag.
|
||||
with torch.inference_mode(False), torch.set_grad_enabled(True):
|
||||
target = z_full.clone().float().to(device)
|
||||
param = nn.Parameter(z_small.clone().float().to(device))
|
||||
opt = torch.optim.Adam([param], lr=lr)
|
||||
size = tuple(target.shape[2:])
|
||||
for i in range(steps):
|
||||
opt.zero_grad()
|
||||
up = F.interpolate(param, size=size, mode="trilinear", align_corners=False)
|
||||
loss = F.mse_loss(up, target)
|
||||
loss.backward()
|
||||
opt.step()
|
||||
if progress_every and (i + 1) % progress_every == 0:
|
||||
print(f"[RefMod] identity {i + 1}/{steps}")
|
||||
# materialize inside the scope so the result is a normal tensor, not
|
||||
# an inference-mode tensor (it gets stored in the mod and reused)
|
||||
refined = param.detach().to(z_small.dtype)
|
||||
return refined
|
||||
|
||||
|
||||
def optimize_latent_multi(z_init, targets, steps=150, lr=0.02, device=None,
|
||||
progress_every=0, strategy="grouped"):
|
||||
"""Mean reconstruction objective with bounded activation memory."""
|
||||
if not targets or steps <= 0:
|
||||
return z_init
|
||||
if strategy not in ("grouped", "stream", "resident"):
|
||||
raise ValueError("Unknown multi-reference optimization strategy.")
|
||||
device = device or z_init.device
|
||||
with torch.inference_mode(False), torch.set_grad_enabled(True):
|
||||
count = len(targets)
|
||||
if strategy == "grouped":
|
||||
groups = {}
|
||||
for target in targets:
|
||||
key = tuple(target.shape)
|
||||
value = target.detach().to(device="cpu", dtype=torch.float32).clone()
|
||||
if key in groups:
|
||||
groups[key][0].add_(value)
|
||||
groups[key][1] += 1
|
||||
else:
|
||||
groups[key] = [value, 1]
|
||||
grouped = []
|
||||
for total, group_count in groups.values():
|
||||
grouped.append((total.div_(group_count), group_count))
|
||||
elif strategy == "resident":
|
||||
resident = [target.detach().float().to(device) for target in targets]
|
||||
param = nn.Parameter(z_init.clone().float().to(device))
|
||||
opt = torch.optim.Adam([param], lr=lr)
|
||||
for step_idx in range(steps):
|
||||
opt.zero_grad()
|
||||
loss = None
|
||||
if strategy == "grouped":
|
||||
total_weight = 0
|
||||
for target_mean, group_count in grouped:
|
||||
target = target_mean.to(device)
|
||||
up = F.interpolate(param, size=tuple(target.shape[2:]), mode="trilinear",
|
||||
align_corners=False)
|
||||
contribution = F.mse_loss(up, target)
|
||||
loss = contribution * group_count if loss is None else loss + contribution * group_count
|
||||
total_weight += group_count
|
||||
loss = loss / total_weight
|
||||
elif strategy == "stream":
|
||||
for target_cpu in targets:
|
||||
target = target_cpu.detach().float().to(device)
|
||||
up = F.interpolate(param, size=tuple(target.shape[2:]), mode="trilinear",
|
||||
align_corners=False)
|
||||
contribution = F.mse_loss(up, target)
|
||||
loss = contribution if loss is None else loss + contribution
|
||||
loss = loss / count
|
||||
else:
|
||||
for target in resident:
|
||||
up = F.interpolate(param, size=tuple(target.shape[2:]), mode="trilinear",
|
||||
align_corners=False)
|
||||
contribution = F.mse_loss(up, target)
|
||||
loss = contribution if loss is None else loss + contribution
|
||||
loss = loss / count
|
||||
loss.backward()
|
||||
opt.step()
|
||||
if progress_every and (step_idx + 1) % progress_every == 0:
|
||||
print(f"[RefMod] identity {step_idx + 1}/{steps}")
|
||||
refined = param.detach().to(z_init.dtype)
|
||||
return refined
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════════════
|
||||
# Per-frame strength curve
|
||||
# ═══════════════════════════════════════════════════════════════════════════
|
||||
|
||||
# Direction names describe where the *concept* shows up in the output, which
|
||||
# is the mirror of the strength envelope over the ref's own latent timeline:
|
||||
# concept_at_end locks the ref's literal footage at the START and releases it
|
||||
# toward the end (was "decrease"), concept_at_start opens free and locks onto
|
||||
# the ref near the END (was "increase"). shape is how the envelope travels
|
||||
# between its endpoints; value is the non-zero endpoint ("user input").
|
||||
CURVE_DIRECTIONS = ("constant", "concept_at_start", "concept_at_middle",
|
||||
"concept_at_end", "concept_at_ends")
|
||||
CURVE_SHAPES = ("linear", "ease", "sigmoid", "tanh", "quadratic", "cubic",
|
||||
"exponential", "stair", "elastic", "bump", "dip")
|
||||
|
||||
# old single-combo preset names -> (direction, shape, value) so workflows
|
||||
# saved before the curve was split still resolve
|
||||
_LEGACY_CURVES = {
|
||||
"flat": ("constant", "linear", 1.0),
|
||||
"fade_in": ("concept_at_start", "linear", 1.0),
|
||||
"fade_out": ("concept_at_end", "linear", 1.0),
|
||||
"bump": ("concept_at_start", "bump", 1.0),
|
||||
"dip": ("constant", "dip", 1.0),
|
||||
}
|
||||
|
||||
|
||||
def _ease(shape: str, x: float) -> float:
|
||||
"""Shape a progress ``x`` in [0, 1] per the easing name (may overshoot)."""
|
||||
if shape == "linear":
|
||||
return x
|
||||
if shape == "ease": # smoothstep
|
||||
return x * x * (3.0 - 2.0 * x)
|
||||
if shape == "sigmoid": # logistic S-curve
|
||||
return 1.0 / (1.0 + math.exp(-12.0 * (x - 0.5)))
|
||||
if shape == "tanh": # sharper S-curve (steeper knee than sigmoid)
|
||||
return 0.5 * (math.tanh(8.0 * (x - 0.5)) + 1.0)
|
||||
if shape == "quadratic":
|
||||
return x * x
|
||||
if shape == "cubic":
|
||||
return x * x * x
|
||||
if shape == "exponential":
|
||||
return 2.0 ** x - 1.0
|
||||
if shape == "stair":
|
||||
return min(1.0, math.floor(x * 4) / 3.0)
|
||||
if shape == "elastic":
|
||||
if x <= 0.0:
|
||||
return 0.0
|
||||
if x >= 1.0:
|
||||
return 1.0
|
||||
return 2.0 ** (-10.0 * x) * math.sin((x * 10.0 - 0.75) * (2.0 * math.pi / 3.0)) + 1.0
|
||||
if shape == "bump":
|
||||
return 1.0 - abs(2.0 * x - 1.0)
|
||||
if shape == "dip":
|
||||
return abs(2.0 * x - 1.0)
|
||||
return x
|
||||
|
||||
|
||||
def curve_strengths(spec, t: int) -> Optional[List[float]]:
|
||||
"""Resolve a curve spec to ``t`` per-frame strength multipliers in [0, 1].
|
||||
|
||||
Kept as plain Python so the Apply node exposes the curve as normal
|
||||
combo/float widgets instead of depending on ComfyUI's new Curve widget.
|
||||
Accepts:
|
||||
|
||||
* a ``(direction, shape, value)`` tuple — direction is ``constant``
|
||||
(one strength everywhere), ``concept_at_start`` (0 -> value, a
|
||||
crescent — the concept shows early; legacy ``increase``),
|
||||
``concept_at_end`` (value -> 0, a decrescent — the concept pops at
|
||||
the end; legacy ``decrease``), ``concept_at_middle`` ([0..1..0] —
|
||||
concept peaks mid-timeline) or ``concept_at_ends`` ([1..0..1] —
|
||||
concept at both ends); shape is how the envelope travels between
|
||||
its endpoints (see ``_ease``); value is the non-zero endpoint
|
||||
(1.0 = full strength there);
|
||||
* a legacy preset name (``flat``/``fade_in``/``fade_out``/``bump``/
|
||||
``dip``) from before the split;
|
||||
* a list of per-frame floats (len == t), used as-is;
|
||||
* a list of ``(x, y)`` control points (x in [0, 1]), linearly
|
||||
interpolated across the frames;
|
||||
* any object with ``interp(x) -> float`` (legacy CurveInput saved in an
|
||||
old workflow).
|
||||
|
||||
Returns None for flat/no curve so the caller keeps its single-strength
|
||||
path unchanged (a ``constant`` curve at value 1.0 included).
|
||||
"""
|
||||
if t <= 1 or spec is None or spec == "":
|
||||
return None
|
||||
if isinstance(spec, str):
|
||||
legacy = _LEGACY_CURVES.get(spec)
|
||||
return curve_strengths(legacy, t) if legacy is not None else None
|
||||
if (isinstance(spec, tuple) and len(spec) == 3
|
||||
and isinstance(spec[0], str) and isinstance(spec[1], str)):
|
||||
direction, shape, value = spec
|
||||
value = float(value)
|
||||
if direction == "constant":
|
||||
# linear = flat at ``value`` (1.0 == no curve); the other shapes
|
||||
# modulate around the constant, so constant + dip is a trough,
|
||||
# constant + bump is a peak (both endpoints equal)
|
||||
if shape == "linear":
|
||||
return None if value >= 1.0 else [value] * t
|
||||
p = [_ease(shape, i / (t - 1)) for i in range(t)]
|
||||
return [max(0.0, min(1.0, value * y)) for y in p]
|
||||
p = [_ease(shape, i / (t - 1)) for i in range(t)]
|
||||
# legacy "increase"/"decrease" (pre-rename workflows) map onto the same
|
||||
# envelopes as the concept-placement names
|
||||
if direction in ("increase", "concept_at_start"):
|
||||
return [max(0.0, min(1.0, value * y)) for y in p]
|
||||
if direction in ("decrease", "concept_at_end"):
|
||||
return [max(0.0, min(1.0, value * (1.0 - y))) for y in p]
|
||||
if direction == "concept_at_middle": # [0..1..0]: concept peaks mid-timeline
|
||||
return [max(0.0, min(1.0, value * (1.0 - abs(2.0 * y - 1.0)))) for y in p]
|
||||
if direction == "concept_at_ends": # [1..0..1]: concept at both ends
|
||||
return [max(0.0, min(1.0, value * abs(2.0 * y - 1.0))) for y in p]
|
||||
return None # unknown direction: treat as flat
|
||||
if isinstance(spec, (list, tuple)):
|
||||
if len(spec) == t and all(isinstance(v, (int, float)) for v in spec):
|
||||
return [float(v) for v in spec]
|
||||
pts = [(float(x), float(y)) for x, y in spec
|
||||
if isinstance(x, (int, float)) and isinstance(y, (int, float))]
|
||||
if pts:
|
||||
pts = sorted(pts, key=lambda p: p[0])
|
||||
out = []
|
||||
for i in range(t):
|
||||
x = i * (1.0 / (t - 1))
|
||||
if x <= pts[0][0]:
|
||||
out.append(pts[0][1])
|
||||
elif x >= pts[-1][0]:
|
||||
out.append(pts[-1][1])
|
||||
else:
|
||||
for (x0, y0), (x1, y1) in zip(pts, pts[1:]):
|
||||
if x0 <= x <= x1:
|
||||
out.append(y0 + (y1 - y0) * (x - x0) / (x1 - x0))
|
||||
break
|
||||
return out
|
||||
interp = getattr(spec, "interp", None)
|
||||
if callable(interp):
|
||||
return [float(interp(i / (t - 1))) for i in range(t)]
|
||||
return None
|
||||
|
||||
|
||||
def curve_value_at(spec, x: float) -> float:
|
||||
"""Per-progress strength of a curve spec at one point ``x`` in [0, 1].
|
||||
|
||||
The continuous analog of ``curve_strengths`` — same direction/shape/value
|
||||
math, but evaluated at a single progress value instead of discretized over
|
||||
frames. The step-curve node uses it to re-mix the ref latents once per
|
||||
denoising step, where ``x`` is the denoise progress (0 = first step,
|
||||
1 = last step). Returns 1.0 (no weakening) for anything unrecognized.
|
||||
"""
|
||||
if spec is None or spec == "":
|
||||
return 1.0
|
||||
if isinstance(spec, str):
|
||||
legacy = _LEGACY_CURVES.get(spec)
|
||||
return curve_value_at(legacy, x) if legacy is not None else 1.0
|
||||
if (isinstance(spec, tuple) and len(spec) == 3
|
||||
and isinstance(spec[0], str) and isinstance(spec[1], str)):
|
||||
direction, shape, value = spec
|
||||
value = float(value)
|
||||
if direction == "constant":
|
||||
if shape == "linear":
|
||||
return value
|
||||
return max(0.0, min(1.0, value * _ease(shape, x)))
|
||||
y = _ease(shape, x)
|
||||
if direction in ("increase", "concept_at_start"):
|
||||
return max(0.0, min(1.0, value * y))
|
||||
if direction in ("decrease", "concept_at_end"):
|
||||
return max(0.0, min(1.0, value * (1.0 - y)))
|
||||
if direction == "concept_at_middle": # [0..1..0]: concept peaks mid-timeline
|
||||
return max(0.0, min(1.0, value * (1.0 - abs(2.0 * y - 1.0))))
|
||||
if direction == "concept_at_ends": # [1..0..1]: concept at both ends
|
||||
return max(0.0, min(1.0, value * abs(2.0 * y - 1.0)))
|
||||
return 1.0
|
||||
|
||||
|
||||
def dedup_frame_indices(z: torch.Tensor, threshold: float = 0.02) -> List[int]:
|
||||
"""Indices of latent frames kept by greedy temporal dedup.
|
||||
|
||||
Each frame is compared to the last *kept* frame: if the mean-abs
|
||||
difference between them, normalized by the frames' own magnitude, is
|
||||
below ``threshold`` the frame is dropped as a (near-)duplicate. Video
|
||||
refs (a dance loop, a static shot, a talking head) contain long runs of
|
||||
frames that differ only by codec noise — each one still costs a token
|
||||
per spatial patch in every DiT block, so cutting them is the cheapest
|
||||
way to honor a token cap. Stacked image refs survive: different
|
||||
angles/expressions land well above the threshold.
|
||||
"""
|
||||
t = z.shape[2]
|
||||
if t <= 1:
|
||||
return list(range(t))
|
||||
flat = z[0].float() # [24, T, H, W]
|
||||
kept = [0]
|
||||
prev = flat[:, 0]
|
||||
for i in range(1, t):
|
||||
cur = flat[:, i]
|
||||
denom = (cur.abs().mean() + prev.abs().mean()) / 2 + 1e-6
|
||||
diff = (cur - prev).abs().mean() / denom
|
||||
if diff >= threshold:
|
||||
kept.append(i)
|
||||
prev = cur
|
||||
return kept
|
||||
|
||||
|
||||
def fit_token_budget(latent: torch.Tensor, budget: int, label: str) -> torch.Tensor:
|
||||
"""Bring a stacked latent's injected token count under ``budget``.
|
||||
|
||||
Tokens = (H/2)*(W/2) per frame (the DiT's 2x2 patch). When the stack
|
||||
exceeds the budget it is cut in two cheap, loss-ordered passes:
|
||||
|
||||
1. temporal dedup — drop latent frames that are (near-)identical to
|
||||
the last kept frame (free: they carry no new reference info);
|
||||
2. budget-fit resample — uniformly subsample the remaining frames down
|
||||
to the largest count that fits the budget.
|
||||
|
||||
The spatial dims are never touched: resizing a latent's H/W changes the
|
||||
rope grid and degrades the reference, so time is the only honest lever.
|
||||
``label`` is the mod name for the console notes.
|
||||
"""
|
||||
h, w = latent.shape[3], latent.shape[4]
|
||||
per_frame = (h // 2) * (w // 2)
|
||||
t = latent.shape[2]
|
||||
if per_frame * t <= budget:
|
||||
return latent
|
||||
kept = dedup_frame_indices(latent)
|
||||
if len(kept) < t:
|
||||
print(f"[RefMod] {label}: over {budget}-token cap, "
|
||||
f"dropped {t - len(kept)} near-duplicate frame(s) "
|
||||
f"({t} -> {len(kept)})")
|
||||
latent = latent[:, :, kept]
|
||||
t = latent.shape[2]
|
||||
if per_frame * t > budget:
|
||||
fit_t = max(1, budget // per_frame)
|
||||
if fit_t < t:
|
||||
idx = torch.linspace(0, t - 1, fit_t).round().long()
|
||||
latent = latent[:, :, idx]
|
||||
print(f"[RefMod] {label}: still over {budget}-token cap, "
|
||||
f"resampled {t} -> {fit_t} frame(s) to fit")
|
||||
if per_frame * latent.shape[2] > budget:
|
||||
print(f"[RefMod] {label}: one frame alone is {per_frame} tokens > "
|
||||
f"{budget} — lower ref_resolution or pool_h/pool_w to meet "
|
||||
f"the cap.")
|
||||
return latent
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════════════
|
||||
# H3RefMod — the saved artifact
|
||||
# ═══════════════════════════════════════════════════════════════════════════
|
||||
|
||||
@dataclass
|
||||
class H3RefMod:
|
||||
"""
|
||||
A compressed reference for MiniMax H3.
|
||||
|
||||
``latent`` is the VAE latent ``[1, 24, latent_t, latent_h, latent_w]`` — a
|
||||
full-resolution encode (``mode="encode"``) or a pooled thumbnail
|
||||
(``mode="training"``). ``kind`` is ``"image"`` (single latent frame) or
|
||||
``"video"`` (a few frames), matching the native ref block kinds the
|
||||
model's ``PackedLayout`` understands.
|
||||
|
||||
``mode="encode"`` stores the encode at the resolution the official ref2video
|
||||
path uses, so the injected ref carries real identity detail;
|
||||
``mode="training"`` stores a tiny average-pooled grid refined by gradient
|
||||
steps (concept/motion, or identity at high pool sizes).
|
||||
"""
|
||||
|
||||
name: str
|
||||
kind: str
|
||||
latent: torch.Tensor
|
||||
latent_h: int = 4
|
||||
latent_w: int = 4
|
||||
latent_t: int = 1
|
||||
mode: str = "training"
|
||||
source: str = "" # "image" | "video" | "manual"
|
||||
source_shape: str = "" # original latent dims as "TxHxW"
|
||||
pool: str = "4x4x1" # pool_t x pool_h x pool_w
|
||||
optimize_steps: int = 0
|
||||
tags: List[str] = field(default_factory=list)
|
||||
description: str = "" # optional text describing the concept (emitted by the loaders)
|
||||
concept_type: str = "generic" # what this mod represents; see CONCEPT_TYPES above
|
||||
audio_concept_type: str = "" # upstream-style audio role label for embedded audio, if present
|
||||
# Optional audio identity: the audio-VAE latent [1, 32, 2, ref_audio_t] and
|
||||
# its frame count. When present the mod is emitted as a "video_audio" (or
|
||||
# standalone "audio") ref block so the DiT attends to the soundtrack too.
|
||||
audio_latent: Optional[torch.Tensor] = None
|
||||
ref_audio_t: int = 0
|
||||
sample_rate: int = 32000
|
||||
enc_frames: Dict[str, torch.Tensor] = field(default_factory=dict)
|
||||
enc_times: List[float] = field(default_factory=list)
|
||||
enc_fps: float = 0.0
|
||||
|
||||
def __post_init__(self):
|
||||
if self.kind == "audio":
|
||||
if self.latent.ndim != 4 or tuple(self.latent.shape[:3]) != (1, 32, 2) or self.latent.shape[-1] < 1:
|
||||
raise ValueError("H3 audio RefMod must contain [1,32,2,T] with T >= 1.")
|
||||
self.latent_t = int(self.latent.shape[-1])
|
||||
self.latent_h = 0
|
||||
self.latent_w = 0
|
||||
self.audio_latent = None
|
||||
self.ref_audio_t = 0
|
||||
return
|
||||
if self.kind not in ("image", "video"):
|
||||
raise ValueError(f"kind must be 'image', 'video' or 'audio' (got {self.kind!r})")
|
||||
if self.kind == "image":
|
||||
self.latent_t = 1
|
||||
if self.audio_latent is None or self.ref_audio_t <= 0:
|
||||
self.audio_latent = None
|
||||
self.audio_concept_type = ""
|
||||
self.ref_audio_t = 0
|
||||
|
||||
# ── token budget ──────────────────────────────────────────────────
|
||||
|
||||
@property
|
||||
def token_count(self) -> int:
|
||||
"""Number of patchified tokens the mod injects into the packed sequence."""
|
||||
if self.kind == "audio":
|
||||
return 2 * self.latent_t
|
||||
per_frame = (self.latent_h // 2) * (self.latent_w // 2)
|
||||
return self.latent_t * per_frame
|
||||
|
||||
# ── native ref block ──────────────────────────────────────────────
|
||||
|
||||
def ref_block(self, strength: float = 1.0,
|
||||
curve=None, use_video: bool = True,
|
||||
use_audio: bool = True) -> Optional[Dict]:
|
||||
"""
|
||||
Build the ref block dict the model's ``PackedLayout`` / payload consumes.
|
||||
|
||||
The shape mirrors what the native ref2va nodes emit, so the pooled
|
||||
latent rides the exact same path: ``cond_video_latents`` -> patchify ->
|
||||
``video_patch_proj`` -> packed sequence with a 3D RoPE grid.
|
||||
|
||||
``strength`` weakens the ref by mixing the latent toward a heavily
|
||||
blurred copy of itself (see ``_blur_latent``) rather than toward
|
||||
zero or random noise: scaling toward zero pushes values out of the
|
||||
normalized latent distribution (reads as grey output), and mixing
|
||||
toward random noise reads as real-but-garbled content and decodes
|
||||
as a wrong texture rather than "weaker" — a blurred copy stays
|
||||
on-manifold while still discarding the specific detail that makes a
|
||||
reference strong. ``strength <= 0`` drops the block entirely (no
|
||||
tokens injected).
|
||||
|
||||
``curve`` (optional) is a per-frame strength spec for the ref's
|
||||
latent timeline — a ``(direction, shape, value)`` tuple (e.g.
|
||||
``("concept_at_end", "ease", 1.0)``), a legacy preset name, a list of
|
||||
per-frame floats, ``(x, y)`` control points, or any object with
|
||||
``interp(x)`` (see ``curve_strengths``). Each frame is mixed with
|
||||
its own strength ``retention * curve(x)`` instead of one flat value,
|
||||
so the ref can fade in/out across the video. A flat curve is
|
||||
identical to passing ``strength`` alone.
|
||||
"""
|
||||
if strength <= 0.0 or (not use_video and not use_audio):
|
||||
return None
|
||||
if self.kind == "audio":
|
||||
if not use_audio:
|
||||
return None
|
||||
latent = self.latent
|
||||
strengths = curve_strengths(curve, self.latent_t) if curve is not None else None
|
||||
if strengths is not None:
|
||||
weights = [max(0.0, min(1.0, strength * value)) for value in strengths]
|
||||
if any(value < 1.0 for value in weights):
|
||||
st = latent.new_tensor(weights).view(1, 1, 1, self.latent_t)
|
||||
latent = st * latent + (1.0 - st) * _blur_latent(latent)
|
||||
elif strength < 1.0:
|
||||
latent = strength * latent + (1.0 - strength) * _blur_latent(latent)
|
||||
return {
|
||||
"kind": "audio",
|
||||
"ref_audio_t": self.latent_t,
|
||||
"audio_latent": latent,
|
||||
}
|
||||
has_audio = use_audio and self.audio_latent is not None and self.ref_audio_t > 0
|
||||
if not use_video:
|
||||
if has_audio:
|
||||
return {
|
||||
"kind": "audio",
|
||||
"ref_audio_t": self.ref_audio_t,
|
||||
"audio_latent": self.audio_latent,
|
||||
}
|
||||
return None
|
||||
latent = self.latent
|
||||
if curve is not None and self.latent_t > 1:
|
||||
strengths = curve_strengths(curve, self.latent_t)
|
||||
if strengths is not None:
|
||||
t = self.latent_t
|
||||
st = torch.tensor(
|
||||
[max(0.0, min(1.0, strength * s)) for s in strengths],
|
||||
dtype=latent.dtype, device=latent.device)
|
||||
st = st.view(1, 1, t, 1, 1)
|
||||
latent = st * latent + (1.0 - st) * _blur_latent(latent)
|
||||
elif strength < 1.0:
|
||||
latent = strength * latent + (1.0 - strength) * _blur_latent(latent)
|
||||
block: Dict = {
|
||||
"kind": self.kind,
|
||||
"latent_h": self.latent_h,
|
||||
"latent_w": self.latent_w,
|
||||
"latent": latent,
|
||||
}
|
||||
if self.kind == "video" or has_audio:
|
||||
# The DiT only reads audio off video / video_audio / audio blocks —
|
||||
# its "image" branch ignores audio_latent. So a mod carrying audio
|
||||
# is always emitted as video_audio (an image mod is just a 1-frame
|
||||
# video), which packs the audio rows immediately before the video
|
||||
# rows on the shared cursor.
|
||||
block["kind"] = "video_audio" if has_audio else "video"
|
||||
block["latent_t"] = self.latent_t
|
||||
block["ref_audio_t"] = self.ref_audio_t if has_audio else 0
|
||||
block["audio_latent"] = self.audio_latent if has_audio else None
|
||||
return block
|
||||
|
||||
# ── serialization ─────────────────────────────────────────────────
|
||||
|
||||
def metadata(self) -> Dict:
|
||||
meta = {
|
||||
"name": self.name,
|
||||
"kind": self.kind,
|
||||
"latent_h": self.latent_h,
|
||||
"latent_w": self.latent_w,
|
||||
"latent_t": self.latent_t,
|
||||
"mode": self.mode,
|
||||
"source": self.source,
|
||||
"source_shape": self.source_shape,
|
||||
"pool": self.pool,
|
||||
"optimize_steps": self.optimize_steps,
|
||||
"tags": self.tags,
|
||||
"description": self.description,
|
||||
"concept_type": self.concept_type,
|
||||
"_format_version": 4,
|
||||
}
|
||||
if self.kind == "audio" or (self.audio_latent is not None and self.ref_audio_t > 0):
|
||||
meta["audio_concept_type"] = self.audio_concept_type
|
||||
meta["ref_audio_t"] = self.ref_audio_t
|
||||
meta["sample_rate"] = self.sample_rate
|
||||
if self.enc_frames:
|
||||
meta["enc_times"] = self.enc_times
|
||||
meta["enc_fps"] = self.enc_fps
|
||||
return meta
|
||||
|
||||
def save(self, path_no_ext: str) -> str:
|
||||
"""Save as a single ``{path}.safetensors`` with metadata in the header."""
|
||||
os.makedirs(os.path.dirname(path_no_ext) or ".", exist_ok=True)
|
||||
meta = self.metadata()
|
||||
tensors = {"latent": self.latent.contiguous()}
|
||||
tensors.update({key: value.contiguous() for key, value in self.enc_frames.items()})
|
||||
if self.audio_latent is not None and self.ref_audio_t > 0:
|
||||
tensors["audio_latent"] = self.audio_latent.contiguous()
|
||||
save_file(tensors, path_no_ext + ".safetensors",
|
||||
metadata={META_KEY: json.dumps(meta)})
|
||||
return path_no_ext + ".safetensors"
|
||||
|
||||
@classmethod
|
||||
def from_metadata(
|
||||
cls,
|
||||
meta: Dict,
|
||||
latent: torch.Tensor,
|
||||
*,
|
||||
audio_latent: Optional[torch.Tensor] = None,
|
||||
enc_frames: Optional[Dict[str, torch.Tensor]] = None,
|
||||
) -> "H3RefMod":
|
||||
if not isinstance(meta, dict):
|
||||
raise ValueError("RefMod metadata must be a dict.")
|
||||
record = meta.get("encoder_images") or {}
|
||||
return cls(
|
||||
name=meta.get("name", "refmod"),
|
||||
kind=meta.get("kind", "image"),
|
||||
latent=latent,
|
||||
latent_h=int(meta.get("latent_h", latent.shape[3] if latent.ndim == 5 else 0)),
|
||||
latent_w=int(meta.get("latent_w", latent.shape[4] if latent.ndim == 5 else 0)),
|
||||
latent_t=int(meta.get("latent_t", latent.shape[2] if latent.ndim == 5 else latent.shape[-1])),
|
||||
mode=normalize_mode(meta.get("mode", "training")),
|
||||
source=meta.get("source", ""),
|
||||
source_shape=meta.get("source_shape", ""),
|
||||
pool=meta.get("pool", ""),
|
||||
optimize_steps=int(meta.get("optimize_steps", 0)),
|
||||
tags=list(meta.get("tags", [])),
|
||||
description=str(meta.get("description", "") or ""),
|
||||
concept_type=str(meta.get("concept_type", "generic") or "generic"),
|
||||
audio_concept_type=str(meta.get("audio_concept_type", "") or ""),
|
||||
audio_latent=audio_latent,
|
||||
ref_audio_t=int(meta.get("ref_audio_t", 0) or 0) if audio_latent is not None else 0,
|
||||
sample_rate=int(meta.get("sample_rate", 32000) or 32000),
|
||||
enc_frames=enc_frames or {},
|
||||
enc_times=list(record.get("times", meta.get("enc_times", []))),
|
||||
enc_fps=float(record.get("fps", meta.get("enc_fps", 0.0))),
|
||||
)
|
||||
|
||||
@classmethod
|
||||
def load(cls, path_no_ext: str, device: str = "cpu") -> "H3RefMod":
|
||||
"""Load from ``{path}.safetensors`` (metadata in header, or legacy .json)."""
|
||||
meta = read_refmod_meta(path_no_ext)
|
||||
if meta is None:
|
||||
raise ValueError(
|
||||
f"{path_no_ext}.safetensors has no RefMod metadata "
|
||||
f"(header key '{META_KEY}' or sidecar .json missing).")
|
||||
if meta.get("kind") == "bundle":
|
||||
raise ValueError("This is a RefMod bundle. Load its members instead of treating it as one latent.")
|
||||
# clone drops the file mmap, so the file isn't locked on Windows and
|
||||
# can be re-saved over the same name
|
||||
all_tensors = load_file(path_no_ext + ".safetensors", device=device)
|
||||
latent = all_tensors["latent"].clone()
|
||||
audio_latent = all_tensors["audio_latent"].clone() if "audio_latent" in all_tensors else None
|
||||
enc_frames = _read_encoder_images(meta, all_tensors.__getitem__)
|
||||
return cls.from_metadata(meta, latent, audio_latent=audio_latent, enc_frames=enc_frames)
|
||||
|
||||
@@ -0,0 +1,60 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
from typing import Any, Dict, Optional, Tuple
|
||||
|
||||
|
||||
_H3_VAE_CACHE: Dict[Tuple[str, str, str, bool], Tuple[Any, Optional[Any]]] = {}
|
||||
|
||||
|
||||
def _cache_key(video_path: str, audio_path: str, device, load_audio: bool) -> Tuple[str, str, str, bool]:
|
||||
return (
|
||||
os.path.abspath(video_path or ""),
|
||||
os.path.abspath(audio_path or ""),
|
||||
str(device),
|
||||
bool(load_audio),
|
||||
)
|
||||
|
||||
|
||||
def load_h3_vae_pair(video_path: str, audio_path: str = "", device=None,
|
||||
load_audio: bool = True) -> Tuple[Any, Optional[Any]]:
|
||||
"""Load and cache the H3 video/audio VAE wrappers from safetensors paths."""
|
||||
import comfy.model_management
|
||||
import comfy.sd
|
||||
import comfy.utils
|
||||
|
||||
if not video_path:
|
||||
raise ValueError("load_h3_vae_pair: video_path is required")
|
||||
device = device or comfy.model_management.get_torch_device()
|
||||
key = _cache_key(video_path, audio_path, device, load_audio)
|
||||
cached = _H3_VAE_CACHE.get(key)
|
||||
if cached is not None:
|
||||
return cached
|
||||
|
||||
sd, metadata = comfy.utils.load_torch_file(video_path, return_metadata=True)
|
||||
video_vae = comfy.sd.VAE(sd=sd, metadata=metadata, device=device)
|
||||
video_vae.throw_exception_if_invalid()
|
||||
|
||||
audio_vae = None
|
||||
if load_audio and audio_path:
|
||||
asd, ameta = comfy.utils.load_torch_file(audio_path, return_metadata=True)
|
||||
audio_vae = comfy.sd.VAE(sd=asd, metadata=ameta, device=device)
|
||||
audio_vae.throw_exception_if_invalid()
|
||||
|
||||
pair = (video_vae, audio_vae)
|
||||
_H3_VAE_CACHE[key] = pair
|
||||
return pair
|
||||
|
||||
|
||||
def load_h3_vaes_from_av_encoder(av_encoder, device=None,
|
||||
load_audio: bool = True) -> Tuple[Any, Optional[Any]]:
|
||||
"""Resolve a MiniMax H3 VAE-loader payload into local video/audio VAE wrappers."""
|
||||
video_path = getattr(av_encoder, "video_path", None)
|
||||
audio_path = getattr(av_encoder, "audio_path", None) or ""
|
||||
if not video_path:
|
||||
raise ValueError(
|
||||
"H3RefModExtract: av_encoder is missing video_path. Connect a valid "
|
||||
"MiniMax H3 VAE loader output."
|
||||
)
|
||||
return load_h3_vae_pair(video_path, audio_path, device=device,
|
||||
load_audio=load_audio)
|
||||
@@ -0,0 +1,138 @@
|
||||
"""Generate random PNG thumbnails for MP4 files in a directory.
|
||||
|
||||
Each thumbnail is extracted at full frame resolution from a random point
|
||||
between 25% and 75% of the video's duration, then saved beside the source MP4
|
||||
with the same basename and a `.png` extension.
|
||||
|
||||
Rerun the script to regenerate different thumbnails.
|
||||
|
||||
Usage:
|
||||
python3 generate_video_thumbnails.py /path/to/folder
|
||||
python3 generate_video_thumbnails.py /path/to/folder --seed 1234
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import random
|
||||
import subprocess # nosec B404
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def _probe_duration(path: Path) -> float:
|
||||
cmd = [
|
||||
"ffprobe",
|
||||
"-v",
|
||||
"error",
|
||||
"-show_entries",
|
||||
"format=duration",
|
||||
"-of",
|
||||
"default=noprint_wrappers=1:nokey=1",
|
||||
str(path),
|
||||
]
|
||||
result = subprocess.run(cmd, capture_output=True, text=True, check=True) # nosec B603
|
||||
value = (result.stdout or "").strip()
|
||||
if not value:
|
||||
raise ValueError(f"Could not read duration for {path}")
|
||||
duration = float(value)
|
||||
if duration <= 0.0:
|
||||
raise ValueError(f"Invalid duration for {path}: {duration}")
|
||||
return duration
|
||||
|
||||
|
||||
def _pick_timestamp(duration: float, rng: random.Random) -> float:
|
||||
if duration <= 0.0:
|
||||
return 0.0
|
||||
start = duration * 0.25
|
||||
end = duration * 0.75
|
||||
if end <= start:
|
||||
return duration * 0.5
|
||||
return rng.uniform(start, end)
|
||||
|
||||
|
||||
def _render_thumbnail(video_path: Path, timestamp: float) -> Path:
|
||||
output_path = video_path.with_suffix(".png")
|
||||
cmd = [
|
||||
"ffmpeg",
|
||||
"-y",
|
||||
"-ss",
|
||||
f"{timestamp:.6f}",
|
||||
"-i",
|
||||
str(video_path),
|
||||
"-frames:v",
|
||||
"1",
|
||||
"-update",
|
||||
"1",
|
||||
str(output_path),
|
||||
]
|
||||
subprocess.run(cmd, check=True, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) # nosec B603
|
||||
return output_path
|
||||
|
||||
|
||||
def _iter_mp4s(folder: Path) -> list[Path]:
|
||||
return sorted(
|
||||
path for path in folder.iterdir()
|
||||
if path.is_file() and path.suffix.lower() == ".mp4"
|
||||
)
|
||||
|
||||
|
||||
def _resolve_folder(folder_arg: str | None) -> Path:
|
||||
folder_text = folder_arg
|
||||
if not folder_text:
|
||||
try:
|
||||
folder_text = input("Folder containing MP4 files: ").strip()
|
||||
except EOFError:
|
||||
folder_text = ""
|
||||
if not folder_text:
|
||||
raise ValueError("No folder provided.")
|
||||
return Path(folder_text).expanduser().resolve()
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(
|
||||
description=(
|
||||
"Generate a random full-resolution PNG thumbnail for every MP4 in a folder. "
|
||||
"The capture time is chosen between 25% and 75% of each video's duration."
|
||||
)
|
||||
)
|
||||
parser.add_argument("folder", nargs="?", help="Folder containing MP4 files.")
|
||||
parser.add_argument(
|
||||
"--seed",
|
||||
type=int,
|
||||
default=None,
|
||||
help="Optional random seed for repeatable thumbnail choices.",
|
||||
)
|
||||
args = parser.parse_args()
|
||||
|
||||
try:
|
||||
folder = _resolve_folder(args.folder)
|
||||
except ValueError as exc:
|
||||
print(str(exc), file=sys.stderr)
|
||||
return 1
|
||||
if not folder.is_dir():
|
||||
print(f"Folder not found: {folder}", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
rng = random.Random(args.seed)
|
||||
videos = _iter_mp4s(folder)
|
||||
if not videos:
|
||||
print(f"No MP4 files found in {folder}", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
failures = 0
|
||||
for video_path in videos:
|
||||
try:
|
||||
duration = _probe_duration(video_path)
|
||||
timestamp = _pick_timestamp(duration, rng)
|
||||
output_path = _render_thumbnail(video_path, timestamp)
|
||||
print(f"{video_path.name} -> {output_path.name} at {timestamp:.2f}s / {duration:.2f}s")
|
||||
except Exception as exc:
|
||||
failures += 1
|
||||
print(f"Failed: {video_path.name}: {exc}", file=sys.stderr)
|
||||
|
||||
return 1 if failures else 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||