Merge commit 'c9eb575e2fd25f74b15f1d7336af4932c7fe6338' as 'vendor/dumas_h3_refmod_picker'

This commit is contained in:
2026-10-20 13:43:48 +00:00
30 changed files with 5783 additions and 0 deletions
+22
View File
@@ -0,0 +1,22 @@
# Python bytecode / caches
__pycache__/
*.py[cod]
*$py.class
# Virtual environments
.venv/
venv/
env/
# Editor / OS clutter
*.swp
*.swo
*~
.DS_Store
.directory
Thumbs.db
# Tooling
.history/
*.code-workspace
test/
+21
View File
@@ -0,0 +1,21 @@
MIT License
Copyright (c) 2026 Luisa (luisacaotica)
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
+89
View File
@@ -0,0 +1,89 @@
# ComfyUI-H3RefModPicker
Companion add-on for [ComfyUI-MiniMaxH3Mod](https://github.com/Luisacaotica/ComfyUI-MiniMaxH3Mod)
This add-on is meant to work alongside **MiniMaxH3Mod**. It provides a small, focused subset of RefMod features: a Visual Picker, a simple loader, an apply axis, a batch Create From Folder node, and grounded RefMods-to-Video generation with an optional reference collector. It keeps compatibility with both the original RefMod format and the newer bundle format.
If you want the fuller MiniMax H3 RefMod toolset and more creation/apply options, the recommended install is [ComfyUI-MiniMaxH3Mod](https://github.com/Luisacaotica/ComfyUI-MiniMaxH3Mod).
Please note, the `Create from Input` node was removed from this add-on. The same functionality is available in the official MiniMaxH3Mod add-on, using the `Create H3 RefMod Master` node.
## Latest Updates
Video-file support has been removed from `Create H3 RefMod From Folder`. Video clips are not ideal for the simple, lightweight RefMods this add-on aims to create: around **six to ten well-chosen reference images** are usually sufficient. Keep their aspect ratios consistent within each dataset folder. This is a recommendation, not an image limit; optional audio remains supported, and video files in the folder are ignored.
## Preview image naming
To have the Visual Picker pick up a preview image, place the image beside the RefMod using the same base name as the RefMod file.
- For the newer bundle format, the image and the RefMod just need to share the exact same name. Example: `character_refMod.safetensors` with `character_refMod.png`.
- For older split RefMods, do not name the preview image `_audio` or `_visual`. Use only the shared base name. Example: `character_refMod_visual.safetensors` and `character_refMod_audio.safetensors` should use `character_refMod.png`.
- When a matching `_visual` and `_audio` pair exists, plus that shared preview image, the picker sees them as one logical item.
<p align="center">
<img src="docs/examples/browser_example.png" alt="Extract H3 RefMod in use" />
<br />
<em>Example using refMods from <a href="https://huggingface.co/spaces/malcolmrey/browser">Malcolm Reynolds</a>.</em>
</p>
<p align="center">
<img src="docs/examples/workflow_example.png" alt="Workflow example" />
<br />
<em>Use Left and Right arrows when over Picker to switch quickly.</em>
</p>
<p align="center">
<img src="docs/examples/create_from_folder_example.png" alt="Create from folder example" />
<br />
<em>Create refMods from Folder example.</em>
</p>
## Main additions
- `Visual RefMod Picker` lets you browse using a **RefMods** browser. It supports legacy split RefMods and the newer single-file bundle format.
- For legacy split RefMods, found pairs are grouped as one item in the picker. They are shown as one entry and loaded together, with separate video/audio weight controls. This uses a weight behavior: `0..1` is regular strength, values above 1 expand into repeated copies. For example, a weight of 2.7 would be the same as strength: 1.0, copies: 2.7.
- `Create H3 RefMod From Folder` scans a folder of reference images and optional audio, and saves in the bundle format by default. Around six to ten images can be sufficient. It can also batch-create RefMods for all subfolders found. The optional `include_images` switch adds reconstructed encoder pictures to visual RefMods while keeping the existing latent format compatible; it is off by default.
- `Load RefMod Simple` is a singular loader. It supports standalone visual/audio RefMods and the new bundle format, while still using the same weight behavior as the Picker.
- `MiniMax H3 RefMods to Video` grounds prompt labels with RefMod pictures and applies their visual/audio latents, returning positive conditioning, an empty AV latent, and a reference map. Optional ordinary media can be added through `Collect H3 References`. It replaced minimax's Reference to Video node. It takes in mods as an input, bypassing the need to apply the RefMods using an apply node.
- `Apply H3 RefMod Simple` is a streamlined version of the Apply H3 RefMod, without the extra controls.
## RefMods to Video Node
The grounded RefMods-to-Video node builds on work by [Adudeguyman's ComfyUI-Fantastic-MiniMaxH3-PromptBuilder](https://github.com/Adudeguyman/ComfyUI-Fantastic-MiniMaxH3-PromptBuilder). This also revisits earlier visual-grounding experiments this add-on had initially, which were perhaps set aside a little too quickly :)
Unlike plain prompt encoding followed by RefMod Apply, this node gives the text encoder visual content behind labels such as `<Picture 1>` and `<Video 1>`, while supplying the corresponding latents to diffusion. This lets you refer to specific references in your prompt, but does not guarantee exclusive character or voice assignment.
Connect an H3 CLIP and the `mods` output from the Picker, Prompt Manager's `Prompt Composer`, a RefMod loader, or MiniMaxH3Mod's standard stack to `MiniMax H3 RefMods to Video`. Prompt Composer preserves embedded encoder images and returns weights above 1 as repeated stack rows, matching the Picker. For RefMod-only generation, leave `references` disconnected. To add ordinary media, connect `Collect H3 References`; connect video frame batches to `ref_video_N` and their soundtracks to the same-numbered `ref_video_audio_N`. The reference inputs grow as connections are added.
`video_vae` reconstructs visual RefMods without stored encoder pictures, and encodes raw images/videos. Encoder pictures stored by this creator or Fantastic can be read without decoding. `audio_vae` is needed to encode raw audio, not audio RefMods. As in the native reference node, raw media without its VAE only enters the encoder presentation. Latent-only visual RefMods require `video_vae`. Video playback is assumed to be 24 fps.
The `reference_map` string output contains only prompt labels and reference names, such as `<Video 1> = Alice_visual`. Connect it to a text display to see the mapping. Raw references are presented first (pictures, videos with soundtracks, standalone audio), followed by active RefMods in stack order. Counters are separate per media type. For example, if the map lists Alice as `<Video 1>` and Bob as `<Video 2>`:
```text
<Subject 1> is based on <Video 1>.
<Subject 2> is based on <Video 2>.
```
`stack_pictures` controls which reconstructed pictures from an image-stack RefMod the encoder sees. It never removes latent frames from the diffusion reference. Grounding helps the model associate labels with visual content; it does not enforce exclusive character assignments. Legacy combined visual/audio RefMods are supported. Do not Apply the same RefMods again after this node, as their latent references are already included.
### Include Encoder Images
`Create H3 RefMod From Folder` has an `include_images` switch, off by default. When enabled, it stores JPEG pictures reconstructed from the final visual latent, like Fantastic's creator, retaining every stored latent frame for image stacks. These are reconstructions, not copies of the original source images. Latents and audio are unchanged.
With the switch off, creation retains the existing latent-only behavior. Older RefMods without pictures remain supported by RefMods-to-Video through VAE reconstruction; adding pictures does not require converting existing files or change how their latents are applied.
The pictures are retained in both bundle and separate-file layouts and can be used by this add-on's RefMods-to-Video node without decoding again. Both layouts use Fantastic's `enc_N` tensor keys and `enc_times` / `enc_fps` metadata. In bundles, the picture metadata stays on the visual member; the optional audio member has no pictures. The version-5 bundle container is unchanged, so other loaders can read its standard latents, but reuse of its pictures depends on their bundle support. Enabling the switch adds a VAE decode during creation and increases file size. It also applies in subfolder mode and to unsaved mods passed directly to another node.
## Notes
- Supported RefMod files include older split `_visual` / `_audio` saves, single-file MiniMax bundles, and RefMods with stored encoder images, including the image-inclusive bundles added by this add-on. Existing latent-only files remain supported.
- For more options and the complete MiniMax H3 RefMod feature set, use [ComfyUI-MiniMaxH3Mod](https://github.com/Luisacaotica/ComfyUI-MiniMaxH3Mod).
## Installation
### Manual
```bash
cd ComfyUI/custom_nodes
git clone https://github.com/FranckyB/ComfyUI-H3RefModPicker
```
+36
View File
@@ -0,0 +1,36 @@
"""
ComfyUI-H3RefModPicker — Addons to MiniMaxH3Mod, adds a visual picker and a batch creation tool.
"""
__author__ = "FranckB"
__version__ = "0.2.0"
from .nodes import refmod_apply
from .nodes import refmod_loader
from .nodes import refmod_visual_picker
from .nodes import refmod_create
from .nodes import refmod_axis
from .nodes import refmods_to_video
# from .nodes import refmod_decode
NODE_CLASS_MAPPINGS = {
**refmod_apply.NODE_CLASS_MAPPINGS,
**refmod_loader.NODE_CLASS_MAPPINGS,
**refmod_visual_picker.NODE_CLASS_MAPPINGS,
**refmod_create.NODE_CLASS_MAPPINGS,
**refmod_axis.NODE_CLASS_MAPPINGS,
**refmods_to_video.NODE_CLASS_MAPPINGS,
# **refmod_decode.NODE_CLASS_MAPPINGS,
}
NODE_DISPLAY_NAME_MAPPINGS = {
**refmod_apply.NODE_DISPLAY_NAME_MAPPINGS,
**refmod_loader.NODE_DISPLAY_NAME_MAPPINGS,
**refmod_visual_picker.NODE_DISPLAY_NAME_MAPPINGS,
**refmod_create.NODE_DISPLAY_NAME_MAPPINGS,
**refmod_axis.NODE_DISPLAY_NAME_MAPPINGS,
**refmods_to_video.NODE_DISPLAY_NAME_MAPPINGS,
# **refmod_decode.NODE_DISPLAY_NAME_MAPPINGS,
}
WEB_DIRECTORY = "./js"
__all__ = ["NODE_CLASS_MAPPINGS", "NODE_DISPLAY_NAME_MAPPINGS", "WEB_DIRECTORY"]
Binary file not shown.

After

Width:  |  Height:  |  Size: 1.6 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 75 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.3 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 43 KiB

File diff suppressed because one or more lines are too long
Binary file not shown.

After

Width:  |  Height:  |  Size: 112 KiB

File diff suppressed because one or more lines are too long
Binary file not shown.

After

Width:  |  Height:  |  Size: 1.0 MiB

File diff suppressed because one or more lines are too long
Binary file not shown.

After

Width:  |  Height:  |  Size: 75 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 29 KiB

+392
View File
@@ -0,0 +1,392 @@
import { app } from "../../scripts/app.js";
const TARGET_NODES = new Set([
"H3RefModCreateFromFolder",
]);
const MIN_NODE_WIDTH = 320;
const CREATE_FOLDER_INSTRUCTIONS = [
"Keep image count to a minimum; around six to ten well-chosen",
"images can suffice. Add audio if wanted. Videos are ignored.",
"",
"Folder: absolute path or folder inside ComfyUI's input/.",
"Connect the H3 video VAE; audio VAE is optional.",
"use_subfolders creates one RefMod per immediate subfolder.",
"include_images stores grounding pictures; off by default.",
"Saves to models/refmods/ when save is enabled.",
"Or use directly as a mod, by connecting to mods output.",
].join("\n");
function addAspectRatioReminder(node) {
let tooltip = null;
let tooltipStyle = null;
function hideTooltip() {
tooltip?.remove();
tooltip = null;
tooltipStyle?.remove();
tooltipStyle = null;
}
const onDrawForeground = node.onDrawForeground;
node.onDrawForeground = function (context) {
const result = onDrawForeground?.apply(this, arguments);
if (!this.flags?.collapsed) {
context.save();
context.fillStyle = "#facc15";
context.font = "bold 18px sans-serif";
context.textAlign = "center";
context.textBaseline = "middle";
context.fillText("?", this.size[0] - 18, -15);
context.restore();
}
return result;
};
const onMouseMove = node.onMouseMove;
node.onMouseMove = function (event, position) {
const result = onMouseMove?.apply(this, arguments);
const overReminder = !this.flags?.collapsed && position &&
Math.abs(position[0] - (this.size[0] - 18)) <= 12 &&
Math.abs(position[1] + 15) <= 12;
if (!overReminder) {
hideTooltip();
return result;
}
if (!tooltip) {
tooltip = document.createElement("div");
tooltip.setAttribute("role", "tooltip");
const warning = document.createElement("strong");
warning.textContent = "Warning: images should all have the same aspect ratio within each dataset folder.";
warning.style.cssText = "display:block;margin-bottom:8px;color:#facc15;";
tooltip.append(warning, document.createTextNode(CREATE_FOLDER_INSTRUCTIONS));
tooltip.style.cssText = "position:fixed;z-index:10000;pointer-events:none;box-sizing:border-box;max-width:min(480px,calc(100vw - 16px));padding:12px 14px;border:1px solid #facc15;border-radius:4px;background:#252525;color:#fff;font-size:15px;line-height:1.45;white-space:pre-line;overflow-wrap:anywhere;";
tooltipStyle = document.createElement("style");
tooltipStyle.textContent = ".node-tooltip { display: none !important; }";
document.body.append(tooltip, tooltipStyle);
}
const bounds = tooltip.getBoundingClientRect();
tooltip.style.left = `${Math.max(8, Math.min(event.clientX + 12, window.innerWidth - bounds.width - 8))}px`;
tooltip.style.top = `${Math.max(8, Math.min(event.clientY + 12, window.innerHeight - bounds.height - 8))}px`;
return result;
};
const onMouseLeave = node.onMouseLeave;
node.onMouseLeave = function () {
hideTooltip();
return onMouseLeave?.apply(this, arguments);
};
const onRemoved = node.onRemoved;
node.onRemoved = function () {
hideTooltip();
return onRemoved?.apply(this, arguments);
};
}
const COMMON_PRESET_FIELDS = [
"concept_type",
"audio_concept_type",
"mode",
"ref_resolution",
"pool_h",
"pool_w",
"max_tokens",
"identity",
"merge",
"multiplier",
"audio_max_seconds",
"audio_max_tokens",
"audio_budget_policy",
"max_total_tokens",
"budget_policy",
];
const PRESET_CONFIG = {
H3RefModCreateFromFolder: {
manual: {
values: {
concept_type: "identity",
audio_concept_type: "voice",
mode: "Full Reference",
ref_resolution: 1024,
pool_h: 16,
pool_w: 16,
max_tokens: 8192,
identity: 500,
merge: false,
multiplier: 1,
audio_max_seconds: 30.0,
audio_max_tokens: 5120,
audio_budget_policy: "error",
max_total_tokens: 0,
budget_policy: "truncate",
},
fields: [],
compactFields: [],
},
identity_encode: {
values: {
concept_type: "identity",
audio_concept_type: "voice",
mode: "Full Reference",
ref_resolution: 1024,
pool_h: 16,
pool_w: 16,
max_tokens: 8192,
identity: 0,
merge: false,
multiplier: 1,
audio_max_seconds: 30.0,
audio_max_tokens: 5120,
audio_budget_policy: "error",
max_total_tokens: 0,
budget_policy: "truncate",
},
fields: [
"concept_type",
"audio_concept_type",
"mode",
"ref_resolution",
"identity",
"merge",
],
compactFields: COMMON_PRESET_FIELDS,
},
style_experimental: {
values: {
concept_type: "style",
audio_concept_type: "voice",
mode: "Compressed Reference",
ref_resolution: 1024,
pool_h: 8,
pool_w: 8,
max_tokens: 8192,
identity: 150,
merge: false,
multiplier: 1,
audio_max_seconds: 30.0,
audio_max_tokens: 5120,
audio_budget_policy: "error",
max_total_tokens: 0,
budget_policy: "truncate",
},
fields: [
"concept_type",
"audio_concept_type",
"mode",
"pool_h",
"pool_w",
"identity",
"merge",
],
compactFields: COMMON_PRESET_FIELDS,
},
motion_sequence: {
values: {
concept_type: "pose_motion",
audio_concept_type: "voice",
mode: "Compressed Reference",
ref_resolution: 1024,
pool_h: 16,
pool_w: 16,
max_tokens: 8192,
identity: 500,
merge: false,
multiplier: 1,
audio_max_seconds: 30.0,
audio_max_tokens: 5120,
audio_budget_policy: "error",
max_total_tokens: 0,
budget_policy: "truncate",
},
fields: [
"concept_type",
"audio_concept_type",
"mode",
"pool_h",
"pool_w",
"merge",
],
compactFields: COMMON_PRESET_FIELDS,
},
},
};
function findWidget(node, name) {
return node.widgets?.find((widget) => widget?.name === name) || null;
}
function moveWidgetAfter(node, widgetName, afterName) {
if (!node.widgets) return;
const widget = findWidget(node, widgetName);
const after = findWidget(node, afterName);
if (!widget || !after) return;
const from = node.widgets.indexOf(widget);
const afterIndex = node.widgets.indexOf(after);
if (from < 0 || afterIndex < 0) return;
node.widgets.splice(from, 1);
const insertAt = node.widgets.indexOf(after) + 1;
node.widgets.splice(insertAt, 0, widget);
}
function setWidgetVisible(widget, visible) {
if (!widget) return;
if (!widget.__h3VisibilityWrapped) {
widget.__h3VisibilityWrapped = true;
widget.__h3BaseComputeSize = widget.computeSize;
widget.computeSize = function (...args) {
if (widget.__h3Hidden) {
return [0, -4];
}
if (typeof widget.__h3BaseComputeSize === "function") {
return widget.__h3BaseComputeSize.apply(this, args);
}
return [MIN_NODE_WIDTH, Number(LiteGraph?.NODE_WIDGET_HEIGHT || 24)];
};
}
if (typeof widget.__h3OriginalDisplay === "undefined" && widget.inputEl) {
widget.__h3OriginalDisplay = widget.inputEl.style.display;
}
widget.__h3Hidden = !visible;
widget.hidden = !visible;
if (widget.inputEl) {
widget.inputEl.style.display = visible ? (widget.__h3OriginalDisplay || "") : "none";
}
}
function refreshNodeLayout(node) {
if (Array.isArray(node.size)) {
node.size[0] = Math.max(Number(node.size[0]) || 0, MIN_NODE_WIDTH);
}
const nextWidth = Math.max(node.size?.[0] || 0, MIN_NODE_WIDTH);
const nextHeight = node.computeSize?.()?.[1] || node.size?.[1] || 0;
node.setSize?.([nextWidth, nextHeight]);
node.onResize?.(node.size);
node.setDirtyCanvas(true, true);
app.graph?.setDirtyCanvas(true, true);
}
function syncPresetValues(node) {
const presetWidget = findWidget(node, "extraction_preset");
const preset = String(presetWidget?.value || "manual");
const nodeType = String(node.comfyClass || "");
const nodePresets = PRESET_CONFIG[nodeType];
if (!nodePresets) return;
const config = nodePresets[preset] || nodePresets.manual;
for (const [name, value] of Object.entries(config.values)) {
const widget = findWidget(node, name);
if (!widget) continue;
widget.value = value;
if (widget.inputEl) {
widget.inputEl.value = String(value);
if (typeof widget.inputEl.checked === "boolean" && typeof value === "boolean") {
widget.inputEl.checked = value;
}
}
}
}
function applyFolderNamingVisibility(node) {
if (String(node.comfyClass || "") !== "H3RefModCreateFromFolder") return;
const useSubfoldersWidget = findWidget(node, "use_subfolders");
const useFolderAsNameWidget = findWidget(node, "use_folder_as_name");
const nameWidget = findWidget(node, "name");
const useSubfolders = Boolean(useSubfoldersWidget?.value);
const useFolderAsName = Boolean(useFolderAsNameWidget?.value);
setWidgetVisible(useFolderAsNameWidget, !useSubfolders);
setWidgetVisible(nameWidget, !useSubfolders && !useFolderAsName);
}
function applyPresetVisibility(node) {
const presetWidget = findWidget(node, "extraction_preset");
const advancedWidget = findWidget(node, "advanced");
const preset = String(presetWidget?.value || "manual");
const advanced = Boolean(advancedWidget?.value);
const nodeType = String(node.comfyClass || "");
const nodePresets = PRESET_CONFIG[nodeType];
if (!nodePresets) return;
const config = nodePresets[preset] || nodePresets.manual;
const active = new Set(advanced ? config.fields : (config.compactFields || config.fields));
const known = new Set(Object.values(nodePresets).flatMap((configEntry) => [
...(configEntry.fields || []),
...(configEntry.compactFields || []),
]));
for (const name of known) {
setWidgetVisible(findWidget(node, name), !(preset !== "manual" && active.has(name)));
}
setWidgetVisible(advancedWidget, preset !== "manual");
applyFolderNamingVisibility(node);
refreshNodeLayout(node);
}
function attachWidgetRefresh(node, widgetName, onAfterChange) {
const widget = findWidget(node, widgetName);
if (!widget || widget.__h3RefreshAttached) return;
widget.__h3RefreshAttached = true;
const originalCallback = widget.callback;
widget.callback = function () {
const result = originalCallback?.apply(this, arguments);
onAfterChange();
return result;
};
}
function enhanceCreateNode(node) {
if (node.__h3CreateUiEnhanced) return;
node.__h3CreateUiEnhanced = true;
const originalOnResize = node.onResize;
node.onResize = function (size) {
if (size) {
size[0] = Math.max(size[0] || 0, MIN_NODE_WIDTH);
}
return originalOnResize?.apply(this, arguments);
};
addAspectRatioReminder(node);
moveWidgetAfter(node, "extraction_preset", "name");
moveWidgetAfter(node, "advanced", "extraction_preset");
moveWidgetAfter(node, "budget_policy", "max_total_tokens");
syncPresetValues(node);
applyPresetVisibility(node);
attachWidgetRefresh(node, "extraction_preset", () => {
syncPresetValues(node);
applyPresetVisibility(node);
});
attachWidgetRefresh(node, "advanced", () => applyPresetVisibility(node));
attachWidgetRefresh(node, "use_subfolders", () => applyPresetVisibility(node));
attachWidgetRefresh(node, "use_folder_as_name", () => applyPresetVisibility(node));
const onConfigure = node.onConfigure;
node.onConfigure = function () {
const result = onConfigure?.apply(this, arguments);
moveWidgetAfter(this, "extraction_preset", "name");
moveWidgetAfter(this, "advanced", "extraction_preset");
moveWidgetAfter(this, "budget_policy", "max_total_tokens");
syncPresetValues(this);
applyPresetVisibility(this);
attachWidgetRefresh(this, "extraction_preset", () => {
syncPresetValues(this);
applyPresetVisibility(this);
});
attachWidgetRefresh(this, "advanced", () => applyPresetVisibility(this));
attachWidgetRefresh(this, "use_subfolders", () => applyPresetVisibility(this));
attachWidgetRefresh(this, "use_folder_as_name", () => applyPresetVisibility(this));
return result;
};
}
app.registerExtension({
name: "H3RefModPicker.RefModCreate",
async beforeRegisterNodeDef(nodeType, nodeData) {
if (!TARGET_NODES.has(nodeData?.name)) return;
const onNodeCreated = nodeType.prototype.onNodeCreated;
nodeType.prototype.onNodeCreated = function () {
const result = onNodeCreated?.apply(this, arguments);
enhanceCreateNode(this);
return result;
};
},
});
console.log("[H3RefModPicker] RefModCreate extension loaded");
+116
View File
@@ -0,0 +1,116 @@
import { app } from "../../scripts/app.js";
import { api } from "../../scripts/api.js";
function findWidget(node, name) {
return node.widgets?.find((widget) => widget?.name === name) || null;
}
function setWidgetVisible(widget, visible) {
if (!widget) return;
if (!widget.__h3LoaderVisibilityWrapped) {
widget.__h3LoaderVisibilityWrapped = true;
widget.__h3LoaderBaseComputeSize = widget.computeSize;
widget.computeSize = function (...args) {
if (widget.__h3LoaderHidden) {
return [0, -4];
}
if (typeof widget.__h3LoaderBaseComputeSize === "function") {
return widget.__h3LoaderBaseComputeSize.apply(this, args);
}
return [200, Number(LiteGraph?.NODE_WIDGET_HEIGHT || 24)];
};
}
if (typeof widget.__h3LoaderOriginalDisplay === "undefined" && widget.inputEl) {
widget.__h3LoaderOriginalDisplay = widget.inputEl.style.display;
}
widget.__h3LoaderHidden = !visible;
widget.hidden = !visible;
if (widget.inputEl) {
widget.inputEl.style.display = visible ? (widget.__h3LoaderOriginalDisplay || "") : "none";
}
}
function refreshNodeLayout(node) {
node.setDirtyCanvas(true, true);
app.graph?.setDirtyCanvas(true, true);
}
async function fetchLoaderCapabilities(mod) {
const query = mod ? `?mod=${encodeURIComponent(mod)}` : "";
const resp = await api.fetchApi(`/h3refmods/refmod-loader/capabilities${query}`);
if (!resp.ok) {
throw new Error(`Request failed (${resp.status})`);
}
return await resp.json();
}
function applyModalityVisibility(node, caps) {
const videoWidget = findWidget(node, "video_weight");
const audioWidget = findWidget(node, "audio_weight");
const hasVisual = typeof caps?.has_visual === "boolean" ? caps.has_visual : true;
const hasAudio = typeof caps?.has_audio === "boolean" ? caps.has_audio : true;
setWidgetVisible(videoWidget, hasVisual);
setWidgetVisible(audioWidget, hasAudio);
refreshNodeLayout(node);
}
async function refreshLoaderVisibility(node) {
const modWidget = findWidget(node, "mod");
const mod = String(modWidget?.value || "").trim();
if (!mod) {
applyModalityVisibility(node, { has_visual: true, has_audio: true });
return;
}
try {
const data = await fetchLoaderCapabilities(mod);
applyModalityVisibility(node, data.ok ? data : { has_visual: true, has_audio: true });
} catch (error) {
console.warn("[H3RefModLoader] Could not refresh capabilities:", error);
applyModalityVisibility(node, { has_visual: true, has_audio: true });
}
}
app.registerExtension({
name: "H3RefModPicker.RefModLoader",
async beforeRegisterNodeDef(nodeType, nodeData) {
if (nodeData?.name !== "H3RefModLoader") return;
const onNodeCreated = nodeType.prototype.onNodeCreated;
nodeType.prototype.onNodeCreated = function () {
const result = onNodeCreated?.apply(this, arguments);
const node = this;
const legacyWeightWidget = findWidget(node, "weight");
const modWidget = findWidget(node, "mod");
if (legacyWeightWidget) {
setWidgetVisible(legacyWeightWidget, false);
}
if (modWidget && !modWidget.__h3LoaderRefreshAttached) {
modWidget.__h3LoaderRefreshAttached = true;
const originalCallback = modWidget.callback;
modWidget.callback = function () {
const callbackResult = originalCallback?.apply(this, arguments);
void refreshLoaderVisibility(node);
return callbackResult;
};
}
const onConfigure = node.onConfigure;
node.onConfigure = function () {
const configureResult = onConfigure?.apply(this, arguments);
if (legacyWeightWidget) {
setWidgetVisible(legacyWeightWidget, false);
}
void refreshLoaderVisibility(this);
return configureResult;
};
void refreshLoaderVisibility(node);
return result;
};
},
});
console.log("[H3RefModPicker] RefModLoader extension loaded");
File diff suppressed because it is too large Load Diff
+183
View File
@@ -0,0 +1,183 @@
"""
H3RefModApplySimple — identity-oriented RefMod apply node with one strength dial
"""
from __future__ import annotations
import random
from dataclasses import replace
from typing import Dict, List, Optional
from comfy_api.latest import io
# reference retention presets (master strength multiplier on Apply)
RETENTION = {
"fully_preserved": 1.0,
"partially_preserved": 0.7,
"attribute_transfer": 0.4,
"weak_reference": 0.15,
}
def _filter_legacy_ref_block(block: Optional[Dict],
use_video: bool,
use_audio: bool) -> Optional[Dict]:
"""Apply media toggles to ref blocks produced by older RefMod APIs."""
if block is None or (not use_video and not use_audio):
return None
kind = str(block.get("kind") or "").strip().lower()
has_audio = block.get("audio_latent") is not None and int(block.get("ref_audio_t") or 0) > 0
if kind == "audio":
return block if use_audio else None
if not use_video:
if use_audio and has_audio:
return {
"kind": "audio",
"ref_audio_t": int(block.get("ref_audio_t") or 0),
"audio_latent": block.get("audio_latent"),
}
return None
if not use_audio and has_audio:
filtered = dict(block)
filtered["kind"] = "video"
filtered["ref_audio_t"] = 0
filtered["audio_latent"] = None
return filtered
return block
def _build_ref_block(mod, eff: float, curve, use_video: bool, use_audio: bool) -> Optional[Dict]:
try:
return mod.ref_block(eff, curve=curve, use_video=use_video, use_audio=use_audio)
except TypeError as exc:
message = str(exc)
if "unexpected keyword argument 'use_video'" not in message and "unexpected keyword argument 'use_audio'" not in message:
raise
legacy_block = mod.ref_block(eff, curve=curve)
return _filter_legacy_ref_block(legacy_block, use_video=use_video, use_audio=use_audio)
def _ref_blocks(mods, retention, curve=None, seed=-1,
use_video: bool = True, use_audio: bool = True) -> List[Dict]:
"""Ref blocks for a loader bundle, scaled by row strength x retention.
``retention`` is a master strength multiplier: a float 0-1 (1.0 =
fully_preserved, 0.7 = partially_preserved, 0.4 = attribute_transfer,
0.15 = weak_reference), or one of those preset names for legacy
workflows saved with the old combo widget.
``curve`` (optional) is a per-frame strength spec — a ``(direction,
shape, value)`` tuple, a legacy preset name, per-frame values, control
points (see ``core.curve_strengths``) — applied on top of the row
strength. A flat/no curve keeps today's behavior.
``seed`` (default -1 = off) enables ref scrambling: with 2+ refs in the
bundle, the order is shuffled and a random subset kept, so a different
ref leads each run instead of the same one always "popping". Same seed
-> same scramble; connect/randomize the seed for per-run variation.
"""
if isinstance(retention, str):
factor = RETENTION.get(retention, 1.0)
else:
factor = float(retention)
items = [] if mods is None else list(mods)
if int(seed) >= 0 and len(items) > 1:
rng = random.Random(int(seed))
rng.shuffle(items)
keep = rng.randint(max(1, len(items) // 2), len(items))
items = items[:keep]
print(f"[H3RefModApply] scramble seed={int(seed)}: "
f"{len(mods)} refs -> kept {len(items)} (order shuffled)")
blocks = []
debug_rows = []
for mod, strength in items:
eff = min(1.0, max(0.0, strength * factor))
block = _build_ref_block(mod, eff, curve=curve, use_video=use_video, use_audio=use_audio)
if block is not None:
blocks.append(block)
debug_rows.append(f"{mod.name}@{eff:.2f}")
else:
debug_rows.append(f"{mod.name}@{eff:.2f} (skipped)")
if debug_rows:
print("[H3RefModApply] effective refs: " + ", ".join(debug_rows))
return blocks
class H3RefModApplySimple(io.ComfyNode):
"""Identity-oriented RefMod apply node with one strength dial."""
@classmethod
def define_schema(cls):
template = io.MatchType.Template(
"cond",
allowed_types=[io.Custom("MINIMAX_H3_COND"), io.Conditioning])
return io.Schema(
node_id="H3RefModApplySimple",
display_name="Apply H3 RefMod Simple",
description=(
"Apply one or more RefMods to a MiniMax H3 conditioning with a single "
"Strength control. This identity-oriented version uses a flat full-length "
"reference curve under the hood."
),
category="H3RefModPicker",
inputs=[
io.MatchType.Input("conditioning", template=template,
tooltip="MINIMAX_H3_COND (ComfyUI-MiniMaxH3 pack) or CONDITIONING "
"(core MiniMaxH3ReferenceToVideo)."),
io.Custom("H3_REF_MODS").Input("mods",
optional=True,
tooltip="Bundle of RefMods to inject. Leave unconnected to bypass unchanged."),
io.Boolean.Input("use_video", default=True,
tooltip="Apply the visual latent from each RefMod. Turn this off to use only embedded audio."),
io.Boolean.Input("use_audio", default=True,
tooltip="Apply the embedded audio latent from each RefMod when present. Turn this off for visual-only application."),
io.Float.Input("strength", default=1.0, min=0.0, max=1.0, step=0.01,
tooltip="Master reference strength. For identity work this is the main "
"dial: higher = tighter identity lock, lower = more freedom but "
"more drift."),
],
outputs=[
io.MatchType.Output(template=template, display_name="conditioning",
tooltip="The conditioning with the ref blocks injected, same type as the input."),
],
)
@classmethod
def execute(cls, conditioning, mods, use_video=True, use_audio=True, strength=1.0):
if mods is None:
print("[H3RefModApplySimple] no mods connected — bypassing conditioning unchanged")
return io.NodeOutput(conditioning)
if not use_video and not use_audio:
print("[H3RefModApplySimple] use_video=False and use_audio=False — bypassing conditioning unchanged")
return io.NodeOutput(conditioning)
blocks = _ref_blocks(mods, strength, ("constant", "linear", 1.0), seed=-1,
use_video=bool(use_video), use_audio=bool(use_audio))
if isinstance(conditioning, list):
out = []
for t in conditioning:
d = dict(t[1])
d["minimax_refs"] = list(d.get("minimax_refs", [])) + blocks
out.append([t[0], d])
print(f"[H3RefModApplySimple] strength={strength} "
f"({len(blocks)} ref block(s) injected)")
else:
out = replace(conditioning, refs=list(conditioning.refs) + blocks)
print(f"[H3RefModApplySimple] strength={strength} "
f"({len(blocks)} ref block(s) injected, {len(out.refs)} total)")
return io.NodeOutput(out)
NODE_CLASS_MAPPINGS = {
"H3RefModApplySimple": H3RefModApplySimple,
}
NODE_DISPLAY_NAME_MAPPINGS = {
"H3RefModApplySimple": "Apply H3 RefMod Simple",
}
__all__ = ["NODE_CLASS_MAPPINGS", "NODE_DISPLAY_NAME_MAPPINGS"]
+186
View File
@@ -0,0 +1,186 @@
"""
H3RefModApplyAxis — choose between two picker bundles with separate video/audio signed values
Think of it as creating a Slider from two RefMods
"""
from __future__ import annotations
import os
from typing import List, Optional, Tuple
from ..py.refmod_common import (
_prompt_hint,
)
from ..py.refmod_core import (
H3RefMod,
)
_PACK_DIR = os.path.dirname(os.path.abspath(__file__))
LEGACY_MODS_DIR = os.path.join(_PACK_DIR, "mods") # pre-models/refmods storage, still read
# reference retention presets (master strength multiplier on Apply)
RETENTION = {
"fully_preserved": 1.0,
"partially_preserved": 0.7,
"attribute_transfer": 0.4,
"weak_reference": 0.15,
}
def _info_lines(mod):
opt = mod.optimize_steps
if mod.mode == "encode":
opt = f"n/a ({mod.optimize_steps} — encode mode stores the actual encode)"
return [
"=" * 52,
f" MiniMax H3 RefMod: {mod.name}",
f" {'concept_type':<18} {mod.concept_type}",
f" {'mode':<18} {mod.mode}",
f" {'kind':<18} {mod.kind}",
f" {'latent':<18} {tuple(mod.latent.shape)}",
f" {'tokens injected':<18} {mod.token_count}",
f" {'source':<18} {mod.source} ({mod.source_shape})",
f" {'pool':<18} {mod.pool}",
f" {'identity':<18} {opt}",
f" {'tags':<18} {', '.join(mod.tags) if mod.tags else '-'}",
f" {'description':<18} {mod.description or '-'}",
"=" * 52,
]
def _mod_group_key(mod: H3RefMod) -> str:
name = str(getattr(mod, "name", "") or "")
lower_name = name.lower()
for suffix in ("_Video", "_Audio"):
if lower_name.endswith(suffix.lower()):
return name[:-len(suffix)]
return name
def _split_trailing_selection(rows, label: str) -> Tuple[List[Tuple[H3RefMod, float]], Optional[Tuple[H3RefMod, float]], Optional[Tuple[H3RefMod, float]]]:
"""Return the last logical RefMod selection from a bundle."""
items = [] if rows is None else list(rows)
if not items:
return [], None, None
last_key = _mod_group_key(items[-1][0])
start = len(items) - 1
while start > 0 and _mod_group_key(items[start - 1][0]) == last_key:
start -= 1
group = items[start:]
if start > 0:
print(f"[H3RefModApplyAxis] warning: {label} input contains {start} earlier row(s); using only the last RefMod selection '{last_key}'.")
video_row = None
audio_row = None
for row in group:
mod, _strength = row
if mod.kind == "audio":
audio_row = row
else:
video_row = row
return group, video_row, audio_row
class H3RefModApplyAxis:
"""Choose between two picker bundles with separate video/audio signed values."""
@classmethod
def INPUT_TYPES(cls):
return {
"required": {
"mods_a": ("H3_REF_MODS", {
"tooltip": "RefMod bundle from one Visual RefMod Picker. If older chained selections are present too, only the last logical RefMod selection is used.",
}),
"mods_b": ("H3_REF_MODS", {
"tooltip": "RefMod bundle from another Visual RefMod Picker. If older chained selections are present too, only the last logical RefMod selection is used.",
}),
"video_value": ("FLOAT", {
"default": 0.0, "min": -1.0, "max": 1.0, "step": 0.01,
"display": "number",
"tooltip": "Signed video selector: negative uses A video, positive uses B video. Magnitude becomes the output video strength. 0 skips video.",
}),
"audio_value": ("FLOAT", {
"default": 0.0, "min": -1.0, "max": 1.0, "step": 0.01,
"display": "number",
"tooltip": "Signed audio selector: negative uses A audio, positive uses B audio. If only one side has audio, that audio is used whenever this is non-zero.",
}),
"show_info": ("BOOLEAN", {
"default": False,
"tooltip": "Print the selected output RefMods and strengths to the console.",
}),
},
}
RETURN_TYPES = ("H3_REF_MODS", "STRING")
RETURN_NAMES = ("mods", "prompt_hint")
FUNCTION = "select"
CATEGORY = "H3RefModPicker"
def select(self, mods_a, mods_b, video_value=0.0, audio_value=0.0, show_info=False):
_group_a, video_a, audio_a = _split_trailing_selection(mods_a, "A")
_group_b, video_b, audio_b = _split_trailing_selection(mods_b, "B")
loads: List[Tuple[H3RefMod, float]] = []
def pick_signed(value: float, row_a, row_b, kind: str):
mag = min(1.0, abs(float(value)))
if mag < 1e-6:
return None
chosen = row_b if value > 0 else row_a
chosen_side = "B" if value > 0 else "A"
fallback = row_a if value > 0 else row_b
fallback_side = "A" if value > 0 else "B"
if chosen is None:
if fallback is not None:
print(f"[H3RefModApplyAxis] warning: {kind} {chosen_side} is missing; using {fallback_side} {kind} at {mag:.2f}.")
return fallback[0], mag
print(f"[H3RefModApplyAxis] warning: no {kind} RefMod available on either side.")
return None
return chosen[0], mag
video_pick = pick_signed(video_value, video_a, video_b, "video")
if video_pick is not None:
loads.append(video_pick)
if audio_a is not None and audio_b is not None:
audio_pick = pick_signed(audio_value, audio_a, audio_b, "audio")
elif audio_a is not None or audio_b is not None:
mag = min(1.0, abs(float(audio_value)))
if mag < 1e-6:
audio_pick = None
else:
only = audio_a or audio_b
side = "A" if audio_a is not None else "B"
print(f"[H3RefModApplyAxis] audio exists only on {side}; using that audio at {mag:.2f}.")
audio_pick = (only[0], mag)
else:
audio_pick = None
if audio_pick is not None:
loads.append(audio_pick)
if loads:
print("[H3RefModApplyAxis] " + ", ".join(
f"{mod.name}@{strength:.2f}" for mod, strength in loads
))
else:
print("[H3RefModApplyAxis] no output refs selected")
if show_info:
for mod, strength in loads:
print("\n".join(_info_lines(mod)))
print(f" {'strength':<18} {strength:.2f}")
hint = _prompt_hint(loads)
if hint:
print(f"[H3RefModApplyAxis] prompt_hint: {hint}")
return (loads, hint)
# ═══════════════════════════════════════════════════════════════════════════
# Registration
# ═══════════════════════════════════════════════════════════════════════════
NODE_CLASS_MAPPINGS = {
"H3RefModApplyAxis": H3RefModApplyAxis,
}
NODE_DISPLAY_NAME_MAPPINGS = {
"H3RefModApplyAxis": "Apply H3 RefMod Axis",
}
__all__ = ["NODE_CLASS_MAPPINGS", "NODE_DISPLAY_NAME_MAPPINGS"]
File diff suppressed because it is too large Load Diff
+116
View File
@@ -0,0 +1,116 @@
"""
decode_refmod.py — H3RefModDecode node.
Decode a saved RefMod's latent back into images, so you can see what the mod
actually captured. Purely a debugging / inspection tool — it plays no part in
generation.
A RefMod stores a video-VAE latent [1, 24, T, h, w] (the reference). This node
runs it back through the H3 video VAE decoder and returns the frames as an IMAGE
batch [T, H, W, 3], so you can preview a mod, sanity-check an extraction, or
eyeball what a pooled (training-mode) mod actually kept vs an encode-mode one.
It decodes the full latent timeline exactly as stored in the RefMod; it does
not resample or guess a frame subset.
Note: a training-mode (pooled) mod decodes to a tiny, blurry grid — that's the
point of the mode, it only stores a concept thumbnail. An encode-mode mod
decodes to recognizable full-res frames. Audio (if the mod has it) is not
decoded here — this is the visual reference only.
"""
from __future__ import annotations
import torch
import torch.nn.functional as F
import comfy.utils
from comfy_api.latest import io
from .refmod_loader import _list_mod_names, _load_mod
def _reshape_decoded_frames(frames: torch.Tensor) -> torch.Tensor:
if frames.dim() == 5: # [B, T, H, W, C] -> [B*T, H, W, C]
frames = frames.reshape(-1, frames.shape[-3], frames.shape[-2], frames.shape[-1])
return frames
def _upscale_for_inspection(frames: torch.Tensor, min_edge: int, mode: str) -> torch.Tensor:
if min_edge <= 0 or frames.dim() != 4 or frames.shape[0] <= 0:
return frames
height, width = int(frames.shape[1]), int(frames.shape[2])
shortest = min(height, width)
if shortest <= 0 or shortest >= min_edge:
return frames
scale = float(min_edge) / float(shortest)
target_h = max(1, int(round(height * scale)))
target_w = max(1, int(round(width * scale)))
nchw = frames.movedim(-1, 1)
if mode == "training":
upscaled = F.interpolate(nchw, size=(target_h, target_w), mode="nearest")
else:
upscaled = comfy.utils.common_upscale(nchw, target_w, target_h, "lanczos", "disabled")
return upscaled.movedim(1, -1)
class H3RefModDecode(io.ComfyNode):
"""Decode a RefMod's latent back to images for inspection."""
@classmethod
def define_schema(cls):
return io.Schema(
node_id="H3RefModDecode",
display_name="Decode H3 RefMod",
category="H3RefMod",
description="Decode a saved RefMod's latent back into images so you can see "
"what it captured. Inspection only — not part of generation. "
"The full stored latent timeline is decoded as-is; no frame "
"sampling is applied. "
"An encode-mode mod decodes to recognizable frames; a "
"training-mode (pooled) mod decodes to a tiny blurry grid "
"(that's all it stores). Audio is not decoded.",
inputs=[
io.Combo.Input("mod", options=_list_mod_names(),
tooltip="The RefMod to decode (from models/refmods/)."),
io.Vae.Input("vae",
tooltip="The MiniMax H3 video VAE."),
io.Int.Input("inspect_min_edge", default=320, min=0, max=2048, step=32,
tooltip="Upscale small decoded previews so pooled/training mods are easier "
"to inspect. 0 disables inspection upscaling. Training-mode uses "
"nearest-neighbor to keep the stored cell structure visible."),
],
outputs=[
io.Image.Output("images", display_name="images",
tooltip="The decoded reference frames [T, H, W, 3]."),
],
)
@classmethod
def execute(cls, mod, vae, inspect_min_edge=320) -> io.NodeOutput:
m = _load_mod(mod)
if m.kind == "audio":
raise ValueError("H3RefModDecode only decodes visual RefMods; audio-only mods are not supported.")
z = m.latent # [1, 24, T, h, w]
total_t = z.shape[2]
with torch.no_grad():
frames = vae.decode(z)
# video latent decodes channel-last; frame count rides the batch dim, so a
# 5-dim result is reshaped like the core VAEDecode node does
frames = _reshape_decoded_frames(frames)
frames = frames.clamp(0.0, 1.0).float().cpu()
frames = _upscale_for_inspection(frames, int(inspect_min_edge), m.mode)
print(f"[H3RefModDecode] '{m.name}' ({m.mode}, {m.kind}, "
f"latent_t={total_t}, inspect_min_edge={inspect_min_edge}) -> {tuple(frames.shape)}")
return io.NodeOutput(frames)
NODE_CLASS_MAPPINGS = {
"H3RefModDecode": H3RefModDecode,
}
NODE_DISPLAY_NAME_MAPPINGS = {
"H3RefModDecode": "Decode H3 RefMod",
}
+283
View File
@@ -0,0 +1,283 @@
"""
H3RefModLoader — load one RefMod and optionally append it to another
"""
from __future__ import annotations
from typing import Any, Dict, List, Tuple
import os
import folder_paths
import server
from ..py.refmod_core import H3RefMod, load_refmods_from_file, read_refmod_meta, refmod_capabilities
from ..py.refmod_common import (
_prompt_hint,
)
from ..py.refmod_common import (
refmods_dir,
refmods_dirs,
)
_MOD_CACHE: Dict[str, Tuple[H3RefMod, ...]] = {}
_MOD_CACHE_MAX = 24 # cap: never pin more mods in RAM than this (FIFO eviction)
_MOD_LIST_CACHE_KEY = None # (dirs, mtimes, sizes) signature of the last _list_mod_names() scan
_MOD_LIST_CACHE_VAL = None
_MOD_PATH_CACHE_KEY = None
_MOD_PATH_CACHE_VAL: Dict[str, str] = {}
_MOD_SKIP_DIRS = {"graph_presets", ".git", "__pycache__"}
_MAX_WEIGHT = 10.0
# Mod storage lives in ComfyUI's models/ tree (created on first run) and is
# registered as a first-class folder type so it shows up next to loras/unet.
try:
folder_paths.add_model_folder_path("refmods", refmods_dir())
except Exception:
pass
# ═══════════════════════════════════════════════════════════════════════════
# Mods folder helpers
# ═══════════════════════════════════════════════════════════════════════════
def _iter_mod_paths(base_dir: str):
"""Yield ``(relative_stem, absolute_stem)`` for RefMods under ``base_dir``."""
for root, dirnames, filenames in os.walk(base_dir):
dirnames[:] = sorted(d for d in dirnames if d not in _MOD_SKIP_DIRS)
for fn in sorted(filenames):
if not fn.endswith(".safetensors"):
continue
abs_stem = os.path.join(root, fn[:-len(".safetensors")])
rel_stem = os.path.relpath(abs_stem, base_dir).replace(os.sep, "/")
yield rel_stem, abs_stem
def _mod_scan_signature() -> tuple[str, List[str]]:
mods_dirs = refmods_dirs()
sig = []
for mods_dir in mods_dirs:
for rel_stem, abs_stem in _iter_mod_paths(mods_dir):
try:
st = os.stat(abs_stem + ".safetensors")
sig.append(f"{mods_dir}:{rel_stem}:{st.st_size}:{int(st.st_mtime)}")
except OSError:
pass
return "\n".join(sig), mods_dirs
def _root_display_labels(roots: List[str]) -> List[str]:
labels: List[str] = []
counts: Dict[str, int] = {}
for index, root in enumerate(roots, start=1):
base = os.path.basename(os.path.normpath(root)) or f"root{index}"
count = counts.get(base, 0) + 1
counts[base] = count
labels.append(base if count == 1 else f"{base}#{count}")
return labels
def _refresh_mod_lookup() -> Dict[str, str]:
global _MOD_LIST_CACHE_KEY, _MOD_LIST_CACHE_VAL, _MOD_PATH_CACHE_KEY, _MOD_PATH_CACHE_VAL
key, mods_dirs = _mod_scan_signature()
if key == _MOD_LIST_CACHE_KEY and key == _MOD_PATH_CACHE_KEY and _MOD_LIST_CACHE_VAL is not None:
return _MOD_PATH_CACHE_VAL
names_by_rel: Dict[str, List[Tuple[int, str]]] = {}
for index, mods_dir in enumerate(mods_dirs):
for rel_stem, abs_stem in _iter_mod_paths(mods_dir):
meta = read_refmod_meta(abs_stem)
if meta is None or meta.get("kind") not in ("image", "video", "audio", "bundle"):
continue
names_by_rel.setdefault(rel_stem, []).append((index, abs_stem))
labels = _root_display_labels(mods_dirs)
lookup: Dict[str, str] = {}
names: List[str] = []
for rel_stem in sorted(names_by_rel, key=str.lower):
matches = names_by_rel[rel_stem]
first_index, first_path = matches[0]
lookup[rel_stem] = first_path
names.append(rel_stem)
for dup_index, dup_path in matches[1:]:
display_name = f"[{labels[dup_index]}] {rel_stem}"
suffix = 2
while display_name in lookup:
display_name = f"[{labels[dup_index]} #{suffix}] {rel_stem}"
suffix += 1
lookup[display_name] = dup_path
names.append(display_name)
_MOD_LIST_CACHE_KEY = key
_MOD_LIST_CACHE_VAL = names
_MOD_PATH_CACHE_KEY = key
_MOD_PATH_CACHE_VAL = lookup
return lookup
def _list_mod_names() -> List[str]:
"""Available RefMod names across the search dirs (for the loader dropdown).
Only entries with valid RefMod metadata (embedded in the safetensors header
or a legacy sidecar .json) are listed, so other mod formats in
models/mods/ (e.g. LTXMod files) don't show up.
Called by INPUT_TYPES/VALIDATE_INPUTS on every prompt validation, so the
result is cached until any mod file appears/disappears/changes (checked
via cheap os.stat, not by re-reading every safetensors header).
"""
_refresh_mod_lookup()
return _MOD_LIST_CACHE_VAL
def _find_mod_path(name: str) -> str:
lookup = _refresh_mod_lookup()
if name in lookup:
return lookup[name]
mods_dirs = refmods_dirs()
raise FileNotFoundError(
f"RefMod '{name}' not found. Searched:\n" +
"\n".join(f" - {d}/{name}.safetensors" for d in mods_dirs))
def _load_mods(name: str) -> List[H3RefMod]:
if name in _MOD_CACHE:
return list(_MOD_CACHE[name])
path = _find_mod_path(name)
mods = tuple(load_refmods_from_file(path, device="cpu"))
_MOD_CACHE[name] = mods
if len(_MOD_CACHE) > _MOD_CACHE_MAX:
_MOD_CACHE.pop(next(iter(_MOD_CACHE)))
return list(mods)
def _load_mod(name: str) -> H3RefMod:
mods = _load_mods(name)
for mod in mods:
if mod.kind != "audio":
return mod
return mods[0]
def _mod_capabilities(name: str) -> Dict[str, bool]:
path = _find_mod_path(name)
meta = read_refmod_meta(path)
if meta is None:
return {"has_visual": True, "has_audio": False}
caps = refmod_capabilities(meta)
return {
"has_visual": bool(caps.get("has_visual", False)),
"has_audio": bool(caps.get("has_audio", False)),
}
def _normalize_weight(weight: float) -> float:
return min(_MAX_WEIGHT, max(0.0, float(weight)))
def _expand_weight(weight: float) -> List[float]:
clipped = _normalize_weight(weight)
whole = int(clipped)
remainder = clipped - whole
strengths = [1.0] * whole
if remainder > 1e-6:
strengths.append(remainder)
return strengths
def _append_weighted_mod(rows: List[tuple[H3RefMod, float]], mod: H3RefMod, weight: float) -> float:
clipped = _normalize_weight(weight)
rows.extend((mod, strength) for strength in _expand_weight(clipped))
return clipped
def _weight_display(weight: float) -> str:
clipped = _normalize_weight(weight)
whole = int(clipped)
remainder = clipped - whole
if whole <= 0:
return f"{clipped:.2f}"
if remainder <= 1e-6:
return f"x{whole}"
return f"x{whole} + {remainder:.2f}"
class H3RefModLoader:
"""Load one RefMod and append it to an existing bundle."""
@classmethod
def INPUT_TYPES(cls):
return {
"required": {
"mod": (_list_mod_names(), {"tooltip": "The RefMod to load."}),
"weight": ("FLOAT", {"default": 1.0, "min": 0.0, "max": _MAX_WEIGHT, "step": 0.01,
"display": "number",
"tooltip": "Legacy unified weight control kept for workflow compatibility. New graphs should use video_weight and audio_weight."}),
"video_weight": ("FLOAT", {"default": 1.0, "min": 0.0, "max": _MAX_WEIGHT, "step": 0.01,
"display": "number",
"tooltip": "Unified video weight control. 0 skips video. 0..1 behaves like the old strength control. Values above 1 repeat the same visual RefMod as extra copies."}),
"audio_weight": ("FLOAT", {"default": 1.0, "min": 0.0, "max": _MAX_WEIGHT, "step": 0.01,
"display": "number",
"tooltip": "Unified audio weight control. 0 skips audio. 0..1 behaves like the old strength control. Values above 1 repeat the same audio RefMod as extra copies."}),
},
"optional": {
"mods": ("H3_REF_MODS",),
},
}
RETURN_TYPES = ("H3_REF_MODS", "STRING")
RETURN_NAMES = ("mods", "prompt_hint")
FUNCTION = "load"
CATEGORY = "H3RefModPicker"
@classmethod
def VALIDATE_INPUTS(cls, mod, **kwargs):
if mod not in set(_list_mod_names()):
return f"RefMod '{mod}' not found in any configured refmods folder. Run Create H3 RefMod first."
return True
def load(self, mod, weight=1.0, video_weight=1.0, audio_weight=1.0, mods=None, strength=None, audio_strength=None):
rows = list(mods) if mods is not None else []
if strength is not None:
weight = strength
if audio_strength is not None:
audio_weight = audio_strength
if strength is not None or (weight != 1.0 and video_weight == 1.0 and audio_weight == 1.0):
video_weight = weight
audio_weight = weight
loaded = _load_mods(mod)
summaries = []
hint_rows = list(mods) if mods is not None else []
for item in loaded:
current_weight = audio_weight if item.kind == "audio" else video_weight
clipped = _append_weighted_mod(rows, item, current_weight)
summaries.append(f"{item.name}({_weight_display(clipped)})")
if clipped > 0.0:
hint_rows.append((item, min(1.0, clipped)))
print(f"[H3RefModLoader] {mod}: " + ", ".join(summaries))
hint = _prompt_hint(hint_rows)
if hint:
print(f"[H3RefModLoader] prompt_hint: {hint}")
return (rows, hint)
NODE_CLASS_MAPPINGS = {
"H3RefModLoader": H3RefModLoader
}
NODE_DISPLAY_NAME_MAPPINGS = {
"H3RefModLoader": "Load H3 RefMod Simple"
}
@server.PromptServer.instance.routes.get("/h3refmods/refmod-loader/capabilities")
async def refmod_loader_capabilities(request):
try:
name = str(request.query.get("mod", "") or "").strip()
if not name:
return server.web.json_response({"ok": True, "has_visual": True, "has_audio": True})
caps = _mod_capabilities(name)
return server.web.json_response({"ok": True, **caps})
except Exception as exc:
return server.web.json_response({"ok": False, "error": str(exc)}, status=500)
@@ -0,0 +1,165 @@
from __future__ import annotations
import os
from typing import Dict, List, Tuple
import server
from ..py.refmod_browser import (
PREVIEW_EXTS,
browser_root,
list_browser_dir,
paired_mod_path,
safe_file_path,
)
from ..py.refmod_core import H3RefMod, load_refmods_from_file
from ..py.refmod_common import _prompt_hint
from .refmod_loader import _MAX_WEIGHT, _append_weighted_mod, _weight_display
_VISUAL_MOD_CACHE: Dict[str, Tuple[H3RefMod, ...]] = {}
_VISUAL_MOD_CACHE_MAX = 24
def _load_mods_from_path(mod_path: str) -> List[H3RefMod]:
path = safe_file_path(mod_path)
if not path.lower().endswith(".safetensors"):
raise ValueError("Selected file is not a RefMod safetensors file.")
if path in _VISUAL_MOD_CACHE:
return list(_VISUAL_MOD_CACHE[path])
mods = list(load_refmods_from_file(path[:-len(".safetensors")], device="cpu"))
if not any(mod.kind == "audio" for mod in mods):
paired_audio = paired_mod_path(path[:-len(".safetensors")], "audio")
if paired_audio is not None:
mods.extend(load_refmods_from_file(paired_audio[:-len(".safetensors")], device="cpu"))
_VISUAL_MOD_CACHE[path] = tuple(mods)
if len(_VISUAL_MOD_CACHE) > _VISUAL_MOD_CACHE_MAX:
_VISUAL_MOD_CACHE.pop(next(iter(_VISUAL_MOD_CACHE)))
return mods
class H3RefModVisualPicker:
"""Pick a RefMod by browsing models/refmods thumbnails and append it to a bundle."""
NAME = "Visual RefMod Picker"
CATEGORY = "H3RefModPicker"
@classmethod
def INPUT_TYPES(cls):
return {
"required": {
"mod_path": (
"STRING",
{
"default": "",
"tooltip": "Selected RefMod safetensors path from the thumbnail browser.",
},
),
"video_weight": (
"FLOAT",
{
"default": 1.0,
"min": 0.0,
"max": _MAX_WEIGHT,
"step": 0.01,
"tooltip": "Unified video weight control. 0 skips video. 0..1 behaves like the old strength control. Values above 1 repeat the same video RefMod as extra copies.",
},
),
"audio_weight": (
"FLOAT",
{
"default": 1.0,
"min": 0.0,
"max": _MAX_WEIGHT,
"step": 0.01,
"tooltip": "Unified audio weight control. 0 skips audio. 0..1 behaves like the old audio strength control. Values above 1 repeat the same audio RefMod as extra copies.",
},
),
},
"optional": {
"mods": ("H3_REF_MODS",),
},
}
RETURN_TYPES = ("H3_REF_MODS", "STRING")
RETURN_NAMES = ("mods", "prompt_hint")
FUNCTION = "pick"
DESCRIPTION = (
"Pick a RefMod by browsing models/refmods and its subfolders. "
"A matching preview image with the same base name is shown when present; "
"otherwise a placeholder is used. When a matching *_Audio file exists "
"beside a *_Video or *_Visual file, both are loaded together and appended to the bundle, "
"with separate video/audio weights. A weight in 0..1 behaves like the old strength control; "
"a weight above 1 repeats the same RefMod as extra copies. Legacy combined RefMods still use one shared video weight."
)
@classmethod
def VALIDATE_INPUTS(cls, **kwargs):
return True
def pick(self, mod_path: str, video_weight: float = 1.0, audio_weight: float = 1.0,
mods=None, strength=None, audio_strength=None):
rows: List[Tuple[H3RefMod, float]] = []
if mods is not None and isinstance(mods, (list, tuple)):
rows.extend(list(mods))
selected = str(mod_path or "").strip()
if not selected or selected.lower() == "(none)":
hint = _prompt_hint(rows)
return (rows, hint)
if strength is not None:
video_weight = strength
if audio_strength is not None:
audio_weight = audio_strength
selected_mods = _load_mods_from_path(selected)
hint_rows = list(mods) if mods is not None and isinstance(mods, (list, tuple)) else []
summaries = []
for mod in selected_mods:
weight = audio_weight if mod.kind == "audio" else video_weight
clipped = _append_weighted_mod(rows, mod, weight)
if clipped > 0.0:
hint_rows.append((mod, min(1.0, clipped)))
summaries.append(f"{mod.name}({_weight_display(clipped)})")
print("[H3RefModVisualPicker] " + ", ".join(summaries))
hint = _prompt_hint(hint_rows)
if hint:
print(f"[H3RefModVisualPicker] prompt_hint: {hint}")
return (rows, hint)
NODE_CLASS_MAPPINGS = {"H3RefModVisualPicker": H3RefModVisualPicker}
NODE_DISPLAY_NAME_MAPPINGS = {"H3RefModVisualPicker": "Visual H3 RefMod Picker"}
@server.PromptServer.instance.routes.get("/h3refmods/refmod-browser/root")
async def refmod_browser_root(request):
try:
return server.web.json_response({"ok": True, "root": browser_root()})
except Exception as exc:
return server.web.json_response({"ok": False, "error": str(exc)}, status=500)
@server.PromptServer.instance.routes.get("/h3refmods/refmod-browser/list")
async def refmod_browser_list(request):
try:
data = list_browser_dir(request.query.get("path", "").strip())
return server.web.json_response({"ok": True, **data})
except ValueError as exc:
return server.web.json_response({"ok": False, "error": str(exc)}, status=404)
except Exception as exc:
return server.web.json_response({"ok": False, "error": str(exc)}, status=500)
@server.PromptServer.instance.routes.get("/h3refmods/refmod-browser/file")
async def refmod_browser_file(request):
try:
path = safe_file_path(request.query.get("path", "").strip())
if os.path.splitext(path)[1].lower() not in PREVIEW_EXTS:
return server.web.json_response(
{"ok": False, "error": "Unsupported file type"}, status=403
)
return server.web.FileResponse(path, headers={"Cache-Control": "no-cache"})
except ValueError as exc:
return server.web.json_response({"ok": False, "error": str(exc)}, status=404)
except Exception as exc:
return server.web.json_response({"ok": False, "error": str(exc)}, status=500)
+293
View File
@@ -0,0 +1,293 @@
"""Native H3 reference conditioning with grounded RefMod stacks."""
from __future__ import annotations
import io as bytes_io
import math
import torch
import torch.nn.functional as functional
from comfy_api.latest import io
REFERENCE_TYPE = "H3_REFMOD_REFERENCES"
def _mod_rows(mods):
if mods is None:
return []
if hasattr(mods, "ref_block"):
return [(mods, 1.0)]
if isinstance(mods, (tuple, list)) and mods and hasattr(mods[0], "ref_block"):
mods = [mods]
rows = []
for row in mods:
if not isinstance(row, (tuple, list)) or len(row) not in (2, 3):
raise ValueError("mods must contain (RefMod, strength) rows.")
mod, strength = row[:2]
if mod is None:
continue
if not callable(getattr(mod, "ref_block", None)):
raise ValueError("mods contains an unsupported RefMod object.")
strength = float(strength)
if not math.isfinite(strength):
raise ValueError("RefMod strengths must be finite.")
if strength > 0:
rows.append((mod, min(1.0, strength)))
return rows
def _decoded_frames(video_vae, latent):
with torch.no_grad():
frames = video_vae.decode(latent)
if frames.ndim == 5 and frames.shape[0] == 1:
frames = frames[0]
if frames.ndim != 4 or frames.shape[-1] != 3 or frames.shape[0] < 1:
raise ValueError(f"H3 video VAE returned invalid images: {tuple(frames.shape)}")
return frames.detach().float().clamp(0, 1).cpu()
def _encoder_frames(mod, video_vae, reference_fps):
times = getattr(mod, "enc_times", None)
path = getattr(mod, "path", "")
packed_frames = getattr(mod, "enc_frames", None)
is_stack = getattr(mod, "source", "") == "stack"
if times and (packed_frames or path) and (is_stack or abs(getattr(mod, "enc_fps", 0) - reference_fps) < 1e-6):
import numpy as np
from PIL import Image
def unpack(get_tensor):
frames = []
for index in range(len(times)):
packed = get_tensor(f"enc_{index}").numpy().tobytes()
with Image.open(bytes_io.BytesIO(packed)) as image:
frames.append(torch.from_numpy(np.array(image.convert("RGB"))).float() / 255)
return torch.stack(frames), list(times), "stored pictures"
if packed_frames:
return unpack(packed_frames.__getitem__)
from safetensors import safe_open
filename = path if path.endswith(".safetensors") else path + ".safetensors"
with safe_open(filename, framework="pt", device="cpu") as handle:
return unpack(handle.get_tensor)
if video_vae is None:
raise ValueError(f"Connect video_vae to reconstruct visual RefMod '{mod.name}'.")
if is_stack:
frames = [_decoded_frames(video_vae, mod.latent[:, :, index:index + 1])[:1]
for index in range(mod.latent.shape[2])]
return torch.cat(frames), [float(index) for index in range(len(frames))], "decoded stack"
frames = _decoded_frames(video_vae, mod.latent)
if mod.kind == "image":
return frames[:1], [0.0], "decoded image"
times = [index / 2 for index in range(math.ceil(frames.shape[0] * 2 / reference_fps))]
indices = [min(round(time * reference_fps), frames.shape[0] - 1) for time in times]
return frames[indices], times, "decoded video"
def _visual_item(mod, frames, times, strength, stack_pictures, visual_kind=None):
if strength < 1:
samples = frames.movedim(-1, 1)
blurred = functional.adaptive_avg_pool2d(samples, (max(1, mod.latent_h // 8),
max(1, mod.latent_w // 8)))
blurred = functional.interpolate(blurred, size=samples.shape[-2:], mode="bilinear", align_corners=False)
frames = (strength * samples + (1 - strength) * blurred).movedim(1, -1)
if (visual_kind or mod.kind) == "image":
return {"type": "image", "data": frames[:1]}, 1
if getattr(mod, "source", "") == "stack":
count = frames.shape[0]
if stack_pictures == "every 4th":
indices = list(range(0, count, 4))
frames = frames[indices]
times = [index / 2 for index in range(len(indices))]
else:
indices = ([round(index * (count - 1) / 7) for index in range(8)]
if stack_pictures == "up to 8" and count > 8 else list(range(count)))
frames = frames[indices].repeat_interleave(2, dim=0)
times = [index + offset for index in range(len(indices)) for offset in (0.0, 0.5)]
shown = len(indices)
else:
shown = frames.shape[0]
return {"type": "video", "data": frames, "timestamps": times}, shown
class H3RefModReferences(io.ComfyNode):
@classmethod
def define_schema(cls):
return io.Schema(
node_id="H3RefModReferences",
display_name="Collect H3 References",
category="H3RefModPicker",
inputs=[
io.Custom(REFERENCE_TYPE).Input("references", optional=True,
tooltip="Optional references from another collector."),
io.Autogrow.Input("ref_images", optional=True, template=io.Autogrow.TemplatePrefix(
input=io.Image.Input("ref_image"), prefix="ref_image_", min=0, max=9)),
io.Autogrow.Input("ref_videos", optional=True, template=io.Autogrow.TemplatePrefix(
input=io.Image.Input("ref_video", tooltip="Video frame batch at 24 fps."),
prefix="ref_video_", min=0, max=3)),
io.Autogrow.Input("ref_video_audios", optional=True, template=io.Autogrow.TemplatePrefix(
input=io.Audio.Input("ref_video_audio", tooltip="Soundtrack for the same-numbered video input."),
prefix="ref_video_audio_", min=0, max=3)),
io.Autogrow.Input("ref_audios", optional=True, template=io.Autogrow.TemplatePrefix(
input=io.Audio.Input("ref_audio"), prefix="ref_audio_", min=0, max=3)),
],
outputs=[io.Custom(REFERENCE_TYPE).Output("references")],
)
@classmethod
def execute(cls, references=None, ref_images=None, ref_videos=None,
ref_video_audios=None, ref_audios=None):
entries = list(references or [])
for name, image in (ref_images or {}).items():
if image is not None:
entries.append({"kind": "image", "data": image, "name": name})
videos = ref_videos or {}
soundtracks = ref_video_audios or {}
for name, soundtrack in soundtracks.items():
if soundtrack is not None and videos.get("ref_video_" + name.rsplit("_", 1)[-1]) is None:
raise ValueError(f"{name} needs its same-numbered ref_video input; use ref_audio for standalone audio.")
for name, frames in videos.items():
if frames is not None:
entries.append({"kind": "video", "data": frames, "name": name,
"audio": soundtracks.get("ref_video_audio_" + name.rsplit("_", 1)[-1])})
for name, audio in (ref_audios or {}).items():
if audio is not None:
entries.append({"kind": "audio", "data": audio, "name": name})
return io.NodeOutput(entries)
class _ReferencePresentation:
def __init__(self):
self.items = []
def tokenize(self, prompt, minimax_ref_items):
self.items = minimax_ref_items
return None
def encode_from_tokens_scheduled(self, tokens):
return [[None, {}]]
def _native_references(references, video_vae, audio_vae, width, height, length, ref_image_size):
from comfy_extras.nodes_minimax_h3 import MiniMaxH3ReferenceToVideo
images, videos, soundtracks, audios = {}, {}, {}, {}
names = []
for kind in ("image", "video", "audio"):
for index, entry in enumerate(references or []):
if entry.get("kind") != kind:
continue
name = entry["name"]
if kind == "image":
images[f"ref_image_{index}"] = entry["data"]
elif kind == "video":
videos[f"ref_video_{index}"] = entry["data"]
if entry.get("audio") is not None:
soundtracks[f"ref_video_audio_{index}"] = entry["audio"]
names.append(name + " soundtrack")
else:
audios[f"ref_audio_{index}"] = entry["data"]
names.append(name)
presentation = _ReferencePresentation()
output = MiniMaxH3ReferenceToVideo.execute(
presentation, "", width, height, length, ref_image_size=ref_image_size,
vae=video_vae, audio_vae=audio_vae, ref_images=images, ref_videos=videos,
ref_video_audios=soundtracks, ref_audios=audios)
conditioning, latent = output.result
return presentation.items, list(conditioning[0][1].get("minimax_refs", [])), names, latent
class H3RefModsToVideo(io.ComfyNode):
@classmethod
def define_schema(cls):
return io.Schema(
node_id="H3RefModsToVideo",
display_name="MiniMax H3 RefMods to Video",
category="model/conditioning/minimax",
description="Reference-to-Video with grounded RefMods and optional collected media references.",
inputs=[
io.Clip.Input("clip"),
io.Vae.Input("video_vae", optional=True,
tooltip="Encodes raw pictures/videos and reconstructs RefMods without stored encoder pictures."),
io.Vae.Input("audio_vae", optional=True,
tooltip="Encodes raw reference audio. Audio RefMods are already encoded."),
io.Custom(REFERENCE_TYPE).Input("references", optional=True),
io.MultiType.Input("mods", types=[io.Custom("H3_REFMOD"), io.Custom("H3_REF_MODS")], optional=True),
io.String.Input("prompt", multiline=True, dynamic_prompts=True),
io.Int.Input("width", default=1344, min=32, max=8192, step=32),
io.Int.Input("height", default=768, min=32, max=8192, step=32),
io.Int.Input("length", default=124, min=5, max=3600, step=17),
io.Combo.Input("ref_image_size", options=["match", "max"], default="match"),
io.Combo.Input("stack_pictures", options=["every 4th", "up to 8", "all"], default="every 4th",
tooltip="Pictures shown to the text encoder for a stacked RefMod. All latent frames remain applied."),
],
outputs=[io.Conditioning.Output(display_name="positive"), io.Latent.Output(),
io.String.Output("reference_map")],
)
@classmethod
def execute(cls, clip, prompt, width, height, length, video_vae=None, audio_vae=None,
references=None, mods=None, ref_image_size="match", stack_pictures="every 4th"):
if stack_pictures not in ("every 4th", "up to 8", "all"):
raise ValueError("Unknown stack_pictures mode.")
items, blocks, names, latent = _native_references(
references, video_vae, audio_vae, width, height, length, ref_image_size)
counters = {"image": 0, "video": 0, "audio": 0}
labels = {"image": "Picture", "video": "Video", "audio": "Audio"}
mapping = []
def append_label(item, name):
kind = item["type"]
counters[kind] += 1
mapping.append(f"<{labels[kind]} {counters[kind]}> = {name}")
for item, name in zip(items, names):
append_label(item, name)
decoded = {}
for mod, strength in _mod_rows(mods):
block = mod.ref_block(strength)
if block is None:
continue
block = dict(block)
kind = block["kind"]
name = mod.name
if kind not in ("image", "video", "video_audio", "audio"):
raise ValueError(f"Unsupported RefMod kind: {kind}")
if kind == "audio" or (block.get("audio_latent") is not None and block.get("ref_audio_t", 0) > 0):
item = {"type": "audio"}
items.append(item)
append_label(item, name + (" soundtrack" if kind != "audio" else ""))
if kind != "audio":
if id(mod) not in decoded:
decoded[id(mod)] = _encoder_frames(mod, video_vae, 24.0)
frames, times, _source = decoded[id(mod)]
visual_kind = "video" if kind == "video_audio" else kind
item, _shown = _visual_item(mod, frames, times, strength, stack_pictures, visual_kind)
items.append(item)
append_label(item, name)
block["refmod"] = True
blocks.append(block)
if video_vae is not None or audio_vae is not None:
import comfy.model_management
comfy.model_management.soft_empty_cache()
tokens = clip.tokenize(prompt, minimax_ref_items=items)
conditioning = clip.encode_from_tokens_scheduled(tokens)
output = []
for embedding, metadata in conditioning:
if items and "minimax_token_tags" not in metadata:
raise ValueError("Connect a MiniMax H3 CLIP that supports reference token tags.")
metadata = dict(metadata)
if blocks:
metadata["minimax_refs"] = list(metadata.get("minimax_refs", [])) + blocks
output.append([embedding, metadata])
reference_map = "\n".join(mapping) or "No references."
return io.NodeOutput(output, latent, reference_map)
NODE_CLASS_MAPPINGS = {
"H3RefModsToVideo": H3RefModsToVideo,
"H3RefModReferences": H3RefModReferences,
}
NODE_DISPLAY_NAME_MAPPINGS = {
"H3RefModsToVideo": "MiniMax H3 RefMods to Video",
"H3RefModReferences": "Collect H3 References",
}
+253
View File
@@ -0,0 +1,253 @@
from __future__ import annotations
import os
from typing import Dict, List, Optional
from .refmod_common import refmods_dir, refmods_dirs
from .refmod_core import read_refmod_meta, refmod_capabilities
PREVIEW_EXTS = (".png", ".jpg", ".jpeg", ".webp")
HIDDEN_BROWSER_DIRS = {"graph_presets"}
VISUAL_SUFFIXES = ("_visual", "_video")
AUDIO_SUFFIXES = ("_audio",)
VISUAL_FILE_SUFFIXES = ("_visual", "_Visual", "_video", "_Video")
AUDIO_FILE_SUFFIXES = ("_audio", "_Audio")
def browser_root() -> str:
root = os.path.abspath(refmods_dir())
os.makedirs(root, exist_ok=True)
return root
def browser_roots() -> List[str]:
"""Every declared RefMod folder; the first one is browser_root()."""
return refmods_dirs()
def _browser_root_labels() -> Dict[str, str]:
labels: Dict[str, str] = {}
counts: Dict[str, int] = {}
for index, root in enumerate(browser_roots(), start=1):
base = os.path.basename(os.path.normpath(root)) or f"root{index}"
count = counts.get(base, 0) + 1
counts[base] = count
labels[os.path.realpath(root)] = base if count == 1 else f"{base}#{count}"
return labels
def _is_root(path: str) -> bool:
real = os.path.realpath(_safe_abspath(path))
return any(real == os.path.realpath(root) for root in browser_roots())
def _root_label_for_path(path: str) -> str:
current = os.path.realpath(_safe_abspath(path))
for root_real, label in _browser_root_labels().items():
if current == root_real or current.startswith(root_real + os.sep):
return label
return "refmods"
def _safe_abspath(path: str) -> str:
return os.path.abspath(os.path.expanduser(path))
def _resolve_browser_path(path: str) -> str:
raw = str(path or "").strip()
if not raw:
return browser_root()
expanded = os.path.expanduser(raw)
if os.path.isabs(expanded):
return os.path.abspath(expanded)
normalized = raw.replace("\\", os.sep).replace("/", os.sep)
for root in browser_roots():
candidate = os.path.abspath(os.path.join(root, normalized))
if os.path.exists(candidate):
return candidate
return os.path.abspath(os.path.join(browser_root(), normalized))
def _is_under_root(path: str) -> bool:
current = os.path.realpath(_safe_abspath(path))
for root_dir in browser_roots():
root = os.path.realpath(root_dir)
if current == root or current.startswith(root + os.sep):
return True
return False
def safe_dir_path(path: str = "") -> str:
if not path:
return browser_root()
current = _resolve_browser_path(path)
if not _is_under_root(current):
raise ValueError("Path is outside models/refmods")
if not os.path.isdir(current):
raise ValueError("Folder not found")
return current
def safe_file_path(path: str) -> str:
if not path:
raise ValueError("Missing path")
current = _resolve_browser_path(path)
if not _is_under_root(current):
raise ValueError("Path is outside models/refmods")
if not os.path.isfile(current):
raise ValueError("File not found")
return current
def _strip_known_suffix(path_no_ext: str) -> tuple[str, str | None, str | None]:
name = os.path.basename(path_no_ext)
lower_name = name.lower()
for suffix in VISUAL_SUFFIXES:
if lower_name.endswith(suffix):
return (path_no_ext[:-len(suffix)], "visual", suffix)
for suffix in AUDIO_SUFFIXES:
if lower_name.endswith(suffix):
return (path_no_ext[:-len(suffix)], "audio", suffix)
return (path_no_ext, None, None)
def paired_mod_path(path_no_ext: str, target_kind: str) -> Optional[str]:
base, current_kind, _suffix = _strip_known_suffix(path_no_ext)
if current_kind is None:
return None
current_path = path_no_ext + ".safetensors"
suffixes = VISUAL_FILE_SUFFIXES if target_kind == "visual" else AUDIO_FILE_SUFFIXES
for suffix in suffixes:
candidate = base + suffix + ".safetensors"
if candidate != current_path and os.path.isfile(candidate):
return candidate
return None
def _find_preview(path_no_ext: str) -> Optional[str]:
candidates = [path_no_ext]
base, kind, _suffix = _strip_known_suffix(path_no_ext)
if kind is not None:
candidates.insert(0, base)
for stem in candidates:
for ext in PREVIEW_EXTS:
candidate = stem + ext
if os.path.isfile(candidate):
return candidate
return None
def _display_mod_name(path_no_ext: str) -> str:
base, kind, _suffix = _strip_known_suffix(path_no_ext)
if kind is not None:
name = os.path.basename(base)
return name or os.path.basename(path_no_ext)
return os.path.basename(path_no_ext)
def _mod_entry(path: str) -> Optional[Dict]:
if not path.lower().endswith(".safetensors"):
return None
path_no_ext = path[:-len(".safetensors")]
meta = read_refmod_meta(path_no_ext)
if meta is None:
return None
caps = refmod_capabilities(meta)
meta_kind = str(caps.get("kind", "") or "")
if meta_kind not in ("image", "video", "audio", "bundle"):
return None
_base, paired_kind, _suffix = _strip_known_suffix(path_no_ext)
paired_visual = paired_mod_path(path_no_ext, "visual")
paired_audio = paired_mod_path(path_no_ext, "audio")
paired_split = bool(paired_visual or paired_audio)
if paired_kind == "audio" and paired_visual:
return None
has_visual = bool(caps.get("has_visual", False))
has_audio = bool(caps.get("has_audio", False))
if paired_split:
has_visual = True
has_audio = True
preview = _find_preview(path_no_ext) if has_visual else None
return {
"name": _display_mod_name(path_no_ext) + ".safetensors",
"path": path,
"preview_path": preview,
"kind": meta_kind or "unknown",
"has_visual": has_visual,
"has_audio": has_audio,
"paired_split": paired_split,
"concept_type": str(meta.get("concept_type", "generic") or "generic"),
"description": str(meta.get("description", "") or ""),
}
def _scan_dir(current: str) -> tuple[List[Dict], List[Dict]]:
dirs: List[Dict] = []
mods: List[Dict] = []
try:
entries = list(os.scandir(current))
except PermissionError as exc:
raise ValueError("Access denied") from exc
for entry in entries:
name = entry.name
if name.startswith(".") or name in HIDDEN_BROWSER_DIRS:
continue
try:
if entry.is_dir(follow_symlinks=False):
dirs.append({"name": name, "path": os.path.abspath(entry.path)})
elif entry.is_file(follow_symlinks=False) and name.lower().endswith(".safetensors"):
item = _mod_entry(os.path.abspath(entry.path))
if item is not None:
mods.append(item)
except OSError:
continue
return dirs, mods
def _annotate_merged_duplicates(items: List[Dict]) -> None:
counts: Dict[str, int] = {}
for item in items:
key = str(item.get("name", "")).lower()
counts[key] = counts.get(key, 0) + 1
for item in items:
key = str(item.get("name", "")).lower()
if counts.get(key, 0) > 1:
item["name"] = f"{item['name']} [{_root_label_for_path(str(item.get('path', '')))}]"
def list_browser_dir(path: str = "") -> Dict:
current = safe_dir_path(path)
if _is_root(current):
dirs: List[Dict] = []
mods: List[Dict] = []
for root in browser_roots():
root_dirs, root_mods = _scan_dir(root)
dirs.extend(root_dirs)
mods.extend(root_mods)
_annotate_merged_duplicates(dirs)
_annotate_merged_duplicates(mods)
current = browser_root()
parent_path = None
else:
dirs, mods = _scan_dir(current)
parent = os.path.dirname(current.rstrip("\\/"))
if parent and _is_root(parent):
parent_path = browser_root()
elif parent and _is_under_root(parent) and parent != current:
parent_path = parent
else:
parent_path = None
dirs.sort(key=lambda item: item["name"].lower())
mods.sort(key=lambda item: item["name"].lower())
return {
"root": browser_root(),
"current_path": current,
"parent_path": parent_path,
"dirs": dirs,
"mods": mods,
}
+96
View File
@@ -0,0 +1,96 @@
"""Shared helpers for the RefMod pack: media loading and the refmods folder."""
from __future__ import annotations
import os
from typing import List, Optional
import torch
IMAGE_EXTS = {".png", ".jpg", ".jpeg", ".webp", ".bmp", ".gif"}
def refmods_dir() -> str:
"""ComfyUI models/refmods — the mod storage folder, created on first use.
Sits next to loras/, unet/ etc. instead of inside the pack folder, so mods
live in the standard model tree.
"""
import folder_paths
d = os.path.join(folder_paths.models_dir, "refmods")
os.makedirs(d, exist_ok=True)
return d
def refmods_dirs() -> List[str]:
"""All RefMod folders ComfyUI knows about, write folder first.
Includes folders declared under ``refmods:`` in extra_model_paths.yaml
(or --extra-model-paths-config). Duplicates are removed by real path,
since a root may be a symlink to another one.
"""
import folder_paths
candidates = [refmods_dir()]
try:
candidates += list(folder_paths.get_folder_paths("refmods"))
except Exception:
pass
roots: List[str] = []
seen = set()
for directory in candidates:
if not os.path.isdir(directory):
continue
real = os.path.realpath(directory)
if real in seen:
continue
seen.add(real)
roots.append(os.path.abspath(directory))
return roots
def list_image_files(folder: str) -> List[str]:
"""Images directly under ``folder`` (top level only), sorted by name."""
images = []
if os.path.isdir(folder):
for fn in sorted(os.listdir(folder)):
ext = os.path.splitext(fn)[1].lower()
p = os.path.join(folder, fn)
if os.path.isfile(p):
if ext in IMAGE_EXTS:
images.append(p)
return images
def load_image_file(path: str, max_edge: Optional[int] = None) -> torch.Tensor:
"""Load one image file -> [1, H, W, 3] float32 in [0, 1]."""
import numpy as np
from PIL import Image, ImageOps
with Image.open(path) as img:
img = ImageOps.exif_transpose(img)
img = img.convert("RGB")
w, h = img.size
if max_edge is not None:
scale = min(1.0, max_edge / max(w, h))
if scale < 1.0:
img = img.resize((max(1, round(w * scale)), max(1, round(h * scale))),
Image.LANCZOS)
arr = torch.from_numpy(np.asarray(img).copy()).float() / 255.0
return arr.unsqueeze(0) # [1, H, W, 3]
def _prompt_hint(loads) -> str:
"""Merge loaded mods' concept_type + description into one prompt-ready string.
e.g. "identity: ginger woman, tattooed neck, black lipstick; pose_motion:
slow twirl into camera, hair whipping". Concat this onto your positive
prompt (a string-concat node ahead of CLIP Text Encode) instead of
retyping each mod's description by hand. Mods with no description are
skipped — a bare concept_type with nothing to say isn't a useful clue.
"""
parts = []
for mod, _strength in loads:
if mod.description:
parts.append(f"{mod.concept_type}: {mod.description}")
return "; ".join(parts)
+958
View File
@@ -0,0 +1,958 @@
"""
core.py — "RefMod" adapter for MiniMax H3 (no training required)
The reference-video path of MiniMax H3 works by injecting *reference tokens*
into the packed sequence: the ref is VAE-encoded, patchified, projected and
placed on the 3D RoPE grid, then every DiT block attends to it. A video ref
is expensive because it contributes thousands of tokens.
Mode naming: ``encode`` (was ``full``) = straight full-res VAE encode;
``training`` (was ``pooled``) = compressed grid refined by gradient steps
(the only mode that "trains" — the refinement loop). The old names are
accepted everywhere as legacy aliases so saved workflows and mods keep
working.
A RefMod is the same reference, compressed to a handful of tokens:
* the ref is VAE-encoded to its full latent [1, 24, T, H, W],
* the latent is average-pooled to a tiny grid (default 4x4) and a few
latent frames, so the patchified token count drops to ~4-16,
* optionally the small latent is refined with a few gradient steps that
reconstruct the full latent (still no model weights involved).
At generation time the pooled latent is handed back to the model through the
native ``refs`` payload, so it flows through the exact same per-block
attention machinery as a full reference — the only thing that changes is the
token budget.
This is the MiniMax H3 analog of the LTX "Mod" concept tokens + per-block
injection: instead of a trained hypernetwork predicting AdaLN deltas, the
model's own cross-attention over the compressed ref tokens does the work.
"""
from __future__ import annotations
import json
import math
import os
import tempfile
from dataclasses import dataclass, field
from typing import Dict, List, Optional, Tuple
import torch
import torch.nn as nn
import torch.nn.functional as F
from safetensors import safe_open
from safetensors.torch import load_file, save_file
# Metadata key embedded in the safetensors header (keeps a mod in one file,
# so it can be shared/uploaded as a single artifact).
META_KEY = "refmod_meta"
# What a mod is standing in for. This is documentation + a routing hint, not
# a hard constraint on the tensors — but it drives two things: (1) the
# "identity in training mode" warning in nodes.py, and (2) the loader's
# prompt_hint output, which merges each loaded mod's concept_type +
# description into a string you can concat onto your CLIP Text Encode prompt.
# The model only reads these labels through the text prompt, so prompt_hint is
# the only place they are surfaced.
CONCEPT_TYPES = (
"generic", # unspecified / mixed
"identity", # a specific person/character — use mode="encode" for this
"pose_motion", # a pose, dance, gesture, camera move
"clothing", # an outfit / garment, decoupled from who's wearing it
"background", # environment / set / location plate
"voice",
"singing",
"music_style",
"sound_fx",
"ambience",
"style", # look/grade/animation style, not a concrete subject
)
AUDIO_CONCEPT_TYPES = ("voice", "singing", "music_style", "sound_fx", "ambience")
def _blur_latent(z: torch.Tensor, factor: int = 8) -> torch.Tensor:
"""Heavy spatial low-pass (downsample then upsample) used as the
weakening target for ``ref_block``'s ``strength`` below ``1.0``.
Originally this mixed toward ``torch.randn()`` noise, mirroring what the
docstring below describes as the model's own ``visual_cond_noise_aug``
mechanism. In practice, at the retention range people actually use this
dial for (0.15-0.7, i.e. large noise fractions), it produces a visible
woven/static texture instead of a gracefully weaker reference: a VAE
latent's channels are correlated, so raw iid noise isn't read as "less
reference," it's read as real-but-garbled content and gets rendered as
an actual (wrong) texture. ``visual_cond_noise_aug`` in the model's own
training is likely a small-magnitude robustness augmentation, not
something swept this far — so copying its math doesn't reproduce its
behavior at these strengths. A blurred copy stays on the latent
manifold (smooth, plausible) while still discarding detail as strength
drops, which is what "weaker reference" should actually look like.
"""
if z.dim() == 4:
b, c, stereo, t = z.shape
if t <= 1:
return z
flat = z.reshape(b * c * stereo, 1, t).float()
down = F.adaptive_avg_pool1d(flat, max(1, t // factor))
up = F.interpolate(down, size=t, mode="linear", align_corners=False)
return up.reshape_as(z).to(z.dtype)
if z.dim() != 5:
raise ValueError(f"Unsupported RefMod latent shape: {tuple(z.shape)}")
t, h, w = z.shape[2], z.shape[3], z.shape[4]
sh, sw = max(1, h // factor), max(1, w // factor)
down = F.adaptive_avg_pool3d(z.float(), (t, sh, sw))
up = F.interpolate(down, size=(t, h, w), mode="trilinear", align_corners=False)
return up.to(z.dtype)
def read_refmod_meta(path_no_ext: str) -> Optional[Dict]:
"""
Read the metadata block from a mod/preset file without loading tensors.
Prefers the metadata embedded in the safetensors header; falls back to
the legacy sidecar ``{path}.json`` written by older versions.
"""
try:
with safe_open(path_no_ext + ".safetensors", framework="pt") as f:
meta = f.metadata()
if meta:
for key in (META_KEY, "audio_refmod_meta"):
if key in meta:
return json.loads(meta[key])
except Exception:
pass
jpath = path_no_ext + ".json"
if os.path.isfile(jpath):
try:
with open(jpath) as f:
return json.load(f)
except Exception:
return None
return None
def refmod_bundle_members(meta: Dict) -> List[Dict]:
if not isinstance(meta, dict) or meta.get("_format_version") != 5 or meta.get("kind") != "bundle":
raise ValueError("Unsupported RefMod bundle format.")
refs = meta.get("members")
if not isinstance(refs, list) or not refs or len(refs) > 256:
raise ValueError("A RefMod bundle must contain 1-256 members.")
for ref in refs:
if not isinstance(ref, dict) or ref.get("kind") not in ("image", "video", "audio"):
raise ValueError("Invalid RefMod bundle member.")
return refs
def refmod_capabilities(meta: Optional[Dict]) -> Dict[str, object]:
caps: Dict[str, object] = {
"kind": "unknown",
"has_visual": False,
"has_audio": False,
"member_count": 0,
}
if not isinstance(meta, dict):
return caps
kind = str(meta.get("kind", "") or "unknown")
caps["kind"] = kind
if kind == "bundle":
refs = refmod_bundle_members(meta)
caps["member_count"] = len(refs)
caps["has_visual"] = any(ref.get("kind") in ("image", "video") for ref in refs)
caps["has_audio"] = any(ref.get("kind") == "audio" for ref in refs)
elif kind in ("image", "video"):
caps["has_visual"] = True
caps["has_audio"] = int(meta.get("ref_audio_t", 0) or 0) > 0
caps["member_count"] = 1 + (1 if caps["has_audio"] else 0)
elif kind == "audio":
caps["has_audio"] = True
caps["member_count"] = 1
return caps
def _standalone_audio_member_meta(meta: Dict) -> Dict:
name = str(meta.get("name", "refmod") or "refmod")
if not name.lower().endswith("_audio"):
name = f"{name}_Audio"
return {
**meta,
"name": name,
"kind": "audio",
"latent_h": 0,
"latent_w": 0,
"latent_t": int(meta.get("ref_audio_t", 0) or 0),
"concept_type": str(meta.get("audio_concept_type", meta.get("concept_type", "voice")) or "voice"),
"audio_concept_type": str(meta.get("audio_concept_type", "") or ""),
"ref_audio_t": int(meta.get("ref_audio_t", 0) or 0),
"sample_rate": int(meta.get("sample_rate", 32000) or 32000),
}
def load_refmods_from_file(path_no_ext: str, device: str = "cpu") -> List["H3RefMod"]:
meta = read_refmod_meta(path_no_ext)
if not isinstance(meta, dict):
raise ValueError(
f"{path_no_ext}.safetensors has no RefMod metadata "
f"(header key '{META_KEY}' or sidecar .json missing)."
)
if meta.get("kind") == "bundle":
loaded: List[H3RefMod] = []
refs = refmod_bundle_members(meta)
with safe_open(path_no_ext + ".safetensors", framework="pt", device=device) as f:
for idx, member_meta in enumerate(refs):
latent = f.get_tensor(f"ref_{idx}").clone()
enc_frames = _read_encoder_images(member_meta, f.get_tensor)
loaded.append(H3RefMod.from_metadata(member_meta, latent, enc_frames=enc_frames))
return loaded
tensors = load_file(path_no_ext + ".safetensors", device=device)
if "latent" not in tensors:
raise ValueError(f"{path_no_ext}.safetensors is missing its latent tensor.")
latent = tensors["latent"].clone()
audio_latent = tensors["audio_latent"].clone() if "audio_latent" in tensors else None
if meta.get("kind") == "audio":
return [H3RefMod.from_metadata(meta, latent)]
visual_meta = dict(meta)
visual_meta["ref_audio_t"] = 0
visual_meta["audio_concept_type"] = ""
enc_frames = _read_encoder_images(meta, tensors.__getitem__)
loaded = [H3RefMod.from_metadata(visual_meta, latent, enc_frames=enc_frames)]
if audio_latent is not None and int(meta.get("ref_audio_t", 0) or 0) > 0:
loaded.append(H3RefMod.from_metadata(_standalone_audio_member_meta(meta), audio_latent))
return loaded
def save_refmod_bundle(path_no_ext: str, name: str, mods: List["H3RefMod"]) -> str:
unique = list({id(mod): mod for mod in mods}.values())
if not unique:
raise ValueError("RefMod bundle is empty.")
if sum(bool(mod.enc_frames) for mod in unique) > 1:
raise ValueError("Save RefMods with encoder pictures in separate files when more than one member has pictures.")
metadata = {
"_format_version": 5,
"kind": "bundle",
"name": str(name or "refmod"),
"members": [mod.metadata() for mod in unique],
}
tensors = {
f"ref_{idx}": mod.latent.detach().cpu().contiguous().clone()
for idx, mod in enumerate(unique)
}
for mod in unique:
tensors.update({key: value.contiguous() for key, value in mod.enc_frames.items()})
destination = path_no_ext + ".safetensors"
directory = os.path.dirname(destination) or "."
os.makedirs(directory, exist_ok=True)
fd, temporary = tempfile.mkstemp(prefix=".refmod-", suffix=".tmp", dir=directory)
os.close(fd)
try:
save_file(tensors, temporary, metadata={META_KEY: json.dumps(metadata)})
os.replace(temporary, destination)
finally:
if os.path.exists(temporary):
os.unlink(temporary)
return destination
def _read_encoder_images(meta, get_tensor):
record = meta.get("encoder_images") or {}
times = record.get("times", meta.get("enc_times", []))
prefix = record.get("prefix", "enc_")
return {f"enc_{index}": get_tensor(f"{prefix}{index}").detach().cpu().clone()
for index in range(len(times))}
def pack_encoder_images(mod, frames, times, fps=24.0):
import io as bytes_io
from PIL import Image
if frames.shape[0] != len(times):
raise ValueError("Encoder picture count does not match its timestamps.")
packed = {}
for index, frame in enumerate(frames):
buffer = bytes_io.BytesIO()
pixels = (frame.detach().cpu().float().clamp(0, 1) * 255).round().to(torch.uint8).numpy()
Image.fromarray(pixels).save(buffer, "JPEG", quality=95)
packed[f"enc_{index}"] = torch.frombuffer(bytearray(buffer.getvalue()), dtype=torch.uint8)
mod.enc_frames = packed
mod.enc_times = list(times)
mod.enc_fps = float(fps)
# ═══════════════════════════════════════════════════════════════════════════
# Latent compression
# ═══════════════════════════════════════════════════════════════════════════
MODE_ALIASES = {
"full": "encode",
"pooled": "training",
"Full Reference": "encode",
"Compressed Reference": "training",
}
def normalize_mode(mode: str) -> str:
"""Map legacy mode names (``full``/``pooled``) to the current ones."""
return MODE_ALIASES.get(mode, mode)
def aspect_grid(pool_h: int, pool_w: int, aspect: float):
"""Even pool grid dims that match ``aspect`` (h/w) within the dial caps.
The DiT patches each 2x2 latent cell into one token and the rope grid is
area-normalized per axis, so a pooled grid that ignores the source aspect
squishes the subject: a portrait latent (tall person) forced into a square
16x16 grid is compressed 5x in height but only 3x in width, and comes out
"fat". The official ref2video node never pools — it feeds the model
aspect-correct latent dims, which is what the DiT was trained on.
The long edge follows the user's dial (``max(pool_h, pool_w)``) and the
short edge is derived from the source aspect, both rounded to even for the
2x2 patch. Square sources keep the exact dial value (16x16 stays 16x16).
"""
long_edge = max(pool_h, pool_w)
if aspect >= 1.0:
h, w = long_edge, long_edge / aspect
else:
w, h = long_edge, long_edge * aspect
h = max(2, round(h / 2) * 2)
w = max(2, round(w / 2) * 2)
return h, w
def pool_latent(
z: torch.Tensor,
latent_t: int,
latent_h: int,
latent_w: int,
) -> torch.Tensor:
"""
Average-pool a VAE latent ``[1, 24, T, H, W]`` down to a tiny grid.
``latent_h/latent_w`` are the *latent* grid dims (the DiT patches each 2x2
latent cell into one token), so a 4x4 latent = 2x2 = 4 tokens per frame.
Dims must be even (the DiT's 2x2 patch). Pooling runs in fp32; the result
keeps the input dtype.
"""
if z.shape[2] == latent_t and z.shape[3] == latent_h and z.shape[4] == latent_w:
return z
if latent_h % 2 != 0 or latent_w % 2 != 0:
raise ValueError(f"pool_h/pool_w must be even (got {latent_h}x{latent_w})")
pooled = F.adaptive_avg_pool3d(
z.float(), (latent_t, latent_h, latent_w)
)
return pooled.to(z.dtype)
def optimize_latent(
z_small: torch.Tensor,
z_full: torch.Tensor,
steps: int = 150,
lr: float = 00.02,
device: Optional[torch.device] = None,
progress_every: int = 0,
) -> torch.Tensor:
"""
Model-free refinement of the compressed latent.
Optimizes the small latent so its trilinearly upsampled reconstruction
matches the full reference latent. This pulls the pooled representation
closer to what the DiT would see from the full ref, with nothing but the
tiny latent trainable (~1-2K params) and no diffusion model loaded.
``progress_every`` > 0 prints a ``[RefMod] identity <step>/<steps>`` line
every N steps so long pooled-mode refinement isn't silent (the default
``identity=500`` takes a while per ref on CPU/VRAM-bound setups).
Returns the refined latent detached, same shape/dtype as ``z_small``.
"""
if steps <= 0:
return z_small
device = device or z_full.device
# Node execution runs under ComfyUI's global inference mode, which would
# disable the autograd this refine loop needs. Re-enable it for this
# scope and clone the inputs to drop the inference flag.
with torch.inference_mode(False), torch.set_grad_enabled(True):
target = z_full.clone().float().to(device)
param = nn.Parameter(z_small.clone().float().to(device))
opt = torch.optim.Adam([param], lr=lr)
size = tuple(target.shape[2:])
for i in range(steps):
opt.zero_grad()
up = F.interpolate(param, size=size, mode="trilinear", align_corners=False)
loss = F.mse_loss(up, target)
loss.backward()
opt.step()
if progress_every and (i + 1) % progress_every == 0:
print(f"[RefMod] identity {i + 1}/{steps}")
# materialize inside the scope so the result is a normal tensor, not
# an inference-mode tensor (it gets stored in the mod and reused)
refined = param.detach().to(z_small.dtype)
return refined
def optimize_latent_multi(z_init, targets, steps=150, lr=0.02, device=None,
progress_every=0, strategy="grouped"):
"""Mean reconstruction objective with bounded activation memory."""
if not targets or steps <= 0:
return z_init
if strategy not in ("grouped", "stream", "resident"):
raise ValueError("Unknown multi-reference optimization strategy.")
device = device or z_init.device
with torch.inference_mode(False), torch.set_grad_enabled(True):
count = len(targets)
if strategy == "grouped":
groups = {}
for target in targets:
key = tuple(target.shape)
value = target.detach().to(device="cpu", dtype=torch.float32).clone()
if key in groups:
groups[key][0].add_(value)
groups[key][1] += 1
else:
groups[key] = [value, 1]
grouped = []
for total, group_count in groups.values():
grouped.append((total.div_(group_count), group_count))
elif strategy == "resident":
resident = [target.detach().float().to(device) for target in targets]
param = nn.Parameter(z_init.clone().float().to(device))
opt = torch.optim.Adam([param], lr=lr)
for step_idx in range(steps):
opt.zero_grad()
loss = None
if strategy == "grouped":
total_weight = 0
for target_mean, group_count in grouped:
target = target_mean.to(device)
up = F.interpolate(param, size=tuple(target.shape[2:]), mode="trilinear",
align_corners=False)
contribution = F.mse_loss(up, target)
loss = contribution * group_count if loss is None else loss + contribution * group_count
total_weight += group_count
loss = loss / total_weight
elif strategy == "stream":
for target_cpu in targets:
target = target_cpu.detach().float().to(device)
up = F.interpolate(param, size=tuple(target.shape[2:]), mode="trilinear",
align_corners=False)
contribution = F.mse_loss(up, target)
loss = contribution if loss is None else loss + contribution
loss = loss / count
else:
for target in resident:
up = F.interpolate(param, size=tuple(target.shape[2:]), mode="trilinear",
align_corners=False)
contribution = F.mse_loss(up, target)
loss = contribution if loss is None else loss + contribution
loss = loss / count
loss.backward()
opt.step()
if progress_every and (step_idx + 1) % progress_every == 0:
print(f"[RefMod] identity {step_idx + 1}/{steps}")
refined = param.detach().to(z_init.dtype)
return refined
# ═══════════════════════════════════════════════════════════════════════════
# Per-frame strength curve
# ═══════════════════════════════════════════════════════════════════════════
# Direction names describe where the *concept* shows up in the output, which
# is the mirror of the strength envelope over the ref's own latent timeline:
# concept_at_end locks the ref's literal footage at the START and releases it
# toward the end (was "decrease"), concept_at_start opens free and locks onto
# the ref near the END (was "increase"). shape is how the envelope travels
# between its endpoints; value is the non-zero endpoint ("user input").
CURVE_DIRECTIONS = ("constant", "concept_at_start", "concept_at_middle",
"concept_at_end", "concept_at_ends")
CURVE_SHAPES = ("linear", "ease", "sigmoid", "tanh", "quadratic", "cubic",
"exponential", "stair", "elastic", "bump", "dip")
# old single-combo preset names -> (direction, shape, value) so workflows
# saved before the curve was split still resolve
_LEGACY_CURVES = {
"flat": ("constant", "linear", 1.0),
"fade_in": ("concept_at_start", "linear", 1.0),
"fade_out": ("concept_at_end", "linear", 1.0),
"bump": ("concept_at_start", "bump", 1.0),
"dip": ("constant", "dip", 1.0),
}
def _ease(shape: str, x: float) -> float:
"""Shape a progress ``x`` in [0, 1] per the easing name (may overshoot)."""
if shape == "linear":
return x
if shape == "ease": # smoothstep
return x * x * (3.0 - 2.0 * x)
if shape == "sigmoid": # logistic S-curve
return 1.0 / (1.0 + math.exp(-12.0 * (x - 0.5)))
if shape == "tanh": # sharper S-curve (steeper knee than sigmoid)
return 0.5 * (math.tanh(8.0 * (x - 0.5)) + 1.0)
if shape == "quadratic":
return x * x
if shape == "cubic":
return x * x * x
if shape == "exponential":
return 2.0 ** x - 1.0
if shape == "stair":
return min(1.0, math.floor(x * 4) / 3.0)
if shape == "elastic":
if x <= 0.0:
return 0.0
if x >= 1.0:
return 1.0
return 2.0 ** (-10.0 * x) * math.sin((x * 10.0 - 0.75) * (2.0 * math.pi / 3.0)) + 1.0
if shape == "bump":
return 1.0 - abs(2.0 * x - 1.0)
if shape == "dip":
return abs(2.0 * x - 1.0)
return x
def curve_strengths(spec, t: int) -> Optional[List[float]]:
"""Resolve a curve spec to ``t`` per-frame strength multipliers in [0, 1].
Kept as plain Python so the Apply node exposes the curve as normal
combo/float widgets instead of depending on ComfyUI's new Curve widget.
Accepts:
* a ``(direction, shape, value)`` tuple — direction is ``constant``
(one strength everywhere), ``concept_at_start`` (0 -> value, a
crescent — the concept shows early; legacy ``increase``),
``concept_at_end`` (value -> 0, a decrescent — the concept pops at
the end; legacy ``decrease``), ``concept_at_middle`` ([0..1..0] —
concept peaks mid-timeline) or ``concept_at_ends`` ([1..0..1] —
concept at both ends); shape is how the envelope travels between
its endpoints (see ``_ease``); value is the non-zero endpoint
(1.0 = full strength there);
* a legacy preset name (``flat``/``fade_in``/``fade_out``/``bump``/
``dip``) from before the split;
* a list of per-frame floats (len == t), used as-is;
* a list of ``(x, y)`` control points (x in [0, 1]), linearly
interpolated across the frames;
* any object with ``interp(x) -> float`` (legacy CurveInput saved in an
old workflow).
Returns None for flat/no curve so the caller keeps its single-strength
path unchanged (a ``constant`` curve at value 1.0 included).
"""
if t <= 1 or spec is None or spec == "":
return None
if isinstance(spec, str):
legacy = _LEGACY_CURVES.get(spec)
return curve_strengths(legacy, t) if legacy is not None else None
if (isinstance(spec, tuple) and len(spec) == 3
and isinstance(spec[0], str) and isinstance(spec[1], str)):
direction, shape, value = spec
value = float(value)
if direction == "constant":
# linear = flat at ``value`` (1.0 == no curve); the other shapes
# modulate around the constant, so constant + dip is a trough,
# constant + bump is a peak (both endpoints equal)
if shape == "linear":
return None if value >= 1.0 else [value] * t
p = [_ease(shape, i / (t - 1)) for i in range(t)]
return [max(0.0, min(1.0, value * y)) for y in p]
p = [_ease(shape, i / (t - 1)) for i in range(t)]
# legacy "increase"/"decrease" (pre-rename workflows) map onto the same
# envelopes as the concept-placement names
if direction in ("increase", "concept_at_start"):
return [max(0.0, min(1.0, value * y)) for y in p]
if direction in ("decrease", "concept_at_end"):
return [max(0.0, min(1.0, value * (1.0 - y))) for y in p]
if direction == "concept_at_middle": # [0..1..0]: concept peaks mid-timeline
return [max(0.0, min(1.0, value * (1.0 - abs(2.0 * y - 1.0)))) for y in p]
if direction == "concept_at_ends": # [1..0..1]: concept at both ends
return [max(0.0, min(1.0, value * abs(2.0 * y - 1.0))) for y in p]
return None # unknown direction: treat as flat
if isinstance(spec, (list, tuple)):
if len(spec) == t and all(isinstance(v, (int, float)) for v in spec):
return [float(v) for v in spec]
pts = [(float(x), float(y)) for x, y in spec
if isinstance(x, (int, float)) and isinstance(y, (int, float))]
if pts:
pts = sorted(pts, key=lambda p: p[0])
out = []
for i in range(t):
x = i * (1.0 / (t - 1))
if x <= pts[0][0]:
out.append(pts[0][1])
elif x >= pts[-1][0]:
out.append(pts[-1][1])
else:
for (x0, y0), (x1, y1) in zip(pts, pts[1:]):
if x0 <= x <= x1:
out.append(y0 + (y1 - y0) * (x - x0) / (x1 - x0))
break
return out
interp = getattr(spec, "interp", None)
if callable(interp):
return [float(interp(i / (t - 1))) for i in range(t)]
return None
def curve_value_at(spec, x: float) -> float:
"""Per-progress strength of a curve spec at one point ``x`` in [0, 1].
The continuous analog of ``curve_strengths`` — same direction/shape/value
math, but evaluated at a single progress value instead of discretized over
frames. The step-curve node uses it to re-mix the ref latents once per
denoising step, where ``x`` is the denoise progress (0 = first step,
1 = last step). Returns 1.0 (no weakening) for anything unrecognized.
"""
if spec is None or spec == "":
return 1.0
if isinstance(spec, str):
legacy = _LEGACY_CURVES.get(spec)
return curve_value_at(legacy, x) if legacy is not None else 1.0
if (isinstance(spec, tuple) and len(spec) == 3
and isinstance(spec[0], str) and isinstance(spec[1], str)):
direction, shape, value = spec
value = float(value)
if direction == "constant":
if shape == "linear":
return value
return max(0.0, min(1.0, value * _ease(shape, x)))
y = _ease(shape, x)
if direction in ("increase", "concept_at_start"):
return max(0.0, min(1.0, value * y))
if direction in ("decrease", "concept_at_end"):
return max(0.0, min(1.0, value * (1.0 - y)))
if direction == "concept_at_middle": # [0..1..0]: concept peaks mid-timeline
return max(0.0, min(1.0, value * (1.0 - abs(2.0 * y - 1.0))))
if direction == "concept_at_ends": # [1..0..1]: concept at both ends
return max(0.0, min(1.0, value * abs(2.0 * y - 1.0)))
return 1.0
def dedup_frame_indices(z: torch.Tensor, threshold: float = 0.02) -> List[int]:
"""Indices of latent frames kept by greedy temporal dedup.
Each frame is compared to the last *kept* frame: if the mean-abs
difference between them, normalized by the frames' own magnitude, is
below ``threshold`` the frame is dropped as a (near-)duplicate. Video
refs (a dance loop, a static shot, a talking head) contain long runs of
frames that differ only by codec noise — each one still costs a token
per spatial patch in every DiT block, so cutting them is the cheapest
way to honor a token cap. Stacked image refs survive: different
angles/expressions land well above the threshold.
"""
t = z.shape[2]
if t <= 1:
return list(range(t))
flat = z[0].float() # [24, T, H, W]
kept = [0]
prev = flat[:, 0]
for i in range(1, t):
cur = flat[:, i]
denom = (cur.abs().mean() + prev.abs().mean()) / 2 + 1e-6
diff = (cur - prev).abs().mean() / denom
if diff >= threshold:
kept.append(i)
prev = cur
return kept
def fit_token_budget(latent: torch.Tensor, budget: int, label: str) -> torch.Tensor:
"""Bring a stacked latent's injected token count under ``budget``.
Tokens = (H/2)*(W/2) per frame (the DiT's 2x2 patch). When the stack
exceeds the budget it is cut in two cheap, loss-ordered passes:
1. temporal dedup — drop latent frames that are (near-)identical to
the last kept frame (free: they carry no new reference info);
2. budget-fit resample — uniformly subsample the remaining frames down
to the largest count that fits the budget.
The spatial dims are never touched: resizing a latent's H/W changes the
rope grid and degrades the reference, so time is the only honest lever.
``label`` is the mod name for the console notes.
"""
h, w = latent.shape[3], latent.shape[4]
per_frame = (h // 2) * (w // 2)
t = latent.shape[2]
if per_frame * t <= budget:
return latent
kept = dedup_frame_indices(latent)
if len(kept) < t:
print(f"[RefMod] {label}: over {budget}-token cap, "
f"dropped {t - len(kept)} near-duplicate frame(s) "
f"({t} -> {len(kept)})")
latent = latent[:, :, kept]
t = latent.shape[2]
if per_frame * t > budget:
fit_t = max(1, budget // per_frame)
if fit_t < t:
idx = torch.linspace(0, t - 1, fit_t).round().long()
latent = latent[:, :, idx]
print(f"[RefMod] {label}: still over {budget}-token cap, "
f"resampled {t} -> {fit_t} frame(s) to fit")
if per_frame * latent.shape[2] > budget:
print(f"[RefMod] {label}: one frame alone is {per_frame} tokens > "
f"{budget} — lower ref_resolution or pool_h/pool_w to meet "
f"the cap.")
return latent
# ═══════════════════════════════════════════════════════════════════════════
# H3RefMod — the saved artifact
# ═══════════════════════════════════════════════════════════════════════════
@dataclass
class H3RefMod:
"""
A compressed reference for MiniMax H3.
``latent`` is the VAE latent ``[1, 24, latent_t, latent_h, latent_w]`` — a
full-resolution encode (``mode="encode"``) or a pooled thumbnail
(``mode="training"``). ``kind`` is ``"image"`` (single latent frame) or
``"video"`` (a few frames), matching the native ref block kinds the
model's ``PackedLayout`` understands.
``mode="encode"`` stores the encode at the resolution the official ref2video
path uses, so the injected ref carries real identity detail;
``mode="training"`` stores a tiny average-pooled grid refined by gradient
steps (concept/motion, or identity at high pool sizes).
"""
name: str
kind: str
latent: torch.Tensor
latent_h: int = 4
latent_w: int = 4
latent_t: int = 1
mode: str = "training"
source: str = "" # "image" | "video" | "manual"
source_shape: str = "" # original latent dims as "TxHxW"
pool: str = "4x4x1" # pool_t x pool_h x pool_w
optimize_steps: int = 0
tags: List[str] = field(default_factory=list)
description: str = "" # optional text describing the concept (emitted by the loaders)
concept_type: str = "generic" # what this mod represents; see CONCEPT_TYPES above
audio_concept_type: str = "" # upstream-style audio role label for embedded audio, if present
# Optional audio identity: the audio-VAE latent [1, 32, 2, ref_audio_t] and
# its frame count. When present the mod is emitted as a "video_audio" (or
# standalone "audio") ref block so the DiT attends to the soundtrack too.
audio_latent: Optional[torch.Tensor] = None
ref_audio_t: int = 0
sample_rate: int = 32000
enc_frames: Dict[str, torch.Tensor] = field(default_factory=dict)
enc_times: List[float] = field(default_factory=list)
enc_fps: float = 0.0
def __post_init__(self):
if self.kind == "audio":
if self.latent.ndim != 4 or tuple(self.latent.shape[:3]) != (1, 32, 2) or self.latent.shape[-1] < 1:
raise ValueError("H3 audio RefMod must contain [1,32,2,T] with T >= 1.")
self.latent_t = int(self.latent.shape[-1])
self.latent_h = 0
self.latent_w = 0
self.audio_latent = None
self.ref_audio_t = 0
return
if self.kind not in ("image", "video"):
raise ValueError(f"kind must be 'image', 'video' or 'audio' (got {self.kind!r})")
if self.kind == "image":
self.latent_t = 1
if self.audio_latent is None or self.ref_audio_t <= 0:
self.audio_latent = None
self.audio_concept_type = ""
self.ref_audio_t = 0
# ── token budget ──────────────────────────────────────────────────
@property
def token_count(self) -> int:
"""Number of patchified tokens the mod injects into the packed sequence."""
if self.kind == "audio":
return 2 * self.latent_t
per_frame = (self.latent_h // 2) * (self.latent_w // 2)
return self.latent_t * per_frame
# ── native ref block ──────────────────────────────────────────────
def ref_block(self, strength: float = 1.0,
curve=None, use_video: bool = True,
use_audio: bool = True) -> Optional[Dict]:
"""
Build the ref block dict the model's ``PackedLayout`` / payload consumes.
The shape mirrors what the native ref2va nodes emit, so the pooled
latent rides the exact same path: ``cond_video_latents`` -> patchify ->
``video_patch_proj`` -> packed sequence with a 3D RoPE grid.
``strength`` weakens the ref by mixing the latent toward a heavily
blurred copy of itself (see ``_blur_latent``) rather than toward
zero or random noise: scaling toward zero pushes values out of the
normalized latent distribution (reads as grey output), and mixing
toward random noise reads as real-but-garbled content and decodes
as a wrong texture rather than "weaker" — a blurred copy stays
on-manifold while still discarding the specific detail that makes a
reference strong. ``strength <= 0`` drops the block entirely (no
tokens injected).
``curve`` (optional) is a per-frame strength spec for the ref's
latent timeline — a ``(direction, shape, value)`` tuple (e.g.
``("concept_at_end", "ease", 1.0)``), a legacy preset name, a list of
per-frame floats, ``(x, y)`` control points, or any object with
``interp(x)`` (see ``curve_strengths``). Each frame is mixed with
its own strength ``retention * curve(x)`` instead of one flat value,
so the ref can fade in/out across the video. A flat curve is
identical to passing ``strength`` alone.
"""
if strength <= 0.0 or (not use_video and not use_audio):
return None
if self.kind == "audio":
if not use_audio:
return None
latent = self.latent
strengths = curve_strengths(curve, self.latent_t) if curve is not None else None
if strengths is not None:
weights = [max(0.0, min(1.0, strength * value)) for value in strengths]
if any(value < 1.0 for value in weights):
st = latent.new_tensor(weights).view(1, 1, 1, self.latent_t)
latent = st * latent + (1.0 - st) * _blur_latent(latent)
elif strength < 1.0:
latent = strength * latent + (1.0 - strength) * _blur_latent(latent)
return {
"kind": "audio",
"ref_audio_t": self.latent_t,
"audio_latent": latent,
}
has_audio = use_audio and self.audio_latent is not None and self.ref_audio_t > 0
if not use_video:
if has_audio:
return {
"kind": "audio",
"ref_audio_t": self.ref_audio_t,
"audio_latent": self.audio_latent,
}
return None
latent = self.latent
if curve is not None and self.latent_t > 1:
strengths = curve_strengths(curve, self.latent_t)
if strengths is not None:
t = self.latent_t
st = torch.tensor(
[max(0.0, min(1.0, strength * s)) for s in strengths],
dtype=latent.dtype, device=latent.device)
st = st.view(1, 1, t, 1, 1)
latent = st * latent + (1.0 - st) * _blur_latent(latent)
elif strength < 1.0:
latent = strength * latent + (1.0 - strength) * _blur_latent(latent)
block: Dict = {
"kind": self.kind,
"latent_h": self.latent_h,
"latent_w": self.latent_w,
"latent": latent,
}
if self.kind == "video" or has_audio:
# The DiT only reads audio off video / video_audio / audio blocks —
# its "image" branch ignores audio_latent. So a mod carrying audio
# is always emitted as video_audio (an image mod is just a 1-frame
# video), which packs the audio rows immediately before the video
# rows on the shared cursor.
block["kind"] = "video_audio" if has_audio else "video"
block["latent_t"] = self.latent_t
block["ref_audio_t"] = self.ref_audio_t if has_audio else 0
block["audio_latent"] = self.audio_latent if has_audio else None
return block
# ── serialization ─────────────────────────────────────────────────
def metadata(self) -> Dict:
meta = {
"name": self.name,
"kind": self.kind,
"latent_h": self.latent_h,
"latent_w": self.latent_w,
"latent_t": self.latent_t,
"mode": self.mode,
"source": self.source,
"source_shape": self.source_shape,
"pool": self.pool,
"optimize_steps": self.optimize_steps,
"tags": self.tags,
"description": self.description,
"concept_type": self.concept_type,
"_format_version": 4,
}
if self.kind == "audio" or (self.audio_latent is not None and self.ref_audio_t > 0):
meta["audio_concept_type"] = self.audio_concept_type
meta["ref_audio_t"] = self.ref_audio_t
meta["sample_rate"] = self.sample_rate
if self.enc_frames:
meta["enc_times"] = self.enc_times
meta["enc_fps"] = self.enc_fps
return meta
def save(self, path_no_ext: str) -> str:
"""Save as a single ``{path}.safetensors`` with metadata in the header."""
os.makedirs(os.path.dirname(path_no_ext) or ".", exist_ok=True)
meta = self.metadata()
tensors = {"latent": self.latent.contiguous()}
tensors.update({key: value.contiguous() for key, value in self.enc_frames.items()})
if self.audio_latent is not None and self.ref_audio_t > 0:
tensors["audio_latent"] = self.audio_latent.contiguous()
save_file(tensors, path_no_ext + ".safetensors",
metadata={META_KEY: json.dumps(meta)})
return path_no_ext + ".safetensors"
@classmethod
def from_metadata(
cls,
meta: Dict,
latent: torch.Tensor,
*,
audio_latent: Optional[torch.Tensor] = None,
enc_frames: Optional[Dict[str, torch.Tensor]] = None,
) -> "H3RefMod":
if not isinstance(meta, dict):
raise ValueError("RefMod metadata must be a dict.")
record = meta.get("encoder_images") or {}
return cls(
name=meta.get("name", "refmod"),
kind=meta.get("kind", "image"),
latent=latent,
latent_h=int(meta.get("latent_h", latent.shape[3] if latent.ndim == 5 else 0)),
latent_w=int(meta.get("latent_w", latent.shape[4] if latent.ndim == 5 else 0)),
latent_t=int(meta.get("latent_t", latent.shape[2] if latent.ndim == 5 else latent.shape[-1])),
mode=normalize_mode(meta.get("mode", "training")),
source=meta.get("source", ""),
source_shape=meta.get("source_shape", ""),
pool=meta.get("pool", ""),
optimize_steps=int(meta.get("optimize_steps", 0)),
tags=list(meta.get("tags", [])),
description=str(meta.get("description", "") or ""),
concept_type=str(meta.get("concept_type", "generic") or "generic"),
audio_concept_type=str(meta.get("audio_concept_type", "") or ""),
audio_latent=audio_latent,
ref_audio_t=int(meta.get("ref_audio_t", 0) or 0) if audio_latent is not None else 0,
sample_rate=int(meta.get("sample_rate", 32000) or 32000),
enc_frames=enc_frames or {},
enc_times=list(record.get("times", meta.get("enc_times", []))),
enc_fps=float(record.get("fps", meta.get("enc_fps", 0.0))),
)
@classmethod
def load(cls, path_no_ext: str, device: str = "cpu") -> "H3RefMod":
"""Load from ``{path}.safetensors`` (metadata in header, or legacy .json)."""
meta = read_refmod_meta(path_no_ext)
if meta is None:
raise ValueError(
f"{path_no_ext}.safetensors has no RefMod metadata "
f"(header key '{META_KEY}' or sidecar .json missing).")
if meta.get("kind") == "bundle":
raise ValueError("This is a RefMod bundle. Load its members instead of treating it as one latent.")
# clone drops the file mmap, so the file isn't locked on Windows and
# can be re-saved over the same name
all_tensors = load_file(path_no_ext + ".safetensors", device=device)
latent = all_tensors["latent"].clone()
audio_latent = all_tensors["audio_latent"].clone() if "audio_latent" in all_tensors else None
enc_frames = _read_encoder_images(meta, all_tensors.__getitem__)
return cls.from_metadata(meta, latent, audio_latent=audio_latent, enc_frames=enc_frames)
+60
View File
@@ -0,0 +1,60 @@
from __future__ import annotations
import os
from typing import Any, Dict, Optional, Tuple
_H3_VAE_CACHE: Dict[Tuple[str, str, str, bool], Tuple[Any, Optional[Any]]] = {}
def _cache_key(video_path: str, audio_path: str, device, load_audio: bool) -> Tuple[str, str, str, bool]:
return (
os.path.abspath(video_path or ""),
os.path.abspath(audio_path or ""),
str(device),
bool(load_audio),
)
def load_h3_vae_pair(video_path: str, audio_path: str = "", device=None,
load_audio: bool = True) -> Tuple[Any, Optional[Any]]:
"""Load and cache the H3 video/audio VAE wrappers from safetensors paths."""
import comfy.model_management
import comfy.sd
import comfy.utils
if not video_path:
raise ValueError("load_h3_vae_pair: video_path is required")
device = device or comfy.model_management.get_torch_device()
key = _cache_key(video_path, audio_path, device, load_audio)
cached = _H3_VAE_CACHE.get(key)
if cached is not None:
return cached
sd, metadata = comfy.utils.load_torch_file(video_path, return_metadata=True)
video_vae = comfy.sd.VAE(sd=sd, metadata=metadata, device=device)
video_vae.throw_exception_if_invalid()
audio_vae = None
if load_audio and audio_path:
asd, ameta = comfy.utils.load_torch_file(audio_path, return_metadata=True)
audio_vae = comfy.sd.VAE(sd=asd, metadata=ameta, device=device)
audio_vae.throw_exception_if_invalid()
pair = (video_vae, audio_vae)
_H3_VAE_CACHE[key] = pair
return pair
def load_h3_vaes_from_av_encoder(av_encoder, device=None,
load_audio: bool = True) -> Tuple[Any, Optional[Any]]:
"""Resolve a MiniMax H3 VAE-loader payload into local video/audio VAE wrappers."""
video_path = getattr(av_encoder, "video_path", None)
audio_path = getattr(av_encoder, "audio_path", None) or ""
if not video_path:
raise ValueError(
"H3RefModExtract: av_encoder is missing video_path. Connect a valid "
"MiniMax H3 VAE loader output."
)
return load_h3_vae_pair(video_path, audio_path, device=device,
load_audio=load_audio)
@@ -0,0 +1,138 @@
"""Generate random PNG thumbnails for MP4 files in a directory.
Each thumbnail is extracted at full frame resolution from a random point
between 25% and 75% of the video's duration, then saved beside the source MP4
with the same basename and a `.png` extension.
Rerun the script to regenerate different thumbnails.
Usage:
python3 generate_video_thumbnails.py /path/to/folder
python3 generate_video_thumbnails.py /path/to/folder --seed 1234
"""
from __future__ import annotations
import argparse
import random
import subprocess # nosec B404
import sys
from pathlib import Path
def _probe_duration(path: Path) -> float:
cmd = [
"ffprobe",
"-v",
"error",
"-show_entries",
"format=duration",
"-of",
"default=noprint_wrappers=1:nokey=1",
str(path),
]
result = subprocess.run(cmd, capture_output=True, text=True, check=True) # nosec B603
value = (result.stdout or "").strip()
if not value:
raise ValueError(f"Could not read duration for {path}")
duration = float(value)
if duration <= 0.0:
raise ValueError(f"Invalid duration for {path}: {duration}")
return duration
def _pick_timestamp(duration: float, rng: random.Random) -> float:
if duration <= 0.0:
return 0.0
start = duration * 0.25
end = duration * 0.75
if end <= start:
return duration * 0.5
return rng.uniform(start, end)
def _render_thumbnail(video_path: Path, timestamp: float) -> Path:
output_path = video_path.with_suffix(".png")
cmd = [
"ffmpeg",
"-y",
"-ss",
f"{timestamp:.6f}",
"-i",
str(video_path),
"-frames:v",
"1",
"-update",
"1",
str(output_path),
]
subprocess.run(cmd, check=True, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) # nosec B603
return output_path
def _iter_mp4s(folder: Path) -> list[Path]:
return sorted(
path for path in folder.iterdir()
if path.is_file() and path.suffix.lower() == ".mp4"
)
def _resolve_folder(folder_arg: str | None) -> Path:
folder_text = folder_arg
if not folder_text:
try:
folder_text = input("Folder containing MP4 files: ").strip()
except EOFError:
folder_text = ""
if not folder_text:
raise ValueError("No folder provided.")
return Path(folder_text).expanduser().resolve()
def main() -> int:
parser = argparse.ArgumentParser(
description=(
"Generate a random full-resolution PNG thumbnail for every MP4 in a folder. "
"The capture time is chosen between 25% and 75% of each video's duration."
)
)
parser.add_argument("folder", nargs="?", help="Folder containing MP4 files.")
parser.add_argument(
"--seed",
type=int,
default=None,
help="Optional random seed for repeatable thumbnail choices.",
)
args = parser.parse_args()
try:
folder = _resolve_folder(args.folder)
except ValueError as exc:
print(str(exc), file=sys.stderr)
return 1
if not folder.is_dir():
print(f"Folder not found: {folder}", file=sys.stderr)
return 1
rng = random.Random(args.seed)
videos = _iter_mp4s(folder)
if not videos:
print(f"No MP4 files found in {folder}", file=sys.stderr)
return 1
failures = 0
for video_path in videos:
try:
duration = _probe_duration(video_path)
timestamp = _pick_timestamp(duration, rng)
output_path = _render_thumbnail(video_path, timestamp)
print(f"{video_path.name} -> {output_path.name} at {timestamp:.2f}s / {duration:.2f}s")
except Exception as exc:
failures += 1
print(f"Failed: {video_path.name}: {exc}", file=sys.stderr)
return 1 if failures else 0
if __name__ == "__main__":
raise SystemExit(main())