Merge commit '70d0672ee306019c7dba8f799a273ec486bec95b' as 'vendor/dumas_minimax_h3_refmod'

This commit is contained in:
2026-10-20 13:43:44 +00:00
49 changed files with 7097 additions and 0 deletions
+2
View File
@@ -0,0 +1,2 @@
/tests/
/node.zip
@@ -0,0 +1,3 @@
__pycache__/
*.pyc
/tests/
+82
View File
@@ -0,0 +1,82 @@
# RefMod single-file bundle format
Version 5 is a container of independent H3 references. It does not concatenate
audio with visual latents or change H3 conditioning semantics. Existing
standalone version-4 files remain supported and keep their `latent` tensor key.
## Detection and layout
Read JSON from the safetensors header's `refmod_meta` key without loading tensors.
A bundle has `_format_version: 5`, `kind: "bundle"`, a display `name`, and an
ordered `members` array containing 1–256 ordinary RefMod metadata objects.
Nested bundles are not supported. A member's `kind` is `image`, `video`, or `audio`.
```json
{
"_format_version": 5,
"kind": "bundle",
"name": "hero",
"members": [
{
"_format_version": 4,
"name": "hero_visual",
"kind": "video",
"latent_t": 3,
"latent_h": 32,
"latent_w": 32
},
{
"_format_version": 4,
"name": "hero_audio",
"kind": "audio",
"latent_t": 200,
"sample_rate": 32000
}
]
}
```
This example omits optional member metadata. Each member keeps the usual mode,
source, tags, description, concept type and optional `refmod_config` JSON string.
Tensor `ref_0` belongs to member 0, `ref_1` to member 1, and so on.
| Kind | Tensor layout | Tokens |
| --- | --- | --- |
| image | `[1,24,1,H,W]` | `(H/2) × (W/2)` |
| video | `[1,24,T,H,W]` | `T × (H/2) × (W/2)` |
| audio | `[1,32,2,T]` | `2 × T` |
Visual H/W must be positive and even; T must be positive. Audio uses the existing
32 kHz H3 codec convention. Each member retains its own latent dimensions.
## Picker and loader integration
1. Detect `kind == "bundle"` and validate the supported format version.
2. Inspect members to display whether the file contains visuals, audio, or both.
3. Filter by **All**, **Visual** (`image` + `video`), or **Audio** before reading
tensor data. Skip members whose modality strength is zero.
4. Load the selected `ref_i` tensors and expand them into ordinary
`(H3RefMod, strength)` entries, preserving relative member order.
5. Multiply the slot strength by the modality strength. Account for each selected
member's token cost and runtime copies.
In this pack, `bundle.load_bundle(path_without_extension, selection,
visual_strength, audio_strength)` performs selection and returns ordinary
entries with modality strengths. The caller applies the slot strength/copies.
Members retain their container `path` and `bundle_index` for Config updates.
Save writes each distinct reference object once; runtime strengths and duplicate
copies are not stored. Repacking a loaded bundle flattens its selected members.
Config updates preserve other members and unknown container metadata/tensors.
## Compatibility and limits
New readers load older standalone files. Older readers expecting only a `latent`
tensor will need bundle support; there is no promise of forward compatibility.
The standalone Save node remains available for exporting members.
The container carries encoded references, not original audio bytes, explicit
cross-modal timestamps, or speaker-to-character assignments. Shared storage does
not guarantee synchronized generation or stop multiple references from mixing.
Per-slot Audio strength affects all audio members in that slot; this initial UI
does not select individual audio tracks within a multi-audio file.
+234
View File
@@ -0,0 +1,234 @@
# Changelog
All notable changes are tracked here. Each version is also published as a
[GitHub Release](https://github.com/Luisacaotica/ComfyUI-MiniMaxH3Mod/releases),
so you can keep using an older version if a new one changes something you
rely on.
## Unreleased
- Correct visual reference sampling to H3's 17k+5 video grid in the node and CLI. Preserve all frames of aligned clips and both endpoints when uniformly sampling; references/caps below five frames use the first image.
- Support MPS-resident RefMod latents without global CPU fallback: pool and blur explicitly on CPU, preserving return device, dtype and autograd. Synchronize MPS in the refinement timing harness.
- Add a reference_map output to Inspect H3 RefMod for LLM prompt generation before Text Encode, using the same reference numbering (#19).
- Fix Loader/Axis settings resetting when switching workflow tabs or reopening workflows: defer slot regrouping until widget restoration finishes (#18).
- Repair Apply's reactive MatchType socket on older frontends whose canvas socket classes use private fields, preventing missing controls while preserving conditioning type matching (#17).
## v0.2.6 — 2026-09-12
- Make Loader/Axis additive: start with one visible slot, add/remove slots up to eight, group modality controls with their reference, preserve old workflow values and protect connected slots.
- Add version-5 single-file RefMod bundles containing independent image/video/audio members, with atomic saves and preserved member order.
- Add Save H3 RefMod Bundle as an output node and opt-in Master save_layout=bundle. Keep standalone saving and existing workflow defaults.
- Add All/Visual/Audio selection and separate modality strengths to every Loader/Axis slot. Skip excluded tensors and count only selected references in token budgets.
- Discover combined files in the library; preserve unselected members during Config updates and support export to standalone files.
- Document the container schema for third-party pickers. Bundling does not add voice binding, AV synchronization, or original-waveform passthrough.
## v0.2.5 — 2026-09-08
- Rename Extract nodes to Create, with unchanged node IDs. Full Reference and Compressed Reference replace the visible encode/training names; old mode values remain accepted. Rename the identity control to Refinement Steps.
- Add visual budget_policy: truncate preserves existing behavior; error stops before saving when the visual token budget is exceeded. Master inherits the option.
- Allow larger token/frame limits and preserve the requested frame count in the motion preset. Loader budgets default to 0 (disabled).
- Add experimental H3 RefMod Text Encode with numbered reference presentation and a reference map. Support compatible ClipProj encoders and native batched video-VAE output.
- Add optional full-video inspection with correct IMAGE output dimensions; retain the lightweight first-frame default.
- Add playable music/video README examples and exclude local tests from distribution.
Validation: 51 local regression tests passed; creation-label hooks checked with Node.js. Full GPU generation, projected-encoder quality and voice transfer remain unverified. Voice cloning is not claimed to work.
Text Encode already attaches references: do not inject the same bundle again with Apply/Bridge. Integrated Continuum samplers still need a dedicated integration to use this conditioning. Full visual previews and text/vision presentation increase memory use.
## v0.2.0 — 2026-09-07
### Known limitation: voice transfer
Audio reference support includes an observed music-transfer example, but current
speaker-identity tests failed. This release does not provide working voice cloning.
The cause has not been isolated between H3/checkpoint behavior and the reference
presentation difference in the current RefMod integration.
### Follow-up implementation (2026-09-07)
- Clarify Master saving logs: deferred extraction, Created/Replaced destinations,
explicit save=False and a final token/file summary; report saved_paths in details.
- Add Save H3 RefMods as an output node: save mixed bundles without connecting
a Preview or sampler, with prefix/subfolder options and copy deduplication.
- Save new mods/presets in the first registered refmods root (extra_model_paths);
keep the standard models/refmods fallback and existing-file Config locations.
- Remove sibling-pack imports and startup dependency warning. Legacy av_encoder
inputs use native ComfyUI video VAE loading; standard VAE inputs are preferred.
- Add Extract H3 RefMod Master: optional image/video and audio inputs, sequential
extraction through the shared extractors, combined token budget, one bundle
output and separate compatible visual/audio files under a shared name.
- Step Curve now mixes the current payload without retaining tensors and handles
keyframe offsets, audio refs, empty conditioning and chained curves.
- Bridge uses the MODEL outer-sampling hook with finally cleanup; removes global
state and third-party monkey patches. The old Disarm node is deprecated.
- Align image/mask center crops; share resize and causal-frame helpers with CLI.
- Invalidate loaded files and queue cache on changes; cap cache at 256 MiB;
enforce copies/numeric limits; bound video decode buffers with unknown lengths.
- Add native audio RefMods, legacy AudioMod import, audio Extract, library UI,
inspection/optional previews, save subfolders, explicit scramble modes,
extraction presets and total-token budgets.
- Atomic saves, deduplicated config writes and grouped mean-MSE refinement.
- Historical entries below describe earlier unreleased revisions; the behavior
above and the current README supersede their bridge/cache descriptions.
### Fixed: linked loader inputs and subfolders
- Both loaders accept unresolved/empty mod names and connected STRING/combo
outputs. Fixed missing names still fail validation; connected names resolve
at execution with a missing-file error when necessary.
- Discover RefMods recursively as `folder/name`, preserving metadata filtering
and skipping graph presets. Listing cache tracks nested files, nanosecond
timestamps, and legacy JSON sidecars.
- Normalize Windows separators and check relative paths at execution. Cache
mods by file path so identical metadata names in different folders do not
collide when saving their configuration.
- Add regression tests using real safetensors and ComfyUI queue validation.
### New: Continuum integration
- **Continuum RefMod Bridge** node — injects your mods into
[ComfyUI-H3-Continuum](https://github.com/xiaolibai-sys/ComfyUI-H3-Continuum)'s
chunked long-video samplers (V2/V3/V3.4), which build their conditioning
internally and have no CONDITIONING socket. While armed, every chunk
carries the bundle's ref blocks as native `minimax_refs` — persistent
across all chunks, no Continuum markers so its continuity layout never
re-times them. Version-checked against one public seam; if Continuum
changes internally the bridge reports and degrades to no-injection rather
than breaking a run. One active bundle per graph; per-run scramble seeds
work normally. Wire it on the sampler's model line (Load Model -> bridge
-> sampler `model`): ComfyUI never executes fully disconnected nodes, so
the earlier "anywhere upstream" placement silently did nothing.
### Fixed: why insertions felt weak, random, or "possessed"
Three defects were found and fixed; each is reproducible with the new
`gauntlet_harness.py` (run it with ComfyUI's Python — no server needed).
- **Apply's default curve silently HALVED every multi-frame mod.** The old
default (`concept_at_end` + `ease`) runs its envelope across the mod's own
ref frames — mean multiplier ≈ **0.50** at any frame count — so video/
stacked mods shipped half-blurred unless you touched widgets. Worse, ref
tokens are not bound to output-video time at all: no frame curve can place
a concept "at the end of the video"; it only weights which reference
content dominates. Defaults are now `constant` + `linear` + `1.0`
(= full strength, official-ref parity); saved workflows keep their saved
values. The curve widgets remain for stack weighting, with honest tooltips.
For real output-timing control use **H3 RefMod Step Curve** (denoise
timeline).
- **Image-kind mods ignored every curve dial silently** (`curve_strengths`
short-circuits at 1 latent frame), so single-image identity mods always
injected at full strength — the "concept bleeds into everything" case.
Now `curve_value` acts as a plain strength cap on single-frame mods
(e.g. value 0.4 = the ref blends 40% toward its blurred self), with a
console note.
- **H3 RefMod Step Curve leaked state across generations.** The wrapper
cached pristine/blurred latents plus the marked-ref index mapping on the
first forward and never reset them — but ComfyUI reuses the patched MODEL
and conditioning payload across queued runs, so run #2 mixed run #1's
cached tensors whenever the scramble seed/mods/strengths changed
(corrupted, "ghost" output). The wrapper now keys its state on the
injected refs' identities (which it never mutates), re-keying cleanly on
every new run while keeping pristine stable within one; chaining multiple
Step Curve nodes composes deterministically, and progress is normalized to
each run's actual schedule start (partial denoise included).
Also new: Apply now prints each row's *effective* strength (row × retention
× curve-mean) and flags anything below 0.30 — the usual suspect when an
insertion feels weak.
### New: fixed configs, shipped with the mod
- **Fix H3 RefMod Config** node — bakes tuned Apply/Step-Curve settings
(`retention` + `curve` + `step_curve`) into a mod's safetensors metadata
and re-saves the file in place, so a concept ships with the settings that
make it work.
- **`override` toggle** on Apply H3 RefMod and H3 RefMod Step Curve — reads
the first mod in the bundle that carries a fixed config and uses those
settings instead of the widgets; falls back to the manual parameters
(with a console note) when no mod has one.
- The debug curve graph reflects the overridden curve, so what you see is
what actually ran.
### New: merge mode on Extract H3 RefMod
- **`merge` toggle** (training mode): one shared latent refined jointly
against every reference's full encode (mean reconstruction error) instead
of stacking each ref's own pooled latent. A whole collection becomes ONE
tiny consensus mod — pulled toward what's common across all the views
(structure/motion/identity) rather than any single shot's framing or
background, at one ref block's worth of tokens. `identity` dials the
joint refinement; masks and `background_retention` still apply per ref.
- **`motion_only` toggle** (training mode, experimental): video refs are
converted to per-frame temporal differences before encoding, so the mod
carries where/how things move instead of what they look like — static
appearance (lineart look, background) never enters the latent. A soft
motion guide, not a ControlNet (the ref channel is content-based).
### Fixed
- **Curve directions now mean what they say.** `concept_at_end` actually
puts the concept at the **end** of the video (and at the end of the
denoise timeline on H3 RefMod Step Curve); `concept_at_start` puts it at
the start. The old envelopes were inverted relative to the names. Legacy
`decrease`/`increase` workflow values and the preset names
(`fade_in`/`fade_out`/`bump`/`dip`) still resolve to their original
envelopes, so old workflows keep their behavior.
- **H3 RefMod Step Curve no longer touches your input video.** The wrapper
used to re-mix *every* ref latent, including the original video's
identity anchor — that blurred the source subject and let a different
person "pop". It now identifies the refs injected by Apply H3 RefMod (via
a marker on the blocks) and only re-mixes those; native ref2va refs pass
through untouched, and a console note explains when there is nothing to
re-mix.
## v0.1.0 — 2026-08-16
First tagged release. Extract references once as tiny `.safetensors`
"mods" and inject them through conditioning — no full video/image loading
every generation, no training.
### Nodes
- **Extract H3 RefMod** — image / video / GIF → one small mod file. Modes:
`training` (pooled concept/identity thumbnails; the `pool` dial trades
concept ↔ identity) and `encode` (full-resolution VAE encode). Identity
refinement steps, token cap with dedup, data multiplier for short clips,
optional `av_encoder` input, and folder bulk-loading.
- **Load H3 RefMods** — LoRA-loader-style rows with a typed strength and a
`copies` multiplier (2-10x — the manual row-duplication trick as a knob).
- **Load H3 RefMod Axis** — signed A/B sliders: negative picks the A mod,
positive the B mod, one dial controls both.
- **Load H3 RefMod Folder** — every image/video in a folder as an ordered
ref list.
- **Apply H3 RefMod** — one node for the pack's `MINIMAX_H3_COND` and the
built-in `CONDITIONING`. `retention` master strength; a curve split into
`curve_direction` (constant / concept_at_start / concept_at_middle /
concept_at_end / concept_at_ends) + `curve_shape` (linear / ease /
sigmoid / tanh / quadratic / cubic / exponential / stair / elastic /
bump / dip) + `curve_value`; `scramble_seed`; optional curve-graph
`debug` IMAGE output; shareable PNG graph presets (graph embedded in the
image metadata, legacy `.json` still loads).
- **H3 RefMod Step Curve** — the same curve widgets, but over the **denoise
timeline**: re-mixes every ref latent once per step (early steps lock
composition/identity, late steps stay clean or refine detail) via a
ComfyUI `DIFFUSION_MODEL` wrapper, attached between the model loader and
the sampler.
### Reference math
- Weakening a ref blends toward a blurred copy of itself instead of noise
or zero — stays on the latent manifold, so no grey/static output.
- The per-frame curve mixes each ref latent frame with
`retention * curve(x)` instead of one flat strength.
- Greedy temporal dedup + budget-fit resampling make the token cap cheap.
### Misc
- `extract_mod.py` standalone CLI.
- Mods live in `ComfyUI/models/refmods/` (created on first run, next to
loras/ and unet/); older mods in the pack's `mods/` folder still load.
+21
View File
@@ -0,0 +1,21 @@
MIT License
Copyright (c) 2026 Luisa (luisacaotica)
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
+131
View File
@@ -0,0 +1,131 @@
# MiniMaxH3Mod 0.3.0 plan
Status: proposed scope, 2026-09-13. Direction selected by the maintainer: **Library and workflow QoL, with measured quality improvements.** This document plans implementation; it does not certify features or experimental quality results.
## Release outcome
Users should be able to recognize a saved RefMod, find it quickly, understand its reference-token cost, combine it with other assets, and see how to refer to it in a prompt. Existing workflows must preserve their settings. Quality claims must come from comparable generation results.
The release has four headline additions: a visual library, cost feedback, contextual controls/reference guidance, and stack composition. Most work extends existing nodes; only stack composition needs a new node in the committed scope.
## What the research changes—and what it does not establish
Both supplied reports informed this plan. Their embedded instructions and example code are proposals, not implementation requirements. Their opaque citation markers are not independently reusable evidence.
| Research point | Verified baseline / planning decision |
|---|---|
| Add aspect-aware grids | Already implemented by `core.py:aspect_grid()` and used during extraction. Show the resolved grid and token count instead of implementing another pooling path. |
| Add subject masking | `mask` and `background_retention` already exist. Improve their workflow and preview; automatic segmentation is optional external preprocessing. |
| Add a library and tags | Search/type filtering and token counts already exist in the library; RefMod objects already have tags. Extend metadata exposure and browsing. |
| Add reference labels | Text Encode already produces a `reference_map`. Make the mapping visible and usable rather than introducing a second incompatible numbering scheme. |
| RefMods solve character bleeding / merge removes backgrounds | Not established. Treat these as quality questions, not release claims. |
| Voice failure has a known architectural cause | Not established by the supplied evidence. Audio cleanup is an experiment, not a voice-cloning fix. |
| Hard 16,384-token limit / exact VRAM from tokens | Do not adopt either claim. Report reference tokens and stored latent bytes separately from measured total generation VRAM. |
| Denoise presets imply output-time morphing | Denoise progress and video playback time differ. Name presets by their actual schedule. |
| Tests are unavailable | Tests exist locally, but `/tests/` is ignored and `git ls-files tests` returns no files. Make selected regressions reproducible from a clean clone. |
Current checkout also has `pyproject.toml` at 0.2.7 while `__init__.py` reports 0.2.6. Reconcile release metadata during packaging. Existing user changes to `pyproject.toml` and `node.zip` are not part of this planning edit.
## Committed scope
### 1. Visual RefMod library — P0, medium effort
Extend the existing dialog with thumbnail cards, list/grid view, tag filtering, and sorting by name, modified date, and tokens. Expand bundle details to show member names, modalities, and per-member costs. Keep the existing slot picker and connected-slot protection.
Creation can embed an optional small source thumbnail for browsing. Label it **Source preview**; it is not a reconstruction or a prediction of generated quality. Keep Inspect's VAE reconstruction as a separate action. Audio assets get a modality placeholder in this release; playable original audio is outside this thumbnail feature.
Proposed storage: bounded image bytes encoded into optional metadata, with a 160-pixel longest edge, a 32 KiB encoded limit per preview, and a 256 KiB aggregate preview limit per file. Final encoding must be checked against existing save/load/update paths. Missing or oversized previews fall back to a placeholder. List responses should omit image payloads; fetch previews on demand. Never load a VAE or full latent tensors just to browse.
Preserve optional preview metadata through standalone and bundle round trips, Config updates, and export. Test v4/v5 readers before deciding whether an additive field is sufficient; do not create v6 merely for thumbnails. Older files remain usable without regeneration, and adding previews to old files is an explicit action.
**Acceptance:** browse a 1,000-entry metadata fixture with incremental rendering; no VAE/GPU work; search/sort/tag filters compose correctly; bundle inspection and selection preserve member order; broken previews cannot prevent loading valid RefMods; reopening the dialog preserves useful view preferences.
### 2. Reference-cost feedback — P0, medium effort
Show each selected slot's active reference count and token contribution, plus the combined total and budget remaining. Account for bundle member selection, copies, zero strengths, and visual/audio strengths using the same rules as execution.
Use metadata and tensor headers for the preflight. Clearly distinguish **reference tokens**, **stored latent bytes**, and **total generation VRAM**. The first two can be computed for known inputs; the third depends on the model, output size, attention backend, and offloading. Connected or dynamically produced values show “resolved at execution” rather than a false exact total. Apply-level curves, retention, or scrambling may further change the effective total; the loader estimate must name its scope.
Explain directly that reducing a nonzero strength generally does not save reference tokens, while copies increase them. Offer actionable budget guidance; keep execution-time validation authoritative.
**Acceptance:** preflight agrees with loader execution for standalone/bundled visual and audio fixtures, modality exclusions, copies, and zero strength; unknown connected values are explicit; changing settings updates the display without queuing generation.
### 3. Contextual controls and reference guidance — P0, medium effort
Dim controls that the effective mode or preset ignores, and explain why. Preserve values and sockets so returning to Manual restores the user's choices. Derive behavior from actual backend preset resolution: do not copy the report's blanket rule that every preset disables every refinement control.
Expose the existing Text Encode reference map clearly, with a copy action and matching asset names/previews. Explain the difference between Apply's latent injection and Text Encode's multimodal presentation. Make the single-injection workflow obvious in the example graph.
Mapping must reflect actual active member order, modalities, exclusions, and copies. When the graph is dynamic, show the executed mapping rather than promising fixed numbers. Semantic aliases such as `hero` are a later extension, not a new model-recognized identity token.
Add descriptive Step Curve presets such as Constant, Early emphasis, and Late emphasis only after schedule tests. Preserve manual settings. Avoid “identity lock” or “texture lock” names until output evidence supports them.
**Acceptance:** Full/Compressed and preset transitions retain manual values; connected controls stay connected; graph reload/tab switching preserves all settings; displayed labels match the Text Encode payload; examples do not inject the same bundle twice.
### 4. Stack composition — P1, small-to-medium effort
Add **Join H3 RefMods**, with two `H3_REF_MODS` inputs and one `H3_REF_MODS` output. Repeated joins combine any number of loader outputs without changing the existing eight-slot loader schema or introducing another bundle type.
Preserve A-then-B order, per-reference strengths, duplicates, and bundle provenance. Do not mutate inputs or concatenate/copy latent tensors. Existing downstream budgets remain authoritative. A larger list does not imply an unlimited model context or free memory.
**Acceptance:** two loaders can produce more than eight selected references through joins; empty input lists work; order/strengths are exact; source lists remain unchanged; loader, Apply, Text Encode, Inspect, and Save Bundle accept the result.
### 5. Compatibility and reproducible delivery — P0, medium effort
Track the relevant tests in Git while continuing to exclude tests and fixtures from the install artifact where appropriate. Add CPU/schema and JavaScript checks to CI, plus real frontend workflow fixtures. Preserve node IDs, old widget positions, mode aliases, v4/v5 assets, and existing defaults.
Test the reported legacy combination (ComfyUI 0.34.0 / frontend 1.39.2), the reported newer combination (0.35.0 / 1.52.7), and one current supported combination selected at implementation time. Record tested combinations instead of claiming every intermediate version works. Exercise classic canvas and Nodes 2 where available, including tabs, save/reopen, copy/paste, and subgraphs.
Verify #17/#18 in real browsers; the current JavaScript regressions alone do not establish that matrix. Generate the release ZIP from the selected source revision, verify extraction/import, and synchronize version strings. Publish a small set of runnable workflow JSONs for library selection, composition, prompting, and quality comparison.
Audit V3 registration/replacement lifecycle early. Migrate only with an API version that supports the required features and with old-workflow coverage. A wholesale V1-to-V3 conversion is not a prerequisite for library QoL. Official guidance recommends `ComfyExtension` / `comfy_entrypoint` and documents `latest` as under development: [V3 migration](https://docs.comfy.org/custom-nodes/v3_migration), [node replacement](https://docs.comfy.org/custom-nodes/backend/node-replacement).
## Measured quality work
Establish a baseline before changing extraction math or promoting quality presets. Use three visual cases: one identity, two similar subjects, and a short motion reference. Run at least three fixed seeds across four paths:
1. Native Ref2VA.
2. Full RefMod + Apply.
3. Full RefMod + Text Encode.
4. Compressed RefMod + Text Encode.
That is 36 planned visual runs. Match source media, crop/frame selection, checkpoint and encoder, prompt intent, output dimensions/duration, sampler, steps, and seed. Record differences that cannot be equalized. Add a small native-versus-RefMod music smoke comparison; speaker identity remains a separately scored experiment.
Store workflows, settings, asset/checkpoint hashes, output media, reference tokens, VAE/encoder/sampling timings, and peak VRAM. Score identity, prompt adherence, subject swaps/bleeding, and motion consistency through paired visual review. Optional embedding metrics supplement that review.
Only promote a quality preset when its results show a repeatable benefit on its stated use case and disclose cost/tradeoffs. If no improvement is demonstrated, ship the QoL release with the baseline and keep experimental paths labeled. Do not change defaults to manufacture a quality headline.
## Stretch scope and deferred research
| Proposal | Decision / promotion gate |
|---|---|
| Native text/tag loader | First stretch item after core QoL. Reuse the loader resolver, define escaping and line errors, preserve order, and support the existing runtime type. No `eval`, code execution, or ad hoc path bypasses. |
| Budget auto-fit | Experimental preview-only planner first: display proposed per-reference reductions and protected refs. Applying reductions must be explicit and leave source files intact. Automatic spatial/temporal changes require quality comparisons before promotion. |
| Stable prompt aliases / character profiles | Stretch after reference-map correctness. Aliases resolve to actual runtime labels; they do not guarantee identity separation. Defer richer profiles and a format revision until an actual consumer needs them. |
| SAM/YOLO auto-segmentation | Provide an optional example feeding the existing MASK socket. Avoid a mandatory segmentation dependency or hidden model download inside Master. |
| Background-plate latent subtraction / zeroed backgrounds | Research only. These operations do not establish clean foreground separation; compare with the existing mask path before considering production. |
| Vocal separation / audio normalization | Optional external preprocessing workflow and A/B experiment. Do not prescribe an arbitrary level or advertise a voice-cloning fix. |
| Spatial attention / bounding-box binding | Defer beyond 0.3.0 until the target H3 integration exposes a workable supported mechanism and experiments show less bleeding. Merely attaching metadata is insufficient. |
| Continuum per-chunk scheduling | Defer until chunk identity/timing and resume behavior have an explicit integration contract. Denoise schedules cannot substitute for a chunk timeline. |
| Conditioning cache / LRU changes | Profile first; defer unless current measurements show a user-visible bottleneck. Any diagnostics stay local. |
| Bundle v6 / provenance framework | Defer unless additive metadata demonstrably cannot serve a committed feature. Preserve existing asset readers. |
| Server path-policy changes | Separate opt-in deployment work; do not break desktop absolute-folder workflows as an incidental QoL change. |
## Implementation order and release gates
| Milestone | Deliverables | Exit condition |
|---|---|---|
| A — Baseline | Track tests, freeze old workflow fixtures, choose compatibility targets, capture quality inputs | Clean clone reproduces automated checks; benchmark workflows are runnable |
| B — Visual library | Metadata/preview persistence and endpoint, then cards/filtering/bundle details | Old and new assets round-trip; browsing performs no tensor/VAE loading |
| C — Workflow polish | Cost feedback, contextual controls, reference map, Join node, examples | Preflight/runtime agreement; stack and persistence regressions pass |
| D — Quality and RC | Execute comparisons, qualify presets, browser matrix, artifact verification | Results published, no settings-loss or connection regressions, install artifact validated |
Start baseline generation before UI implementation; it need not wait for the library. Preview storage precedes the gallery, and correct member accounting precedes cost feedback. The text/tag loader is the first item to cut if scope grows. Auto-fit, spatial binding, and voice research never block the committed release.
No calendar promise is attached yet: GPU benchmark availability and the oldest supported frontend are the main scheduling uncertainties. The next implementation step is milestone A followed by the preview metadata contract—not a broad node rewrite.
## Source notes
- Supplied `C:/Users/Pichau/Downloads/deep-research-report.md`: architecture, reproducibility, compatibility, and benchmark proposals.
- Supplied `C:/Users/Pichau/Downloads/research.md`: feature and QoL candidates, treated as hypotheses where their quality claims exceed evidence.
- Locally checked: `core.py`, `nodes.py`, `prompt.py`, `library.py`, `web/library_dialog.js`, `.gitignore`, `CHANGELOG.md`, `__init__.py`, and `pyproject.toml`.
+885
View File
@@ -0,0 +1,885 @@
*Update: https://github.com/Luisacaotica/ComfyUI-MiniMaxH3Mod/discussions/36
# ComfyUI-MiniMaxH3Mod - Changing the F#cking world with cigarretes and coffe.
> 🚧 **Under construction** — API and node schemas are still evolving. Mods
> stay compatible, but expect node names/inputs to shift between versions.
[![ko-fi](https://ko-fi.com/img/githubbutton_sm.svg)](https://ko-fi.com/C0C2EV9GW)
## What's new — v0.2.6
- **Standalone creators** — Create H3 RefMod and Master now run without a
downstream output node. Both return a `details` string with token counts and
saved paths. `save=False` still produces an in-memory bundle without files.
- **Different-size masks** — route a MASK list through **Collect H3 RefMod
Masks** into the creator's `mask_list` input. Each mask retains its dimensions
until it is resized/cropped with its corresponding reference. The collector
consumes the list once, avoiding one extraction per list item. Use `mask` or
`mask_list`; one mask broadcasts, otherwise provide one per reference in
image-slot, video-slot, then folder-bundle order. Existing MASK batches work
as before. Pixels cropped by an upstream batch node cannot be recovered.
- **Full Reference video sampling** — choose a causal frame count before
sampling, so frame limits such as 16 no longer drop the end of the clip.
- **Compact additive loaders** — Loader and Axis start with one visible slot.
Use **+ Add RefMod** to reveal more, up to the existing eight slots. Each
reference's controls stay together; connected slots are protected from removal.
- **Single-file visual + audio bundles** — Master can save both modalities in
one `.safetensors` with `save_layout=bundle`. **Save H3 RefMod Bundle** also
packs existing references without running extraction again.
- **Independent loader controls** — each Loader/Axis slot offers **All / Visual /
Audio**, plus visual and audio strengths. Excluded modalities and modalities
with strength zero are not loaded. Existing standalone files remain supported.
- **Bundle-aware library and Config** — find combined files in the library and
edit selected components without dropping the other contents.
This is a storage and selection update. It does not fix voice identity or enforce
audiovisual synchronization. See [single-file bundles](#single-file-bundles).
### v0.2.5 and subsequent fixes
On `main` after v0.2.5: **Refresh RefMods** updates the current Loader/Axis
dropdowns from the RefMod folders without refreshing all ComfyUI models.
Selections and connected inputs are preserved. The library's Refresh also
updates that loader's dropdowns.
- **Clearer creation controls** — Create node titles, Full/Compressed Reference modes and Refinement Steps; old workflows remain accepted.
- **Explicit visual budget policy** — choose truncate or error before saving, including Master.
- **Compatibility fixes** — projected ClipProj encoders and batched H3 VAE decoding in Text Encode.
- **Full visual inspection** — optional full-video preview, with a lightweight first-frame default.
Since v0.2.0: extraction widgets accept token budgets above
65,536 and frame limits above 16, including Master. The `motion_sequence`
preset now respects the requested frame limit. Counts remain integers;
defaults are unchanged.
**Experimental: H3 RefMod Text Encode** presents saved references to the native
H3 text/vision encoder and reports their `<Picture n>`, `<Video n>` and
`<Audio n>` labels. Character binding and voice quality still require generation
tests; this is not a validated voice-cloning fix.
- **RefMod Master** — extract image/video and audio references in one node,
with separate VAE inputs, one output bundle and a combined token budget.
- **Integrated audio** — create and load audio RefMods directly in this pack;
no H3AudioMod installation required. Mix visual and audio refs in Apply.
- **Known failure: voice transfer** — audio support includes an observed music
reference result, but speaker-identity transfer failed in current tests.
This update does not provide working voice cloning. See
[voice-transfer limitations](#voice-transfer-limitations).
- **Library and inspector** — search/filter your collection, select loader
slots, inspect token costs and optionally preview stored visuals or audio.
- **Save H3 RefMods** — a dedicated output node saves bundles without needing
a Preview, sampler or another downstream consumer.
- **Linked loaders fixed** — unresolved names from upstream nodes no longer
fail queue validation, including both sides of the Axis loader.
- **Subfolders and external drives** — recursive discovery and saving to the
registered RefMod root respect `extra_model_paths.yaml`.
- **Stale references and cache fixes** — Step Curve no longer retains refs
from previous calls; loaders detect overwritten files. Bridge injection is
scoped to the MODEL branch, with cleanup after sampling errors.
- **Extraction fixes** — masks follow image crops, video sampling stays
bounded when frame counts are unknown, and runtime token/copy limits are
enforced. Saves use atomic file replacement.
- **Fewer dependencies and repeated operations** — removed sibling-pack imports;
shared extraction helpers and grouped multi-ref refinement reduce duplication.
Full history: [CHANGELOG.md](CHANGELOG.md). Validation and remaining limitations:
[REVIEW_FOLLOWUP.md](REVIEW_FOLLOWUP.md).
## RefMod in brief
RefMod saves MiniMax H3 image, video or audio references as reusable
`.safetensors` files. Load a saved reference, combine it with others, and pass
the bundle to **Apply H3 RefMod**. **Create H3 RefMod Master** brings visual
and audio extraction into one node.
Extraction uses the corresponding VAE. It does not train H3 weights or require
the diffusion model. The visual mode named `training` refines a compressed
latent; it is not LoRA training or supervised concept learning.
Saving avoids re-encoding the source on each run. Compression can reduce the
number of reference tokens processed during generation, at the cost of lost
information. Encode mode keeps more detail but does not remove the attention
cost of those tokens. Neither mode guarantees that only the desired attribute
will transfer: identity, clothing, background and composition can still mix.
## Try it
A ready-made example mod ships in the repo: **`mods/vanellope_example.safetensors`**.
It appears as `vanellope_example` in the `Load H3 RefMods` dropdown after
install — plug it into `Apply H3 RefMod` at strength 1.0 and prompt for
a candy racer in a karting scene.
## Install
1. Use a current ComfyUI with native MiniMax H3 support. RefMod does not
import or require ComfyUI-MiniMaxH3. Connect a standard H3 video VAE to
`vae`, the audio VAE to `audio_vae`, and native CONDITIONING to Apply.
Old `av_encoder` connections are supported by reading the VAERef video
checkpoint through ComfyUI's native VAE loader; they no longer load the
sibling pack's VAE implementation or its audio checkpoint. Prefer `vae`
to share an already loaded instance. Existing pack-conditioning objects
remain accepted by Apply without importing their package.
2. **This pack**: clone into `custom_nodes/` and restart ComfyUI. Python
deps (`safetensors`, `numpy`, `Pillow`) are in `requirements.txt` and are
installed automatically by ComfyUI Manager (or `pip install -r
requirements.txt` manually). `opencv-python`/`imageio` are optional video
backends for the folder loader.
```bash
git clone https://github.com/Luisacaotica/ComfyUI-MiniMaxH3Mod custom_nodes/ComfyUI-MiniMaxH3Mod
```
Originally tested on Windows. Bounded [Apple Silicon checks](#apple-silicon)
also cover preprocessing and the real H3 VAE on M5 Ultra, macOS 27 and
PyTorch 2.14; they do not qualify every workflow or reference quality.
Visual creation also offers `budget_policy`: `truncate` (the default) uses
existing frame reduction to fit `max_tokens`; `error` stops before saving if
the combined visual latent exceeds the budget after the multiplier. This applies
to both Create and Master. `max_tokens=0` disables the cap. Master's audio policy
and combined budget remain separate.
## What extraction actually optimizes
Create/Master now print which visual controls the effective configuration ignores
and which values a preset replaces. Apply reports actual curve/retention overrides
and inactive scrambling controls. These are execution-time notices; widgets remain
visible and editable, and cached nodes do not rerun just to print a notice.
The preset tooltip lists exact assignments. `motion_sequence` preserves the frame
limit and Refinement Steps and disables frame-difference extraction (`motion_only`)
to retain the visual sequence. `concept_type` is metadata, not an algorithm selector.
On Apply, `curve_shape` still matters with `constant` for non-linear shapes; a missing
saved override does not disable manual controls or discard a valid graph preset.
The UI calls the modes **Full Reference** (`encode` internally) and
**Compressed Reference** (`training` internally). **Refinement Steps** retains
its historical input ID `identity`. Saved files and API workflows keep these
internal identifiers; the descriptions below use them when discussing code.
Node titles now use **Create** instead of **Extract**, with unchanged node IDs.
A saved visual latent enters H3 through the native reference-token path.
That shares the model's reference mechanism, but does **not** establish parity
with a complete native reference workflow: resizing, conditioning, prompt
processing, sampling and other references can differ.
| Visual mode | Operation | Tradeoff |
| --- | --- | --- |
| `encode` | Resize/preprocess and store the VAE encode | Keeps more reference detail; larger token cost. Useful as the baseline for identity comparisons. |
| `training` | Pool the latent, then optionally refine it against the original VAE latent | Fewer tokens; detail and motion may be lost. |
The `training` loss is MSE between the full latent and a trilinearly enlarged
small latent. Only the small latent is optimized. No DiT, text instruction,
identity recognizer or motion objective participates. `identity` is the number
of refinement steps, despite its historical name; 0 means pooling only.
Increasing it does not teach the model which attribute to preserve or discard.
`concept_type` is descriptive metadata, not a separate learning algorithm.
The aliases `full` → `encode` and `pooled` → `training` remain supported.
## Storage
Mods are saved in the **first registered `refmods` folder**, including mappings
from `extra_model_paths.yaml`. The fallback is `ComfyUI/models/refmods/`, created
when saving. Set `is_default: true` in the YAML mapping to prioritize that root.
A write failure is reported rather than silently switching drives. Config updates
keep the original path of a loaded file. Loaders search all registered roots and legacy locations, including
the pack's own `mods/` folder, so existing files keep working. The standalone CLI
uses the folder registry available in its process; it does not load YAML mappings
on its own.
Both **Load H3 RefMods** and **Load H3 RefMod Axis** scan subfolders too:
`models/refmods/celebs/person.safetensors` appears as `celebs/person`.
Names use `/` without the `.safetensors` extension; linked Windows names
using `\` work too. Only files with RefMod metadata (`kind: image`, `video` or `audio`) are listed. `graph_presets/`, `.git/`, and `__pycache__/` trees
are skipped. Extract saves to the selected root by default; `subfolder` is optional.
You can convert `mod_#`, `mod_a_#`, or `mod_b_#` to inputs and connect a
STRING/combo output. Unresolved linked values are checked when the loader
runs. `None`, an empty string, `"None"`, and `"(none)"` skip the slot;
a resolved missing filename raises a missing-file error. Fixed dropdown
values are still checked when queuing. After adding or moving files,
refresh ComfyUI's node definitions to update the visible dropdowns.
Loader regression checks (run with ComfyUI's Python from this pack):
`python tests/test_loaders.py`. They use temporary safetensors files and
ComfyUI's queue validator; no generation models are needed.
## Integrated audio, library and inspection
### One character extractor
**Create H3 RefMod Master** combines the visual extractor's inputs and controls
with optional `audio` and `audio_vae` inputs. Connect the visual H3 VAE to `vae`
(or the existing `av_encoder`) and the H3 audio VAE to `audio_vae`. Either modality
can be omitted. Video-frame inputs do not implicitly include an audio track;
connect the source loader's AUDIO output separately when needed.
The master extracts visual references first, then audio, returning one `mods`
bundle for Apply/Inspect and a `details` string with paths and token totals.
Visual `max_tokens` and `audio_max_tokens` are separate; `max_total_tokens` limits
their combined cost (0 disables that extra limit). A failed extraction or budget
check does not save either result. Each final file is saved atomically, but saving
the pair is not a filesystem transaction.
With the default `save_layout=separate_files`, `name=hero` and
`subfolder=characters`, saving creates
`characters/hero_visual.safetensors` and `characters/hero_audio.safetensors`.
Select both in Load H3 RefMods to reconstruct the bundle after restarting.
Choose `save_layout=bundle` to save `characters/hero.safetensors` instead;
load that one file and select its modalities in the loader.
The internal visual extractor reports `(not saved)` because Master postpones
saving until both extractions and the total-budget check succeed. Master's final
`Created` / `Replaced` messages show the actual paths. `encode` and `training`
both replace existing files at those paths; an older file without the modality
suffix is a different destination. With `save=False`, Master only returns the bundle.
These are appearance and voice references, not joint character training or a
guarantee of audiovisual synchronization. The individual extractors remain available.
**Create H3 Audio RefMod** accepts `AUDIO` and the H3 **audio** VAE (the
32 kHz codec, not the video VAE). The implementation follows the tested
ComfyUI-H3AudioMod encode path: stereo normalized `[1,32,2,T]` latents,
40 frames/second, 2 tokens/frame, bounded 10-second encode chunks.
`max_seconds` explicitly selects the initial duration of the clip. A token
budget overflow raises an error by default; optional `truncate` keeps a
contiguous prefix rather than resampling speech/music in latent time.
Chunk boundaries may affect codec continuity; this is reference conditioning,
not a promise of lossless audio reconstruction.
Save audio to the selected RefMod storage root with an optional `subfolder`. Existing files
from `models/audio_refmods/` using `audio_refmod_meta` load directly. Both
common loaders, Apply, Step Curve, Config and the model-scoped bridge accept
mixed visual/audio bundles. Single-file bundles have per-slot visual/audio
strengths; multiple audio references inside one file share its audio strength.
Keep different voices in separate files if they need individual controls.
Standalone saves use format version 4; combined files use version 5. Older visual
mods remain readable. No `<Name>` prompt triggers are implemented.
Both loaders now have a **RefMod library** button: search names, subfolders,
concepts and descriptions, filter image/video/audio/bundle, choose a slot and Use.
Refresh rescans the disk. Connected slots are protected from widget replacement.
Close and Escape dismiss the library. Registered extra model roots are honored.
**Inspect H3 RefMod** reports paths, shapes, metadata, saved config and total
cost after copies. Optional matching `vae` enables `stored` or
`compare_strength` previews: `visual_preview=first_frame` decodes the first
latent frame by default; `full_video` decodes the complete stored visual latent
as an IMAGE batch for a video-combine/save node. Full previews use more memory.
In comparison mode, the stored sequence is followed by the weakened sequence;
audio outputs preview up to two seconds. Comparison appends the weakened
preview after the stored preview. This shows stored information, not a
prediction of generated identity/style quality.
`max_total_tokens` on loaders/Apply/bridge limits the sum after copies
(0 disables it). Oversized bundles fail explicitly. Extraction also fails
when one frame alone exceeds its cap. Loaded-file cache checks file and
sidecar changes, and is bounded to 256 MiB and 24 entries.
Visual Extract offers opt-in presets: `manual` preserves current widgets;
`identity_encode` selects encode/1024px; `style_experimental` selects an
8×8 training grid; `motion_sequence` keeps an ordered temporal sequence.
These are starting settings, not quality guarantees or semantic disentanglement.
Both visual and audio Extract support an optional save subfolder.
Validation: `python -m unittest discover -s tests -v` runs production-code
regressions. `python gauntlet_harness.py --device cuda` compares resident,
streaming and grouped multi-ref refinement with numerical parity checks.
`python issue_regression_test.py` checks standalone creators, different-size
mask lists and video sampling without loading model weights.
`python tests/audio_smoke.py PATH_TO_H3_AUDIO_VAE` exercises the real codec,
safetensors roundtrip and native H3 reference layout without loading the DiT.
## File format, resolution and token budget
### How many references should I encode?
There is no validated optimum image count or clip duration. As a practical
starting experiment, use 5–10 distinct, clear images of the same subject,
one short video that shows the desired motion, or 5–15 seconds of clean audio.
These are starting points, not enforced limits or measured quality optima.
Compare against a smaller set with the same generation settings before adding
more. Duplicate or conflicting examples can add cost without useful detail.
Check what actually survives extraction:
- **Images:** Folder Loader defaults to `max_items=32`, so a folder containing
40 images is not fully loaded unless you raise that limit. The creator's
default stack mode stores images along one temporal axis; it does not train
a character model. More inputs can trigger the visual token cap and discard
frames. Inspect the saved shape and token count.
- **Video:** Folder Loader's `max_frames` and the creator's `latent_frames`
both affect the sequence. Full Reference limits source frames; Compressed
Reference limits latent frames after VAE encoding. Frame reduction can lose
motion and timing. Start with one relevant short clip and inspect its preview.
- **Audio:** `max_seconds` (Master: `audio_max_seconds`) defaults to 30 and
selects the **beginning** of the input. A two-minute file therefore does not
mean two minutes were encoded. Audio uses about **80 tokens/second**: 15s is
about 1,200 tokens, 30s about 2,400, and 120s about 9,600. Keeping all 120s
requires raising the duration limit and the default 5,120-token budget
(or disabling that budget with 0). The default overflow policy is an error;
`truncate` keeps only a prefix. Longer input is not guaranteed to improve voice
similarity and raises sampling attention cost.
Small saved files and chunked encoding do not make reference conditioning free.
Generation cost depends on the retained tokens across all active references,
including copies. Use Inspect H3 RefMod to check the actual total before raising
budgets. Loader, Apply and bridge `max_total_tokens` limits are separate from
extraction limits.
Standalone files contain a latent and JSON metadata in the safetensors header.
Visual latents have shape `[1,24,T,H,W]`; audio latents use `[1,32,2,T]`.
Version-5 bundles contain an ordered list of these references in one file;
they keep separate tensors, shapes and metadata. See [BUNDLE_FORMAT.md](BUNDLE_FORMAT.md)
for the schema used by loaders and third-party pickers.
Visual token count is **T × (H/2) × (W/2)**. H and W are even latent-grid
dimensions, not image pixels. Audio uses **2 × T** tokens.
| Visual latent grid | Tokens per latent frame |
| --- | --- |
| 8×8 | 16 |
| 16×16 | 64 |
| 32×32 | 256 |
| 64×64 | 1024 |
For example, a square 1024×1024 image encoded to a 64×64 latent costs 1024
tokens and about 192 KiB of fp16 tensor data, plus metadata. A rectangular
image with a 1024-pixel short edge can cost more. Four 16×16 latent frames
cost 256 tokens; sixteen 64×64 frames cost 16,384 tokens before budget fitting.
A small grid is spatial compression, **not a concept extractor**. It may retain
colors and large structures while losing face detail, texture or useful motion.
A larger grid preserves more information, including unwanted content. There
is no validated universal 8×8 concept / 16×16 identity sweet spot.
`multiplier` repeats the extracted visual latent along its time axis; loader
`copies` repeats a reference block. Both increase token cost. Neither adds new
information, and neither promises a proportional increase in influence.
Visual `max_tokens` defaults to 5120 (0 disables it). Budget fitting drops
near-duplicate latent frames, then resamples time if needed, after repetition.
This can damage motion timing; inspect the resulting frame/token count. If a
single spatial frame exceeds the cap, extraction raises an error. Audio instead
uses the explicit error/prefix-truncation policy described above.
## Nodes (`MiniMax-H3/mod`)
| Node | Purpose |
| --- | --- |
| H3 RefMod Text Encode | Encode a prompt with numbered saved references; outputs conditioning and the reference map. |
| Create H3 RefMod Master | Visual and/or audio extraction into one bundle, with separate VAE inputs. |
| Create H3 RefMod | Visual extraction, masks, compression and optional refinement. |
| Collect H3 RefMod Masks | Pack a MASK list or batch without resizing; connect to the creator's `mask_list`. |
| Create H3 Audio RefMod | Audio extraction with duration and token limits. |
| Save H3 RefMods | Save a bundle as an output node; no downstream connection required. |
| Save H3 RefMod Bundle | Pack selected references into one `.safetensors`; no downstream connection required. |
| Load H3 RefMod Folder | Ordered image/video references from a folder. |
| Load H3 RefMods | Up to 8 slots with strength, copies and optional total budget. |
| Load H3 RefMod Axis | Select A or B with a signed strength; 0 skips the slot. |
| Apply H3 RefMod | Append refs to native CONDITIONING or an existing pack-conditioning object. |
| H3 RefMod Step Curve | Change marked RefMod latents during denoising via the MODEL connection. |
| Fix H3 RefMod Config | Persist Apply/Step Curve settings in mod metadata. |
| Inspect H3 RefMod | Metadata, token totals, reference map and optional VAE previews. |
| Continuum RefMod Bridge | Inject refs through the MODEL sampling hook. |
Older `Apply H3 RefMod (Cond)` workflows migrate to the unified Apply node.
The legacy Bridge Disarm node remains for compatibility; bridge state now
belongs to the MODEL branch.
### Prompting with numbered RefMods (experimental)
Connect `Load H3 RefMods → H3 RefMod Text Encode.mods`, the native H3 CLIP
or an H3-compatible projected CLIP (current ClipProj Loader all-in-one),
and the H3 video VAE for visual references. Enter your prompt and connect the
CONDITIONING output to the sampler's positive input. Keep the workflow's normal
negative conditioning and generation latent. This node already attaches the
references: do not Apply or Bridge the same bundle again.
To generate a prompt with an LLM, connect the same `mods` to Inspect H3 RefMod
and H3 RefMod Text Encode. Set Inspect's `preview=off`, send its `reference_map`
output to the LLM, then connect the generated prompt to Text Encode. Inspect
produces the map without CLIP or VAE execution, avoiding a dependency cycle.
Connect `reference_map` to a text display to see the actual mapping. For example,
if it reports `<Picture 1> = alice` and `<Picture 2> = beth`, try:
```text
[Shot1]
The woman in <Picture 1> stands on the left.
The woman in <Picture 2> stands on the right.
They turn toward each other and smile.
```
The map counts each modality separately, in bundle order, excluding zero-strength
entries. Copies receive additional labels. Stacked photos saved as one video-kind
RefMod receive one `<Video n>` label, not one Picture label per original photo.
Keep different characters in separate files. Neither the filename nor description
is a trigger, and `<Subject n>` is not automatically bound to a loader slot.
Visual presentation requires decoding the stored latent for Qwen; it adds VAE and
vision-encoder work. Compressed latents reconstruct less detail than the original
photos. Videos are decoded and then sampled for Qwen at 2 fps; `reference_fps`
sets reconstructed playback timing (default 24). Original timing is not recovered
from pooled or stacked refs. Audio presentation uses the native numbered label
without decoding audio. Loader strengths affect the latents; saved Apply curves
and overrides are not applied by this node. `max_total_tokens` limits DiT reference
tokens, not Qwen tokens or VAE decode memory.
Presentation/payload tests pass, but full GPU generation and identity/voice
quality remain unverified. Compare against the old Apply workflow with the same
references, prompt and seed.
### Saving without a Preview or sampler
Connect `Extract / Master / Loader → Save H3 RefMods` and queue the workflow.
The Save node is an execution output, so its own output sockets may stay
unconnected. Set the extractor's `save=False` to avoid saving twice; the Save
node handles persistence. It also provides `mods` and `saved_paths` outputs.
`filename_prefix` is prepended to each mod name, and `subfolder` is relative to
the configured RefMod storage root. Existing destination files are replaced.
Repeated copies of the same mod are written once; distinct mods with the same
name receive numbered suffixes within the bundle. Files keep their stored
latents/config; loader strengths remain in the output bundle and are not baked
into the saved latents. As with other output nodes, ComfyUI may reuse cached
results when inputs have not changed.
### Single-file bundles
For new references, set **Create H3 RefMod Master → save_layout = bundle**.
For existing files, connect **Load H3 RefMods → Save H3 RefMod Bundle**, choose
a name/subfolder, and queue. The output node writes one file; it does not require
a sampler or Preview. Existing destinations are replaced atomically.
Both loaders expose these controls for each slot:
Only added slots are shown. **Remove RefMod N** clears that slot and restores its
defaults; slots with connected inputs cannot be removed. Old workflows reveal
their configured/connected slots automatically, and the library can reveal a
hidden slot when selecting a file. Slot numbers stay stable when another slot
is removed. Refresh the browser after updating to load the new interface.
| Control | Effect |
| --- | --- |
| `components_N = All` | Load every visual and audio member in that file. |
| `components_N = Visual` | Load images/videos only. |
| `components_N = Audio` | Load audio only. |
| `visual_strength_N`, `audio_strength_N` | Multiply the slot strength for the chosen modality; zero skips it. |
For example, load character A with **All**, and character B with **Visual** to
use both appearances and only A's audio reference. With slot strength `0.8`
and audio strength `0.5`, the resulting audio reference strength is `0.4`.
The total-token budget counts only loaded components, including copies.
The file stores each distinct reference object once, preserving member order.
Runtime copies and strengths are retained on the Save node's output but are not
baked into the file. Loading the file later uses the loader's current controls.
Config preserves unselected members when updating a combined file. To export
members back into standalone files, use **Save H3 RefMods** with a different
prefix or subfolder.
Existing standalone RefMods and workflows keep working. Version-5 files require
a bundle-aware reader; v0.2.5 and older third-party readers are not guaranteed
to load them. The default Master save layout remains `separate_files`.
Packaging visual and audio references together does **not** bind a voice to a
character, fix reference mixing, or add synchronization timestamps. Original
audio passthrough is also separate: this format stores encoded audio latents,
not the original waveform. To preserve the original soundtrack, connect the
source loader's AUDIO output to your video-saving workflow.
### Strength and reference-frame curves
Loader strength and Apply retention are multiplied. For a positive resulting
weight `w`, the stored latent is transformed as:
```text
w * latent + (1 - w) * blur(latent)
```
A zero row/master strength drops the reference block. A zero **frame-curve**
weight instead leaves a blurred frame in the block; it does not remove tokens.
At 1, the stored latent is unchanged. At 0.4, the mixture is 40% original and
60% blurred. This is not an attention weight or a percentage of identity.
Blur can leave palette, framing and broad structure while discarding desired
texture. Smooth identity fading and an in-distribution result are not guaranteed.
The curve on Apply runs across **the reference's latent frames**, including
stacked images. `concept_at_start`, `concept_at_middle`, `concept_at_end` and
`concept_at_ends` are historical labels for where the envelope is strongest
in that reference sequence. **They do not schedule an event in the output video.**
`constant` is the default. `curve_shape` selects the envelope shape and
`curve_value` its value; for one latent frame, value acts as a scalar cap.
Printed effective strength is a summary of the configured weights, not a
measurement of the model's actual attention or the resulting concept transfer.
### Using RefMods with ComfyUI-H3-Continuum
Connect `Load Model → Continuum RefMod Bridge → sampler.model`, and connect
`mods` to the bridge. The bridge clones the MODEL and appends refs through
ComfyUI's public outer-sampling hook before each chunk's conditioning is
prepared. It restores the previous conditioning on completion, error or
cancellation. There is no global bundle, TTL, or patch of Continuum code.
Different model branches can carry different bundles. `enable=False` removes
this bridge from its output clone. The old Disarm node is retained for saved
workflows; use the bridge toggle or bypass its MODEL output instead.
### Ref scrambling (seed)
`scramble_seed=-1` disables scrambling and keeps all bundle entries in order.
With a nonnegative seed, `scramble_mode=shuffle` changes only their order;
`subset` selects up to `scramble_keep` entries. The selection is deterministic
for the same seed and bundle. `legacy_subset` retains the old random-size subset
behavior for workflows missing the new option. These operations act on bundle
entries, not on frames inside one saved mod.
### Curve graph and presets
Apply's `debug` IMAGE output plots the configured envelope, not a prediction
of where a concept will appear. `save_preset_as` writes a PNG with embedded
curve metadata under the selected RefMod root's `graph_presets/` folder.
Choose it in `graph_preset` to override the manual curve widgets. Share that
PNG to share the settings. Older JSON presets remain readable.
### Ref strength over denoising steps
Connect **H3 RefMod Step Curve** between the model loader and sampler.
It applies an envelope over denoising progress, relative to the run's schedule
start. This is different from the reference-frame curve and from output time.
Early/late weighting can change the result, but there is no guarantee that it
isolates composition, identity, texture or any particular semantic attribute.
The wrapper processes the current payload without retaining reference tensors
between calls. It transforms only marked RefMod references; empty payloads pass
through. Other reference latents are not directly modified, although generated
content can still change through interactions among all conditioning inputs.
Chained Step Curve nodes compose their transformations.
### Fixing configs into a mod (`override`)
**Fix H3 RefMod Config** stores your chosen Apply and Step Curve settings
in the mod file. Recipients can reuse those settings, although results still
depend on their model, prompt, sampler and other references:
1. **Creator:** tune the Apply + Step Curve until the concept behaves,
then set the same values on `Fix H3 RefMod Config` (wire `mods`
through it — `Loader → Config → Apply`). Run once: the config
(`retention` + `curve` + `step_curve`) is written into the mod's
safetensors metadata and the file is re-saved in place. Share the
`.safetensors` as usual.
2. **User:** load the mod, flip **`override`** on Apply H3 RefMod (and/or
connect the bundle to H3 RefMod Step Curve's optional `mods` input and
flip its `override`) — the node reads the first mod in the bundle that
carries a config and uses *those* settings instead of the widgets.
`override` off (default) = today's behavior, manual parameters.
If a bundle has no mod with a saved config, `override` falls back to the
manual widgets and prints a note to the console — it never silently does
nothing. The curve graph `debug` output shows the overridden curve, so it's
obvious which settings actually ran.
### Concept axes (signed A/B sliders)
`Load H3 RefMod Axis` pairs an A-side mod and a B-side mod on **one signed
`value` slider** per row ([-1, 1]): negative values use the A mod, positive
values use the B mod, and the magnitude is the reference strength (same 0-1
math as the loader). A value of 0 skips the row. This selects between two references; it does not learn a continuous semantic
age axis. For example:
1. Extract a mod from your **young** refs (baby photos) and another from your
**old** refs (elder man).
2. `Load H3 RefMod Axis`: `mod_a` = young mod, `mod_b` = old mod,
`value = -0.6` → young at 60% strength, `value = +0.8` → old at 80%,
`value = 0` → no age reference at all.
Same for other contrasting references: clean ↔ weathered, modern ↔ vintage, calm ↔
energetic. Rows are independent, so several axes can live in one node (up to
8), and the output feeds the same `Apply H3 RefMod` nodes with the
same `retention` master control.
### Minimal workflow
1. Connect images/video frames to **Create H3 RefMod** or **Master**, with the
standard H3 video VAE. On Master, connect AUDIO and the H3 audio VAE if wanted.
Image slots use the first image of a batch; video slots preserve a sequence.
2. Choose a name and optional subfolder. For a visual baseline, compare `encode`
against `training` using the same refs and generation settings. The visual
node defaults are `training`, 16×16 grid, `latent_frames=16`, `identity=500`,
`ref_resolution=1024`, and `max_tokens=5120`; presets can override some of them.
3. Connect the output bundle directly to **Apply H3 RefMod**, or load the saved
file(s) with **Load H3 RefMods**. Connect your H3 conditioning to Apply and
its result to the sampler's matching conditioning input.
4. Use **Inspect H3 RefMod** for stored information and token cost. Evaluate the
generation against a no-RefMod baseline with the same prompt and seed.
Describe the desired subject/action in the prompt and what should remain from
other inputs. A prompt can clarify intent, but it does not guarantee selective
transfer, exact dance/camera reproduction, face replacement or silence between
spoken phrases. RefMod currently has no `<Name>` trigger parser.
## Examples
Development examples; screenshots may show older names/defaults. Individual
results are illustrations, not controlled benchmarks or quality guarantees.
### Voice-transfer limitations
Current voice tests did not reproduce the reference speaker's voice reliably.
In one test, a masculine voice reference was supplied for a female character,
but the output retained a feminine voice instead of matching the reference.
Treat speaker-identity transfer / voice cloning as not working in this update,
even though music-reference conditioning has produced a successful example.
H3's architecture or checkpoint behavior may contribute, but the cause has
not been isolated. There is also a known difference in the current RefMod
integration: Apply appends audio latents after text encoding, while the native
H3 reference node also presents numbered audio labels during tokenization.
The experimental H3 RefMod Text Encode node now uses that native presentation
path, but voice transfer has not been retested with it.
Until a matched native-versus-RefMod comparison rules out that difference,
we cannot attribute the failure exclusively to H3 or rule out RefMod integration.
### Music reference with RefMod
In this example, the author supplied a music clip to RefMod and reported that
the generated video included the reference music as background audio.
**Input music** — waveform preview with audio. Enable sound in the player.
https://github.com/user-attachments/assets/a3e1c07a-cc88-47e2-b754-2d8ca2bc3c19
[Original MP3](examples/audio_input.mp3) · [Waveform MP4](examples/audio_input_preview.mp4)
**Generated video** — the RefMod music-reference result.
https://github.com/user-attachments/assets/ef72642c-80c2-4152-82fe-915e6dd57824
[Original output MP4](examples/audio_refmod_example.mp4)
Prompt used, unchanged:
```text
[Shot1]
The ginger woman is walking on an office and dancing, the camera viewer is tracking in her head following every rotation and position.
music playing on background.
```
### Extracting
An image and a video ref being extracted and fed into a conditioning node:
![Create H3 RefMod in use](examples/Extracting_example.png)
### Loading mods
`Load H3 RefMods` with several mods stacked (LoRA-loader style, one
strength per row):
![Load H3 RefMods in use](examples/loading_ref_example.png)
### Loading a ref folder
`Load H3 RefMod Folder` pointed at an absolute path (a whole movie
dataset) — images + videos loaded in one shot, with the count shown in
the preview text:
![Load H3 RefMod Folder in use](examples/load_ref_folder_example.png)
### Pool-size examples
Observed results with 8×8 and 16×16 pools. These examples do not establish
a universal concept/identity split:
![Concept pool 8x8 — with and without the mod](examples/concept_example_with_without_comparission.gif)
![Identity pool 16x16](examples/id_example.gif)
An identity success from real use — the author's own face extracted as a
mod and prompted as **"ginger woman"**, and a third example pushed into
JoJo Bizarre-style rendering:
![Identity success — ginger woman / JoJo style](examples/id_sucess.gif)
### Curve controls
The same mod run three ways — no curve, `concept_at_end`, and
`concept_at_end` with the loader's `copies` set to 3 (the same row injected
three times; this example showed a stronger effect):
![Curve controls — nothing vs concept at the end vs concept at the end + 3 copies](examples/concept_curve.gif)
### Known limitations
Fast, high-speed motion is the hard case. A quick sequence — Sasuke doing
hand signs, a fight flurry, a rapid dance step — gets smeared into
something slower and softer, because the reference is compressed into a
handful of latent frames and the model fills the gaps with its own idea of
how motion looks:
![Fast motion gets smeared](examples/know_issues_1.gif)
Things that help today: extract only the *moments* that matter (trim the
clip to the actual hand-sign burst instead of the whole scene), use more
`latent_frames` / a higher `ref_resolution`, and describe the motion in
the prompt so the model has an anchor for what it's seeing.
### Bulk folder loading
`Load H3 RefMod Folder` reads supported images (png/jpg/webp/bmp/gif) and video
(mp4/webm/mov/mkv/avi) in a folder — images first, then videos, by filename.
Type an absolute path, or a folder name inside ComfyUI's `input/` (empty =
`input/` itself). Feed its `refs_bundle` output into `Create H3 RefMod` to
bulk-extract a whole character shoot in one go:
```
Load H3 RefMod Folder (folder: E:/vanellope_refs) ──refs_bundle──┐
├─ Create H3 RefMod ─> vanellope.safetensors
ref_image_1 (hand-picked shots) ─────────────────────────────────┘
```
Bundle refs are appended after the autogrow refs, so `ref_image_1` still
anchors the canvas. Unreadable files are skipped with a note; `max_items`
caps the item count and `max_frames` caps the number of sampled video frames.
### Multiple references and merge
By default, visual refs are encoded separately and stacked along the latent
time axis. A video contributes a sequence, not just one frame. Spatial resizing,
temporal pooling/sampling and the token cap can still discard information.
In encode mode, multiple refs use a common canvas based on the first source;
other aspect ratios can be center-cropped to fit it.
With `merge=True` in training mode, one grid minimizes mean reconstruction
error across targets. For equal target shapes, this has the same gradient as
reconstructing their average. That is not semantic discovery of what the
examples share: misaligned faces, motion and backgrounds can average into blur.
Compare merge against stacking rather than assuming it removes unwanted content.
The current refinement groups same-shaped targets on CPU and processes gradient
contributions sequentially. This reduces repeated work; it does not change the
objective into DreamBooth, Textual Inversion or control/target edit training.
### Motion-only extraction (experimental)
For a motion-transfer comparison with native Ref2VA, start with one video,
Full Reference, `multiplier=1`, `max_tokens=0` (or an adequate budget with
`budget_policy=error`), and enough source frames. On Apply, use `retention=1`,
`curve_direction=constant`, `curve_shape=linear`, `curve_value=1`, no preset
override, and `scramble_seed=-1`. A constant linear curve at **0** replaces
every reference frame with its blurred version; it does not disable the curve.
Spatial pooling and Refinement Steps do not affect Full Reference mode.
Use a **Ref2VA checkpoint and reference workflow** for this comparison.
[MiniMax's model card](https://huggingface.co/MiniMaxAI/MiniMax-H3#model-variants-and-input-specifications)
distinguishes FL2VA first/last-frame conditioning from Ref2VA video references.
Using the same first image in extraction and FL2VA generation does not make a
RefMod a motion controller. A still alone contains no temporal sequence;
connect the source clip to `ref_video`. RefMods guide generation through visual
references and do not guarantee an exact motion replay, even with Ref2VA.
Apply alone does not present the reference video to Qwen. Compare with H3
RefMod Text Encode when the workflow accepts external conditioning, using the
reported Video label and without applying the same refs twice. This reconstructs
visual presentation from saved latents, so it is still not an exact replay of
native Ref2VA preprocessing or source timing. Dense textual motion descriptions
are not a replacement for the reference's visual input. These are diagnostic
settings, not validated optimal settings or a guarantee of motion fidelity.
For longer video references, increase `latent_frames` and the token budget
together. In `encode`, `latent_frames` limits sampled **source frames** before
the VAE's temporal compression. The count is snapped down to H3's **17k+5**
video grid (5, 22, 39, ...), then sampled across the full clip, including its
last frame when more than one frame survives. For example, a limit of 16
selects 5 frames; a limit of 22 preserves all 22 frames of a 22-frame clip.
A source or limit below 5 uses only the first image, retaining RefMod's
still-image fallback. Already-aligned clips at or below the limit are unchanged.
The same grid applies before compressed-mode encoding and in the CLI;
motion-only differences are aligned after they are calculated. Uniform
sampling preserves the endpoints but changes the temporal spacing: it does
not preserve original timestamps or synchronize a separate soundtrack.
In `training`, `latent_frames` limits the
**latent frames** retained after encoding. It is not a duration in seconds.
`max_tokens=0` disables the extraction token cap; loader/Apply/bridge budgets
are separate. Higher values increase memory and attention cost and do not
guarantee faithful motion transfer.
`motion_only` in training mode encodes normalized absolute frame differences,
`abs(frame[t+1] - frame[t])`. These highlight change, including camera motion,
lighting variation and compression artifacts. They can still reveal contours
and appearance, and absolute differences do not explicitly encode direction.
This is not optical flow, a pose trajectory or a trained motion-conditioning
channel. H3 may interpret the difference images as visual content. Still-image
refs retain their appearance with a warning. Treat this mode as an experiment,
not as guaranteed separation of motion from identity or background.
## Standalone extraction (visual only)
Use the Python environment that runs ComfyUI. Both commands set their mode
explicitly; the CLI default is `training`.
```bash
# Store the VAE encode as an identity comparison baseline
python custom_nodes/ComfyUI-MiniMaxH3Mod/extract_mod.py \
--image char.png --vae path/to/h3_video_vae.safetensors \
--name my_character --mode encode --resolution 1024
# Compressed video reference; evaluate motion loss against encode
python custom_nodes/ComfyUI-MiniMaxH3Mod/extract_mod.py \
--video dance.mp4 --vae path/to/h3_video_vae.safetensors \
--name dance --mode training --pool 16 --latent-frames 16 --identity 500
```
Other options include `--pool-w`, `--max-tokens`, `--multiplier`, `--subfolder`,
`--output`, `--max-edge`, `--max-frames`, `--description`, `--concept-type` and
`--device`. Run `--help` for defaults. Video decode uses OpenCV or imageio.
Files use embedded metadata; legacy JSON sidecars remain supported.
## Validation and experimental training status
### Apple Silicon
RefMod uses native ComfyUI H3 models and ordinary PyTorch operations. Visual
pooling and strength blurring explicitly use CPU for MPS-resident latents,
then return the result to the original device and dtype. This covers GPU-only
intermediates without requiring `PYTORCH_ENABLE_MPS_FALLBACK=1`. CPU and CUDA
paths retain their existing arithmetic; model/VAE execution can still use MPS.
The transfers add preprocessing work for GPU-resident references, rather than
moving diffusion inference to CPU.
Run `PYTORCH_ENABLE_MPS_FALLBACK=0 python -m unittest compatibility_regression_test -v`
with ComfyUI's Python for the frame-grid, CPU/MPS pooling, blur, gradient and
timing-backend checks. MPS tests skip on other platforms. A checkout outside
ComfyUI needs its ComfyUI directory on `PYTHONPATH`. The existing
`python issue_regression_test.py` also checks node/CLI frame selection.
`python gauntlet_harness.py --device mps --output gauntlet_mps.json` synchronizes
MPS before and after each timed refinement. Its CUDA memory field remains null
on MPS; it is not an Apple GPU peak-memory measurement.
The corrected video grid changes newly extracted video RefMods that previously
used 4k+1 sampling/trimming. Existing saved RefMods remain readable, but must be
re-extracted to recover source frames that were discarded. These compatibility
fixes are not a claim of improved identity, motion or voice quality.
### Other validation
Run production regressions with `python -m unittest discover -s tests -v`.
The suite covers loaders/queue validation, storage, caches, curves, bridge,
Master orchestration and audio storage. Codec checks used a real audio VAE;
these are not full H3 voice/visual quality evaluations.
`gauntlet_harness.py` compares three implementations of the existing model-free
MSE refinement. `tests/h3_gradient_probe.py` is a separate gradient-feasibility
experiment with a small randomly initialized H3 architecture. Its success does
not establish compatibility with a full INT8/ConvRot checkpoint, memory fit,
concept learning or edit quality. A train-through-H3 ComfyUI node is not yet
implemented. See [REVIEW_FOLLOWUP.md](REVIEW_FOLLOWUP.md) for recorded results.
## License
[MIT](LICENSE) — © 2026 Luisa (luisacaotica).
+71
View File
@@ -0,0 +1,71 @@
# RefMod — implementação e validação
Atualizado em 07/09/2026. Relatório de implementação e validação da v0.2.0.
## Correções concluídas
- Salvamento de mods e presets respeita a primeira raiz registrada em `folder_paths`, incluindo YAML externo; apenas a ausência de raízes usa `models/refmods`. Selecionar a pasta não cria diretórios.
- Removidos imports e aviso de dependência do pack ComfyUI-MiniMaxH3. O socket legado `av_encoder` usa seu caminho de vídeo com o VAE nativo; Apply continua aceitando conditioning nativo e objetos antigos sem importar o pack.
- Loaders normal e Axis aceitam inputs conectados ainda não resolvidos na fila. Nomes fixos inválidos continuam sendo rejeitados; arquivos conectados inexistentes falham na execução.
- Descoberta recursiva, barras normalizadas, filtro de metadados e exclusão de graph_presets. Raízes registradas e caminhos legados têm precedência consistente entre listagem e carga.
- Step Curve processa o payload atual sem guardar tensores entre chamadas. Cobre refs visuais/áudio, payload vazio, CFG e curvas encadeadas. Schedule isolado por execução.
- Bridge usa hooks públicos no MODEL clonado, sem lista global ou monkeypatch do Continuum. Restaura o conditioning mesmo se o sampler falhar. Disarm permanece por compatibilidade; desligue a bridge no ramo MODEL correspondente.
- Máscaras seguem o crop da imagem. Vídeos sem contagem confiável têm amostragem limitada em memória e fechamento do decoder em finally.
- Cache invalida por fingerprint de arquivo/sidecar; loaders expõem IS_CHANGED. Teto de 24 arquivos e 256 MiB de tensores.
- Copies e valores numéricos são verificados na execução. Orçamento opcional soma todas as cópias; hard cap insuficiente para um frame gera erro explícito.
- Salvamento atômico e gravação de config deduplicada por caminho.
## Manutenção e recursos entregues
- **Extract H3 RefMod Master** reúne imagem/vídeo e áudio opcional, usa os extratores compartilhados em sequência e retorna um bundle único. Salva arquivos separados com sufixos `_visual` e `_audio`; inclui limite total de tokens e diagnóstico. Não representa treinamento conjunto de personagem.
- Blur centralizado; resize e preparo causal compartilhados entre CLI e nodes. Removidos loops e estado redundantes. Harness importa código de produção.
- **Extract H3 Audio RefMod**: VAE H3, mono/estéreo, resample para 32 kHz, encode em blocos de 10 segundos, limite de duração e orçamento por erro ou prefixo contíguo.
- Formato v4 suporta áudio [1,32,2,T], preservando leitura dos visuais antigos e audio_refmod_meta do H3AudioMod. Áudio e visual coexistem no bundle em arquivos independentes.
- Loaders, Apply, Config, Step Curve e bridge reconhecem áudio; models/audio_refmods é consultado.
- **Inspect H3 RefMod**: tipo, caminho, dimensões, descrição, config e custo total. Preview opcional da primeira imagem ou primeiros 2 segundos de áudio, com comparação de força e VAE correspondente.
- Biblioteca nos loaders: busca por nome/pasta/conceito/descrição, filtro de tipo, seleção de slot e Refresh. Endpoint lê apenas metadados. Slots conectados são protegidos contra seleção pelo widget.
- Subpasta opcional no Extract visual/áudio e CLI. Scramble separa embaralhamento de seleção de subconjunto; workflows antigos preservam o comportamento sem o novo parâmetro.
- Presets de identidade, estilo experimental e sequência de movimento. São configurações convenientes, sem alegação de melhoria de qualidade ou separação semântica.
## Validação
**39 testes aprovados na suíte completa antes da publicação da v0.2.0**, com Python embarcado e módulos reais do ComfyUI:
```powershell
../../../python_embeded/python.exe -m unittest discover -s tests -v
```
Cobertura: fila real com upstream não executado, ambos os loaders, caminhos, cache, limites, máscara, vídeo, bridge, curvas, áudio, escrita atômica, inspector, endpoint da biblioteca sem tensores e paridade contra a perda MSE completa com geometrias diferentes.
O módulo real do diálogo foi verificado no navegador em fixture local: layout visível, pesquisa, seleção e Escape com restauração de foco. Ainda é necessário experimentar a extensão no ComfyUI completo após reiniciar.
### VAE real de áudio
Checkpoint: E:\pinokio\api\comfy.git\app\models\vae\minimax_h3_audio_vae_fp32.safetensors. GPU: RTX 4060 Ti.
| Entrada sintética | Latente | Tokens | Decode estéreo a 32 kHz | Pico CUDA |
| --- | --- | --- | --- | --- |
| 0,5 s | [1,32,2,20] | 40 | 16.000 amostras | 668 MiB |
| 10,125 s | [1,32,2,405] | 810 | 324.000 amostras | 1.371 MiB |
Ambos passaram: layout nativo, valores finitos e roundtrip safetensors exato. Resultados: tests/audio_smoke_result.json e tests/audio_chunk_result.json. Reprodução: tests/audio_smoke.py.
### Otimização multi-ref
Três implementações reais, mesma GPU/processo, 4 targets sintéticos de 8x64x64 e 20 passos:
| Estratégia | Tempo | Pico CUDA | Erro máximo contra resident |
| --- | --- | --- | --- |
| Targets residentes, backward sequencial | 0,0538 s | 21,05 MiB | 0 |
| Transferência por target | 0,0934 s | 12,05 MiB | 8,05e-7 |
| Média ponderada por geometria | 0,0264 s | 12,04 MiB | 2,38e-6 |
A terceira é o padrão: preserva o gradiente da média MSE e evita reconstruções repetidas. Paridade passou com rtol=1e-4, atol=1e-5. Resultados em tests/gauntlet_result.json. Medição somente do refinamento sintético: não representa encode, geração completa ou ganho contra todo o pipeline anterior.
## Limites e próximo teste prático
Reinicie o ComfyUI para carregar os novos nodes e a biblioteca. Compare uma voz conhecida com/sem RefMod usando prompts e seeds fixos. Não foi executada geração completa H3/Continuum nesta tarefa: identidade vocal, música, qualidade visual, sincronização audiovisual e VRAM do pipeline inteiro ainda não foram medidos.
Estilo/conceito continuam baseados em reconstrução de latentes; metadados não criam um extrator semântico independente. Presets experimentais precisam de avaliação visual. Disparo por <Nome> permanece fora do escopo solicitado.
+29
View File
@@ -0,0 +1,29 @@
"""
ComfyUI-MiniMaxH3Mod — no-training "RefMod" reference adapters for MiniMax H3
Reference videos are expensive because they inject thousands of tokens into
the H3 packed sequence. A RefMod compresses a reference image/video into a
tiny pooled latent (~4-16 tokens) that still rides the model's native ref2va
path — the DiT attends to it through all 50 blocks like a real reference, but
at a fraction of the compute. Extraction needs only the H3 VAE: no diffusion
model load, no training.
Nodes
─────
MiniMaxH3RefModExtract — ref_image_1..N stills + ref_video_1..N clips -> saved mod
MiniMaxH3RefModFolderLoader — load every image/video in a folder as an ordered ref list
MiniMaxH3RefModsLoader — load 1-8 mods with a typed strength each (LoRA-style)
MiniMaxH3RefModsAxis — A/B mod pairs on one signed slider each (negative -> A, positive -> B)
MiniMaxH3RefModApply — inject the bundle into MINIMAX_H3_COND (pack) or CONDITIONING (built-in);
the old split Apply/ApplyCond merged into one node (old workflows migrate)
Standalone extraction (image/video files): see extract_mod.py
"""
__author__ = "Luisa (luisacaotica)"
__version__ = "0.2.6"
WEB_DIRECTORY = "./web"
from .nodes import NODE_CLASS_MAPPINGS, NODE_DISPLAY_NAME_MAPPINGS
__all__ = ["NODE_CLASS_MAPPINGS", "NODE_DISPLAY_NAME_MAPPINGS", "WEB_DIRECTORY"]
+65
View File
@@ -0,0 +1,65 @@
"""H3 audio encoding, adapted from the user's ComfyUI-H3AudioMod.
Uses its tested normalized [1,32,2,T] representation and bounded 10-second
encode path. Curves, storage and loaders are shared with visual RefMods.
"""
import math
import torch
import comfy.model_management as model_management
from comfy.ldm.minimax.audio_vae import MiniMaxH3AudioVAE
from .core import H3RefMod
def encode_audio(vae, audio, max_seconds=30.0, chunk_seconds=10.0):
waveform, sample_rate = audio["waveform"], int(audio["sample_rate"])
if waveform.ndim != 3 or waveform.shape[0] != 1 or waveform.shape[1] not in (1, 2):
raise ValueError("Audio must have one batch of mono or stereo samples [1,C,L].")
if sample_rate <= 0 or waveform.shape[-1] < 1:
raise ValueError("Audio is empty or has an invalid sample rate.")
if not math.isfinite(max_seconds) or not math.isfinite(chunk_seconds) or max_seconds <= 0 or chunk_seconds <= 0:
raise ValueError("Audio duration and chunk length must be positive.")
model = vae.first_stage_model
if not isinstance(model, MiniMaxH3AudioVAE):
raise ValueError("Connect the MiniMax H3 audio VAE, not the video VAE or another audio codec.")
waveform = waveform[..., :max(1, round(max_seconds * sample_rate))]
if waveform.shape[1] == 1:
waveform = waveform.repeat(1, 2, 1)
if sample_rate != 32000:
try:
from comfy.audio import resample
except ModuleNotFoundError as exc:
if exc.name != "comfy.audio":
raise
# ComfyUI versions before the torchaudio removal lack comfy.audio.
from torchaudio.functional import resample
waveform = resample(waveform, sample_rate, 32000)
model_management.load_models_gpu([vae.patcher], memory_required=0, force_full_load=True)
chunk = max(800, round(chunk_seconds * 40) * 800)
latents = []
with model_management.cuda_device_context(vae.device):
for start in range(0, waveform.shape[-1], chunk):
piece = waveform[..., start:start + chunk].to(device=vae.device, dtype=vae.vae_dtype)
z = model.encode(piece).detach().cpu()
if z.ndim != 4 or tuple(z.shape[:3]) != (1, 32, 2):
raise ValueError(f"H3 audio VAE returned an invalid latent: {tuple(z.shape)}")
latents.append(z)
return torch.cat(latents, dim=-1)
def make_audio_mod(vae, audio, name, max_seconds=30.0, max_tokens=5120,
budget_policy="error", description="", concept_type="voice"):
if not isinstance(max_tokens, int) or isinstance(max_tokens, bool) or max_tokens < 0:
raise ValueError("Audio token budget must be a non-negative integer.")
if budget_policy not in ("error", "truncate"):
raise ValueError("Audio budget policy must be error or truncate.")
latent = encode_audio(vae, audio, max_seconds)
if max_tokens and latent.shape[-1] * 2 > max_tokens:
if max_tokens < 2 or budget_policy == "error":
raise ValueError(f"Audio requires {latent.shape[-1] * 2} tokens; budget is {max_tokens}. "
"Lower max_seconds or choose truncate.")
if budget_policy != "truncate":
raise ValueError("Audio budget policy must be error or truncate.")
latent = latent[..., :max_tokens // 2].clone()
return H3RefMod(name=name, kind="audio", latent=latent, mode="encode", source="audio",
description=description, concept_type=concept_type, sample_rate=32000)
+98
View File
@@ -0,0 +1,98 @@
"""Version 5 containers; members remain ordinary independent H3 references."""
import json
import math
import os
import tempfile
from safetensors import safe_open
from safetensors.torch import save_file
from .core import H3RefMod, META_KEY, read_refmod_meta
def members(meta):
if not isinstance(meta, dict) or meta.get("_format_version") != 5 or meta.get("kind") != "bundle":
raise ValueError("Unsupported RefMod bundle format.")
refs = meta.get("members")
if not isinstance(refs, list) or not refs or len(refs) > 256:
raise ValueError("A RefMod bundle must contain 1–256 references.")
for ref in refs:
if not isinstance(ref, dict) or ref.get("kind") not in ("image", "video", "audio"):
raise ValueError("Invalid RefMod bundle member.")
return refs
def write_bundle(path, metadata, tensors, extra_metadata=None):
members(metadata)
directory = os.path.dirname(path) or "."
os.makedirs(directory, exist_ok=True)
fd, temporary = tempfile.mkstemp(prefix=".refmod-", suffix=".tmp", dir=directory)
os.close(fd)
try:
header = dict(extra_metadata or {})
header[META_KEY] = json.dumps(metadata)
save_file(tensors, temporary, metadata=header)
os.replace(temporary, path + ".safetensors")
finally:
if os.path.exists(temporary):
os.unlink(temporary)
return path + ".safetensors"
def save_bundle(path, name, mods):
# Copies affect runtime, not stored content; strengths are not baked in.
unique = list({id(mod): mod for mod, _ in mods}.values())
metadata = {"_format_version": 5, "kind": "bundle", "name": name,
"members": [mod.metadata() for mod in unique]}
tensors = {f"ref_{i}": mod.latent.detach().cpu().contiguous().clone()
for i, mod in enumerate(unique)}
return write_bundle(path, metadata, tensors)
def load_bundle(path, selection="All", visual_strength=1.0, audio_strength=1.0):
if selection not in ("All", "Visual", "Audio"):
raise ValueError("Components must be All, Visual or Audio.")
if any(not math.isfinite(weight) or not 0 <= weight <= 1
for weight in (visual_strength, audio_strength)):
raise ValueError("Component strengths must be finite numbers between 0 and 1.")
metadata = read_refmod_meta(path)
refs = members(metadata)
loaded = []
with safe_open(path + ".safetensors", framework="pt", device="cpu") as file:
for i, meta in enumerate(refs):
audio = meta["kind"] == "audio"
strength = audio_strength if audio else visual_strength
if strength <= 0 or (selection == "Visual" and audio) or (selection == "Audio" and not audio):
continue
latent = file.get_tensor(f"ref_{i}").clone()
if audio:
valid = (latent.ndim == 4 and tuple(latent.shape[:3]) == (1, 32, 2)
and latent.shape[-1] > 0)
else:
valid = (latent.ndim == 5 and tuple(latent.shape[:2]) == (1, 24)
and all(n > 0 for n in latent.shape[2:])
and all(n % 2 == 0 for n in latent.shape[-2:]))
valid = valid and (meta["kind"] != "image" or latent.shape[2] == 1)
if not valid:
raise ValueError(f"Invalid tensor layout for bundle member {i}.")
mod = H3RefMod.from_metadata(meta, latent, path, bundle_index=i)
if not audio:
mod.latent_t, mod.latent_h, mod.latent_w = latent.shape[2:]
loaded.append((mod, strength))
return loaded
def update_member(mod):
# Read the current container: updating only Visual must retain Audio, and
# unknown tensors/metadata must survive a metadata-only Config operation.
with safe_open(mod.path + ".safetensors", framework="pt", device="cpu") as file:
header = file.metadata()
metadata = json.loads(header[META_KEY])
refs = members(metadata)
if not 0 <= mod.bundle_index < len(refs) or refs[mod.bundle_index].get("name") != mod.name:
raise ValueError("Bundle changed since loading. Reload before updating its config.")
tensors = {key: file.get_tensor(key).clone() for key in file.keys()}
refs[mod.bundle_index] = {**refs[mod.bundle_index], **mod.metadata()}
tensors[f"ref_{mod.bundle_index}"] = mod.latent.detach().cpu().contiguous().clone()
return write_bundle(mod.path, metadata, tensors, header)
+237
View File
@@ -0,0 +1,237 @@
"""Shared helpers for the RefMod pack: media loading and the refmods folder."""
from __future__ import annotations
import os
import ntpath
import math
import random
from typing import List, Optional, Tuple
import torch
import comfy.utils
IMAGE_EXTS = {".png", ".jpg", ".jpeg", ".webp", ".bmp", ".gif"}
VIDEO_EXTS = {".mp4", ".webm", ".mov", ".mkv", ".avi", ".m4v"}
def refmods_dir() -> str:
"""First registered RefMod root, or the default; selection does not write."""
import folder_paths
try:
roots = folder_paths.get_folder_paths("refmods")
except KeyError:
roots = []
return roots[0] if roots else os.path.join(folder_paths.models_dir, "refmods")
def list_media_files(folder: str) -> Tuple[List[str], List[str]]:
"""(images, videos) directly under ``folder`` (top level only), sorted by name."""
images, videos = [], []
if os.path.isdir(folder):
for fn in sorted(os.listdir(folder)):
ext = os.path.splitext(fn)[1].lower()
p = os.path.join(folder, fn)
if os.path.isfile(p):
if ext in IMAGE_EXTS:
images.append(p)
elif ext in VIDEO_EXTS:
videos.append(p)
return images, videos
def load_image_file(path: str, max_edge: Optional[int] = None) -> torch.Tensor:
"""Load one image file -> [1, H, W, 3] float32 in [0, 1]."""
import numpy as np
from PIL import Image
with Image.open(path) as img:
img = img.convert("RGB")
w, h = img.size
if max_edge is not None:
scale = min(1.0, max_edge / max(w, h))
if scale < 1.0:
img = img.resize((max(1, round(w * scale)), max(1, round(h * scale))),
Image.LANCZOS)
arr = torch.from_numpy(np.asarray(img).copy()).float() / 255.0
return arr.unsqueeze(0) # [1, H, W, 3]
def _sample_video_frames(frames, total, max_frames, max_edge):
"""Uniform known-length sampling; bounded deterministic reservoir otherwise."""
import numpy as np
from PIL import Image
known = math.isfinite(total) and total > 0
total = int(total) if known else 0
targets = None
if total > max_frames:
targets = {round(i * (total - 1) / max(1, max_frames - 1)) for i in range(max_frames)}
kept = []
rng = random.Random(0)
for index, frame in enumerate(frames):
if targets is not None and index not in targets:
continue
# Treat underestimated counts as unknown after filling the reservoir.
slot = len(kept) if len(kept) < max_frames else rng.randrange(index + 1)
if slot >= max_frames:
continue
frame = np.asarray(frame)
if max_edge is not None:
h, w = frame.shape[:2]
scale = min(1.0, max_edge / max(h, w))
if scale < 1.0:
frame = np.asarray(Image.fromarray(frame).resize(
(max(1, round(w * scale)), max(1, round(h * scale))), Image.Resampling.BILINEAR))
item = (index, frame.copy())
if slot == len(kept):
kept.append(item)
else:
kept[slot] = item
if targets is not None and len(kept) == len(targets):
break
if not kept:
raise ValueError("Video contains no decodable frames.")
return torch.from_numpy(np.stack([frame for _, frame in sorted(kept)])).float() / 255.0
def load_video_file(path: str, max_frames: int = 240,
max_edge: Optional[int] = None) -> torch.Tensor:
"""Load bounded, ordered RGB frames; release either decoder on every exit."""
if max_frames < 1:
raise ValueError("max_frames must be at least 1.")
errors = []
try:
import cv2
cap = cv2.VideoCapture(path)
try:
total = cap.get(cv2.CAP_PROP_FRAME_COUNT) or 0
def frames():
while True:
ok, frame = cap.read()
if not ok:
break
yield cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
return _sample_video_frames(frames(), total, max_frames, max_edge)
finally:
cap.release()
except (ImportError, OSError, ValueError, RuntimeError) as exc:
errors.append(str(exc))
try:
import imageio.v2 as imageio
reader = imageio.get_reader(path)
try:
total = reader.get_meta_data().get("nframes", 0) or 0
return _sample_video_frames(iter(reader), total, max_frames, max_edge)
finally:
reader.close()
except (ImportError, OSError, ValueError, RuntimeError) as exc:
errors.append(str(exc))
raise RuntimeError(f"Cannot decode video '{path}': " + "; ".join(errors))
def mod_output_path(name, subfolder=""):
relative = "/".join(part for part in (subfolder.replace("\\", "/"), name) if part)
if (ntpath.splitdrive(relative)[0] or relative.startswith("/") or
any(p in ("..", ".", "graph_presets", ".git", "__pycache__") for p in relative.split("/"))):
raise ValueError("Use a relative RefMod subfolder without '..' or reserved directories.")
root = os.path.realpath(refmods_dir())
path = os.path.join(root, relative)
if os.path.commonpath((root, os.path.realpath(path + ".safetensors"))) != root:
raise ValueError("RefMod save path leaves the model folder.")
return path
def resize_ref(image, short_edge: int, canvas=None):
"""Aspect-preserving downscale (never upscale) to ``short_edge`` px; dims to /32.
When several refs are stacked into one mod they must share a single spatial
canvas, so ``canvas`` (tw, th) cover-crops each ref to it (like the official
node's follower keyframes). Mirrors the official ref2video node: refs are
resized before VAE encode, so the stored latent rides the same
full-resolution path the model was trained with (the pooled path below is
the cheap "thumbnail" alternative).
"""
h, w = image.shape[1], image.shape[2]
if h <= 0 or w <= 0:
raise ValueError(
f"_resize_ref: source has an empty frame ({h}x{w}) before any "
f"resize — the reference itself is invalid.")
scale = min(1.0, short_edge / min(h, w))
tw = max(32, round(w * scale / 32) * 32)
th = max(32, round(h * scale / 32) * 32)
crop = "disabled"
if canvas is not None:
tw, th = canvas
crop = "center"
if tw <= 0 or th <= 0:
raise ValueError(
f"_resize_ref: computed a zero-size resize target ({tw}x{th}) "
f"for a {h}x{w} source (short_edge={short_edge}, canvas={canvas}). "
f"This should be impossible — please report this shape.")
samples = image[..., :3].movedim(-1, 1)
samples = comfy.utils.common_upscale(samples, tw, th, "lanczos", crop)
if samples.shape[2] <= 0 or samples.shape[3] <= 0:
raise ValueError(
f"_resize_ref: common_upscale produced an empty result "
f"{tuple(samples.shape)} from a {h}x{w} source targeting "
f"{tw}x{th} (crop={crop}, canvas={canvas}). This points to a bug "
f"in comfy.utils.common_upscale for this input, not in RefMod's "
f"own math.")
return samples.movedim(1, -1)
def snap_to_causal_grid(n_frames: int) -> int:
"""Round down to H3's reference-video grid: 5, 22, 39, ... (17k+5).
The VAE encodes 17-frame chunks, pads the final chunk and drops three
trailing latent tokens. Its effective video grid is not the encoder's
standalone 4x temporal stride. RefMod keeps its single-image fallback
when fewer than five frames are selected.
"""
if n_frames < 1:
raise ValueError("A reference must contain at least one frame.")
if n_frames < 5:
return 1
return ((n_frames - 5) // 17) * 17 + 5
def sample_video_for_vae(frames: torch.Tensor, max_frames: Optional[int] = None) -> torch.Tensor:
"""Sample the full clip onto H3's grid without exceeding the frame cap.
Keep both endpoints whenever the selected count is greater than one.
Already-aligned clips are returned unchanged. A cap/input shorter than
five frames selects only the first image, using H3's still-image path.
"""
if max_frames is not None and max_frames < 1:
raise ValueError("max_frames must be at least 1.")
count = snap_to_causal_grid(min(frames.shape[0], max_frames)
if max_frames is not None else frames.shape[0])
if count == frames.shape[0]:
return frames
indices = torch.linspace(0, frames.shape[0] - 1, count, device=frames.device).round().long()
return frames[indices]
def ensure_min_size(image, floor: int = 320):
"""Upscale (never downscale) so both spatial dims are >= ``floor`` px.
The MiniMax H3 VAE encodes with internal tiled_encode (~256px tiles). A
reference smaller than the tile size in one dimension can make the tiler
compute a zero-size edge tile, which crashes deep inside conv_in with a
cryptic 'Expected 4D or 5D... but got [1,3,1,0,W]' error. This applies
regardless of extraction mode ('encode' already resizes down to
ref_resolution but never guarantees a floor; 'training' now resizes to
the same cap, also without a floor), so it's a separate, unconditional
safety net right before encode.
"""
import comfy.utils
h, w = image.shape[1], image.shape[2]
if h >= floor and w >= floor:
return image
scale = floor / min(h, w)
tw = max(floor, round(w * scale / 32) * 32)
th = max(floor, round(h * scale / 32) * 32)
samples = image[..., :3].movedim(-1, 1)
samples = comfy.utils.common_upscale(samples, tw, th, "lanczos", "disabled")
return samples.movedim(1, -1)
@@ -0,0 +1,110 @@
"""CPU/MPS preprocessing regressions, without model weights.
Run with ComfyUI's Python from this directory:
PYTORCH_ENABLE_MPS_FALLBACK=0 python -m unittest compatibility_regression_test -v
MPS tests skip when no Apple GPU is available. For a standalone checkout, add
your ComfyUI directory to PYTHONPATH so common.py can import comfy.utils.
"""
from pathlib import Path
import sys
import unittest
from unittest.mock import patch
sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
import torch
import torch.nn.functional as F
from common import sample_video_for_vae, snap_to_causal_grid
from core import _blur_latent, pool_latent
from gauntlet_harness import synchronize
class VideoGridTests(unittest.TestCase):
def test_h3_grid_and_image_special_case(self):
for n, expected in ((1,1),(2,1),(4,1),(5,5),(16,5),(17,5),(21,5),
(22,22),(23,22),(38,22),(39,39),(101,90),(124,124)):
with self.subTest(frames=n):
self.assertEqual(snap_to_causal_grid(n), expected)
def test_aligned_clips_preserve_every_frame(self):
for n in (1,5,22,39,124):
clip = torch.arange(n).reshape(n,1,1,1)
self.assertIs(sample_video_for_vae(clip), clip)
self.assertIs(sample_video_for_vae(clip, n), clip)
self.assertIs(sample_video_for_vae(clip, n+10), clip)
def test_frame_caps_keep_order_and_endpoints(self):
for n in (2,4,16,21,22,38,39,101):
clip = torch.arange(n).reshape(n,1,1,1)
for limit in (1,4,5,16,22,39,101):
with self.subTest(frames=n, limit=limit):
sampled = sample_video_for_vae(clip, limit).flatten()
self.assertLessEqual(len(sampled), min(n, limit))
self.assertEqual(sampled[0], 0)
if len(sampled) > 1:
self.assertEqual(len(sampled) % 17, 5)
self.assertEqual(sampled[-1], n-1)
self.assertTrue(torch.all(sampled[1:] > sampled[:-1]))
def test_empty_input_and_invalid_cap_fail_before_encoding(self):
with self.assertRaises(ValueError):
sample_video_for_vae(torch.empty(0,1,1,3))
with self.assertRaises(ValueError):
sample_video_for_vae(torch.zeros(5,1,1,3), 0)
class PoolingTests(unittest.TestCase):
def test_cpu_pool_matches_existing_fp32_arithmetic(self):
torch.manual_seed(19)
z = torch.randn(1,24,3,20,28).half()
expected = F.adaptive_avg_pool3d(z.float(), (2,8,8)).to(z.dtype)
actual = pool_latent(z,2,8,8)
self.assertTrue(torch.equal(actual.view(torch.uint8), expected.view(torch.uint8)))
self.assertIs(pool_latent(z,3,20,28), z)
@unittest.skipUnless(torch.backends.mps.is_available(), "requires MPS")
def test_mps_pool_and_visual_audio_blur_match_cpu_bytes(self):
torch.manual_seed(20)
for dtype in (torch.float32, torch.float16, torch.bfloat16):
for shape in ((1,24,3,20,28), (1,32,2,19)):
with self.subTest(dtype=dtype, shape=shape):
z = torch.randn(shape).to(dtype)
gpu = z.to('mps')
functions = [_blur_latent]
if len(shape) == 5:
functions.append(lambda x: pool_latent(x,2,8,8))
for function in functions:
expected = function(z)
actual = function(gpu)
self.assertEqual(actual.device, gpu.device)
self.assertEqual(actual.dtype, dtype)
self.assertTrue(torch.equal(actual.cpu().view(torch.uint8),
expected.view(torch.uint8)))
one = torch.ones(1,32,2,1,device='mps')
self.assertEqual(_blur_latent(one).device, one.device)
@unittest.skipUnless(torch.backends.mps.is_available(), "requires MPS")
def test_explicit_cpu_transfers_preserve_autograd(self):
for function in (_blur_latent, lambda x: pool_latent(x,2,8,8)):
cpu = torch.randn(1,2,3,20,28,requires_grad=True)
gpu = cpu.detach().to('mps').requires_grad_()
function(cpu).square().sum().backward()
function(gpu).square().sum().backward()
torch.testing.assert_close(gpu.grad.cpu(), cpu.grad, rtol=1e-5, atol=1e-6)
class TimingTests(unittest.TestCase):
def test_synchronizes_only_the_selected_backend(self):
with patch.object(torch.mps, 'synchronize') as mps, \
patch.object(torch.cuda, 'synchronize') as cuda:
synchronize('cpu')
mps.assert_not_called()
cuda.assert_not_called()
synchronize('mps')
mps.assert_called_once_with()
synchronize('cuda:1')
cuda.assert_called_once_with(torch.device('cuda:1'))
if __name__ == '__main__':
unittest.main()
+39
View File
@@ -0,0 +1,39 @@
"""Model-scoped RefMod injection before ComfyUI prepares each sampling run.
Works with Continuum and standard H3 samplers through the public model patcher
OUTER_SAMPLE hook. No process-global bundle or third-party function patch.
"""
from comfy.patcher_extension import WrappersMP
BRIDGE_KEY = "minimax_h3_refmod_bridge"
def attach(model, blocks, enabled=True):
model = model.clone()
model.remove_wrappers_with_key(WrappersMP.OUTER_SAMPLE, BRIDGE_KEY)
if not enabled or not blocks:
return model
refs = tuple(dict(b) for b in blocks)
def inject(executor, *args, **kwargs):
guider = executor.class_obj
original = guider.conds
guider.conds = {
key: [dict(entry, minimax_refs=list(entry.get("minimax_refs") or [])
+ [dict(ref) for ref in refs]) for entry in entries]
for key, entries in original.items()
}
try:
return executor(*args, **kwargs)
finally:
guider.conds = original
model.add_wrapper_with_key(WrappersMP.OUTER_SAMPLE, BRIDGE_KEY, inject)
return model
def disarm():
# Preserve the old output-node ID in saved workflows. There is no longer
# global state to disarm; enable=False on the model line removes the hook.
return "Bridge is model-scoped. Bypass it or set enable=False on the model line."
+752
View File
@@ -0,0 +1,752 @@
"""
core.py — "RefMod" adapter for MiniMax H3 (no training required)
The reference-video path of MiniMax H3 works by injecting *reference tokens*
into the packed sequence: the ref is VAE-encoded, patchified, projected and
placed on the 3D RoPE grid, then every DiT block attends to it. A video ref
is expensive because it contributes thousands of tokens.
Mode naming: ``encode`` (was ``full``) = straight full-res VAE encode;
``training`` (was ``pooled``) = compressed grid refined by gradient steps
(the only mode that "trains" — the refinement loop). The old names are
accepted everywhere as legacy aliases so saved workflows and mods keep
working.
A RefMod is the same reference, compressed to a handful of tokens:
* the ref is VAE-encoded to its full latent [1, 24, T, H, W],
* the latent is average-pooled to a tiny grid (default 4x4) and a few
latent frames, so the patchified token count drops to ~4-16,
* optionally the small latent is refined with a few gradient steps that
reconstruct the full latent (still no model weights involved).
At generation time the pooled latent is handed back to the model through the
native ``refs`` payload, so it flows through the exact same per-block
attention machinery as a full reference — the only thing that changes is the
token budget.
This is the MiniMax H3 analog of the LTX "Mod" concept tokens + per-block
injection: instead of a trained hypernetwork predicting AdaLN deltas, the
model's own cross-attention over the compressed ref tokens does the work.
"""
from __future__ import annotations
import json
import math
import os
import tempfile
from dataclasses import dataclass, field
from typing import Dict, List, Optional, Tuple
import torch
import torch.nn as nn
import torch.nn.functional as F
from safetensors import safe_open
from safetensors.torch import load_file, save_file
# Metadata key embedded in the safetensors header (keeps a mod in one file,
# so it can be shared/uploaded as a single artifact).
META_KEY = "refmod_meta"
# What a mod is standing in for. This is documentation + a routing hint, not
# a hard constraint on the tensors — but it drives two things: (1) the
# "identity in training mode" warning in nodes.py, and (2) the loader's
# prompt_hint output, which merges each loaded mod's concept_type +
# description into a string you can concat onto your CLIP Text Encode prompt.
# There's no CLIP-Vision / image-embedding injection point on H3's ref2va
# path to hang a "visual clue" off of — text is the only channel the model's
# text encoder reads, so that's the channel this uses.
CONCEPT_TYPES = (
"generic", # unspecified / mixed
"identity", # a specific person/character — use mode="encode" for this
"pose_motion", # a pose, dance, gesture, camera move
"clothing", # an outfit / garment, decoupled from who's wearing it
"background", # environment / set / location plate
"voice", "singing", "music_style", "sound_fx", "ambience",
"style", # look/grade/animation style, not a concrete subject
)
def _blur_latent(z: torch.Tensor, factor: int = 8) -> torch.Tensor:
"""Heavy spatial low-pass (downsample then upsample) used as the
weakening target for ``ref_block``'s ``strength`` below ``1.0``.
Originally this mixed toward ``torch.randn()`` noise, mirroring what the
docstring below describes as the model's own ``visual_cond_noise_aug``
mechanism. In practice, at the retention range people actually use this
dial for (0.15-0.7, i.e. large noise fractions), it produces a visible
woven/static texture instead of a gracefully weaker reference: a VAE
latent's channels are correlated, so raw iid noise isn't read as "less
reference," it's read as real-but-garbled content and gets rendered as
an actual (wrong) texture. ``visual_cond_noise_aug`` in the model's own
training is likely a small-magnitude robustness augmentation, not
something swept this far — so copying its math doesn't reproduce its
behavior at these strengths. A blurred copy stays on the latent
manifold (smooth, plausible) while still discarding detail as strength
drops, which is what "weaker reference" should actually look like.
"""
if z.device.type == "mps":
# MPS lacks adaptive_avg_pool3d (and some non-divisible pooling
# shapes). Keep this small preprocessing chain on CPU explicitly,
# then restore the caller's device; global MPS fallback is not needed.
return _blur_latent(z.cpu(), factor).to(z.device)
if z.dim() == 4:
b, c, stereo, t = z.shape
if t <= 1:
return z
flat = z.reshape(b * c * stereo, 1, t).float()
down = F.adaptive_avg_pool1d(flat, max(1, t // factor))
return F.interpolate(down, size=t, mode="linear", align_corners=False).reshape_as(z).to(z.dtype)
if z.dim() != 5:
raise ValueError(f"Unsupported RefMod latent shape: {tuple(z.shape)}")
t, h, w = z.shape[2], z.shape[3], z.shape[4]
sh, sw = max(1, h // factor), max(1, w // factor)
down = F.adaptive_avg_pool3d(z.float(), (t, sh, sw))
up = F.interpolate(down, size=(t, h, w), mode="trilinear", align_corners=False)
return up.to(z.dtype)
def read_refmod_meta(path_no_ext: str) -> Optional[Dict]:
"""
Read the metadata block from a mod/preset file without loading tensors.
Prefers the metadata embedded in the safetensors header; falls back to
the legacy sidecar ``{path}.json`` written by older versions.
"""
try:
with safe_open(path_no_ext + ".safetensors", framework="pt") as f:
meta = f.metadata()
if meta:
for key in (META_KEY, "audio_refmod_meta"):
if key in meta:
return json.loads(meta[key])
except Exception:
pass
jpath = path_no_ext + ".json"
if os.path.isfile(jpath):
try:
with open(jpath) as f:
return json.load(f)
except Exception:
return None
return None
# ═══════════════════════════════════════════════════════════════════════════
# Latent compression
# ═══════════════════════════════════════════════════════════════════════════
MODE_ALIASES = {"full": "encode", "pooled": "training",
"Full Reference": "encode", "Compressed Reference": "training"}
def normalize_mode(mode: str) -> str:
"""Map legacy mode names (``full``/``pooled``) to the current ones."""
return MODE_ALIASES.get(mode, mode)
def aspect_grid(pool_h: int, pool_w: int, aspect: float):
"""Even pool grid dims that match ``aspect`` (h/w) within the dial caps.
The DiT patches each 2x2 latent cell into one token and the rope grid is
area-normalized per axis, so a pooled grid that ignores the source aspect
squishes the subject: a portrait latent (tall person) forced into a square
16x16 grid is compressed 5x in height but only 3x in width, and comes out
"fat". The official ref2video node never pools — it feeds the model
aspect-correct latent dims, which is what the DiT was trained on.
The long edge follows the user's dial (``max(pool_h, pool_w)``) and the
short edge is derived from the source aspect, both rounded to even for the
2x2 patch. Square sources keep the exact dial value (16x16 stays 16x16).
"""
long_edge = max(pool_h, pool_w)
if aspect >= 1.0:
h, w = long_edge, long_edge / aspect
else:
w, h = long_edge, long_edge * aspect
h = max(2, round(h / 2) * 2)
w = max(2, round(w / 2) * 2)
return h, w
def pool_latent(
z: torch.Tensor,
latent_t: int,
latent_h: int,
latent_w: int,
) -> torch.Tensor:
"""
Average-pool a VAE latent ``[1, 24, T, H, W]`` down to a tiny grid.
``latent_h/latent_w`` are the *latent* grid dims (the DiT patches each 2x2
latent cell into one token), so a 4x4 latent = 2x2 = 4 tokens per frame.
Dims must be even (the DiT's 2x2 patch). Pooling runs in fp32; the result
keeps the input dtype.
"""
if z.shape[2] == latent_t and z.shape[3] == latent_h and z.shape[4] == latent_w:
return z
if latent_h % 2 != 0 or latent_w % 2 != 0:
raise ValueError(f"pool_h/pool_w must be even (got {latent_h}x{latent_w})")
# Normal ComfyUI intermediates are already on CPU. GPU-only workflows can
# supply MPS latents, where adaptive_avg_pool3d is not implemented.
work = z.cpu() if z.device.type == "mps" else z
pooled = F.adaptive_avg_pool3d(work.float(), (latent_t, latent_h, latent_w))
return pooled.to(device=z.device, dtype=z.dtype)
def optimize_latent(
z_small: torch.Tensor,
z_full: torch.Tensor,
steps: int = 150,
lr: float = 0.02,
device: Optional[torch.device] = None,
progress_every: int = 0,
) -> torch.Tensor:
"""
Model-free refinement of the compressed latent.
Optimizes the small latent so its trilinearly upsampled reconstruction
matches the full reference latent. This pulls the pooled representation
closer to what the DiT would see from the full ref, with nothing but the
tiny latent trainable (~1-2K params) and no diffusion model loaded.
``progress_every`` > 0 prints a ``[RefMod] identity <step>/<steps>`` line
every N steps so long pooled-mode refinement isn't silent (the default
``identity=500`` takes a while per ref on CPU/VRAM-bound setups).
Returns the refined latent detached, same shape/dtype as ``z_small``.
"""
if steps <= 0:
return z_small
device = device or z_full.device
# Node execution runs under ComfyUI's global inference mode, which would
# disable the autograd this refine loop needs. Re-enable it for this
# scope and clone the inputs to drop the inference flag.
with torch.inference_mode(False), torch.set_grad_enabled(True):
target = z_full.clone().float().to(device)
param = nn.Parameter(z_small.clone().float().to(device))
opt = torch.optim.Adam([param], lr=lr)
size = tuple(target.shape[2:])
for i in range(steps):
opt.zero_grad()
up = F.interpolate(param, size=size, mode="trilinear", align_corners=False)
loss = F.mse_loss(up, target)
loss.backward()
opt.step()
if progress_every and (i + 1) % progress_every == 0:
print(f"[RefMod] identity {i + 1}/{steps}")
# materialize inside the scope so the result is a normal tensor, not
# an inference-mode tensor (it gets stored in the mod and reused)
refined = param.detach().to(z_small.dtype)
return refined
def optimize_latent_multi(z_init, targets, steps=150, lr=0.02, device=None,
progress_every=0, strategy="grouped"):
"""Mean reconstruction objective with bounded activation memory.
grouped: average targets with identical shapes on CPU before refinement.
stream: transfer one target per gradient contribution.
resident: retain targets on the compute device, backward one at a time.
All strategies optimize the same mean MSE; grouping drops a constant term.
"""
if not targets or steps <= 0:
return z_init
if strategy not in ("grouped", "stream", "resident"):
raise ValueError("Unknown multi-reference optimization strategy.")
device = device or z_init.device
with torch.inference_mode(False), torch.set_grad_enabled(True):
count = len(targets)
if strategy == "grouped":
groups = {}
for target in targets:
key = tuple(target.shape)
value = target.detach().to(device="cpu", dtype=torch.float32).clone()
if key in groups:
groups[key][0].add_(value)
groups[key][1] += 1
else:
groups[key] = [value, 1]
prepared = [(total.div_(n), n / count) for total, n in groups.values()]
else:
target_device = device if strategy == "resident" else "cpu"
prepared = [(t.detach().to(device=target_device, dtype=torch.float32).clone(), 1 / count) for t in targets]
param = nn.Parameter(z_init.clone().float().to(device))
opt = torch.optim.Adam([param], lr=lr)
for i in range(steps):
opt.zero_grad()
for target, weight in prepared:
current = target.to(device)
up = F.interpolate(param, size=tuple(current.shape[2:]), mode="trilinear", align_corners=False)
(F.mse_loss(up, current) * weight).backward()
del up, current
opt.step()
if progress_every and (i + 1) % progress_every == 0:
print(f"[RefMod] merge {i + 1}/{steps}")
return param.detach().to(z_init.dtype)
# ═══════════════════════════════════════════════════════════════════════════
# Per-frame strength curve
# ═══════════════════════════════════════════════════════════════════════════
# Direction names describe where the *concept* shows up in the output, and
# the envelope is strong exactly where the concept lands: concept_at_start
# keeps the ref at full strength at the START of the video and releases it
# toward the end, concept_at_end fades it in toward the END. The old
# "decrease"/"increase" envelopes are kept as legacy aliases (decrease =
# concept_at_start, increase = concept_at_end) so pre-rename workflows keep
# their behavior. shape is how the envelope travels between its endpoints;
# value is the non-zero endpoint ("user input").
CURVE_DIRECTIONS = ("constant", "concept_at_start", "concept_at_middle",
"concept_at_end", "concept_at_ends")
CURVE_SHAPES = ("linear", "ease", "sigmoid", "tanh", "quadratic", "cubic",
"exponential", "stair", "elastic", "bump", "dip")
# old single-combo preset names -> (direction, shape, value) so workflows
# saved before the curve was split still resolve
_LEGACY_CURVES = {
"flat": ("constant", "linear", 1.0),
# keep each preset's historical envelope: fade_in = crescent (0 -> value),
# fade_out = decrescent (value -> 0), bump = mid peak
"fade_in": ("concept_at_end", "linear", 1.0),
"fade_out": ("concept_at_start", "linear", 1.0),
"bump": ("concept_at_end", "bump", 1.0),
"dip": ("constant", "dip", 1.0),
}
def _ease(shape: str, x: float) -> float:
"""Shape a progress ``x`` in [0, 1] per the easing name (may overshoot)."""
if shape == "linear":
return x
if shape == "ease": # smoothstep
return x * x * (3.0 - 2.0 * x)
if shape == "sigmoid": # logistic S-curve
return 1.0 / (1.0 + math.exp(-12.0 * (x - 0.5)))
if shape == "tanh": # sharper S-curve (steeper knee than sigmoid)
return 0.5 * (math.tanh(8.0 * (x - 0.5)) + 1.0)
if shape == "quadratic":
return x * x
if shape == "cubic":
return x * x * x
if shape == "exponential":
return 2.0 ** x - 1.0
if shape == "stair":
return min(1.0, math.floor(x * 4) / 3.0)
if shape == "elastic":
if x <= 0.0:
return 0.0
if x >= 1.0:
return 1.0
return 2.0 ** (-10.0 * x) * math.sin((x * 10.0 - 0.75) * (2.0 * math.pi / 3.0)) + 1.0
if shape == "bump":
return 1.0 - abs(2.0 * x - 1.0)
if shape == "dip":
return abs(2.0 * x - 1.0)
return x
def curve_strengths(spec, t: int) -> Optional[List[float]]:
"""Resolve a curve spec to ``t`` per-frame strength multipliers in [0, 1].
Kept as plain Python so the Apply node exposes the curve as normal
combo/float widgets instead of depending on ComfyUI's new Curve widget.
Accepts:
* a ``(direction, shape, value)`` tuple — direction is ``constant``
(one strength everywhere), ``concept_at_start`` (value -> 0, a
decrescent — the concept shows in the first half; legacy
``decrease``), ``concept_at_end`` (0 -> value, a crescent — the
concept shows in the second half; legacy ``increase``),
``concept_at_middle`` ([0..1..0] — concept peaks mid-timeline) or
``concept_at_ends`` ([1..0..1] — concept at both ends); shape is how
the envelope travels between its endpoints (see ``_ease``); value is
the non-zero endpoint (1.0 = full strength there);
* a legacy preset name (``flat``/``fade_in``/``fade_out``/``bump``/
``dip``) from before the split;
* a list of per-frame floats (len == t), used as-is;
* a list of ``(x, y)`` control points (x in [0, 1]), linearly
interpolated across the frames;
* any object with ``interp(x) -> float`` (legacy CurveInput saved in an
old workflow).
Returns None for flat/no curve so the caller keeps its single-strength
path unchanged (a ``constant`` curve at value 1.0 included).
"""
if t <= 1 or spec is None or spec == "":
return None
if isinstance(spec, str):
legacy = _LEGACY_CURVES.get(spec)
return curve_strengths(legacy, t) if legacy is not None else None
if (isinstance(spec, tuple) and len(spec) == 3
and isinstance(spec[0], str) and isinstance(spec[1], str)):
direction, shape, value = spec
value = float(value)
if direction == "constant":
# linear = flat at ``value`` (1.0 == no curve); the other shapes
# modulate around the constant, so constant + dip is a trough,
# constant + bump is a peak (both endpoints equal)
if shape == "linear":
return None if value >= 1.0 else [value] * t
p = [_ease(shape, i / (t - 1)) for i in range(t)]
return [max(0.0, min(1.0, value * y)) for y in p]
p = [_ease(shape, i / (t - 1)) for i in range(t)]
# legacy "decrease"/"increase" (pre-rename workflows) keep their
# original envelopes: decrease = strong at the start, increase =
# strong at the end. The concept_* names mean exactly what they say.
if direction in ("concept_at_end", "increase"):
return [max(0.0, min(1.0, value * y)) for y in p]
if direction in ("concept_at_start", "decrease"):
return [max(0.0, min(1.0, value * (1.0 - y))) for y in p]
if direction == "concept_at_middle": # [0..1..0]: concept peaks mid-timeline
return [max(0.0, min(1.0, value * (1.0 - abs(2.0 * y - 1.0)))) for y in p]
if direction == "concept_at_ends": # [1..0..1]: concept at both ends
return [max(0.0, min(1.0, value * abs(2.0 * y - 1.0))) for y in p]
return None # unknown direction: treat as flat
if isinstance(spec, (list, tuple)):
if len(spec) == t and all(isinstance(v, (int, float)) for v in spec):
return [float(v) for v in spec]
pts = [(float(x), float(y)) for x, y in spec
if isinstance(x, (int, float)) and isinstance(y, (int, float))]
if pts:
pts = sorted(pts, key=lambda p: p[0])
out = []
for i in range(t):
x = i * (1.0 / (t - 1))
if x <= pts[0][0]:
out.append(pts[0][1])
elif x >= pts[-1][0]:
out.append(pts[-1][1])
else:
for (x0, y0), (x1, y1) in zip(pts, pts[1:]):
if x0 <= x <= x1:
out.append(y0 + (y1 - y0) * (x - x0) / (x1 - x0))
break
return out
interp = getattr(spec, "interp", None)
if callable(interp):
return [float(interp(i / (t - 1))) for i in range(t)]
return None
def curve_value_at(spec, x: float) -> float:
"""Per-progress strength of a curve spec at one point ``x`` in [0, 1].
The continuous analog of ``curve_strengths`` — same direction/shape/value
math, but evaluated at a single progress value instead of discretized over
frames. The step-curve node uses it to re-mix the ref latents once per
denoising step, where ``x`` is the denoise progress (0 = first step,
1 = last step). Returns 1.0 (no weakening) for anything unrecognized.
"""
if spec is None or spec == "":
return 1.0
if isinstance(spec, str):
legacy = _LEGACY_CURVES.get(spec)
return curve_value_at(legacy, x) if legacy is not None else 1.0
if (isinstance(spec, tuple) and len(spec) == 3
and isinstance(spec[0], str) and isinstance(spec[1], str)):
direction, shape, value = spec
value = float(value)
if direction == "constant":
if shape == "linear":
return value
return max(0.0, min(1.0, value * _ease(shape, x)))
y = _ease(shape, x)
if direction in ("concept_at_end", "increase"):
return max(0.0, min(1.0, value * y))
if direction in ("concept_at_start", "decrease"):
return max(0.0, min(1.0, value * (1.0 - y)))
if direction == "concept_at_middle": # [0..1..0]: concept peaks mid-timeline
return max(0.0, min(1.0, value * (1.0 - abs(2.0 * y - 1.0))))
if direction == "concept_at_ends": # [1..0..1]: concept at both ends
return max(0.0, min(1.0, value * abs(2.0 * y - 1.0)))
return 1.0
# ═══════════════════════════════════════════════════════════════════════════
# Token budget cap (shared by the Extract node and extract_mod.py)
# ═══════════════════════════════════════════════════════════════════════════
def dedup_frame_indices(z: torch.Tensor, threshold: float = 0.02) -> List[int]:
"""Indices of latent frames kept by greedy temporal dedup.
Each frame is compared to the last *kept* frame: if the mean-abs
difference between them, normalized by the frames' own magnitude, is
below ``threshold`` the frame is dropped as a (near-)duplicate. Video
refs (a dance loop, a static shot, a talking head) contain long runs of
frames that differ only by codec noise — each one still costs a token
per spatial patch in every DiT block, so cutting them is the cheapest
way to honor a token cap. Stacked image refs survive: different
angles/expressions land well above the threshold.
"""
t = z.shape[2]
if t <= 1:
return list(range(t))
flat = z[0].float() # [24, T, H, W]
kept = [0]
prev = flat[:, 0]
for i in range(1, t):
cur = flat[:, i]
denom = (cur.abs().mean() + prev.abs().mean()) / 2 + 1e-6
diff = (cur - prev).abs().mean() / denom
if diff >= threshold:
kept.append(i)
prev = cur
return kept
def fit_token_budget(latent: torch.Tensor, budget: int, label: str) -> torch.Tensor:
"""Bring a stacked latent's injected token count under ``budget``.
Tokens = (H/2)*(W/2) per frame (the DiT's 2x2 patch). When the stack
exceeds the budget it is cut in two cheap, loss-ordered passes:
1. temporal dedup — drop latent frames that are (near-)identical to
the last kept frame (free: they carry no new reference info);
2. budget-fit resample — uniformly subsample the remaining frames down
to the largest count that fits the budget.
The spatial dims are never touched: resizing a latent's H/W changes the
rope grid and degrades the reference, so time is the only honest lever.
``label`` is the mod name for the console notes.
"""
if budget <= 0:
return latent
h, w = latent.shape[3], latent.shape[4]
per_frame = (h // 2) * (w // 2)
t = latent.shape[2]
if per_frame > budget:
raise ValueError(f"{label}: one frame costs {per_frame} tokens, above budget {budget}. Lower ref_resolution or pool size.")
if per_frame * t <= budget:
return latent
kept = dedup_frame_indices(latent)
if len(kept) < t:
print(f"[RefMod] {label}: over {budget}-token cap, "
f"dropped {t - len(kept)} near-duplicate frame(s) "
f"({t} -> {len(kept)})")
latent = latent[:, :, kept]
t = latent.shape[2]
if per_frame * t > budget:
fit_t = max(1, budget // per_frame)
if fit_t < t:
idx = torch.linspace(0, t - 1, fit_t).round().long()
latent = latent[:, :, idx]
print(f"[RefMod] {label}: still over {budget}-token cap, "
f"resampled {t} -> {fit_t} frame(s) to fit")
if per_frame * latent.shape[2] > budget:
print(f"[RefMod] {label}: one frame alone is {per_frame} tokens > "
f"{budget} — lower ref_resolution or pool_h/pool_w to meet "
f"the cap.")
return latent
# ═══════════════════════════════════════════════════════════════════════════
# H3RefMod — the saved artifact
# ═══════════════════════════════════════════════════════════════════════════
@dataclass
class H3RefMod:
"""
A compressed reference for MiniMax H3.
``latent`` is the VAE latent ``[1, 24, latent_t, latent_h, latent_w]`` — a
full-resolution encode (``mode="encode"``) or a pooled thumbnail
(``mode="training"``). ``kind`` is ``"image"`` (single latent frame) or
``"video"`` (a few frames), matching the native ref block kinds the
model's ``PackedLayout`` understands.
``mode="encode"`` stores the encode at the resolution the official ref2video
path uses, so the injected ref carries real identity detail;
``mode="training"`` stores a tiny average-pooled grid refined by gradient
steps (concept/motion, or identity at high pool sizes).
"""
name: str
kind: str
latent: torch.Tensor
latent_h: int = 4
latent_w: int = 4
latent_t: int = 1
mode: str = "training"
source: str = "" # "image" | "video" | "manual"
source_shape: str = "" # original latent dims as "TxHxW"
pool: str = "4x4x1" # pool_t x pool_h x pool_w
optimize_steps: int = 0
tags: List[str] = field(default_factory=list)
description: str = "" # optional text describing the concept (emitted by the loaders)
concept_type: str = "generic" # what this mod represents; see CONCEPT_TYPES above
config: Dict = field(default_factory=dict)
# Recommended Apply/Step-Curve settings baked in by the "Fix H3 RefMod
# Config" node: {"retention": float, "curve": [direction, shape, value],
# "step_curve": [direction, shape, value]}. The Apply / Step Curve
# nodes' ``override`` toggle reads it back so a concept ships with the
# settings that make it work — newbies never have to tune.
sample_rate: int = 32000
path: str = "" # path_no_ext the mod was loaded from / saved to (for re-saving)
bundle_index: int = -1 # index in the container at path; -1 for standalone refs
def __post_init__(self):
if self.kind == "audio":
if self.latent.ndim != 4 or tuple(self.latent.shape[:3]) != (1, 32, 2) or self.latent.shape[-1] < 1:
raise ValueError("H3 audio RefMod must contain [1,32,2,T] with T >= 1.")
self.latent_t = self.latent.shape[-1]
self.latent_h = self.latent_w = 0
return
if self.kind not in ("image", "video"):
raise ValueError(f"kind must be 'image' or 'video' (got {self.kind!r})")
if self.kind == "image":
self.latent_t = 1
# ── token budget ──────────────────────────────────────────────────
@property
def token_count(self) -> int:
"""Number of patchified tokens the mod injects into the packed sequence."""
if self.kind == "audio":
return 2 * self.latent_t
per_frame = (self.latent_h // 2) * (self.latent_w // 2)
return self.latent_t * per_frame
# ── native ref block ──────────────────────────────────────────────
def ref_block(self, strength: float = 1.0,
curve=None) -> Optional[Dict]:
"""
Build the ref block dict the model's ``PackedLayout`` / payload consumes.
The shape mirrors what the native ref2va nodes emit, so the pooled
latent rides the exact same path: ``cond_video_latents`` -> patchify ->
``video_patch_proj`` -> packed sequence with a 3D RoPE grid.
``strength`` weakens the ref by mixing the latent toward a heavily
blurred copy of itself (see ``_blur_latent``) rather than toward
zero or random noise: scaling toward zero pushes values out of the
normalized latent distribution (reads as grey output), and mixing
toward random noise reads as real-but-garbled content and decodes
as a wrong texture rather than "weaker" — a blurred copy stays
on-manifold while still discarding the specific detail that makes a
reference strong. ``strength <= 0`` drops the block entirely (no
tokens injected).
``curve`` (optional) is a per-frame strength spec for the ref's
latent timeline — a ``(direction, shape, value)`` tuple (e.g.
``("concept_at_end", "ease", 1.0)``), a legacy preset name, a list of
per-frame floats, ``(x, y)`` control points, or any object with
``interp(x)`` (see ``curve_strengths``). Each frame is mixed with
its own strength ``retention * curve(x)`` instead of one flat value,
so the ref can fade in/out across the video. A flat curve is
identical to passing ``strength`` alone.
"""
if strength <= 0.0:
return None
latent = self.latent
strengths = curve_strengths(curve, self.latent_t) if curve is not None else None
if strengths is not None:
weights = [max(0.0, min(1.0, strength * value)) for value in strengths]
if any(value < 1.0 for value in weights):
shape = (1, 1, 1, self.latent_t) if self.kind == "audio" else (1, 1, self.latent_t, 1, 1)
st = latent.new_tensor(weights).view(shape)
latent = st * latent + (1.0 - st) * _blur_latent(latent)
elif strength < 1.0:
latent = strength * latent + (1.0 - strength) * _blur_latent(latent)
if self.kind == "audio":
return {"kind": "audio", "ref_audio_t": self.latent_t, "audio_latent": latent}
block: Dict = {
"kind": self.kind,
"latent_h": self.latent_h,
"latent_w": self.latent_w,
"latent": latent,
}
if self.kind == "video":
block["latent_t"] = self.latent_t
block["ref_audio_t"] = 0
block["audio_latent"] = None
return block
# ── serialization ─────────────────────────────────────────────────
def metadata(self) -> dict:
meta = {
"name": self.name,
"kind": self.kind,
"latent_h": self.latent_h,
"latent_w": self.latent_w,
"latent_t": self.latent_t,
"mode": self.mode,
"source": self.source,
"source_shape": self.source_shape,
"pool": self.pool,
"optimize_steps": self.optimize_steps,
"tags": self.tags,
"description": self.description,
"concept_type": self.concept_type,
"_format_version": 4,
"sample_rate": self.sample_rate,
}
if self.config:
meta["refmod_config"] = json.dumps(self.config)
return meta
def save(self, path_no_ext: str) -> str:
"""Save standalone, or update this member without dropping its siblings."""
if self.bundle_index >= 0 and os.path.normcase(os.path.abspath(path_no_ext)) == os.path.normcase(os.path.abspath(self.path)):
from .bundle import update_member
return update_member(self)
os.makedirs(os.path.dirname(path_no_ext) or ".", exist_ok=True)
meta = self.metadata()
destination = path_no_ext + ".safetensors"
fd, temporary = tempfile.mkstemp(prefix=".refmod-", suffix=".tmp", dir=os.path.dirname(destination) or ".")
os.close(fd)
try:
save_file({"latent": self.latent.contiguous()}, temporary,
metadata={META_KEY: json.dumps(meta)})
os.replace(temporary, destination)
finally:
if os.path.exists(temporary):
os.unlink(temporary)
return destination
@classmethod
def load(cls, path_no_ext: str, device: str = "cpu") -> "H3RefMod":
"""Load from ``{path}.safetensors`` (metadata in header, or legacy .json)."""
meta = read_refmod_meta(path_no_ext)
if not isinstance(meta, dict):
raise ValueError(
f"{path_no_ext}.safetensors has no RefMod metadata "
f"(header key '{META_KEY}' or sidecar .json missing).")
if meta.get("kind") == "bundle":
raise ValueError("This is a RefMod bundle. Use the current RefMod Loader to select its components.")
# Clone releases the file mapping before a possible Windows overwrite.
latent = load_file(path_no_ext + ".safetensors", device=device)["latent"].clone()
return cls.from_metadata(meta, latent, path_no_ext)
@classmethod
def from_metadata(cls, meta, latent, path_no_ext="", bundle_index=-1):
raw_config = meta.get("refmod_config")
try:
config = json.loads(raw_config) if isinstance(raw_config, str) else {}
except ValueError:
config = {}
return cls(
name=meta.get("name", os.path.basename(path_no_ext)),
kind=meta.get("kind", "image"),
latent=latent,
latent_h=int(meta.get("latent_h", latent.shape[3] if latent.ndim == 5 else 0)),
latent_w=int(meta.get("latent_w", latent.shape[4] if latent.ndim == 5 else 0)),
latent_t=int(meta.get("latent_t", latent.shape[2] if latent.ndim == 5 else latent.shape[-1])),
mode=normalize_mode(meta.get("mode", "training")),
source=meta.get("source", ""),
source_shape=meta.get("source_shape", ""),
pool=meta.get("pool", ""),
optimize_steps=int(meta.get("optimize_steps", 0)),
tags=list(meta.get("tags", [])),
description=str(meta.get("description", "") or ""),
concept_type=str(meta.get("concept_type", "generic") or "generic"),
config=config if isinstance(config, dict) else {},
path=path_no_ext,
bundle_index=bundle_index,
sample_rate=int(meta.get("sample_rate", 32000)),
)
+205
View File
@@ -0,0 +1,205 @@
"""
debug_grid.py — 1024x1024 curve-graph visualization for Apply H3 RefMod.
Renders the per-frame strength envelope (direction/shape/value) across the
full canvas, with the "concept zone" shaded where the concept shows up. It
is both the Apply's optional debug IMAGE output and the saved graph-preset
PNG (graph embedded in the image's tEXt metadata).
Pure PIL; ``pil_to_tensor`` converts the result to a torch IMAGE tensor.
"""
from __future__ import annotations
import json
import os
from typing import Dict, Optional, Tuple
import numpy as np
import torch
from PIL import Image, ImageDraw, ImageFont, PngImagePlugin
from .core import CURVE_DIRECTIONS, CURVE_SHAPES, curve_strengths
SIZE = 1024
BG = (18, 20, 24)
TITLE = (232, 230, 227)
TEXT = (207, 203, 196)
DIM = (107, 111, 118)
GRID_LINE = (42, 46, 54)
ACCENT = (255, 158, 61)
ACCENT_FILL = (58, 34, 15)
ACCENT_DIM = (122, 69, 21)
_CURVE_SAMPLES = 160
# where the concept shows up in the output, per direction — the envelope is
# drawn on the same timeline, so it is strong exactly where the zone is
_ZONES = {
"concept_at_start": ((0.0, 1.0 / 3.0),),
"concept_at_middle": ((1.0 / 3.0, 2.0 / 3.0),),
"concept_at_end": ((2.0 / 3.0, 1.0),),
"concept_at_ends": ((0.0, 1.0 / 3.0), (2.0 / 3.0, 1.0)),
}
# ── fonts ────────────────────────────────────────────────────────────────────
_FONT_CACHE: Dict[Tuple[int, bool, bool], ImageFont.FreeTypeFont] = {}
def _font(size: int, bold: bool = False, mono: bool = False):
key = (size, bold, mono)
if key in _FONT_CACHE:
return _FONT_CACHE[key]
candidates = []
if os.name == "nt":
base = "C:/Windows/Fonts/"
name = "consola.ttf" if mono else ("arialbd.ttf" if bold else "arial.ttf")
candidates = [base + name]
else:
if mono:
candidates = [
"/usr/share/fonts/truetype/dejavu/DejaVuSansMono.ttf",
"/System/Library/Fonts/Menlo.ttc",
]
else:
candidates = [
"/usr/share/fonts/truetype/dejavu/DejaVuSans-Bold.ttf" if bold
else "/usr/share/fonts/truetype/dejavu/DejaVuSans.ttf",
"/System/Library/Fonts/Supplemental/Arial.ttf",
]
for path in candidates:
if os.path.isfile(path):
try:
f = ImageFont.truetype(path, size)
_FONT_CACHE[key] = f
return f
except OSError:
pass
try:
f = ImageFont.load_default(size=size)
except Exception:
f = ImageFont.load_default()
_FONT_CACHE[key] = f
return f
def _text_w(draw: ImageDraw.ImageDraw, text: str, font) -> int:
l, _t, r, _b = draw.textbbox((0, 0), text, font=font)
return r - l
# ── the curve graph ──────────────────────────────────────────────────────────
def render_debug_grid(spec, preset_name: str = "") -> Image.Image:
"""Render the 1024x1024 curve graph for an Apply configuration.
``spec`` is the resolved (direction, shape, value) curve tuple and
``preset_name`` the active graph preset (shown when one is loaded).
"""
img = Image.new("RGB", (SIZE, SIZE), BG)
draw = ImageDraw.Draw(img)
draw.text((32, 26), "curve graph", font=_font(28, bold=True), fill=TITLE)
if preset_name:
draw.text((SIZE - 32, 34), f"preset: {preset_name}",
font=_font(20), fill=DIM, anchor="rs")
x0, y0, x1, y1 = 90, 120, SIZE - 10, 940
direction = spec[0]
# concept zone shading (where the concept shows up in the output)
for start, end in _ZONES.get(direction, ()):
zx0 = x0 + start * (x1 - x0)
zx1 = x0 + end * (x1 - x0)
draw.rectangle([zx0, y0, zx1, y1], fill=ACCENT_FILL)
draw.text((zx0 + 8, y1 + 8), "concept zone", font=_font(20), fill=ACCENT_DIM)
# grid + axis labels
for f in (0.0, 0.25, 0.5, 0.75, 1.0):
gy = y1 - f * (y1 - y0)
draw.line([x0, gy, x1, gy], fill=GRID_LINE, width=2)
draw.text((16, gy - 12), f"{1.0 - f:.2f}", font=_font(20), fill=DIM)
for fx, label in ((0.0, "0"), (0.25, "0.25"), (0.5, "0.5"),
(0.75, "0.75"), (1.0, "1")):
gx = x0 + fx * (x1 - x0)
draw.line([gx, y0, gx, y1], fill=GRID_LINE, width=2)
draw.text((gx, y1 + 8), label, font=_font(20), fill=DIM, anchor="ma")
draw.text((x1 - 6, y1 + 8), "video →", font=_font(20), fill=DIM, anchor="ra")
# the envelope
strengths = curve_strengths(spec, _CURVE_SAMPLES)
if strengths is None:
strengths = [1.0] * _CURVE_SAMPLES
pts = [(x0 + i * (x1 - x0) / (_CURVE_SAMPLES - 1),
y1 - s * (y1 - y0)) for i, s in enumerate(strengths)]
draw.polygon([(x0, y1)] + pts + [(x1, y1)], fill=ACCENT_FILL)
draw.line(pts, fill=ACCENT, width=6)
# peak marker
peak = max(range(_CURVE_SAMPLES), key=lambda i: strengths[i])
px, py = pts[peak]
draw.line([px, y0, px, y1], fill=ACCENT_DIM, width=2)
draw.ellipse([px - 9, py - 9, px + 9, py + 9], fill=ACCENT)
# spec readout on a dark backdrop
lines = [f"direction: {direction}", f"shape: {spec[1]}",
f"value: {float(spec[2]):.2f}"]
font = _font(24, mono=True)
tw = max(_text_w(draw, ln, font) for ln in lines)
draw.rectangle([x0 + 18, y0 + 18, x0 + 18 + tw + 24, y0 + 18 + 3 * 34 + 12],
fill=(14, 17, 22))
for i, ln in enumerate(lines):
draw.text((x0 + 30, y0 + 28 + i * 34), ln, font=font, fill=TEXT)
return img
# ── graph presets (PNG metadata) ────────────────────────────────────────────
# The graph spec is embedded in the preset PNG's tEXt chunk, so a saved curve
# graph IS the preset: share the image, and the curve comes with it.
GRAPH_META_KEY = "graph"
def graph_pnginfo(spec):
"""PIL pnginfo carrying the (direction, shape, value) spec as tEXt."""
info = PngImagePlugin.PngInfo()
info.add_text(GRAPH_META_KEY, json.dumps({
"direction": spec[0], "shape": spec[1], "value": float(spec[2])}))
return info
def read_graph_meta(path: str) -> Optional[tuple]:
"""Read a (direction, shape, value) spec from a PNG's tEXt metadata.
Returns None for a missing file, a missing/empty chunk, or an invalid
spec (unknown direction/shape, bad value) so callers can fall back.
"""
try:
with Image.open(path) as img:
raw = img.text.get(GRAPH_META_KEY)
except Exception:
return None
if not raw:
return None
try:
data = json.loads(raw)
except ValueError:
return None
direction, shape = data.get("direction"), data.get("shape")
if direction not in CURVE_DIRECTIONS or shape not in CURVE_SHAPES:
return None
try:
value = float(data.get("value", 1.0))
except (TypeError, ValueError):
return None
return (direction, shape, value)
def pil_to_tensor(img: Image.Image) -> torch.Tensor:
"""Convert a rendered PIL graph to a [1, 1024, 1024, 3] float32 tensor."""
arr = np.asarray(img).copy()
return torch.from_numpy(arr).float().unsqueeze(0) / 255.0 # [1, 1024, 1024, 3]
Binary file not shown.

After

Width:  |  Height:  |  Size: 110 KiB

Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.

After

Width:  |  Height:  |  Size: 2.9 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 4.1 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 2.4 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 7.0 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 5.7 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 71 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 116 KiB

+271
View File
@@ -0,0 +1,271 @@
"""
extract_mod.py — standalone MiniMax H3 RefMod extractor (no training)
Turns reference images/videos into a ``.safetensors`` mod that the ComfyUI
Load H3 RefMods / Apply nodes can inject into MiniMax H3 generation. Only
the H3 video VAE is needed; the 29B DiT is never loaded.
Two modes:
* ``encode`` (default) — refs are resized to ``--resolution`` short edge (down
only) and encoded at that resolution, exactly like the official ref2video
node. This is what carries identity (a face, an outfit); files are
~0.2-1 MB per frame. (Old name: ``full``.)
* ``training`` — refs are resized to ``--resolution`` short edge too (the
latent is pooled to a tiny grid anyway, so encoding at native resolution
is wasted compute), then average-pooled to a small grid (4x4 by default
= 4 tokens per latent frame) and optionally refined with a few gradient
steps (still model-free; that refinement loop is the only "training" in
the whole pack). Nearly free to inject but only carries concept / motion,
not fine identity. (Old name: ``pooled``.)
Usage
-----
# full-res identity mod (recommended for characters)
python custom_nodes/ComfyUI-MiniMaxH3Mod/extract_mod.py \
--image char.png --vae path/to/h3_video_vae.safetensors \
--name my_character --mode encode --resolution 1024
# tiny concept/motion mod
python custom_nodes/ComfyUI-MiniMaxH3Mod/extract_mod.py \
--video dance.mp4 --vae path/to/h3_video_vae.safetensors \
--name dance --mode training --pool 4 --latent-frames 2
Multi-reference concept (each ref becomes its own latent frame):
python custom_nodes/ComfyUI-MiniMaxH3Mod/extract_mod.py \
--image face_a.png --image face_b.png --image full.png \
--video dance.mp4 --vae path/to/h3_video_vae.safetensors \
--name disney_char --mode encode --resolution 1024
Output goes to ``ComfyUI/models/refmods/`` by default (created on first run,
next to loras/ and unet/).
Run it with the same Python/venv that runs ComfyUI (it imports ``comfy``
from the install the script lives in).
"""
from __future__ import annotations
import argparse
import os
import sys
HERE = os.path.dirname(os.path.abspath(__file__))
if HERE not in sys.path:
sys.path.insert(0, HERE)
ROOT = os.path.abspath(os.path.join(HERE, "..", ".."))
if ROOT not in sys.path:
sys.path.insert(0, ROOT)
import torch
from common import (load_image_file, load_video_file, refmods_dir, mod_output_path,
resize_ref as _resize_ref, ensure_min_size, sample_video_for_vae)
from core import (CONCEPT_TYPES, H3RefMod, aspect_grid, fit_token_budget,
normalize_mode, optimize_latent, pool_latent)
MAX_VIDEO_FRAMES = 60 # uniform sample cap; temporal pooling averages anyway
# ═══════════════════════════════════════════════════════════════════════════
# Media loading (delegates to common.py: image/video -> [B, H, W, 3] float32)
# ═══════════════════════════════════════════════════════════════════════════
def _load_image(path: str, max_edge: int) -> torch.Tensor:
return load_image_file(path, max_edge=max_edge)
def _load_video(path: str, max_edge: int, max_frames: int = MAX_VIDEO_FRAMES) -> torch.Tensor:
return load_video_file(path, max_frames=max_frames, max_edge=max_edge)
# ═══════════════════════════════════════════════════════════════════════════
# Main
# ═══════════════════════════════════════════════════════════════════════════
def main():
ap = argparse.ArgumentParser(
description="Extract a MiniMax H3 RefMod from one or more images/videos "
"of the same concept (they are stacked into a video-kind mod)")
ap.add_argument("--image", action="append", default=[], metavar="PATH",
help="reference image path (repeatable for multi-view concepts)")
ap.add_argument("--video", action="append", default=[], metavar="PATH",
help="reference video path (repeatable)")
ap.add_argument("--vae", required=True, help="MiniMax H3 video VAE .safetensors")
ap.add_argument("--name", default=None, help="mod name (default: first source file stem)")
ap.add_argument("--subfolder", default="", help="optional subfolder inside models/refmods")
ap.add_argument("--output", default=None,
help="output dir (default: ComfyUI/models/refmods)")
ap.add_argument("--mode", choices=["training", "encode", "full", "pooled"],
default="training",
help="training = compressed grid refined by --identity (default, good balance); encode = straight full-res VAE encode (~1K tokens/img); full/pooled = old names, still accepted")
ap.add_argument("--resolution", type=int, default=1024,
help="both modes: target short edge in px, downscale only (default 1024; 2048 = max fidelity). training mode encodes at this size too — the main speed dial for it")
ap.add_argument("--pool", type=int, default=16, help="training mode: spatial latent grid (even, default 16)")
ap.add_argument("--pool-w", type=int, default=None, help="pool width (default: == --pool)")
ap.add_argument("--latent-frames", type=int, default=16,
help="frames kept per video ref (default 16; images use 1): training mode pools latent frames; encode mode samples source frames on H3's 17k+5 grid (16 selects 5, below 5 selects the first image)")
ap.add_argument("--identity", type=int, default=500,
help="training mode: how tightly the mod clings to the reference (gradient refinement steps; default 500, 0 = pure pooling)")
ap.add_argument("--multiplier", type=int, default=1,
help="data multiplier: repeat the extracted ref N times along time so a short video/GIF isn't drowned out by the main video's tokens (default 1 = no repeat)")
ap.add_argument("--max-tokens", type=int, default=5120,
help="hard cap on the total injected tokens (0 = off; default 5120): drops near-duplicate latent frames first, then resamples the rest to fit")
ap.add_argument("--max-edge", type=int, default=1536,
help="resize source so the longest edge is <= this before encoding (default 1536)")
ap.add_argument("--max-frames", type=int, default=60,
help="max video frames sampled (uniform) before encoding (default 60; lower for CPU)")
ap.add_argument("--description", default="",
help="optional text describing the concept (e.g. 'a ginger woman with messy hair', "
"'an animation style'). Stored in the mod and emitted by the loaders so it "
"can be merged into the prompt without retyping it.")
ap.add_argument("--concept-type", choices=list(CONCEPT_TYPES), default="generic",
help="what this mod represents (default: generic). 'identity' in --mode training "
"prints a warning: pooling is lossy in exactly the way that destroys faces — "
"use --mode encode for people.")
ap.add_argument("--device", default="auto", help="auto / cuda / mps / cpu (VAE device)")
args = ap.parse_args()
args.mode = normalize_mode(args.mode) # accept legacy 'full'/'pooled'
if not args.image and not args.video:
ap.error("provide at least one --image or --video")
if args.concept_type == "identity" and args.mode == "training" and args.pool < 16:
print(
f"[extract] warning: --concept-type identity with --mode training at "
f"--pool {args.pool} — pooling averages away exactly the detail that "
f"carries a face. Use --mode encode for people, or at least --pool 32."
)
import comfy.model_management
import comfy.sd
import comfy.utils
device = comfy.model_management.get_torch_device() if args.device == "auto" \
else torch.device(args.device)
# ── load VAE ─────────────────────────────────────────────────────
print(f"[extract] loading VAE {args.vae} (device={args.device})")
sd, metadata = comfy.utils.load_torch_file(args.vae, return_metadata=True)
vae = comfy.sd.VAE(sd=sd, metadata=metadata, device=device)
vae.throw_exception_if_invalid()
# ── encode each source (full-res or pooled), then stack along time ──
pool_w = args.pool_w or args.pool
# in encode mode --resolution is the target short edge, so keep the
# load-time longest-edge cap from interfering with it
# both modes resize to --resolution before encode now, so the load-time
# longest-edge cap just needs to stay ahead of that target
load_max_edge = max(args.resolution * 2, args.max_edge)
# full-res refs must share one spatial canvas so the stacked latent has a
# single H/W: anchor on the first source, cover-crop the rest to it
canvas = None
if args.mode == "encode" and len(args.image) + len(args.video) > 1:
if args.image:
first = _load_image(args.image[0], load_max_edge)
else:
first = _load_video(args.video[0], load_max_edge, args.max_frames)
h, w = first.shape[1], first.shape[2]
scale = min(1.0, args.resolution / min(h, w))
canvas = (max(32, round(w * scale / 32) * 32),
max(32, round(h * scale / 32) * 32))
# training mode: anchor the pool grid to the first source's aspect so a
# portrait person isn't squished into a square grid ("fat" distortion).
pool_grid = None
if args.mode == "training":
if args.image:
first = _load_image(args.image[0], load_max_edge)
else:
first = _load_video(args.video[0], load_max_edge, args.max_frames)
h0, w0 = first.shape[1], first.shape[2]
pool_grid = aspect_grid(args.pool, pool_w, h0 / w0)
if pool_grid != (args.pool, pool_w):
print(f"[extract] pooled grid {args.pool}x{pool_w} -> "
f"{pool_grid[0]}x{pool_grid[1]} to match source aspect "
f"{w0}x{h0} (avoids squishing the subject wide)")
frames = []
shapes = []
n_img = n_vid = 0
for path in args.image:
src = _load_image(path, load_max_edge)
src = ensure_min_size(_resize_ref(src, args.resolution, canvas))
print(f"[extract] image {path}: {tuple(src.shape)} (mode={args.mode})")
with torch.no_grad():
z = vae.encode(src.to(device)).float().cpu()
if args.mode == "encode":
pooled = z.to(torch.float16)
else:
gh, gw = pool_grid if pool_grid is not None else (args.pool, pool_w)
pooled = pool_latent(z, 1, gh, gw).to(torch.float16)
if args.identity > 0:
pooled = optimize_latent(pooled, z, steps=args.identity, device=device,
progress_every=100)
frames.append(pooled)
shapes.append(f"{z.shape[2]}x{z.shape[3]}x{z.shape[4]}")
n_img += 1
for path in args.video:
src = _load_video(path, load_max_edge, args.max_frames)
src = sample_video_for_vae(src, args.latent_frames if args.mode == "encode" else None)
src = ensure_min_size(_resize_ref(src, args.resolution, canvas))
print(f"[extract] video {path}: {tuple(src.shape)} (mode={args.mode})")
with torch.no_grad():
z = vae.encode(src.to(device)).float().cpu()
if args.mode == "encode":
pooled = z.to(torch.float16)
else:
pool_t = min(args.latent_frames, z.shape[2])
gh, gw = pool_grid if pool_grid is not None else (args.pool, pool_w)
pooled = pool_latent(z, pool_t, gh, gw).to(torch.float16)
if args.identity > 0:
pooled = optimize_latent(pooled, z, steps=args.identity, device=device,
progress_every=100)
frames.append(pooled)
shapes.append(f"{z.shape[2]}x{z.shape[3]}x{z.shape[4]}")
n_vid += 1
if args.mode == "encode" and args.identity > 0:
print(f"[extract] warning: --identity only applies to training mode — "
f"encode mode stores the actual encode, so --identity {args.identity} "
f"was ignored.")
latent = torch.cat(frames, dim=2)
if args.multiplier > 1:
latent = latent.repeat(1, 1, args.multiplier, 1, 1)
name = args.name or (os.path.splitext(os.path.basename(args.image[0]))[0]
if args.image else os.path.splitext(os.path.basename(args.video[0]))[0])
if args.max_tokens > 0:
latent = fit_token_budget(latent, args.max_tokens, name)
total_t = latent.shape[2]
kind = "video" if total_t > 1 else "image"
out_dir = args.output or refmods_dir()
px_w, px_h = latent.shape[4] * 16, latent.shape[3] * 16
mod = H3RefMod(
name=name,
kind=kind,
latent=latent,
latent_h=latent.shape[3],
latent_w=latent.shape[4],
latent_t=total_t,
mode=args.mode,
source="stack" if len(frames) > 1 else ("video" if n_vid else "image"),
source_shape=" +".join(shapes),
pool=f"full-res {px_w}x{px_h}px (short-edge cap {args.resolution}px)" if args.mode == "encode" else f"{total_t}x{gh}x{gw}",
optimize_steps=args.identity if args.mode == "training" else 0,
tags=[f"{n_img} img, {n_vid} vid"] + ([f"x{args.multiplier} repeat"] if args.multiplier > 1 else []),
description=args.description.strip(),
concept_type=args.concept_type,
)
mod.path = os.path.join(out_dir, name) if args.output else mod_output_path(name, args.subfolder) # so Fix H3 RefMod Config can re-save in place
path = mod.save(mod.path)
mb = latent.numel() * latent.element_size() / 1024 / 1024
print(f"[extract] saved {kind} mod '{name}' "
f"({mod.token_count} tokens, {mb:.2f} MB) -> {path}")
print(f"[extract] load it in ComfyUI with the Load H3 RefMods node "
f"(dropdown '{name}', strength 1.0 = full ref), then chain "
f"Apply H3 RefMod (Cond) into your sampling.")
if __name__ == "__main__":
main()
+62
View File
@@ -0,0 +1,62 @@
"""Benchmark the three production multi-ref strategies; reject parity failures.
Run: python gauntlet_harness.py --device cuda --refs 4 --frames 8 --edge 64
Run regressions: python -m unittest discover -s tests -v
"""
import argparse
import gc
import json
import time
import sys
from pathlib import Path
import torch
sys.path.insert(0, str(Path(__file__).resolve().parent))
from core import optimize_latent_multi
def synchronize(device):
device = torch.device(device)
if device.type == "cuda":
torch.cuda.synchronize(device)
elif device.type == "mps":
torch.mps.synchronize()
def run():
parser = argparse.ArgumentParser()
parser.add_argument("--device", default="cuda" if torch.cuda.is_available() else "cpu")
parser.add_argument("--refs", type=int, default=4)
parser.add_argument("--frames", type=int, default=8)
parser.add_argument("--edge", type=int, default=64)
parser.add_argument("--steps", type=int, default=20)
parser.add_argument("--output", default="tests/gauntlet_result.json")
args = parser.parse_args()
device = torch.device(args.device)
torch.manual_seed(123)
initial = torch.randn(1,24,2,8,8)
targets = [torch.randn(1,24,args.frames,args.edge,args.edge) for _ in range(args.refs)]
# Warm up optimizer and device initialization outside the measurements.
optimize_latent_multi(initial, targets[:1], steps=1, device=args.device, strategy="stream")
results, baseline = [], None
for strategy in ("resident", "stream", "grouped"):
gc.collect()
if device.type == "cuda":
torch.cuda.empty_cache()
torch.cuda.reset_peak_memory_stats(device)
synchronize(device)
start = time.perf_counter()
value = optimize_latent_multi(initial, targets, steps=args.steps, device=args.device, strategy=strategy)
synchronize(device)
elapsed = time.perf_counter() - start
peak = torch.cuda.max_memory_allocated(device)/1024**2 if device.type == "cuda" else None
value = value.cpu()
if baseline is None: baseline = value
error = (value-baseline).abs().max().item()
torch.testing.assert_close(value,baseline,rtol=1e-4,atol=1e-5)
results.append({"strategy":strategy,"seconds":elapsed,"peak_cuda_mib":peak,"max_error":error,"parity":True})
del value
Path(args.output).write_text(json.dumps({"parameters":vars(args),"results":results},indent=2),encoding="utf-8")
print(json.dumps(results,indent=2))
if __name__ == "__main__":
run()
+209
View File
@@ -0,0 +1,209 @@
"""Issue #21-23 regressions. Run with ComfyUI's Python; no model weights needed.
python issue_regression_test.py
"""
import asyncio
import contextlib
import importlib
import io
import json
from pathlib import Path
import sys
import tempfile
import types
import unittest
from unittest.mock import patch
ROOT = Path(__file__).resolve().parent
sys.path.insert(0, str(ROOT.parents[1]))
from comfy.cli_args import args
args.cpu = True
import folder_paths
import torch
with tempfile.TemporaryDirectory() as import_dir:
with patch.object(folder_paths, "models_dir", import_dir):
N = importlib.import_module(f"custom_nodes.{ROOT.name}.nodes")
import execution
import nodes as comfy_nodes
COMMON = importlib.import_module(N.__package__ + ".common")
class IssueRegressions(unittest.TestCase):
def setUp(self):
temp = tempfile.TemporaryDirectory()
self.addCleanup(temp.cleanup)
self.root = Path(temp.name) / "refmods"
self.root.mkdir()
N._MOD_CACHE.clear()
N._MOD_CACHE_STAMPS.clear()
self.addCleanup(N._MOD_CACHE.clear)
output = contextlib.redirect_stdout(io.StringIO())
output.__enter__()
self.addCleanup(output.__exit__, None, None, None)
def test_creators_are_queue_outputs_without_save_node(self):
for cls in (N.MiniMaxH3RefModExtract, N.MiniMaxH3RefModMasterExtract):
cls.INPUT_TYPES() # initialize the V3 compatibility attributes
schema = cls.define_schema()
self.assertTrue(schema.is_output_node)
self.assertEqual(len(schema.outputs), 2)
inputs = {}
for key, (kind, options) in cls.INPUT_TYPES()["required"].items():
if "default" in options:
inputs[key] = options["default"]
prompt = {"create": {"class_type": schema.node_id, "inputs": inputs}}
with patch.dict(comfy_nodes.NODE_CLASS_MAPPINGS, {schema.node_id: cls}):
result = asyncio.run(execution.validate_prompt("creator", prompt, None))
self.assertTrue(result[0], result)
self.assertIn("create", result[2])
def test_create_details_report_save_and_no_save(self):
vae = types.SimpleNamespace(encode=lambda pixels: torch.ones(1,24,1,4,4))
for save in (False, True):
with patch.object(COMMON, "refmods_dir", return_value=str(self.root)):
result = N.MiniMaxH3RefModExtract.execute(
"details", mode="encode", refs_image={"ref_image_0": torch.zeros(1,64,64,3)},
vae=vae, save=save)
details = json.loads(result[1])
self.assertEqual(len(details["saved_paths"]), int(save))
self.assertEqual(details["tokens"], result[0][0][0].token_count)
self.assertEqual(result.ui.as_dict()["text"], (result[1],))
def test_mask_list_keeps_sizes_and_maps_each_reference(self):
masks = [torch.zeros(1,64,96), torch.ones(1,96,64)]
collected = N.MiniMaxH3RefModMaskList().collect(masks)[0]
self.assertEqual([tuple(m.shape) for m in collected], [(1,64,96),(1,96,64)])
refs = {"ref_image_0": torch.zeros(1,64,96,3), "ref_image_1": torch.zeros(1,96,64,3)}
seen = []
def mask_latent(z, mask_px, *args, **kwargs):
seen.append(mask_px.clone())
return z
vae = types.SimpleNamespace(encode=lambda pixels: torch.ones(1,24,1,4,4))
for cls in (N.MiniMaxH3RefModExtract, N.MiniMaxH3RefModMasterExtract):
seen.clear()
with patch.object(N, "_mask_latent", side_effect=mask_latent):
cls.execute("masked", mode="training", identity=0, refs_image=refs,
vae=vae, mask_list=collected, save=False)
self.assertEqual(len(seen), 2)
self.assertEqual([tuple(m.shape) for m in seen], [(1,320,480),(1,480,320)])
self.assertEqual([m.mean().item() for m in seen], [0,1])
def test_mask_list_consumed_once_by_comfy_execution(self):
masks = [torch.zeros(1,64,96), torch.ones(1,96,64)]
node = N.MiniMaxH3RefModMaskList()
result = asyncio.run(execution._async_map_node_over_list(
"masks", "collect", node, {"masks": masks}, node.FUNCTION))
self.assertEqual(len(result), 1)
self.assertEqual(len(result[0][0]), 2)
def test_mixed_aspect_masks_follow_reference_resize_and_crop(self):
masks = [torch.linspace(0,1,640).expand(320,640).unsqueeze(0),
torch.linspace(0,1,640).view(640,1).expand(640,320).unsqueeze(0)]
refs = {f"ref_image_{i}": mask.unsqueeze(-1).expand(-1,-1,-1,3)
for i, mask in enumerate(masks)}
pixels_seen, masks_seen = [], []
def encode(pixels):
pixels_seen.append(pixels[...,0].clone())
return torch.ones(1,24,1,pixels.shape[1]//16,pixels.shape[2]//16)
def apply_mask(z, mask_px, *args, **kwargs):
masks_seen.append(mask_px.clone())
return z
for mode in ("training", "encode"):
pixels_seen.clear()
masks_seen.clear()
with patch.object(N, "_mask_latent", side_effect=apply_mask):
N.MiniMaxH3RefModExtract.execute(
"alignment", mode=mode, identity=0, refs_image=refs,
mask_list=masks, vae=types.SimpleNamespace(encode=encode), save=False)
for pixels, mask in zip(pixels_seen, masks_seen):
# Lanczos/PIL images and float bilinear masks differ by up to
# two uint8 levels from quantization and interpolation kernels.
torch.testing.assert_close(mask, pixels, atol=2/255, rtol=0)
def test_mask_broadcast_batch_and_mapping_errors(self):
vae = types.SimpleNamespace(encode=lambda pixels: torch.ones(1,24,1,4,4))
args = dict(name="masks", mode="encode", vae=vae, save=False,
refs_image={"ref_image_0": torch.zeros(1,64,64,3),
"ref_image_1": torch.zeros(1,64,64,3)})
for kwargs in ({"mask": torch.ones(1,64,64)},
{"mask": torch.ones(2,64,64)},
{"mask_list": [torch.ones(1,64,64)]}):
self.assertEqual(len(N.MiniMaxH3RefModExtract.execute(**args, **kwargs)[0]), 1)
with patch.object(vae, "encode", side_effect=AssertionError("must validate before encoding")):
with self.assertRaisesRegex(ValueError, "3 entries"):
N.MiniMaxH3RefModExtract.execute(**args, mask_list=[torch.ones(3,64,64)])
with self.assertRaisesRegex(ValueError, "not both"):
N.MiniMaxH3RefModExtract.execute(**args, mask=torch.ones(1,64,64),
mask_list=[torch.ones(1,64,64)])
def test_encode_samples_causal_count_without_losing_clip_endpoint(self):
clip = torch.arange(101).reshape(101,1,1,1).expand(-1,64,64,3).float() / 100
seen = []
def encode(pixels):
seen.append(pixels[:,0,0,0].clone())
n = pixels.shape[0]
return torch.ones(1,24,1 if n == 1 else (n-5)//17*5+2,4,4)
for limit, expected in ((16,5),(17,5),(22,22),(39,39),(1,1),(4,1),(101,90)):
N.MiniMaxH3RefModExtract.execute(
"motion", mode="encode", refs_video={"ref_video_0":clip},
vae=types.SimpleNamespace(encode=encode), latent_frames=limit, save=False)
self.assertEqual(len(seen[-1]), expected)
self.assertEqual(seen[-1][0].item(), 0)
if expected > 1:
self.assertEqual(seen[-1][-1].item(), 1)
def test_valid_22_frame_reference_reaches_vae_without_trimming(self):
clip = torch.arange(22).reshape(22,1,1,1).expand(-1,32,32,3).float() / 21
seen = []
vae = types.SimpleNamespace(encode=lambda pixels:
seen.append(pixels.clone()) or torch.ones(1,24,7,4,4))
# Isolate temporal policy from image-resize quantization.
with patch.object(N, '_resize_ref', side_effect=lambda src,*a:src), \
patch.object(N, '_ensure_min_size', side_effect=lambda src:src):
for mode in ('encode', 'training'):
N.MiniMaxH3RefModExtract.execute(
'aligned', mode=mode, refs_video={'ref_video_0':clip},
vae=vae, latent_frames=22, identity=0, save=False)
self.assertTrue(torch.equal(seen[-1], clip))
def test_motion_only_realigns_differences_and_keeps_last_motion(self):
clip = torch.linspace(0,1,23).square().reshape(23,1,1,1).expand(-1,32,32,3)
seen = []
vae = types.SimpleNamespace(encode=lambda pixels:
seen.append(pixels.clone()) or torch.ones(1,24,7,4,4))
with patch.object(N, '_resize_ref', side_effect=lambda src,*a:src), \
patch.object(N, '_ensure_min_size', side_effect=lambda src:src):
N.MiniMaxH3RefModExtract.execute(
'motion', mode='training', refs_video={'ref_video_0':clip},
vae=vae, latent_frames=2, identity=0, motion_only=True, save=False)
expected = (clip[1:]-clip[:-1]).abs()
expected /= expected.max()
self.assertTrue(torch.equal(seen[-1], expected))
def test_cli_samples_the_same_frames_as_the_node_helper(self):
sys.path.insert(0, str(ROOT))
self.addCleanup(sys.path.remove, str(ROOT))
cli = importlib.import_module('extract_mod')
import comfy.sd
import comfy.utils
clip = torch.arange(101).reshape(101,1,1,1).expand(-1,32,32,3).float()/100
seen = []
vae = types.SimpleNamespace(throw_exception_if_invalid=lambda:None,
encode=lambda pixels:seen.append(pixels.clone()) or torch.ones(1,24,7,4,4))
with patch.object(cli, '_load_video', return_value=clip), \
patch.object(cli, '_resize_ref', side_effect=lambda src,*a:src), \
patch.object(cli, 'ensure_min_size', side_effect=lambda src:src), \
patch.object(comfy.sd, 'VAE', return_value=vae), \
patch.object(comfy.utils, 'load_torch_file', return_value=({},{})):
for mode, limit in (('encode',16),('encode',22),('training',2)):
argv = ['extract_mod.py','--video','test.mp4','--vae','stub.safetensors',
'--device','cpu','--mode',mode,'--latent-frames',str(limit),
'--identity','0','--output',str(self.root),'--name','cli']
with patch.object(sys, 'argv', argv):
cli.main()
expected = COMMON.sample_video_for_vae(clip,limit if mode=='encode' else None)
self.assertTrue(torch.equal(seen[-1], expected))
if __name__ == "__main__":
unittest.main()
+47
View File
@@ -0,0 +1,47 @@
"""Metadata-only library endpoint; no tensor allocation or VAE loading."""
from aiohttp import web
from .bundle import members
def library_entries(names, find_path, read_meta):
entries = []
for name in names():
try:
path = find_path(name)
meta = read_meta(path)
except (OSError, ValueError):
continue
if not isinstance(meta, dict):
continue
kind = meta.get("kind")
if kind == "bundle":
try:
refs = members(meta)
tokens = sum(int(ref["latent_t"]) * (2 if ref["kind"] == "audio" else
(int(ref["latent_h"]) // 2) * (int(ref["latent_w"]) // 2)) for ref in refs)
except (ValueError, KeyError, TypeError):
continue
entries.append({"name": name, "kind": "bundle", "path": path + ".safetensors",
"concept": "bundle", "description": "; ".join(
f"{ref.get('name', '')} ({ref['kind']})" for ref in refs),
"tokens": tokens, "shape": None, "config": ""})
continue
try:
t = int(meta.get("latent_t", 0))
h, w = int(meta.get("latent_h", 0)), int(meta.get("latent_w", 0))
except (TypeError, ValueError):
continue
tokens = t * 2 if kind == "audio" else t * (h // 2) * (w // 2)
entries.append({"name": name, "kind": kind, "path": path + ".safetensors",
"concept": meta.get("concept_type", "generic"),
"description": meta.get("description", ""),
"tokens": tokens or None, "shape": [t, h, w],
"config": meta.get("refmod_config", "")})
return entries
def register_routes(server, names, find_path, read_meta):
@server.routes.get("/refmods/library")
async def library(request):
return web.json_response(library_entries(names, find_path, read_meta))
Binary file not shown.
Binary file not shown.
File diff suppressed because it is too large Load Diff
+102
View File
@@ -0,0 +1,102 @@
"""Present saved RefMod references to H3's native text/vision encoder."""
import math
import inspect
from comfy.text_encoders.minimax import MiniMaxH3Tokenizer
from comfy.ldm.minimax.vae import MiniMaxH3VideoVAE
def reference_map(mods):
counters = {"image": 0, "video": 0, "audio": 0}
labels = {"image": "Picture", "video": "Video", "audio": "Audio"}
mapping = []
for mod, strength in mods:
if not math.isfinite(strength) or not 0 <= strength <= 1:
raise ValueError("RefMod strength must be between 0 and 1.")
if strength == 0:
continue
counters[mod.kind] += 1
mapping.append(f"<{labels[mod.kind]} {counters[mod.kind]}> = {mod.name}")
return "\n".join(mapping) or "No active RefMods."
class MiniMaxH3RefModTextEncode:
@classmethod
def INPUT_TYPES(cls):
return {"required": {
"clip": ("CLIP",),
"mods": ("H3_REF_MODS",),
"prompt": ("STRING", {"multiline": True, "dynamicPrompts": True}),
"reference_fps": ("FLOAT", {"default": 24.0, "min": 1.0, "max": 120.0,
"tooltip": "Playback FPS of the reconstructed video, used for Qwen timestamps. Saved compressed or stacked refs do not preserve original timing."}),
"max_total_tokens": ("INT", {"default": 0, "min": 0, "max": 2147483647}),
}, "optional": {
"vae": ("VAE", {"tooltip": "H3 video VAE: reconstructs visual refs for Qwen. Not needed for audio-only bundles."}),
}}
RETURN_TYPES = ("CONDITIONING", "STRING")
RETURN_NAMES = ("conditioning", "reference_map")
FUNCTION = "encode"
CATEGORY = "MiniMax-H3/mod"
DESCRIPTION = "Encode the prompt together with numbered RefMod references. Already attaches refs; connect directly to the sampler, without applying the same mods again."
def encode(self, clip, mods, prompt, reference_fps=24.0, max_total_tokens=0, vae=None):
from .nodes import _check_token_budget
native = isinstance(clip.tokenizer, MiniMaxH3Tokenizer)
# Projected encoders retain their smaller model's tokenizer, but expose
# H3 reference presentation on the CLIP wrapper itself (e.g. ClipProj).
if not native and "minimax_ref_items" not in inspect.signature(clip.tokenize).parameters:
raise ValueError("Connect an H3 CLIP or a projected CLIP supporting minimax_ref_items (such as current ClipProj).")
if not math.isfinite(reference_fps) or not 1 <= reference_fps <= 120:
raise ValueError("reference_fps must be between 1 and 120.")
# Keep zero-strength slots out of both the presentation and DiT payload.
mapping = reference_map(mods)
active = [(mod, strength) for mod, strength in mods if strength > 0]
_check_token_budget(active, max_total_tokens)
if any(mod.kind != "audio" for mod, _ in active) and vae is None:
raise ValueError("Connect the H3 video VAE to present visual RefMods to CLIP.")
if any(mod.kind != "audio" for mod, _ in active) and not isinstance(vae.first_stage_model, MiniMaxH3VideoVAE):
raise ValueError("Visual RefMods require the MiniMax H3 video VAE.")
items, blocks = [], []
for mod, strength in active:
block = mod.ref_block(strength)
block["refmod"] = True
kind = block["kind"]
item = {"type": kind}
if kind != "audio":
# Decode the same weakened latent that the DiT receives. ComfyUI
# owns device placement and its decode OOM/tiled fallback.
pixels = vae.decode(block["latent"])
# ComfyUI video VAEs return BTHWC; older wrappers may already
# expose THWC. Each RefMod is one video, never a batch of videos.
if pixels.ndim == 5 and pixels.shape[0] == 1:
pixels = pixels[0]
if pixels.ndim != 4 or pixels.shape[-1] != 3 or pixels.shape[0] < 1:
raise ValueError(f"H3 video VAE must decode to [1, frames, height, width, 3] or [frames, height, width, 3]; got {tuple(pixels.shape)}.")
if kind == "image":
item["data"] = pixels[:1].cpu().clone()
else:
# Native H3 presents video at 2 fps. Index by timestamps so
# non-integer frame rates do not accumulate rounding drift.
times = [i / 2 for i in range(math.ceil(pixels.shape[0] * 2 / reference_fps))]
indices = [min(round(t * reference_fps), pixels.shape[0] - 1) for t in times]
item["data"] = pixels[indices].cpu()
item["timestamps"] = times
del pixels
items.append(item)
blocks.append(block)
tokens = clip.tokenize(prompt, minimax_ref_items=items)
conditioning = clip.encode_from_tokens_scheduled(tokens)
out = []
for embedding, metadata in conditioning:
if "minimax_token_tags" not in metadata:
raise ValueError("The CLIP encoder did not return H3 minimax_token_tags. Use an H3-compatible encoder/projection.")
metadata = dict(metadata)
if blocks:
metadata["minimax_refs"] = list(blocks)
out.append([embedding, metadata])
return out, mapping
+17
View File
@@ -0,0 +1,17 @@
[project]
name = "ComfyUI-MiniMaxH3Mod"
description = "Extract, save and combine reusable MiniMax H3 image, video and audio references as RefMod safetensors, with a unified Master extractor, library, inspector and strength curves."
version = "0.2.7"
license = { file = "LICENSE" }
readme = "README.md"
dependencies = ["torch", "safetensors", "numpy", "Pillow"]
[project.urls]
Repository = "https://github.com/Luisacaotica/ComfyUI-MiniMaxH3Mod"
Documentation = "https://github.com/Luisacaotica/ComfyUI-MiniMaxH3Mod#readme"
"Bug Tracker" = "https://github.com/Luisacaotica/ComfyUI-MiniMaxH3Mod/issues"
[tool.comfy]
# Fill with the exact publisher ID from your Comfy Registry profile before publishing.
PublisherId = "luisacaotica"
DisplayName = "MiniMax H3 RefMod"
+56
View File
@@ -0,0 +1,56 @@
# RefMod v0.2.0 — Master, audio references and loader fixes
## Highlights
- **Extract H3 RefMod Master:** image/video and optional audio in one node,
sequential extraction with separate VAEs, a combined token budget and one
bundle output. Visual and audio references save as separate compatible files.
- **Integrated audio:** extract and load audio RefMods without installing
H3AudioMod. Includes an input-music and generated-video example in the README.
- **Save H3 RefMods:** a dedicated output node saves bundles without a connected
Preview or sampler. Supports prefixes, subfolders and copy deduplication.
- **Library and inspection:** search/filter the collection, select loader slots,
inspect metadata/token counts and optionally preview stored visual/audio latents.
## Fixes and maintenance
- Linked mod names no longer fail queue validation before upstream nodes run;
covers both the standard and Axis loaders.
- Recursive subfolder discovery and saving to registered external RefMod roots
respect ComfyUI's folder configuration.
- File changes invalidate loader caches; loaded tensors have a bounded cache.
- Step Curve no longer retains reference tensors between calls. Bridge injection
is scoped to the MODEL branch and restores conditioning after errors.
- Matching image/mask crops, bounded video sampling, runtime numeric/copy limits
and optional total-token budgets.
- Atomic file saves and explicit Master Created/Replaced destination logs.
- Removed sibling-pack imports and the startup dependency warning. Legacy
av_encoder inputs resolve through ComfyUI's native video VAE loader.
- Shared extraction helpers and grouped multi-reference MSE refinement; explicit
shuffle/subset options and experimental extraction presets.
## Known failure: voice transfer
**Audio support is not working voice cloning.** Music-reference conditioning has
produced an observed result, but current speaker-identity transfer tests failed.
The cause has not been isolated: H3/checkpoint behavior and the difference in how
RefMod presents audio references to text encoding both remain under investigation.
The visual mode named `training` refines a compressed latent; it does not train
H3 weights or learn a supervised control-to-target edit. The separate gradient
probe is experimental and does not establish full-checkpoint training support.
## Updating
Validation: all 39 regression tests passed before release, using ComfyUI's
embedded Python and real queue-validation modules. Previous audio codec checks
used the real H3 audio VAE; these checks do not establish voice-transfer quality.
Update the node pack and restart ComfyUI to register the new nodes and library.
Existing visual RefMods and legacy audio metadata remain readable. When using
Save H3 RefMods after an extractor, set the extractor's save option to false to
avoid writing twice. Master saves `name_visual` and `name_audio`; load both to
reconstruct a mixed bundle. Existing files at those destinations are replaced.
The old Bridge Disarm node is retained for workflow compatibility; disable or
bypass the bridge on its MODEL branch instead of relying on global state.
+12
View File
@@ -0,0 +1,12 @@
# v0.2.5 — RefMod creation and compatibility fixes
- Rename Extract nodes to Create, with unchanged node IDs. Full Reference and Compressed Reference replace the visible encode/training names; old mode values remain accepted. Rename the identity control to Refinement Steps.
- Add visual budget_policy: truncate preserves existing behavior; error stops before saving when the visual token budget is exceeded. Master inherits the option.
- Allow larger token/frame limits and preserve the requested frame count in the motion preset. Loader budgets default to 0 (disabled).
- Add experimental H3 RefMod Text Encode with numbered reference presentation and a reference map. Support compatible ClipProj encoders and native batched video-VAE output.
- Add optional full-video inspection with correct IMAGE output dimensions; retain the lightweight first-frame default.
- Add playable music/video README examples and exclude local tests from distribution.
Validation: 51 local regression tests passed; creation-label hooks checked with Node.js. Full GPU generation, projected-encoder quality and voice transfer remain unverified. Voice cloning is not claimed to work.
Text Encode already attaches references: do not inject the same bundle again with Apply/Bridge. Integrated Continuum samplers still need a dedicated integration to use this conditioning. Full visual previews and text/vision presentation increase memory use.
+15
View File
@@ -0,0 +1,15 @@
# v0.2.6 — RefMod bundles and compact loaders
- Save visual and audio references together in one `.safetensors`: choose `save_layout=bundle` in Master, or pack existing references with **Save H3 RefMod Bundle**.
- Select **All / Visual / Audio** per Loader/Axis slot, with independent modality strengths. Excluded components are not loaded.
- Loader and Axis now start with one visible slot. **+ Add RefMod** reveals more slots, up to eight, with each reference's controls grouped together. Removal clears the slot; connected slots are protected.
- The library recognizes combined files, and Config preserves unselected members. The bundle schema is documented in `BUNDLE_FORMAT.md` for third-party pickers.
- Includes RefMod-only dropdown refresh and clearer effective-setting logs added after v0.2.5.
Existing standalone files and workflow values remain supported. Master still defaults to separate files. Combined version-5 files require a bundle-aware reader.
Restart ComfyUI for the Python changes and refresh the browser for the compact interface.
Bundling does not add voice-to-character binding, audiovisual synchronization, or original-waveform passthrough. Voice transfer remains experimental and failed in previous tests.
Validation: 61 local Python regression tests passed, plus frontend checks for adding/removing slots, connected inputs, legacy workflow restoration, serialization and library selection. The user confirmed the compact interface works in ComfyUI. Full GPU generation quality was not validated for this release.
+11
View File
@@ -0,0 +1,11 @@
# Hard dependencies (ComfyUI Manager installs these automatically).
# torch is provided by ComfyUI itself, listed here for completeness.
torch
safetensors
numpy
Pillow
# Optional video loader backends (soft-imported in common.py). Install one if
# the Load H3 RefMod Folder node can't read your video files.
# opencv-python
# imageio
@@ -0,0 +1,23 @@
{
"properties": {"refmod_visible_slots": [1, 8]},
"widgets_values": [
false,
"characters/alice", 0.73, 2,
"(none)", 1, 1,
"(none)", 1, 1,
"(none)", 1, 1,
"(none)", 1, 1,
"(none)", 1, 1,
"(none)", 1, 1,
"motion/walk", 0.6, 1,
8192,
"Visual", 0.85, 0.25,
"All", 1, 1,
"All", 1, 1,
"All", 1, 1,
"All", 1, 1,
"All", 1, 1,
"All", 1, 1,
"All", 1, 0.4
]
}
@@ -0,0 +1,157 @@
import assert from 'node:assert/strict';
import {readFileSync} from 'node:fs';
const source = readFileSync(new URL('../web/refmod_slots.js', import.meta.url), 'utf8')
.replace('import { app } from "../../scripts/app.js";', 'const app = {registerExtension() {}};');
const {installRefModSlots} = await import('data:text/javascript;base64,' + Buffer.from(source).toString('base64'));
function fixture(axis = false, stableView = false) {
class Node {
constructor() {
if (stableView) {
// Frontend v1.53.6 widgetsView.ts mutates a stable array on assignment.
const view = [];
Object.defineProperty(this, 'widgets', {
get: () => view,
set: widgets => view.splice(0, view.length, ...widgets),
});
}
this.widgets = []; this.properties = {}; this.inputs = []; this.size = [360, 1400];
this.addWidget('toggle', 'show_info', false);
for (let i=1;i<=8;i++) {
if (axis) {
this.addWidget('combo',`mod_a_${i}`,'(none)');
this.addWidget('combo',`mod_b_${i}`,'(none)');
this.addWidget('number',`value_${i}`,0);
} else {
this.addWidget('combo',`mod_${i}`,'(none)');
this.addWidget('number',`strength_${i}`,1);
this.addWidget('number',`copies_${i}`,1);
}
}
this.addWidget('number','max_total_tokens',0);
for (let i=1;i<=8;i++) {
this.addWidget('combo',`components_${i}`,'All');
this.addWidget('number',`visual_strength_${i}`,1);
this.addWidget('number',`audio_strength_${i}`,1);
}
this.onNodeCreated();
}
addWidget(type,name,value,callback,options={}) {
const widget={type,name,value,callback,options}; this.widgets.push(widget); return widget;
}
computeSize() { return [360, 60 + this.widgets.reduce((h,w)=>h+(w.computeSize?.()[1] ?? 20)+4,0)]; }
setSize(size) { this.size=size; }
setDirtyCanvas() {}
serialize() { return {properties:structuredClone(this.properties),widgets_values:this.widgets.map(w=>w.options.serialize===false?null:w.value)}; }
// PR #20: modern frontend saves directly, without calling serialize().
serializeFromStoreState() {
return {properties:structuredClone(this.properties),widgets_values:this.widgets.map(w=>w.options.serialize===false?null:w.value)};
}
configure(info) {
this.properties = structuredClone(info.properties ?? {});
// LiteGraph invokes these even for unconnected input sockets,
// before restoring widgets_values (frontend 1.39.2 and 1.52.7).
this.inputs=info.inputs ?? [{name:'show_info',link:null}];
for (const [i,input] of this.inputs.entries()) {
this.onConnectionsChange?.(1,i,true,null,input);
}
info.widgets_values.forEach((value,i)=>{if(this.widgets[i])this.widgets[i].value=value;});
}
}
installRefModSlots(Node);
return Node;
}
const get=(node,name)=>node.widgets.find(w=>w.name===name);
const click=(node,name)=>get(node,name).callback();
for (const stableView of [false,true]) for (const axis of [false,true]) {
const Node=fixture(axis, stableView), node=new Node();
const mod=axis?'mod_a_':'mod_';
const initial=node.size[1];
assert.deepEqual(node.properties.refmod_visible_slots,[1]);
assert.equal(get(node,mod+'8').type,'combo');
assert.equal(get(node,mod+'8').options.hidden,true);
// Nodes 2.0 renders converted-widget socket rows even when hidden. Real
// widget types plus options.hidden eliminate those blank rows entirely.
const rendered = () => node.widgets.filter(w => w.type === 'converted-widget' || !w.options.hidden).map(w => w.name);
assert.ok(!rendered().includes(mod+'8'));
assert.deepEqual(rendered().slice(0,6), axis
? ['mod_a_1','mod_b_1','value_1','components_1','visual_strength_1','audio_strength_1']
: ['mod_1','strength_1','copies_1','components_1','visual_strength_1','audio_strength_1']);
click(node,'+ Add RefMod');
assert.deepEqual(node.properties.refmod_visible_slots,[1,2]);
assert.ok(node.size[1]>initial);
get(node,mod+'2').value='folder/hero';
get(node,'audio_strength_2').value=.25;
const saved=node.serialize();
assert.equal(saved.widgets_values[4],'folder/hero');
const displayOrder = [...node.widgets];
node.serialize = () => { throw new Error('store serializer must be independent'); };
assert.deepEqual(node.serializeFromStoreState(), saved);
assert.deepEqual(node.widgets, displayOrder);
assert.equal(node._refmodSlots.schemaOrderDepth, 0);
// Vue/store snapshots may restore visible rows as an indexed object.
const fromStore = new Node();
fromStore.configure({...saved, properties:{refmod_visible_slots:{0:1,1:2,2:4}}});
assert.deepEqual(fromStore.properties.refmod_visible_slots,[1,2,4]);
assert.equal(get(fromStore,mod+'2').value,'folder/hero');
assert.equal(get(fromStore,'audio_strength_2').value,.25);
assert.deepEqual(fromStore.serializeFromStoreState().widgets_values,saved.widgets_values);
const restored=new Node(); restored.configure(saved);
assert.equal(get(restored,mod+'2').value,'folder/hero');
assert.equal(get(restored,'audio_strength_2').value,.25);
assert.deepEqual(restored.serialize().widgets_values,saved.widgets_values);
assert.equal(restored.widgets.indexOf(get(restored,'audio_strength_1')),5);
// Repeated tab changes must keep all schema values and visible rows stable.
let current = restored;
for (let i=0;i<5;i++) {
const snapshot = current.serialize();
current = new Node();
current.configure(snapshot);
assert.deepEqual(current.serialize(), snapshot);
}
// A saved connected input also fires callbacks before value restoration.
const connected = new Node();
connected.configure({...saved,inputs:[{name:mod+'8',link:99}]});
assert.equal(get(connected,mod+'2').value,'folder/hero');
assert.equal(get(connected,'audio_strength_2').value,.25);
assert.ok(connected.properties.refmod_visible_slots.includes(8));
// Connected rows cannot be removed, even when their widget has no filename.
restored.inputs=[{name:mod+'8',link:99}]; restored.onConnectionsChange();
click(restored,'Remove RefMod 8');
assert.ok(restored.properties.refmod_visible_slots.includes(8));
assert.equal(restored.inputs[0].link,99);
click(restored,'Remove RefMod 2');
assert.equal(get(restored,mod+'2').value,'(none)');
assert.equal(get(restored,'audio_strength_2').value,1);
assert.ok(!restored.properties.refmod_visible_slots.includes(2));
// Legacy positional workflow, including a high selected slot and no UI properties.
const legacy=new Node().serialize(); delete legacy.properties;
legacy.widgets_values=legacy.widgets_values.slice(0,26);
legacy.widgets_values[22]='legacy/eighth';
const migrated=new Node(); migrated.configure(legacy);
assert.equal(get(migrated,mod+'8').value,'legacy/eighth');
assert.ok(migrated.properties.refmod_visible_slots.includes(8));
assert.equal(get(migrated,'audio_strength_8').value,1);
// Library selection in a hidden slot reveals it via its existing callback.
get(migrated,mod+'5').value='library/pick'; get(migrated,mod+'5').callback();
assert.ok(migrated.properties.refmod_visible_slots.includes(5));
while(migrated.properties.refmod_visible_slots.length<8) click(migrated,'+ Add RefMod');
assert.equal(get(migrated,'+ Add RefMod').options.hidden,true);
}
// Frozen pre-library schema values: restoration must not depend on a snapshot
// produced by the current implementation under test.
const legacyFixture = JSON.parse(readFileSync(new URL('./fixtures/loader_widgets.json',import.meta.url),'utf8'));
const FixtureNode = fixture();
const fixtureNode = new FixtureNode();
fixtureNode.configure(legacyFixture);
assert.equal(get(fixtureNode,'mod_1').value,'characters/alice');
assert.equal(get(fixtureNode,'strength_1').value,.73);
assert.equal(get(fixtureNode,'copies_1').value,2);
assert.equal(get(fixtureNode,'mod_8').value,'motion/walk');
assert.equal(get(fixtureNode,'components_1').value,'Visual');
assert.equal(get(fixtureNode,'visual_strength_1').value,.85);
assert.equal(get(fixtureNode,'audio_strength_8').value,.4);
assert.equal(get(fixtureNode,'max_total_tokens').value,8192);
assert.deepEqual(fixtureNode.properties.refmod_visible_slots,[1,8]);
assert.deepEqual(fixtureNode.serializeFromStoreState().widgets_values.slice(0,50),legacyFixture.widgets_values);
console.log('PASS: Loader + Axis, legacy/store serialization, indexed visible slots, frozen fixture, linked slots and library selection');
@@ -0,0 +1,84 @@
"""Audio backend regressions without loading codec weights."""
import builtins
import contextlib
import importlib
import math
import sys
import types
import unittest
from unittest.mock import Mock, patch
import test_loaders as loaders
torch = loaders.torch
AUDIO = importlib.import_module(loaders.N.__package__ + ".audio")
class AudioResampleTests(unittest.TestCase):
def setUp(self):
self.pieces = []
pieces = self.pieces
class Codec:
def encode(self, waveform):
pieces.append(waveform.clone())
return torch.zeros(1, 32, 2, math.ceil(waveform.shape[-1] / 800))
self.vae = types.SimpleNamespace(first_stage_model=Codec(), patcher=object(),
device="cpu", vae_dtype=torch.float32)
for mocked in (patch.object(AUDIO, "MiniMaxH3AudioVAE", Codec),
patch.object(AUDIO.model_management, "load_models_gpu"),
patch.object(AUDIO.model_management, "cuda_device_context",
side_effect=lambda device: contextlib.nullcontext())):
mocked.start()
self.addCleanup(mocked.stop)
def backend(self):
def resample(waveform, orig_freq, new_freq):
length = math.ceil(waveform.shape[-1] * new_freq / orig_freq)
return torch.ones(*waveform.shape[:-1], length)
return types.SimpleNamespace(resample=Mock(side_effect=resample))
def test_native_backend_without_torchaudio_clips_before_resampling(self):
for rate, channels in ((44100, 1), (48000, 2)):
with self.subTest(rate=rate, channels=channels):
native = self.backend()
self.pieces.clear()
with patch.dict(sys.modules, {"comfy.audio": native, "torchaudio": None,
"torchaudio.functional": None}):
z = AUDIO.encode_audio(self.vae, {
"waveform": torch.zeros(1,channels,rate*2), "sample_rate": rate},
max_seconds=1.25, chunk_seconds=1)
wave, orig, new = native.resample.call_args.args
self.assertEqual(tuple(wave.shape), (1,2,round(rate*1.25)))
self.assertEqual((orig,new), (rate,32000))
self.assertEqual([p.shape[-1] for p in self.pieces], [32000,8000])
self.assertEqual(tuple(z.shape), (1,32,2,50))
def test_older_comfyui_uses_torchaudio(self):
legacy = self.backend()
with patch.dict(sys.modules, {"comfy.audio": None, "torchaudio.functional": legacy}):
AUDIO.encode_audio(self.vae, {"waveform": torch.zeros(1,2,4800), "sample_rate": 48000})
legacy.resample.assert_called_once()
self.assertEqual(self.pieces[0].shape[-1], 3200)
def test_32000_needs_neither_resampler(self):
with patch.dict(sys.modules, {"comfy.audio": None, "torchaudio": None,
"torchaudio.functional": None}):
AUDIO.encode_audio(self.vae, {"waveform": torch.ones(1,1,3200), "sample_rate": 32000})
torch.testing.assert_close(self.pieces[0], torch.ones(1,2,3200))
def test_native_dependency_error_is_not_hidden_by_fallback(self):
original_import = builtins.__import__
def failing_import(name, *args, **kwargs):
if name == "comfy.audio":
raise ModuleNotFoundError("Missing native dependency", name="scipy")
return original_import(name, *args, **kwargs)
with patch.object(builtins, "__import__", side_effect=failing_import):
with self.assertRaises(ModuleNotFoundError) as error:
AUDIO.encode_audio(self.vae, {"waveform": torch.zeros(1,2,4800), "sample_rate": 48000})
self.assertEqual(error.exception.name, "scipy")
if __name__ == "__main__":
unittest.main()
+229
View File
@@ -0,0 +1,229 @@
"""Loader regressions using real safetensors and ComfyUI's queue validator.
Run with ComfyUI's Python: python tests/test_loaders.py
No models or server are started; all mod files live in temporary folders.
"""
import asyncio
import contextlib
import importlib
import io
import json
import os
from pathlib import Path
import sys
import tempfile
import unittest
from unittest.mock import patch
ROOT = Path(__file__).resolve().parents[1]
sys.path.insert(0, str(ROOT.parents[1]))
from comfy.cli_args import args
args.cpu = True
import folder_paths
import torch
from safetensors.torch import save_file
with tempfile.TemporaryDirectory() as import_dir:
with patch.object(folder_paths, "models_dir", import_dir):
N = importlib.import_module(f"custom_nodes.{ROOT.name}.nodes")
import execution
import nodes as comfy_nodes
class UpstreamString:
RETURN_TYPES = ("STRING",)
FUNCTION = "run"
@classmethod
def INPUT_TYPES(cls):
return {"required": {"text": ("STRING",)}}
def run(self, text):
raise AssertionError("Queue validation must not execute upstream nodes")
class LoaderTests(unittest.TestCase):
def setUp(self):
self.temp = tempfile.TemporaryDirectory()
self.addCleanup(self.temp.cleanup)
self.root = Path(self.temp.name) / "refmods"
self.legacy = Path(self.temp.name) / "mods"
self.root.mkdir()
self.legacy.mkdir()
search = patch.object(N, "_mod_search_dirs", return_value=[str(self.root), str(self.legacy)])
search.start()
self.addCleanup(search.stop)
N._MOD_CACHE.clear()
N._MOD_LIST_CACHE_KEY = N._MOD_LIST_CACHE_VAL = None
output = contextlib.redirect_stdout(io.StringIO())
output.__enter__()
self.addCleanup(output.__exit__, None, None, None)
def save_mod(self, name, kind="image", root=None, value=1.0, metadata=None):
path = (root or self.root) / name
path.parent.mkdir(parents=True, exist_ok=True)
mod = N.H3RefMod(name=path.name, kind=kind,
latent=torch.full((1, 24, 1, 4, 4), value),
latent_h=4, latent_w=4)
if metadata is None:
mod.save(str(path))
else:
save_file({"latent": mod.latent}, str(path) + ".safetensors",
metadata={"refmod_meta": json.dumps(metadata)})
return path
def test_empty_names_validate_and_execute_in_every_slot(self):
for cls in (N.MiniMaxH3RefModsLoader, N.MiniMaxH3RefModsAxis):
for field in cls.INPUT_TYPES()["required"]:
if not field.startswith("mod_"):
continue
for empty in (None, "", "None", "(none)"):
with self.subTest(node=cls.__name__, field=field, empty=empty):
inputs = {field: empty}
if "_a_" in field or "_b_" in field:
inputs["value_" + field.rsplit("_", 1)[1]] = -1 if "_a_" in field else 1
self.assertIs(cls.VALIDATE_INPUTS(**inputs), True)
self.assertEqual(cls().load(**inputs), ([], ""))
def test_recursive_listing_filters_metadata_and_presets(self):
self.save_mod("root")
self.save_mod("celebs/person", kind="video")
self.save_mod("oc/deep/person", root=self.legacy)
self.save_mod("celebs/audio", metadata={"kind": "other"})
self.save_mod("celebs/bad_json_type", metadata=["image"])
self.save_mod("graph_presets/hidden")
self.save_mod("oc/graph_presets/hidden")
(self.root / "curve.png").touch()
(self.root / "junk.safetensors").write_bytes(b"invalid")
expected = ["celebs/person", "oc/deep/person", "root"]
self.assertEqual(N._list_mod_names(), expected)
for cls in (N.MiniMaxH3RefModsLoader, N.MiniMaxH3RefModsAxis):
for field, schema in cls.INPUT_TYPES()["required"].items():
if field.startswith("mod_"):
self.assertEqual(schema[0], ["(none)"] + expected)
def test_listing_cache_reuses_metadata_reads(self):
self.save_mod("deep/a")
with patch.object(N, "read_refmod_meta", wraps=N.read_refmod_meta) as read:
self.assertEqual(N._list_mod_names(), ["deep/a"])
self.assertEqual(N._list_mod_names(), ["deep/a"])
self.assertEqual(read.call_count, 1)
def test_nested_add_rename_delete_refresh_listing(self):
self.assertEqual(N._list_mod_names(), [])
path = self.save_mod("new/deep/a")
self.assertEqual(N._list_mod_names(), ["new/deep/a"])
file = Path(str(path) + ".safetensors")
renamed = file.with_name("b.safetensors")
file.rename(renamed)
self.assertEqual(N._list_mod_names(), ["new/deep/b"])
renamed.unlink()
self.assertEqual(N._list_mod_names(), [])
def test_subsecond_metadata_change_refreshes_listing(self):
path = self.save_mod("deep/a", metadata={"kind": "image"})
file = Path(str(path) + ".safetensors")
stamp = 1_700_000_000_100_000_000
os.utime(file, ns=(stamp, stamp))
self.assertEqual(N._list_mod_names(), ["deep/a"])
self.save_mod("deep/a", metadata={"kind": "other"})
os.utime(file, ns=(stamp + 1_000_000, stamp + 1_000_000))
self.assertEqual(N._list_mod_names(), [])
def test_legacy_sidecar_add_change_delete_refresh_listing(self):
path = self.root / "deep" / "legacy"
path.parent.mkdir()
save_file({"latent": torch.zeros(1, 24, 1, 4, 4)}, str(path) + ".safetensors")
self.assertEqual(N._list_mod_names(), [])
sidecar = Path(str(path) + ".json")
sidecar.write_text('{"kind": "image"}')
self.assertEqual(N._list_mod_names(), ["deep/legacy"])
self.assertEqual(N._load_mod("deep/legacy").kind, "image")
sidecar.write_text('{"kind": "unsupported"}')
self.assertEqual(N._list_mod_names(), [])
sidecar.unlink()
self.assertEqual(N._list_mod_names(), [])
def test_search_root_identity_and_precedence(self):
self.save_mod("a", value=1)
self.save_mod("a", root=self.legacy, value=2)
self.assertEqual(N._list_mod_names(), ["a"])
self.assertEqual(N._load_mod("a").latent.flatten()[0].item(), 1)
with patch.object(N, "_mod_search_dirs", return_value=[str(self.legacy)]):
self.assertEqual(N._load_mod("a").latent.flatten()[0].item(), 2)
self.save_mod("a", metadata={"kind": "other"})
self.assertEqual(N._list_mod_names(), [])
def test_nested_runtime_both_loaders_and_windows_separators(self):
self.save_mod("celebs/person", value=3)
self.assertIs(N.MiniMaxH3RefModsLoader.VALIDATE_INPUTS(mod_1="celebs\\person"), True)
bundle, _ = N.MiniMaxH3RefModsLoader().load(mod_1="celebs\\person", strength_1=0.6, copies_1=2)
self.assertEqual(len(bundle), 2)
self.assertEqual(bundle[0][1], 0.6)
self.assertEqual(bundle[0][0].latent.flatten()[0].item(), 3)
for side, value in (("a", -0.4), ("b", 0.4)):
bundle, _ = N.MiniMaxH3RefModsAxis().load(**{f"mod_{side}_1": "celebs/person", "value_1": value})
self.assertEqual(bundle[0][1], 0.4)
self.assertIs(bundle[0][0], N._load_mod("celebs\\person"))
def test_same_metadata_name_config_does_not_poison_root_cache(self):
self.save_mod("person", value=1)
nested_path = self.save_mod("celebs/person", value=2)
root_mod = N._load_mod("person")
nested_mod = N._load_mod("celebs/person")
N.MiniMaxH3RefModConfig().fix([(nested_mod, 1.0)], retention=0.4)
self.assertIs(N._load_mod("person"), root_mod)
self.assertEqual(root_mod.config, {})
self.assertEqual(N.H3RefMod.load(str(nested_path)).config["retention"], 0.4)
def test_fixed_missing_names_rejected_and_linked_missing_fail_at_load(self):
for cls, field in ((N.MiniMaxH3RefModsLoader, "mod_1"),
(N.MiniMaxH3RefModsAxis, "mod_a_1"),
(N.MiniMaxH3RefModsAxis, "mod_b_1")):
self.assertIn("not found", cls.VALIDATE_INPUTS(**{field: "missing"}))
self.assertIs(cls.VALIDATE_INPUTS(input_types={field: "STRING"}, **{field: None}), True)
with self.assertRaisesRegex(FileNotFoundError, "missing"):
N.MiniMaxH3RefModsLoader().load(mod_1="missing")
def test_path_boundary_is_checked_at_execution(self):
for name in ("../outside", "a/../../outside", "/absolute", "C:/absolute", "C:relative",
"\\\\server\\share\\mod", "graph_presets/a"):
with self.subTest(name=name), self.assertRaises(ValueError):
N._load_mod(name)
def test_actual_comfy_queue_with_unexecuted_upstream(self):
for cls, field in ((N.MiniMaxH3RefModsLoader, "mod_1"),
(N.MiniMaxH3RefModsAxis, "mod_a_1"),
(N.MiniMaxH3RefModsAxis, "mod_b_1")):
inputs = {}
for name, (kind, options) in cls.INPUT_TYPES()["required"].items():
inputs[name] = kind[0] if isinstance(kind, list) else options["default"]
inputs[field] = ["source", 0]
prompt = {"source": {"class_type": "TestRefModString", "inputs": {"text": "future/person"}},
"loader": {"class_type": "TestRefModLoader", "inputs": inputs}}
with patch.dict(comfy_nodes.NODE_CLASS_MAPPINGS, TestRefModString=UpstreamString, TestRefModLoader=cls):
for output_type in ("STRING", "COMBO", ["future/person"]):
with self.subTest(node=cls.__name__, field=field, output_type=output_type):
with patch.object(UpstreamString, "RETURN_TYPES", (output_type,)):
result = asyncio.run(execution.validate_inputs("test-refmods", prompt, "loader", {}))
self.assertTrue(result[0], result)
with patch.object(UpstreamString, "RETURN_TYPES", ("IMAGE",)):
result = asyncio.run(execution.validate_inputs("test-refmods", prompt, "loader", {}))
self.assertFalse(result[0], result)
def test_connected_non_mod_inputs_keep_type_checks(self):
for cls, field in ((N.MiniMaxH3RefModsLoader, "strength_1"),
(N.MiniMaxH3RefModsLoader, "copies_1"),
(N.MiniMaxH3RefModsLoader, "show_info"),
(N.MiniMaxH3RefModsAxis, "value_1")):
with self.subTest(field=field):
self.assertIn("expected", cls.VALIDATE_INPUTS(input_types={field: "IMAGE"}))
if __name__ == "__main__":
unittest.main(verbosity=2)
+128
View File
@@ -0,0 +1,128 @@
export function openRefModLibrary(node, fetchEntries) {
const previousFocus = document.activeElement;
const dialog = document.createElement("dialog");
dialog.className = "refmod-library";
dialog.setAttribute("aria-label", "RefMod library");
const style = document.createElement("style");
style.textContent = `
.refmod-library {box-sizing:border-box;width:min(1100px,96vw);height:90vh;max-height:90vh;padding:24px;border:1px solid #455266;border-radius:14px;background:#111820;color:#e8edf5;font:14px system-ui;}
.refmod-library::backdrop {background:#000b}
.refmod-library .layout {height:100%;display:flex;flex-direction:column;gap:16px;min-height:0}
.refmod-library header {display:flex;gap:12px;align-items:center;justify-content:space-between}
.refmod-library h2 {margin:0;font-size:23px}
.refmod-library .controls {display:flex;gap:10px;flex-wrap:wrap}
.refmod-library input,.refmod-library select,.refmod-library button {font:inherit;color:inherit;background:#222f3e;border:1px solid #455266;border-radius:7px;padding:9px 12px}
.refmod-library input {flex:1;min-width:180px}
.refmod-library button {cursor:pointer}
.refmod-library button:focus-visible,.refmod-library input:focus-visible,.refmod-library select:focus-visible {outline:2px solid #ffb257;outline-offset:2px}
.refmod-library .results {overflow:auto;min-height:0;flex:1;display:flex;flex-direction:column;gap:10px}
.refmod-library article {padding:16px;border:1px solid #334254;border-radius:9px;background:#18222d;display:grid;grid-template-columns:1fr auto;gap:8px 18px}
.refmod-library .description,.refmod-library .path {grid-column:1/-1;color:#afbed0;overflow-wrap:anywhere}
.refmod-library .path {font-size:12px}
.refmod-library .use {background:#ffb257;border-color:#ffb257;color:#171c22;font-weight:650}
.refmod-library .status {color:#afbed0;min-height:20px}
@media(max-width:600px){.refmod-library{padding:14px}.refmod-library article{grid-template-columns:1fr}.refmod-library header{flex-wrap:wrap}}
`;
const make = (tag, text, className) => {
const el = document.createElement(tag);
if (text !== undefined) el.textContent = text;
if (className) el.className = className;
return el;
};
const layout = make("div", undefined, "layout");
const header = make("header");
const close = make("button", "Close");
header.append(make("h2", "RefMod library"), close);
const controls = make("div", undefined, "controls");
const search = make("input");
search.placeholder = "Search name, folder or description";
search.setAttribute("aria-label", "Search RefMods");
const kind = make("select");
kind.setAttribute("aria-label", "Reference type");
for (const value of ["all", "image", "video", "audio", "bundle"]) {
const option = make("option", value === "all" ? "All reference types" : value);
option.value = value;
kind.append(option);
}
const slot = make("select");
slot.setAttribute("aria-label", "Loader slot");
const widgets = (node.widgets || []).filter(w => /^mod_(?:[ab]_)?\d+$/.test(w.name));
for (const widget of widgets) {
const option = make("option", widget.name);
option.value = widget.name;
slot.append(option);
}
const refresh = make("button", "Refresh");
controls.append(search, kind, slot, refresh);
const status = make("div", "Loading library…", "status");
status.setAttribute("role", "status");
const results = make("div", undefined, "results");
layout.append(header, controls, status, results);
dialog.append(style, layout);
document.body.append(dialog);
let entries = [], disposed = false, controller;
const cleanup = () => {
if (disposed) return;
disposed = true;
controller?.abort();
dialog.remove();
previousFocus?.focus();
};
close.onclick = () => dialog.close();
dialog.addEventListener("close", cleanup, {once:true});
const render = () => {
const query = search.value.trim().toLocaleLowerCase();
const filtered = entries.filter(e => (kind.value === "all" || e.kind === kind.value)
&& [e.name,e.concept,e.description].join(" ").toLocaleLowerCase().includes(query));
status.textContent = `${filtered.length} of ${entries.length} RefMods · choose a loader slot, then Use`;
results.replaceChildren();
for (const entry of filtered) {
const row = make("article");
const title = make("strong", entry.name);
const use = make("button", "Use", "use");
use.disabled = widgets.length === 0;
use.onclick = () => {
const widget = widgets.find(w => w.name === slot.value);
const linked = node.inputs?.find(i => i.name === slot.value)?.link;
if (linked != null) {
status.textContent = `${slot.value} is connected. Select an unconnected slot.`;
return;
}
if (!widget) return;
widget.value = entry.name;
widget.callback?.(entry.name);
node.setDirtyCanvas?.(true, true);
dialog.close();
};
const tokens = entry.tokens == null ? "tokens available after loading" : `${entry.tokens.toLocaleString()} tokens / copy`;
row.append(title, use, make("div", `${entry.kind} · ${entry.concept} · ${tokens}`, "description"),
make("div", entry.description || "No description saved", "description"),
make("div", entry.path, "path"));
results.append(row);
}
if (!filtered.length) results.append(make("p", "No matching RefMods. Try another search or Refresh after adding files."));
};
const load = async () => {
controller?.abort();
controller = new AbortController();
const signal = controller.signal;
refresh.disabled = true;
try {
const data = await fetchEntries(signal);
if (disposed || signal.aborted) return;
entries = data;
render();
} catch (error) {
if (!disposed && !signal.aborted) status.textContent = `Could not load library: ${error.message}`;
} finally {
if (!disposed && !signal.aborted) refresh.disabled = false;
}
};
search.oninput = render;
kind.onchange = render;
refresh.onclick = load;
dialog.showModal();
search.focus();
load();
return dialog;
}
+34
View File
@@ -0,0 +1,34 @@
import { app } from "../../scripts/app.js";
export function installRefModApply(nodeType) {
const created = nodeType.prototype.onNodeCreated;
nodeType.prototype.onNodeCreated = function(...args) {
const result = created?.apply(this, args);
// MatchType installs its callback on the instance during construction.
const connections = this.onConnectionsChange;
this.onConnectionsChange = function(...args) {
const index = this.inputs?.findIndex(input => input.name === "conditioning") ?? -1;
if (index >= 0) {
const input = this.inputs[index];
try {
// Frontend 1.39.2 proxies MatchType slots, but NodeSlot.node
// reads a #private field. That throws during canvas drawing.
void input.node;
} catch (error) {
if (!(error instanceof TypeError)) throw error;
this.inputs[index] = new input.constructor({...input}, this);
if (args[4] === input) args[4] = this.inputs[index];
}
}
return connections?.apply(this, args);
};
return result;
};
}
app.registerExtension({
name: "MiniMaxH3.ApplyCanvasCompatibility",
beforeRegisterNodeDef(nodeType, nodeData) {
if (nodeData.name === "MiniMaxH3RefModApply") installRefModApply(nodeType);
},
});
@@ -0,0 +1,28 @@
import { app } from "../../scripts/app.js";
const modes = { training: "Compressed Reference", encode: "Full Reference" };
function updateLabels(node) {
const mode = node.widgets?.find(widget => widget.name === "mode");
if (mode) {
mode.options.values = Object.values(modes);
mode.value = modes[mode.value] ?? mode.value;
}
const steps = node.widgets?.find(widget => widget.name === "identity");
if (steps) steps.label = "Refinement Steps";
}
app.registerExtension({
name: "MiniMaxH3.CreationLabels",
beforeRegisterNodeDef(nodeType, nodeData) {
if (!["MiniMaxH3RefModExtract", "MiniMaxH3RefModMasterExtract"].includes(nodeData.name)) return;
for (const hook of ["onNodeCreated", "onConfigure"]) {
const previous = nodeType.prototype[hook];
nodeType.prototype[hook] = function(...args) {
const result = previous?.apply(this, args);
updateLabels(this);
return result;
};
}
}
});
+67
View File
@@ -0,0 +1,67 @@
import { app } from "../../scripts/app.js";
import { api } from "../../scripts/api.js";
import { openRefModLibrary } from "./library_dialog.js";
export function updateRefModChoices(node, entries) {
const names = ["(none)", ...new Set(entries.map(entry => entry.name))];
for (const widget of node.widgets || []) {
if (!/^mod_(?:[ab]_)?\d+$/.test(widget.name)) continue;
// Preserve selections, including missing files, for an explicit load
// error instead of silently switching the subject after a refresh.
widget.options.values = names.includes(widget.value) || !widget.value
? [...names] : [...names, widget.value];
}
node.setDirtyCanvas?.(true, true);
}
app.registerExtension({
name: "MiniMaxH3.RefModLibrary",
beforeRegisterNodeDef(nodeType, nodeData) {
if (!["MiniMaxH3RefModsLoader", "MiniMaxH3RefModsAxis"].includes(nodeData.name)) return;
const removed = nodeType.prototype.onRemoved;
nodeType.prototype.onRemoved = function(...args) {
this._refmodRefresh?.abort();
this._refmodDialog?.close();
return removed?.apply(this, args);
};
const previous = nodeType.prototype.onNodeCreated;
nodeType.prototype.onNodeCreated = function(...args) {
const result = previous?.apply(this, args);
const refresh = this.addWidget("button", "Refresh RefMods", null, async () => {
this._refmodRefresh?.abort();
const controller = new AbortController();
this._refmodRefresh = controller;
refresh.name = "Refreshing RefMods…";
this.setDirtyCanvas?.(true, true);
try {
const response = await api.fetchApi("/refmods/library", {signal: controller.signal, cache: "no-store"});
if (!response.ok) throw new Error(`HTTP ${response.status}`);
const entries = await response.json();
if (controller.signal.aborted) return;
updateRefModChoices(this, entries);
refresh.name = "Refresh RefMods";
} catch (error) {
if (controller.signal.aborted) return;
refresh.name = "Refresh failed — click to retry";
console.error("RefMod refresh failed:", error);
} finally {
if (!controller.signal.aborted) {
this._refmodRefresh = null;
this.setDirtyCanvas?.(true, true);
}
}
}, {serialize:false});
this.addWidget("button", "RefMod library", null, () => {
this._refmodDialog?.close();
this._refmodDialog = openRefModLibrary(this, async signal => {
const response = await api.fetchApi("/refmods/library", {signal});
if (!response.ok) throw new Error(`HTTP ${response.status}`);
const entries = await response.json();
if (!signal.aborted) updateRefModChoices(this, entries);
return entries;
});
}, {serialize:false});
return result;
};
}
});
+158
View File
@@ -0,0 +1,158 @@
import { app } from "../../scripts/app.js";
const SLOT = /^(?:mod_(?:[ab]_)?|strength_|copies_|value_|components_|visual_strength_|audio_strength_)(\d+)$/;
const NONE = new Set(["", "None", "(none)"]);
export function installRefModSlots(nodeType) {
const created = nodeType.prototype.onNodeCreated;
nodeType.prototype.onNodeCreated = function(...args) {
const result = created?.apply(this, args);
setupSlots(this);
return result;
};
// LiteGraph stores widget values by position. Keep the original schema
// order for saving/loading even though the canvas groups them by slot.
// ComfyUI frontend >= 1.5x saves via serializeFromStoreState() and never
// calls serialize(), so that path has to be covered as well or workflows
// get written in display order and shift on the next load.
for (const method of ["serialize", "serializeFromStoreState", "configure"]) {
const original = nodeType.prototype[method];
nodeType.prototype[method] = function(...args) {
const state = this._refmodSlots;
if (!state) return original?.apply(this, args);
// Nodes 2.0 exposes a stable array view: assigning widgets mutates
// that same view, so retain a snapshot rather than its reference.
const displayOrder = [...this.widgets];
state.schemaOrderDepth++;
this.widgets = [...state.originals, ...displayOrder.filter(w => !state.originals.includes(w))];
try {
return original?.apply(this, args);
} finally {
this.widgets = displayOrder;
state.schemaOrderDepth--;
if (method === "configure") state.sync();
}
};
}
const connections = nodeType.prototype.onConnectionsChange;
nodeType.prototype.onConnectionsChange = function(...args) {
const result = connections?.apply(this, args);
this._refmodSlots?.sync();
return result;
};
}
function setupSlots(node) {
if (node._refmodSlots) return;
const originals = [...(node.widgets || [])];
const groups = new Map();
for (const widget of originals) {
const match = widget.name.match(SLOT);
if (!match) continue;
const slot = Number(match[1]);
if (!groups.has(slot)) groups.set(slot, []);
groups.get(slot).push(widget);
}
const slots = [...groups.keys()].sort((a, b) => a - b);
if (!slots.length) return;
const defaults = new Map(originals.map(w => [w, w.value]));
const hidden = new Map();
const removers = new Map();
node.properties ??= {};
node.properties.refmod_visible_slots ??= [slots[0]];
const linked = slot => node.inputs?.some(input => Number(input.name.match(SLOT)?.[1]) === slot && input.link != null);
const used = slot => groups.get(slot).some(w => w.name.startsWith("mod_")
? w.value != null && !NONE.has(String(w.value).trim())
: w.value !== defaults.get(w));
function show(widget, visible) {
if (!visible && !hidden.has(widget)) {
hidden.set(widget, {computeSize: widget.computeSize,
draw: widget.draw, mouse: widget.mouse, hidden: widget.hidden,
optionsHidden: widget.options?.hidden});
// converted-widget means a socket row in Nodes 2.0, not a hidden
// widget. Its empty row was responsible for the large gaps.
widget.options = {...widget.options, hidden: true};
widget.computeSize = () => [0, -4];
widget.draw = () => {};
widget.mouse = () => false;
widget.hidden = true;
} else if (visible && hidden.has(widget)) {
const saved = hidden.get(widget);
widget.options = {...widget.options, hidden: saved.optionsHidden};
widget.computeSize = saved.computeSize;
widget.draw = saved.draw;
widget.mouse = saved.mouse;
widget.hidden = saved.hidden;
hidden.delete(widget);
}
}
function sync() {
// configure fires connection callbacks before assigning widget values.
// Keep schema order until the entire positional restore has finished.
if (node._refmodSlots.schemaOrderDepth) return;
const requested = node.properties.refmod_visible_slots;
// A saved workflow can bring this back as {0: 1, 1: 2, ...} rather than
// an array, so accept either shape before falling back to slot one.
const requestedSlots = Array.isArray(requested)
? requested
: requested && typeof requested === "object"
? Object.values(requested)
: [slots[0]];
const visible = new Set(requestedSlots.filter(slot => groups.has(slot)));
for (const slot of slots) if (used(slot) || linked(slot)) visible.add(slot);
node.properties.refmod_visible_slots = [...visible].sort((a, b) => a - b);
const grouped = [...groups.values()].flat();
const extras = node.widgets.filter(w => !originals.includes(w) && ![...removers.values(), add].includes(w));
const displayOrder = [];
for (const slot of slots) {
for (const widget of groups.get(slot)) {
show(widget, visible.has(slot));
displayOrder.push(widget);
}
const remove = removers.get(slot);
remove.disabled = !!linked(slot);
show(remove, visible.has(slot));
displayOrder.push(remove);
}
show(add, visible.size < slots.length);
displayOrder.push(add, ...originals.filter(w => !grouped.includes(w)), ...extras);
node.widgets = displayOrder;
node.setSize?.([node.size[0], node.computeSize()[1]]);
node.setDirtyCanvas?.(true, true);
}
for (const slot of slots) {
removers.set(slot, node.addWidget("button", `Remove RefMod ${slot}`, null, () => {
if (linked(slot)) return;
for (const widget of groups.get(slot)) widget.value = defaults.get(widget);
node.properties.refmod_visible_slots = node.properties.refmod_visible_slots.filter(i => i !== slot);
sync();
}, {serialize: false}));
for (const widget of groups.get(slot)) {
const callback = widget.callback;
widget.callback = function(...args) {
const result = callback?.apply(this, args);
sync();
return result;
};
}
}
const add = node.addWidget("button", "+ Add RefMod", null, () => {
const next = slots.find(slot => !node.properties.refmod_visible_slots.includes(slot));
if (next != null) node.properties.refmod_visible_slots.push(next);
sync();
}, {serialize: false});
node._refmodSlots = {originals, sync, schemaOrderDepth: 0};
sync();
}
app.registerExtension({
name: "MiniMaxH3.AdditiveSlots",
beforeRegisterNodeDef(nodeType, nodeData) {
if (["MiniMaxH3RefModsLoader", "MiniMaxH3RefModsAxis"].includes(nodeData.name)) {
installRefModSlots(nodeType);
}
},
});