Merge commit '70d0672ee306019c7dba8f799a273ec486bec95b' as 'vendor/dumas_minimax_h3_refmod'
@@ -0,0 +1,2 @@
|
||||
/tests/
|
||||
/node.zip
|
||||
@@ -0,0 +1,3 @@
|
||||
__pycache__/
|
||||
*.pyc
|
||||
/tests/
|
||||
@@ -0,0 +1,82 @@
|
||||
# RefMod single-file bundle format
|
||||
|
||||
Version 5 is a container of independent H3 references. It does not concatenate
|
||||
audio with visual latents or change H3 conditioning semantics. Existing
|
||||
standalone version-4 files remain supported and keep their `latent` tensor key.
|
||||
|
||||
## Detection and layout
|
||||
|
||||
Read JSON from the safetensors header's `refmod_meta` key without loading tensors.
|
||||
A bundle has `_format_version: 5`, `kind: "bundle"`, a display `name`, and an
|
||||
ordered `members` array containing 1–256 ordinary RefMod metadata objects.
|
||||
Nested bundles are not supported. A member's `kind` is `image`, `video`, or `audio`.
|
||||
|
||||
```json
|
||||
{
|
||||
"_format_version": 5,
|
||||
"kind": "bundle",
|
||||
"name": "hero",
|
||||
"members": [
|
||||
{
|
||||
"_format_version": 4,
|
||||
"name": "hero_visual",
|
||||
"kind": "video",
|
||||
"latent_t": 3,
|
||||
"latent_h": 32,
|
||||
"latent_w": 32
|
||||
},
|
||||
{
|
||||
"_format_version": 4,
|
||||
"name": "hero_audio",
|
||||
"kind": "audio",
|
||||
"latent_t": 200,
|
||||
"sample_rate": 32000
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
This example omits optional member metadata. Each member keeps the usual mode,
|
||||
source, tags, description, concept type and optional `refmod_config` JSON string.
|
||||
Tensor `ref_0` belongs to member 0, `ref_1` to member 1, and so on.
|
||||
|
||||
| Kind | Tensor layout | Tokens |
|
||||
| --- | --- | --- |
|
||||
| image | `[1,24,1,H,W]` | `(H/2) × (W/2)` |
|
||||
| video | `[1,24,T,H,W]` | `T × (H/2) × (W/2)` |
|
||||
| audio | `[1,32,2,T]` | `2 × T` |
|
||||
|
||||
Visual H/W must be positive and even; T must be positive. Audio uses the existing
|
||||
32 kHz H3 codec convention. Each member retains its own latent dimensions.
|
||||
|
||||
## Picker and loader integration
|
||||
|
||||
1. Detect `kind == "bundle"` and validate the supported format version.
|
||||
2. Inspect members to display whether the file contains visuals, audio, or both.
|
||||
3. Filter by **All**, **Visual** (`image` + `video`), or **Audio** before reading
|
||||
tensor data. Skip members whose modality strength is zero.
|
||||
4. Load the selected `ref_i` tensors and expand them into ordinary
|
||||
`(H3RefMod, strength)` entries, preserving relative member order.
|
||||
5. Multiply the slot strength by the modality strength. Account for each selected
|
||||
member's token cost and runtime copies.
|
||||
|
||||
In this pack, `bundle.load_bundle(path_without_extension, selection,
|
||||
visual_strength, audio_strength)` performs selection and returns ordinary
|
||||
entries with modality strengths. The caller applies the slot strength/copies.
|
||||
Members retain their container `path` and `bundle_index` for Config updates.
|
||||
|
||||
Save writes each distinct reference object once; runtime strengths and duplicate
|
||||
copies are not stored. Repacking a loaded bundle flattens its selected members.
|
||||
Config updates preserve other members and unknown container metadata/tensors.
|
||||
|
||||
## Compatibility and limits
|
||||
|
||||
New readers load older standalone files. Older readers expecting only a `latent`
|
||||
tensor will need bundle support; there is no promise of forward compatibility.
|
||||
The standalone Save node remains available for exporting members.
|
||||
|
||||
The container carries encoded references, not original audio bytes, explicit
|
||||
cross-modal timestamps, or speaker-to-character assignments. Shared storage does
|
||||
not guarantee synchronized generation or stop multiple references from mixing.
|
||||
Per-slot Audio strength affects all audio members in that slot; this initial UI
|
||||
does not select individual audio tracks within a multi-audio file.
|
||||
@@ -0,0 +1,234 @@
|
||||
# Changelog
|
||||
|
||||
All notable changes are tracked here. Each version is also published as a
|
||||
[GitHub Release](https://github.com/Luisacaotica/ComfyUI-MiniMaxH3Mod/releases),
|
||||
so you can keep using an older version if a new one changes something you
|
||||
rely on.
|
||||
|
||||
## Unreleased
|
||||
|
||||
- Correct visual reference sampling to H3's 17k+5 video grid in the node and CLI. Preserve all frames of aligned clips and both endpoints when uniformly sampling; references/caps below five frames use the first image.
|
||||
- Support MPS-resident RefMod latents without global CPU fallback: pool and blur explicitly on CPU, preserving return device, dtype and autograd. Synchronize MPS in the refinement timing harness.
|
||||
|
||||
- Add a reference_map output to Inspect H3 RefMod for LLM prompt generation before Text Encode, using the same reference numbering (#19).
|
||||
- Fix Loader/Axis settings resetting when switching workflow tabs or reopening workflows: defer slot regrouping until widget restoration finishes (#18).
|
||||
- Repair Apply's reactive MatchType socket on older frontends whose canvas socket classes use private fields, preventing missing controls while preserving conditioning type matching (#17).
|
||||
|
||||
## v0.2.6 — 2026-09-12
|
||||
|
||||
- Make Loader/Axis additive: start with one visible slot, add/remove slots up to eight, group modality controls with their reference, preserve old workflow values and protect connected slots.
|
||||
- Add version-5 single-file RefMod bundles containing independent image/video/audio members, with atomic saves and preserved member order.
|
||||
- Add Save H3 RefMod Bundle as an output node and opt-in Master save_layout=bundle. Keep standalone saving and existing workflow defaults.
|
||||
- Add All/Visual/Audio selection and separate modality strengths to every Loader/Axis slot. Skip excluded tensors and count only selected references in token budgets.
|
||||
- Discover combined files in the library; preserve unselected members during Config updates and support export to standalone files.
|
||||
- Document the container schema for third-party pickers. Bundling does not add voice binding, AV synchronization, or original-waveform passthrough.
|
||||
|
||||
## v0.2.5 — 2026-09-08
|
||||
|
||||
- Rename Extract nodes to Create, with unchanged node IDs. Full Reference and Compressed Reference replace the visible encode/training names; old mode values remain accepted. Rename the identity control to Refinement Steps.
|
||||
- Add visual budget_policy: truncate preserves existing behavior; error stops before saving when the visual token budget is exceeded. Master inherits the option.
|
||||
- Allow larger token/frame limits and preserve the requested frame count in the motion preset. Loader budgets default to 0 (disabled).
|
||||
- Add experimental H3 RefMod Text Encode with numbered reference presentation and a reference map. Support compatible ClipProj encoders and native batched video-VAE output.
|
||||
- Add optional full-video inspection with correct IMAGE output dimensions; retain the lightweight first-frame default.
|
||||
- Add playable music/video README examples and exclude local tests from distribution.
|
||||
|
||||
Validation: 51 local regression tests passed; creation-label hooks checked with Node.js. Full GPU generation, projected-encoder quality and voice transfer remain unverified. Voice cloning is not claimed to work.
|
||||
|
||||
Text Encode already attaches references: do not inject the same bundle again with Apply/Bridge. Integrated Continuum samplers still need a dedicated integration to use this conditioning. Full visual previews and text/vision presentation increase memory use.
|
||||
|
||||
## v0.2.0 — 2026-09-07
|
||||
|
||||
### Known limitation: voice transfer
|
||||
|
||||
Audio reference support includes an observed music-transfer example, but current
|
||||
speaker-identity tests failed. This release does not provide working voice cloning.
|
||||
The cause has not been isolated between H3/checkpoint behavior and the reference
|
||||
presentation difference in the current RefMod integration.
|
||||
|
||||
### Follow-up implementation (2026-09-07)
|
||||
|
||||
- Clarify Master saving logs: deferred extraction, Created/Replaced destinations,
|
||||
explicit save=False and a final token/file summary; report saved_paths in details.
|
||||
|
||||
- Add Save H3 RefMods as an output node: save mixed bundles without connecting
|
||||
a Preview or sampler, with prefix/subfolder options and copy deduplication.
|
||||
|
||||
- Save new mods/presets in the first registered refmods root (extra_model_paths);
|
||||
keep the standard models/refmods fallback and existing-file Config locations.
|
||||
- Remove sibling-pack imports and startup dependency warning. Legacy av_encoder
|
||||
inputs use native ComfyUI video VAE loading; standard VAE inputs are preferred.
|
||||
|
||||
- Add Extract H3 RefMod Master: optional image/video and audio inputs, sequential
|
||||
extraction through the shared extractors, combined token budget, one bundle
|
||||
output and separate compatible visual/audio files under a shared name.
|
||||
|
||||
- Step Curve now mixes the current payload without retaining tensors and handles
|
||||
keyframe offsets, audio refs, empty conditioning and chained curves.
|
||||
- Bridge uses the MODEL outer-sampling hook with finally cleanup; removes global
|
||||
state and third-party monkey patches. The old Disarm node is deprecated.
|
||||
- Align image/mask center crops; share resize and causal-frame helpers with CLI.
|
||||
- Invalidate loaded files and queue cache on changes; cap cache at 256 MiB;
|
||||
enforce copies/numeric limits; bound video decode buffers with unknown lengths.
|
||||
- Add native audio RefMods, legacy AudioMod import, audio Extract, library UI,
|
||||
inspection/optional previews, save subfolders, explicit scramble modes,
|
||||
extraction presets and total-token budgets.
|
||||
- Atomic saves, deduplicated config writes and grouped mean-MSE refinement.
|
||||
- Historical entries below describe earlier unreleased revisions; the behavior
|
||||
above and the current README supersede their bridge/cache descriptions.
|
||||
|
||||
### Fixed: linked loader inputs and subfolders
|
||||
|
||||
- Both loaders accept unresolved/empty mod names and connected STRING/combo
|
||||
outputs. Fixed missing names still fail validation; connected names resolve
|
||||
at execution with a missing-file error when necessary.
|
||||
- Discover RefMods recursively as `folder/name`, preserving metadata filtering
|
||||
and skipping graph presets. Listing cache tracks nested files, nanosecond
|
||||
timestamps, and legacy JSON sidecars.
|
||||
- Normalize Windows separators and check relative paths at execution. Cache
|
||||
mods by file path so identical metadata names in different folders do not
|
||||
collide when saving their configuration.
|
||||
- Add regression tests using real safetensors and ComfyUI queue validation.
|
||||
|
||||
### New: Continuum integration
|
||||
|
||||
- **Continuum RefMod Bridge** node — injects your mods into
|
||||
[ComfyUI-H3-Continuum](https://github.com/xiaolibai-sys/ComfyUI-H3-Continuum)'s
|
||||
chunked long-video samplers (V2/V3/V3.4), which build their conditioning
|
||||
internally and have no CONDITIONING socket. While armed, every chunk
|
||||
carries the bundle's ref blocks as native `minimax_refs` — persistent
|
||||
across all chunks, no Continuum markers so its continuity layout never
|
||||
re-times them. Version-checked against one public seam; if Continuum
|
||||
changes internally the bridge reports and degrades to no-injection rather
|
||||
than breaking a run. One active bundle per graph; per-run scramble seeds
|
||||
work normally. Wire it on the sampler's model line (Load Model -> bridge
|
||||
-> sampler `model`): ComfyUI never executes fully disconnected nodes, so
|
||||
the earlier "anywhere upstream" placement silently did nothing.
|
||||
|
||||
### Fixed: why insertions felt weak, random, or "possessed"
|
||||
|
||||
Three defects were found and fixed; each is reproducible with the new
|
||||
`gauntlet_harness.py` (run it with ComfyUI's Python — no server needed).
|
||||
|
||||
- **Apply's default curve silently HALVED every multi-frame mod.** The old
|
||||
default (`concept_at_end` + `ease`) runs its envelope across the mod's own
|
||||
ref frames — mean multiplier ≈ **0.50** at any frame count — so video/
|
||||
stacked mods shipped half-blurred unless you touched widgets. Worse, ref
|
||||
tokens are not bound to output-video time at all: no frame curve can place
|
||||
a concept "at the end of the video"; it only weights which reference
|
||||
content dominates. Defaults are now `constant` + `linear` + `1.0`
|
||||
(= full strength, official-ref parity); saved workflows keep their saved
|
||||
values. The curve widgets remain for stack weighting, with honest tooltips.
|
||||
For real output-timing control use **H3 RefMod Step Curve** (denoise
|
||||
timeline).
|
||||
- **Image-kind mods ignored every curve dial silently** (`curve_strengths`
|
||||
short-circuits at 1 latent frame), so single-image identity mods always
|
||||
injected at full strength — the "concept bleeds into everything" case.
|
||||
Now `curve_value` acts as a plain strength cap on single-frame mods
|
||||
(e.g. value 0.4 = the ref blends 40% toward its blurred self), with a
|
||||
console note.
|
||||
- **H3 RefMod Step Curve leaked state across generations.** The wrapper
|
||||
cached pristine/blurred latents plus the marked-ref index mapping on the
|
||||
first forward and never reset them — but ComfyUI reuses the patched MODEL
|
||||
and conditioning payload across queued runs, so run #2 mixed run #1's
|
||||
cached tensors whenever the scramble seed/mods/strengths changed
|
||||
(corrupted, "ghost" output). The wrapper now keys its state on the
|
||||
injected refs' identities (which it never mutates), re-keying cleanly on
|
||||
every new run while keeping pristine stable within one; chaining multiple
|
||||
Step Curve nodes composes deterministically, and progress is normalized to
|
||||
each run's actual schedule start (partial denoise included).
|
||||
|
||||
Also new: Apply now prints each row's *effective* strength (row × retention
|
||||
× curve-mean) and flags anything below 0.30 — the usual suspect when an
|
||||
insertion feels weak.
|
||||
|
||||
### New: fixed configs, shipped with the mod
|
||||
|
||||
- **Fix H3 RefMod Config** node — bakes tuned Apply/Step-Curve settings
|
||||
(`retention` + `curve` + `step_curve`) into a mod's safetensors metadata
|
||||
and re-saves the file in place, so a concept ships with the settings that
|
||||
make it work.
|
||||
- **`override` toggle** on Apply H3 RefMod and H3 RefMod Step Curve — reads
|
||||
the first mod in the bundle that carries a fixed config and uses those
|
||||
settings instead of the widgets; falls back to the manual parameters
|
||||
(with a console note) when no mod has one.
|
||||
- The debug curve graph reflects the overridden curve, so what you see is
|
||||
what actually ran.
|
||||
|
||||
### New: merge mode on Extract H3 RefMod
|
||||
|
||||
- **`merge` toggle** (training mode): one shared latent refined jointly
|
||||
against every reference's full encode (mean reconstruction error) instead
|
||||
of stacking each ref's own pooled latent. A whole collection becomes ONE
|
||||
tiny consensus mod — pulled toward what's common across all the views
|
||||
(structure/motion/identity) rather than any single shot's framing or
|
||||
background, at one ref block's worth of tokens. `identity` dials the
|
||||
joint refinement; masks and `background_retention` still apply per ref.
|
||||
- **`motion_only` toggle** (training mode, experimental): video refs are
|
||||
converted to per-frame temporal differences before encoding, so the mod
|
||||
carries where/how things move instead of what they look like — static
|
||||
appearance (lineart look, background) never enters the latent. A soft
|
||||
motion guide, not a ControlNet (the ref channel is content-based).
|
||||
|
||||
### Fixed
|
||||
|
||||
- **Curve directions now mean what they say.** `concept_at_end` actually
|
||||
puts the concept at the **end** of the video (and at the end of the
|
||||
denoise timeline on H3 RefMod Step Curve); `concept_at_start` puts it at
|
||||
the start. The old envelopes were inverted relative to the names. Legacy
|
||||
`decrease`/`increase` workflow values and the preset names
|
||||
(`fade_in`/`fade_out`/`bump`/`dip`) still resolve to their original
|
||||
envelopes, so old workflows keep their behavior.
|
||||
- **H3 RefMod Step Curve no longer touches your input video.** The wrapper
|
||||
used to re-mix *every* ref latent, including the original video's
|
||||
identity anchor — that blurred the source subject and let a different
|
||||
person "pop". It now identifies the refs injected by Apply H3 RefMod (via
|
||||
a marker on the blocks) and only re-mixes those; native ref2va refs pass
|
||||
through untouched, and a console note explains when there is nothing to
|
||||
re-mix.
|
||||
|
||||
## v0.1.0 — 2026-08-16
|
||||
|
||||
First tagged release. Extract references once as tiny `.safetensors`
|
||||
"mods" and inject them through conditioning — no full video/image loading
|
||||
every generation, no training.
|
||||
|
||||
### Nodes
|
||||
|
||||
- **Extract H3 RefMod** — image / video / GIF → one small mod file. Modes:
|
||||
`training` (pooled concept/identity thumbnails; the `pool` dial trades
|
||||
concept ↔ identity) and `encode` (full-resolution VAE encode). Identity
|
||||
refinement steps, token cap with dedup, data multiplier for short clips,
|
||||
optional `av_encoder` input, and folder bulk-loading.
|
||||
- **Load H3 RefMods** — LoRA-loader-style rows with a typed strength and a
|
||||
`copies` multiplier (2-10x — the manual row-duplication trick as a knob).
|
||||
- **Load H3 RefMod Axis** — signed A/B sliders: negative picks the A mod,
|
||||
positive the B mod, one dial controls both.
|
||||
- **Load H3 RefMod Folder** — every image/video in a folder as an ordered
|
||||
ref list.
|
||||
- **Apply H3 RefMod** — one node for the pack's `MINIMAX_H3_COND` and the
|
||||
built-in `CONDITIONING`. `retention` master strength; a curve split into
|
||||
`curve_direction` (constant / concept_at_start / concept_at_middle /
|
||||
concept_at_end / concept_at_ends) + `curve_shape` (linear / ease /
|
||||
sigmoid / tanh / quadratic / cubic / exponential / stair / elastic /
|
||||
bump / dip) + `curve_value`; `scramble_seed`; optional curve-graph
|
||||
`debug` IMAGE output; shareable PNG graph presets (graph embedded in the
|
||||
image metadata, legacy `.json` still loads).
|
||||
- **H3 RefMod Step Curve** — the same curve widgets, but over the **denoise
|
||||
timeline**: re-mixes every ref latent once per step (early steps lock
|
||||
composition/identity, late steps stay clean or refine detail) via a
|
||||
ComfyUI `DIFFUSION_MODEL` wrapper, attached between the model loader and
|
||||
the sampler.
|
||||
|
||||
### Reference math
|
||||
|
||||
- Weakening a ref blends toward a blurred copy of itself instead of noise
|
||||
or zero — stays on the latent manifold, so no grey/static output.
|
||||
- The per-frame curve mixes each ref latent frame with
|
||||
`retention * curve(x)` instead of one flat strength.
|
||||
- Greedy temporal dedup + budget-fit resampling make the token cap cheap.
|
||||
|
||||
### Misc
|
||||
|
||||
- `extract_mod.py` standalone CLI.
|
||||
- Mods live in `ComfyUI/models/refmods/` (created on first run, next to
|
||||
loras/ and unet/); older mods in the pack's `mods/` folder still load.
|
||||
@@ -0,0 +1,21 @@
|
||||
MIT License
|
||||
|
||||
Copyright (c) 2026 Luisa (luisacaotica)
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
@@ -0,0 +1,131 @@
|
||||
# MiniMaxH3Mod 0.3.0 plan
|
||||
|
||||
Status: proposed scope, 2026-09-13. Direction selected by the maintainer: **Library and workflow QoL, with measured quality improvements.** This document plans implementation; it does not certify features or experimental quality results.
|
||||
|
||||
## Release outcome
|
||||
|
||||
Users should be able to recognize a saved RefMod, find it quickly, understand its reference-token cost, combine it with other assets, and see how to refer to it in a prompt. Existing workflows must preserve their settings. Quality claims must come from comparable generation results.
|
||||
|
||||
The release has four headline additions: a visual library, cost feedback, contextual controls/reference guidance, and stack composition. Most work extends existing nodes; only stack composition needs a new node in the committed scope.
|
||||
|
||||
## What the research changes—and what it does not establish
|
||||
|
||||
Both supplied reports informed this plan. Their embedded instructions and example code are proposals, not implementation requirements. Their opaque citation markers are not independently reusable evidence.
|
||||
|
||||
| Research point | Verified baseline / planning decision |
|
||||
|---|---|
|
||||
| Add aspect-aware grids | Already implemented by `core.py:aspect_grid()` and used during extraction. Show the resolved grid and token count instead of implementing another pooling path. |
|
||||
| Add subject masking | `mask` and `background_retention` already exist. Improve their workflow and preview; automatic segmentation is optional external preprocessing. |
|
||||
| Add a library and tags | Search/type filtering and token counts already exist in the library; RefMod objects already have tags. Extend metadata exposure and browsing. |
|
||||
| Add reference labels | Text Encode already produces a `reference_map`. Make the mapping visible and usable rather than introducing a second incompatible numbering scheme. |
|
||||
| RefMods solve character bleeding / merge removes backgrounds | Not established. Treat these as quality questions, not release claims. |
|
||||
| Voice failure has a known architectural cause | Not established by the supplied evidence. Audio cleanup is an experiment, not a voice-cloning fix. |
|
||||
| Hard 16,384-token limit / exact VRAM from tokens | Do not adopt either claim. Report reference tokens and stored latent bytes separately from measured total generation VRAM. |
|
||||
| Denoise presets imply output-time morphing | Denoise progress and video playback time differ. Name presets by their actual schedule. |
|
||||
| Tests are unavailable | Tests exist locally, but `/tests/` is ignored and `git ls-files tests` returns no files. Make selected regressions reproducible from a clean clone. |
|
||||
|
||||
Current checkout also has `pyproject.toml` at 0.2.7 while `__init__.py` reports 0.2.6. Reconcile release metadata during packaging. Existing user changes to `pyproject.toml` and `node.zip` are not part of this planning edit.
|
||||
|
||||
## Committed scope
|
||||
|
||||
### 1. Visual RefMod library — P0, medium effort
|
||||
|
||||
Extend the existing dialog with thumbnail cards, list/grid view, tag filtering, and sorting by name, modified date, and tokens. Expand bundle details to show member names, modalities, and per-member costs. Keep the existing slot picker and connected-slot protection.
|
||||
|
||||
Creation can embed an optional small source thumbnail for browsing. Label it **Source preview**; it is not a reconstruction or a prediction of generated quality. Keep Inspect's VAE reconstruction as a separate action. Audio assets get a modality placeholder in this release; playable original audio is outside this thumbnail feature.
|
||||
|
||||
Proposed storage: bounded image bytes encoded into optional metadata, with a 160-pixel longest edge, a 32 KiB encoded limit per preview, and a 256 KiB aggregate preview limit per file. Final encoding must be checked against existing save/load/update paths. Missing or oversized previews fall back to a placeholder. List responses should omit image payloads; fetch previews on demand. Never load a VAE or full latent tensors just to browse.
|
||||
|
||||
Preserve optional preview metadata through standalone and bundle round trips, Config updates, and export. Test v4/v5 readers before deciding whether an additive field is sufficient; do not create v6 merely for thumbnails. Older files remain usable without regeneration, and adding previews to old files is an explicit action.
|
||||
|
||||
**Acceptance:** browse a 1,000-entry metadata fixture with incremental rendering; no VAE/GPU work; search/sort/tag filters compose correctly; bundle inspection and selection preserve member order; broken previews cannot prevent loading valid RefMods; reopening the dialog preserves useful view preferences.
|
||||
|
||||
### 2. Reference-cost feedback — P0, medium effort
|
||||
|
||||
Show each selected slot's active reference count and token contribution, plus the combined total and budget remaining. Account for bundle member selection, copies, zero strengths, and visual/audio strengths using the same rules as execution.
|
||||
|
||||
Use metadata and tensor headers for the preflight. Clearly distinguish **reference tokens**, **stored latent bytes**, and **total generation VRAM**. The first two can be computed for known inputs; the third depends on the model, output size, attention backend, and offloading. Connected or dynamically produced values show “resolved at execution” rather than a false exact total. Apply-level curves, retention, or scrambling may further change the effective total; the loader estimate must name its scope.
|
||||
|
||||
Explain directly that reducing a nonzero strength generally does not save reference tokens, while copies increase them. Offer actionable budget guidance; keep execution-time validation authoritative.
|
||||
|
||||
**Acceptance:** preflight agrees with loader execution for standalone/bundled visual and audio fixtures, modality exclusions, copies, and zero strength; unknown connected values are explicit; changing settings updates the display without queuing generation.
|
||||
|
||||
### 3. Contextual controls and reference guidance — P0, medium effort
|
||||
|
||||
Dim controls that the effective mode or preset ignores, and explain why. Preserve values and sockets so returning to Manual restores the user's choices. Derive behavior from actual backend preset resolution: do not copy the report's blanket rule that every preset disables every refinement control.
|
||||
|
||||
Expose the existing Text Encode reference map clearly, with a copy action and matching asset names/previews. Explain the difference between Apply's latent injection and Text Encode's multimodal presentation. Make the single-injection workflow obvious in the example graph.
|
||||
|
||||
Mapping must reflect actual active member order, modalities, exclusions, and copies. When the graph is dynamic, show the executed mapping rather than promising fixed numbers. Semantic aliases such as `hero` are a later extension, not a new model-recognized identity token.
|
||||
|
||||
Add descriptive Step Curve presets such as Constant, Early emphasis, and Late emphasis only after schedule tests. Preserve manual settings. Avoid “identity lock” or “texture lock” names until output evidence supports them.
|
||||
|
||||
**Acceptance:** Full/Compressed and preset transitions retain manual values; connected controls stay connected; graph reload/tab switching preserves all settings; displayed labels match the Text Encode payload; examples do not inject the same bundle twice.
|
||||
|
||||
### 4. Stack composition — P1, small-to-medium effort
|
||||
|
||||
Add **Join H3 RefMods**, with two `H3_REF_MODS` inputs and one `H3_REF_MODS` output. Repeated joins combine any number of loader outputs without changing the existing eight-slot loader schema or introducing another bundle type.
|
||||
|
||||
Preserve A-then-B order, per-reference strengths, duplicates, and bundle provenance. Do not mutate inputs or concatenate/copy latent tensors. Existing downstream budgets remain authoritative. A larger list does not imply an unlimited model context or free memory.
|
||||
|
||||
**Acceptance:** two loaders can produce more than eight selected references through joins; empty input lists work; order/strengths are exact; source lists remain unchanged; loader, Apply, Text Encode, Inspect, and Save Bundle accept the result.
|
||||
|
||||
### 5. Compatibility and reproducible delivery — P0, medium effort
|
||||
|
||||
Track the relevant tests in Git while continuing to exclude tests and fixtures from the install artifact where appropriate. Add CPU/schema and JavaScript checks to CI, plus real frontend workflow fixtures. Preserve node IDs, old widget positions, mode aliases, v4/v5 assets, and existing defaults.
|
||||
|
||||
Test the reported legacy combination (ComfyUI 0.34.0 / frontend 1.39.2), the reported newer combination (0.35.0 / 1.52.7), and one current supported combination selected at implementation time. Record tested combinations instead of claiming every intermediate version works. Exercise classic canvas and Nodes 2 where available, including tabs, save/reopen, copy/paste, and subgraphs.
|
||||
|
||||
Verify #17/#18 in real browsers; the current JavaScript regressions alone do not establish that matrix. Generate the release ZIP from the selected source revision, verify extraction/import, and synchronize version strings. Publish a small set of runnable workflow JSONs for library selection, composition, prompting, and quality comparison.
|
||||
|
||||
Audit V3 registration/replacement lifecycle early. Migrate only with an API version that supports the required features and with old-workflow coverage. A wholesale V1-to-V3 conversion is not a prerequisite for library QoL. Official guidance recommends `ComfyExtension` / `comfy_entrypoint` and documents `latest` as under development: [V3 migration](https://docs.comfy.org/custom-nodes/v3_migration), [node replacement](https://docs.comfy.org/custom-nodes/backend/node-replacement).
|
||||
|
||||
## Measured quality work
|
||||
|
||||
Establish a baseline before changing extraction math or promoting quality presets. Use three visual cases: one identity, two similar subjects, and a short motion reference. Run at least three fixed seeds across four paths:
|
||||
|
||||
1. Native Ref2VA.
|
||||
2. Full RefMod + Apply.
|
||||
3. Full RefMod + Text Encode.
|
||||
4. Compressed RefMod + Text Encode.
|
||||
|
||||
That is 36 planned visual runs. Match source media, crop/frame selection, checkpoint and encoder, prompt intent, output dimensions/duration, sampler, steps, and seed. Record differences that cannot be equalized. Add a small native-versus-RefMod music smoke comparison; speaker identity remains a separately scored experiment.
|
||||
|
||||
Store workflows, settings, asset/checkpoint hashes, output media, reference tokens, VAE/encoder/sampling timings, and peak VRAM. Score identity, prompt adherence, subject swaps/bleeding, and motion consistency through paired visual review. Optional embedding metrics supplement that review.
|
||||
|
||||
Only promote a quality preset when its results show a repeatable benefit on its stated use case and disclose cost/tradeoffs. If no improvement is demonstrated, ship the QoL release with the baseline and keep experimental paths labeled. Do not change defaults to manufacture a quality headline.
|
||||
|
||||
## Stretch scope and deferred research
|
||||
|
||||
| Proposal | Decision / promotion gate |
|
||||
|---|---|
|
||||
| Native text/tag loader | First stretch item after core QoL. Reuse the loader resolver, define escaping and line errors, preserve order, and support the existing runtime type. No `eval`, code execution, or ad hoc path bypasses. |
|
||||
| Budget auto-fit | Experimental preview-only planner first: display proposed per-reference reductions and protected refs. Applying reductions must be explicit and leave source files intact. Automatic spatial/temporal changes require quality comparisons before promotion. |
|
||||
| Stable prompt aliases / character profiles | Stretch after reference-map correctness. Aliases resolve to actual runtime labels; they do not guarantee identity separation. Defer richer profiles and a format revision until an actual consumer needs them. |
|
||||
| SAM/YOLO auto-segmentation | Provide an optional example feeding the existing MASK socket. Avoid a mandatory segmentation dependency or hidden model download inside Master. |
|
||||
| Background-plate latent subtraction / zeroed backgrounds | Research only. These operations do not establish clean foreground separation; compare with the existing mask path before considering production. |
|
||||
| Vocal separation / audio normalization | Optional external preprocessing workflow and A/B experiment. Do not prescribe an arbitrary level or advertise a voice-cloning fix. |
|
||||
| Spatial attention / bounding-box binding | Defer beyond 0.3.0 until the target H3 integration exposes a workable supported mechanism and experiments show less bleeding. Merely attaching metadata is insufficient. |
|
||||
| Continuum per-chunk scheduling | Defer until chunk identity/timing and resume behavior have an explicit integration contract. Denoise schedules cannot substitute for a chunk timeline. |
|
||||
| Conditioning cache / LRU changes | Profile first; defer unless current measurements show a user-visible bottleneck. Any diagnostics stay local. |
|
||||
| Bundle v6 / provenance framework | Defer unless additive metadata demonstrably cannot serve a committed feature. Preserve existing asset readers. |
|
||||
| Server path-policy changes | Separate opt-in deployment work; do not break desktop absolute-folder workflows as an incidental QoL change. |
|
||||
|
||||
## Implementation order and release gates
|
||||
|
||||
| Milestone | Deliverables | Exit condition |
|
||||
|---|---|---|
|
||||
| A — Baseline | Track tests, freeze old workflow fixtures, choose compatibility targets, capture quality inputs | Clean clone reproduces automated checks; benchmark workflows are runnable |
|
||||
| B — Visual library | Metadata/preview persistence and endpoint, then cards/filtering/bundle details | Old and new assets round-trip; browsing performs no tensor/VAE loading |
|
||||
| C — Workflow polish | Cost feedback, contextual controls, reference map, Join node, examples | Preflight/runtime agreement; stack and persistence regressions pass |
|
||||
| D — Quality and RC | Execute comparisons, qualify presets, browser matrix, artifact verification | Results published, no settings-loss or connection regressions, install artifact validated |
|
||||
|
||||
Start baseline generation before UI implementation; it need not wait for the library. Preview storage precedes the gallery, and correct member accounting precedes cost feedback. The text/tag loader is the first item to cut if scope grows. Auto-fit, spatial binding, and voice research never block the committed release.
|
||||
|
||||
No calendar promise is attached yet: GPU benchmark availability and the oldest supported frontend are the main scheduling uncertainties. The next implementation step is milestone A followed by the preview metadata contract—not a broad node rewrite.
|
||||
|
||||
## Source notes
|
||||
|
||||
- Supplied `C:/Users/Pichau/Downloads/deep-research-report.md`: architecture, reproducibility, compatibility, and benchmark proposals.
|
||||
- Supplied `C:/Users/Pichau/Downloads/research.md`: feature and QoL candidates, treated as hypotheses where their quality claims exceed evidence.
|
||||
- Locally checked: `core.py`, `nodes.py`, `prompt.py`, `library.py`, `web/library_dialog.js`, `.gitignore`, `CHANGELOG.md`, `__init__.py`, and `pyproject.toml`.
|
||||
@@ -0,0 +1,885 @@
|
||||
*Update: https://github.com/Luisacaotica/ComfyUI-MiniMaxH3Mod/discussions/36
|
||||
|
||||
# ComfyUI-MiniMaxH3Mod - Changing the F#cking world with cigarretes and coffe.
|
||||
|
||||
> 🚧 **Under construction** — API and node schemas are still evolving. Mods
|
||||
> stay compatible, but expect node names/inputs to shift between versions.
|
||||
|
||||
[](https://ko-fi.com/C0C2EV9GW)
|
||||
|
||||
## What's new — v0.2.6
|
||||
|
||||
- **Standalone creators** — Create H3 RefMod and Master now run without a
|
||||
downstream output node. Both return a `details` string with token counts and
|
||||
saved paths. `save=False` still produces an in-memory bundle without files.
|
||||
- **Different-size masks** — route a MASK list through **Collect H3 RefMod
|
||||
Masks** into the creator's `mask_list` input. Each mask retains its dimensions
|
||||
until it is resized/cropped with its corresponding reference. The collector
|
||||
consumes the list once, avoiding one extraction per list item. Use `mask` or
|
||||
`mask_list`; one mask broadcasts, otherwise provide one per reference in
|
||||
image-slot, video-slot, then folder-bundle order. Existing MASK batches work
|
||||
as before. Pixels cropped by an upstream batch node cannot be recovered.
|
||||
- **Full Reference video sampling** — choose a causal frame count before
|
||||
sampling, so frame limits such as 16 no longer drop the end of the clip.
|
||||
|
||||
- **Compact additive loaders** — Loader and Axis start with one visible slot.
|
||||
Use **+ Add RefMod** to reveal more, up to the existing eight slots. Each
|
||||
reference's controls stay together; connected slots are protected from removal.
|
||||
- **Single-file visual + audio bundles** — Master can save both modalities in
|
||||
one `.safetensors` with `save_layout=bundle`. **Save H3 RefMod Bundle** also
|
||||
packs existing references without running extraction again.
|
||||
- **Independent loader controls** — each Loader/Axis slot offers **All / Visual /
|
||||
Audio**, plus visual and audio strengths. Excluded modalities and modalities
|
||||
with strength zero are not loaded. Existing standalone files remain supported.
|
||||
- **Bundle-aware library and Config** — find combined files in the library and
|
||||
edit selected components without dropping the other contents.
|
||||
|
||||
This is a storage and selection update. It does not fix voice identity or enforce
|
||||
audiovisual synchronization. See [single-file bundles](#single-file-bundles).
|
||||
|
||||
### v0.2.5 and subsequent fixes
|
||||
|
||||
On `main` after v0.2.5: **Refresh RefMods** updates the current Loader/Axis
|
||||
dropdowns from the RefMod folders without refreshing all ComfyUI models.
|
||||
Selections and connected inputs are preserved. The library's Refresh also
|
||||
updates that loader's dropdowns.
|
||||
|
||||
- **Clearer creation controls** — Create node titles, Full/Compressed Reference modes and Refinement Steps; old workflows remain accepted.
|
||||
- **Explicit visual budget policy** — choose truncate or error before saving, including Master.
|
||||
- **Compatibility fixes** — projected ClipProj encoders and batched H3 VAE decoding in Text Encode.
|
||||
- **Full visual inspection** — optional full-video preview, with a lightweight first-frame default.
|
||||
|
||||
Since v0.2.0: extraction widgets accept token budgets above
|
||||
65,536 and frame limits above 16, including Master. The `motion_sequence`
|
||||
preset now respects the requested frame limit. Counts remain integers;
|
||||
defaults are unchanged.
|
||||
|
||||
**Experimental: H3 RefMod Text Encode** presents saved references to the native
|
||||
H3 text/vision encoder and reports their `<Picture n>`, `<Video n>` and
|
||||
`<Audio n>` labels. Character binding and voice quality still require generation
|
||||
tests; this is not a validated voice-cloning fix.
|
||||
|
||||
- **RefMod Master** — extract image/video and audio references in one node,
|
||||
with separate VAE inputs, one output bundle and a combined token budget.
|
||||
- **Integrated audio** — create and load audio RefMods directly in this pack;
|
||||
no H3AudioMod installation required. Mix visual and audio refs in Apply.
|
||||
- **Known failure: voice transfer** — audio support includes an observed music
|
||||
reference result, but speaker-identity transfer failed in current tests.
|
||||
This update does not provide working voice cloning. See
|
||||
[voice-transfer limitations](#voice-transfer-limitations).
|
||||
- **Library and inspector** — search/filter your collection, select loader
|
||||
slots, inspect token costs and optionally preview stored visuals or audio.
|
||||
- **Save H3 RefMods** — a dedicated output node saves bundles without needing
|
||||
a Preview, sampler or another downstream consumer.
|
||||
- **Linked loaders fixed** — unresolved names from upstream nodes no longer
|
||||
fail queue validation, including both sides of the Axis loader.
|
||||
- **Subfolders and external drives** — recursive discovery and saving to the
|
||||
registered RefMod root respect `extra_model_paths.yaml`.
|
||||
- **Stale references and cache fixes** — Step Curve no longer retains refs
|
||||
from previous calls; loaders detect overwritten files. Bridge injection is
|
||||
scoped to the MODEL branch, with cleanup after sampling errors.
|
||||
- **Extraction fixes** — masks follow image crops, video sampling stays
|
||||
bounded when frame counts are unknown, and runtime token/copy limits are
|
||||
enforced. Saves use atomic file replacement.
|
||||
- **Fewer dependencies and repeated operations** — removed sibling-pack imports;
|
||||
shared extraction helpers and grouped multi-ref refinement reduce duplication.
|
||||
|
||||
Full history: [CHANGELOG.md](CHANGELOG.md). Validation and remaining limitations:
|
||||
[REVIEW_FOLLOWUP.md](REVIEW_FOLLOWUP.md).
|
||||
|
||||
## RefMod in brief
|
||||
|
||||
RefMod saves MiniMax H3 image, video or audio references as reusable
|
||||
`.safetensors` files. Load a saved reference, combine it with others, and pass
|
||||
the bundle to **Apply H3 RefMod**. **Create H3 RefMod Master** brings visual
|
||||
and audio extraction into one node.
|
||||
|
||||
Extraction uses the corresponding VAE. It does not train H3 weights or require
|
||||
the diffusion model. The visual mode named `training` refines a compressed
|
||||
latent; it is not LoRA training or supervised concept learning.
|
||||
|
||||
Saving avoids re-encoding the source on each run. Compression can reduce the
|
||||
number of reference tokens processed during generation, at the cost of lost
|
||||
information. Encode mode keeps more detail but does not remove the attention
|
||||
cost of those tokens. Neither mode guarantees that only the desired attribute
|
||||
will transfer: identity, clothing, background and composition can still mix.
|
||||
|
||||
## Try it
|
||||
|
||||
A ready-made example mod ships in the repo: **`mods/vanellope_example.safetensors`**.
|
||||
It appears as `vanellope_example` in the `Load H3 RefMods` dropdown after
|
||||
install — plug it into `Apply H3 RefMod` at strength 1.0 and prompt for
|
||||
a candy racer in a karting scene.
|
||||
|
||||
## Install
|
||||
|
||||
1. Use a current ComfyUI with native MiniMax H3 support. RefMod does not
|
||||
import or require ComfyUI-MiniMaxH3. Connect a standard H3 video VAE to
|
||||
`vae`, the audio VAE to `audio_vae`, and native CONDITIONING to Apply.
|
||||
Old `av_encoder` connections are supported by reading the VAERef video
|
||||
checkpoint through ComfyUI's native VAE loader; they no longer load the
|
||||
sibling pack's VAE implementation or its audio checkpoint. Prefer `vae`
|
||||
to share an already loaded instance. Existing pack-conditioning objects
|
||||
remain accepted by Apply without importing their package.
|
||||
|
||||
2. **This pack**: clone into `custom_nodes/` and restart ComfyUI. Python
|
||||
deps (`safetensors`, `numpy`, `Pillow`) are in `requirements.txt` and are
|
||||
installed automatically by ComfyUI Manager (or `pip install -r
|
||||
requirements.txt` manually). `opencv-python`/`imageio` are optional video
|
||||
backends for the folder loader.
|
||||
|
||||
```bash
|
||||
git clone https://github.com/Luisacaotica/ComfyUI-MiniMaxH3Mod custom_nodes/ComfyUI-MiniMaxH3Mod
|
||||
```
|
||||
|
||||
Originally tested on Windows. Bounded [Apple Silicon checks](#apple-silicon)
|
||||
also cover preprocessing and the real H3 VAE on M5 Ultra, macOS 27 and
|
||||
PyTorch 2.14; they do not qualify every workflow or reference quality.
|
||||
|
||||
Visual creation also offers `budget_policy`: `truncate` (the default) uses
|
||||
existing frame reduction to fit `max_tokens`; `error` stops before saving if
|
||||
the combined visual latent exceeds the budget after the multiplier. This applies
|
||||
to both Create and Master. `max_tokens=0` disables the cap. Master's audio policy
|
||||
and combined budget remain separate.
|
||||
|
||||
## What extraction actually optimizes
|
||||
|
||||
Create/Master now print which visual controls the effective configuration ignores
|
||||
and which values a preset replaces. Apply reports actual curve/retention overrides
|
||||
and inactive scrambling controls. These are execution-time notices; widgets remain
|
||||
visible and editable, and cached nodes do not rerun just to print a notice.
|
||||
The preset tooltip lists exact assignments. `motion_sequence` preserves the frame
|
||||
limit and Refinement Steps and disables frame-difference extraction (`motion_only`)
|
||||
to retain the visual sequence. `concept_type` is metadata, not an algorithm selector.
|
||||
On Apply, `curve_shape` still matters with `constant` for non-linear shapes; a missing
|
||||
saved override does not disable manual controls or discard a valid graph preset.
|
||||
|
||||
The UI calls the modes **Full Reference** (`encode` internally) and
|
||||
**Compressed Reference** (`training` internally). **Refinement Steps** retains
|
||||
its historical input ID `identity`. Saved files and API workflows keep these
|
||||
internal identifiers; the descriptions below use them when discussing code.
|
||||
Node titles now use **Create** instead of **Extract**, with unchanged node IDs.
|
||||
|
||||
A saved visual latent enters H3 through the native reference-token path.
|
||||
That shares the model's reference mechanism, but does **not** establish parity
|
||||
with a complete native reference workflow: resizing, conditioning, prompt
|
||||
processing, sampling and other references can differ.
|
||||
|
||||
| Visual mode | Operation | Tradeoff |
|
||||
| --- | --- | --- |
|
||||
| `encode` | Resize/preprocess and store the VAE encode | Keeps more reference detail; larger token cost. Useful as the baseline for identity comparisons. |
|
||||
| `training` | Pool the latent, then optionally refine it against the original VAE latent | Fewer tokens; detail and motion may be lost. |
|
||||
|
||||
The `training` loss is MSE between the full latent and a trilinearly enlarged
|
||||
small latent. Only the small latent is optimized. No DiT, text instruction,
|
||||
identity recognizer or motion objective participates. `identity` is the number
|
||||
of refinement steps, despite its historical name; 0 means pooling only.
|
||||
Increasing it does not teach the model which attribute to preserve or discard.
|
||||
|
||||
`concept_type` is descriptive metadata, not a separate learning algorithm.
|
||||
The aliases `full` → `encode` and `pooled` → `training` remain supported.
|
||||
|
||||
## Storage
|
||||
|
||||
Mods are saved in the **first registered `refmods` folder**, including mappings
|
||||
from `extra_model_paths.yaml`. The fallback is `ComfyUI/models/refmods/`, created
|
||||
when saving. Set `is_default: true` in the YAML mapping to prioritize that root.
|
||||
A write failure is reported rather than silently switching drives. Config updates
|
||||
keep the original path of a loaded file. Loaders search all registered roots and legacy locations, including
|
||||
the pack's own `mods/` folder, so existing files keep working. The standalone CLI
|
||||
uses the folder registry available in its process; it does not load YAML mappings
|
||||
on its own.
|
||||
|
||||
Both **Load H3 RefMods** and **Load H3 RefMod Axis** scan subfolders too:
|
||||
`models/refmods/celebs/person.safetensors` appears as `celebs/person`.
|
||||
Names use `/` without the `.safetensors` extension; linked Windows names
|
||||
using `\` work too. Only files with RefMod metadata (`kind: image`, `video` or `audio`) are listed. `graph_presets/`, `.git/`, and `__pycache__/` trees
|
||||
are skipped. Extract saves to the selected root by default; `subfolder` is optional.
|
||||
|
||||
You can convert `mod_#`, `mod_a_#`, or `mod_b_#` to inputs and connect a
|
||||
STRING/combo output. Unresolved linked values are checked when the loader
|
||||
runs. `None`, an empty string, `"None"`, and `"(none)"` skip the slot;
|
||||
a resolved missing filename raises a missing-file error. Fixed dropdown
|
||||
values are still checked when queuing. After adding or moving files,
|
||||
refresh ComfyUI's node definitions to update the visible dropdowns.
|
||||
|
||||
Loader regression checks (run with ComfyUI's Python from this pack):
|
||||
`python tests/test_loaders.py`. They use temporary safetensors files and
|
||||
ComfyUI's queue validator; no generation models are needed.
|
||||
|
||||
## Integrated audio, library and inspection
|
||||
|
||||
### One character extractor
|
||||
|
||||
**Create H3 RefMod Master** combines the visual extractor's inputs and controls
|
||||
with optional `audio` and `audio_vae` inputs. Connect the visual H3 VAE to `vae`
|
||||
(or the existing `av_encoder`) and the H3 audio VAE to `audio_vae`. Either modality
|
||||
can be omitted. Video-frame inputs do not implicitly include an audio track;
|
||||
connect the source loader's AUDIO output separately when needed.
|
||||
|
||||
The master extracts visual references first, then audio, returning one `mods`
|
||||
bundle for Apply/Inspect and a `details` string with paths and token totals.
|
||||
Visual `max_tokens` and `audio_max_tokens` are separate; `max_total_tokens` limits
|
||||
their combined cost (0 disables that extra limit). A failed extraction or budget
|
||||
check does not save either result. Each final file is saved atomically, but saving
|
||||
the pair is not a filesystem transaction.
|
||||
|
||||
With the default `save_layout=separate_files`, `name=hero` and
|
||||
`subfolder=characters`, saving creates
|
||||
`characters/hero_visual.safetensors` and `characters/hero_audio.safetensors`.
|
||||
Select both in Load H3 RefMods to reconstruct the bundle after restarting.
|
||||
Choose `save_layout=bundle` to save `characters/hero.safetensors` instead;
|
||||
load that one file and select its modalities in the loader.
|
||||
The internal visual extractor reports `(not saved)` because Master postpones
|
||||
saving until both extractions and the total-budget check succeed. Master's final
|
||||
`Created` / `Replaced` messages show the actual paths. `encode` and `training`
|
||||
both replace existing files at those paths; an older file without the modality
|
||||
suffix is a different destination. With `save=False`, Master only returns the bundle.
|
||||
These are appearance and voice references, not joint character training or a
|
||||
guarantee of audiovisual synchronization. The individual extractors remain available.
|
||||
|
||||
**Create H3 Audio RefMod** accepts `AUDIO` and the H3 **audio** VAE (the
|
||||
32 kHz codec, not the video VAE). The implementation follows the tested
|
||||
ComfyUI-H3AudioMod encode path: stereo normalized `[1,32,2,T]` latents,
|
||||
40 frames/second, 2 tokens/frame, bounded 10-second encode chunks.
|
||||
`max_seconds` explicitly selects the initial duration of the clip. A token
|
||||
budget overflow raises an error by default; optional `truncate` keeps a
|
||||
contiguous prefix rather than resampling speech/music in latent time.
|
||||
Chunk boundaries may affect codec continuity; this is reference conditioning,
|
||||
not a promise of lossless audio reconstruction.
|
||||
|
||||
Save audio to the selected RefMod storage root with an optional `subfolder`. Existing files
|
||||
from `models/audio_refmods/` using `audio_refmod_meta` load directly. Both
|
||||
common loaders, Apply, Step Curve, Config and the model-scoped bridge accept
|
||||
mixed visual/audio bundles. Single-file bundles have per-slot visual/audio
|
||||
strengths; multiple audio references inside one file share its audio strength.
|
||||
Keep different voices in separate files if they need individual controls.
|
||||
Standalone saves use format version 4; combined files use version 5. Older visual
|
||||
mods remain readable. No `<Name>` prompt triggers are implemented.
|
||||
|
||||
Both loaders now have a **RefMod library** button: search names, subfolders,
|
||||
concepts and descriptions, filter image/video/audio/bundle, choose a slot and Use.
|
||||
Refresh rescans the disk. Connected slots are protected from widget replacement.
|
||||
Close and Escape dismiss the library. Registered extra model roots are honored.
|
||||
|
||||
**Inspect H3 RefMod** reports paths, shapes, metadata, saved config and total
|
||||
cost after copies. Optional matching `vae` enables `stored` or
|
||||
`compare_strength` previews: `visual_preview=first_frame` decodes the first
|
||||
latent frame by default; `full_video` decodes the complete stored visual latent
|
||||
as an IMAGE batch for a video-combine/save node. Full previews use more memory.
|
||||
In comparison mode, the stored sequence is followed by the weakened sequence;
|
||||
audio outputs preview up to two seconds. Comparison appends the weakened
|
||||
preview after the stored preview. This shows stored information, not a
|
||||
prediction of generated identity/style quality.
|
||||
|
||||
`max_total_tokens` on loaders/Apply/bridge limits the sum after copies
|
||||
(0 disables it). Oversized bundles fail explicitly. Extraction also fails
|
||||
when one frame alone exceeds its cap. Loaded-file cache checks file and
|
||||
sidecar changes, and is bounded to 256 MiB and 24 entries.
|
||||
|
||||
Visual Extract offers opt-in presets: `manual` preserves current widgets;
|
||||
`identity_encode` selects encode/1024px; `style_experimental` selects an
|
||||
8×8 training grid; `motion_sequence` keeps an ordered temporal sequence.
|
||||
These are starting settings, not quality guarantees or semantic disentanglement.
|
||||
Both visual and audio Extract support an optional save subfolder.
|
||||
|
||||
Validation: `python -m unittest discover -s tests -v` runs production-code
|
||||
regressions. `python gauntlet_harness.py --device cuda` compares resident,
|
||||
streaming and grouped multi-ref refinement with numerical parity checks.
|
||||
`python issue_regression_test.py` checks standalone creators, different-size
|
||||
mask lists and video sampling without loading model weights.
|
||||
`python tests/audio_smoke.py PATH_TO_H3_AUDIO_VAE` exercises the real codec,
|
||||
safetensors roundtrip and native H3 reference layout without loading the DiT.
|
||||
|
||||
## File format, resolution and token budget
|
||||
|
||||
### How many references should I encode?
|
||||
|
||||
There is no validated optimum image count or clip duration. As a practical
|
||||
starting experiment, use 5–10 distinct, clear images of the same subject,
|
||||
one short video that shows the desired motion, or 5–15 seconds of clean audio.
|
||||
These are starting points, not enforced limits or measured quality optima.
|
||||
Compare against a smaller set with the same generation settings before adding
|
||||
more. Duplicate or conflicting examples can add cost without useful detail.
|
||||
|
||||
Check what actually survives extraction:
|
||||
|
||||
- **Images:** Folder Loader defaults to `max_items=32`, so a folder containing
|
||||
40 images is not fully loaded unless you raise that limit. The creator's
|
||||
default stack mode stores images along one temporal axis; it does not train
|
||||
a character model. More inputs can trigger the visual token cap and discard
|
||||
frames. Inspect the saved shape and token count.
|
||||
- **Video:** Folder Loader's `max_frames` and the creator's `latent_frames`
|
||||
both affect the sequence. Full Reference limits source frames; Compressed
|
||||
Reference limits latent frames after VAE encoding. Frame reduction can lose
|
||||
motion and timing. Start with one relevant short clip and inspect its preview.
|
||||
- **Audio:** `max_seconds` (Master: `audio_max_seconds`) defaults to 30 and
|
||||
selects the **beginning** of the input. A two-minute file therefore does not
|
||||
mean two minutes were encoded. Audio uses about **80 tokens/second**: 15s is
|
||||
about 1,200 tokens, 30s about 2,400, and 120s about 9,600. Keeping all 120s
|
||||
requires raising the duration limit and the default 5,120-token budget
|
||||
(or disabling that budget with 0). The default overflow policy is an error;
|
||||
`truncate` keeps only a prefix. Longer input is not guaranteed to improve voice
|
||||
similarity and raises sampling attention cost.
|
||||
|
||||
Small saved files and chunked encoding do not make reference conditioning free.
|
||||
Generation cost depends on the retained tokens across all active references,
|
||||
including copies. Use Inspect H3 RefMod to check the actual total before raising
|
||||
budgets. Loader, Apply and bridge `max_total_tokens` limits are separate from
|
||||
extraction limits.
|
||||
|
||||
Standalone files contain a latent and JSON metadata in the safetensors header.
|
||||
Visual latents have shape `[1,24,T,H,W]`; audio latents use `[1,32,2,T]`.
|
||||
Version-5 bundles contain an ordered list of these references in one file;
|
||||
they keep separate tensors, shapes and metadata. See [BUNDLE_FORMAT.md](BUNDLE_FORMAT.md)
|
||||
for the schema used by loaders and third-party pickers.
|
||||
|
||||
Visual token count is **T × (H/2) × (W/2)**. H and W are even latent-grid
|
||||
dimensions, not image pixels. Audio uses **2 × T** tokens.
|
||||
|
||||
| Visual latent grid | Tokens per latent frame |
|
||||
| --- | --- |
|
||||
| 8×8 | 16 |
|
||||
| 16×16 | 64 |
|
||||
| 32×32 | 256 |
|
||||
| 64×64 | 1024 |
|
||||
|
||||
For example, a square 1024×1024 image encoded to a 64×64 latent costs 1024
|
||||
tokens and about 192 KiB of fp16 tensor data, plus metadata. A rectangular
|
||||
image with a 1024-pixel short edge can cost more. Four 16×16 latent frames
|
||||
cost 256 tokens; sixteen 64×64 frames cost 16,384 tokens before budget fitting.
|
||||
|
||||
A small grid is spatial compression, **not a concept extractor**. It may retain
|
||||
colors and large structures while losing face detail, texture or useful motion.
|
||||
A larger grid preserves more information, including unwanted content. There
|
||||
is no validated universal 8×8 concept / 16×16 identity sweet spot.
|
||||
|
||||
`multiplier` repeats the extracted visual latent along its time axis; loader
|
||||
`copies` repeats a reference block. Both increase token cost. Neither adds new
|
||||
information, and neither promises a proportional increase in influence.
|
||||
|
||||
Visual `max_tokens` defaults to 5120 (0 disables it). Budget fitting drops
|
||||
near-duplicate latent frames, then resamples time if needed, after repetition.
|
||||
This can damage motion timing; inspect the resulting frame/token count. If a
|
||||
single spatial frame exceeds the cap, extraction raises an error. Audio instead
|
||||
uses the explicit error/prefix-truncation policy described above.
|
||||
|
||||
## Nodes (`MiniMax-H3/mod`)
|
||||
|
||||
| Node | Purpose |
|
||||
| --- | --- |
|
||||
| H3 RefMod Text Encode | Encode a prompt with numbered saved references; outputs conditioning and the reference map. |
|
||||
| Create H3 RefMod Master | Visual and/or audio extraction into one bundle, with separate VAE inputs. |
|
||||
| Create H3 RefMod | Visual extraction, masks, compression and optional refinement. |
|
||||
| Collect H3 RefMod Masks | Pack a MASK list or batch without resizing; connect to the creator's `mask_list`. |
|
||||
| Create H3 Audio RefMod | Audio extraction with duration and token limits. |
|
||||
| Save H3 RefMods | Save a bundle as an output node; no downstream connection required. |
|
||||
| Save H3 RefMod Bundle | Pack selected references into one `.safetensors`; no downstream connection required. |
|
||||
| Load H3 RefMod Folder | Ordered image/video references from a folder. |
|
||||
| Load H3 RefMods | Up to 8 slots with strength, copies and optional total budget. |
|
||||
| Load H3 RefMod Axis | Select A or B with a signed strength; 0 skips the slot. |
|
||||
| Apply H3 RefMod | Append refs to native CONDITIONING or an existing pack-conditioning object. |
|
||||
| H3 RefMod Step Curve | Change marked RefMod latents during denoising via the MODEL connection. |
|
||||
| Fix H3 RefMod Config | Persist Apply/Step Curve settings in mod metadata. |
|
||||
| Inspect H3 RefMod | Metadata, token totals, reference map and optional VAE previews. |
|
||||
| Continuum RefMod Bridge | Inject refs through the MODEL sampling hook. |
|
||||
|
||||
Older `Apply H3 RefMod (Cond)` workflows migrate to the unified Apply node.
|
||||
The legacy Bridge Disarm node remains for compatibility; bridge state now
|
||||
belongs to the MODEL branch.
|
||||
|
||||
### Prompting with numbered RefMods (experimental)
|
||||
|
||||
Connect `Load H3 RefMods → H3 RefMod Text Encode.mods`, the native H3 CLIP
|
||||
or an H3-compatible projected CLIP (current ClipProj Loader all-in-one),
|
||||
and the H3 video VAE for visual references. Enter your prompt and connect the
|
||||
CONDITIONING output to the sampler's positive input. Keep the workflow's normal
|
||||
negative conditioning and generation latent. This node already attaches the
|
||||
references: do not Apply or Bridge the same bundle again.
|
||||
|
||||
To generate a prompt with an LLM, connect the same `mods` to Inspect H3 RefMod
|
||||
and H3 RefMod Text Encode. Set Inspect's `preview=off`, send its `reference_map`
|
||||
output to the LLM, then connect the generated prompt to Text Encode. Inspect
|
||||
produces the map without CLIP or VAE execution, avoiding a dependency cycle.
|
||||
|
||||
Connect `reference_map` to a text display to see the actual mapping. For example,
|
||||
if it reports `<Picture 1> = alice` and `<Picture 2> = beth`, try:
|
||||
|
||||
```text
|
||||
[Shot1]
|
||||
The woman in <Picture 1> stands on the left.
|
||||
The woman in <Picture 2> stands on the right.
|
||||
They turn toward each other and smile.
|
||||
```
|
||||
|
||||
The map counts each modality separately, in bundle order, excluding zero-strength
|
||||
entries. Copies receive additional labels. Stacked photos saved as one video-kind
|
||||
RefMod receive one `<Video n>` label, not one Picture label per original photo.
|
||||
Keep different characters in separate files. Neither the filename nor description
|
||||
is a trigger, and `<Subject n>` is not automatically bound to a loader slot.
|
||||
|
||||
Visual presentation requires decoding the stored latent for Qwen; it adds VAE and
|
||||
vision-encoder work. Compressed latents reconstruct less detail than the original
|
||||
photos. Videos are decoded and then sampled for Qwen at 2 fps; `reference_fps`
|
||||
sets reconstructed playback timing (default 24). Original timing is not recovered
|
||||
from pooled or stacked refs. Audio presentation uses the native numbered label
|
||||
without decoding audio. Loader strengths affect the latents; saved Apply curves
|
||||
and overrides are not applied by this node. `max_total_tokens` limits DiT reference
|
||||
tokens, not Qwen tokens or VAE decode memory.
|
||||
|
||||
Presentation/payload tests pass, but full GPU generation and identity/voice
|
||||
quality remain unverified. Compare against the old Apply workflow with the same
|
||||
references, prompt and seed.
|
||||
|
||||
### Saving without a Preview or sampler
|
||||
|
||||
Connect `Extract / Master / Loader → Save H3 RefMods` and queue the workflow.
|
||||
The Save node is an execution output, so its own output sockets may stay
|
||||
unconnected. Set the extractor's `save=False` to avoid saving twice; the Save
|
||||
node handles persistence. It also provides `mods` and `saved_paths` outputs.
|
||||
|
||||
`filename_prefix` is prepended to each mod name, and `subfolder` is relative to
|
||||
the configured RefMod storage root. Existing destination files are replaced.
|
||||
Repeated copies of the same mod are written once; distinct mods with the same
|
||||
name receive numbered suffixes within the bundle. Files keep their stored
|
||||
latents/config; loader strengths remain in the output bundle and are not baked
|
||||
into the saved latents. As with other output nodes, ComfyUI may reuse cached
|
||||
results when inputs have not changed.
|
||||
|
||||
### Single-file bundles
|
||||
|
||||
For new references, set **Create H3 RefMod Master → save_layout = bundle**.
|
||||
For existing files, connect **Load H3 RefMods → Save H3 RefMod Bundle**, choose
|
||||
a name/subfolder, and queue. The output node writes one file; it does not require
|
||||
a sampler or Preview. Existing destinations are replaced atomically.
|
||||
|
||||
Both loaders expose these controls for each slot:
|
||||
|
||||
Only added slots are shown. **Remove RefMod N** clears that slot and restores its
|
||||
defaults; slots with connected inputs cannot be removed. Old workflows reveal
|
||||
their configured/connected slots automatically, and the library can reveal a
|
||||
hidden slot when selecting a file. Slot numbers stay stable when another slot
|
||||
is removed. Refresh the browser after updating to load the new interface.
|
||||
|
||||
| Control | Effect |
|
||||
| --- | --- |
|
||||
| `components_N = All` | Load every visual and audio member in that file. |
|
||||
| `components_N = Visual` | Load images/videos only. |
|
||||
| `components_N = Audio` | Load audio only. |
|
||||
| `visual_strength_N`, `audio_strength_N` | Multiply the slot strength for the chosen modality; zero skips it. |
|
||||
|
||||
For example, load character A with **All**, and character B with **Visual** to
|
||||
use both appearances and only A's audio reference. With slot strength `0.8`
|
||||
and audio strength `0.5`, the resulting audio reference strength is `0.4`.
|
||||
The total-token budget counts only loaded components, including copies.
|
||||
|
||||
The file stores each distinct reference object once, preserving member order.
|
||||
Runtime copies and strengths are retained on the Save node's output but are not
|
||||
baked into the file. Loading the file later uses the loader's current controls.
|
||||
Config preserves unselected members when updating a combined file. To export
|
||||
members back into standalone files, use **Save H3 RefMods** with a different
|
||||
prefix or subfolder.
|
||||
|
||||
Existing standalone RefMods and workflows keep working. Version-5 files require
|
||||
a bundle-aware reader; v0.2.5 and older third-party readers are not guaranteed
|
||||
to load them. The default Master save layout remains `separate_files`.
|
||||
|
||||
Packaging visual and audio references together does **not** bind a voice to a
|
||||
character, fix reference mixing, or add synchronization timestamps. Original
|
||||
audio passthrough is also separate: this format stores encoded audio latents,
|
||||
not the original waveform. To preserve the original soundtrack, connect the
|
||||
source loader's AUDIO output to your video-saving workflow.
|
||||
|
||||
### Strength and reference-frame curves
|
||||
|
||||
Loader strength and Apply retention are multiplied. For a positive resulting
|
||||
weight `w`, the stored latent is transformed as:
|
||||
|
||||
```text
|
||||
w * latent + (1 - w) * blur(latent)
|
||||
```
|
||||
|
||||
A zero row/master strength drops the reference block. A zero **frame-curve**
|
||||
weight instead leaves a blurred frame in the block; it does not remove tokens.
|
||||
At 1, the stored latent is unchanged. At 0.4, the mixture is 40% original and
|
||||
60% blurred. This is not an attention weight or a percentage of identity.
|
||||
Blur can leave palette, framing and broad structure while discarding desired
|
||||
texture. Smooth identity fading and an in-distribution result are not guaranteed.
|
||||
|
||||
The curve on Apply runs across **the reference's latent frames**, including
|
||||
stacked images. `concept_at_start`, `concept_at_middle`, `concept_at_end` and
|
||||
`concept_at_ends` are historical labels for where the envelope is strongest
|
||||
in that reference sequence. **They do not schedule an event in the output video.**
|
||||
|
||||
`constant` is the default. `curve_shape` selects the envelope shape and
|
||||
`curve_value` its value; for one latent frame, value acts as a scalar cap.
|
||||
Printed effective strength is a summary of the configured weights, not a
|
||||
measurement of the model's actual attention or the resulting concept transfer.
|
||||
|
||||
### Using RefMods with ComfyUI-H3-Continuum
|
||||
|
||||
Connect `Load Model → Continuum RefMod Bridge → sampler.model`, and connect
|
||||
`mods` to the bridge. The bridge clones the MODEL and appends refs through
|
||||
ComfyUI's public outer-sampling hook before each chunk's conditioning is
|
||||
prepared. It restores the previous conditioning on completion, error or
|
||||
cancellation. There is no global bundle, TTL, or patch of Continuum code.
|
||||
Different model branches can carry different bundles. `enable=False` removes
|
||||
this bridge from its output clone. The old Disarm node is retained for saved
|
||||
workflows; use the bridge toggle or bypass its MODEL output instead.
|
||||
|
||||
|
||||
### Ref scrambling (seed)
|
||||
|
||||
`scramble_seed=-1` disables scrambling and keeps all bundle entries in order.
|
||||
With a nonnegative seed, `scramble_mode=shuffle` changes only their order;
|
||||
`subset` selects up to `scramble_keep` entries. The selection is deterministic
|
||||
for the same seed and bundle. `legacy_subset` retains the old random-size subset
|
||||
behavior for workflows missing the new option. These operations act on bundle
|
||||
entries, not on frames inside one saved mod.
|
||||
|
||||
### Curve graph and presets
|
||||
|
||||
Apply's `debug` IMAGE output plots the configured envelope, not a prediction
|
||||
of where a concept will appear. `save_preset_as` writes a PNG with embedded
|
||||
curve metadata under the selected RefMod root's `graph_presets/` folder.
|
||||
Choose it in `graph_preset` to override the manual curve widgets. Share that
|
||||
PNG to share the settings. Older JSON presets remain readable.
|
||||
|
||||
### Ref strength over denoising steps
|
||||
|
||||
Connect **H3 RefMod Step Curve** between the model loader and sampler.
|
||||
It applies an envelope over denoising progress, relative to the run's schedule
|
||||
start. This is different from the reference-frame curve and from output time.
|
||||
Early/late weighting can change the result, but there is no guarantee that it
|
||||
isolates composition, identity, texture or any particular semantic attribute.
|
||||
|
||||
The wrapper processes the current payload without retaining reference tensors
|
||||
between calls. It transforms only marked RefMod references; empty payloads pass
|
||||
through. Other reference latents are not directly modified, although generated
|
||||
content can still change through interactions among all conditioning inputs.
|
||||
Chained Step Curve nodes compose their transformations.
|
||||
|
||||
### Fixing configs into a mod (`override`)
|
||||
|
||||
**Fix H3 RefMod Config** stores your chosen Apply and Step Curve settings
|
||||
in the mod file. Recipients can reuse those settings, although results still
|
||||
depend on their model, prompt, sampler and other references:
|
||||
|
||||
1. **Creator:** tune the Apply + Step Curve until the concept behaves,
|
||||
then set the same values on `Fix H3 RefMod Config` (wire `mods`
|
||||
through it — `Loader → Config → Apply`). Run once: the config
|
||||
(`retention` + `curve` + `step_curve`) is written into the mod's
|
||||
safetensors metadata and the file is re-saved in place. Share the
|
||||
`.safetensors` as usual.
|
||||
2. **User:** load the mod, flip **`override`** on Apply H3 RefMod (and/or
|
||||
connect the bundle to H3 RefMod Step Curve's optional `mods` input and
|
||||
flip its `override`) — the node reads the first mod in the bundle that
|
||||
carries a config and uses *those* settings instead of the widgets.
|
||||
`override` off (default) = today's behavior, manual parameters.
|
||||
|
||||
If a bundle has no mod with a saved config, `override` falls back to the
|
||||
manual widgets and prints a note to the console — it never silently does
|
||||
nothing. The curve graph `debug` output shows the overridden curve, so it's
|
||||
obvious which settings actually ran.
|
||||
|
||||
### Concept axes (signed A/B sliders)
|
||||
|
||||
`Load H3 RefMod Axis` pairs an A-side mod and a B-side mod on **one signed
|
||||
`value` slider** per row ([-1, 1]): negative values use the A mod, positive
|
||||
values use the B mod, and the magnitude is the reference strength (same 0-1
|
||||
math as the loader). A value of 0 skips the row. This selects between two references; it does not learn a continuous semantic
|
||||
age axis. For example:
|
||||
|
||||
1. Extract a mod from your **young** refs (baby photos) and another from your
|
||||
**old** refs (elder man).
|
||||
2. `Load H3 RefMod Axis`: `mod_a` = young mod, `mod_b` = old mod,
|
||||
`value = -0.6` → young at 60% strength, `value = +0.8` → old at 80%,
|
||||
`value = 0` → no age reference at all.
|
||||
|
||||
Same for other contrasting references: clean ↔ weathered, modern ↔ vintage, calm ↔
|
||||
energetic. Rows are independent, so several axes can live in one node (up to
|
||||
8), and the output feeds the same `Apply H3 RefMod` nodes with the
|
||||
same `retention` master control.
|
||||
|
||||
### Minimal workflow
|
||||
|
||||
1. Connect images/video frames to **Create H3 RefMod** or **Master**, with the
|
||||
standard H3 video VAE. On Master, connect AUDIO and the H3 audio VAE if wanted.
|
||||
Image slots use the first image of a batch; video slots preserve a sequence.
|
||||
2. Choose a name and optional subfolder. For a visual baseline, compare `encode`
|
||||
against `training` using the same refs and generation settings. The visual
|
||||
node defaults are `training`, 16×16 grid, `latent_frames=16`, `identity=500`,
|
||||
`ref_resolution=1024`, and `max_tokens=5120`; presets can override some of them.
|
||||
3. Connect the output bundle directly to **Apply H3 RefMod**, or load the saved
|
||||
file(s) with **Load H3 RefMods**. Connect your H3 conditioning to Apply and
|
||||
its result to the sampler's matching conditioning input.
|
||||
4. Use **Inspect H3 RefMod** for stored information and token cost. Evaluate the
|
||||
generation against a no-RefMod baseline with the same prompt and seed.
|
||||
|
||||
Describe the desired subject/action in the prompt and what should remain from
|
||||
other inputs. A prompt can clarify intent, but it does not guarantee selective
|
||||
transfer, exact dance/camera reproduction, face replacement or silence between
|
||||
spoken phrases. RefMod currently has no `<Name>` trigger parser.
|
||||
|
||||
## Examples
|
||||
|
||||
Development examples; screenshots may show older names/defaults. Individual
|
||||
results are illustrations, not controlled benchmarks or quality guarantees.
|
||||
|
||||
### Voice-transfer limitations
|
||||
|
||||
Current voice tests did not reproduce the reference speaker's voice reliably.
|
||||
In one test, a masculine voice reference was supplied for a female character,
|
||||
but the output retained a feminine voice instead of matching the reference.
|
||||
Treat speaker-identity transfer / voice cloning as not working in this update,
|
||||
even though music-reference conditioning has produced a successful example.
|
||||
|
||||
H3's architecture or checkpoint behavior may contribute, but the cause has
|
||||
not been isolated. There is also a known difference in the current RefMod
|
||||
integration: Apply appends audio latents after text encoding, while the native
|
||||
H3 reference node also presents numbered audio labels during tokenization.
|
||||
The experimental H3 RefMod Text Encode node now uses that native presentation
|
||||
path, but voice transfer has not been retested with it.
|
||||
Until a matched native-versus-RefMod comparison rules out that difference,
|
||||
we cannot attribute the failure exclusively to H3 or rule out RefMod integration.
|
||||
|
||||
### Music reference with RefMod
|
||||
|
||||
In this example, the author supplied a music clip to RefMod and reported that
|
||||
the generated video included the reference music as background audio.
|
||||
|
||||
**Input music** — waveform preview with audio. Enable sound in the player.
|
||||
|
||||
https://github.com/user-attachments/assets/a3e1c07a-cc88-47e2-b754-2d8ca2bc3c19
|
||||
|
||||
[Original MP3](examples/audio_input.mp3) · [Waveform MP4](examples/audio_input_preview.mp4)
|
||||
|
||||
**Generated video** — the RefMod music-reference result.
|
||||
|
||||
https://github.com/user-attachments/assets/ef72642c-80c2-4152-82fe-915e6dd57824
|
||||
|
||||
[Original output MP4](examples/audio_refmod_example.mp4)
|
||||
|
||||
Prompt used, unchanged:
|
||||
|
||||
```text
|
||||
[Shot1]
|
||||
The ginger woman is walking on an office and dancing, the camera viewer is tracking in her head following every rotation and position.
|
||||
|
||||
music playing on background.
|
||||
```
|
||||
|
||||
### Extracting
|
||||
|
||||
An image and a video ref being extracted and fed into a conditioning node:
|
||||
|
||||

|
||||
|
||||
### Loading mods
|
||||
|
||||
`Load H3 RefMods` with several mods stacked (LoRA-loader style, one
|
||||
strength per row):
|
||||
|
||||

|
||||
|
||||
### Loading a ref folder
|
||||
|
||||
`Load H3 RefMod Folder` pointed at an absolute path (a whole movie
|
||||
dataset) — images + videos loaded in one shot, with the count shown in
|
||||
the preview text:
|
||||
|
||||

|
||||
|
||||
### Pool-size examples
|
||||
|
||||
Observed results with 8×8 and 16×16 pools. These examples do not establish
|
||||
a universal concept/identity split:
|
||||
|
||||

|
||||
|
||||

|
||||
|
||||
An identity success from real use — the author's own face extracted as a
|
||||
mod and prompted as **"ginger woman"**, and a third example pushed into
|
||||
JoJo Bizarre-style rendering:
|
||||
|
||||

|
||||
|
||||
### Curve controls
|
||||
|
||||
The same mod run three ways — no curve, `concept_at_end`, and
|
||||
`concept_at_end` with the loader's `copies` set to 3 (the same row injected
|
||||
three times; this example showed a stronger effect):
|
||||
|
||||

|
||||
|
||||
### Known limitations
|
||||
|
||||
Fast, high-speed motion is the hard case. A quick sequence — Sasuke doing
|
||||
hand signs, a fight flurry, a rapid dance step — gets smeared into
|
||||
something slower and softer, because the reference is compressed into a
|
||||
handful of latent frames and the model fills the gaps with its own idea of
|
||||
how motion looks:
|
||||
|
||||

|
||||
|
||||
Things that help today: extract only the *moments* that matter (trim the
|
||||
clip to the actual hand-sign burst instead of the whole scene), use more
|
||||
`latent_frames` / a higher `ref_resolution`, and describe the motion in
|
||||
the prompt so the model has an anchor for what it's seeing.
|
||||
|
||||
### Bulk folder loading
|
||||
|
||||
`Load H3 RefMod Folder` reads supported images (png/jpg/webp/bmp/gif) and video
|
||||
(mp4/webm/mov/mkv/avi) in a folder — images first, then videos, by filename.
|
||||
Type an absolute path, or a folder name inside ComfyUI's `input/` (empty =
|
||||
`input/` itself). Feed its `refs_bundle` output into `Create H3 RefMod` to
|
||||
bulk-extract a whole character shoot in one go:
|
||||
|
||||
```
|
||||
Load H3 RefMod Folder (folder: E:/vanellope_refs) ──refs_bundle──┐
|
||||
├─ Create H3 RefMod ─> vanellope.safetensors
|
||||
ref_image_1 (hand-picked shots) ─────────────────────────────────┘
|
||||
```
|
||||
|
||||
Bundle refs are appended after the autogrow refs, so `ref_image_1` still
|
||||
anchors the canvas. Unreadable files are skipped with a note; `max_items`
|
||||
caps the item count and `max_frames` caps the number of sampled video frames.
|
||||
|
||||
### Multiple references and merge
|
||||
|
||||
By default, visual refs are encoded separately and stacked along the latent
|
||||
time axis. A video contributes a sequence, not just one frame. Spatial resizing,
|
||||
temporal pooling/sampling and the token cap can still discard information.
|
||||
In encode mode, multiple refs use a common canvas based on the first source;
|
||||
other aspect ratios can be center-cropped to fit it.
|
||||
|
||||
With `merge=True` in training mode, one grid minimizes mean reconstruction
|
||||
error across targets. For equal target shapes, this has the same gradient as
|
||||
reconstructing their average. That is not semantic discovery of what the
|
||||
examples share: misaligned faces, motion and backgrounds can average into blur.
|
||||
Compare merge against stacking rather than assuming it removes unwanted content.
|
||||
|
||||
The current refinement groups same-shaped targets on CPU and processes gradient
|
||||
contributions sequentially. This reduces repeated work; it does not change the
|
||||
objective into DreamBooth, Textual Inversion or control/target edit training.
|
||||
|
||||
### Motion-only extraction (experimental)
|
||||
|
||||
For a motion-transfer comparison with native Ref2VA, start with one video,
|
||||
Full Reference, `multiplier=1`, `max_tokens=0` (or an adequate budget with
|
||||
`budget_policy=error`), and enough source frames. On Apply, use `retention=1`,
|
||||
`curve_direction=constant`, `curve_shape=linear`, `curve_value=1`, no preset
|
||||
override, and `scramble_seed=-1`. A constant linear curve at **0** replaces
|
||||
every reference frame with its blurred version; it does not disable the curve.
|
||||
Spatial pooling and Refinement Steps do not affect Full Reference mode.
|
||||
|
||||
Use a **Ref2VA checkpoint and reference workflow** for this comparison.
|
||||
[MiniMax's model card](https://huggingface.co/MiniMaxAI/MiniMax-H3#model-variants-and-input-specifications)
|
||||
distinguishes FL2VA first/last-frame conditioning from Ref2VA video references.
|
||||
Using the same first image in extraction and FL2VA generation does not make a
|
||||
RefMod a motion controller. A still alone contains no temporal sequence;
|
||||
connect the source clip to `ref_video`. RefMods guide generation through visual
|
||||
references and do not guarantee an exact motion replay, even with Ref2VA.
|
||||
|
||||
Apply alone does not present the reference video to Qwen. Compare with H3
|
||||
RefMod Text Encode when the workflow accepts external conditioning, using the
|
||||
reported Video label and without applying the same refs twice. This reconstructs
|
||||
visual presentation from saved latents, so it is still not an exact replay of
|
||||
native Ref2VA preprocessing or source timing. Dense textual motion descriptions
|
||||
are not a replacement for the reference's visual input. These are diagnostic
|
||||
settings, not validated optimal settings or a guarantee of motion fidelity.
|
||||
|
||||
For longer video references, increase `latent_frames` and the token budget
|
||||
together. In `encode`, `latent_frames` limits sampled **source frames** before
|
||||
the VAE's temporal compression. The count is snapped down to H3's **17k+5**
|
||||
video grid (5, 22, 39, ...), then sampled across the full clip, including its
|
||||
last frame when more than one frame survives. For example, a limit of 16
|
||||
selects 5 frames; a limit of 22 preserves all 22 frames of a 22-frame clip.
|
||||
A source or limit below 5 uses only the first image, retaining RefMod's
|
||||
still-image fallback. Already-aligned clips at or below the limit are unchanged.
|
||||
The same grid applies before compressed-mode encoding and in the CLI;
|
||||
motion-only differences are aligned after they are calculated. Uniform
|
||||
sampling preserves the endpoints but changes the temporal spacing: it does
|
||||
not preserve original timestamps or synchronize a separate soundtrack.
|
||||
In `training`, `latent_frames` limits the
|
||||
**latent frames** retained after encoding. It is not a duration in seconds.
|
||||
`max_tokens=0` disables the extraction token cap; loader/Apply/bridge budgets
|
||||
are separate. Higher values increase memory and attention cost and do not
|
||||
guarantee faithful motion transfer.
|
||||
|
||||
`motion_only` in training mode encodes normalized absolute frame differences,
|
||||
`abs(frame[t+1] - frame[t])`. These highlight change, including camera motion,
|
||||
lighting variation and compression artifacts. They can still reveal contours
|
||||
and appearance, and absolute differences do not explicitly encode direction.
|
||||
|
||||
This is not optical flow, a pose trajectory or a trained motion-conditioning
|
||||
channel. H3 may interpret the difference images as visual content. Still-image
|
||||
refs retain their appearance with a warning. Treat this mode as an experiment,
|
||||
not as guaranteed separation of motion from identity or background.
|
||||
|
||||
## Standalone extraction (visual only)
|
||||
|
||||
Use the Python environment that runs ComfyUI. Both commands set their mode
|
||||
explicitly; the CLI default is `training`.
|
||||
|
||||
```bash
|
||||
# Store the VAE encode as an identity comparison baseline
|
||||
python custom_nodes/ComfyUI-MiniMaxH3Mod/extract_mod.py \
|
||||
--image char.png --vae path/to/h3_video_vae.safetensors \
|
||||
--name my_character --mode encode --resolution 1024
|
||||
|
||||
# Compressed video reference; evaluate motion loss against encode
|
||||
python custom_nodes/ComfyUI-MiniMaxH3Mod/extract_mod.py \
|
||||
--video dance.mp4 --vae path/to/h3_video_vae.safetensors \
|
||||
--name dance --mode training --pool 16 --latent-frames 16 --identity 500
|
||||
```
|
||||
|
||||
Other options include `--pool-w`, `--max-tokens`, `--multiplier`, `--subfolder`,
|
||||
`--output`, `--max-edge`, `--max-frames`, `--description`, `--concept-type` and
|
||||
`--device`. Run `--help` for defaults. Video decode uses OpenCV or imageio.
|
||||
Files use embedded metadata; legacy JSON sidecars remain supported.
|
||||
|
||||
## Validation and experimental training status
|
||||
|
||||
### Apple Silicon
|
||||
|
||||
RefMod uses native ComfyUI H3 models and ordinary PyTorch operations. Visual
|
||||
pooling and strength blurring explicitly use CPU for MPS-resident latents,
|
||||
then return the result to the original device and dtype. This covers GPU-only
|
||||
intermediates without requiring `PYTORCH_ENABLE_MPS_FALLBACK=1`. CPU and CUDA
|
||||
paths retain their existing arithmetic; model/VAE execution can still use MPS.
|
||||
The transfers add preprocessing work for GPU-resident references, rather than
|
||||
moving diffusion inference to CPU.
|
||||
|
||||
Run `PYTORCH_ENABLE_MPS_FALLBACK=0 python -m unittest compatibility_regression_test -v`
|
||||
with ComfyUI's Python for the frame-grid, CPU/MPS pooling, blur, gradient and
|
||||
timing-backend checks. MPS tests skip on other platforms. A checkout outside
|
||||
ComfyUI needs its ComfyUI directory on `PYTHONPATH`. The existing
|
||||
`python issue_regression_test.py` also checks node/CLI frame selection.
|
||||
`python gauntlet_harness.py --device mps --output gauntlet_mps.json` synchronizes
|
||||
MPS before and after each timed refinement. Its CUDA memory field remains null
|
||||
on MPS; it is not an Apple GPU peak-memory measurement.
|
||||
|
||||
The corrected video grid changes newly extracted video RefMods that previously
|
||||
used 4k+1 sampling/trimming. Existing saved RefMods remain readable, but must be
|
||||
re-extracted to recover source frames that were discarded. These compatibility
|
||||
fixes are not a claim of improved identity, motion or voice quality.
|
||||
|
||||
### Other validation
|
||||
|
||||
Run production regressions with `python -m unittest discover -s tests -v`.
|
||||
The suite covers loaders/queue validation, storage, caches, curves, bridge,
|
||||
Master orchestration and audio storage. Codec checks used a real audio VAE;
|
||||
these are not full H3 voice/visual quality evaluations.
|
||||
|
||||
`gauntlet_harness.py` compares three implementations of the existing model-free
|
||||
MSE refinement. `tests/h3_gradient_probe.py` is a separate gradient-feasibility
|
||||
experiment with a small randomly initialized H3 architecture. Its success does
|
||||
not establish compatibility with a full INT8/ConvRot checkpoint, memory fit,
|
||||
concept learning or edit quality. A train-through-H3 ComfyUI node is not yet
|
||||
implemented. See [REVIEW_FOLLOWUP.md](REVIEW_FOLLOWUP.md) for recorded results.
|
||||
|
||||
## License
|
||||
|
||||
[MIT](LICENSE) — © 2026 Luisa (luisacaotica).
|
||||
@@ -0,0 +1,71 @@
|
||||
# RefMod — implementação e validação
|
||||
|
||||
Atualizado em 07/09/2026. Relatório de implementação e validação da v0.2.0.
|
||||
|
||||
## Correções concluídas
|
||||
|
||||
- Salvamento de mods e presets respeita a primeira raiz registrada em `folder_paths`, incluindo YAML externo; apenas a ausência de raízes usa `models/refmods`. Selecionar a pasta não cria diretórios.
|
||||
- Removidos imports e aviso de dependência do pack ComfyUI-MiniMaxH3. O socket legado `av_encoder` usa seu caminho de vídeo com o VAE nativo; Apply continua aceitando conditioning nativo e objetos antigos sem importar o pack.
|
||||
|
||||
- Loaders normal e Axis aceitam inputs conectados ainda não resolvidos na fila. Nomes fixos inválidos continuam sendo rejeitados; arquivos conectados inexistentes falham na execução.
|
||||
- Descoberta recursiva, barras normalizadas, filtro de metadados e exclusão de graph_presets. Raízes registradas e caminhos legados têm precedência consistente entre listagem e carga.
|
||||
- Step Curve processa o payload atual sem guardar tensores entre chamadas. Cobre refs visuais/áudio, payload vazio, CFG e curvas encadeadas. Schedule isolado por execução.
|
||||
- Bridge usa hooks públicos no MODEL clonado, sem lista global ou monkeypatch do Continuum. Restaura o conditioning mesmo se o sampler falhar. Disarm permanece por compatibilidade; desligue a bridge no ramo MODEL correspondente.
|
||||
- Máscaras seguem o crop da imagem. Vídeos sem contagem confiável têm amostragem limitada em memória e fechamento do decoder em finally.
|
||||
- Cache invalida por fingerprint de arquivo/sidecar; loaders expõem IS_CHANGED. Teto de 24 arquivos e 256 MiB de tensores.
|
||||
- Copies e valores numéricos são verificados na execução. Orçamento opcional soma todas as cópias; hard cap insuficiente para um frame gera erro explícito.
|
||||
- Salvamento atômico e gravação de config deduplicada por caminho.
|
||||
|
||||
## Manutenção e recursos entregues
|
||||
|
||||
- **Extract H3 RefMod Master** reúne imagem/vídeo e áudio opcional, usa os extratores compartilhados em sequência e retorna um bundle único. Salva arquivos separados com sufixos `_visual` e `_audio`; inclui limite total de tokens e diagnóstico. Não representa treinamento conjunto de personagem.
|
||||
|
||||
- Blur centralizado; resize e preparo causal compartilhados entre CLI e nodes. Removidos loops e estado redundantes. Harness importa código de produção.
|
||||
- **Extract H3 Audio RefMod**: VAE H3, mono/estéreo, resample para 32 kHz, encode em blocos de 10 segundos, limite de duração e orçamento por erro ou prefixo contíguo.
|
||||
- Formato v4 suporta áudio [1,32,2,T], preservando leitura dos visuais antigos e audio_refmod_meta do H3AudioMod. Áudio e visual coexistem no bundle em arquivos independentes.
|
||||
- Loaders, Apply, Config, Step Curve e bridge reconhecem áudio; models/audio_refmods é consultado.
|
||||
- **Inspect H3 RefMod**: tipo, caminho, dimensões, descrição, config e custo total. Preview opcional da primeira imagem ou primeiros 2 segundos de áudio, com comparação de força e VAE correspondente.
|
||||
- Biblioteca nos loaders: busca por nome/pasta/conceito/descrição, filtro de tipo, seleção de slot e Refresh. Endpoint lê apenas metadados. Slots conectados são protegidos contra seleção pelo widget.
|
||||
- Subpasta opcional no Extract visual/áudio e CLI. Scramble separa embaralhamento de seleção de subconjunto; workflows antigos preservam o comportamento sem o novo parâmetro.
|
||||
- Presets de identidade, estilo experimental e sequência de movimento. São configurações convenientes, sem alegação de melhoria de qualidade ou separação semântica.
|
||||
|
||||
## Validação
|
||||
|
||||
**39 testes aprovados na suíte completa antes da publicação da v0.2.0**, com Python embarcado e módulos reais do ComfyUI:
|
||||
|
||||
```powershell
|
||||
../../../python_embeded/python.exe -m unittest discover -s tests -v
|
||||
```
|
||||
|
||||
Cobertura: fila real com upstream não executado, ambos os loaders, caminhos, cache, limites, máscara, vídeo, bridge, curvas, áudio, escrita atômica, inspector, endpoint da biblioteca sem tensores e paridade contra a perda MSE completa com geometrias diferentes.
|
||||
|
||||
O módulo real do diálogo foi verificado no navegador em fixture local: layout visível, pesquisa, seleção e Escape com restauração de foco. Ainda é necessário experimentar a extensão no ComfyUI completo após reiniciar.
|
||||
|
||||
### VAE real de áudio
|
||||
|
||||
Checkpoint: E:\pinokio\api\comfy.git\app\models\vae\minimax_h3_audio_vae_fp32.safetensors. GPU: RTX 4060 Ti.
|
||||
|
||||
| Entrada sintética | Latente | Tokens | Decode estéreo a 32 kHz | Pico CUDA |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| 0,5 s | [1,32,2,20] | 40 | 16.000 amostras | 668 MiB |
|
||||
| 10,125 s | [1,32,2,405] | 810 | 324.000 amostras | 1.371 MiB |
|
||||
|
||||
Ambos passaram: layout nativo, valores finitos e roundtrip safetensors exato. Resultados: tests/audio_smoke_result.json e tests/audio_chunk_result.json. Reprodução: tests/audio_smoke.py.
|
||||
|
||||
### Otimização multi-ref
|
||||
|
||||
Três implementações reais, mesma GPU/processo, 4 targets sintéticos de 8x64x64 e 20 passos:
|
||||
|
||||
| Estratégia | Tempo | Pico CUDA | Erro máximo contra resident |
|
||||
| --- | --- | --- | --- |
|
||||
| Targets residentes, backward sequencial | 0,0538 s | 21,05 MiB | 0 |
|
||||
| Transferência por target | 0,0934 s | 12,05 MiB | 8,05e-7 |
|
||||
| Média ponderada por geometria | 0,0264 s | 12,04 MiB | 2,38e-6 |
|
||||
|
||||
A terceira é o padrão: preserva o gradiente da média MSE e evita reconstruções repetidas. Paridade passou com rtol=1e-4, atol=1e-5. Resultados em tests/gauntlet_result.json. Medição somente do refinamento sintético: não representa encode, geração completa ou ganho contra todo o pipeline anterior.
|
||||
|
||||
## Limites e próximo teste prático
|
||||
|
||||
Reinicie o ComfyUI para carregar os novos nodes e a biblioteca. Compare uma voz conhecida com/sem RefMod usando prompts e seeds fixos. Não foi executada geração completa H3/Continuum nesta tarefa: identidade vocal, música, qualidade visual, sincronização audiovisual e VRAM do pipeline inteiro ainda não foram medidos.
|
||||
|
||||
Estilo/conceito continuam baseados em reconstrução de latentes; metadados não criam um extrator semântico independente. Presets experimentais precisam de avaliação visual. Disparo por <Nome> permanece fora do escopo solicitado.
|
||||
@@ -0,0 +1,29 @@
|
||||
"""
|
||||
ComfyUI-MiniMaxH3Mod — no-training "RefMod" reference adapters for MiniMax H3
|
||||
|
||||
Reference videos are expensive because they inject thousands of tokens into
|
||||
the H3 packed sequence. A RefMod compresses a reference image/video into a
|
||||
tiny pooled latent (~4-16 tokens) that still rides the model's native ref2va
|
||||
path — the DiT attends to it through all 50 blocks like a real reference, but
|
||||
at a fraction of the compute. Extraction needs only the H3 VAE: no diffusion
|
||||
model load, no training.
|
||||
|
||||
Nodes
|
||||
─────
|
||||
MiniMaxH3RefModExtract — ref_image_1..N stills + ref_video_1..N clips -> saved mod
|
||||
MiniMaxH3RefModFolderLoader — load every image/video in a folder as an ordered ref list
|
||||
MiniMaxH3RefModsLoader — load 1-8 mods with a typed strength each (LoRA-style)
|
||||
MiniMaxH3RefModsAxis — A/B mod pairs on one signed slider each (negative -> A, positive -> B)
|
||||
MiniMaxH3RefModApply — inject the bundle into MINIMAX_H3_COND (pack) or CONDITIONING (built-in);
|
||||
the old split Apply/ApplyCond merged into one node (old workflows migrate)
|
||||
|
||||
Standalone extraction (image/video files): see extract_mod.py
|
||||
"""
|
||||
|
||||
__author__ = "Luisa (luisacaotica)"
|
||||
__version__ = "0.2.6"
|
||||
WEB_DIRECTORY = "./web"
|
||||
|
||||
from .nodes import NODE_CLASS_MAPPINGS, NODE_DISPLAY_NAME_MAPPINGS
|
||||
|
||||
__all__ = ["NODE_CLASS_MAPPINGS", "NODE_DISPLAY_NAME_MAPPINGS", "WEB_DIRECTORY"]
|
||||
@@ -0,0 +1,65 @@
|
||||
"""H3 audio encoding, adapted from the user's ComfyUI-H3AudioMod.
|
||||
|
||||
Uses its tested normalized [1,32,2,T] representation and bounded 10-second
|
||||
encode path. Curves, storage and loaders are shared with visual RefMods.
|
||||
"""
|
||||
|
||||
import math
|
||||
import torch
|
||||
import comfy.model_management as model_management
|
||||
from comfy.ldm.minimax.audio_vae import MiniMaxH3AudioVAE
|
||||
from .core import H3RefMod
|
||||
|
||||
|
||||
def encode_audio(vae, audio, max_seconds=30.0, chunk_seconds=10.0):
|
||||
waveform, sample_rate = audio["waveform"], int(audio["sample_rate"])
|
||||
if waveform.ndim != 3 or waveform.shape[0] != 1 or waveform.shape[1] not in (1, 2):
|
||||
raise ValueError("Audio must have one batch of mono or stereo samples [1,C,L].")
|
||||
if sample_rate <= 0 or waveform.shape[-1] < 1:
|
||||
raise ValueError("Audio is empty or has an invalid sample rate.")
|
||||
if not math.isfinite(max_seconds) or not math.isfinite(chunk_seconds) or max_seconds <= 0 or chunk_seconds <= 0:
|
||||
raise ValueError("Audio duration and chunk length must be positive.")
|
||||
model = vae.first_stage_model
|
||||
if not isinstance(model, MiniMaxH3AudioVAE):
|
||||
raise ValueError("Connect the MiniMax H3 audio VAE, not the video VAE or another audio codec.")
|
||||
waveform = waveform[..., :max(1, round(max_seconds * sample_rate))]
|
||||
if waveform.shape[1] == 1:
|
||||
waveform = waveform.repeat(1, 2, 1)
|
||||
if sample_rate != 32000:
|
||||
try:
|
||||
from comfy.audio import resample
|
||||
except ModuleNotFoundError as exc:
|
||||
if exc.name != "comfy.audio":
|
||||
raise
|
||||
# ComfyUI versions before the torchaudio removal lack comfy.audio.
|
||||
from torchaudio.functional import resample
|
||||
waveform = resample(waveform, sample_rate, 32000)
|
||||
model_management.load_models_gpu([vae.patcher], memory_required=0, force_full_load=True)
|
||||
chunk = max(800, round(chunk_seconds * 40) * 800)
|
||||
latents = []
|
||||
with model_management.cuda_device_context(vae.device):
|
||||
for start in range(0, waveform.shape[-1], chunk):
|
||||
piece = waveform[..., start:start + chunk].to(device=vae.device, dtype=vae.vae_dtype)
|
||||
z = model.encode(piece).detach().cpu()
|
||||
if z.ndim != 4 or tuple(z.shape[:3]) != (1, 32, 2):
|
||||
raise ValueError(f"H3 audio VAE returned an invalid latent: {tuple(z.shape)}")
|
||||
latents.append(z)
|
||||
return torch.cat(latents, dim=-1)
|
||||
|
||||
|
||||
def make_audio_mod(vae, audio, name, max_seconds=30.0, max_tokens=5120,
|
||||
budget_policy="error", description="", concept_type="voice"):
|
||||
if not isinstance(max_tokens, int) or isinstance(max_tokens, bool) or max_tokens < 0:
|
||||
raise ValueError("Audio token budget must be a non-negative integer.")
|
||||
if budget_policy not in ("error", "truncate"):
|
||||
raise ValueError("Audio budget policy must be error or truncate.")
|
||||
latent = encode_audio(vae, audio, max_seconds)
|
||||
if max_tokens and latent.shape[-1] * 2 > max_tokens:
|
||||
if max_tokens < 2 or budget_policy == "error":
|
||||
raise ValueError(f"Audio requires {latent.shape[-1] * 2} tokens; budget is {max_tokens}. "
|
||||
"Lower max_seconds or choose truncate.")
|
||||
if budget_policy != "truncate":
|
||||
raise ValueError("Audio budget policy must be error or truncate.")
|
||||
latent = latent[..., :max_tokens // 2].clone()
|
||||
return H3RefMod(name=name, kind="audio", latent=latent, mode="encode", source="audio",
|
||||
description=description, concept_type=concept_type, sample_rate=32000)
|
||||
@@ -0,0 +1,98 @@
|
||||
"""Version 5 containers; members remain ordinary independent H3 references."""
|
||||
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import tempfile
|
||||
|
||||
from safetensors import safe_open
|
||||
from safetensors.torch import save_file
|
||||
|
||||
from .core import H3RefMod, META_KEY, read_refmod_meta
|
||||
|
||||
|
||||
def members(meta):
|
||||
if not isinstance(meta, dict) or meta.get("_format_version") != 5 or meta.get("kind") != "bundle":
|
||||
raise ValueError("Unsupported RefMod bundle format.")
|
||||
refs = meta.get("members")
|
||||
if not isinstance(refs, list) or not refs or len(refs) > 256:
|
||||
raise ValueError("A RefMod bundle must contain 1–256 references.")
|
||||
for ref in refs:
|
||||
if not isinstance(ref, dict) or ref.get("kind") not in ("image", "video", "audio"):
|
||||
raise ValueError("Invalid RefMod bundle member.")
|
||||
return refs
|
||||
|
||||
|
||||
def write_bundle(path, metadata, tensors, extra_metadata=None):
|
||||
members(metadata)
|
||||
directory = os.path.dirname(path) or "."
|
||||
os.makedirs(directory, exist_ok=True)
|
||||
fd, temporary = tempfile.mkstemp(prefix=".refmod-", suffix=".tmp", dir=directory)
|
||||
os.close(fd)
|
||||
try:
|
||||
header = dict(extra_metadata or {})
|
||||
header[META_KEY] = json.dumps(metadata)
|
||||
save_file(tensors, temporary, metadata=header)
|
||||
os.replace(temporary, path + ".safetensors")
|
||||
finally:
|
||||
if os.path.exists(temporary):
|
||||
os.unlink(temporary)
|
||||
return path + ".safetensors"
|
||||
|
||||
|
||||
def save_bundle(path, name, mods):
|
||||
# Copies affect runtime, not stored content; strengths are not baked in.
|
||||
unique = list({id(mod): mod for mod, _ in mods}.values())
|
||||
metadata = {"_format_version": 5, "kind": "bundle", "name": name,
|
||||
"members": [mod.metadata() for mod in unique]}
|
||||
tensors = {f"ref_{i}": mod.latent.detach().cpu().contiguous().clone()
|
||||
for i, mod in enumerate(unique)}
|
||||
return write_bundle(path, metadata, tensors)
|
||||
|
||||
|
||||
def load_bundle(path, selection="All", visual_strength=1.0, audio_strength=1.0):
|
||||
if selection not in ("All", "Visual", "Audio"):
|
||||
raise ValueError("Components must be All, Visual or Audio.")
|
||||
if any(not math.isfinite(weight) or not 0 <= weight <= 1
|
||||
for weight in (visual_strength, audio_strength)):
|
||||
raise ValueError("Component strengths must be finite numbers between 0 and 1.")
|
||||
metadata = read_refmod_meta(path)
|
||||
refs = members(metadata)
|
||||
loaded = []
|
||||
with safe_open(path + ".safetensors", framework="pt", device="cpu") as file:
|
||||
for i, meta in enumerate(refs):
|
||||
audio = meta["kind"] == "audio"
|
||||
strength = audio_strength if audio else visual_strength
|
||||
if strength <= 0 or (selection == "Visual" and audio) or (selection == "Audio" and not audio):
|
||||
continue
|
||||
latent = file.get_tensor(f"ref_{i}").clone()
|
||||
if audio:
|
||||
valid = (latent.ndim == 4 and tuple(latent.shape[:3]) == (1, 32, 2)
|
||||
and latent.shape[-1] > 0)
|
||||
else:
|
||||
valid = (latent.ndim == 5 and tuple(latent.shape[:2]) == (1, 24)
|
||||
and all(n > 0 for n in latent.shape[2:])
|
||||
and all(n % 2 == 0 for n in latent.shape[-2:]))
|
||||
valid = valid and (meta["kind"] != "image" or latent.shape[2] == 1)
|
||||
if not valid:
|
||||
raise ValueError(f"Invalid tensor layout for bundle member {i}.")
|
||||
mod = H3RefMod.from_metadata(meta, latent, path, bundle_index=i)
|
||||
if not audio:
|
||||
mod.latent_t, mod.latent_h, mod.latent_w = latent.shape[2:]
|
||||
loaded.append((mod, strength))
|
||||
return loaded
|
||||
|
||||
|
||||
def update_member(mod):
|
||||
# Read the current container: updating only Visual must retain Audio, and
|
||||
# unknown tensors/metadata must survive a metadata-only Config operation.
|
||||
with safe_open(mod.path + ".safetensors", framework="pt", device="cpu") as file:
|
||||
header = file.metadata()
|
||||
metadata = json.loads(header[META_KEY])
|
||||
refs = members(metadata)
|
||||
if not 0 <= mod.bundle_index < len(refs) or refs[mod.bundle_index].get("name") != mod.name:
|
||||
raise ValueError("Bundle changed since loading. Reload before updating its config.")
|
||||
tensors = {key: file.get_tensor(key).clone() for key in file.keys()}
|
||||
refs[mod.bundle_index] = {**refs[mod.bundle_index], **mod.metadata()}
|
||||
tensors[f"ref_{mod.bundle_index}"] = mod.latent.detach().cpu().contiguous().clone()
|
||||
return write_bundle(mod.path, metadata, tensors, header)
|
||||
@@ -0,0 +1,237 @@
|
||||
"""Shared helpers for the RefMod pack: media loading and the refmods folder."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import ntpath
|
||||
import math
|
||||
import random
|
||||
from typing import List, Optional, Tuple
|
||||
|
||||
import torch
|
||||
import comfy.utils
|
||||
|
||||
IMAGE_EXTS = {".png", ".jpg", ".jpeg", ".webp", ".bmp", ".gif"}
|
||||
VIDEO_EXTS = {".mp4", ".webm", ".mov", ".mkv", ".avi", ".m4v"}
|
||||
|
||||
|
||||
def refmods_dir() -> str:
|
||||
"""First registered RefMod root, or the default; selection does not write."""
|
||||
import folder_paths
|
||||
try:
|
||||
roots = folder_paths.get_folder_paths("refmods")
|
||||
except KeyError:
|
||||
roots = []
|
||||
return roots[0] if roots else os.path.join(folder_paths.models_dir, "refmods")
|
||||
|
||||
|
||||
def list_media_files(folder: str) -> Tuple[List[str], List[str]]:
|
||||
"""(images, videos) directly under ``folder`` (top level only), sorted by name."""
|
||||
images, videos = [], []
|
||||
if os.path.isdir(folder):
|
||||
for fn in sorted(os.listdir(folder)):
|
||||
ext = os.path.splitext(fn)[1].lower()
|
||||
p = os.path.join(folder, fn)
|
||||
if os.path.isfile(p):
|
||||
if ext in IMAGE_EXTS:
|
||||
images.append(p)
|
||||
elif ext in VIDEO_EXTS:
|
||||
videos.append(p)
|
||||
return images, videos
|
||||
|
||||
|
||||
def load_image_file(path: str, max_edge: Optional[int] = None) -> torch.Tensor:
|
||||
"""Load one image file -> [1, H, W, 3] float32 in [0, 1]."""
|
||||
import numpy as np
|
||||
from PIL import Image
|
||||
with Image.open(path) as img:
|
||||
img = img.convert("RGB")
|
||||
w, h = img.size
|
||||
if max_edge is not None:
|
||||
scale = min(1.0, max_edge / max(w, h))
|
||||
if scale < 1.0:
|
||||
img = img.resize((max(1, round(w * scale)), max(1, round(h * scale))),
|
||||
Image.LANCZOS)
|
||||
arr = torch.from_numpy(np.asarray(img).copy()).float() / 255.0
|
||||
return arr.unsqueeze(0) # [1, H, W, 3]
|
||||
|
||||
|
||||
def _sample_video_frames(frames, total, max_frames, max_edge):
|
||||
"""Uniform known-length sampling; bounded deterministic reservoir otherwise."""
|
||||
import numpy as np
|
||||
from PIL import Image
|
||||
known = math.isfinite(total) and total > 0
|
||||
total = int(total) if known else 0
|
||||
targets = None
|
||||
if total > max_frames:
|
||||
targets = {round(i * (total - 1) / max(1, max_frames - 1)) for i in range(max_frames)}
|
||||
kept = []
|
||||
rng = random.Random(0)
|
||||
for index, frame in enumerate(frames):
|
||||
if targets is not None and index not in targets:
|
||||
continue
|
||||
# Treat underestimated counts as unknown after filling the reservoir.
|
||||
slot = len(kept) if len(kept) < max_frames else rng.randrange(index + 1)
|
||||
if slot >= max_frames:
|
||||
continue
|
||||
frame = np.asarray(frame)
|
||||
if max_edge is not None:
|
||||
h, w = frame.shape[:2]
|
||||
scale = min(1.0, max_edge / max(h, w))
|
||||
if scale < 1.0:
|
||||
frame = np.asarray(Image.fromarray(frame).resize(
|
||||
(max(1, round(w * scale)), max(1, round(h * scale))), Image.Resampling.BILINEAR))
|
||||
item = (index, frame.copy())
|
||||
if slot == len(kept):
|
||||
kept.append(item)
|
||||
else:
|
||||
kept[slot] = item
|
||||
if targets is not None and len(kept) == len(targets):
|
||||
break
|
||||
if not kept:
|
||||
raise ValueError("Video contains no decodable frames.")
|
||||
return torch.from_numpy(np.stack([frame for _, frame in sorted(kept)])).float() / 255.0
|
||||
|
||||
|
||||
def load_video_file(path: str, max_frames: int = 240,
|
||||
max_edge: Optional[int] = None) -> torch.Tensor:
|
||||
"""Load bounded, ordered RGB frames; release either decoder on every exit."""
|
||||
if max_frames < 1:
|
||||
raise ValueError("max_frames must be at least 1.")
|
||||
errors = []
|
||||
try:
|
||||
import cv2
|
||||
cap = cv2.VideoCapture(path)
|
||||
try:
|
||||
total = cap.get(cv2.CAP_PROP_FRAME_COUNT) or 0
|
||||
def frames():
|
||||
while True:
|
||||
ok, frame = cap.read()
|
||||
if not ok:
|
||||
break
|
||||
yield cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
|
||||
return _sample_video_frames(frames(), total, max_frames, max_edge)
|
||||
finally:
|
||||
cap.release()
|
||||
except (ImportError, OSError, ValueError, RuntimeError) as exc:
|
||||
errors.append(str(exc))
|
||||
try:
|
||||
import imageio.v2 as imageio
|
||||
reader = imageio.get_reader(path)
|
||||
try:
|
||||
total = reader.get_meta_data().get("nframes", 0) or 0
|
||||
return _sample_video_frames(iter(reader), total, max_frames, max_edge)
|
||||
finally:
|
||||
reader.close()
|
||||
except (ImportError, OSError, ValueError, RuntimeError) as exc:
|
||||
errors.append(str(exc))
|
||||
raise RuntimeError(f"Cannot decode video '{path}': " + "; ".join(errors))
|
||||
|
||||
|
||||
def mod_output_path(name, subfolder=""):
|
||||
relative = "/".join(part for part in (subfolder.replace("\\", "/"), name) if part)
|
||||
if (ntpath.splitdrive(relative)[0] or relative.startswith("/") or
|
||||
any(p in ("..", ".", "graph_presets", ".git", "__pycache__") for p in relative.split("/"))):
|
||||
raise ValueError("Use a relative RefMod subfolder without '..' or reserved directories.")
|
||||
root = os.path.realpath(refmods_dir())
|
||||
path = os.path.join(root, relative)
|
||||
if os.path.commonpath((root, os.path.realpath(path + ".safetensors"))) != root:
|
||||
raise ValueError("RefMod save path leaves the model folder.")
|
||||
return path
|
||||
|
||||
|
||||
def resize_ref(image, short_edge: int, canvas=None):
|
||||
"""Aspect-preserving downscale (never upscale) to ``short_edge`` px; dims to /32.
|
||||
|
||||
When several refs are stacked into one mod they must share a single spatial
|
||||
canvas, so ``canvas`` (tw, th) cover-crops each ref to it (like the official
|
||||
node's follower keyframes). Mirrors the official ref2video node: refs are
|
||||
resized before VAE encode, so the stored latent rides the same
|
||||
full-resolution path the model was trained with (the pooled path below is
|
||||
the cheap "thumbnail" alternative).
|
||||
"""
|
||||
h, w = image.shape[1], image.shape[2]
|
||||
if h <= 0 or w <= 0:
|
||||
raise ValueError(
|
||||
f"_resize_ref: source has an empty frame ({h}x{w}) before any "
|
||||
f"resize — the reference itself is invalid.")
|
||||
scale = min(1.0, short_edge / min(h, w))
|
||||
tw = max(32, round(w * scale / 32) * 32)
|
||||
th = max(32, round(h * scale / 32) * 32)
|
||||
crop = "disabled"
|
||||
if canvas is not None:
|
||||
tw, th = canvas
|
||||
crop = "center"
|
||||
if tw <= 0 or th <= 0:
|
||||
raise ValueError(
|
||||
f"_resize_ref: computed a zero-size resize target ({tw}x{th}) "
|
||||
f"for a {h}x{w} source (short_edge={short_edge}, canvas={canvas}). "
|
||||
f"This should be impossible — please report this shape.")
|
||||
samples = image[..., :3].movedim(-1, 1)
|
||||
samples = comfy.utils.common_upscale(samples, tw, th, "lanczos", crop)
|
||||
if samples.shape[2] <= 0 or samples.shape[3] <= 0:
|
||||
raise ValueError(
|
||||
f"_resize_ref: common_upscale produced an empty result "
|
||||
f"{tuple(samples.shape)} from a {h}x{w} source targeting "
|
||||
f"{tw}x{th} (crop={crop}, canvas={canvas}). This points to a bug "
|
||||
f"in comfy.utils.common_upscale for this input, not in RefMod's "
|
||||
f"own math.")
|
||||
return samples.movedim(1, -1)
|
||||
|
||||
|
||||
def snap_to_causal_grid(n_frames: int) -> int:
|
||||
"""Round down to H3's reference-video grid: 5, 22, 39, ... (17k+5).
|
||||
|
||||
The VAE encodes 17-frame chunks, pads the final chunk and drops three
|
||||
trailing latent tokens. Its effective video grid is not the encoder's
|
||||
standalone 4x temporal stride. RefMod keeps its single-image fallback
|
||||
when fewer than five frames are selected.
|
||||
"""
|
||||
if n_frames < 1:
|
||||
raise ValueError("A reference must contain at least one frame.")
|
||||
if n_frames < 5:
|
||||
return 1
|
||||
return ((n_frames - 5) // 17) * 17 + 5
|
||||
|
||||
|
||||
def sample_video_for_vae(frames: torch.Tensor, max_frames: Optional[int] = None) -> torch.Tensor:
|
||||
"""Sample the full clip onto H3's grid without exceeding the frame cap.
|
||||
|
||||
Keep both endpoints whenever the selected count is greater than one.
|
||||
Already-aligned clips are returned unchanged. A cap/input shorter than
|
||||
five frames selects only the first image, using H3's still-image path.
|
||||
"""
|
||||
if max_frames is not None and max_frames < 1:
|
||||
raise ValueError("max_frames must be at least 1.")
|
||||
count = snap_to_causal_grid(min(frames.shape[0], max_frames)
|
||||
if max_frames is not None else frames.shape[0])
|
||||
if count == frames.shape[0]:
|
||||
return frames
|
||||
indices = torch.linspace(0, frames.shape[0] - 1, count, device=frames.device).round().long()
|
||||
return frames[indices]
|
||||
|
||||
|
||||
|
||||
|
||||
def ensure_min_size(image, floor: int = 320):
|
||||
"""Upscale (never downscale) so both spatial dims are >= ``floor`` px.
|
||||
|
||||
The MiniMax H3 VAE encodes with internal tiled_encode (~256px tiles). A
|
||||
reference smaller than the tile size in one dimension can make the tiler
|
||||
compute a zero-size edge tile, which crashes deep inside conv_in with a
|
||||
cryptic 'Expected 4D or 5D... but got [1,3,1,0,W]' error. This applies
|
||||
regardless of extraction mode ('encode' already resizes down to
|
||||
ref_resolution but never guarantees a floor; 'training' now resizes to
|
||||
the same cap, also without a floor), so it's a separate, unconditional
|
||||
safety net right before encode.
|
||||
"""
|
||||
import comfy.utils
|
||||
h, w = image.shape[1], image.shape[2]
|
||||
if h >= floor and w >= floor:
|
||||
return image
|
||||
scale = floor / min(h, w)
|
||||
tw = max(floor, round(w * scale / 32) * 32)
|
||||
th = max(floor, round(h * scale / 32) * 32)
|
||||
samples = image[..., :3].movedim(-1, 1)
|
||||
samples = comfy.utils.common_upscale(samples, tw, th, "lanczos", "disabled")
|
||||
return samples.movedim(1, -1)
|
||||
@@ -0,0 +1,110 @@
|
||||
"""CPU/MPS preprocessing regressions, without model weights.
|
||||
|
||||
Run with ComfyUI's Python from this directory:
|
||||
PYTORCH_ENABLE_MPS_FALLBACK=0 python -m unittest compatibility_regression_test -v
|
||||
MPS tests skip when no Apple GPU is available. For a standalone checkout, add
|
||||
your ComfyUI directory to PYTHONPATH so common.py can import comfy.utils.
|
||||
"""
|
||||
from pathlib import Path
|
||||
import sys
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
|
||||
import torch
|
||||
import torch.nn.functional as F
|
||||
|
||||
from common import sample_video_for_vae, snap_to_causal_grid
|
||||
from core import _blur_latent, pool_latent
|
||||
from gauntlet_harness import synchronize
|
||||
|
||||
|
||||
class VideoGridTests(unittest.TestCase):
|
||||
def test_h3_grid_and_image_special_case(self):
|
||||
for n, expected in ((1,1),(2,1),(4,1),(5,5),(16,5),(17,5),(21,5),
|
||||
(22,22),(23,22),(38,22),(39,39),(101,90),(124,124)):
|
||||
with self.subTest(frames=n):
|
||||
self.assertEqual(snap_to_causal_grid(n), expected)
|
||||
|
||||
def test_aligned_clips_preserve_every_frame(self):
|
||||
for n in (1,5,22,39,124):
|
||||
clip = torch.arange(n).reshape(n,1,1,1)
|
||||
self.assertIs(sample_video_for_vae(clip), clip)
|
||||
self.assertIs(sample_video_for_vae(clip, n), clip)
|
||||
self.assertIs(sample_video_for_vae(clip, n+10), clip)
|
||||
|
||||
def test_frame_caps_keep_order_and_endpoints(self):
|
||||
for n in (2,4,16,21,22,38,39,101):
|
||||
clip = torch.arange(n).reshape(n,1,1,1)
|
||||
for limit in (1,4,5,16,22,39,101):
|
||||
with self.subTest(frames=n, limit=limit):
|
||||
sampled = sample_video_for_vae(clip, limit).flatten()
|
||||
self.assertLessEqual(len(sampled), min(n, limit))
|
||||
self.assertEqual(sampled[0], 0)
|
||||
if len(sampled) > 1:
|
||||
self.assertEqual(len(sampled) % 17, 5)
|
||||
self.assertEqual(sampled[-1], n-1)
|
||||
self.assertTrue(torch.all(sampled[1:] > sampled[:-1]))
|
||||
|
||||
def test_empty_input_and_invalid_cap_fail_before_encoding(self):
|
||||
with self.assertRaises(ValueError):
|
||||
sample_video_for_vae(torch.empty(0,1,1,3))
|
||||
with self.assertRaises(ValueError):
|
||||
sample_video_for_vae(torch.zeros(5,1,1,3), 0)
|
||||
|
||||
|
||||
class PoolingTests(unittest.TestCase):
|
||||
def test_cpu_pool_matches_existing_fp32_arithmetic(self):
|
||||
torch.manual_seed(19)
|
||||
z = torch.randn(1,24,3,20,28).half()
|
||||
expected = F.adaptive_avg_pool3d(z.float(), (2,8,8)).to(z.dtype)
|
||||
actual = pool_latent(z,2,8,8)
|
||||
self.assertTrue(torch.equal(actual.view(torch.uint8), expected.view(torch.uint8)))
|
||||
self.assertIs(pool_latent(z,3,20,28), z)
|
||||
|
||||
@unittest.skipUnless(torch.backends.mps.is_available(), "requires MPS")
|
||||
def test_mps_pool_and_visual_audio_blur_match_cpu_bytes(self):
|
||||
torch.manual_seed(20)
|
||||
for dtype in (torch.float32, torch.float16, torch.bfloat16):
|
||||
for shape in ((1,24,3,20,28), (1,32,2,19)):
|
||||
with self.subTest(dtype=dtype, shape=shape):
|
||||
z = torch.randn(shape).to(dtype)
|
||||
gpu = z.to('mps')
|
||||
functions = [_blur_latent]
|
||||
if len(shape) == 5:
|
||||
functions.append(lambda x: pool_latent(x,2,8,8))
|
||||
for function in functions:
|
||||
expected = function(z)
|
||||
actual = function(gpu)
|
||||
self.assertEqual(actual.device, gpu.device)
|
||||
self.assertEqual(actual.dtype, dtype)
|
||||
self.assertTrue(torch.equal(actual.cpu().view(torch.uint8),
|
||||
expected.view(torch.uint8)))
|
||||
one = torch.ones(1,32,2,1,device='mps')
|
||||
self.assertEqual(_blur_latent(one).device, one.device)
|
||||
|
||||
@unittest.skipUnless(torch.backends.mps.is_available(), "requires MPS")
|
||||
def test_explicit_cpu_transfers_preserve_autograd(self):
|
||||
for function in (_blur_latent, lambda x: pool_latent(x,2,8,8)):
|
||||
cpu = torch.randn(1,2,3,20,28,requires_grad=True)
|
||||
gpu = cpu.detach().to('mps').requires_grad_()
|
||||
function(cpu).square().sum().backward()
|
||||
function(gpu).square().sum().backward()
|
||||
torch.testing.assert_close(gpu.grad.cpu(), cpu.grad, rtol=1e-5, atol=1e-6)
|
||||
|
||||
|
||||
class TimingTests(unittest.TestCase):
|
||||
def test_synchronizes_only_the_selected_backend(self):
|
||||
with patch.object(torch.mps, 'synchronize') as mps, \
|
||||
patch.object(torch.cuda, 'synchronize') as cuda:
|
||||
synchronize('cpu')
|
||||
mps.assert_not_called()
|
||||
cuda.assert_not_called()
|
||||
synchronize('mps')
|
||||
mps.assert_called_once_with()
|
||||
synchronize('cuda:1')
|
||||
cuda.assert_called_once_with(torch.device('cuda:1'))
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main()
|
||||
@@ -0,0 +1,39 @@
|
||||
"""Model-scoped RefMod injection before ComfyUI prepares each sampling run.
|
||||
|
||||
Works with Continuum and standard H3 samplers through the public model patcher
|
||||
OUTER_SAMPLE hook. No process-global bundle or third-party function patch.
|
||||
"""
|
||||
|
||||
from comfy.patcher_extension import WrappersMP
|
||||
|
||||
BRIDGE_KEY = "minimax_h3_refmod_bridge"
|
||||
|
||||
|
||||
def attach(model, blocks, enabled=True):
|
||||
model = model.clone()
|
||||
model.remove_wrappers_with_key(WrappersMP.OUTER_SAMPLE, BRIDGE_KEY)
|
||||
if not enabled or not blocks:
|
||||
return model
|
||||
refs = tuple(dict(b) for b in blocks)
|
||||
|
||||
def inject(executor, *args, **kwargs):
|
||||
guider = executor.class_obj
|
||||
original = guider.conds
|
||||
guider.conds = {
|
||||
key: [dict(entry, minimax_refs=list(entry.get("minimax_refs") or [])
|
||||
+ [dict(ref) for ref in refs]) for entry in entries]
|
||||
for key, entries in original.items()
|
||||
}
|
||||
try:
|
||||
return executor(*args, **kwargs)
|
||||
finally:
|
||||
guider.conds = original
|
||||
|
||||
model.add_wrapper_with_key(WrappersMP.OUTER_SAMPLE, BRIDGE_KEY, inject)
|
||||
return model
|
||||
|
||||
|
||||
def disarm():
|
||||
# Preserve the old output-node ID in saved workflows. There is no longer
|
||||
# global state to disarm; enable=False on the model line removes the hook.
|
||||
return "Bridge is model-scoped. Bypass it or set enable=False on the model line."
|
||||
@@ -0,0 +1,752 @@
|
||||
"""
|
||||
core.py — "RefMod" adapter for MiniMax H3 (no training required)
|
||||
|
||||
The reference-video path of MiniMax H3 works by injecting *reference tokens*
|
||||
into the packed sequence: the ref is VAE-encoded, patchified, projected and
|
||||
placed on the 3D RoPE grid, then every DiT block attends to it. A video ref
|
||||
is expensive because it contributes thousands of tokens.
|
||||
|
||||
Mode naming: ``encode`` (was ``full``) = straight full-res VAE encode;
|
||||
``training`` (was ``pooled``) = compressed grid refined by gradient steps
|
||||
(the only mode that "trains" — the refinement loop). The old names are
|
||||
accepted everywhere as legacy aliases so saved workflows and mods keep
|
||||
working.
|
||||
|
||||
A RefMod is the same reference, compressed to a handful of tokens:
|
||||
|
||||
* the ref is VAE-encoded to its full latent [1, 24, T, H, W],
|
||||
* the latent is average-pooled to a tiny grid (default 4x4) and a few
|
||||
latent frames, so the patchified token count drops to ~4-16,
|
||||
* optionally the small latent is refined with a few gradient steps that
|
||||
reconstruct the full latent (still no model weights involved).
|
||||
|
||||
At generation time the pooled latent is handed back to the model through the
|
||||
native ``refs`` payload, so it flows through the exact same per-block
|
||||
attention machinery as a full reference — the only thing that changes is the
|
||||
token budget.
|
||||
|
||||
This is the MiniMax H3 analog of the LTX "Mod" concept tokens + per-block
|
||||
injection: instead of a trained hypernetwork predicting AdaLN deltas, the
|
||||
model's own cross-attention over the compressed ref tokens does the work.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import tempfile
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Dict, List, Optional, Tuple
|
||||
|
||||
import torch
|
||||
import torch.nn as nn
|
||||
import torch.nn.functional as F
|
||||
from safetensors import safe_open
|
||||
from safetensors.torch import load_file, save_file
|
||||
|
||||
# Metadata key embedded in the safetensors header (keeps a mod in one file,
|
||||
# so it can be shared/uploaded as a single artifact).
|
||||
META_KEY = "refmod_meta"
|
||||
|
||||
# What a mod is standing in for. This is documentation + a routing hint, not
|
||||
# a hard constraint on the tensors — but it drives two things: (1) the
|
||||
# "identity in training mode" warning in nodes.py, and (2) the loader's
|
||||
# prompt_hint output, which merges each loaded mod's concept_type +
|
||||
# description into a string you can concat onto your CLIP Text Encode prompt.
|
||||
# There's no CLIP-Vision / image-embedding injection point on H3's ref2va
|
||||
# path to hang a "visual clue" off of — text is the only channel the model's
|
||||
# text encoder reads, so that's the channel this uses.
|
||||
CONCEPT_TYPES = (
|
||||
"generic", # unspecified / mixed
|
||||
"identity", # a specific person/character — use mode="encode" for this
|
||||
"pose_motion", # a pose, dance, gesture, camera move
|
||||
"clothing", # an outfit / garment, decoupled from who's wearing it
|
||||
"background", # environment / set / location plate
|
||||
"voice", "singing", "music_style", "sound_fx", "ambience",
|
||||
"style", # look/grade/animation style, not a concrete subject
|
||||
)
|
||||
|
||||
|
||||
def _blur_latent(z: torch.Tensor, factor: int = 8) -> torch.Tensor:
|
||||
"""Heavy spatial low-pass (downsample then upsample) used as the
|
||||
weakening target for ``ref_block``'s ``strength`` below ``1.0``.
|
||||
|
||||
Originally this mixed toward ``torch.randn()`` noise, mirroring what the
|
||||
docstring below describes as the model's own ``visual_cond_noise_aug``
|
||||
mechanism. In practice, at the retention range people actually use this
|
||||
dial for (0.15-0.7, i.e. large noise fractions), it produces a visible
|
||||
woven/static texture instead of a gracefully weaker reference: a VAE
|
||||
latent's channels are correlated, so raw iid noise isn't read as "less
|
||||
reference," it's read as real-but-garbled content and gets rendered as
|
||||
an actual (wrong) texture. ``visual_cond_noise_aug`` in the model's own
|
||||
training is likely a small-magnitude robustness augmentation, not
|
||||
something swept this far — so copying its math doesn't reproduce its
|
||||
behavior at these strengths. A blurred copy stays on the latent
|
||||
manifold (smooth, plausible) while still discarding detail as strength
|
||||
drops, which is what "weaker reference" should actually look like.
|
||||
"""
|
||||
if z.device.type == "mps":
|
||||
# MPS lacks adaptive_avg_pool3d (and some non-divisible pooling
|
||||
# shapes). Keep this small preprocessing chain on CPU explicitly,
|
||||
# then restore the caller's device; global MPS fallback is not needed.
|
||||
return _blur_latent(z.cpu(), factor).to(z.device)
|
||||
if z.dim() == 4:
|
||||
b, c, stereo, t = z.shape
|
||||
if t <= 1:
|
||||
return z
|
||||
flat = z.reshape(b * c * stereo, 1, t).float()
|
||||
down = F.adaptive_avg_pool1d(flat, max(1, t // factor))
|
||||
return F.interpolate(down, size=t, mode="linear", align_corners=False).reshape_as(z).to(z.dtype)
|
||||
if z.dim() != 5:
|
||||
raise ValueError(f"Unsupported RefMod latent shape: {tuple(z.shape)}")
|
||||
t, h, w = z.shape[2], z.shape[3], z.shape[4]
|
||||
sh, sw = max(1, h // factor), max(1, w // factor)
|
||||
down = F.adaptive_avg_pool3d(z.float(), (t, sh, sw))
|
||||
up = F.interpolate(down, size=(t, h, w), mode="trilinear", align_corners=False)
|
||||
return up.to(z.dtype)
|
||||
|
||||
|
||||
def read_refmod_meta(path_no_ext: str) -> Optional[Dict]:
|
||||
"""
|
||||
Read the metadata block from a mod/preset file without loading tensors.
|
||||
|
||||
Prefers the metadata embedded in the safetensors header; falls back to
|
||||
the legacy sidecar ``{path}.json`` written by older versions.
|
||||
"""
|
||||
try:
|
||||
with safe_open(path_no_ext + ".safetensors", framework="pt") as f:
|
||||
meta = f.metadata()
|
||||
if meta:
|
||||
for key in (META_KEY, "audio_refmod_meta"):
|
||||
if key in meta:
|
||||
return json.loads(meta[key])
|
||||
except Exception:
|
||||
pass
|
||||
jpath = path_no_ext + ".json"
|
||||
if os.path.isfile(jpath):
|
||||
try:
|
||||
with open(jpath) as f:
|
||||
return json.load(f)
|
||||
except Exception:
|
||||
return None
|
||||
return None
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════════════
|
||||
# Latent compression
|
||||
# ═══════════════════════════════════════════════════════════════════════════
|
||||
|
||||
MODE_ALIASES = {"full": "encode", "pooled": "training",
|
||||
"Full Reference": "encode", "Compressed Reference": "training"}
|
||||
|
||||
|
||||
def normalize_mode(mode: str) -> str:
|
||||
"""Map legacy mode names (``full``/``pooled``) to the current ones."""
|
||||
return MODE_ALIASES.get(mode, mode)
|
||||
|
||||
|
||||
def aspect_grid(pool_h: int, pool_w: int, aspect: float):
|
||||
"""Even pool grid dims that match ``aspect`` (h/w) within the dial caps.
|
||||
|
||||
The DiT patches each 2x2 latent cell into one token and the rope grid is
|
||||
area-normalized per axis, so a pooled grid that ignores the source aspect
|
||||
squishes the subject: a portrait latent (tall person) forced into a square
|
||||
16x16 grid is compressed 5x in height but only 3x in width, and comes out
|
||||
"fat". The official ref2video node never pools — it feeds the model
|
||||
aspect-correct latent dims, which is what the DiT was trained on.
|
||||
|
||||
The long edge follows the user's dial (``max(pool_h, pool_w)``) and the
|
||||
short edge is derived from the source aspect, both rounded to even for the
|
||||
2x2 patch. Square sources keep the exact dial value (16x16 stays 16x16).
|
||||
"""
|
||||
long_edge = max(pool_h, pool_w)
|
||||
if aspect >= 1.0:
|
||||
h, w = long_edge, long_edge / aspect
|
||||
else:
|
||||
w, h = long_edge, long_edge * aspect
|
||||
h = max(2, round(h / 2) * 2)
|
||||
w = max(2, round(w / 2) * 2)
|
||||
return h, w
|
||||
|
||||
|
||||
def pool_latent(
|
||||
z: torch.Tensor,
|
||||
latent_t: int,
|
||||
latent_h: int,
|
||||
latent_w: int,
|
||||
) -> torch.Tensor:
|
||||
"""
|
||||
Average-pool a VAE latent ``[1, 24, T, H, W]`` down to a tiny grid.
|
||||
|
||||
``latent_h/latent_w`` are the *latent* grid dims (the DiT patches each 2x2
|
||||
latent cell into one token), so a 4x4 latent = 2x2 = 4 tokens per frame.
|
||||
Dims must be even (the DiT's 2x2 patch). Pooling runs in fp32; the result
|
||||
keeps the input dtype.
|
||||
"""
|
||||
if z.shape[2] == latent_t and z.shape[3] == latent_h and z.shape[4] == latent_w:
|
||||
return z
|
||||
if latent_h % 2 != 0 or latent_w % 2 != 0:
|
||||
raise ValueError(f"pool_h/pool_w must be even (got {latent_h}x{latent_w})")
|
||||
# Normal ComfyUI intermediates are already on CPU. GPU-only workflows can
|
||||
# supply MPS latents, where adaptive_avg_pool3d is not implemented.
|
||||
work = z.cpu() if z.device.type == "mps" else z
|
||||
pooled = F.adaptive_avg_pool3d(work.float(), (latent_t, latent_h, latent_w))
|
||||
return pooled.to(device=z.device, dtype=z.dtype)
|
||||
|
||||
|
||||
def optimize_latent(
|
||||
z_small: torch.Tensor,
|
||||
z_full: torch.Tensor,
|
||||
steps: int = 150,
|
||||
lr: float = 0.02,
|
||||
device: Optional[torch.device] = None,
|
||||
progress_every: int = 0,
|
||||
) -> torch.Tensor:
|
||||
"""
|
||||
Model-free refinement of the compressed latent.
|
||||
|
||||
Optimizes the small latent so its trilinearly upsampled reconstruction
|
||||
matches the full reference latent. This pulls the pooled representation
|
||||
closer to what the DiT would see from the full ref, with nothing but the
|
||||
tiny latent trainable (~1-2K params) and no diffusion model loaded.
|
||||
|
||||
``progress_every`` > 0 prints a ``[RefMod] identity <step>/<steps>`` line
|
||||
every N steps so long pooled-mode refinement isn't silent (the default
|
||||
``identity=500`` takes a while per ref on CPU/VRAM-bound setups).
|
||||
|
||||
Returns the refined latent detached, same shape/dtype as ``z_small``.
|
||||
"""
|
||||
if steps <= 0:
|
||||
return z_small
|
||||
device = device or z_full.device
|
||||
# Node execution runs under ComfyUI's global inference mode, which would
|
||||
# disable the autograd this refine loop needs. Re-enable it for this
|
||||
# scope and clone the inputs to drop the inference flag.
|
||||
with torch.inference_mode(False), torch.set_grad_enabled(True):
|
||||
target = z_full.clone().float().to(device)
|
||||
param = nn.Parameter(z_small.clone().float().to(device))
|
||||
opt = torch.optim.Adam([param], lr=lr)
|
||||
size = tuple(target.shape[2:])
|
||||
for i in range(steps):
|
||||
opt.zero_grad()
|
||||
up = F.interpolate(param, size=size, mode="trilinear", align_corners=False)
|
||||
loss = F.mse_loss(up, target)
|
||||
loss.backward()
|
||||
opt.step()
|
||||
if progress_every and (i + 1) % progress_every == 0:
|
||||
print(f"[RefMod] identity {i + 1}/{steps}")
|
||||
# materialize inside the scope so the result is a normal tensor, not
|
||||
# an inference-mode tensor (it gets stored in the mod and reused)
|
||||
refined = param.detach().to(z_small.dtype)
|
||||
return refined
|
||||
|
||||
|
||||
def optimize_latent_multi(z_init, targets, steps=150, lr=0.02, device=None,
|
||||
progress_every=0, strategy="grouped"):
|
||||
"""Mean reconstruction objective with bounded activation memory.
|
||||
|
||||
grouped: average targets with identical shapes on CPU before refinement.
|
||||
stream: transfer one target per gradient contribution.
|
||||
resident: retain targets on the compute device, backward one at a time.
|
||||
All strategies optimize the same mean MSE; grouping drops a constant term.
|
||||
"""
|
||||
if not targets or steps <= 0:
|
||||
return z_init
|
||||
if strategy not in ("grouped", "stream", "resident"):
|
||||
raise ValueError("Unknown multi-reference optimization strategy.")
|
||||
device = device or z_init.device
|
||||
with torch.inference_mode(False), torch.set_grad_enabled(True):
|
||||
count = len(targets)
|
||||
if strategy == "grouped":
|
||||
groups = {}
|
||||
for target in targets:
|
||||
key = tuple(target.shape)
|
||||
value = target.detach().to(device="cpu", dtype=torch.float32).clone()
|
||||
if key in groups:
|
||||
groups[key][0].add_(value)
|
||||
groups[key][1] += 1
|
||||
else:
|
||||
groups[key] = [value, 1]
|
||||
prepared = [(total.div_(n), n / count) for total, n in groups.values()]
|
||||
else:
|
||||
target_device = device if strategy == "resident" else "cpu"
|
||||
prepared = [(t.detach().to(device=target_device, dtype=torch.float32).clone(), 1 / count) for t in targets]
|
||||
param = nn.Parameter(z_init.clone().float().to(device))
|
||||
opt = torch.optim.Adam([param], lr=lr)
|
||||
for i in range(steps):
|
||||
opt.zero_grad()
|
||||
for target, weight in prepared:
|
||||
current = target.to(device)
|
||||
up = F.interpolate(param, size=tuple(current.shape[2:]), mode="trilinear", align_corners=False)
|
||||
(F.mse_loss(up, current) * weight).backward()
|
||||
del up, current
|
||||
opt.step()
|
||||
if progress_every and (i + 1) % progress_every == 0:
|
||||
print(f"[RefMod] merge {i + 1}/{steps}")
|
||||
return param.detach().to(z_init.dtype)
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════════════
|
||||
# Per-frame strength curve
|
||||
# ═══════════════════════════════════════════════════════════════════════════
|
||||
|
||||
# Direction names describe where the *concept* shows up in the output, and
|
||||
# the envelope is strong exactly where the concept lands: concept_at_start
|
||||
# keeps the ref at full strength at the START of the video and releases it
|
||||
# toward the end, concept_at_end fades it in toward the END. The old
|
||||
# "decrease"/"increase" envelopes are kept as legacy aliases (decrease =
|
||||
# concept_at_start, increase = concept_at_end) so pre-rename workflows keep
|
||||
# their behavior. shape is how the envelope travels between its endpoints;
|
||||
# value is the non-zero endpoint ("user input").
|
||||
CURVE_DIRECTIONS = ("constant", "concept_at_start", "concept_at_middle",
|
||||
"concept_at_end", "concept_at_ends")
|
||||
CURVE_SHAPES = ("linear", "ease", "sigmoid", "tanh", "quadratic", "cubic",
|
||||
"exponential", "stair", "elastic", "bump", "dip")
|
||||
|
||||
# old single-combo preset names -> (direction, shape, value) so workflows
|
||||
# saved before the curve was split still resolve
|
||||
_LEGACY_CURVES = {
|
||||
"flat": ("constant", "linear", 1.0),
|
||||
# keep each preset's historical envelope: fade_in = crescent (0 -> value),
|
||||
# fade_out = decrescent (value -> 0), bump = mid peak
|
||||
"fade_in": ("concept_at_end", "linear", 1.0),
|
||||
"fade_out": ("concept_at_start", "linear", 1.0),
|
||||
"bump": ("concept_at_end", "bump", 1.0),
|
||||
"dip": ("constant", "dip", 1.0),
|
||||
}
|
||||
|
||||
|
||||
def _ease(shape: str, x: float) -> float:
|
||||
"""Shape a progress ``x`` in [0, 1] per the easing name (may overshoot)."""
|
||||
if shape == "linear":
|
||||
return x
|
||||
if shape == "ease": # smoothstep
|
||||
return x * x * (3.0 - 2.0 * x)
|
||||
if shape == "sigmoid": # logistic S-curve
|
||||
return 1.0 / (1.0 + math.exp(-12.0 * (x - 0.5)))
|
||||
if shape == "tanh": # sharper S-curve (steeper knee than sigmoid)
|
||||
return 0.5 * (math.tanh(8.0 * (x - 0.5)) + 1.0)
|
||||
if shape == "quadratic":
|
||||
return x * x
|
||||
if shape == "cubic":
|
||||
return x * x * x
|
||||
if shape == "exponential":
|
||||
return 2.0 ** x - 1.0
|
||||
if shape == "stair":
|
||||
return min(1.0, math.floor(x * 4) / 3.0)
|
||||
if shape == "elastic":
|
||||
if x <= 0.0:
|
||||
return 0.0
|
||||
if x >= 1.0:
|
||||
return 1.0
|
||||
return 2.0 ** (-10.0 * x) * math.sin((x * 10.0 - 0.75) * (2.0 * math.pi / 3.0)) + 1.0
|
||||
if shape == "bump":
|
||||
return 1.0 - abs(2.0 * x - 1.0)
|
||||
if shape == "dip":
|
||||
return abs(2.0 * x - 1.0)
|
||||
return x
|
||||
|
||||
|
||||
def curve_strengths(spec, t: int) -> Optional[List[float]]:
|
||||
"""Resolve a curve spec to ``t`` per-frame strength multipliers in [0, 1].
|
||||
|
||||
Kept as plain Python so the Apply node exposes the curve as normal
|
||||
combo/float widgets instead of depending on ComfyUI's new Curve widget.
|
||||
Accepts:
|
||||
|
||||
* a ``(direction, shape, value)`` tuple — direction is ``constant``
|
||||
(one strength everywhere), ``concept_at_start`` (value -> 0, a
|
||||
decrescent — the concept shows in the first half; legacy
|
||||
``decrease``), ``concept_at_end`` (0 -> value, a crescent — the
|
||||
concept shows in the second half; legacy ``increase``),
|
||||
``concept_at_middle`` ([0..1..0] — concept peaks mid-timeline) or
|
||||
``concept_at_ends`` ([1..0..1] — concept at both ends); shape is how
|
||||
the envelope travels between its endpoints (see ``_ease``); value is
|
||||
the non-zero endpoint (1.0 = full strength there);
|
||||
* a legacy preset name (``flat``/``fade_in``/``fade_out``/``bump``/
|
||||
``dip``) from before the split;
|
||||
* a list of per-frame floats (len == t), used as-is;
|
||||
* a list of ``(x, y)`` control points (x in [0, 1]), linearly
|
||||
interpolated across the frames;
|
||||
* any object with ``interp(x) -> float`` (legacy CurveInput saved in an
|
||||
old workflow).
|
||||
|
||||
Returns None for flat/no curve so the caller keeps its single-strength
|
||||
path unchanged (a ``constant`` curve at value 1.0 included).
|
||||
"""
|
||||
if t <= 1 or spec is None or spec == "":
|
||||
return None
|
||||
if isinstance(spec, str):
|
||||
legacy = _LEGACY_CURVES.get(spec)
|
||||
return curve_strengths(legacy, t) if legacy is not None else None
|
||||
if (isinstance(spec, tuple) and len(spec) == 3
|
||||
and isinstance(spec[0], str) and isinstance(spec[1], str)):
|
||||
direction, shape, value = spec
|
||||
value = float(value)
|
||||
if direction == "constant":
|
||||
# linear = flat at ``value`` (1.0 == no curve); the other shapes
|
||||
# modulate around the constant, so constant + dip is a trough,
|
||||
# constant + bump is a peak (both endpoints equal)
|
||||
if shape == "linear":
|
||||
return None if value >= 1.0 else [value] * t
|
||||
p = [_ease(shape, i / (t - 1)) for i in range(t)]
|
||||
return [max(0.0, min(1.0, value * y)) for y in p]
|
||||
p = [_ease(shape, i / (t - 1)) for i in range(t)]
|
||||
# legacy "decrease"/"increase" (pre-rename workflows) keep their
|
||||
# original envelopes: decrease = strong at the start, increase =
|
||||
# strong at the end. The concept_* names mean exactly what they say.
|
||||
if direction in ("concept_at_end", "increase"):
|
||||
return [max(0.0, min(1.0, value * y)) for y in p]
|
||||
if direction in ("concept_at_start", "decrease"):
|
||||
return [max(0.0, min(1.0, value * (1.0 - y))) for y in p]
|
||||
if direction == "concept_at_middle": # [0..1..0]: concept peaks mid-timeline
|
||||
return [max(0.0, min(1.0, value * (1.0 - abs(2.0 * y - 1.0)))) for y in p]
|
||||
if direction == "concept_at_ends": # [1..0..1]: concept at both ends
|
||||
return [max(0.0, min(1.0, value * abs(2.0 * y - 1.0))) for y in p]
|
||||
return None # unknown direction: treat as flat
|
||||
if isinstance(spec, (list, tuple)):
|
||||
if len(spec) == t and all(isinstance(v, (int, float)) for v in spec):
|
||||
return [float(v) for v in spec]
|
||||
pts = [(float(x), float(y)) for x, y in spec
|
||||
if isinstance(x, (int, float)) and isinstance(y, (int, float))]
|
||||
if pts:
|
||||
pts = sorted(pts, key=lambda p: p[0])
|
||||
out = []
|
||||
for i in range(t):
|
||||
x = i * (1.0 / (t - 1))
|
||||
if x <= pts[0][0]:
|
||||
out.append(pts[0][1])
|
||||
elif x >= pts[-1][0]:
|
||||
out.append(pts[-1][1])
|
||||
else:
|
||||
for (x0, y0), (x1, y1) in zip(pts, pts[1:]):
|
||||
if x0 <= x <= x1:
|
||||
out.append(y0 + (y1 - y0) * (x - x0) / (x1 - x0))
|
||||
break
|
||||
return out
|
||||
interp = getattr(spec, "interp", None)
|
||||
if callable(interp):
|
||||
return [float(interp(i / (t - 1))) for i in range(t)]
|
||||
return None
|
||||
|
||||
|
||||
def curve_value_at(spec, x: float) -> float:
|
||||
"""Per-progress strength of a curve spec at one point ``x`` in [0, 1].
|
||||
|
||||
The continuous analog of ``curve_strengths`` — same direction/shape/value
|
||||
math, but evaluated at a single progress value instead of discretized over
|
||||
frames. The step-curve node uses it to re-mix the ref latents once per
|
||||
denoising step, where ``x`` is the denoise progress (0 = first step,
|
||||
1 = last step). Returns 1.0 (no weakening) for anything unrecognized.
|
||||
"""
|
||||
if spec is None or spec == "":
|
||||
return 1.0
|
||||
if isinstance(spec, str):
|
||||
legacy = _LEGACY_CURVES.get(spec)
|
||||
return curve_value_at(legacy, x) if legacy is not None else 1.0
|
||||
if (isinstance(spec, tuple) and len(spec) == 3
|
||||
and isinstance(spec[0], str) and isinstance(spec[1], str)):
|
||||
direction, shape, value = spec
|
||||
value = float(value)
|
||||
if direction == "constant":
|
||||
if shape == "linear":
|
||||
return value
|
||||
return max(0.0, min(1.0, value * _ease(shape, x)))
|
||||
y = _ease(shape, x)
|
||||
if direction in ("concept_at_end", "increase"):
|
||||
return max(0.0, min(1.0, value * y))
|
||||
if direction in ("concept_at_start", "decrease"):
|
||||
return max(0.0, min(1.0, value * (1.0 - y)))
|
||||
if direction == "concept_at_middle": # [0..1..0]: concept peaks mid-timeline
|
||||
return max(0.0, min(1.0, value * (1.0 - abs(2.0 * y - 1.0))))
|
||||
if direction == "concept_at_ends": # [1..0..1]: concept at both ends
|
||||
return max(0.0, min(1.0, value * abs(2.0 * y - 1.0)))
|
||||
return 1.0
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════════════
|
||||
# Token budget cap (shared by the Extract node and extract_mod.py)
|
||||
# ═══════════════════════════════════════════════════════════════════════════
|
||||
|
||||
|
||||
def dedup_frame_indices(z: torch.Tensor, threshold: float = 0.02) -> List[int]:
|
||||
"""Indices of latent frames kept by greedy temporal dedup.
|
||||
|
||||
Each frame is compared to the last *kept* frame: if the mean-abs
|
||||
difference between them, normalized by the frames' own magnitude, is
|
||||
below ``threshold`` the frame is dropped as a (near-)duplicate. Video
|
||||
refs (a dance loop, a static shot, a talking head) contain long runs of
|
||||
frames that differ only by codec noise — each one still costs a token
|
||||
per spatial patch in every DiT block, so cutting them is the cheapest
|
||||
way to honor a token cap. Stacked image refs survive: different
|
||||
angles/expressions land well above the threshold.
|
||||
"""
|
||||
t = z.shape[2]
|
||||
if t <= 1:
|
||||
return list(range(t))
|
||||
flat = z[0].float() # [24, T, H, W]
|
||||
kept = [0]
|
||||
prev = flat[:, 0]
|
||||
for i in range(1, t):
|
||||
cur = flat[:, i]
|
||||
denom = (cur.abs().mean() + prev.abs().mean()) / 2 + 1e-6
|
||||
diff = (cur - prev).abs().mean() / denom
|
||||
if diff >= threshold:
|
||||
kept.append(i)
|
||||
prev = cur
|
||||
return kept
|
||||
|
||||
|
||||
def fit_token_budget(latent: torch.Tensor, budget: int, label: str) -> torch.Tensor:
|
||||
"""Bring a stacked latent's injected token count under ``budget``.
|
||||
|
||||
Tokens = (H/2)*(W/2) per frame (the DiT's 2x2 patch). When the stack
|
||||
exceeds the budget it is cut in two cheap, loss-ordered passes:
|
||||
|
||||
1. temporal dedup — drop latent frames that are (near-)identical to
|
||||
the last kept frame (free: they carry no new reference info);
|
||||
2. budget-fit resample — uniformly subsample the remaining frames down
|
||||
to the largest count that fits the budget.
|
||||
|
||||
The spatial dims are never touched: resizing a latent's H/W changes the
|
||||
rope grid and degrades the reference, so time is the only honest lever.
|
||||
``label`` is the mod name for the console notes.
|
||||
"""
|
||||
if budget <= 0:
|
||||
return latent
|
||||
h, w = latent.shape[3], latent.shape[4]
|
||||
per_frame = (h // 2) * (w // 2)
|
||||
t = latent.shape[2]
|
||||
if per_frame > budget:
|
||||
raise ValueError(f"{label}: one frame costs {per_frame} tokens, above budget {budget}. Lower ref_resolution or pool size.")
|
||||
if per_frame * t <= budget:
|
||||
return latent
|
||||
kept = dedup_frame_indices(latent)
|
||||
if len(kept) < t:
|
||||
print(f"[RefMod] {label}: over {budget}-token cap, "
|
||||
f"dropped {t - len(kept)} near-duplicate frame(s) "
|
||||
f"({t} -> {len(kept)})")
|
||||
latent = latent[:, :, kept]
|
||||
t = latent.shape[2]
|
||||
if per_frame * t > budget:
|
||||
fit_t = max(1, budget // per_frame)
|
||||
if fit_t < t:
|
||||
idx = torch.linspace(0, t - 1, fit_t).round().long()
|
||||
latent = latent[:, :, idx]
|
||||
print(f"[RefMod] {label}: still over {budget}-token cap, "
|
||||
f"resampled {t} -> {fit_t} frame(s) to fit")
|
||||
if per_frame * latent.shape[2] > budget:
|
||||
print(f"[RefMod] {label}: one frame alone is {per_frame} tokens > "
|
||||
f"{budget} — lower ref_resolution or pool_h/pool_w to meet "
|
||||
f"the cap.")
|
||||
return latent
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════════════
|
||||
# H3RefMod — the saved artifact
|
||||
# ═══════════════════════════════════════════════════════════════════════════
|
||||
|
||||
@dataclass
|
||||
class H3RefMod:
|
||||
"""
|
||||
A compressed reference for MiniMax H3.
|
||||
|
||||
``latent`` is the VAE latent ``[1, 24, latent_t, latent_h, latent_w]`` — a
|
||||
full-resolution encode (``mode="encode"``) or a pooled thumbnail
|
||||
(``mode="training"``). ``kind`` is ``"image"`` (single latent frame) or
|
||||
``"video"`` (a few frames), matching the native ref block kinds the
|
||||
model's ``PackedLayout`` understands.
|
||||
|
||||
``mode="encode"`` stores the encode at the resolution the official ref2video
|
||||
path uses, so the injected ref carries real identity detail;
|
||||
``mode="training"`` stores a tiny average-pooled grid refined by gradient
|
||||
steps (concept/motion, or identity at high pool sizes).
|
||||
"""
|
||||
|
||||
name: str
|
||||
kind: str
|
||||
latent: torch.Tensor
|
||||
latent_h: int = 4
|
||||
latent_w: int = 4
|
||||
latent_t: int = 1
|
||||
mode: str = "training"
|
||||
source: str = "" # "image" | "video" | "manual"
|
||||
source_shape: str = "" # original latent dims as "TxHxW"
|
||||
pool: str = "4x4x1" # pool_t x pool_h x pool_w
|
||||
optimize_steps: int = 0
|
||||
tags: List[str] = field(default_factory=list)
|
||||
description: str = "" # optional text describing the concept (emitted by the loaders)
|
||||
concept_type: str = "generic" # what this mod represents; see CONCEPT_TYPES above
|
||||
config: Dict = field(default_factory=dict)
|
||||
# Recommended Apply/Step-Curve settings baked in by the "Fix H3 RefMod
|
||||
# Config" node: {"retention": float, "curve": [direction, shape, value],
|
||||
# "step_curve": [direction, shape, value]}. The Apply / Step Curve
|
||||
# nodes' ``override`` toggle reads it back so a concept ships with the
|
||||
# settings that make it work — newbies never have to tune.
|
||||
sample_rate: int = 32000
|
||||
path: str = "" # path_no_ext the mod was loaded from / saved to (for re-saving)
|
||||
bundle_index: int = -1 # index in the container at path; -1 for standalone refs
|
||||
|
||||
def __post_init__(self):
|
||||
if self.kind == "audio":
|
||||
if self.latent.ndim != 4 or tuple(self.latent.shape[:3]) != (1, 32, 2) or self.latent.shape[-1] < 1:
|
||||
raise ValueError("H3 audio RefMod must contain [1,32,2,T] with T >= 1.")
|
||||
self.latent_t = self.latent.shape[-1]
|
||||
self.latent_h = self.latent_w = 0
|
||||
return
|
||||
if self.kind not in ("image", "video"):
|
||||
raise ValueError(f"kind must be 'image' or 'video' (got {self.kind!r})")
|
||||
if self.kind == "image":
|
||||
self.latent_t = 1
|
||||
|
||||
# ── token budget ──────────────────────────────────────────────────
|
||||
|
||||
@property
|
||||
def token_count(self) -> int:
|
||||
"""Number of patchified tokens the mod injects into the packed sequence."""
|
||||
if self.kind == "audio":
|
||||
return 2 * self.latent_t
|
||||
per_frame = (self.latent_h // 2) * (self.latent_w // 2)
|
||||
return self.latent_t * per_frame
|
||||
|
||||
# ── native ref block ──────────────────────────────────────────────
|
||||
|
||||
def ref_block(self, strength: float = 1.0,
|
||||
curve=None) -> Optional[Dict]:
|
||||
"""
|
||||
Build the ref block dict the model's ``PackedLayout`` / payload consumes.
|
||||
|
||||
The shape mirrors what the native ref2va nodes emit, so the pooled
|
||||
latent rides the exact same path: ``cond_video_latents`` -> patchify ->
|
||||
``video_patch_proj`` -> packed sequence with a 3D RoPE grid.
|
||||
|
||||
``strength`` weakens the ref by mixing the latent toward a heavily
|
||||
blurred copy of itself (see ``_blur_latent``) rather than toward
|
||||
zero or random noise: scaling toward zero pushes values out of the
|
||||
normalized latent distribution (reads as grey output), and mixing
|
||||
toward random noise reads as real-but-garbled content and decodes
|
||||
as a wrong texture rather than "weaker" — a blurred copy stays
|
||||
on-manifold while still discarding the specific detail that makes a
|
||||
reference strong. ``strength <= 0`` drops the block entirely (no
|
||||
tokens injected).
|
||||
|
||||
``curve`` (optional) is a per-frame strength spec for the ref's
|
||||
latent timeline — a ``(direction, shape, value)`` tuple (e.g.
|
||||
``("concept_at_end", "ease", 1.0)``), a legacy preset name, a list of
|
||||
per-frame floats, ``(x, y)`` control points, or any object with
|
||||
``interp(x)`` (see ``curve_strengths``). Each frame is mixed with
|
||||
its own strength ``retention * curve(x)`` instead of one flat value,
|
||||
so the ref can fade in/out across the video. A flat curve is
|
||||
identical to passing ``strength`` alone.
|
||||
"""
|
||||
if strength <= 0.0:
|
||||
return None
|
||||
latent = self.latent
|
||||
strengths = curve_strengths(curve, self.latent_t) if curve is not None else None
|
||||
if strengths is not None:
|
||||
weights = [max(0.0, min(1.0, strength * value)) for value in strengths]
|
||||
if any(value < 1.0 for value in weights):
|
||||
shape = (1, 1, 1, self.latent_t) if self.kind == "audio" else (1, 1, self.latent_t, 1, 1)
|
||||
st = latent.new_tensor(weights).view(shape)
|
||||
latent = st * latent + (1.0 - st) * _blur_latent(latent)
|
||||
elif strength < 1.0:
|
||||
latent = strength * latent + (1.0 - strength) * _blur_latent(latent)
|
||||
if self.kind == "audio":
|
||||
return {"kind": "audio", "ref_audio_t": self.latent_t, "audio_latent": latent}
|
||||
block: Dict = {
|
||||
"kind": self.kind,
|
||||
"latent_h": self.latent_h,
|
||||
"latent_w": self.latent_w,
|
||||
"latent": latent,
|
||||
}
|
||||
if self.kind == "video":
|
||||
block["latent_t"] = self.latent_t
|
||||
block["ref_audio_t"] = 0
|
||||
block["audio_latent"] = None
|
||||
return block
|
||||
|
||||
# ── serialization ─────────────────────────────────────────────────
|
||||
|
||||
def metadata(self) -> dict:
|
||||
meta = {
|
||||
"name": self.name,
|
||||
"kind": self.kind,
|
||||
"latent_h": self.latent_h,
|
||||
"latent_w": self.latent_w,
|
||||
"latent_t": self.latent_t,
|
||||
"mode": self.mode,
|
||||
"source": self.source,
|
||||
"source_shape": self.source_shape,
|
||||
"pool": self.pool,
|
||||
"optimize_steps": self.optimize_steps,
|
||||
"tags": self.tags,
|
||||
"description": self.description,
|
||||
"concept_type": self.concept_type,
|
||||
"_format_version": 4,
|
||||
"sample_rate": self.sample_rate,
|
||||
}
|
||||
if self.config:
|
||||
meta["refmod_config"] = json.dumps(self.config)
|
||||
return meta
|
||||
|
||||
def save(self, path_no_ext: str) -> str:
|
||||
"""Save standalone, or update this member without dropping its siblings."""
|
||||
if self.bundle_index >= 0 and os.path.normcase(os.path.abspath(path_no_ext)) == os.path.normcase(os.path.abspath(self.path)):
|
||||
from .bundle import update_member
|
||||
return update_member(self)
|
||||
os.makedirs(os.path.dirname(path_no_ext) or ".", exist_ok=True)
|
||||
meta = self.metadata()
|
||||
destination = path_no_ext + ".safetensors"
|
||||
fd, temporary = tempfile.mkstemp(prefix=".refmod-", suffix=".tmp", dir=os.path.dirname(destination) or ".")
|
||||
os.close(fd)
|
||||
try:
|
||||
save_file({"latent": self.latent.contiguous()}, temporary,
|
||||
metadata={META_KEY: json.dumps(meta)})
|
||||
os.replace(temporary, destination)
|
||||
finally:
|
||||
if os.path.exists(temporary):
|
||||
os.unlink(temporary)
|
||||
return destination
|
||||
|
||||
@classmethod
|
||||
def load(cls, path_no_ext: str, device: str = "cpu") -> "H3RefMod":
|
||||
"""Load from ``{path}.safetensors`` (metadata in header, or legacy .json)."""
|
||||
meta = read_refmod_meta(path_no_ext)
|
||||
if not isinstance(meta, dict):
|
||||
raise ValueError(
|
||||
f"{path_no_ext}.safetensors has no RefMod metadata "
|
||||
f"(header key '{META_KEY}' or sidecar .json missing).")
|
||||
if meta.get("kind") == "bundle":
|
||||
raise ValueError("This is a RefMod bundle. Use the current RefMod Loader to select its components.")
|
||||
# Clone releases the file mapping before a possible Windows overwrite.
|
||||
latent = load_file(path_no_ext + ".safetensors", device=device)["latent"].clone()
|
||||
return cls.from_metadata(meta, latent, path_no_ext)
|
||||
|
||||
@classmethod
|
||||
def from_metadata(cls, meta, latent, path_no_ext="", bundle_index=-1):
|
||||
raw_config = meta.get("refmod_config")
|
||||
try:
|
||||
config = json.loads(raw_config) if isinstance(raw_config, str) else {}
|
||||
except ValueError:
|
||||
config = {}
|
||||
return cls(
|
||||
name=meta.get("name", os.path.basename(path_no_ext)),
|
||||
kind=meta.get("kind", "image"),
|
||||
latent=latent,
|
||||
latent_h=int(meta.get("latent_h", latent.shape[3] if latent.ndim == 5 else 0)),
|
||||
latent_w=int(meta.get("latent_w", latent.shape[4] if latent.ndim == 5 else 0)),
|
||||
latent_t=int(meta.get("latent_t", latent.shape[2] if latent.ndim == 5 else latent.shape[-1])),
|
||||
mode=normalize_mode(meta.get("mode", "training")),
|
||||
source=meta.get("source", ""),
|
||||
source_shape=meta.get("source_shape", ""),
|
||||
pool=meta.get("pool", ""),
|
||||
optimize_steps=int(meta.get("optimize_steps", 0)),
|
||||
tags=list(meta.get("tags", [])),
|
||||
description=str(meta.get("description", "") or ""),
|
||||
concept_type=str(meta.get("concept_type", "generic") or "generic"),
|
||||
config=config if isinstance(config, dict) else {},
|
||||
path=path_no_ext,
|
||||
bundle_index=bundle_index,
|
||||
sample_rate=int(meta.get("sample_rate", 32000)),
|
||||
)
|
||||
|
||||
@@ -0,0 +1,205 @@
|
||||
"""
|
||||
debug_grid.py — 1024x1024 curve-graph visualization for Apply H3 RefMod.
|
||||
|
||||
Renders the per-frame strength envelope (direction/shape/value) across the
|
||||
full canvas, with the "concept zone" shaded where the concept shows up. It
|
||||
is both the Apply's optional debug IMAGE output and the saved graph-preset
|
||||
PNG (graph embedded in the image's tEXt metadata).
|
||||
|
||||
Pure PIL; ``pil_to_tensor`` converts the result to a torch IMAGE tensor.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
from typing import Dict, Optional, Tuple
|
||||
|
||||
import numpy as np
|
||||
import torch
|
||||
from PIL import Image, ImageDraw, ImageFont, PngImagePlugin
|
||||
|
||||
from .core import CURVE_DIRECTIONS, CURVE_SHAPES, curve_strengths
|
||||
|
||||
SIZE = 1024
|
||||
|
||||
BG = (18, 20, 24)
|
||||
TITLE = (232, 230, 227)
|
||||
TEXT = (207, 203, 196)
|
||||
DIM = (107, 111, 118)
|
||||
GRID_LINE = (42, 46, 54)
|
||||
ACCENT = (255, 158, 61)
|
||||
ACCENT_FILL = (58, 34, 15)
|
||||
ACCENT_DIM = (122, 69, 21)
|
||||
|
||||
_CURVE_SAMPLES = 160
|
||||
|
||||
# where the concept shows up in the output, per direction — the envelope is
|
||||
# drawn on the same timeline, so it is strong exactly where the zone is
|
||||
_ZONES = {
|
||||
"concept_at_start": ((0.0, 1.0 / 3.0),),
|
||||
"concept_at_middle": ((1.0 / 3.0, 2.0 / 3.0),),
|
||||
"concept_at_end": ((2.0 / 3.0, 1.0),),
|
||||
"concept_at_ends": ((0.0, 1.0 / 3.0), (2.0 / 3.0, 1.0)),
|
||||
}
|
||||
|
||||
|
||||
# ── fonts ────────────────────────────────────────────────────────────────────
|
||||
|
||||
_FONT_CACHE: Dict[Tuple[int, bool, bool], ImageFont.FreeTypeFont] = {}
|
||||
|
||||
|
||||
def _font(size: int, bold: bool = False, mono: bool = False):
|
||||
key = (size, bold, mono)
|
||||
if key in _FONT_CACHE:
|
||||
return _FONT_CACHE[key]
|
||||
candidates = []
|
||||
if os.name == "nt":
|
||||
base = "C:/Windows/Fonts/"
|
||||
name = "consola.ttf" if mono else ("arialbd.ttf" if bold else "arial.ttf")
|
||||
candidates = [base + name]
|
||||
else:
|
||||
if mono:
|
||||
candidates = [
|
||||
"/usr/share/fonts/truetype/dejavu/DejaVuSansMono.ttf",
|
||||
"/System/Library/Fonts/Menlo.ttc",
|
||||
]
|
||||
else:
|
||||
candidates = [
|
||||
"/usr/share/fonts/truetype/dejavu/DejaVuSans-Bold.ttf" if bold
|
||||
else "/usr/share/fonts/truetype/dejavu/DejaVuSans.ttf",
|
||||
"/System/Library/Fonts/Supplemental/Arial.ttf",
|
||||
]
|
||||
for path in candidates:
|
||||
if os.path.isfile(path):
|
||||
try:
|
||||
f = ImageFont.truetype(path, size)
|
||||
_FONT_CACHE[key] = f
|
||||
return f
|
||||
except OSError:
|
||||
pass
|
||||
try:
|
||||
f = ImageFont.load_default(size=size)
|
||||
except Exception:
|
||||
f = ImageFont.load_default()
|
||||
_FONT_CACHE[key] = f
|
||||
return f
|
||||
|
||||
|
||||
def _text_w(draw: ImageDraw.ImageDraw, text: str, font) -> int:
|
||||
l, _t, r, _b = draw.textbbox((0, 0), text, font=font)
|
||||
return r - l
|
||||
|
||||
|
||||
# ── the curve graph ──────────────────────────────────────────────────────────
|
||||
|
||||
def render_debug_grid(spec, preset_name: str = "") -> Image.Image:
|
||||
"""Render the 1024x1024 curve graph for an Apply configuration.
|
||||
|
||||
``spec`` is the resolved (direction, shape, value) curve tuple and
|
||||
``preset_name`` the active graph preset (shown when one is loaded).
|
||||
"""
|
||||
img = Image.new("RGB", (SIZE, SIZE), BG)
|
||||
draw = ImageDraw.Draw(img)
|
||||
|
||||
draw.text((32, 26), "curve graph", font=_font(28, bold=True), fill=TITLE)
|
||||
if preset_name:
|
||||
draw.text((SIZE - 32, 34), f"preset: {preset_name}",
|
||||
font=_font(20), fill=DIM, anchor="rs")
|
||||
|
||||
x0, y0, x1, y1 = 90, 120, SIZE - 10, 940
|
||||
direction = spec[0]
|
||||
|
||||
# concept zone shading (where the concept shows up in the output)
|
||||
for start, end in _ZONES.get(direction, ()):
|
||||
zx0 = x0 + start * (x1 - x0)
|
||||
zx1 = x0 + end * (x1 - x0)
|
||||
draw.rectangle([zx0, y0, zx1, y1], fill=ACCENT_FILL)
|
||||
draw.text((zx0 + 8, y1 + 8), "concept zone", font=_font(20), fill=ACCENT_DIM)
|
||||
|
||||
# grid + axis labels
|
||||
for f in (0.0, 0.25, 0.5, 0.75, 1.0):
|
||||
gy = y1 - f * (y1 - y0)
|
||||
draw.line([x0, gy, x1, gy], fill=GRID_LINE, width=2)
|
||||
draw.text((16, gy - 12), f"{1.0 - f:.2f}", font=_font(20), fill=DIM)
|
||||
for fx, label in ((0.0, "0"), (0.25, "0.25"), (0.5, "0.5"),
|
||||
(0.75, "0.75"), (1.0, "1")):
|
||||
gx = x0 + fx * (x1 - x0)
|
||||
draw.line([gx, y0, gx, y1], fill=GRID_LINE, width=2)
|
||||
draw.text((gx, y1 + 8), label, font=_font(20), fill=DIM, anchor="ma")
|
||||
draw.text((x1 - 6, y1 + 8), "video →", font=_font(20), fill=DIM, anchor="ra")
|
||||
|
||||
# the envelope
|
||||
strengths = curve_strengths(spec, _CURVE_SAMPLES)
|
||||
if strengths is None:
|
||||
strengths = [1.0] * _CURVE_SAMPLES
|
||||
pts = [(x0 + i * (x1 - x0) / (_CURVE_SAMPLES - 1),
|
||||
y1 - s * (y1 - y0)) for i, s in enumerate(strengths)]
|
||||
draw.polygon([(x0, y1)] + pts + [(x1, y1)], fill=ACCENT_FILL)
|
||||
draw.line(pts, fill=ACCENT, width=6)
|
||||
|
||||
# peak marker
|
||||
peak = max(range(_CURVE_SAMPLES), key=lambda i: strengths[i])
|
||||
px, py = pts[peak]
|
||||
draw.line([px, y0, px, y1], fill=ACCENT_DIM, width=2)
|
||||
draw.ellipse([px - 9, py - 9, px + 9, py + 9], fill=ACCENT)
|
||||
|
||||
# spec readout on a dark backdrop
|
||||
lines = [f"direction: {direction}", f"shape: {spec[1]}",
|
||||
f"value: {float(spec[2]):.2f}"]
|
||||
font = _font(24, mono=True)
|
||||
tw = max(_text_w(draw, ln, font) for ln in lines)
|
||||
draw.rectangle([x0 + 18, y0 + 18, x0 + 18 + tw + 24, y0 + 18 + 3 * 34 + 12],
|
||||
fill=(14, 17, 22))
|
||||
for i, ln in enumerate(lines):
|
||||
draw.text((x0 + 30, y0 + 28 + i * 34), ln, font=font, fill=TEXT)
|
||||
|
||||
return img
|
||||
|
||||
|
||||
# ── graph presets (PNG metadata) ────────────────────────────────────────────
|
||||
|
||||
# The graph spec is embedded in the preset PNG's tEXt chunk, so a saved curve
|
||||
# graph IS the preset: share the image, and the curve comes with it.
|
||||
GRAPH_META_KEY = "graph"
|
||||
|
||||
|
||||
def graph_pnginfo(spec):
|
||||
"""PIL pnginfo carrying the (direction, shape, value) spec as tEXt."""
|
||||
info = PngImagePlugin.PngInfo()
|
||||
info.add_text(GRAPH_META_KEY, json.dumps({
|
||||
"direction": spec[0], "shape": spec[1], "value": float(spec[2])}))
|
||||
return info
|
||||
|
||||
|
||||
def read_graph_meta(path: str) -> Optional[tuple]:
|
||||
"""Read a (direction, shape, value) spec from a PNG's tEXt metadata.
|
||||
|
||||
Returns None for a missing file, a missing/empty chunk, or an invalid
|
||||
spec (unknown direction/shape, bad value) so callers can fall back.
|
||||
"""
|
||||
try:
|
||||
with Image.open(path) as img:
|
||||
raw = img.text.get(GRAPH_META_KEY)
|
||||
except Exception:
|
||||
return None
|
||||
if not raw:
|
||||
return None
|
||||
try:
|
||||
data = json.loads(raw)
|
||||
except ValueError:
|
||||
return None
|
||||
direction, shape = data.get("direction"), data.get("shape")
|
||||
if direction not in CURVE_DIRECTIONS or shape not in CURVE_SHAPES:
|
||||
return None
|
||||
try:
|
||||
value = float(data.get("value", 1.0))
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
return (direction, shape, value)
|
||||
|
||||
|
||||
def pil_to_tensor(img: Image.Image) -> torch.Tensor:
|
||||
"""Convert a rendered PIL graph to a [1, 1024, 1024, 3] float32 tensor."""
|
||||
arr = np.asarray(img).copy()
|
||||
return torch.from_numpy(arr).float().unsqueeze(0) / 255.0 # [1, 1024, 1024, 3]
|
||||
|
After Width: | Height: | Size: 110 KiB |
|
After Width: | Height: | Size: 2.9 MiB |
|
After Width: | Height: | Size: 4.1 MiB |
|
After Width: | Height: | Size: 2.4 MiB |
|
After Width: | Height: | Size: 7.0 MiB |
|
After Width: | Height: | Size: 5.7 MiB |
|
After Width: | Height: | Size: 71 KiB |
|
After Width: | Height: | Size: 116 KiB |
@@ -0,0 +1,271 @@
|
||||
"""
|
||||
extract_mod.py — standalone MiniMax H3 RefMod extractor (no training)
|
||||
|
||||
Turns reference images/videos into a ``.safetensors`` mod that the ComfyUI
|
||||
Load H3 RefMods / Apply nodes can inject into MiniMax H3 generation. Only
|
||||
the H3 video VAE is needed; the 29B DiT is never loaded.
|
||||
|
||||
Two modes:
|
||||
|
||||
* ``encode`` (default) — refs are resized to ``--resolution`` short edge (down
|
||||
only) and encoded at that resolution, exactly like the official ref2video
|
||||
node. This is what carries identity (a face, an outfit); files are
|
||||
~0.2-1 MB per frame. (Old name: ``full``.)
|
||||
* ``training`` — refs are resized to ``--resolution`` short edge too (the
|
||||
latent is pooled to a tiny grid anyway, so encoding at native resolution
|
||||
is wasted compute), then average-pooled to a small grid (4x4 by default
|
||||
= 4 tokens per latent frame) and optionally refined with a few gradient
|
||||
steps (still model-free; that refinement loop is the only "training" in
|
||||
the whole pack). Nearly free to inject but only carries concept / motion,
|
||||
not fine identity. (Old name: ``pooled``.)
|
||||
|
||||
Usage
|
||||
-----
|
||||
# full-res identity mod (recommended for characters)
|
||||
python custom_nodes/ComfyUI-MiniMaxH3Mod/extract_mod.py \
|
||||
--image char.png --vae path/to/h3_video_vae.safetensors \
|
||||
--name my_character --mode encode --resolution 1024
|
||||
|
||||
# tiny concept/motion mod
|
||||
python custom_nodes/ComfyUI-MiniMaxH3Mod/extract_mod.py \
|
||||
--video dance.mp4 --vae path/to/h3_video_vae.safetensors \
|
||||
--name dance --mode training --pool 4 --latent-frames 2
|
||||
|
||||
Multi-reference concept (each ref becomes its own latent frame):
|
||||
|
||||
python custom_nodes/ComfyUI-MiniMaxH3Mod/extract_mod.py \
|
||||
--image face_a.png --image face_b.png --image full.png \
|
||||
--video dance.mp4 --vae path/to/h3_video_vae.safetensors \
|
||||
--name disney_char --mode encode --resolution 1024
|
||||
|
||||
Output goes to ``ComfyUI/models/refmods/`` by default (created on first run,
|
||||
next to loras/ and unet/).
|
||||
|
||||
Run it with the same Python/venv that runs ComfyUI (it imports ``comfy``
|
||||
from the install the script lives in).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import sys
|
||||
|
||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
if HERE not in sys.path:
|
||||
sys.path.insert(0, HERE)
|
||||
ROOT = os.path.abspath(os.path.join(HERE, "..", ".."))
|
||||
if ROOT not in sys.path:
|
||||
sys.path.insert(0, ROOT)
|
||||
|
||||
import torch
|
||||
|
||||
from common import (load_image_file, load_video_file, refmods_dir, mod_output_path,
|
||||
resize_ref as _resize_ref, ensure_min_size, sample_video_for_vae)
|
||||
from core import (CONCEPT_TYPES, H3RefMod, aspect_grid, fit_token_budget,
|
||||
normalize_mode, optimize_latent, pool_latent)
|
||||
|
||||
MAX_VIDEO_FRAMES = 60 # uniform sample cap; temporal pooling averages anyway
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════════════
|
||||
# Media loading (delegates to common.py: image/video -> [B, H, W, 3] float32)
|
||||
# ═══════════════════════════════════════════════════════════════════════════
|
||||
|
||||
def _load_image(path: str, max_edge: int) -> torch.Tensor:
|
||||
return load_image_file(path, max_edge=max_edge)
|
||||
|
||||
|
||||
def _load_video(path: str, max_edge: int, max_frames: int = MAX_VIDEO_FRAMES) -> torch.Tensor:
|
||||
return load_video_file(path, max_frames=max_frames, max_edge=max_edge)
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════════════
|
||||
# Main
|
||||
# ═══════════════════════════════════════════════════════════════════════════
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(
|
||||
description="Extract a MiniMax H3 RefMod from one or more images/videos "
|
||||
"of the same concept (they are stacked into a video-kind mod)")
|
||||
ap.add_argument("--image", action="append", default=[], metavar="PATH",
|
||||
help="reference image path (repeatable for multi-view concepts)")
|
||||
ap.add_argument("--video", action="append", default=[], metavar="PATH",
|
||||
help="reference video path (repeatable)")
|
||||
ap.add_argument("--vae", required=True, help="MiniMax H3 video VAE .safetensors")
|
||||
ap.add_argument("--name", default=None, help="mod name (default: first source file stem)")
|
||||
ap.add_argument("--subfolder", default="", help="optional subfolder inside models/refmods")
|
||||
ap.add_argument("--output", default=None,
|
||||
help="output dir (default: ComfyUI/models/refmods)")
|
||||
ap.add_argument("--mode", choices=["training", "encode", "full", "pooled"],
|
||||
default="training",
|
||||
help="training = compressed grid refined by --identity (default, good balance); encode = straight full-res VAE encode (~1K tokens/img); full/pooled = old names, still accepted")
|
||||
ap.add_argument("--resolution", type=int, default=1024,
|
||||
help="both modes: target short edge in px, downscale only (default 1024; 2048 = max fidelity). training mode encodes at this size too — the main speed dial for it")
|
||||
ap.add_argument("--pool", type=int, default=16, help="training mode: spatial latent grid (even, default 16)")
|
||||
ap.add_argument("--pool-w", type=int, default=None, help="pool width (default: == --pool)")
|
||||
ap.add_argument("--latent-frames", type=int, default=16,
|
||||
help="frames kept per video ref (default 16; images use 1): training mode pools latent frames; encode mode samples source frames on H3's 17k+5 grid (16 selects 5, below 5 selects the first image)")
|
||||
ap.add_argument("--identity", type=int, default=500,
|
||||
help="training mode: how tightly the mod clings to the reference (gradient refinement steps; default 500, 0 = pure pooling)")
|
||||
ap.add_argument("--multiplier", type=int, default=1,
|
||||
help="data multiplier: repeat the extracted ref N times along time so a short video/GIF isn't drowned out by the main video's tokens (default 1 = no repeat)")
|
||||
ap.add_argument("--max-tokens", type=int, default=5120,
|
||||
help="hard cap on the total injected tokens (0 = off; default 5120): drops near-duplicate latent frames first, then resamples the rest to fit")
|
||||
ap.add_argument("--max-edge", type=int, default=1536,
|
||||
help="resize source so the longest edge is <= this before encoding (default 1536)")
|
||||
ap.add_argument("--max-frames", type=int, default=60,
|
||||
help="max video frames sampled (uniform) before encoding (default 60; lower for CPU)")
|
||||
ap.add_argument("--description", default="",
|
||||
help="optional text describing the concept (e.g. 'a ginger woman with messy hair', "
|
||||
"'an animation style'). Stored in the mod and emitted by the loaders so it "
|
||||
"can be merged into the prompt without retyping it.")
|
||||
ap.add_argument("--concept-type", choices=list(CONCEPT_TYPES), default="generic",
|
||||
help="what this mod represents (default: generic). 'identity' in --mode training "
|
||||
"prints a warning: pooling is lossy in exactly the way that destroys faces — "
|
||||
"use --mode encode for people.")
|
||||
ap.add_argument("--device", default="auto", help="auto / cuda / mps / cpu (VAE device)")
|
||||
args = ap.parse_args()
|
||||
args.mode = normalize_mode(args.mode) # accept legacy 'full'/'pooled'
|
||||
|
||||
if not args.image and not args.video:
|
||||
ap.error("provide at least one --image or --video")
|
||||
if args.concept_type == "identity" and args.mode == "training" and args.pool < 16:
|
||||
print(
|
||||
f"[extract] warning: --concept-type identity with --mode training at "
|
||||
f"--pool {args.pool} — pooling averages away exactly the detail that "
|
||||
f"carries a face. Use --mode encode for people, or at least --pool 32."
|
||||
)
|
||||
|
||||
import comfy.model_management
|
||||
import comfy.sd
|
||||
import comfy.utils
|
||||
|
||||
device = comfy.model_management.get_torch_device() if args.device == "auto" \
|
||||
else torch.device(args.device)
|
||||
|
||||
# ── load VAE ─────────────────────────────────────────────────────
|
||||
print(f"[extract] loading VAE {args.vae} (device={args.device})")
|
||||
sd, metadata = comfy.utils.load_torch_file(args.vae, return_metadata=True)
|
||||
vae = comfy.sd.VAE(sd=sd, metadata=metadata, device=device)
|
||||
vae.throw_exception_if_invalid()
|
||||
|
||||
# ── encode each source (full-res or pooled), then stack along time ──
|
||||
pool_w = args.pool_w or args.pool
|
||||
# in encode mode --resolution is the target short edge, so keep the
|
||||
# load-time longest-edge cap from interfering with it
|
||||
# both modes resize to --resolution before encode now, so the load-time
|
||||
# longest-edge cap just needs to stay ahead of that target
|
||||
load_max_edge = max(args.resolution * 2, args.max_edge)
|
||||
# full-res refs must share one spatial canvas so the stacked latent has a
|
||||
# single H/W: anchor on the first source, cover-crop the rest to it
|
||||
canvas = None
|
||||
if args.mode == "encode" and len(args.image) + len(args.video) > 1:
|
||||
if args.image:
|
||||
first = _load_image(args.image[0], load_max_edge)
|
||||
else:
|
||||
first = _load_video(args.video[0], load_max_edge, args.max_frames)
|
||||
h, w = first.shape[1], first.shape[2]
|
||||
scale = min(1.0, args.resolution / min(h, w))
|
||||
canvas = (max(32, round(w * scale / 32) * 32),
|
||||
max(32, round(h * scale / 32) * 32))
|
||||
# training mode: anchor the pool grid to the first source's aspect so a
|
||||
# portrait person isn't squished into a square grid ("fat" distortion).
|
||||
pool_grid = None
|
||||
if args.mode == "training":
|
||||
if args.image:
|
||||
first = _load_image(args.image[0], load_max_edge)
|
||||
else:
|
||||
first = _load_video(args.video[0], load_max_edge, args.max_frames)
|
||||
h0, w0 = first.shape[1], first.shape[2]
|
||||
pool_grid = aspect_grid(args.pool, pool_w, h0 / w0)
|
||||
if pool_grid != (args.pool, pool_w):
|
||||
print(f"[extract] pooled grid {args.pool}x{pool_w} -> "
|
||||
f"{pool_grid[0]}x{pool_grid[1]} to match source aspect "
|
||||
f"{w0}x{h0} (avoids squishing the subject wide)")
|
||||
|
||||
frames = []
|
||||
shapes = []
|
||||
n_img = n_vid = 0
|
||||
for path in args.image:
|
||||
src = _load_image(path, load_max_edge)
|
||||
src = ensure_min_size(_resize_ref(src, args.resolution, canvas))
|
||||
print(f"[extract] image {path}: {tuple(src.shape)} (mode={args.mode})")
|
||||
with torch.no_grad():
|
||||
z = vae.encode(src.to(device)).float().cpu()
|
||||
if args.mode == "encode":
|
||||
pooled = z.to(torch.float16)
|
||||
else:
|
||||
gh, gw = pool_grid if pool_grid is not None else (args.pool, pool_w)
|
||||
pooled = pool_latent(z, 1, gh, gw).to(torch.float16)
|
||||
if args.identity > 0:
|
||||
pooled = optimize_latent(pooled, z, steps=args.identity, device=device,
|
||||
progress_every=100)
|
||||
frames.append(pooled)
|
||||
shapes.append(f"{z.shape[2]}x{z.shape[3]}x{z.shape[4]}")
|
||||
n_img += 1
|
||||
for path in args.video:
|
||||
src = _load_video(path, load_max_edge, args.max_frames)
|
||||
src = sample_video_for_vae(src, args.latent_frames if args.mode == "encode" else None)
|
||||
src = ensure_min_size(_resize_ref(src, args.resolution, canvas))
|
||||
print(f"[extract] video {path}: {tuple(src.shape)} (mode={args.mode})")
|
||||
with torch.no_grad():
|
||||
z = vae.encode(src.to(device)).float().cpu()
|
||||
if args.mode == "encode":
|
||||
pooled = z.to(torch.float16)
|
||||
else:
|
||||
pool_t = min(args.latent_frames, z.shape[2])
|
||||
gh, gw = pool_grid if pool_grid is not None else (args.pool, pool_w)
|
||||
pooled = pool_latent(z, pool_t, gh, gw).to(torch.float16)
|
||||
if args.identity > 0:
|
||||
pooled = optimize_latent(pooled, z, steps=args.identity, device=device,
|
||||
progress_every=100)
|
||||
frames.append(pooled)
|
||||
shapes.append(f"{z.shape[2]}x{z.shape[3]}x{z.shape[4]}")
|
||||
n_vid += 1
|
||||
|
||||
if args.mode == "encode" and args.identity > 0:
|
||||
print(f"[extract] warning: --identity only applies to training mode — "
|
||||
f"encode mode stores the actual encode, so --identity {args.identity} "
|
||||
f"was ignored.")
|
||||
|
||||
latent = torch.cat(frames, dim=2)
|
||||
if args.multiplier > 1:
|
||||
latent = latent.repeat(1, 1, args.multiplier, 1, 1)
|
||||
name = args.name or (os.path.splitext(os.path.basename(args.image[0]))[0]
|
||||
if args.image else os.path.splitext(os.path.basename(args.video[0]))[0])
|
||||
if args.max_tokens > 0:
|
||||
latent = fit_token_budget(latent, args.max_tokens, name)
|
||||
total_t = latent.shape[2]
|
||||
kind = "video" if total_t > 1 else "image"
|
||||
|
||||
out_dir = args.output or refmods_dir()
|
||||
px_w, px_h = latent.shape[4] * 16, latent.shape[3] * 16
|
||||
mod = H3RefMod(
|
||||
name=name,
|
||||
kind=kind,
|
||||
latent=latent,
|
||||
latent_h=latent.shape[3],
|
||||
latent_w=latent.shape[4],
|
||||
latent_t=total_t,
|
||||
mode=args.mode,
|
||||
source="stack" if len(frames) > 1 else ("video" if n_vid else "image"),
|
||||
source_shape=" +".join(shapes),
|
||||
pool=f"full-res {px_w}x{px_h}px (short-edge cap {args.resolution}px)" if args.mode == "encode" else f"{total_t}x{gh}x{gw}",
|
||||
optimize_steps=args.identity if args.mode == "training" else 0,
|
||||
tags=[f"{n_img} img, {n_vid} vid"] + ([f"x{args.multiplier} repeat"] if args.multiplier > 1 else []),
|
||||
description=args.description.strip(),
|
||||
concept_type=args.concept_type,
|
||||
)
|
||||
mod.path = os.path.join(out_dir, name) if args.output else mod_output_path(name, args.subfolder) # so Fix H3 RefMod Config can re-save in place
|
||||
path = mod.save(mod.path)
|
||||
mb = latent.numel() * latent.element_size() / 1024 / 1024
|
||||
print(f"[extract] saved {kind} mod '{name}' "
|
||||
f"({mod.token_count} tokens, {mb:.2f} MB) -> {path}")
|
||||
print(f"[extract] load it in ComfyUI with the Load H3 RefMods node "
|
||||
f"(dropdown '{name}', strength 1.0 = full ref), then chain "
|
||||
f"Apply H3 RefMod (Cond) into your sampling.")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,62 @@
|
||||
"""Benchmark the three production multi-ref strategies; reject parity failures.
|
||||
Run: python gauntlet_harness.py --device cuda --refs 4 --frames 8 --edge 64
|
||||
Run regressions: python -m unittest discover -s tests -v
|
||||
"""
|
||||
import argparse
|
||||
import gc
|
||||
import json
|
||||
import time
|
||||
import sys
|
||||
from pathlib import Path
|
||||
import torch
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||
from core import optimize_latent_multi
|
||||
|
||||
|
||||
def synchronize(device):
|
||||
device = torch.device(device)
|
||||
if device.type == "cuda":
|
||||
torch.cuda.synchronize(device)
|
||||
elif device.type == "mps":
|
||||
torch.mps.synchronize()
|
||||
|
||||
|
||||
def run():
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--device", default="cuda" if torch.cuda.is_available() else "cpu")
|
||||
parser.add_argument("--refs", type=int, default=4)
|
||||
parser.add_argument("--frames", type=int, default=8)
|
||||
parser.add_argument("--edge", type=int, default=64)
|
||||
parser.add_argument("--steps", type=int, default=20)
|
||||
parser.add_argument("--output", default="tests/gauntlet_result.json")
|
||||
args = parser.parse_args()
|
||||
device = torch.device(args.device)
|
||||
torch.manual_seed(123)
|
||||
initial = torch.randn(1,24,2,8,8)
|
||||
targets = [torch.randn(1,24,args.frames,args.edge,args.edge) for _ in range(args.refs)]
|
||||
# Warm up optimizer and device initialization outside the measurements.
|
||||
optimize_latent_multi(initial, targets[:1], steps=1, device=args.device, strategy="stream")
|
||||
results, baseline = [], None
|
||||
for strategy in ("resident", "stream", "grouped"):
|
||||
gc.collect()
|
||||
if device.type == "cuda":
|
||||
torch.cuda.empty_cache()
|
||||
torch.cuda.reset_peak_memory_stats(device)
|
||||
synchronize(device)
|
||||
start = time.perf_counter()
|
||||
value = optimize_latent_multi(initial, targets, steps=args.steps, device=args.device, strategy=strategy)
|
||||
synchronize(device)
|
||||
elapsed = time.perf_counter() - start
|
||||
peak = torch.cuda.max_memory_allocated(device)/1024**2 if device.type == "cuda" else None
|
||||
value = value.cpu()
|
||||
if baseline is None: baseline = value
|
||||
error = (value-baseline).abs().max().item()
|
||||
torch.testing.assert_close(value,baseline,rtol=1e-4,atol=1e-5)
|
||||
results.append({"strategy":strategy,"seconds":elapsed,"peak_cuda_mib":peak,"max_error":error,"parity":True})
|
||||
del value
|
||||
Path(args.output).write_text(json.dumps({"parameters":vars(args),"results":results},indent=2),encoding="utf-8")
|
||||
print(json.dumps(results,indent=2))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
run()
|
||||
@@ -0,0 +1,209 @@
|
||||
"""Issue #21-23 regressions. Run with ComfyUI's Python; no model weights needed.
|
||||
|
||||
python issue_regression_test.py
|
||||
"""
|
||||
import asyncio
|
||||
import contextlib
|
||||
import importlib
|
||||
import io
|
||||
import json
|
||||
from pathlib import Path
|
||||
import sys
|
||||
import tempfile
|
||||
import types
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
|
||||
ROOT = Path(__file__).resolve().parent
|
||||
sys.path.insert(0, str(ROOT.parents[1]))
|
||||
from comfy.cli_args import args
|
||||
args.cpu = True
|
||||
import folder_paths
|
||||
import torch
|
||||
with tempfile.TemporaryDirectory() as import_dir:
|
||||
with patch.object(folder_paths, "models_dir", import_dir):
|
||||
N = importlib.import_module(f"custom_nodes.{ROOT.name}.nodes")
|
||||
import execution
|
||||
import nodes as comfy_nodes
|
||||
COMMON = importlib.import_module(N.__package__ + ".common")
|
||||
|
||||
|
||||
class IssueRegressions(unittest.TestCase):
|
||||
def setUp(self):
|
||||
temp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(temp.cleanup)
|
||||
self.root = Path(temp.name) / "refmods"
|
||||
self.root.mkdir()
|
||||
N._MOD_CACHE.clear()
|
||||
N._MOD_CACHE_STAMPS.clear()
|
||||
self.addCleanup(N._MOD_CACHE.clear)
|
||||
output = contextlib.redirect_stdout(io.StringIO())
|
||||
output.__enter__()
|
||||
self.addCleanup(output.__exit__, None, None, None)
|
||||
|
||||
def test_creators_are_queue_outputs_without_save_node(self):
|
||||
for cls in (N.MiniMaxH3RefModExtract, N.MiniMaxH3RefModMasterExtract):
|
||||
cls.INPUT_TYPES() # initialize the V3 compatibility attributes
|
||||
schema = cls.define_schema()
|
||||
self.assertTrue(schema.is_output_node)
|
||||
self.assertEqual(len(schema.outputs), 2)
|
||||
inputs = {}
|
||||
for key, (kind, options) in cls.INPUT_TYPES()["required"].items():
|
||||
if "default" in options:
|
||||
inputs[key] = options["default"]
|
||||
prompt = {"create": {"class_type": schema.node_id, "inputs": inputs}}
|
||||
with patch.dict(comfy_nodes.NODE_CLASS_MAPPINGS, {schema.node_id: cls}):
|
||||
result = asyncio.run(execution.validate_prompt("creator", prompt, None))
|
||||
self.assertTrue(result[0], result)
|
||||
self.assertIn("create", result[2])
|
||||
|
||||
def test_create_details_report_save_and_no_save(self):
|
||||
vae = types.SimpleNamespace(encode=lambda pixels: torch.ones(1,24,1,4,4))
|
||||
for save in (False, True):
|
||||
with patch.object(COMMON, "refmods_dir", return_value=str(self.root)):
|
||||
result = N.MiniMaxH3RefModExtract.execute(
|
||||
"details", mode="encode", refs_image={"ref_image_0": torch.zeros(1,64,64,3)},
|
||||
vae=vae, save=save)
|
||||
details = json.loads(result[1])
|
||||
self.assertEqual(len(details["saved_paths"]), int(save))
|
||||
self.assertEqual(details["tokens"], result[0][0][0].token_count)
|
||||
self.assertEqual(result.ui.as_dict()["text"], (result[1],))
|
||||
|
||||
def test_mask_list_keeps_sizes_and_maps_each_reference(self):
|
||||
masks = [torch.zeros(1,64,96), torch.ones(1,96,64)]
|
||||
collected = N.MiniMaxH3RefModMaskList().collect(masks)[0]
|
||||
self.assertEqual([tuple(m.shape) for m in collected], [(1,64,96),(1,96,64)])
|
||||
refs = {"ref_image_0": torch.zeros(1,64,96,3), "ref_image_1": torch.zeros(1,96,64,3)}
|
||||
seen = []
|
||||
def mask_latent(z, mask_px, *args, **kwargs):
|
||||
seen.append(mask_px.clone())
|
||||
return z
|
||||
vae = types.SimpleNamespace(encode=lambda pixels: torch.ones(1,24,1,4,4))
|
||||
for cls in (N.MiniMaxH3RefModExtract, N.MiniMaxH3RefModMasterExtract):
|
||||
seen.clear()
|
||||
with patch.object(N, "_mask_latent", side_effect=mask_latent):
|
||||
cls.execute("masked", mode="training", identity=0, refs_image=refs,
|
||||
vae=vae, mask_list=collected, save=False)
|
||||
self.assertEqual(len(seen), 2)
|
||||
self.assertEqual([tuple(m.shape) for m in seen], [(1,320,480),(1,480,320)])
|
||||
self.assertEqual([m.mean().item() for m in seen], [0,1])
|
||||
|
||||
def test_mask_list_consumed_once_by_comfy_execution(self):
|
||||
masks = [torch.zeros(1,64,96), torch.ones(1,96,64)]
|
||||
node = N.MiniMaxH3RefModMaskList()
|
||||
result = asyncio.run(execution._async_map_node_over_list(
|
||||
"masks", "collect", node, {"masks": masks}, node.FUNCTION))
|
||||
self.assertEqual(len(result), 1)
|
||||
self.assertEqual(len(result[0][0]), 2)
|
||||
|
||||
def test_mixed_aspect_masks_follow_reference_resize_and_crop(self):
|
||||
masks = [torch.linspace(0,1,640).expand(320,640).unsqueeze(0),
|
||||
torch.linspace(0,1,640).view(640,1).expand(640,320).unsqueeze(0)]
|
||||
refs = {f"ref_image_{i}": mask.unsqueeze(-1).expand(-1,-1,-1,3)
|
||||
for i, mask in enumerate(masks)}
|
||||
pixels_seen, masks_seen = [], []
|
||||
def encode(pixels):
|
||||
pixels_seen.append(pixels[...,0].clone())
|
||||
return torch.ones(1,24,1,pixels.shape[1]//16,pixels.shape[2]//16)
|
||||
def apply_mask(z, mask_px, *args, **kwargs):
|
||||
masks_seen.append(mask_px.clone())
|
||||
return z
|
||||
for mode in ("training", "encode"):
|
||||
pixels_seen.clear()
|
||||
masks_seen.clear()
|
||||
with patch.object(N, "_mask_latent", side_effect=apply_mask):
|
||||
N.MiniMaxH3RefModExtract.execute(
|
||||
"alignment", mode=mode, identity=0, refs_image=refs,
|
||||
mask_list=masks, vae=types.SimpleNamespace(encode=encode), save=False)
|
||||
for pixels, mask in zip(pixels_seen, masks_seen):
|
||||
# Lanczos/PIL images and float bilinear masks differ by up to
|
||||
# two uint8 levels from quantization and interpolation kernels.
|
||||
torch.testing.assert_close(mask, pixels, atol=2/255, rtol=0)
|
||||
|
||||
def test_mask_broadcast_batch_and_mapping_errors(self):
|
||||
vae = types.SimpleNamespace(encode=lambda pixels: torch.ones(1,24,1,4,4))
|
||||
args = dict(name="masks", mode="encode", vae=vae, save=False,
|
||||
refs_image={"ref_image_0": torch.zeros(1,64,64,3),
|
||||
"ref_image_1": torch.zeros(1,64,64,3)})
|
||||
for kwargs in ({"mask": torch.ones(1,64,64)},
|
||||
{"mask": torch.ones(2,64,64)},
|
||||
{"mask_list": [torch.ones(1,64,64)]}):
|
||||
self.assertEqual(len(N.MiniMaxH3RefModExtract.execute(**args, **kwargs)[0]), 1)
|
||||
with patch.object(vae, "encode", side_effect=AssertionError("must validate before encoding")):
|
||||
with self.assertRaisesRegex(ValueError, "3 entries"):
|
||||
N.MiniMaxH3RefModExtract.execute(**args, mask_list=[torch.ones(3,64,64)])
|
||||
with self.assertRaisesRegex(ValueError, "not both"):
|
||||
N.MiniMaxH3RefModExtract.execute(**args, mask=torch.ones(1,64,64),
|
||||
mask_list=[torch.ones(1,64,64)])
|
||||
|
||||
def test_encode_samples_causal_count_without_losing_clip_endpoint(self):
|
||||
clip = torch.arange(101).reshape(101,1,1,1).expand(-1,64,64,3).float() / 100
|
||||
seen = []
|
||||
def encode(pixels):
|
||||
seen.append(pixels[:,0,0,0].clone())
|
||||
n = pixels.shape[0]
|
||||
return torch.ones(1,24,1 if n == 1 else (n-5)//17*5+2,4,4)
|
||||
for limit, expected in ((16,5),(17,5),(22,22),(39,39),(1,1),(4,1),(101,90)):
|
||||
N.MiniMaxH3RefModExtract.execute(
|
||||
"motion", mode="encode", refs_video={"ref_video_0":clip},
|
||||
vae=types.SimpleNamespace(encode=encode), latent_frames=limit, save=False)
|
||||
self.assertEqual(len(seen[-1]), expected)
|
||||
self.assertEqual(seen[-1][0].item(), 0)
|
||||
if expected > 1:
|
||||
self.assertEqual(seen[-1][-1].item(), 1)
|
||||
|
||||
def test_valid_22_frame_reference_reaches_vae_without_trimming(self):
|
||||
clip = torch.arange(22).reshape(22,1,1,1).expand(-1,32,32,3).float() / 21
|
||||
seen = []
|
||||
vae = types.SimpleNamespace(encode=lambda pixels:
|
||||
seen.append(pixels.clone()) or torch.ones(1,24,7,4,4))
|
||||
# Isolate temporal policy from image-resize quantization.
|
||||
with patch.object(N, '_resize_ref', side_effect=lambda src,*a:src), \
|
||||
patch.object(N, '_ensure_min_size', side_effect=lambda src:src):
|
||||
for mode in ('encode', 'training'):
|
||||
N.MiniMaxH3RefModExtract.execute(
|
||||
'aligned', mode=mode, refs_video={'ref_video_0':clip},
|
||||
vae=vae, latent_frames=22, identity=0, save=False)
|
||||
self.assertTrue(torch.equal(seen[-1], clip))
|
||||
|
||||
def test_motion_only_realigns_differences_and_keeps_last_motion(self):
|
||||
clip = torch.linspace(0,1,23).square().reshape(23,1,1,1).expand(-1,32,32,3)
|
||||
seen = []
|
||||
vae = types.SimpleNamespace(encode=lambda pixels:
|
||||
seen.append(pixels.clone()) or torch.ones(1,24,7,4,4))
|
||||
with patch.object(N, '_resize_ref', side_effect=lambda src,*a:src), \
|
||||
patch.object(N, '_ensure_min_size', side_effect=lambda src:src):
|
||||
N.MiniMaxH3RefModExtract.execute(
|
||||
'motion', mode='training', refs_video={'ref_video_0':clip},
|
||||
vae=vae, latent_frames=2, identity=0, motion_only=True, save=False)
|
||||
expected = (clip[1:]-clip[:-1]).abs()
|
||||
expected /= expected.max()
|
||||
self.assertTrue(torch.equal(seen[-1], expected))
|
||||
|
||||
def test_cli_samples_the_same_frames_as_the_node_helper(self):
|
||||
sys.path.insert(0, str(ROOT))
|
||||
self.addCleanup(sys.path.remove, str(ROOT))
|
||||
cli = importlib.import_module('extract_mod')
|
||||
import comfy.sd
|
||||
import comfy.utils
|
||||
clip = torch.arange(101).reshape(101,1,1,1).expand(-1,32,32,3).float()/100
|
||||
seen = []
|
||||
vae = types.SimpleNamespace(throw_exception_if_invalid=lambda:None,
|
||||
encode=lambda pixels:seen.append(pixels.clone()) or torch.ones(1,24,7,4,4))
|
||||
with patch.object(cli, '_load_video', return_value=clip), \
|
||||
patch.object(cli, '_resize_ref', side_effect=lambda src,*a:src), \
|
||||
patch.object(cli, 'ensure_min_size', side_effect=lambda src:src), \
|
||||
patch.object(comfy.sd, 'VAE', return_value=vae), \
|
||||
patch.object(comfy.utils, 'load_torch_file', return_value=({},{})):
|
||||
for mode, limit in (('encode',16),('encode',22),('training',2)):
|
||||
argv = ['extract_mod.py','--video','test.mp4','--vae','stub.safetensors',
|
||||
'--device','cpu','--mode',mode,'--latent-frames',str(limit),
|
||||
'--identity','0','--output',str(self.root),'--name','cli']
|
||||
with patch.object(sys, 'argv', argv):
|
||||
cli.main()
|
||||
expected = COMMON.sample_video_for_vae(clip,limit if mode=='encode' else None)
|
||||
self.assertTrue(torch.equal(seen[-1], expected))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,47 @@
|
||||
"""Metadata-only library endpoint; no tensor allocation or VAE loading."""
|
||||
|
||||
from aiohttp import web
|
||||
from .bundle import members
|
||||
|
||||
|
||||
def library_entries(names, find_path, read_meta):
|
||||
entries = []
|
||||
for name in names():
|
||||
try:
|
||||
path = find_path(name)
|
||||
meta = read_meta(path)
|
||||
except (OSError, ValueError):
|
||||
continue
|
||||
if not isinstance(meta, dict):
|
||||
continue
|
||||
kind = meta.get("kind")
|
||||
if kind == "bundle":
|
||||
try:
|
||||
refs = members(meta)
|
||||
tokens = sum(int(ref["latent_t"]) * (2 if ref["kind"] == "audio" else
|
||||
(int(ref["latent_h"]) // 2) * (int(ref["latent_w"]) // 2)) for ref in refs)
|
||||
except (ValueError, KeyError, TypeError):
|
||||
continue
|
||||
entries.append({"name": name, "kind": "bundle", "path": path + ".safetensors",
|
||||
"concept": "bundle", "description": "; ".join(
|
||||
f"{ref.get('name', '')} ({ref['kind']})" for ref in refs),
|
||||
"tokens": tokens, "shape": None, "config": ""})
|
||||
continue
|
||||
try:
|
||||
t = int(meta.get("latent_t", 0))
|
||||
h, w = int(meta.get("latent_h", 0)), int(meta.get("latent_w", 0))
|
||||
except (TypeError, ValueError):
|
||||
continue
|
||||
tokens = t * 2 if kind == "audio" else t * (h // 2) * (w // 2)
|
||||
entries.append({"name": name, "kind": kind, "path": path + ".safetensors",
|
||||
"concept": meta.get("concept_type", "generic"),
|
||||
"description": meta.get("description", ""),
|
||||
"tokens": tokens or None, "shape": [t, h, w],
|
||||
"config": meta.get("refmod_config", "")})
|
||||
return entries
|
||||
|
||||
|
||||
def register_routes(server, names, find_path, read_meta):
|
||||
@server.routes.get("/refmods/library")
|
||||
async def library(request):
|
||||
return web.json_response(library_entries(names, find_path, read_meta))
|
||||
@@ -0,0 +1,102 @@
|
||||
"""Present saved RefMod references to H3's native text/vision encoder."""
|
||||
|
||||
import math
|
||||
import inspect
|
||||
|
||||
from comfy.text_encoders.minimax import MiniMaxH3Tokenizer
|
||||
from comfy.ldm.minimax.vae import MiniMaxH3VideoVAE
|
||||
|
||||
|
||||
def reference_map(mods):
|
||||
counters = {"image": 0, "video": 0, "audio": 0}
|
||||
labels = {"image": "Picture", "video": "Video", "audio": "Audio"}
|
||||
mapping = []
|
||||
for mod, strength in mods:
|
||||
if not math.isfinite(strength) or not 0 <= strength <= 1:
|
||||
raise ValueError("RefMod strength must be between 0 and 1.")
|
||||
if strength == 0:
|
||||
continue
|
||||
counters[mod.kind] += 1
|
||||
mapping.append(f"<{labels[mod.kind]} {counters[mod.kind]}> = {mod.name}")
|
||||
return "\n".join(mapping) or "No active RefMods."
|
||||
|
||||
|
||||
class MiniMaxH3RefModTextEncode:
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
return {"required": {
|
||||
"clip": ("CLIP",),
|
||||
"mods": ("H3_REF_MODS",),
|
||||
"prompt": ("STRING", {"multiline": True, "dynamicPrompts": True}),
|
||||
"reference_fps": ("FLOAT", {"default": 24.0, "min": 1.0, "max": 120.0,
|
||||
"tooltip": "Playback FPS of the reconstructed video, used for Qwen timestamps. Saved compressed or stacked refs do not preserve original timing."}),
|
||||
"max_total_tokens": ("INT", {"default": 0, "min": 0, "max": 2147483647}),
|
||||
}, "optional": {
|
||||
"vae": ("VAE", {"tooltip": "H3 video VAE: reconstructs visual refs for Qwen. Not needed for audio-only bundles."}),
|
||||
}}
|
||||
|
||||
RETURN_TYPES = ("CONDITIONING", "STRING")
|
||||
RETURN_NAMES = ("conditioning", "reference_map")
|
||||
FUNCTION = "encode"
|
||||
CATEGORY = "MiniMax-H3/mod"
|
||||
DESCRIPTION = "Encode the prompt together with numbered RefMod references. Already attaches refs; connect directly to the sampler, without applying the same mods again."
|
||||
|
||||
def encode(self, clip, mods, prompt, reference_fps=24.0, max_total_tokens=0, vae=None):
|
||||
from .nodes import _check_token_budget
|
||||
|
||||
native = isinstance(clip.tokenizer, MiniMaxH3Tokenizer)
|
||||
# Projected encoders retain their smaller model's tokenizer, but expose
|
||||
# H3 reference presentation on the CLIP wrapper itself (e.g. ClipProj).
|
||||
if not native and "minimax_ref_items" not in inspect.signature(clip.tokenize).parameters:
|
||||
raise ValueError("Connect an H3 CLIP or a projected CLIP supporting minimax_ref_items (such as current ClipProj).")
|
||||
if not math.isfinite(reference_fps) or not 1 <= reference_fps <= 120:
|
||||
raise ValueError("reference_fps must be between 1 and 120.")
|
||||
# Keep zero-strength slots out of both the presentation and DiT payload.
|
||||
mapping = reference_map(mods)
|
||||
active = [(mod, strength) for mod, strength in mods if strength > 0]
|
||||
_check_token_budget(active, max_total_tokens)
|
||||
if any(mod.kind != "audio" for mod, _ in active) and vae is None:
|
||||
raise ValueError("Connect the H3 video VAE to present visual RefMods to CLIP.")
|
||||
if any(mod.kind != "audio" for mod, _ in active) and not isinstance(vae.first_stage_model, MiniMaxH3VideoVAE):
|
||||
raise ValueError("Visual RefMods require the MiniMax H3 video VAE.")
|
||||
|
||||
items, blocks = [], []
|
||||
for mod, strength in active:
|
||||
block = mod.ref_block(strength)
|
||||
block["refmod"] = True
|
||||
kind = block["kind"]
|
||||
item = {"type": kind}
|
||||
if kind != "audio":
|
||||
# Decode the same weakened latent that the DiT receives. ComfyUI
|
||||
# owns device placement and its decode OOM/tiled fallback.
|
||||
pixels = vae.decode(block["latent"])
|
||||
# ComfyUI video VAEs return BTHWC; older wrappers may already
|
||||
# expose THWC. Each RefMod is one video, never a batch of videos.
|
||||
if pixels.ndim == 5 and pixels.shape[0] == 1:
|
||||
pixels = pixels[0]
|
||||
if pixels.ndim != 4 or pixels.shape[-1] != 3 or pixels.shape[0] < 1:
|
||||
raise ValueError(f"H3 video VAE must decode to [1, frames, height, width, 3] or [frames, height, width, 3]; got {tuple(pixels.shape)}.")
|
||||
if kind == "image":
|
||||
item["data"] = pixels[:1].cpu().clone()
|
||||
else:
|
||||
# Native H3 presents video at 2 fps. Index by timestamps so
|
||||
# non-integer frame rates do not accumulate rounding drift.
|
||||
times = [i / 2 for i in range(math.ceil(pixels.shape[0] * 2 / reference_fps))]
|
||||
indices = [min(round(t * reference_fps), pixels.shape[0] - 1) for t in times]
|
||||
item["data"] = pixels[indices].cpu()
|
||||
item["timestamps"] = times
|
||||
del pixels
|
||||
items.append(item)
|
||||
blocks.append(block)
|
||||
|
||||
tokens = clip.tokenize(prompt, minimax_ref_items=items)
|
||||
conditioning = clip.encode_from_tokens_scheduled(tokens)
|
||||
out = []
|
||||
for embedding, metadata in conditioning:
|
||||
if "minimax_token_tags" not in metadata:
|
||||
raise ValueError("The CLIP encoder did not return H3 minimax_token_tags. Use an H3-compatible encoder/projection.")
|
||||
metadata = dict(metadata)
|
||||
if blocks:
|
||||
metadata["minimax_refs"] = list(blocks)
|
||||
out.append([embedding, metadata])
|
||||
return out, mapping
|
||||
@@ -0,0 +1,17 @@
|
||||
[project]
|
||||
name = "ComfyUI-MiniMaxH3Mod"
|
||||
description = "Extract, save and combine reusable MiniMax H3 image, video and audio references as RefMod safetensors, with a unified Master extractor, library, inspector and strength curves."
|
||||
version = "0.2.7"
|
||||
license = { file = "LICENSE" }
|
||||
readme = "README.md"
|
||||
dependencies = ["torch", "safetensors", "numpy", "Pillow"]
|
||||
|
||||
[project.urls]
|
||||
Repository = "https://github.com/Luisacaotica/ComfyUI-MiniMaxH3Mod"
|
||||
Documentation = "https://github.com/Luisacaotica/ComfyUI-MiniMaxH3Mod#readme"
|
||||
"Bug Tracker" = "https://github.com/Luisacaotica/ComfyUI-MiniMaxH3Mod/issues"
|
||||
|
||||
[tool.comfy]
|
||||
# Fill with the exact publisher ID from your Comfy Registry profile before publishing.
|
||||
PublisherId = "luisacaotica"
|
||||
DisplayName = "MiniMax H3 RefMod"
|
||||
@@ -0,0 +1,56 @@
|
||||
# RefMod v0.2.0 — Master, audio references and loader fixes
|
||||
|
||||
## Highlights
|
||||
|
||||
- **Extract H3 RefMod Master:** image/video and optional audio in one node,
|
||||
sequential extraction with separate VAEs, a combined token budget and one
|
||||
bundle output. Visual and audio references save as separate compatible files.
|
||||
- **Integrated audio:** extract and load audio RefMods without installing
|
||||
H3AudioMod. Includes an input-music and generated-video example in the README.
|
||||
- **Save H3 RefMods:** a dedicated output node saves bundles without a connected
|
||||
Preview or sampler. Supports prefixes, subfolders and copy deduplication.
|
||||
- **Library and inspection:** search/filter the collection, select loader slots,
|
||||
inspect metadata/token counts and optionally preview stored visual/audio latents.
|
||||
|
||||
## Fixes and maintenance
|
||||
|
||||
- Linked mod names no longer fail queue validation before upstream nodes run;
|
||||
covers both the standard and Axis loaders.
|
||||
- Recursive subfolder discovery and saving to registered external RefMod roots
|
||||
respect ComfyUI's folder configuration.
|
||||
- File changes invalidate loader caches; loaded tensors have a bounded cache.
|
||||
- Step Curve no longer retains reference tensors between calls. Bridge injection
|
||||
is scoped to the MODEL branch and restores conditioning after errors.
|
||||
- Matching image/mask crops, bounded video sampling, runtime numeric/copy limits
|
||||
and optional total-token budgets.
|
||||
- Atomic file saves and explicit Master Created/Replaced destination logs.
|
||||
- Removed sibling-pack imports and the startup dependency warning. Legacy
|
||||
av_encoder inputs resolve through ComfyUI's native video VAE loader.
|
||||
- Shared extraction helpers and grouped multi-reference MSE refinement; explicit
|
||||
shuffle/subset options and experimental extraction presets.
|
||||
|
||||
## Known failure: voice transfer
|
||||
|
||||
**Audio support is not working voice cloning.** Music-reference conditioning has
|
||||
produced an observed result, but current speaker-identity transfer tests failed.
|
||||
The cause has not been isolated: H3/checkpoint behavior and the difference in how
|
||||
RefMod presents audio references to text encoding both remain under investigation.
|
||||
|
||||
The visual mode named `training` refines a compressed latent; it does not train
|
||||
H3 weights or learn a supervised control-to-target edit. The separate gradient
|
||||
probe is experimental and does not establish full-checkpoint training support.
|
||||
|
||||
## Updating
|
||||
|
||||
Validation: all 39 regression tests passed before release, using ComfyUI's
|
||||
embedded Python and real queue-validation modules. Previous audio codec checks
|
||||
used the real H3 audio VAE; these checks do not establish voice-transfer quality.
|
||||
|
||||
Update the node pack and restart ComfyUI to register the new nodes and library.
|
||||
Existing visual RefMods and legacy audio metadata remain readable. When using
|
||||
Save H3 RefMods after an extractor, set the extractor's save option to false to
|
||||
avoid writing twice. Master saves `name_visual` and `name_audio`; load both to
|
||||
reconstruct a mixed bundle. Existing files at those destinations are replaced.
|
||||
|
||||
The old Bridge Disarm node is retained for workflow compatibility; disable or
|
||||
bypass the bridge on its MODEL branch instead of relying on global state.
|
||||
@@ -0,0 +1,12 @@
|
||||
# v0.2.5 — RefMod creation and compatibility fixes
|
||||
|
||||
- Rename Extract nodes to Create, with unchanged node IDs. Full Reference and Compressed Reference replace the visible encode/training names; old mode values remain accepted. Rename the identity control to Refinement Steps.
|
||||
- Add visual budget_policy: truncate preserves existing behavior; error stops before saving when the visual token budget is exceeded. Master inherits the option.
|
||||
- Allow larger token/frame limits and preserve the requested frame count in the motion preset. Loader budgets default to 0 (disabled).
|
||||
- Add experimental H3 RefMod Text Encode with numbered reference presentation and a reference map. Support compatible ClipProj encoders and native batched video-VAE output.
|
||||
- Add optional full-video inspection with correct IMAGE output dimensions; retain the lightweight first-frame default.
|
||||
- Add playable music/video README examples and exclude local tests from distribution.
|
||||
|
||||
Validation: 51 local regression tests passed; creation-label hooks checked with Node.js. Full GPU generation, projected-encoder quality and voice transfer remain unverified. Voice cloning is not claimed to work.
|
||||
|
||||
Text Encode already attaches references: do not inject the same bundle again with Apply/Bridge. Integrated Continuum samplers still need a dedicated integration to use this conditioning. Full visual previews and text/vision presentation increase memory use.
|
||||
@@ -0,0 +1,15 @@
|
||||
# v0.2.6 — RefMod bundles and compact loaders
|
||||
|
||||
- Save visual and audio references together in one `.safetensors`: choose `save_layout=bundle` in Master, or pack existing references with **Save H3 RefMod Bundle**.
|
||||
- Select **All / Visual / Audio** per Loader/Axis slot, with independent modality strengths. Excluded components are not loaded.
|
||||
- Loader and Axis now start with one visible slot. **+ Add RefMod** reveals more slots, up to eight, with each reference's controls grouped together. Removal clears the slot; connected slots are protected.
|
||||
- The library recognizes combined files, and Config preserves unselected members. The bundle schema is documented in `BUNDLE_FORMAT.md` for third-party pickers.
|
||||
- Includes RefMod-only dropdown refresh and clearer effective-setting logs added after v0.2.5.
|
||||
|
||||
Existing standalone files and workflow values remain supported. Master still defaults to separate files. Combined version-5 files require a bundle-aware reader.
|
||||
|
||||
Restart ComfyUI for the Python changes and refresh the browser for the compact interface.
|
||||
|
||||
Bundling does not add voice-to-character binding, audiovisual synchronization, or original-waveform passthrough. Voice transfer remains experimental and failed in previous tests.
|
||||
|
||||
Validation: 61 local Python regression tests passed, plus frontend checks for adding/removing slots, connected inputs, legacy workflow restoration, serialization and library selection. The user confirmed the compact interface works in ComfyUI. Full GPU generation quality was not validated for this release.
|
||||
@@ -0,0 +1,11 @@
|
||||
# Hard dependencies (ComfyUI Manager installs these automatically).
|
||||
# torch is provided by ComfyUI itself, listed here for completeness.
|
||||
torch
|
||||
safetensors
|
||||
numpy
|
||||
Pillow
|
||||
|
||||
# Optional video loader backends (soft-imported in common.py). Install one if
|
||||
# the Load H3 RefMod Folder node can't read your video files.
|
||||
# opencv-python
|
||||
# imageio
|
||||
@@ -0,0 +1,23 @@
|
||||
{
|
||||
"properties": {"refmod_visible_slots": [1, 8]},
|
||||
"widgets_values": [
|
||||
false,
|
||||
"characters/alice", 0.73, 2,
|
||||
"(none)", 1, 1,
|
||||
"(none)", 1, 1,
|
||||
"(none)", 1, 1,
|
||||
"(none)", 1, 1,
|
||||
"(none)", 1, 1,
|
||||
"(none)", 1, 1,
|
||||
"motion/walk", 0.6, 1,
|
||||
8192,
|
||||
"Visual", 0.85, 0.25,
|
||||
"All", 1, 1,
|
||||
"All", 1, 1,
|
||||
"All", 1, 1,
|
||||
"All", 1, 1,
|
||||
"All", 1, 1,
|
||||
"All", 1, 1,
|
||||
"All", 1, 0.4
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,157 @@
|
||||
import assert from 'node:assert/strict';
|
||||
import {readFileSync} from 'node:fs';
|
||||
const source = readFileSync(new URL('../web/refmod_slots.js', import.meta.url), 'utf8')
|
||||
.replace('import { app } from "../../scripts/app.js";', 'const app = {registerExtension() {}};');
|
||||
const {installRefModSlots} = await import('data:text/javascript;base64,' + Buffer.from(source).toString('base64'));
|
||||
|
||||
function fixture(axis = false, stableView = false) {
|
||||
class Node {
|
||||
constructor() {
|
||||
if (stableView) {
|
||||
// Frontend v1.53.6 widgetsView.ts mutates a stable array on assignment.
|
||||
const view = [];
|
||||
Object.defineProperty(this, 'widgets', {
|
||||
get: () => view,
|
||||
set: widgets => view.splice(0, view.length, ...widgets),
|
||||
});
|
||||
}
|
||||
this.widgets = []; this.properties = {}; this.inputs = []; this.size = [360, 1400];
|
||||
this.addWidget('toggle', 'show_info', false);
|
||||
for (let i=1;i<=8;i++) {
|
||||
if (axis) {
|
||||
this.addWidget('combo',`mod_a_${i}`,'(none)');
|
||||
this.addWidget('combo',`mod_b_${i}`,'(none)');
|
||||
this.addWidget('number',`value_${i}`,0);
|
||||
} else {
|
||||
this.addWidget('combo',`mod_${i}`,'(none)');
|
||||
this.addWidget('number',`strength_${i}`,1);
|
||||
this.addWidget('number',`copies_${i}`,1);
|
||||
}
|
||||
}
|
||||
this.addWidget('number','max_total_tokens',0);
|
||||
for (let i=1;i<=8;i++) {
|
||||
this.addWidget('combo',`components_${i}`,'All');
|
||||
this.addWidget('number',`visual_strength_${i}`,1);
|
||||
this.addWidget('number',`audio_strength_${i}`,1);
|
||||
}
|
||||
this.onNodeCreated();
|
||||
}
|
||||
addWidget(type,name,value,callback,options={}) {
|
||||
const widget={type,name,value,callback,options}; this.widgets.push(widget); return widget;
|
||||
}
|
||||
computeSize() { return [360, 60 + this.widgets.reduce((h,w)=>h+(w.computeSize?.()[1] ?? 20)+4,0)]; }
|
||||
setSize(size) { this.size=size; }
|
||||
setDirtyCanvas() {}
|
||||
serialize() { return {properties:structuredClone(this.properties),widgets_values:this.widgets.map(w=>w.options.serialize===false?null:w.value)}; }
|
||||
// PR #20: modern frontend saves directly, without calling serialize().
|
||||
serializeFromStoreState() {
|
||||
return {properties:structuredClone(this.properties),widgets_values:this.widgets.map(w=>w.options.serialize===false?null:w.value)};
|
||||
}
|
||||
configure(info) {
|
||||
this.properties = structuredClone(info.properties ?? {});
|
||||
// LiteGraph invokes these even for unconnected input sockets,
|
||||
// before restoring widgets_values (frontend 1.39.2 and 1.52.7).
|
||||
this.inputs=info.inputs ?? [{name:'show_info',link:null}];
|
||||
for (const [i,input] of this.inputs.entries()) {
|
||||
this.onConnectionsChange?.(1,i,true,null,input);
|
||||
}
|
||||
info.widgets_values.forEach((value,i)=>{if(this.widgets[i])this.widgets[i].value=value;});
|
||||
}
|
||||
}
|
||||
installRefModSlots(Node);
|
||||
return Node;
|
||||
}
|
||||
const get=(node,name)=>node.widgets.find(w=>w.name===name);
|
||||
const click=(node,name)=>get(node,name).callback();
|
||||
for (const stableView of [false,true]) for (const axis of [false,true]) {
|
||||
const Node=fixture(axis, stableView), node=new Node();
|
||||
const mod=axis?'mod_a_':'mod_';
|
||||
const initial=node.size[1];
|
||||
assert.deepEqual(node.properties.refmod_visible_slots,[1]);
|
||||
assert.equal(get(node,mod+'8').type,'combo');
|
||||
assert.equal(get(node,mod+'8').options.hidden,true);
|
||||
// Nodes 2.0 renders converted-widget socket rows even when hidden. Real
|
||||
// widget types plus options.hidden eliminate those blank rows entirely.
|
||||
const rendered = () => node.widgets.filter(w => w.type === 'converted-widget' || !w.options.hidden).map(w => w.name);
|
||||
assert.ok(!rendered().includes(mod+'8'));
|
||||
assert.deepEqual(rendered().slice(0,6), axis
|
||||
? ['mod_a_1','mod_b_1','value_1','components_1','visual_strength_1','audio_strength_1']
|
||||
: ['mod_1','strength_1','copies_1','components_1','visual_strength_1','audio_strength_1']);
|
||||
click(node,'+ Add RefMod');
|
||||
assert.deepEqual(node.properties.refmod_visible_slots,[1,2]);
|
||||
assert.ok(node.size[1]>initial);
|
||||
get(node,mod+'2').value='folder/hero';
|
||||
get(node,'audio_strength_2').value=.25;
|
||||
const saved=node.serialize();
|
||||
assert.equal(saved.widgets_values[4],'folder/hero');
|
||||
const displayOrder = [...node.widgets];
|
||||
node.serialize = () => { throw new Error('store serializer must be independent'); };
|
||||
assert.deepEqual(node.serializeFromStoreState(), saved);
|
||||
assert.deepEqual(node.widgets, displayOrder);
|
||||
assert.equal(node._refmodSlots.schemaOrderDepth, 0);
|
||||
// Vue/store snapshots may restore visible rows as an indexed object.
|
||||
const fromStore = new Node();
|
||||
fromStore.configure({...saved, properties:{refmod_visible_slots:{0:1,1:2,2:4}}});
|
||||
assert.deepEqual(fromStore.properties.refmod_visible_slots,[1,2,4]);
|
||||
assert.equal(get(fromStore,mod+'2').value,'folder/hero');
|
||||
assert.equal(get(fromStore,'audio_strength_2').value,.25);
|
||||
assert.deepEqual(fromStore.serializeFromStoreState().widgets_values,saved.widgets_values);
|
||||
const restored=new Node(); restored.configure(saved);
|
||||
assert.equal(get(restored,mod+'2').value,'folder/hero');
|
||||
assert.equal(get(restored,'audio_strength_2').value,.25);
|
||||
assert.deepEqual(restored.serialize().widgets_values,saved.widgets_values);
|
||||
assert.equal(restored.widgets.indexOf(get(restored,'audio_strength_1')),5);
|
||||
// Repeated tab changes must keep all schema values and visible rows stable.
|
||||
let current = restored;
|
||||
for (let i=0;i<5;i++) {
|
||||
const snapshot = current.serialize();
|
||||
current = new Node();
|
||||
current.configure(snapshot);
|
||||
assert.deepEqual(current.serialize(), snapshot);
|
||||
}
|
||||
// A saved connected input also fires callbacks before value restoration.
|
||||
const connected = new Node();
|
||||
connected.configure({...saved,inputs:[{name:mod+'8',link:99}]});
|
||||
assert.equal(get(connected,mod+'2').value,'folder/hero');
|
||||
assert.equal(get(connected,'audio_strength_2').value,.25);
|
||||
assert.ok(connected.properties.refmod_visible_slots.includes(8));
|
||||
// Connected rows cannot be removed, even when their widget has no filename.
|
||||
restored.inputs=[{name:mod+'8',link:99}]; restored.onConnectionsChange();
|
||||
click(restored,'Remove RefMod 8');
|
||||
assert.ok(restored.properties.refmod_visible_slots.includes(8));
|
||||
assert.equal(restored.inputs[0].link,99);
|
||||
click(restored,'Remove RefMod 2');
|
||||
assert.equal(get(restored,mod+'2').value,'(none)');
|
||||
assert.equal(get(restored,'audio_strength_2').value,1);
|
||||
assert.ok(!restored.properties.refmod_visible_slots.includes(2));
|
||||
// Legacy positional workflow, including a high selected slot and no UI properties.
|
||||
const legacy=new Node().serialize(); delete legacy.properties;
|
||||
legacy.widgets_values=legacy.widgets_values.slice(0,26);
|
||||
legacy.widgets_values[22]='legacy/eighth';
|
||||
const migrated=new Node(); migrated.configure(legacy);
|
||||
assert.equal(get(migrated,mod+'8').value,'legacy/eighth');
|
||||
assert.ok(migrated.properties.refmod_visible_slots.includes(8));
|
||||
assert.equal(get(migrated,'audio_strength_8').value,1);
|
||||
// Library selection in a hidden slot reveals it via its existing callback.
|
||||
get(migrated,mod+'5').value='library/pick'; get(migrated,mod+'5').callback();
|
||||
assert.ok(migrated.properties.refmod_visible_slots.includes(5));
|
||||
while(migrated.properties.refmod_visible_slots.length<8) click(migrated,'+ Add RefMod');
|
||||
assert.equal(get(migrated,'+ Add RefMod').options.hidden,true);
|
||||
}
|
||||
// Frozen pre-library schema values: restoration must not depend on a snapshot
|
||||
// produced by the current implementation under test.
|
||||
const legacyFixture = JSON.parse(readFileSync(new URL('./fixtures/loader_widgets.json',import.meta.url),'utf8'));
|
||||
const FixtureNode = fixture();
|
||||
const fixtureNode = new FixtureNode();
|
||||
fixtureNode.configure(legacyFixture);
|
||||
assert.equal(get(fixtureNode,'mod_1').value,'characters/alice');
|
||||
assert.equal(get(fixtureNode,'strength_1').value,.73);
|
||||
assert.equal(get(fixtureNode,'copies_1').value,2);
|
||||
assert.equal(get(fixtureNode,'mod_8').value,'motion/walk');
|
||||
assert.equal(get(fixtureNode,'components_1').value,'Visual');
|
||||
assert.equal(get(fixtureNode,'visual_strength_1').value,.85);
|
||||
assert.equal(get(fixtureNode,'audio_strength_8').value,.4);
|
||||
assert.equal(get(fixtureNode,'max_total_tokens').value,8192);
|
||||
assert.deepEqual(fixtureNode.properties.refmod_visible_slots,[1,8]);
|
||||
assert.deepEqual(fixtureNode.serializeFromStoreState().widgets_values.slice(0,50),legacyFixture.widgets_values);
|
||||
console.log('PASS: Loader + Axis, legacy/store serialization, indexed visible slots, frozen fixture, linked slots and library selection');
|
||||
@@ -0,0 +1,84 @@
|
||||
"""Audio backend regressions without loading codec weights."""
|
||||
import builtins
|
||||
import contextlib
|
||||
import importlib
|
||||
import math
|
||||
import sys
|
||||
import types
|
||||
import unittest
|
||||
from unittest.mock import Mock, patch
|
||||
|
||||
import test_loaders as loaders
|
||||
|
||||
torch = loaders.torch
|
||||
AUDIO = importlib.import_module(loaders.N.__package__ + ".audio")
|
||||
|
||||
|
||||
class AudioResampleTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.pieces = []
|
||||
pieces = self.pieces
|
||||
|
||||
class Codec:
|
||||
def encode(self, waveform):
|
||||
pieces.append(waveform.clone())
|
||||
return torch.zeros(1, 32, 2, math.ceil(waveform.shape[-1] / 800))
|
||||
|
||||
self.vae = types.SimpleNamespace(first_stage_model=Codec(), patcher=object(),
|
||||
device="cpu", vae_dtype=torch.float32)
|
||||
for mocked in (patch.object(AUDIO, "MiniMaxH3AudioVAE", Codec),
|
||||
patch.object(AUDIO.model_management, "load_models_gpu"),
|
||||
patch.object(AUDIO.model_management, "cuda_device_context",
|
||||
side_effect=lambda device: contextlib.nullcontext())):
|
||||
mocked.start()
|
||||
self.addCleanup(mocked.stop)
|
||||
|
||||
def backend(self):
|
||||
def resample(waveform, orig_freq, new_freq):
|
||||
length = math.ceil(waveform.shape[-1] * new_freq / orig_freq)
|
||||
return torch.ones(*waveform.shape[:-1], length)
|
||||
return types.SimpleNamespace(resample=Mock(side_effect=resample))
|
||||
|
||||
def test_native_backend_without_torchaudio_clips_before_resampling(self):
|
||||
for rate, channels in ((44100, 1), (48000, 2)):
|
||||
with self.subTest(rate=rate, channels=channels):
|
||||
native = self.backend()
|
||||
self.pieces.clear()
|
||||
with patch.dict(sys.modules, {"comfy.audio": native, "torchaudio": None,
|
||||
"torchaudio.functional": None}):
|
||||
z = AUDIO.encode_audio(self.vae, {
|
||||
"waveform": torch.zeros(1,channels,rate*2), "sample_rate": rate},
|
||||
max_seconds=1.25, chunk_seconds=1)
|
||||
wave, orig, new = native.resample.call_args.args
|
||||
self.assertEqual(tuple(wave.shape), (1,2,round(rate*1.25)))
|
||||
self.assertEqual((orig,new), (rate,32000))
|
||||
self.assertEqual([p.shape[-1] for p in self.pieces], [32000,8000])
|
||||
self.assertEqual(tuple(z.shape), (1,32,2,50))
|
||||
|
||||
def test_older_comfyui_uses_torchaudio(self):
|
||||
legacy = self.backend()
|
||||
with patch.dict(sys.modules, {"comfy.audio": None, "torchaudio.functional": legacy}):
|
||||
AUDIO.encode_audio(self.vae, {"waveform": torch.zeros(1,2,4800), "sample_rate": 48000})
|
||||
legacy.resample.assert_called_once()
|
||||
self.assertEqual(self.pieces[0].shape[-1], 3200)
|
||||
|
||||
def test_32000_needs_neither_resampler(self):
|
||||
with patch.dict(sys.modules, {"comfy.audio": None, "torchaudio": None,
|
||||
"torchaudio.functional": None}):
|
||||
AUDIO.encode_audio(self.vae, {"waveform": torch.ones(1,1,3200), "sample_rate": 32000})
|
||||
torch.testing.assert_close(self.pieces[0], torch.ones(1,2,3200))
|
||||
|
||||
def test_native_dependency_error_is_not_hidden_by_fallback(self):
|
||||
original_import = builtins.__import__
|
||||
def failing_import(name, *args, **kwargs):
|
||||
if name == "comfy.audio":
|
||||
raise ModuleNotFoundError("Missing native dependency", name="scipy")
|
||||
return original_import(name, *args, **kwargs)
|
||||
with patch.object(builtins, "__import__", side_effect=failing_import):
|
||||
with self.assertRaises(ModuleNotFoundError) as error:
|
||||
AUDIO.encode_audio(self.vae, {"waveform": torch.zeros(1,2,4800), "sample_rate": 48000})
|
||||
self.assertEqual(error.exception.name, "scipy")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,229 @@
|
||||
"""Loader regressions using real safetensors and ComfyUI's queue validator.
|
||||
|
||||
Run with ComfyUI's Python: python tests/test_loaders.py
|
||||
No models or server are started; all mod files live in temporary folders.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import contextlib
|
||||
import importlib
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
sys.path.insert(0, str(ROOT.parents[1]))
|
||||
|
||||
from comfy.cli_args import args
|
||||
|
||||
args.cpu = True
|
||||
|
||||
import folder_paths
|
||||
import torch
|
||||
from safetensors.torch import save_file
|
||||
|
||||
with tempfile.TemporaryDirectory() as import_dir:
|
||||
with patch.object(folder_paths, "models_dir", import_dir):
|
||||
N = importlib.import_module(f"custom_nodes.{ROOT.name}.nodes")
|
||||
|
||||
import execution
|
||||
import nodes as comfy_nodes
|
||||
|
||||
|
||||
class UpstreamString:
|
||||
RETURN_TYPES = ("STRING",)
|
||||
FUNCTION = "run"
|
||||
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
return {"required": {"text": ("STRING",)}}
|
||||
|
||||
def run(self, text):
|
||||
raise AssertionError("Queue validation must not execute upstream nodes")
|
||||
|
||||
|
||||
class LoaderTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.temp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(self.temp.cleanup)
|
||||
self.root = Path(self.temp.name) / "refmods"
|
||||
self.legacy = Path(self.temp.name) / "mods"
|
||||
self.root.mkdir()
|
||||
self.legacy.mkdir()
|
||||
search = patch.object(N, "_mod_search_dirs", return_value=[str(self.root), str(self.legacy)])
|
||||
search.start()
|
||||
self.addCleanup(search.stop)
|
||||
N._MOD_CACHE.clear()
|
||||
N._MOD_LIST_CACHE_KEY = N._MOD_LIST_CACHE_VAL = None
|
||||
output = contextlib.redirect_stdout(io.StringIO())
|
||||
output.__enter__()
|
||||
self.addCleanup(output.__exit__, None, None, None)
|
||||
|
||||
def save_mod(self, name, kind="image", root=None, value=1.0, metadata=None):
|
||||
path = (root or self.root) / name
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
mod = N.H3RefMod(name=path.name, kind=kind,
|
||||
latent=torch.full((1, 24, 1, 4, 4), value),
|
||||
latent_h=4, latent_w=4)
|
||||
if metadata is None:
|
||||
mod.save(str(path))
|
||||
else:
|
||||
save_file({"latent": mod.latent}, str(path) + ".safetensors",
|
||||
metadata={"refmod_meta": json.dumps(metadata)})
|
||||
return path
|
||||
|
||||
def test_empty_names_validate_and_execute_in_every_slot(self):
|
||||
for cls in (N.MiniMaxH3RefModsLoader, N.MiniMaxH3RefModsAxis):
|
||||
for field in cls.INPUT_TYPES()["required"]:
|
||||
if not field.startswith("mod_"):
|
||||
continue
|
||||
for empty in (None, "", "None", "(none)"):
|
||||
with self.subTest(node=cls.__name__, field=field, empty=empty):
|
||||
inputs = {field: empty}
|
||||
if "_a_" in field or "_b_" in field:
|
||||
inputs["value_" + field.rsplit("_", 1)[1]] = -1 if "_a_" in field else 1
|
||||
self.assertIs(cls.VALIDATE_INPUTS(**inputs), True)
|
||||
self.assertEqual(cls().load(**inputs), ([], ""))
|
||||
|
||||
def test_recursive_listing_filters_metadata_and_presets(self):
|
||||
self.save_mod("root")
|
||||
self.save_mod("celebs/person", kind="video")
|
||||
self.save_mod("oc/deep/person", root=self.legacy)
|
||||
self.save_mod("celebs/audio", metadata={"kind": "other"})
|
||||
self.save_mod("celebs/bad_json_type", metadata=["image"])
|
||||
self.save_mod("graph_presets/hidden")
|
||||
self.save_mod("oc/graph_presets/hidden")
|
||||
(self.root / "curve.png").touch()
|
||||
(self.root / "junk.safetensors").write_bytes(b"invalid")
|
||||
expected = ["celebs/person", "oc/deep/person", "root"]
|
||||
self.assertEqual(N._list_mod_names(), expected)
|
||||
for cls in (N.MiniMaxH3RefModsLoader, N.MiniMaxH3RefModsAxis):
|
||||
for field, schema in cls.INPUT_TYPES()["required"].items():
|
||||
if field.startswith("mod_"):
|
||||
self.assertEqual(schema[0], ["(none)"] + expected)
|
||||
|
||||
def test_listing_cache_reuses_metadata_reads(self):
|
||||
self.save_mod("deep/a")
|
||||
with patch.object(N, "read_refmod_meta", wraps=N.read_refmod_meta) as read:
|
||||
self.assertEqual(N._list_mod_names(), ["deep/a"])
|
||||
self.assertEqual(N._list_mod_names(), ["deep/a"])
|
||||
self.assertEqual(read.call_count, 1)
|
||||
|
||||
def test_nested_add_rename_delete_refresh_listing(self):
|
||||
self.assertEqual(N._list_mod_names(), [])
|
||||
path = self.save_mod("new/deep/a")
|
||||
self.assertEqual(N._list_mod_names(), ["new/deep/a"])
|
||||
file = Path(str(path) + ".safetensors")
|
||||
renamed = file.with_name("b.safetensors")
|
||||
file.rename(renamed)
|
||||
self.assertEqual(N._list_mod_names(), ["new/deep/b"])
|
||||
renamed.unlink()
|
||||
self.assertEqual(N._list_mod_names(), [])
|
||||
|
||||
def test_subsecond_metadata_change_refreshes_listing(self):
|
||||
path = self.save_mod("deep/a", metadata={"kind": "image"})
|
||||
file = Path(str(path) + ".safetensors")
|
||||
stamp = 1_700_000_000_100_000_000
|
||||
os.utime(file, ns=(stamp, stamp))
|
||||
self.assertEqual(N._list_mod_names(), ["deep/a"])
|
||||
self.save_mod("deep/a", metadata={"kind": "other"})
|
||||
os.utime(file, ns=(stamp + 1_000_000, stamp + 1_000_000))
|
||||
self.assertEqual(N._list_mod_names(), [])
|
||||
|
||||
def test_legacy_sidecar_add_change_delete_refresh_listing(self):
|
||||
path = self.root / "deep" / "legacy"
|
||||
path.parent.mkdir()
|
||||
save_file({"latent": torch.zeros(1, 24, 1, 4, 4)}, str(path) + ".safetensors")
|
||||
self.assertEqual(N._list_mod_names(), [])
|
||||
sidecar = Path(str(path) + ".json")
|
||||
sidecar.write_text('{"kind": "image"}')
|
||||
self.assertEqual(N._list_mod_names(), ["deep/legacy"])
|
||||
self.assertEqual(N._load_mod("deep/legacy").kind, "image")
|
||||
sidecar.write_text('{"kind": "unsupported"}')
|
||||
self.assertEqual(N._list_mod_names(), [])
|
||||
sidecar.unlink()
|
||||
self.assertEqual(N._list_mod_names(), [])
|
||||
|
||||
def test_search_root_identity_and_precedence(self):
|
||||
self.save_mod("a", value=1)
|
||||
self.save_mod("a", root=self.legacy, value=2)
|
||||
self.assertEqual(N._list_mod_names(), ["a"])
|
||||
self.assertEqual(N._load_mod("a").latent.flatten()[0].item(), 1)
|
||||
with patch.object(N, "_mod_search_dirs", return_value=[str(self.legacy)]):
|
||||
self.assertEqual(N._load_mod("a").latent.flatten()[0].item(), 2)
|
||||
self.save_mod("a", metadata={"kind": "other"})
|
||||
self.assertEqual(N._list_mod_names(), [])
|
||||
|
||||
def test_nested_runtime_both_loaders_and_windows_separators(self):
|
||||
self.save_mod("celebs/person", value=3)
|
||||
self.assertIs(N.MiniMaxH3RefModsLoader.VALIDATE_INPUTS(mod_1="celebs\\person"), True)
|
||||
bundle, _ = N.MiniMaxH3RefModsLoader().load(mod_1="celebs\\person", strength_1=0.6, copies_1=2)
|
||||
self.assertEqual(len(bundle), 2)
|
||||
self.assertEqual(bundle[0][1], 0.6)
|
||||
self.assertEqual(bundle[0][0].latent.flatten()[0].item(), 3)
|
||||
for side, value in (("a", -0.4), ("b", 0.4)):
|
||||
bundle, _ = N.MiniMaxH3RefModsAxis().load(**{f"mod_{side}_1": "celebs/person", "value_1": value})
|
||||
self.assertEqual(bundle[0][1], 0.4)
|
||||
self.assertIs(bundle[0][0], N._load_mod("celebs\\person"))
|
||||
|
||||
def test_same_metadata_name_config_does_not_poison_root_cache(self):
|
||||
self.save_mod("person", value=1)
|
||||
nested_path = self.save_mod("celebs/person", value=2)
|
||||
root_mod = N._load_mod("person")
|
||||
nested_mod = N._load_mod("celebs/person")
|
||||
N.MiniMaxH3RefModConfig().fix([(nested_mod, 1.0)], retention=0.4)
|
||||
self.assertIs(N._load_mod("person"), root_mod)
|
||||
self.assertEqual(root_mod.config, {})
|
||||
self.assertEqual(N.H3RefMod.load(str(nested_path)).config["retention"], 0.4)
|
||||
|
||||
def test_fixed_missing_names_rejected_and_linked_missing_fail_at_load(self):
|
||||
for cls, field in ((N.MiniMaxH3RefModsLoader, "mod_1"),
|
||||
(N.MiniMaxH3RefModsAxis, "mod_a_1"),
|
||||
(N.MiniMaxH3RefModsAxis, "mod_b_1")):
|
||||
self.assertIn("not found", cls.VALIDATE_INPUTS(**{field: "missing"}))
|
||||
self.assertIs(cls.VALIDATE_INPUTS(input_types={field: "STRING"}, **{field: None}), True)
|
||||
with self.assertRaisesRegex(FileNotFoundError, "missing"):
|
||||
N.MiniMaxH3RefModsLoader().load(mod_1="missing")
|
||||
|
||||
def test_path_boundary_is_checked_at_execution(self):
|
||||
for name in ("../outside", "a/../../outside", "/absolute", "C:/absolute", "C:relative",
|
||||
"\\\\server\\share\\mod", "graph_presets/a"):
|
||||
with self.subTest(name=name), self.assertRaises(ValueError):
|
||||
N._load_mod(name)
|
||||
|
||||
def test_actual_comfy_queue_with_unexecuted_upstream(self):
|
||||
for cls, field in ((N.MiniMaxH3RefModsLoader, "mod_1"),
|
||||
(N.MiniMaxH3RefModsAxis, "mod_a_1"),
|
||||
(N.MiniMaxH3RefModsAxis, "mod_b_1")):
|
||||
inputs = {}
|
||||
for name, (kind, options) in cls.INPUT_TYPES()["required"].items():
|
||||
inputs[name] = kind[0] if isinstance(kind, list) else options["default"]
|
||||
inputs[field] = ["source", 0]
|
||||
prompt = {"source": {"class_type": "TestRefModString", "inputs": {"text": "future/person"}},
|
||||
"loader": {"class_type": "TestRefModLoader", "inputs": inputs}}
|
||||
with patch.dict(comfy_nodes.NODE_CLASS_MAPPINGS, TestRefModString=UpstreamString, TestRefModLoader=cls):
|
||||
for output_type in ("STRING", "COMBO", ["future/person"]):
|
||||
with self.subTest(node=cls.__name__, field=field, output_type=output_type):
|
||||
with patch.object(UpstreamString, "RETURN_TYPES", (output_type,)):
|
||||
result = asyncio.run(execution.validate_inputs("test-refmods", prompt, "loader", {}))
|
||||
self.assertTrue(result[0], result)
|
||||
with patch.object(UpstreamString, "RETURN_TYPES", ("IMAGE",)):
|
||||
result = asyncio.run(execution.validate_inputs("test-refmods", prompt, "loader", {}))
|
||||
self.assertFalse(result[0], result)
|
||||
|
||||
def test_connected_non_mod_inputs_keep_type_checks(self):
|
||||
for cls, field in ((N.MiniMaxH3RefModsLoader, "strength_1"),
|
||||
(N.MiniMaxH3RefModsLoader, "copies_1"),
|
||||
(N.MiniMaxH3RefModsLoader, "show_info"),
|
||||
(N.MiniMaxH3RefModsAxis, "value_1")):
|
||||
with self.subTest(field=field):
|
||||
self.assertIn("expected", cls.VALIDATE_INPUTS(input_types={field: "IMAGE"}))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main(verbosity=2)
|
||||
@@ -0,0 +1,128 @@
|
||||
export function openRefModLibrary(node, fetchEntries) {
|
||||
const previousFocus = document.activeElement;
|
||||
const dialog = document.createElement("dialog");
|
||||
dialog.className = "refmod-library";
|
||||
dialog.setAttribute("aria-label", "RefMod library");
|
||||
const style = document.createElement("style");
|
||||
style.textContent = `
|
||||
.refmod-library {box-sizing:border-box;width:min(1100px,96vw);height:90vh;max-height:90vh;padding:24px;border:1px solid #455266;border-radius:14px;background:#111820;color:#e8edf5;font:14px system-ui;}
|
||||
.refmod-library::backdrop {background:#000b}
|
||||
.refmod-library .layout {height:100%;display:flex;flex-direction:column;gap:16px;min-height:0}
|
||||
.refmod-library header {display:flex;gap:12px;align-items:center;justify-content:space-between}
|
||||
.refmod-library h2 {margin:0;font-size:23px}
|
||||
.refmod-library .controls {display:flex;gap:10px;flex-wrap:wrap}
|
||||
.refmod-library input,.refmod-library select,.refmod-library button {font:inherit;color:inherit;background:#222f3e;border:1px solid #455266;border-radius:7px;padding:9px 12px}
|
||||
.refmod-library input {flex:1;min-width:180px}
|
||||
.refmod-library button {cursor:pointer}
|
||||
.refmod-library button:focus-visible,.refmod-library input:focus-visible,.refmod-library select:focus-visible {outline:2px solid #ffb257;outline-offset:2px}
|
||||
.refmod-library .results {overflow:auto;min-height:0;flex:1;display:flex;flex-direction:column;gap:10px}
|
||||
.refmod-library article {padding:16px;border:1px solid #334254;border-radius:9px;background:#18222d;display:grid;grid-template-columns:1fr auto;gap:8px 18px}
|
||||
.refmod-library .description,.refmod-library .path {grid-column:1/-1;color:#afbed0;overflow-wrap:anywhere}
|
||||
.refmod-library .path {font-size:12px}
|
||||
.refmod-library .use {background:#ffb257;border-color:#ffb257;color:#171c22;font-weight:650}
|
||||
.refmod-library .status {color:#afbed0;min-height:20px}
|
||||
@media(max-width:600px){.refmod-library{padding:14px}.refmod-library article{grid-template-columns:1fr}.refmod-library header{flex-wrap:wrap}}
|
||||
`;
|
||||
const make = (tag, text, className) => {
|
||||
const el = document.createElement(tag);
|
||||
if (text !== undefined) el.textContent = text;
|
||||
if (className) el.className = className;
|
||||
return el;
|
||||
};
|
||||
const layout = make("div", undefined, "layout");
|
||||
const header = make("header");
|
||||
const close = make("button", "Close");
|
||||
header.append(make("h2", "RefMod library"), close);
|
||||
const controls = make("div", undefined, "controls");
|
||||
const search = make("input");
|
||||
search.placeholder = "Search name, folder or description";
|
||||
search.setAttribute("aria-label", "Search RefMods");
|
||||
const kind = make("select");
|
||||
kind.setAttribute("aria-label", "Reference type");
|
||||
for (const value of ["all", "image", "video", "audio", "bundle"]) {
|
||||
const option = make("option", value === "all" ? "All reference types" : value);
|
||||
option.value = value;
|
||||
kind.append(option);
|
||||
}
|
||||
const slot = make("select");
|
||||
slot.setAttribute("aria-label", "Loader slot");
|
||||
const widgets = (node.widgets || []).filter(w => /^mod_(?:[ab]_)?\d+$/.test(w.name));
|
||||
for (const widget of widgets) {
|
||||
const option = make("option", widget.name);
|
||||
option.value = widget.name;
|
||||
slot.append(option);
|
||||
}
|
||||
const refresh = make("button", "Refresh");
|
||||
controls.append(search, kind, slot, refresh);
|
||||
const status = make("div", "Loading library…", "status");
|
||||
status.setAttribute("role", "status");
|
||||
const results = make("div", undefined, "results");
|
||||
layout.append(header, controls, status, results);
|
||||
dialog.append(style, layout);
|
||||
document.body.append(dialog);
|
||||
let entries = [], disposed = false, controller;
|
||||
const cleanup = () => {
|
||||
if (disposed) return;
|
||||
disposed = true;
|
||||
controller?.abort();
|
||||
dialog.remove();
|
||||
previousFocus?.focus();
|
||||
};
|
||||
close.onclick = () => dialog.close();
|
||||
dialog.addEventListener("close", cleanup, {once:true});
|
||||
const render = () => {
|
||||
const query = search.value.trim().toLocaleLowerCase();
|
||||
const filtered = entries.filter(e => (kind.value === "all" || e.kind === kind.value)
|
||||
&& [e.name,e.concept,e.description].join(" ").toLocaleLowerCase().includes(query));
|
||||
status.textContent = `${filtered.length} of ${entries.length} RefMods · choose a loader slot, then Use`;
|
||||
results.replaceChildren();
|
||||
for (const entry of filtered) {
|
||||
const row = make("article");
|
||||
const title = make("strong", entry.name);
|
||||
const use = make("button", "Use", "use");
|
||||
use.disabled = widgets.length === 0;
|
||||
use.onclick = () => {
|
||||
const widget = widgets.find(w => w.name === slot.value);
|
||||
const linked = node.inputs?.find(i => i.name === slot.value)?.link;
|
||||
if (linked != null) {
|
||||
status.textContent = `${slot.value} is connected. Select an unconnected slot.`;
|
||||
return;
|
||||
}
|
||||
if (!widget) return;
|
||||
widget.value = entry.name;
|
||||
widget.callback?.(entry.name);
|
||||
node.setDirtyCanvas?.(true, true);
|
||||
dialog.close();
|
||||
};
|
||||
const tokens = entry.tokens == null ? "tokens available after loading" : `${entry.tokens.toLocaleString()} tokens / copy`;
|
||||
row.append(title, use, make("div", `${entry.kind} · ${entry.concept} · ${tokens}`, "description"),
|
||||
make("div", entry.description || "No description saved", "description"),
|
||||
make("div", entry.path, "path"));
|
||||
results.append(row);
|
||||
}
|
||||
if (!filtered.length) results.append(make("p", "No matching RefMods. Try another search or Refresh after adding files."));
|
||||
};
|
||||
const load = async () => {
|
||||
controller?.abort();
|
||||
controller = new AbortController();
|
||||
const signal = controller.signal;
|
||||
refresh.disabled = true;
|
||||
try {
|
||||
const data = await fetchEntries(signal);
|
||||
if (disposed || signal.aborted) return;
|
||||
entries = data;
|
||||
render();
|
||||
} catch (error) {
|
||||
if (!disposed && !signal.aborted) status.textContent = `Could not load library: ${error.message}`;
|
||||
} finally {
|
||||
if (!disposed && !signal.aborted) refresh.disabled = false;
|
||||
}
|
||||
};
|
||||
search.oninput = render;
|
||||
kind.onchange = render;
|
||||
refresh.onclick = load;
|
||||
dialog.showModal();
|
||||
search.focus();
|
||||
load();
|
||||
return dialog;
|
||||
}
|
||||
@@ -0,0 +1,34 @@
|
||||
import { app } from "../../scripts/app.js";
|
||||
|
||||
export function installRefModApply(nodeType) {
|
||||
const created = nodeType.prototype.onNodeCreated;
|
||||
nodeType.prototype.onNodeCreated = function(...args) {
|
||||
const result = created?.apply(this, args);
|
||||
// MatchType installs its callback on the instance during construction.
|
||||
const connections = this.onConnectionsChange;
|
||||
this.onConnectionsChange = function(...args) {
|
||||
const index = this.inputs?.findIndex(input => input.name === "conditioning") ?? -1;
|
||||
if (index >= 0) {
|
||||
const input = this.inputs[index];
|
||||
try {
|
||||
// Frontend 1.39.2 proxies MatchType slots, but NodeSlot.node
|
||||
// reads a #private field. That throws during canvas drawing.
|
||||
void input.node;
|
||||
} catch (error) {
|
||||
if (!(error instanceof TypeError)) throw error;
|
||||
this.inputs[index] = new input.constructor({...input}, this);
|
||||
if (args[4] === input) args[4] = this.inputs[index];
|
||||
}
|
||||
}
|
||||
return connections?.apply(this, args);
|
||||
};
|
||||
return result;
|
||||
};
|
||||
}
|
||||
|
||||
app.registerExtension({
|
||||
name: "MiniMaxH3.ApplyCanvasCompatibility",
|
||||
beforeRegisterNodeDef(nodeType, nodeData) {
|
||||
if (nodeData.name === "MiniMaxH3RefModApply") installRefModApply(nodeType);
|
||||
},
|
||||
});
|
||||
@@ -0,0 +1,28 @@
|
||||
import { app } from "../../scripts/app.js";
|
||||
|
||||
const modes = { training: "Compressed Reference", encode: "Full Reference" };
|
||||
|
||||
function updateLabels(node) {
|
||||
const mode = node.widgets?.find(widget => widget.name === "mode");
|
||||
if (mode) {
|
||||
mode.options.values = Object.values(modes);
|
||||
mode.value = modes[mode.value] ?? mode.value;
|
||||
}
|
||||
const steps = node.widgets?.find(widget => widget.name === "identity");
|
||||
if (steps) steps.label = "Refinement Steps";
|
||||
}
|
||||
|
||||
app.registerExtension({
|
||||
name: "MiniMaxH3.CreationLabels",
|
||||
beforeRegisterNodeDef(nodeType, nodeData) {
|
||||
if (!["MiniMaxH3RefModExtract", "MiniMaxH3RefModMasterExtract"].includes(nodeData.name)) return;
|
||||
for (const hook of ["onNodeCreated", "onConfigure"]) {
|
||||
const previous = nodeType.prototype[hook];
|
||||
nodeType.prototype[hook] = function(...args) {
|
||||
const result = previous?.apply(this, args);
|
||||
updateLabels(this);
|
||||
return result;
|
||||
};
|
||||
}
|
||||
}
|
||||
});
|
||||
@@ -0,0 +1,67 @@
|
||||
import { app } from "../../scripts/app.js";
|
||||
import { api } from "../../scripts/api.js";
|
||||
import { openRefModLibrary } from "./library_dialog.js";
|
||||
|
||||
export function updateRefModChoices(node, entries) {
|
||||
const names = ["(none)", ...new Set(entries.map(entry => entry.name))];
|
||||
for (const widget of node.widgets || []) {
|
||||
if (!/^mod_(?:[ab]_)?\d+$/.test(widget.name)) continue;
|
||||
// Preserve selections, including missing files, for an explicit load
|
||||
// error instead of silently switching the subject after a refresh.
|
||||
widget.options.values = names.includes(widget.value) || !widget.value
|
||||
? [...names] : [...names, widget.value];
|
||||
}
|
||||
node.setDirtyCanvas?.(true, true);
|
||||
}
|
||||
|
||||
app.registerExtension({
|
||||
name: "MiniMaxH3.RefModLibrary",
|
||||
beforeRegisterNodeDef(nodeType, nodeData) {
|
||||
if (!["MiniMaxH3RefModsLoader", "MiniMaxH3RefModsAxis"].includes(nodeData.name)) return;
|
||||
const removed = nodeType.prototype.onRemoved;
|
||||
nodeType.prototype.onRemoved = function(...args) {
|
||||
this._refmodRefresh?.abort();
|
||||
this._refmodDialog?.close();
|
||||
return removed?.apply(this, args);
|
||||
};
|
||||
const previous = nodeType.prototype.onNodeCreated;
|
||||
nodeType.prototype.onNodeCreated = function(...args) {
|
||||
const result = previous?.apply(this, args);
|
||||
const refresh = this.addWidget("button", "Refresh RefMods", null, async () => {
|
||||
this._refmodRefresh?.abort();
|
||||
const controller = new AbortController();
|
||||
this._refmodRefresh = controller;
|
||||
refresh.name = "Refreshing RefMods…";
|
||||
this.setDirtyCanvas?.(true, true);
|
||||
try {
|
||||
const response = await api.fetchApi("/refmods/library", {signal: controller.signal, cache: "no-store"});
|
||||
if (!response.ok) throw new Error(`HTTP ${response.status}`);
|
||||
const entries = await response.json();
|
||||
if (controller.signal.aborted) return;
|
||||
updateRefModChoices(this, entries);
|
||||
refresh.name = "Refresh RefMods";
|
||||
} catch (error) {
|
||||
if (controller.signal.aborted) return;
|
||||
refresh.name = "Refresh failed — click to retry";
|
||||
console.error("RefMod refresh failed:", error);
|
||||
} finally {
|
||||
if (!controller.signal.aborted) {
|
||||
this._refmodRefresh = null;
|
||||
this.setDirtyCanvas?.(true, true);
|
||||
}
|
||||
}
|
||||
}, {serialize:false});
|
||||
this.addWidget("button", "RefMod library", null, () => {
|
||||
this._refmodDialog?.close();
|
||||
this._refmodDialog = openRefModLibrary(this, async signal => {
|
||||
const response = await api.fetchApi("/refmods/library", {signal});
|
||||
if (!response.ok) throw new Error(`HTTP ${response.status}`);
|
||||
const entries = await response.json();
|
||||
if (!signal.aborted) updateRefModChoices(this, entries);
|
||||
return entries;
|
||||
});
|
||||
}, {serialize:false});
|
||||
return result;
|
||||
};
|
||||
}
|
||||
});
|
||||
@@ -0,0 +1,158 @@
|
||||
import { app } from "../../scripts/app.js";
|
||||
|
||||
const SLOT = /^(?:mod_(?:[ab]_)?|strength_|copies_|value_|components_|visual_strength_|audio_strength_)(\d+)$/;
|
||||
const NONE = new Set(["", "None", "(none)"]);
|
||||
|
||||
export function installRefModSlots(nodeType) {
|
||||
const created = nodeType.prototype.onNodeCreated;
|
||||
nodeType.prototype.onNodeCreated = function(...args) {
|
||||
const result = created?.apply(this, args);
|
||||
setupSlots(this);
|
||||
return result;
|
||||
};
|
||||
// LiteGraph stores widget values by position. Keep the original schema
|
||||
// order for saving/loading even though the canvas groups them by slot.
|
||||
// ComfyUI frontend >= 1.5x saves via serializeFromStoreState() and never
|
||||
// calls serialize(), so that path has to be covered as well or workflows
|
||||
// get written in display order and shift on the next load.
|
||||
for (const method of ["serialize", "serializeFromStoreState", "configure"]) {
|
||||
const original = nodeType.prototype[method];
|
||||
nodeType.prototype[method] = function(...args) {
|
||||
const state = this._refmodSlots;
|
||||
if (!state) return original?.apply(this, args);
|
||||
// Nodes 2.0 exposes a stable array view: assigning widgets mutates
|
||||
// that same view, so retain a snapshot rather than its reference.
|
||||
const displayOrder = [...this.widgets];
|
||||
state.schemaOrderDepth++;
|
||||
this.widgets = [...state.originals, ...displayOrder.filter(w => !state.originals.includes(w))];
|
||||
try {
|
||||
return original?.apply(this, args);
|
||||
} finally {
|
||||
this.widgets = displayOrder;
|
||||
state.schemaOrderDepth--;
|
||||
if (method === "configure") state.sync();
|
||||
}
|
||||
};
|
||||
}
|
||||
const connections = nodeType.prototype.onConnectionsChange;
|
||||
nodeType.prototype.onConnectionsChange = function(...args) {
|
||||
const result = connections?.apply(this, args);
|
||||
this._refmodSlots?.sync();
|
||||
return result;
|
||||
};
|
||||
}
|
||||
|
||||
function setupSlots(node) {
|
||||
if (node._refmodSlots) return;
|
||||
const originals = [...(node.widgets || [])];
|
||||
const groups = new Map();
|
||||
for (const widget of originals) {
|
||||
const match = widget.name.match(SLOT);
|
||||
if (!match) continue;
|
||||
const slot = Number(match[1]);
|
||||
if (!groups.has(slot)) groups.set(slot, []);
|
||||
groups.get(slot).push(widget);
|
||||
}
|
||||
const slots = [...groups.keys()].sort((a, b) => a - b);
|
||||
if (!slots.length) return;
|
||||
const defaults = new Map(originals.map(w => [w, w.value]));
|
||||
const hidden = new Map();
|
||||
const removers = new Map();
|
||||
node.properties ??= {};
|
||||
node.properties.refmod_visible_slots ??= [slots[0]];
|
||||
const linked = slot => node.inputs?.some(input => Number(input.name.match(SLOT)?.[1]) === slot && input.link != null);
|
||||
const used = slot => groups.get(slot).some(w => w.name.startsWith("mod_")
|
||||
? w.value != null && !NONE.has(String(w.value).trim())
|
||||
: w.value !== defaults.get(w));
|
||||
|
||||
function show(widget, visible) {
|
||||
if (!visible && !hidden.has(widget)) {
|
||||
hidden.set(widget, {computeSize: widget.computeSize,
|
||||
draw: widget.draw, mouse: widget.mouse, hidden: widget.hidden,
|
||||
optionsHidden: widget.options?.hidden});
|
||||
// converted-widget means a socket row in Nodes 2.0, not a hidden
|
||||
// widget. Its empty row was responsible for the large gaps.
|
||||
widget.options = {...widget.options, hidden: true};
|
||||
widget.computeSize = () => [0, -4];
|
||||
widget.draw = () => {};
|
||||
widget.mouse = () => false;
|
||||
widget.hidden = true;
|
||||
} else if (visible && hidden.has(widget)) {
|
||||
const saved = hidden.get(widget);
|
||||
widget.options = {...widget.options, hidden: saved.optionsHidden};
|
||||
widget.computeSize = saved.computeSize;
|
||||
widget.draw = saved.draw;
|
||||
widget.mouse = saved.mouse;
|
||||
widget.hidden = saved.hidden;
|
||||
hidden.delete(widget);
|
||||
}
|
||||
}
|
||||
|
||||
function sync() {
|
||||
// configure fires connection callbacks before assigning widget values.
|
||||
// Keep schema order until the entire positional restore has finished.
|
||||
if (node._refmodSlots.schemaOrderDepth) return;
|
||||
const requested = node.properties.refmod_visible_slots;
|
||||
// A saved workflow can bring this back as {0: 1, 1: 2, ...} rather than
|
||||
// an array, so accept either shape before falling back to slot one.
|
||||
const requestedSlots = Array.isArray(requested)
|
||||
? requested
|
||||
: requested && typeof requested === "object"
|
||||
? Object.values(requested)
|
||||
: [slots[0]];
|
||||
const visible = new Set(requestedSlots.filter(slot => groups.has(slot)));
|
||||
for (const slot of slots) if (used(slot) || linked(slot)) visible.add(slot);
|
||||
node.properties.refmod_visible_slots = [...visible].sort((a, b) => a - b);
|
||||
const grouped = [...groups.values()].flat();
|
||||
const extras = node.widgets.filter(w => !originals.includes(w) && ![...removers.values(), add].includes(w));
|
||||
const displayOrder = [];
|
||||
for (const slot of slots) {
|
||||
for (const widget of groups.get(slot)) {
|
||||
show(widget, visible.has(slot));
|
||||
displayOrder.push(widget);
|
||||
}
|
||||
const remove = removers.get(slot);
|
||||
remove.disabled = !!linked(slot);
|
||||
show(remove, visible.has(slot));
|
||||
displayOrder.push(remove);
|
||||
}
|
||||
show(add, visible.size < slots.length);
|
||||
displayOrder.push(add, ...originals.filter(w => !grouped.includes(w)), ...extras);
|
||||
node.widgets = displayOrder;
|
||||
node.setSize?.([node.size[0], node.computeSize()[1]]);
|
||||
node.setDirtyCanvas?.(true, true);
|
||||
}
|
||||
|
||||
for (const slot of slots) {
|
||||
removers.set(slot, node.addWidget("button", `Remove RefMod ${slot}`, null, () => {
|
||||
if (linked(slot)) return;
|
||||
for (const widget of groups.get(slot)) widget.value = defaults.get(widget);
|
||||
node.properties.refmod_visible_slots = node.properties.refmod_visible_slots.filter(i => i !== slot);
|
||||
sync();
|
||||
}, {serialize: false}));
|
||||
for (const widget of groups.get(slot)) {
|
||||
const callback = widget.callback;
|
||||
widget.callback = function(...args) {
|
||||
const result = callback?.apply(this, args);
|
||||
sync();
|
||||
return result;
|
||||
};
|
||||
}
|
||||
}
|
||||
const add = node.addWidget("button", "+ Add RefMod", null, () => {
|
||||
const next = slots.find(slot => !node.properties.refmod_visible_slots.includes(slot));
|
||||
if (next != null) node.properties.refmod_visible_slots.push(next);
|
||||
sync();
|
||||
}, {serialize: false});
|
||||
node._refmodSlots = {originals, sync, schemaOrderDepth: 0};
|
||||
sync();
|
||||
}
|
||||
|
||||
app.registerExtension({
|
||||
name: "MiniMaxH3.AdditiveSlots",
|
||||
beforeRegisterNodeDef(nodeType, nodeData) {
|
||||
if (["MiniMaxH3RefModsLoader", "MiniMaxH3RefModsAxis"].includes(nodeData.name)) {
|
||||
installRefModSlots(nodeType);
|
||||
}
|
||||
},
|
||||
});
|
||||