Merge commit '70d0672ee306019c7dba8f799a273ec486bec95b' as 'vendor/dumas_minimax_h3_refmod'

This commit is contained in:
2026-10-20 13:43:44 +00:00
49 changed files with 7097 additions and 0 deletions
+234
View File
@@ -0,0 +1,234 @@
# Changelog
All notable changes are tracked here. Each version is also published as a
[GitHub Release](https://github.com/Luisacaotica/ComfyUI-MiniMaxH3Mod/releases),
so you can keep using an older version if a new one changes something you
rely on.
## Unreleased
- Correct visual reference sampling to H3's 17k+5 video grid in the node and CLI. Preserve all frames of aligned clips and both endpoints when uniformly sampling; references/caps below five frames use the first image.
- Support MPS-resident RefMod latents without global CPU fallback: pool and blur explicitly on CPU, preserving return device, dtype and autograd. Synchronize MPS in the refinement timing harness.
- Add a reference_map output to Inspect H3 RefMod for LLM prompt generation before Text Encode, using the same reference numbering (#19).
- Fix Loader/Axis settings resetting when switching workflow tabs or reopening workflows: defer slot regrouping until widget restoration finishes (#18).
- Repair Apply's reactive MatchType socket on older frontends whose canvas socket classes use private fields, preventing missing controls while preserving conditioning type matching (#17).
## v0.2.6 — 2026-09-12
- Make Loader/Axis additive: start with one visible slot, add/remove slots up to eight, group modality controls with their reference, preserve old workflow values and protect connected slots.
- Add version-5 single-file RefMod bundles containing independent image/video/audio members, with atomic saves and preserved member order.
- Add Save H3 RefMod Bundle as an output node and opt-in Master save_layout=bundle. Keep standalone saving and existing workflow defaults.
- Add All/Visual/Audio selection and separate modality strengths to every Loader/Axis slot. Skip excluded tensors and count only selected references in token budgets.
- Discover combined files in the library; preserve unselected members during Config updates and support export to standalone files.
- Document the container schema for third-party pickers. Bundling does not add voice binding, AV synchronization, or original-waveform passthrough.
## v0.2.5 — 2026-09-08
- Rename Extract nodes to Create, with unchanged node IDs. Full Reference and Compressed Reference replace the visible encode/training names; old mode values remain accepted. Rename the identity control to Refinement Steps.
- Add visual budget_policy: truncate preserves existing behavior; error stops before saving when the visual token budget is exceeded. Master inherits the option.
- Allow larger token/frame limits and preserve the requested frame count in the motion preset. Loader budgets default to 0 (disabled).
- Add experimental H3 RefMod Text Encode with numbered reference presentation and a reference map. Support compatible ClipProj encoders and native batched video-VAE output.
- Add optional full-video inspection with correct IMAGE output dimensions; retain the lightweight first-frame default.
- Add playable music/video README examples and exclude local tests from distribution.
Validation: 51 local regression tests passed; creation-label hooks checked with Node.js. Full GPU generation, projected-encoder quality and voice transfer remain unverified. Voice cloning is not claimed to work.
Text Encode already attaches references: do not inject the same bundle again with Apply/Bridge. Integrated Continuum samplers still need a dedicated integration to use this conditioning. Full visual previews and text/vision presentation increase memory use.
## v0.2.0 — 2026-09-07
### Known limitation: voice transfer
Audio reference support includes an observed music-transfer example, but current
speaker-identity tests failed. This release does not provide working voice cloning.
The cause has not been isolated between H3/checkpoint behavior and the reference
presentation difference in the current RefMod integration.
### Follow-up implementation (2026-09-07)
- Clarify Master saving logs: deferred extraction, Created/Replaced destinations,
explicit save=False and a final token/file summary; report saved_paths in details.
- Add Save H3 RefMods as an output node: save mixed bundles without connecting
a Preview or sampler, with prefix/subfolder options and copy deduplication.
- Save new mods/presets in the first registered refmods root (extra_model_paths);
keep the standard models/refmods fallback and existing-file Config locations.
- Remove sibling-pack imports and startup dependency warning. Legacy av_encoder
inputs use native ComfyUI video VAE loading; standard VAE inputs are preferred.
- Add Extract H3 RefMod Master: optional image/video and audio inputs, sequential
extraction through the shared extractors, combined token budget, one bundle
output and separate compatible visual/audio files under a shared name.
- Step Curve now mixes the current payload without retaining tensors and handles
keyframe offsets, audio refs, empty conditioning and chained curves.
- Bridge uses the MODEL outer-sampling hook with finally cleanup; removes global
state and third-party monkey patches. The old Disarm node is deprecated.
- Align image/mask center crops; share resize and causal-frame helpers with CLI.
- Invalidate loaded files and queue cache on changes; cap cache at 256 MiB;
enforce copies/numeric limits; bound video decode buffers with unknown lengths.
- Add native audio RefMods, legacy AudioMod import, audio Extract, library UI,
inspection/optional previews, save subfolders, explicit scramble modes,
extraction presets and total-token budgets.
- Atomic saves, deduplicated config writes and grouped mean-MSE refinement.
- Historical entries below describe earlier unreleased revisions; the behavior
above and the current README supersede their bridge/cache descriptions.
### Fixed: linked loader inputs and subfolders
- Both loaders accept unresolved/empty mod names and connected STRING/combo
outputs. Fixed missing names still fail validation; connected names resolve
at execution with a missing-file error when necessary.
- Discover RefMods recursively as `folder/name`, preserving metadata filtering
and skipping graph presets. Listing cache tracks nested files, nanosecond
timestamps, and legacy JSON sidecars.
- Normalize Windows separators and check relative paths at execution. Cache
mods by file path so identical metadata names in different folders do not
collide when saving their configuration.
- Add regression tests using real safetensors and ComfyUI queue validation.
### New: Continuum integration
- **Continuum RefMod Bridge** node — injects your mods into
[ComfyUI-H3-Continuum](https://github.com/xiaolibai-sys/ComfyUI-H3-Continuum)'s
chunked long-video samplers (V2/V3/V3.4), which build their conditioning
internally and have no CONDITIONING socket. While armed, every chunk
carries the bundle's ref blocks as native `minimax_refs` — persistent
across all chunks, no Continuum markers so its continuity layout never
re-times them. Version-checked against one public seam; if Continuum
changes internally the bridge reports and degrades to no-injection rather
than breaking a run. One active bundle per graph; per-run scramble seeds
work normally. Wire it on the sampler's model line (Load Model -> bridge
-> sampler `model`): ComfyUI never executes fully disconnected nodes, so
the earlier "anywhere upstream" placement silently did nothing.
### Fixed: why insertions felt weak, random, or "possessed"
Three defects were found and fixed; each is reproducible with the new
`gauntlet_harness.py` (run it with ComfyUI's Python — no server needed).
- **Apply's default curve silently HALVED every multi-frame mod.** The old
default (`concept_at_end` + `ease`) runs its envelope across the mod's own
ref frames — mean multiplier ≈ **0.50** at any frame count — so video/
stacked mods shipped half-blurred unless you touched widgets. Worse, ref
tokens are not bound to output-video time at all: no frame curve can place
a concept "at the end of the video"; it only weights which reference
content dominates. Defaults are now `constant` + `linear` + `1.0`
(= full strength, official-ref parity); saved workflows keep their saved
values. The curve widgets remain for stack weighting, with honest tooltips.
For real output-timing control use **H3 RefMod Step Curve** (denoise
timeline).
- **Image-kind mods ignored every curve dial silently** (`curve_strengths`
short-circuits at 1 latent frame), so single-image identity mods always
injected at full strength — the "concept bleeds into everything" case.
Now `curve_value` acts as a plain strength cap on single-frame mods
(e.g. value 0.4 = the ref blends 40% toward its blurred self), with a
console note.
- **H3 RefMod Step Curve leaked state across generations.** The wrapper
cached pristine/blurred latents plus the marked-ref index mapping on the
first forward and never reset them — but ComfyUI reuses the patched MODEL
and conditioning payload across queued runs, so run #2 mixed run #1's
cached tensors whenever the scramble seed/mods/strengths changed
(corrupted, "ghost" output). The wrapper now keys its state on the
injected refs' identities (which it never mutates), re-keying cleanly on
every new run while keeping pristine stable within one; chaining multiple
Step Curve nodes composes deterministically, and progress is normalized to
each run's actual schedule start (partial denoise included).
Also new: Apply now prints each row's *effective* strength (row × retention
× curve-mean) and flags anything below 0.30 — the usual suspect when an
insertion feels weak.
### New: fixed configs, shipped with the mod
- **Fix H3 RefMod Config** node — bakes tuned Apply/Step-Curve settings
(`retention` + `curve` + `step_curve`) into a mod's safetensors metadata
and re-saves the file in place, so a concept ships with the settings that
make it work.
- **`override` toggle** on Apply H3 RefMod and H3 RefMod Step Curve — reads
the first mod in the bundle that carries a fixed config and uses those
settings instead of the widgets; falls back to the manual parameters
(with a console note) when no mod has one.
- The debug curve graph reflects the overridden curve, so what you see is
what actually ran.
### New: merge mode on Extract H3 RefMod
- **`merge` toggle** (training mode): one shared latent refined jointly
against every reference's full encode (mean reconstruction error) instead
of stacking each ref's own pooled latent. A whole collection becomes ONE
tiny consensus mod — pulled toward what's common across all the views
(structure/motion/identity) rather than any single shot's framing or
background, at one ref block's worth of tokens. `identity` dials the
joint refinement; masks and `background_retention` still apply per ref.
- **`motion_only` toggle** (training mode, experimental): video refs are
converted to per-frame temporal differences before encoding, so the mod
carries where/how things move instead of what they look like — static
appearance (lineart look, background) never enters the latent. A soft
motion guide, not a ControlNet (the ref channel is content-based).
### Fixed
- **Curve directions now mean what they say.** `concept_at_end` actually
puts the concept at the **end** of the video (and at the end of the
denoise timeline on H3 RefMod Step Curve); `concept_at_start` puts it at
the start. The old envelopes were inverted relative to the names. Legacy
`decrease`/`increase` workflow values and the preset names
(`fade_in`/`fade_out`/`bump`/`dip`) still resolve to their original
envelopes, so old workflows keep their behavior.
- **H3 RefMod Step Curve no longer touches your input video.** The wrapper
used to re-mix *every* ref latent, including the original video's
identity anchor — that blurred the source subject and let a different
person "pop". It now identifies the refs injected by Apply H3 RefMod (via
a marker on the blocks) and only re-mixes those; native ref2va refs pass
through untouched, and a console note explains when there is nothing to
re-mix.
## v0.1.0 — 2026-08-16
First tagged release. Extract references once as tiny `.safetensors`
"mods" and inject them through conditioning — no full video/image loading
every generation, no training.
### Nodes
- **Extract H3 RefMod** — image / video / GIF → one small mod file. Modes:
`training` (pooled concept/identity thumbnails; the `pool` dial trades
concept ↔ identity) and `encode` (full-resolution VAE encode). Identity
refinement steps, token cap with dedup, data multiplier for short clips,
optional `av_encoder` input, and folder bulk-loading.
- **Load H3 RefMods** — LoRA-loader-style rows with a typed strength and a
`copies` multiplier (2-10x — the manual row-duplication trick as a knob).
- **Load H3 RefMod Axis** — signed A/B sliders: negative picks the A mod,
positive the B mod, one dial controls both.
- **Load H3 RefMod Folder** — every image/video in a folder as an ordered
ref list.
- **Apply H3 RefMod** — one node for the pack's `MINIMAX_H3_COND` and the
built-in `CONDITIONING`. `retention` master strength; a curve split into
`curve_direction` (constant / concept_at_start / concept_at_middle /
concept_at_end / concept_at_ends) + `curve_shape` (linear / ease /
sigmoid / tanh / quadratic / cubic / exponential / stair / elastic /
bump / dip) + `curve_value`; `scramble_seed`; optional curve-graph
`debug` IMAGE output; shareable PNG graph presets (graph embedded in the
image metadata, legacy `.json` still loads).
- **H3 RefMod Step Curve** — the same curve widgets, but over the **denoise
timeline**: re-mixes every ref latent once per step (early steps lock
composition/identity, late steps stay clean or refine detail) via a
ComfyUI `DIFFUSION_MODEL` wrapper, attached between the model loader and
the sampler.
### Reference math
- Weakening a ref blends toward a blurred copy of itself instead of noise
or zero — stays on the latent manifold, so no grey/static output.
- The per-frame curve mixes each ref latent frame with
`retention * curve(x)` instead of one flat strength.
- Greedy temporal dedup + budget-fit resampling make the token cap cheap.
### Misc
- `extract_mod.py` standalone CLI.
- Mods live in `ComfyUI/models/refmods/` (created on first run, next to
loras/ and unet/); older mods in the pack's `mods/` folder still load.