diff --git a/README.md b/README.md index dc9c6f3..6f0d13c 100644 --- a/README.md +++ b/README.md @@ -1,60 +1,375 @@ # ComfyUI-H3-Continuum 2.1.7 -Long-form native MiniMax H3 audio/video continuation for ComfyUI. +Native long-form MiniMax H3 video and audio continuation for ComfyUI. -**H3 Continuum Sampler** generates 1–16 H3 chunks in one compact facade, carries the previous raw video/audio latent directly into the next chunk, stores accepted chunks on CPU, and defers VAE decode until sampling is complete. +**H3 Continuum Sampler** generates 1 to 16 H3 chunks as one sequence. It carries the previous raw video and audio latent into the next chunk, keeps accepted chunk latents on CPU, and defers VAE decoding until sampling is complete. -## V2.1.7 Stable Facade UI +## Quick start -V2.1.7 replaces the dynamic V2.1.6 display controller with a small static facade. It uses standard ComfyUI rendering and delegates generation to the fully compatible `H3ContinuumSamplerV2` execution core. +1. Install the custom node and restart ComfyUI. +2. Add **H3 Continuum Sampler**. +3. Connect the normal H3 model, CLIP, VAEs, sampler, and sigmas. +4. Connect one **Text (Multiline)** node to **Sequence Prompt**. +5. For image-to-video, connect the source image to **first_frame**. +6. Connect **images** and **audio** to your normal video output node. +7. Start with two 5-second chunks and the settings shown below. -- `H3 Continuum Sampler` contains only the normal sequence controls and three optional pack inputs. -- `H3 Continuum Clip Overrides` supplies per-clip prompts through one static pack socket. -- `H3 Continuum Advanced` contains resume/session inputs and infrequent settings. -- `H3 Continuum Result` expands `last_state`, `session`, and `report` only when needed. -- The old `H3ContinuumSamplerV2` identifier remains registered as a Legacy/Core compatibility node. -- No dynamic sockets, proxy widgets, DOM hiding, MutationObserver, widget-size overrides, or `widgets_values` reconstruction. -- State, Session, Prompt Plan, Sampling, Native Continuity, and Spectrum Interop contracts are unchanged. +Most users only need **H3 Continuum Sampler**. The helper nodes are optional. -## Core features +## Installation -- Native raw H3 video/audio latent continuation; no decode/re-encode between chunks. -- 5 / 22 / 39-frame context profiles plus conservative Auto. -- Native H3 17k+5 temporal grid and signed 40/24 audio-grid alignment. -- Call-local MODEL clone per chunk; Spectrum runtime/history does not leak across chunks. -- Spectrum Interop API v1 requests Actual Prefix 2 only when valid continuation context exists. -- `Sequence Prompt` with Auto / Fixed / List / Timeline parsing and per-clip overrides. -- Seam Correction: `Auto / Off / Basic`. -- Session resume and branch-safe reroll. -- Exact cumulative audio boundaries and exact total-duration correction. -- No additional pip dependencies beyond the normal ComfyUI environment. +### Git installation -## Recommended graph +From the ComfyUI custom_nodes directory: -```text -H3 UNET -> SageAttention -> Sol-Attn -> LoRA -> Spectrum - | -CLIP / Video VAE / Audio VAE / Sampler / Sigmas | - v - H3 Continuum Sampler - | | - IMAGE AUDIO -``` +~~~bash +git clone https://github.com/ukr8b3g-cmyk/ComfyUI-H3-Continuum.git +~~~ -Recommended first validation: `chunks=2`, `chunk_seconds=5`, `Balanced — 22 frames`, `Seam Correction=Off` for native-continuity A/B, then test Auto separately. +Restart ComfyUI, then search for **H3 Continuum Sampler**. -## Install +For Stability Matrix, the destination is normally similar to: -Copy the extracted `ComfyUI-H3-Continuum` folder into `ComfyUI/custom_nodes/` and restart ComfyUI, or run `install_windows.bat`. +~~~text +D:\StabilityMatrix\Data\Packages\\custom_nodes\ +~~~ -If replacing the older `ComfyUI-H3-Continuum-Join` folder, remove or rename the old folder so both packages are not loaded at once. The included installer backs up both old and new destination names. +### ZIP or Windows installer + +1. Extract the repository as **ComfyUI-H3-Continuum**. +2. Place it in **ComfyUI/custom_nodes/**. +3. Run **install_windows.bat** if you want the included Windows installer to copy and validate the package. +4. Restart ComfyUI. + +If an older **ComfyUI-H3-Continuum-Join** folder exists, remove or rename it before starting ComfyUI. Do not load both folders at the same time. + +No additional pip packages are required beyond the normal ComfyUI MiniMax H3 environment. + +## Standard connection + +~~~text +H3 MODEL loader + | + +-> optional SageAttention + +-> optional Sol-Attn + +-> optional LoRA + +-> optional Spectrum + | + v +H3 Continuum Sampler.model + +CLIP loader -----------------------> clip +MiniMax H3 Video VAE -------------> video_vae +MiniMax H3 Audio VAE -------------> audio_vae +Sampler --------------------------> sampler +Sigmas ---------------------------> sigmas +Text (Multiline) -----------------> Sequence Prompt +Load Image -----------------------> first_frame + +H3 Continuum Sampler.images ------> Create Video.images +H3 Continuum Sampler.audio -------> Create Video.audio +~~~ + +The accelerator chain is optional. A typical accelerated order is: + +~~~text +H3 MODEL -> SageAttention -> Sol-Attn -> LoRA -> Spectrum +~~~ + +When troubleshooting, start with fewer accelerators and add them back one at a time. + +### Main inputs + +| Input | Connect | +| --- | --- | +| **model** | The final MODEL from the normal H3 model or accelerator chain | +| **clip** | MiniMax H3 text encoder / CLIP output | +| **video_vae** | MiniMax H3 Video VAE | +| **audio_vae** | MiniMax H3 Audio VAE | +| **sampler** | The sampler used by the normal H3 workflow | +| **sigmas** | The corresponding sigma schedule | +| **Sequence Prompt** | One external Text (Multiline) STRING | +| **first_frame** | Source image for image-to-video | +| **prompt_overrides** | Optional H3 Continuum Clip Overrides pack | +| **advanced** | Optional H3 Continuum Advanced pack | + +### Main outputs + +| Output | Use | +| --- | --- | +| **images** | Connect directly to the image/video frames input of the output node | +| **audio** | Connect directly to the audio input of the output node | +| **result** | Optional packed State, Session, and report data | + +## Recommended first run + +| Setting | Recommended value | +| --- | --- | +| **Prompt Format** | Auto | +| **chunks** | 2 | +| **chunk_seconds** | 5.0 | +| **width / height** | Match the intended H3 generation resolution | +| **continuity** | Balanced - 22 frames | +| **base_seed** | Fixed while comparing settings | +| **control after generate** | fixed for A/B testing | +| **seam_correction** | Auto for normal use | + +This produces approximately 10 seconds. + +For a native-continuity comparison, use the same prompt and seed and compare: + +~~~text +Seam Correction = Off +Seam Correction = Auto +~~~ + +**Auto** is the normal recommendation. It keeps the native seam unless the guarded correction produces a safely measurable improvement. + +### Seam Correction modes + +- **Auto**: Native-first mode. Applies post-decode correction only when the guarded candidate improves the measured seam. +- **Off**: No post-decode seam correction. Use this to inspect Native Continuity directly. +- **Basic**: Runs the V2.1 seam-correction path directly. + +### Continuity profiles + +- **Auto - conservative**: Chooses a conservative context profile from the previous state. +- **Fast - 5 frames**: Lowest overlap and lowest continuity cost. +- **Balanced - 22 frames**: Recommended starting point. +- **Strong - 39 frames**: More context, with higher cost and a greater risk of over-constraining motion. + +## Sequence Prompt + +Connect a single **Text (Multiline)** node to **Sequence Prompt**. + +### Auto + +Auto detects Fixed, List, or Timeline syntax. This is recommended for normal use. + +### Fixed + +One prompt is used for every chunk. + +~~~text +A single continuous cinematic performance. Preserve the same person, +lighting, camera direction, motion, voice, and music across every chunk. +~~~ + +### List + +Separate one prompt per chunk with a line containing three hyphens. + +~~~text +The woman begins singing while the camera slowly pushes in. +--- +She continues the same phrase without a cut, reset, or pose jump. +--- +She resolves the final note naturally and releases a soft breath. +~~~ + +If fewer List sections are supplied than the configured chunk count, the last section is reused. + +### Timeline + +Use time or chunk sections in one multiline prompt. + +~~~text +[0-5s] +The woman begins singing. The piano phrase remains unresolved. + +[5-10s] +Continue from the exact preceding breath, pose, camera movement, and melody. + +[10-15s] +Complete the melody naturally during the final second. +~~~ + +Chunk headers are also accepted: + +~~~text +[Chunk 1] +Opening action. + +[Chunk 2] +Continuous middle action. + +[Chunk 3] +Natural ending. +~~~ + +## How continuation works + +For every continuation chunk, Continuum: + +1. Reads the accepted previous raw H3 video and audio latent. +2. Extracts a phase-valid video and audio context window. +3. Adds that context as fixed reference rows. +4. Keeps new target audio and video rows separate and writable. +5. Clones the workflow MODEL once for the current chunk. +6. Samples the next chunk without decoding and re-encoding the prior result. +7. Stores the accepted full chunk latent on CPU. +8. Decodes and assembles all accepted chunks after sampling finishes. + +The workflow input MODEL is not modified. Each chunk clone has its own Spectrum runtime/history. + +When valid continuation context exists, Continuum emits Spectrum Interop API v1 with an Actual Prefix request of 2. Older or unsupported Spectrum versions ignore the hint and continue normally. + +## Optional helper nodes + +These nodes are not needed for the basic workflow. + +### H3 Continuum Clip Overrides + +Use this only when selected clips need prompts different from the main Sequence Prompt. + +Connect one **Text (Multiline)** to **Override Script** and use explicit sparse sections: + +~~~text +[Clip 2] +Move to a closer shot while preserving the same motion and voice. + +[Clip 5] +Finish the song naturally and hold the final note. +~~~ + +Only Clip 2 and Clip 5 are replaced. All unspecified clips continue using the main Sequence Prompt. + +**[Chunk N]** is also accepted. Duplicate clip numbers and clip numbers outside the configured chunk count are rejected. + +Connect the pack output to **H3 Continuum Sampler.prompt_overrides**. + +### H3 Continuum Advanced + +Use this node for resume data and infrequent controls. Connect its output to **H3 Continuum Sampler.advanced**. + +Important controls include: + +- **last_frame**: Optional final-frame anchor for the final chunk. +- **session**: Continue or reroll from a prior V2 Session. +- **initial_state**: Start from a compatible saved State. Do not connect both Session and Initial State. +- **prompt_plan**: Advanced or compatibility prompt-plan input. +- **audio_continuity**: Continue the prior raw audio latent. +- **exact_total_duration**: Correct the assembled output to the requested total duration. +- **diagnostics**: Basic, Detailed Report, or Off. +- **reroll_from_chunk**: Keep accepted earlier chunks and regenerate from this one-based chunk number. +- **reroll_nonce**: Derive a different reroll seed without changing the base seed. +- **strict_compatibility**: Stop on unknown or unsafe H3 layout contracts. +- **debug**: Enable internal diagnostic logging. +- **show_preview**: Enable the normal sampling preview. + +### H3 Continuum Result + +Connect the Sampler **result** output when you need to expose: + +- **last_state** +- **session** +- **report** + +For ordinary final video generation, this node is unnecessary. + +## Sessions and rerolling + +A Session stores accepted CPU latents, prompts, seeds, dimensions, and continuation metadata. + +Typical advanced use: + +~~~text +H3 Continuum Sampler.result + -> H3 Continuum Result.session + -> H3 Continuum Save Session +~~~ + +Later: + +~~~text +H3 Continuum Load Session + -> H3 Continuum Advanced.session + -> H3 Continuum Sampler.advanced +~~~ + +Resolution changes are rejected rather than silently resizing State or Session data. + +## Diagnostics and debugging + +Use **Basic** diagnostics for normal testing. Use **Detailed Report** when inspecting seam scores, selected cuts, audio correlation, correction fallback, and per-chunk context. + +Set **debug=true** only when deeper runtime information is needed. The single-clone implementation reports object IDs without retaining the objects: + +~~~text +Continuum model lifetime: input= chunk= parent= base= +~~~ + +For the expected single-clone path, every chunk parent should be the same workflow input MODEL: + +~~~text +chunk1.parent -> input MODEL +chunk2.parent -> input MODEL +chunk3.parent -> input MODEL +~~~ + +With a compatible Spectrum version, continuation chunks should also log: + +~~~text +Spectrum H3: accepted H3 Continuum API v1, actual prefix=2 +~~~ + +**0 models unloaded** by itself is normal model-management information. The relevant leak warnings are: + +~~~text +Potential memory leak detected with model MiniMaxH3 +WARNING, memory leak with model MiniMaxH3 +~~~ + +If those warnings occur, compare identical one-chunk and two-chunk runs, then test Spectrum, Sol-Attn, and Sage individually. + +## Special and compatibility nodes + +The following nodes remain available for advanced workflows or saved-workflow compatibility: + +- **H3 Continuum Sampler V2**: Legacy/Core identifier used by older V2 workflows. +- **H3 Continuum Prompt Plan**: Explicit prompt-plan construction and inspection. +- **H3 Continuum Save Session** +- **H3 Continuum Load Session** +- **H3 Continuum Session Info** +- V1 Join, Finish, Assemble, Save State, and Load State nodes. + +New workflows should normally use the compact **H3 Continuum Sampler** facade. + +## Compatibility and safety + +- Native H3 PackedLayout is preserved. +- layout.position_ids identity is preserved when MM-RoPE coordinates are adjusted. +- State and Session schema remain compatible. +- V1 node identifiers remain registered. +- Video and audio VAEs are not run between H3 chunks. +- Full accepted chunk latents are stored on CPU. +- SageAttention, Sol-Attn, LoRA, and Spectrum remain external packages. +- Unknown H3 layout contracts fail clearly instead of producing a subtly incorrect continuation. ## Development checks -```bash +Use the same Python environment as ComfyUI: + +~~~bash python -m compileall -q . pytest -q -``` +~~~ + +Current local validation for this change: + +~~~text +74 passed +~~~ + +GPU validation should compare identical prompts, seeds, resolutions, samplers, and sigmas across: + +~~~text +Native +Sage +Sage + Sol-Attn +Sage + Spectrum +Sage + Sol-Attn + Spectrum +~~~ ## License diff --git a/model_patch.py b/model_patch.py index b220618..27bf800 100644 --- a/model_patch.py +++ b/model_patch.py @@ -2,6 +2,8 @@ from __future__ import annotations + +import logging from typing import Any, Callable from .compatibility import CompatibilityError, ensure_native_h3_base_model @@ -51,12 +53,12 @@ def _wrapper_factory(*, strict: bool, debug: bool) -> Callable[..., Any]: return apply_model_wrapper -def patch_model(model: Any, *, strict: bool, debug: bool): +def configure_continuum_model(model: Any, *, strict: bool, debug: bool): try: from comfy.patcher_extension import WrappersMP except Exception as exc: # pragma: no cover raise RuntimeError(f"ComfyUI model-wrapper API unavailable: {exc}") from exc - required_api = ("clone", "add_wrapper_with_key", "remove_wrappers_with_key") + required_api = ("add_wrapper_with_key", "remove_wrappers_with_key") missing_api = [name for name in required_api if not hasattr(model, name)] if missing_api: raise RuntimeError( @@ -66,7 +68,7 @@ def patch_model(model: Any, *, strict: bool, debug: bool): if base_model is None: raise RuntimeError("MODEL does not expose the ComfyUI BaseModel") ensure_native_h3_base_model(base_model) - patched = model.clone() + patched = model patched.remove_wrappers_with_key(WrappersMP.APPLY_MODEL, MODEL_WRAPPER_KEY) patched.add_wrapper_with_key( WrappersMP.APPLY_MODEL, @@ -80,6 +82,13 @@ def patch_model(model: Any, *, strict: bool, debug: bool): patched.model_options = model_options return patched +def patch_model(model: Any, *, strict: bool, debug: bool): + """Clone once, then install the Continuum wrapper on that clone.""" + if not hasattr(model, "clone"): + raise RuntimeError( + "MODEL lacks the wrapper API required by H3 Continuum Join: clone" + ) + return configure_continuum_model(model.clone(), strict=strict, debug=debug) def continuum_interop_request(*, chunk_index: int, context_frames: int) -> dict[str, Any]: return { @@ -100,7 +109,23 @@ def clone_model_for_chunk( context_frames: int | None, ): """Create a call-local MODEL and attach an optional read-only Spectrum hint.""" - chunk_model = patch_model(model, strict=bool(strict), debug=bool(debug)) + if not hasattr(model, "clone"): + raise RuntimeError( + "MODEL lacks the wrapper API required by H3 Continuum Join: clone" + ) + chunk_model = model.clone() + configure_continuum_model( + chunk_model, strict=bool(strict), debug=bool(debug) + ) + if debug: + parent = getattr(chunk_model, "parent", None) + logging.getLogger(__name__).info( + "Continuum model lifetime: input=%s chunk=%s parent=%s base=%s", + id(model), + id(chunk_model), + id(parent) if parent is not None else None, + id(getattr(chunk_model, "model", None)), + ) model_options = dict(getattr(chunk_model, "model_options", None) or {}) transformer_options = dict(model_options.get("transformer_options") or {}) if context_frames is None: diff --git a/tests/test_model_patch.py b/tests/test_model_patch.py index d477b62..5e3919b 100644 --- a/tests/test_model_patch.py +++ b/tests/test_model_patch.py @@ -1,15 +1,21 @@ from types import SimpleNamespace +import pytest import torch from ComfyUI_H3_Continuum_Join.constants import ( + CONTINUUM_INTEROP_KEY, MARK_AUDIO_CONTEXT, MARK_AUDIO_END_FRAME, MARK_AUDIO_OVERHANG, MARK_CONTEXT_FRAMES, MARK_VIDEO_CONTEXT, ) -from ComfyUI_H3_Continuum_Join.model_patch import _wrapper_factory, patch_model +from ComfyUI_H3_Continuum_Join.model_patch import ( + _wrapper_factory, + clone_model_for_chunk, + patch_model, +) class Executor: @@ -75,6 +81,36 @@ def test_apply_model_wrapper_repairs_payload_before_executor(): assert torch.equal(result["layout"].position_ids[3:4], result["layout"].position_ids[6:7]) +def test_apply_model_wrapper_fails_when_actual_topology_changes(): + video = torch.zeros(1, 24, 1, 1, 1) + audio = torch.zeros(1, 32, 2, 1) + ref = { + "kind": "video_audio", + "latent_t": 1, + "latent_h": 1, + "latent_w": 1, + "ref_audio_t": 1, + "latent": video, + "audio_latent": audio, + MARK_VIDEO_CONTEXT: True, + MARK_CONTEXT_FRAMES: 1, + MARK_AUDIO_CONTEXT: True, + MARK_AUDIO_END_FRAME: 1.0, + MARK_AUDIO_OVERHANG: 0.0, + } + payload = {"layout": _layout(), "keyframes": [], "refs": [ref]} + wrapper = _wrapper_factory(strict=True, debug=False) + options = { + "spectrum_h3_actual": True, + "cond_or_uncond": [0], + "uuids": ["positive"], + } + wrapper(Executor(), torch.zeros(1), transformer_options=options, minimax_payload=payload) + payload["layout"].position_ids[-1, 1] = 1.0 + with pytest.raises(RuntimeError, match="topology changed"): + wrapper(Executor(), torch.zeros(1), transformer_options=options, minimax_payload=payload) + + class FakeModelPatcher: def __init__(self): inner_type = type( @@ -122,3 +158,43 @@ def test_patch_model_validates_and_installs_per_model_wrapper(monkeypatch): assert patched.removed[0][1] == "h3_continuum_join.apply_model.v1" assert patched.added[0][1] == "h3_continuum_join.apply_model.v1" assert patched.model_options["h3_continuum_join"]["api_version"] == 1 + + +def test_chunk_model_copy_on_write_and_context_scoped_hint(monkeypatch): + import sys + from types import ModuleType + + extension = ModuleType("comfy.patcher_extension") + extension.WrappersMP = SimpleNamespace(APPLY_MODEL="apply_model") + comfy = sys.modules.get("comfy") or ModuleType("comfy") + monkeypatch.setitem(sys.modules, "comfy", comfy) + monkeypatch.setitem(sys.modules, "comfy.patcher_extension", extension) + + source = FakeModelPatcher() + clone_calls = 0 + original_clone = FakeModelPatcher.clone + + def counted_clone(self): + nonlocal clone_calls + clone_calls += 1 + return original_clone(self) + + monkeypatch.setattr(FakeModelPatcher, "clone", counted_clone) + source.model_options = {"transformer_options": {"existing": "kept"}} + first = clone_model_for_chunk(source, strict=True, debug=False, chunk_index=1, context_frames=None) + second = clone_model_for_chunk(source, strict=True, debug=False, chunk_index=2, context_frames=22) + third = clone_model_for_chunk(source, strict=True, debug=False, chunk_index=3, context_frames=22) + + assert clone_calls == 3 + assert source.model_options == {"transformer_options": {"existing": "kept"}} + assert CONTINUUM_INTEROP_KEY not in first.model_options["transformer_options"] + request = second.model_options["transformer_options"][CONTINUUM_INTEROP_KEY] + assert request == { + "api": 1, + "active": True, + "min_actual_prefix_steps": 2, + "chunk_index": 2, + "context_frames": 22, + } + assert request is not third.model_options["transformer_options"][CONTINUUM_INTEROP_KEY] + assert CONTINUUM_INTEROP_KEY not in source.model_options["transformer_options"] diff --git a/tests/test_temporal.py b/tests/test_temporal.py index 7b11da2..54c7832 100644 --- a/tests/test_temporal.py +++ b/tests/test_temporal.py @@ -1,3 +1,5 @@ +import pytest + from ComfyUI_H3_Continuum_Join.temporal import ( align_frame_count_nearest, audio_grid_offset, @@ -5,6 +7,7 @@ from ComfyUI_H3_Continuum_Join.temporal import ( context_slots, latent_slot_offsets, largest_context_capacity, + make_av_context_window, make_extension_shape, pixel_frames_for_latent_t, video_latent_t, @@ -47,6 +50,23 @@ def test_audio_grid_offset_is_signed_and_bounded_on_valid_h3_lengths(): assert any(value > 0 for value in offsets) +def test_native_context_window_uses_source_av_grid_and_cycle_phase(): + balanced = make_av_context_window(141, audio_latent_t(141), 22) + assert (balanced.video_start_slot, balanced.video_stop_slot) == (35, 42) + assert (balanced.audio_start_tick, balanced.audio_stop_tick) == (198, 235) + assert balanced.audio_steps == 37 + + shifted = make_av_context_window(158, audio_latent_t(158), 22) + assert (shifted.video_start_slot, shifted.video_stop_slot) == (40, 47) + assert (shifted.audio_start_tick, shifted.audio_stop_tick) == (226, 263) + assert shifted.audio_grid_offset < 0 + + +def test_native_context_window_rejects_unknown_audio_grid_phase(): + with pytest.raises(ValueError, match="audio-grid offset"): + make_av_context_window(141, audio_latent_t(141) + 2, 22) + + def test_dynamic_state_capacity_for_short_extensions(): assert largest_context_capacity(119) == 39 assert largest_context_capacity(34) == 22 diff --git a/tests/test_v217_stable_facade.py b/tests/test_v217_stable_facade.py index 4b73c61..16e8209 100644 --- a/tests/test_v217_stable_facade.py +++ b/tests/test_v217_stable_facade.py @@ -1,5 +1,7 @@ from pathlib import Path +import pytest + from ComfyUI_H3_Continuum_Join.constants import PROMPT_FORMAT_OPTIONS from ComfyUI_H3_Continuum_Join.v2.nodes import ( H3ContinuumAdvanced, @@ -9,6 +11,7 @@ from ComfyUI_H3_Continuum_Join.v2.nodes import ( H3ContinuumSamplerV2, NODE_CLASS_MAPPINGS, ) +from ComfyUI_H3_Continuum_Join.v2.prompts import PromptPlanError ROOT = Path(__file__).resolve().parents[1] @@ -88,6 +91,45 @@ def test_v217_facade_delegates_to_legacy_core(monkeypatch): assert captured["show_preview"] is True +def test_v217_facade_passes_only_sparse_override_clips(monkeypatch): + captured = {} + + def fake_run(_self, **kwargs): + captured.update(kwargs) + return "images", "audio", "state", "session", "report" + + monkeypatch.setattr(H3ContinuumSamplerV2, "run", fake_run) + pack = H3ContinuumClipOverrides().build( + PROMPT_FORMAT_OPTIONS[0], + "[Clip 2]\nclose-up\n\n[Clip 5]\nfinish naturally", + )[0] + H3ContinuumSampler().run( + model="model", clip="clip", video_vae="video_vae", audio_vae="audio_vae", + sampler="sampler", sigmas="sigmas", sequence_prompt="base", + prompt_mode=PROMPT_FORMAT_OPTIONS[0], chunks=5, chunk_seconds=5.0, + width=1344, height=768, continuity="Balanced - 22 frames", base_seed=123, + seam_correction="Auto", prompt_overrides=pack, + ) + assert captured["clip_2_prompt"] == "close-up" + assert captured["clip_5_prompt"] == "finish naturally" + assert not any(f"clip_{index}_prompt" in captured for index in (1, 3, 4)) + + +def test_v217_facade_rejects_sparse_override_outside_chunks(monkeypatch): + monkeypatch.setattr(H3ContinuumSamplerV2, "run", lambda _self, **kwargs: None) + pack = H3ContinuumClipOverrides().build( + PROMPT_FORMAT_OPTIONS[0], "[Clip 6]\noutside" + )[0] + with pytest.raises(PromptPlanError, match="outside"): + H3ContinuumSampler().run( + model="model", clip="clip", video_vae="video_vae", audio_vae="audio_vae", + sampler="sampler", sigmas="sigmas", sequence_prompt="base", + prompt_mode=PROMPT_FORMAT_OPTIONS[0], chunks=5, chunk_seconds=5.0, + width=1344, height=768, continuity="Balanced - 22 frames", base_seed=123, + seam_correction="Auto", prompt_overrides=pack, + ) + + def test_v217_result_pack_expands_legacy_outputs(): assert H3ContinuumResult().unpack( {"last_state": "state", "session": "session", "report": "report"} diff --git a/tests/test_v2_prompts.py b/tests/test_v2_prompts.py index c0a765c..e1d0a03 100644 --- a/tests/test_v2_prompts.py +++ b/tests/test_v2_prompts.py @@ -1,13 +1,20 @@ import pytest from ComfyUI_H3_Continuum_Join.constants import ( + PROMPT_FORMAT_AUTO, + PROMPT_FORMAT_FIXED, PROMPT_MODE_FIXED, PROMPT_MODE_LIST, PROMPT_MODE_TIMELINE, ) from ComfyUI_H3_Continuum_Join.v2.prompts import ( PromptPlanError, + apply_prompt_overrides, + build_sampler_prompt_plan, + detect_prompt_mode, make_prompt_plan, + parse_sparse_prompt_overrides, + validate_sparse_prompt_overrides, validate_prompt_plan, ) @@ -19,6 +26,31 @@ def test_fixed_prompt_repeats_without_mutation(): assert validate_prompt_plan(plan) is plan +@pytest.mark.parametrize( + ("script", "expected"), + [ + ("one continuous prompt", PROMPT_MODE_FIXED), + ("first\n---\nsecond", PROMPT_MODE_LIST), + ("[0-5s]\nfirst", PROMPT_MODE_TIMELINE), + ("[Chunk 1]\nfirst", PROMPT_MODE_TIMELINE), + ], +) +def test_auto_detects_prompt_format(script, expected): + assert detect_prompt_mode(script) == expected + + +def test_auto_plan_records_detected_format_without_changing_schema_mode(): + plan = make_prompt_plan(mode=PROMPT_FORMAT_AUTO, script="first\n---\nsecond", chunks=2, chunk_seconds=5) + assert plan["mode"] == PROMPT_MODE_LIST + assert plan["prompts"] == ["first", "second"] + assert plan["notes"][0] == "Auto detected List" + + +def test_simple_fixed_format_maps_to_stable_fixed_plan_mode(): + plan = make_prompt_plan(mode=PROMPT_FORMAT_FIXED, script="same", chunks=2, chunk_seconds=5) + assert plan["mode"] == PROMPT_MODE_FIXED + + def test_list_prompt_uses_separator_and_repeats_last(): plan = make_prompt_plan( mode=PROMPT_MODE_LIST, @@ -53,3 +85,66 @@ def test_timeline_requires_coverage(): def test_native_h3_chunk_duration_rejects_sub_four_seconds(): with pytest.raises(PromptPlanError): make_prompt_plan(mode=PROMPT_MODE_FIXED, script="x", chunks=1, chunk_seconds=3.9) + + +def test_external_prompt_overrides_only_the_connected_clip_and_rehashes_it(): + plan = make_prompt_plan(mode=PROMPT_MODE_LIST, script="first\n---\nsecond\n---\nthird", chunks=3, chunk_seconds=5) + updated = apply_prompt_overrides(plan, [None, "external second", None]) + assert updated["prompts"] == ["first", "external second", "third"] + assert updated["hashes"][0] == plan["hashes"][0] + assert updated["hashes"][1] != plan["hashes"][1] + assert updated["hashes"][2] == plan["hashes"][2] + assert updated["notes"][-1] == "external Clip Prompt input(s): 2" + + +def test_external_prompt_with_same_text_keeps_the_same_hash(): + plan = make_prompt_plan(mode=PROMPT_MODE_FIXED, script="same", chunks=2, chunk_seconds=5) + assert apply_prompt_overrides(plan, [None, "same"])["hashes"] == plan["hashes"] + + +def test_external_prompt_rejects_empty_connected_text(): + plan = make_prompt_plan(mode=PROMPT_MODE_FIXED, script="fallback", chunks=1, chunk_seconds=5) + with pytest.raises(PromptPlanError): + apply_prompt_overrides(plan, [" "]) + + +def test_sequence_prompt_precedes_connected_plan_and_legacy_script(): + connected = make_prompt_plan(mode=PROMPT_FORMAT_FIXED, script="connected plan", chunks=2, chunk_seconds=5) + resolved = build_sampler_prompt_plan(prompt_mode=PROMPT_FORMAT_AUTO, prompt_script="legacy", sequence_prompt="first\n---\nsecond", prompt_plan=connected, chunks=2, chunk_seconds=5) + assert resolved["prompts"] == ["first", "second"] + + +def test_connected_plan_precedes_legacy_script_without_sequence_input(): + connected = make_prompt_plan(mode=PROMPT_FORMAT_FIXED, script="connected plan", chunks=2, chunk_seconds=5) + resolved = build_sampler_prompt_plan(prompt_mode=PROMPT_FORMAT_AUTO, prompt_script="legacy", sequence_prompt=None, prompt_plan=connected, chunks=2, chunk_seconds=5) + assert resolved is connected + + +def test_sparse_overrides_replace_only_explicit_clips_and_rehash(): + plan = make_prompt_plan(mode=PROMPT_FORMAT_FIXED, script="Sequence Prompt", chunks=5, chunk_seconds=5) + sparse = validate_sparse_prompt_overrides(parse_sparse_prompt_overrides("[Clip 2]\nclose-up\n\n[Clip 5]\nfinish naturally"), chunks=5) + updated = apply_prompt_overrides(plan, [sparse.get(index) for index in range(1, 6)]) + assert updated["prompts"] == ["Sequence Prompt", "close-up", "Sequence Prompt", "Sequence Prompt", "finish naturally"] + assert updated["hashes"][0] == plan["hashes"][0] + assert updated["hashes"][1] != plan["hashes"][1] + assert updated["hashes"][2] == plan["hashes"][2] + assert updated["hashes"][3] == plan["hashes"][3] + assert updated["hashes"][4] != plan["hashes"][4] + + +def test_sparse_override_clip_one_leaves_remaining_clips_unchanged(): + plan = make_prompt_plan(mode=PROMPT_FORMAT_FIXED, script="base", chunks=5, chunk_seconds=5) + sparse = validate_sparse_prompt_overrides(parse_sparse_prompt_overrides("[Clip 1]\nopening override"), chunks=5) + updated = apply_prompt_overrides(plan, [sparse.get(index) for index in range(1, 6)]) + assert updated["prompts"] == ["opening override", "base", "base", "base", "base"] + + +def test_sparse_override_rejects_out_of_range_clip(): + sparse = parse_sparse_prompt_overrides("[Clip 6]\noutside") + with pytest.raises(PromptPlanError, match="outside"): + validate_sparse_prompt_overrides(sparse, chunks=5) + + +def test_sparse_override_rejects_duplicate_clip(): + with pytest.raises(PromptPlanError, match="more than once"): + parse_sparse_prompt_overrides("[Clip 2]\nfirst\n\n[Chunk 2]\nsecond") diff --git a/tests/test_v2_registration.py b/tests/test_v2_registration.py index fd24306..4ef11bb 100644 --- a/tests/test_v2_registration.py +++ b/tests/test_v2_registration.py @@ -1,3 +1,7 @@ +from ComfyUI_H3_Continuum_Join.constants import ( + PROMPT_FORMAT_OPTIONS, + SEAM_CORRECTION_AUTO, +) from ComfyUI_H3_Continuum_Join.nodes import NODE_CLASS_MAPPINGS @@ -20,3 +24,22 @@ def test_v2_nodes_are_registered_without_removing_v1(): def test_v2_sampler_accepts_connectable_prompt_plan(): schema = NODE_CLASS_MAPPINGS["H3ContinuumSamplerV2"].INPUT_TYPES() assert "prompt_plan" in schema["optional"] + + +def test_v2_sampler_registers_preview_setting_bridge_and_sixteen_prompt_inputs(): + schema = NODE_CLASS_MAPPINGS["H3ContinuumSamplerV2"].INPUT_TYPES() + optional = schema["optional"] + assert schema["required"]["prompt_mode"][0] == PROMPT_FORMAT_OPTIONS + assert schema["required"]["prompt_mode"][1]["default"] == "Auto" + assert optional["sequence_prompt"][0] == "STRING" + assert optional["sequence_prompt"][1]["forceInput"] is True + assert optional["show_preview"][1]["default"] is True + for index in range(1, 17): + prompt_input = optional[f"clip_{index}_prompt"] + assert prompt_input[0] == "STRING" + assert prompt_input[1]["forceInput"] is True + + +def test_v2_sampler_defaults_to_auto_seam_correction(): + schema = NODE_CLASS_MAPPINGS["H3ContinuumSamplerV2"].INPUT_TYPES() + assert schema["required"]["seam_correction"][1]["default"] == SEAM_CORRECTION_AUTO diff --git a/tests/test_v2_sequence_orchestration.py b/tests/test_v2_sequence_orchestration.py index 82500fd..9c53bf5 100644 --- a/tests/test_v2_sequence_orchestration.py +++ b/tests/test_v2_sequence_orchestration.py @@ -22,7 +22,6 @@ def test_v2_samples_every_chunk_before_any_decode(monkeypatch): fake_model=SimpleNamespace(model=SimpleNamespace(diffusion_model=SimpleNamespace()),model_options={},wrappers={},model_dtype=lambda:torch.bfloat16,model_size=lambda:0) assets=IdentityAssets(None,None,None,None,"none") monkeypatch.setattr(sequence,"check_comfy_h3_runtime",lambda:[]) - monkeypatch.setattr(sequence,"patch_model",lambda model,strict,debug:model) def clone_model_for_chunk(model,*,strict,debug,chunk_index,context_frames): options=dict(model.model_options); transformer_options=dict(options.get("transformer_options") or {}) if context_frames is not None: transformer_options[CONTINUUM_INTEROP_KEY]={"api":1,"active":True,"min_actual_prefix_steps":2} diff --git a/tests/test_v2_session.py b/tests/test_v2_session.py index 75f4ff8..56aeb80 100644 --- a/tests/test_v2_session.py +++ b/tests/test_v2_session.py @@ -77,3 +77,36 @@ def test_session_roundtrip_and_last_state(tmp_path, monkeypatch): loaded = session_io.load_session(prefix="test", slot=1) assert torch.equal(loaded["chunks"][0]["video"], session["chunks"][0]["video"]) assert loaded["session_id"] == session["session_id"] + +def test_model_fingerprint_can_add_continuum_wrapper_without_mutating_model(): + from types import SimpleNamespace + + from ComfyUI_H3_Continuum_Join.v2.session import model_fingerprint + + base = SimpleNamespace(diffusion_model=object()) + model = SimpleNamespace( + model=base, + model_options={"transformer_options": {}}, + wrappers={}, + model_dtype=lambda: "float32", + model_size=lambda: 1, + ) + wrapped = SimpleNamespace( + model=base, + model_options={"transformer_options": {}}, + wrappers={ + "apply_model": { + "h3_continuum_join.apply_model.v1": [object()], + } + }, + model_dtype=model.model_dtype, + model_size=model.model_size, + ) + + logical = model_fingerprint( + model, + extra_wrapper_keys=("h3_continuum_join.apply_model.v1",), + ) + + assert logical == model_fingerprint(wrapped) + assert model.wrappers == {} diff --git a/v2/nodes.py b/v2/nodes.py index c63b24e..a930d24 100644 --- a/v2/nodes.py +++ b/v2/nodes.py @@ -19,7 +19,9 @@ from .prompts import ( apply_prompt_overrides, build_sampler_prompt_plan, make_prompt_plan, + parse_sparse_prompt_overrides, prompt_plan_report, + validate_sparse_prompt_overrides, ) from .sequence import run_sequence from .session import entry_to_state, session_summary, validate_session @@ -32,6 +34,7 @@ from .session_io import ( CATEGORY = "MiniMax H3/Continuum" LOG = logging.getLogger("h3_continuum_join") +SPARSE_OVERRIDE_SCHEMA_VERSION = 1 def _repair_v200_example_widget_shift( @@ -493,6 +496,10 @@ class H3ContinuumClipOverrides: { "default": PROMPT_FORMAT_OPTIONS[0], "display_name": "Prompt Format", + "tooltip": ( + "Retained for V2.1.7 workflow compatibility. Sparse overrides " + "use explicit [Clip N] sections." + ), }, ), "override_script": ( @@ -501,8 +508,8 @@ class H3ContinuumClipOverrides: "forceInput": True, "display_name": "Override Script", "tooltip": ( - "Connect one Text (Multiline). Use ---, [Chunk N], or " - "timeline sections according to Prompt Format." + "Connect one Text (Multiline). Define only the clips to replace " + "with [Clip N] or [Chunk N] sections." ), }, ), @@ -517,8 +524,8 @@ class H3ContinuumClipOverrides: def build(self, prompt_mode, override_script): return ( { - "prompt_mode": str(prompt_mode), - "prompt_script": str(override_script), + "schema_version": SPARSE_OVERRIDE_SCHEMA_VERSION, + "overrides": parse_sparse_prompt_overrides(override_script), }, ) @@ -724,15 +731,16 @@ class H3ContinuumSampler: clip_prompt_inputs = {} if prompt_overrides: - override_plan = make_prompt_plan( - mode=prompt_overrides.get("prompt_mode", PROMPT_FORMAT_OPTIONS[0]), - script=prompt_overrides.get("prompt_script", ""), - chunks=int(chunks), - chunk_seconds=float(chunk_seconds), + if ( + type(prompt_overrides.get("schema_version")) is not int + or prompt_overrides.get("schema_version") != SPARSE_OVERRIDE_SCHEMA_VERSION + ): + raise ValueError("unsupported H3_CONTINUUM_CLIP_OVERRIDES schema") + sparse_overrides = validate_sparse_prompt_overrides( + prompt_overrides.get("overrides"), chunks=int(chunks) ) - for index, prompt in enumerate(override_plan.get("prompts", []), start=1): - if isinstance(prompt, str) and prompt.strip(): - clip_prompt_inputs[f"clip_{index}_prompt"] = prompt + for index, prompt in sparse_overrides.items(): + clip_prompt_inputs[f"clip_{index}_prompt"] = prompt images, audio, last_state, session, report = H3ContinuumSamplerV2().run( model=model, diff --git a/v2/prompts.py b/v2/prompts.py index d9e2543..e59a242 100644 --- a/v2/prompts.py +++ b/v2/prompts.py @@ -60,6 +60,24 @@ def _parse_timeline_sections(script): finish() if not sections: raise PromptPlanError("timeline contains no sections") return sections +def parse_sparse_prompt_overrides(script): + sections=_parse_timeline_sections(script); overrides={} + for item in sections: + if item["kind"]!="chunk": raise PromptPlanError("Sparse Clip Overrides only accepts [Clip N] or [Chunk N] sections") + index=int(item["index"]) + if index in overrides: raise PromptPlanError(f"Sparse Clip Overrides defines Clip {index} more than once") + overrides[index]=item["prompt"] + return overrides +def validate_sparse_prompt_overrides(overrides,*,chunks): + chunks=int(chunks) + if not isinstance(overrides,dict): raise PromptPlanError("Sparse Clip Overrides must be a dictionary") + validated={} + for index,prompt in overrides.items(): + if type(index) is not int: raise PromptPlanError("Sparse Clip Override numbers must be integers") + if not 1<=index<=chunks: raise PromptPlanError(f"Sparse Clip Override {index} is outside the configured 1-{chunks} chunks") + validated[index]=_normalize_prompt(prompt,label=f"Clip {index} Override") + if not validated: raise PromptPlanError("Sparse Clip Overrides contains no overrides") + return validated def _parse_timeline(script,chunks,chunk_seconds): sections=_parse_timeline_sections(script); prompts=[]; notes=[] for chunk_index in range(1,chunks+1): diff --git a/v2/sequence.py b/v2/sequence.py index c3c0e34..6a28222 100644 --- a/v2/sequence.py +++ b/v2/sequence.py @@ -16,7 +16,7 @@ from ..constants import ( SEAM_CORRECTION_OPTIONS, V2_CONTINUITY_AUTO, normalize_diagnostics_mode, ) from ..continuation import POLICY_REPLACE, prepare_conditioning -from ..model_patch import clone_model_for_chunk, patch_model +from ..model_patch import clone_model_for_chunk from ..state import assert_context_unchanged, context_fingerprint, make_plan, select_context, validate_state from ..temporal import align_frame_count_up, largest_context_capacity, make_extension_shape from ..version import PACKAGE_VERSION @@ -87,7 +87,9 @@ def run_sequence(*,model:Any,clip:Any,video_vae:Any,audio_vae:Any,sampler:Any,si issues=check_comfy_h3_runtime() if issues and strict_compatibility: raise SequenceRuntimeError("H3 runtime is incompatible: "+"; ".join(issues)) assets=prepare_identity_assets(video_vae,width=width,height=height,first_frame=first_frame,last_frame=last_frame,encode_latents=False) - patched_model=patch_model(model,strict=bool(strict_compatibility),debug=bool(debug)); current_model_fingerprint=model_fingerprint(patched_model) + current_model_fingerprint=model_fingerprint(model,extra_wrapper_keys=("h3_continuum_join.apply_model.v1",)) + accelerators=accelerator_summary(model) + if "Continuum APPLY_MODEL wrapper installed" not in accelerators: accelerators+="; Continuum APPLY_MODEL wrapper installed" preserved,reuse_notes=_preserved_prefix(session=session,prompt_hashes=prompt_hashes,chunks=chunks,reroll_from_chunk=int(reroll_from_chunk),width=width,height=height,chunk_seconds=chunk_seconds,identity_hash=assets.identity_hash) if session is not None: old_fingerprint=str(session.get("model_fingerprint","")) @@ -112,7 +114,7 @@ def run_sequence(*,model:Any,clip:Any,video_vae:Any,audio_vae:Any,sampler:Any,si video_context,audio_context,grid_offset=select_context(previous_state,context_frames,include_audio=bool(audio_continuity)); context_before=context_fingerprint(video_context,audio_context) conditioning=prepare_conditioning(base_conditioning,video_context=video_context,audio_context=audio_context,audio_grid_offset=grid_offset,context_frames=context_frames,new_frame_count=shape.total_frames,first_frame_policy=POLICY_REPLACE,preserve_last_frame=True) clip_index=int(previous_state["clip_index"])+1; chunk_plan=make_plan(continuation=True,clip_index=clip_index,total_frames=shape.total_frames,trim_frames=context_frames,width=width,height=height,context_frames=context_frames,state_capacity_frames=largest_context_capacity(shape.net_new_frames),requested_extend_seconds=chunk_seconds,debug=debug) - chunk_model=clone_model_for_chunk(patched_model,strict=bool(strict_compatibility),debug=bool(debug),chunk_index=clip_index,context_frames=context_frames if previous_state is not None else None) + chunk_model=clone_model_for_chunk(model,strict=bool(strict_compatibility),debug=bool(debug),chunk_index=clip_index,context_frames=context_frames if previous_state is not None else None) sampled=sample_chunk(model=chunk_model,conditioning=conditioning,latent=latent,sampler=sampler,sigmas=sigmas,seed=seed,enable_preview=bool(enable_preview)) if context_before is not None and video_context is not None: assert_context_unchanged(video_context,audio_context,context_before) entry=make_chunk_entry(latent=sampled,plan=chunk_plan,prompt=prompt,prompt_hash=prompt_hash_value,seed=seed,context_frames=context_frames,motion_score=motion_score,reused=False); previous_state=entry_to_state(entry); entries.append(entry); retained_frames+=int(chunk_plan["net_frames"]) @@ -128,7 +130,7 @@ def run_sequence(*,model:Any,clip:Any,video_vae:Any,audio_vae:Any,sampler:Any,si settings={"continuity":continuity,"audio_continuity":bool(audio_continuity),"exact_total_duration":bool(exact_total_duration),"prompt_mode":plan["mode"],"base_seed":int(base_seed),"reroll_nonce":int(reroll_nonce),"diagnostics_mode":diagnostics_mode,"initial_state_source":initial_state is not None} new_session=make_session(chunks=entries,width=width,height=height,chunk_seconds=chunk_seconds,identity_hash=assets.identity_hash,model_fingerprint_value=current_model_fingerprint,parent_session_id=parent_id,reroll_from_chunk=int(reroll_from_chunk),settings=settings) decoded_gib=float(images.shape[0])*float(width)*float(height)*3.0*4.0/(1024.0**3) - report_lines=[f"H3 Continuum V2 {PACKAGE_VERSION}",prompt_plan_report(plan),f"Seam correction: {seam_correction}.",accelerator_summary(patched_model),"Execution: conditioning precomputed; call-local MODEL clone per chunk; decode deferred until sampling completed.",*reuse_notes] + report_lines=[f"H3 Continuum V2 {PACKAGE_VERSION}",prompt_plan_report(plan),f"Seam correction: {seam_correction}.",accelerators,"Execution: conditioning precomputed; single call-local MODEL clone per chunk; decode deferred until sampling completed.",*reuse_notes] if diagnostics_mode!=DIAGNOSTICS_OFF: report_lines.extend([f"Decode RAM estimate: {decode_estimate_gib:.2f} GiB including transient headroom"+(f"; available at start {available_ram_gib:.2f} GiB." if available_ram_gib is not None else "."),*sampling_reports,*decode_reports]) if duration_report: report_lines.append(duration_report) report_lines.extend([session_summary(new_session),f"Output: {images.shape[0]} frames ({images.shape[0]/FPS:.3f}s), audio samples={audio['waveform'].shape[-1]}, decoded tensor≈{decoded_gib:.2f} GiB."]) diff --git a/v2/session.py b/v2/session.py index e7747f1..0d07b7a 100644 --- a/v2/session.py +++ b/v2/session.py @@ -33,7 +33,9 @@ def _now_iso() -> str: return datetime.now(timezone.utc).replace(microsecond=0).isoformat() -def model_fingerprint(model: Any) -> str: +def model_fingerprint( + model: Any, *, extra_wrapper_keys: tuple[str, ...] = () +) -> str: base = getattr(model, "model", None) inner = getattr(base, "diffusion_model", None) options = getattr(model, "model_options", {}) or {} @@ -44,6 +46,8 @@ def model_fingerprint(model: Any) -> str: for group in wrappers.values(): if isinstance(group, dict): wrapper_keys.extend(str(key) for key in group) + if extra_wrapper_keys: + wrapper_keys = sorted(set(wrapper_keys).union(map(str, extra_wrapper_keys))) descriptor = { "base": f"{type(base).__module__}.{type(base).__name__}" if base is not None else "missing", "inner": f"{type(inner).__module__}.{type(inner).__name__}" if inner is not None else "missing",