diff --git a/README.md b/README.md index 7f91632..8f3c5be 100644 --- a/README.md +++ b/README.md @@ -1,230 +1,585 @@ -# ComfyUI-H3-Continuum 3.0.0 +# ComfyUI-H3-Continuum 2.1.7 -Long-form native MiniMax H3 video and audio continuation for ComfyUI. +Native long-form MiniMax H3 video and audio continuation for ComfyUI. -V3 is a latent-first workflow. It samples several manageable H3 chunks, carries raw video and audio context into each continuation chunk, and sends the complete raw chunks to ComfyUI Core VAE nodes for decoding. **H3 Continuum Assemble V3** then trims decoded context, aligns audio, and produces the final video frames and audio. +**H3 Continuum Sampler** generates 1 to 16 H3 chunks as one sequence. It carries the previous raw video and audio latent into the next chunk, keeps accepted chunk latents on CPU, and defers VAE decoding until sampling is complete. -## V3 highlights + +## Standard workflow -- Generate 1 to 16 native H3 chunks as one continuous sequence. -- Preserve raw paired video/audio latent context without decode and re-encode between chunks. -- Keep accepted full chunk latents on CPU instead of accumulating them in active VRAM. -- Use ComfyUI Core **VAE Decode** or **VAE Decode (Tiled)** for video. -- Use ComfyUI Core **VAE Decode Audio** for audio. -- Trim decoded overlap and assemble the final result with cumulative audio sample alignment. -- Optionally accelerate sampling with external SageAttention and Spectrum nodes. -- Automatically request Spectrum Actual Prefix 2 for continuation chunks. -- Preserve V1/V2 node identifiers and State/Session schema compatibility. +[Download the standard **H3 Continuum V2** template bundle](examples/H3_Continuum_V2.zip) (recommended). -## Standard V3 connection +[Download the workflow JSON only](examples/H3_Continuum_V2.json). + +The ZIP contains the workflow, its reference image, and a short setup guide. Extract it, copy `H3_Continuum_V2_reference.webp` into `ComfyUI/input/`, then drag `H3_Continuum_V2.json` onto the ComfyUI canvas. + +It is a practical 3 x 5-second starting point for compatible MiniMax H3 ComfyUI systems. The bundled profile was validated on an RTX 5060 Ti 16GB system with 64GB of system RAM, but the workflow is not limited to that GPU. Adjust `chunks`, `chunk_seconds`, resolution, seed, prompts, and optional accelerators to suit the available hardware. The graph uses the stable facade nodes and keeps session, reroll, and report wiring optional. + +Related projects: + +- [ComfyUI Spectrum MiniMax H3](https://github.com/xmarre/ComfyUI-Spectrum-MiniMax-H3) is the optional sampling accelerator used by the bundled graph. + +### Models referenced by the workflow + +All model URLs stored in the JSON point to the `Comfy-Org/MiniMax-H3` distribution used for ComfyUI workflows. No third-party checkpoint host is referenced: + +- `diffusion_models/minimax/minimax_h3_fl2va_pruned_int8_convrot.safetensors` +- `text_encoders/qwen3vl_32b_minimax_h3_nvfp4_awq.safetensors` +- `vae/minimax_h3_video_vae_fp16.safetensors` +- `vae/minimax_h3_audio_vae_fp32.safetensors` + +No model weights are bundled in this repository. The workflow also contains optional/custom helper nodes from Spectrum, KJNodes (Sage Attention), rgthree, and ComfyUI-Easy-Use. Install those packages or replace/bypass their helper nodes. + +## Why H3 Continuum exists + +**H3 Continuum Sampler is an integrated sampler that includes H3 sampling, deferred Video/Audio VAE decoding, Context trimming, and automatic multi-chunk assembly.** The VAE model weights are not bundled or loaded internally; connect externally loaded `video_vae` and `audio_vae` inputs. + +Continuum was created to make SVI-style long-form continuation practical with native MiniMax H3: generate several manageable chunks, carry motion and audio context forward, and assemble one result without manually rebuilding the graph for every clip. + +The main benefit is not only continuity. A native monolithic 15-second H3 generation can put substantial pressure on VRAM. Continuum samples shorter chunks separately, carries only the required raw AV latent context, stores completed chunk data outside the active sampling path, and defers decoding until sampling is complete. This can make a 3 x 5-second sequence less likely to run out of VRAM on a 16GB GPU, although resolution, model precision, accelerators, offload settings, and available system RAM still determine the actual limit. + +The standard workflow uses the INT8 ConvRot H3 checkpoint, SageAttention CUDA++, and optional Spectrum acceleration as the validated reference profile. On different GPUs or software stacks, start with SageAttention `Auto` or bypass optional accelerators and confirm compatibility locally. Spectrum is not required for continuity and should be disabled for native-trajectory quality comparisons. + +Continuum was developed independently around its integrated multi-chunk workflow, AV timeline validation, and compatibility requirements. Other H3 continuation approaches are discussed only where their workflow and implementation differences are relevant. + +## Recommended quality-first profile + +Continuum targets a useful reduction in long-form generation time while retaining comparatively high quality. It does not select every available accelerator for maximum speed. + +| Component | Bundled RTX 5060 Ti profile | General recommendation | +| --- | --- | --- | +| MiniMax H3 model | `minimax_h3_fl2va_pruned_int8_convrot.safetensors` | Use a compatible official H3 model; the INT8 ConvRot build is the tested 16GB profile. | +| SageAttention | `sageattn_qk_int8_pv_fp8_cuda++`, compile off | Use `Auto` when the installed SageAttention build and GPU-specific backend are unknown. | +| Spectrum | Enabled with the settings below | Optional; disable it for a native-trajectory quality reference. | +| Sol-Attn | Disabled | Currently not recommended in the standard continuity profile. Local A/B runs showed more visible boundary artifacts in some outputs; this is not a universal incompatibility claim. | +| Turbo LoRA | Disabled | Not tested with this Continuum profile. No quality or compatibility claim is made yet. | +| Seam Correction | `Auto` | Safe general starting point. In the current local A/B sample, `Off` produced the least noticeable visual boundary. Compare `Auto` and `Off` first. | + +Recommended Spectrum values used by this profile: ```text -H3 MODEL - -> optional SageAttention - -> optional Spectrum - -> H3 Continuum Sampler V3.model - -CLIP / text encoder ----------------------> Sampler V3.clip -MiniMax H3 Video VAE --------------------> Sampler V3.video_vae -Sampler ---------------------------------> Sampler V3.sampler -Sigmas ----------------------------------> Sampler V3.sigmas -Text (Multiline) ------------------------> Sampler V3.Sequence Prompt -Scaled source image ---------------------> Sampler V3.first_frame - -Sampler V3.video_latents ---------------> Core VAE Decode.samples -MiniMax H3 Video VAE --------------------> Core VAE Decode.vae - -Sampler V3.audio_latents ---------------> Core VAE Decode Audio.samples -MiniMax H3 Audio VAE --------------------> Core VAE Decode Audio.vae - -Core VAE Decode.IMAGE -------------------> Assemble V3.images -Core VAE Decode Audio.AUDIO -------------> Assemble V3.audio -Sampler V3.assembly_plan ----------------> Assemble V3.assembly_plan - -Assemble V3.images ----------------------> Create Video.images -Assemble V3.audio -----------------------> Create Video.audio +blend_weight = 0.50 +warmup_steps = 1 +offline_smoothing_replay = true +bootstrap_first_forecast = true +audio_blend_weight = 0.00 ``` -The grid marker on `video_latents` and `audio_latents` means that each output is a chunk list. ComfyUI automatically maps the Core VAE Decode nodes across that list. You do not need to duplicate the decode nodes for every chunk. +Continuum automatically requests Actual Prefix 2 for continuation chunks. This is not an extra user-facing Spectrum setting and does not increase the configured solver-step count. -`video_vae` remains connected to the sampler because V3 uses it to encode first-frame and optional last-frame conditioning. V3 does not use it for output decoding. +### Seam Correction A/B note -## Main nodes +The bundled workflow keeps `Auto` as a conservative general default. In the current 3 x 5-second local comparison, however, `Off` gave the most natural-looking join. `Auto` may apply a measured color adjustment or short blend that improves its numeric seam score but is not always preferred by visual inspection. -### H3 Continuum Sampler V3 +For a new model, prompt, resolution, or accelerator combination, render the same seed with `Auto` and `Off`. Prefer `Off` when Native Continuity already produces a clean boundary; use `Auto` when it visibly reduces a remaining luminance, color, or audio discontinuity. -Runs H3 sampling and native continuation, then outputs: +## Quick start -| Output | Purpose | -| --- | --- | -| `video_latents` | Full raw video latent chunks for Core VAE Decode | -| `audio_latents` | Full raw audio latent chunks for Core VAE Decode Audio | -| `assembly_plan` | Context trim and exact-duration contract for Assemble V3 | -| `result` | Optional packed State, Session, and sampler report | +1. Install the custom node and restart ComfyUI. +2. Add **H3 Continuum Sampler**. +3. Connect the normal H3 model, CLIP, VAEs, sampler, and sigmas. +4. Connect one **Text (Multiline)** node to **Sequence Prompt**. +5. For image-to-video, connect the source image to **first_frame**. +6. Connect **images** and **audio** to your normal video output node. +7. Start with two 5-second chunks and the settings shown below. -### H3 Continuum Assemble V3 - -Receives all externally decoded chunks and: - -- validates chunk counts against the assembly plan; -- trims the decoded continuation context; -- aligns audio using cumulative frame boundaries; -- optionally applies **Audio Seam Auto** without changing video frames; -- adjusts the final output to the requested exact duration. - -### H3 Continuum Advanced V3 - -Optional pack for Session/State input, reroll controls, diagnostics, audio continuity, strict compatibility, and sampling preview control. Leave it disconnected for normal defaults. - -### H3 Continuum Clip Overrides - -Optional sparse per-clip prompt replacement. Unspecified clips continue using the main Sequence Prompt. - -### H3 Continuum Result - -Optional helper that exposes `last_state`, `session`, and the sampler report from the packed `result` output. - -## Recommended starting settings - -| Setting | Starting value | -| --- | --- | -| Prompt Format | `Auto` | -| Chunks | `2` or `3` for the first run | -| Chunk seconds | `5.0` | -| Continuity | `Balanced - 22 frames` | -| Audio Seam | `Off` | -| Exact total duration | `true` | -| Report Detail | `Basic` | -| Resolution | Approximately `0.5 MP` for practical detail | - -Use approximately `0.3 MP` first when testing longer sequences or limited hardware. For a 3:2 landscape source, typical rounded sizes are approximately: - -```text -0.3 MP -> 672 x 448 -0.5 MP -> 864 x 576 -0.6 MP -> 960 x 640 -``` - -Resolution, model precision, ComfyUI offload behavior, and other loaded nodes determine the actual memory limit. - -## Prompt formats - -Connect one **Text (Multiline)** node to **Sequence Prompt**. - -- **Fixed**: one prompt is used for every chunk. -- **List**: separate one prompt per chunk with a line containing `---`. -- **Timeline**: use sections such as `[0-5s]`, `[5-10s]`, or `[Chunk 1]`. -- **Auto**: detects Fixed, List, or Timeline syntax and is the recommended default. - -Example List prompt: - -```text -The skateboarder begins moving through the city in one continuous shot. ---- -She continues the same motion and camera direction without a cut or reset. ---- -She completes the action naturally while preserving identity and environment. -``` - -## Spectrum interoperability - -Spectrum is optional. Continuum does not import Spectrum private runtime code. - -When valid previous context exists, Continuum adds a small read-only API v1 hint requesting at least two Actual Transformer evaluations at the beginning of that continuation chunk. A compatible Spectrum build reports: - -```text -Spectrum H3: accepted H3 Continuum API v1, actual prefix=2 -``` - -Actual Prefix 2 does not add solver steps. It changes the early Actual/Forecast schedule for continuation chunks. Chunk 1 emits no hint, and unsupported Spectrum versions ignore it and continue normally. - -A practical accelerated starting point is: - -```text -Spectrum enabled = true -warmup_steps = 1 -tail_actual_steps = 1 -bootstrap_first_forecast = true -offline_smoothing_replay = true -audio_blend_weight = 0.00 -``` - -SageAttention and Spectrum remain external and optional. Sol-Attn and Turbo LoRA are not required by the current tested V3 profile. - -## Current tested baseline - -The current local V3 baseline was tested on: - -- NVIDIA GeForce RTX 5060 Ti 16GB -- 64GB system RAM -- MiniMax H3 INT8 ConvRot workflow -- SageAttention plus Spectrum -- no Sol-Attn and no Turbo LoRA -- `Balanced - 22 frames` -- `Audio Seam: Off` - -Observed long-form run: - -```text -6 x 5-second chunks -672 x 448 (approximately 0.3 MP) -24 fps / 720 frames / 30.0 seconds -Spectrum Actual Prefix 2 accepted for chunks 2-6 -Total execution time: 12 minutes 48 seconds -``` - -This is one validated local baseline, not a universal performance guarantee. A 60-second run and higher-resolution long-form profiles are still pending validation. - -Development and A/B notes are available in [docs/AB_TEST_LOG_JA.md](docs/AB_TEST_LOG_JA.md). +Most users only need **H3 Continuum Sampler**. The helper nodes are optional. ## Installation -From the ComfyUI `custom_nodes` directory: +### Git installation -```bash +From the ComfyUI custom_nodes directory: + +~~~bash git clone https://github.com/ukr8b3g-cmyk/ComfyUI-H3-Continuum.git -``` +~~~ -Restart ComfyUI and search for: +Restart ComfyUI, then search for **H3 Continuum Sampler**. + +For Stability Matrix, the destination is normally similar to: + +~~~text +D:\StabilityMatrix\Data\Packages\\custom_nodes\ +~~~ + +### ZIP or Windows installer + +1. Extract the repository as **ComfyUI-H3-Continuum**. +2. Place it in **ComfyUI/custom_nodes/**. +3. Run **install_windows.bat** if you want the included Windows installer to copy and validate the package. +4. Restart ComfyUI. + +If an older **ComfyUI-H3-Continuum-Join** folder exists, remove or rename it before starting ComfyUI. Do not load both folders at the same time. + +No additional pip packages are required beyond the normal ComfyUI MiniMax H3 environment. + +## Standard connection + +~~~text +H3 MODEL loader + | + +-> optional SageAttention + +-> optional Sol-Attn + +-> optional LoRA + +-> optional Spectrum + | + v +H3 Continuum Sampler.model + +CLIP loader -----------------------> clip +MiniMax H3 Video VAE -------------> video_vae +MiniMax H3 Audio VAE -------------> audio_vae +Sampler --------------------------> sampler +Sigmas ---------------------------> sigmas +Text (Multiline) -----------------> Sequence Prompt +Load Image -----------------------> first_frame + +H3 Continuum Sampler.images ------> Create Video.images +H3 Continuum Sampler.audio -------> Create Video.audio +~~~ + +The accelerator chain is optional. A typical accelerated order is: + +~~~text +H3 MODEL -> SageAttention -> Sol-Attn -> LoRA -> Spectrum +~~~ + +When troubleshooting, start with fewer accelerators and add them back one at a time. + +### Main inputs + +| Input | Connect | +| --- | --- | +| **model** | The final MODEL from the normal H3 model or accelerator chain | +| **clip** | MiniMax H3 text encoder / CLIP output | +| **video_vae** | MiniMax H3 Video VAE | +| **audio_vae** | MiniMax H3 Audio VAE | +| **sampler** | The sampler used by the normal H3 workflow | +| **sigmas** | The corresponding sigma schedule | +| **Sequence Prompt** | One external Text (Multiline) STRING | +| **first_frame** | Source image for image-to-video | +| **prompt_overrides** | Optional H3 Continuum Clip Overrides pack | +| **advanced** | Optional H3 Continuum Advanced pack | + +### Main outputs + +| Output | Use | +| --- | --- | +| **images** | Connect directly to the image/video frames input of the output node | +| **audio** | Connect directly to the audio input of the output node | +| **result** | Optional packed State, Session, and report data | + + +## Sampler and Advanced controls + +### Input image scaling + +![Scale Image to Total Pixels settings](docs/images/h3-continuum-image-scale.png) + +The bundled workflow uses **Scale Image to Total Pixels** before `first_frame`: + +- `upscale_method: area` is appropriate for reducing a larger source image with stable averaging. +- `megapixels: 0.50` preserves the source aspect ratio while targeting approximately 0.5 million pixels. +- `resolution_steps: 32` rounds width and height to model-friendly multiples of 32. +- The bundled portrait is reduced from 848 x 1264 to approximately 576 x 864; the exact rounded size is supplied to the Continuum sampler through the image-size nodes. +- If a 16GB GPU runs out of VRAM, reduce this first to `0.30 MP`, which is approximately 448 x 672 for the same portrait. + +This node changes generation resolution, not the Continuum context length. Increasing megapixels raises VRAM use and generation time for every chunk. + +The screenshot shows the same control at 0.60 MP for illustration. The current bundled JSON uses 0.50 MP. + +![H3 Continuum Sampler controls](docs/images/h3-continuum-sampler.png) + +The main sampler is sufficient for a normal first run: + +- `Sequence Prompt`: one fixed prompt, a `---`-separated list, or a timeline prompt. +- `Prompt Format`: `Auto` is the recommended default. +- `chunks`: number of generated clips. +- `chunk_seconds`: requested duration of each clip. +- `width` / `height`: inherited from the connected image-size path when linked. +- `continuity`: `Balanced - 22 frames` is the general-purpose default. +- `base_seed` and `control after generate`: define the chunk seed sequence. +- `seam_correction`: `Auto` is the safe default; it keeps the legacy video seam when correction does not improve the measured boundary. +- `prompt_overrides`: optional sparse per-clip replacement pack. +- `advanced`: optional settings pack. Leave it disconnected to use normal defaults. + +![H3 Continuum Advanced controls](docs/images/h3-continuum-advanced.png) + +Use **H3 Continuum Advanced** only when the basic node is not enough: + +- `last_frame`: optional final-frame anchor. +- `session`: resumes or reuses a Continuum session. +- `initial_state`: begins from a saved Continuum state. +- `prompt_plan`: accepts an externally prepared prompt plan. +- `audio_continuity`: keeps audio context across chunk boundaries. +- `exact_total_duration`: trims or pads the assembled result to the requested duration. +- `Report Detail`: `Basic` is the standard default. Use Detailed Report only for troubleshooting. +- `reroll_from_chunk`: regenerates from a selected chunk; `0` disables reroll. +- `reroll_nonce`: changes a reroll without replacing the main seed. +- `strict_compatibility`: fails closed when the native H3 layout contract is incompatible. +- `debug`: enables additional diagnostic logging. +- `show_preview`: enables the sampling preview. + +### Wiring reminder + +For a standard graph, connect the patched MODEL path to `model`, the MiniMax H3 text encoder to `clip`, the video/audio VAEs to their matching inputs, the sampler and sigmas, a multiline text node to `Sequence Prompt`, and the scaled source image to `first_frame`. Connect `images` and `audio` to Create Video or Save Video. + +`prompt_overrides`, `advanced`, and the Result helper are optional. Use **H3 Continuum Result** only when you need `last_state`, `session`, or the text report. + +## Recommended first run + +| Setting | Recommended value | +| --- | --- | +| **Prompt Format** | Auto | +| **chunks** | 2 | +| **chunk_seconds** | 5.0 | +| **width / height** | Match the intended H3 generation resolution | +| **continuity** | Balanced - 22 frames | +| **base_seed** | Fixed while comparing settings | +| **control after generate** | fixed for A/B testing | +| **seam_correction** | Auto for normal use | + +This produces approximately 10 seconds. + +For a native-continuity comparison, use the same prompt and seed and compare: + +~~~text +Seam Correction = Off +Seam Correction = Auto +~~~ + +**Auto** is the normal recommendation. It keeps the native seam unless the guarded correction produces a safely measurable improvement. + +### Seam Correction modes + +- **Auto**: Native-first mode. Applies post-decode correction only when the guarded candidate improves the measured seam. +- **Off**: No post-decode seam correction. Use this to inspect Native Continuity directly. +- **Basic**: Runs the V2.1 seam-correction path directly. + +### Continuity profiles + +- **Auto - conservative**: Chooses a conservative context profile from the previous state. +- **Fast - 5 frames**: Lowest overlap and lowest continuity cost. +- **Balanced - 22 frames**: Recommended starting point. +- **Strong - 39 frames**: More context, with higher cost and a greater risk of over-constraining motion. + +## Sequence Prompt + +Connect a single **Text (Multiline)** node to **Sequence Prompt**. + +### Auto + +Auto detects Fixed, List, or Timeline syntax. This is recommended for normal use. + +### Fixed + +One prompt is used for every chunk. + +~~~text +A single continuous cinematic performance. Preserve the same person, +lighting, camera direction, motion, voice, and music across every chunk. +~~~ + +### List + +Separate one prompt per chunk with a line containing three hyphens. + +~~~text +The woman begins singing while the camera slowly pushes in. +--- +She continues the same phrase without a cut, reset, or pose jump. +--- +She resolves the final note naturally and releases a soft breath. +~~~ + +If fewer List sections are supplied than the configured chunk count, the last section is reused. + +### Timeline + +Use time or chunk sections in one multiline prompt. + +~~~text +[0-5s] +The woman begins singing. The piano phrase remains unresolved. + +[5-10s] +Continue from the exact preceding breath, pose, camera movement, and melody. + +[10-15s] +Complete the melody naturally during the final second. +~~~ + +Chunk headers are also accepted: + +~~~text +[Chunk 1] +Opening action. + +[Chunk 2] +Continuous middle action. + +[Chunk 3] +Natural ending. +~~~ + +## How continuation works + +For every continuation chunk, Continuum: + +1. Reads the accepted previous raw H3 video and audio latent. +2. Extracts a phase-valid video and audio context window. +3. Adds that context as fixed reference rows. +4. Keeps new target audio and video rows separate and writable. +5. Clones the workflow MODEL once for the current chunk. +6. Samples the next chunk without decoding and re-encoding the prior result. +7. Stores the accepted full chunk latent on CPU. +8. Decodes and assembles all accepted chunks after sampling finishes. + +The workflow input MODEL is not modified. Each chunk clone has its own Spectrum runtime/history. + +When valid continuation context exists, Continuum emits Spectrum Interop API v1 with an Actual Prefix request of 2. Older or unsupported Spectrum versions ignore the hint and continue normally. + +## Optional helper nodes + +These nodes are not needed for the basic workflow. + +### H3 Continuum Clip Overrides + +Use this only when selected clips need prompts different from the main Sequence Prompt. + +Connect one **Text (Multiline)** to **Override Script** and use explicit sparse sections: + +~~~text +[Clip 2] +Move to a closer shot while preserving the same motion and voice. + +[Clip 5] +Finish the song naturally and hold the final note. +~~~ + +Only Clip 2 and Clip 5 are replaced. All unspecified clips continue using the main Sequence Prompt. + +**[Chunk N]** is also accepted. Duplicate clip numbers and clip numbers outside the configured chunk count are rejected. + +Connect the pack output to **H3 Continuum Sampler.prompt_overrides**. + +### H3 Continuum Advanced + +Use this node for resume data and infrequent controls. Connect its output to **H3 Continuum Sampler.advanced**. + +Important controls include: + +- **last_frame**: Optional final-frame anchor for the final chunk. +- **session**: Continue or reroll from a prior V2 Session. +- **initial_state**: Start from a compatible saved State. Do not connect both Session and Initial State. +- **prompt_plan**: Advanced or compatibility prompt-plan input. +- **audio_continuity**: Continue the prior raw audio latent. +- **exact_total_duration**: Correct the assembled output to the requested total duration. +- **diagnostics**: Basic, Detailed Report, or Off. +- **reroll_from_chunk**: Keep accepted earlier chunks and regenerate from this one-based chunk number. +- **reroll_nonce**: Derive a different reroll seed without changing the base seed. +- **strict_compatibility**: Stop on unknown or unsafe H3 layout contracts. +- **debug**: Enable internal diagnostic logging. +- **show_preview**: Enable the normal sampling preview. + +### H3 Continuum Result + +Connect the Sampler **result** output when you need to expose: + +- **last_state** +- **session** +- **report** + +For ordinary final video generation, this node is unnecessary. + +## Sessions and rerolling + +A Session stores accepted CPU latents, prompts, seeds, dimensions, and continuation metadata. + +Typical advanced use: + +~~~text +H3 Continuum Sampler.result + -> H3 Continuum Result.session + -> H3 Continuum Save Session +~~~ + +Later: + +~~~text +H3 Continuum Load Session + -> H3 Continuum Advanced.session + -> H3 Continuum Sampler.advanced +~~~ + +Resolution changes are rejected rather than silently resizing State or Session data. + +## Diagnostics and debugging + +Use **Basic** diagnostics for normal testing. Use **Detailed Report** when inspecting seam scores, selected cuts, audio correlation, correction fallback, and per-chunk context. + +Set **debug=true** only when deeper runtime information is needed. The single-clone implementation reports object IDs without retaining the objects: + +~~~text +Continuum model lifetime: input= chunk= parent= base= +~~~ + +For the expected single-clone path, every chunk parent should be the same workflow input MODEL: + +~~~text +chunk1.parent -> input MODEL +chunk2.parent -> input MODEL +chunk3.parent -> input MODEL +~~~ + +With a compatible Spectrum version, continuation chunks should also log: + +~~~text +Spectrum H3: accepted H3 Continuum API v1, actual prefix=2 +~~~ + +**0 models unloaded** by itself is normal model-management information. The relevant leak warnings are: + +~~~text +Potential memory leak detected with model MiniMaxH3 +WARNING, memory leak with model MiniMaxH3 +~~~ + +If those warnings occur, compare identical one-chunk and two-chunk runs, then test Spectrum, Sol-Attn, and Sage individually. + +## Special and compatibility nodes + +The following nodes remain available for advanced workflows or saved-workflow compatibility: + +- **H3 Continuum Sampler V2**: Legacy/Core identifier used by older V2 workflows. +- **H3 Continuum Prompt Plan**: Explicit prompt-plan construction and inspection. +- **H3 Continuum Save Session** +- **H3 Continuum Load Session** +- **H3 Continuum Session Info** +- V1 Join, Finish, Assemble, Save State, and Load State nodes. + +New workflows should normally use the compact **H3 Continuum Sampler** facade. + +## Compatibility and safety + +- Native H3 PackedLayout is preserved. +- layout.position_ids identity is preserved when MM-RoPE coordinates are adjusted. +- State and Session schema remain compatible. +- V1 node identifiers remain registered. +- Video and audio VAEs are not run between H3 chunks. +- Full accepted chunk latents are stored on CPU. +- SageAttention, Sol-Attn, LoRA, and Spectrum remain external packages. +- Unknown H3 layout contracts fail clearly instead of producing a subtly incorrect continuation. + + +## Related projects and implementation scope + +| Project | Primary purpose | Workflow model | Main technical difference | +| --- | --- | --- | --- | +| **H3 Continuum** | Generate and assemble 1-16 H3 chunks | One integrated sampler with optional facade packs | Carries raw paired AV latent context, manages prompts/seeds/session/reroll, defers decode, and can apply post-decode seam correction. | +| **H3 Motion Context** | Continue motion and sound from one clip into another | Explicit context, trim, and optional save/load nodes across runs | Pins prior video/audio context into the next native H3 generation and trims the repeated context from the delivered clip. | +| **Spectrum MiniMax H3** | Reduce H3 sampling time | MODEL wrapper before the guider/sampler | Forecasts selected post-transformer hidden features. It is an approximate accelerator, not a clip-continuity system, and may change the generated result. | + +Continuum does not copy, vendor, or bundle source code from Spectrum or H3 Motion Context. Compatibility uses ComfyUI's MODEL/payload behavior. For a supported Spectrum build, Continuum emits a small API v1 metadata hint requesting at least two actual solver steps at the start of a continuation chunk; unsupported builds ignore it and continue normally. + +## Spectrum Actual Prefix 2 interoperability + +Spectrum accelerates sampling by forecasting selected post-transformer hidden features from Actual H3 evaluations. A continuation chunk introduces a new AV Context and Packed Layout. Forecasting immediately after that transition could start before Spectrum has enough Actual features for the new context. + +Continuum avoids that conflict without disabling or modifying Spectrum: ```text -H3 Continuum Sampler V3 -H3 Continuum Assemble V3 -H3 Continuum Advanced V3 +Stable input MODEL + -> call-local MODEL clone for the chunk + -> build and validate the continuation AV Context + -> freeze the branch-specific Packed Layout signature + -> Actual solver step 0 + -> Actual solver step 1 + -> allow the normal Spectrum forecast schedule ``` -No additional Python package is required beyond a working ComfyUI MiniMax H3 environment. Model files and external accelerator nodes are not bundled. +- The input MODEL is not mutated; every chunk receives an isolated clone and Spectrum runtime. +- The read-only `h3_continuum` API v1 hint is emitted only when valid previous Context exists. +- The effective prefix is the larger of Spectrum's own warmup and Continuum's requested prefix, clamped to the configured solver-step count. +- Context rows and generated Target rows remain separate in the native H3 Packed Layout. +- The Packed Layout is checked per logical branch and must remain structurally stable during Native Actual calls. +- Chunk 1 emits no hint and follows the normal Spectrum path. +- Offline Replay reuses the recorded Actual anchors and does not apply the prefix a second time. +- Older or unsupported Spectrum builds ignore the metadata and continue normally. -If an older `ComfyUI-H3-Continuum-Join` folder exists, remove or rename it before starting ComfyUI. Do not load both package folders at the same time. +Actual Prefix 2 does not add two solver steps and does not run the whole sample twice. With a 20-step schedule, the result still uses 20 solver steps. Under the standard Spectrum `warmup_steps=1` profile, it normally replaces one early forecast with one additional Actual Transformer evaluation. If the configured warmup is already two or more, it adds no evaluation. -## Compatibility and legacy workflows +This design gives Spectrum two Context-specific Actual anchors before forecasting. It trades a small amount of the maximum possible acceleration for more stable continuation behavior. -- Existing V1 and V2 node identifiers remain registered as Legacy nodes. -- State, Session, and Prompt Plan schema versions remain unchanged. -- Unknown or unsafe native H3 layout contracts fail clearly. -- The workflow input MODEL is not mutated; each chunk uses a call-local clone. -- Full continuation chunks are decoded before overlap trimming to preserve VAE temporal context. +### Report output and optional Detailed Report -New workflows should use the V3 sampler, ComfyUI Core VAE Decode nodes, and Assemble V3. +![H3 Continuum Detailed Report](docs/images/h3-continuum-detailed-report.png) -## Current limitation and roadmap +The bundled workflow uses `Basic` reporting and leaves `H3 Continuum Result.report` unconnected. No additional report-display extension is required for the standard workflow. -V3.0 keeps raw chunk latents on CPU and delegates decoding to ComfyUI Core, which substantially improves VRAM flexibility. Decoded IMAGE tensors still remain in system RAM until Assemble V3 runs, so system RAM can become the limit for very long or high-resolution output. +The screenshot above is an optional troubleshooting example using `Detailed Report` and an external text display node. That display node is not included in the bundled template. -Planned work: +```text +Continuum report: +interop=emitted actual_prefix=2 consumer=not_observable -- V3.0.x: contract validation, diagnostics, and ownership/lifetime hardening without changing sampling semantics. -- V3.1: disk-backed chunks, chunk-wise decode/release, and streaming assembly for longer sequences. +Spectrum log: +Spectrum H3: accepted H3 Continuum API v1, actual prefix=2 +``` + +`consumer=not_observable` is not an error. Interop API v1 is deliberately a one-way, read-only hint without an acknowledgement side channel. Spectrum's `accepted` log confirms that the hint was consumed. + +### Independently developed scope + +The general idea of continuing one clip from another is not claimed as unique. Continuum's AV timeline validation, branch-local Packed Layout guard, call-local MODEL isolation, Spectrum Actual Prefix 2 contract, integrated multi-chunk orchestration, and seam-correction pipeline were developed independently for this project. No Spectrum or H3 Motion Context source code is copied or bundled. + +### License and external-content boundaries + +- This repository is licensed under [GPL-3.0](LICENSE). +- [Spectrum MiniMax H3](https://github.com/xmarre/ComfyUI-Spectrum-MiniMax-H3/blob/main/LICENSE) is GPL-3.0. +- [H3 Motion Context](https://github.com/NikoDemon80/ComfyUI-H3-Motion-Context/blob/main/LICENSE) is GPL-3.0. +- The separate [ComfyUI workflow templates repository](https://github.com/Comfy-Org/workflow_templates) is MIT-licensed. +- Model files, model cards, and other custom-node packages remain external and keep their own licenses and usage terms. + +The MIT license of the template repository does not change the licenses of model weights or third-party custom nodes. This project does not redistribute MiniMax H3 weights. + +### Tested environment + +- GPU: NVIDIA GeForce RTX 5060 Ti 16GB. +- System memory: 64GB RAM. +- Runtime: Python 3.13.12, PyTorch 2.13.0+cu130, CUDA 13.0. +- SageAttention: 2.2.0+cu130, using KJNodes FP8 CUDA++ with compile disabled. +- Verified baseline environment: 3 x 5-second runs at approximately 448 x 672 (0.3 MP), using the INT8 ConvRot checkpoint and SageAttention CUDA++. +- Bundled template profile: 3 x 5 seconds at 0.5 MP. Treat this higher-resolution profile as a starting point and monitor VRAM on the first run. +- Longer-than-15-second chains are supported by the node design but have not yet been validated in this environment. +- OOM avoidance is not guaranteed; peak usage depends on resolution, precision, model patches, Spectrum settings, ComfyUI offload behavior, and other loaded nodes. ## Development checks Use the same Python environment as ComfyUI: -```bash +~~~bash python -m compileall -q . pytest -q -``` +~~~ + +Current local validation for this change: + +~~~text +74 passed +~~~ + +GPU validation should compare identical prompts, seeds, resolutions, samplers, and sigmas across: + +~~~text +Native +Sage +Sage + Sol-Attn +Sage + Spectrum +Sage + Sol-Attn + Spectrum +~~~ ## License -GPL-3.0-or-later. External model weights and accelerator packages retain their own licenses. +GPL-3.0-or-later. External accelerator source code is not bundled. diff --git a/hardening.py b/hardening.py new file mode 100644 index 0000000..74a6a03 --- /dev/null +++ b/hardening.py @@ -0,0 +1,534 @@ +"""V3.0.x validation and diagnostics that do not alter generation semantics.""" + +from __future__ import annotations + +import inspect +import math +from collections.abc import Mapping +from typing import Any + +import torch + + +_MIB = 1024.0 * 1024.0 + + +def _strict_int(name: str, value: Any, minimum: int | None = None) -> int: + if type(value) is not int: + raise ValueError(f"{name} must be an integer") + if minimum is not None and value < minimum: + raise ValueError(f"{name} must be >= {minimum}") + return value + + +def enrich_assembly_plan(plan: dict[str, Any]) -> dict[str, Any]: + """Add redundant metadata without correcting any existing value.""" + chunks = plan.get("chunks") + if not isinstance(chunks, list) or not chunks: + return plan + + cursor = 0 + for chunk in chunks: + if not isinstance(chunk, dict): + return plan + net_frames = chunk.get("net_frames") + if type(net_frames) is not int: + return plan + chunk.setdefault("frame_start", cursor) + chunk.setdefault("frame_stop", cursor + net_frames) + cursor += net_frames + + plan.setdefault("chunk_count", len(chunks)) + plan.setdefault("natural_frames", cursor) + return plan + + +def validate_assembly_plan_contract(plan: Mapping[str, Any]) -> None: + """Fail closed on contradictions while accepting valid legacy schema-1 plans.""" + if not isinstance(plan, Mapping): + raise ValueError("assembly_plan must be a mapping") + + chunks = plan.get("chunks") + if not isinstance(chunks, list) or not chunks: + raise ValueError("assembly_plan.chunks must be a non-empty list") + + width = _strict_int("assembly_plan.width", plan.get("width"), 1) + height = _strict_int("assembly_plan.height", plan.get("height"), 1) + if width <= 0 or height <= 0: + raise ValueError("assembly_plan dimensions must be positive") + + fps = plan.get("fps") + if not isinstance(fps, (int, float)) or isinstance(fps, bool): + raise ValueError("assembly_plan.fps must be numeric") + fps = float(fps) + if not math.isfinite(fps) or fps <= 0: + raise ValueError("assembly_plan.fps must be finite and positive") + + chunk_seconds = plan.get("chunk_seconds") + if not isinstance(chunk_seconds, (int, float)) or isinstance(chunk_seconds, bool): + raise ValueError("assembly_plan.chunk_seconds must be numeric") + chunk_seconds = float(chunk_seconds) + if not math.isfinite(chunk_seconds) or chunk_seconds <= 0: + raise ValueError("assembly_plan.chunk_seconds must be finite and positive") + + target_frames = _strict_int("assembly_plan.target_frames", plan.get("target_frames"), 1) + expected_target = int(round(len(chunks) * chunk_seconds * fps)) + if target_frames != expected_target: + raise ValueError( + "assembly_plan.target_frames does not match chunks * chunk_seconds * fps" + ) + + if "chunk_count" in plan: + chunk_count = _strict_int("assembly_plan.chunk_count", plan["chunk_count"], 1) + if chunk_count != len(chunks): + raise ValueError("assembly_plan.chunk_count does not match chunks") + + if "preserve_final_frame" in plan and type(plan["preserve_final_frame"]) is not bool: + raise ValueError("assembly_plan.preserve_final_frame must be a boolean") + + natural_frames = 0 + previous_sequence_index: int | None = None + previous_chunk_index: int | None = None + + for position, chunk in enumerate(chunks, start=1): + if not isinstance(chunk, Mapping): + raise ValueError(f"assembly_plan chunk {position} must be a mapping") + + sequence_index = _strict_int( + f"assembly_plan chunk {position}.sequence_index", + chunk.get("sequence_index"), + 0, + ) + chunk_index = _strict_int( + f"assembly_plan chunk {position}.chunk_index", + chunk.get("chunk_index"), + 1, + ) + if previous_sequence_index is not None and sequence_index != previous_sequence_index + 1: + raise ValueError("assembly_plan sequence_index values must be contiguous") + if previous_chunk_index is not None and chunk_index != previous_chunk_index + 1: + raise ValueError("assembly_plan chunk_index values must be contiguous") + previous_sequence_index = sequence_index + previous_chunk_index = chunk_index + + total_frames = _strict_int( + f"assembly_plan chunk {position}.total_frames", + chunk.get("total_frames"), + 5, + ) + trim_frames = _strict_int( + f"assembly_plan chunk {position}.trim_frames", + chunk.get("trim_frames"), + 0, + ) + net_frames = _strict_int( + f"assembly_plan chunk {position}.net_frames", + chunk.get("net_frames"), + 1, + ) + context_frames = _strict_int( + f"assembly_plan chunk {position}.context_frames", + chunk.get("context_frames"), + 0, + ) + expected_video_t = _strict_int( + f"assembly_plan chunk {position}.expected_video_latent_t", + chunk.get("expected_video_latent_t"), + 1, + ) + expected_audio_t = _strict_int( + f"assembly_plan chunk {position}.expected_audio_latent_t", + chunk.get("expected_audio_latent_t"), + 1, + ) + + if (total_frames - 5) % 17 != 0: + raise ValueError( + f"assembly_plan chunk {position}.total_frames is not on the H3 17k+5 grid" + ) + if total_frames - trim_frames != net_frames: + raise ValueError( + f"assembly_plan chunk {position}: total_frames - trim_frames != net_frames" + ) + if trim_frames > 0 and context_frames != trim_frames: + raise ValueError( + f"assembly_plan chunk {position}: continuation context and trim differ" + ) + if expected_video_t <= 0 or expected_audio_t <= 0: + raise ValueError(f"assembly_plan chunk {position}: latent lengths must be positive") + + if "frame_start" in chunk or "frame_stop" in chunk: + frame_start = _strict_int( + f"assembly_plan chunk {position}.frame_start", + chunk.get("frame_start"), + 0, + ) + frame_stop = _strict_int( + f"assembly_plan chunk {position}.frame_stop", + chunk.get("frame_stop"), + 1, + ) + if frame_start != natural_frames or frame_stop != natural_frames + net_frames: + raise ValueError( + f"assembly_plan chunk {position}: natural frame boundary is discontinuous" + ) + + natural_frames += net_frames + + if "natural_frames" in plan: + recorded_natural = _strict_int( + "assembly_plan.natural_frames", plan["natural_frames"], 1 + ) + if recorded_natural != natural_frames: + raise ValueError( + "sum(chunk.net_frames) does not match assembly_plan.natural_frames" + ) + + +def preflight_decoded_chunks( + images: Any, + audio: Any, + plan: Mapping[str, Any], +) -> None: + """Validate all decoded chunks before the assembler allocates output buffers.""" + validate_assembly_plan_contract(plan) + chunks = plan["chunks"] + + if not isinstance(images, (list, tuple)): + raise ValueError("decoded images must be a chunk list") + if not isinstance(audio, (list, tuple)): + raise ValueError("decoded audio must be a chunk list") + if len(images) != len(chunks) or len(audio) != len(chunks): + raise ValueError("decoded chunk counts do not match assembly_plan") + + image_signature: tuple[Any, ...] | None = None + audio_signature: tuple[Any, ...] | None = None + sample_rate: int | None = None + + for position, (image, audio_item, chunk) in enumerate( + zip(images, audio, chunks), start=1 + ): + if not torch.is_tensor(image) or image.ndim != 4: + raise ValueError(f"decoded image chunk {position} must be a 4D tensor") + total_frames = int(chunk["total_frames"]) + if int(image.shape[0]) < total_frames: + raise ValueError( + f"decoded image chunk {position} has fewer frames than assembly_plan" + ) + + current_image_signature = ( + int(image.shape[1]), + int(image.shape[2]), + int(image.shape[3]), + image.dtype, + ) + if image_signature is None: + image_signature = current_image_signature + elif current_image_signature != image_signature: + raise ValueError("decoded image chunk geometry or dtype changed") + + if not isinstance(audio_item, Mapping): + raise ValueError(f"decoded audio chunk {position} must be an AUDIO mapping") + waveform = audio_item.get("waveform") + current_rate = audio_item.get("sample_rate") + if not torch.is_tensor(waveform) or waveform.ndim != 3: + raise ValueError( + f"decoded audio chunk {position}.waveform must be a 3D tensor" + ) + if type(current_rate) is not int or current_rate <= 0: + raise ValueError( + f"decoded audio chunk {position}.sample_rate must be positive" + ) + if int(waveform.shape[-1]) <= 0: + raise ValueError(f"decoded audio chunk {position} is empty") + + current_audio_signature = ( + int(waveform.shape[0]), + int(waveform.shape[1]), + waveform.dtype, + ) + if audio_signature is None: + audio_signature = current_audio_signature + sample_rate = current_rate + elif current_audio_signature != audio_signature: + raise ValueError("decoded audio chunk batch, channels, or dtype changed") + if current_rate != sample_rate: + raise ValueError("decoded audio sample_rate changed between chunks") + + +def _iter_tensors(value: Any, seen_containers: set[int] | None = None): + if seen_containers is None: + seen_containers = set() + + if torch.is_tensor(value): + if bool(getattr(value, "is_nested", False)): + try: + for item in value.unbind(): + yield from _iter_tensors(item, seen_containers) + return + except Exception: + pass + yield value + return + + if isinstance(value, Mapping): + identity = id(value) + if identity in seen_containers: + return + seen_containers.add(identity) + for item in value.values(): + yield from _iter_tensors(item, seen_containers) + return + + if isinstance(value, (list, tuple, set)): + identity = id(value) + if identity in seen_containers: + return + seen_containers.add(identity) + for item in value: + yield from _iter_tensors(item, seen_containers) + + +def _storage_key(tensor: torch.Tensor) -> tuple[Any, ...]: + device = (tensor.device.type, tensor.device.index) + try: + storage = tensor.untyped_storage() + pointer = int(storage.data_ptr()) + byte_count = int(storage.nbytes()) + except Exception: + pointer = int(tensor.data_ptr()) if tensor.numel() else 0 + byte_count = int(tensor.numel() * tensor.element_size()) + if pointer == 0: + return device + ("empty", id(tensor), byte_count) + return device + (pointer, byte_count) + + +def tensor_storage_stats(*values: Any) -> dict[str, Any]: + references = 0 + storages: dict[tuple[Any, ...], tuple[int, str]] = {} + for value in values: + for tensor in _iter_tensors(value): + references += 1 + key = _storage_key(tensor) + storages.setdefault(key, (int(key[-1]), tensor.device.type)) + + cpu_bytes = sum(size for size, device in storages.values() if device == "cpu") + gpu_bytes = sum(size for size, device in storages.values() if device != "cpu") + return { + "references": references, + "unique_storages": len(storages), + "cpu_bytes": cpu_bytes, + "gpu_bytes": gpu_bytes, + "all_cpu": all(device == "cpu" for _, device in storages.values()), + "keys": frozenset(storages), + } + + +def _mib(value: int | None) -> str: + if value is None: + return "n/a" + return f"{value / _MIB:.1f} MiB" + + +def format_memory_snapshot(label: str, *values: Any) -> str: + """Capture non-synchronizing process/system/CUDA counters for Detailed Report.""" + rss: int | None = None + available: int | None = None + try: + import psutil + + rss = int(psutil.Process().memory_info().rss) + available = int(psutil.virtual_memory().available) + except Exception: + pass + + cuda_allocated: int | None = None + cuda_reserved: int | None = None + cuda_free: int | None = None + try: + if torch.cuda.is_available(): + cuda_allocated = int(torch.cuda.memory_allocated()) + cuda_reserved = int(torch.cuda.memory_reserved()) + cuda_free = int(torch.cuda.mem_get_info()[0]) + except Exception: + pass + + stats = tensor_storage_stats(*values) + return ( + f"Memory [{label}]: RSS={_mib(rss)}, system_available={_mib(available)}, " + f"CUDA_allocated={_mib(cuda_allocated)}, CUDA_reserved={_mib(cuda_reserved)}, " + f"CUDA_free={_mib(cuda_free)}, tensor_refs={stats['references']}, " + f"unique_storage={stats['unique_storages']}, " + f"CPU_storage={_mib(stats['cpu_bytes'])}, GPU_storage={_mib(stats['gpu_bytes'])}" + ) + + +def format_latent_storage_audit(entries: Any, session: Any) -> str: + entry_values: list[Any] = [] + if isinstance(entries, (list, tuple)): + for entry in entries: + if isinstance(entry, Mapping): + entry_values.extend((entry.get("video"), entry.get("audio"))) + + session_values: list[Any] = [] + if isinstance(session, Mapping): + session_chunks = session.get("chunks") + if isinstance(session_chunks, (list, tuple)): + for entry in session_chunks: + if isinstance(entry, Mapping): + session_values.extend((entry.get("video"), entry.get("audio"))) + + entry_stats = tensor_storage_stats(entry_values) + session_stats = tensor_storage_stats(session_values) + session_keys = session_stats["keys"] + if not session_keys: + shared = "n/a" + else: + shared = "yes" if session_keys.issubset(entry_stats["keys"]) else "no" + + return ( + "CPU latent storage audit: " + f"entry_refs={entry_stats['references']}, " + f"entry_unique={entry_stats['unique_storages']}, " + f"entry_bytes={_mib(entry_stats['cpu_bytes'])}, " + f"session_shared={shared}, " + f"all_cpu={'yes' if entry_stats['all_cpu'] else 'no'}" + ) + + +def core_video_vae_chunked_io_status(video_vae: Any) -> str: + first_stage_model = getattr(video_vae, "first_stage_model", None) + if first_stage_model is None: + return "Unknown" + capability = getattr(first_stage_model, "comfy_has_chunked_io", None) + if capability is True: + return ( + "Supported" + if callable(getattr(first_stage_model, "decode_output_shape", None)) + else "Unknown" + ) + if capability is None or capability is False: + return "Unsupported" + return "Unknown" + + +def diagnostics_is_full(value: Any) -> bool: + if not isinstance(value, str): + return False + normalized = value.strip().lower() + return "detailed" in normalized or normalized == "full" + + +def _bound_arguments(function: Any, args: tuple[Any, ...], kwargs: dict[str, Any]): + try: + return inspect.signature(function).bind_partial(*args, **kwargs).arguments + except Exception: + return {} + + +def _first_argument(arguments: Mapping[str, Any], *names: str) -> Any: + for name in names: + if name in arguments: + return arguments[name] + return None + + +def _append_report(result: Any, lines: list[str]) -> Any: + if not lines: + return result + suffix = "\n".join(lines) + + if isinstance(result, tuple): + items = list(result) + for index in range(len(items) - 1, -1, -1): + if isinstance(items[index], str): + items[index] = items[index].rstrip() + "\n" + suffix + return tuple(items) + return result + + if isinstance(result, list): + items = list(result) + for index in range(len(items) - 1, -1, -1): + if isinstance(items[index], str): + items[index] = items[index].rstrip() + "\n" + suffix + return items + return result + + if isinstance(result, dict) and isinstance(result.get("report"), str): + updated = dict(result) + updated["report"] = updated["report"].rstrip() + "\n" + suffix + return updated + + if isinstance(result, str): + return result.rstrip() + "\n" + suffix + return result + + +def run_sequence_with_hardening(base: Any, args: tuple[Any, ...], kwargs: dict[str, Any]): + arguments = _bound_arguments(base, args, kwargs) + latent_only = bool(_first_argument(arguments, "latent_only")) + diagnostics = _first_argument(arguments, "diagnostics", "diagnostics_mode") + if not latent_only or not diagnostics_is_full(diagnostics): + return base(*args, **kwargs) + + video_vae = _first_argument(arguments, "video_vae") + lines = [ + f"Core Video VAE chunked I/O: {core_video_vae_chunked_io_status(video_vae)}", + format_memory_snapshot("sequence start"), + ] + result = base(*args, **kwargs) + + entries = result[0] if isinstance(result, (tuple, list)) and len(result) > 0 else None + session = result[2] if isinstance(result, (tuple, list)) and len(result) > 2 else None + lines.append(format_latent_storage_audit(entries, session)) + lines.append(format_memory_snapshot("sequence complete", entries, session)) + return _append_report(result, lines) + + +def assemble_with_hardening(base: Any, args: tuple[Any, ...], kwargs: dict[str, Any]): + arguments = _bound_arguments(base, args, kwargs) + plan = _first_argument(arguments, "assembly_plan", "plan") + images = _first_argument(arguments, "images", "decoded_images") + audio = _first_argument(arguments, "audio", "decoded_audio") + + if plan is None: + for value in arguments.values(): + if isinstance(value, Mapping) and "chunks" in value and "target_frames" in value: + plan = value + break + if images is None: + for value in arguments.values(): + if isinstance(value, (list, tuple)) and value and torch.is_tensor(value[0]): + images = value + break + if audio is None: + for value in arguments.values(): + if ( + isinstance(value, (list, tuple)) + and value + and isinstance(value[0], Mapping) + and "waveform" in value[0] + ): + audio = value + break + + preflight_decoded_chunks(images, audio, plan) + + diagnostics = _first_argument(arguments, "diagnostics", "diagnostics_mode") + lines: list[str] = [] + if diagnostics_is_full(diagnostics): + lines.append(format_memory_snapshot("assemble preflight", images, audio)) + natural_frames = sum(int(chunk["net_frames"]) for chunk in plan["chunks"]) + target_frames = int(plan["target_frames"]) + correction = target_frames - natural_frames + if abs(correction) > 2: + sign = "+" if correction > 0 else "" + lines.append( + f"Warning: Exact Duration requires {sign}{correction} trailing frames; " + "existing tail trim/pad policy is unchanged." + ) + + result = base(*args, **kwargs) + if diagnostics_is_full(diagnostics): + lines.append(format_memory_snapshot("assemble complete", result)) + return _append_report(result, lines) diff --git a/metadata.ini b/metadata.ini index b4d66b2..b0eff50 100644 --- a/metadata.ini +++ b/metadata.ini @@ -1,4 +1,4 @@ [project] name = H3 Continuum -version = 3.0.0 +version = 3.0.1 description = Integrated native MiniMax H3 long-video continuation with stable Vue UI, V1 compatibility and external SageAttention, Sol-Attn and Spectrum composition. diff --git a/pyproject.toml b/pyproject.toml index 35db0cd..0e5c8fa 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,6 +1,6 @@ [project] name = "comfyui-h3-continuum" -version = "3.0.0" +version = "3.0.1" description = "Integrated N-chunk continuation for ComfyUI native MiniMax H3 video and audio" readme = "README.md" requires-python = ">=3.10" diff --git a/tests/test_public_api.py b/tests/test_public_api.py index 264d285..41c53c5 100644 --- a/tests/test_public_api.py +++ b/tests/test_public_api.py @@ -1,8 +1,8 @@ from ComfyUI_H3_Continuum_Join import public_api -def test_public_api_is_v2_ready_and_complete(): - assert public_api.PUBLIC_API_VERSION == 2 +def test_public_api_is_v3_ready_and_complete(): + assert public_api.PUBLIC_API_VERSION == 3 for name in ( "make_extension_shape", "prepare_conditioning", diff --git a/tests/test_v217_stable_facade.py b/tests/test_v217_stable_facade.py index 16e8209..f6a1349 100644 --- a/tests/test_v217_stable_facade.py +++ b/tests/test_v217_stable_facade.py @@ -139,4 +139,4 @@ def test_v217_result_pack_expands_legacy_outputs(): def test_v217_has_no_frontend_display_controller(): assert not (ROOT / "web" / "h3_continuum_v2.js").exists() version_source = (ROOT / "version.py").read_text(encoding="utf-8") - assert 'PACKAGE_VERSION = "2.1.7"' in version_source + assert 'PACKAGE_VERSION = "3.0.1"' in version_source diff --git a/tests/test_v3_hardening.py b/tests/test_v3_hardening.py new file mode 100644 index 0000000..f715639 --- /dev/null +++ b/tests/test_v3_hardening.py @@ -0,0 +1,183 @@ +from copy import deepcopy +import importlib.util +from pathlib import Path +from types import SimpleNamespace + +import pytest +import torch + + +_SPEC = importlib.util.spec_from_file_location( + "h3_continuum_v3_hardening", + Path(__file__).parents[1] / "hardening.py", +) +hardening = importlib.util.module_from_spec(_SPEC) +assert _SPEC.loader is not None +_SPEC.loader.exec_module(hardening) + + +def _plan(): + plan = { + "schema_version": 1, + "fps": 24, + "width": 448, + "height": 672, + "chunk_seconds": 5.0, + "target_frames": 240, + "preserve_final_frame": True, + "chunks": [ + { + "sequence_index": 0, + "chunk_index": 7, + "total_frames": 124, + "trim_frames": 0, + "net_frames": 124, + "context_frames": 5, + "expected_video_latent_t": 32, + "expected_audio_latent_t": 207, + }, + { + "sequence_index": 1, + "chunk_index": 8, + "total_frames": 141, + "trim_frames": 22, + "net_frames": 119, + "context_frames": 22, + "expected_video_latent_t": 36, + "expected_audio_latent_t": 235, + }, + ], + } + return hardening.enrich_assembly_plan(plan) + + +def _decoded(): + images = [ + torch.zeros((124, 8, 8, 3), dtype=torch.float32), + torch.zeros((141, 8, 8, 3), dtype=torch.float32), + ] + audio = [ + {"waveform": torch.zeros((1, 2, 160000)), "sample_rate": 32000}, + {"waveform": torch.zeros((1, 2, 188000)), "sample_rate": 32000}, + ] + return images, audio + + +def test_plan_records_natural_boundaries_without_equating_target(): + plan = _plan() + assert plan["chunk_count"] == 2 + assert plan["natural_frames"] == 243 + assert plan["target_frames"] == 240 + assert plan["chunks"][0]["frame_start"] == 0 + assert plan["chunks"][0]["frame_stop"] == 124 + assert plan["chunks"][1]["frame_start"] == 124 + assert plan["chunks"][1]["frame_stop"] == 243 + hardening.validate_assembly_plan_contract(plan) + + +def test_plan_uses_canonical_latent_length_fields(): + plan = _plan() + chunk = plan["chunks"][0] + assert chunk["expected_video_latent_t"] == 32 + assert chunk["expected_audio_latent_t"] == 207 + + chunk["expected_video_t"] = chunk.pop("expected_video_latent_t") + with pytest.raises(ValueError, match="expected_video_latent_t"): + hardening.validate_assembly_plan_contract(plan) + + +def test_plan_builder_reads_audio_time_from_last_axis(): + source = (Path(__file__).parents[1] / "v3" / "plan.py").read_text( + encoding="utf-8" + ) + assert '"expected_video_latent_t": int(video.shape[2])' in source + assert '"expected_audio_latent_t": int(audio.shape[-1])' in source + + +def test_plan_accepts_resumed_indices_but_rejects_discontinuity(): + plan = _plan() + hardening.validate_assembly_plan_contract(plan) + plan["chunks"][1]["chunk_index"] = 9 + with pytest.raises(ValueError, match="chunk_index"): + hardening.validate_assembly_plan_contract(plan) + + +def test_plan_rejects_invalid_h3_grid_and_arithmetic(): + plan = _plan() + plan["chunks"][1]["total_frames"] = 140 + with pytest.raises(ValueError, match=r"17k\+5"): + hardening.validate_assembly_plan_contract(plan) + + plan = _plan() + plan["chunks"][1]["net_frames"] = 118 + with pytest.raises(ValueError, match="total_frames - trim_frames"): + hardening.validate_assembly_plan_contract(plan) + + +def test_legacy_schema_one_plan_without_redundant_fields_is_accepted(): + plan = _plan() + plan.pop("chunk_count") + plan.pop("natural_frames") + for chunk in plan["chunks"]: + chunk.pop("frame_start") + chunk.pop("frame_stop") + hardening.validate_assembly_plan_contract(plan) + + +def test_decoded_preflight_rejects_geometry_before_assembly(): + plan = _plan() + images, audio = _decoded() + images[1] = torch.zeros((141, 9, 8, 3), dtype=torch.float32) + with pytest.raises(ValueError, match="geometry"): + hardening.preflight_decoded_chunks(images, audio, plan) + + +def test_decoded_preflight_rejects_sample_rate_change(): + plan = _plan() + images, audio = _decoded() + audio[1]["sample_rate"] = 44100 + with pytest.raises(ValueError, match="sample_rate"): + hardening.preflight_decoded_chunks(images, audio, plan) + + +def test_decoded_preflight_accepts_current_two_chunk_contract(): + plan = _plan() + images, audio = _decoded() + hardening.preflight_decoded_chunks(images, audio, plan) + + +def test_storage_stats_deduplicate_shared_storage(): + tensor = torch.zeros((2, 3)) + stats = hardening.tensor_storage_stats(tensor, tensor) + assert stats["references"] == 2 + assert stats["unique_storages"] == 1 + + +def test_session_storage_audit_reports_shared_cpu_storage(): + video = torch.zeros((1, 2, 3)) + audio = torch.zeros((1, 2, 4)) + entry = {"video": video, "audio": audio} + text = hardening.format_latent_storage_audit([entry], {"chunks": [dict(entry)]}) + assert "session_shared=yes" in text + assert "all_cpu=yes" in text + + +def test_core_chunked_io_capability_is_optional(): + supported_model = SimpleNamespace( + comfy_has_chunked_io=True, + decode_output_shape=lambda *args: None, + ) + unsupported_model = SimpleNamespace() + assert ( + hardening.core_video_vae_chunked_io_status( + SimpleNamespace(first_stage_model=supported_model) + ) + == "Supported" + ) + assert ( + hardening.core_video_vae_chunked_io_status( + SimpleNamespace(first_stage_model=unsupported_model) + ) + == "Unsupported" + ) + assert hardening.core_video_vae_chunked_io_status(object()) == "Unknown" diff --git a/v2/sequence.py b/v2/sequence.py index 63127aa..59e178f 100644 --- a/v2/sequence.py +++ b/v2/sequence.py @@ -144,3 +144,11 @@ def run_sequence(*,model:Any,clip:Any,video_vae:Any,audio_vae:Any,sampler:Any,si if duration_report: report_lines.append(duration_report) report_lines.extend([session_summary(new_session),f"Output: {images.shape[0]} frames ({images.shape[0]/FPS:.3f}s), audio samples={audio['waveform'].shape[-1]}, decoded tensor≈{decoded_gib:.2f} GiB."]) return images,audio,last_state,new_session,"\n".join(report_lines) +# V3.0.1 hardening integration: Detailed Report only; generation semantics unchanged. +from ..hardening import run_sequence_with_hardening as _run_sequence_with_hardening + +_run_sequence_v300 = run_sequence + + +def run_sequence(*args, **kwargs): + return _run_sequence_with_hardening(_run_sequence_v300, args, kwargs) \ No newline at end of file diff --git a/v3/assembly.py b/v3/assembly.py index 1aa7b2a..27b6a18 100644 --- a/v3/assembly.py +++ b/v3/assembly.py @@ -267,3 +267,11 @@ class H3ContinuumAssembleV3: audio_seam=str(_singleton(audio_seam, "audio_seam")), diagnostics=str(_singleton(diagnostics, "diagnostics")), ) +# V3.0.1 hardening integration: preflight before allocation; Detailed Report only. +from ..hardening import assemble_with_hardening as _assemble_with_hardening + +_assemble_decoded_chunks_v300 = assemble_decoded_chunks + + +def assemble_decoded_chunks(*args, **kwargs): + return _assemble_with_hardening(_assemble_decoded_chunks_v300, args, kwargs) \ No newline at end of file diff --git a/v3/plan.py b/v3/plan.py index ace92c0..cef89b4 100644 --- a/v3/plan.py +++ b/v3/plan.py @@ -55,7 +55,7 @@ def make_assembly_plan( "net_frames": int(plan["net_frames"]), "context_frames": int(plan["context_frames"]), "expected_video_latent_t": int(video.shape[2]), - "expected_audio_latent_t": int(audio.shape[2]), + "expected_audio_latent_t": int(audio.shape[-1]), } ) @@ -120,3 +120,24 @@ def validate_assembly_plan(plan: Any) -> dict[str, Any]: f"assembly chunk {expected_index} audio latent length is invalid" ) return plan +# V3.0.1 hardening integration: redundant metadata plus fail-closed validation. +from ..hardening import ( + enrich_assembly_plan as _enrich_assembly_plan, + validate_assembly_plan_contract as _validate_assembly_plan_contract, +) + +_make_assembly_plan_v300 = make_assembly_plan +_validate_assembly_plan_v300 = validate_assembly_plan + + +def validate_assembly_plan(plan): + result = _validate_assembly_plan_v300(plan) + _validate_assembly_plan_contract(plan) + return result + + +def make_assembly_plan(*args, **kwargs): + plan = _make_assembly_plan_v300(*args, **kwargs) + _enrich_assembly_plan(plan) + validate_assembly_plan(plan) + return plan diff --git a/version.py b/version.py index d21f522..70e3137 100644 --- a/version.py +++ b/version.py @@ -1,6 +1,6 @@ """Package version and public schema identifiers.""" -PACKAGE_VERSION = "3.0.0" +PACKAGE_VERSION = "3.0.1" STATE_SCHEMA_VERSION = 1 PLAN_SCHEMA_VERSION = 1 PUBLIC_API_VERSION = 3