Release H3 Continuum V3.4
This commit is contained in:
@@ -1,488 +1,228 @@
|
||||
# ComfyUI-H3-Continuum 3.3.0
|
||||
<img width="5250" height="2020" alt="workflow (8)" src="https://github.com/user-attachments/assets/58bc27d7-cab4-4f7d-9f58-ed94bdad4a1d" />
|
||||
# ComfyUI-H3-Continuum 3.4.0
|
||||
|
||||
Long-form MiniMax H3 video generation for ComfyUI with restartable chunks, persistent references, Driving Audio, Video Reference, seam correction, and optional Spectrum interoperability.
|
||||
|
||||

|
||||
|
||||
Native long-form MiniMax H3 video and audio continuation for ComfyUI.
|
||||
> V3.4 is the current release candidate. V3.3 legacy nodes remain available for existing workflows.
|
||||
|
||||
**Stable release:** V3.3.0. It adds chunk-local Timeline Video and guarded decoded-video seam correction while preserving the stable sampler, Run Storage contracts, and Spectrum Interop API v1.
|
||||
## Example workflows
|
||||
|
||||
https://github.com/user-attachments/assets/bfa8c683-fc9d-48f1-9cdb-477ca110cdf2
|
||||
- [V3.4 standard](examples/workflows/MiniMax_H3_Continuum_V34.json)
|
||||
- [V3.4 Turbo](examples/workflows/MiniMax_H3_Continuum_V34_turbo.json)
|
||||
|
||||
**H3 Continuum Sampler V3.3** generates 1 to 16 linked H3 chunks, carries raw video/audio latent context between chunks, supports T2VA, I2VA, FL2VA, multi-image Reference conditioning, and optional Timeline Video, and can resume completed chunks from disk.
|
||||
The templates include optional external nodes such as Spectrum, Video Helper Suite, rgthree, EasyUse, and RTX Video Super Resolution. Install, replace, connect, or bypass them according to your installation. Media and acceleration choices remain under user control.
|
||||
|
||||
V3 delegates video and audio decoding to normal **ComfyUI Core VAE Decode nodes**. No model weights or third-party accelerator code are bundled.
|
||||
## What's new in V3.4
|
||||
|
||||
## Sample workflows
|
||||
### Driving Audio
|
||||
|
||||
- [Standard workflow](examples/workflows/MiniMax_H3_Continuum_V33.json) - quality-oriented `res_multistep` profile with Spectrum enabled.
|
||||
- [Turbo workflow](examples/workflows/MiniMax_H3_Continuum_V33_turbo.json) - 8-step `euler` profile with Spectrum bypassed.
|
||||
Driving Audio is the primary V3.4 audio path.
|
||||
|
||||
Download a JSON file and drag it onto the ComfyUI canvas. The examples use optional external custom nodes and local image/audio inputs; replace or bypass unavailable assets for your environment.
|
||||
- Source audio guides each chunk at its absolute sequence position.
|
||||
- The original effective stream is preserved for final output.
|
||||
- Generated audio and Audio Seam processing are bypassed while Driving Audio is active.
|
||||
- Audio is not independently rewritten at every chunk boundary.
|
||||
- Perfect lip sync is not guaranteed; results still depend on H3, source material, prompts, and sampling.
|
||||
|
||||
Turbo weights are available from [LightX2V MiniMax H3 Turbo](https://huggingface.co/lightx2v/Minimax-h3-Turbo/tree/main).
|
||||
### Video Reference
|
||||
|
||||
## Key features
|
||||
Video Reference provides a persistent native H3 video reference across all chunks.
|
||||
|
||||
- Raw paired video/audio latent continuation without decode/re-encode between chunks.
|
||||
- One production sampler UI with normal controls first and infrequent controls marked Advanced.
|
||||
- T2VA, First Frame, Last Frame, First + Last Frame, and Reference conditioning.
|
||||
- Up to three active Reference images across every chunk, compacted in Core connection order.
|
||||
- One native Reference Audio item across every chunk.
|
||||
- Disk-backed Run Storage with automatic resume and compatible Revision reuse.
|
||||
- Fixed, List, and Timeline prompts through one Sequence Prompt input.
|
||||
- ComfyUI Core Video/Audio VAE Decode and a separate Continuum Assemble stage.
|
||||
- Chunk-local Timeline Video conditioning with an efficient 0.4 MP default.
|
||||
- Guarded Audio Seam and Video Seam correction after Core decode.
|
||||
- Optional Spectrum Actual Prefix 2 interoperability.
|
||||
- External SageAttention, LoRA, Turbo, and Spectrum MODEL chains remain composable.
|
||||
- Existing legacy node identifiers remain registered for saved-workflow compatibility.
|
||||
- It guides motion, framing, timing, and appearance.
|
||||
- It is not a direct pixel-copy or deterministic identity-transfer system.
|
||||
- Video and audio are independent. Route a loader's IMAGE output to Video Reference and AUDIO output to Driving Audio.
|
||||
- Source decoding and frame-rate conversion remain loader responsibilities. Continuum does not impose an unnecessary forced 24 fps conversion.
|
||||
|
||||
| Video Reference Size | Purpose |
|
||||
|---|---|
|
||||
| Efficient - 0.4 MP | Default; lower token cost and faster iteration. |
|
||||
| Balanced - 0.6 MP | More detail at a higher compute cost. |
|
||||
| Match Output | Largest reference input; potentially much slower and heavier. |
|
||||
|
||||
### Core-first, permissive execution
|
||||
|
||||
V3.4 removes or relaxes Continuum-only rejection rules that blocked otherwise runnable Core H3 experiments.
|
||||
|
||||
- Prompt problems prefer warnings and fallback behavior instead of stopping.
|
||||
- Unknown upstream/custom wrapper classes are not blanket-rejected.
|
||||
- There is no model allowlist and no automatic model replacement.
|
||||
- The obsolete strict_compatibility control is removed from the V3.4 interface.
|
||||
- Current Core PackedLayout behavior is supported without requiring the removed legacy frame_count argument.
|
||||
|
||||
Hard stops remain only for states that cannot execute safely, including corrupt latent topology, incompatible assembly data, invalid persisted revisions, or unusable mandatory payloads.
|
||||
|
||||
### Cleaner interface
|
||||
|
||||
The public nodes focus on normal production controls. Developer diagnostics and detailed reports are available through ComfyUI settings.
|
||||
|
||||

|
||||
|
||||
### Timeline paths
|
||||
|
||||
Experimental Timeline Video and earlier reference/timeline audio paths are no longer public V3.4 inputs. They are hidden rather than destructively removed. Existing V3.3 workflows can continue through legacy nodes.
|
||||
|
||||
## Installation
|
||||
|
||||
### ComfyUI Manager
|
||||
Search for H3 Continuum or Continuum in ComfyUI Manager, or install manually:
|
||||
|
||||
Open **ComfyUI Manager**, search for **H3 Continuum** or **Continuum**, and select **Install**.
|
||||
|
||||

|
||||
|
||||
Restart ComfyUI after installation, then search for **H3 Continuum Sampler V3.3** in the node menu.
|
||||
|
||||
### Manual installation
|
||||
|
||||
From `ComfyUI/custom_nodes/`:
|
||||
|
||||
```bash
|
||||
~~~bash
|
||||
cd ComfyUI/custom_nodes
|
||||
git clone https://github.com/ukr8b3g-cmyk/ComfyUI-H3-Continuum.git
|
||||
```
|
||||
~~~
|
||||
|
||||
Restart ComfyUI and search for **H3 Continuum Sampler V3.3**.
|
||||
Restart ComfyUI after installation or update.
|
||||
|
||||
For ZIP installation, extract the repository as `ComfyUI-H3-Continuum` under `ComfyUI/custom_nodes/`, then restart ComfyUI.
|
||||
## V3.4 nodes
|
||||
|
||||
Use an up-to-date ComfyUI. ComfyUI v0.32.0 or later is recommended for the current MiniMax H3 Core VAE fixes and memory improvements.
|
||||
### H3 Continuum Sampler V3.4
|
||||
|
||||
## Current and legacy nodes
|
||||
Main inputs:
|
||||
|
||||
V3.3 uses two current workflow nodes:
|
||||
- model, clip, video_vae, sampler, sigmas
|
||||
- Sequence Prompt
|
||||
- first_frame, last_frame
|
||||
- reference_image_1 through reference_image_3
|
||||
- Video Reference
|
||||
- driving_audio, audio_vae
|
||||
|
||||
- **H3 Continuum Sampler V3.3** - unified sampler for normal generation and optional Timeline Video conditioning.
|
||||
- **H3 Continuum Assemble + Seam** - Core-decoded chunk assembly with Audio Seam and guarded Video Seam correction.
|
||||
Outputs:
|
||||
|
||||
The V3.2.4 nodes remain registered only so existing saved workflows continue to load:
|
||||
- video_latents, audio_latents
|
||||
- assembly_plan, status
|
||||
- driving_audio
|
||||
|
||||
- **[Legacy] H3 Continuum Sampler V3.2.4**
|
||||
- **[Legacy] H3 Continuum Assemble V3.2.4**
|
||||
Visible controls:
|
||||
|
||||
New workflows should use the two V3.3 nodes. The legacy Node IDs are preserved, but they do not receive the integrated Timeline Video and current seam workflow surface.
|
||||
- Prompt Format, chunks, chunk_seconds
|
||||
- width, height, continuity
|
||||
- base_seed, control after generate
|
||||
- audio_continuity
|
||||
- Run Storage
|
||||
- Reference Size
|
||||
- Video Reference Size
|
||||
|
||||
No additional Python package is required beyond a working ComfyUI MiniMax H3 environment.
|
||||
### H3 Continuum Assemble + Seam V3.4
|
||||
|
||||
## Standard connection
|
||||
Inputs: images, audio, assembly_plan, and driving_audio.
|
||||
|
||||
```text
|
||||
H3 MODEL Loader
|
||||
-> optional SageAttention
|
||||
-> optional Turbo/LoRA
|
||||
-> optional Spectrum
|
||||
-> H3 Continuum Sampler V3.3.model
|
||||
Controls: Audio Seam and Video Seam.
|
||||
|
||||
MiniMax H3 CLIP/Text Encoder ------> clip
|
||||
MiniMax H3 Video VAE -------------> video_vae
|
||||
Sampler --------------------------> sampler
|
||||
Sigmas ---------------------------> sigmas
|
||||
Text (Multiline) -----------------> Sequence Prompt
|
||||
Optional images ------------------> first_frame / last_frame / reference_image_1/2/3
|
||||
When Driving Audio is connected, preserved source audio is selected for final output and generated audio seam processing is bypassed.
|
||||
|
||||
video_latents --> Core VAE Decode -----------+
|
||||
audio_latents --> Core VAE Decode Audio -----+--> H3 Continuum Assemble V3
|
||||
assembly_plan -------------------------------+
|
||||
## Connection order
|
||||
|
||||
H3 Continuum Assemble V3.images/audio --> Create Video
|
||||
```
|
||||
~~~text
|
||||
H3 model / CLIP / Video VAE / sampler / sigmas
|
||||
|
|
||||
v
|
||||
H3 Continuum Sampler V3.4
|
||||
| | |
|
||||
video_latents | assembly_plan
|
||||
|
|
||||
audio_latents
|
||||
|
||||
The square grid icon on `video_latents` and `audio_latents` means a list of chunk latents. Connect each output to one normal Core decode node; ComfyUI maps the decoder across the chunk list.
|
||||
video_latents -> Core VAE Decode ------- images --+
|
||||
audio_latents -> Core VAE Decode Audio -- audio --+--> H3 Continuum Assemble + Seam V3.4
|
||||
assembly_plan -------------------------------------+
|
||||
driving_audio -------------------------------------+
|
||||
~~~
|
||||
|
||||
The `video_vae` input is used for image conditioning when needed. Continuum does not perform final Video VAE decoding internally.
|
||||
For a source video with sound:
|
||||
|
||||
## Conditioning modes
|
||||
~~~text
|
||||
Video loader IMAGE -> Video Reference
|
||||
Video loader AUDIO -> driving_audio
|
||||
~~~
|
||||
|
||||
The sampler selects the mode from connected image inputs.
|
||||
The supplied templates use Video Helper Suite for this split. Any compatible IMAGE/AUDIO loader may be substituted.
|
||||
|
||||
| Connected images | Mode |
|
||||
| --- | --- |
|
||||
| None | T2VA + Continuation |
|
||||
| First Frame | I2VA + Continuation |
|
||||
| Last Frame | Last-frame-conditioned + Continuation |
|
||||
| First + Last Frame | FL2VA + Continuation |
|
||||
| Reference 1, optionally Reference 2 and 3 | Reference + Continuation |
|
||||
## Prompt formats
|
||||
|
||||
Reference images and First/Last Frame are mutually exclusive. Active Reference inputs are compacted in Core connection order; bypassed sockets are ignored.
|
||||
Timeline:
|
||||
|
||||
Reference Image 1, 1+2, or 1+2+3 may be connected. A single active image is treated as Picture 1, and active images are numbered without gaps. **Ref2VA** is the reference-specialized checkpoint and may provide stronger reference fidelity, while **FL2VA + Reference** is also allowed. Checkpoint classification is diagnostic only; the sampler never switches MODELs automatically. **Strict Compatibility** remains reserved for H3 contracts that are actually unsafe or unsupported.
|
||||
~~~text
|
||||
[0-5s] First section: action, camera movement, and scene progression.
|
||||
[5-10s] Continue from the exact final state of the previous section.
|
||||
[10-15s] Continue naturally without resetting the scene.
|
||||
~~~
|
||||
|
||||
Reference prompts should identify the images explicitly:
|
||||
List prompts use --- separators. Fixed reuses one prompt for every chunk. Auto detects the available structure. If a timeline cannot be parsed, V3.4 reports a warning and uses an applicable fallback rather than rejecting an otherwise runnable workflow.
|
||||
|
||||
```text
|
||||
<Picture 1> defines the subject's face and identity.
|
||||
<Picture 2> defines the subject's clothing, body design, and skateboard.
|
||||
<Picture 3> optionally defines another ordered identity, object, or environment reference.
|
||||
```
|
||||
## Run Storage
|
||||
|
||||
Reference images persist across all Continuum chunks.
|
||||
Run Storage preserves completed chunks and can reuse them after restarting ComfyUI. Regenerate From selects the first chunk to regenerate; earlier compatible chunks remain unchanged.
|
||||
|
||||
### Reference Size
|
||||

|
||||
|
||||
- **Match Output**: preserve aspect ratio and downscale toward the output pixel area.
|
||||
- **Max Identity**: retain a larger identity reference, downscaling only when necessary.
|
||||
- Use a fixed seed for reproducible resume.
|
||||
- Changing persistent models, LoRAs, references, prompts, resolution, or sampling settings can create a new revision.
|
||||
- Unknown upstream node classes no longer cause rejection by themselves; reuse still depends on compatible observable contracts and hashes.
|
||||
|
||||
Both modes avoid stretching and use crop-free Reference preprocessing.
|
||||
## Spectrum interoperability
|
||||
|
||||
## Quick start
|
||||
Spectrum is optional. Current Spectrum releases officially support H3 Continuum Interop API v1.
|
||||
|
||||
1. Add **H3 Continuum Sampler V3.3**.
|
||||
2. Connect MODEL, CLIP, Video VAE, sampler, sigmas, and one Text (Multiline) node.
|
||||
3. Set `Prompt Format = Auto`.
|
||||
4. Start with `chunks = 3`, `chunk_seconds = 5.0`, and `Balanced - 22 frames`.
|
||||
5. Connect the raw latent outputs to Core VAE Decode and Core VAE Decode Audio.
|
||||
6. Connect decoded outputs and `assembly_plan` to **H3 Continuum Assemble + Seam**.
|
||||
7. Connect Assemble images/audio to Create Video.
|
||||
- Chunk 1 uses the normal initial path.
|
||||
- Chunk 2 and later request Actual Prefix 2.
|
||||
- Successful continuation logs accepted H3 Continuum API v1, actual prefix=2.
|
||||
- Unsupported Spectrum versions fall back without a private patch.
|
||||
|
||||
Recommended safe defaults:
|
||||
See [Spectrum v0.2.15 H3 Continuum interoperability](https://github.com/xmarre/ComfyUI-Spectrum-MiniMax-H3#v0215-h3-continuum-interoperability).
|
||||
|
||||
| Setting | Starting value |
|
||||
| --- | --- |
|
||||
| Prompt Format | Auto |
|
||||
| Continuity | Balanced - 22 frames |
|
||||
| Audio Continuity | On |
|
||||
| Audio Seam | Auto |
|
||||
| Regenerate From | Auto |
|
||||
| Run Storage | Off for short tests; Save + Auto Resume for long runs |
|
||||
| SageAttention | Auto when backend compatibility is unknown |
|
||||
Turbo LoRA and Spectrum are not mutually exclusive. Quality and speed remain workflow-dependent.
|
||||
|
||||
## H3 Continuum Sampler UI
|
||||
## Issue and pull-request response
|
||||
|
||||
The V3.3 production node keeps normal generation controls visible and moves developer-only options to ComfyUI Settings. Connection-specific controls appear only when the related input or Run Storage mode is active.
|
||||
### Issue #3
|
||||
|
||||

|
||||
V3.4 removes blanket rejection of unknown upstream/custom class names. Run Storage evaluates the observable generation contract instead of treating an unfamiliar wrapper as automatically incompatible.
|
||||
|
||||
### Main inputs and controls
|
||||
See [Issue #3](https://github.com/ukr8b3g-cmyk/ComfyUI-H3-Continuum/issues/3).
|
||||
|
||||
| Group | Input or control | Purpose |
|
||||
| --- | --- | --- |
|
||||
| H3 pipeline | `model`, `clip`, `video_vae`, `sampler`, `sigmas` | Connect the normal MiniMax H3 MODEL, text encoder, Video VAE, sampler, and sigma schedule. |
|
||||
| Prompt | `Sequence Prompt` | Fixed, List, or Timeline text for the complete Continuum run. |
|
||||
| Keyframes | `first_frame`, `last_frame` | Optional I2VA, last-frame-conditioned, or FL2VA keyframes. |
|
||||
| Reference images | `reference_image_1`, `reference_image_2`, `reference_image_3` | Up to three ordered Picture references. Bypassed inputs are ignored. |
|
||||
| Reference audio | `reference_audio_1`, `reference_audio_vae` | Optional native H3 Reference Audio. Connect the Audio VAE only when Reference Audio is used. |
|
||||
| Duration | `chunks`, `chunk_seconds` | Number of linked chunks and duration of each chunk. Start with `3 x 5.0` seconds. |
|
||||
| Canvas | `width`, `height` | Output dimensions. Use values compatible with the current H3 Core path, normally multiples of 32. |
|
||||
| Continuity | `continuity` | Number of previous raw frames carried into the next chunk. `Balanced - 22 frames` is the practical default. |
|
||||
| Seed | `base_seed` | Base seed used to derive deterministic per-chunk seeds. |
|
||||
### Issue #4
|
||||
|
||||
Reference images and First/Last Frame are mutually exclusive. Reference Audio can be combined with either keyframe conditioning or Reference Image conditioning.
|
||||
V3.4 follows the current Core H3 layout contract and no longer requires the legacy frame_count parameter. The old strict-compatibility toggle is not part of the public interface.
|
||||
|
||||
### Advanced controls
|
||||
See [Issue #4](https://github.com/ukr8b3g-cmyk/ComfyUI-H3-Continuum/issues/4).
|
||||
|
||||
| Control | Default | Use |
|
||||
| --- | --- | --- |
|
||||
| Prompt Format | Auto | Detect Fixed, List, or Timeline prompts. |
|
||||
| Audio Continuity | On | Carry raw audio context between chunks. |
|
||||
| Run Storage | Off | Use `Save + Auto Resume` for long or interruptible runs. |
|
||||
| Run Name | blank | Appears with `Save + Auto Resume`; optionally overrides the automatic storage identity. |
|
||||
| Regenerate From | Auto | Appears with `Save + Auto Resume`; reuse normally or regenerate from a selected saved chunk. |
|
||||
| Variation Nonce | 0 | Appears only for explicit chunk regeneration and creates a deliberate alternate revision. |
|
||||
| Reference Size | Match Output | Appears when a Reference Image is connected. |
|
||||
| Timeline Video Size | Efficient - 0.4 MP | Appears when Timeline Video is connected. |
|
||||
### Pull request #1
|
||||
|
||||
### ComfyUI settings
|
||||
Relevant current-Core compatibility ideas are selectively adapted to V3.4 rather than copying the pull request wholesale. The pull request itself is not described as merged.
|
||||
|
||||
The following package-wide options are available in ComfyUI Settings instead of occupying every sampler and assembler node:
|
||||
See [Pull request #1](https://github.com/ukr8b3g-cmyk/ComfyUI-H3-Continuum/pull/1).
|
||||
|
||||

|
||||
### Pull request #2
|
||||
|
||||
| Setting | Default | Use |
|
||||
| --- | --- | --- |
|
||||
| H3 Continuum: Sampling Preview | On | Show normal sampling preview/progress information. |
|
||||
| H3 Continuum: Developer Diagnostics | Off | Enable developer-oriented diagnostics when troubleshooting. |
|
||||
| H3 Continuum: Detailed Report | Off | Expand sampler and assembly status text. Normal generation does not require it. |
|
||||
The proposed eight-reference expansion and additional high-resolution assembly RAM changes are not included in this update. They remain a separate review item.
|
||||
|
||||
These settings are applied when the workflow is queued. Hidden legacy widgets remain compatible with older saved workflows, while unknown or merged MODEL filenames are not rejected by name.
|
||||
See [Pull request #2](https://github.com/ukr8b3g-cmyk/ComfyUI-H3-Continuum/pull/2).
|
||||
|
||||
### Outputs
|
||||
## Current validation status
|
||||
|
||||
| Output | Connect to |
|
||||
| --- | --- |
|
||||
| `video_latents` | One normal ComfyUI Core Video VAE Decode node. |
|
||||
| `audio_latents` | One normal ComfyUI Core Audio VAE Decode node. |
|
||||
| `assembly_plan` | **H3 Continuum Assemble V3**. |
|
||||
| `status` | Core **Preview as Text** when a readable run report is needed. |
|
||||
V3.4 has been exercised locally with:
|
||||
|
||||
## Sequence Prompt
|
||||
- 1, 2, and 3 chunks
|
||||
- standard and Turbo paths
|
||||
- Spectrum enabled and disabled
|
||||
- I2VA, Reference, and selected FL2VA configurations
|
||||
- Driving Audio with short, exact-length, and longer sources
|
||||
- Video Reference with source audio routed separately
|
||||
- 0.4 MP and 0.6 MP reference sizing
|
||||
- Run Storage reuse and selected-chunk regeneration
|
||||
- Core VAE Decode and final assembly
|
||||
|
||||
### Fixed
|
||||
No OOM was observed in the cited recent local V3.4 checks, including two-chunk 800 x 800 runs. This is not a universal memory guarantee. Model precision, LoRAs, source resolution, optional nodes, GPU, and RAM affect memory use.
|
||||
|
||||
One prompt is reused for all chunks.
|
||||
## Limits
|
||||
|
||||
```text
|
||||
A continuous cinematic shot. Preserve the same subject, camera direction,
|
||||
lighting, movement, voice, soundscape, and music across every continuation.
|
||||
```
|
||||
|
||||
### List
|
||||
|
||||
Separate one prompt per chunk with a line containing three hyphens.
|
||||
|
||||
```text
|
||||
Opening action.
|
||||
---
|
||||
Continue seamlessly from the exact previous movement.
|
||||
---
|
||||
Complete the action naturally without a cut or pose reset.
|
||||
```
|
||||
|
||||
If fewer sections are provided than `chunks`, the last section is repeated.
|
||||
|
||||
### Timeline
|
||||
|
||||
```text
|
||||
[0-5s]
|
||||
Opening action.
|
||||
|
||||
[5-10s]
|
||||
Continuous middle action.
|
||||
|
||||
[10-15s]
|
||||
Natural ending.
|
||||
```
|
||||
|
||||
`Prompt Format = Auto` detects Fixed, List, or Timeline syntax.
|
||||
|
||||
### Local LLM / Qwen prompt generation
|
||||
|
||||
MiniMax provides the official [H3 prompt-writing skill](https://github.com/MiniMax-AI/MiniMax-H3/tree/main/skills/h3-prompt-writing) for T2VA, I2VA, FL2VA, L2VA, and Ref2VA. Give a local agent the complete `h3-prompt-writing` folder, not only `SKILL.md`, because the detailed formats and examples are stored under `references/`.
|
||||
|
||||
The official skill defines the H3 prompt fields and reference-label rules. H3 Continuum adds one routing requirement: divide the visual progression into one prompt section per Continuum chunk. The currently validated workflow uses 5-second chunks.
|
||||
|
||||
For reliable output, use `Prompt Format = Timeline` or `Auto` and require:
|
||||
|
||||
- exactly one non-empty section per configured chunk;
|
||||
- sequential ranges such as `[0-5s]`, `[5-10s]`, and `[10-15s]`;
|
||||
- shared Ref2VA subject and reference definitions only once, before the first timeline section;
|
||||
- only labels for active connected inputs, using compact Core numbering;
|
||||
- a different action, camera, and scene progression for each section;
|
||||
- no explanatory text outside the final H3 prompt.
|
||||
|
||||
If fewer usable sections are produced, Continuum may reuse the previous available section. Matching the requested chunk count is therefore strongly recommended.
|
||||
|
||||
Copy-paste system instruction for a local LLM:
|
||||
|
||||
```text
|
||||
You write production prompts for ComfyUI-H3-Continuum V3.3.
|
||||
|
||||
Use the complete official MiniMax H3 prompt-writing skill as the authority for
|
||||
T2VA, I2VA, FL2VA, L2VA, and Ref2VA field order, syntax, and reference labels.
|
||||
|
||||
The user will provide:
|
||||
- generation mode;
|
||||
- chunk count;
|
||||
- seconds per chunk;
|
||||
- active image, video, and audio inputs;
|
||||
- scene requirements.
|
||||
|
||||
Output rules:
|
||||
1. Output only the final H3 prompt. Do not add explanations or Markdown fences.
|
||||
2. Produce exactly one non-empty Timeline section per requested chunk.
|
||||
3. Use sequential headers calculated from the supplied chunk duration. For three
|
||||
5-second chunks, use exactly [0-5s], [5-10s], and [10-15s].
|
||||
4. Every section must describe the action, camera movement, scene progression,
|
||||
and relevant sound for that interval. Continue naturally from the previous
|
||||
section; do not repeat the first section.
|
||||
5. Mention only active connected references. Number Picture, Video, and Audio
|
||||
labels compactly in ComfyUI Core order. Never mention bypassed, missing, or
|
||||
unconnected inputs.
|
||||
6. For Ref2VA, write subject_definitions, summary, and retention_analysis once
|
||||
before the first Timeline header. In each Timeline section, provide that
|
||||
chunk's detailed_description, overall_soundscape, and non_diegetic_music.
|
||||
7. For T2VA, I2VA, FL2VA, and L2VA, each Timeline section must contain the
|
||||
official integrated_multimodal_description, overall_soundscape, and
|
||||
non_diegetic_music structure for that chunk.
|
||||
8. Preserve requested dialogue, lyrics, and visible text verbatim and in their
|
||||
requested language. Follow the official H3 dialogue and reference syntax.
|
||||
9. Do not claim guaranteed frame-perfect continuity or exact audio copying.
|
||||
```
|
||||
|
||||
## Run Storage and automatic resume
|
||||
|
||||
For long generations:
|
||||
|
||||
```text
|
||||
Run Storage = Save + Auto Resume
|
||||
Regenerate From = Auto
|
||||
reroll_nonce = 0
|
||||
```
|
||||
|
||||

|
||||
|
||||
Each completed raw AV chunk is atomically saved. If generation stops, Queue the same compatible workflow again:
|
||||
|
||||
```text
|
||||
Chunk 1 saved
|
||||
Chunk 2 saved
|
||||
Chunk 3 interrupted
|
||||
|
||||
Next Queue:
|
||||
Chunk 1 reused
|
||||
Chunk 2 reused
|
||||
Chunk 3 generated
|
||||
```
|
||||
|
||||
Run Storage creates deterministic Revisions from the sampling contract. When Run Name and Auto Resume ID are blank, a stable storage ID is derived from the sampler node. It tracks observable inputs including:
|
||||
|
||||
- H3 MODEL loader and ordered MODEL patch/wrapper chain.
|
||||
- LoRA/Turbo enabled state, filename, and strength.
|
||||
- CLIP/Qwen encoder and Video VAE loaders.
|
||||
- Sampler, exact sigma schedule, resolution, duration, continuity, seed, and prompts.
|
||||
- Conditioning mode, First/Last Frame hashes, and Reference image hashes/settings.
|
||||
|
||||
Changing FL2VA to Ref2VA, changing LoRA strength, or changing another tracked setting creates or selects a different Revision. Returning to an earlier compatible configuration reuses its earlier Revision.
|
||||
|
||||
The upstream graph and runtime MODEL/CLIP/VAE weight probes are both retained. Unknown wrappers or incomplete runtime probes fail safe: chunks may still be saved, but automatic reuse is disabled when identity cannot be established reliably.
|
||||
|
||||
Saved chunk files are verified by size and SHA-256 before reuse. Storage schema changes preserve old files but do not automatically reuse unverifiable older chunks.
|
||||
|
||||
Even when all sampling chunks are reused, Core VAE Decode and final assembly run again. A reused 15-second test can therefore still take roughly one minute.
|
||||
|
||||
### Regenerate from a chunk
|
||||
|
||||
Select `Chunk N` in `Regenerate From` to reuse chunks before N and regenerate N onward. Keep `Auto` for normal resume. Chunk regeneration requires `Run Storage = Save + Auto Resume`.
|
||||
|
||||
`reroll_nonce = 0` uses automatic Revision behavior. A positive nonce is retained for compatibility and explicit branch selection.
|
||||
|
||||
## Spectrum-aware continuation
|
||||
|
||||
With [ComfyUI Spectrum MiniMax H3](https://github.com/xmarre/ComfyUI-Spectrum-MiniMax-H3), Continuum requests **Actual Prefix 2** for continuation chunks through Interop API v1. First-class H3 Continuum API v1 interoperability is included upstream in Spectrum v0.2.15 and later, so a current standard Spectrum installation can receive this request without a separate fork. Confirm the acceptance log below during runtime validation.
|
||||
|
||||
```text
|
||||
Chunk 1: normal path
|
||||
Chunk 2+: inherited raw AV Context
|
||||
-> at least two Actual Transformer steps
|
||||
-> normal Spectrum forecast schedule
|
||||
```
|
||||
|
||||
This does not add solver steps. It protects the first evaluations after a new continuation Context before forecasting begins.
|
||||
|
||||
A compatible receiver logs:
|
||||
|
||||
```text
|
||||
Spectrum H3: accepted H3 Continuum API v1, actual prefix=2
|
||||
```
|
||||
|
||||
Actual Prefix 2 is active only when the Spectrum log reports that the Continuum API request was accepted. Older or incompatible Spectrum versions continue without that receiver behavior.
|
||||
|
||||
### Benefits of the upstream Spectrum integration
|
||||
|
||||
The Continuum receiver is included in the official Spectrum release, so users no longer need a Continuum-specific Spectrum fork or local patch. Installing or updating Spectrum through its normal distribution path preserves the integration.
|
||||
|
||||
- **Automatic handshake:** Continuum emits the request only for continuation chunks, and Spectrum applies Actual Prefix 2 when it recognizes Interop API v1.
|
||||
- **Safer chunk transitions:** the first two Transformer evaluations after a new raw AV Context are evaluated normally before Spectrum forecasting resumes. This is intended to reduce forecast instability immediately after a chunk boundary; it does not guarantee that every visible seam or flicker will disappear.
|
||||
- **No additional solver steps:** Actual Prefix 2 changes which existing evaluations are calculated as Actual steps; it does not increase the configured step count.
|
||||
- **Update-friendly operation:** the receiver becomes part of upstream Spectrum, so a normal Spectrum update does not overwrite a separate local interoperability patch.
|
||||
- **Optional composition:** Continuum still works without Spectrum. MODEL selection, Turbo/LoRA, SageAttention, and Spectrum remain external workflow choices, and Continuum does not install or switch them automatically.
|
||||
|
||||
After updating Spectrum, confirm that Chunk 2 and later print the acceptance log shown above. Its presence is the runtime proof that the upstream receiver is active.
|
||||
|
||||
### Standard and Turbo profiles
|
||||
|
||||
```text
|
||||
Quality
|
||||
standard approximately 20-step H3 sampling
|
||||
Spectrum ON
|
||||
Actual Prefix 2 automatic
|
||||
|
||||
Fast
|
||||
Turbo / 8-step profile
|
||||
Spectrum OFF
|
||||
|
||||
Experimental
|
||||
Turbo + Spectrum
|
||||
```
|
||||
|
||||
Turbo + Spectrum is not the recommended default because local testing showed a higher risk of artifacts. Sol-Attn is also not part of the current recommended continuity profile.
|
||||
|
||||
Turbo LoRA files are available from [LightX2V MiniMax H3 Turbo](https://huggingface.co/lightx2v/Minimax-h3-Turbo/tree/main). The ComfyUI-specific FL2VA and Ref2VA variants can be selected in the upstream MODEL/LoRA chain; Continuum does not bundle or switch them automatically.
|
||||
|
||||
## Timeline Video
|
||||
|
||||
`H3 Continuum Sampler V3.3` includes an optional `timeline_video` input. When connected, the source is sliced per 5-second chunk, resized independently from the output, encoded only for chunks that must be generated, and released after use. When it is not connected, the same node operates as the normal Continuum sampler.
|
||||
|
||||
- `Efficient - 0.4 MP` is the default and recommended starting point.
|
||||
- `Match Output` preserves more source detail but can be substantially slower and heavier.
|
||||
- Timeline-video audio is not used; connect Reference Audio separately when needed.
|
||||
- Results still depend on the checkpoint, prompt, source motion, and reference compatibility.
|
||||
|
||||
## Assemble + Seam
|
||||
|
||||
Connect decoded chunk images, audio, and `assembly_plan` to `H3 Continuum Assemble + Seam`.
|
||||
|
||||
- Default: `Audio Seam = Auto`, `Video Seam = Auto`.
|
||||
- Video modes: `Auto`, `Auto 2`, `Analyze Only`, `Off`.
|
||||
- `Auto` applies guarded transient and micro-flash correction without frame deletion.
|
||||
- `Auto 2` additionally enables guarded exposure-ramp correction and remains experimental.
|
||||
- `Analyze Only` reports the boundary classification without changing images or audio.
|
||||
## Why Core Decode is external
|
||||
|
||||
Continuum owns sampling, continuation planning, chunk metadata, context trimming, audio alignment, exact duration, and final assembly.
|
||||
|
||||
ComfyUI Core owns Video/Audio VAE execution, tiled decode behavior, memory strategy, and future H3 VAE improvements.
|
||||
|
||||
This allows the normal Core `VAE Decode` to benefit from ComfyUI updates without copying H3 VAE implementation code into Continuum. Use Core `VAE Decode (Tiled)` when the normal decode path is unsuitable for the selected resolution or available memory.
|
||||
|
||||
Full raw chunks are decoded before Continuum trims repeated context. Trimming latents before decode would change temporal VAE conditioning.
|
||||
|
||||
## Validation summary
|
||||
|
||||
V3.3.0 passed the automated suite and Windows runtime checks for the production sampler, Reference Image/Audio, Run Storage, Timeline Video, Spectrum Actual Prefix 2, Audio Seam Auto, and guarded Video Seam Auto.
|
||||
|
||||
`Auto 2` is included as an experimental fallback when `Auto` does not sufficiently reduce a boundary exposure ramp. Detailed test conditions, results, and remaining limits are recorded in [docs/VALIDATION_RESULTS.md](docs/VALIDATION_RESULTS.md).
|
||||
## Diagnostics
|
||||
|
||||
`status` is a normal STRING output. Connect it to Core **Preview as Text** when you need the sampling and Run Storage report.
|
||||
|
||||
Useful lines include:
|
||||
|
||||
```text
|
||||
Conditioning mode: Reference + Continuation.
|
||||
interop=emitted actual_prefix=2
|
||||
3 reused, 0 generated
|
||||
resume=complete
|
||||
```
|
||||
|
||||
`accelerator markers not detected (informational only)` is not an error. It only means the optional accelerator marker was not observable.
|
||||
|
||||
Keep **Detailed Report** and **Developer Diagnostics** disabled normally. Enable them in ComfyUI Settings only when troubleshooting or collecting a diagnostic report.
|
||||
|
||||
## Compatibility and scope
|
||||
|
||||
- Native MiniMax H3 PackedLayout is preserved.
|
||||
- Video and audio VAEs are not run between sampling chunks.
|
||||
- Accepted raw chunk latents are retained on CPU.
|
||||
- MODEL input is cloned once per chunk; the workflow MODEL is not mutated.
|
||||
- SageAttention, LoRA/Turbo, Spectrum, and model weights remain external.
|
||||
- Unknown H3 layout contracts fail clearly.
|
||||
- Legacy V1/V2/V3 node identifiers remain registered for older workflows but are not the recommended starting point.
|
||||
|
||||
Continuum is independently implemented. It does not copy or bundle source code from Spectrum or [H3 Motion Context](https://github.com/NikoDemon80/ComfyUI-H3-Motion-Context).
|
||||
|
||||
## Development checks
|
||||
|
||||
```bash
|
||||
python -m compileall -q .
|
||||
python -m pytest -q
|
||||
```
|
||||
|
||||
The GitHub Actions workflow runs compileall and the full pytest suite with its explicit test dependencies.
|
||||
- Continuation does not guarantee frame-perfect identity or motion.
|
||||
- Video Reference guides H3; it does not reproduce every source frame.
|
||||
- Driving Audio preserves selected audio, but visual lip synchronization remains model-dependent.
|
||||
- Match Output can be substantially slower than 0.4 MP or 0.6 MP.
|
||||
- Seam correction may keep the native boundary when a proposed correction is not safer.
|
||||
- Optional template nodes must be installed, replaced, or bypassed by the user.
|
||||
|
||||
## License
|
||||
|
||||
MIT. Model files and third-party custom nodes retain their own licenses. MiniMax H3 weights are not redistributed.
|
||||
See [LICENSE](LICENSE).
|
||||
|
||||
+4
-6
@@ -82,7 +82,6 @@ def check_comfy_h3_runtime() -> list[str]:
|
||||
keywords=(
|
||||
"keyframes",
|
||||
"refs",
|
||||
"frame_count",
|
||||
),
|
||||
)
|
||||
for name in missing:
|
||||
@@ -171,7 +170,7 @@ def run_native_layout_self_test() -> str:
|
||||
MARK_AUDIO_OVERHANG: -1.0 / 3.0,
|
||||
}
|
||||
layout = h3_model.PackedLayout(
|
||||
3, 7, 2, 2, 10, keyframes=None, refs=[ref], frame_count=22
|
||||
3, 7, 2, 2, 10, keyframes=None, refs=[ref]
|
||||
)
|
||||
position_id = id(layout.position_ids)
|
||||
payload = {"layout": layout, "keyframes": [], "refs": [ref]}
|
||||
@@ -196,9 +195,6 @@ def ensure_native_h3_base_model(base_model: Any) -> None:
|
||||
inner = getattr(base_model, "diffusion_model", None)
|
||||
if inner is None:
|
||||
raise CompatibilityError("MODEL does not expose diffusion_model")
|
||||
actual = f"{type(inner).__module__}.{type(inner).__name__}"
|
||||
if actual != "comfy.ldm.minimax.model.MiniMaxH3Model":
|
||||
raise CompatibilityError(f"native MiniMaxH3Model required; got {actual}")
|
||||
missing = [
|
||||
name
|
||||
for name in (
|
||||
@@ -212,8 +208,10 @@ def ensure_native_h3_base_model(base_model: Any) -> None:
|
||||
if not hasattr(inner, name)
|
||||
]
|
||||
if missing:
|
||||
actual = f"{type(inner).__module__}.{type(inner).__name__}"
|
||||
raise CompatibilityError(
|
||||
"MiniMaxH3Model contract changed; missing " + ", ".join(missing)
|
||||
f"MiniMax H3 model contract is unavailable on {actual}; missing "
|
||||
+ ", ".join(missing)
|
||||
)
|
||||
|
||||
|
||||
|
||||
@@ -31,10 +31,6 @@ _MODE_LABELS = {
|
||||
def conditioning_mode_from_presence(
|
||||
*, has_first: bool, has_last: bool, has_reference: bool
|
||||
) -> str:
|
||||
if has_reference and (has_first or has_last):
|
||||
raise ValueError(
|
||||
"Reference Images cannot be combined with First Frame or Last Frame"
|
||||
)
|
||||
if has_reference:
|
||||
return CONDITIONING_MODE_REFERENCE
|
||||
if has_first and has_last:
|
||||
|
||||
+6
-3
@@ -66,6 +66,7 @@ def prepare_conditioning(
|
||||
new_frame_count: int,
|
||||
first_frame_policy: str,
|
||||
preserve_last_frame: bool,
|
||||
source_frame_count: int | None = None,
|
||||
):
|
||||
"""Add one native H3 video/video-audio reference as timeline context."""
|
||||
if first_frame_policy not in FIRST_FRAME_POLICIES:
|
||||
@@ -110,6 +111,8 @@ def prepare_conditioning(
|
||||
output = clone_conditioning(conditioning)
|
||||
for _, metadata in output:
|
||||
old_frame_count = metadata.get("minimax_frame_count")
|
||||
if old_frame_count is None:
|
||||
old_frame_count = source_frame_count
|
||||
old_keyframes = [dict(item) for item in (metadata.get("minimax_keyframes") or [])]
|
||||
kept = []
|
||||
for keyframe in old_keyframes:
|
||||
@@ -130,9 +133,9 @@ def prepare_conditioning(
|
||||
elif is_last:
|
||||
continue
|
||||
else:
|
||||
raise ValueError(
|
||||
f"unsupported existing H3 keyframe index {frame_index}; only first/last are safe"
|
||||
)
|
||||
# Core H3 accepts arbitrary guide positions. Preserve them and
|
||||
# let the native keyframe/layout path decide their semantics.
|
||||
kept.append(keyframe)
|
||||
if kept:
|
||||
metadata["minimax_keyframes"] = kept
|
||||
else:
|
||||
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 61 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 433 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 569 KiB |
@@ -0,0 +1,178 @@
|
||||
"""Core-aligned Driving Audio preparation for H3 Continuum V3.4."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
import hashlib
|
||||
import json
|
||||
from typing import Any
|
||||
|
||||
import torch
|
||||
|
||||
|
||||
DRIVING_AUDIO_CONTRACT_VERSION = 1
|
||||
DRIVING_AUDIO_PREPROCESS_VERSION = 1
|
||||
H3_AUDIO_LATENT_FPS = 40
|
||||
|
||||
|
||||
def _tensor_sha256(value: torch.Tensor) -> str:
|
||||
tensor = value.detach().to("cpu").contiguous()
|
||||
digest = hashlib.sha256()
|
||||
digest.update(str(tuple(int(v) for v in tensor.shape)).encode("ascii"))
|
||||
digest.update(str(tensor.dtype).encode("ascii"))
|
||||
digest.update(tensor.numpy().tobytes(order="C"))
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def _canonical_sha256(value: Any) -> str:
|
||||
payload = json.dumps(
|
||||
value,
|
||||
sort_keys=True,
|
||||
separators=(",", ":"),
|
||||
ensure_ascii=True,
|
||||
).encode("utf-8")
|
||||
return hashlib.sha256(payload).hexdigest()
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class DrivingAudioSource:
|
||||
source_audio: dict[str, Any]
|
||||
resampled_waveform: torch.Tensor
|
||||
resolved_vae_sample_rate: int
|
||||
contract: dict[str, Any]
|
||||
combined_hash: str
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class DrivingAudioAssets:
|
||||
source: DrivingAudioSource
|
||||
audio_latent: torch.Tensor
|
||||
|
||||
|
||||
def prepare_driving_audio_source(
|
||||
audio: dict[str, Any] | None,
|
||||
audio_vae: Any,
|
||||
*,
|
||||
target_frames: int,
|
||||
fps: int,
|
||||
) -> DrivingAudioSource | None:
|
||||
"""Prepare only the effective target-duration prefix; never pad a short source."""
|
||||
if audio is None:
|
||||
return None
|
||||
if audio_vae is None:
|
||||
raise ValueError("anchoring guide audio needs the audio_vae input")
|
||||
|
||||
waveform = audio["waveform"]
|
||||
sample_rate = int(audio["sample_rate"])
|
||||
if not isinstance(waveform, torch.Tensor):
|
||||
raise TypeError("AUDIO waveform must be a torch.Tensor")
|
||||
|
||||
target_samples = max(0, round(int(target_frames) * sample_rate / int(fps)))
|
||||
effective_waveform = waveform[..., : min(int(waveform.shape[-1]), target_samples)]
|
||||
source_audio = {
|
||||
"waveform": effective_waveform,
|
||||
"sample_rate": sample_rate,
|
||||
}
|
||||
|
||||
resolved_rate = int(getattr(audio_vae, "audio_sample_rate", 32000))
|
||||
resampled = effective_waveform
|
||||
if sample_rate != resolved_rate:
|
||||
import torchaudio
|
||||
|
||||
resampled = torchaudio.functional.resample(
|
||||
effective_waveform,
|
||||
sample_rate,
|
||||
resolved_rate,
|
||||
)
|
||||
|
||||
contract = {
|
||||
"driving_audio_contract_version": DRIVING_AUDIO_CONTRACT_VERSION,
|
||||
"source_sample_rate": sample_rate,
|
||||
"source_shape": list(effective_waveform.shape),
|
||||
"source_dtype": str(effective_waveform.dtype),
|
||||
"source_sha256": _tensor_sha256(effective_waveform),
|
||||
"resolved_vae_sample_rate": resolved_rate,
|
||||
"resampled_shape": list(resampled.shape),
|
||||
"resampled_dtype": str(resampled.dtype),
|
||||
"resampled_sha256": _tensor_sha256(resampled),
|
||||
"preprocess_version": DRIVING_AUDIO_PREPROCESS_VERSION,
|
||||
}
|
||||
return DrivingAudioSource(
|
||||
source_audio=source_audio,
|
||||
resampled_waveform=resampled,
|
||||
resolved_vae_sample_rate=resolved_rate,
|
||||
contract=contract,
|
||||
combined_hash=_canonical_sha256(contract),
|
||||
)
|
||||
|
||||
|
||||
def encode_driving_audio(
|
||||
source: DrivingAudioSource | None,
|
||||
audio_vae: Any,
|
||||
) -> DrivingAudioAssets | None:
|
||||
if source is None:
|
||||
return None
|
||||
latent = audio_vae.encode(source.resampled_waveform[:1].movedim(1, -1))
|
||||
return DrivingAudioAssets(source=source, audio_latent=latent)
|
||||
|
||||
|
||||
def slice_driving_audio_latent(
|
||||
assets: DrivingAudioAssets | None,
|
||||
*,
|
||||
cumulative_retained_before: int,
|
||||
total_frames: int,
|
||||
trim_frames: int,
|
||||
fps: int,
|
||||
) -> torch.Tensor | None:
|
||||
if assets is None:
|
||||
return None
|
||||
source_start_frame = max(
|
||||
0,
|
||||
int(cumulative_retained_before) - int(trim_frames),
|
||||
)
|
||||
source_end_frame = source_start_frame + int(total_frames)
|
||||
latent_start = round(source_start_frame * H3_AUDIO_LATENT_FPS / int(fps))
|
||||
latent_end = round(source_end_frame * H3_AUDIO_LATENT_FPS / int(fps))
|
||||
latent_length = int(assets.audio_latent.shape[-1])
|
||||
latent_start = min(max(0, latent_start), latent_length)
|
||||
latent_end = min(max(latent_start, latent_end), latent_length)
|
||||
if latent_end <= latent_start:
|
||||
return None
|
||||
return assets.audio_latent[..., latent_start:latent_end].clone()
|
||||
|
||||
|
||||
def combine_driving_audio_identity(
|
||||
visual_identity_hash: str,
|
||||
source: DrivingAudioSource | None,
|
||||
) -> str:
|
||||
if source is None:
|
||||
return visual_identity_hash
|
||||
return _canonical_sha256(
|
||||
{
|
||||
"driving_audio_identity_version": 1,
|
||||
"visual_identity_hash": visual_identity_hash,
|
||||
"driving_audio_hash": source.combined_hash,
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
def attach_driving_audio(
|
||||
conditioning: list,
|
||||
audio_latent: torch.Tensor | None,
|
||||
) -> list:
|
||||
"""Attach a Core-compatible guide-audio keyframe without tokenizer refs."""
|
||||
if audio_latent is None:
|
||||
return conditioning
|
||||
output = []
|
||||
for tokens, metadata in conditioning:
|
||||
copied = dict(metadata)
|
||||
keyframes = list(copied.get("minimax_keyframes", []))
|
||||
keyframes.append(
|
||||
{
|
||||
"resolved_frame_index": 0,
|
||||
"audio_latent": audio_latent,
|
||||
}
|
||||
)
|
||||
copied["minimax_keyframes"] = keyframes
|
||||
output.append([tokens, copied])
|
||||
return output
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
+15
-34
@@ -1,4 +1,4 @@
|
||||
"""Deterministic, fail-closed fingerprints for ComfyUI upstream loader graphs."""
|
||||
"""Deterministic fingerprints for arbitrary ComfyUI upstream loader graphs."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -7,29 +7,7 @@ import json
|
||||
from typing import Any
|
||||
|
||||
|
||||
GRAPH_CONTRACT_VERSION = 2
|
||||
|
||||
# Keep this deliberately explicit. New loader/patch nodes must be audited before
|
||||
# their saved chunks can be resumed automatically.
|
||||
_KNOWN_MODEL_NODES = {
|
||||
"UNETLoader",
|
||||
"LoraLoader",
|
||||
"LoraLoaderModelOnly",
|
||||
"Power Lora Loader (rgthree)",
|
||||
"PathchSageAttentionKJ",
|
||||
"PatchSageAttentionKJ",
|
||||
"SpectrumApplyMiniMaxH3",
|
||||
"MiniMaxH3MemoryEfficientSageAttention",
|
||||
"MiniMaxH3MemoryEfficientSageAttentionPatch",
|
||||
"MiniMaxH3SigmaShift",
|
||||
"ModelSamplingMiniMaxH3",
|
||||
}
|
||||
_KNOWN_CLIP_NODES = {"CLIPLoader", "DualCLIPLoader", "TripleCLIPLoader"}
|
||||
_KNOWN_VAE_NODES = {"VAELoader"}
|
||||
_KNOWN_SHARED_NODES = {
|
||||
"Reroute",
|
||||
"Any Switch (rgthree)",
|
||||
}
|
||||
GRAPH_CONTRACT_VERSION = 3
|
||||
|
||||
CHECKPOINT_REF2VA = "ref2va"
|
||||
CHECKPOINT_FL2VA = "fl2va"
|
||||
@@ -66,15 +44,15 @@ def _literal(value: Any) -> tuple[Any, bool]:
|
||||
return {"type": f"{type(value).__module__}.{type(value).__qualname__}"}, False
|
||||
|
||||
|
||||
def _node(prompt: dict[str, Any], node_id: str, *, allowed: set[str], stack: set[str]) -> tuple[dict[str, Any], bool, list[str]]:
|
||||
def _node(prompt: dict[str, Any], node_id: str, *, stack: set[str]) -> tuple[dict[str, Any], bool, list[str]]:
|
||||
if node_id in stack:
|
||||
return {"node_id": node_id, "error": "cycle"}, False, [f"upstream graph contains a cycle at node {node_id}"]
|
||||
raw = prompt.get(str(node_id))
|
||||
if not isinstance(raw, dict):
|
||||
return {"node_id": node_id, "error": "missing"}, False, [f"upstream node {node_id} is missing"]
|
||||
class_type = str(raw.get("class_type", ""))
|
||||
safe = class_type in allowed or class_type in _KNOWN_SHARED_NODES
|
||||
reasons = [] if safe else [f"unknown upstream node class: {class_type or '<missing>'}"]
|
||||
safe = bool(class_type)
|
||||
reasons = [] if safe else [f"upstream node {node_id} has no class_type"]
|
||||
inputs = raw.get("inputs") or {}
|
||||
if not isinstance(inputs, dict):
|
||||
return {"node_id": node_id, "class_type": class_type, "error": "invalid inputs"}, False, reasons + [f"upstream node {node_id} inputs are invalid"]
|
||||
@@ -91,7 +69,7 @@ def _node(prompt: dict[str, Any], node_id: str, *, allowed: set[str], stack: set
|
||||
and str(value[0]) in prompt
|
||||
):
|
||||
child, child_safe, child_reasons = _node(
|
||||
prompt, str(value[0]), allowed=allowed, stack=next_stack
|
||||
prompt, str(value[0]), stack=next_stack
|
||||
)
|
||||
rendered_inputs[str(name)] = {"output": int(value[1]), "node": child}
|
||||
safe = safe and child_safe
|
||||
@@ -112,6 +90,7 @@ def _node(prompt: dict[str, Any], node_id: str, *, allowed: set[str], stack: set
|
||||
def build_upstream_graph_contract(
|
||||
prompt: Any, unique_id: Any, *, require_video_vae: bool,
|
||||
require_reference_audio_vae: bool = False,
|
||||
require_audio_vae: bool = False,
|
||||
) -> tuple[dict[str, Any], bool, list[str]]:
|
||||
"""Fingerprint MODEL/CLIP/VAE routes feeding the Continuum sampler."""
|
||||
if not isinstance(prompt, dict):
|
||||
@@ -125,13 +104,15 @@ def build_upstream_graph_contract(
|
||||
safe = True
|
||||
reasons: list[str] = []
|
||||
specs = [
|
||||
("model", _KNOWN_MODEL_NODES, True),
|
||||
("clip", _KNOWN_CLIP_NODES, True),
|
||||
("video_vae", _KNOWN_VAE_NODES, bool(require_video_vae)),
|
||||
("model", True),
|
||||
("clip", True),
|
||||
("video_vae", bool(require_video_vae)),
|
||||
]
|
||||
if require_reference_audio_vae:
|
||||
specs.append(("reference_audio_vae", _KNOWN_VAE_NODES, True))
|
||||
for name, allowed, required in specs:
|
||||
specs.append(("reference_audio_vae", True))
|
||||
if require_audio_vae:
|
||||
specs.append(("audio_vae", True))
|
||||
for name, required in specs:
|
||||
if not required:
|
||||
continue
|
||||
link = inputs.get(name)
|
||||
@@ -146,7 +127,7 @@ def build_upstream_graph_contract(
|
||||
reasons.append(f"{name} upstream connection is unavailable")
|
||||
continue
|
||||
route, route_safe, route_reasons = _node(
|
||||
prompt, str(link[0]), allowed=allowed, stack={node_id}
|
||||
prompt, str(link[0]), stack={node_id}
|
||||
)
|
||||
routes[name] = {"output": int(link[1]), "node": route}
|
||||
safe = safe and route_safe
|
||||
|
||||
+39
-10
@@ -36,8 +36,10 @@ def normalize_condition_latents(payload):
|
||||
keyframes=list(payload.get("keyframes") or ()); refs=list(payload.get("refs") or ())
|
||||
video_latents=[item["latent"] for item in keyframes if item.get("latent") is not None]
|
||||
video_latents.extend(item["latent"] for item in refs if item.get("latent") is not None)
|
||||
audio_latents=[item["audio_latent"] for item in keyframes if item.get("audio_latent") is not None]
|
||||
audio_latents.extend(item["audio_latent"] for item in refs if item.get("audio_latent") is not None)
|
||||
payload["cond_video_latents"]=video_latents
|
||||
payload["cond_audio_latents"]=[item["audio_latent"] for item in refs if item.get("audio_latent") is not None]
|
||||
payload["cond_audio_latents"]=audio_latents
|
||||
def _input_device(x):
|
||||
device=getattr(x,"device",None)
|
||||
if isinstance(device,torch.device): return device
|
||||
@@ -104,6 +106,24 @@ def _map_refs_to_segments(layout,refs):
|
||||
result.append(mapping)
|
||||
if cursor!=len(available): raise LayoutCompatibilityError(f"layout contains {len(available)-cursor} unmapped reference segments")
|
||||
return result
|
||||
def _map_keyframes_to_segments(layout,keyframes):
|
||||
available=[(int(a),int(b),str(kind)) for a,b,kind in layout.segments if kind in ("cond","cond_audio")]
|
||||
cursor=0; result=[]
|
||||
def consume(expected):
|
||||
nonlocal cursor
|
||||
if cursor>=len(available): raise LayoutCompatibilityError(f"layout ended while mapping keyframe '{expected}'")
|
||||
a,b,kind=available[cursor]; cursor+=1
|
||||
if kind!=expected: raise LayoutCompatibilityError(f"keyframe layout mismatch: expected '{expected}', found '{kind}'")
|
||||
return a,b
|
||||
for keyframe in keyframes:
|
||||
mapping={"audio":None,"video":None}
|
||||
if keyframe.get("latent") is not None: mapping["video"]=consume("cond")
|
||||
if keyframe.get("audio_latent") is not None: mapping["audio"]=consume("cond_audio")
|
||||
if mapping["video"] is None and mapping["audio"] is None:
|
||||
raise LayoutCompatibilityError("H3 keyframe has neither video nor audio latent")
|
||||
result.append(mapping)
|
||||
if cursor!=len(available): raise LayoutCompatibilityError(f"layout contains {len(available)-cursor} unmapped keyframe segments")
|
||||
return result
|
||||
def _patch_signature(payload,layout):
|
||||
return (
|
||||
tuple((int(kf.get("resolved_frame_index",-1)),int(kf.get(MARK_VIDEO_SLOT,-1))) for kf in (payload.get("keyframes") or ())),
|
||||
@@ -185,22 +205,31 @@ def patch_layout_in_place(payload,*,strict=True,debug=False):
|
||||
original_time=getattr(layout,LAYOUT_ORIGINAL_TIME_ATTR)
|
||||
if not torch.is_tensor(original_time) or original_time.shape!=position_ids[:,0].shape: raise LayoutCompatibilityError("stored layout baseline is incompatible")
|
||||
position_ids[:,0].copy_(original_time)
|
||||
keyframes=list(payload.get("keyframes") or ()); refs=list(payload.get("refs") or ()); cond_segments=[(int(a),int(b)) for a,b,kind in layout.segments if kind=="cond"]
|
||||
if len(cond_segments)!=len(keyframes): raise LayoutCompatibilityError(f"layout has {len(cond_segments)} cond segments for {len(keyframes)} keyframes")
|
||||
keyframes=list(payload.get("keyframes") or ()); refs=list(payload.get("refs") or ()); keyframe_mappings=_map_keyframes_to_segments(layout,keyframes)
|
||||
video_start,video_stop=_single_segment(layout,"video"); _single_segment(layout,"audio")
|
||||
text_segments=[(int(a),int(b)) for a,b,kind in layout.segments if kind=="text"]
|
||||
if len(text_segments)!=1 or text_segments[0][0]!=0: raise LayoutCompatibilityError("unexpected H3 text segment")
|
||||
text_len=text_segments[0][1]; latent_t=int(layout.signature[1]); video_rows=video_stop-video_start
|
||||
if latent_t<=0 or video_rows%latent_t: raise LayoutCompatibilityError(f"video rows {video_rows} are incompatible with latent T={latent_t}")
|
||||
frame_rows=video_rows//latent_t; target_origin=float(position_ids[video_start,0]); reference_shift=target_origin-float(text_len)
|
||||
patched_video_slots=0
|
||||
for (start,stop),keyframe in zip(cond_segments,keyframes):
|
||||
if stop-start!=frame_rows: raise LayoutCompatibilityError("keyframe rows no longer equal one target slot")
|
||||
frame_rows=video_rows//latent_t; target_origin=float(position_ids[video_start,0])
|
||||
patched_video_slots=0; patched_keyframe_audio=0
|
||||
for keyframe,mapping in zip(keyframes,keyframe_mappings):
|
||||
frame_index=int(keyframe.get("resolved_frame_index",0))
|
||||
if frame_index<0: raise LayoutCompatibilityError(f"negative H3 keyframe index {frame_index}")
|
||||
desired_start=target_origin+FRAME_RESCALE*float(frame_index)
|
||||
video_segment=mapping.get("video")
|
||||
if MARK_VIDEO_SLOT in keyframe:
|
||||
if video_segment is None: raise LayoutCompatibilityError("legacy Continuum video slot has no cond rows")
|
||||
start,stop=video_segment
|
||||
if stop-start!=frame_rows: raise LayoutCompatibilityError("keyframe rows no longer equal one target slot")
|
||||
slot=int(keyframe[MARK_VIDEO_SLOT])
|
||||
if not (0<=slot<latent_t): raise LayoutCompatibilityError(f"context slot {slot} is outside target latent T={latent_t}")
|
||||
target_row=video_start+slot*frame_rows; position_ids[start:stop].copy_(position_ids[target_row:target_row+frame_rows]); patched_video_slots+=1
|
||||
elif reference_shift: position_ids[start:stop,0].add_(reference_shift)
|
||||
elif video_segment is not None:
|
||||
start,stop=video_segment; old_start=float(position_ids[start,0]); position_ids[start:stop,0].add_(desired_start-old_start)
|
||||
audio_segment=mapping.get("audio")
|
||||
if audio_segment is not None:
|
||||
start,stop=audio_segment; old_start=float(position_ids[start,0]); position_ids[start:stop,0].add_(desired_start-old_start); patched_keyframe_audio+=1
|
||||
ref_mappings=_map_refs_to_segments(layout,refs); patched_video_refs=0; patched_audio=0
|
||||
for ref,mapping in zip(refs,ref_mappings):
|
||||
if _is_video_context(ref):
|
||||
@@ -222,5 +251,5 @@ def patch_layout_in_place(payload,*,strict=True,debug=False):
|
||||
if strict: raise LayoutCompatibilityError(message)
|
||||
LOG.warning(message)
|
||||
setattr(layout,LAYOUT_SIGNATURE_ATTR,signature)
|
||||
if debug: LOG.info("Continuum layout: video_refs=%d legacy_slots=%d audio_windows=%d origin=%.6f rows=%d pos_id=%s",patched_video_refs,patched_video_slots,patched_audio,target_origin,position_ids.shape[0],id(position_ids))
|
||||
return {"status":"patched","video_contexts":patched_video,"video_refs":patched_video_refs,"legacy_video_slots":patched_video_slots,"audio_windows":patched_audio,"target_origin":target_origin,"position_ids_id":id(position_ids)}
|
||||
if debug: LOG.info("Continuum layout: video_refs=%d legacy_slots=%d keyframe_audio=%d audio_windows=%d origin=%.6f rows=%d pos_id=%s",patched_video_refs,patched_video_slots,patched_keyframe_audio,patched_audio,target_origin,position_ids.shape[0],id(position_ids))
|
||||
return {"status":"patched","video_contexts":patched_video,"video_refs":patched_video_refs,"legacy_video_slots":patched_video_slots,"keyframe_audio_windows":patched_keyframe_audio,"audio_windows":patched_audio,"target_origin":target_origin,"position_ids_id":id(position_ids)}
|
||||
|
||||
+1
-1
@@ -1,4 +1,4 @@
|
||||
[project]
|
||||
name = H3 Continuum
|
||||
version = 3.3.0
|
||||
version = 3.4.0
|
||||
description = Native MiniMax H3 continuation with Core VAE decode, Reference conditioning and crash-safe Run Storage.
|
||||
|
||||
+16
-2
@@ -37,7 +37,14 @@ def _wrapper_factory(*, strict: bool, debug: bool) -> Callable[..., Any]:
|
||||
if not has_continuum and not has_mixed_keyframe_refs:
|
||||
return executor(*args, **kwargs)
|
||||
try:
|
||||
ensure_native_h3_base_model(executor.class_obj)
|
||||
try:
|
||||
ensure_native_h3_base_model(executor.class_obj)
|
||||
except CompatibilityError as exc:
|
||||
logging.getLogger(__name__).warning(
|
||||
"H3 Continuum could not pre-classify this model; continuing with "
|
||||
"ComfyUI runtime validation: %s",
|
||||
exc,
|
||||
)
|
||||
patched_payload = dict(payload)
|
||||
if has_continuum:
|
||||
patch_layout_in_place(patched_payload, strict=True, debug=debug)
|
||||
@@ -77,7 +84,14 @@ def configure_continuum_model(model: Any, *, strict: bool, debug: bool):
|
||||
base_model = getattr(model, "model", None)
|
||||
if base_model is None:
|
||||
raise RuntimeError("MODEL does not expose the ComfyUI BaseModel")
|
||||
ensure_native_h3_base_model(base_model)
|
||||
try:
|
||||
ensure_native_h3_base_model(base_model)
|
||||
except CompatibilityError as exc:
|
||||
logging.getLogger(__name__).warning(
|
||||
"H3 Continuum could not pre-classify this model; installing the wrapper "
|
||||
"and deferring compatibility to ComfyUI runtime validation: %s",
|
||||
exc,
|
||||
)
|
||||
patched = model
|
||||
patched.remove_wrappers_with_key(WrappersMP.APPLY_MODEL, MODEL_WRAPPER_KEY)
|
||||
patched.add_wrapper_with_key(
|
||||
|
||||
@@ -181,6 +181,7 @@ class H3ContinuumJoin:
|
||||
new_frame_count=shape.total_frames,
|
||||
first_frame_policy=first_frame_policy,
|
||||
preserve_last_frame=bool(preserve_last_frame),
|
||||
source_frame_count=current_frames,
|
||||
)
|
||||
clip_index = int(state.get("clip_index", 0)) + 1
|
||||
plan = make_plan(
|
||||
@@ -373,12 +374,20 @@ from .v3.nodes import (
|
||||
NODE_CLASS_MAPPINGS.update(V3_NODE_CLASS_MAPPINGS)
|
||||
NODE_DISPLAY_NAME_MAPPINGS.update(V3_NODE_DISPLAY_NAME_MAPPINGS)
|
||||
|
||||
from .v3.driving_nodes import (
|
||||
NODE_CLASS_MAPPINGS as V34_NODE_CLASS_MAPPINGS,
|
||||
NODE_DISPLAY_NAME_MAPPINGS as V34_NODE_DISPLAY_NAME_MAPPINGS,
|
||||
)
|
||||
|
||||
NODE_CLASS_MAPPINGS.update(V34_NODE_CLASS_MAPPINGS)
|
||||
NODE_DISPLAY_NAME_MAPPINGS.update(V34_NODE_DISPLAY_NAME_MAPPINGS)
|
||||
|
||||
# The stable facade and its pack helpers are the public workflow surface. Keep
|
||||
# every older identifier registered so saved workflows still load, while
|
||||
# ComfyUI's native deprecated-node filter hides legacy building blocks.
|
||||
_primary_node_ids = {
|
||||
"H3ContinuumSamplerTimelineVideo",
|
||||
"H3ContinuumAssembleSeamExperimental",
|
||||
"H3ContinuumSamplerV34",
|
||||
"H3ContinuumAssembleSeamV34",
|
||||
"H3ContinuumClipOverrides",
|
||||
"H3ContinuumResult",
|
||||
}
|
||||
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
[project]
|
||||
name = "comfyui-h3-continuum"
|
||||
version = "3.3.0"
|
||||
version = "3.4.0"
|
||||
description = "Integrated N-chunk continuation for ComfyUI native MiniMax H3 video and audio"
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.10"
|
||||
|
||||
@@ -0,0 +1,257 @@
|
||||
"""Persistent MiniMax H3 video-reference conditioning for the V3.4 sampler."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
import hashlib
|
||||
import json
|
||||
import math
|
||||
from typing import Any
|
||||
|
||||
import torch
|
||||
|
||||
|
||||
REFERENCE_VIDEO_CONTRACT_VERSION = 1
|
||||
REFERENCE_VIDEO_PREPROCESS_VERSION = 2
|
||||
REFERENCE_VIDEO_SIZE_EFFICIENT = "Efficient - 0.4 MP"
|
||||
REFERENCE_VIDEO_SIZE_BALANCED = "Balanced - 0.6 MP"
|
||||
REFERENCE_VIDEO_SIZE_MATCH_OUTPUT = "Match Output"
|
||||
REFERENCE_VIDEO_SIZE_OPTIONS = (
|
||||
REFERENCE_VIDEO_SIZE_EFFICIENT,
|
||||
REFERENCE_VIDEO_SIZE_BALANCED,
|
||||
REFERENCE_VIDEO_SIZE_MATCH_OUTPUT,
|
||||
)
|
||||
_CANVAS_MULTIPLE = 32
|
||||
_EFFICIENT_AREA = 400_000
|
||||
_BALANCED_AREA = 600_000
|
||||
_SOURCE_FPS = 24
|
||||
|
||||
|
||||
class ReferenceVideoError(ValueError):
|
||||
pass
|
||||
|
||||
|
||||
def _canonical_hash(value: Any) -> str:
|
||||
payload = json.dumps(
|
||||
value, sort_keys=True, separators=(",", ":"), ensure_ascii=True
|
||||
).encode("utf-8")
|
||||
return hashlib.sha256(payload).hexdigest()
|
||||
|
||||
|
||||
def _tensor_hash(value: torch.Tensor) -> str:
|
||||
tensor = value.detach().to(device="cpu").contiguous()
|
||||
digest = hashlib.sha256()
|
||||
digest.update(str(tuple(int(v) for v in tensor.shape)).encode("ascii"))
|
||||
digest.update(str(tensor.dtype).encode("ascii"))
|
||||
digest.update(tensor.numpy().tobytes(order="C"))
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def _round_canvas(value: float) -> int:
|
||||
return max(
|
||||
_CANVAS_MULTIPLE,
|
||||
int(round(float(value) / _CANVAS_MULTIPLE)) * _CANVAS_MULTIPLE,
|
||||
)
|
||||
|
||||
|
||||
def _resolved_size(
|
||||
source_width: int,
|
||||
source_height: int,
|
||||
*,
|
||||
output_width: int,
|
||||
output_height: int,
|
||||
size_mode: str,
|
||||
) -> tuple[int, int]:
|
||||
if size_mode == REFERENCE_VIDEO_SIZE_MATCH_OUTPUT:
|
||||
target_area = int(output_width) * int(output_height)
|
||||
elif size_mode == REFERENCE_VIDEO_SIZE_BALANCED:
|
||||
target_area = _BALANCED_AREA
|
||||
else:
|
||||
target_area = _EFFICIENT_AREA
|
||||
source_area = int(source_width) * int(source_height)
|
||||
scale = min(1.0, math.sqrt(float(target_area) / float(source_area)))
|
||||
target_width = _round_canvas(int(source_width) * scale)
|
||||
target_height = _round_canvas(int(source_height) * scale)
|
||||
if source_width >= _CANVAS_MULTIPLE:
|
||||
target_width = min(
|
||||
target_width,
|
||||
max(_CANVAS_MULTIPLE, (int(source_width) // _CANVAS_MULTIPLE) * _CANVAS_MULTIPLE),
|
||||
)
|
||||
if source_height >= _CANVAS_MULTIPLE:
|
||||
target_height = min(
|
||||
target_height,
|
||||
max(_CANVAS_MULTIPLE, (int(source_height) // _CANVAS_MULTIPLE) * _CANVAS_MULTIPLE),
|
||||
)
|
||||
return target_width, target_height
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ReferenceVideoSource:
|
||||
frames: torch.Tensor
|
||||
source_shape: tuple[int, ...]
|
||||
source_dtype: str
|
||||
source_sha256: str
|
||||
size_mode: str
|
||||
target_width: int
|
||||
target_height: int
|
||||
frame_count: int
|
||||
combined_hash: str
|
||||
|
||||
@property
|
||||
def contract(self) -> dict[str, Any]:
|
||||
return {
|
||||
"reference_video_contract_version": REFERENCE_VIDEO_CONTRACT_VERSION,
|
||||
"source_shape": list(self.source_shape),
|
||||
"source_dtype": self.source_dtype,
|
||||
"source_sha256": self.source_sha256,
|
||||
"source_fps": _SOURCE_FPS,
|
||||
"size_mode": self.size_mode,
|
||||
"target_width": self.target_width,
|
||||
"target_height": self.target_height,
|
||||
"frame_count": self.frame_count,
|
||||
"preprocess_version": REFERENCE_VIDEO_PREPROCESS_VERSION,
|
||||
"combined_hash": self.combined_hash,
|
||||
}
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ReferenceVideoAssets:
|
||||
source: ReferenceVideoSource
|
||||
item: dict[str, Any]
|
||||
block: dict[str, Any]
|
||||
|
||||
|
||||
def prepare_reference_video_source(
|
||||
reference_video_1: Any,
|
||||
*,
|
||||
target_frames: int,
|
||||
output_width: int,
|
||||
output_height: int,
|
||||
size_mode: str = REFERENCE_VIDEO_SIZE_EFFICIENT,
|
||||
) -> ReferenceVideoSource | None:
|
||||
if reference_video_1 is None:
|
||||
return None
|
||||
if not torch.is_tensor(reference_video_1) or reference_video_1.ndim != 4:
|
||||
raise ReferenceVideoError(
|
||||
"Video Reference must be a ComfyUI IMAGE batch with shape [T, H, W, C]"
|
||||
)
|
||||
if int(reference_video_1.shape[-1]) < 3:
|
||||
raise ReferenceVideoError("Video Reference must contain RGB channels")
|
||||
frames = reference_video_1[:, :, :, :3].detach().to(
|
||||
device="cpu", dtype=torch.float32
|
||||
).contiguous()
|
||||
if not bool(torch.isfinite(frames).all().item()):
|
||||
raise ReferenceVideoError("Video Reference contains NaN or infinity")
|
||||
source_shape = tuple(int(value) for value in frames.shape)
|
||||
source_sha256 = _tensor_hash(frames)
|
||||
frame_count = min(int(frames.shape[0]), int(target_frames))
|
||||
if frame_count < 5:
|
||||
raise ReferenceVideoError("Video Reference requires at least 5 frames")
|
||||
while frame_count > 5 and frame_count % 17 != 5:
|
||||
frame_count -= 1
|
||||
normalized_size_mode = (
|
||||
str(size_mode)
|
||||
if str(size_mode) in REFERENCE_VIDEO_SIZE_OPTIONS
|
||||
else REFERENCE_VIDEO_SIZE_EFFICIENT
|
||||
)
|
||||
target_width, target_height = _resolved_size(
|
||||
int(frames.shape[2]),
|
||||
int(frames.shape[1]),
|
||||
output_width=int(output_width),
|
||||
output_height=int(output_height),
|
||||
size_mode=normalized_size_mode,
|
||||
)
|
||||
contract_base = {
|
||||
"reference_video_contract_version": REFERENCE_VIDEO_CONTRACT_VERSION,
|
||||
"source_shape": list(source_shape),
|
||||
"source_dtype": str(frames.dtype),
|
||||
"source_sha256": source_sha256,
|
||||
"source_fps": _SOURCE_FPS,
|
||||
"size_mode": normalized_size_mode,
|
||||
"target_width": target_width,
|
||||
"target_height": target_height,
|
||||
"frame_count": frame_count,
|
||||
"preprocess_version": REFERENCE_VIDEO_PREPROCESS_VERSION,
|
||||
}
|
||||
return ReferenceVideoSource(
|
||||
frames=frames,
|
||||
source_shape=source_shape,
|
||||
source_dtype=str(frames.dtype),
|
||||
source_sha256=source_sha256,
|
||||
size_mode=normalized_size_mode,
|
||||
target_width=target_width,
|
||||
target_height=target_height,
|
||||
frame_count=frame_count,
|
||||
combined_hash=_canonical_hash(contract_base),
|
||||
)
|
||||
|
||||
|
||||
def encode_reference_video(
|
||||
video_vae: Any,
|
||||
source: ReferenceVideoSource,
|
||||
) -> ReferenceVideoAssets:
|
||||
frames = source.frames[: source.frame_count]
|
||||
if (
|
||||
int(frames.shape[2]) != source.target_width
|
||||
or int(frames.shape[1]) != source.target_height
|
||||
):
|
||||
try:
|
||||
import comfy.utils
|
||||
except Exception as exc:
|
||||
raise ReferenceVideoError(
|
||||
"ComfyUI Core image resize support is unavailable"
|
||||
) from exc
|
||||
frames = comfy.utils.common_upscale(
|
||||
frames.movedim(-1, 1),
|
||||
source.target_width,
|
||||
source.target_height,
|
||||
"lanczos",
|
||||
"disabled",
|
||||
).movedim(1, -1).contiguous()
|
||||
try:
|
||||
latent = video_vae.encode(frames).detach().to("cpu").contiguous()
|
||||
except Exception as exc:
|
||||
raise ReferenceVideoError("Video Reference VAE Encode failed") from exc
|
||||
qwen_frames = frames[::_SOURCE_FPS // 2].contiguous()
|
||||
item = {
|
||||
"type": "video",
|
||||
"data": qwen_frames,
|
||||
"timestamps": [index / 2.0 for index in range(int(qwen_frames.shape[0]))],
|
||||
}
|
||||
block = {
|
||||
"kind": "video",
|
||||
"latent_t": int(latent.shape[2]),
|
||||
"latent_h": source.target_height // 16,
|
||||
"latent_w": source.target_width // 16,
|
||||
"ref_audio_t": 0,
|
||||
"latent": latent,
|
||||
"audio_latent": None,
|
||||
}
|
||||
return ReferenceVideoAssets(source=source, item=item, block=block)
|
||||
|
||||
|
||||
def combine_reference_video_identity(
|
||||
visual_identity_hash: str,
|
||||
source: ReferenceVideoSource | None,
|
||||
) -> str:
|
||||
if source is None:
|
||||
return str(visual_identity_hash)
|
||||
return _canonical_hash(
|
||||
{
|
||||
"reference_video_identity_version": 1,
|
||||
"visual_identity_hash": str(visual_identity_hash),
|
||||
"reference_video_hash": source.combined_hash,
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
def validate_reference_video_prompts(
|
||||
prompts: list[str], source: ReferenceVideoSource | None
|
||||
) -> str:
|
||||
if source is None or any("<Video 1>" in str(prompt) for prompt in prompts):
|
||||
return ""
|
||||
return (
|
||||
"H3C-P103 Warning: Video Reference is connected but the prompt contains no "
|
||||
"<Video 1> tag; video still conditions generation, but an explicit tag is "
|
||||
"recommended."
|
||||
)
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
"""Package version and public schema identifiers."""
|
||||
|
||||
PACKAGE_VERSION = "3.3.0"
|
||||
PACKAGE_VERSION = "3.4.0"
|
||||
STATE_SCHEMA_VERSION = 1
|
||||
PLAN_SCHEMA_VERSION = 1
|
||||
PUBLIC_API_VERSION = 3
|
||||
|
||||
Reference in New Issue
Block a user