Files

178 lines
6.7 KiB
Markdown

# Architecture — H3 Continuum Join 2.0
## 1. Design goals
1. Preserve the V1 native H3 continuation behavior.
2. Integrate N chunks without duplicating V1 temporal/layout logic.
3. Keep SageAttention, Sol-Attn, and Spectrum external and independently updatable.
4. Avoid global ComfyUI class mutation.
5. Fail closed when native H3 layout contracts change.
6. Prevent H3/VAE/CLIP model switching between continuation chunks.
7. Keep accepted chunk latents on CPU and bound transient decode memory.
8. Preserve exact long-chain A/V timing.
## 2. Module layers
```text
UI
nodes.py V1 compatibility nodes
v2/nodes.py integrated V2 nodes
V2 orchestration
v2/sequence.py phases, session reuse, reroll, reports
v2/sampling.py ComfyUI-native sampler path
v2/h3_builder.py empty AV latent, prompt and keyframe conditioning
v2/decoder.py deferred decode, preallocated assembly
v2/prompts.py Fixed/List/Timeline parser
v2/session.py in-memory accepted chunk/session model
v2/session_io.py atomic session persistence
v2/motion.py conservative latent motion analysis
v2/seeds.py deterministic per-chunk seed derivation
v2/diagnostics.py lightweight seam measurements
Shared continuation core
continuation.py native H3 video/video_audio reference construction
model_patch.py per-MODEL APPLY_MODEL wrapper
layout_adapter.py in-place MM-RoPE placement and payload normalization
temporal.py H3 17k+5 / 24fps / 40Hz grid
state.py V1 State and Plan schemas
media.py V1 trim/assembly helpers
compatibility.py runtime contract checks
```
V2 calls the shared continuation core. It does not implement a second continuation algorithm.
## 3. Three execution phases
### Phase 1 — conditioning preparation
- Resize first/last images once.
- VAE-encode their keyframe latents once.
- Tokenize and encode each unique Prompt once.
- Clone the externally patched MODEL once and add the Continuum APPLY_MODEL wrapper.
No H3 diffusion sampling begins until this phase completes.
### Phase 2 — H3-only loop
For each chunk:
1. Create native H3 NestedTensor AV latent.
2. Select 5/22/39-frame Context or Auto.
3. Add one native `video` / `video_audio` reference block.
4. Call the standard ComfyUI guider/sampler path.
5. Let Spectrum create and tear down its own run through its public wrappers.
6. Copy the sampled full AV latent to CPU once.
7. Derive the next V1-compatible State from that CPU chunk.
Video and Audio VAE decode are deliberately absent from this loop.
### Phase 3 — deferred decode
- Decode each CPU chunk after all H3 sampling.
- Immediately move decoded tensors to CPU.
- Remove the overlap.
- Copy images into one preallocated final IMAGE tensor.
- Place audio into cumulative 24fps-derived sample ranges.
- Release each raw chunk before decoding the next.
- Apply exact final duration if enabled.
## 4. Native H3 continuation representation
V1 and V2 both represent the previous tail as one reference block:
```text
kind: video or video_audio
latent: native H3 video latent tail
latent_t/h/w: native geometry
audio_latent: optional native H3 audio latent tail
ref_audio_t: audio latent length
_h3cj_* metadata: timeline placement contract
```
The outer APPLY_MODEL wrapper receives the payload built by ComfyUI, moves only marked Continuum reference rows onto the target timeline, and normalizes keyframe/reference latent lists.
## 5. Accelerator composition
### SageAttention
Continuum never replaces the attention kernel. Existing attention overrides remain intact.
### Sol-Attn
Sol-Attn registers H3 target spans by the identity of `PackedLayout.position_ids`. Continuum modifies this tensor in place and never replaces it. Sol conditioning sink and Morton tracking therefore retain their registration.
### Spectrum
Spectrum remains an external OUTER_SAMPLE / PREDICT_NOISE / SAMPLER_SAMPLE / DIFFUSION_MODEL wrapper stack. V2 calls the normal ComfyUI `guider.sample()` once per chunk. Spectrum starts and ends an independent runtime run for each call. Continuum imports no Spectrum private functions.
The recommended model chain is:
```text
UNET → Sage → Sol → LoRA → Spectrum → V2 Sampler
```
V2 then clones this ModelPatcher once; Spectrum's ON_CLONE callback creates an independent runtime binding on the clone.
## 6. State and Session
### V1 State
Contains only the maximum retained continuation tail and temporal metadata. It can start V2 through `initial_state`.
### V2 Session
Contains all accepted raw chunk AV latents plus:
- Prompt and SHA-256 Prompt hash
- per-chunk Seed
- Continuum Plan
- selected Context frames and motion score
- model/accelerator fingerprint
- branch parent/session identifiers
Tensors remain on CPU. Save Session writes safetensors and a JSON mirror atomically. The authoritative metadata is embedded in safetensors.
## 7. Reroll semantics
- `reroll_from_chunk=0`: reuse every matching accepted prefix.
- `reroll_from_chunk=N`: preserve chunks before N and regenerate N onward.
- A Prompt mismatch automatically stops reuse before the changed chunk.
- A first-frame fingerprint mismatch fails explicitly.
- A model/accelerator fingerprint mismatch is reported but accepted chunks remain fixed, because their latent output is already approved.
- The new output creates a child Session; the input Session is not mutated.
## 8. Audio timeline
H3 video is 24fps and audio latent is 40Hz. Decoded PCM is placed using cumulative integer boundaries:
```text
start_sample = round(cumulative_frames / 24 * sample_rate)
stop_sample = round(next_cumulative_frames / 24 * sample_rate)
```
This avoids the one-sample drift that can occur when every 119-frame segment is rounded independently.
## 9. Memory model
Full accepted AV latents are comparatively small and stored on CPU. The final decoded IMAGE is large, so V2:
- estimates final float32 IMAGE + one raw chunk + headroom before sampling;
- refuses unsafe jobs before expensive H3 work;
- allocates the final IMAGE exactly once;
- copies each decoded segment into its final range;
- avoids progressive whole-sequence concatenation.
## 10. Compatibility policy
The package checks:
- native `MiniMaxH3Model` class and required attributes;
- `PackedLayout` capability, including transparent `*args/**kwargs` wrappers;
- H3 temporal grid `(1,4,4,4,4)`;
- APPLY_MODEL and DIFFUSION_MODEL wrapper APIs;
- NestedTensor support;
- real PackedLayout row placement through the installer self-test.
Marked Continuum payload failures never silently fall back to stock reference placement, because that could render a plausible but semantically wrong join.