Release V2.1.4 UI simplification
This commit is contained in:
@@ -0,0 +1,177 @@
|
||||
# Architecture — H3 Continuum Join 2.0
|
||||
|
||||
## 1. Design goals
|
||||
|
||||
1. Preserve the V1 native H3 continuation behavior.
|
||||
2. Integrate N chunks without duplicating V1 temporal/layout logic.
|
||||
3. Keep SageAttention, Sol-Attn, and Spectrum external and independently updatable.
|
||||
4. Avoid global ComfyUI class mutation.
|
||||
5. Fail closed when native H3 layout contracts change.
|
||||
6. Prevent H3/VAE/CLIP model switching between continuation chunks.
|
||||
7. Keep accepted chunk latents on CPU and bound transient decode memory.
|
||||
8. Preserve exact long-chain A/V timing.
|
||||
|
||||
## 2. Module layers
|
||||
|
||||
```text
|
||||
UI
|
||||
nodes.py V1 compatibility nodes
|
||||
v2/nodes.py integrated V2 nodes
|
||||
|
||||
V2 orchestration
|
||||
v2/sequence.py phases, session reuse, reroll, reports
|
||||
v2/sampling.py ComfyUI-native sampler path
|
||||
v2/h3_builder.py empty AV latent, prompt and keyframe conditioning
|
||||
v2/decoder.py deferred decode, preallocated assembly
|
||||
v2/prompts.py Fixed/List/Timeline parser
|
||||
v2/session.py in-memory accepted chunk/session model
|
||||
v2/session_io.py atomic session persistence
|
||||
v2/motion.py conservative latent motion analysis
|
||||
v2/seeds.py deterministic per-chunk seed derivation
|
||||
v2/diagnostics.py lightweight seam measurements
|
||||
|
||||
Shared continuation core
|
||||
continuation.py native H3 video/video_audio reference construction
|
||||
model_patch.py per-MODEL APPLY_MODEL wrapper
|
||||
layout_adapter.py in-place MM-RoPE placement and payload normalization
|
||||
temporal.py H3 17k+5 / 24fps / 40Hz grid
|
||||
state.py V1 State and Plan schemas
|
||||
media.py V1 trim/assembly helpers
|
||||
compatibility.py runtime contract checks
|
||||
```
|
||||
|
||||
V2 calls the shared continuation core. It does not implement a second continuation algorithm.
|
||||
|
||||
## 3. Three execution phases
|
||||
|
||||
### Phase 1 — conditioning preparation
|
||||
|
||||
- Resize first/last images once.
|
||||
- VAE-encode their keyframe latents once.
|
||||
- Tokenize and encode each unique Prompt once.
|
||||
- Clone the externally patched MODEL once and add the Continuum APPLY_MODEL wrapper.
|
||||
|
||||
No H3 diffusion sampling begins until this phase completes.
|
||||
|
||||
### Phase 2 — H3-only loop
|
||||
|
||||
For each chunk:
|
||||
|
||||
1. Create native H3 NestedTensor AV latent.
|
||||
2. Select 5/22/39-frame Context or Auto.
|
||||
3. Add one native `video` / `video_audio` reference block.
|
||||
4. Call the standard ComfyUI guider/sampler path.
|
||||
5. Let Spectrum create and tear down its own run through its public wrappers.
|
||||
6. Copy the sampled full AV latent to CPU once.
|
||||
7. Derive the next V1-compatible State from that CPU chunk.
|
||||
|
||||
Video and Audio VAE decode are deliberately absent from this loop.
|
||||
|
||||
### Phase 3 — deferred decode
|
||||
|
||||
- Decode each CPU chunk after all H3 sampling.
|
||||
- Immediately move decoded tensors to CPU.
|
||||
- Remove the overlap.
|
||||
- Copy images into one preallocated final IMAGE tensor.
|
||||
- Place audio into cumulative 24fps-derived sample ranges.
|
||||
- Release each raw chunk before decoding the next.
|
||||
- Apply exact final duration if enabled.
|
||||
|
||||
## 4. Native H3 continuation representation
|
||||
|
||||
V1 and V2 both represent the previous tail as one reference block:
|
||||
|
||||
```text
|
||||
kind: video or video_audio
|
||||
latent: native H3 video latent tail
|
||||
latent_t/h/w: native geometry
|
||||
audio_latent: optional native H3 audio latent tail
|
||||
ref_audio_t: audio latent length
|
||||
_h3cj_* metadata: timeline placement contract
|
||||
```
|
||||
|
||||
The outer APPLY_MODEL wrapper receives the payload built by ComfyUI, moves only marked Continuum reference rows onto the target timeline, and normalizes keyframe/reference latent lists.
|
||||
|
||||
## 5. Accelerator composition
|
||||
|
||||
### SageAttention
|
||||
|
||||
Continuum never replaces the attention kernel. Existing attention overrides remain intact.
|
||||
|
||||
### Sol-Attn
|
||||
|
||||
Sol-Attn registers H3 target spans by the identity of `PackedLayout.position_ids`. Continuum modifies this tensor in place and never replaces it. Sol conditioning sink and Morton tracking therefore retain their registration.
|
||||
|
||||
### Spectrum
|
||||
|
||||
Spectrum remains an external OUTER_SAMPLE / PREDICT_NOISE / SAMPLER_SAMPLE / DIFFUSION_MODEL wrapper stack. V2 calls the normal ComfyUI `guider.sample()` once per chunk. Spectrum starts and ends an independent runtime run for each call. Continuum imports no Spectrum private functions.
|
||||
|
||||
The recommended model chain is:
|
||||
|
||||
```text
|
||||
UNET → Sage → Sol → LoRA → Spectrum → V2 Sampler
|
||||
```
|
||||
|
||||
V2 then clones this ModelPatcher once; Spectrum's ON_CLONE callback creates an independent runtime binding on the clone.
|
||||
|
||||
## 6. State and Session
|
||||
|
||||
### V1 State
|
||||
|
||||
Contains only the maximum retained continuation tail and temporal metadata. It can start V2 through `initial_state`.
|
||||
|
||||
### V2 Session
|
||||
|
||||
Contains all accepted raw chunk AV latents plus:
|
||||
|
||||
- Prompt and SHA-256 Prompt hash
|
||||
- per-chunk Seed
|
||||
- Continuum Plan
|
||||
- selected Context frames and motion score
|
||||
- model/accelerator fingerprint
|
||||
- branch parent/session identifiers
|
||||
|
||||
Tensors remain on CPU. Save Session writes safetensors and a JSON mirror atomically. The authoritative metadata is embedded in safetensors.
|
||||
|
||||
## 7. Reroll semantics
|
||||
|
||||
- `reroll_from_chunk=0`: reuse every matching accepted prefix.
|
||||
- `reroll_from_chunk=N`: preserve chunks before N and regenerate N onward.
|
||||
- A Prompt mismatch automatically stops reuse before the changed chunk.
|
||||
- A first-frame fingerprint mismatch fails explicitly.
|
||||
- A model/accelerator fingerprint mismatch is reported but accepted chunks remain fixed, because their latent output is already approved.
|
||||
- The new output creates a child Session; the input Session is not mutated.
|
||||
|
||||
## 8. Audio timeline
|
||||
|
||||
H3 video is 24fps and audio latent is 40Hz. Decoded PCM is placed using cumulative integer boundaries:
|
||||
|
||||
```text
|
||||
start_sample = round(cumulative_frames / 24 * sample_rate)
|
||||
stop_sample = round(next_cumulative_frames / 24 * sample_rate)
|
||||
```
|
||||
|
||||
This avoids the one-sample drift that can occur when every 119-frame segment is rounded independently.
|
||||
|
||||
## 9. Memory model
|
||||
|
||||
Full accepted AV latents are comparatively small and stored on CPU. The final decoded IMAGE is large, so V2:
|
||||
|
||||
- estimates final float32 IMAGE + one raw chunk + headroom before sampling;
|
||||
- refuses unsafe jobs before expensive H3 work;
|
||||
- allocates the final IMAGE exactly once;
|
||||
- copies each decoded segment into its final range;
|
||||
- avoids progressive whole-sequence concatenation.
|
||||
|
||||
## 10. Compatibility policy
|
||||
|
||||
The package checks:
|
||||
|
||||
- native `MiniMaxH3Model` class and required attributes;
|
||||
- `PackedLayout` capability, including transparent `*args/**kwargs` wrappers;
|
||||
- H3 temporal grid `(1,4,4,4,4)`;
|
||||
- APPLY_MODEL and DIFFUSION_MODEL wrapper APIs;
|
||||
- NestedTensor support;
|
||||
- real PackedLayout row placement through the installer self-test.
|
||||
|
||||
Marked Continuum payload failures never silently fall back to stock reference placement, because that could render a plausible but semantically wrong join.
|
||||
@@ -0,0 +1,16 @@
|
||||
# Future roadmap after V2
|
||||
|
||||
V2 is the integrated production path. Future changes must preserve V1/V2 schemas and remain optional.
|
||||
|
||||
Possible additions:
|
||||
|
||||
- streaming decode/save that does not retain the complete final IMAGE tensor;
|
||||
- interactive per-chunk approval UI with a frontend timeline;
|
||||
- persisted multiple child branches and branch comparison;
|
||||
- richer identity diagnostics using optional external encoders;
|
||||
- adaptive Spectrum seam policy only if Spectrum exposes a stable public hook;
|
||||
- reference-video / Ref2VA identity-memory mode;
|
||||
- multi-resolution continuation through explicit regeneration, never silent resize;
|
||||
- batch-independent server/API orchestration.
|
||||
|
||||
These are not promises for a specific release. The current V2 avoids private accelerator APIs so updates remain maintainable.
|
||||
@@ -0,0 +1,35 @@
|
||||
# V2.1.4 UI Simplification
|
||||
|
||||
The integrated `H3 Continuum Sampler V2` is the primary workflow node.
|
||||
|
||||
## Default workflow
|
||||
|
||||
```text
|
||||
Text (Multiline) -> Sequence Prompt
|
||||
Image -> first_frame
|
||||
Sampler images/audio -> Create Video
|
||||
```
|
||||
|
||||
The default node surface shows Sequence Prompt, first frame, chunk timing,
|
||||
continuity, seed, and seam correction. Individual clip overrides and Advanced
|
||||
settings are collapsed.
|
||||
|
||||
## Advanced settings
|
||||
|
||||
Advanced settings expose `last_frame`, `initial_state`, `session`,
|
||||
`prompt_plan`, reroll controls, diagnostics, `last_state`, session output, and
|
||||
report output. A saved workflow with an existing Advanced connection expands
|
||||
the section automatically.
|
||||
|
||||
The Advanced Settings button remains the section header when expanded. The
|
||||
button is placed before the expanded controls instead of moving below them.
|
||||
|
||||
## Legacy nodes
|
||||
|
||||
V1 Join, Finish, Assemble, state utilities, and auxiliary V2 plan/session nodes
|
||||
remain registered for saved-workflow compatibility. They are marked deprecated
|
||||
and moved to `MiniMax H3/Continuum/Legacy`, so ComfyUI can omit them from the
|
||||
normal node picker.
|
||||
|
||||
No sampling, State/Session schema, output order, or accelerator behavior changes
|
||||
in V2.1.4.
|
||||
@@ -0,0 +1,150 @@
|
||||
# H3 Continuum Join 2.0 — 実機検証
|
||||
|
||||
## 1. インストール検査
|
||||
|
||||
ComfyUIを終了後、展開フォルダーで実行します。
|
||||
|
||||
```powershell
|
||||
.\install_windows.bat
|
||||
```
|
||||
|
||||
または:
|
||||
|
||||
```powershell
|
||||
D:\StabilityMatrix\Data\Packages\ComfyUI_W\venv\Scripts\python.exe `
|
||||
.\tools\verify_runtime.py `
|
||||
--comfy-root D:\StabilityMatrix\Data\Packages\ComfyUI_W
|
||||
```
|
||||
|
||||
確認項目:
|
||||
|
||||
- native H3 API
|
||||
- Sol-Attn形式の`*args/**kwargs` PackedLayout wrapper互換
|
||||
- real PackedLayout行構造とin-place座標補正
|
||||
- V1/V2計10ノード登録
|
||||
- Fixed 3×5秒Prompt Plan
|
||||
|
||||
## 2. 最初のV2基準生成
|
||||
|
||||
```text
|
||||
chunks: 3
|
||||
chunk_seconds: 5.0
|
||||
continuity: Balanced — 22 frames
|
||||
audio_continuity: true
|
||||
exact_total_duration: true
|
||||
diagnostics: Basic
|
||||
reroll_from_chunk: 0
|
||||
reroll_nonce: 0
|
||||
strict_compatibility: true
|
||||
```
|
||||
|
||||
Model:
|
||||
|
||||
```text
|
||||
H3 INT8 ConvRot
|
||||
20 steps / CFG 1
|
||||
Turbo LoRA OFF
|
||||
```
|
||||
|
||||
同じ入力画像とPromptで、既存V1の3×5秒ワークフローとV2を比較します。
|
||||
|
||||
## 3. アクセラレータ比較
|
||||
|
||||
同じBase Seed、Prompt、解像度で次を順番に確認します。
|
||||
|
||||
1. SageAttentionのみ
|
||||
2. Sage + Sol-Attn
|
||||
3. Sage + Spectrum
|
||||
4. Sage + Sol-Attn + Spectrum
|
||||
|
||||
RTX 5060 Ti環境で既に動作したSage backendを基準にします。クラッシュしたbackendへ切り替えません。
|
||||
|
||||
確認:
|
||||
|
||||
- 全チャンクが順番に完了
|
||||
- Spectrumのrunが各チャンクで開始・終了
|
||||
- `Sol-Attn takes first refusal...`ログ
|
||||
- 黒出力、NaN、CUDA errorなし
|
||||
- VAE Decodeが全H3 Sampling後に始まる
|
||||
- 最終出力が指定尺になる
|
||||
|
||||
## 4. 接続品質
|
||||
|
||||
Basic確認:
|
||||
|
||||
- 人物ID、髪、服、背景
|
||||
- 前チャンクからの姿勢・移動方向・カメラ速度
|
||||
- 口形、歌唱・発話の継続
|
||||
- 音声tick、無音、語尾重複
|
||||
- 最終フレーム数と音声samples
|
||||
|
||||
Full DiagnosticsではReportに次を出します。
|
||||
|
||||
```text
|
||||
video overlap MAE
|
||||
video overlap PSNR
|
||||
audio overlap correlation
|
||||
audio boundary jump
|
||||
```
|
||||
|
||||
絶対的な合格値ではなく、V1と各アクセラレータ構成の相対比較に使います。
|
||||
|
||||
## 5. Session再開
|
||||
|
||||
1. 3チャンク生成
|
||||
2. `Save Session slot 1`
|
||||
3. ComfyUI再起動
|
||||
4. `Load Session slot 1`
|
||||
5. chunksを4へ変更し、sessionへ接続
|
||||
6. Chunk 1〜3がreused、Chunk 4だけSamplingされることをReportで確認
|
||||
|
||||
## 6. リロール
|
||||
|
||||
```text
|
||||
reroll_from_chunk: 3
|
||||
reroll_nonce: 1
|
||||
```
|
||||
|
||||
確認:
|
||||
|
||||
- Chunk 1〜2はSamplingされない
|
||||
- Chunk 3以降だけ新Seed
|
||||
- 入力Sessionファイルは変更されない
|
||||
- 出力Sessionの`parent_session_id`が入力Sessionを示す
|
||||
|
||||
## 7. Promptモード
|
||||
|
||||
- Fixed:Qwen encodeが1回
|
||||
- List:`---`で3 Prompt
|
||||
- Timeline:`[0-5s] / [5-10s] / [10-15s]`
|
||||
- Prompt変更時、変更前チャンクだけSession再利用される
|
||||
|
||||
## 8. Auto Context
|
||||
|
||||
Balanced 22fを品質基準にします。その後Autoと比較します。
|
||||
|
||||
- 静かな動き:5fを選ぶ可能性
|
||||
- 通常:22f
|
||||
- 強いlatent motionかつ39f State:39f
|
||||
|
||||
Autoの結果が悪い内容ではBalancedへ固定します。
|
||||
|
||||
## 9. RAM guard
|
||||
|
||||
高解像度・多数チャンクで、Sampling前にRAM見積りエラーが出ることを確認します。これは意図した安全停止です。chunks、秒数、解像度を下げます。
|
||||
|
||||
## 10. 報告時に必要な情報
|
||||
|
||||
```text
|
||||
ComfyUI version / revision
|
||||
Python / Torch / CUDA
|
||||
GPU / VRAM / RAM
|
||||
H3 checkpoint
|
||||
Sage backend
|
||||
Sol settings
|
||||
Spectrum settings
|
||||
chunks / seconds / context
|
||||
steps / scheduler / sampler
|
||||
Console log
|
||||
V2 report
|
||||
```
|
||||
Reference in New Issue
Block a user