feat: finalize FL2VA terminal merge
This commit is contained in:
+47
-1
@@ -51,7 +51,8 @@ Static checks do not replace a real GPU generation. Record workflow, prompt, med
|
|||||||
- PASS `MiniMax_H3_Continuum_V31_00032_.mp4`: standard Ref2VA model; visual result accepted.
|
- PASS `MiniMax_H3_Continuum_V31_00032_.mp4`: standard Ref2VA model; visual result accepted.
|
||||||
- FAIL `MiniMax_H3_Continuum_V31_00034_.mp4`: Hybrid FLF+Reference; visible jumps near 5 s and 9 s.
|
- FAIL `MiniMax_H3_Continuum_V31_00034_.mp4`: Hybrid FLF+Reference; visible jumps near 5 s and 9 s.
|
||||||
- FAIL `MiniMax_H3_Continuum_V31_00035_.mp4`: Ref2VA model plus FL2V Turbo 8-step LoRA; two visible flashes/jumps.
|
- FAIL `MiniMax_H3_Continuum_V31_00035_.mp4`: Ref2VA model plus FL2V Turbo 8-step LoRA; two visible flashes/jumps.
|
||||||
- Current FLF baseline is `context_bound_terminal_v1`: final-chunk context-tail and Last Frame anchors only.
|
- Current FLF candidate is `context_bound_terminal_last_only_v2`: the final chunk keeps its 22-frame continuation context and the single Last Frame anchor retained by `prepare_conditioning()`, without adding a duplicate context-tail boundary keyframe.
|
||||||
|
- GPU verification of this candidate is pending; it is not yet a stable baseline.
|
||||||
- `context_bound_terminal_bridge_v2` is withdrawn because its synthetic intermediate latent correlated with the later terminal flash regression.
|
- `context_bound_terminal_bridge_v2` is withdrawn because its synthetic intermediate latent correlated with the later terminal flash regression.
|
||||||
- Preserve Hybrid/Reference support, Run Storage identity, UI, audio paths, and permissive prompt handling when repairing FLF.
|
- Preserve Hybrid/Reference support, Run Storage identity, UI, audio paths, and permissive prompt handling when repairing FLF.
|
||||||
|
|
||||||
@@ -78,3 +79,48 @@ Static checks do not replace a real GPU generation. Record workflow, prompt, med
|
|||||||
- The initial rollback after 00038 temporarily returned to bridge v2, but the user clarified that the required target is the exact code state used for PASS 00031.
|
- The initial rollback after 00038 temporarily returned to bridge v2, but the user clarified that the required target is the exact code state used for PASS 00031.
|
||||||
- `00031` completed at 2026-08-21 02:32:59; `pre-flf-terminal-bridge-v2-repo-20260821_023553` was captured immediately afterward and before bridge v2 was introduced.
|
- `00031` completed at 2026-08-21 02:32:59; `pre-flf-terminal-bridge-v2-repo-20260821_023553` was captured immediately afterward and before bridge v2 was introduced.
|
||||||
- Current FLF strategy is restored exactly from that snapshot: `context_bound_terminal_v1`. Bridge v2 is withdrawn and is not active.
|
- Current FLF strategy is restored exactly from that snapshot: `context_bound_terminal_v1`. Bridge v2 is withdrawn and is not active.
|
||||||
|
|
||||||
|
## M1 Hybrid diagnostics (2026-08-21)
|
||||||
|
|
||||||
|
- Canonical implementation source: ComfyUI_W runtime.
|
||||||
|
- Runtime pre-change snapshot: D:\Codex\_snapshots\ComfyUI-H3-Continuum\pre-m1-comfyui-w-canonical-20260821_172324.
|
||||||
|
- Added display-only Hybrid conditioning labels; the stored/runtime conditioning mode remains unchanged.
|
||||||
|
- Hybrid reference warnings now use the actual global Qwen <Picture N> numbering.
|
||||||
|
- Added README numbering guidance and focused CPU-only regression tests.
|
||||||
|
- No Sampling, Conditioning payload, Run Storage contract, UI socket, or model-selection behavior changed.
|
||||||
|
- Tests were added but not executed in this implementation step.
|
||||||
|
|
||||||
|
## Terminal Merge Physical Seed v2 (2026-08-22 00:49:57)
|
||||||
|
- Terminal Merge uses one physical seed per merged pair; an initial 2x5s pair uses base_seed directly when reroll is inactive.
|
||||||
|
- Later terminal pairs derive once from the zero-based pair-start index; both logical entries store that same physical seed.
|
||||||
|
- Run Storage distinguishes the scoped terminal_merged_10s_seed_v2 / physical_window_seed_v2 execution semantics without changing the global schema version.
|
||||||
|
|
||||||
|
|
||||||
|
## Terminal Merge diagnostic harness (2026-08-22)
|
||||||
|
|
||||||
|
- Production generation semantics remain unchanged; diagnostics live under tests only.
|
||||||
|
- Diagnostic stages are ordered A Pre-Sampler, B First MODEL Call, C sampled physical AV, D split/recombine, E decode/assembly.
|
||||||
|
- Tensor identity uses shape, dtype, and SHA-256 of contiguous CPU raw bytes; numerical mismatch also records max/mean absolute difference and allclose.
|
||||||
|
- Terminal split/recombine requires bit-exact equality.
|
||||||
|
- Real Core vs Continuum GPU capture remains pending; this unit-test phase does not load checkpoints.
|
||||||
|
|
||||||
|
## GPU Core/Continuum diagnostic runner (2026-08-22)
|
||||||
|
- Source: `tools/gpu_diagnostic_node/`
|
||||||
|
- Installer: `tools/install_gpu_diagnostic_node.ps1`
|
||||||
|
- Runtime: `D:\StabilityMatrix\Data\Packages\ComfyUI_W\custom_nodes\H3_Continuum_GPU_Diagnostics`
|
||||||
|
- Node: `H3 Core vs Continuum GPU Diagnostic`
|
||||||
|
- Production Continuum files and generation semantics are not modified by this diagnostic plugin.
|
||||||
|
- Captures A: pre-sampler, B: first MODEL call, C: physical sampled AV latent, D: terminal split/recombine, E1: decode comparison, E2: common 243-to-240 adjustment.
|
||||||
|
- Artifacts: JSON manifest plus safetensors under the configured ComfyUI output diagnostics directory.
|
||||||
|
- Pre-change snapshot: `D:\Codex\_snapshots\ComfyUI-H3-Continuum\pre-gpu-core-continuum-diagnostic-runner-20260822_022625` (138/138 files verified).
|
||||||
|
- GPU comparison has not yet been executed.
|
||||||
|
|
||||||
|
## Long Terminal Merge for distinct timeline sections (2026-08-22)
|
||||||
|
|
||||||
|
- For FL2VA with 3 or more 5-second logical chunks, the final two chunks may now share one physical Terminal Merge sample even when their prompt sections differ.
|
||||||
|
- Distinct final prompts are rebased to a local physical timeline (`[0-5s]` and `[5-10s]`); identical prompts retain the accepted 2x5 Core-equivalent prompt path unchanged.
|
||||||
|
- Expected 3x5 execution: 3 logical chunks, 2 physical sampling passes, and 2 physical VAE decode groups; the terminal group is 260 frames with a 22-frame decoded prefix trim and 238 retained frames.
|
||||||
|
- Timeline Video remains excluded from Terminal Merge. UI, public sockets, Conditioning payload construction, Physical Seed v2, Assembly, and Seam semantics are otherwise unchanged.
|
||||||
|
- GPU acceptance passed for `MiniMax_H3_Continuum_V31_00008_.mp4`: 3 logical chunks, 2 physical sampling passes, decoded groups of 124 and 238 retained frames, 362-to-360 exact-duration adjustment, and 15.000-second output.
|
||||||
|
- Core comparison `MiniMax_H3_00005_.mp4` showed the same short terminal still period. This is accepted as normal FL2VA Last Frame convergence, not a Continuum-specific defect.
|
||||||
|
- Long Terminal Merge is formally adopted. Do not retune Conditioning, Physical Seed v2, physical decode grouping, or terminal prompt pairing without new contrary regression evidence.
|
||||||
|
|||||||
@@ -15,6 +15,7 @@ V3.4 focuses on the two reference workflows that are most useful in normal produ
|
|||||||
- **Driving Audio**: use an existing audio source as the preserved final audio while it guides the H3 generation across chunks.
|
- **Driving Audio**: use an existing audio source as the preserved final audio while it guides the H3 generation across chunks.
|
||||||
- **Video Reference**: use an existing video as persistent visual reference for appearance, motion, framing, and timing without treating it as a frame-by-frame copy.
|
- **Video Reference**: use an existing video as persistent visual reference for appearance, motion, framing, and timing without treating it as a frame-by-frame copy.
|
||||||
- **Hybrid FLF + Reference**: use First Frame, Last Frame, and persistent Reference Images together without adding a separate mode or model allowlist.
|
- **Hybrid FLF + Reference**: use First Frame, Last Frame, and persistent Reference Images together without adding a separate mode or model allowlist.
|
||||||
|
- **FL2VA Terminal Merge**: keep the final two 5-second FL2VA chunks as one physical 10-second sampling and decode unit for Core-equivalent Last Frame handling.
|
||||||
- **Restartable chunks**: reuse completed chunks with Run Storage and regenerate only the selected part when the generation contract remains compatible.
|
- **Restartable chunks**: reuse completed chunks with Run Storage and regenerate only the selected part when the generation contract remains compatible.
|
||||||
- **Core compatibility**: remove Continuum-only rejection of unknown upstream/custom nodes and remove the obsolete `strict_compatibility` control.
|
- **Core compatibility**: remove Continuum-only rejection of unknown upstream/custom nodes and remove the obsolete `strict_compatibility` control.
|
||||||
- **Spectrum interoperability**: use the official H3 Continuum Interop API v1 when Spectrum is installed; Spectrum remains optional.
|
- **Spectrum interoperability**: use the official H3 Continuum Interop API v1 when Spectrum is installed; Spectrum remains optional.
|
||||||
@@ -83,6 +84,18 @@ V3.4 supports First Frame, Last Frame, and persistent Reference Images in the sa
|
|||||||
|
|
||||||
Local checks passed with the B2049 hybrid variant. FL-only models could run, but showed less reliable transitions when FLF and Reference Images were combined; a hybrid-capable model is therefore recommended for this path.
|
Local checks passed with the B2049 hybrid variant. FL-only models could run, but showed less reliable transitions when FLF and Reference Images were combined; a hybrid-capable model is therefore recommended for this path.
|
||||||
|
|
||||||
|
### FL2VA Terminal Merge
|
||||||
|
|
||||||
|
For 5-second FL2VA workflows, Continuum keeps the final two logical chunks together as one physical sampling and VAE decode unit.
|
||||||
|
|
||||||
|
- `2 x 5s`: one 243-frame physical sample, decoded once, then adjusted to the requested 240-frame output.
|
||||||
|
- `3 x 5s` and longer: earlier chunks use normal Continuation; the final two chunks use one 260-frame physical sample containing the 22-frame continuation context.
|
||||||
|
- After the terminal physical decode, the 22-frame context prefix is removed. For `3 x 5s`, this gives `124 + 238 = 362` frames before exact-duration adjustment to 360 frames.
|
||||||
|
- If the final two prompt sections differ, they are rebased internally to a local `[0-5s]` / `[5-10s]` timeline. Identical prompts retain the accepted shared-prompt path.
|
||||||
|
- The terminal pair uses one physical seed and one physical decode while Run Storage continues to preserve normal logical chunk entries.
|
||||||
|
|
||||||
|
Local GPU comparison confirmed that `2 x 5s` matches Core 10-second FL2VA sampling and terminal decode behavior. A `3 x 5s` Timeline run also passed with three logical chunks, two physical sampling passes, two physical decode groups, and a 15.000-second output. The short still period near the supplied Last Frame was also present in the equivalent Core result and is treated as normal FL2VA convergence rather than a Continuum-specific failure.
|
||||||
|
|
||||||
### Core-first, permissive execution
|
### Core-first, permissive execution
|
||||||
|
|
||||||
V3.4 removes or relaxes Continuum-only rejection rules that blocked otherwise runnable Core H3 experiments.
|
V3.4 removes or relaxes Continuum-only rejection rules that blocked otherwise runnable Core H3 experiments.
|
||||||
@@ -295,6 +308,8 @@ V3.4 has been exercised locally with:
|
|||||||
- Spectrum enabled and disabled
|
- Spectrum enabled and disabled
|
||||||
- I2VA, Reference, and selected FL2VA configurations
|
- I2VA, Reference, and selected FL2VA configurations
|
||||||
- Hybrid FLF + Reference with the B2049 hybrid variant
|
- Hybrid FLF + Reference with the B2049 hybrid variant
|
||||||
|
- Core-equivalent `2 x 5s` FL2VA Terminal Merge
|
||||||
|
- `3 x 5s` FL2VA with the final two logical chunks merged into one 260-frame physical sample and decode group
|
||||||
- Driving Audio with short, exact-length, and longer sources
|
- Driving Audio with short, exact-length, and longer sources
|
||||||
- Video Reference with source audio routed separately
|
- Video Reference with source audio routed separately
|
||||||
- 0.4 MP and 0.6 MP reference sizing
|
- 0.4 MP and 0.6 MP reference sizing
|
||||||
@@ -308,6 +323,8 @@ No OOM was observed in the cited recent local V3.4 checks, including two-chunk 8
|
|||||||
- Continuation does not guarantee frame-perfect identity or motion.
|
- Continuation does not guarantee frame-perfect identity or motion.
|
||||||
- Video Reference guides H3; it does not reproduce every source frame.
|
- Video Reference guides H3; it does not reproduce every source frame.
|
||||||
- Driving Audio preserves selected audio, but visual lip synchronization remains model-dependent.
|
- Driving Audio preserves selected audio, but visual lip synchronization remains model-dependent.
|
||||||
|
- FL2VA may settle on the supplied Last Frame before the requested duration ends; equivalent Core runs can show the same behavior.
|
||||||
|
- Long, high-resolution sequences may exceed system RAM during final decode and assembly even when chunk sampling succeeds.
|
||||||
- Match Output can be substantially slower than 0.4 MP or 0.6 MP.
|
- Match Output can be substantially slower than 0.4 MP or 0.6 MP.
|
||||||
- Seam correction may keep the native boundary when a proposed correction is not safer.
|
- Seam correction may keep the native boundary when a proposed correction is not safer.
|
||||||
- Optional template nodes must be installed, replaced, or bypassed by the user.
|
- Optional template nodes must be installed, replaced, or bypassed by the user.
|
||||||
|
|||||||
+62
@@ -60,3 +60,65 @@ D:\Codex\_snapshots\ComfyUI-H3-Continuum\pre-rollback-after-00038-fail-20260821_
|
|||||||
- Restored only `v2/sequence.py` from `pre-flf-terminal-bridge-v2-repo-20260821_023553` to the repository and active ComfyUI_W runtime.
|
- Restored only `v2/sequence.py` from `pre-flf-terminal-bridge-v2-repo-20260821_023553` to the repository and active ComfyUI_W runtime.
|
||||||
- Active FLF strategy is `context_bound_terminal_v1`; bridge v2 is no longer active.
|
- Active FLF strategy is `context_bound_terminal_v1`; bridge v2 is no longer active.
|
||||||
- No UI or unrelated feature files were changed. GPU retest was not run.
|
- No UI or unrelated feature files were changed. GPU retest was not run.
|
||||||
|
|
||||||
|
## 2026-08-21 - M1 Hybrid diagnostics
|
||||||
|
|
||||||
|
- Used ComfyUI_W as the authoritative source after confirming repository/runtime differences.
|
||||||
|
- Created and verified a runtime snapshot before edits.
|
||||||
|
- Implemented display-only Hybrid labels and global Picture-number warnings.
|
||||||
|
- Documented Hybrid Picture numbering and added focused regression tests.
|
||||||
|
- Runtime generation behavior and public UI were intentionally left unchanged.
|
||||||
|
- Tests were not run in this implementation step.
|
||||||
|
## 2026-08-21 - FLF duplicate boundary anchor removal
|
||||||
|
|
||||||
|
- Preserved the active ComfyUI_W M3 display-label and M4 Last Frame session checks while synchronizing `v2/sequence.py` to the repository.
|
||||||
|
- Changed the FLF contract to `context_bound_terminal_last_only_v2` so Run Storage cannot reuse chunks produced by the previous strategy.
|
||||||
|
- Removed only the final-chunk call that duplicated the continuation-context tail as a boundary keyframe.
|
||||||
|
- Kept the 22-frame continuation context and the single supplied Last Frame anchor retained by `prepare_conditioning()`.
|
||||||
|
- No UI, public sockets, Assembly, audio, Spectrum, or general continuation behavior changed.
|
||||||
|
- Automated and GPU validation were not run in this implementation step.
|
||||||
|
|
||||||
|
- 2026-08-22 00:49:57 Terminal Merge Physical Seed v2 implemented locally after snapshot pre-change-20260822_004423. Changed v2/sequence.py and run_storage.py; added seed and revision regressions. No commit or push.
|
||||||
|
|
||||||
|
- 2026-08-22 00:51:45 Validation: 220 passed, 1 pre-existing failure in test_v2_sequence_orchestration because conditioning_display_label is absent from conditioning.py. All Physical Seed v2 and Run Storage revision tests passed. GPU generation not run.
|
||||||
|
|
||||||
|
|
||||||
|
## 2026-08-22 - Terminal Merge staged diagnostics
|
||||||
|
|
||||||
|
- Snapshot: D:\Codex\_snapshots\ComfyUI-H3-Continuum\pre-terminal-merge-diagnostics-20260822_020527
|
||||||
|
- Added tests-only deterministic capture/comparison helpers for stages A-E.
|
||||||
|
- Added unit coverage for noise, PackedLayout/position_ids, sampled-stage ordering, numerical tolerance, and bit-exact 243f Terminal split/recombine.
|
||||||
|
- No Production code, UI, Conditioning, Seed v2, Terminal Merge, Assembly, or Seam behavior changed.
|
||||||
|
- Commit/push not performed.
|
||||||
|
|
||||||
|
## 2026-08-22 - Standalone Core vs Continuum GPU diagnostics
|
||||||
|
- Added a standalone diagnostic node without changing Production runtime semantics.
|
||||||
|
- Added capture helpers and regression tests.
|
||||||
|
- Installed to ComfyUI_W as a separate custom node and verified source/runtime SHA-256 equality.
|
||||||
|
- Registration import: PASS (`H3CoreContinuumGpuDiagnostic`).
|
||||||
|
- Targeted tests: 6 passed.
|
||||||
|
- Full suite: 232 passed, 1 pre-existing unrelated failure (`conditioning_display_label` missing for unfinished M3 wiring).
|
||||||
|
- GPU execution: not run.
|
||||||
|
- Commit/push: not performed.
|
||||||
|
## 2026-08-22 - Terminal Merge physical decode groups
|
||||||
|
|
||||||
|
- Preserved logical chunks for Session and Run Storage.
|
||||||
|
- Recombined terminal logical AV latent slices bit-exactly before external Core VAE Decode.
|
||||||
|
- Added assembly_plan.decode_groups so one terminal physical sample is decoded once.
|
||||||
|
- Assembly hardening and Video Seam now operate at physical decode-group boundaries.
|
||||||
|
- Sampling, Physical Seed v2, Conditioning, UI, sockets, and Run Storage schema were unchanged.
|
||||||
|
- Synced implementation to ComfyUI_W.
|
||||||
|
- Pending validation: targeted pytest, full pytest, and GPU Core-vs-Continuum decode comparison.
|
||||||
|
|
||||||
|
## 2026-08-22 - Long Terminal Merge prompt pairing
|
||||||
|
|
||||||
|
- Snapshots: `pre-long-terminal-merge-repo`, `pre-long-terminal-merge-comfyui-w`, and `pre-long-terminal-merge-comfyui-wan` under `D:\Codex\_snapshots\ComfyUI-H3-Continuum`.
|
||||||
|
- Removed only the final-prompt-hash equality gate from Terminal Merge eligibility.
|
||||||
|
- Added a scoped physical prompt policy: identical prompts remain byte-identical; distinct final logical prompts are rebased to `[0-5s]` / `[5-10s]` for one terminal physical sample.
|
||||||
|
- Added scoped Run Storage execution semantics and 3-logical-to-2-physical decode-group regression coverage.
|
||||||
|
- No UI, public sockets, model selection, Spectrum, Assembly, Seam, or global Run Storage schema changes.
|
||||||
|
- Focused regression result: 22 passed; GPU generation not yet run for the new 3x5 path.
|
||||||
|
- Runtime correction: `run_storage.py` was also synchronized to ComfyUI_W and ComfyUI_WAN after the first GPU attempt exposed an old `RunStorageController.prepare()` signature. Run Storage plus Terminal Merge regressions: 47 passed.
|
||||||
|
- GPU PASS: `MiniMax_H3_Continuum_V31_00008_.mp4` exercised 3x5s FL2VA with `terminal_pair=2-3`, `physical_frames=260`, `sampling_passes=1` for the pair, `paired_timeline_v1`, two physical decode groups, and final 360 frames / 15.000 seconds.
|
||||||
|
- Core comparison `MiniMax_H3_00005_.mp4` reproduced the same short terminal still period. Long Terminal Merge was formally accepted; terminal stillness is Core-equivalent FL2VA convergence and is not a Continuum repair target.
|
||||||
|
- Final release validation: full pytest `249 passed`; `tools/verify_runtime.py --comfy-root D:\StabilityMatrix\Data\Packages\ComfyUI_W` passed V3.4 registration, native PackedLayout, and Fixed 3x5s prompt-plan checks. GPU acceptance is recorded above.
|
||||||
|
|||||||
@@ -59,6 +59,46 @@ def conditioning_mode_label(mode: str) -> str:
|
|||||||
raise ValueError(f"unknown conditioning mode: {mode!r}") from exc
|
raise ValueError(f"unknown conditioning mode: {mode!r}") from exc
|
||||||
|
|
||||||
|
|
||||||
|
def conditioning_display_label(
|
||||||
|
*,
|
||||||
|
has_first: bool,
|
||||||
|
has_last: bool,
|
||||||
|
has_reference: bool,
|
||||||
|
) -> str:
|
||||||
|
"""Return a user-facing label without changing the runtime mode contract."""
|
||||||
|
if has_reference:
|
||||||
|
if has_first and has_last:
|
||||||
|
return "Hybrid FL2VA + Reference + Continuation"
|
||||||
|
if has_first:
|
||||||
|
return "Hybrid I2VA + Reference + Continuation"
|
||||||
|
if has_last:
|
||||||
|
return "Hybrid L2VA + Reference + Continuation"
|
||||||
|
return "Reference + Continuation"
|
||||||
|
mode = conditioning_mode_from_presence(
|
||||||
|
has_first=has_first,
|
||||||
|
has_last=has_last,
|
||||||
|
has_reference=False,
|
||||||
|
)
|
||||||
|
return conditioning_mode_label(mode)
|
||||||
|
|
||||||
|
|
||||||
|
def conditioning_mode_display_label(
|
||||||
|
mode: str,
|
||||||
|
*,
|
||||||
|
has_first: bool = False,
|
||||||
|
has_last: bool = False,
|
||||||
|
has_reference: bool = False,
|
||||||
|
) -> str:
|
||||||
|
"""Legacy display helper retained for existing callers and tests."""
|
||||||
|
if has_first or has_last or has_reference:
|
||||||
|
return conditioning_display_label(
|
||||||
|
has_first=has_first,
|
||||||
|
has_last=has_last,
|
||||||
|
has_reference=has_reference,
|
||||||
|
)
|
||||||
|
return conditioning_mode_label(mode)
|
||||||
|
|
||||||
|
|
||||||
def conditioning_mode_uses_video_vae(mode: str) -> bool:
|
def conditioning_mode_uses_video_vae(mode: str) -> bool:
|
||||||
if mode not in CONDITIONING_MODES:
|
if mode not in CONDITIONING_MODES:
|
||||||
raise ValueError(f"unknown conditioning mode: {mode!r}")
|
raise ValueError(f"unknown conditioning mode: {mode!r}")
|
||||||
|
|||||||
+124
-2
@@ -21,6 +21,15 @@ def _strict_int(name: str, value: Any, minimum: int | None = None) -> int:
|
|||||||
return value
|
return value
|
||||||
|
|
||||||
|
|
||||||
|
def _assembly_decode_units(plan: Mapping[str, Any]) -> list[Any]:
|
||||||
|
if "decode_groups" not in plan:
|
||||||
|
return plan["chunks"]
|
||||||
|
groups = plan["decode_groups"]
|
||||||
|
if not isinstance(groups, list) or not groups:
|
||||||
|
raise ValueError("assembly_plan.decode_groups must be a non-empty list")
|
||||||
|
return groups
|
||||||
|
|
||||||
|
|
||||||
def enrich_assembly_plan(plan: dict[str, Any]) -> dict[str, Any]:
|
def enrich_assembly_plan(plan: dict[str, Any]) -> dict[str, Any]:
|
||||||
"""Add redundant metadata without correcting any existing value."""
|
"""Add redundant metadata without correcting any existing value."""
|
||||||
chunks = plan.get("chunks")
|
chunks = plan.get("chunks")
|
||||||
@@ -184,6 +193,117 @@ def validate_assembly_plan_contract(plan: Mapping[str, Any]) -> None:
|
|||||||
"sum(chunk.net_frames) does not match assembly_plan.natural_frames"
|
"sum(chunk.net_frames) does not match assembly_plan.natural_frames"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
if "decode_groups" in plan:
|
||||||
|
decode_groups = _assembly_decode_units(plan)
|
||||||
|
if "physical_decode_group_count" in plan:
|
||||||
|
group_count = _strict_int(
|
||||||
|
"assembly_plan.physical_decode_group_count",
|
||||||
|
plan["physical_decode_group_count"],
|
||||||
|
1,
|
||||||
|
)
|
||||||
|
if group_count != len(decode_groups):
|
||||||
|
raise ValueError(
|
||||||
|
"assembly_plan.physical_decode_group_count does not match decode_groups"
|
||||||
|
)
|
||||||
|
expected_logical_indices = [int(chunk["chunk_index"]) for chunk in chunks]
|
||||||
|
covered_logical_indices: list[int] = []
|
||||||
|
decode_natural_frames = 0
|
||||||
|
for position, group in enumerate(decode_groups, start=1):
|
||||||
|
if not isinstance(group, Mapping):
|
||||||
|
raise ValueError(
|
||||||
|
f"assembly_plan decode group {position} must be a mapping"
|
||||||
|
)
|
||||||
|
total_frames = _strict_int(
|
||||||
|
f"assembly_plan decode group {position}.total_frames",
|
||||||
|
group.get("total_frames"),
|
||||||
|
5,
|
||||||
|
)
|
||||||
|
trim_frames = _strict_int(
|
||||||
|
f"assembly_plan decode group {position}.trim_frames",
|
||||||
|
group.get("trim_frames"),
|
||||||
|
0,
|
||||||
|
)
|
||||||
|
net_frames = _strict_int(
|
||||||
|
f"assembly_plan decode group {position}.net_frames",
|
||||||
|
group.get("net_frames"),
|
||||||
|
1,
|
||||||
|
)
|
||||||
|
context_frames = _strict_int(
|
||||||
|
f"assembly_plan decode group {position}.context_frames",
|
||||||
|
group.get("context_frames"),
|
||||||
|
0,
|
||||||
|
)
|
||||||
|
_strict_int(
|
||||||
|
f"assembly_plan decode group {position}.expected_video_latent_t",
|
||||||
|
group.get("expected_video_latent_t"),
|
||||||
|
1,
|
||||||
|
)
|
||||||
|
_strict_int(
|
||||||
|
f"assembly_plan decode group {position}.expected_audio_latent_t",
|
||||||
|
group.get("expected_audio_latent_t"),
|
||||||
|
1,
|
||||||
|
)
|
||||||
|
if (total_frames - 5) % 17 != 0:
|
||||||
|
raise ValueError(
|
||||||
|
f"assembly_plan decode group {position}.total_frames is not on "
|
||||||
|
"the H3 17k+5 grid"
|
||||||
|
)
|
||||||
|
if total_frames - trim_frames != net_frames:
|
||||||
|
raise ValueError(
|
||||||
|
f"assembly_plan decode group {position}: "
|
||||||
|
"total_frames - trim_frames != net_frames"
|
||||||
|
)
|
||||||
|
if trim_frames > 0 and context_frames != trim_frames:
|
||||||
|
raise ValueError(
|
||||||
|
f"assembly_plan decode group {position}: "
|
||||||
|
"continuation context and trim differ"
|
||||||
|
)
|
||||||
|
logical_indices = group.get("logical_chunk_indices")
|
||||||
|
if not isinstance(logical_indices, list) or not logical_indices:
|
||||||
|
raise ValueError(
|
||||||
|
f"assembly_plan decode group {position}.logical_chunk_indices "
|
||||||
|
"must be a non-empty list"
|
||||||
|
)
|
||||||
|
covered_logical_indices.extend(
|
||||||
|
_strict_int(
|
||||||
|
f"assembly_plan decode group {position}.logical_chunk_indices",
|
||||||
|
value,
|
||||||
|
1,
|
||||||
|
)
|
||||||
|
for value in logical_indices
|
||||||
|
)
|
||||||
|
if "terminal_merged" in group and type(group["terminal_merged"]) is not bool:
|
||||||
|
raise ValueError(
|
||||||
|
f"assembly_plan decode group {position}.terminal_merged must be boolean"
|
||||||
|
)
|
||||||
|
frame_start = _strict_int(
|
||||||
|
f"assembly_plan decode group {position}.frame_start",
|
||||||
|
group.get("frame_start"),
|
||||||
|
0,
|
||||||
|
)
|
||||||
|
frame_stop = _strict_int(
|
||||||
|
f"assembly_plan decode group {position}.frame_stop",
|
||||||
|
group.get("frame_stop"),
|
||||||
|
1,
|
||||||
|
)
|
||||||
|
if (
|
||||||
|
frame_start != decode_natural_frames
|
||||||
|
or frame_stop != decode_natural_frames + net_frames
|
||||||
|
):
|
||||||
|
raise ValueError(
|
||||||
|
f"assembly_plan decode group {position}: natural frame boundary "
|
||||||
|
"is discontinuous"
|
||||||
|
)
|
||||||
|
decode_natural_frames += net_frames
|
||||||
|
if covered_logical_indices != expected_logical_indices:
|
||||||
|
raise ValueError(
|
||||||
|
"assembly_plan.decode_groups do not cover logical chunks exactly once"
|
||||||
|
)
|
||||||
|
if decode_natural_frames != natural_frames:
|
||||||
|
raise ValueError(
|
||||||
|
"sum(decode_group.net_frames) does not match logical natural frames"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def preflight_decoded_chunks(
|
def preflight_decoded_chunks(
|
||||||
images: Any,
|
images: Any,
|
||||||
@@ -192,7 +312,7 @@ def preflight_decoded_chunks(
|
|||||||
) -> None:
|
) -> None:
|
||||||
"""Validate all decoded chunks before the assembler allocates output buffers."""
|
"""Validate all decoded chunks before the assembler allocates output buffers."""
|
||||||
validate_assembly_plan_contract(plan)
|
validate_assembly_plan_contract(plan)
|
||||||
chunks = plan["chunks"]
|
chunks = _assembly_decode_units(plan)
|
||||||
|
|
||||||
if not isinstance(images, (list, tuple)):
|
if not isinstance(images, (list, tuple)):
|
||||||
raise ValueError("decoded images must be a chunk list")
|
raise ValueError("decoded images must be a chunk list")
|
||||||
@@ -518,7 +638,9 @@ def assemble_with_hardening(base: Any, args: tuple[Any, ...], kwargs: dict[str,
|
|||||||
lines: list[str] = []
|
lines: list[str] = []
|
||||||
if diagnostics_is_full(diagnostics):
|
if diagnostics_is_full(diagnostics):
|
||||||
lines.append(format_memory_snapshot("assemble preflight", images, audio))
|
lines.append(format_memory_snapshot("assemble preflight", images, audio))
|
||||||
natural_frames = sum(int(chunk["net_frames"]) for chunk in plan["chunks"])
|
natural_frames = sum(
|
||||||
|
int(chunk["net_frames"]) for chunk in _assembly_decode_units(plan)
|
||||||
|
)
|
||||||
target_frames = int(plan["target_frames"])
|
target_frames = int(plan["target_frames"])
|
||||||
correction = target_frames - natural_frames
|
correction = target_frames - natural_frames
|
||||||
if abs(correction) > 2:
|
if abs(correction) > 2:
|
||||||
|
|||||||
@@ -571,6 +571,7 @@ def build_sampling_contract(
|
|||||||
driving_audio_vae: Any = None,
|
driving_audio_vae: Any = None,
|
||||||
reference_video_contract: dict[str, Any] | None = None,
|
reference_video_contract: dict[str, Any] | None = None,
|
||||||
timeline_video_contract: dict[str, Any] | None = None,
|
timeline_video_contract: dict[str, Any] | None = None,
|
||||||
|
execution_semantics: dict[str, Any] | None = None,
|
||||||
) -> tuple[dict[str, Any], bool, list[str]]:
|
) -> tuple[dict[str, Any], bool, list[str]]:
|
||||||
model_value, model_safe = _model_signature(model, model_fingerprint_value)
|
model_value, model_safe = _model_signature(model, model_fingerprint_value)
|
||||||
clip_value, clip_safe = _clip_signature(clip)
|
clip_value, clip_safe = _clip_signature(clip)
|
||||||
@@ -660,6 +661,8 @@ def build_sampling_contract(
|
|||||||
}
|
}
|
||||||
if graph_authoritative:
|
if graph_authoritative:
|
||||||
global_contract["upstream_graph"] = dict(upstream_graph_contract)
|
global_contract["upstream_graph"] = dict(upstream_graph_contract)
|
||||||
|
if execution_semantics is not None:
|
||||||
|
global_contract["execution_semantics"] = dict(execution_semantics)
|
||||||
if uses_video_vae:
|
if uses_video_vae:
|
||||||
global_contract["video_vae"] = video_vae_value
|
global_contract["video_vae"] = video_vae_value
|
||||||
if has_first:
|
if has_first:
|
||||||
@@ -1017,6 +1020,7 @@ class RunStorageController:
|
|||||||
driving_audio_vae: Any = None,
|
driving_audio_vae: Any = None,
|
||||||
reference_video_contract: dict[str, Any] | None = None,
|
reference_video_contract: dict[str, Any] | None = None,
|
||||||
timeline_video_contract: dict[str, Any] | None = None,
|
timeline_video_contract: dict[str, Any] | None = None,
|
||||||
|
execution_semantics: dict[str, Any] | None = None,
|
||||||
) -> dict[str, Any] | None:
|
) -> dict[str, Any] | None:
|
||||||
if existing_session is not None:
|
if existing_session is not None:
|
||||||
raise RunStorageError("Run Storage cannot be combined with an explicit Session")
|
raise RunStorageError("Run Storage cannot be combined with an explicit Session")
|
||||||
@@ -1054,6 +1058,7 @@ class RunStorageController:
|
|||||||
driving_audio_vae=driving_audio_vae,
|
driving_audio_vae=driving_audio_vae,
|
||||||
reference_video_contract=reference_video_contract,
|
reference_video_contract=reference_video_contract,
|
||||||
timeline_video_contract=timeline_video_contract,
|
timeline_video_contract=timeline_video_contract,
|
||||||
|
execution_semantics=execution_semantics,
|
||||||
)
|
)
|
||||||
effective_nonce, nonce_decision = self._resolve_effective_nonce(
|
effective_nonce, nonce_decision = self._resolve_effective_nonce(
|
||||||
contract, requested_nonce=int(reroll_nonce), resume_safe=safe
|
contract, requested_nonce=int(reroll_nonce), resume_safe=safe
|
||||||
|
|||||||
@@ -0,0 +1,378 @@
|
|||||||
|
"""Tests-only capture and comparison helpers for Terminal Merge diagnostics.
|
||||||
|
|
||||||
|
This module deliberately has no production imports or runtime hooks. Tests can
|
||||||
|
monkeypatch Core and Continuum boundaries, feed the captured values here, and
|
||||||
|
identify the first pipeline stage that differs without changing generation
|
||||||
|
semantics.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from dataclasses import dataclass
|
||||||
|
from enum import Enum
|
||||||
|
from hashlib import sha256
|
||||||
|
import json
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any, Mapping
|
||||||
|
|
||||||
|
import torch
|
||||||
|
|
||||||
|
|
||||||
|
STAGE_ORDER = (
|
||||||
|
"A_pre_sampler",
|
||||||
|
"B_first_model_call",
|
||||||
|
"C_sampled_physical_av",
|
||||||
|
"D_split_recombine",
|
||||||
|
"E_decode_assembly",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class TensorFingerprint:
|
||||||
|
shape: tuple[int, ...]
|
||||||
|
dtype: str
|
||||||
|
sha256: str
|
||||||
|
|
||||||
|
def as_dict(self) -> dict[str, Any]:
|
||||||
|
return {
|
||||||
|
"shape": list(self.shape),
|
||||||
|
"dtype": self.dtype,
|
||||||
|
"sha256": self.sha256,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class TensorDifference:
|
||||||
|
same_shape: bool
|
||||||
|
same_dtype: bool
|
||||||
|
bit_exact: bool
|
||||||
|
allclose: bool
|
||||||
|
max_abs_diff: float | None
|
||||||
|
mean_abs_diff: float | None
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class StageSnapshot:
|
||||||
|
name: str
|
||||||
|
sha256: str
|
||||||
|
payload: Any
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class StageComparison:
|
||||||
|
name: str
|
||||||
|
equal: bool
|
||||||
|
core_sha256: str | None
|
||||||
|
continuum_sha256: str | None
|
||||||
|
missing_from: str | None = None
|
||||||
|
|
||||||
|
|
||||||
|
def _tensor_raw_bytes(tensor: torch.Tensor) -> bytes:
|
||||||
|
value = tensor.detach().to("cpu").contiguous()
|
||||||
|
if value.numel() == 0:
|
||||||
|
return b""
|
||||||
|
return value.reshape(-1).view(torch.uint8).numpy().tobytes()
|
||||||
|
|
||||||
|
|
||||||
|
def tensor_fingerprint(tensor: torch.Tensor) -> TensorFingerprint:
|
||||||
|
if not torch.is_tensor(tensor):
|
||||||
|
raise TypeError("tensor_fingerprint expects a torch.Tensor")
|
||||||
|
return TensorFingerprint(
|
||||||
|
shape=tuple(int(size) for size in tensor.shape),
|
||||||
|
dtype=str(tensor.dtype),
|
||||||
|
sha256=sha256(_tensor_raw_bytes(tensor)).hexdigest(),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def tensor_difference(
|
||||||
|
core: torch.Tensor,
|
||||||
|
continuum: torch.Tensor,
|
||||||
|
*,
|
||||||
|
rtol: float = 1.0e-5,
|
||||||
|
atol: float = 1.0e-6,
|
||||||
|
) -> TensorDifference:
|
||||||
|
if not torch.is_tensor(core) or not torch.is_tensor(continuum):
|
||||||
|
raise TypeError("tensor_difference expects two torch.Tensor values")
|
||||||
|
|
||||||
|
same_shape = tuple(core.shape) == tuple(continuum.shape)
|
||||||
|
same_dtype = core.dtype == continuum.dtype
|
||||||
|
bit_exact = (
|
||||||
|
same_shape
|
||||||
|
and same_dtype
|
||||||
|
and tensor_fingerprint(core).sha256 == tensor_fingerprint(continuum).sha256
|
||||||
|
)
|
||||||
|
if not same_shape:
|
||||||
|
return TensorDifference(
|
||||||
|
same_shape=False,
|
||||||
|
same_dtype=same_dtype,
|
||||||
|
bit_exact=False,
|
||||||
|
allclose=False,
|
||||||
|
max_abs_diff=None,
|
||||||
|
mean_abs_diff=None,
|
||||||
|
)
|
||||||
|
|
||||||
|
core_value = core.detach().to("cpu")
|
||||||
|
continuum_value = continuum.detach().to("cpu")
|
||||||
|
if core_value.is_complex() or continuum_value.is_complex():
|
||||||
|
core_numeric = core_value.to(torch.complex128)
|
||||||
|
continuum_numeric = continuum_value.to(torch.complex128)
|
||||||
|
else:
|
||||||
|
core_numeric = core_value.to(torch.float64)
|
||||||
|
continuum_numeric = continuum_value.to(torch.float64)
|
||||||
|
|
||||||
|
if core_numeric.numel() == 0:
|
||||||
|
max_abs_diff = 0.0
|
||||||
|
mean_abs_diff = 0.0
|
||||||
|
else:
|
||||||
|
absolute = (core_numeric - continuum_numeric).abs()
|
||||||
|
max_abs_diff = float(absolute.max().item())
|
||||||
|
mean_abs_diff = float(absolute.mean().item())
|
||||||
|
|
||||||
|
return TensorDifference(
|
||||||
|
same_shape=True,
|
||||||
|
same_dtype=same_dtype,
|
||||||
|
bit_exact=bit_exact,
|
||||||
|
allclose=bool(
|
||||||
|
torch.allclose(
|
||||||
|
core_numeric,
|
||||||
|
continuum_numeric,
|
||||||
|
rtol=rtol,
|
||||||
|
atol=atol,
|
||||||
|
equal_nan=True,
|
||||||
|
)
|
||||||
|
),
|
||||||
|
max_abs_diff=max_abs_diff,
|
||||||
|
mean_abs_diff=mean_abs_diff,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def packed_layout_snapshot(layout: Any) -> dict[str, Any]:
|
||||||
|
return {
|
||||||
|
"signature": stable_snapshot(getattr(layout, "signature", None)),
|
||||||
|
"segments": stable_snapshot(getattr(layout, "segments", None)),
|
||||||
|
"position_ids": stable_snapshot(getattr(layout, "position_ids", None)),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def stable_snapshot(value: Any) -> Any:
|
||||||
|
"""Convert diagnostic data to deterministic JSON-compatible state."""
|
||||||
|
|
||||||
|
if torch.is_tensor(value):
|
||||||
|
return {"kind": "tensor", **tensor_fingerprint(value).as_dict()}
|
||||||
|
if isinstance(value, TensorFingerprint):
|
||||||
|
return {"kind": "tensor", **value.as_dict()}
|
||||||
|
if value is None or isinstance(value, (bool, int, str)):
|
||||||
|
return value
|
||||||
|
if isinstance(value, float):
|
||||||
|
return {"kind": "float", "hex": value.hex()}
|
||||||
|
if isinstance(value, bytes):
|
||||||
|
return {
|
||||||
|
"kind": "bytes",
|
||||||
|
"length": len(value),
|
||||||
|
"sha256": sha256(value).hexdigest(),
|
||||||
|
}
|
||||||
|
if isinstance(value, (Path, torch.device, torch.dtype)):
|
||||||
|
return str(value)
|
||||||
|
if isinstance(value, Enum):
|
||||||
|
return {
|
||||||
|
"kind": "enum",
|
||||||
|
"type": f"{type(value).__module__}.{type(value).__qualname__}",
|
||||||
|
"value": stable_snapshot(value.value),
|
||||||
|
}
|
||||||
|
if isinstance(value, slice):
|
||||||
|
return {
|
||||||
|
"kind": "slice",
|
||||||
|
"start": value.start,
|
||||||
|
"stop": value.stop,
|
||||||
|
"step": value.step,
|
||||||
|
}
|
||||||
|
if isinstance(value, Mapping):
|
||||||
|
return {
|
||||||
|
"kind": "mapping",
|
||||||
|
"items": [
|
||||||
|
[str(key), stable_snapshot(item)]
|
||||||
|
for key, item in sorted(value.items(), key=lambda pair: str(pair[0]))
|
||||||
|
],
|
||||||
|
}
|
||||||
|
if isinstance(value, tuple):
|
||||||
|
return {"kind": "tuple", "items": [stable_snapshot(item) for item in value]}
|
||||||
|
if isinstance(value, list):
|
||||||
|
return {"kind": "list", "items": [stable_snapshot(item) for item in value]}
|
||||||
|
if isinstance(value, (set, frozenset)):
|
||||||
|
items = [stable_snapshot(item) for item in value]
|
||||||
|
items.sort(key=lambda item: json.dumps(item, sort_keys=True, separators=(",", ":")))
|
||||||
|
return {"kind": type(value).__name__, "items": items}
|
||||||
|
if callable(value):
|
||||||
|
return {
|
||||||
|
"kind": "callable",
|
||||||
|
"module": getattr(value, "__module__", type(value).__module__),
|
||||||
|
"qualname": getattr(value, "__qualname__", type(value).__qualname__),
|
||||||
|
}
|
||||||
|
if all(hasattr(value, name) for name in ("signature", "segments", "position_ids")):
|
||||||
|
return {"kind": "packed_layout", **packed_layout_snapshot(value)}
|
||||||
|
if hasattr(value, "__dict__"):
|
||||||
|
public_state = {
|
||||||
|
key: item
|
||||||
|
for key, item in vars(value).items()
|
||||||
|
if not key.startswith("_") and not callable(item)
|
||||||
|
}
|
||||||
|
return {
|
||||||
|
"kind": "object",
|
||||||
|
"type": f"{type(value).__module__}.{type(value).__qualname__}",
|
||||||
|
"state": stable_snapshot(public_state),
|
||||||
|
}
|
||||||
|
raise TypeError(f"Unsupported diagnostic value: {type(value)!r}")
|
||||||
|
|
||||||
|
|
||||||
|
def snapshot_sha256(payload: Any) -> str:
|
||||||
|
normalized = stable_snapshot(payload)
|
||||||
|
encoded = json.dumps(
|
||||||
|
normalized,
|
||||||
|
ensure_ascii=True,
|
||||||
|
sort_keys=True,
|
||||||
|
separators=(",", ":"),
|
||||||
|
).encode("utf-8")
|
||||||
|
return sha256(encoded).hexdigest()
|
||||||
|
|
||||||
|
|
||||||
|
class DiagnosticCapture:
|
||||||
|
def __init__(self) -> None:
|
||||||
|
self._stages: dict[str, StageSnapshot] = {}
|
||||||
|
|
||||||
|
@property
|
||||||
|
def stages(self) -> Mapping[str, StageSnapshot]:
|
||||||
|
return self._stages
|
||||||
|
|
||||||
|
def capture(self, stage: str, **payload: Any) -> StageSnapshot:
|
||||||
|
if stage not in STAGE_ORDER:
|
||||||
|
raise ValueError(f"Unknown diagnostic stage: {stage}")
|
||||||
|
if stage in self._stages:
|
||||||
|
raise ValueError(f"Diagnostic stage already captured: {stage}")
|
||||||
|
normalized = stable_snapshot(payload)
|
||||||
|
snapshot = StageSnapshot(
|
||||||
|
name=stage,
|
||||||
|
sha256=snapshot_sha256(normalized),
|
||||||
|
payload=normalized,
|
||||||
|
)
|
||||||
|
self._stages[stage] = snapshot
|
||||||
|
return snapshot
|
||||||
|
|
||||||
|
def capture_pre_sampler(
|
||||||
|
self,
|
||||||
|
*,
|
||||||
|
physical_frames: int,
|
||||||
|
seed: int,
|
||||||
|
conditioning: Any,
|
||||||
|
latent: Any,
|
||||||
|
noise: torch.Tensor,
|
||||||
|
keyframes: Any = None,
|
||||||
|
qwen_images: Any = None,
|
||||||
|
minimax_refs: Any = None,
|
||||||
|
) -> StageSnapshot:
|
||||||
|
return self.capture(
|
||||||
|
"A_pre_sampler",
|
||||||
|
physical_frames=physical_frames,
|
||||||
|
seed=seed,
|
||||||
|
conditioning=conditioning,
|
||||||
|
latent=latent,
|
||||||
|
noise=noise,
|
||||||
|
keyframes=keyframes,
|
||||||
|
qwen_images=qwen_images,
|
||||||
|
minimax_refs=minimax_refs,
|
||||||
|
)
|
||||||
|
|
||||||
|
def capture_first_model_call(
|
||||||
|
self,
|
||||||
|
*,
|
||||||
|
minimax_payload: Any,
|
||||||
|
transformer_options: Any,
|
||||||
|
) -> StageSnapshot:
|
||||||
|
return self.capture(
|
||||||
|
"B_first_model_call",
|
||||||
|
minimax_payload=minimax_payload,
|
||||||
|
transformer_options=transformer_options,
|
||||||
|
)
|
||||||
|
|
||||||
|
def capture_sampled_physical_av(
|
||||||
|
self,
|
||||||
|
*,
|
||||||
|
video: torch.Tensor,
|
||||||
|
audio: torch.Tensor,
|
||||||
|
) -> StageSnapshot:
|
||||||
|
return self.capture(
|
||||||
|
"C_sampled_physical_av",
|
||||||
|
video=video,
|
||||||
|
audio=audio,
|
||||||
|
)
|
||||||
|
|
||||||
|
def capture_split_recombine(
|
||||||
|
self,
|
||||||
|
*,
|
||||||
|
physical_video: torch.Tensor,
|
||||||
|
physical_audio: torch.Tensor,
|
||||||
|
recombined_video: torch.Tensor,
|
||||||
|
recombined_audio: torch.Tensor,
|
||||||
|
) -> StageSnapshot:
|
||||||
|
return self.capture(
|
||||||
|
"D_split_recombine",
|
||||||
|
physical_video=physical_video,
|
||||||
|
physical_audio=physical_audio,
|
||||||
|
recombined_video=recombined_video,
|
||||||
|
recombined_audio=recombined_audio,
|
||||||
|
video_difference=tensor_difference(physical_video, recombined_video).__dict__,
|
||||||
|
audio_difference=tensor_difference(physical_audio, recombined_audio).__dict__,
|
||||||
|
)
|
||||||
|
|
||||||
|
def capture_decode_assembly(
|
||||||
|
self,
|
||||||
|
*,
|
||||||
|
physical_decoded: torch.Tensor,
|
||||||
|
assembled_decoded: torch.Tensor,
|
||||||
|
) -> StageSnapshot:
|
||||||
|
return self.capture(
|
||||||
|
"E_decode_assembly",
|
||||||
|
physical_decoded=physical_decoded,
|
||||||
|
assembled_decoded=assembled_decoded,
|
||||||
|
difference=tensor_difference(physical_decoded, assembled_decoded).__dict__,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def compare_captures(
|
||||||
|
core: DiagnosticCapture,
|
||||||
|
continuum: DiagnosticCapture,
|
||||||
|
) -> tuple[StageComparison, ...]:
|
||||||
|
comparisons = []
|
||||||
|
for stage in STAGE_ORDER:
|
||||||
|
core_stage = core.stages.get(stage)
|
||||||
|
continuum_stage = continuum.stages.get(stage)
|
||||||
|
missing = None
|
||||||
|
if core_stage is None and continuum_stage is None:
|
||||||
|
missing = "both"
|
||||||
|
elif core_stage is None:
|
||||||
|
missing = "core"
|
||||||
|
elif continuum_stage is None:
|
||||||
|
missing = "continuum"
|
||||||
|
comparisons.append(
|
||||||
|
StageComparison(
|
||||||
|
name=stage,
|
||||||
|
equal=(
|
||||||
|
core_stage is not None
|
||||||
|
and continuum_stage is not None
|
||||||
|
and core_stage.sha256 == continuum_stage.sha256
|
||||||
|
),
|
||||||
|
core_sha256=None if core_stage is None else core_stage.sha256,
|
||||||
|
continuum_sha256=None if continuum_stage is None else continuum_stage.sha256,
|
||||||
|
missing_from=missing,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return tuple(comparisons)
|
||||||
|
|
||||||
|
|
||||||
|
def first_divergent_stage(
|
||||||
|
core: DiagnosticCapture,
|
||||||
|
continuum: DiagnosticCapture,
|
||||||
|
) -> str | None:
|
||||||
|
for comparison in compare_captures(core, continuum):
|
||||||
|
if not comparison.equal:
|
||||||
|
return comparison.name
|
||||||
|
return None
|
||||||
@@ -0,0 +1,81 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import importlib.util
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import torch
|
||||||
|
|
||||||
|
|
||||||
|
MODULE_PATH = Path(__file__).parents[1] / "tools" / "gpu_diagnostic_node" / "capture.py"
|
||||||
|
SPEC = importlib.util.spec_from_file_location("h3_gpu_diagnostic_capture", MODULE_PATH)
|
||||||
|
capture = importlib.util.module_from_spec(SPEC)
|
||||||
|
assert SPEC.loader is not None
|
||||||
|
sys.modules[SPEC.name] = capture
|
||||||
|
SPEC.loader.exec_module(capture)
|
||||||
|
|
||||||
|
|
||||||
|
def test_tensor_fingerprint_is_deterministic():
|
||||||
|
value = torch.arange(12, dtype=torch.float32).reshape(1, 3, 4)
|
||||||
|
assert capture.tensor_fingerprint(value) == capture.tensor_fingerprint(value.clone())
|
||||||
|
|
||||||
|
|
||||||
|
def test_tensor_comparison_records_numerical_difference():
|
||||||
|
left = torch.tensor([1.0, 2.0], dtype=torch.float32)
|
||||||
|
right = torch.tensor([1.0, 2.000001], dtype=torch.float32)
|
||||||
|
result = capture.compare_tensors(left, right)
|
||||||
|
assert not result.bit_exact
|
||||||
|
assert result.allclose
|
||||||
|
assert result.matches
|
||||||
|
assert result.max_abs_diff > 0.0
|
||||||
|
|
||||||
|
|
||||||
|
def test_terminal_video_recombine_is_bit_exact():
|
||||||
|
physical = torch.arange(72, dtype=torch.float32).reshape(1, 1, 72, 1, 1)
|
||||||
|
first = physical[:, :, 0:37]
|
||||||
|
second = physical[:, :, 30:72]
|
||||||
|
rebuilt = capture.recombine_terminal_streams(
|
||||||
|
first,
|
||||||
|
second,
|
||||||
|
first_range=(0, 37),
|
||||||
|
second_range=(30, 72),
|
||||||
|
dim=2,
|
||||||
|
)
|
||||||
|
assert torch.equal(rebuilt, physical)
|
||||||
|
|
||||||
|
|
||||||
|
def test_terminal_audio_recombine_is_bit_exact():
|
||||||
|
physical = torch.arange(405, dtype=torch.float32).reshape(1, 1, 405)
|
||||||
|
first = physical[..., 0:207]
|
||||||
|
second = physical[..., 170:405]
|
||||||
|
rebuilt = capture.recombine_terminal_streams(
|
||||||
|
first,
|
||||||
|
second,
|
||||||
|
first_range=(0, 207),
|
||||||
|
second_range=(170, 405),
|
||||||
|
dim=-1,
|
||||||
|
)
|
||||||
|
assert torch.equal(rebuilt, physical)
|
||||||
|
|
||||||
|
|
||||||
|
def test_stage_comparison_reports_first_field():
|
||||||
|
left = capture.DiagnosticCapture("core")
|
||||||
|
right = capture.DiagnosticCapture("continuum")
|
||||||
|
left.add_value("A_pre_sampler", "seed", 42)
|
||||||
|
right.add_value("A_pre_sampler", "seed", 43)
|
||||||
|
left.add_tensor("A_pre_sampler", "sigmas", torch.ones(2))
|
||||||
|
right.add_tensor("A_pre_sampler", "sigmas", torch.ones(2))
|
||||||
|
result = capture.compare_stage(left, right, "A_pre_sampler")
|
||||||
|
assert result["status"] == "DIFFER"
|
||||||
|
assert result["first_field"] == "seed"
|
||||||
|
|
||||||
|
|
||||||
|
def test_summary_uses_stage_order_and_first_divergence(tmp_path):
|
||||||
|
comparison = {
|
||||||
|
"A_pre_sampler": {"status": "MATCH", "first_field": None},
|
||||||
|
"B_first_model_call": {"status": "DIFFER", "first_field": "position_ids"},
|
||||||
|
}
|
||||||
|
text = capture.format_comparison(comparison, tmp_path)
|
||||||
|
assert "A_pre_sampler: MATCH" in text
|
||||||
|
assert "B_first_model_call: DIFFER" in text
|
||||||
|
assert "First divergence: B_first_model_call.position_ids" in text
|
||||||
@@ -0,0 +1,59 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from ComfyUI_H3_Continuum_Join.conditioning import (
|
||||||
|
conditioning_display_label,
|
||||||
|
conditioning_mode_from_presence,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
("has_first", "has_last", "has_reference", "expected"),
|
||||||
|
[
|
||||||
|
(False, False, False, "T2VA + Continuation"),
|
||||||
|
(True, False, False, "I2VA + Continuation"),
|
||||||
|
(False, True, False, "Last Frame + Continuation"),
|
||||||
|
(True, True, False, "FL2VA + Continuation"),
|
||||||
|
(False, False, True, "Reference + Continuation"),
|
||||||
|
(True, False, True, "Hybrid I2VA + Reference + Continuation"),
|
||||||
|
(False, True, True, "Hybrid L2VA + Reference + Continuation"),
|
||||||
|
(True, True, True, "Hybrid FL2VA + Reference + Continuation"),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_conditioning_display_label_matrix(
|
||||||
|
has_first: bool,
|
||||||
|
has_last: bool,
|
||||||
|
has_reference: bool,
|
||||||
|
expected: str,
|
||||||
|
) -> None:
|
||||||
|
assert (
|
||||||
|
conditioning_display_label(
|
||||||
|
has_first=has_first,
|
||||||
|
has_last=has_last,
|
||||||
|
has_reference=has_reference,
|
||||||
|
)
|
||||||
|
== expected
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
("has_first", "has_last"),
|
||||||
|
[
|
||||||
|
(True, False),
|
||||||
|
(False, True),
|
||||||
|
(True, True),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_hybrid_display_does_not_change_internal_reference_mode(
|
||||||
|
has_first: bool,
|
||||||
|
has_last: bool,
|
||||||
|
) -> None:
|
||||||
|
assert (
|
||||||
|
conditioning_mode_from_presence(
|
||||||
|
has_first=has_first,
|
||||||
|
has_last=has_last,
|
||||||
|
has_reference=True,
|
||||||
|
)
|
||||||
|
== "reference"
|
||||||
|
)
|
||||||
@@ -0,0 +1,183 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from types import SimpleNamespace
|
||||||
|
|
||||||
|
import torch
|
||||||
|
|
||||||
|
from ComfyUI_H3_Continuum_Join.v2.sequence import (
|
||||||
|
_split_terminal_merged_latents,
|
||||||
|
_terminal_pair_contract,
|
||||||
|
)
|
||||||
|
from ComfyUI_H3_Continuum_Join.tests.terminal_merge_diagnostics import (
|
||||||
|
DiagnosticCapture,
|
||||||
|
first_divergent_stage,
|
||||||
|
tensor_difference,
|
||||||
|
tensor_fingerprint,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _capture_all_stages(*, sampled_offset: float = 0.0) -> DiagnosticCapture:
|
||||||
|
capture = DiagnosticCapture()
|
||||||
|
capture.capture_pre_sampler(
|
||||||
|
physical_frames=243,
|
||||||
|
seed=42,
|
||||||
|
conditioning=[[torch.tensor([1.0]), {"minimax_frame_count": 243}]],
|
||||||
|
latent={"video": torch.zeros(1, 24, 72, 2, 2)},
|
||||||
|
noise=torch.arange(8, dtype=torch.float32),
|
||||||
|
keyframes=(0, 242),
|
||||||
|
qwen_images=("first", "last"),
|
||||||
|
minimax_refs=None,
|
||||||
|
)
|
||||||
|
layout = SimpleNamespace(
|
||||||
|
signature=(1, 2, 3),
|
||||||
|
segments=(("target_video", 0, 72),),
|
||||||
|
position_ids=torch.arange(12, dtype=torch.float32).reshape(6, 2),
|
||||||
|
)
|
||||||
|
capture.capture_first_model_call(
|
||||||
|
minimax_payload={
|
||||||
|
"layout": layout,
|
||||||
|
"cond_video_latents": torch.ones(1, 24, 2, 2, 2),
|
||||||
|
"target_video_shape": (1, 24, 72, 2, 2),
|
||||||
|
"target_audio_shape": (1, 32, 2, 405),
|
||||||
|
},
|
||||||
|
transformer_options={"patches": {}},
|
||||||
|
)
|
||||||
|
video = torch.full((1, 24, 72, 2, 2), sampled_offset)
|
||||||
|
audio = torch.zeros(1, 32, 2, 405)
|
||||||
|
capture.capture_sampled_physical_av(video=video, audio=audio)
|
||||||
|
capture.capture_split_recombine(
|
||||||
|
physical_video=video,
|
||||||
|
physical_audio=audio,
|
||||||
|
recombined_video=video.clone(),
|
||||||
|
recombined_audio=audio.clone(),
|
||||||
|
)
|
||||||
|
decoded = torch.zeros(243, 2, 2, 3)
|
||||||
|
capture.capture_decode_assembly(
|
||||||
|
physical_decoded=decoded,
|
||||||
|
assembled_decoded=decoded.clone(),
|
||||||
|
)
|
||||||
|
return capture
|
||||||
|
|
||||||
|
|
||||||
|
def test_tensor_fingerprint_is_deterministic_and_dtype_sensitive():
|
||||||
|
value = torch.arange(12, dtype=torch.float32).reshape(3, 4)
|
||||||
|
first = tensor_fingerprint(value)
|
||||||
|
second = tensor_fingerprint(value.clone())
|
||||||
|
|
||||||
|
assert first == second
|
||||||
|
assert first.shape == (3, 4)
|
||||||
|
assert first.dtype == "torch.float32"
|
||||||
|
assert first.sha256 != tensor_fingerprint(value.to(torch.float64)).sha256
|
||||||
|
|
||||||
|
|
||||||
|
def test_tensor_difference_keeps_hash_mismatch_distinct_from_allclose():
|
||||||
|
core = torch.tensor([1.0], dtype=torch.float32)
|
||||||
|
continuum = torch.tensor([1.0 + 1.0e-7], dtype=torch.float32)
|
||||||
|
|
||||||
|
difference = tensor_difference(core, continuum)
|
||||||
|
|
||||||
|
assert difference.bit_exact is False
|
||||||
|
assert difference.allclose is True
|
||||||
|
assert difference.max_abs_diff is not None
|
||||||
|
assert difference.max_abs_diff > 0.0
|
||||||
|
assert difference.mean_abs_diff == difference.max_abs_diff
|
||||||
|
|
||||||
|
|
||||||
|
def test_pre_sampler_capture_includes_initial_noise():
|
||||||
|
core = _capture_all_stages()
|
||||||
|
continuum = _capture_all_stages()
|
||||||
|
assert first_divergent_stage(core, continuum) is None
|
||||||
|
|
||||||
|
changed = DiagnosticCapture()
|
||||||
|
changed.capture_pre_sampler(
|
||||||
|
physical_frames=243,
|
||||||
|
seed=42,
|
||||||
|
conditioning=[[torch.tensor([1.0]), {"minimax_frame_count": 243}]],
|
||||||
|
latent={"video": torch.zeros(1, 24, 72, 2, 2)},
|
||||||
|
noise=torch.arange(8, dtype=torch.float32) + 1.0,
|
||||||
|
keyframes=(0, 242),
|
||||||
|
qwen_images=("first", "last"),
|
||||||
|
minimax_refs=None,
|
||||||
|
)
|
||||||
|
|
||||||
|
assert first_divergent_stage(core, changed) == "A_pre_sampler"
|
||||||
|
|
||||||
|
|
||||||
|
def test_first_model_call_capture_detects_packed_layout_difference():
|
||||||
|
core = DiagnosticCapture()
|
||||||
|
continuum = DiagnosticCapture()
|
||||||
|
shared = dict(
|
||||||
|
physical_frames=243,
|
||||||
|
seed=42,
|
||||||
|
conditioning=[torch.zeros(1)],
|
||||||
|
latent=torch.zeros(1),
|
||||||
|
noise=torch.zeros(1),
|
||||||
|
)
|
||||||
|
core.capture_pre_sampler(**shared)
|
||||||
|
continuum.capture_pre_sampler(**shared)
|
||||||
|
|
||||||
|
core_layout = SimpleNamespace(
|
||||||
|
signature=(1, 2),
|
||||||
|
segments=(("target_video", 0, 72),),
|
||||||
|
position_ids=torch.zeros(4, 2),
|
||||||
|
)
|
||||||
|
continuum_layout = SimpleNamespace(
|
||||||
|
signature=(1, 2),
|
||||||
|
segments=(("target_video", 0, 72),),
|
||||||
|
position_ids=torch.ones(4, 2),
|
||||||
|
)
|
||||||
|
core.capture_first_model_call(
|
||||||
|
minimax_payload={"layout": core_layout},
|
||||||
|
transformer_options={},
|
||||||
|
)
|
||||||
|
continuum.capture_first_model_call(
|
||||||
|
minimax_payload={"layout": continuum_layout},
|
||||||
|
transformer_options={},
|
||||||
|
)
|
||||||
|
|
||||||
|
assert first_divergent_stage(core, continuum) == "B_first_model_call"
|
||||||
|
|
||||||
|
|
||||||
|
def test_first_divergent_stage_uses_pipeline_order():
|
||||||
|
core = _capture_all_stages(sampled_offset=0.0)
|
||||||
|
continuum = _capture_all_stages(sampled_offset=1.0)
|
||||||
|
|
||||||
|
assert first_divergent_stage(core, continuum) == "C_sampled_physical_av"
|
||||||
|
|
||||||
|
|
||||||
|
def test_terminal_split_recombine_is_bit_exact():
|
||||||
|
contract = _terminal_pair_contract(initial_pair=True, chunk_seconds=5.0)
|
||||||
|
video = torch.arange(
|
||||||
|
1 * 24 * contract["video_slices"][-1][1] * 2 * 2,
|
||||||
|
dtype=torch.float32,
|
||||||
|
).reshape(1, 24, contract["video_slices"][-1][1], 2, 2)
|
||||||
|
audio = torch.arange(
|
||||||
|
1 * 32 * 2 * contract["audio_slices"][-1][1],
|
||||||
|
dtype=torch.float32,
|
||||||
|
).reshape(1, 32, 2, contract["audio_slices"][-1][1])
|
||||||
|
|
||||||
|
(video_1, audio_1), (video_2, audio_2) = _split_terminal_merged_latents(
|
||||||
|
video,
|
||||||
|
audio,
|
||||||
|
contract,
|
||||||
|
)
|
||||||
|
video_overlap = contract["video_slices"][0][1] - contract["video_slices"][1][0]
|
||||||
|
audio_overlap = contract["audio_slices"][0][1] - contract["audio_slices"][1][0]
|
||||||
|
recombined_video = torch.cat((video_1, video_2[:, :, video_overlap:]), dim=2)
|
||||||
|
recombined_audio = torch.cat((audio_1, audio_2[..., audio_overlap:]), dim=-1)
|
||||||
|
|
||||||
|
assert contract["physical_frames"] == 243
|
||||||
|
assert contract["logical_frames"] == (124, 141)
|
||||||
|
assert torch.equal(recombined_video, video)
|
||||||
|
assert torch.equal(recombined_audio, audio)
|
||||||
|
assert tensor_difference(video, recombined_video).bit_exact is True
|
||||||
|
assert tensor_difference(audio, recombined_audio).bit_exact is True
|
||||||
|
|
||||||
|
capture = DiagnosticCapture()
|
||||||
|
stage = capture.capture_split_recombine(
|
||||||
|
physical_video=video,
|
||||||
|
physical_audio=audio,
|
||||||
|
recombined_video=recombined_video,
|
||||||
|
recombined_audio=recombined_audio,
|
||||||
|
)
|
||||||
|
assert stage.name == "D_split_recombine"
|
||||||
@@ -0,0 +1,165 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
import torch
|
||||||
|
|
||||||
|
from ComfyUI_H3_Continuum_Join.v2.sequence import (
|
||||||
|
_atomic_terminal_prefix,
|
||||||
|
_split_terminal_merged_latents,
|
||||||
|
_terminal_flf_merge_enabled,
|
||||||
|
_terminal_pair_prompt,
|
||||||
|
_terminal_physical_seed_plan,
|
||||||
|
_terminal_pair_contract,
|
||||||
|
_terminal_sampling_plan,
|
||||||
|
)
|
||||||
|
from ComfyUI_H3_Continuum_Join.v2.seeds import derive_chunk_seed
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
("initial_pair", "physical_frames", "logical_frames", "video_slices", "audio_slices"),
|
||||||
|
[
|
||||||
|
(True, 243, (124, 141), ((0, 37), (30, 72)), ((0, 207), (170, 405))),
|
||||||
|
(False, 260, (141, 141), ((0, 42), (35, 77)), ((0, 235), (198, 433))),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_terminal_pair_contract_uses_native_temporal_grid(
|
||||||
|
initial_pair,
|
||||||
|
physical_frames,
|
||||||
|
logical_frames,
|
||||||
|
video_slices,
|
||||||
|
audio_slices,
|
||||||
|
):
|
||||||
|
contract = _terminal_pair_contract(initial_pair=initial_pair, chunk_seconds=5.0)
|
||||||
|
assert contract["physical_frames"] == physical_frames
|
||||||
|
assert contract["logical_frames"] == logical_frames
|
||||||
|
assert contract["video_slices"] == video_slices
|
||||||
|
assert contract["audio_slices"] == audio_slices
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("initial_pair", [True, False])
|
||||||
|
def test_terminal_pair_split_preserves_expected_overlap(initial_pair):
|
||||||
|
contract = _terminal_pair_contract(initial_pair=initial_pair, chunk_seconds=5.0)
|
||||||
|
video_t = contract["video_slices"][-1][1]
|
||||||
|
audio_t = contract["audio_slices"][-1][1]
|
||||||
|
video = torch.arange(video_t, dtype=torch.float32).reshape(1, 1, video_t, 1, 1)
|
||||||
|
audio = torch.arange(audio_t, dtype=torch.float32).reshape(1, 1, 1, audio_t)
|
||||||
|
first, second = _split_terminal_merged_latents(video, audio, contract)
|
||||||
|
assert first[0].shape[2] == contract["video_slices"][0][1]
|
||||||
|
assert first[1].shape[-1] == contract["audio_slices"][0][1]
|
||||||
|
assert second[0][0, 0, 0, 0, 0].item() == contract["video_slices"][1][0]
|
||||||
|
assert second[1][0, 0, 0, 0].item() == contract["audio_slices"][1][0]
|
||||||
|
|
||||||
|
|
||||||
|
def test_two_by_five_uses_one_physical_sample():
|
||||||
|
normal, terminal = _terminal_sampling_plan(chunks=2, completed=0, merge_enabled=True)
|
||||||
|
assert normal == ()
|
||||||
|
assert terminal is True
|
||||||
|
assert len(normal) + int(terminal) == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_three_by_five_uses_one_normal_and_one_terminal_sample():
|
||||||
|
normal, terminal = _terminal_sampling_plan(chunks=3, completed=0, merge_enabled=True)
|
||||||
|
assert normal == (0,)
|
||||||
|
assert terminal is True
|
||||||
|
assert len(normal) + int(terminal) == 2
|
||||||
|
|
||||||
|
|
||||||
|
def test_terminal_merge_accepts_distinct_prompts_but_not_timeline_video():
|
||||||
|
common = dict(multi_chunk_flf=True, chunks=2, chunk_seconds=5.0)
|
||||||
|
assert _terminal_flf_merge_enabled(**common, prompt_hashes=["same", "same"], timeline_video_source=None)
|
||||||
|
assert _terminal_flf_merge_enabled(**common, prompt_hashes=["a", "b"], timeline_video_source=None)
|
||||||
|
assert not _terminal_flf_merge_enabled(**common, prompt_hashes=["same", "same"], timeline_video_source=object())
|
||||||
|
|
||||||
|
|
||||||
|
def test_terminal_pair_prompt_preserves_shared_prompt_exactly():
|
||||||
|
prompt, policy = _terminal_pair_prompt(["same", "same"], pair_start=0, chunk_seconds=5.0)
|
||||||
|
assert prompt == "same"
|
||||||
|
assert policy == "shared_prompt_v1"
|
||||||
|
|
||||||
|
|
||||||
|
def test_terminal_pair_prompt_rebases_distinct_sections_to_local_timeline():
|
||||||
|
prompt, policy = _terminal_pair_prompt(["first action", "second action"], pair_start=0, chunk_seconds=5.0)
|
||||||
|
assert prompt == "[0-5s]\nfirst action\n\n[5-10s]\nsecond action"
|
||||||
|
assert policy == "paired_timeline_v1"
|
||||||
|
|
||||||
|
|
||||||
|
def test_partial_terminal_pair_is_discarded_for_atomic_resume():
|
||||||
|
preserved = [{"chunk": 1}, {"chunk": 2}]
|
||||||
|
result, reset = _atomic_terminal_prefix(preserved, chunks=3, reroll_from_chunk=0)
|
||||||
|
assert result == preserved[:1]
|
||||||
|
assert reset is True
|
||||||
|
|
||||||
|
|
||||||
|
def test_rerolling_final_chunk_regenerates_both_terminal_chunks():
|
||||||
|
preserved = [{"chunk": 1}, {"chunk": 2}, {"chunk": 3}]
|
||||||
|
result, reset = _atomic_terminal_prefix(preserved, chunks=3, reroll_from_chunk=3)
|
||||||
|
assert result == preserved[:1]
|
||||||
|
assert reset is True
|
||||||
|
|
||||||
|
|
||||||
|
def test_complete_terminal_pair_remains_reusable_without_reroll():
|
||||||
|
preserved = [{"chunk": 1}, {"chunk": 2}, {"chunk": 3}]
|
||||||
|
result, reset = _atomic_terminal_prefix(preserved, chunks=3, reroll_from_chunk=0)
|
||||||
|
assert result == preserved
|
||||||
|
assert reset is False
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("base_seed", [42, 123456789])
|
||||||
|
def test_initial_terminal_pair_uses_base_seed_directly(base_seed):
|
||||||
|
plan = _terminal_physical_seed_plan(
|
||||||
|
base_seed=base_seed,
|
||||||
|
physical_window_start_index=0,
|
||||||
|
reroll_nonce=0,
|
||||||
|
)
|
||||||
|
assert plan["physical_seed"] == base_seed
|
||||||
|
assert plan["logical_entry_seeds"] == (base_seed, base_seed)
|
||||||
|
|
||||||
|
|
||||||
|
def test_later_terminal_pair_derives_once_from_pair_start():
|
||||||
|
plan = _terminal_physical_seed_plan(
|
||||||
|
base_seed=42,
|
||||||
|
physical_window_start_index=2,
|
||||||
|
reroll_nonce=0,
|
||||||
|
)
|
||||||
|
assert plan["physical_seed"] == derive_chunk_seed(42, 2, 0)
|
||||||
|
assert plan["physical_seed"] == 1808285636802909398
|
||||||
|
assert plan["logical_entry_seeds"] == (plan["physical_seed"],) * 2
|
||||||
|
|
||||||
|
|
||||||
|
def test_terminal_pair_reroll_derives_once_and_shares_new_seed():
|
||||||
|
original = _terminal_physical_seed_plan(
|
||||||
|
base_seed=42,
|
||||||
|
physical_window_start_index=2,
|
||||||
|
reroll_nonce=0,
|
||||||
|
)
|
||||||
|
rerolled = _terminal_physical_seed_plan(
|
||||||
|
base_seed=42,
|
||||||
|
physical_window_start_index=2,
|
||||||
|
reroll_nonce=1,
|
||||||
|
)
|
||||||
|
assert rerolled["physical_seed"] == derive_chunk_seed(42, 2, 1)
|
||||||
|
assert rerolled["physical_seed"] == 4306926667627841302
|
||||||
|
assert rerolled["physical_seed"] != original["physical_seed"]
|
||||||
|
assert rerolled["logical_entry_seeds"] == (rerolled["physical_seed"],) * 2
|
||||||
|
|
||||||
|
|
||||||
|
def test_non_terminal_seed_derivation_golden_values_are_unchanged():
|
||||||
|
assert derive_chunk_seed(42, 0, 0) == 15348518480342951850
|
||||||
|
assert derive_chunk_seed(42, 1, 0) == 13021120982267964226
|
||||||
|
|
||||||
|
|
||||||
|
def test_last_frame_absent_does_not_enable_terminal_merge():
|
||||||
|
assert not _terminal_flf_merge_enabled(
|
||||||
|
multi_chunk_flf=False,
|
||||||
|
chunks=2,
|
||||||
|
chunk_seconds=5.0,
|
||||||
|
prompt_hashes=["same", "same"],
|
||||||
|
timeline_video_source=None,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_one_chunk_keeps_normal_sampling_and_seed_behavior():
|
||||||
|
normal, terminal = _terminal_sampling_plan(chunks=1, completed=0, merge_enabled=False)
|
||||||
|
assert normal == (0,)
|
||||||
|
assert terminal is False
|
||||||
|
assert derive_chunk_seed(42, normal[0], 0) == 15348518480342951850
|
||||||
@@ -283,6 +283,7 @@ def _sampling_contract(
|
|||||||
last_frame_hash="", reference_contract=None,
|
last_frame_hash="", reference_contract=None,
|
||||||
upstream_graph_contract=None, upstream_graph_safe=None,
|
upstream_graph_contract=None, upstream_graph_safe=None,
|
||||||
upstream_graph_reasons=None,
|
upstream_graph_reasons=None,
|
||||||
|
execution_semantics=None,
|
||||||
):
|
):
|
||||||
return build_sampling_contract(
|
return build_sampling_contract(
|
||||||
model=model or _Model(),
|
model=model or _Model(),
|
||||||
@@ -312,9 +313,69 @@ def _sampling_contract(
|
|||||||
upstream_graph_contract=upstream_graph_contract,
|
upstream_graph_contract=upstream_graph_contract,
|
||||||
upstream_graph_safe=upstream_graph_safe,
|
upstream_graph_safe=upstream_graph_safe,
|
||||||
upstream_graph_reasons=upstream_graph_reasons,
|
upstream_graph_reasons=upstream_graph_reasons,
|
||||||
|
execution_semantics=execution_semantics,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_terminal_seed_v2_revision_is_scoped_and_distinct():
|
||||||
|
old_contract, old_safe, old_reasons = _sampling_contract(
|
||||||
|
conditioning_mode="fl2va",
|
||||||
|
first_frame_hash="a" * 64,
|
||||||
|
last_frame_hash="b" * 64,
|
||||||
|
)
|
||||||
|
new_contract, new_safe, new_reasons = _sampling_contract(
|
||||||
|
conditioning_mode="fl2va",
|
||||||
|
first_frame_hash="a" * 64,
|
||||||
|
last_frame_hash="b" * 64,
|
||||||
|
execution_semantics={
|
||||||
|
"flf_execution": "terminal_merged_10s_seed_v2",
|
||||||
|
"terminal_seed_policy": "physical_window_seed_v2",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
assert old_safe, old_reasons
|
||||||
|
assert new_safe, new_reasons
|
||||||
|
assert "execution_semantics" not in old_contract["global"]
|
||||||
|
assert new_contract["global"]["execution_semantics"] == {
|
||||||
|
"flf_execution": "terminal_merged_10s_seed_v2",
|
||||||
|
"terminal_seed_policy": "physical_window_seed_v2",
|
||||||
|
}
|
||||||
|
assert revision_identity(old_contract) != revision_identity(new_contract)
|
||||||
|
|
||||||
|
|
||||||
|
def test_terminal_seed_v2_revision_identity_is_deterministic():
|
||||||
|
semantics = {
|
||||||
|
"flf_execution": "terminal_merged_10s_seed_v2",
|
||||||
|
"terminal_seed_policy": "physical_window_seed_v2",
|
||||||
|
}
|
||||||
|
model = _Model()
|
||||||
|
clip = _Clip()
|
||||||
|
video_vae = _VideoVAE()
|
||||||
|
sampler = _Sampler()
|
||||||
|
first, first_safe, first_reasons = _sampling_contract(
|
||||||
|
model=model,
|
||||||
|
clip=clip,
|
||||||
|
video_vae=video_vae,
|
||||||
|
sampler=sampler,
|
||||||
|
conditioning_mode="fl2va",
|
||||||
|
first_frame_hash="a" * 64,
|
||||||
|
last_frame_hash="b" * 64,
|
||||||
|
execution_semantics=semantics,
|
||||||
|
)
|
||||||
|
second, second_safe, second_reasons = _sampling_contract(
|
||||||
|
model=model,
|
||||||
|
clip=clip,
|
||||||
|
video_vae=video_vae,
|
||||||
|
sampler=sampler,
|
||||||
|
conditioning_mode="fl2va",
|
||||||
|
first_frame_hash="a" * 64,
|
||||||
|
last_frame_hash="b" * 64,
|
||||||
|
execution_semantics=semantics,
|
||||||
|
)
|
||||||
|
assert first_safe, first_reasons
|
||||||
|
assert second_safe, second_reasons
|
||||||
|
assert revision_identity(first) == revision_identity(second)
|
||||||
|
|
||||||
|
|
||||||
def test_v322_golden_reference_contract_preserves_revision_identity():
|
def test_v322_golden_reference_contract_preserves_revision_identity():
|
||||||
model = _Model()
|
model = _Model()
|
||||||
clip = _Clip()
|
clip = _Clip()
|
||||||
|
|||||||
@@ -0,0 +1,149 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
import torch
|
||||||
|
|
||||||
|
from ComfyUI_H3_Continuum_Join.v3 import plan as plan_module
|
||||||
|
|
||||||
|
|
||||||
|
def _logical_plan():
|
||||||
|
return {
|
||||||
|
"chunks": [
|
||||||
|
{
|
||||||
|
"sequence_index": 0,
|
||||||
|
"chunk_index": 1,
|
||||||
|
"total_frames": 124,
|
||||||
|
"trim_frames": 0,
|
||||||
|
"net_frames": 124,
|
||||||
|
"context_frames": 0,
|
||||||
|
"expected_video_latent_t": 37,
|
||||||
|
"expected_audio_latent_t": 207,
|
||||||
|
"frame_start": 0,
|
||||||
|
"frame_stop": 124,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"sequence_index": 1,
|
||||||
|
"chunk_index": 2,
|
||||||
|
"total_frames": 141,
|
||||||
|
"trim_frames": 22,
|
||||||
|
"net_frames": 119,
|
||||||
|
"context_frames": 22,
|
||||||
|
"expected_video_latent_t": 42,
|
||||||
|
"expected_audio_latent_t": 235,
|
||||||
|
"frame_start": 124,
|
||||||
|
"frame_stop": 243,
|
||||||
|
},
|
||||||
|
],
|
||||||
|
"target_frames": 240,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _terminal_entries():
|
||||||
|
physical_video = torch.arange(72, dtype=torch.float32).reshape(1, 1, 72, 1, 1)
|
||||||
|
physical_audio = torch.arange(405, dtype=torch.float32).reshape(1, 1, 405)
|
||||||
|
entries = [
|
||||||
|
{
|
||||||
|
"video": physical_video[:, :, 0:37].clone(),
|
||||||
|
"audio": physical_audio[..., 0:207].clone(),
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"video": physical_video[:, :, 30:72].clone(),
|
||||||
|
"audio": physical_audio[..., 170:405].clone(),
|
||||||
|
},
|
||||||
|
]
|
||||||
|
return entries, physical_video, physical_audio
|
||||||
|
|
||||||
|
|
||||||
|
def test_terminal_logical_entries_recombine_bit_exact(monkeypatch):
|
||||||
|
entries, physical_video, physical_audio = _terminal_entries()
|
||||||
|
monkeypatch.setattr(
|
||||||
|
plan_module,
|
||||||
|
"make_assembly_plan",
|
||||||
|
lambda *args, **kwargs: _logical_plan(),
|
||||||
|
)
|
||||||
|
monkeypatch.setattr(plan_module, "validate_assembly_plan", lambda plan: plan)
|
||||||
|
|
||||||
|
decode_entries, assembly_plan = plan_module.prepare_physical_decode_entries(
|
||||||
|
entries,
|
||||||
|
chunk_seconds=5.0,
|
||||||
|
preserve_final_frame=True,
|
||||||
|
terminal_merged=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
assert len(decode_entries) == 1
|
||||||
|
assert torch.equal(decode_entries[0]["video"], physical_video)
|
||||||
|
assert torch.equal(decode_entries[0]["audio"], physical_audio)
|
||||||
|
assert assembly_plan["logical_chunk_count"] == 2
|
||||||
|
assert assembly_plan["physical_decode_group_count"] == 1
|
||||||
|
assert assembly_plan["decode_groups"][0]["logical_chunk_indices"] == [1, 2]
|
||||||
|
assert assembly_plan["decode_groups"][0]["total_frames"] == 243
|
||||||
|
assert assembly_plan["decode_groups"][0]["trim_frames"] == 0
|
||||||
|
assert assembly_plan["decode_groups"][0]["net_frames"] == 243
|
||||||
|
|
||||||
|
|
||||||
|
def test_terminal_recombine_rejects_changed_overlap(monkeypatch):
|
||||||
|
entries, _, _ = _terminal_entries()
|
||||||
|
entries[1]["video"][:, :, 0].add_(1.0)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
plan_module,
|
||||||
|
"make_assembly_plan",
|
||||||
|
lambda *args, **kwargs: _logical_plan(),
|
||||||
|
)
|
||||||
|
|
||||||
|
with pytest.raises(ValueError, match="overlap differs"):
|
||||||
|
plan_module.prepare_physical_decode_entries(
|
||||||
|
entries,
|
||||||
|
chunk_seconds=5.0,
|
||||||
|
preserve_final_frame=True,
|
||||||
|
terminal_merged=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_three_logical_chunks_use_two_physical_decode_groups(monkeypatch):
|
||||||
|
from ComfyUI_H3_Continuum_Join.v2.sequence import (
|
||||||
|
_split_terminal_merged_latents,
|
||||||
|
_terminal_pair_contract,
|
||||||
|
)
|
||||||
|
|
||||||
|
contract = _terminal_pair_contract(initial_pair=False, chunk_seconds=5.0)
|
||||||
|
video_t = contract["video_slices"][-1][1]
|
||||||
|
audio_t = contract["audio_slices"][-1][1]
|
||||||
|
physical_video = torch.arange(video_t, dtype=torch.float32).reshape(1, 1, video_t, 1, 1)
|
||||||
|
physical_audio = torch.arange(audio_t, dtype=torch.float32).reshape(1, 1, audio_t)
|
||||||
|
terminal_parts = _split_terminal_merged_latents(physical_video, physical_audio, contract)
|
||||||
|
first_entry = {
|
||||||
|
"video": torch.zeros((1, 1, 37, 1, 1), dtype=torch.float32),
|
||||||
|
"audio": torch.zeros((1, 1, 207), dtype=torch.float32),
|
||||||
|
}
|
||||||
|
logical_plan = {
|
||||||
|
"chunks": [
|
||||||
|
{"sequence_index": 0, "chunk_index": 1, "total_frames": 124, "trim_frames": 0, "net_frames": 124, "context_frames": 0, "expected_video_latent_t": 37, "expected_audio_latent_t": 207, "frame_start": 0, "frame_stop": 124},
|
||||||
|
{"sequence_index": 1, "chunk_index": 2, "total_frames": 141, "trim_frames": 22, "net_frames": 119, "context_frames": 22, "expected_video_latent_t": 42, "expected_audio_latent_t": 235, "frame_start": 124, "frame_stop": 243},
|
||||||
|
{"sequence_index": 2, "chunk_index": 3, "total_frames": 141, "trim_frames": 22, "net_frames": 119, "context_frames": 22, "expected_video_latent_t": 42, "expected_audio_latent_t": 235, "frame_start": 243, "frame_stop": 362},
|
||||||
|
],
|
||||||
|
"target_frames": 360,
|
||||||
|
}
|
||||||
|
entries = [
|
||||||
|
first_entry,
|
||||||
|
{"video": terminal_parts[0][0], "audio": terminal_parts[0][1]},
|
||||||
|
{"video": terminal_parts[1][0], "audio": terminal_parts[1][1]},
|
||||||
|
]
|
||||||
|
monkeypatch.setattr(plan_module, "make_assembly_plan", lambda *args, **kwargs: logical_plan)
|
||||||
|
monkeypatch.setattr(plan_module, "validate_assembly_plan", lambda plan: plan)
|
||||||
|
|
||||||
|
decode_entries, assembly_plan = plan_module.prepare_physical_decode_entries(
|
||||||
|
entries,
|
||||||
|
chunk_seconds=5.0,
|
||||||
|
preserve_final_frame=True,
|
||||||
|
terminal_merged=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
assert len(decode_entries) == 2
|
||||||
|
assert torch.equal(decode_entries[1]["video"], physical_video)
|
||||||
|
assert torch.equal(decode_entries[1]["audio"], physical_audio)
|
||||||
|
assert assembly_plan["physical_decode_group_count"] == 2
|
||||||
|
terminal_group = assembly_plan["decode_groups"][1]
|
||||||
|
assert terminal_group["logical_chunk_indices"] == [2, 3]
|
||||||
|
assert terminal_group["total_frames"] == 260
|
||||||
|
assert terminal_group["trim_frames"] == 22
|
||||||
|
assert terminal_group["net_frames"] == 238
|
||||||
@@ -0,0 +1,32 @@
|
|||||||
|
# H3 Core vs Continuum GPU Diagnostic
|
||||||
|
|
||||||
|
This is a standalone diagnostic node. It does not change or register through the
|
||||||
|
production H3 Continuum package.
|
||||||
|
|
||||||
|
## Install for ComfyUI_W
|
||||||
|
|
||||||
|
Run from the repository root:
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
.\tools\install_gpu_diagnostic_node.ps1
|
||||||
|
```
|
||||||
|
|
||||||
|
Restart ComfyUI and search for `H3 Core vs Continuum GPU Diagnostic`.
|
||||||
|
|
||||||
|
## Connections
|
||||||
|
|
||||||
|
- Connect the same direct MODEL used by both tests. Keep Spectrum off.
|
||||||
|
- Connect the same CLIP, Video VAE, Audio VAE, sampler, and sigmas.
|
||||||
|
- Connect the exact same First Frame and Last Frame.
|
||||||
|
- Use 640x640, seed 42, 8 steps, Euler/simple for the baseline comparison.
|
||||||
|
- Run Storage and Session are not used by this node.
|
||||||
|
|
||||||
|
One queue executes Core FL2VA at 243 frames, then Continuum Terminal Merge 2x5s.
|
||||||
|
Results are written below:
|
||||||
|
|
||||||
|
```text
|
||||||
|
ComfyUI/output/h3_continuum/diagnostics/<timestamp>/
|
||||||
|
```
|
||||||
|
|
||||||
|
The report identifies the first difference across pre-sampler data, first H3
|
||||||
|
MODEL call, sampled physical latent, split/recombine, and decode/assembly.
|
||||||
@@ -0,0 +1,9 @@
|
|||||||
|
"""Standalone GPU diagnostics for H3 Continuum.
|
||||||
|
|
||||||
|
This package is intentionally separate from the production node package. Copy the
|
||||||
|
directory into ComfyUI/custom_nodes only while diagnostics are needed.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from .nodes import NODE_CLASS_MAPPINGS, NODE_DISPLAY_NAME_MAPPINGS
|
||||||
|
|
||||||
|
__all__ = ["NODE_CLASS_MAPPINGS", "NODE_DISPLAY_NAME_MAPPINGS"]
|
||||||
@@ -0,0 +1,302 @@
|
|||||||
|
"""Deterministic tensor capture and comparison helpers for the GPU runner."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from collections.abc import Mapping, Sequence
|
||||||
|
from dataclasses import asdict, dataclass
|
||||||
|
from hashlib import sha256
|
||||||
|
import json
|
||||||
|
from pathlib import Path
|
||||||
|
import re
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
import torch
|
||||||
|
|
||||||
|
|
||||||
|
STAGE_ORDER = (
|
||||||
|
"A_pre_sampler",
|
||||||
|
"B_first_model_call",
|
||||||
|
"C_physical_sample",
|
||||||
|
"D_split_recombine",
|
||||||
|
"E1_decode_243",
|
||||||
|
"E2_final_240",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _dense_cpu(tensor: torch.Tensor) -> torch.Tensor:
|
||||||
|
return tensor.detach().to("cpu").contiguous()
|
||||||
|
|
||||||
|
|
||||||
|
def _raw_sha256(tensor: torch.Tensor) -> str:
|
||||||
|
value = _dense_cpu(tensor)
|
||||||
|
raw = value.view(torch.uint8).numpy().tobytes()
|
||||||
|
return sha256(raw).hexdigest()
|
||||||
|
|
||||||
|
|
||||||
|
def tensor_fingerprint(tensor: torch.Tensor) -> dict[str, Any]:
|
||||||
|
value = _dense_cpu(tensor)
|
||||||
|
return {
|
||||||
|
"shape": list(value.shape),
|
||||||
|
"dtype": str(value.dtype),
|
||||||
|
"sha256": _raw_sha256(value),
|
||||||
|
"numel": int(value.numel()),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class TensorDifference:
|
||||||
|
comparable: bool
|
||||||
|
bit_exact: bool
|
||||||
|
allclose: bool
|
||||||
|
max_abs_diff: float | None
|
||||||
|
mean_abs_diff: float | None
|
||||||
|
reason: str | None = None
|
||||||
|
|
||||||
|
@property
|
||||||
|
def matches(self) -> bool:
|
||||||
|
return self.comparable and (self.bit_exact or self.allclose)
|
||||||
|
|
||||||
|
def to_dict(self) -> dict[str, Any]:
|
||||||
|
value = asdict(self)
|
||||||
|
value["matches"] = self.matches
|
||||||
|
return value
|
||||||
|
|
||||||
|
|
||||||
|
def compare_tensors(
|
||||||
|
left: torch.Tensor,
|
||||||
|
right: torch.Tensor,
|
||||||
|
*,
|
||||||
|
rtol: float = 1.0e-4,
|
||||||
|
atol: float = 1.0e-5,
|
||||||
|
) -> TensorDifference:
|
||||||
|
a = _dense_cpu(left)
|
||||||
|
b = _dense_cpu(right)
|
||||||
|
if tuple(a.shape) != tuple(b.shape):
|
||||||
|
return TensorDifference(False, False, False, None, None, "shape mismatch")
|
||||||
|
if a.dtype != b.dtype:
|
||||||
|
return TensorDifference(False, False, False, None, None, "dtype mismatch")
|
||||||
|
bit_exact = bool(torch.equal(a, b))
|
||||||
|
if not (a.is_floating_point() or a.is_complex()):
|
||||||
|
return TensorDifference(True, bit_exact, bit_exact, 0.0 if bit_exact else None, 0.0 if bit_exact else None)
|
||||||
|
af = a.to(torch.float64)
|
||||||
|
bf = b.to(torch.float64)
|
||||||
|
delta = (af - bf).abs()
|
||||||
|
max_abs = float(delta.max().item()) if delta.numel() else 0.0
|
||||||
|
mean_abs = float(delta.mean().item()) if delta.numel() else 0.0
|
||||||
|
close = bool(torch.allclose(af, bf, rtol=rtol, atol=atol, equal_nan=True))
|
||||||
|
return TensorDifference(True, bit_exact, close, max_abs, mean_abs)
|
||||||
|
|
||||||
|
|
||||||
|
def _safe_tensor_key(value: str) -> str:
|
||||||
|
return re.sub(r"[^A-Za-z0-9_.-]+", "_", value).strip("_") or "tensor"
|
||||||
|
|
||||||
|
|
||||||
|
def _is_primitive(value: Any) -> bool:
|
||||||
|
return value is None or isinstance(value, (bool, int, float, str))
|
||||||
|
|
||||||
|
|
||||||
|
def _primitive_sequence(value: Any) -> bool:
|
||||||
|
return isinstance(value, (list, tuple)) and all(_is_primitive(item) for item in value)
|
||||||
|
|
||||||
|
|
||||||
|
class DiagnosticCapture:
|
||||||
|
def __init__(self, side: str):
|
||||||
|
self.side = side
|
||||||
|
self.manifest: dict[str, Any] = {
|
||||||
|
"format": "h3-core-continuum-gpu-diagnostic-v1",
|
||||||
|
"side": side,
|
||||||
|
"stages": {},
|
||||||
|
}
|
||||||
|
self.tensors: dict[str, torch.Tensor] = {}
|
||||||
|
self.runtime: dict[str, Any] = {}
|
||||||
|
|
||||||
|
def _stage(self, stage: str) -> dict[str, Any]:
|
||||||
|
return self.manifest["stages"].setdefault(stage, {"values": {}, "tensors": {}})
|
||||||
|
|
||||||
|
def add_value(self, stage: str, field: str, value: Any) -> None:
|
||||||
|
if not (_is_primitive(value) or _primitive_sequence(value)):
|
||||||
|
value = json.loads(json.dumps(value, default=str))
|
||||||
|
self._stage(stage)["values"][field] = value
|
||||||
|
|
||||||
|
def add_tensor(self, stage: str, field: str, tensor: torch.Tensor, *, persist: bool = True) -> None:
|
||||||
|
if not torch.is_tensor(tensor):
|
||||||
|
raise TypeError(f"{field} is not a tensor")
|
||||||
|
if getattr(tensor, "is_nested", False):
|
||||||
|
parts = list(tensor.unbind())
|
||||||
|
self.add_value(stage, f"{field}.components", len(parts))
|
||||||
|
for index, part in enumerate(parts):
|
||||||
|
self.add_tensor(stage, f"{field}.component_{index}", part, persist=persist)
|
||||||
|
return
|
||||||
|
value = _dense_cpu(tensor)
|
||||||
|
key = _safe_tensor_key(f"{stage}.{field}")
|
||||||
|
self._stage(stage)["tensors"][field] = {
|
||||||
|
**tensor_fingerprint(value),
|
||||||
|
"stored_as": key if persist else None,
|
||||||
|
}
|
||||||
|
if persist:
|
||||||
|
self.tensors[key] = value.clone()
|
||||||
|
|
||||||
|
def capture_value(
|
||||||
|
self,
|
||||||
|
stage: str,
|
||||||
|
field: str,
|
||||||
|
value: Any,
|
||||||
|
*,
|
||||||
|
persist_tensors: bool = True,
|
||||||
|
depth: int = 0,
|
||||||
|
) -> None:
|
||||||
|
if depth > 7:
|
||||||
|
self.add_value(stage, f"{field}.truncated", True)
|
||||||
|
return
|
||||||
|
if torch.is_tensor(value):
|
||||||
|
self.add_tensor(stage, field, value, persist=persist_tensors)
|
||||||
|
return
|
||||||
|
if _is_primitive(value):
|
||||||
|
self.add_value(stage, field, value)
|
||||||
|
return
|
||||||
|
if _primitive_sequence(value):
|
||||||
|
self.add_value(stage, field, list(value))
|
||||||
|
return
|
||||||
|
if isinstance(value, Mapping):
|
||||||
|
keys = sorted(str(key) for key in value.keys())
|
||||||
|
self.add_value(stage, f"{field}.keys", keys)
|
||||||
|
for key in keys[:256]:
|
||||||
|
try:
|
||||||
|
child = value[key]
|
||||||
|
except (KeyError, TypeError):
|
||||||
|
child = next((item for raw, item in value.items() if str(raw) == key), None)
|
||||||
|
self.capture_value(
|
||||||
|
stage,
|
||||||
|
f"{field}.{key}",
|
||||||
|
child,
|
||||||
|
persist_tensors=persist_tensors,
|
||||||
|
depth=depth + 1,
|
||||||
|
)
|
||||||
|
return
|
||||||
|
if isinstance(value, Sequence) and not isinstance(value, (str, bytes, bytearray)):
|
||||||
|
self.add_value(stage, f"{field}.length", len(value))
|
||||||
|
for index, child in enumerate(value[:256]):
|
||||||
|
self.capture_value(
|
||||||
|
stage,
|
||||||
|
f"{field}.{index}",
|
||||||
|
child,
|
||||||
|
persist_tensors=persist_tensors,
|
||||||
|
depth=depth + 1,
|
||||||
|
)
|
||||||
|
return
|
||||||
|
if hasattr(value, "unbind") and value.__class__.__name__ == "NestedTensor":
|
||||||
|
parts = list(value.unbind())
|
||||||
|
self.add_value(stage, f"{field}.components", len(parts))
|
||||||
|
for index, child in enumerate(parts):
|
||||||
|
label = ("video", "audio")[index] if index < 2 else f"component_{index}"
|
||||||
|
self.capture_value(
|
||||||
|
stage,
|
||||||
|
f"{field}.{label}",
|
||||||
|
child,
|
||||||
|
persist_tensors=persist_tensors,
|
||||||
|
depth=depth + 1,
|
||||||
|
)
|
||||||
|
return
|
||||||
|
if all(hasattr(value, name) for name in ("signature", "segments", "position_ids")):
|
||||||
|
self.add_value(stage, f"{field}.type", value.__class__.__name__)
|
||||||
|
self.add_value(stage, f"{field}.signature", list(value.signature))
|
||||||
|
self.add_value(stage, f"{field}.segments", [list(item) for item in value.segments])
|
||||||
|
for name in ("position_ids", "img_pos", "img_update", "audio_pos", "audio_update"):
|
||||||
|
child = getattr(value, name, None)
|
||||||
|
if torch.is_tensor(child):
|
||||||
|
self.add_tensor(stage, f"{field}.{name}", child, persist=persist_tensors)
|
||||||
|
return
|
||||||
|
self.add_value(stage, f"{field}.type", value.__class__.__name__)
|
||||||
|
|
||||||
|
def save(self, directory: Path) -> None:
|
||||||
|
directory.mkdir(parents=True, exist_ok=True)
|
||||||
|
from safetensors.torch import save_file
|
||||||
|
|
||||||
|
save_file(self.tensors, str(directory / "tensors.safetensors"))
|
||||||
|
(directory / "manifest.json").write_text(
|
||||||
|
json.dumps(self.manifest, indent=2, ensure_ascii=False),
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def compare_stage(
|
||||||
|
left: DiagnosticCapture,
|
||||||
|
right: DiagnosticCapture,
|
||||||
|
stage: str,
|
||||||
|
) -> dict[str, Any]:
|
||||||
|
a = left.manifest["stages"].get(stage, {"values": {}, "tensors": {}})
|
||||||
|
b = right.manifest["stages"].get(stage, {"values": {}, "tensors": {}})
|
||||||
|
details: dict[str, Any] = {}
|
||||||
|
first_field: str | None = None
|
||||||
|
|
||||||
|
value_fields = list(a["values"].keys()) + [key for key in b["values"] if key not in a["values"]]
|
||||||
|
for field in value_fields:
|
||||||
|
av = a["values"].get(field, {"missing": True})
|
||||||
|
bv = b["values"].get(field, {"missing": True})
|
||||||
|
matches = av == bv
|
||||||
|
details[field] = {"kind": "value", "matches": matches, "left": av, "right": bv}
|
||||||
|
if not matches and first_field is None:
|
||||||
|
first_field = field
|
||||||
|
|
||||||
|
tensor_fields = list(a["tensors"].keys()) + [key for key in b["tensors"] if key not in a["tensors"]]
|
||||||
|
for field in tensor_fields:
|
||||||
|
a_meta = a["tensors"].get(field)
|
||||||
|
b_meta = b["tensors"].get(field)
|
||||||
|
if a_meta is None or b_meta is None:
|
||||||
|
result = {"kind": "tensor", "matches": False, "reason": "missing tensor"}
|
||||||
|
else:
|
||||||
|
a_key = a_meta.get("stored_as")
|
||||||
|
b_key = b_meta.get("stored_as")
|
||||||
|
if a_key in left.tensors and b_key in right.tensors:
|
||||||
|
result = {"kind": "tensor", **compare_tensors(left.tensors[a_key], right.tensors[b_key]).to_dict()}
|
||||||
|
else:
|
||||||
|
matches = a_meta == b_meta
|
||||||
|
result = {"kind": "fingerprint", "matches": matches, "left": a_meta, "right": b_meta}
|
||||||
|
details[field] = result
|
||||||
|
if not result["matches"] and first_field is None:
|
||||||
|
first_field = field
|
||||||
|
|
||||||
|
return {
|
||||||
|
"status": "MATCH" if first_field is None else "DIFFER",
|
||||||
|
"first_field": first_field,
|
||||||
|
"fields": details,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def recombine_terminal_streams(
|
||||||
|
first: torch.Tensor,
|
||||||
|
second: torch.Tensor,
|
||||||
|
*,
|
||||||
|
first_range: tuple[int, int] | list[int],
|
||||||
|
second_range: tuple[int, int] | list[int],
|
||||||
|
dim: int,
|
||||||
|
) -> torch.Tensor:
|
||||||
|
first_start, first_stop = (int(value) for value in first_range)
|
||||||
|
second_start, second_stop = (int(value) for value in second_range)
|
||||||
|
overlap = first_stop - second_start
|
||||||
|
if first_start != 0 or overlap < 0:
|
||||||
|
raise ValueError("terminal split ranges do not form an overlapping sequence")
|
||||||
|
if first.shape[dim] != first_stop - first_start:
|
||||||
|
raise ValueError("first logical stream length does not match its range")
|
||||||
|
if second.shape[dim] != second_stop - second_start:
|
||||||
|
raise ValueError("second logical stream length does not match its range")
|
||||||
|
if overlap > second.shape[dim]:
|
||||||
|
raise ValueError("terminal overlap exceeds the second logical stream")
|
||||||
|
slices = [slice(None)] * second.ndim
|
||||||
|
slices[dim] = slice(overlap, None)
|
||||||
|
return torch.cat((first, second[tuple(slices)]), dim=dim).contiguous()
|
||||||
|
|
||||||
|
|
||||||
|
def format_comparison(comparison: Mapping[str, Any], output_directory: Path) -> str:
|
||||||
|
lines = []
|
||||||
|
first_divergence = None
|
||||||
|
for stage in STAGE_ORDER:
|
||||||
|
result = comparison.get(stage, {"status": "NOT_RUN", "first_field": None})
|
||||||
|
lines.append(f"{stage}: {result['status']}")
|
||||||
|
if first_divergence is None and result["status"] == "DIFFER":
|
||||||
|
field = result.get("first_field")
|
||||||
|
first_divergence = f"{stage}.{field}" if field else stage
|
||||||
|
lines.append("")
|
||||||
|
lines.append(f"First divergence: {first_divergence or 'none'}")
|
||||||
|
lines.append(f"Output: {output_directory}")
|
||||||
|
return "\n".join(lines)
|
||||||
@@ -0,0 +1,715 @@
|
|||||||
|
"""Standalone ComfyUI node for Core-vs-Continuum GPU diagnostics."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from contextlib import contextmanager
|
||||||
|
from datetime import datetime
|
||||||
|
import gc
|
||||||
|
import importlib
|
||||||
|
import json
|
||||||
|
from pathlib import Path, PurePosixPath
|
||||||
|
import random
|
||||||
|
import sys
|
||||||
|
import traceback
|
||||||
|
from types import SimpleNamespace
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
import torch
|
||||||
|
|
||||||
|
from .capture import (
|
||||||
|
DiagnosticCapture,
|
||||||
|
STAGE_ORDER,
|
||||||
|
compare_stage,
|
||||||
|
compare_tensors,
|
||||||
|
format_comparison,
|
||||||
|
recombine_terminal_streams,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _resolve_loaded_module(path: Path):
|
||||||
|
target = path.resolve()
|
||||||
|
for module in tuple(sys.modules.values()):
|
||||||
|
source = getattr(module, "__file__", None)
|
||||||
|
if source and Path(source).resolve() == target:
|
||||||
|
return module
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def _resolve_continuum_modules() -> SimpleNamespace:
|
||||||
|
import nodes as comfy_nodes
|
||||||
|
|
||||||
|
sampler_class = comfy_nodes.NODE_CLASS_MAPPINGS.get("H3ContinuumSamplerV34")
|
||||||
|
if sampler_class is None:
|
||||||
|
raise RuntimeError("H3ContinuumSamplerV34 is not loaded; install and enable H3 Continuum V3.4 first")
|
||||||
|
owner = sys.modules.get(sampler_class.__module__)
|
||||||
|
source = getattr(owner, "__file__", None)
|
||||||
|
if source is None:
|
||||||
|
raise RuntimeError("cannot resolve the active H3 Continuum package")
|
||||||
|
root = Path(source).resolve().parent.parent
|
||||||
|
module_name = sampler_class.__module__
|
||||||
|
base_name = module_name.split(".v3.", 1)[0] if ".v3." in module_name else module_name.rsplit(".", 2)[0]
|
||||||
|
|
||||||
|
def load(relative: str, suffix: str):
|
||||||
|
target = root / relative
|
||||||
|
loaded = _resolve_loaded_module(target)
|
||||||
|
if loaded is not None:
|
||||||
|
return loaded
|
||||||
|
return importlib.import_module(f"{base_name}.{suffix}")
|
||||||
|
|
||||||
|
return SimpleNamespace(
|
||||||
|
root=root,
|
||||||
|
constants=load("constants.py", "constants"),
|
||||||
|
state=load("state.py", "state"),
|
||||||
|
sequence=load("v2/sequence.py", "v2.sequence"),
|
||||||
|
sampling=load("v2/sampling.py", "v2.sampling"),
|
||||||
|
prompts=load("v2/prompts.py", "v2.prompts"),
|
||||||
|
decoder=load("v2/decoder.py", "v2.decoder"),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _reset_rng(seed: int) -> None:
|
||||||
|
random.seed(seed)
|
||||||
|
np.random.seed(seed & 0xFFFFFFFF)
|
||||||
|
torch.manual_seed(seed)
|
||||||
|
if torch.cuda.is_available():
|
||||||
|
torch.cuda.manual_seed_all(seed)
|
||||||
|
|
||||||
|
|
||||||
|
def _release_cuda() -> None:
|
||||||
|
gc.collect()
|
||||||
|
if torch.cuda.is_available():
|
||||||
|
torch.cuda.empty_cache()
|
||||||
|
|
||||||
|
|
||||||
|
def _safe_output_directory(subdir: str) -> Path:
|
||||||
|
import folder_paths
|
||||||
|
|
||||||
|
clean = str(subdir).replace("\\", "/").strip("/") or "h3_continuum/diagnostics"
|
||||||
|
parts = PurePosixPath(clean).parts
|
||||||
|
if any(part in ("", ".", "..") for part in parts):
|
||||||
|
raise ValueError("output_subdir must stay below the ComfyUI output directory")
|
||||||
|
stamp = datetime.now().strftime("%Y%m%d_%H%M%S_%f")
|
||||||
|
return Path(folder_paths.get_output_directory()).joinpath(*parts, stamp)
|
||||||
|
|
||||||
|
|
||||||
|
def _capture_av_value(capture: DiagnosticCapture, stage: str, prefix: str, value: Any) -> None:
|
||||||
|
candidate = value.get("samples") if isinstance(value, dict) and "samples" in value else value
|
||||||
|
if hasattr(candidate, "unbind") and candidate.__class__.__name__ == "NestedTensor":
|
||||||
|
parts = list(candidate.unbind())
|
||||||
|
capture.add_value(stage, f"{prefix}.components", len(parts))
|
||||||
|
for index, part in enumerate(parts):
|
||||||
|
label = ("video", "audio")[index] if index < 2 else f"component_{index}"
|
||||||
|
capture.add_tensor(stage, f"{prefix}.{label}", part)
|
||||||
|
return
|
||||||
|
if torch.is_tensor(candidate) and getattr(candidate, "is_nested", False):
|
||||||
|
parts = list(candidate.unbind())
|
||||||
|
capture.add_value(stage, f"{prefix}.components", len(parts))
|
||||||
|
for index, part in enumerate(parts):
|
||||||
|
label = ("video", "audio")[index] if index < 2 else f"component_{index}"
|
||||||
|
capture.add_tensor(stage, f"{prefix}.{label}", part)
|
||||||
|
return
|
||||||
|
capture.capture_value(stage, prefix, candidate)
|
||||||
|
|
||||||
|
|
||||||
|
def _capture_pre_sampler(
|
||||||
|
capture: DiagnosticCapture,
|
||||||
|
modules: SimpleNamespace,
|
||||||
|
*,
|
||||||
|
conditioning: Any,
|
||||||
|
latent: dict[str, Any],
|
||||||
|
sigmas: torch.Tensor,
|
||||||
|
seed: int,
|
||||||
|
) -> None:
|
||||||
|
stage = "A_pre_sampler"
|
||||||
|
capture.add_value(stage, "seed", int(seed))
|
||||||
|
capture.add_value(stage, "presentation_order", ["First Frame", "Last Frame"])
|
||||||
|
capture.add_tensor(stage, "sigmas", sigmas)
|
||||||
|
video, audio = modules.state.extract_av_streams(latent)
|
||||||
|
capture.add_tensor(stage, "target.video", video)
|
||||||
|
capture.add_tensor(stage, "target.audio", audio)
|
||||||
|
capture.add_value(stage, "target.video_shape", list(video.shape))
|
||||||
|
capture.add_value(stage, "target.audio_shape", list(audio.shape))
|
||||||
|
|
||||||
|
if not conditioning:
|
||||||
|
capture.add_value(stage, "conditioning.empty", True)
|
||||||
|
return
|
||||||
|
first = conditioning[0]
|
||||||
|
cond_tensor = first[0]
|
||||||
|
metadata = first[1] if len(first) > 1 and isinstance(first[1], dict) else {}
|
||||||
|
capture.capture_value(stage, "conditioning.positive", cond_tensor)
|
||||||
|
capture.add_value(stage, "conditioning.metadata_keys", sorted(str(key) for key in metadata))
|
||||||
|
capture.add_value(stage, "physical_frames", int(metadata.get("minimax_frame_count", 243)))
|
||||||
|
for key in ("minimax_token_tags", "minimax_keyframes", "minimax_refs", "minimax_frame_count"):
|
||||||
|
if key in metadata:
|
||||||
|
capture.capture_value(stage, f"conditioning.{key}", metadata[key])
|
||||||
|
|
||||||
|
|
||||||
|
def _capture_physical_sample(
|
||||||
|
capture: DiagnosticCapture,
|
||||||
|
modules: SimpleNamespace,
|
||||||
|
result: dict[str, Any],
|
||||||
|
) -> tuple[torch.Tensor, torch.Tensor]:
|
||||||
|
video, audio = modules.state.extract_av_streams(result)
|
||||||
|
video_cpu = video.detach().to("cpu").contiguous()
|
||||||
|
audio_cpu = audio.detach().to("cpu").contiguous()
|
||||||
|
capture.add_tensor("C_physical_sample", "video", video_cpu)
|
||||||
|
capture.add_tensor("C_physical_sample", "audio", audio_cpu)
|
||||||
|
capture.runtime["physical_streams"] = (video_cpu, audio_cpu)
|
||||||
|
return video_cpu, audio_cpu
|
||||||
|
|
||||||
|
|
||||||
|
@contextmanager
|
||||||
|
def _runtime_capture(capture: DiagnosticCapture):
|
||||||
|
import comfy.model_base
|
||||||
|
import comfy.sample
|
||||||
|
|
||||||
|
original_noise = comfy.sample.prepare_noise
|
||||||
|
original_apply = comfy.model_base.BaseModel._apply_model
|
||||||
|
noise_seen = False
|
||||||
|
model_seen = False
|
||||||
|
|
||||||
|
def prepare_noise(*args, **kwargs):
|
||||||
|
nonlocal noise_seen
|
||||||
|
result = original_noise(*args, **kwargs)
|
||||||
|
if not noise_seen:
|
||||||
|
noise_seen = True
|
||||||
|
_capture_av_value(capture, "A_pre_sampler", "actual_noise", result)
|
||||||
|
return result
|
||||||
|
|
||||||
|
def apply_model(
|
||||||
|
self,
|
||||||
|
x,
|
||||||
|
t,
|
||||||
|
c_concat=None,
|
||||||
|
c_crossattn=None,
|
||||||
|
control=None,
|
||||||
|
transformer_options={},
|
||||||
|
**kwargs,
|
||||||
|
):
|
||||||
|
nonlocal model_seen
|
||||||
|
if not model_seen and self.__class__.__name__ == "MiniMaxH3":
|
||||||
|
model_seen = True
|
||||||
|
capture.capture_value("B_first_model_call", "input_x", x)
|
||||||
|
capture.capture_value("B_first_model_call", "timestep", t)
|
||||||
|
capture.capture_value(
|
||||||
|
"B_first_model_call",
|
||||||
|
"transformer_options",
|
||||||
|
transformer_options,
|
||||||
|
persist_tensors=False,
|
||||||
|
)
|
||||||
|
capture.capture_value("B_first_model_call", "minimax_payload", kwargs.get("minimax_payload"))
|
||||||
|
capture.capture_value("B_first_model_call", "latent_shapes", kwargs.get("latent_shapes"))
|
||||||
|
return original_apply(
|
||||||
|
self,
|
||||||
|
x,
|
||||||
|
t,
|
||||||
|
c_concat,
|
||||||
|
c_crossattn,
|
||||||
|
control,
|
||||||
|
transformer_options,
|
||||||
|
**kwargs,
|
||||||
|
)
|
||||||
|
|
||||||
|
comfy.sample.prepare_noise = prepare_noise
|
||||||
|
comfy.model_base.BaseModel._apply_model = apply_model
|
||||||
|
try:
|
||||||
|
yield
|
||||||
|
finally:
|
||||||
|
comfy.sample.prepare_noise = original_noise
|
||||||
|
comfy.model_base.BaseModel._apply_model = original_apply
|
||||||
|
restored = (
|
||||||
|
comfy.sample.prepare_noise is original_noise
|
||||||
|
and comfy.model_base.BaseModel._apply_model is original_apply
|
||||||
|
)
|
||||||
|
capture.add_value("B_first_model_call", "patches_restored", restored)
|
||||||
|
capture.add_value("A_pre_sampler", "noise_captured", noise_seen)
|
||||||
|
capture.add_value("B_first_model_call", "model_call_captured", model_seen)
|
||||||
|
|
||||||
|
|
||||||
|
def _run_core(
|
||||||
|
capture: DiagnosticCapture,
|
||||||
|
modules: SimpleNamespace,
|
||||||
|
*,
|
||||||
|
model: Any,
|
||||||
|
clip: Any,
|
||||||
|
video_vae: Any,
|
||||||
|
sampler: Any,
|
||||||
|
sigmas: torch.Tensor,
|
||||||
|
first_frame: torch.Tensor,
|
||||||
|
last_frame: torch.Tensor,
|
||||||
|
prompt: str,
|
||||||
|
width: int,
|
||||||
|
height: int,
|
||||||
|
seed: int,
|
||||||
|
) -> None:
|
||||||
|
from comfy_extras.nodes_minimax_h3 import MiniMaxH3ImageToVideo
|
||||||
|
|
||||||
|
core_model = model.clone()
|
||||||
|
output = MiniMaxH3ImageToVideo().execute(
|
||||||
|
clip,
|
||||||
|
video_vae,
|
||||||
|
prompt,
|
||||||
|
int(width),
|
||||||
|
int(height),
|
||||||
|
243,
|
||||||
|
first_frame=first_frame,
|
||||||
|
last_frame=last_frame,
|
||||||
|
)
|
||||||
|
conditioning, latent = output[0], output[1]
|
||||||
|
_capture_pre_sampler(
|
||||||
|
capture,
|
||||||
|
modules,
|
||||||
|
conditioning=conditioning,
|
||||||
|
latent=latent,
|
||||||
|
sigmas=sigmas,
|
||||||
|
seed=seed,
|
||||||
|
)
|
||||||
|
with _runtime_capture(capture):
|
||||||
|
result = modules.sampling.sample_chunk(
|
||||||
|
model=core_model,
|
||||||
|
conditioning=conditioning,
|
||||||
|
latent=latent,
|
||||||
|
sampler=sampler,
|
||||||
|
sigmas=sigmas,
|
||||||
|
seed=int(seed),
|
||||||
|
enable_preview=False,
|
||||||
|
)
|
||||||
|
_capture_physical_sample(capture, modules, result)
|
||||||
|
capture.add_value("C_physical_sample", "sampling_passes", 1)
|
||||||
|
capture.add_value("C_physical_sample", "physical_seed", int(seed))
|
||||||
|
del result, latent, conditioning, core_model
|
||||||
|
|
||||||
|
|
||||||
|
def _capture_terminal_split(
|
||||||
|
capture: DiagnosticCapture,
|
||||||
|
video: torch.Tensor,
|
||||||
|
audio: torch.Tensor,
|
||||||
|
contract: dict[str, Any],
|
||||||
|
result: Any,
|
||||||
|
) -> None:
|
||||||
|
(video_1, audio_1), (video_2, audio_2) = result
|
||||||
|
video_ranges = contract["video_slices"]
|
||||||
|
audio_ranges = contract["audio_slices"]
|
||||||
|
video_recombined = recombine_terminal_streams(
|
||||||
|
video_1,
|
||||||
|
video_2,
|
||||||
|
first_range=video_ranges[0],
|
||||||
|
second_range=video_ranges[1],
|
||||||
|
dim=2,
|
||||||
|
)
|
||||||
|
audio_recombined = recombine_terminal_streams(
|
||||||
|
audio_1,
|
||||||
|
audio_2,
|
||||||
|
first_range=audio_ranges[0],
|
||||||
|
second_range=audio_ranges[1],
|
||||||
|
dim=-1,
|
||||||
|
)
|
||||||
|
video_diff = compare_tensors(video.detach().to("cpu"), video_recombined.detach().to("cpu"))
|
||||||
|
audio_diff = compare_tensors(audio.detach().to("cpu"), audio_recombined.detach().to("cpu"))
|
||||||
|
stage = "D_split_recombine"
|
||||||
|
capture.add_value(stage, "contract", contract)
|
||||||
|
capture.add_tensor(stage, "video.physical", video)
|
||||||
|
capture.add_tensor(stage, "video.chunk_1", video_1)
|
||||||
|
capture.add_tensor(stage, "video.chunk_2", video_2)
|
||||||
|
capture.add_tensor(stage, "video.recombined", video_recombined)
|
||||||
|
capture.add_tensor(stage, "audio.physical", audio)
|
||||||
|
capture.add_tensor(stage, "audio.chunk_1", audio_1)
|
||||||
|
capture.add_tensor(stage, "audio.chunk_2", audio_2)
|
||||||
|
capture.add_tensor(stage, "audio.recombined", audio_recombined)
|
||||||
|
capture.add_value(stage, "video_difference", video_diff.to_dict())
|
||||||
|
capture.add_value(stage, "audio_difference", audio_diff.to_dict())
|
||||||
|
capture.runtime["logical_streams"] = (
|
||||||
|
(video_1.detach().to("cpu").contiguous(), audio_1.detach().to("cpu").contiguous()),
|
||||||
|
(video_2.detach().to("cpu").contiguous(), audio_2.detach().to("cpu").contiguous()),
|
||||||
|
)
|
||||||
|
capture.runtime["terminal_contract"] = contract
|
||||||
|
|
||||||
|
|
||||||
|
def _run_continuum(
|
||||||
|
capture: DiagnosticCapture,
|
||||||
|
modules: SimpleNamespace,
|
||||||
|
*,
|
||||||
|
model: Any,
|
||||||
|
clip: Any,
|
||||||
|
video_vae: Any,
|
||||||
|
audio_vae: Any,
|
||||||
|
sampler: Any,
|
||||||
|
sigmas: torch.Tensor,
|
||||||
|
first_frame: torch.Tensor,
|
||||||
|
last_frame: torch.Tensor,
|
||||||
|
prompt: str,
|
||||||
|
width: int,
|
||||||
|
height: int,
|
||||||
|
seed: int,
|
||||||
|
) -> None:
|
||||||
|
sequence = modules.sequence
|
||||||
|
original_sample = sequence.sample_chunk
|
||||||
|
original_split = sequence._split_terminal_merged_latents
|
||||||
|
sample_calls = 0
|
||||||
|
split_calls = 0
|
||||||
|
|
||||||
|
def sample_chunk(**kwargs):
|
||||||
|
nonlocal sample_calls
|
||||||
|
sample_calls += 1
|
||||||
|
if sample_calls == 1:
|
||||||
|
_capture_pre_sampler(
|
||||||
|
capture,
|
||||||
|
modules,
|
||||||
|
conditioning=kwargs["conditioning"],
|
||||||
|
latent=kwargs["latent"],
|
||||||
|
sigmas=kwargs["sigmas"],
|
||||||
|
seed=int(kwargs["seed"]),
|
||||||
|
)
|
||||||
|
result = original_sample(**kwargs)
|
||||||
|
if sample_calls == 1:
|
||||||
|
_capture_physical_sample(capture, modules, result)
|
||||||
|
capture.add_value("C_physical_sample", "physical_seed", int(kwargs["seed"]))
|
||||||
|
return result
|
||||||
|
|
||||||
|
def split_terminal(video, audio, contract):
|
||||||
|
nonlocal split_calls
|
||||||
|
split_calls += 1
|
||||||
|
result = original_split(video, audio, contract)
|
||||||
|
if split_calls == 1:
|
||||||
|
_capture_terminal_split(capture, video, audio, contract, result)
|
||||||
|
return result
|
||||||
|
|
||||||
|
sequence.sample_chunk = sample_chunk
|
||||||
|
sequence._split_terminal_merged_latents = split_terminal
|
||||||
|
try:
|
||||||
|
prompt_plan = modules.prompts.make_prompt_plan(
|
||||||
|
mode=modules.constants.PROMPT_MODE_FIXED,
|
||||||
|
script=prompt,
|
||||||
|
chunks=2,
|
||||||
|
chunk_seconds=5.0,
|
||||||
|
)
|
||||||
|
continuum_model = model.clone()
|
||||||
|
with _runtime_capture(capture):
|
||||||
|
sequence.run_sequence(
|
||||||
|
model=continuum_model,
|
||||||
|
clip=clip,
|
||||||
|
video_vae=video_vae,
|
||||||
|
audio_vae=audio_vae,
|
||||||
|
sampler=sampler,
|
||||||
|
sigmas=sigmas,
|
||||||
|
first_frame=first_frame,
|
||||||
|
last_frame=last_frame,
|
||||||
|
prompt_plan=prompt_plan,
|
||||||
|
width=int(width),
|
||||||
|
height=int(height),
|
||||||
|
continuity=modules.constants.CONTINUITY_OPTIONS[0],
|
||||||
|
base_seed=int(seed),
|
||||||
|
audio_continuity=True,
|
||||||
|
exact_total_duration=True,
|
||||||
|
diagnostics_mode=modules.constants.DIAGNOSTICS_BASIC,
|
||||||
|
reroll_from_chunk=0,
|
||||||
|
reroll_nonce=0,
|
||||||
|
strict_compatibility=False,
|
||||||
|
debug=False,
|
||||||
|
seam_correction=modules.constants.SEAM_CORRECTION_OFF,
|
||||||
|
enable_preview=False,
|
||||||
|
session=None,
|
||||||
|
initial_state=None,
|
||||||
|
latent_only=True,
|
||||||
|
reference_assets=None,
|
||||||
|
reference_audio_source=None,
|
||||||
|
reference_audio_vae=None,
|
||||||
|
driving_audio_source=None,
|
||||||
|
driving_audio_vae=None,
|
||||||
|
reference_video_source=None,
|
||||||
|
timeline_video_source=None,
|
||||||
|
)
|
||||||
|
capture.add_value("C_physical_sample", "sampling_passes", sample_calls)
|
||||||
|
capture.add_value("D_split_recombine", "split_calls", split_calls)
|
||||||
|
del continuum_model
|
||||||
|
finally:
|
||||||
|
sequence.sample_chunk = original_sample
|
||||||
|
sequence._split_terminal_merged_latents = original_split
|
||||||
|
capture.add_value(
|
||||||
|
"D_split_recombine",
|
||||||
|
"sequence_patches_restored",
|
||||||
|
sequence.sample_chunk is original_sample
|
||||||
|
and sequence._split_terminal_merged_latents is original_split,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _make_av_latent(video: torch.Tensor, audio: torch.Tensor) -> dict[str, Any]:
|
||||||
|
from comfy.nested_tensor import NestedTensor
|
||||||
|
|
||||||
|
return {"samples": NestedTensor([video, audio])}
|
||||||
|
|
||||||
|
|
||||||
|
def _copy_audio(audio: dict[str, Any]) -> dict[str, Any]:
|
||||||
|
return {
|
||||||
|
"waveform": audio["waveform"].clone(),
|
||||||
|
"sample_rate": int(audio["sample_rate"]),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _run_decode_diagnostics(capture: DiagnosticCapture, modules: SimpleNamespace, video_vae: Any, audio_vae: Any) -> dict[str, Any]:
|
||||||
|
physical = capture.runtime.get("physical_streams")
|
||||||
|
logical = capture.runtime.get("logical_streams")
|
||||||
|
contract = capture.runtime.get("terminal_contract")
|
||||||
|
if physical is None or logical is None or contract is None:
|
||||||
|
return {
|
||||||
|
"E1_decode_243": {"status": "NOT_RUN", "first_field": "missing terminal capture"},
|
||||||
|
"E2_final_240": {"status": "NOT_RUN", "first_field": "missing terminal capture"},
|
||||||
|
}
|
||||||
|
|
||||||
|
decoder = modules.decoder
|
||||||
|
physical_video, physical_audio = physical
|
||||||
|
direct_latent = _make_av_latent(physical_video, physical_audio)
|
||||||
|
direct_images = decoder.decode_video(video_vae, direct_latent)
|
||||||
|
direct_audio = decoder.decode_audio(audio_vae, direct_latent)
|
||||||
|
|
||||||
|
decoded_images = []
|
||||||
|
decoded_audio = []
|
||||||
|
for video, audio in logical:
|
||||||
|
latent = _make_av_latent(video, audio)
|
||||||
|
decoded_images.append(decoder.decode_video(video_vae, latent))
|
||||||
|
decoded_audio.append(decoder.decode_audio(audio_vae, latent))
|
||||||
|
|
||||||
|
trim_frames = 22
|
||||||
|
reconstructed_images = torch.cat(
|
||||||
|
(decoded_images[0][:124], decoded_images[1][:141][trim_frames:]),
|
||||||
|
dim=0,
|
||||||
|
).contiguous()
|
||||||
|
sample_rate = int(decoded_audio[0]["sample_rate"])
|
||||||
|
if int(decoded_audio[1]["sample_rate"]) != sample_rate:
|
||||||
|
raise ValueError("chunk audio sample rates differ during decode diagnostics")
|
||||||
|
total_samples = int(round(243 / 24.0 * sample_rate))
|
||||||
|
reconstructed_waveform = torch.empty(
|
||||||
|
(*decoded_audio[0]["waveform"].shape[:-1], total_samples),
|
||||||
|
dtype=decoded_audio[0]["waveform"].dtype,
|
||||||
|
device="cpu",
|
||||||
|
)
|
||||||
|
segment_1 = decoder._slice_audio_for_timeline(
|
||||||
|
decoded_audio[0], trim_frames=0, frame_start=0, frame_stop=124
|
||||||
|
)
|
||||||
|
segment_2 = decoder._slice_audio_for_timeline(
|
||||||
|
decoded_audio[1], trim_frames=trim_frames, frame_start=124, frame_stop=243
|
||||||
|
)
|
||||||
|
split_sample = int(round(124 / 24.0 * sample_rate))
|
||||||
|
reconstructed_waveform[..., :split_sample].copy_(segment_1)
|
||||||
|
reconstructed_waveform[..., split_sample:].copy_(segment_2)
|
||||||
|
reconstructed_audio = {"waveform": reconstructed_waveform, "sample_rate": sample_rate}
|
||||||
|
|
||||||
|
direct_images, direct_audio, _ = decoder.enforce_total_frames(
|
||||||
|
direct_images,
|
||||||
|
direct_audio,
|
||||||
|
target_frames=243,
|
||||||
|
preserve_final_frame=False,
|
||||||
|
)
|
||||||
|
reconstructed_images, reconstructed_audio, _ = decoder.enforce_total_frames(
|
||||||
|
reconstructed_images,
|
||||||
|
reconstructed_audio,
|
||||||
|
target_frames=243,
|
||||||
|
preserve_final_frame=False,
|
||||||
|
)
|
||||||
|
|
||||||
|
image_e1 = compare_tensors(direct_images, reconstructed_images)
|
||||||
|
audio_e1 = compare_tensors(direct_audio["waveform"], reconstructed_audio["waveform"])
|
||||||
|
stage = "E1_decode_243"
|
||||||
|
capture.add_tensor(stage, "direct.images", direct_images, persist=False)
|
||||||
|
capture.add_tensor(stage, "split.images", reconstructed_images, persist=False)
|
||||||
|
capture.add_tensor(stage, "direct.audio", direct_audio["waveform"], persist=False)
|
||||||
|
capture.add_tensor(stage, "split.audio", reconstructed_audio["waveform"], persist=False)
|
||||||
|
capture.add_value(stage, "images_difference", image_e1.to_dict())
|
||||||
|
capture.add_value(stage, "audio_difference", audio_e1.to_dict())
|
||||||
|
boundary = slice(112, 129)
|
||||||
|
capture.add_tensor(stage, "direct.boundary_images", direct_images[boundary])
|
||||||
|
capture.add_tensor(stage, "split.boundary_images", reconstructed_images[boundary])
|
||||||
|
boundary_diff = compare_tensors(direct_images[boundary], reconstructed_images[boundary])
|
||||||
|
capture.add_value(stage, "boundary_images_difference", boundary_diff.to_dict())
|
||||||
|
|
||||||
|
direct_240_images, direct_240_audio, _ = decoder.enforce_total_frames(
|
||||||
|
direct_images,
|
||||||
|
_copy_audio(direct_audio),
|
||||||
|
target_frames=240,
|
||||||
|
preserve_final_frame=True,
|
||||||
|
)
|
||||||
|
split_240_images, split_240_audio, _ = decoder.enforce_total_frames(
|
||||||
|
reconstructed_images,
|
||||||
|
_copy_audio(reconstructed_audio),
|
||||||
|
target_frames=240,
|
||||||
|
preserve_final_frame=True,
|
||||||
|
)
|
||||||
|
image_e2 = compare_tensors(direct_240_images, split_240_images)
|
||||||
|
audio_e2 = compare_tensors(direct_240_audio["waveform"], split_240_audio["waveform"])
|
||||||
|
stage = "E2_final_240"
|
||||||
|
capture.add_tensor(stage, "direct.images", direct_240_images, persist=False)
|
||||||
|
capture.add_tensor(stage, "split.images", split_240_images, persist=False)
|
||||||
|
capture.add_tensor(stage, "direct.audio", direct_240_audio["waveform"], persist=False)
|
||||||
|
capture.add_tensor(stage, "split.audio", split_240_audio["waveform"], persist=False)
|
||||||
|
capture.add_value(stage, "images_difference", image_e2.to_dict())
|
||||||
|
capture.add_value(stage, "audio_difference", audio_e2.to_dict())
|
||||||
|
capture.add_tensor(stage, "direct.boundary_images", direct_240_images[boundary])
|
||||||
|
capture.add_tensor(stage, "split.boundary_images", split_240_images[boundary])
|
||||||
|
|
||||||
|
return {
|
||||||
|
"E1_decode_243": {
|
||||||
|
"status": "MATCH" if image_e1.matches and audio_e1.matches else "DIFFER",
|
||||||
|
"first_field": None if image_e1.matches and audio_e1.matches else ("images" if not image_e1.matches else "audio"),
|
||||||
|
"images": image_e1.to_dict(),
|
||||||
|
"audio": audio_e1.to_dict(),
|
||||||
|
"boundary_images": boundary_diff.to_dict(),
|
||||||
|
},
|
||||||
|
"E2_final_240": {
|
||||||
|
"status": "MATCH" if image_e2.matches and audio_e2.matches else "DIFFER",
|
||||||
|
"first_field": None if image_e2.matches and audio_e2.matches else ("images" if not image_e2.matches else "audio"),
|
||||||
|
"images": image_e2.to_dict(),
|
||||||
|
"audio": audio_e2.to_dict(),
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
class H3CoreContinuumGpuDiagnostic:
|
||||||
|
"""Run an isolated Core 243f and Continuum 2x5s comparison in one queue."""
|
||||||
|
|
||||||
|
OUTPUT_NODE = True
|
||||||
|
CATEGORY = "H3-Continuum/Diagnostics"
|
||||||
|
FUNCTION = "run"
|
||||||
|
RETURN_TYPES = ("STRING",)
|
||||||
|
RETURN_NAMES = ("diagnostic_report",)
|
||||||
|
|
||||||
|
@classmethod
|
||||||
|
def INPUT_TYPES(cls):
|
||||||
|
return {
|
||||||
|
"required": {
|
||||||
|
"model": ("MODEL",),
|
||||||
|
"clip": ("CLIP",),
|
||||||
|
"video_vae": ("VAE",),
|
||||||
|
"audio_vae": ("VAE",),
|
||||||
|
"sampler": ("SAMPLER",),
|
||||||
|
"sigmas": ("SIGMAS",),
|
||||||
|
"first_frame": ("IMAGE",),
|
||||||
|
"last_frame": ("IMAGE",),
|
||||||
|
"prompt": ("STRING", {"multiline": True, "default": "The subject moves continuously from the First Frame to the Last Frame."}),
|
||||||
|
"width": ("INT", {"default": 640, "min": 64, "max": 4096, "step": 16}),
|
||||||
|
"height": ("INT", {"default": 640, "min": 64, "max": 4096, "step": 16}),
|
||||||
|
"seed": ("INT", {"default": 42, "min": 0, "max": 0xFFFFFFFFFFFFFFFF}),
|
||||||
|
"output_subdir": ("STRING", {"default": "h3_continuum/diagnostics"}),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
def run(
|
||||||
|
self,
|
||||||
|
model,
|
||||||
|
clip,
|
||||||
|
video_vae,
|
||||||
|
audio_vae,
|
||||||
|
sampler,
|
||||||
|
sigmas,
|
||||||
|
first_frame,
|
||||||
|
last_frame,
|
||||||
|
prompt,
|
||||||
|
width,
|
||||||
|
height,
|
||||||
|
seed,
|
||||||
|
output_subdir,
|
||||||
|
):
|
||||||
|
output_directory = _safe_output_directory(output_subdir)
|
||||||
|
core = DiagnosticCapture("core")
|
||||||
|
continuum = DiagnosticCapture("continuum")
|
||||||
|
comparison: dict[str, Any] = {}
|
||||||
|
modules = _resolve_continuum_modules()
|
||||||
|
common = {
|
||||||
|
"seed": int(seed),
|
||||||
|
"width": int(width),
|
||||||
|
"height": int(height),
|
||||||
|
"physical_frames": 243,
|
||||||
|
"spectrum": "must be OFF at the model input",
|
||||||
|
"run_storage": "Off",
|
||||||
|
"session": None,
|
||||||
|
"continuum_root": str(modules.root),
|
||||||
|
}
|
||||||
|
core.manifest["conditions"] = common
|
||||||
|
continuum.manifest["conditions"] = common
|
||||||
|
|
||||||
|
try:
|
||||||
|
_reset_rng(int(seed))
|
||||||
|
_run_core(
|
||||||
|
core,
|
||||||
|
modules,
|
||||||
|
model=model,
|
||||||
|
clip=clip,
|
||||||
|
video_vae=video_vae,
|
||||||
|
sampler=sampler,
|
||||||
|
sigmas=sigmas,
|
||||||
|
first_frame=first_frame,
|
||||||
|
last_frame=last_frame,
|
||||||
|
prompt=prompt,
|
||||||
|
width=width,
|
||||||
|
height=height,
|
||||||
|
seed=seed,
|
||||||
|
)
|
||||||
|
_release_cuda()
|
||||||
|
_reset_rng(int(seed))
|
||||||
|
_run_continuum(
|
||||||
|
continuum,
|
||||||
|
modules,
|
||||||
|
model=model,
|
||||||
|
clip=clip,
|
||||||
|
video_vae=video_vae,
|
||||||
|
audio_vae=audio_vae,
|
||||||
|
sampler=sampler,
|
||||||
|
sigmas=sigmas,
|
||||||
|
first_frame=first_frame,
|
||||||
|
last_frame=last_frame,
|
||||||
|
prompt=prompt,
|
||||||
|
width=width,
|
||||||
|
height=height,
|
||||||
|
seed=seed,
|
||||||
|
)
|
||||||
|
|
||||||
|
for stage in ("A_pre_sampler", "B_first_model_call", "C_physical_sample"):
|
||||||
|
comparison[stage] = compare_stage(core, continuum, stage)
|
||||||
|
|
||||||
|
d_values = continuum.manifest["stages"].get("D_split_recombine", {}).get("values", {})
|
||||||
|
d_video = d_values.get("video_difference", {})
|
||||||
|
d_audio = d_values.get("audio_difference", {})
|
||||||
|
d_match = bool(d_video.get("matches") and d_audio.get("matches"))
|
||||||
|
comparison["D_split_recombine"] = {
|
||||||
|
"status": "MATCH" if d_match else "DIFFER",
|
||||||
|
"first_field": None if d_match else ("video" if not d_video.get("matches") else "audio"),
|
||||||
|
"video": d_video,
|
||||||
|
"audio": d_audio,
|
||||||
|
}
|
||||||
|
comparison.update(_run_decode_diagnostics(continuum, modules, video_vae, audio_vae))
|
||||||
|
except Exception as exc:
|
||||||
|
comparison["error"] = {
|
||||||
|
"type": type(exc).__name__,
|
||||||
|
"message": str(exc),
|
||||||
|
"traceback": traceback.format_exc(),
|
||||||
|
}
|
||||||
|
finally:
|
||||||
|
output_directory.mkdir(parents=True, exist_ok=True)
|
||||||
|
core.save(output_directory / "core")
|
||||||
|
continuum.save(output_directory / "continuum")
|
||||||
|
for stage in STAGE_ORDER:
|
||||||
|
comparison.setdefault(stage, {"status": "NOT_RUN", "first_field": None})
|
||||||
|
summary = format_comparison(comparison, output_directory)
|
||||||
|
(output_directory / "comparison.json").write_text(
|
||||||
|
json.dumps(comparison, indent=2, ensure_ascii=False),
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
(output_directory / "comparison.txt").write_text(summary + "\n", encoding="utf-8")
|
||||||
|
_release_cuda()
|
||||||
|
|
||||||
|
if "error" in comparison:
|
||||||
|
error = comparison["error"]
|
||||||
|
raise RuntimeError(
|
||||||
|
f"GPU diagnostic failed; partial results were saved to {output_directory}: "
|
||||||
|
f"{error['type']}: {error['message']}"
|
||||||
|
)
|
||||||
|
return (summary,)
|
||||||
|
|
||||||
|
|
||||||
|
NODE_CLASS_MAPPINGS = {
|
||||||
|
"H3CoreContinuumGpuDiagnostic": H3CoreContinuumGpuDiagnostic,
|
||||||
|
}
|
||||||
|
|
||||||
|
NODE_DISPLAY_NAME_MAPPINGS = {
|
||||||
|
"H3CoreContinuumGpuDiagnostic": "H3 Core vs Continuum GPU Diagnostic",
|
||||||
|
}
|
||||||
@@ -0,0 +1,29 @@
|
|||||||
|
param(
|
||||||
|
[string]$ComfyUIRoot = 'D:\StabilityMatrix\Data\Packages\ComfyUI_W'
|
||||||
|
)
|
||||||
|
|
||||||
|
$ErrorActionPreference = 'Stop'
|
||||||
|
$source = Join-Path $PSScriptRoot 'gpu_diagnostic_node'
|
||||||
|
$customNodes = Join-Path $ComfyUIRoot 'custom_nodes'
|
||||||
|
$destination = Join-Path $customNodes 'H3_Continuum_GPU_Diagnostics'
|
||||||
|
|
||||||
|
if (-not (Test-Path -LiteralPath $source -PathType Container)) {
|
||||||
|
throw "Diagnostic source was not found: $source"
|
||||||
|
}
|
||||||
|
if (-not (Test-Path -LiteralPath $customNodes -PathType Container)) {
|
||||||
|
throw "ComfyUI custom_nodes directory was not found: $customNodes"
|
||||||
|
}
|
||||||
|
|
||||||
|
New-Item -ItemType Directory -Path $destination -Force | Out-Null
|
||||||
|
foreach ($name in @('__init__.py', 'capture.py', 'nodes.py', 'README.md')) {
|
||||||
|
Copy-Item -LiteralPath (Join-Path $source $name) -Destination (Join-Path $destination $name) -Force
|
||||||
|
}
|
||||||
|
|
||||||
|
$sourceCount = (Get-ChildItem -LiteralPath $source -File).Count
|
||||||
|
$destinationCount = (Get-ChildItem -LiteralPath $destination -File).Count
|
||||||
|
if ($destinationCount -lt $sourceCount) {
|
||||||
|
throw "Diagnostic install verification failed: $destinationCount < $sourceCount"
|
||||||
|
}
|
||||||
|
|
||||||
|
Write-Output "Installed standalone diagnostics: $destination"
|
||||||
|
Write-Output 'Restart ComfyUI before using the diagnostic node.'
|
||||||
+283
-12
@@ -24,14 +24,21 @@ from ..driving_audio import (
|
|||||||
)
|
)
|
||||||
from ..model_patch import clone_model_for_chunk
|
from ..model_patch import clone_model_for_chunk
|
||||||
from ..state import assert_context_unchanged, context_fingerprint, make_plan, select_context, validate_state
|
from ..state import assert_context_unchanged, context_fingerprint, make_plan, select_context, validate_state
|
||||||
from ..temporal import align_frame_count_up, largest_context_capacity, make_extension_shape
|
from ..temporal import (
|
||||||
|
align_frame_count_up,
|
||||||
|
audio_latent_t,
|
||||||
|
context_slots,
|
||||||
|
largest_context_capacity,
|
||||||
|
make_extension_shape,
|
||||||
|
video_latent_t,
|
||||||
|
)
|
||||||
from ..version import PACKAGE_VERSION
|
from ..version import PACKAGE_VERSION
|
||||||
from .decoder import decode_sequence, decode_sequence_with_seam, enforce_total_frames
|
from .decoder import decode_sequence, decode_sequence_with_seam, enforce_total_frames
|
||||||
from .context_diagnostics import ContextDiagnosticsTracker
|
from .context_diagnostics import ContextDiagnosticsTracker
|
||||||
from .h3_builder import attach_keyframes, empty_h3_latent, encode_identity_latents, encode_prompt_conditioning, prepare_identity_assets
|
from .h3_builder import attach_keyframes, empty_h3_latent, encode_identity_latents, encode_prompt_conditioning, prepare_identity_assets
|
||||||
from .motion import choose_context_frames
|
from .motion import choose_context_frames
|
||||||
from .prompts import prompt_plan_report, validate_prompt_plan
|
from .prompts import prompt_plan_report, validate_prompt_plan
|
||||||
from .sampling import sample_chunk
|
from .sampling import latent_from_cpu, latent_to_cpu, sample_chunk
|
||||||
from .seeds import derive_chunk_seed
|
from .seeds import derive_chunk_seed
|
||||||
from .session import (
|
from .session import (
|
||||||
SessionValidationError, entry_to_state, make_chunk_entry, make_session, model_fingerprint,
|
SessionValidationError, entry_to_state, make_chunk_entry, make_session, model_fingerprint,
|
||||||
@@ -62,7 +69,7 @@ def _check_decode_memory_budget(*,width,height,chunks,chunk_seconds):
|
|||||||
def _clone_entry_for_reuse(entry):
|
def _clone_entry_for_reuse(entry):
|
||||||
entry=validate_chunk_entry(entry); result=dict(entry); result["plan"]=copy.deepcopy(entry["plan"]); result["reused"]=True; return result
|
entry=validate_chunk_entry(entry); result=dict(entry); result["plan"]=copy.deepcopy(entry["plan"]); result["reused"]=True; return result
|
||||||
|
|
||||||
def _preserved_prefix(*,session,prompt_hashes,chunks,reroll_from_chunk,width,height,chunk_seconds,identity_hash):
|
def _preserved_prefix(*,session,prompt_hashes,chunks,reroll_from_chunk,width,height,chunk_seconds,identity_hash,last_frame_hash):
|
||||||
if session is None: return [],[]
|
if session is None: return [],[]
|
||||||
notes=[]
|
notes=[]
|
||||||
try: session=validate_session(session)
|
try: session=validate_session(session)
|
||||||
@@ -70,6 +77,8 @@ def _preserved_prefix(*,session,prompt_hashes,chunks,reroll_from_chunk,width,hei
|
|||||||
if int(session["width"])!=int(width) or int(session["height"])!=int(height): return [],["saved session resolution differs; generated a fresh run"]
|
if int(session["width"])!=int(width) or int(session["height"])!=int(height): return [],["saved session resolution differs; generated a fresh run"]
|
||||||
if abs(float(session["chunk_seconds"])-float(chunk_seconds))>1e-6: return [],["saved session chunk duration differs; generated a fresh run"]
|
if abs(float(session["chunk_seconds"])-float(chunk_seconds))>1e-6: return [],["saved session chunk duration differs; generated a fresh run"]
|
||||||
if str(session.get("identity_hash","none"))!=str(identity_hash): return [],["saved session identity differs; generated a fresh run"]
|
if str(session.get("identity_hash","none"))!=str(identity_hash): return [],["saved session identity differs; generated a fresh run"]
|
||||||
|
saved_last_frame_hash=(session.get("settings") or {}).get("last_frame_hash")
|
||||||
|
if str(saved_last_frame_hash or "none")!=str(last_frame_hash or "none"): return [],["saved session Last Frame differs; generated a fresh run"]
|
||||||
if reroll_from_chunk<0 or reroll_from_chunk>chunks: raise SessionValidationError("reroll_from_chunk must be 0 or a valid one-based chunk index")
|
if reroll_from_chunk<0 or reroll_from_chunk>chunks: raise SessionValidationError("reroll_from_chunk must be 0 or a valid one-based chunk index")
|
||||||
limit=min(len(session["chunks"]),chunks)
|
limit=min(len(session["chunks"]),chunks)
|
||||||
if reroll_from_chunk>0: limit=min(limit,reroll_from_chunk-1)
|
if reroll_from_chunk>0: limit=min(limit,reroll_from_chunk-1)
|
||||||
@@ -103,7 +112,12 @@ def _conditioning_cache(*,clip,prompts,assets,final_has_last_frame,reference_ass
|
|||||||
return cache
|
return cache
|
||||||
|
|
||||||
|
|
||||||
FLF_STRATEGY = "context_bound_terminal_v1"
|
FLF_STRATEGY = "terminal_merged_10s_seed_v2"
|
||||||
|
TERMINAL_SEED_POLICY = "physical_window_seed_v2"
|
||||||
|
TERMINAL_MERGE_CHUNK_SECONDS = 5.0
|
||||||
|
TERMINAL_MERGE_CONTEXT_FRAMES = 22
|
||||||
|
TERMINAL_PROMPT_POLICY_SHARED = "shared_prompt_v1"
|
||||||
|
TERMINAL_PROMPT_POLICY_TIMELINE = "paired_timeline_v1"
|
||||||
|
|
||||||
|
|
||||||
def _flf_contract_identity(base_hash: str) -> str:
|
def _flf_contract_identity(base_hash: str) -> str:
|
||||||
@@ -111,6 +125,191 @@ def _flf_contract_identity(base_hash: str) -> str:
|
|||||||
return hashlib.sha256(payload.encode("utf-8")).hexdigest()
|
return hashlib.sha256(payload.encode("utf-8")).hexdigest()
|
||||||
|
|
||||||
|
|
||||||
|
def _terminal_flf_merge_enabled(
|
||||||
|
*,
|
||||||
|
multi_chunk_flf: bool,
|
||||||
|
chunks: int,
|
||||||
|
chunk_seconds: float,
|
||||||
|
prompt_hashes: list[str],
|
||||||
|
timeline_video_source,
|
||||||
|
) -> bool:
|
||||||
|
return bool(
|
||||||
|
multi_chunk_flf
|
||||||
|
and int(chunks) >= 2
|
||||||
|
and abs(float(chunk_seconds) - TERMINAL_MERGE_CHUNK_SECONDS) <= 1e-6
|
||||||
|
and timeline_video_source is None
|
||||||
|
and len(prompt_hashes) >= 2
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _terminal_pair_prompt(
|
||||||
|
prompts: list[str],
|
||||||
|
*,
|
||||||
|
pair_start: int,
|
||||||
|
chunk_seconds: float,
|
||||||
|
) -> tuple[str, str]:
|
||||||
|
"""Build one physical prompt without changing either logical prompt."""
|
||||||
|
|
||||||
|
first = str(prompts[int(pair_start)])
|
||||||
|
second = str(prompts[int(pair_start) + 1])
|
||||||
|
if first == second:
|
||||||
|
return first, TERMINAL_PROMPT_POLICY_SHARED
|
||||||
|
split = f"{float(chunk_seconds):g}"
|
||||||
|
stop = f"{float(chunk_seconds) * 2.0:g}"
|
||||||
|
return (
|
||||||
|
f"[0-{split}s]\n{first}\n\n[{split}-{stop}s]\n{second}",
|
||||||
|
TERMINAL_PROMPT_POLICY_TIMELINE,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _atomic_terminal_prefix(
|
||||||
|
preserved: list[dict[str, Any]],
|
||||||
|
*,
|
||||||
|
chunks: int,
|
||||||
|
reroll_from_chunk: int,
|
||||||
|
) -> tuple[list[dict[str, Any]], bool]:
|
||||||
|
"""Never reuse only one logical half of a shared terminal sample."""
|
||||||
|
pair_start = int(chunks) - 2
|
||||||
|
reroll_hits_pair = int(reroll_from_chunk) in (int(chunks) - 1, int(chunks))
|
||||||
|
partial_pair = pair_start < len(preserved) < int(chunks)
|
||||||
|
if (reroll_hits_pair or partial_pair) and len(preserved) > pair_start:
|
||||||
|
return preserved[:pair_start], True
|
||||||
|
return preserved, False
|
||||||
|
|
||||||
|
|
||||||
|
def _terminal_sampling_plan(
|
||||||
|
*,
|
||||||
|
chunks: int,
|
||||||
|
completed: int,
|
||||||
|
merge_enabled: bool,
|
||||||
|
) -> tuple[tuple[int, ...], bool]:
|
||||||
|
if completed >= int(chunks):
|
||||||
|
return (), False
|
||||||
|
if not merge_enabled:
|
||||||
|
return tuple(range(completed, int(chunks))), False
|
||||||
|
pair_start = int(chunks) - 2
|
||||||
|
if completed > pair_start:
|
||||||
|
raise SequenceRuntimeError("terminal merged pair cannot resume from only one logical half")
|
||||||
|
return tuple(range(completed, pair_start)), True
|
||||||
|
|
||||||
|
|
||||||
|
def _terminal_physical_seed_plan(
|
||||||
|
*,
|
||||||
|
base_seed: int,
|
||||||
|
physical_window_start_index: int,
|
||||||
|
reroll_nonce: int,
|
||||||
|
) -> dict[str, Any]:
|
||||||
|
"""Return the one physical seed and the matching logical-entry metadata."""
|
||||||
|
start_index = int(physical_window_start_index)
|
||||||
|
nonce = int(reroll_nonce)
|
||||||
|
if start_index < 0:
|
||||||
|
raise ValueError("physical_window_start_index must be non-negative")
|
||||||
|
if nonce < 0:
|
||||||
|
raise ValueError("reroll_nonce must be non-negative")
|
||||||
|
if start_index == 0 and nonce == 0:
|
||||||
|
physical_seed = int(base_seed)
|
||||||
|
else:
|
||||||
|
physical_seed = derive_chunk_seed(base_seed, start_index, nonce)
|
||||||
|
return {
|
||||||
|
"physical_seed": physical_seed,
|
||||||
|
"logical_entry_seeds": (physical_seed, physical_seed),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _terminal_execution_semantics(
|
||||||
|
*,
|
||||||
|
merge_enabled: bool,
|
||||||
|
prompt_policy: str | None = None,
|
||||||
|
) -> dict[str, str] | None:
|
||||||
|
if not merge_enabled:
|
||||||
|
return None
|
||||||
|
semantics = {
|
||||||
|
"flf_execution": FLF_STRATEGY,
|
||||||
|
"terminal_seed_policy": TERMINAL_SEED_POLICY,
|
||||||
|
}
|
||||||
|
if prompt_policy:
|
||||||
|
semantics["terminal_prompt_policy"] = str(prompt_policy)
|
||||||
|
return semantics
|
||||||
|
|
||||||
|
|
||||||
|
def _terminal_pair_contract(*, initial_pair: bool, chunk_seconds: float) -> dict[str, Any]:
|
||||||
|
"""Derive physical and logical AV ranges without fixed latent indices."""
|
||||||
|
if abs(float(chunk_seconds) - TERMINAL_MERGE_CHUNK_SECONDS) > 1e-6:
|
||||||
|
raise SequenceRuntimeError("terminal merged sampling currently requires 5-second logical chunks")
|
||||||
|
initial_frames = align_frame_count_up(int(round(float(chunk_seconds) * FPS)))
|
||||||
|
extension = make_extension_shape(
|
||||||
|
TERMINAL_MERGE_CONTEXT_FRAMES,
|
||||||
|
float(chunk_seconds),
|
||||||
|
)
|
||||||
|
if initial_pair:
|
||||||
|
physical_frames = align_frame_count_up(int(round(2.0 * float(chunk_seconds) * FPS)))
|
||||||
|
logical_frames = (initial_frames, extension.total_frames)
|
||||||
|
logical_trims = (0, TERMINAL_MERGE_CONTEXT_FRAMES)
|
||||||
|
first_video_stop = video_latent_t(initial_frames)
|
||||||
|
first_audio_stop = audio_latent_t(initial_frames)
|
||||||
|
else:
|
||||||
|
physical_frames = make_extension_shape(
|
||||||
|
TERMINAL_MERGE_CONTEXT_FRAMES,
|
||||||
|
2.0 * float(chunk_seconds),
|
||||||
|
).total_frames
|
||||||
|
logical_frames = (extension.total_frames, extension.total_frames)
|
||||||
|
logical_trims = (
|
||||||
|
TERMINAL_MERGE_CONTEXT_FRAMES,
|
||||||
|
TERMINAL_MERGE_CONTEXT_FRAMES,
|
||||||
|
)
|
||||||
|
first_video_stop = video_latent_t(extension.total_frames)
|
||||||
|
first_audio_stop = audio_latent_t(extension.total_frames)
|
||||||
|
physical_video_stop = video_latent_t(physical_frames)
|
||||||
|
physical_audio_stop = audio_latent_t(physical_frames)
|
||||||
|
video_overlap = context_slots(TERMINAL_MERGE_CONTEXT_FRAMES)
|
||||||
|
audio_overlap = audio_latent_t(TERMINAL_MERGE_CONTEXT_FRAMES)
|
||||||
|
video_slices = (
|
||||||
|
(0, first_video_stop),
|
||||||
|
(first_video_stop - video_overlap, physical_video_stop),
|
||||||
|
)
|
||||||
|
audio_slices = (
|
||||||
|
(0, first_audio_stop),
|
||||||
|
(first_audio_stop - audio_overlap, physical_audio_stop),
|
||||||
|
)
|
||||||
|
for index in range(2):
|
||||||
|
expected_video = video_latent_t(logical_frames[index])
|
||||||
|
expected_audio = audio_latent_t(logical_frames[index])
|
||||||
|
if video_slices[index][1] - video_slices[index][0] != expected_video:
|
||||||
|
raise SequenceRuntimeError("derived terminal video split does not match its logical frame count")
|
||||||
|
if audio_slices[index][1] - audio_slices[index][0] != expected_audio:
|
||||||
|
raise SequenceRuntimeError("derived terminal audio split does not match its logical frame count")
|
||||||
|
return {
|
||||||
|
"initial_pair": bool(initial_pair),
|
||||||
|
"physical_frames": int(physical_frames),
|
||||||
|
"physical_context_frames": 0 if initial_pair else TERMINAL_MERGE_CONTEXT_FRAMES,
|
||||||
|
"logical_frames": logical_frames,
|
||||||
|
"logical_trims": logical_trims,
|
||||||
|
"video_slices": video_slices,
|
||||||
|
"audio_slices": audio_slices,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _split_terminal_merged_latents(
|
||||||
|
video: torch.Tensor,
|
||||||
|
audio: torch.Tensor,
|
||||||
|
contract: dict[str, Any],
|
||||||
|
) -> tuple[tuple[torch.Tensor, torch.Tensor], tuple[torch.Tensor, torch.Tensor]]:
|
||||||
|
expected_video = video_latent_t(int(contract["physical_frames"]))
|
||||||
|
expected_audio = audio_latent_t(int(contract["physical_frames"]))
|
||||||
|
if int(video.shape[2]) != expected_video or int(audio.shape[-1]) != expected_audio:
|
||||||
|
raise SequenceRuntimeError(
|
||||||
|
"terminal merged sampler output does not match the derived physical AV grid: "
|
||||||
|
f"video T={video.shape[2]} (expected {expected_video}), "
|
||||||
|
f"audio T={audio.shape[-1]} (expected {expected_audio})"
|
||||||
|
)
|
||||||
|
parts=[]
|
||||||
|
for video_range,audio_range in zip(contract["video_slices"],contract["audio_slices"]):
|
||||||
|
video_part=video[:,:,video_range[0]:video_range[1],...].contiguous()
|
||||||
|
audio_part=audio[...,audio_range[0]:audio_range[1]].contiguous()
|
||||||
|
parts.append((video_part,audio_part))
|
||||||
|
return parts[0],parts[1]
|
||||||
|
|
||||||
|
|
||||||
def _attach_terminal_flf_keyframes(
|
def _attach_terminal_flf_keyframes(
|
||||||
conditioning,
|
conditioning,
|
||||||
*,
|
*,
|
||||||
@@ -141,7 +340,7 @@ def _attach_terminal_flf_keyframes(
|
|||||||
return conditioning
|
return conditioning
|
||||||
|
|
||||||
def run_sequence(*,model:Any,clip:Any,video_vae:Any,audio_vae:Any,sampler:Any,sigmas:torch.Tensor,first_frame:torch.Tensor|None,last_frame:torch.Tensor|None,prompt_plan:dict[str,Any],width:int,height:int,continuity:str,base_seed:int,audio_continuity:bool,exact_total_duration:bool,diagnostics_mode:str,reroll_from_chunk:int,reroll_nonce:int,strict_compatibility:bool,debug:bool,seam_correction:str=SEAM_CORRECTION_OFF,enable_preview:bool=True,session:dict[str,Any]|None=None,initial_state:dict[str,Any]|None=None,latent_only:bool=False,reference_assets=None,reference_audio_source=None,reference_audio_vae=None,driving_audio_source=None,driving_audio_vae=None,reference_video_source=None,timeline_video_source=None):
|
def run_sequence(*,model:Any,clip:Any,video_vae:Any,audio_vae:Any,sampler:Any,sigmas:torch.Tensor,first_frame:torch.Tensor|None,last_frame:torch.Tensor|None,prompt_plan:dict[str,Any],width:int,height:int,continuity:str,base_seed:int,audio_continuity:bool,exact_total_duration:bool,diagnostics_mode:str,reroll_from_chunk:int,reroll_nonce:int,strict_compatibility:bool,debug:bool,seam_correction:str=SEAM_CORRECTION_OFF,enable_preview:bool=True,session:dict[str,Any]|None=None,initial_state:dict[str,Any]|None=None,latent_only:bool=False,reference_assets=None,reference_audio_source=None,reference_audio_vae=None,driving_audio_source=None,driving_audio_vae=None,reference_video_source=None,timeline_video_source=None):
|
||||||
from ..conditioning import detect_conditioning_mode, conditioning_mode_label
|
from ..conditioning import detect_conditioning_mode, conditioning_display_label
|
||||||
from ..run_storage import get_active_run_storage
|
from ..run_storage import get_active_run_storage
|
||||||
storage_controller=get_active_run_storage()
|
storage_controller=get_active_run_storage()
|
||||||
diagnostics_mode=normalize_diagnostics_mode(diagnostics_mode); plan=validate_prompt_plan(prompt_plan); chunks=int(plan["chunks"]); chunk_seconds=float(plan["chunk_seconds"]); prompts=list(plan["prompts"]); prompt_hashes=list(plan["hashes"]); width,height=int(width),int(height)
|
diagnostics_mode=normalize_diagnostics_mode(diagnostics_mode); plan=validate_prompt_plan(prompt_plan); chunks=int(plan["chunks"]); chunk_seconds=float(plan["chunk_seconds"]); prompts=list(plan["prompts"]); prompt_hashes=list(plan["hashes"]); width,height=int(width),int(height)
|
||||||
@@ -157,6 +356,7 @@ def run_sequence(*,model:Any,clip:Any,video_vae:Any,audio_vae:Any,sampler:Any,si
|
|||||||
if initial_state is not None and int(reroll_from_chunk) not in (0,1): raise SequenceRuntimeError("with initial_state, reroll_from_chunk can only be 0 or 1")
|
if initial_state is not None and int(reroll_from_chunk) not in (0,1): raise SequenceRuntimeError("with initial_state, reroll_from_chunk can only be 0 or 1")
|
||||||
try: conditioning_mode=detect_conditioning_mode(first_frame=first_frame,last_frame=last_frame,reference_assets=reference_assets)
|
try: conditioning_mode=detect_conditioning_mode(first_frame=first_frame,last_frame=last_frame,reference_assets=reference_assets)
|
||||||
except ValueError as exc: raise SequenceRuntimeError(str(exc)) from exc
|
except ValueError as exc: raise SequenceRuntimeError(str(exc)) from exc
|
||||||
|
conditioning_display=conditioning_display_label(has_first=first_frame is not None,has_last=last_frame is not None,has_reference=reference_assets is not None)
|
||||||
reference_warning=""
|
reference_warning=""
|
||||||
if reference_assets is not None:
|
if reference_assets is not None:
|
||||||
from ..reference import validate_reference_prompts
|
from ..reference import validate_reference_prompts
|
||||||
@@ -181,6 +381,11 @@ def run_sequence(*,model:Any,clip:Any,video_vae:Any,audio_vae:Any,sampler:Any,si
|
|||||||
issues=check_comfy_h3_runtime()
|
issues=check_comfy_h3_runtime()
|
||||||
assets=prepare_identity_assets(video_vae,width=width,height=height,first_frame=first_frame,last_frame=last_frame,encode_latents=False)
|
assets=prepare_identity_assets(video_vae,width=width,height=height,first_frame=first_frame,last_frame=last_frame,encode_latents=False)
|
||||||
multi_chunk_flf=bool(chunks>=2 and first_frame is not None and last_frame is not None)
|
multi_chunk_flf=bool(chunks>=2 and first_frame is not None and last_frame is not None)
|
||||||
|
terminal_merge_enabled=_terminal_flf_merge_enabled(multi_chunk_flf=multi_chunk_flf,chunks=chunks,chunk_seconds=chunk_seconds,prompt_hashes=prompt_hashes,timeline_video_source=timeline_video_source)
|
||||||
|
terminal_prompt=None
|
||||||
|
terminal_prompt_policy=None
|
||||||
|
if terminal_merge_enabled:
|
||||||
|
terminal_prompt,terminal_prompt_policy=_terminal_pair_prompt(prompts,pair_start=chunks-2,chunk_seconds=chunk_seconds)
|
||||||
from ..reference import combine_hybrid_visual_identity
|
from ..reference import combine_hybrid_visual_identity
|
||||||
visual_identity_hash=combine_hybrid_visual_identity(keyframe_identity_hash=assets.identity_hash,reference_assets=reference_assets,has_first=first_frame is not None,has_last=last_frame is not None)
|
visual_identity_hash=combine_hybrid_visual_identity(keyframe_identity_hash=assets.identity_hash,reference_assets=reference_assets,has_first=first_frame is not None,has_last=last_frame is not None)
|
||||||
sequence_identity_hash=combine_reference_audio_identity(visual_identity_hash,reference_audio_source)
|
sequence_identity_hash=combine_reference_audio_identity(visual_identity_hash,reference_audio_source)
|
||||||
@@ -191,20 +396,27 @@ def run_sequence(*,model:Any,clip:Any,video_vae:Any,audio_vae:Any,sampler:Any,si
|
|||||||
sequence_identity_hash=_flf_contract_identity(sequence_identity_hash)
|
sequence_identity_hash=_flf_contract_identity(sequence_identity_hash)
|
||||||
current_model_fingerprint=model_fingerprint(model,extra_wrapper_keys=("h3_continuum_join.apply_model.v1",))
|
current_model_fingerprint=model_fingerprint(model,extra_wrapper_keys=("h3_continuum_join.apply_model.v1",))
|
||||||
if storage_controller is not None:
|
if storage_controller is not None:
|
||||||
stored_session=storage_controller.prepare(model=model,model_fingerprint_value=current_model_fingerprint,clip=clip,video_vae=video_vae,sampler=sampler,sigmas=sigmas,prompt_plan=plan,width=width,height=height,chunk_seconds=chunk_seconds,continuity=continuity,audio_continuity=audio_continuity,base_seed=base_seed,reroll_from_chunk=reroll_from_chunk,reroll_nonce=reroll_nonce,first_frame_hash=assets.first_frame_hash,last_frame_hash=assets.last_frame_hash,identity_hash=sequence_identity_hash,strict_compatibility=strict_compatibility,existing_session=session,reference_contract=reference_assets.contract if reference_assets is not None else None,conditioning_mode=conditioning_mode,reference_audio_contract=reference_audio_source.contract if reference_audio_source is not None else None,reference_audio_vae=reference_audio_vae,driving_audio_contract=driving_audio_source.contract if driving_audio_source is not None else None,driving_audio_vae=driving_audio_vae,reference_video_contract=reference_video_source.contract if reference_video_source is not None else None,timeline_video_contract=timeline_video_source.contract if timeline_video_source is not None else None)
|
stored_session=storage_controller.prepare(model=model,model_fingerprint_value=current_model_fingerprint,clip=clip,video_vae=video_vae,sampler=sampler,sigmas=sigmas,prompt_plan=plan,width=width,height=height,chunk_seconds=chunk_seconds,continuity=continuity,audio_continuity=audio_continuity,base_seed=base_seed,reroll_from_chunk=reroll_from_chunk,reroll_nonce=reroll_nonce,first_frame_hash=assets.first_frame_hash,last_frame_hash=assets.last_frame_hash,identity_hash=sequence_identity_hash,strict_compatibility=strict_compatibility,existing_session=session,reference_contract=reference_assets.contract if reference_assets is not None else None,conditioning_mode=conditioning_mode,reference_audio_contract=reference_audio_source.contract if reference_audio_source is not None else None,reference_audio_vae=reference_audio_vae,driving_audio_contract=driving_audio_source.contract if driving_audio_source is not None else None,driving_audio_vae=driving_audio_vae,reference_video_contract=reference_video_source.contract if reference_video_source is not None else None,timeline_video_contract=timeline_video_source.contract if timeline_video_source is not None else None,execution_semantics=_terminal_execution_semantics(merge_enabled=terminal_merge_enabled,prompt_policy=terminal_prompt_policy))
|
||||||
reroll_nonce=storage_controller.effective_reroll_nonce
|
reroll_nonce=storage_controller.effective_reroll_nonce
|
||||||
if stored_session is not None: session=stored_session
|
if stored_session is not None: session=stored_session
|
||||||
accelerators=accelerator_summary(model)
|
accelerators=accelerator_summary(model)
|
||||||
if "Continuum APPLY_MODEL wrapper installed" not in accelerators: accelerators+="; Continuum APPLY_MODEL wrapper installed"
|
if "Continuum APPLY_MODEL wrapper installed" not in accelerators: accelerators+="; Continuum APPLY_MODEL wrapper installed"
|
||||||
effective_reroll_from_chunk=0 if storage_controller is not None and session is not None and bool((session.get("settings") or {}).get("run_storage_validated_prefix")) else int(reroll_from_chunk)
|
effective_reroll_from_chunk=0 if storage_controller is not None and session is not None and bool((session.get("settings") or {}).get("run_storage_validated_prefix")) else int(reroll_from_chunk)
|
||||||
preserved,reuse_notes=_preserved_prefix(session=session,prompt_hashes=prompt_hashes,chunks=chunks,reroll_from_chunk=effective_reroll_from_chunk,width=width,height=height,chunk_seconds=chunk_seconds,identity_hash=sequence_identity_hash)
|
preserved,reuse_notes=_preserved_prefix(session=session,prompt_hashes=prompt_hashes,chunks=chunks,reroll_from_chunk=effective_reroll_from_chunk,width=width,height=height,chunk_seconds=chunk_seconds,identity_hash=sequence_identity_hash,last_frame_hash=assets.last_frame_hash)
|
||||||
if multi_chunk_flf and session is not None:
|
if multi_chunk_flf and session is not None:
|
||||||
saved_strategy=str((session.get("settings") or {}).get("flf_execution", ""))
|
saved_strategy=str((session.get("settings") or {}).get("flf_execution", ""))
|
||||||
if saved_strategy!=FLF_STRATEGY:
|
if saved_strategy!=FLF_STRATEGY:
|
||||||
preserved=[]
|
preserved=[]
|
||||||
reuse_notes.append("FLF chunk contract changed; regenerated the requested FLF sequence.")
|
reuse_notes.append("FLF chunk contract changed; regenerated the requested FLF sequence.")
|
||||||
|
if terminal_merge_enabled:
|
||||||
|
preserved,atomic_reset=_atomic_terminal_prefix(preserved,chunks=chunks,reroll_from_chunk=int(reroll_from_chunk))
|
||||||
|
if atomic_reset:
|
||||||
|
reuse_notes.append("terminal merged pair is atomic; regenerated both final logical chunks")
|
||||||
if multi_chunk_flf:
|
if multi_chunk_flf:
|
||||||
reuse_notes.insert(0,"FLF execution: context-bound terminal v1; one sampling pass per chunk; the final chunk is anchored from the continuation-context tail to the supplied Last Frame.")
|
if terminal_merge_enabled:
|
||||||
|
reuse_notes.insert(0,"FLF execution: terminal merged 10s seed v2; the final two 5-second logical chunks share one physical sample and are split back into normal chunk entries.")
|
||||||
|
else:
|
||||||
|
reuse_notes.insert(0,"FLF execution: terminal merge unavailable for this request; retained one sampling pass per logical chunk.")
|
||||||
if reference_assets is not None:
|
if reference_assets is not None:
|
||||||
reuse_notes.insert(0,f"Reference conditioning: {reference_assets.count} image(s), size={reference_assets.size_mode}; persistent across all chunks.")
|
reuse_notes.insert(0,f"Reference conditioning: {reference_assets.count} image(s), size={reference_assets.size_mode}; persistent across all chunks.")
|
||||||
if reference_warning: reuse_notes.append(reference_warning)
|
if reference_warning: reuse_notes.append(reference_warning)
|
||||||
@@ -239,6 +451,10 @@ def run_sequence(*,model:Any,clip:Any,video_vae:Any,audio_vae:Any,sampler:Any,si
|
|||||||
reference_video_assets=encode_reference_video(video_vae,reference_video_source)
|
reference_video_assets=encode_reference_video(video_vae,reference_video_source)
|
||||||
if timeline_video_source is None:
|
if timeline_video_source is None:
|
||||||
cache=_conditioning_cache(clip=clip,prompts=prompts,assets=assets,final_has_last_frame=last_frame is not None,reference_assets=reference_assets,reference_audio_assets=reference_audio_assets,timeline_video_assets=reference_video_assets)
|
cache=_conditioning_cache(clip=clip,prompts=prompts,assets=assets,final_has_last_frame=last_frame is not None,reference_assets=reference_assets,reference_audio_assets=reference_audio_assets,timeline_video_assets=reference_video_assets)
|
||||||
|
if terminal_merge_enabled and terminal_prompt is not None:
|
||||||
|
terminal_key=_conditioning_cache_key(terminal_prompt,include_last=True,reference_assets=reference_assets)
|
||||||
|
if terminal_key not in cache:
|
||||||
|
cache.update(_conditioning_cache(clip=clip,prompts=[terminal_prompt],assets=assets,final_has_last_frame=True,reference_assets=reference_assets,reference_audio_assets=reference_audio_assets,timeline_video_assets=reference_video_assets))
|
||||||
entries=preserved[:]; previous_state=None
|
entries=preserved[:]; previous_state=None
|
||||||
if entries:
|
if entries:
|
||||||
try: previous_state=entry_to_state(entries[-1])
|
try: previous_state=entry_to_state(entries[-1])
|
||||||
@@ -261,7 +477,8 @@ def run_sequence(*,model:Any,clip:Any,video_vae:Any,audio_vae:Any,sampler:Any,si
|
|||||||
_record_context_diagnostics(tracker=context_diagnostics,reports=sampling_reports,state=entry_to_state(reused_entry),continuity=continuity,reused=True)
|
_record_context_diagnostics(tracker=context_diagnostics,reports=sampling_reports,state=entry_to_state(reused_entry),continuity=continuity,reused=True)
|
||||||
elif previous_state is not None:
|
elif previous_state is not None:
|
||||||
_record_context_diagnostics(tracker=context_diagnostics,reports=sampling_reports,state=previous_state,continuity=continuity,reused=True)
|
_record_context_diagnostics(tracker=context_diagnostics,reports=sampling_reports,state=previous_state,continuity=continuity,reused=True)
|
||||||
for sequence_index in range(len(entries),chunks):
|
normal_indices,terminal_merge_pending=_terminal_sampling_plan(chunks=chunks,completed=len(entries),merge_enabled=terminal_merge_enabled)
|
||||||
|
for sequence_index in normal_indices:
|
||||||
prompt=prompts[sequence_index]; prompt_hash_value=prompt_hashes[sequence_index]; is_final=sequence_index==chunks-1; effective_reroll_nonce=int(reroll_nonce) if int(reroll_from_chunk)>0 and sequence_index+1>=int(reroll_from_chunk) else 0; seed=derive_chunk_seed(base_seed,sequence_index,effective_reroll_nonce); motion_score=0.0; video_context=None; audio_context=None; context_before=None
|
prompt=prompts[sequence_index]; prompt_hash_value=prompt_hashes[sequence_index]; is_final=sequence_index==chunks-1; effective_reroll_nonce=int(reroll_nonce) if int(reroll_from_chunk)>0 and sequence_index+1>=int(reroll_from_chunk) else 0; seed=derive_chunk_seed(base_seed,sequence_index,effective_reroll_nonce); motion_score=0.0; video_context=None; audio_context=None; context_before=None
|
||||||
timeline_video_assets=reference_video_assets
|
timeline_video_assets=reference_video_assets
|
||||||
chunk_cache=cache
|
chunk_cache=cache
|
||||||
@@ -283,8 +500,6 @@ def run_sequence(*,model:Any,clip:Any,video_vae:Any,audio_vae:Any,sampler:Any,si
|
|||||||
base_conditioning=attach_keyframes(chunk_cache[conditioning_key],frame_count=shape.total_frames,first_latent=chunk_assets.first_latent,last_latent=chunk_assets.last_latent if include_chunk_last else None)
|
base_conditioning=attach_keyframes(chunk_cache[conditioning_key],frame_count=shape.total_frames,first_latent=chunk_assets.first_latent,last_latent=chunk_assets.last_latent if include_chunk_last else None)
|
||||||
video_context,audio_context,grid_offset=select_context(previous_state,context_frames,include_audio=bool(audio_continuity)); context_before=context_fingerprint(video_context,audio_context)
|
video_context,audio_context,grid_offset=select_context(previous_state,context_frames,include_audio=bool(audio_continuity)); context_before=context_fingerprint(video_context,audio_context)
|
||||||
conditioning=prepare_conditioning(base_conditioning,video_context=video_context,audio_context=audio_context,audio_grid_offset=grid_offset,context_frames=context_frames,new_frame_count=shape.total_frames,first_frame_policy=POLICY_REPLACE,preserve_last_frame=True)
|
conditioning=prepare_conditioning(base_conditioning,video_context=video_context,audio_context=audio_context,audio_grid_offset=grid_offset,context_frames=context_frames,new_frame_count=shape.total_frames,first_frame_policy=POLICY_REPLACE,preserve_last_frame=True)
|
||||||
if multi_chunk_flf and is_final:
|
|
||||||
conditioning=_attach_terminal_flf_keyframes(conditioning,video_context=video_context,context_frames=context_frames,frame_count=shape.total_frames,last_latent=assets.last_latent)
|
|
||||||
clip_index=int(previous_state["clip_index"])+1; chunk_plan=make_plan(continuation=True,clip_index=clip_index,total_frames=shape.total_frames,trim_frames=context_frames,width=width,height=height,context_frames=context_frames,state_capacity_frames=largest_context_capacity(shape.net_new_frames),requested_extend_seconds=chunk_seconds,debug=debug)
|
clip_index=int(previous_state["clip_index"])+1; chunk_plan=make_plan(continuation=True,clip_index=clip_index,total_frames=shape.total_frames,trim_frames=context_frames,width=width,height=height,context_frames=context_frames,state_capacity_frames=largest_context_capacity(shape.net_new_frames),requested_extend_seconds=chunk_seconds,debug=debug)
|
||||||
driving_audio_latent=slice_driving_audio_latent(driving_audio_assets,cumulative_retained_before=retained_frames,total_frames=int(chunk_plan["total_frames"]),trim_frames=int(chunk_plan["trim_frames"]),fps=FPS)
|
driving_audio_latent=slice_driving_audio_latent(driving_audio_assets,cumulative_retained_before=retained_frames,total_frames=int(chunk_plan["total_frames"]),trim_frames=int(chunk_plan["trim_frames"]),fps=FPS)
|
||||||
conditioning=attach_driving_audio(conditioning,driving_audio_latent)
|
conditioning=attach_driving_audio(conditioning,driving_audio_latent)
|
||||||
@@ -298,6 +513,62 @@ def run_sequence(*,model:Any,clip:Any,video_vae:Any,audio_vae:Any,sampler:Any,si
|
|||||||
sampling_reports.append(f"chunk {sequence_index+1}/{chunks}: seed={seed}, frames={chunk_plan['total_frames']}, trim={chunk_plan['trim_frames']}, retained_total={retained_frames}, context={context_frames} ({reason}), motion={motion_score:.6f}, "+(f"interop=emitted actual_prefix={CONTINUUM_ACTUAL_PREFIX_STEPS} consumer=not_observable" if context_before is not None else "interop=not_emitted"))
|
sampling_reports.append(f"chunk {sequence_index+1}/{chunks}: seed={seed}, frames={chunk_plan['total_frames']}, trim={chunk_plan['trim_frames']}, retained_total={retained_frames}, context={context_frames} ({reason}), motion={motion_score:.6f}, "+(f"interop=emitted actual_prefix={CONTINUUM_ACTUAL_PREFIX_STEPS} consumer=not_observable" if context_before is not None else "interop=not_emitted"))
|
||||||
del sampled,latent,conditioning,chunk_model,chunk_cache
|
del sampled,latent,conditioning,chunk_model,chunk_cache
|
||||||
if timeline_video_source is not None and timeline_video_assets is not None: del timeline_video_assets
|
if timeline_video_source is not None and timeline_video_assets is not None: del timeline_video_assets
|
||||||
|
if terminal_merge_pending:
|
||||||
|
pair_start=chunks-2
|
||||||
|
if len(entries)!=pair_start:
|
||||||
|
raise SequenceRuntimeError(f"terminal merged pair expected {pair_start} completed chunks, got {len(entries)}")
|
||||||
|
prompt=terminal_prompt
|
||||||
|
seed_nonce=int(reroll_nonce) if int(reroll_from_chunk)>0 else 0
|
||||||
|
terminal_seed_plan=_terminal_physical_seed_plan(base_seed=base_seed,physical_window_start_index=pair_start,reroll_nonce=seed_nonce)
|
||||||
|
physical_seed=int(terminal_seed_plan["physical_seed"])
|
||||||
|
initial_pair=previous_state is None
|
||||||
|
contract=_terminal_pair_contract(initial_pair=initial_pair,chunk_seconds=chunk_seconds)
|
||||||
|
physical_frames=int(contract["physical_frames"])
|
||||||
|
physical_context_frames=int(contract["physical_context_frames"])
|
||||||
|
latent=empty_h3_latent(width,height,physical_frames)
|
||||||
|
conditioning_key=_conditioning_cache_key(prompt,include_last=True,reference_assets=reference_assets)
|
||||||
|
base_conditioning=attach_keyframes(cache[conditioning_key],frame_count=physical_frames,first_latent=assets.first_latent,last_latent=assets.last_latent)
|
||||||
|
video_context=None; audio_context=None; context_before=None; motion_score=0.0
|
||||||
|
if initial_pair:
|
||||||
|
conditioning=base_conditioning
|
||||||
|
physical_clip_index=1
|
||||||
|
reason="terminal merged initial 10-second sample"
|
||||||
|
else:
|
||||||
|
selected_context,motion_score,selected_reason=choose_context_frames(continuity,previous_state)
|
||||||
|
video_context,audio_context,grid_offset=select_context(previous_state,TERMINAL_MERGE_CONTEXT_FRAMES,include_audio=bool(audio_continuity))
|
||||||
|
context_before=context_fingerprint(video_context,audio_context)
|
||||||
|
conditioning=prepare_conditioning(base_conditioning,video_context=video_context,audio_context=audio_context,audio_grid_offset=grid_offset,context_frames=TERMINAL_MERGE_CONTEXT_FRAMES,new_frame_count=physical_frames,first_frame_policy=POLICY_REPLACE,preserve_last_frame=True)
|
||||||
|
physical_clip_index=int(previous_state["clip_index"])+1
|
||||||
|
reason=f"terminal merged 10-second sample with 22-frame context; continuity selection was {selected_context} ({selected_reason})"
|
||||||
|
driving_audio_latent=slice_driving_audio_latent(driving_audio_assets,cumulative_retained_before=retained_frames,total_frames=physical_frames,trim_frames=physical_context_frames,fps=FPS)
|
||||||
|
conditioning=attach_driving_audio(conditioning,driving_audio_latent)
|
||||||
|
chunk_model=clone_model_for_chunk(model,strict=bool(strict_compatibility),debug=bool(debug),chunk_index=physical_clip_index,context_frames=physical_context_frames if not initial_pair else None)
|
||||||
|
sampled=sample_chunk(model=chunk_model,conditioning=conditioning,latent=latent,sampler=sampler,sigmas=sigmas,seed=physical_seed,enable_preview=bool(enable_preview))
|
||||||
|
if context_before is not None and video_context is not None:
|
||||||
|
assert_context_unchanged(video_context,audio_context,context_before)
|
||||||
|
sampled_video,sampled_audio=latent_to_cpu(sampled)
|
||||||
|
logical_parts=_split_terminal_merged_latents(sampled_video,sampled_audio,contract)
|
||||||
|
sampling_reports.append(f"terminal physical sample: terminal_pair={pair_start+1}-{chunks}, physical_seed={physical_seed}, physical_frames={physical_frames}, sampling_passes=1, shared_physical_sample=true, terminal_prompt_policy={terminal_prompt_policy}, trim={physical_context_frames}, reason={reason}")
|
||||||
|
for offset,(video_part,audio_part) in enumerate(logical_parts):
|
||||||
|
sequence_index=pair_start+offset
|
||||||
|
total_frames=int(contract["logical_frames"][offset])
|
||||||
|
trim_frames=int(contract["logical_trims"][offset])
|
||||||
|
continuation=bool(trim_frames)
|
||||||
|
clip_index=physical_clip_index+offset
|
||||||
|
context_frames=trim_frames if continuation else 5
|
||||||
|
chunk_plan=make_plan(continuation=continuation,clip_index=clip_index,total_frames=total_frames,trim_frames=trim_frames,width=width,height=height,context_frames=context_frames,state_capacity_frames=largest_context_capacity(total_frames-trim_frames),requested_extend_seconds=chunk_seconds,debug=debug)
|
||||||
|
logical_latent=latent_from_cpu(video_part,audio_part)
|
||||||
|
logical_seed=int(terminal_seed_plan["logical_entry_seeds"][offset])
|
||||||
|
entry=make_chunk_entry(latent=logical_latent,plan=chunk_plan,prompt=prompts[sequence_index],prompt_hash=prompt_hashes[sequence_index],seed=logical_seed,context_frames=trim_frames,motion_score=motion_score,reused=False)
|
||||||
|
entries.append(entry)
|
||||||
|
previous_state=entry_to_state(entry)
|
||||||
|
_record_context_diagnostics(tracker=context_diagnostics,reports=sampling_reports,state=previous_state,continuity=continuity,reused=False)
|
||||||
|
if storage_controller is not None:
|
||||||
|
storage_controller.commit_chunk(entry,position=sequence_index)
|
||||||
|
retained_frames+=int(chunk_plan["net_frames"])
|
||||||
|
sampling_reports.append(f"chunk {sequence_index+1}/{chunks}: seed={logical_seed}, frames={total_frames}, trim={trim_frames}, retained_total={retained_frames}, context={trim_frames}, shared_physical_sample=terminal_10s_seed_v2")
|
||||||
|
del logical_latent
|
||||||
|
del sampled,sampled_video,sampled_audio,logical_parts,latent,conditioning,chunk_model
|
||||||
if len(entries)!=chunks: raise SequenceRuntimeError(f"internal sequence length mismatch: expected {chunks}, got {len(entries)}")
|
if len(entries)!=chunks: raise SequenceRuntimeError(f"internal sequence length mismatch: expected {chunks}, got {len(entries)}")
|
||||||
if latent_only:
|
if latent_only:
|
||||||
last_state=entry_to_state(entries[-1]); parent_id=session.get("session_id") if session is not None else None
|
last_state=entry_to_state(entries[-1]); parent_id=session.get("session_id") if session is not None else None
|
||||||
@@ -308,7 +579,7 @@ def run_sequence(*,model:Any,clip:Any,video_vae:Any,audio_vae:Any,sampler:Any,si
|
|||||||
if reference_video_source is not None: settings["reference_video_contract"]=reference_video_source.contract
|
if reference_video_source is not None: settings["reference_video_contract"]=reference_video_source.contract
|
||||||
if timeline_video_source is not None: settings["timeline_video_contract"]=timeline_video_source.contract
|
if timeline_video_source is not None: settings["timeline_video_contract"]=timeline_video_source.contract
|
||||||
new_session=make_session(chunks=entries,width=width,height=height,chunk_seconds=chunk_seconds,identity_hash=sequence_identity_hash,model_fingerprint_value=current_model_fingerprint,parent_session_id=parent_id,reroll_from_chunk=int(reroll_from_chunk),settings=settings)
|
new_session=make_session(chunks=entries,width=width,height=height,chunk_seconds=chunk_seconds,identity_hash=sequence_identity_hash,model_fingerprint_value=current_model_fingerprint,parent_session_id=parent_id,reroll_from_chunk=int(reroll_from_chunk),settings=settings)
|
||||||
report_lines=[f"H3 Continuum V3 {PACKAGE_VERSION}",f"Conditioning mode: {conditioning_mode_label(conditioning_mode)}.",prompt_plan_report(plan),"Decode: external ComfyUI Core VAE nodes; full raw AV chunks retained.",accelerators,"Execution: conditioning precomputed; single call-local MODEL clone per chunk; no internal VAE decode.",*reuse_notes]
|
report_lines=[f"H3 Continuum V3 {PACKAGE_VERSION}",f"Conditioning mode: {conditioning_display}.",prompt_plan_report(plan),"Decode: external ComfyUI Core VAE nodes; full raw AV chunks retained.",accelerators,"Execution: conditioning precomputed; single call-local MODEL clone per chunk; no internal VAE decode.",*reuse_notes]
|
||||||
if diagnostics_mode!=DIAGNOSTICS_OFF: report_lines.extend(sampling_reports)
|
if diagnostics_mode!=DIAGNOSTICS_OFF: report_lines.extend(sampling_reports)
|
||||||
report_lines.extend([session_summary(new_session),f"Output: {len(entries)} raw AV latent chunk(s); connect Core VAE Decode nodes, then H3 Continuum Assemble V3."])
|
report_lines.extend([session_summary(new_session),f"Output: {len(entries)} raw AV latent chunk(s); connect Core VAE Decode nodes, then H3 Continuum Assemble V3."])
|
||||||
return entries,last_state,new_session,"\n".join(report_lines)
|
return entries,last_state,new_session,"\n".join(report_lines)
|
||||||
|
|||||||
+3
-1
@@ -71,7 +71,9 @@ def assemble_decoded_chunks(
|
|||||||
if audio_seam not in AUDIO_SEAM_OPTIONS:
|
if audio_seam not in AUDIO_SEAM_OPTIONS:
|
||||||
raise ValueError(f"unknown Audio Seam mode: {audio_seam!r}")
|
raise ValueError(f"unknown Audio Seam mode: {audio_seam!r}")
|
||||||
|
|
||||||
chunk_plans = list(plan["chunks"])
|
chunk_plans = list(
|
||||||
|
plan["decode_groups"] if "decode_groups" in plan else plan["chunks"]
|
||||||
|
)
|
||||||
if len(images) != len(chunk_plans) or len(audio) != len(chunk_plans):
|
if len(images) != len(chunk_plans) or len(audio) != len(chunk_plans):
|
||||||
raise ValueError(
|
raise ValueError(
|
||||||
"decoded chunk count does not match the V3 assembly plan: "
|
"decoded chunk count does not match the V3 assembly plan: "
|
||||||
|
|||||||
+23
-3
@@ -338,13 +338,33 @@ class H3ContinuumSamplerV3:
|
|||||||
**clip_prompt_inputs,
|
**clip_prompt_inputs,
|
||||||
)
|
)
|
||||||
|
|
||||||
assembly_plan = make_assembly_plan(
|
from ..v2.sequence import _terminal_flf_merge_enabled
|
||||||
|
from .plan import prepare_physical_decode_entries
|
||||||
|
|
||||||
|
terminal_merged = _terminal_flf_merge_enabled(
|
||||||
|
multi_chunk_flf=(
|
||||||
|
first_frame is not None
|
||||||
|
and advanced_values["last_frame"] is not None
|
||||||
|
and len(entries) > 1
|
||||||
|
),
|
||||||
|
chunks=len(entries),
|
||||||
|
chunk_seconds=float(chunk_seconds),
|
||||||
|
prompt_hashes=[str(entry["prompt_hash"]) for entry in entries],
|
||||||
|
timeline_video_source=timeline_video_source,
|
||||||
|
)
|
||||||
|
decode_entries, assembly_plan = prepare_physical_decode_entries(
|
||||||
entries,
|
entries,
|
||||||
chunk_seconds=float(chunk_seconds),
|
chunk_seconds=float(chunk_seconds),
|
||||||
preserve_final_frame=advanced_values["last_frame"] is not None,
|
preserve_final_frame=advanced_values["last_frame"] is not None,
|
||||||
|
terminal_merged=terminal_merged,
|
||||||
|
)
|
||||||
|
video_latents = [{"samples": entry["video"]} for entry in decode_entries]
|
||||||
|
audio_latents = [{"samples": entry["audio"]} for entry in decode_entries]
|
||||||
|
if terminal_merged:
|
||||||
|
report = str(report).rstrip() + (
|
||||||
|
"\nDecode: terminal merged latent retained as one physical external "
|
||||||
|
"Core VAE decode group; logical Run Storage entries are unchanged."
|
||||||
)
|
)
|
||||||
video_latents = [{"samples": entry["video"]} for entry in entries]
|
|
||||||
audio_latents = [{"samples": entry["audio"]} for entry in entries]
|
|
||||||
result = {
|
result = {
|
||||||
"last_state": last_state,
|
"last_state": last_state,
|
||||||
"session": session,
|
"session": session,
|
||||||
|
|||||||
+148
@@ -141,3 +141,151 @@ def make_assembly_plan(*args, **kwargs):
|
|||||||
_enrich_assembly_plan(plan)
|
_enrich_assembly_plan(plan)
|
||||||
validate_assembly_plan(plan)
|
validate_assembly_plan(plan)
|
||||||
return plan
|
return plan
|
||||||
|
|
||||||
|
|
||||||
|
def _recombine_terminal_tensor(
|
||||||
|
first,
|
||||||
|
second,
|
||||||
|
*,
|
||||||
|
slices,
|
||||||
|
time_dim: int,
|
||||||
|
name: str,
|
||||||
|
):
|
||||||
|
"""Invert the terminal logical split without changing any tensor value."""
|
||||||
|
|
||||||
|
import torch
|
||||||
|
|
||||||
|
if not torch.is_tensor(first) or not torch.is_tensor(second):
|
||||||
|
raise ValueError(f"terminal {name} entries must be tensors")
|
||||||
|
if first.ndim != second.ndim:
|
||||||
|
raise ValueError(f"terminal {name} tensor ranks differ")
|
||||||
|
dim = int(time_dim)
|
||||||
|
if dim < 0:
|
||||||
|
dim += first.ndim
|
||||||
|
if dim < 0 or dim >= first.ndim:
|
||||||
|
raise ValueError(f"terminal {name} time dimension is invalid")
|
||||||
|
for axis, (left, right) in enumerate(zip(first.shape, second.shape)):
|
||||||
|
if axis != dim and int(left) != int(right):
|
||||||
|
raise ValueError(f"terminal {name} tensor geometry differs")
|
||||||
|
|
||||||
|
(first_start, first_stop), (second_start, second_stop) = slices
|
||||||
|
first_start = int(first_start)
|
||||||
|
first_stop = int(first_stop)
|
||||||
|
second_start = int(second_start)
|
||||||
|
second_stop = int(second_stop)
|
||||||
|
if first_start != 0 or second_start > first_stop or second_stop <= first_stop:
|
||||||
|
raise ValueError(f"terminal {name} split ranges do not form one physical unit")
|
||||||
|
if int(first.shape[dim]) != first_stop - first_start:
|
||||||
|
raise ValueError(f"terminal {name} first slice length is inconsistent")
|
||||||
|
if int(second.shape[dim]) != second_stop - second_start:
|
||||||
|
raise ValueError(f"terminal {name} second slice length is inconsistent")
|
||||||
|
|
||||||
|
overlap_start = max(first_start, second_start)
|
||||||
|
overlap_stop = min(first_stop, second_stop)
|
||||||
|
overlap_length = max(0, overlap_stop - overlap_start)
|
||||||
|
if overlap_length:
|
||||||
|
first_overlap = first.narrow(dim, overlap_start - first_start, overlap_length)
|
||||||
|
second_overlap = second.narrow(dim, overlap_start - second_start, overlap_length)
|
||||||
|
if not torch.equal(first_overlap, second_overlap):
|
||||||
|
raise ValueError(
|
||||||
|
f"terminal {name} overlap differs; refusing lossy physical reconstruction"
|
||||||
|
)
|
||||||
|
|
||||||
|
output_shape = list(first.shape)
|
||||||
|
output_shape[dim] = second_stop
|
||||||
|
output = first.new_empty(output_shape)
|
||||||
|
output.narrow(dim, first_start, first_stop - first_start).copy_(first)
|
||||||
|
output.narrow(dim, second_start, second_stop - second_start).copy_(second)
|
||||||
|
return output.contiguous()
|
||||||
|
|
||||||
|
|
||||||
|
def prepare_physical_decode_entries(
|
||||||
|
entries,
|
||||||
|
*,
|
||||||
|
chunk_seconds: float,
|
||||||
|
preserve_final_frame: bool,
|
||||||
|
terminal_merged: bool,
|
||||||
|
):
|
||||||
|
"""Return external-VAE decode units while preserving logical plan metadata."""
|
||||||
|
|
||||||
|
logical_entries = list(entries)
|
||||||
|
plan = make_assembly_plan(
|
||||||
|
logical_entries,
|
||||||
|
chunk_seconds=float(chunk_seconds),
|
||||||
|
preserve_final_frame=bool(preserve_final_frame),
|
||||||
|
)
|
||||||
|
if not terminal_merged:
|
||||||
|
return logical_entries, plan
|
||||||
|
if len(logical_entries) < 2:
|
||||||
|
raise ValueError("terminal physical decode requires two logical entries")
|
||||||
|
|
||||||
|
from ..v2.sequence import _terminal_pair_contract
|
||||||
|
|
||||||
|
initial_pair = len(logical_entries) == 2
|
||||||
|
contract = _terminal_pair_contract(
|
||||||
|
initial_pair=initial_pair,
|
||||||
|
chunk_seconds=float(chunk_seconds),
|
||||||
|
)
|
||||||
|
first_entry = logical_entries[-2]
|
||||||
|
second_entry = logical_entries[-1]
|
||||||
|
physical_video = _recombine_terminal_tensor(
|
||||||
|
first_entry["video"],
|
||||||
|
second_entry["video"],
|
||||||
|
slices=contract["video_slices"],
|
||||||
|
time_dim=2,
|
||||||
|
name="video",
|
||||||
|
)
|
||||||
|
physical_audio = _recombine_terminal_tensor(
|
||||||
|
first_entry["audio"],
|
||||||
|
second_entry["audio"],
|
||||||
|
slices=contract["audio_slices"],
|
||||||
|
time_dim=-1,
|
||||||
|
name="audio",
|
||||||
|
)
|
||||||
|
|
||||||
|
physical_entry = dict(first_entry)
|
||||||
|
physical_entry["video"] = physical_video
|
||||||
|
physical_entry["audio"] = physical_audio
|
||||||
|
decode_entries = [*logical_entries[:-2], physical_entry]
|
||||||
|
|
||||||
|
logical_chunks = list(plan["chunks"])
|
||||||
|
decode_groups = []
|
||||||
|
for chunk in logical_chunks[:-2]:
|
||||||
|
group = dict(chunk)
|
||||||
|
group["logical_chunk_indices"] = [int(chunk["chunk_index"])]
|
||||||
|
group["terminal_merged"] = False
|
||||||
|
decode_groups.append(group)
|
||||||
|
|
||||||
|
first_chunk = logical_chunks[-2]
|
||||||
|
second_chunk = logical_chunks[-1]
|
||||||
|
frame_start = int(first_chunk.get("frame_start", 0))
|
||||||
|
net_frames = int(contract["physical_frames"]) - int(
|
||||||
|
contract["physical_context_frames"]
|
||||||
|
)
|
||||||
|
terminal_group = dict(first_chunk)
|
||||||
|
terminal_group.update(
|
||||||
|
{
|
||||||
|
"total_frames": int(contract["physical_frames"]),
|
||||||
|
"trim_frames": int(contract["physical_context_frames"]),
|
||||||
|
"net_frames": net_frames,
|
||||||
|
"context_frames": int(contract["physical_context_frames"]),
|
||||||
|
"expected_video_latent_t": int(physical_video.shape[2]),
|
||||||
|
"expected_audio_latent_t": int(physical_audio.shape[-1]),
|
||||||
|
"frame_start": frame_start,
|
||||||
|
"frame_stop": frame_start + net_frames,
|
||||||
|
"logical_chunk_indices": [
|
||||||
|
int(first_chunk["chunk_index"]),
|
||||||
|
int(second_chunk["chunk_index"]),
|
||||||
|
],
|
||||||
|
"terminal_merged": True,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
decode_groups.append(terminal_group)
|
||||||
|
|
||||||
|
plan = dict(plan)
|
||||||
|
plan["decode_group_version"] = 1
|
||||||
|
plan["logical_chunk_count"] = len(logical_chunks)
|
||||||
|
plan["physical_decode_group_count"] = len(decode_groups)
|
||||||
|
plan["decode_groups"] = decode_groups
|
||||||
|
validate_assembly_plan(plan)
|
||||||
|
return decode_entries, plan
|
||||||
|
|||||||
+6
-2
@@ -390,7 +390,9 @@ def analyze_decoded_boundaries(
|
|||||||
"""Analyze every decoded chunk boundary without changing decoded content."""
|
"""Analyze every decoded chunk boundary without changing decoded content."""
|
||||||
|
|
||||||
plan = validate_assembly_plan(assembly_plan)
|
plan = validate_assembly_plan(assembly_plan)
|
||||||
chunks = list(plan["chunks"])
|
chunks = list(
|
||||||
|
plan["decode_groups"] if "decode_groups" in plan else plan["chunks"]
|
||||||
|
)
|
||||||
if len(images) != len(chunks):
|
if len(images) != len(chunks):
|
||||||
raise ValueError("decoded image count does not match the assembly plan")
|
raise ValueError("decoded image count does not match the assembly plan")
|
||||||
analyses: list[VideoBoundaryAnalysis] = []
|
analyses: list[VideoBoundaryAnalysis] = []
|
||||||
@@ -428,7 +430,9 @@ def correct_decoded_boundaries(
|
|||||||
"""Normalize qualified transient exposure shifts without changing motion."""
|
"""Normalize qualified transient exposure shifts without changing motion."""
|
||||||
|
|
||||||
plan = validate_assembly_plan(assembly_plan)
|
plan = validate_assembly_plan(assembly_plan)
|
||||||
chunks = list(plan["chunks"])
|
chunks = list(
|
||||||
|
plan["decode_groups"] if "decode_groups" in plan else plan["chunks"]
|
||||||
|
)
|
||||||
if len(images) != len(chunks):
|
if len(images) != len(chunks):
|
||||||
raise ValueError("decoded image count does not match the assembly plan")
|
raise ValueError("decoded image count does not match the assembly plan")
|
||||||
corrected = list(images)
|
corrected = list(images)
|
||||||
|
|||||||
Reference in New Issue
Block a user