diff --git a/PROJECT_STATE.md b/PROJECT_STATE.md index 283ac36..aca6e36 100644 --- a/PROJECT_STATE.md +++ b/PROJECT_STATE.md @@ -51,7 +51,8 @@ Static checks do not replace a real GPU generation. Record workflow, prompt, med - PASS `MiniMax_H3_Continuum_V31_00032_.mp4`: standard Ref2VA model; visual result accepted. - FAIL `MiniMax_H3_Continuum_V31_00034_.mp4`: Hybrid FLF+Reference; visible jumps near 5 s and 9 s. - FAIL `MiniMax_H3_Continuum_V31_00035_.mp4`: Ref2VA model plus FL2V Turbo 8-step LoRA; two visible flashes/jumps. -- Current FLF baseline is `context_bound_terminal_v1`: final-chunk context-tail and Last Frame anchors only. +- Current FLF candidate is `context_bound_terminal_last_only_v2`: the final chunk keeps its 22-frame continuation context and the single Last Frame anchor retained by `prepare_conditioning()`, without adding a duplicate context-tail boundary keyframe. +- GPU verification of this candidate is pending; it is not yet a stable baseline. - `context_bound_terminal_bridge_v2` is withdrawn because its synthetic intermediate latent correlated with the later terminal flash regression. - Preserve Hybrid/Reference support, Run Storage identity, UI, audio paths, and permissive prompt handling when repairing FLF. @@ -78,3 +79,48 @@ Static checks do not replace a real GPU generation. Record workflow, prompt, med - The initial rollback after 00038 temporarily returned to bridge v2, but the user clarified that the required target is the exact code state used for PASS 00031. - `00031` completed at 2026-08-21 02:32:59; `pre-flf-terminal-bridge-v2-repo-20260821_023553` was captured immediately afterward and before bridge v2 was introduced. - Current FLF strategy is restored exactly from that snapshot: `context_bound_terminal_v1`. Bridge v2 is withdrawn and is not active. + +## M1 Hybrid diagnostics (2026-08-21) + +- Canonical implementation source: ComfyUI_W runtime. +- Runtime pre-change snapshot: D:\Codex\_snapshots\ComfyUI-H3-Continuum\pre-m1-comfyui-w-canonical-20260821_172324. +- Added display-only Hybrid conditioning labels; the stored/runtime conditioning mode remains unchanged. +- Hybrid reference warnings now use the actual global Qwen numbering. +- Added README numbering guidance and focused CPU-only regression tests. +- No Sampling, Conditioning payload, Run Storage contract, UI socket, or model-selection behavior changed. +- Tests were added but not executed in this implementation step. + +## Terminal Merge Physical Seed v2 (2026-08-22 00:49:57) +- Terminal Merge uses one physical seed per merged pair; an initial 2x5s pair uses base_seed directly when reroll is inactive. +- Later terminal pairs derive once from the zero-based pair-start index; both logical entries store that same physical seed. +- Run Storage distinguishes the scoped terminal_merged_10s_seed_v2 / physical_window_seed_v2 execution semantics without changing the global schema version. + + +## Terminal Merge diagnostic harness (2026-08-22) + +- Production generation semantics remain unchanged; diagnostics live under tests only. +- Diagnostic stages are ordered A Pre-Sampler, B First MODEL Call, C sampled physical AV, D split/recombine, E decode/assembly. +- Tensor identity uses shape, dtype, and SHA-256 of contiguous CPU raw bytes; numerical mismatch also records max/mean absolute difference and allclose. +- Terminal split/recombine requires bit-exact equality. +- Real Core vs Continuum GPU capture remains pending; this unit-test phase does not load checkpoints. + +## GPU Core/Continuum diagnostic runner (2026-08-22) +- Source: `tools/gpu_diagnostic_node/` +- Installer: `tools/install_gpu_diagnostic_node.ps1` +- Runtime: `D:\StabilityMatrix\Data\Packages\ComfyUI_W\custom_nodes\H3_Continuum_GPU_Diagnostics` +- Node: `H3 Core vs Continuum GPU Diagnostic` +- Production Continuum files and generation semantics are not modified by this diagnostic plugin. +- Captures A: pre-sampler, B: first MODEL call, C: physical sampled AV latent, D: terminal split/recombine, E1: decode comparison, E2: common 243-to-240 adjustment. +- Artifacts: JSON manifest plus safetensors under the configured ComfyUI output diagnostics directory. +- Pre-change snapshot: `D:\Codex\_snapshots\ComfyUI-H3-Continuum\pre-gpu-core-continuum-diagnostic-runner-20260822_022625` (138/138 files verified). +- GPU comparison has not yet been executed. + +## Long Terminal Merge for distinct timeline sections (2026-08-22) + +- For FL2VA with 3 or more 5-second logical chunks, the final two chunks may now share one physical Terminal Merge sample even when their prompt sections differ. +- Distinct final prompts are rebased to a local physical timeline (`[0-5s]` and `[5-10s]`); identical prompts retain the accepted 2x5 Core-equivalent prompt path unchanged. +- Expected 3x5 execution: 3 logical chunks, 2 physical sampling passes, and 2 physical VAE decode groups; the terminal group is 260 frames with a 22-frame decoded prefix trim and 238 retained frames. +- Timeline Video remains excluded from Terminal Merge. UI, public sockets, Conditioning payload construction, Physical Seed v2, Assembly, and Seam semantics are otherwise unchanged. +- GPU acceptance passed for `MiniMax_H3_Continuum_V31_00008_.mp4`: 3 logical chunks, 2 physical sampling passes, decoded groups of 124 and 238 retained frames, 362-to-360 exact-duration adjustment, and 15.000-second output. +- Core comparison `MiniMax_H3_00005_.mp4` showed the same short terminal still period. This is accepted as normal FL2VA Last Frame convergence, not a Continuum-specific defect. +- Long Terminal Merge is formally adopted. Do not retune Conditioning, Physical Seed v2, physical decode grouping, or terminal prompt pairing without new contrary regression evidence. diff --git a/README.md b/README.md index 8d18777..53ad9e6 100644 --- a/README.md +++ b/README.md @@ -15,6 +15,7 @@ V3.4 focuses on the two reference workflows that are most useful in normal produ - **Driving Audio**: use an existing audio source as the preserved final audio while it guides the H3 generation across chunks. - **Video Reference**: use an existing video as persistent visual reference for appearance, motion, framing, and timing without treating it as a frame-by-frame copy. - **Hybrid FLF + Reference**: use First Frame, Last Frame, and persistent Reference Images together without adding a separate mode or model allowlist. +- **FL2VA Terminal Merge**: keep the final two 5-second FL2VA chunks as one physical 10-second sampling and decode unit for Core-equivalent Last Frame handling. - **Restartable chunks**: reuse completed chunks with Run Storage and regenerate only the selected part when the generation contract remains compatible. - **Core compatibility**: remove Continuum-only rejection of unknown upstream/custom nodes and remove the obsolete `strict_compatibility` control. - **Spectrum interoperability**: use the official H3 Continuum Interop API v1 when Spectrum is installed; Spectrum remains optional. @@ -83,6 +84,18 @@ V3.4 supports First Frame, Last Frame, and persistent Reference Images in the sa Local checks passed with the B2049 hybrid variant. FL-only models could run, but showed less reliable transitions when FLF and Reference Images were combined; a hybrid-capable model is therefore recommended for this path. +### FL2VA Terminal Merge + +For 5-second FL2VA workflows, Continuum keeps the final two logical chunks together as one physical sampling and VAE decode unit. + +- `2 x 5s`: one 243-frame physical sample, decoded once, then adjusted to the requested 240-frame output. +- `3 x 5s` and longer: earlier chunks use normal Continuation; the final two chunks use one 260-frame physical sample containing the 22-frame continuation context. +- After the terminal physical decode, the 22-frame context prefix is removed. For `3 x 5s`, this gives `124 + 238 = 362` frames before exact-duration adjustment to 360 frames. +- If the final two prompt sections differ, they are rebased internally to a local `[0-5s]` / `[5-10s]` timeline. Identical prompts retain the accepted shared-prompt path. +- The terminal pair uses one physical seed and one physical decode while Run Storage continues to preserve normal logical chunk entries. + +Local GPU comparison confirmed that `2 x 5s` matches Core 10-second FL2VA sampling and terminal decode behavior. A `3 x 5s` Timeline run also passed with three logical chunks, two physical sampling passes, two physical decode groups, and a 15.000-second output. The short still period near the supplied Last Frame was also present in the equivalent Core result and is treated as normal FL2VA convergence rather than a Continuum-specific failure. + ### Core-first, permissive execution V3.4 removes or relaxes Continuum-only rejection rules that blocked otherwise runnable Core H3 experiments. @@ -295,6 +308,8 @@ V3.4 has been exercised locally with: - Spectrum enabled and disabled - I2VA, Reference, and selected FL2VA configurations - Hybrid FLF + Reference with the B2049 hybrid variant +- Core-equivalent `2 x 5s` FL2VA Terminal Merge +- `3 x 5s` FL2VA with the final two logical chunks merged into one 260-frame physical sample and decode group - Driving Audio with short, exact-length, and longer sources - Video Reference with source audio routed separately - 0.4 MP and 0.6 MP reference sizing @@ -308,6 +323,8 @@ No OOM was observed in the cited recent local V3.4 checks, including two-chunk 8 - Continuation does not guarantee frame-perfect identity or motion. - Video Reference guides H3; it does not reproduce every source frame. - Driving Audio preserves selected audio, but visual lip synchronization remains model-dependent. +- FL2VA may settle on the supplied Last Frame before the requested duration ends; equivalent Core runs can show the same behavior. +- Long, high-resolution sequences may exceed system RAM during final decode and assembly even when chunk sampling succeeds. - Match Output can be substantially slower than 0.4 MP or 0.6 MP. - Seam correction may keep the native boundary when a proposed correction is not safer. - Optional template nodes must be installed, replaced, or bypassed by the user. diff --git a/WORKLOG.md b/WORKLOG.md index 8b79159..4071348 100644 --- a/WORKLOG.md +++ b/WORKLOG.md @@ -60,3 +60,65 @@ D:\Codex\_snapshots\ComfyUI-H3-Continuum\pre-rollback-after-00038-fail-20260821_ - Restored only `v2/sequence.py` from `pre-flf-terminal-bridge-v2-repo-20260821_023553` to the repository and active ComfyUI_W runtime. - Active FLF strategy is `context_bound_terminal_v1`; bridge v2 is no longer active. - No UI or unrelated feature files were changed. GPU retest was not run. + +## 2026-08-21 - M1 Hybrid diagnostics + +- Used ComfyUI_W as the authoritative source after confirming repository/runtime differences. +- Created and verified a runtime snapshot before edits. +- Implemented display-only Hybrid labels and global Picture-number warnings. +- Documented Hybrid Picture numbering and added focused regression tests. +- Runtime generation behavior and public UI were intentionally left unchanged. +- Tests were not run in this implementation step. +## 2026-08-21 - FLF duplicate boundary anchor removal + +- Preserved the active ComfyUI_W M3 display-label and M4 Last Frame session checks while synchronizing `v2/sequence.py` to the repository. +- Changed the FLF contract to `context_bound_terminal_last_only_v2` so Run Storage cannot reuse chunks produced by the previous strategy. +- Removed only the final-chunk call that duplicated the continuation-context tail as a boundary keyframe. +- Kept the 22-frame continuation context and the single supplied Last Frame anchor retained by `prepare_conditioning()`. +- No UI, public sockets, Assembly, audio, Spectrum, or general continuation behavior changed. +- Automated and GPU validation were not run in this implementation step. + +- 2026-08-22 00:49:57 Terminal Merge Physical Seed v2 implemented locally after snapshot pre-change-20260822_004423. Changed v2/sequence.py and run_storage.py; added seed and revision regressions. No commit or push. + +- 2026-08-22 00:51:45 Validation: 220 passed, 1 pre-existing failure in test_v2_sequence_orchestration because conditioning_display_label is absent from conditioning.py. All Physical Seed v2 and Run Storage revision tests passed. GPU generation not run. + + +## 2026-08-22 - Terminal Merge staged diagnostics + +- Snapshot: D:\Codex\_snapshots\ComfyUI-H3-Continuum\pre-terminal-merge-diagnostics-20260822_020527 +- Added tests-only deterministic capture/comparison helpers for stages A-E. +- Added unit coverage for noise, PackedLayout/position_ids, sampled-stage ordering, numerical tolerance, and bit-exact 243f Terminal split/recombine. +- No Production code, UI, Conditioning, Seed v2, Terminal Merge, Assembly, or Seam behavior changed. +- Commit/push not performed. + +## 2026-08-22 - Standalone Core vs Continuum GPU diagnostics +- Added a standalone diagnostic node without changing Production runtime semantics. +- Added capture helpers and regression tests. +- Installed to ComfyUI_W as a separate custom node and verified source/runtime SHA-256 equality. +- Registration import: PASS (`H3CoreContinuumGpuDiagnostic`). +- Targeted tests: 6 passed. +- Full suite: 232 passed, 1 pre-existing unrelated failure (`conditioning_display_label` missing for unfinished M3 wiring). +- GPU execution: not run. +- Commit/push: not performed. +## 2026-08-22 - Terminal Merge physical decode groups + +- Preserved logical chunks for Session and Run Storage. +- Recombined terminal logical AV latent slices bit-exactly before external Core VAE Decode. +- Added assembly_plan.decode_groups so one terminal physical sample is decoded once. +- Assembly hardening and Video Seam now operate at physical decode-group boundaries. +- Sampling, Physical Seed v2, Conditioning, UI, sockets, and Run Storage schema were unchanged. +- Synced implementation to ComfyUI_W. +- Pending validation: targeted pytest, full pytest, and GPU Core-vs-Continuum decode comparison. + +## 2026-08-22 - Long Terminal Merge prompt pairing + +- Snapshots: `pre-long-terminal-merge-repo`, `pre-long-terminal-merge-comfyui-w`, and `pre-long-terminal-merge-comfyui-wan` under `D:\Codex\_snapshots\ComfyUI-H3-Continuum`. +- Removed only the final-prompt-hash equality gate from Terminal Merge eligibility. +- Added a scoped physical prompt policy: identical prompts remain byte-identical; distinct final logical prompts are rebased to `[0-5s]` / `[5-10s]` for one terminal physical sample. +- Added scoped Run Storage execution semantics and 3-logical-to-2-physical decode-group regression coverage. +- No UI, public sockets, model selection, Spectrum, Assembly, Seam, or global Run Storage schema changes. +- Focused regression result: 22 passed; GPU generation not yet run for the new 3x5 path. +- Runtime correction: `run_storage.py` was also synchronized to ComfyUI_W and ComfyUI_WAN after the first GPU attempt exposed an old `RunStorageController.prepare()` signature. Run Storage plus Terminal Merge regressions: 47 passed. +- GPU PASS: `MiniMax_H3_Continuum_V31_00008_.mp4` exercised 3x5s FL2VA with `terminal_pair=2-3`, `physical_frames=260`, `sampling_passes=1` for the pair, `paired_timeline_v1`, two physical decode groups, and final 360 frames / 15.000 seconds. +- Core comparison `MiniMax_H3_00005_.mp4` reproduced the same short terminal still period. Long Terminal Merge was formally accepted; terminal stillness is Core-equivalent FL2VA convergence and is not a Continuum repair target. +- Final release validation: full pytest `249 passed`; `tools/verify_runtime.py --comfy-root D:\StabilityMatrix\Data\Packages\ComfyUI_W` passed V3.4 registration, native PackedLayout, and Fixed 3x5s prompt-plan checks. GPU acceptance is recorded above. diff --git a/conditioning.py b/conditioning.py index b349246..892bf33 100644 --- a/conditioning.py +++ b/conditioning.py @@ -59,6 +59,46 @@ def conditioning_mode_label(mode: str) -> str: raise ValueError(f"unknown conditioning mode: {mode!r}") from exc +def conditioning_display_label( + *, + has_first: bool, + has_last: bool, + has_reference: bool, +) -> str: + """Return a user-facing label without changing the runtime mode contract.""" + if has_reference: + if has_first and has_last: + return "Hybrid FL2VA + Reference + Continuation" + if has_first: + return "Hybrid I2VA + Reference + Continuation" + if has_last: + return "Hybrid L2VA + Reference + Continuation" + return "Reference + Continuation" + mode = conditioning_mode_from_presence( + has_first=has_first, + has_last=has_last, + has_reference=False, + ) + return conditioning_mode_label(mode) + + +def conditioning_mode_display_label( + mode: str, + *, + has_first: bool = False, + has_last: bool = False, + has_reference: bool = False, +) -> str: + """Legacy display helper retained for existing callers and tests.""" + if has_first or has_last or has_reference: + return conditioning_display_label( + has_first=has_first, + has_last=has_last, + has_reference=has_reference, + ) + return conditioning_mode_label(mode) + + def conditioning_mode_uses_video_vae(mode: str) -> bool: if mode not in CONDITIONING_MODES: raise ValueError(f"unknown conditioning mode: {mode!r}") diff --git a/hardening.py b/hardening.py index 74a6a03..d519713 100644 --- a/hardening.py +++ b/hardening.py @@ -21,6 +21,15 @@ def _strict_int(name: str, value: Any, minimum: int | None = None) -> int: return value +def _assembly_decode_units(plan: Mapping[str, Any]) -> list[Any]: + if "decode_groups" not in plan: + return plan["chunks"] + groups = plan["decode_groups"] + if not isinstance(groups, list) or not groups: + raise ValueError("assembly_plan.decode_groups must be a non-empty list") + return groups + + def enrich_assembly_plan(plan: dict[str, Any]) -> dict[str, Any]: """Add redundant metadata without correcting any existing value.""" chunks = plan.get("chunks") @@ -184,6 +193,117 @@ def validate_assembly_plan_contract(plan: Mapping[str, Any]) -> None: "sum(chunk.net_frames) does not match assembly_plan.natural_frames" ) + if "decode_groups" in plan: + decode_groups = _assembly_decode_units(plan) + if "physical_decode_group_count" in plan: + group_count = _strict_int( + "assembly_plan.physical_decode_group_count", + plan["physical_decode_group_count"], + 1, + ) + if group_count != len(decode_groups): + raise ValueError( + "assembly_plan.physical_decode_group_count does not match decode_groups" + ) + expected_logical_indices = [int(chunk["chunk_index"]) for chunk in chunks] + covered_logical_indices: list[int] = [] + decode_natural_frames = 0 + for position, group in enumerate(decode_groups, start=1): + if not isinstance(group, Mapping): + raise ValueError( + f"assembly_plan decode group {position} must be a mapping" + ) + total_frames = _strict_int( + f"assembly_plan decode group {position}.total_frames", + group.get("total_frames"), + 5, + ) + trim_frames = _strict_int( + f"assembly_plan decode group {position}.trim_frames", + group.get("trim_frames"), + 0, + ) + net_frames = _strict_int( + f"assembly_plan decode group {position}.net_frames", + group.get("net_frames"), + 1, + ) + context_frames = _strict_int( + f"assembly_plan decode group {position}.context_frames", + group.get("context_frames"), + 0, + ) + _strict_int( + f"assembly_plan decode group {position}.expected_video_latent_t", + group.get("expected_video_latent_t"), + 1, + ) + _strict_int( + f"assembly_plan decode group {position}.expected_audio_latent_t", + group.get("expected_audio_latent_t"), + 1, + ) + if (total_frames - 5) % 17 != 0: + raise ValueError( + f"assembly_plan decode group {position}.total_frames is not on " + "the H3 17k+5 grid" + ) + if total_frames - trim_frames != net_frames: + raise ValueError( + f"assembly_plan decode group {position}: " + "total_frames - trim_frames != net_frames" + ) + if trim_frames > 0 and context_frames != trim_frames: + raise ValueError( + f"assembly_plan decode group {position}: " + "continuation context and trim differ" + ) + logical_indices = group.get("logical_chunk_indices") + if not isinstance(logical_indices, list) or not logical_indices: + raise ValueError( + f"assembly_plan decode group {position}.logical_chunk_indices " + "must be a non-empty list" + ) + covered_logical_indices.extend( + _strict_int( + f"assembly_plan decode group {position}.logical_chunk_indices", + value, + 1, + ) + for value in logical_indices + ) + if "terminal_merged" in group and type(group["terminal_merged"]) is not bool: + raise ValueError( + f"assembly_plan decode group {position}.terminal_merged must be boolean" + ) + frame_start = _strict_int( + f"assembly_plan decode group {position}.frame_start", + group.get("frame_start"), + 0, + ) + frame_stop = _strict_int( + f"assembly_plan decode group {position}.frame_stop", + group.get("frame_stop"), + 1, + ) + if ( + frame_start != decode_natural_frames + or frame_stop != decode_natural_frames + net_frames + ): + raise ValueError( + f"assembly_plan decode group {position}: natural frame boundary " + "is discontinuous" + ) + decode_natural_frames += net_frames + if covered_logical_indices != expected_logical_indices: + raise ValueError( + "assembly_plan.decode_groups do not cover logical chunks exactly once" + ) + if decode_natural_frames != natural_frames: + raise ValueError( + "sum(decode_group.net_frames) does not match logical natural frames" + ) + def preflight_decoded_chunks( images: Any, @@ -192,7 +312,7 @@ def preflight_decoded_chunks( ) -> None: """Validate all decoded chunks before the assembler allocates output buffers.""" validate_assembly_plan_contract(plan) - chunks = plan["chunks"] + chunks = _assembly_decode_units(plan) if not isinstance(images, (list, tuple)): raise ValueError("decoded images must be a chunk list") @@ -518,7 +638,9 @@ def assemble_with_hardening(base: Any, args: tuple[Any, ...], kwargs: dict[str, lines: list[str] = [] if diagnostics_is_full(diagnostics): lines.append(format_memory_snapshot("assemble preflight", images, audio)) - natural_frames = sum(int(chunk["net_frames"]) for chunk in plan["chunks"]) + natural_frames = sum( + int(chunk["net_frames"]) for chunk in _assembly_decode_units(plan) + ) target_frames = int(plan["target_frames"]) correction = target_frames - natural_frames if abs(correction) > 2: diff --git a/run_storage.py b/run_storage.py index 929a84d..e4adf3e 100644 --- a/run_storage.py +++ b/run_storage.py @@ -571,6 +571,7 @@ def build_sampling_contract( driving_audio_vae: Any = None, reference_video_contract: dict[str, Any] | None = None, timeline_video_contract: dict[str, Any] | None = None, + execution_semantics: dict[str, Any] | None = None, ) -> tuple[dict[str, Any], bool, list[str]]: model_value, model_safe = _model_signature(model, model_fingerprint_value) clip_value, clip_safe = _clip_signature(clip) @@ -660,6 +661,8 @@ def build_sampling_contract( } if graph_authoritative: global_contract["upstream_graph"] = dict(upstream_graph_contract) + if execution_semantics is not None: + global_contract["execution_semantics"] = dict(execution_semantics) if uses_video_vae: global_contract["video_vae"] = video_vae_value if has_first: @@ -1017,6 +1020,7 @@ class RunStorageController: driving_audio_vae: Any = None, reference_video_contract: dict[str, Any] | None = None, timeline_video_contract: dict[str, Any] | None = None, + execution_semantics: dict[str, Any] | None = None, ) -> dict[str, Any] | None: if existing_session is not None: raise RunStorageError("Run Storage cannot be combined with an explicit Session") @@ -1054,6 +1058,7 @@ class RunStorageController: driving_audio_vae=driving_audio_vae, reference_video_contract=reference_video_contract, timeline_video_contract=timeline_video_contract, + execution_semantics=execution_semantics, ) effective_nonce, nonce_decision = self._resolve_effective_nonce( contract, requested_nonce=int(reroll_nonce), resume_safe=safe diff --git a/tests/terminal_merge_diagnostics.py b/tests/terminal_merge_diagnostics.py new file mode 100644 index 0000000..b050682 --- /dev/null +++ b/tests/terminal_merge_diagnostics.py @@ -0,0 +1,378 @@ +"""Tests-only capture and comparison helpers for Terminal Merge diagnostics. + +This module deliberately has no production imports or runtime hooks. Tests can +monkeypatch Core and Continuum boundaries, feed the captured values here, and +identify the first pipeline stage that differs without changing generation +semantics. +""" + +from __future__ import annotations + +from dataclasses import dataclass +from enum import Enum +from hashlib import sha256 +import json +from pathlib import Path +from typing import Any, Mapping + +import torch + + +STAGE_ORDER = ( + "A_pre_sampler", + "B_first_model_call", + "C_sampled_physical_av", + "D_split_recombine", + "E_decode_assembly", +) + + +@dataclass(frozen=True) +class TensorFingerprint: + shape: tuple[int, ...] + dtype: str + sha256: str + + def as_dict(self) -> dict[str, Any]: + return { + "shape": list(self.shape), + "dtype": self.dtype, + "sha256": self.sha256, + } + + +@dataclass(frozen=True) +class TensorDifference: + same_shape: bool + same_dtype: bool + bit_exact: bool + allclose: bool + max_abs_diff: float | None + mean_abs_diff: float | None + + +@dataclass(frozen=True) +class StageSnapshot: + name: str + sha256: str + payload: Any + + +@dataclass(frozen=True) +class StageComparison: + name: str + equal: bool + core_sha256: str | None + continuum_sha256: str | None + missing_from: str | None = None + + +def _tensor_raw_bytes(tensor: torch.Tensor) -> bytes: + value = tensor.detach().to("cpu").contiguous() + if value.numel() == 0: + return b"" + return value.reshape(-1).view(torch.uint8).numpy().tobytes() + + +def tensor_fingerprint(tensor: torch.Tensor) -> TensorFingerprint: + if not torch.is_tensor(tensor): + raise TypeError("tensor_fingerprint expects a torch.Tensor") + return TensorFingerprint( + shape=tuple(int(size) for size in tensor.shape), + dtype=str(tensor.dtype), + sha256=sha256(_tensor_raw_bytes(tensor)).hexdigest(), + ) + + +def tensor_difference( + core: torch.Tensor, + continuum: torch.Tensor, + *, + rtol: float = 1.0e-5, + atol: float = 1.0e-6, +) -> TensorDifference: + if not torch.is_tensor(core) or not torch.is_tensor(continuum): + raise TypeError("tensor_difference expects two torch.Tensor values") + + same_shape = tuple(core.shape) == tuple(continuum.shape) + same_dtype = core.dtype == continuum.dtype + bit_exact = ( + same_shape + and same_dtype + and tensor_fingerprint(core).sha256 == tensor_fingerprint(continuum).sha256 + ) + if not same_shape: + return TensorDifference( + same_shape=False, + same_dtype=same_dtype, + bit_exact=False, + allclose=False, + max_abs_diff=None, + mean_abs_diff=None, + ) + + core_value = core.detach().to("cpu") + continuum_value = continuum.detach().to("cpu") + if core_value.is_complex() or continuum_value.is_complex(): + core_numeric = core_value.to(torch.complex128) + continuum_numeric = continuum_value.to(torch.complex128) + else: + core_numeric = core_value.to(torch.float64) + continuum_numeric = continuum_value.to(torch.float64) + + if core_numeric.numel() == 0: + max_abs_diff = 0.0 + mean_abs_diff = 0.0 + else: + absolute = (core_numeric - continuum_numeric).abs() + max_abs_diff = float(absolute.max().item()) + mean_abs_diff = float(absolute.mean().item()) + + return TensorDifference( + same_shape=True, + same_dtype=same_dtype, + bit_exact=bit_exact, + allclose=bool( + torch.allclose( + core_numeric, + continuum_numeric, + rtol=rtol, + atol=atol, + equal_nan=True, + ) + ), + max_abs_diff=max_abs_diff, + mean_abs_diff=mean_abs_diff, + ) + + +def packed_layout_snapshot(layout: Any) -> dict[str, Any]: + return { + "signature": stable_snapshot(getattr(layout, "signature", None)), + "segments": stable_snapshot(getattr(layout, "segments", None)), + "position_ids": stable_snapshot(getattr(layout, "position_ids", None)), + } + + +def stable_snapshot(value: Any) -> Any: + """Convert diagnostic data to deterministic JSON-compatible state.""" + + if torch.is_tensor(value): + return {"kind": "tensor", **tensor_fingerprint(value).as_dict()} + if isinstance(value, TensorFingerprint): + return {"kind": "tensor", **value.as_dict()} + if value is None or isinstance(value, (bool, int, str)): + return value + if isinstance(value, float): + return {"kind": "float", "hex": value.hex()} + if isinstance(value, bytes): + return { + "kind": "bytes", + "length": len(value), + "sha256": sha256(value).hexdigest(), + } + if isinstance(value, (Path, torch.device, torch.dtype)): + return str(value) + if isinstance(value, Enum): + return { + "kind": "enum", + "type": f"{type(value).__module__}.{type(value).__qualname__}", + "value": stable_snapshot(value.value), + } + if isinstance(value, slice): + return { + "kind": "slice", + "start": value.start, + "stop": value.stop, + "step": value.step, + } + if isinstance(value, Mapping): + return { + "kind": "mapping", + "items": [ + [str(key), stable_snapshot(item)] + for key, item in sorted(value.items(), key=lambda pair: str(pair[0])) + ], + } + if isinstance(value, tuple): + return {"kind": "tuple", "items": [stable_snapshot(item) for item in value]} + if isinstance(value, list): + return {"kind": "list", "items": [stable_snapshot(item) for item in value]} + if isinstance(value, (set, frozenset)): + items = [stable_snapshot(item) for item in value] + items.sort(key=lambda item: json.dumps(item, sort_keys=True, separators=(",", ":"))) + return {"kind": type(value).__name__, "items": items} + if callable(value): + return { + "kind": "callable", + "module": getattr(value, "__module__", type(value).__module__), + "qualname": getattr(value, "__qualname__", type(value).__qualname__), + } + if all(hasattr(value, name) for name in ("signature", "segments", "position_ids")): + return {"kind": "packed_layout", **packed_layout_snapshot(value)} + if hasattr(value, "__dict__"): + public_state = { + key: item + for key, item in vars(value).items() + if not key.startswith("_") and not callable(item) + } + return { + "kind": "object", + "type": f"{type(value).__module__}.{type(value).__qualname__}", + "state": stable_snapshot(public_state), + } + raise TypeError(f"Unsupported diagnostic value: {type(value)!r}") + + +def snapshot_sha256(payload: Any) -> str: + normalized = stable_snapshot(payload) + encoded = json.dumps( + normalized, + ensure_ascii=True, + sort_keys=True, + separators=(",", ":"), + ).encode("utf-8") + return sha256(encoded).hexdigest() + + +class DiagnosticCapture: + def __init__(self) -> None: + self._stages: dict[str, StageSnapshot] = {} + + @property + def stages(self) -> Mapping[str, StageSnapshot]: + return self._stages + + def capture(self, stage: str, **payload: Any) -> StageSnapshot: + if stage not in STAGE_ORDER: + raise ValueError(f"Unknown diagnostic stage: {stage}") + if stage in self._stages: + raise ValueError(f"Diagnostic stage already captured: {stage}") + normalized = stable_snapshot(payload) + snapshot = StageSnapshot( + name=stage, + sha256=snapshot_sha256(normalized), + payload=normalized, + ) + self._stages[stage] = snapshot + return snapshot + + def capture_pre_sampler( + self, + *, + physical_frames: int, + seed: int, + conditioning: Any, + latent: Any, + noise: torch.Tensor, + keyframes: Any = None, + qwen_images: Any = None, + minimax_refs: Any = None, + ) -> StageSnapshot: + return self.capture( + "A_pre_sampler", + physical_frames=physical_frames, + seed=seed, + conditioning=conditioning, + latent=latent, + noise=noise, + keyframes=keyframes, + qwen_images=qwen_images, + minimax_refs=minimax_refs, + ) + + def capture_first_model_call( + self, + *, + minimax_payload: Any, + transformer_options: Any, + ) -> StageSnapshot: + return self.capture( + "B_first_model_call", + minimax_payload=minimax_payload, + transformer_options=transformer_options, + ) + + def capture_sampled_physical_av( + self, + *, + video: torch.Tensor, + audio: torch.Tensor, + ) -> StageSnapshot: + return self.capture( + "C_sampled_physical_av", + video=video, + audio=audio, + ) + + def capture_split_recombine( + self, + *, + physical_video: torch.Tensor, + physical_audio: torch.Tensor, + recombined_video: torch.Tensor, + recombined_audio: torch.Tensor, + ) -> StageSnapshot: + return self.capture( + "D_split_recombine", + physical_video=physical_video, + physical_audio=physical_audio, + recombined_video=recombined_video, + recombined_audio=recombined_audio, + video_difference=tensor_difference(physical_video, recombined_video).__dict__, + audio_difference=tensor_difference(physical_audio, recombined_audio).__dict__, + ) + + def capture_decode_assembly( + self, + *, + physical_decoded: torch.Tensor, + assembled_decoded: torch.Tensor, + ) -> StageSnapshot: + return self.capture( + "E_decode_assembly", + physical_decoded=physical_decoded, + assembled_decoded=assembled_decoded, + difference=tensor_difference(physical_decoded, assembled_decoded).__dict__, + ) + + +def compare_captures( + core: DiagnosticCapture, + continuum: DiagnosticCapture, +) -> tuple[StageComparison, ...]: + comparisons = [] + for stage in STAGE_ORDER: + core_stage = core.stages.get(stage) + continuum_stage = continuum.stages.get(stage) + missing = None + if core_stage is None and continuum_stage is None: + missing = "both" + elif core_stage is None: + missing = "core" + elif continuum_stage is None: + missing = "continuum" + comparisons.append( + StageComparison( + name=stage, + equal=( + core_stage is not None + and continuum_stage is not None + and core_stage.sha256 == continuum_stage.sha256 + ), + core_sha256=None if core_stage is None else core_stage.sha256, + continuum_sha256=None if continuum_stage is None else continuum_stage.sha256, + missing_from=missing, + ) + ) + return tuple(comparisons) + + +def first_divergent_stage( + core: DiagnosticCapture, + continuum: DiagnosticCapture, +) -> str | None: + for comparison in compare_captures(core, continuum): + if not comparison.equal: + return comparison.name + return None diff --git a/tests/test_gpu_diagnostic_runner.py b/tests/test_gpu_diagnostic_runner.py new file mode 100644 index 0000000..35217c2 --- /dev/null +++ b/tests/test_gpu_diagnostic_runner.py @@ -0,0 +1,81 @@ +from __future__ import annotations + +import importlib.util +import sys +from pathlib import Path + +import torch + + +MODULE_PATH = Path(__file__).parents[1] / "tools" / "gpu_diagnostic_node" / "capture.py" +SPEC = importlib.util.spec_from_file_location("h3_gpu_diagnostic_capture", MODULE_PATH) +capture = importlib.util.module_from_spec(SPEC) +assert SPEC.loader is not None +sys.modules[SPEC.name] = capture +SPEC.loader.exec_module(capture) + + +def test_tensor_fingerprint_is_deterministic(): + value = torch.arange(12, dtype=torch.float32).reshape(1, 3, 4) + assert capture.tensor_fingerprint(value) == capture.tensor_fingerprint(value.clone()) + + +def test_tensor_comparison_records_numerical_difference(): + left = torch.tensor([1.0, 2.0], dtype=torch.float32) + right = torch.tensor([1.0, 2.000001], dtype=torch.float32) + result = capture.compare_tensors(left, right) + assert not result.bit_exact + assert result.allclose + assert result.matches + assert result.max_abs_diff > 0.0 + + +def test_terminal_video_recombine_is_bit_exact(): + physical = torch.arange(72, dtype=torch.float32).reshape(1, 1, 72, 1, 1) + first = physical[:, :, 0:37] + second = physical[:, :, 30:72] + rebuilt = capture.recombine_terminal_streams( + first, + second, + first_range=(0, 37), + second_range=(30, 72), + dim=2, + ) + assert torch.equal(rebuilt, physical) + + +def test_terminal_audio_recombine_is_bit_exact(): + physical = torch.arange(405, dtype=torch.float32).reshape(1, 1, 405) + first = physical[..., 0:207] + second = physical[..., 170:405] + rebuilt = capture.recombine_terminal_streams( + first, + second, + first_range=(0, 207), + second_range=(170, 405), + dim=-1, + ) + assert torch.equal(rebuilt, physical) + + +def test_stage_comparison_reports_first_field(): + left = capture.DiagnosticCapture("core") + right = capture.DiagnosticCapture("continuum") + left.add_value("A_pre_sampler", "seed", 42) + right.add_value("A_pre_sampler", "seed", 43) + left.add_tensor("A_pre_sampler", "sigmas", torch.ones(2)) + right.add_tensor("A_pre_sampler", "sigmas", torch.ones(2)) + result = capture.compare_stage(left, right, "A_pre_sampler") + assert result["status"] == "DIFFER" + assert result["first_field"] == "seed" + + +def test_summary_uses_stage_order_and_first_divergence(tmp_path): + comparison = { + "A_pre_sampler": {"status": "MATCH", "first_field": None}, + "B_first_model_call": {"status": "DIFFER", "first_field": "position_ids"}, + } + text = capture.format_comparison(comparison, tmp_path) + assert "A_pre_sampler: MATCH" in text + assert "B_first_model_call: DIFFER" in text + assert "First divergence: B_first_model_call.position_ids" in text diff --git a/tests/test_m3_conditioning_display.py b/tests/test_m3_conditioning_display.py new file mode 100644 index 0000000..58ceb2c --- /dev/null +++ b/tests/test_m3_conditioning_display.py @@ -0,0 +1,59 @@ +from __future__ import annotations + +import pytest + +from ComfyUI_H3_Continuum_Join.conditioning import ( + conditioning_display_label, + conditioning_mode_from_presence, +) + + +@pytest.mark.parametrize( + ("has_first", "has_last", "has_reference", "expected"), + [ + (False, False, False, "T2VA + Continuation"), + (True, False, False, "I2VA + Continuation"), + (False, True, False, "Last Frame + Continuation"), + (True, True, False, "FL2VA + Continuation"), + (False, False, True, "Reference + Continuation"), + (True, False, True, "Hybrid I2VA + Reference + Continuation"), + (False, True, True, "Hybrid L2VA + Reference + Continuation"), + (True, True, True, "Hybrid FL2VA + Reference + Continuation"), + ], +) +def test_conditioning_display_label_matrix( + has_first: bool, + has_last: bool, + has_reference: bool, + expected: str, +) -> None: + assert ( + conditioning_display_label( + has_first=has_first, + has_last=has_last, + has_reference=has_reference, + ) + == expected + ) + + +@pytest.mark.parametrize( + ("has_first", "has_last"), + [ + (True, False), + (False, True), + (True, True), + ], +) +def test_hybrid_display_does_not_change_internal_reference_mode( + has_first: bool, + has_last: bool, +) -> None: + assert ( + conditioning_mode_from_presence( + has_first=has_first, + has_last=has_last, + has_reference=True, + ) + == "reference" + ) \ No newline at end of file diff --git a/tests/test_terminal_merge_diagnostics.py b/tests/test_terminal_merge_diagnostics.py new file mode 100644 index 0000000..65ab724 --- /dev/null +++ b/tests/test_terminal_merge_diagnostics.py @@ -0,0 +1,183 @@ +from __future__ import annotations + +from types import SimpleNamespace + +import torch + +from ComfyUI_H3_Continuum_Join.v2.sequence import ( + _split_terminal_merged_latents, + _terminal_pair_contract, +) +from ComfyUI_H3_Continuum_Join.tests.terminal_merge_diagnostics import ( + DiagnosticCapture, + first_divergent_stage, + tensor_difference, + tensor_fingerprint, +) + + +def _capture_all_stages(*, sampled_offset: float = 0.0) -> DiagnosticCapture: + capture = DiagnosticCapture() + capture.capture_pre_sampler( + physical_frames=243, + seed=42, + conditioning=[[torch.tensor([1.0]), {"minimax_frame_count": 243}]], + latent={"video": torch.zeros(1, 24, 72, 2, 2)}, + noise=torch.arange(8, dtype=torch.float32), + keyframes=(0, 242), + qwen_images=("first", "last"), + minimax_refs=None, + ) + layout = SimpleNamespace( + signature=(1, 2, 3), + segments=(("target_video", 0, 72),), + position_ids=torch.arange(12, dtype=torch.float32).reshape(6, 2), + ) + capture.capture_first_model_call( + minimax_payload={ + "layout": layout, + "cond_video_latents": torch.ones(1, 24, 2, 2, 2), + "target_video_shape": (1, 24, 72, 2, 2), + "target_audio_shape": (1, 32, 2, 405), + }, + transformer_options={"patches": {}}, + ) + video = torch.full((1, 24, 72, 2, 2), sampled_offset) + audio = torch.zeros(1, 32, 2, 405) + capture.capture_sampled_physical_av(video=video, audio=audio) + capture.capture_split_recombine( + physical_video=video, + physical_audio=audio, + recombined_video=video.clone(), + recombined_audio=audio.clone(), + ) + decoded = torch.zeros(243, 2, 2, 3) + capture.capture_decode_assembly( + physical_decoded=decoded, + assembled_decoded=decoded.clone(), + ) + return capture + + +def test_tensor_fingerprint_is_deterministic_and_dtype_sensitive(): + value = torch.arange(12, dtype=torch.float32).reshape(3, 4) + first = tensor_fingerprint(value) + second = tensor_fingerprint(value.clone()) + + assert first == second + assert first.shape == (3, 4) + assert first.dtype == "torch.float32" + assert first.sha256 != tensor_fingerprint(value.to(torch.float64)).sha256 + + +def test_tensor_difference_keeps_hash_mismatch_distinct_from_allclose(): + core = torch.tensor([1.0], dtype=torch.float32) + continuum = torch.tensor([1.0 + 1.0e-7], dtype=torch.float32) + + difference = tensor_difference(core, continuum) + + assert difference.bit_exact is False + assert difference.allclose is True + assert difference.max_abs_diff is not None + assert difference.max_abs_diff > 0.0 + assert difference.mean_abs_diff == difference.max_abs_diff + + +def test_pre_sampler_capture_includes_initial_noise(): + core = _capture_all_stages() + continuum = _capture_all_stages() + assert first_divergent_stage(core, continuum) is None + + changed = DiagnosticCapture() + changed.capture_pre_sampler( + physical_frames=243, + seed=42, + conditioning=[[torch.tensor([1.0]), {"minimax_frame_count": 243}]], + latent={"video": torch.zeros(1, 24, 72, 2, 2)}, + noise=torch.arange(8, dtype=torch.float32) + 1.0, + keyframes=(0, 242), + qwen_images=("first", "last"), + minimax_refs=None, + ) + + assert first_divergent_stage(core, changed) == "A_pre_sampler" + + +def test_first_model_call_capture_detects_packed_layout_difference(): + core = DiagnosticCapture() + continuum = DiagnosticCapture() + shared = dict( + physical_frames=243, + seed=42, + conditioning=[torch.zeros(1)], + latent=torch.zeros(1), + noise=torch.zeros(1), + ) + core.capture_pre_sampler(**shared) + continuum.capture_pre_sampler(**shared) + + core_layout = SimpleNamespace( + signature=(1, 2), + segments=(("target_video", 0, 72),), + position_ids=torch.zeros(4, 2), + ) + continuum_layout = SimpleNamespace( + signature=(1, 2), + segments=(("target_video", 0, 72),), + position_ids=torch.ones(4, 2), + ) + core.capture_first_model_call( + minimax_payload={"layout": core_layout}, + transformer_options={}, + ) + continuum.capture_first_model_call( + minimax_payload={"layout": continuum_layout}, + transformer_options={}, + ) + + assert first_divergent_stage(core, continuum) == "B_first_model_call" + + +def test_first_divergent_stage_uses_pipeline_order(): + core = _capture_all_stages(sampled_offset=0.0) + continuum = _capture_all_stages(sampled_offset=1.0) + + assert first_divergent_stage(core, continuum) == "C_sampled_physical_av" + + +def test_terminal_split_recombine_is_bit_exact(): + contract = _terminal_pair_contract(initial_pair=True, chunk_seconds=5.0) + video = torch.arange( + 1 * 24 * contract["video_slices"][-1][1] * 2 * 2, + dtype=torch.float32, + ).reshape(1, 24, contract["video_slices"][-1][1], 2, 2) + audio = torch.arange( + 1 * 32 * 2 * contract["audio_slices"][-1][1], + dtype=torch.float32, + ).reshape(1, 32, 2, contract["audio_slices"][-1][1]) + + (video_1, audio_1), (video_2, audio_2) = _split_terminal_merged_latents( + video, + audio, + contract, + ) + video_overlap = contract["video_slices"][0][1] - contract["video_slices"][1][0] + audio_overlap = contract["audio_slices"][0][1] - contract["audio_slices"][1][0] + recombined_video = torch.cat((video_1, video_2[:, :, video_overlap:]), dim=2) + recombined_audio = torch.cat((audio_1, audio_2[..., audio_overlap:]), dim=-1) + + assert contract["physical_frames"] == 243 + assert contract["logical_frames"] == (124, 141) + assert torch.equal(recombined_video, video) + assert torch.equal(recombined_audio, audio) + assert tensor_difference(video, recombined_video).bit_exact is True + assert tensor_difference(audio, recombined_audio).bit_exact is True + + capture = DiagnosticCapture() + stage = capture.capture_split_recombine( + physical_video=video, + physical_audio=audio, + recombined_video=recombined_video, + recombined_audio=recombined_audio, + ) + assert stage.name == "D_split_recombine" diff --git a/tests/test_v2_terminal_merge.py b/tests/test_v2_terminal_merge.py new file mode 100644 index 0000000..c0d28f2 --- /dev/null +++ b/tests/test_v2_terminal_merge.py @@ -0,0 +1,165 @@ +from __future__ import annotations + +import pytest +import torch + +from ComfyUI_H3_Continuum_Join.v2.sequence import ( + _atomic_terminal_prefix, + _split_terminal_merged_latents, + _terminal_flf_merge_enabled, + _terminal_pair_prompt, + _terminal_physical_seed_plan, + _terminal_pair_contract, + _terminal_sampling_plan, +) +from ComfyUI_H3_Continuum_Join.v2.seeds import derive_chunk_seed + + +@pytest.mark.parametrize( + ("initial_pair", "physical_frames", "logical_frames", "video_slices", "audio_slices"), + [ + (True, 243, (124, 141), ((0, 37), (30, 72)), ((0, 207), (170, 405))), + (False, 260, (141, 141), ((0, 42), (35, 77)), ((0, 235), (198, 433))), + ], +) +def test_terminal_pair_contract_uses_native_temporal_grid( + initial_pair, + physical_frames, + logical_frames, + video_slices, + audio_slices, +): + contract = _terminal_pair_contract(initial_pair=initial_pair, chunk_seconds=5.0) + assert contract["physical_frames"] == physical_frames + assert contract["logical_frames"] == logical_frames + assert contract["video_slices"] == video_slices + assert contract["audio_slices"] == audio_slices + + +@pytest.mark.parametrize("initial_pair", [True, False]) +def test_terminal_pair_split_preserves_expected_overlap(initial_pair): + contract = _terminal_pair_contract(initial_pair=initial_pair, chunk_seconds=5.0) + video_t = contract["video_slices"][-1][1] + audio_t = contract["audio_slices"][-1][1] + video = torch.arange(video_t, dtype=torch.float32).reshape(1, 1, video_t, 1, 1) + audio = torch.arange(audio_t, dtype=torch.float32).reshape(1, 1, 1, audio_t) + first, second = _split_terminal_merged_latents(video, audio, contract) + assert first[0].shape[2] == contract["video_slices"][0][1] + assert first[1].shape[-1] == contract["audio_slices"][0][1] + assert second[0][0, 0, 0, 0, 0].item() == contract["video_slices"][1][0] + assert second[1][0, 0, 0, 0].item() == contract["audio_slices"][1][0] + + +def test_two_by_five_uses_one_physical_sample(): + normal, terminal = _terminal_sampling_plan(chunks=2, completed=0, merge_enabled=True) + assert normal == () + assert terminal is True + assert len(normal) + int(terminal) == 1 + + +def test_three_by_five_uses_one_normal_and_one_terminal_sample(): + normal, terminal = _terminal_sampling_plan(chunks=3, completed=0, merge_enabled=True) + assert normal == (0,) + assert terminal is True + assert len(normal) + int(terminal) == 2 + + +def test_terminal_merge_accepts_distinct_prompts_but_not_timeline_video(): + common = dict(multi_chunk_flf=True, chunks=2, chunk_seconds=5.0) + assert _terminal_flf_merge_enabled(**common, prompt_hashes=["same", "same"], timeline_video_source=None) + assert _terminal_flf_merge_enabled(**common, prompt_hashes=["a", "b"], timeline_video_source=None) + assert not _terminal_flf_merge_enabled(**common, prompt_hashes=["same", "same"], timeline_video_source=object()) + + +def test_terminal_pair_prompt_preserves_shared_prompt_exactly(): + prompt, policy = _terminal_pair_prompt(["same", "same"], pair_start=0, chunk_seconds=5.0) + assert prompt == "same" + assert policy == "shared_prompt_v1" + + +def test_terminal_pair_prompt_rebases_distinct_sections_to_local_timeline(): + prompt, policy = _terminal_pair_prompt(["first action", "second action"], pair_start=0, chunk_seconds=5.0) + assert prompt == "[0-5s]\nfirst action\n\n[5-10s]\nsecond action" + assert policy == "paired_timeline_v1" + + +def test_partial_terminal_pair_is_discarded_for_atomic_resume(): + preserved = [{"chunk": 1}, {"chunk": 2}] + result, reset = _atomic_terminal_prefix(preserved, chunks=3, reroll_from_chunk=0) + assert result == preserved[:1] + assert reset is True + + +def test_rerolling_final_chunk_regenerates_both_terminal_chunks(): + preserved = [{"chunk": 1}, {"chunk": 2}, {"chunk": 3}] + result, reset = _atomic_terminal_prefix(preserved, chunks=3, reroll_from_chunk=3) + assert result == preserved[:1] + assert reset is True + + +def test_complete_terminal_pair_remains_reusable_without_reroll(): + preserved = [{"chunk": 1}, {"chunk": 2}, {"chunk": 3}] + result, reset = _atomic_terminal_prefix(preserved, chunks=3, reroll_from_chunk=0) + assert result == preserved + assert reset is False + + +@pytest.mark.parametrize("base_seed", [42, 123456789]) +def test_initial_terminal_pair_uses_base_seed_directly(base_seed): + plan = _terminal_physical_seed_plan( + base_seed=base_seed, + physical_window_start_index=0, + reroll_nonce=0, + ) + assert plan["physical_seed"] == base_seed + assert plan["logical_entry_seeds"] == (base_seed, base_seed) + + +def test_later_terminal_pair_derives_once_from_pair_start(): + plan = _terminal_physical_seed_plan( + base_seed=42, + physical_window_start_index=2, + reroll_nonce=0, + ) + assert plan["physical_seed"] == derive_chunk_seed(42, 2, 0) + assert plan["physical_seed"] == 1808285636802909398 + assert plan["logical_entry_seeds"] == (plan["physical_seed"],) * 2 + + +def test_terminal_pair_reroll_derives_once_and_shares_new_seed(): + original = _terminal_physical_seed_plan( + base_seed=42, + physical_window_start_index=2, + reroll_nonce=0, + ) + rerolled = _terminal_physical_seed_plan( + base_seed=42, + physical_window_start_index=2, + reroll_nonce=1, + ) + assert rerolled["physical_seed"] == derive_chunk_seed(42, 2, 1) + assert rerolled["physical_seed"] == 4306926667627841302 + assert rerolled["physical_seed"] != original["physical_seed"] + assert rerolled["logical_entry_seeds"] == (rerolled["physical_seed"],) * 2 + + +def test_non_terminal_seed_derivation_golden_values_are_unchanged(): + assert derive_chunk_seed(42, 0, 0) == 15348518480342951850 + assert derive_chunk_seed(42, 1, 0) == 13021120982267964226 + + +def test_last_frame_absent_does_not_enable_terminal_merge(): + assert not _terminal_flf_merge_enabled( + multi_chunk_flf=False, + chunks=2, + chunk_seconds=5.0, + prompt_hashes=["same", "same"], + timeline_video_source=None, + ) + + +def test_one_chunk_keeps_normal_sampling_and_seed_behavior(): + normal, terminal = _terminal_sampling_plan(chunks=1, completed=0, merge_enabled=False) + assert normal == (0,) + assert terminal is False + assert derive_chunk_seed(42, normal[0], 0) == 15348518480342951850 diff --git a/tests/test_v31_run_storage.py b/tests/test_v31_run_storage.py index d9e5d63..ef0e0c8 100644 --- a/tests/test_v31_run_storage.py +++ b/tests/test_v31_run_storage.py @@ -283,6 +283,7 @@ def _sampling_contract( last_frame_hash="", reference_contract=None, upstream_graph_contract=None, upstream_graph_safe=None, upstream_graph_reasons=None, + execution_semantics=None, ): return build_sampling_contract( model=model or _Model(), @@ -312,9 +313,69 @@ def _sampling_contract( upstream_graph_contract=upstream_graph_contract, upstream_graph_safe=upstream_graph_safe, upstream_graph_reasons=upstream_graph_reasons, + execution_semantics=execution_semantics, ) +def test_terminal_seed_v2_revision_is_scoped_and_distinct(): + old_contract, old_safe, old_reasons = _sampling_contract( + conditioning_mode="fl2va", + first_frame_hash="a" * 64, + last_frame_hash="b" * 64, + ) + new_contract, new_safe, new_reasons = _sampling_contract( + conditioning_mode="fl2va", + first_frame_hash="a" * 64, + last_frame_hash="b" * 64, + execution_semantics={ + "flf_execution": "terminal_merged_10s_seed_v2", + "terminal_seed_policy": "physical_window_seed_v2", + }, + ) + assert old_safe, old_reasons + assert new_safe, new_reasons + assert "execution_semantics" not in old_contract["global"] + assert new_contract["global"]["execution_semantics"] == { + "flf_execution": "terminal_merged_10s_seed_v2", + "terminal_seed_policy": "physical_window_seed_v2", + } + assert revision_identity(old_contract) != revision_identity(new_contract) + + +def test_terminal_seed_v2_revision_identity_is_deterministic(): + semantics = { + "flf_execution": "terminal_merged_10s_seed_v2", + "terminal_seed_policy": "physical_window_seed_v2", + } + model = _Model() + clip = _Clip() + video_vae = _VideoVAE() + sampler = _Sampler() + first, first_safe, first_reasons = _sampling_contract( + model=model, + clip=clip, + video_vae=video_vae, + sampler=sampler, + conditioning_mode="fl2va", + first_frame_hash="a" * 64, + last_frame_hash="b" * 64, + execution_semantics=semantics, + ) + second, second_safe, second_reasons = _sampling_contract( + model=model, + clip=clip, + video_vae=video_vae, + sampler=sampler, + conditioning_mode="fl2va", + first_frame_hash="a" * 64, + last_frame_hash="b" * 64, + execution_semantics=semantics, + ) + assert first_safe, first_reasons + assert second_safe, second_reasons + assert revision_identity(first) == revision_identity(second) + + def test_v322_golden_reference_contract_preserves_revision_identity(): model = _Model() clip = _Clip() diff --git a/tests/test_v3_physical_decode_groups.py b/tests/test_v3_physical_decode_groups.py new file mode 100644 index 0000000..251408a --- /dev/null +++ b/tests/test_v3_physical_decode_groups.py @@ -0,0 +1,149 @@ +from __future__ import annotations + +import pytest +import torch + +from ComfyUI_H3_Continuum_Join.v3 import plan as plan_module + + +def _logical_plan(): + return { + "chunks": [ + { + "sequence_index": 0, + "chunk_index": 1, + "total_frames": 124, + "trim_frames": 0, + "net_frames": 124, + "context_frames": 0, + "expected_video_latent_t": 37, + "expected_audio_latent_t": 207, + "frame_start": 0, + "frame_stop": 124, + }, + { + "sequence_index": 1, + "chunk_index": 2, + "total_frames": 141, + "trim_frames": 22, + "net_frames": 119, + "context_frames": 22, + "expected_video_latent_t": 42, + "expected_audio_latent_t": 235, + "frame_start": 124, + "frame_stop": 243, + }, + ], + "target_frames": 240, + } + + +def _terminal_entries(): + physical_video = torch.arange(72, dtype=torch.float32).reshape(1, 1, 72, 1, 1) + physical_audio = torch.arange(405, dtype=torch.float32).reshape(1, 1, 405) + entries = [ + { + "video": physical_video[:, :, 0:37].clone(), + "audio": physical_audio[..., 0:207].clone(), + }, + { + "video": physical_video[:, :, 30:72].clone(), + "audio": physical_audio[..., 170:405].clone(), + }, + ] + return entries, physical_video, physical_audio + + +def test_terminal_logical_entries_recombine_bit_exact(monkeypatch): + entries, physical_video, physical_audio = _terminal_entries() + monkeypatch.setattr( + plan_module, + "make_assembly_plan", + lambda *args, **kwargs: _logical_plan(), + ) + monkeypatch.setattr(plan_module, "validate_assembly_plan", lambda plan: plan) + + decode_entries, assembly_plan = plan_module.prepare_physical_decode_entries( + entries, + chunk_seconds=5.0, + preserve_final_frame=True, + terminal_merged=True, + ) + + assert len(decode_entries) == 1 + assert torch.equal(decode_entries[0]["video"], physical_video) + assert torch.equal(decode_entries[0]["audio"], physical_audio) + assert assembly_plan["logical_chunk_count"] == 2 + assert assembly_plan["physical_decode_group_count"] == 1 + assert assembly_plan["decode_groups"][0]["logical_chunk_indices"] == [1, 2] + assert assembly_plan["decode_groups"][0]["total_frames"] == 243 + assert assembly_plan["decode_groups"][0]["trim_frames"] == 0 + assert assembly_plan["decode_groups"][0]["net_frames"] == 243 + + +def test_terminal_recombine_rejects_changed_overlap(monkeypatch): + entries, _, _ = _terminal_entries() + entries[1]["video"][:, :, 0].add_(1.0) + monkeypatch.setattr( + plan_module, + "make_assembly_plan", + lambda *args, **kwargs: _logical_plan(), + ) + + with pytest.raises(ValueError, match="overlap differs"): + plan_module.prepare_physical_decode_entries( + entries, + chunk_seconds=5.0, + preserve_final_frame=True, + terminal_merged=True, + ) + + +def test_three_logical_chunks_use_two_physical_decode_groups(monkeypatch): + from ComfyUI_H3_Continuum_Join.v2.sequence import ( + _split_terminal_merged_latents, + _terminal_pair_contract, + ) + + contract = _terminal_pair_contract(initial_pair=False, chunk_seconds=5.0) + video_t = contract["video_slices"][-1][1] + audio_t = contract["audio_slices"][-1][1] + physical_video = torch.arange(video_t, dtype=torch.float32).reshape(1, 1, video_t, 1, 1) + physical_audio = torch.arange(audio_t, dtype=torch.float32).reshape(1, 1, audio_t) + terminal_parts = _split_terminal_merged_latents(physical_video, physical_audio, contract) + first_entry = { + "video": torch.zeros((1, 1, 37, 1, 1), dtype=torch.float32), + "audio": torch.zeros((1, 1, 207), dtype=torch.float32), + } + logical_plan = { + "chunks": [ + {"sequence_index": 0, "chunk_index": 1, "total_frames": 124, "trim_frames": 0, "net_frames": 124, "context_frames": 0, "expected_video_latent_t": 37, "expected_audio_latent_t": 207, "frame_start": 0, "frame_stop": 124}, + {"sequence_index": 1, "chunk_index": 2, "total_frames": 141, "trim_frames": 22, "net_frames": 119, "context_frames": 22, "expected_video_latent_t": 42, "expected_audio_latent_t": 235, "frame_start": 124, "frame_stop": 243}, + {"sequence_index": 2, "chunk_index": 3, "total_frames": 141, "trim_frames": 22, "net_frames": 119, "context_frames": 22, "expected_video_latent_t": 42, "expected_audio_latent_t": 235, "frame_start": 243, "frame_stop": 362}, + ], + "target_frames": 360, + } + entries = [ + first_entry, + {"video": terminal_parts[0][0], "audio": terminal_parts[0][1]}, + {"video": terminal_parts[1][0], "audio": terminal_parts[1][1]}, + ] + monkeypatch.setattr(plan_module, "make_assembly_plan", lambda *args, **kwargs: logical_plan) + monkeypatch.setattr(plan_module, "validate_assembly_plan", lambda plan: plan) + + decode_entries, assembly_plan = plan_module.prepare_physical_decode_entries( + entries, + chunk_seconds=5.0, + preserve_final_frame=True, + terminal_merged=True, + ) + + assert len(decode_entries) == 2 + assert torch.equal(decode_entries[1]["video"], physical_video) + assert torch.equal(decode_entries[1]["audio"], physical_audio) + assert assembly_plan["physical_decode_group_count"] == 2 + terminal_group = assembly_plan["decode_groups"][1] + assert terminal_group["logical_chunk_indices"] == [2, 3] + assert terminal_group["total_frames"] == 260 + assert terminal_group["trim_frames"] == 22 + assert terminal_group["net_frames"] == 238 diff --git a/tools/gpu_diagnostic_node/README.md b/tools/gpu_diagnostic_node/README.md new file mode 100644 index 0000000..1bd6cdb --- /dev/null +++ b/tools/gpu_diagnostic_node/README.md @@ -0,0 +1,32 @@ +# H3 Core vs Continuum GPU Diagnostic + +This is a standalone diagnostic node. It does not change or register through the +production H3 Continuum package. + +## Install for ComfyUI_W + +Run from the repository root: + +```powershell +.\tools\install_gpu_diagnostic_node.ps1 +``` + +Restart ComfyUI and search for `H3 Core vs Continuum GPU Diagnostic`. + +## Connections + +- Connect the same direct MODEL used by both tests. Keep Spectrum off. +- Connect the same CLIP, Video VAE, Audio VAE, sampler, and sigmas. +- Connect the exact same First Frame and Last Frame. +- Use 640x640, seed 42, 8 steps, Euler/simple for the baseline comparison. +- Run Storage and Session are not used by this node. + +One queue executes Core FL2VA at 243 frames, then Continuum Terminal Merge 2x5s. +Results are written below: + +```text +ComfyUI/output/h3_continuum/diagnostics// +``` + +The report identifies the first difference across pre-sampler data, first H3 +MODEL call, sampled physical latent, split/recombine, and decode/assembly. diff --git a/tools/gpu_diagnostic_node/__init__.py b/tools/gpu_diagnostic_node/__init__.py new file mode 100644 index 0000000..13db33d --- /dev/null +++ b/tools/gpu_diagnostic_node/__init__.py @@ -0,0 +1,9 @@ +"""Standalone GPU diagnostics for H3 Continuum. + +This package is intentionally separate from the production node package. Copy the +directory into ComfyUI/custom_nodes only while diagnostics are needed. +""" + +from .nodes import NODE_CLASS_MAPPINGS, NODE_DISPLAY_NAME_MAPPINGS + +__all__ = ["NODE_CLASS_MAPPINGS", "NODE_DISPLAY_NAME_MAPPINGS"] diff --git a/tools/gpu_diagnostic_node/capture.py b/tools/gpu_diagnostic_node/capture.py new file mode 100644 index 0000000..757ce93 --- /dev/null +++ b/tools/gpu_diagnostic_node/capture.py @@ -0,0 +1,302 @@ +"""Deterministic tensor capture and comparison helpers for the GPU runner.""" + +from __future__ import annotations + +from collections.abc import Mapping, Sequence +from dataclasses import asdict, dataclass +from hashlib import sha256 +import json +from pathlib import Path +import re +from typing import Any + +import torch + + +STAGE_ORDER = ( + "A_pre_sampler", + "B_first_model_call", + "C_physical_sample", + "D_split_recombine", + "E1_decode_243", + "E2_final_240", +) + + +def _dense_cpu(tensor: torch.Tensor) -> torch.Tensor: + return tensor.detach().to("cpu").contiguous() + + +def _raw_sha256(tensor: torch.Tensor) -> str: + value = _dense_cpu(tensor) + raw = value.view(torch.uint8).numpy().tobytes() + return sha256(raw).hexdigest() + + +def tensor_fingerprint(tensor: torch.Tensor) -> dict[str, Any]: + value = _dense_cpu(tensor) + return { + "shape": list(value.shape), + "dtype": str(value.dtype), + "sha256": _raw_sha256(value), + "numel": int(value.numel()), + } + + +@dataclass(frozen=True) +class TensorDifference: + comparable: bool + bit_exact: bool + allclose: bool + max_abs_diff: float | None + mean_abs_diff: float | None + reason: str | None = None + + @property + def matches(self) -> bool: + return self.comparable and (self.bit_exact or self.allclose) + + def to_dict(self) -> dict[str, Any]: + value = asdict(self) + value["matches"] = self.matches + return value + + +def compare_tensors( + left: torch.Tensor, + right: torch.Tensor, + *, + rtol: float = 1.0e-4, + atol: float = 1.0e-5, +) -> TensorDifference: + a = _dense_cpu(left) + b = _dense_cpu(right) + if tuple(a.shape) != tuple(b.shape): + return TensorDifference(False, False, False, None, None, "shape mismatch") + if a.dtype != b.dtype: + return TensorDifference(False, False, False, None, None, "dtype mismatch") + bit_exact = bool(torch.equal(a, b)) + if not (a.is_floating_point() or a.is_complex()): + return TensorDifference(True, bit_exact, bit_exact, 0.0 if bit_exact else None, 0.0 if bit_exact else None) + af = a.to(torch.float64) + bf = b.to(torch.float64) + delta = (af - bf).abs() + max_abs = float(delta.max().item()) if delta.numel() else 0.0 + mean_abs = float(delta.mean().item()) if delta.numel() else 0.0 + close = bool(torch.allclose(af, bf, rtol=rtol, atol=atol, equal_nan=True)) + return TensorDifference(True, bit_exact, close, max_abs, mean_abs) + + +def _safe_tensor_key(value: str) -> str: + return re.sub(r"[^A-Za-z0-9_.-]+", "_", value).strip("_") or "tensor" + + +def _is_primitive(value: Any) -> bool: + return value is None or isinstance(value, (bool, int, float, str)) + + +def _primitive_sequence(value: Any) -> bool: + return isinstance(value, (list, tuple)) and all(_is_primitive(item) for item in value) + + +class DiagnosticCapture: + def __init__(self, side: str): + self.side = side + self.manifest: dict[str, Any] = { + "format": "h3-core-continuum-gpu-diagnostic-v1", + "side": side, + "stages": {}, + } + self.tensors: dict[str, torch.Tensor] = {} + self.runtime: dict[str, Any] = {} + + def _stage(self, stage: str) -> dict[str, Any]: + return self.manifest["stages"].setdefault(stage, {"values": {}, "tensors": {}}) + + def add_value(self, stage: str, field: str, value: Any) -> None: + if not (_is_primitive(value) or _primitive_sequence(value)): + value = json.loads(json.dumps(value, default=str)) + self._stage(stage)["values"][field] = value + + def add_tensor(self, stage: str, field: str, tensor: torch.Tensor, *, persist: bool = True) -> None: + if not torch.is_tensor(tensor): + raise TypeError(f"{field} is not a tensor") + if getattr(tensor, "is_nested", False): + parts = list(tensor.unbind()) + self.add_value(stage, f"{field}.components", len(parts)) + for index, part in enumerate(parts): + self.add_tensor(stage, f"{field}.component_{index}", part, persist=persist) + return + value = _dense_cpu(tensor) + key = _safe_tensor_key(f"{stage}.{field}") + self._stage(stage)["tensors"][field] = { + **tensor_fingerprint(value), + "stored_as": key if persist else None, + } + if persist: + self.tensors[key] = value.clone() + + def capture_value( + self, + stage: str, + field: str, + value: Any, + *, + persist_tensors: bool = True, + depth: int = 0, + ) -> None: + if depth > 7: + self.add_value(stage, f"{field}.truncated", True) + return + if torch.is_tensor(value): + self.add_tensor(stage, field, value, persist=persist_tensors) + return + if _is_primitive(value): + self.add_value(stage, field, value) + return + if _primitive_sequence(value): + self.add_value(stage, field, list(value)) + return + if isinstance(value, Mapping): + keys = sorted(str(key) for key in value.keys()) + self.add_value(stage, f"{field}.keys", keys) + for key in keys[:256]: + try: + child = value[key] + except (KeyError, TypeError): + child = next((item for raw, item in value.items() if str(raw) == key), None) + self.capture_value( + stage, + f"{field}.{key}", + child, + persist_tensors=persist_tensors, + depth=depth + 1, + ) + return + if isinstance(value, Sequence) and not isinstance(value, (str, bytes, bytearray)): + self.add_value(stage, f"{field}.length", len(value)) + for index, child in enumerate(value[:256]): + self.capture_value( + stage, + f"{field}.{index}", + child, + persist_tensors=persist_tensors, + depth=depth + 1, + ) + return + if hasattr(value, "unbind") and value.__class__.__name__ == "NestedTensor": + parts = list(value.unbind()) + self.add_value(stage, f"{field}.components", len(parts)) + for index, child in enumerate(parts): + label = ("video", "audio")[index] if index < 2 else f"component_{index}" + self.capture_value( + stage, + f"{field}.{label}", + child, + persist_tensors=persist_tensors, + depth=depth + 1, + ) + return + if all(hasattr(value, name) for name in ("signature", "segments", "position_ids")): + self.add_value(stage, f"{field}.type", value.__class__.__name__) + self.add_value(stage, f"{field}.signature", list(value.signature)) + self.add_value(stage, f"{field}.segments", [list(item) for item in value.segments]) + for name in ("position_ids", "img_pos", "img_update", "audio_pos", "audio_update"): + child = getattr(value, name, None) + if torch.is_tensor(child): + self.add_tensor(stage, f"{field}.{name}", child, persist=persist_tensors) + return + self.add_value(stage, f"{field}.type", value.__class__.__name__) + + def save(self, directory: Path) -> None: + directory.mkdir(parents=True, exist_ok=True) + from safetensors.torch import save_file + + save_file(self.tensors, str(directory / "tensors.safetensors")) + (directory / "manifest.json").write_text( + json.dumps(self.manifest, indent=2, ensure_ascii=False), + encoding="utf-8", + ) + + +def compare_stage( + left: DiagnosticCapture, + right: DiagnosticCapture, + stage: str, +) -> dict[str, Any]: + a = left.manifest["stages"].get(stage, {"values": {}, "tensors": {}}) + b = right.manifest["stages"].get(stage, {"values": {}, "tensors": {}}) + details: dict[str, Any] = {} + first_field: str | None = None + + value_fields = list(a["values"].keys()) + [key for key in b["values"] if key not in a["values"]] + for field in value_fields: + av = a["values"].get(field, {"missing": True}) + bv = b["values"].get(field, {"missing": True}) + matches = av == bv + details[field] = {"kind": "value", "matches": matches, "left": av, "right": bv} + if not matches and first_field is None: + first_field = field + + tensor_fields = list(a["tensors"].keys()) + [key for key in b["tensors"] if key not in a["tensors"]] + for field in tensor_fields: + a_meta = a["tensors"].get(field) + b_meta = b["tensors"].get(field) + if a_meta is None or b_meta is None: + result = {"kind": "tensor", "matches": False, "reason": "missing tensor"} + else: + a_key = a_meta.get("stored_as") + b_key = b_meta.get("stored_as") + if a_key in left.tensors and b_key in right.tensors: + result = {"kind": "tensor", **compare_tensors(left.tensors[a_key], right.tensors[b_key]).to_dict()} + else: + matches = a_meta == b_meta + result = {"kind": "fingerprint", "matches": matches, "left": a_meta, "right": b_meta} + details[field] = result + if not result["matches"] and first_field is None: + first_field = field + + return { + "status": "MATCH" if first_field is None else "DIFFER", + "first_field": first_field, + "fields": details, + } + + +def recombine_terminal_streams( + first: torch.Tensor, + second: torch.Tensor, + *, + first_range: tuple[int, int] | list[int], + second_range: tuple[int, int] | list[int], + dim: int, +) -> torch.Tensor: + first_start, first_stop = (int(value) for value in first_range) + second_start, second_stop = (int(value) for value in second_range) + overlap = first_stop - second_start + if first_start != 0 or overlap < 0: + raise ValueError("terminal split ranges do not form an overlapping sequence") + if first.shape[dim] != first_stop - first_start: + raise ValueError("first logical stream length does not match its range") + if second.shape[dim] != second_stop - second_start: + raise ValueError("second logical stream length does not match its range") + if overlap > second.shape[dim]: + raise ValueError("terminal overlap exceeds the second logical stream") + slices = [slice(None)] * second.ndim + slices[dim] = slice(overlap, None) + return torch.cat((first, second[tuple(slices)]), dim=dim).contiguous() + + +def format_comparison(comparison: Mapping[str, Any], output_directory: Path) -> str: + lines = [] + first_divergence = None + for stage in STAGE_ORDER: + result = comparison.get(stage, {"status": "NOT_RUN", "first_field": None}) + lines.append(f"{stage}: {result['status']}") + if first_divergence is None and result["status"] == "DIFFER": + field = result.get("first_field") + first_divergence = f"{stage}.{field}" if field else stage + lines.append("") + lines.append(f"First divergence: {first_divergence or 'none'}") + lines.append(f"Output: {output_directory}") + return "\n".join(lines) diff --git a/tools/gpu_diagnostic_node/nodes.py b/tools/gpu_diagnostic_node/nodes.py new file mode 100644 index 0000000..6b749a4 --- /dev/null +++ b/tools/gpu_diagnostic_node/nodes.py @@ -0,0 +1,715 @@ +"""Standalone ComfyUI node for Core-vs-Continuum GPU diagnostics.""" + +from __future__ import annotations + +from contextlib import contextmanager +from datetime import datetime +import gc +import importlib +import json +from pathlib import Path, PurePosixPath +import random +import sys +import traceback +from types import SimpleNamespace +from typing import Any + +import numpy as np +import torch + +from .capture import ( + DiagnosticCapture, + STAGE_ORDER, + compare_stage, + compare_tensors, + format_comparison, + recombine_terminal_streams, +) + + +def _resolve_loaded_module(path: Path): + target = path.resolve() + for module in tuple(sys.modules.values()): + source = getattr(module, "__file__", None) + if source and Path(source).resolve() == target: + return module + return None + + +def _resolve_continuum_modules() -> SimpleNamespace: + import nodes as comfy_nodes + + sampler_class = comfy_nodes.NODE_CLASS_MAPPINGS.get("H3ContinuumSamplerV34") + if sampler_class is None: + raise RuntimeError("H3ContinuumSamplerV34 is not loaded; install and enable H3 Continuum V3.4 first") + owner = sys.modules.get(sampler_class.__module__) + source = getattr(owner, "__file__", None) + if source is None: + raise RuntimeError("cannot resolve the active H3 Continuum package") + root = Path(source).resolve().parent.parent + module_name = sampler_class.__module__ + base_name = module_name.split(".v3.", 1)[0] if ".v3." in module_name else module_name.rsplit(".", 2)[0] + + def load(relative: str, suffix: str): + target = root / relative + loaded = _resolve_loaded_module(target) + if loaded is not None: + return loaded + return importlib.import_module(f"{base_name}.{suffix}") + + return SimpleNamespace( + root=root, + constants=load("constants.py", "constants"), + state=load("state.py", "state"), + sequence=load("v2/sequence.py", "v2.sequence"), + sampling=load("v2/sampling.py", "v2.sampling"), + prompts=load("v2/prompts.py", "v2.prompts"), + decoder=load("v2/decoder.py", "v2.decoder"), + ) + + +def _reset_rng(seed: int) -> None: + random.seed(seed) + np.random.seed(seed & 0xFFFFFFFF) + torch.manual_seed(seed) + if torch.cuda.is_available(): + torch.cuda.manual_seed_all(seed) + + +def _release_cuda() -> None: + gc.collect() + if torch.cuda.is_available(): + torch.cuda.empty_cache() + + +def _safe_output_directory(subdir: str) -> Path: + import folder_paths + + clean = str(subdir).replace("\\", "/").strip("/") or "h3_continuum/diagnostics" + parts = PurePosixPath(clean).parts + if any(part in ("", ".", "..") for part in parts): + raise ValueError("output_subdir must stay below the ComfyUI output directory") + stamp = datetime.now().strftime("%Y%m%d_%H%M%S_%f") + return Path(folder_paths.get_output_directory()).joinpath(*parts, stamp) + + +def _capture_av_value(capture: DiagnosticCapture, stage: str, prefix: str, value: Any) -> None: + candidate = value.get("samples") if isinstance(value, dict) and "samples" in value else value + if hasattr(candidate, "unbind") and candidate.__class__.__name__ == "NestedTensor": + parts = list(candidate.unbind()) + capture.add_value(stage, f"{prefix}.components", len(parts)) + for index, part in enumerate(parts): + label = ("video", "audio")[index] if index < 2 else f"component_{index}" + capture.add_tensor(stage, f"{prefix}.{label}", part) + return + if torch.is_tensor(candidate) and getattr(candidate, "is_nested", False): + parts = list(candidate.unbind()) + capture.add_value(stage, f"{prefix}.components", len(parts)) + for index, part in enumerate(parts): + label = ("video", "audio")[index] if index < 2 else f"component_{index}" + capture.add_tensor(stage, f"{prefix}.{label}", part) + return + capture.capture_value(stage, prefix, candidate) + + +def _capture_pre_sampler( + capture: DiagnosticCapture, + modules: SimpleNamespace, + *, + conditioning: Any, + latent: dict[str, Any], + sigmas: torch.Tensor, + seed: int, +) -> None: + stage = "A_pre_sampler" + capture.add_value(stage, "seed", int(seed)) + capture.add_value(stage, "presentation_order", ["First Frame", "Last Frame"]) + capture.add_tensor(stage, "sigmas", sigmas) + video, audio = modules.state.extract_av_streams(latent) + capture.add_tensor(stage, "target.video", video) + capture.add_tensor(stage, "target.audio", audio) + capture.add_value(stage, "target.video_shape", list(video.shape)) + capture.add_value(stage, "target.audio_shape", list(audio.shape)) + + if not conditioning: + capture.add_value(stage, "conditioning.empty", True) + return + first = conditioning[0] + cond_tensor = first[0] + metadata = first[1] if len(first) > 1 and isinstance(first[1], dict) else {} + capture.capture_value(stage, "conditioning.positive", cond_tensor) + capture.add_value(stage, "conditioning.metadata_keys", sorted(str(key) for key in metadata)) + capture.add_value(stage, "physical_frames", int(metadata.get("minimax_frame_count", 243))) + for key in ("minimax_token_tags", "minimax_keyframes", "minimax_refs", "minimax_frame_count"): + if key in metadata: + capture.capture_value(stage, f"conditioning.{key}", metadata[key]) + + +def _capture_physical_sample( + capture: DiagnosticCapture, + modules: SimpleNamespace, + result: dict[str, Any], +) -> tuple[torch.Tensor, torch.Tensor]: + video, audio = modules.state.extract_av_streams(result) + video_cpu = video.detach().to("cpu").contiguous() + audio_cpu = audio.detach().to("cpu").contiguous() + capture.add_tensor("C_physical_sample", "video", video_cpu) + capture.add_tensor("C_physical_sample", "audio", audio_cpu) + capture.runtime["physical_streams"] = (video_cpu, audio_cpu) + return video_cpu, audio_cpu + + +@contextmanager +def _runtime_capture(capture: DiagnosticCapture): + import comfy.model_base + import comfy.sample + + original_noise = comfy.sample.prepare_noise + original_apply = comfy.model_base.BaseModel._apply_model + noise_seen = False + model_seen = False + + def prepare_noise(*args, **kwargs): + nonlocal noise_seen + result = original_noise(*args, **kwargs) + if not noise_seen: + noise_seen = True + _capture_av_value(capture, "A_pre_sampler", "actual_noise", result) + return result + + def apply_model( + self, + x, + t, + c_concat=None, + c_crossattn=None, + control=None, + transformer_options={}, + **kwargs, + ): + nonlocal model_seen + if not model_seen and self.__class__.__name__ == "MiniMaxH3": + model_seen = True + capture.capture_value("B_first_model_call", "input_x", x) + capture.capture_value("B_first_model_call", "timestep", t) + capture.capture_value( + "B_first_model_call", + "transformer_options", + transformer_options, + persist_tensors=False, + ) + capture.capture_value("B_first_model_call", "minimax_payload", kwargs.get("minimax_payload")) + capture.capture_value("B_first_model_call", "latent_shapes", kwargs.get("latent_shapes")) + return original_apply( + self, + x, + t, + c_concat, + c_crossattn, + control, + transformer_options, + **kwargs, + ) + + comfy.sample.prepare_noise = prepare_noise + comfy.model_base.BaseModel._apply_model = apply_model + try: + yield + finally: + comfy.sample.prepare_noise = original_noise + comfy.model_base.BaseModel._apply_model = original_apply + restored = ( + comfy.sample.prepare_noise is original_noise + and comfy.model_base.BaseModel._apply_model is original_apply + ) + capture.add_value("B_first_model_call", "patches_restored", restored) + capture.add_value("A_pre_sampler", "noise_captured", noise_seen) + capture.add_value("B_first_model_call", "model_call_captured", model_seen) + + +def _run_core( + capture: DiagnosticCapture, + modules: SimpleNamespace, + *, + model: Any, + clip: Any, + video_vae: Any, + sampler: Any, + sigmas: torch.Tensor, + first_frame: torch.Tensor, + last_frame: torch.Tensor, + prompt: str, + width: int, + height: int, + seed: int, +) -> None: + from comfy_extras.nodes_minimax_h3 import MiniMaxH3ImageToVideo + + core_model = model.clone() + output = MiniMaxH3ImageToVideo().execute( + clip, + video_vae, + prompt, + int(width), + int(height), + 243, + first_frame=first_frame, + last_frame=last_frame, + ) + conditioning, latent = output[0], output[1] + _capture_pre_sampler( + capture, + modules, + conditioning=conditioning, + latent=latent, + sigmas=sigmas, + seed=seed, + ) + with _runtime_capture(capture): + result = modules.sampling.sample_chunk( + model=core_model, + conditioning=conditioning, + latent=latent, + sampler=sampler, + sigmas=sigmas, + seed=int(seed), + enable_preview=False, + ) + _capture_physical_sample(capture, modules, result) + capture.add_value("C_physical_sample", "sampling_passes", 1) + capture.add_value("C_physical_sample", "physical_seed", int(seed)) + del result, latent, conditioning, core_model + + +def _capture_terminal_split( + capture: DiagnosticCapture, + video: torch.Tensor, + audio: torch.Tensor, + contract: dict[str, Any], + result: Any, +) -> None: + (video_1, audio_1), (video_2, audio_2) = result + video_ranges = contract["video_slices"] + audio_ranges = contract["audio_slices"] + video_recombined = recombine_terminal_streams( + video_1, + video_2, + first_range=video_ranges[0], + second_range=video_ranges[1], + dim=2, + ) + audio_recombined = recombine_terminal_streams( + audio_1, + audio_2, + first_range=audio_ranges[0], + second_range=audio_ranges[1], + dim=-1, + ) + video_diff = compare_tensors(video.detach().to("cpu"), video_recombined.detach().to("cpu")) + audio_diff = compare_tensors(audio.detach().to("cpu"), audio_recombined.detach().to("cpu")) + stage = "D_split_recombine" + capture.add_value(stage, "contract", contract) + capture.add_tensor(stage, "video.physical", video) + capture.add_tensor(stage, "video.chunk_1", video_1) + capture.add_tensor(stage, "video.chunk_2", video_2) + capture.add_tensor(stage, "video.recombined", video_recombined) + capture.add_tensor(stage, "audio.physical", audio) + capture.add_tensor(stage, "audio.chunk_1", audio_1) + capture.add_tensor(stage, "audio.chunk_2", audio_2) + capture.add_tensor(stage, "audio.recombined", audio_recombined) + capture.add_value(stage, "video_difference", video_diff.to_dict()) + capture.add_value(stage, "audio_difference", audio_diff.to_dict()) + capture.runtime["logical_streams"] = ( + (video_1.detach().to("cpu").contiguous(), audio_1.detach().to("cpu").contiguous()), + (video_2.detach().to("cpu").contiguous(), audio_2.detach().to("cpu").contiguous()), + ) + capture.runtime["terminal_contract"] = contract + + +def _run_continuum( + capture: DiagnosticCapture, + modules: SimpleNamespace, + *, + model: Any, + clip: Any, + video_vae: Any, + audio_vae: Any, + sampler: Any, + sigmas: torch.Tensor, + first_frame: torch.Tensor, + last_frame: torch.Tensor, + prompt: str, + width: int, + height: int, + seed: int, +) -> None: + sequence = modules.sequence + original_sample = sequence.sample_chunk + original_split = sequence._split_terminal_merged_latents + sample_calls = 0 + split_calls = 0 + + def sample_chunk(**kwargs): + nonlocal sample_calls + sample_calls += 1 + if sample_calls == 1: + _capture_pre_sampler( + capture, + modules, + conditioning=kwargs["conditioning"], + latent=kwargs["latent"], + sigmas=kwargs["sigmas"], + seed=int(kwargs["seed"]), + ) + result = original_sample(**kwargs) + if sample_calls == 1: + _capture_physical_sample(capture, modules, result) + capture.add_value("C_physical_sample", "physical_seed", int(kwargs["seed"])) + return result + + def split_terminal(video, audio, contract): + nonlocal split_calls + split_calls += 1 + result = original_split(video, audio, contract) + if split_calls == 1: + _capture_terminal_split(capture, video, audio, contract, result) + return result + + sequence.sample_chunk = sample_chunk + sequence._split_terminal_merged_latents = split_terminal + try: + prompt_plan = modules.prompts.make_prompt_plan( + mode=modules.constants.PROMPT_MODE_FIXED, + script=prompt, + chunks=2, + chunk_seconds=5.0, + ) + continuum_model = model.clone() + with _runtime_capture(capture): + sequence.run_sequence( + model=continuum_model, + clip=clip, + video_vae=video_vae, + audio_vae=audio_vae, + sampler=sampler, + sigmas=sigmas, + first_frame=first_frame, + last_frame=last_frame, + prompt_plan=prompt_plan, + width=int(width), + height=int(height), + continuity=modules.constants.CONTINUITY_OPTIONS[0], + base_seed=int(seed), + audio_continuity=True, + exact_total_duration=True, + diagnostics_mode=modules.constants.DIAGNOSTICS_BASIC, + reroll_from_chunk=0, + reroll_nonce=0, + strict_compatibility=False, + debug=False, + seam_correction=modules.constants.SEAM_CORRECTION_OFF, + enable_preview=False, + session=None, + initial_state=None, + latent_only=True, + reference_assets=None, + reference_audio_source=None, + reference_audio_vae=None, + driving_audio_source=None, + driving_audio_vae=None, + reference_video_source=None, + timeline_video_source=None, + ) + capture.add_value("C_physical_sample", "sampling_passes", sample_calls) + capture.add_value("D_split_recombine", "split_calls", split_calls) + del continuum_model + finally: + sequence.sample_chunk = original_sample + sequence._split_terminal_merged_latents = original_split + capture.add_value( + "D_split_recombine", + "sequence_patches_restored", + sequence.sample_chunk is original_sample + and sequence._split_terminal_merged_latents is original_split, + ) + + +def _make_av_latent(video: torch.Tensor, audio: torch.Tensor) -> dict[str, Any]: + from comfy.nested_tensor import NestedTensor + + return {"samples": NestedTensor([video, audio])} + + +def _copy_audio(audio: dict[str, Any]) -> dict[str, Any]: + return { + "waveform": audio["waveform"].clone(), + "sample_rate": int(audio["sample_rate"]), + } + + +def _run_decode_diagnostics(capture: DiagnosticCapture, modules: SimpleNamespace, video_vae: Any, audio_vae: Any) -> dict[str, Any]: + physical = capture.runtime.get("physical_streams") + logical = capture.runtime.get("logical_streams") + contract = capture.runtime.get("terminal_contract") + if physical is None or logical is None or contract is None: + return { + "E1_decode_243": {"status": "NOT_RUN", "first_field": "missing terminal capture"}, + "E2_final_240": {"status": "NOT_RUN", "first_field": "missing terminal capture"}, + } + + decoder = modules.decoder + physical_video, physical_audio = physical + direct_latent = _make_av_latent(physical_video, physical_audio) + direct_images = decoder.decode_video(video_vae, direct_latent) + direct_audio = decoder.decode_audio(audio_vae, direct_latent) + + decoded_images = [] + decoded_audio = [] + for video, audio in logical: + latent = _make_av_latent(video, audio) + decoded_images.append(decoder.decode_video(video_vae, latent)) + decoded_audio.append(decoder.decode_audio(audio_vae, latent)) + + trim_frames = 22 + reconstructed_images = torch.cat( + (decoded_images[0][:124], decoded_images[1][:141][trim_frames:]), + dim=0, + ).contiguous() + sample_rate = int(decoded_audio[0]["sample_rate"]) + if int(decoded_audio[1]["sample_rate"]) != sample_rate: + raise ValueError("chunk audio sample rates differ during decode diagnostics") + total_samples = int(round(243 / 24.0 * sample_rate)) + reconstructed_waveform = torch.empty( + (*decoded_audio[0]["waveform"].shape[:-1], total_samples), + dtype=decoded_audio[0]["waveform"].dtype, + device="cpu", + ) + segment_1 = decoder._slice_audio_for_timeline( + decoded_audio[0], trim_frames=0, frame_start=0, frame_stop=124 + ) + segment_2 = decoder._slice_audio_for_timeline( + decoded_audio[1], trim_frames=trim_frames, frame_start=124, frame_stop=243 + ) + split_sample = int(round(124 / 24.0 * sample_rate)) + reconstructed_waveform[..., :split_sample].copy_(segment_1) + reconstructed_waveform[..., split_sample:].copy_(segment_2) + reconstructed_audio = {"waveform": reconstructed_waveform, "sample_rate": sample_rate} + + direct_images, direct_audio, _ = decoder.enforce_total_frames( + direct_images, + direct_audio, + target_frames=243, + preserve_final_frame=False, + ) + reconstructed_images, reconstructed_audio, _ = decoder.enforce_total_frames( + reconstructed_images, + reconstructed_audio, + target_frames=243, + preserve_final_frame=False, + ) + + image_e1 = compare_tensors(direct_images, reconstructed_images) + audio_e1 = compare_tensors(direct_audio["waveform"], reconstructed_audio["waveform"]) + stage = "E1_decode_243" + capture.add_tensor(stage, "direct.images", direct_images, persist=False) + capture.add_tensor(stage, "split.images", reconstructed_images, persist=False) + capture.add_tensor(stage, "direct.audio", direct_audio["waveform"], persist=False) + capture.add_tensor(stage, "split.audio", reconstructed_audio["waveform"], persist=False) + capture.add_value(stage, "images_difference", image_e1.to_dict()) + capture.add_value(stage, "audio_difference", audio_e1.to_dict()) + boundary = slice(112, 129) + capture.add_tensor(stage, "direct.boundary_images", direct_images[boundary]) + capture.add_tensor(stage, "split.boundary_images", reconstructed_images[boundary]) + boundary_diff = compare_tensors(direct_images[boundary], reconstructed_images[boundary]) + capture.add_value(stage, "boundary_images_difference", boundary_diff.to_dict()) + + direct_240_images, direct_240_audio, _ = decoder.enforce_total_frames( + direct_images, + _copy_audio(direct_audio), + target_frames=240, + preserve_final_frame=True, + ) + split_240_images, split_240_audio, _ = decoder.enforce_total_frames( + reconstructed_images, + _copy_audio(reconstructed_audio), + target_frames=240, + preserve_final_frame=True, + ) + image_e2 = compare_tensors(direct_240_images, split_240_images) + audio_e2 = compare_tensors(direct_240_audio["waveform"], split_240_audio["waveform"]) + stage = "E2_final_240" + capture.add_tensor(stage, "direct.images", direct_240_images, persist=False) + capture.add_tensor(stage, "split.images", split_240_images, persist=False) + capture.add_tensor(stage, "direct.audio", direct_240_audio["waveform"], persist=False) + capture.add_tensor(stage, "split.audio", split_240_audio["waveform"], persist=False) + capture.add_value(stage, "images_difference", image_e2.to_dict()) + capture.add_value(stage, "audio_difference", audio_e2.to_dict()) + capture.add_tensor(stage, "direct.boundary_images", direct_240_images[boundary]) + capture.add_tensor(stage, "split.boundary_images", split_240_images[boundary]) + + return { + "E1_decode_243": { + "status": "MATCH" if image_e1.matches and audio_e1.matches else "DIFFER", + "first_field": None if image_e1.matches and audio_e1.matches else ("images" if not image_e1.matches else "audio"), + "images": image_e1.to_dict(), + "audio": audio_e1.to_dict(), + "boundary_images": boundary_diff.to_dict(), + }, + "E2_final_240": { + "status": "MATCH" if image_e2.matches and audio_e2.matches else "DIFFER", + "first_field": None if image_e2.matches and audio_e2.matches else ("images" if not image_e2.matches else "audio"), + "images": image_e2.to_dict(), + "audio": audio_e2.to_dict(), + }, + } + + +class H3CoreContinuumGpuDiagnostic: + """Run an isolated Core 243f and Continuum 2x5s comparison in one queue.""" + + OUTPUT_NODE = True + CATEGORY = "H3-Continuum/Diagnostics" + FUNCTION = "run" + RETURN_TYPES = ("STRING",) + RETURN_NAMES = ("diagnostic_report",) + + @classmethod + def INPUT_TYPES(cls): + return { + "required": { + "model": ("MODEL",), + "clip": ("CLIP",), + "video_vae": ("VAE",), + "audio_vae": ("VAE",), + "sampler": ("SAMPLER",), + "sigmas": ("SIGMAS",), + "first_frame": ("IMAGE",), + "last_frame": ("IMAGE",), + "prompt": ("STRING", {"multiline": True, "default": "The subject moves continuously from the First Frame to the Last Frame."}), + "width": ("INT", {"default": 640, "min": 64, "max": 4096, "step": 16}), + "height": ("INT", {"default": 640, "min": 64, "max": 4096, "step": 16}), + "seed": ("INT", {"default": 42, "min": 0, "max": 0xFFFFFFFFFFFFFFFF}), + "output_subdir": ("STRING", {"default": "h3_continuum/diagnostics"}), + } + } + + def run( + self, + model, + clip, + video_vae, + audio_vae, + sampler, + sigmas, + first_frame, + last_frame, + prompt, + width, + height, + seed, + output_subdir, + ): + output_directory = _safe_output_directory(output_subdir) + core = DiagnosticCapture("core") + continuum = DiagnosticCapture("continuum") + comparison: dict[str, Any] = {} + modules = _resolve_continuum_modules() + common = { + "seed": int(seed), + "width": int(width), + "height": int(height), + "physical_frames": 243, + "spectrum": "must be OFF at the model input", + "run_storage": "Off", + "session": None, + "continuum_root": str(modules.root), + } + core.manifest["conditions"] = common + continuum.manifest["conditions"] = common + + try: + _reset_rng(int(seed)) + _run_core( + core, + modules, + model=model, + clip=clip, + video_vae=video_vae, + sampler=sampler, + sigmas=sigmas, + first_frame=first_frame, + last_frame=last_frame, + prompt=prompt, + width=width, + height=height, + seed=seed, + ) + _release_cuda() + _reset_rng(int(seed)) + _run_continuum( + continuum, + modules, + model=model, + clip=clip, + video_vae=video_vae, + audio_vae=audio_vae, + sampler=sampler, + sigmas=sigmas, + first_frame=first_frame, + last_frame=last_frame, + prompt=prompt, + width=width, + height=height, + seed=seed, + ) + + for stage in ("A_pre_sampler", "B_first_model_call", "C_physical_sample"): + comparison[stage] = compare_stage(core, continuum, stage) + + d_values = continuum.manifest["stages"].get("D_split_recombine", {}).get("values", {}) + d_video = d_values.get("video_difference", {}) + d_audio = d_values.get("audio_difference", {}) + d_match = bool(d_video.get("matches") and d_audio.get("matches")) + comparison["D_split_recombine"] = { + "status": "MATCH" if d_match else "DIFFER", + "first_field": None if d_match else ("video" if not d_video.get("matches") else "audio"), + "video": d_video, + "audio": d_audio, + } + comparison.update(_run_decode_diagnostics(continuum, modules, video_vae, audio_vae)) + except Exception as exc: + comparison["error"] = { + "type": type(exc).__name__, + "message": str(exc), + "traceback": traceback.format_exc(), + } + finally: + output_directory.mkdir(parents=True, exist_ok=True) + core.save(output_directory / "core") + continuum.save(output_directory / "continuum") + for stage in STAGE_ORDER: + comparison.setdefault(stage, {"status": "NOT_RUN", "first_field": None}) + summary = format_comparison(comparison, output_directory) + (output_directory / "comparison.json").write_text( + json.dumps(comparison, indent=2, ensure_ascii=False), + encoding="utf-8", + ) + (output_directory / "comparison.txt").write_text(summary + "\n", encoding="utf-8") + _release_cuda() + + if "error" in comparison: + error = comparison["error"] + raise RuntimeError( + f"GPU diagnostic failed; partial results were saved to {output_directory}: " + f"{error['type']}: {error['message']}" + ) + return (summary,) + + +NODE_CLASS_MAPPINGS = { + "H3CoreContinuumGpuDiagnostic": H3CoreContinuumGpuDiagnostic, +} + +NODE_DISPLAY_NAME_MAPPINGS = { + "H3CoreContinuumGpuDiagnostic": "H3 Core vs Continuum GPU Diagnostic", +} diff --git a/tools/install_gpu_diagnostic_node.ps1 b/tools/install_gpu_diagnostic_node.ps1 new file mode 100644 index 0000000..60cc220 --- /dev/null +++ b/tools/install_gpu_diagnostic_node.ps1 @@ -0,0 +1,29 @@ +param( + [string]$ComfyUIRoot = 'D:\StabilityMatrix\Data\Packages\ComfyUI_W' +) + +$ErrorActionPreference = 'Stop' +$source = Join-Path $PSScriptRoot 'gpu_diagnostic_node' +$customNodes = Join-Path $ComfyUIRoot 'custom_nodes' +$destination = Join-Path $customNodes 'H3_Continuum_GPU_Diagnostics' + +if (-not (Test-Path -LiteralPath $source -PathType Container)) { + throw "Diagnostic source was not found: $source" +} +if (-not (Test-Path -LiteralPath $customNodes -PathType Container)) { + throw "ComfyUI custom_nodes directory was not found: $customNodes" +} + +New-Item -ItemType Directory -Path $destination -Force | Out-Null +foreach ($name in @('__init__.py', 'capture.py', 'nodes.py', 'README.md')) { + Copy-Item -LiteralPath (Join-Path $source $name) -Destination (Join-Path $destination $name) -Force +} + +$sourceCount = (Get-ChildItem -LiteralPath $source -File).Count +$destinationCount = (Get-ChildItem -LiteralPath $destination -File).Count +if ($destinationCount -lt $sourceCount) { + throw "Diagnostic install verification failed: $destinationCount < $sourceCount" +} + +Write-Output "Installed standalone diagnostics: $destination" +Write-Output 'Restart ComfyUI before using the diagnostic node.' diff --git a/v2/sequence.py b/v2/sequence.py index ac0dd52..07e9c48 100644 --- a/v2/sequence.py +++ b/v2/sequence.py @@ -24,14 +24,21 @@ from ..driving_audio import ( ) from ..model_patch import clone_model_for_chunk from ..state import assert_context_unchanged, context_fingerprint, make_plan, select_context, validate_state -from ..temporal import align_frame_count_up, largest_context_capacity, make_extension_shape +from ..temporal import ( + align_frame_count_up, + audio_latent_t, + context_slots, + largest_context_capacity, + make_extension_shape, + video_latent_t, +) from ..version import PACKAGE_VERSION from .decoder import decode_sequence, decode_sequence_with_seam, enforce_total_frames from .context_diagnostics import ContextDiagnosticsTracker from .h3_builder import attach_keyframes, empty_h3_latent, encode_identity_latents, encode_prompt_conditioning, prepare_identity_assets from .motion import choose_context_frames from .prompts import prompt_plan_report, validate_prompt_plan -from .sampling import sample_chunk +from .sampling import latent_from_cpu, latent_to_cpu, sample_chunk from .seeds import derive_chunk_seed from .session import ( SessionValidationError, entry_to_state, make_chunk_entry, make_session, model_fingerprint, @@ -62,7 +69,7 @@ def _check_decode_memory_budget(*,width,height,chunks,chunk_seconds): def _clone_entry_for_reuse(entry): entry=validate_chunk_entry(entry); result=dict(entry); result["plan"]=copy.deepcopy(entry["plan"]); result["reused"]=True; return result -def _preserved_prefix(*,session,prompt_hashes,chunks,reroll_from_chunk,width,height,chunk_seconds,identity_hash): +def _preserved_prefix(*,session,prompt_hashes,chunks,reroll_from_chunk,width,height,chunk_seconds,identity_hash,last_frame_hash): if session is None: return [],[] notes=[] try: session=validate_session(session) @@ -70,6 +77,8 @@ def _preserved_prefix(*,session,prompt_hashes,chunks,reroll_from_chunk,width,hei if int(session["width"])!=int(width) or int(session["height"])!=int(height): return [],["saved session resolution differs; generated a fresh run"] if abs(float(session["chunk_seconds"])-float(chunk_seconds))>1e-6: return [],["saved session chunk duration differs; generated a fresh run"] if str(session.get("identity_hash","none"))!=str(identity_hash): return [],["saved session identity differs; generated a fresh run"] + saved_last_frame_hash=(session.get("settings") or {}).get("last_frame_hash") + if str(saved_last_frame_hash or "none")!=str(last_frame_hash or "none"): return [],["saved session Last Frame differs; generated a fresh run"] if reroll_from_chunk<0 or reroll_from_chunk>chunks: raise SessionValidationError("reroll_from_chunk must be 0 or a valid one-based chunk index") limit=min(len(session["chunks"]),chunks) if reroll_from_chunk>0: limit=min(limit,reroll_from_chunk-1) @@ -103,7 +112,12 @@ def _conditioning_cache(*,clip,prompts,assets,final_has_last_frame,reference_ass return cache -FLF_STRATEGY = "context_bound_terminal_v1" +FLF_STRATEGY = "terminal_merged_10s_seed_v2" +TERMINAL_SEED_POLICY = "physical_window_seed_v2" +TERMINAL_MERGE_CHUNK_SECONDS = 5.0 +TERMINAL_MERGE_CONTEXT_FRAMES = 22 +TERMINAL_PROMPT_POLICY_SHARED = "shared_prompt_v1" +TERMINAL_PROMPT_POLICY_TIMELINE = "paired_timeline_v1" def _flf_contract_identity(base_hash: str) -> str: @@ -111,6 +125,191 @@ def _flf_contract_identity(base_hash: str) -> str: return hashlib.sha256(payload.encode("utf-8")).hexdigest() +def _terminal_flf_merge_enabled( + *, + multi_chunk_flf: bool, + chunks: int, + chunk_seconds: float, + prompt_hashes: list[str], + timeline_video_source, +) -> bool: + return bool( + multi_chunk_flf + and int(chunks) >= 2 + and abs(float(chunk_seconds) - TERMINAL_MERGE_CHUNK_SECONDS) <= 1e-6 + and timeline_video_source is None + and len(prompt_hashes) >= 2 + ) + + +def _terminal_pair_prompt( + prompts: list[str], + *, + pair_start: int, + chunk_seconds: float, +) -> tuple[str, str]: + """Build one physical prompt without changing either logical prompt.""" + + first = str(prompts[int(pair_start)]) + second = str(prompts[int(pair_start) + 1]) + if first == second: + return first, TERMINAL_PROMPT_POLICY_SHARED + split = f"{float(chunk_seconds):g}" + stop = f"{float(chunk_seconds) * 2.0:g}" + return ( + f"[0-{split}s]\n{first}\n\n[{split}-{stop}s]\n{second}", + TERMINAL_PROMPT_POLICY_TIMELINE, + ) + + +def _atomic_terminal_prefix( + preserved: list[dict[str, Any]], + *, + chunks: int, + reroll_from_chunk: int, +) -> tuple[list[dict[str, Any]], bool]: + """Never reuse only one logical half of a shared terminal sample.""" + pair_start = int(chunks) - 2 + reroll_hits_pair = int(reroll_from_chunk) in (int(chunks) - 1, int(chunks)) + partial_pair = pair_start < len(preserved) < int(chunks) + if (reroll_hits_pair or partial_pair) and len(preserved) > pair_start: + return preserved[:pair_start], True + return preserved, False + + +def _terminal_sampling_plan( + *, + chunks: int, + completed: int, + merge_enabled: bool, +) -> tuple[tuple[int, ...], bool]: + if completed >= int(chunks): + return (), False + if not merge_enabled: + return tuple(range(completed, int(chunks))), False + pair_start = int(chunks) - 2 + if completed > pair_start: + raise SequenceRuntimeError("terminal merged pair cannot resume from only one logical half") + return tuple(range(completed, pair_start)), True + + +def _terminal_physical_seed_plan( + *, + base_seed: int, + physical_window_start_index: int, + reroll_nonce: int, +) -> dict[str, Any]: + """Return the one physical seed and the matching logical-entry metadata.""" + start_index = int(physical_window_start_index) + nonce = int(reroll_nonce) + if start_index < 0: + raise ValueError("physical_window_start_index must be non-negative") + if nonce < 0: + raise ValueError("reroll_nonce must be non-negative") + if start_index == 0 and nonce == 0: + physical_seed = int(base_seed) + else: + physical_seed = derive_chunk_seed(base_seed, start_index, nonce) + return { + "physical_seed": physical_seed, + "logical_entry_seeds": (physical_seed, physical_seed), + } + + +def _terminal_execution_semantics( + *, + merge_enabled: bool, + prompt_policy: str | None = None, +) -> dict[str, str] | None: + if not merge_enabled: + return None + semantics = { + "flf_execution": FLF_STRATEGY, + "terminal_seed_policy": TERMINAL_SEED_POLICY, + } + if prompt_policy: + semantics["terminal_prompt_policy"] = str(prompt_policy) + return semantics + + +def _terminal_pair_contract(*, initial_pair: bool, chunk_seconds: float) -> dict[str, Any]: + """Derive physical and logical AV ranges without fixed latent indices.""" + if abs(float(chunk_seconds) - TERMINAL_MERGE_CHUNK_SECONDS) > 1e-6: + raise SequenceRuntimeError("terminal merged sampling currently requires 5-second logical chunks") + initial_frames = align_frame_count_up(int(round(float(chunk_seconds) * FPS))) + extension = make_extension_shape( + TERMINAL_MERGE_CONTEXT_FRAMES, + float(chunk_seconds), + ) + if initial_pair: + physical_frames = align_frame_count_up(int(round(2.0 * float(chunk_seconds) * FPS))) + logical_frames = (initial_frames, extension.total_frames) + logical_trims = (0, TERMINAL_MERGE_CONTEXT_FRAMES) + first_video_stop = video_latent_t(initial_frames) + first_audio_stop = audio_latent_t(initial_frames) + else: + physical_frames = make_extension_shape( + TERMINAL_MERGE_CONTEXT_FRAMES, + 2.0 * float(chunk_seconds), + ).total_frames + logical_frames = (extension.total_frames, extension.total_frames) + logical_trims = ( + TERMINAL_MERGE_CONTEXT_FRAMES, + TERMINAL_MERGE_CONTEXT_FRAMES, + ) + first_video_stop = video_latent_t(extension.total_frames) + first_audio_stop = audio_latent_t(extension.total_frames) + physical_video_stop = video_latent_t(physical_frames) + physical_audio_stop = audio_latent_t(physical_frames) + video_overlap = context_slots(TERMINAL_MERGE_CONTEXT_FRAMES) + audio_overlap = audio_latent_t(TERMINAL_MERGE_CONTEXT_FRAMES) + video_slices = ( + (0, first_video_stop), + (first_video_stop - video_overlap, physical_video_stop), + ) + audio_slices = ( + (0, first_audio_stop), + (first_audio_stop - audio_overlap, physical_audio_stop), + ) + for index in range(2): + expected_video = video_latent_t(logical_frames[index]) + expected_audio = audio_latent_t(logical_frames[index]) + if video_slices[index][1] - video_slices[index][0] != expected_video: + raise SequenceRuntimeError("derived terminal video split does not match its logical frame count") + if audio_slices[index][1] - audio_slices[index][0] != expected_audio: + raise SequenceRuntimeError("derived terminal audio split does not match its logical frame count") + return { + "initial_pair": bool(initial_pair), + "physical_frames": int(physical_frames), + "physical_context_frames": 0 if initial_pair else TERMINAL_MERGE_CONTEXT_FRAMES, + "logical_frames": logical_frames, + "logical_trims": logical_trims, + "video_slices": video_slices, + "audio_slices": audio_slices, + } + + +def _split_terminal_merged_latents( + video: torch.Tensor, + audio: torch.Tensor, + contract: dict[str, Any], +) -> tuple[tuple[torch.Tensor, torch.Tensor], tuple[torch.Tensor, torch.Tensor]]: + expected_video = video_latent_t(int(contract["physical_frames"])) + expected_audio = audio_latent_t(int(contract["physical_frames"])) + if int(video.shape[2]) != expected_video or int(audio.shape[-1]) != expected_audio: + raise SequenceRuntimeError( + "terminal merged sampler output does not match the derived physical AV grid: " + f"video T={video.shape[2]} (expected {expected_video}), " + f"audio T={audio.shape[-1]} (expected {expected_audio})" + ) + parts=[] + for video_range,audio_range in zip(contract["video_slices"],contract["audio_slices"]): + video_part=video[:,:,video_range[0]:video_range[1],...].contiguous() + audio_part=audio[...,audio_range[0]:audio_range[1]].contiguous() + parts.append((video_part,audio_part)) + return parts[0],parts[1] + + def _attach_terminal_flf_keyframes( conditioning, *, @@ -141,7 +340,7 @@ def _attach_terminal_flf_keyframes( return conditioning def run_sequence(*,model:Any,clip:Any,video_vae:Any,audio_vae:Any,sampler:Any,sigmas:torch.Tensor,first_frame:torch.Tensor|None,last_frame:torch.Tensor|None,prompt_plan:dict[str,Any],width:int,height:int,continuity:str,base_seed:int,audio_continuity:bool,exact_total_duration:bool,diagnostics_mode:str,reroll_from_chunk:int,reroll_nonce:int,strict_compatibility:bool,debug:bool,seam_correction:str=SEAM_CORRECTION_OFF,enable_preview:bool=True,session:dict[str,Any]|None=None,initial_state:dict[str,Any]|None=None,latent_only:bool=False,reference_assets=None,reference_audio_source=None,reference_audio_vae=None,driving_audio_source=None,driving_audio_vae=None,reference_video_source=None,timeline_video_source=None): - from ..conditioning import detect_conditioning_mode, conditioning_mode_label + from ..conditioning import detect_conditioning_mode, conditioning_display_label from ..run_storage import get_active_run_storage storage_controller=get_active_run_storage() diagnostics_mode=normalize_diagnostics_mode(diagnostics_mode); plan=validate_prompt_plan(prompt_plan); chunks=int(plan["chunks"]); chunk_seconds=float(plan["chunk_seconds"]); prompts=list(plan["prompts"]); prompt_hashes=list(plan["hashes"]); width,height=int(width),int(height) @@ -157,6 +356,7 @@ def run_sequence(*,model:Any,clip:Any,video_vae:Any,audio_vae:Any,sampler:Any,si if initial_state is not None and int(reroll_from_chunk) not in (0,1): raise SequenceRuntimeError("with initial_state, reroll_from_chunk can only be 0 or 1") try: conditioning_mode=detect_conditioning_mode(first_frame=first_frame,last_frame=last_frame,reference_assets=reference_assets) except ValueError as exc: raise SequenceRuntimeError(str(exc)) from exc + conditioning_display=conditioning_display_label(has_first=first_frame is not None,has_last=last_frame is not None,has_reference=reference_assets is not None) reference_warning="" if reference_assets is not None: from ..reference import validate_reference_prompts @@ -181,6 +381,11 @@ def run_sequence(*,model:Any,clip:Any,video_vae:Any,audio_vae:Any,sampler:Any,si issues=check_comfy_h3_runtime() assets=prepare_identity_assets(video_vae,width=width,height=height,first_frame=first_frame,last_frame=last_frame,encode_latents=False) multi_chunk_flf=bool(chunks>=2 and first_frame is not None and last_frame is not None) + terminal_merge_enabled=_terminal_flf_merge_enabled(multi_chunk_flf=multi_chunk_flf,chunks=chunks,chunk_seconds=chunk_seconds,prompt_hashes=prompt_hashes,timeline_video_source=timeline_video_source) + terminal_prompt=None + terminal_prompt_policy=None + if terminal_merge_enabled: + terminal_prompt,terminal_prompt_policy=_terminal_pair_prompt(prompts,pair_start=chunks-2,chunk_seconds=chunk_seconds) from ..reference import combine_hybrid_visual_identity visual_identity_hash=combine_hybrid_visual_identity(keyframe_identity_hash=assets.identity_hash,reference_assets=reference_assets,has_first=first_frame is not None,has_last=last_frame is not None) sequence_identity_hash=combine_reference_audio_identity(visual_identity_hash,reference_audio_source) @@ -191,20 +396,27 @@ def run_sequence(*,model:Any,clip:Any,video_vae:Any,audio_vae:Any,sampler:Any,si sequence_identity_hash=_flf_contract_identity(sequence_identity_hash) current_model_fingerprint=model_fingerprint(model,extra_wrapper_keys=("h3_continuum_join.apply_model.v1",)) if storage_controller is not None: - stored_session=storage_controller.prepare(model=model,model_fingerprint_value=current_model_fingerprint,clip=clip,video_vae=video_vae,sampler=sampler,sigmas=sigmas,prompt_plan=plan,width=width,height=height,chunk_seconds=chunk_seconds,continuity=continuity,audio_continuity=audio_continuity,base_seed=base_seed,reroll_from_chunk=reroll_from_chunk,reroll_nonce=reroll_nonce,first_frame_hash=assets.first_frame_hash,last_frame_hash=assets.last_frame_hash,identity_hash=sequence_identity_hash,strict_compatibility=strict_compatibility,existing_session=session,reference_contract=reference_assets.contract if reference_assets is not None else None,conditioning_mode=conditioning_mode,reference_audio_contract=reference_audio_source.contract if reference_audio_source is not None else None,reference_audio_vae=reference_audio_vae,driving_audio_contract=driving_audio_source.contract if driving_audio_source is not None else None,driving_audio_vae=driving_audio_vae,reference_video_contract=reference_video_source.contract if reference_video_source is not None else None,timeline_video_contract=timeline_video_source.contract if timeline_video_source is not None else None) + stored_session=storage_controller.prepare(model=model,model_fingerprint_value=current_model_fingerprint,clip=clip,video_vae=video_vae,sampler=sampler,sigmas=sigmas,prompt_plan=plan,width=width,height=height,chunk_seconds=chunk_seconds,continuity=continuity,audio_continuity=audio_continuity,base_seed=base_seed,reroll_from_chunk=reroll_from_chunk,reroll_nonce=reroll_nonce,first_frame_hash=assets.first_frame_hash,last_frame_hash=assets.last_frame_hash,identity_hash=sequence_identity_hash,strict_compatibility=strict_compatibility,existing_session=session,reference_contract=reference_assets.contract if reference_assets is not None else None,conditioning_mode=conditioning_mode,reference_audio_contract=reference_audio_source.contract if reference_audio_source is not None else None,reference_audio_vae=reference_audio_vae,driving_audio_contract=driving_audio_source.contract if driving_audio_source is not None else None,driving_audio_vae=driving_audio_vae,reference_video_contract=reference_video_source.contract if reference_video_source is not None else None,timeline_video_contract=timeline_video_source.contract if timeline_video_source is not None else None,execution_semantics=_terminal_execution_semantics(merge_enabled=terminal_merge_enabled,prompt_policy=terminal_prompt_policy)) reroll_nonce=storage_controller.effective_reroll_nonce if stored_session is not None: session=stored_session accelerators=accelerator_summary(model) if "Continuum APPLY_MODEL wrapper installed" not in accelerators: accelerators+="; Continuum APPLY_MODEL wrapper installed" effective_reroll_from_chunk=0 if storage_controller is not None and session is not None and bool((session.get("settings") or {}).get("run_storage_validated_prefix")) else int(reroll_from_chunk) - preserved,reuse_notes=_preserved_prefix(session=session,prompt_hashes=prompt_hashes,chunks=chunks,reroll_from_chunk=effective_reroll_from_chunk,width=width,height=height,chunk_seconds=chunk_seconds,identity_hash=sequence_identity_hash) + preserved,reuse_notes=_preserved_prefix(session=session,prompt_hashes=prompt_hashes,chunks=chunks,reroll_from_chunk=effective_reroll_from_chunk,width=width,height=height,chunk_seconds=chunk_seconds,identity_hash=sequence_identity_hash,last_frame_hash=assets.last_frame_hash) if multi_chunk_flf and session is not None: saved_strategy=str((session.get("settings") or {}).get("flf_execution", "")) if saved_strategy!=FLF_STRATEGY: preserved=[] reuse_notes.append("FLF chunk contract changed; regenerated the requested FLF sequence.") + if terminal_merge_enabled: + preserved,atomic_reset=_atomic_terminal_prefix(preserved,chunks=chunks,reroll_from_chunk=int(reroll_from_chunk)) + if atomic_reset: + reuse_notes.append("terminal merged pair is atomic; regenerated both final logical chunks") if multi_chunk_flf: - reuse_notes.insert(0,"FLF execution: context-bound terminal v1; one sampling pass per chunk; the final chunk is anchored from the continuation-context tail to the supplied Last Frame.") + if terminal_merge_enabled: + reuse_notes.insert(0,"FLF execution: terminal merged 10s seed v2; the final two 5-second logical chunks share one physical sample and are split back into normal chunk entries.") + else: + reuse_notes.insert(0,"FLF execution: terminal merge unavailable for this request; retained one sampling pass per logical chunk.") if reference_assets is not None: reuse_notes.insert(0,f"Reference conditioning: {reference_assets.count} image(s), size={reference_assets.size_mode}; persistent across all chunks.") if reference_warning: reuse_notes.append(reference_warning) @@ -239,6 +451,10 @@ def run_sequence(*,model:Any,clip:Any,video_vae:Any,audio_vae:Any,sampler:Any,si reference_video_assets=encode_reference_video(video_vae,reference_video_source) if timeline_video_source is None: cache=_conditioning_cache(clip=clip,prompts=prompts,assets=assets,final_has_last_frame=last_frame is not None,reference_assets=reference_assets,reference_audio_assets=reference_audio_assets,timeline_video_assets=reference_video_assets) + if terminal_merge_enabled and terminal_prompt is not None: + terminal_key=_conditioning_cache_key(terminal_prompt,include_last=True,reference_assets=reference_assets) + if terminal_key not in cache: + cache.update(_conditioning_cache(clip=clip,prompts=[terminal_prompt],assets=assets,final_has_last_frame=True,reference_assets=reference_assets,reference_audio_assets=reference_audio_assets,timeline_video_assets=reference_video_assets)) entries=preserved[:]; previous_state=None if entries: try: previous_state=entry_to_state(entries[-1]) @@ -261,7 +477,8 @@ def run_sequence(*,model:Any,clip:Any,video_vae:Any,audio_vae:Any,sampler:Any,si _record_context_diagnostics(tracker=context_diagnostics,reports=sampling_reports,state=entry_to_state(reused_entry),continuity=continuity,reused=True) elif previous_state is not None: _record_context_diagnostics(tracker=context_diagnostics,reports=sampling_reports,state=previous_state,continuity=continuity,reused=True) - for sequence_index in range(len(entries),chunks): + normal_indices,terminal_merge_pending=_terminal_sampling_plan(chunks=chunks,completed=len(entries),merge_enabled=terminal_merge_enabled) + for sequence_index in normal_indices: prompt=prompts[sequence_index]; prompt_hash_value=prompt_hashes[sequence_index]; is_final=sequence_index==chunks-1; effective_reroll_nonce=int(reroll_nonce) if int(reroll_from_chunk)>0 and sequence_index+1>=int(reroll_from_chunk) else 0; seed=derive_chunk_seed(base_seed,sequence_index,effective_reroll_nonce); motion_score=0.0; video_context=None; audio_context=None; context_before=None timeline_video_assets=reference_video_assets chunk_cache=cache @@ -283,8 +500,6 @@ def run_sequence(*,model:Any,clip:Any,video_vae:Any,audio_vae:Any,sampler:Any,si base_conditioning=attach_keyframes(chunk_cache[conditioning_key],frame_count=shape.total_frames,first_latent=chunk_assets.first_latent,last_latent=chunk_assets.last_latent if include_chunk_last else None) video_context,audio_context,grid_offset=select_context(previous_state,context_frames,include_audio=bool(audio_continuity)); context_before=context_fingerprint(video_context,audio_context) conditioning=prepare_conditioning(base_conditioning,video_context=video_context,audio_context=audio_context,audio_grid_offset=grid_offset,context_frames=context_frames,new_frame_count=shape.total_frames,first_frame_policy=POLICY_REPLACE,preserve_last_frame=True) - if multi_chunk_flf and is_final: - conditioning=_attach_terminal_flf_keyframes(conditioning,video_context=video_context,context_frames=context_frames,frame_count=shape.total_frames,last_latent=assets.last_latent) clip_index=int(previous_state["clip_index"])+1; chunk_plan=make_plan(continuation=True,clip_index=clip_index,total_frames=shape.total_frames,trim_frames=context_frames,width=width,height=height,context_frames=context_frames,state_capacity_frames=largest_context_capacity(shape.net_new_frames),requested_extend_seconds=chunk_seconds,debug=debug) driving_audio_latent=slice_driving_audio_latent(driving_audio_assets,cumulative_retained_before=retained_frames,total_frames=int(chunk_plan["total_frames"]),trim_frames=int(chunk_plan["trim_frames"]),fps=FPS) conditioning=attach_driving_audio(conditioning,driving_audio_latent) @@ -298,6 +513,62 @@ def run_sequence(*,model:Any,clip:Any,video_vae:Any,audio_vae:Any,sampler:Any,si sampling_reports.append(f"chunk {sequence_index+1}/{chunks}: seed={seed}, frames={chunk_plan['total_frames']}, trim={chunk_plan['trim_frames']}, retained_total={retained_frames}, context={context_frames} ({reason}), motion={motion_score:.6f}, "+(f"interop=emitted actual_prefix={CONTINUUM_ACTUAL_PREFIX_STEPS} consumer=not_observable" if context_before is not None else "interop=not_emitted")) del sampled,latent,conditioning,chunk_model,chunk_cache if timeline_video_source is not None and timeline_video_assets is not None: del timeline_video_assets + if terminal_merge_pending: + pair_start=chunks-2 + if len(entries)!=pair_start: + raise SequenceRuntimeError(f"terminal merged pair expected {pair_start} completed chunks, got {len(entries)}") + prompt=terminal_prompt + seed_nonce=int(reroll_nonce) if int(reroll_from_chunk)>0 else 0 + terminal_seed_plan=_terminal_physical_seed_plan(base_seed=base_seed,physical_window_start_index=pair_start,reroll_nonce=seed_nonce) + physical_seed=int(terminal_seed_plan["physical_seed"]) + initial_pair=previous_state is None + contract=_terminal_pair_contract(initial_pair=initial_pair,chunk_seconds=chunk_seconds) + physical_frames=int(contract["physical_frames"]) + physical_context_frames=int(contract["physical_context_frames"]) + latent=empty_h3_latent(width,height,physical_frames) + conditioning_key=_conditioning_cache_key(prompt,include_last=True,reference_assets=reference_assets) + base_conditioning=attach_keyframes(cache[conditioning_key],frame_count=physical_frames,first_latent=assets.first_latent,last_latent=assets.last_latent) + video_context=None; audio_context=None; context_before=None; motion_score=0.0 + if initial_pair: + conditioning=base_conditioning + physical_clip_index=1 + reason="terminal merged initial 10-second sample" + else: + selected_context,motion_score,selected_reason=choose_context_frames(continuity,previous_state) + video_context,audio_context,grid_offset=select_context(previous_state,TERMINAL_MERGE_CONTEXT_FRAMES,include_audio=bool(audio_continuity)) + context_before=context_fingerprint(video_context,audio_context) + conditioning=prepare_conditioning(base_conditioning,video_context=video_context,audio_context=audio_context,audio_grid_offset=grid_offset,context_frames=TERMINAL_MERGE_CONTEXT_FRAMES,new_frame_count=physical_frames,first_frame_policy=POLICY_REPLACE,preserve_last_frame=True) + physical_clip_index=int(previous_state["clip_index"])+1 + reason=f"terminal merged 10-second sample with 22-frame context; continuity selection was {selected_context} ({selected_reason})" + driving_audio_latent=slice_driving_audio_latent(driving_audio_assets,cumulative_retained_before=retained_frames,total_frames=physical_frames,trim_frames=physical_context_frames,fps=FPS) + conditioning=attach_driving_audio(conditioning,driving_audio_latent) + chunk_model=clone_model_for_chunk(model,strict=bool(strict_compatibility),debug=bool(debug),chunk_index=physical_clip_index,context_frames=physical_context_frames if not initial_pair else None) + sampled=sample_chunk(model=chunk_model,conditioning=conditioning,latent=latent,sampler=sampler,sigmas=sigmas,seed=physical_seed,enable_preview=bool(enable_preview)) + if context_before is not None and video_context is not None: + assert_context_unchanged(video_context,audio_context,context_before) + sampled_video,sampled_audio=latent_to_cpu(sampled) + logical_parts=_split_terminal_merged_latents(sampled_video,sampled_audio,contract) + sampling_reports.append(f"terminal physical sample: terminal_pair={pair_start+1}-{chunks}, physical_seed={physical_seed}, physical_frames={physical_frames}, sampling_passes=1, shared_physical_sample=true, terminal_prompt_policy={terminal_prompt_policy}, trim={physical_context_frames}, reason={reason}") + for offset,(video_part,audio_part) in enumerate(logical_parts): + sequence_index=pair_start+offset + total_frames=int(contract["logical_frames"][offset]) + trim_frames=int(contract["logical_trims"][offset]) + continuation=bool(trim_frames) + clip_index=physical_clip_index+offset + context_frames=trim_frames if continuation else 5 + chunk_plan=make_plan(continuation=continuation,clip_index=clip_index,total_frames=total_frames,trim_frames=trim_frames,width=width,height=height,context_frames=context_frames,state_capacity_frames=largest_context_capacity(total_frames-trim_frames),requested_extend_seconds=chunk_seconds,debug=debug) + logical_latent=latent_from_cpu(video_part,audio_part) + logical_seed=int(terminal_seed_plan["logical_entry_seeds"][offset]) + entry=make_chunk_entry(latent=logical_latent,plan=chunk_plan,prompt=prompts[sequence_index],prompt_hash=prompt_hashes[sequence_index],seed=logical_seed,context_frames=trim_frames,motion_score=motion_score,reused=False) + entries.append(entry) + previous_state=entry_to_state(entry) + _record_context_diagnostics(tracker=context_diagnostics,reports=sampling_reports,state=previous_state,continuity=continuity,reused=False) + if storage_controller is not None: + storage_controller.commit_chunk(entry,position=sequence_index) + retained_frames+=int(chunk_plan["net_frames"]) + sampling_reports.append(f"chunk {sequence_index+1}/{chunks}: seed={logical_seed}, frames={total_frames}, trim={trim_frames}, retained_total={retained_frames}, context={trim_frames}, shared_physical_sample=terminal_10s_seed_v2") + del logical_latent + del sampled,sampled_video,sampled_audio,logical_parts,latent,conditioning,chunk_model if len(entries)!=chunks: raise SequenceRuntimeError(f"internal sequence length mismatch: expected {chunks}, got {len(entries)}") if latent_only: last_state=entry_to_state(entries[-1]); parent_id=session.get("session_id") if session is not None else None @@ -308,7 +579,7 @@ def run_sequence(*,model:Any,clip:Any,video_vae:Any,audio_vae:Any,sampler:Any,si if reference_video_source is not None: settings["reference_video_contract"]=reference_video_source.contract if timeline_video_source is not None: settings["timeline_video_contract"]=timeline_video_source.contract new_session=make_session(chunks=entries,width=width,height=height,chunk_seconds=chunk_seconds,identity_hash=sequence_identity_hash,model_fingerprint_value=current_model_fingerprint,parent_session_id=parent_id,reroll_from_chunk=int(reroll_from_chunk),settings=settings) - report_lines=[f"H3 Continuum V3 {PACKAGE_VERSION}",f"Conditioning mode: {conditioning_mode_label(conditioning_mode)}.",prompt_plan_report(plan),"Decode: external ComfyUI Core VAE nodes; full raw AV chunks retained.",accelerators,"Execution: conditioning precomputed; single call-local MODEL clone per chunk; no internal VAE decode.",*reuse_notes] + report_lines=[f"H3 Continuum V3 {PACKAGE_VERSION}",f"Conditioning mode: {conditioning_display}.",prompt_plan_report(plan),"Decode: external ComfyUI Core VAE nodes; full raw AV chunks retained.",accelerators,"Execution: conditioning precomputed; single call-local MODEL clone per chunk; no internal VAE decode.",*reuse_notes] if diagnostics_mode!=DIAGNOSTICS_OFF: report_lines.extend(sampling_reports) report_lines.extend([session_summary(new_session),f"Output: {len(entries)} raw AV latent chunk(s); connect Core VAE Decode nodes, then H3 Continuum Assemble V3."]) return entries,last_state,new_session,"\n".join(report_lines) diff --git a/v3/assembly.py b/v3/assembly.py index 10ae62c..2e5c984 100644 --- a/v3/assembly.py +++ b/v3/assembly.py @@ -71,7 +71,9 @@ def assemble_decoded_chunks( if audio_seam not in AUDIO_SEAM_OPTIONS: raise ValueError(f"unknown Audio Seam mode: {audio_seam!r}") - chunk_plans = list(plan["chunks"]) + chunk_plans = list( + plan["decode_groups"] if "decode_groups" in plan else plan["chunks"] + ) if len(images) != len(chunk_plans) or len(audio) != len(chunk_plans): raise ValueError( "decoded chunk count does not match the V3 assembly plan: " diff --git a/v3/nodes.py b/v3/nodes.py index 5b60916..1ab2133 100644 --- a/v3/nodes.py +++ b/v3/nodes.py @@ -338,13 +338,33 @@ class H3ContinuumSamplerV3: **clip_prompt_inputs, ) - assembly_plan = make_assembly_plan( + from ..v2.sequence import _terminal_flf_merge_enabled + from .plan import prepare_physical_decode_entries + + terminal_merged = _terminal_flf_merge_enabled( + multi_chunk_flf=( + first_frame is not None + and advanced_values["last_frame"] is not None + and len(entries) > 1 + ), + chunks=len(entries), + chunk_seconds=float(chunk_seconds), + prompt_hashes=[str(entry["prompt_hash"]) for entry in entries], + timeline_video_source=timeline_video_source, + ) + decode_entries, assembly_plan = prepare_physical_decode_entries( entries, chunk_seconds=float(chunk_seconds), preserve_final_frame=advanced_values["last_frame"] is not None, + terminal_merged=terminal_merged, ) - video_latents = [{"samples": entry["video"]} for entry in entries] - audio_latents = [{"samples": entry["audio"]} for entry in entries] + video_latents = [{"samples": entry["video"]} for entry in decode_entries] + audio_latents = [{"samples": entry["audio"]} for entry in decode_entries] + if terminal_merged: + report = str(report).rstrip() + ( + "\nDecode: terminal merged latent retained as one physical external " + "Core VAE decode group; logical Run Storage entries are unchanged." + ) result = { "last_state": last_state, "session": session, diff --git a/v3/plan.py b/v3/plan.py index cef89b4..d3e3d32 100644 --- a/v3/plan.py +++ b/v3/plan.py @@ -141,3 +141,151 @@ def make_assembly_plan(*args, **kwargs): _enrich_assembly_plan(plan) validate_assembly_plan(plan) return plan + + +def _recombine_terminal_tensor( + first, + second, + *, + slices, + time_dim: int, + name: str, +): + """Invert the terminal logical split without changing any tensor value.""" + + import torch + + if not torch.is_tensor(first) or not torch.is_tensor(second): + raise ValueError(f"terminal {name} entries must be tensors") + if first.ndim != second.ndim: + raise ValueError(f"terminal {name} tensor ranks differ") + dim = int(time_dim) + if dim < 0: + dim += first.ndim + if dim < 0 or dim >= first.ndim: + raise ValueError(f"terminal {name} time dimension is invalid") + for axis, (left, right) in enumerate(zip(first.shape, second.shape)): + if axis != dim and int(left) != int(right): + raise ValueError(f"terminal {name} tensor geometry differs") + + (first_start, first_stop), (second_start, second_stop) = slices + first_start = int(first_start) + first_stop = int(first_stop) + second_start = int(second_start) + second_stop = int(second_stop) + if first_start != 0 or second_start > first_stop or second_stop <= first_stop: + raise ValueError(f"terminal {name} split ranges do not form one physical unit") + if int(first.shape[dim]) != first_stop - first_start: + raise ValueError(f"terminal {name} first slice length is inconsistent") + if int(second.shape[dim]) != second_stop - second_start: + raise ValueError(f"terminal {name} second slice length is inconsistent") + + overlap_start = max(first_start, second_start) + overlap_stop = min(first_stop, second_stop) + overlap_length = max(0, overlap_stop - overlap_start) + if overlap_length: + first_overlap = first.narrow(dim, overlap_start - first_start, overlap_length) + second_overlap = second.narrow(dim, overlap_start - second_start, overlap_length) + if not torch.equal(first_overlap, second_overlap): + raise ValueError( + f"terminal {name} overlap differs; refusing lossy physical reconstruction" + ) + + output_shape = list(first.shape) + output_shape[dim] = second_stop + output = first.new_empty(output_shape) + output.narrow(dim, first_start, first_stop - first_start).copy_(first) + output.narrow(dim, second_start, second_stop - second_start).copy_(second) + return output.contiguous() + + +def prepare_physical_decode_entries( + entries, + *, + chunk_seconds: float, + preserve_final_frame: bool, + terminal_merged: bool, +): + """Return external-VAE decode units while preserving logical plan metadata.""" + + logical_entries = list(entries) + plan = make_assembly_plan( + logical_entries, + chunk_seconds=float(chunk_seconds), + preserve_final_frame=bool(preserve_final_frame), + ) + if not terminal_merged: + return logical_entries, plan + if len(logical_entries) < 2: + raise ValueError("terminal physical decode requires two logical entries") + + from ..v2.sequence import _terminal_pair_contract + + initial_pair = len(logical_entries) == 2 + contract = _terminal_pair_contract( + initial_pair=initial_pair, + chunk_seconds=float(chunk_seconds), + ) + first_entry = logical_entries[-2] + second_entry = logical_entries[-1] + physical_video = _recombine_terminal_tensor( + first_entry["video"], + second_entry["video"], + slices=contract["video_slices"], + time_dim=2, + name="video", + ) + physical_audio = _recombine_terminal_tensor( + first_entry["audio"], + second_entry["audio"], + slices=contract["audio_slices"], + time_dim=-1, + name="audio", + ) + + physical_entry = dict(first_entry) + physical_entry["video"] = physical_video + physical_entry["audio"] = physical_audio + decode_entries = [*logical_entries[:-2], physical_entry] + + logical_chunks = list(plan["chunks"]) + decode_groups = [] + for chunk in logical_chunks[:-2]: + group = dict(chunk) + group["logical_chunk_indices"] = [int(chunk["chunk_index"])] + group["terminal_merged"] = False + decode_groups.append(group) + + first_chunk = logical_chunks[-2] + second_chunk = logical_chunks[-1] + frame_start = int(first_chunk.get("frame_start", 0)) + net_frames = int(contract["physical_frames"]) - int( + contract["physical_context_frames"] + ) + terminal_group = dict(first_chunk) + terminal_group.update( + { + "total_frames": int(contract["physical_frames"]), + "trim_frames": int(contract["physical_context_frames"]), + "net_frames": net_frames, + "context_frames": int(contract["physical_context_frames"]), + "expected_video_latent_t": int(physical_video.shape[2]), + "expected_audio_latent_t": int(physical_audio.shape[-1]), + "frame_start": frame_start, + "frame_stop": frame_start + net_frames, + "logical_chunk_indices": [ + int(first_chunk["chunk_index"]), + int(second_chunk["chunk_index"]), + ], + "terminal_merged": True, + } + ) + decode_groups.append(terminal_group) + + plan = dict(plan) + plan["decode_group_version"] = 1 + plan["logical_chunk_count"] = len(logical_chunks) + plan["physical_decode_group_count"] = len(decode_groups) + plan["decode_groups"] = decode_groups + validate_assembly_plan(plan) + return decode_entries, plan diff --git a/v3/video_seam.py b/v3/video_seam.py index 0887f56..2548790 100644 --- a/v3/video_seam.py +++ b/v3/video_seam.py @@ -390,7 +390,9 @@ def analyze_decoded_boundaries( """Analyze every decoded chunk boundary without changing decoded content.""" plan = validate_assembly_plan(assembly_plan) - chunks = list(plan["chunks"]) + chunks = list( + plan["decode_groups"] if "decode_groups" in plan else plan["chunks"] + ) if len(images) != len(chunks): raise ValueError("decoded image count does not match the assembly plan") analyses: list[VideoBoundaryAnalysis] = [] @@ -428,7 +430,9 @@ def correct_decoded_boundaries( """Normalize qualified transient exposure shifts without changing motion.""" plan = validate_assembly_plan(assembly_plan) - chunks = list(plan["chunks"]) + chunks = list( + plan["decode_groups"] if "decode_groups" in plan else plan["chunks"] + ) if len(images) != len(chunks): raise ValueError("decoded image count does not match the assembly plan") corrected = list(images)