feat: finalize FL2VA terminal merge
This commit is contained in:
+47
-1
@@ -51,7 +51,8 @@ Static checks do not replace a real GPU generation. Record workflow, prompt, med
|
||||
- PASS `MiniMax_H3_Continuum_V31_00032_.mp4`: standard Ref2VA model; visual result accepted.
|
||||
- FAIL `MiniMax_H3_Continuum_V31_00034_.mp4`: Hybrid FLF+Reference; visible jumps near 5 s and 9 s.
|
||||
- FAIL `MiniMax_H3_Continuum_V31_00035_.mp4`: Ref2VA model plus FL2V Turbo 8-step LoRA; two visible flashes/jumps.
|
||||
- Current FLF baseline is `context_bound_terminal_v1`: final-chunk context-tail and Last Frame anchors only.
|
||||
- Current FLF candidate is `context_bound_terminal_last_only_v2`: the final chunk keeps its 22-frame continuation context and the single Last Frame anchor retained by `prepare_conditioning()`, without adding a duplicate context-tail boundary keyframe.
|
||||
- GPU verification of this candidate is pending; it is not yet a stable baseline.
|
||||
- `context_bound_terminal_bridge_v2` is withdrawn because its synthetic intermediate latent correlated with the later terminal flash regression.
|
||||
- Preserve Hybrid/Reference support, Run Storage identity, UI, audio paths, and permissive prompt handling when repairing FLF.
|
||||
|
||||
@@ -78,3 +79,48 @@ Static checks do not replace a real GPU generation. Record workflow, prompt, med
|
||||
- The initial rollback after 00038 temporarily returned to bridge v2, but the user clarified that the required target is the exact code state used for PASS 00031.
|
||||
- `00031` completed at 2026-08-21 02:32:59; `pre-flf-terminal-bridge-v2-repo-20260821_023553` was captured immediately afterward and before bridge v2 was introduced.
|
||||
- Current FLF strategy is restored exactly from that snapshot: `context_bound_terminal_v1`. Bridge v2 is withdrawn and is not active.
|
||||
|
||||
## M1 Hybrid diagnostics (2026-08-21)
|
||||
|
||||
- Canonical implementation source: ComfyUI_W runtime.
|
||||
- Runtime pre-change snapshot: D:\Codex\_snapshots\ComfyUI-H3-Continuum\pre-m1-comfyui-w-canonical-20260821_172324.
|
||||
- Added display-only Hybrid conditioning labels; the stored/runtime conditioning mode remains unchanged.
|
||||
- Hybrid reference warnings now use the actual global Qwen <Picture N> numbering.
|
||||
- Added README numbering guidance and focused CPU-only regression tests.
|
||||
- No Sampling, Conditioning payload, Run Storage contract, UI socket, or model-selection behavior changed.
|
||||
- Tests were added but not executed in this implementation step.
|
||||
|
||||
## Terminal Merge Physical Seed v2 (2026-08-22 00:49:57)
|
||||
- Terminal Merge uses one physical seed per merged pair; an initial 2x5s pair uses base_seed directly when reroll is inactive.
|
||||
- Later terminal pairs derive once from the zero-based pair-start index; both logical entries store that same physical seed.
|
||||
- Run Storage distinguishes the scoped terminal_merged_10s_seed_v2 / physical_window_seed_v2 execution semantics without changing the global schema version.
|
||||
|
||||
|
||||
## Terminal Merge diagnostic harness (2026-08-22)
|
||||
|
||||
- Production generation semantics remain unchanged; diagnostics live under tests only.
|
||||
- Diagnostic stages are ordered A Pre-Sampler, B First MODEL Call, C sampled physical AV, D split/recombine, E decode/assembly.
|
||||
- Tensor identity uses shape, dtype, and SHA-256 of contiguous CPU raw bytes; numerical mismatch also records max/mean absolute difference and allclose.
|
||||
- Terminal split/recombine requires bit-exact equality.
|
||||
- Real Core vs Continuum GPU capture remains pending; this unit-test phase does not load checkpoints.
|
||||
|
||||
## GPU Core/Continuum diagnostic runner (2026-08-22)
|
||||
- Source: `tools/gpu_diagnostic_node/`
|
||||
- Installer: `tools/install_gpu_diagnostic_node.ps1`
|
||||
- Runtime: `D:\StabilityMatrix\Data\Packages\ComfyUI_W\custom_nodes\H3_Continuum_GPU_Diagnostics`
|
||||
- Node: `H3 Core vs Continuum GPU Diagnostic`
|
||||
- Production Continuum files and generation semantics are not modified by this diagnostic plugin.
|
||||
- Captures A: pre-sampler, B: first MODEL call, C: physical sampled AV latent, D: terminal split/recombine, E1: decode comparison, E2: common 243-to-240 adjustment.
|
||||
- Artifacts: JSON manifest plus safetensors under the configured ComfyUI output diagnostics directory.
|
||||
- Pre-change snapshot: `D:\Codex\_snapshots\ComfyUI-H3-Continuum\pre-gpu-core-continuum-diagnostic-runner-20260822_022625` (138/138 files verified).
|
||||
- GPU comparison has not yet been executed.
|
||||
|
||||
## Long Terminal Merge for distinct timeline sections (2026-08-22)
|
||||
|
||||
- For FL2VA with 3 or more 5-second logical chunks, the final two chunks may now share one physical Terminal Merge sample even when their prompt sections differ.
|
||||
- Distinct final prompts are rebased to a local physical timeline (`[0-5s]` and `[5-10s]`); identical prompts retain the accepted 2x5 Core-equivalent prompt path unchanged.
|
||||
- Expected 3x5 execution: 3 logical chunks, 2 physical sampling passes, and 2 physical VAE decode groups; the terminal group is 260 frames with a 22-frame decoded prefix trim and 238 retained frames.
|
||||
- Timeline Video remains excluded from Terminal Merge. UI, public sockets, Conditioning payload construction, Physical Seed v2, Assembly, and Seam semantics are otherwise unchanged.
|
||||
- GPU acceptance passed for `MiniMax_H3_Continuum_V31_00008_.mp4`: 3 logical chunks, 2 physical sampling passes, decoded groups of 124 and 238 retained frames, 362-to-360 exact-duration adjustment, and 15.000-second output.
|
||||
- Core comparison `MiniMax_H3_00005_.mp4` showed the same short terminal still period. This is accepted as normal FL2VA Last Frame convergence, not a Continuum-specific defect.
|
||||
- Long Terminal Merge is formally adopted. Do not retune Conditioning, Physical Seed v2, physical decode grouping, or terminal prompt pairing without new contrary regression evidence.
|
||||
|
||||
@@ -15,6 +15,7 @@ V3.4 focuses on the two reference workflows that are most useful in normal produ
|
||||
- **Driving Audio**: use an existing audio source as the preserved final audio while it guides the H3 generation across chunks.
|
||||
- **Video Reference**: use an existing video as persistent visual reference for appearance, motion, framing, and timing without treating it as a frame-by-frame copy.
|
||||
- **Hybrid FLF + Reference**: use First Frame, Last Frame, and persistent Reference Images together without adding a separate mode or model allowlist.
|
||||
- **FL2VA Terminal Merge**: keep the final two 5-second FL2VA chunks as one physical 10-second sampling and decode unit for Core-equivalent Last Frame handling.
|
||||
- **Restartable chunks**: reuse completed chunks with Run Storage and regenerate only the selected part when the generation contract remains compatible.
|
||||
- **Core compatibility**: remove Continuum-only rejection of unknown upstream/custom nodes and remove the obsolete `strict_compatibility` control.
|
||||
- **Spectrum interoperability**: use the official H3 Continuum Interop API v1 when Spectrum is installed; Spectrum remains optional.
|
||||
@@ -83,6 +84,18 @@ V3.4 supports First Frame, Last Frame, and persistent Reference Images in the sa
|
||||
|
||||
Local checks passed with the B2049 hybrid variant. FL-only models could run, but showed less reliable transitions when FLF and Reference Images were combined; a hybrid-capable model is therefore recommended for this path.
|
||||
|
||||
### FL2VA Terminal Merge
|
||||
|
||||
For 5-second FL2VA workflows, Continuum keeps the final two logical chunks together as one physical sampling and VAE decode unit.
|
||||
|
||||
- `2 x 5s`: one 243-frame physical sample, decoded once, then adjusted to the requested 240-frame output.
|
||||
- `3 x 5s` and longer: earlier chunks use normal Continuation; the final two chunks use one 260-frame physical sample containing the 22-frame continuation context.
|
||||
- After the terminal physical decode, the 22-frame context prefix is removed. For `3 x 5s`, this gives `124 + 238 = 362` frames before exact-duration adjustment to 360 frames.
|
||||
- If the final two prompt sections differ, they are rebased internally to a local `[0-5s]` / `[5-10s]` timeline. Identical prompts retain the accepted shared-prompt path.
|
||||
- The terminal pair uses one physical seed and one physical decode while Run Storage continues to preserve normal logical chunk entries.
|
||||
|
||||
Local GPU comparison confirmed that `2 x 5s` matches Core 10-second FL2VA sampling and terminal decode behavior. A `3 x 5s` Timeline run also passed with three logical chunks, two physical sampling passes, two physical decode groups, and a 15.000-second output. The short still period near the supplied Last Frame was also present in the equivalent Core result and is treated as normal FL2VA convergence rather than a Continuum-specific failure.
|
||||
|
||||
### Core-first, permissive execution
|
||||
|
||||
V3.4 removes or relaxes Continuum-only rejection rules that blocked otherwise runnable Core H3 experiments.
|
||||
@@ -295,6 +308,8 @@ V3.4 has been exercised locally with:
|
||||
- Spectrum enabled and disabled
|
||||
- I2VA, Reference, and selected FL2VA configurations
|
||||
- Hybrid FLF + Reference with the B2049 hybrid variant
|
||||
- Core-equivalent `2 x 5s` FL2VA Terminal Merge
|
||||
- `3 x 5s` FL2VA with the final two logical chunks merged into one 260-frame physical sample and decode group
|
||||
- Driving Audio with short, exact-length, and longer sources
|
||||
- Video Reference with source audio routed separately
|
||||
- 0.4 MP and 0.6 MP reference sizing
|
||||
@@ -308,6 +323,8 @@ No OOM was observed in the cited recent local V3.4 checks, including two-chunk 8
|
||||
- Continuation does not guarantee frame-perfect identity or motion.
|
||||
- Video Reference guides H3; it does not reproduce every source frame.
|
||||
- Driving Audio preserves selected audio, but visual lip synchronization remains model-dependent.
|
||||
- FL2VA may settle on the supplied Last Frame before the requested duration ends; equivalent Core runs can show the same behavior.
|
||||
- Long, high-resolution sequences may exceed system RAM during final decode and assembly even when chunk sampling succeeds.
|
||||
- Match Output can be substantially slower than 0.4 MP or 0.6 MP.
|
||||
- Seam correction may keep the native boundary when a proposed correction is not safer.
|
||||
- Optional template nodes must be installed, replaced, or bypassed by the user.
|
||||
|
||||
+62
@@ -60,3 +60,65 @@ D:\Codex\_snapshots\ComfyUI-H3-Continuum\pre-rollback-after-00038-fail-20260821_
|
||||
- Restored only `v2/sequence.py` from `pre-flf-terminal-bridge-v2-repo-20260821_023553` to the repository and active ComfyUI_W runtime.
|
||||
- Active FLF strategy is `context_bound_terminal_v1`; bridge v2 is no longer active.
|
||||
- No UI or unrelated feature files were changed. GPU retest was not run.
|
||||
|
||||
## 2026-08-21 - M1 Hybrid diagnostics
|
||||
|
||||
- Used ComfyUI_W as the authoritative source after confirming repository/runtime differences.
|
||||
- Created and verified a runtime snapshot before edits.
|
||||
- Implemented display-only Hybrid labels and global Picture-number warnings.
|
||||
- Documented Hybrid Picture numbering and added focused regression tests.
|
||||
- Runtime generation behavior and public UI were intentionally left unchanged.
|
||||
- Tests were not run in this implementation step.
|
||||
## 2026-08-21 - FLF duplicate boundary anchor removal
|
||||
|
||||
- Preserved the active ComfyUI_W M3 display-label and M4 Last Frame session checks while synchronizing `v2/sequence.py` to the repository.
|
||||
- Changed the FLF contract to `context_bound_terminal_last_only_v2` so Run Storage cannot reuse chunks produced by the previous strategy.
|
||||
- Removed only the final-chunk call that duplicated the continuation-context tail as a boundary keyframe.
|
||||
- Kept the 22-frame continuation context and the single supplied Last Frame anchor retained by `prepare_conditioning()`.
|
||||
- No UI, public sockets, Assembly, audio, Spectrum, or general continuation behavior changed.
|
||||
- Automated and GPU validation were not run in this implementation step.
|
||||
|
||||
- 2026-08-22 00:49:57 Terminal Merge Physical Seed v2 implemented locally after snapshot pre-change-20260822_004423. Changed v2/sequence.py and run_storage.py; added seed and revision regressions. No commit or push.
|
||||
|
||||
- 2026-08-22 00:51:45 Validation: 220 passed, 1 pre-existing failure in test_v2_sequence_orchestration because conditioning_display_label is absent from conditioning.py. All Physical Seed v2 and Run Storage revision tests passed. GPU generation not run.
|
||||
|
||||
|
||||
## 2026-08-22 - Terminal Merge staged diagnostics
|
||||
|
||||
- Snapshot: D:\Codex\_snapshots\ComfyUI-H3-Continuum\pre-terminal-merge-diagnostics-20260822_020527
|
||||
- Added tests-only deterministic capture/comparison helpers for stages A-E.
|
||||
- Added unit coverage for noise, PackedLayout/position_ids, sampled-stage ordering, numerical tolerance, and bit-exact 243f Terminal split/recombine.
|
||||
- No Production code, UI, Conditioning, Seed v2, Terminal Merge, Assembly, or Seam behavior changed.
|
||||
- Commit/push not performed.
|
||||
|
||||
## 2026-08-22 - Standalone Core vs Continuum GPU diagnostics
|
||||
- Added a standalone diagnostic node without changing Production runtime semantics.
|
||||
- Added capture helpers and regression tests.
|
||||
- Installed to ComfyUI_W as a separate custom node and verified source/runtime SHA-256 equality.
|
||||
- Registration import: PASS (`H3CoreContinuumGpuDiagnostic`).
|
||||
- Targeted tests: 6 passed.
|
||||
- Full suite: 232 passed, 1 pre-existing unrelated failure (`conditioning_display_label` missing for unfinished M3 wiring).
|
||||
- GPU execution: not run.
|
||||
- Commit/push: not performed.
|
||||
## 2026-08-22 - Terminal Merge physical decode groups
|
||||
|
||||
- Preserved logical chunks for Session and Run Storage.
|
||||
- Recombined terminal logical AV latent slices bit-exactly before external Core VAE Decode.
|
||||
- Added assembly_plan.decode_groups so one terminal physical sample is decoded once.
|
||||
- Assembly hardening and Video Seam now operate at physical decode-group boundaries.
|
||||
- Sampling, Physical Seed v2, Conditioning, UI, sockets, and Run Storage schema were unchanged.
|
||||
- Synced implementation to ComfyUI_W.
|
||||
- Pending validation: targeted pytest, full pytest, and GPU Core-vs-Continuum decode comparison.
|
||||
|
||||
## 2026-08-22 - Long Terminal Merge prompt pairing
|
||||
|
||||
- Snapshots: `pre-long-terminal-merge-repo`, `pre-long-terminal-merge-comfyui-w`, and `pre-long-terminal-merge-comfyui-wan` under `D:\Codex\_snapshots\ComfyUI-H3-Continuum`.
|
||||
- Removed only the final-prompt-hash equality gate from Terminal Merge eligibility.
|
||||
- Added a scoped physical prompt policy: identical prompts remain byte-identical; distinct final logical prompts are rebased to `[0-5s]` / `[5-10s]` for one terminal physical sample.
|
||||
- Added scoped Run Storage execution semantics and 3-logical-to-2-physical decode-group regression coverage.
|
||||
- No UI, public sockets, model selection, Spectrum, Assembly, Seam, or global Run Storage schema changes.
|
||||
- Focused regression result: 22 passed; GPU generation not yet run for the new 3x5 path.
|
||||
- Runtime correction: `run_storage.py` was also synchronized to ComfyUI_W and ComfyUI_WAN after the first GPU attempt exposed an old `RunStorageController.prepare()` signature. Run Storage plus Terminal Merge regressions: 47 passed.
|
||||
- GPU PASS: `MiniMax_H3_Continuum_V31_00008_.mp4` exercised 3x5s FL2VA with `terminal_pair=2-3`, `physical_frames=260`, `sampling_passes=1` for the pair, `paired_timeline_v1`, two physical decode groups, and final 360 frames / 15.000 seconds.
|
||||
- Core comparison `MiniMax_H3_00005_.mp4` reproduced the same short terminal still period. Long Terminal Merge was formally accepted; terminal stillness is Core-equivalent FL2VA convergence and is not a Continuum repair target.
|
||||
- Final release validation: full pytest `249 passed`; `tools/verify_runtime.py --comfy-root D:\StabilityMatrix\Data\Packages\ComfyUI_W` passed V3.4 registration, native PackedLayout, and Fixed 3x5s prompt-plan checks. GPU acceptance is recorded above.
|
||||
|
||||
@@ -59,6 +59,46 @@ def conditioning_mode_label(mode: str) -> str:
|
||||
raise ValueError(f"unknown conditioning mode: {mode!r}") from exc
|
||||
|
||||
|
||||
def conditioning_display_label(
|
||||
*,
|
||||
has_first: bool,
|
||||
has_last: bool,
|
||||
has_reference: bool,
|
||||
) -> str:
|
||||
"""Return a user-facing label without changing the runtime mode contract."""
|
||||
if has_reference:
|
||||
if has_first and has_last:
|
||||
return "Hybrid FL2VA + Reference + Continuation"
|
||||
if has_first:
|
||||
return "Hybrid I2VA + Reference + Continuation"
|
||||
if has_last:
|
||||
return "Hybrid L2VA + Reference + Continuation"
|
||||
return "Reference + Continuation"
|
||||
mode = conditioning_mode_from_presence(
|
||||
has_first=has_first,
|
||||
has_last=has_last,
|
||||
has_reference=False,
|
||||
)
|
||||
return conditioning_mode_label(mode)
|
||||
|
||||
|
||||
def conditioning_mode_display_label(
|
||||
mode: str,
|
||||
*,
|
||||
has_first: bool = False,
|
||||
has_last: bool = False,
|
||||
has_reference: bool = False,
|
||||
) -> str:
|
||||
"""Legacy display helper retained for existing callers and tests."""
|
||||
if has_first or has_last or has_reference:
|
||||
return conditioning_display_label(
|
||||
has_first=has_first,
|
||||
has_last=has_last,
|
||||
has_reference=has_reference,
|
||||
)
|
||||
return conditioning_mode_label(mode)
|
||||
|
||||
|
||||
def conditioning_mode_uses_video_vae(mode: str) -> bool:
|
||||
if mode not in CONDITIONING_MODES:
|
||||
raise ValueError(f"unknown conditioning mode: {mode!r}")
|
||||
|
||||
+124
-2
@@ -21,6 +21,15 @@ def _strict_int(name: str, value: Any, minimum: int | None = None) -> int:
|
||||
return value
|
||||
|
||||
|
||||
def _assembly_decode_units(plan: Mapping[str, Any]) -> list[Any]:
|
||||
if "decode_groups" not in plan:
|
||||
return plan["chunks"]
|
||||
groups = plan["decode_groups"]
|
||||
if not isinstance(groups, list) or not groups:
|
||||
raise ValueError("assembly_plan.decode_groups must be a non-empty list")
|
||||
return groups
|
||||
|
||||
|
||||
def enrich_assembly_plan(plan: dict[str, Any]) -> dict[str, Any]:
|
||||
"""Add redundant metadata without correcting any existing value."""
|
||||
chunks = plan.get("chunks")
|
||||
@@ -184,6 +193,117 @@ def validate_assembly_plan_contract(plan: Mapping[str, Any]) -> None:
|
||||
"sum(chunk.net_frames) does not match assembly_plan.natural_frames"
|
||||
)
|
||||
|
||||
if "decode_groups" in plan:
|
||||
decode_groups = _assembly_decode_units(plan)
|
||||
if "physical_decode_group_count" in plan:
|
||||
group_count = _strict_int(
|
||||
"assembly_plan.physical_decode_group_count",
|
||||
plan["physical_decode_group_count"],
|
||||
1,
|
||||
)
|
||||
if group_count != len(decode_groups):
|
||||
raise ValueError(
|
||||
"assembly_plan.physical_decode_group_count does not match decode_groups"
|
||||
)
|
||||
expected_logical_indices = [int(chunk["chunk_index"]) for chunk in chunks]
|
||||
covered_logical_indices: list[int] = []
|
||||
decode_natural_frames = 0
|
||||
for position, group in enumerate(decode_groups, start=1):
|
||||
if not isinstance(group, Mapping):
|
||||
raise ValueError(
|
||||
f"assembly_plan decode group {position} must be a mapping"
|
||||
)
|
||||
total_frames = _strict_int(
|
||||
f"assembly_plan decode group {position}.total_frames",
|
||||
group.get("total_frames"),
|
||||
5,
|
||||
)
|
||||
trim_frames = _strict_int(
|
||||
f"assembly_plan decode group {position}.trim_frames",
|
||||
group.get("trim_frames"),
|
||||
0,
|
||||
)
|
||||
net_frames = _strict_int(
|
||||
f"assembly_plan decode group {position}.net_frames",
|
||||
group.get("net_frames"),
|
||||
1,
|
||||
)
|
||||
context_frames = _strict_int(
|
||||
f"assembly_plan decode group {position}.context_frames",
|
||||
group.get("context_frames"),
|
||||
0,
|
||||
)
|
||||
_strict_int(
|
||||
f"assembly_plan decode group {position}.expected_video_latent_t",
|
||||
group.get("expected_video_latent_t"),
|
||||
1,
|
||||
)
|
||||
_strict_int(
|
||||
f"assembly_plan decode group {position}.expected_audio_latent_t",
|
||||
group.get("expected_audio_latent_t"),
|
||||
1,
|
||||
)
|
||||
if (total_frames - 5) % 17 != 0:
|
||||
raise ValueError(
|
||||
f"assembly_plan decode group {position}.total_frames is not on "
|
||||
"the H3 17k+5 grid"
|
||||
)
|
||||
if total_frames - trim_frames != net_frames:
|
||||
raise ValueError(
|
||||
f"assembly_plan decode group {position}: "
|
||||
"total_frames - trim_frames != net_frames"
|
||||
)
|
||||
if trim_frames > 0 and context_frames != trim_frames:
|
||||
raise ValueError(
|
||||
f"assembly_plan decode group {position}: "
|
||||
"continuation context and trim differ"
|
||||
)
|
||||
logical_indices = group.get("logical_chunk_indices")
|
||||
if not isinstance(logical_indices, list) or not logical_indices:
|
||||
raise ValueError(
|
||||
f"assembly_plan decode group {position}.logical_chunk_indices "
|
||||
"must be a non-empty list"
|
||||
)
|
||||
covered_logical_indices.extend(
|
||||
_strict_int(
|
||||
f"assembly_plan decode group {position}.logical_chunk_indices",
|
||||
value,
|
||||
1,
|
||||
)
|
||||
for value in logical_indices
|
||||
)
|
||||
if "terminal_merged" in group and type(group["terminal_merged"]) is not bool:
|
||||
raise ValueError(
|
||||
f"assembly_plan decode group {position}.terminal_merged must be boolean"
|
||||
)
|
||||
frame_start = _strict_int(
|
||||
f"assembly_plan decode group {position}.frame_start",
|
||||
group.get("frame_start"),
|
||||
0,
|
||||
)
|
||||
frame_stop = _strict_int(
|
||||
f"assembly_plan decode group {position}.frame_stop",
|
||||
group.get("frame_stop"),
|
||||
1,
|
||||
)
|
||||
if (
|
||||
frame_start != decode_natural_frames
|
||||
or frame_stop != decode_natural_frames + net_frames
|
||||
):
|
||||
raise ValueError(
|
||||
f"assembly_plan decode group {position}: natural frame boundary "
|
||||
"is discontinuous"
|
||||
)
|
||||
decode_natural_frames += net_frames
|
||||
if covered_logical_indices != expected_logical_indices:
|
||||
raise ValueError(
|
||||
"assembly_plan.decode_groups do not cover logical chunks exactly once"
|
||||
)
|
||||
if decode_natural_frames != natural_frames:
|
||||
raise ValueError(
|
||||
"sum(decode_group.net_frames) does not match logical natural frames"
|
||||
)
|
||||
|
||||
|
||||
def preflight_decoded_chunks(
|
||||
images: Any,
|
||||
@@ -192,7 +312,7 @@ def preflight_decoded_chunks(
|
||||
) -> None:
|
||||
"""Validate all decoded chunks before the assembler allocates output buffers."""
|
||||
validate_assembly_plan_contract(plan)
|
||||
chunks = plan["chunks"]
|
||||
chunks = _assembly_decode_units(plan)
|
||||
|
||||
if not isinstance(images, (list, tuple)):
|
||||
raise ValueError("decoded images must be a chunk list")
|
||||
@@ -518,7 +638,9 @@ def assemble_with_hardening(base: Any, args: tuple[Any, ...], kwargs: dict[str,
|
||||
lines: list[str] = []
|
||||
if diagnostics_is_full(diagnostics):
|
||||
lines.append(format_memory_snapshot("assemble preflight", images, audio))
|
||||
natural_frames = sum(int(chunk["net_frames"]) for chunk in plan["chunks"])
|
||||
natural_frames = sum(
|
||||
int(chunk["net_frames"]) for chunk in _assembly_decode_units(plan)
|
||||
)
|
||||
target_frames = int(plan["target_frames"])
|
||||
correction = target_frames - natural_frames
|
||||
if abs(correction) > 2:
|
||||
|
||||
@@ -571,6 +571,7 @@ def build_sampling_contract(
|
||||
driving_audio_vae: Any = None,
|
||||
reference_video_contract: dict[str, Any] | None = None,
|
||||
timeline_video_contract: dict[str, Any] | None = None,
|
||||
execution_semantics: dict[str, Any] | None = None,
|
||||
) -> tuple[dict[str, Any], bool, list[str]]:
|
||||
model_value, model_safe = _model_signature(model, model_fingerprint_value)
|
||||
clip_value, clip_safe = _clip_signature(clip)
|
||||
@@ -660,6 +661,8 @@ def build_sampling_contract(
|
||||
}
|
||||
if graph_authoritative:
|
||||
global_contract["upstream_graph"] = dict(upstream_graph_contract)
|
||||
if execution_semantics is not None:
|
||||
global_contract["execution_semantics"] = dict(execution_semantics)
|
||||
if uses_video_vae:
|
||||
global_contract["video_vae"] = video_vae_value
|
||||
if has_first:
|
||||
@@ -1017,6 +1020,7 @@ class RunStorageController:
|
||||
driving_audio_vae: Any = None,
|
||||
reference_video_contract: dict[str, Any] | None = None,
|
||||
timeline_video_contract: dict[str, Any] | None = None,
|
||||
execution_semantics: dict[str, Any] | None = None,
|
||||
) -> dict[str, Any] | None:
|
||||
if existing_session is not None:
|
||||
raise RunStorageError("Run Storage cannot be combined with an explicit Session")
|
||||
@@ -1054,6 +1058,7 @@ class RunStorageController:
|
||||
driving_audio_vae=driving_audio_vae,
|
||||
reference_video_contract=reference_video_contract,
|
||||
timeline_video_contract=timeline_video_contract,
|
||||
execution_semantics=execution_semantics,
|
||||
)
|
||||
effective_nonce, nonce_decision = self._resolve_effective_nonce(
|
||||
contract, requested_nonce=int(reroll_nonce), resume_safe=safe
|
||||
|
||||
@@ -0,0 +1,378 @@
|
||||
"""Tests-only capture and comparison helpers for Terminal Merge diagnostics.
|
||||
|
||||
This module deliberately has no production imports or runtime hooks. Tests can
|
||||
monkeypatch Core and Continuum boundaries, feed the captured values here, and
|
||||
identify the first pipeline stage that differs without changing generation
|
||||
semantics.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from enum import Enum
|
||||
from hashlib import sha256
|
||||
import json
|
||||
from pathlib import Path
|
||||
from typing import Any, Mapping
|
||||
|
||||
import torch
|
||||
|
||||
|
||||
STAGE_ORDER = (
|
||||
"A_pre_sampler",
|
||||
"B_first_model_call",
|
||||
"C_sampled_physical_av",
|
||||
"D_split_recombine",
|
||||
"E_decode_assembly",
|
||||
)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class TensorFingerprint:
|
||||
shape: tuple[int, ...]
|
||||
dtype: str
|
||||
sha256: str
|
||||
|
||||
def as_dict(self) -> dict[str, Any]:
|
||||
return {
|
||||
"shape": list(self.shape),
|
||||
"dtype": self.dtype,
|
||||
"sha256": self.sha256,
|
||||
}
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class TensorDifference:
|
||||
same_shape: bool
|
||||
same_dtype: bool
|
||||
bit_exact: bool
|
||||
allclose: bool
|
||||
max_abs_diff: float | None
|
||||
mean_abs_diff: float | None
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class StageSnapshot:
|
||||
name: str
|
||||
sha256: str
|
||||
payload: Any
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class StageComparison:
|
||||
name: str
|
||||
equal: bool
|
||||
core_sha256: str | None
|
||||
continuum_sha256: str | None
|
||||
missing_from: str | None = None
|
||||
|
||||
|
||||
def _tensor_raw_bytes(tensor: torch.Tensor) -> bytes:
|
||||
value = tensor.detach().to("cpu").contiguous()
|
||||
if value.numel() == 0:
|
||||
return b""
|
||||
return value.reshape(-1).view(torch.uint8).numpy().tobytes()
|
||||
|
||||
|
||||
def tensor_fingerprint(tensor: torch.Tensor) -> TensorFingerprint:
|
||||
if not torch.is_tensor(tensor):
|
||||
raise TypeError("tensor_fingerprint expects a torch.Tensor")
|
||||
return TensorFingerprint(
|
||||
shape=tuple(int(size) for size in tensor.shape),
|
||||
dtype=str(tensor.dtype),
|
||||
sha256=sha256(_tensor_raw_bytes(tensor)).hexdigest(),
|
||||
)
|
||||
|
||||
|
||||
def tensor_difference(
|
||||
core: torch.Tensor,
|
||||
continuum: torch.Tensor,
|
||||
*,
|
||||
rtol: float = 1.0e-5,
|
||||
atol: float = 1.0e-6,
|
||||
) -> TensorDifference:
|
||||
if not torch.is_tensor(core) or not torch.is_tensor(continuum):
|
||||
raise TypeError("tensor_difference expects two torch.Tensor values")
|
||||
|
||||
same_shape = tuple(core.shape) == tuple(continuum.shape)
|
||||
same_dtype = core.dtype == continuum.dtype
|
||||
bit_exact = (
|
||||
same_shape
|
||||
and same_dtype
|
||||
and tensor_fingerprint(core).sha256 == tensor_fingerprint(continuum).sha256
|
||||
)
|
||||
if not same_shape:
|
||||
return TensorDifference(
|
||||
same_shape=False,
|
||||
same_dtype=same_dtype,
|
||||
bit_exact=False,
|
||||
allclose=False,
|
||||
max_abs_diff=None,
|
||||
mean_abs_diff=None,
|
||||
)
|
||||
|
||||
core_value = core.detach().to("cpu")
|
||||
continuum_value = continuum.detach().to("cpu")
|
||||
if core_value.is_complex() or continuum_value.is_complex():
|
||||
core_numeric = core_value.to(torch.complex128)
|
||||
continuum_numeric = continuum_value.to(torch.complex128)
|
||||
else:
|
||||
core_numeric = core_value.to(torch.float64)
|
||||
continuum_numeric = continuum_value.to(torch.float64)
|
||||
|
||||
if core_numeric.numel() == 0:
|
||||
max_abs_diff = 0.0
|
||||
mean_abs_diff = 0.0
|
||||
else:
|
||||
absolute = (core_numeric - continuum_numeric).abs()
|
||||
max_abs_diff = float(absolute.max().item())
|
||||
mean_abs_diff = float(absolute.mean().item())
|
||||
|
||||
return TensorDifference(
|
||||
same_shape=True,
|
||||
same_dtype=same_dtype,
|
||||
bit_exact=bit_exact,
|
||||
allclose=bool(
|
||||
torch.allclose(
|
||||
core_numeric,
|
||||
continuum_numeric,
|
||||
rtol=rtol,
|
||||
atol=atol,
|
||||
equal_nan=True,
|
||||
)
|
||||
),
|
||||
max_abs_diff=max_abs_diff,
|
||||
mean_abs_diff=mean_abs_diff,
|
||||
)
|
||||
|
||||
|
||||
def packed_layout_snapshot(layout: Any) -> dict[str, Any]:
|
||||
return {
|
||||
"signature": stable_snapshot(getattr(layout, "signature", None)),
|
||||
"segments": stable_snapshot(getattr(layout, "segments", None)),
|
||||
"position_ids": stable_snapshot(getattr(layout, "position_ids", None)),
|
||||
}
|
||||
|
||||
|
||||
def stable_snapshot(value: Any) -> Any:
|
||||
"""Convert diagnostic data to deterministic JSON-compatible state."""
|
||||
|
||||
if torch.is_tensor(value):
|
||||
return {"kind": "tensor", **tensor_fingerprint(value).as_dict()}
|
||||
if isinstance(value, TensorFingerprint):
|
||||
return {"kind": "tensor", **value.as_dict()}
|
||||
if value is None or isinstance(value, (bool, int, str)):
|
||||
return value
|
||||
if isinstance(value, float):
|
||||
return {"kind": "float", "hex": value.hex()}
|
||||
if isinstance(value, bytes):
|
||||
return {
|
||||
"kind": "bytes",
|
||||
"length": len(value),
|
||||
"sha256": sha256(value).hexdigest(),
|
||||
}
|
||||
if isinstance(value, (Path, torch.device, torch.dtype)):
|
||||
return str(value)
|
||||
if isinstance(value, Enum):
|
||||
return {
|
||||
"kind": "enum",
|
||||
"type": f"{type(value).__module__}.{type(value).__qualname__}",
|
||||
"value": stable_snapshot(value.value),
|
||||
}
|
||||
if isinstance(value, slice):
|
||||
return {
|
||||
"kind": "slice",
|
||||
"start": value.start,
|
||||
"stop": value.stop,
|
||||
"step": value.step,
|
||||
}
|
||||
if isinstance(value, Mapping):
|
||||
return {
|
||||
"kind": "mapping",
|
||||
"items": [
|
||||
[str(key), stable_snapshot(item)]
|
||||
for key, item in sorted(value.items(), key=lambda pair: str(pair[0]))
|
||||
],
|
||||
}
|
||||
if isinstance(value, tuple):
|
||||
return {"kind": "tuple", "items": [stable_snapshot(item) for item in value]}
|
||||
if isinstance(value, list):
|
||||
return {"kind": "list", "items": [stable_snapshot(item) for item in value]}
|
||||
if isinstance(value, (set, frozenset)):
|
||||
items = [stable_snapshot(item) for item in value]
|
||||
items.sort(key=lambda item: json.dumps(item, sort_keys=True, separators=(",", ":")))
|
||||
return {"kind": type(value).__name__, "items": items}
|
||||
if callable(value):
|
||||
return {
|
||||
"kind": "callable",
|
||||
"module": getattr(value, "__module__", type(value).__module__),
|
||||
"qualname": getattr(value, "__qualname__", type(value).__qualname__),
|
||||
}
|
||||
if all(hasattr(value, name) for name in ("signature", "segments", "position_ids")):
|
||||
return {"kind": "packed_layout", **packed_layout_snapshot(value)}
|
||||
if hasattr(value, "__dict__"):
|
||||
public_state = {
|
||||
key: item
|
||||
for key, item in vars(value).items()
|
||||
if not key.startswith("_") and not callable(item)
|
||||
}
|
||||
return {
|
||||
"kind": "object",
|
||||
"type": f"{type(value).__module__}.{type(value).__qualname__}",
|
||||
"state": stable_snapshot(public_state),
|
||||
}
|
||||
raise TypeError(f"Unsupported diagnostic value: {type(value)!r}")
|
||||
|
||||
|
||||
def snapshot_sha256(payload: Any) -> str:
|
||||
normalized = stable_snapshot(payload)
|
||||
encoded = json.dumps(
|
||||
normalized,
|
||||
ensure_ascii=True,
|
||||
sort_keys=True,
|
||||
separators=(",", ":"),
|
||||
).encode("utf-8")
|
||||
return sha256(encoded).hexdigest()
|
||||
|
||||
|
||||
class DiagnosticCapture:
|
||||
def __init__(self) -> None:
|
||||
self._stages: dict[str, StageSnapshot] = {}
|
||||
|
||||
@property
|
||||
def stages(self) -> Mapping[str, StageSnapshot]:
|
||||
return self._stages
|
||||
|
||||
def capture(self, stage: str, **payload: Any) -> StageSnapshot:
|
||||
if stage not in STAGE_ORDER:
|
||||
raise ValueError(f"Unknown diagnostic stage: {stage}")
|
||||
if stage in self._stages:
|
||||
raise ValueError(f"Diagnostic stage already captured: {stage}")
|
||||
normalized = stable_snapshot(payload)
|
||||
snapshot = StageSnapshot(
|
||||
name=stage,
|
||||
sha256=snapshot_sha256(normalized),
|
||||
payload=normalized,
|
||||
)
|
||||
self._stages[stage] = snapshot
|
||||
return snapshot
|
||||
|
||||
def capture_pre_sampler(
|
||||
self,
|
||||
*,
|
||||
physical_frames: int,
|
||||
seed: int,
|
||||
conditioning: Any,
|
||||
latent: Any,
|
||||
noise: torch.Tensor,
|
||||
keyframes: Any = None,
|
||||
qwen_images: Any = None,
|
||||
minimax_refs: Any = None,
|
||||
) -> StageSnapshot:
|
||||
return self.capture(
|
||||
"A_pre_sampler",
|
||||
physical_frames=physical_frames,
|
||||
seed=seed,
|
||||
conditioning=conditioning,
|
||||
latent=latent,
|
||||
noise=noise,
|
||||
keyframes=keyframes,
|
||||
qwen_images=qwen_images,
|
||||
minimax_refs=minimax_refs,
|
||||
)
|
||||
|
||||
def capture_first_model_call(
|
||||
self,
|
||||
*,
|
||||
minimax_payload: Any,
|
||||
transformer_options: Any,
|
||||
) -> StageSnapshot:
|
||||
return self.capture(
|
||||
"B_first_model_call",
|
||||
minimax_payload=minimax_payload,
|
||||
transformer_options=transformer_options,
|
||||
)
|
||||
|
||||
def capture_sampled_physical_av(
|
||||
self,
|
||||
*,
|
||||
video: torch.Tensor,
|
||||
audio: torch.Tensor,
|
||||
) -> StageSnapshot:
|
||||
return self.capture(
|
||||
"C_sampled_physical_av",
|
||||
video=video,
|
||||
audio=audio,
|
||||
)
|
||||
|
||||
def capture_split_recombine(
|
||||
self,
|
||||
*,
|
||||
physical_video: torch.Tensor,
|
||||
physical_audio: torch.Tensor,
|
||||
recombined_video: torch.Tensor,
|
||||
recombined_audio: torch.Tensor,
|
||||
) -> StageSnapshot:
|
||||
return self.capture(
|
||||
"D_split_recombine",
|
||||
physical_video=physical_video,
|
||||
physical_audio=physical_audio,
|
||||
recombined_video=recombined_video,
|
||||
recombined_audio=recombined_audio,
|
||||
video_difference=tensor_difference(physical_video, recombined_video).__dict__,
|
||||
audio_difference=tensor_difference(physical_audio, recombined_audio).__dict__,
|
||||
)
|
||||
|
||||
def capture_decode_assembly(
|
||||
self,
|
||||
*,
|
||||
physical_decoded: torch.Tensor,
|
||||
assembled_decoded: torch.Tensor,
|
||||
) -> StageSnapshot:
|
||||
return self.capture(
|
||||
"E_decode_assembly",
|
||||
physical_decoded=physical_decoded,
|
||||
assembled_decoded=assembled_decoded,
|
||||
difference=tensor_difference(physical_decoded, assembled_decoded).__dict__,
|
||||
)
|
||||
|
||||
|
||||
def compare_captures(
|
||||
core: DiagnosticCapture,
|
||||
continuum: DiagnosticCapture,
|
||||
) -> tuple[StageComparison, ...]:
|
||||
comparisons = []
|
||||
for stage in STAGE_ORDER:
|
||||
core_stage = core.stages.get(stage)
|
||||
continuum_stage = continuum.stages.get(stage)
|
||||
missing = None
|
||||
if core_stage is None and continuum_stage is None:
|
||||
missing = "both"
|
||||
elif core_stage is None:
|
||||
missing = "core"
|
||||
elif continuum_stage is None:
|
||||
missing = "continuum"
|
||||
comparisons.append(
|
||||
StageComparison(
|
||||
name=stage,
|
||||
equal=(
|
||||
core_stage is not None
|
||||
and continuum_stage is not None
|
||||
and core_stage.sha256 == continuum_stage.sha256
|
||||
),
|
||||
core_sha256=None if core_stage is None else core_stage.sha256,
|
||||
continuum_sha256=None if continuum_stage is None else continuum_stage.sha256,
|
||||
missing_from=missing,
|
||||
)
|
||||
)
|
||||
return tuple(comparisons)
|
||||
|
||||
|
||||
def first_divergent_stage(
|
||||
core: DiagnosticCapture,
|
||||
continuum: DiagnosticCapture,
|
||||
) -> str | None:
|
||||
for comparison in compare_captures(core, continuum):
|
||||
if not comparison.equal:
|
||||
return comparison.name
|
||||
return None
|
||||
@@ -0,0 +1,81 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib.util
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import torch
|
||||
|
||||
|
||||
MODULE_PATH = Path(__file__).parents[1] / "tools" / "gpu_diagnostic_node" / "capture.py"
|
||||
SPEC = importlib.util.spec_from_file_location("h3_gpu_diagnostic_capture", MODULE_PATH)
|
||||
capture = importlib.util.module_from_spec(SPEC)
|
||||
assert SPEC.loader is not None
|
||||
sys.modules[SPEC.name] = capture
|
||||
SPEC.loader.exec_module(capture)
|
||||
|
||||
|
||||
def test_tensor_fingerprint_is_deterministic():
|
||||
value = torch.arange(12, dtype=torch.float32).reshape(1, 3, 4)
|
||||
assert capture.tensor_fingerprint(value) == capture.tensor_fingerprint(value.clone())
|
||||
|
||||
|
||||
def test_tensor_comparison_records_numerical_difference():
|
||||
left = torch.tensor([1.0, 2.0], dtype=torch.float32)
|
||||
right = torch.tensor([1.0, 2.000001], dtype=torch.float32)
|
||||
result = capture.compare_tensors(left, right)
|
||||
assert not result.bit_exact
|
||||
assert result.allclose
|
||||
assert result.matches
|
||||
assert result.max_abs_diff > 0.0
|
||||
|
||||
|
||||
def test_terminal_video_recombine_is_bit_exact():
|
||||
physical = torch.arange(72, dtype=torch.float32).reshape(1, 1, 72, 1, 1)
|
||||
first = physical[:, :, 0:37]
|
||||
second = physical[:, :, 30:72]
|
||||
rebuilt = capture.recombine_terminal_streams(
|
||||
first,
|
||||
second,
|
||||
first_range=(0, 37),
|
||||
second_range=(30, 72),
|
||||
dim=2,
|
||||
)
|
||||
assert torch.equal(rebuilt, physical)
|
||||
|
||||
|
||||
def test_terminal_audio_recombine_is_bit_exact():
|
||||
physical = torch.arange(405, dtype=torch.float32).reshape(1, 1, 405)
|
||||
first = physical[..., 0:207]
|
||||
second = physical[..., 170:405]
|
||||
rebuilt = capture.recombine_terminal_streams(
|
||||
first,
|
||||
second,
|
||||
first_range=(0, 207),
|
||||
second_range=(170, 405),
|
||||
dim=-1,
|
||||
)
|
||||
assert torch.equal(rebuilt, physical)
|
||||
|
||||
|
||||
def test_stage_comparison_reports_first_field():
|
||||
left = capture.DiagnosticCapture("core")
|
||||
right = capture.DiagnosticCapture("continuum")
|
||||
left.add_value("A_pre_sampler", "seed", 42)
|
||||
right.add_value("A_pre_sampler", "seed", 43)
|
||||
left.add_tensor("A_pre_sampler", "sigmas", torch.ones(2))
|
||||
right.add_tensor("A_pre_sampler", "sigmas", torch.ones(2))
|
||||
result = capture.compare_stage(left, right, "A_pre_sampler")
|
||||
assert result["status"] == "DIFFER"
|
||||
assert result["first_field"] == "seed"
|
||||
|
||||
|
||||
def test_summary_uses_stage_order_and_first_divergence(tmp_path):
|
||||
comparison = {
|
||||
"A_pre_sampler": {"status": "MATCH", "first_field": None},
|
||||
"B_first_model_call": {"status": "DIFFER", "first_field": "position_ids"},
|
||||
}
|
||||
text = capture.format_comparison(comparison, tmp_path)
|
||||
assert "A_pre_sampler: MATCH" in text
|
||||
assert "B_first_model_call: DIFFER" in text
|
||||
assert "First divergence: B_first_model_call.position_ids" in text
|
||||
@@ -0,0 +1,59 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
from ComfyUI_H3_Continuum_Join.conditioning import (
|
||||
conditioning_display_label,
|
||||
conditioning_mode_from_presence,
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("has_first", "has_last", "has_reference", "expected"),
|
||||
[
|
||||
(False, False, False, "T2VA + Continuation"),
|
||||
(True, False, False, "I2VA + Continuation"),
|
||||
(False, True, False, "Last Frame + Continuation"),
|
||||
(True, True, False, "FL2VA + Continuation"),
|
||||
(False, False, True, "Reference + Continuation"),
|
||||
(True, False, True, "Hybrid I2VA + Reference + Continuation"),
|
||||
(False, True, True, "Hybrid L2VA + Reference + Continuation"),
|
||||
(True, True, True, "Hybrid FL2VA + Reference + Continuation"),
|
||||
],
|
||||
)
|
||||
def test_conditioning_display_label_matrix(
|
||||
has_first: bool,
|
||||
has_last: bool,
|
||||
has_reference: bool,
|
||||
expected: str,
|
||||
) -> None:
|
||||
assert (
|
||||
conditioning_display_label(
|
||||
has_first=has_first,
|
||||
has_last=has_last,
|
||||
has_reference=has_reference,
|
||||
)
|
||||
== expected
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("has_first", "has_last"),
|
||||
[
|
||||
(True, False),
|
||||
(False, True),
|
||||
(True, True),
|
||||
],
|
||||
)
|
||||
def test_hybrid_display_does_not_change_internal_reference_mode(
|
||||
has_first: bool,
|
||||
has_last: bool,
|
||||
) -> None:
|
||||
assert (
|
||||
conditioning_mode_from_presence(
|
||||
has_first=has_first,
|
||||
has_last=has_last,
|
||||
has_reference=True,
|
||||
)
|
||||
== "reference"
|
||||
)
|
||||
@@ -0,0 +1,183 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from types import SimpleNamespace
|
||||
|
||||
import torch
|
||||
|
||||
from ComfyUI_H3_Continuum_Join.v2.sequence import (
|
||||
_split_terminal_merged_latents,
|
||||
_terminal_pair_contract,
|
||||
)
|
||||
from ComfyUI_H3_Continuum_Join.tests.terminal_merge_diagnostics import (
|
||||
DiagnosticCapture,
|
||||
first_divergent_stage,
|
||||
tensor_difference,
|
||||
tensor_fingerprint,
|
||||
)
|
||||
|
||||
|
||||
def _capture_all_stages(*, sampled_offset: float = 0.0) -> DiagnosticCapture:
|
||||
capture = DiagnosticCapture()
|
||||
capture.capture_pre_sampler(
|
||||
physical_frames=243,
|
||||
seed=42,
|
||||
conditioning=[[torch.tensor([1.0]), {"minimax_frame_count": 243}]],
|
||||
latent={"video": torch.zeros(1, 24, 72, 2, 2)},
|
||||
noise=torch.arange(8, dtype=torch.float32),
|
||||
keyframes=(0, 242),
|
||||
qwen_images=("first", "last"),
|
||||
minimax_refs=None,
|
||||
)
|
||||
layout = SimpleNamespace(
|
||||
signature=(1, 2, 3),
|
||||
segments=(("target_video", 0, 72),),
|
||||
position_ids=torch.arange(12, dtype=torch.float32).reshape(6, 2),
|
||||
)
|
||||
capture.capture_first_model_call(
|
||||
minimax_payload={
|
||||
"layout": layout,
|
||||
"cond_video_latents": torch.ones(1, 24, 2, 2, 2),
|
||||
"target_video_shape": (1, 24, 72, 2, 2),
|
||||
"target_audio_shape": (1, 32, 2, 405),
|
||||
},
|
||||
transformer_options={"patches": {}},
|
||||
)
|
||||
video = torch.full((1, 24, 72, 2, 2), sampled_offset)
|
||||
audio = torch.zeros(1, 32, 2, 405)
|
||||
capture.capture_sampled_physical_av(video=video, audio=audio)
|
||||
capture.capture_split_recombine(
|
||||
physical_video=video,
|
||||
physical_audio=audio,
|
||||
recombined_video=video.clone(),
|
||||
recombined_audio=audio.clone(),
|
||||
)
|
||||
decoded = torch.zeros(243, 2, 2, 3)
|
||||
capture.capture_decode_assembly(
|
||||
physical_decoded=decoded,
|
||||
assembled_decoded=decoded.clone(),
|
||||
)
|
||||
return capture
|
||||
|
||||
|
||||
def test_tensor_fingerprint_is_deterministic_and_dtype_sensitive():
|
||||
value = torch.arange(12, dtype=torch.float32).reshape(3, 4)
|
||||
first = tensor_fingerprint(value)
|
||||
second = tensor_fingerprint(value.clone())
|
||||
|
||||
assert first == second
|
||||
assert first.shape == (3, 4)
|
||||
assert first.dtype == "torch.float32"
|
||||
assert first.sha256 != tensor_fingerprint(value.to(torch.float64)).sha256
|
||||
|
||||
|
||||
def test_tensor_difference_keeps_hash_mismatch_distinct_from_allclose():
|
||||
core = torch.tensor([1.0], dtype=torch.float32)
|
||||
continuum = torch.tensor([1.0 + 1.0e-7], dtype=torch.float32)
|
||||
|
||||
difference = tensor_difference(core, continuum)
|
||||
|
||||
assert difference.bit_exact is False
|
||||
assert difference.allclose is True
|
||||
assert difference.max_abs_diff is not None
|
||||
assert difference.max_abs_diff > 0.0
|
||||
assert difference.mean_abs_diff == difference.max_abs_diff
|
||||
|
||||
|
||||
def test_pre_sampler_capture_includes_initial_noise():
|
||||
core = _capture_all_stages()
|
||||
continuum = _capture_all_stages()
|
||||
assert first_divergent_stage(core, continuum) is None
|
||||
|
||||
changed = DiagnosticCapture()
|
||||
changed.capture_pre_sampler(
|
||||
physical_frames=243,
|
||||
seed=42,
|
||||
conditioning=[[torch.tensor([1.0]), {"minimax_frame_count": 243}]],
|
||||
latent={"video": torch.zeros(1, 24, 72, 2, 2)},
|
||||
noise=torch.arange(8, dtype=torch.float32) + 1.0,
|
||||
keyframes=(0, 242),
|
||||
qwen_images=("first", "last"),
|
||||
minimax_refs=None,
|
||||
)
|
||||
|
||||
assert first_divergent_stage(core, changed) == "A_pre_sampler"
|
||||
|
||||
|
||||
def test_first_model_call_capture_detects_packed_layout_difference():
|
||||
core = DiagnosticCapture()
|
||||
continuum = DiagnosticCapture()
|
||||
shared = dict(
|
||||
physical_frames=243,
|
||||
seed=42,
|
||||
conditioning=[torch.zeros(1)],
|
||||
latent=torch.zeros(1),
|
||||
noise=torch.zeros(1),
|
||||
)
|
||||
core.capture_pre_sampler(**shared)
|
||||
continuum.capture_pre_sampler(**shared)
|
||||
|
||||
core_layout = SimpleNamespace(
|
||||
signature=(1, 2),
|
||||
segments=(("target_video", 0, 72),),
|
||||
position_ids=torch.zeros(4, 2),
|
||||
)
|
||||
continuum_layout = SimpleNamespace(
|
||||
signature=(1, 2),
|
||||
segments=(("target_video", 0, 72),),
|
||||
position_ids=torch.ones(4, 2),
|
||||
)
|
||||
core.capture_first_model_call(
|
||||
minimax_payload={"layout": core_layout},
|
||||
transformer_options={},
|
||||
)
|
||||
continuum.capture_first_model_call(
|
||||
minimax_payload={"layout": continuum_layout},
|
||||
transformer_options={},
|
||||
)
|
||||
|
||||
assert first_divergent_stage(core, continuum) == "B_first_model_call"
|
||||
|
||||
|
||||
def test_first_divergent_stage_uses_pipeline_order():
|
||||
core = _capture_all_stages(sampled_offset=0.0)
|
||||
continuum = _capture_all_stages(sampled_offset=1.0)
|
||||
|
||||
assert first_divergent_stage(core, continuum) == "C_sampled_physical_av"
|
||||
|
||||
|
||||
def test_terminal_split_recombine_is_bit_exact():
|
||||
contract = _terminal_pair_contract(initial_pair=True, chunk_seconds=5.0)
|
||||
video = torch.arange(
|
||||
1 * 24 * contract["video_slices"][-1][1] * 2 * 2,
|
||||
dtype=torch.float32,
|
||||
).reshape(1, 24, contract["video_slices"][-1][1], 2, 2)
|
||||
audio = torch.arange(
|
||||
1 * 32 * 2 * contract["audio_slices"][-1][1],
|
||||
dtype=torch.float32,
|
||||
).reshape(1, 32, 2, contract["audio_slices"][-1][1])
|
||||
|
||||
(video_1, audio_1), (video_2, audio_2) = _split_terminal_merged_latents(
|
||||
video,
|
||||
audio,
|
||||
contract,
|
||||
)
|
||||
video_overlap = contract["video_slices"][0][1] - contract["video_slices"][1][0]
|
||||
audio_overlap = contract["audio_slices"][0][1] - contract["audio_slices"][1][0]
|
||||
recombined_video = torch.cat((video_1, video_2[:, :, video_overlap:]), dim=2)
|
||||
recombined_audio = torch.cat((audio_1, audio_2[..., audio_overlap:]), dim=-1)
|
||||
|
||||
assert contract["physical_frames"] == 243
|
||||
assert contract["logical_frames"] == (124, 141)
|
||||
assert torch.equal(recombined_video, video)
|
||||
assert torch.equal(recombined_audio, audio)
|
||||
assert tensor_difference(video, recombined_video).bit_exact is True
|
||||
assert tensor_difference(audio, recombined_audio).bit_exact is True
|
||||
|
||||
capture = DiagnosticCapture()
|
||||
stage = capture.capture_split_recombine(
|
||||
physical_video=video,
|
||||
physical_audio=audio,
|
||||
recombined_video=recombined_video,
|
||||
recombined_audio=recombined_audio,
|
||||
)
|
||||
assert stage.name == "D_split_recombine"
|
||||
@@ -0,0 +1,165 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
import torch
|
||||
|
||||
from ComfyUI_H3_Continuum_Join.v2.sequence import (
|
||||
_atomic_terminal_prefix,
|
||||
_split_terminal_merged_latents,
|
||||
_terminal_flf_merge_enabled,
|
||||
_terminal_pair_prompt,
|
||||
_terminal_physical_seed_plan,
|
||||
_terminal_pair_contract,
|
||||
_terminal_sampling_plan,
|
||||
)
|
||||
from ComfyUI_H3_Continuum_Join.v2.seeds import derive_chunk_seed
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("initial_pair", "physical_frames", "logical_frames", "video_slices", "audio_slices"),
|
||||
[
|
||||
(True, 243, (124, 141), ((0, 37), (30, 72)), ((0, 207), (170, 405))),
|
||||
(False, 260, (141, 141), ((0, 42), (35, 77)), ((0, 235), (198, 433))),
|
||||
],
|
||||
)
|
||||
def test_terminal_pair_contract_uses_native_temporal_grid(
|
||||
initial_pair,
|
||||
physical_frames,
|
||||
logical_frames,
|
||||
video_slices,
|
||||
audio_slices,
|
||||
):
|
||||
contract = _terminal_pair_contract(initial_pair=initial_pair, chunk_seconds=5.0)
|
||||
assert contract["physical_frames"] == physical_frames
|
||||
assert contract["logical_frames"] == logical_frames
|
||||
assert contract["video_slices"] == video_slices
|
||||
assert contract["audio_slices"] == audio_slices
|
||||
|
||||
|
||||
@pytest.mark.parametrize("initial_pair", [True, False])
|
||||
def test_terminal_pair_split_preserves_expected_overlap(initial_pair):
|
||||
contract = _terminal_pair_contract(initial_pair=initial_pair, chunk_seconds=5.0)
|
||||
video_t = contract["video_slices"][-1][1]
|
||||
audio_t = contract["audio_slices"][-1][1]
|
||||
video = torch.arange(video_t, dtype=torch.float32).reshape(1, 1, video_t, 1, 1)
|
||||
audio = torch.arange(audio_t, dtype=torch.float32).reshape(1, 1, 1, audio_t)
|
||||
first, second = _split_terminal_merged_latents(video, audio, contract)
|
||||
assert first[0].shape[2] == contract["video_slices"][0][1]
|
||||
assert first[1].shape[-1] == contract["audio_slices"][0][1]
|
||||
assert second[0][0, 0, 0, 0, 0].item() == contract["video_slices"][1][0]
|
||||
assert second[1][0, 0, 0, 0].item() == contract["audio_slices"][1][0]
|
||||
|
||||
|
||||
def test_two_by_five_uses_one_physical_sample():
|
||||
normal, terminal = _terminal_sampling_plan(chunks=2, completed=0, merge_enabled=True)
|
||||
assert normal == ()
|
||||
assert terminal is True
|
||||
assert len(normal) + int(terminal) == 1
|
||||
|
||||
|
||||
def test_three_by_five_uses_one_normal_and_one_terminal_sample():
|
||||
normal, terminal = _terminal_sampling_plan(chunks=3, completed=0, merge_enabled=True)
|
||||
assert normal == (0,)
|
||||
assert terminal is True
|
||||
assert len(normal) + int(terminal) == 2
|
||||
|
||||
|
||||
def test_terminal_merge_accepts_distinct_prompts_but_not_timeline_video():
|
||||
common = dict(multi_chunk_flf=True, chunks=2, chunk_seconds=5.0)
|
||||
assert _terminal_flf_merge_enabled(**common, prompt_hashes=["same", "same"], timeline_video_source=None)
|
||||
assert _terminal_flf_merge_enabled(**common, prompt_hashes=["a", "b"], timeline_video_source=None)
|
||||
assert not _terminal_flf_merge_enabled(**common, prompt_hashes=["same", "same"], timeline_video_source=object())
|
||||
|
||||
|
||||
def test_terminal_pair_prompt_preserves_shared_prompt_exactly():
|
||||
prompt, policy = _terminal_pair_prompt(["same", "same"], pair_start=0, chunk_seconds=5.0)
|
||||
assert prompt == "same"
|
||||
assert policy == "shared_prompt_v1"
|
||||
|
||||
|
||||
def test_terminal_pair_prompt_rebases_distinct_sections_to_local_timeline():
|
||||
prompt, policy = _terminal_pair_prompt(["first action", "second action"], pair_start=0, chunk_seconds=5.0)
|
||||
assert prompt == "[0-5s]\nfirst action\n\n[5-10s]\nsecond action"
|
||||
assert policy == "paired_timeline_v1"
|
||||
|
||||
|
||||
def test_partial_terminal_pair_is_discarded_for_atomic_resume():
|
||||
preserved = [{"chunk": 1}, {"chunk": 2}]
|
||||
result, reset = _atomic_terminal_prefix(preserved, chunks=3, reroll_from_chunk=0)
|
||||
assert result == preserved[:1]
|
||||
assert reset is True
|
||||
|
||||
|
||||
def test_rerolling_final_chunk_regenerates_both_terminal_chunks():
|
||||
preserved = [{"chunk": 1}, {"chunk": 2}, {"chunk": 3}]
|
||||
result, reset = _atomic_terminal_prefix(preserved, chunks=3, reroll_from_chunk=3)
|
||||
assert result == preserved[:1]
|
||||
assert reset is True
|
||||
|
||||
|
||||
def test_complete_terminal_pair_remains_reusable_without_reroll():
|
||||
preserved = [{"chunk": 1}, {"chunk": 2}, {"chunk": 3}]
|
||||
result, reset = _atomic_terminal_prefix(preserved, chunks=3, reroll_from_chunk=0)
|
||||
assert result == preserved
|
||||
assert reset is False
|
||||
|
||||
|
||||
@pytest.mark.parametrize("base_seed", [42, 123456789])
|
||||
def test_initial_terminal_pair_uses_base_seed_directly(base_seed):
|
||||
plan = _terminal_physical_seed_plan(
|
||||
base_seed=base_seed,
|
||||
physical_window_start_index=0,
|
||||
reroll_nonce=0,
|
||||
)
|
||||
assert plan["physical_seed"] == base_seed
|
||||
assert plan["logical_entry_seeds"] == (base_seed, base_seed)
|
||||
|
||||
|
||||
def test_later_terminal_pair_derives_once_from_pair_start():
|
||||
plan = _terminal_physical_seed_plan(
|
||||
base_seed=42,
|
||||
physical_window_start_index=2,
|
||||
reroll_nonce=0,
|
||||
)
|
||||
assert plan["physical_seed"] == derive_chunk_seed(42, 2, 0)
|
||||
assert plan["physical_seed"] == 1808285636802909398
|
||||
assert plan["logical_entry_seeds"] == (plan["physical_seed"],) * 2
|
||||
|
||||
|
||||
def test_terminal_pair_reroll_derives_once_and_shares_new_seed():
|
||||
original = _terminal_physical_seed_plan(
|
||||
base_seed=42,
|
||||
physical_window_start_index=2,
|
||||
reroll_nonce=0,
|
||||
)
|
||||
rerolled = _terminal_physical_seed_plan(
|
||||
base_seed=42,
|
||||
physical_window_start_index=2,
|
||||
reroll_nonce=1,
|
||||
)
|
||||
assert rerolled["physical_seed"] == derive_chunk_seed(42, 2, 1)
|
||||
assert rerolled["physical_seed"] == 4306926667627841302
|
||||
assert rerolled["physical_seed"] != original["physical_seed"]
|
||||
assert rerolled["logical_entry_seeds"] == (rerolled["physical_seed"],) * 2
|
||||
|
||||
|
||||
def test_non_terminal_seed_derivation_golden_values_are_unchanged():
|
||||
assert derive_chunk_seed(42, 0, 0) == 15348518480342951850
|
||||
assert derive_chunk_seed(42, 1, 0) == 13021120982267964226
|
||||
|
||||
|
||||
def test_last_frame_absent_does_not_enable_terminal_merge():
|
||||
assert not _terminal_flf_merge_enabled(
|
||||
multi_chunk_flf=False,
|
||||
chunks=2,
|
||||
chunk_seconds=5.0,
|
||||
prompt_hashes=["same", "same"],
|
||||
timeline_video_source=None,
|
||||
)
|
||||
|
||||
|
||||
def test_one_chunk_keeps_normal_sampling_and_seed_behavior():
|
||||
normal, terminal = _terminal_sampling_plan(chunks=1, completed=0, merge_enabled=False)
|
||||
assert normal == (0,)
|
||||
assert terminal is False
|
||||
assert derive_chunk_seed(42, normal[0], 0) == 15348518480342951850
|
||||
@@ -283,6 +283,7 @@ def _sampling_contract(
|
||||
last_frame_hash="", reference_contract=None,
|
||||
upstream_graph_contract=None, upstream_graph_safe=None,
|
||||
upstream_graph_reasons=None,
|
||||
execution_semantics=None,
|
||||
):
|
||||
return build_sampling_contract(
|
||||
model=model or _Model(),
|
||||
@@ -312,9 +313,69 @@ def _sampling_contract(
|
||||
upstream_graph_contract=upstream_graph_contract,
|
||||
upstream_graph_safe=upstream_graph_safe,
|
||||
upstream_graph_reasons=upstream_graph_reasons,
|
||||
execution_semantics=execution_semantics,
|
||||
)
|
||||
|
||||
|
||||
def test_terminal_seed_v2_revision_is_scoped_and_distinct():
|
||||
old_contract, old_safe, old_reasons = _sampling_contract(
|
||||
conditioning_mode="fl2va",
|
||||
first_frame_hash="a" * 64,
|
||||
last_frame_hash="b" * 64,
|
||||
)
|
||||
new_contract, new_safe, new_reasons = _sampling_contract(
|
||||
conditioning_mode="fl2va",
|
||||
first_frame_hash="a" * 64,
|
||||
last_frame_hash="b" * 64,
|
||||
execution_semantics={
|
||||
"flf_execution": "terminal_merged_10s_seed_v2",
|
||||
"terminal_seed_policy": "physical_window_seed_v2",
|
||||
},
|
||||
)
|
||||
assert old_safe, old_reasons
|
||||
assert new_safe, new_reasons
|
||||
assert "execution_semantics" not in old_contract["global"]
|
||||
assert new_contract["global"]["execution_semantics"] == {
|
||||
"flf_execution": "terminal_merged_10s_seed_v2",
|
||||
"terminal_seed_policy": "physical_window_seed_v2",
|
||||
}
|
||||
assert revision_identity(old_contract) != revision_identity(new_contract)
|
||||
|
||||
|
||||
def test_terminal_seed_v2_revision_identity_is_deterministic():
|
||||
semantics = {
|
||||
"flf_execution": "terminal_merged_10s_seed_v2",
|
||||
"terminal_seed_policy": "physical_window_seed_v2",
|
||||
}
|
||||
model = _Model()
|
||||
clip = _Clip()
|
||||
video_vae = _VideoVAE()
|
||||
sampler = _Sampler()
|
||||
first, first_safe, first_reasons = _sampling_contract(
|
||||
model=model,
|
||||
clip=clip,
|
||||
video_vae=video_vae,
|
||||
sampler=sampler,
|
||||
conditioning_mode="fl2va",
|
||||
first_frame_hash="a" * 64,
|
||||
last_frame_hash="b" * 64,
|
||||
execution_semantics=semantics,
|
||||
)
|
||||
second, second_safe, second_reasons = _sampling_contract(
|
||||
model=model,
|
||||
clip=clip,
|
||||
video_vae=video_vae,
|
||||
sampler=sampler,
|
||||
conditioning_mode="fl2va",
|
||||
first_frame_hash="a" * 64,
|
||||
last_frame_hash="b" * 64,
|
||||
execution_semantics=semantics,
|
||||
)
|
||||
assert first_safe, first_reasons
|
||||
assert second_safe, second_reasons
|
||||
assert revision_identity(first) == revision_identity(second)
|
||||
|
||||
|
||||
def test_v322_golden_reference_contract_preserves_revision_identity():
|
||||
model = _Model()
|
||||
clip = _Clip()
|
||||
|
||||
@@ -0,0 +1,149 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
import torch
|
||||
|
||||
from ComfyUI_H3_Continuum_Join.v3 import plan as plan_module
|
||||
|
||||
|
||||
def _logical_plan():
|
||||
return {
|
||||
"chunks": [
|
||||
{
|
||||
"sequence_index": 0,
|
||||
"chunk_index": 1,
|
||||
"total_frames": 124,
|
||||
"trim_frames": 0,
|
||||
"net_frames": 124,
|
||||
"context_frames": 0,
|
||||
"expected_video_latent_t": 37,
|
||||
"expected_audio_latent_t": 207,
|
||||
"frame_start": 0,
|
||||
"frame_stop": 124,
|
||||
},
|
||||
{
|
||||
"sequence_index": 1,
|
||||
"chunk_index": 2,
|
||||
"total_frames": 141,
|
||||
"trim_frames": 22,
|
||||
"net_frames": 119,
|
||||
"context_frames": 22,
|
||||
"expected_video_latent_t": 42,
|
||||
"expected_audio_latent_t": 235,
|
||||
"frame_start": 124,
|
||||
"frame_stop": 243,
|
||||
},
|
||||
],
|
||||
"target_frames": 240,
|
||||
}
|
||||
|
||||
|
||||
def _terminal_entries():
|
||||
physical_video = torch.arange(72, dtype=torch.float32).reshape(1, 1, 72, 1, 1)
|
||||
physical_audio = torch.arange(405, dtype=torch.float32).reshape(1, 1, 405)
|
||||
entries = [
|
||||
{
|
||||
"video": physical_video[:, :, 0:37].clone(),
|
||||
"audio": physical_audio[..., 0:207].clone(),
|
||||
},
|
||||
{
|
||||
"video": physical_video[:, :, 30:72].clone(),
|
||||
"audio": physical_audio[..., 170:405].clone(),
|
||||
},
|
||||
]
|
||||
return entries, physical_video, physical_audio
|
||||
|
||||
|
||||
def test_terminal_logical_entries_recombine_bit_exact(monkeypatch):
|
||||
entries, physical_video, physical_audio = _terminal_entries()
|
||||
monkeypatch.setattr(
|
||||
plan_module,
|
||||
"make_assembly_plan",
|
||||
lambda *args, **kwargs: _logical_plan(),
|
||||
)
|
||||
monkeypatch.setattr(plan_module, "validate_assembly_plan", lambda plan: plan)
|
||||
|
||||
decode_entries, assembly_plan = plan_module.prepare_physical_decode_entries(
|
||||
entries,
|
||||
chunk_seconds=5.0,
|
||||
preserve_final_frame=True,
|
||||
terminal_merged=True,
|
||||
)
|
||||
|
||||
assert len(decode_entries) == 1
|
||||
assert torch.equal(decode_entries[0]["video"], physical_video)
|
||||
assert torch.equal(decode_entries[0]["audio"], physical_audio)
|
||||
assert assembly_plan["logical_chunk_count"] == 2
|
||||
assert assembly_plan["physical_decode_group_count"] == 1
|
||||
assert assembly_plan["decode_groups"][0]["logical_chunk_indices"] == [1, 2]
|
||||
assert assembly_plan["decode_groups"][0]["total_frames"] == 243
|
||||
assert assembly_plan["decode_groups"][0]["trim_frames"] == 0
|
||||
assert assembly_plan["decode_groups"][0]["net_frames"] == 243
|
||||
|
||||
|
||||
def test_terminal_recombine_rejects_changed_overlap(monkeypatch):
|
||||
entries, _, _ = _terminal_entries()
|
||||
entries[1]["video"][:, :, 0].add_(1.0)
|
||||
monkeypatch.setattr(
|
||||
plan_module,
|
||||
"make_assembly_plan",
|
||||
lambda *args, **kwargs: _logical_plan(),
|
||||
)
|
||||
|
||||
with pytest.raises(ValueError, match="overlap differs"):
|
||||
plan_module.prepare_physical_decode_entries(
|
||||
entries,
|
||||
chunk_seconds=5.0,
|
||||
preserve_final_frame=True,
|
||||
terminal_merged=True,
|
||||
)
|
||||
|
||||
|
||||
def test_three_logical_chunks_use_two_physical_decode_groups(monkeypatch):
|
||||
from ComfyUI_H3_Continuum_Join.v2.sequence import (
|
||||
_split_terminal_merged_latents,
|
||||
_terminal_pair_contract,
|
||||
)
|
||||
|
||||
contract = _terminal_pair_contract(initial_pair=False, chunk_seconds=5.0)
|
||||
video_t = contract["video_slices"][-1][1]
|
||||
audio_t = contract["audio_slices"][-1][1]
|
||||
physical_video = torch.arange(video_t, dtype=torch.float32).reshape(1, 1, video_t, 1, 1)
|
||||
physical_audio = torch.arange(audio_t, dtype=torch.float32).reshape(1, 1, audio_t)
|
||||
terminal_parts = _split_terminal_merged_latents(physical_video, physical_audio, contract)
|
||||
first_entry = {
|
||||
"video": torch.zeros((1, 1, 37, 1, 1), dtype=torch.float32),
|
||||
"audio": torch.zeros((1, 1, 207), dtype=torch.float32),
|
||||
}
|
||||
logical_plan = {
|
||||
"chunks": [
|
||||
{"sequence_index": 0, "chunk_index": 1, "total_frames": 124, "trim_frames": 0, "net_frames": 124, "context_frames": 0, "expected_video_latent_t": 37, "expected_audio_latent_t": 207, "frame_start": 0, "frame_stop": 124},
|
||||
{"sequence_index": 1, "chunk_index": 2, "total_frames": 141, "trim_frames": 22, "net_frames": 119, "context_frames": 22, "expected_video_latent_t": 42, "expected_audio_latent_t": 235, "frame_start": 124, "frame_stop": 243},
|
||||
{"sequence_index": 2, "chunk_index": 3, "total_frames": 141, "trim_frames": 22, "net_frames": 119, "context_frames": 22, "expected_video_latent_t": 42, "expected_audio_latent_t": 235, "frame_start": 243, "frame_stop": 362},
|
||||
],
|
||||
"target_frames": 360,
|
||||
}
|
||||
entries = [
|
||||
first_entry,
|
||||
{"video": terminal_parts[0][0], "audio": terminal_parts[0][1]},
|
||||
{"video": terminal_parts[1][0], "audio": terminal_parts[1][1]},
|
||||
]
|
||||
monkeypatch.setattr(plan_module, "make_assembly_plan", lambda *args, **kwargs: logical_plan)
|
||||
monkeypatch.setattr(plan_module, "validate_assembly_plan", lambda plan: plan)
|
||||
|
||||
decode_entries, assembly_plan = plan_module.prepare_physical_decode_entries(
|
||||
entries,
|
||||
chunk_seconds=5.0,
|
||||
preserve_final_frame=True,
|
||||
terminal_merged=True,
|
||||
)
|
||||
|
||||
assert len(decode_entries) == 2
|
||||
assert torch.equal(decode_entries[1]["video"], physical_video)
|
||||
assert torch.equal(decode_entries[1]["audio"], physical_audio)
|
||||
assert assembly_plan["physical_decode_group_count"] == 2
|
||||
terminal_group = assembly_plan["decode_groups"][1]
|
||||
assert terminal_group["logical_chunk_indices"] == [2, 3]
|
||||
assert terminal_group["total_frames"] == 260
|
||||
assert terminal_group["trim_frames"] == 22
|
||||
assert terminal_group["net_frames"] == 238
|
||||
@@ -0,0 +1,32 @@
|
||||
# H3 Core vs Continuum GPU Diagnostic
|
||||
|
||||
This is a standalone diagnostic node. It does not change or register through the
|
||||
production H3 Continuum package.
|
||||
|
||||
## Install for ComfyUI_W
|
||||
|
||||
Run from the repository root:
|
||||
|
||||
```powershell
|
||||
.\tools\install_gpu_diagnostic_node.ps1
|
||||
```
|
||||
|
||||
Restart ComfyUI and search for `H3 Core vs Continuum GPU Diagnostic`.
|
||||
|
||||
## Connections
|
||||
|
||||
- Connect the same direct MODEL used by both tests. Keep Spectrum off.
|
||||
- Connect the same CLIP, Video VAE, Audio VAE, sampler, and sigmas.
|
||||
- Connect the exact same First Frame and Last Frame.
|
||||
- Use 640x640, seed 42, 8 steps, Euler/simple for the baseline comparison.
|
||||
- Run Storage and Session are not used by this node.
|
||||
|
||||
One queue executes Core FL2VA at 243 frames, then Continuum Terminal Merge 2x5s.
|
||||
Results are written below:
|
||||
|
||||
```text
|
||||
ComfyUI/output/h3_continuum/diagnostics/<timestamp>/
|
||||
```
|
||||
|
||||
The report identifies the first difference across pre-sampler data, first H3
|
||||
MODEL call, sampled physical latent, split/recombine, and decode/assembly.
|
||||
@@ -0,0 +1,9 @@
|
||||
"""Standalone GPU diagnostics for H3 Continuum.
|
||||
|
||||
This package is intentionally separate from the production node package. Copy the
|
||||
directory into ComfyUI/custom_nodes only while diagnostics are needed.
|
||||
"""
|
||||
|
||||
from .nodes import NODE_CLASS_MAPPINGS, NODE_DISPLAY_NAME_MAPPINGS
|
||||
|
||||
__all__ = ["NODE_CLASS_MAPPINGS", "NODE_DISPLAY_NAME_MAPPINGS"]
|
||||
@@ -0,0 +1,302 @@
|
||||
"""Deterministic tensor capture and comparison helpers for the GPU runner."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Mapping, Sequence
|
||||
from dataclasses import asdict, dataclass
|
||||
from hashlib import sha256
|
||||
import json
|
||||
from pathlib import Path
|
||||
import re
|
||||
from typing import Any
|
||||
|
||||
import torch
|
||||
|
||||
|
||||
STAGE_ORDER = (
|
||||
"A_pre_sampler",
|
||||
"B_first_model_call",
|
||||
"C_physical_sample",
|
||||
"D_split_recombine",
|
||||
"E1_decode_243",
|
||||
"E2_final_240",
|
||||
)
|
||||
|
||||
|
||||
def _dense_cpu(tensor: torch.Tensor) -> torch.Tensor:
|
||||
return tensor.detach().to("cpu").contiguous()
|
||||
|
||||
|
||||
def _raw_sha256(tensor: torch.Tensor) -> str:
|
||||
value = _dense_cpu(tensor)
|
||||
raw = value.view(torch.uint8).numpy().tobytes()
|
||||
return sha256(raw).hexdigest()
|
||||
|
||||
|
||||
def tensor_fingerprint(tensor: torch.Tensor) -> dict[str, Any]:
|
||||
value = _dense_cpu(tensor)
|
||||
return {
|
||||
"shape": list(value.shape),
|
||||
"dtype": str(value.dtype),
|
||||
"sha256": _raw_sha256(value),
|
||||
"numel": int(value.numel()),
|
||||
}
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class TensorDifference:
|
||||
comparable: bool
|
||||
bit_exact: bool
|
||||
allclose: bool
|
||||
max_abs_diff: float | None
|
||||
mean_abs_diff: float | None
|
||||
reason: str | None = None
|
||||
|
||||
@property
|
||||
def matches(self) -> bool:
|
||||
return self.comparable and (self.bit_exact or self.allclose)
|
||||
|
||||
def to_dict(self) -> dict[str, Any]:
|
||||
value = asdict(self)
|
||||
value["matches"] = self.matches
|
||||
return value
|
||||
|
||||
|
||||
def compare_tensors(
|
||||
left: torch.Tensor,
|
||||
right: torch.Tensor,
|
||||
*,
|
||||
rtol: float = 1.0e-4,
|
||||
atol: float = 1.0e-5,
|
||||
) -> TensorDifference:
|
||||
a = _dense_cpu(left)
|
||||
b = _dense_cpu(right)
|
||||
if tuple(a.shape) != tuple(b.shape):
|
||||
return TensorDifference(False, False, False, None, None, "shape mismatch")
|
||||
if a.dtype != b.dtype:
|
||||
return TensorDifference(False, False, False, None, None, "dtype mismatch")
|
||||
bit_exact = bool(torch.equal(a, b))
|
||||
if not (a.is_floating_point() or a.is_complex()):
|
||||
return TensorDifference(True, bit_exact, bit_exact, 0.0 if bit_exact else None, 0.0 if bit_exact else None)
|
||||
af = a.to(torch.float64)
|
||||
bf = b.to(torch.float64)
|
||||
delta = (af - bf).abs()
|
||||
max_abs = float(delta.max().item()) if delta.numel() else 0.0
|
||||
mean_abs = float(delta.mean().item()) if delta.numel() else 0.0
|
||||
close = bool(torch.allclose(af, bf, rtol=rtol, atol=atol, equal_nan=True))
|
||||
return TensorDifference(True, bit_exact, close, max_abs, mean_abs)
|
||||
|
||||
|
||||
def _safe_tensor_key(value: str) -> str:
|
||||
return re.sub(r"[^A-Za-z0-9_.-]+", "_", value).strip("_") or "tensor"
|
||||
|
||||
|
||||
def _is_primitive(value: Any) -> bool:
|
||||
return value is None or isinstance(value, (bool, int, float, str))
|
||||
|
||||
|
||||
def _primitive_sequence(value: Any) -> bool:
|
||||
return isinstance(value, (list, tuple)) and all(_is_primitive(item) for item in value)
|
||||
|
||||
|
||||
class DiagnosticCapture:
|
||||
def __init__(self, side: str):
|
||||
self.side = side
|
||||
self.manifest: dict[str, Any] = {
|
||||
"format": "h3-core-continuum-gpu-diagnostic-v1",
|
||||
"side": side,
|
||||
"stages": {},
|
||||
}
|
||||
self.tensors: dict[str, torch.Tensor] = {}
|
||||
self.runtime: dict[str, Any] = {}
|
||||
|
||||
def _stage(self, stage: str) -> dict[str, Any]:
|
||||
return self.manifest["stages"].setdefault(stage, {"values": {}, "tensors": {}})
|
||||
|
||||
def add_value(self, stage: str, field: str, value: Any) -> None:
|
||||
if not (_is_primitive(value) or _primitive_sequence(value)):
|
||||
value = json.loads(json.dumps(value, default=str))
|
||||
self._stage(stage)["values"][field] = value
|
||||
|
||||
def add_tensor(self, stage: str, field: str, tensor: torch.Tensor, *, persist: bool = True) -> None:
|
||||
if not torch.is_tensor(tensor):
|
||||
raise TypeError(f"{field} is not a tensor")
|
||||
if getattr(tensor, "is_nested", False):
|
||||
parts = list(tensor.unbind())
|
||||
self.add_value(stage, f"{field}.components", len(parts))
|
||||
for index, part in enumerate(parts):
|
||||
self.add_tensor(stage, f"{field}.component_{index}", part, persist=persist)
|
||||
return
|
||||
value = _dense_cpu(tensor)
|
||||
key = _safe_tensor_key(f"{stage}.{field}")
|
||||
self._stage(stage)["tensors"][field] = {
|
||||
**tensor_fingerprint(value),
|
||||
"stored_as": key if persist else None,
|
||||
}
|
||||
if persist:
|
||||
self.tensors[key] = value.clone()
|
||||
|
||||
def capture_value(
|
||||
self,
|
||||
stage: str,
|
||||
field: str,
|
||||
value: Any,
|
||||
*,
|
||||
persist_tensors: bool = True,
|
||||
depth: int = 0,
|
||||
) -> None:
|
||||
if depth > 7:
|
||||
self.add_value(stage, f"{field}.truncated", True)
|
||||
return
|
||||
if torch.is_tensor(value):
|
||||
self.add_tensor(stage, field, value, persist=persist_tensors)
|
||||
return
|
||||
if _is_primitive(value):
|
||||
self.add_value(stage, field, value)
|
||||
return
|
||||
if _primitive_sequence(value):
|
||||
self.add_value(stage, field, list(value))
|
||||
return
|
||||
if isinstance(value, Mapping):
|
||||
keys = sorted(str(key) for key in value.keys())
|
||||
self.add_value(stage, f"{field}.keys", keys)
|
||||
for key in keys[:256]:
|
||||
try:
|
||||
child = value[key]
|
||||
except (KeyError, TypeError):
|
||||
child = next((item for raw, item in value.items() if str(raw) == key), None)
|
||||
self.capture_value(
|
||||
stage,
|
||||
f"{field}.{key}",
|
||||
child,
|
||||
persist_tensors=persist_tensors,
|
||||
depth=depth + 1,
|
||||
)
|
||||
return
|
||||
if isinstance(value, Sequence) and not isinstance(value, (str, bytes, bytearray)):
|
||||
self.add_value(stage, f"{field}.length", len(value))
|
||||
for index, child in enumerate(value[:256]):
|
||||
self.capture_value(
|
||||
stage,
|
||||
f"{field}.{index}",
|
||||
child,
|
||||
persist_tensors=persist_tensors,
|
||||
depth=depth + 1,
|
||||
)
|
||||
return
|
||||
if hasattr(value, "unbind") and value.__class__.__name__ == "NestedTensor":
|
||||
parts = list(value.unbind())
|
||||
self.add_value(stage, f"{field}.components", len(parts))
|
||||
for index, child in enumerate(parts):
|
||||
label = ("video", "audio")[index] if index < 2 else f"component_{index}"
|
||||
self.capture_value(
|
||||
stage,
|
||||
f"{field}.{label}",
|
||||
child,
|
||||
persist_tensors=persist_tensors,
|
||||
depth=depth + 1,
|
||||
)
|
||||
return
|
||||
if all(hasattr(value, name) for name in ("signature", "segments", "position_ids")):
|
||||
self.add_value(stage, f"{field}.type", value.__class__.__name__)
|
||||
self.add_value(stage, f"{field}.signature", list(value.signature))
|
||||
self.add_value(stage, f"{field}.segments", [list(item) for item in value.segments])
|
||||
for name in ("position_ids", "img_pos", "img_update", "audio_pos", "audio_update"):
|
||||
child = getattr(value, name, None)
|
||||
if torch.is_tensor(child):
|
||||
self.add_tensor(stage, f"{field}.{name}", child, persist=persist_tensors)
|
||||
return
|
||||
self.add_value(stage, f"{field}.type", value.__class__.__name__)
|
||||
|
||||
def save(self, directory: Path) -> None:
|
||||
directory.mkdir(parents=True, exist_ok=True)
|
||||
from safetensors.torch import save_file
|
||||
|
||||
save_file(self.tensors, str(directory / "tensors.safetensors"))
|
||||
(directory / "manifest.json").write_text(
|
||||
json.dumps(self.manifest, indent=2, ensure_ascii=False),
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
|
||||
def compare_stage(
|
||||
left: DiagnosticCapture,
|
||||
right: DiagnosticCapture,
|
||||
stage: str,
|
||||
) -> dict[str, Any]:
|
||||
a = left.manifest["stages"].get(stage, {"values": {}, "tensors": {}})
|
||||
b = right.manifest["stages"].get(stage, {"values": {}, "tensors": {}})
|
||||
details: dict[str, Any] = {}
|
||||
first_field: str | None = None
|
||||
|
||||
value_fields = list(a["values"].keys()) + [key for key in b["values"] if key not in a["values"]]
|
||||
for field in value_fields:
|
||||
av = a["values"].get(field, {"missing": True})
|
||||
bv = b["values"].get(field, {"missing": True})
|
||||
matches = av == bv
|
||||
details[field] = {"kind": "value", "matches": matches, "left": av, "right": bv}
|
||||
if not matches and first_field is None:
|
||||
first_field = field
|
||||
|
||||
tensor_fields = list(a["tensors"].keys()) + [key for key in b["tensors"] if key not in a["tensors"]]
|
||||
for field in tensor_fields:
|
||||
a_meta = a["tensors"].get(field)
|
||||
b_meta = b["tensors"].get(field)
|
||||
if a_meta is None or b_meta is None:
|
||||
result = {"kind": "tensor", "matches": False, "reason": "missing tensor"}
|
||||
else:
|
||||
a_key = a_meta.get("stored_as")
|
||||
b_key = b_meta.get("stored_as")
|
||||
if a_key in left.tensors and b_key in right.tensors:
|
||||
result = {"kind": "tensor", **compare_tensors(left.tensors[a_key], right.tensors[b_key]).to_dict()}
|
||||
else:
|
||||
matches = a_meta == b_meta
|
||||
result = {"kind": "fingerprint", "matches": matches, "left": a_meta, "right": b_meta}
|
||||
details[field] = result
|
||||
if not result["matches"] and first_field is None:
|
||||
first_field = field
|
||||
|
||||
return {
|
||||
"status": "MATCH" if first_field is None else "DIFFER",
|
||||
"first_field": first_field,
|
||||
"fields": details,
|
||||
}
|
||||
|
||||
|
||||
def recombine_terminal_streams(
|
||||
first: torch.Tensor,
|
||||
second: torch.Tensor,
|
||||
*,
|
||||
first_range: tuple[int, int] | list[int],
|
||||
second_range: tuple[int, int] | list[int],
|
||||
dim: int,
|
||||
) -> torch.Tensor:
|
||||
first_start, first_stop = (int(value) for value in first_range)
|
||||
second_start, second_stop = (int(value) for value in second_range)
|
||||
overlap = first_stop - second_start
|
||||
if first_start != 0 or overlap < 0:
|
||||
raise ValueError("terminal split ranges do not form an overlapping sequence")
|
||||
if first.shape[dim] != first_stop - first_start:
|
||||
raise ValueError("first logical stream length does not match its range")
|
||||
if second.shape[dim] != second_stop - second_start:
|
||||
raise ValueError("second logical stream length does not match its range")
|
||||
if overlap > second.shape[dim]:
|
||||
raise ValueError("terminal overlap exceeds the second logical stream")
|
||||
slices = [slice(None)] * second.ndim
|
||||
slices[dim] = slice(overlap, None)
|
||||
return torch.cat((first, second[tuple(slices)]), dim=dim).contiguous()
|
||||
|
||||
|
||||
def format_comparison(comparison: Mapping[str, Any], output_directory: Path) -> str:
|
||||
lines = []
|
||||
first_divergence = None
|
||||
for stage in STAGE_ORDER:
|
||||
result = comparison.get(stage, {"status": "NOT_RUN", "first_field": None})
|
||||
lines.append(f"{stage}: {result['status']}")
|
||||
if first_divergence is None and result["status"] == "DIFFER":
|
||||
field = result.get("first_field")
|
||||
first_divergence = f"{stage}.{field}" if field else stage
|
||||
lines.append("")
|
||||
lines.append(f"First divergence: {first_divergence or 'none'}")
|
||||
lines.append(f"Output: {output_directory}")
|
||||
return "\n".join(lines)
|
||||
@@ -0,0 +1,715 @@
|
||||
"""Standalone ComfyUI node for Core-vs-Continuum GPU diagnostics."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from contextlib import contextmanager
|
||||
from datetime import datetime
|
||||
import gc
|
||||
import importlib
|
||||
import json
|
||||
from pathlib import Path, PurePosixPath
|
||||
import random
|
||||
import sys
|
||||
import traceback
|
||||
from types import SimpleNamespace
|
||||
from typing import Any
|
||||
|
||||
import numpy as np
|
||||
import torch
|
||||
|
||||
from .capture import (
|
||||
DiagnosticCapture,
|
||||
STAGE_ORDER,
|
||||
compare_stage,
|
||||
compare_tensors,
|
||||
format_comparison,
|
||||
recombine_terminal_streams,
|
||||
)
|
||||
|
||||
|
||||
def _resolve_loaded_module(path: Path):
|
||||
target = path.resolve()
|
||||
for module in tuple(sys.modules.values()):
|
||||
source = getattr(module, "__file__", None)
|
||||
if source and Path(source).resolve() == target:
|
||||
return module
|
||||
return None
|
||||
|
||||
|
||||
def _resolve_continuum_modules() -> SimpleNamespace:
|
||||
import nodes as comfy_nodes
|
||||
|
||||
sampler_class = comfy_nodes.NODE_CLASS_MAPPINGS.get("H3ContinuumSamplerV34")
|
||||
if sampler_class is None:
|
||||
raise RuntimeError("H3ContinuumSamplerV34 is not loaded; install and enable H3 Continuum V3.4 first")
|
||||
owner = sys.modules.get(sampler_class.__module__)
|
||||
source = getattr(owner, "__file__", None)
|
||||
if source is None:
|
||||
raise RuntimeError("cannot resolve the active H3 Continuum package")
|
||||
root = Path(source).resolve().parent.parent
|
||||
module_name = sampler_class.__module__
|
||||
base_name = module_name.split(".v3.", 1)[0] if ".v3." in module_name else module_name.rsplit(".", 2)[0]
|
||||
|
||||
def load(relative: str, suffix: str):
|
||||
target = root / relative
|
||||
loaded = _resolve_loaded_module(target)
|
||||
if loaded is not None:
|
||||
return loaded
|
||||
return importlib.import_module(f"{base_name}.{suffix}")
|
||||
|
||||
return SimpleNamespace(
|
||||
root=root,
|
||||
constants=load("constants.py", "constants"),
|
||||
state=load("state.py", "state"),
|
||||
sequence=load("v2/sequence.py", "v2.sequence"),
|
||||
sampling=load("v2/sampling.py", "v2.sampling"),
|
||||
prompts=load("v2/prompts.py", "v2.prompts"),
|
||||
decoder=load("v2/decoder.py", "v2.decoder"),
|
||||
)
|
||||
|
||||
|
||||
def _reset_rng(seed: int) -> None:
|
||||
random.seed(seed)
|
||||
np.random.seed(seed & 0xFFFFFFFF)
|
||||
torch.manual_seed(seed)
|
||||
if torch.cuda.is_available():
|
||||
torch.cuda.manual_seed_all(seed)
|
||||
|
||||
|
||||
def _release_cuda() -> None:
|
||||
gc.collect()
|
||||
if torch.cuda.is_available():
|
||||
torch.cuda.empty_cache()
|
||||
|
||||
|
||||
def _safe_output_directory(subdir: str) -> Path:
|
||||
import folder_paths
|
||||
|
||||
clean = str(subdir).replace("\\", "/").strip("/") or "h3_continuum/diagnostics"
|
||||
parts = PurePosixPath(clean).parts
|
||||
if any(part in ("", ".", "..") for part in parts):
|
||||
raise ValueError("output_subdir must stay below the ComfyUI output directory")
|
||||
stamp = datetime.now().strftime("%Y%m%d_%H%M%S_%f")
|
||||
return Path(folder_paths.get_output_directory()).joinpath(*parts, stamp)
|
||||
|
||||
|
||||
def _capture_av_value(capture: DiagnosticCapture, stage: str, prefix: str, value: Any) -> None:
|
||||
candidate = value.get("samples") if isinstance(value, dict) and "samples" in value else value
|
||||
if hasattr(candidate, "unbind") and candidate.__class__.__name__ == "NestedTensor":
|
||||
parts = list(candidate.unbind())
|
||||
capture.add_value(stage, f"{prefix}.components", len(parts))
|
||||
for index, part in enumerate(parts):
|
||||
label = ("video", "audio")[index] if index < 2 else f"component_{index}"
|
||||
capture.add_tensor(stage, f"{prefix}.{label}", part)
|
||||
return
|
||||
if torch.is_tensor(candidate) and getattr(candidate, "is_nested", False):
|
||||
parts = list(candidate.unbind())
|
||||
capture.add_value(stage, f"{prefix}.components", len(parts))
|
||||
for index, part in enumerate(parts):
|
||||
label = ("video", "audio")[index] if index < 2 else f"component_{index}"
|
||||
capture.add_tensor(stage, f"{prefix}.{label}", part)
|
||||
return
|
||||
capture.capture_value(stage, prefix, candidate)
|
||||
|
||||
|
||||
def _capture_pre_sampler(
|
||||
capture: DiagnosticCapture,
|
||||
modules: SimpleNamespace,
|
||||
*,
|
||||
conditioning: Any,
|
||||
latent: dict[str, Any],
|
||||
sigmas: torch.Tensor,
|
||||
seed: int,
|
||||
) -> None:
|
||||
stage = "A_pre_sampler"
|
||||
capture.add_value(stage, "seed", int(seed))
|
||||
capture.add_value(stage, "presentation_order", ["First Frame", "Last Frame"])
|
||||
capture.add_tensor(stage, "sigmas", sigmas)
|
||||
video, audio = modules.state.extract_av_streams(latent)
|
||||
capture.add_tensor(stage, "target.video", video)
|
||||
capture.add_tensor(stage, "target.audio", audio)
|
||||
capture.add_value(stage, "target.video_shape", list(video.shape))
|
||||
capture.add_value(stage, "target.audio_shape", list(audio.shape))
|
||||
|
||||
if not conditioning:
|
||||
capture.add_value(stage, "conditioning.empty", True)
|
||||
return
|
||||
first = conditioning[0]
|
||||
cond_tensor = first[0]
|
||||
metadata = first[1] if len(first) > 1 and isinstance(first[1], dict) else {}
|
||||
capture.capture_value(stage, "conditioning.positive", cond_tensor)
|
||||
capture.add_value(stage, "conditioning.metadata_keys", sorted(str(key) for key in metadata))
|
||||
capture.add_value(stage, "physical_frames", int(metadata.get("minimax_frame_count", 243)))
|
||||
for key in ("minimax_token_tags", "minimax_keyframes", "minimax_refs", "minimax_frame_count"):
|
||||
if key in metadata:
|
||||
capture.capture_value(stage, f"conditioning.{key}", metadata[key])
|
||||
|
||||
|
||||
def _capture_physical_sample(
|
||||
capture: DiagnosticCapture,
|
||||
modules: SimpleNamespace,
|
||||
result: dict[str, Any],
|
||||
) -> tuple[torch.Tensor, torch.Tensor]:
|
||||
video, audio = modules.state.extract_av_streams(result)
|
||||
video_cpu = video.detach().to("cpu").contiguous()
|
||||
audio_cpu = audio.detach().to("cpu").contiguous()
|
||||
capture.add_tensor("C_physical_sample", "video", video_cpu)
|
||||
capture.add_tensor("C_physical_sample", "audio", audio_cpu)
|
||||
capture.runtime["physical_streams"] = (video_cpu, audio_cpu)
|
||||
return video_cpu, audio_cpu
|
||||
|
||||
|
||||
@contextmanager
|
||||
def _runtime_capture(capture: DiagnosticCapture):
|
||||
import comfy.model_base
|
||||
import comfy.sample
|
||||
|
||||
original_noise = comfy.sample.prepare_noise
|
||||
original_apply = comfy.model_base.BaseModel._apply_model
|
||||
noise_seen = False
|
||||
model_seen = False
|
||||
|
||||
def prepare_noise(*args, **kwargs):
|
||||
nonlocal noise_seen
|
||||
result = original_noise(*args, **kwargs)
|
||||
if not noise_seen:
|
||||
noise_seen = True
|
||||
_capture_av_value(capture, "A_pre_sampler", "actual_noise", result)
|
||||
return result
|
||||
|
||||
def apply_model(
|
||||
self,
|
||||
x,
|
||||
t,
|
||||
c_concat=None,
|
||||
c_crossattn=None,
|
||||
control=None,
|
||||
transformer_options={},
|
||||
**kwargs,
|
||||
):
|
||||
nonlocal model_seen
|
||||
if not model_seen and self.__class__.__name__ == "MiniMaxH3":
|
||||
model_seen = True
|
||||
capture.capture_value("B_first_model_call", "input_x", x)
|
||||
capture.capture_value("B_first_model_call", "timestep", t)
|
||||
capture.capture_value(
|
||||
"B_first_model_call",
|
||||
"transformer_options",
|
||||
transformer_options,
|
||||
persist_tensors=False,
|
||||
)
|
||||
capture.capture_value("B_first_model_call", "minimax_payload", kwargs.get("minimax_payload"))
|
||||
capture.capture_value("B_first_model_call", "latent_shapes", kwargs.get("latent_shapes"))
|
||||
return original_apply(
|
||||
self,
|
||||
x,
|
||||
t,
|
||||
c_concat,
|
||||
c_crossattn,
|
||||
control,
|
||||
transformer_options,
|
||||
**kwargs,
|
||||
)
|
||||
|
||||
comfy.sample.prepare_noise = prepare_noise
|
||||
comfy.model_base.BaseModel._apply_model = apply_model
|
||||
try:
|
||||
yield
|
||||
finally:
|
||||
comfy.sample.prepare_noise = original_noise
|
||||
comfy.model_base.BaseModel._apply_model = original_apply
|
||||
restored = (
|
||||
comfy.sample.prepare_noise is original_noise
|
||||
and comfy.model_base.BaseModel._apply_model is original_apply
|
||||
)
|
||||
capture.add_value("B_first_model_call", "patches_restored", restored)
|
||||
capture.add_value("A_pre_sampler", "noise_captured", noise_seen)
|
||||
capture.add_value("B_first_model_call", "model_call_captured", model_seen)
|
||||
|
||||
|
||||
def _run_core(
|
||||
capture: DiagnosticCapture,
|
||||
modules: SimpleNamespace,
|
||||
*,
|
||||
model: Any,
|
||||
clip: Any,
|
||||
video_vae: Any,
|
||||
sampler: Any,
|
||||
sigmas: torch.Tensor,
|
||||
first_frame: torch.Tensor,
|
||||
last_frame: torch.Tensor,
|
||||
prompt: str,
|
||||
width: int,
|
||||
height: int,
|
||||
seed: int,
|
||||
) -> None:
|
||||
from comfy_extras.nodes_minimax_h3 import MiniMaxH3ImageToVideo
|
||||
|
||||
core_model = model.clone()
|
||||
output = MiniMaxH3ImageToVideo().execute(
|
||||
clip,
|
||||
video_vae,
|
||||
prompt,
|
||||
int(width),
|
||||
int(height),
|
||||
243,
|
||||
first_frame=first_frame,
|
||||
last_frame=last_frame,
|
||||
)
|
||||
conditioning, latent = output[0], output[1]
|
||||
_capture_pre_sampler(
|
||||
capture,
|
||||
modules,
|
||||
conditioning=conditioning,
|
||||
latent=latent,
|
||||
sigmas=sigmas,
|
||||
seed=seed,
|
||||
)
|
||||
with _runtime_capture(capture):
|
||||
result = modules.sampling.sample_chunk(
|
||||
model=core_model,
|
||||
conditioning=conditioning,
|
||||
latent=latent,
|
||||
sampler=sampler,
|
||||
sigmas=sigmas,
|
||||
seed=int(seed),
|
||||
enable_preview=False,
|
||||
)
|
||||
_capture_physical_sample(capture, modules, result)
|
||||
capture.add_value("C_physical_sample", "sampling_passes", 1)
|
||||
capture.add_value("C_physical_sample", "physical_seed", int(seed))
|
||||
del result, latent, conditioning, core_model
|
||||
|
||||
|
||||
def _capture_terminal_split(
|
||||
capture: DiagnosticCapture,
|
||||
video: torch.Tensor,
|
||||
audio: torch.Tensor,
|
||||
contract: dict[str, Any],
|
||||
result: Any,
|
||||
) -> None:
|
||||
(video_1, audio_1), (video_2, audio_2) = result
|
||||
video_ranges = contract["video_slices"]
|
||||
audio_ranges = contract["audio_slices"]
|
||||
video_recombined = recombine_terminal_streams(
|
||||
video_1,
|
||||
video_2,
|
||||
first_range=video_ranges[0],
|
||||
second_range=video_ranges[1],
|
||||
dim=2,
|
||||
)
|
||||
audio_recombined = recombine_terminal_streams(
|
||||
audio_1,
|
||||
audio_2,
|
||||
first_range=audio_ranges[0],
|
||||
second_range=audio_ranges[1],
|
||||
dim=-1,
|
||||
)
|
||||
video_diff = compare_tensors(video.detach().to("cpu"), video_recombined.detach().to("cpu"))
|
||||
audio_diff = compare_tensors(audio.detach().to("cpu"), audio_recombined.detach().to("cpu"))
|
||||
stage = "D_split_recombine"
|
||||
capture.add_value(stage, "contract", contract)
|
||||
capture.add_tensor(stage, "video.physical", video)
|
||||
capture.add_tensor(stage, "video.chunk_1", video_1)
|
||||
capture.add_tensor(stage, "video.chunk_2", video_2)
|
||||
capture.add_tensor(stage, "video.recombined", video_recombined)
|
||||
capture.add_tensor(stage, "audio.physical", audio)
|
||||
capture.add_tensor(stage, "audio.chunk_1", audio_1)
|
||||
capture.add_tensor(stage, "audio.chunk_2", audio_2)
|
||||
capture.add_tensor(stage, "audio.recombined", audio_recombined)
|
||||
capture.add_value(stage, "video_difference", video_diff.to_dict())
|
||||
capture.add_value(stage, "audio_difference", audio_diff.to_dict())
|
||||
capture.runtime["logical_streams"] = (
|
||||
(video_1.detach().to("cpu").contiguous(), audio_1.detach().to("cpu").contiguous()),
|
||||
(video_2.detach().to("cpu").contiguous(), audio_2.detach().to("cpu").contiguous()),
|
||||
)
|
||||
capture.runtime["terminal_contract"] = contract
|
||||
|
||||
|
||||
def _run_continuum(
|
||||
capture: DiagnosticCapture,
|
||||
modules: SimpleNamespace,
|
||||
*,
|
||||
model: Any,
|
||||
clip: Any,
|
||||
video_vae: Any,
|
||||
audio_vae: Any,
|
||||
sampler: Any,
|
||||
sigmas: torch.Tensor,
|
||||
first_frame: torch.Tensor,
|
||||
last_frame: torch.Tensor,
|
||||
prompt: str,
|
||||
width: int,
|
||||
height: int,
|
||||
seed: int,
|
||||
) -> None:
|
||||
sequence = modules.sequence
|
||||
original_sample = sequence.sample_chunk
|
||||
original_split = sequence._split_terminal_merged_latents
|
||||
sample_calls = 0
|
||||
split_calls = 0
|
||||
|
||||
def sample_chunk(**kwargs):
|
||||
nonlocal sample_calls
|
||||
sample_calls += 1
|
||||
if sample_calls == 1:
|
||||
_capture_pre_sampler(
|
||||
capture,
|
||||
modules,
|
||||
conditioning=kwargs["conditioning"],
|
||||
latent=kwargs["latent"],
|
||||
sigmas=kwargs["sigmas"],
|
||||
seed=int(kwargs["seed"]),
|
||||
)
|
||||
result = original_sample(**kwargs)
|
||||
if sample_calls == 1:
|
||||
_capture_physical_sample(capture, modules, result)
|
||||
capture.add_value("C_physical_sample", "physical_seed", int(kwargs["seed"]))
|
||||
return result
|
||||
|
||||
def split_terminal(video, audio, contract):
|
||||
nonlocal split_calls
|
||||
split_calls += 1
|
||||
result = original_split(video, audio, contract)
|
||||
if split_calls == 1:
|
||||
_capture_terminal_split(capture, video, audio, contract, result)
|
||||
return result
|
||||
|
||||
sequence.sample_chunk = sample_chunk
|
||||
sequence._split_terminal_merged_latents = split_terminal
|
||||
try:
|
||||
prompt_plan = modules.prompts.make_prompt_plan(
|
||||
mode=modules.constants.PROMPT_MODE_FIXED,
|
||||
script=prompt,
|
||||
chunks=2,
|
||||
chunk_seconds=5.0,
|
||||
)
|
||||
continuum_model = model.clone()
|
||||
with _runtime_capture(capture):
|
||||
sequence.run_sequence(
|
||||
model=continuum_model,
|
||||
clip=clip,
|
||||
video_vae=video_vae,
|
||||
audio_vae=audio_vae,
|
||||
sampler=sampler,
|
||||
sigmas=sigmas,
|
||||
first_frame=first_frame,
|
||||
last_frame=last_frame,
|
||||
prompt_plan=prompt_plan,
|
||||
width=int(width),
|
||||
height=int(height),
|
||||
continuity=modules.constants.CONTINUITY_OPTIONS[0],
|
||||
base_seed=int(seed),
|
||||
audio_continuity=True,
|
||||
exact_total_duration=True,
|
||||
diagnostics_mode=modules.constants.DIAGNOSTICS_BASIC,
|
||||
reroll_from_chunk=0,
|
||||
reroll_nonce=0,
|
||||
strict_compatibility=False,
|
||||
debug=False,
|
||||
seam_correction=modules.constants.SEAM_CORRECTION_OFF,
|
||||
enable_preview=False,
|
||||
session=None,
|
||||
initial_state=None,
|
||||
latent_only=True,
|
||||
reference_assets=None,
|
||||
reference_audio_source=None,
|
||||
reference_audio_vae=None,
|
||||
driving_audio_source=None,
|
||||
driving_audio_vae=None,
|
||||
reference_video_source=None,
|
||||
timeline_video_source=None,
|
||||
)
|
||||
capture.add_value("C_physical_sample", "sampling_passes", sample_calls)
|
||||
capture.add_value("D_split_recombine", "split_calls", split_calls)
|
||||
del continuum_model
|
||||
finally:
|
||||
sequence.sample_chunk = original_sample
|
||||
sequence._split_terminal_merged_latents = original_split
|
||||
capture.add_value(
|
||||
"D_split_recombine",
|
||||
"sequence_patches_restored",
|
||||
sequence.sample_chunk is original_sample
|
||||
and sequence._split_terminal_merged_latents is original_split,
|
||||
)
|
||||
|
||||
|
||||
def _make_av_latent(video: torch.Tensor, audio: torch.Tensor) -> dict[str, Any]:
|
||||
from comfy.nested_tensor import NestedTensor
|
||||
|
||||
return {"samples": NestedTensor([video, audio])}
|
||||
|
||||
|
||||
def _copy_audio(audio: dict[str, Any]) -> dict[str, Any]:
|
||||
return {
|
||||
"waveform": audio["waveform"].clone(),
|
||||
"sample_rate": int(audio["sample_rate"]),
|
||||
}
|
||||
|
||||
|
||||
def _run_decode_diagnostics(capture: DiagnosticCapture, modules: SimpleNamespace, video_vae: Any, audio_vae: Any) -> dict[str, Any]:
|
||||
physical = capture.runtime.get("physical_streams")
|
||||
logical = capture.runtime.get("logical_streams")
|
||||
contract = capture.runtime.get("terminal_contract")
|
||||
if physical is None or logical is None or contract is None:
|
||||
return {
|
||||
"E1_decode_243": {"status": "NOT_RUN", "first_field": "missing terminal capture"},
|
||||
"E2_final_240": {"status": "NOT_RUN", "first_field": "missing terminal capture"},
|
||||
}
|
||||
|
||||
decoder = modules.decoder
|
||||
physical_video, physical_audio = physical
|
||||
direct_latent = _make_av_latent(physical_video, physical_audio)
|
||||
direct_images = decoder.decode_video(video_vae, direct_latent)
|
||||
direct_audio = decoder.decode_audio(audio_vae, direct_latent)
|
||||
|
||||
decoded_images = []
|
||||
decoded_audio = []
|
||||
for video, audio in logical:
|
||||
latent = _make_av_latent(video, audio)
|
||||
decoded_images.append(decoder.decode_video(video_vae, latent))
|
||||
decoded_audio.append(decoder.decode_audio(audio_vae, latent))
|
||||
|
||||
trim_frames = 22
|
||||
reconstructed_images = torch.cat(
|
||||
(decoded_images[0][:124], decoded_images[1][:141][trim_frames:]),
|
||||
dim=0,
|
||||
).contiguous()
|
||||
sample_rate = int(decoded_audio[0]["sample_rate"])
|
||||
if int(decoded_audio[1]["sample_rate"]) != sample_rate:
|
||||
raise ValueError("chunk audio sample rates differ during decode diagnostics")
|
||||
total_samples = int(round(243 / 24.0 * sample_rate))
|
||||
reconstructed_waveform = torch.empty(
|
||||
(*decoded_audio[0]["waveform"].shape[:-1], total_samples),
|
||||
dtype=decoded_audio[0]["waveform"].dtype,
|
||||
device="cpu",
|
||||
)
|
||||
segment_1 = decoder._slice_audio_for_timeline(
|
||||
decoded_audio[0], trim_frames=0, frame_start=0, frame_stop=124
|
||||
)
|
||||
segment_2 = decoder._slice_audio_for_timeline(
|
||||
decoded_audio[1], trim_frames=trim_frames, frame_start=124, frame_stop=243
|
||||
)
|
||||
split_sample = int(round(124 / 24.0 * sample_rate))
|
||||
reconstructed_waveform[..., :split_sample].copy_(segment_1)
|
||||
reconstructed_waveform[..., split_sample:].copy_(segment_2)
|
||||
reconstructed_audio = {"waveform": reconstructed_waveform, "sample_rate": sample_rate}
|
||||
|
||||
direct_images, direct_audio, _ = decoder.enforce_total_frames(
|
||||
direct_images,
|
||||
direct_audio,
|
||||
target_frames=243,
|
||||
preserve_final_frame=False,
|
||||
)
|
||||
reconstructed_images, reconstructed_audio, _ = decoder.enforce_total_frames(
|
||||
reconstructed_images,
|
||||
reconstructed_audio,
|
||||
target_frames=243,
|
||||
preserve_final_frame=False,
|
||||
)
|
||||
|
||||
image_e1 = compare_tensors(direct_images, reconstructed_images)
|
||||
audio_e1 = compare_tensors(direct_audio["waveform"], reconstructed_audio["waveform"])
|
||||
stage = "E1_decode_243"
|
||||
capture.add_tensor(stage, "direct.images", direct_images, persist=False)
|
||||
capture.add_tensor(stage, "split.images", reconstructed_images, persist=False)
|
||||
capture.add_tensor(stage, "direct.audio", direct_audio["waveform"], persist=False)
|
||||
capture.add_tensor(stage, "split.audio", reconstructed_audio["waveform"], persist=False)
|
||||
capture.add_value(stage, "images_difference", image_e1.to_dict())
|
||||
capture.add_value(stage, "audio_difference", audio_e1.to_dict())
|
||||
boundary = slice(112, 129)
|
||||
capture.add_tensor(stage, "direct.boundary_images", direct_images[boundary])
|
||||
capture.add_tensor(stage, "split.boundary_images", reconstructed_images[boundary])
|
||||
boundary_diff = compare_tensors(direct_images[boundary], reconstructed_images[boundary])
|
||||
capture.add_value(stage, "boundary_images_difference", boundary_diff.to_dict())
|
||||
|
||||
direct_240_images, direct_240_audio, _ = decoder.enforce_total_frames(
|
||||
direct_images,
|
||||
_copy_audio(direct_audio),
|
||||
target_frames=240,
|
||||
preserve_final_frame=True,
|
||||
)
|
||||
split_240_images, split_240_audio, _ = decoder.enforce_total_frames(
|
||||
reconstructed_images,
|
||||
_copy_audio(reconstructed_audio),
|
||||
target_frames=240,
|
||||
preserve_final_frame=True,
|
||||
)
|
||||
image_e2 = compare_tensors(direct_240_images, split_240_images)
|
||||
audio_e2 = compare_tensors(direct_240_audio["waveform"], split_240_audio["waveform"])
|
||||
stage = "E2_final_240"
|
||||
capture.add_tensor(stage, "direct.images", direct_240_images, persist=False)
|
||||
capture.add_tensor(stage, "split.images", split_240_images, persist=False)
|
||||
capture.add_tensor(stage, "direct.audio", direct_240_audio["waveform"], persist=False)
|
||||
capture.add_tensor(stage, "split.audio", split_240_audio["waveform"], persist=False)
|
||||
capture.add_value(stage, "images_difference", image_e2.to_dict())
|
||||
capture.add_value(stage, "audio_difference", audio_e2.to_dict())
|
||||
capture.add_tensor(stage, "direct.boundary_images", direct_240_images[boundary])
|
||||
capture.add_tensor(stage, "split.boundary_images", split_240_images[boundary])
|
||||
|
||||
return {
|
||||
"E1_decode_243": {
|
||||
"status": "MATCH" if image_e1.matches and audio_e1.matches else "DIFFER",
|
||||
"first_field": None if image_e1.matches and audio_e1.matches else ("images" if not image_e1.matches else "audio"),
|
||||
"images": image_e1.to_dict(),
|
||||
"audio": audio_e1.to_dict(),
|
||||
"boundary_images": boundary_diff.to_dict(),
|
||||
},
|
||||
"E2_final_240": {
|
||||
"status": "MATCH" if image_e2.matches and audio_e2.matches else "DIFFER",
|
||||
"first_field": None if image_e2.matches and audio_e2.matches else ("images" if not image_e2.matches else "audio"),
|
||||
"images": image_e2.to_dict(),
|
||||
"audio": audio_e2.to_dict(),
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
class H3CoreContinuumGpuDiagnostic:
|
||||
"""Run an isolated Core 243f and Continuum 2x5s comparison in one queue."""
|
||||
|
||||
OUTPUT_NODE = True
|
||||
CATEGORY = "H3-Continuum/Diagnostics"
|
||||
FUNCTION = "run"
|
||||
RETURN_TYPES = ("STRING",)
|
||||
RETURN_NAMES = ("diagnostic_report",)
|
||||
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
return {
|
||||
"required": {
|
||||
"model": ("MODEL",),
|
||||
"clip": ("CLIP",),
|
||||
"video_vae": ("VAE",),
|
||||
"audio_vae": ("VAE",),
|
||||
"sampler": ("SAMPLER",),
|
||||
"sigmas": ("SIGMAS",),
|
||||
"first_frame": ("IMAGE",),
|
||||
"last_frame": ("IMAGE",),
|
||||
"prompt": ("STRING", {"multiline": True, "default": "The subject moves continuously from the First Frame to the Last Frame."}),
|
||||
"width": ("INT", {"default": 640, "min": 64, "max": 4096, "step": 16}),
|
||||
"height": ("INT", {"default": 640, "min": 64, "max": 4096, "step": 16}),
|
||||
"seed": ("INT", {"default": 42, "min": 0, "max": 0xFFFFFFFFFFFFFFFF}),
|
||||
"output_subdir": ("STRING", {"default": "h3_continuum/diagnostics"}),
|
||||
}
|
||||
}
|
||||
|
||||
def run(
|
||||
self,
|
||||
model,
|
||||
clip,
|
||||
video_vae,
|
||||
audio_vae,
|
||||
sampler,
|
||||
sigmas,
|
||||
first_frame,
|
||||
last_frame,
|
||||
prompt,
|
||||
width,
|
||||
height,
|
||||
seed,
|
||||
output_subdir,
|
||||
):
|
||||
output_directory = _safe_output_directory(output_subdir)
|
||||
core = DiagnosticCapture("core")
|
||||
continuum = DiagnosticCapture("continuum")
|
||||
comparison: dict[str, Any] = {}
|
||||
modules = _resolve_continuum_modules()
|
||||
common = {
|
||||
"seed": int(seed),
|
||||
"width": int(width),
|
||||
"height": int(height),
|
||||
"physical_frames": 243,
|
||||
"spectrum": "must be OFF at the model input",
|
||||
"run_storage": "Off",
|
||||
"session": None,
|
||||
"continuum_root": str(modules.root),
|
||||
}
|
||||
core.manifest["conditions"] = common
|
||||
continuum.manifest["conditions"] = common
|
||||
|
||||
try:
|
||||
_reset_rng(int(seed))
|
||||
_run_core(
|
||||
core,
|
||||
modules,
|
||||
model=model,
|
||||
clip=clip,
|
||||
video_vae=video_vae,
|
||||
sampler=sampler,
|
||||
sigmas=sigmas,
|
||||
first_frame=first_frame,
|
||||
last_frame=last_frame,
|
||||
prompt=prompt,
|
||||
width=width,
|
||||
height=height,
|
||||
seed=seed,
|
||||
)
|
||||
_release_cuda()
|
||||
_reset_rng(int(seed))
|
||||
_run_continuum(
|
||||
continuum,
|
||||
modules,
|
||||
model=model,
|
||||
clip=clip,
|
||||
video_vae=video_vae,
|
||||
audio_vae=audio_vae,
|
||||
sampler=sampler,
|
||||
sigmas=sigmas,
|
||||
first_frame=first_frame,
|
||||
last_frame=last_frame,
|
||||
prompt=prompt,
|
||||
width=width,
|
||||
height=height,
|
||||
seed=seed,
|
||||
)
|
||||
|
||||
for stage in ("A_pre_sampler", "B_first_model_call", "C_physical_sample"):
|
||||
comparison[stage] = compare_stage(core, continuum, stage)
|
||||
|
||||
d_values = continuum.manifest["stages"].get("D_split_recombine", {}).get("values", {})
|
||||
d_video = d_values.get("video_difference", {})
|
||||
d_audio = d_values.get("audio_difference", {})
|
||||
d_match = bool(d_video.get("matches") and d_audio.get("matches"))
|
||||
comparison["D_split_recombine"] = {
|
||||
"status": "MATCH" if d_match else "DIFFER",
|
||||
"first_field": None if d_match else ("video" if not d_video.get("matches") else "audio"),
|
||||
"video": d_video,
|
||||
"audio": d_audio,
|
||||
}
|
||||
comparison.update(_run_decode_diagnostics(continuum, modules, video_vae, audio_vae))
|
||||
except Exception as exc:
|
||||
comparison["error"] = {
|
||||
"type": type(exc).__name__,
|
||||
"message": str(exc),
|
||||
"traceback": traceback.format_exc(),
|
||||
}
|
||||
finally:
|
||||
output_directory.mkdir(parents=True, exist_ok=True)
|
||||
core.save(output_directory / "core")
|
||||
continuum.save(output_directory / "continuum")
|
||||
for stage in STAGE_ORDER:
|
||||
comparison.setdefault(stage, {"status": "NOT_RUN", "first_field": None})
|
||||
summary = format_comparison(comparison, output_directory)
|
||||
(output_directory / "comparison.json").write_text(
|
||||
json.dumps(comparison, indent=2, ensure_ascii=False),
|
||||
encoding="utf-8",
|
||||
)
|
||||
(output_directory / "comparison.txt").write_text(summary + "\n", encoding="utf-8")
|
||||
_release_cuda()
|
||||
|
||||
if "error" in comparison:
|
||||
error = comparison["error"]
|
||||
raise RuntimeError(
|
||||
f"GPU diagnostic failed; partial results were saved to {output_directory}: "
|
||||
f"{error['type']}: {error['message']}"
|
||||
)
|
||||
return (summary,)
|
||||
|
||||
|
||||
NODE_CLASS_MAPPINGS = {
|
||||
"H3CoreContinuumGpuDiagnostic": H3CoreContinuumGpuDiagnostic,
|
||||
}
|
||||
|
||||
NODE_DISPLAY_NAME_MAPPINGS = {
|
||||
"H3CoreContinuumGpuDiagnostic": "H3 Core vs Continuum GPU Diagnostic",
|
||||
}
|
||||
@@ -0,0 +1,29 @@
|
||||
param(
|
||||
[string]$ComfyUIRoot = 'D:\StabilityMatrix\Data\Packages\ComfyUI_W'
|
||||
)
|
||||
|
||||
$ErrorActionPreference = 'Stop'
|
||||
$source = Join-Path $PSScriptRoot 'gpu_diagnostic_node'
|
||||
$customNodes = Join-Path $ComfyUIRoot 'custom_nodes'
|
||||
$destination = Join-Path $customNodes 'H3_Continuum_GPU_Diagnostics'
|
||||
|
||||
if (-not (Test-Path -LiteralPath $source -PathType Container)) {
|
||||
throw "Diagnostic source was not found: $source"
|
||||
}
|
||||
if (-not (Test-Path -LiteralPath $customNodes -PathType Container)) {
|
||||
throw "ComfyUI custom_nodes directory was not found: $customNodes"
|
||||
}
|
||||
|
||||
New-Item -ItemType Directory -Path $destination -Force | Out-Null
|
||||
foreach ($name in @('__init__.py', 'capture.py', 'nodes.py', 'README.md')) {
|
||||
Copy-Item -LiteralPath (Join-Path $source $name) -Destination (Join-Path $destination $name) -Force
|
||||
}
|
||||
|
||||
$sourceCount = (Get-ChildItem -LiteralPath $source -File).Count
|
||||
$destinationCount = (Get-ChildItem -LiteralPath $destination -File).Count
|
||||
if ($destinationCount -lt $sourceCount) {
|
||||
throw "Diagnostic install verification failed: $destinationCount < $sourceCount"
|
||||
}
|
||||
|
||||
Write-Output "Installed standalone diagnostics: $destination"
|
||||
Write-Output 'Restart ComfyUI before using the diagnostic node.'
|
||||
+283
-12
@@ -24,14 +24,21 @@ from ..driving_audio import (
|
||||
)
|
||||
from ..model_patch import clone_model_for_chunk
|
||||
from ..state import assert_context_unchanged, context_fingerprint, make_plan, select_context, validate_state
|
||||
from ..temporal import align_frame_count_up, largest_context_capacity, make_extension_shape
|
||||
from ..temporal import (
|
||||
align_frame_count_up,
|
||||
audio_latent_t,
|
||||
context_slots,
|
||||
largest_context_capacity,
|
||||
make_extension_shape,
|
||||
video_latent_t,
|
||||
)
|
||||
from ..version import PACKAGE_VERSION
|
||||
from .decoder import decode_sequence, decode_sequence_with_seam, enforce_total_frames
|
||||
from .context_diagnostics import ContextDiagnosticsTracker
|
||||
from .h3_builder import attach_keyframes, empty_h3_latent, encode_identity_latents, encode_prompt_conditioning, prepare_identity_assets
|
||||
from .motion import choose_context_frames
|
||||
from .prompts import prompt_plan_report, validate_prompt_plan
|
||||
from .sampling import sample_chunk
|
||||
from .sampling import latent_from_cpu, latent_to_cpu, sample_chunk
|
||||
from .seeds import derive_chunk_seed
|
||||
from .session import (
|
||||
SessionValidationError, entry_to_state, make_chunk_entry, make_session, model_fingerprint,
|
||||
@@ -62,7 +69,7 @@ def _check_decode_memory_budget(*,width,height,chunks,chunk_seconds):
|
||||
def _clone_entry_for_reuse(entry):
|
||||
entry=validate_chunk_entry(entry); result=dict(entry); result["plan"]=copy.deepcopy(entry["plan"]); result["reused"]=True; return result
|
||||
|
||||
def _preserved_prefix(*,session,prompt_hashes,chunks,reroll_from_chunk,width,height,chunk_seconds,identity_hash):
|
||||
def _preserved_prefix(*,session,prompt_hashes,chunks,reroll_from_chunk,width,height,chunk_seconds,identity_hash,last_frame_hash):
|
||||
if session is None: return [],[]
|
||||
notes=[]
|
||||
try: session=validate_session(session)
|
||||
@@ -70,6 +77,8 @@ def _preserved_prefix(*,session,prompt_hashes,chunks,reroll_from_chunk,width,hei
|
||||
if int(session["width"])!=int(width) or int(session["height"])!=int(height): return [],["saved session resolution differs; generated a fresh run"]
|
||||
if abs(float(session["chunk_seconds"])-float(chunk_seconds))>1e-6: return [],["saved session chunk duration differs; generated a fresh run"]
|
||||
if str(session.get("identity_hash","none"))!=str(identity_hash): return [],["saved session identity differs; generated a fresh run"]
|
||||
saved_last_frame_hash=(session.get("settings") or {}).get("last_frame_hash")
|
||||
if str(saved_last_frame_hash or "none")!=str(last_frame_hash or "none"): return [],["saved session Last Frame differs; generated a fresh run"]
|
||||
if reroll_from_chunk<0 or reroll_from_chunk>chunks: raise SessionValidationError("reroll_from_chunk must be 0 or a valid one-based chunk index")
|
||||
limit=min(len(session["chunks"]),chunks)
|
||||
if reroll_from_chunk>0: limit=min(limit,reroll_from_chunk-1)
|
||||
@@ -103,7 +112,12 @@ def _conditioning_cache(*,clip,prompts,assets,final_has_last_frame,reference_ass
|
||||
return cache
|
||||
|
||||
|
||||
FLF_STRATEGY = "context_bound_terminal_v1"
|
||||
FLF_STRATEGY = "terminal_merged_10s_seed_v2"
|
||||
TERMINAL_SEED_POLICY = "physical_window_seed_v2"
|
||||
TERMINAL_MERGE_CHUNK_SECONDS = 5.0
|
||||
TERMINAL_MERGE_CONTEXT_FRAMES = 22
|
||||
TERMINAL_PROMPT_POLICY_SHARED = "shared_prompt_v1"
|
||||
TERMINAL_PROMPT_POLICY_TIMELINE = "paired_timeline_v1"
|
||||
|
||||
|
||||
def _flf_contract_identity(base_hash: str) -> str:
|
||||
@@ -111,6 +125,191 @@ def _flf_contract_identity(base_hash: str) -> str:
|
||||
return hashlib.sha256(payload.encode("utf-8")).hexdigest()
|
||||
|
||||
|
||||
def _terminal_flf_merge_enabled(
|
||||
*,
|
||||
multi_chunk_flf: bool,
|
||||
chunks: int,
|
||||
chunk_seconds: float,
|
||||
prompt_hashes: list[str],
|
||||
timeline_video_source,
|
||||
) -> bool:
|
||||
return bool(
|
||||
multi_chunk_flf
|
||||
and int(chunks) >= 2
|
||||
and abs(float(chunk_seconds) - TERMINAL_MERGE_CHUNK_SECONDS) <= 1e-6
|
||||
and timeline_video_source is None
|
||||
and len(prompt_hashes) >= 2
|
||||
)
|
||||
|
||||
|
||||
def _terminal_pair_prompt(
|
||||
prompts: list[str],
|
||||
*,
|
||||
pair_start: int,
|
||||
chunk_seconds: float,
|
||||
) -> tuple[str, str]:
|
||||
"""Build one physical prompt without changing either logical prompt."""
|
||||
|
||||
first = str(prompts[int(pair_start)])
|
||||
second = str(prompts[int(pair_start) + 1])
|
||||
if first == second:
|
||||
return first, TERMINAL_PROMPT_POLICY_SHARED
|
||||
split = f"{float(chunk_seconds):g}"
|
||||
stop = f"{float(chunk_seconds) * 2.0:g}"
|
||||
return (
|
||||
f"[0-{split}s]\n{first}\n\n[{split}-{stop}s]\n{second}",
|
||||
TERMINAL_PROMPT_POLICY_TIMELINE,
|
||||
)
|
||||
|
||||
|
||||
def _atomic_terminal_prefix(
|
||||
preserved: list[dict[str, Any]],
|
||||
*,
|
||||
chunks: int,
|
||||
reroll_from_chunk: int,
|
||||
) -> tuple[list[dict[str, Any]], bool]:
|
||||
"""Never reuse only one logical half of a shared terminal sample."""
|
||||
pair_start = int(chunks) - 2
|
||||
reroll_hits_pair = int(reroll_from_chunk) in (int(chunks) - 1, int(chunks))
|
||||
partial_pair = pair_start < len(preserved) < int(chunks)
|
||||
if (reroll_hits_pair or partial_pair) and len(preserved) > pair_start:
|
||||
return preserved[:pair_start], True
|
||||
return preserved, False
|
||||
|
||||
|
||||
def _terminal_sampling_plan(
|
||||
*,
|
||||
chunks: int,
|
||||
completed: int,
|
||||
merge_enabled: bool,
|
||||
) -> tuple[tuple[int, ...], bool]:
|
||||
if completed >= int(chunks):
|
||||
return (), False
|
||||
if not merge_enabled:
|
||||
return tuple(range(completed, int(chunks))), False
|
||||
pair_start = int(chunks) - 2
|
||||
if completed > pair_start:
|
||||
raise SequenceRuntimeError("terminal merged pair cannot resume from only one logical half")
|
||||
return tuple(range(completed, pair_start)), True
|
||||
|
||||
|
||||
def _terminal_physical_seed_plan(
|
||||
*,
|
||||
base_seed: int,
|
||||
physical_window_start_index: int,
|
||||
reroll_nonce: int,
|
||||
) -> dict[str, Any]:
|
||||
"""Return the one physical seed and the matching logical-entry metadata."""
|
||||
start_index = int(physical_window_start_index)
|
||||
nonce = int(reroll_nonce)
|
||||
if start_index < 0:
|
||||
raise ValueError("physical_window_start_index must be non-negative")
|
||||
if nonce < 0:
|
||||
raise ValueError("reroll_nonce must be non-negative")
|
||||
if start_index == 0 and nonce == 0:
|
||||
physical_seed = int(base_seed)
|
||||
else:
|
||||
physical_seed = derive_chunk_seed(base_seed, start_index, nonce)
|
||||
return {
|
||||
"physical_seed": physical_seed,
|
||||
"logical_entry_seeds": (physical_seed, physical_seed),
|
||||
}
|
||||
|
||||
|
||||
def _terminal_execution_semantics(
|
||||
*,
|
||||
merge_enabled: bool,
|
||||
prompt_policy: str | None = None,
|
||||
) -> dict[str, str] | None:
|
||||
if not merge_enabled:
|
||||
return None
|
||||
semantics = {
|
||||
"flf_execution": FLF_STRATEGY,
|
||||
"terminal_seed_policy": TERMINAL_SEED_POLICY,
|
||||
}
|
||||
if prompt_policy:
|
||||
semantics["terminal_prompt_policy"] = str(prompt_policy)
|
||||
return semantics
|
||||
|
||||
|
||||
def _terminal_pair_contract(*, initial_pair: bool, chunk_seconds: float) -> dict[str, Any]:
|
||||
"""Derive physical and logical AV ranges without fixed latent indices."""
|
||||
if abs(float(chunk_seconds) - TERMINAL_MERGE_CHUNK_SECONDS) > 1e-6:
|
||||
raise SequenceRuntimeError("terminal merged sampling currently requires 5-second logical chunks")
|
||||
initial_frames = align_frame_count_up(int(round(float(chunk_seconds) * FPS)))
|
||||
extension = make_extension_shape(
|
||||
TERMINAL_MERGE_CONTEXT_FRAMES,
|
||||
float(chunk_seconds),
|
||||
)
|
||||
if initial_pair:
|
||||
physical_frames = align_frame_count_up(int(round(2.0 * float(chunk_seconds) * FPS)))
|
||||
logical_frames = (initial_frames, extension.total_frames)
|
||||
logical_trims = (0, TERMINAL_MERGE_CONTEXT_FRAMES)
|
||||
first_video_stop = video_latent_t(initial_frames)
|
||||
first_audio_stop = audio_latent_t(initial_frames)
|
||||
else:
|
||||
physical_frames = make_extension_shape(
|
||||
TERMINAL_MERGE_CONTEXT_FRAMES,
|
||||
2.0 * float(chunk_seconds),
|
||||
).total_frames
|
||||
logical_frames = (extension.total_frames, extension.total_frames)
|
||||
logical_trims = (
|
||||
TERMINAL_MERGE_CONTEXT_FRAMES,
|
||||
TERMINAL_MERGE_CONTEXT_FRAMES,
|
||||
)
|
||||
first_video_stop = video_latent_t(extension.total_frames)
|
||||
first_audio_stop = audio_latent_t(extension.total_frames)
|
||||
physical_video_stop = video_latent_t(physical_frames)
|
||||
physical_audio_stop = audio_latent_t(physical_frames)
|
||||
video_overlap = context_slots(TERMINAL_MERGE_CONTEXT_FRAMES)
|
||||
audio_overlap = audio_latent_t(TERMINAL_MERGE_CONTEXT_FRAMES)
|
||||
video_slices = (
|
||||
(0, first_video_stop),
|
||||
(first_video_stop - video_overlap, physical_video_stop),
|
||||
)
|
||||
audio_slices = (
|
||||
(0, first_audio_stop),
|
||||
(first_audio_stop - audio_overlap, physical_audio_stop),
|
||||
)
|
||||
for index in range(2):
|
||||
expected_video = video_latent_t(logical_frames[index])
|
||||
expected_audio = audio_latent_t(logical_frames[index])
|
||||
if video_slices[index][1] - video_slices[index][0] != expected_video:
|
||||
raise SequenceRuntimeError("derived terminal video split does not match its logical frame count")
|
||||
if audio_slices[index][1] - audio_slices[index][0] != expected_audio:
|
||||
raise SequenceRuntimeError("derived terminal audio split does not match its logical frame count")
|
||||
return {
|
||||
"initial_pair": bool(initial_pair),
|
||||
"physical_frames": int(physical_frames),
|
||||
"physical_context_frames": 0 if initial_pair else TERMINAL_MERGE_CONTEXT_FRAMES,
|
||||
"logical_frames": logical_frames,
|
||||
"logical_trims": logical_trims,
|
||||
"video_slices": video_slices,
|
||||
"audio_slices": audio_slices,
|
||||
}
|
||||
|
||||
|
||||
def _split_terminal_merged_latents(
|
||||
video: torch.Tensor,
|
||||
audio: torch.Tensor,
|
||||
contract: dict[str, Any],
|
||||
) -> tuple[tuple[torch.Tensor, torch.Tensor], tuple[torch.Tensor, torch.Tensor]]:
|
||||
expected_video = video_latent_t(int(contract["physical_frames"]))
|
||||
expected_audio = audio_latent_t(int(contract["physical_frames"]))
|
||||
if int(video.shape[2]) != expected_video or int(audio.shape[-1]) != expected_audio:
|
||||
raise SequenceRuntimeError(
|
||||
"terminal merged sampler output does not match the derived physical AV grid: "
|
||||
f"video T={video.shape[2]} (expected {expected_video}), "
|
||||
f"audio T={audio.shape[-1]} (expected {expected_audio})"
|
||||
)
|
||||
parts=[]
|
||||
for video_range,audio_range in zip(contract["video_slices"],contract["audio_slices"]):
|
||||
video_part=video[:,:,video_range[0]:video_range[1],...].contiguous()
|
||||
audio_part=audio[...,audio_range[0]:audio_range[1]].contiguous()
|
||||
parts.append((video_part,audio_part))
|
||||
return parts[0],parts[1]
|
||||
|
||||
|
||||
def _attach_terminal_flf_keyframes(
|
||||
conditioning,
|
||||
*,
|
||||
@@ -141,7 +340,7 @@ def _attach_terminal_flf_keyframes(
|
||||
return conditioning
|
||||
|
||||
def run_sequence(*,model:Any,clip:Any,video_vae:Any,audio_vae:Any,sampler:Any,sigmas:torch.Tensor,first_frame:torch.Tensor|None,last_frame:torch.Tensor|None,prompt_plan:dict[str,Any],width:int,height:int,continuity:str,base_seed:int,audio_continuity:bool,exact_total_duration:bool,diagnostics_mode:str,reroll_from_chunk:int,reroll_nonce:int,strict_compatibility:bool,debug:bool,seam_correction:str=SEAM_CORRECTION_OFF,enable_preview:bool=True,session:dict[str,Any]|None=None,initial_state:dict[str,Any]|None=None,latent_only:bool=False,reference_assets=None,reference_audio_source=None,reference_audio_vae=None,driving_audio_source=None,driving_audio_vae=None,reference_video_source=None,timeline_video_source=None):
|
||||
from ..conditioning import detect_conditioning_mode, conditioning_mode_label
|
||||
from ..conditioning import detect_conditioning_mode, conditioning_display_label
|
||||
from ..run_storage import get_active_run_storage
|
||||
storage_controller=get_active_run_storage()
|
||||
diagnostics_mode=normalize_diagnostics_mode(diagnostics_mode); plan=validate_prompt_plan(prompt_plan); chunks=int(plan["chunks"]); chunk_seconds=float(plan["chunk_seconds"]); prompts=list(plan["prompts"]); prompt_hashes=list(plan["hashes"]); width,height=int(width),int(height)
|
||||
@@ -157,6 +356,7 @@ def run_sequence(*,model:Any,clip:Any,video_vae:Any,audio_vae:Any,sampler:Any,si
|
||||
if initial_state is not None and int(reroll_from_chunk) not in (0,1): raise SequenceRuntimeError("with initial_state, reroll_from_chunk can only be 0 or 1")
|
||||
try: conditioning_mode=detect_conditioning_mode(first_frame=first_frame,last_frame=last_frame,reference_assets=reference_assets)
|
||||
except ValueError as exc: raise SequenceRuntimeError(str(exc)) from exc
|
||||
conditioning_display=conditioning_display_label(has_first=first_frame is not None,has_last=last_frame is not None,has_reference=reference_assets is not None)
|
||||
reference_warning=""
|
||||
if reference_assets is not None:
|
||||
from ..reference import validate_reference_prompts
|
||||
@@ -181,6 +381,11 @@ def run_sequence(*,model:Any,clip:Any,video_vae:Any,audio_vae:Any,sampler:Any,si
|
||||
issues=check_comfy_h3_runtime()
|
||||
assets=prepare_identity_assets(video_vae,width=width,height=height,first_frame=first_frame,last_frame=last_frame,encode_latents=False)
|
||||
multi_chunk_flf=bool(chunks>=2 and first_frame is not None and last_frame is not None)
|
||||
terminal_merge_enabled=_terminal_flf_merge_enabled(multi_chunk_flf=multi_chunk_flf,chunks=chunks,chunk_seconds=chunk_seconds,prompt_hashes=prompt_hashes,timeline_video_source=timeline_video_source)
|
||||
terminal_prompt=None
|
||||
terminal_prompt_policy=None
|
||||
if terminal_merge_enabled:
|
||||
terminal_prompt,terminal_prompt_policy=_terminal_pair_prompt(prompts,pair_start=chunks-2,chunk_seconds=chunk_seconds)
|
||||
from ..reference import combine_hybrid_visual_identity
|
||||
visual_identity_hash=combine_hybrid_visual_identity(keyframe_identity_hash=assets.identity_hash,reference_assets=reference_assets,has_first=first_frame is not None,has_last=last_frame is not None)
|
||||
sequence_identity_hash=combine_reference_audio_identity(visual_identity_hash,reference_audio_source)
|
||||
@@ -191,20 +396,27 @@ def run_sequence(*,model:Any,clip:Any,video_vae:Any,audio_vae:Any,sampler:Any,si
|
||||
sequence_identity_hash=_flf_contract_identity(sequence_identity_hash)
|
||||
current_model_fingerprint=model_fingerprint(model,extra_wrapper_keys=("h3_continuum_join.apply_model.v1",))
|
||||
if storage_controller is not None:
|
||||
stored_session=storage_controller.prepare(model=model,model_fingerprint_value=current_model_fingerprint,clip=clip,video_vae=video_vae,sampler=sampler,sigmas=sigmas,prompt_plan=plan,width=width,height=height,chunk_seconds=chunk_seconds,continuity=continuity,audio_continuity=audio_continuity,base_seed=base_seed,reroll_from_chunk=reroll_from_chunk,reroll_nonce=reroll_nonce,first_frame_hash=assets.first_frame_hash,last_frame_hash=assets.last_frame_hash,identity_hash=sequence_identity_hash,strict_compatibility=strict_compatibility,existing_session=session,reference_contract=reference_assets.contract if reference_assets is not None else None,conditioning_mode=conditioning_mode,reference_audio_contract=reference_audio_source.contract if reference_audio_source is not None else None,reference_audio_vae=reference_audio_vae,driving_audio_contract=driving_audio_source.contract if driving_audio_source is not None else None,driving_audio_vae=driving_audio_vae,reference_video_contract=reference_video_source.contract if reference_video_source is not None else None,timeline_video_contract=timeline_video_source.contract if timeline_video_source is not None else None)
|
||||
stored_session=storage_controller.prepare(model=model,model_fingerprint_value=current_model_fingerprint,clip=clip,video_vae=video_vae,sampler=sampler,sigmas=sigmas,prompt_plan=plan,width=width,height=height,chunk_seconds=chunk_seconds,continuity=continuity,audio_continuity=audio_continuity,base_seed=base_seed,reroll_from_chunk=reroll_from_chunk,reroll_nonce=reroll_nonce,first_frame_hash=assets.first_frame_hash,last_frame_hash=assets.last_frame_hash,identity_hash=sequence_identity_hash,strict_compatibility=strict_compatibility,existing_session=session,reference_contract=reference_assets.contract if reference_assets is not None else None,conditioning_mode=conditioning_mode,reference_audio_contract=reference_audio_source.contract if reference_audio_source is not None else None,reference_audio_vae=reference_audio_vae,driving_audio_contract=driving_audio_source.contract if driving_audio_source is not None else None,driving_audio_vae=driving_audio_vae,reference_video_contract=reference_video_source.contract if reference_video_source is not None else None,timeline_video_contract=timeline_video_source.contract if timeline_video_source is not None else None,execution_semantics=_terminal_execution_semantics(merge_enabled=terminal_merge_enabled,prompt_policy=terminal_prompt_policy))
|
||||
reroll_nonce=storage_controller.effective_reroll_nonce
|
||||
if stored_session is not None: session=stored_session
|
||||
accelerators=accelerator_summary(model)
|
||||
if "Continuum APPLY_MODEL wrapper installed" not in accelerators: accelerators+="; Continuum APPLY_MODEL wrapper installed"
|
||||
effective_reroll_from_chunk=0 if storage_controller is not None and session is not None and bool((session.get("settings") or {}).get("run_storage_validated_prefix")) else int(reroll_from_chunk)
|
||||
preserved,reuse_notes=_preserved_prefix(session=session,prompt_hashes=prompt_hashes,chunks=chunks,reroll_from_chunk=effective_reroll_from_chunk,width=width,height=height,chunk_seconds=chunk_seconds,identity_hash=sequence_identity_hash)
|
||||
preserved,reuse_notes=_preserved_prefix(session=session,prompt_hashes=prompt_hashes,chunks=chunks,reroll_from_chunk=effective_reroll_from_chunk,width=width,height=height,chunk_seconds=chunk_seconds,identity_hash=sequence_identity_hash,last_frame_hash=assets.last_frame_hash)
|
||||
if multi_chunk_flf and session is not None:
|
||||
saved_strategy=str((session.get("settings") or {}).get("flf_execution", ""))
|
||||
if saved_strategy!=FLF_STRATEGY:
|
||||
preserved=[]
|
||||
reuse_notes.append("FLF chunk contract changed; regenerated the requested FLF sequence.")
|
||||
if terminal_merge_enabled:
|
||||
preserved,atomic_reset=_atomic_terminal_prefix(preserved,chunks=chunks,reroll_from_chunk=int(reroll_from_chunk))
|
||||
if atomic_reset:
|
||||
reuse_notes.append("terminal merged pair is atomic; regenerated both final logical chunks")
|
||||
if multi_chunk_flf:
|
||||
reuse_notes.insert(0,"FLF execution: context-bound terminal v1; one sampling pass per chunk; the final chunk is anchored from the continuation-context tail to the supplied Last Frame.")
|
||||
if terminal_merge_enabled:
|
||||
reuse_notes.insert(0,"FLF execution: terminal merged 10s seed v2; the final two 5-second logical chunks share one physical sample and are split back into normal chunk entries.")
|
||||
else:
|
||||
reuse_notes.insert(0,"FLF execution: terminal merge unavailable for this request; retained one sampling pass per logical chunk.")
|
||||
if reference_assets is not None:
|
||||
reuse_notes.insert(0,f"Reference conditioning: {reference_assets.count} image(s), size={reference_assets.size_mode}; persistent across all chunks.")
|
||||
if reference_warning: reuse_notes.append(reference_warning)
|
||||
@@ -239,6 +451,10 @@ def run_sequence(*,model:Any,clip:Any,video_vae:Any,audio_vae:Any,sampler:Any,si
|
||||
reference_video_assets=encode_reference_video(video_vae,reference_video_source)
|
||||
if timeline_video_source is None:
|
||||
cache=_conditioning_cache(clip=clip,prompts=prompts,assets=assets,final_has_last_frame=last_frame is not None,reference_assets=reference_assets,reference_audio_assets=reference_audio_assets,timeline_video_assets=reference_video_assets)
|
||||
if terminal_merge_enabled and terminal_prompt is not None:
|
||||
terminal_key=_conditioning_cache_key(terminal_prompt,include_last=True,reference_assets=reference_assets)
|
||||
if terminal_key not in cache:
|
||||
cache.update(_conditioning_cache(clip=clip,prompts=[terminal_prompt],assets=assets,final_has_last_frame=True,reference_assets=reference_assets,reference_audio_assets=reference_audio_assets,timeline_video_assets=reference_video_assets))
|
||||
entries=preserved[:]; previous_state=None
|
||||
if entries:
|
||||
try: previous_state=entry_to_state(entries[-1])
|
||||
@@ -261,7 +477,8 @@ def run_sequence(*,model:Any,clip:Any,video_vae:Any,audio_vae:Any,sampler:Any,si
|
||||
_record_context_diagnostics(tracker=context_diagnostics,reports=sampling_reports,state=entry_to_state(reused_entry),continuity=continuity,reused=True)
|
||||
elif previous_state is not None:
|
||||
_record_context_diagnostics(tracker=context_diagnostics,reports=sampling_reports,state=previous_state,continuity=continuity,reused=True)
|
||||
for sequence_index in range(len(entries),chunks):
|
||||
normal_indices,terminal_merge_pending=_terminal_sampling_plan(chunks=chunks,completed=len(entries),merge_enabled=terminal_merge_enabled)
|
||||
for sequence_index in normal_indices:
|
||||
prompt=prompts[sequence_index]; prompt_hash_value=prompt_hashes[sequence_index]; is_final=sequence_index==chunks-1; effective_reroll_nonce=int(reroll_nonce) if int(reroll_from_chunk)>0 and sequence_index+1>=int(reroll_from_chunk) else 0; seed=derive_chunk_seed(base_seed,sequence_index,effective_reroll_nonce); motion_score=0.0; video_context=None; audio_context=None; context_before=None
|
||||
timeline_video_assets=reference_video_assets
|
||||
chunk_cache=cache
|
||||
@@ -283,8 +500,6 @@ def run_sequence(*,model:Any,clip:Any,video_vae:Any,audio_vae:Any,sampler:Any,si
|
||||
base_conditioning=attach_keyframes(chunk_cache[conditioning_key],frame_count=shape.total_frames,first_latent=chunk_assets.first_latent,last_latent=chunk_assets.last_latent if include_chunk_last else None)
|
||||
video_context,audio_context,grid_offset=select_context(previous_state,context_frames,include_audio=bool(audio_continuity)); context_before=context_fingerprint(video_context,audio_context)
|
||||
conditioning=prepare_conditioning(base_conditioning,video_context=video_context,audio_context=audio_context,audio_grid_offset=grid_offset,context_frames=context_frames,new_frame_count=shape.total_frames,first_frame_policy=POLICY_REPLACE,preserve_last_frame=True)
|
||||
if multi_chunk_flf and is_final:
|
||||
conditioning=_attach_terminal_flf_keyframes(conditioning,video_context=video_context,context_frames=context_frames,frame_count=shape.total_frames,last_latent=assets.last_latent)
|
||||
clip_index=int(previous_state["clip_index"])+1; chunk_plan=make_plan(continuation=True,clip_index=clip_index,total_frames=shape.total_frames,trim_frames=context_frames,width=width,height=height,context_frames=context_frames,state_capacity_frames=largest_context_capacity(shape.net_new_frames),requested_extend_seconds=chunk_seconds,debug=debug)
|
||||
driving_audio_latent=slice_driving_audio_latent(driving_audio_assets,cumulative_retained_before=retained_frames,total_frames=int(chunk_plan["total_frames"]),trim_frames=int(chunk_plan["trim_frames"]),fps=FPS)
|
||||
conditioning=attach_driving_audio(conditioning,driving_audio_latent)
|
||||
@@ -298,6 +513,62 @@ def run_sequence(*,model:Any,clip:Any,video_vae:Any,audio_vae:Any,sampler:Any,si
|
||||
sampling_reports.append(f"chunk {sequence_index+1}/{chunks}: seed={seed}, frames={chunk_plan['total_frames']}, trim={chunk_plan['trim_frames']}, retained_total={retained_frames}, context={context_frames} ({reason}), motion={motion_score:.6f}, "+(f"interop=emitted actual_prefix={CONTINUUM_ACTUAL_PREFIX_STEPS} consumer=not_observable" if context_before is not None else "interop=not_emitted"))
|
||||
del sampled,latent,conditioning,chunk_model,chunk_cache
|
||||
if timeline_video_source is not None and timeline_video_assets is not None: del timeline_video_assets
|
||||
if terminal_merge_pending:
|
||||
pair_start=chunks-2
|
||||
if len(entries)!=pair_start:
|
||||
raise SequenceRuntimeError(f"terminal merged pair expected {pair_start} completed chunks, got {len(entries)}")
|
||||
prompt=terminal_prompt
|
||||
seed_nonce=int(reroll_nonce) if int(reroll_from_chunk)>0 else 0
|
||||
terminal_seed_plan=_terminal_physical_seed_plan(base_seed=base_seed,physical_window_start_index=pair_start,reroll_nonce=seed_nonce)
|
||||
physical_seed=int(terminal_seed_plan["physical_seed"])
|
||||
initial_pair=previous_state is None
|
||||
contract=_terminal_pair_contract(initial_pair=initial_pair,chunk_seconds=chunk_seconds)
|
||||
physical_frames=int(contract["physical_frames"])
|
||||
physical_context_frames=int(contract["physical_context_frames"])
|
||||
latent=empty_h3_latent(width,height,physical_frames)
|
||||
conditioning_key=_conditioning_cache_key(prompt,include_last=True,reference_assets=reference_assets)
|
||||
base_conditioning=attach_keyframes(cache[conditioning_key],frame_count=physical_frames,first_latent=assets.first_latent,last_latent=assets.last_latent)
|
||||
video_context=None; audio_context=None; context_before=None; motion_score=0.0
|
||||
if initial_pair:
|
||||
conditioning=base_conditioning
|
||||
physical_clip_index=1
|
||||
reason="terminal merged initial 10-second sample"
|
||||
else:
|
||||
selected_context,motion_score,selected_reason=choose_context_frames(continuity,previous_state)
|
||||
video_context,audio_context,grid_offset=select_context(previous_state,TERMINAL_MERGE_CONTEXT_FRAMES,include_audio=bool(audio_continuity))
|
||||
context_before=context_fingerprint(video_context,audio_context)
|
||||
conditioning=prepare_conditioning(base_conditioning,video_context=video_context,audio_context=audio_context,audio_grid_offset=grid_offset,context_frames=TERMINAL_MERGE_CONTEXT_FRAMES,new_frame_count=physical_frames,first_frame_policy=POLICY_REPLACE,preserve_last_frame=True)
|
||||
physical_clip_index=int(previous_state["clip_index"])+1
|
||||
reason=f"terminal merged 10-second sample with 22-frame context; continuity selection was {selected_context} ({selected_reason})"
|
||||
driving_audio_latent=slice_driving_audio_latent(driving_audio_assets,cumulative_retained_before=retained_frames,total_frames=physical_frames,trim_frames=physical_context_frames,fps=FPS)
|
||||
conditioning=attach_driving_audio(conditioning,driving_audio_latent)
|
||||
chunk_model=clone_model_for_chunk(model,strict=bool(strict_compatibility),debug=bool(debug),chunk_index=physical_clip_index,context_frames=physical_context_frames if not initial_pair else None)
|
||||
sampled=sample_chunk(model=chunk_model,conditioning=conditioning,latent=latent,sampler=sampler,sigmas=sigmas,seed=physical_seed,enable_preview=bool(enable_preview))
|
||||
if context_before is not None and video_context is not None:
|
||||
assert_context_unchanged(video_context,audio_context,context_before)
|
||||
sampled_video,sampled_audio=latent_to_cpu(sampled)
|
||||
logical_parts=_split_terminal_merged_latents(sampled_video,sampled_audio,contract)
|
||||
sampling_reports.append(f"terminal physical sample: terminal_pair={pair_start+1}-{chunks}, physical_seed={physical_seed}, physical_frames={physical_frames}, sampling_passes=1, shared_physical_sample=true, terminal_prompt_policy={terminal_prompt_policy}, trim={physical_context_frames}, reason={reason}")
|
||||
for offset,(video_part,audio_part) in enumerate(logical_parts):
|
||||
sequence_index=pair_start+offset
|
||||
total_frames=int(contract["logical_frames"][offset])
|
||||
trim_frames=int(contract["logical_trims"][offset])
|
||||
continuation=bool(trim_frames)
|
||||
clip_index=physical_clip_index+offset
|
||||
context_frames=trim_frames if continuation else 5
|
||||
chunk_plan=make_plan(continuation=continuation,clip_index=clip_index,total_frames=total_frames,trim_frames=trim_frames,width=width,height=height,context_frames=context_frames,state_capacity_frames=largest_context_capacity(total_frames-trim_frames),requested_extend_seconds=chunk_seconds,debug=debug)
|
||||
logical_latent=latent_from_cpu(video_part,audio_part)
|
||||
logical_seed=int(terminal_seed_plan["logical_entry_seeds"][offset])
|
||||
entry=make_chunk_entry(latent=logical_latent,plan=chunk_plan,prompt=prompts[sequence_index],prompt_hash=prompt_hashes[sequence_index],seed=logical_seed,context_frames=trim_frames,motion_score=motion_score,reused=False)
|
||||
entries.append(entry)
|
||||
previous_state=entry_to_state(entry)
|
||||
_record_context_diagnostics(tracker=context_diagnostics,reports=sampling_reports,state=previous_state,continuity=continuity,reused=False)
|
||||
if storage_controller is not None:
|
||||
storage_controller.commit_chunk(entry,position=sequence_index)
|
||||
retained_frames+=int(chunk_plan["net_frames"])
|
||||
sampling_reports.append(f"chunk {sequence_index+1}/{chunks}: seed={logical_seed}, frames={total_frames}, trim={trim_frames}, retained_total={retained_frames}, context={trim_frames}, shared_physical_sample=terminal_10s_seed_v2")
|
||||
del logical_latent
|
||||
del sampled,sampled_video,sampled_audio,logical_parts,latent,conditioning,chunk_model
|
||||
if len(entries)!=chunks: raise SequenceRuntimeError(f"internal sequence length mismatch: expected {chunks}, got {len(entries)}")
|
||||
if latent_only:
|
||||
last_state=entry_to_state(entries[-1]); parent_id=session.get("session_id") if session is not None else None
|
||||
@@ -308,7 +579,7 @@ def run_sequence(*,model:Any,clip:Any,video_vae:Any,audio_vae:Any,sampler:Any,si
|
||||
if reference_video_source is not None: settings["reference_video_contract"]=reference_video_source.contract
|
||||
if timeline_video_source is not None: settings["timeline_video_contract"]=timeline_video_source.contract
|
||||
new_session=make_session(chunks=entries,width=width,height=height,chunk_seconds=chunk_seconds,identity_hash=sequence_identity_hash,model_fingerprint_value=current_model_fingerprint,parent_session_id=parent_id,reroll_from_chunk=int(reroll_from_chunk),settings=settings)
|
||||
report_lines=[f"H3 Continuum V3 {PACKAGE_VERSION}",f"Conditioning mode: {conditioning_mode_label(conditioning_mode)}.",prompt_plan_report(plan),"Decode: external ComfyUI Core VAE nodes; full raw AV chunks retained.",accelerators,"Execution: conditioning precomputed; single call-local MODEL clone per chunk; no internal VAE decode.",*reuse_notes]
|
||||
report_lines=[f"H3 Continuum V3 {PACKAGE_VERSION}",f"Conditioning mode: {conditioning_display}.",prompt_plan_report(plan),"Decode: external ComfyUI Core VAE nodes; full raw AV chunks retained.",accelerators,"Execution: conditioning precomputed; single call-local MODEL clone per chunk; no internal VAE decode.",*reuse_notes]
|
||||
if diagnostics_mode!=DIAGNOSTICS_OFF: report_lines.extend(sampling_reports)
|
||||
report_lines.extend([session_summary(new_session),f"Output: {len(entries)} raw AV latent chunk(s); connect Core VAE Decode nodes, then H3 Continuum Assemble V3."])
|
||||
return entries,last_state,new_session,"\n".join(report_lines)
|
||||
|
||||
+3
-1
@@ -71,7 +71,9 @@ def assemble_decoded_chunks(
|
||||
if audio_seam not in AUDIO_SEAM_OPTIONS:
|
||||
raise ValueError(f"unknown Audio Seam mode: {audio_seam!r}")
|
||||
|
||||
chunk_plans = list(plan["chunks"])
|
||||
chunk_plans = list(
|
||||
plan["decode_groups"] if "decode_groups" in plan else plan["chunks"]
|
||||
)
|
||||
if len(images) != len(chunk_plans) or len(audio) != len(chunk_plans):
|
||||
raise ValueError(
|
||||
"decoded chunk count does not match the V3 assembly plan: "
|
||||
|
||||
+23
-3
@@ -338,13 +338,33 @@ class H3ContinuumSamplerV3:
|
||||
**clip_prompt_inputs,
|
||||
)
|
||||
|
||||
assembly_plan = make_assembly_plan(
|
||||
from ..v2.sequence import _terminal_flf_merge_enabled
|
||||
from .plan import prepare_physical_decode_entries
|
||||
|
||||
terminal_merged = _terminal_flf_merge_enabled(
|
||||
multi_chunk_flf=(
|
||||
first_frame is not None
|
||||
and advanced_values["last_frame"] is not None
|
||||
and len(entries) > 1
|
||||
),
|
||||
chunks=len(entries),
|
||||
chunk_seconds=float(chunk_seconds),
|
||||
prompt_hashes=[str(entry["prompt_hash"]) for entry in entries],
|
||||
timeline_video_source=timeline_video_source,
|
||||
)
|
||||
decode_entries, assembly_plan = prepare_physical_decode_entries(
|
||||
entries,
|
||||
chunk_seconds=float(chunk_seconds),
|
||||
preserve_final_frame=advanced_values["last_frame"] is not None,
|
||||
terminal_merged=terminal_merged,
|
||||
)
|
||||
video_latents = [{"samples": entry["video"]} for entry in entries]
|
||||
audio_latents = [{"samples": entry["audio"]} for entry in entries]
|
||||
video_latents = [{"samples": entry["video"]} for entry in decode_entries]
|
||||
audio_latents = [{"samples": entry["audio"]} for entry in decode_entries]
|
||||
if terminal_merged:
|
||||
report = str(report).rstrip() + (
|
||||
"\nDecode: terminal merged latent retained as one physical external "
|
||||
"Core VAE decode group; logical Run Storage entries are unchanged."
|
||||
)
|
||||
result = {
|
||||
"last_state": last_state,
|
||||
"session": session,
|
||||
|
||||
+148
@@ -141,3 +141,151 @@ def make_assembly_plan(*args, **kwargs):
|
||||
_enrich_assembly_plan(plan)
|
||||
validate_assembly_plan(plan)
|
||||
return plan
|
||||
|
||||
|
||||
def _recombine_terminal_tensor(
|
||||
first,
|
||||
second,
|
||||
*,
|
||||
slices,
|
||||
time_dim: int,
|
||||
name: str,
|
||||
):
|
||||
"""Invert the terminal logical split without changing any tensor value."""
|
||||
|
||||
import torch
|
||||
|
||||
if not torch.is_tensor(first) or not torch.is_tensor(second):
|
||||
raise ValueError(f"terminal {name} entries must be tensors")
|
||||
if first.ndim != second.ndim:
|
||||
raise ValueError(f"terminal {name} tensor ranks differ")
|
||||
dim = int(time_dim)
|
||||
if dim < 0:
|
||||
dim += first.ndim
|
||||
if dim < 0 or dim >= first.ndim:
|
||||
raise ValueError(f"terminal {name} time dimension is invalid")
|
||||
for axis, (left, right) in enumerate(zip(first.shape, second.shape)):
|
||||
if axis != dim and int(left) != int(right):
|
||||
raise ValueError(f"terminal {name} tensor geometry differs")
|
||||
|
||||
(first_start, first_stop), (second_start, second_stop) = slices
|
||||
first_start = int(first_start)
|
||||
first_stop = int(first_stop)
|
||||
second_start = int(second_start)
|
||||
second_stop = int(second_stop)
|
||||
if first_start != 0 or second_start > first_stop or second_stop <= first_stop:
|
||||
raise ValueError(f"terminal {name} split ranges do not form one physical unit")
|
||||
if int(first.shape[dim]) != first_stop - first_start:
|
||||
raise ValueError(f"terminal {name} first slice length is inconsistent")
|
||||
if int(second.shape[dim]) != second_stop - second_start:
|
||||
raise ValueError(f"terminal {name} second slice length is inconsistent")
|
||||
|
||||
overlap_start = max(first_start, second_start)
|
||||
overlap_stop = min(first_stop, second_stop)
|
||||
overlap_length = max(0, overlap_stop - overlap_start)
|
||||
if overlap_length:
|
||||
first_overlap = first.narrow(dim, overlap_start - first_start, overlap_length)
|
||||
second_overlap = second.narrow(dim, overlap_start - second_start, overlap_length)
|
||||
if not torch.equal(first_overlap, second_overlap):
|
||||
raise ValueError(
|
||||
f"terminal {name} overlap differs; refusing lossy physical reconstruction"
|
||||
)
|
||||
|
||||
output_shape = list(first.shape)
|
||||
output_shape[dim] = second_stop
|
||||
output = first.new_empty(output_shape)
|
||||
output.narrow(dim, first_start, first_stop - first_start).copy_(first)
|
||||
output.narrow(dim, second_start, second_stop - second_start).copy_(second)
|
||||
return output.contiguous()
|
||||
|
||||
|
||||
def prepare_physical_decode_entries(
|
||||
entries,
|
||||
*,
|
||||
chunk_seconds: float,
|
||||
preserve_final_frame: bool,
|
||||
terminal_merged: bool,
|
||||
):
|
||||
"""Return external-VAE decode units while preserving logical plan metadata."""
|
||||
|
||||
logical_entries = list(entries)
|
||||
plan = make_assembly_plan(
|
||||
logical_entries,
|
||||
chunk_seconds=float(chunk_seconds),
|
||||
preserve_final_frame=bool(preserve_final_frame),
|
||||
)
|
||||
if not terminal_merged:
|
||||
return logical_entries, plan
|
||||
if len(logical_entries) < 2:
|
||||
raise ValueError("terminal physical decode requires two logical entries")
|
||||
|
||||
from ..v2.sequence import _terminal_pair_contract
|
||||
|
||||
initial_pair = len(logical_entries) == 2
|
||||
contract = _terminal_pair_contract(
|
||||
initial_pair=initial_pair,
|
||||
chunk_seconds=float(chunk_seconds),
|
||||
)
|
||||
first_entry = logical_entries[-2]
|
||||
second_entry = logical_entries[-1]
|
||||
physical_video = _recombine_terminal_tensor(
|
||||
first_entry["video"],
|
||||
second_entry["video"],
|
||||
slices=contract["video_slices"],
|
||||
time_dim=2,
|
||||
name="video",
|
||||
)
|
||||
physical_audio = _recombine_terminal_tensor(
|
||||
first_entry["audio"],
|
||||
second_entry["audio"],
|
||||
slices=contract["audio_slices"],
|
||||
time_dim=-1,
|
||||
name="audio",
|
||||
)
|
||||
|
||||
physical_entry = dict(first_entry)
|
||||
physical_entry["video"] = physical_video
|
||||
physical_entry["audio"] = physical_audio
|
||||
decode_entries = [*logical_entries[:-2], physical_entry]
|
||||
|
||||
logical_chunks = list(plan["chunks"])
|
||||
decode_groups = []
|
||||
for chunk in logical_chunks[:-2]:
|
||||
group = dict(chunk)
|
||||
group["logical_chunk_indices"] = [int(chunk["chunk_index"])]
|
||||
group["terminal_merged"] = False
|
||||
decode_groups.append(group)
|
||||
|
||||
first_chunk = logical_chunks[-2]
|
||||
second_chunk = logical_chunks[-1]
|
||||
frame_start = int(first_chunk.get("frame_start", 0))
|
||||
net_frames = int(contract["physical_frames"]) - int(
|
||||
contract["physical_context_frames"]
|
||||
)
|
||||
terminal_group = dict(first_chunk)
|
||||
terminal_group.update(
|
||||
{
|
||||
"total_frames": int(contract["physical_frames"]),
|
||||
"trim_frames": int(contract["physical_context_frames"]),
|
||||
"net_frames": net_frames,
|
||||
"context_frames": int(contract["physical_context_frames"]),
|
||||
"expected_video_latent_t": int(physical_video.shape[2]),
|
||||
"expected_audio_latent_t": int(physical_audio.shape[-1]),
|
||||
"frame_start": frame_start,
|
||||
"frame_stop": frame_start + net_frames,
|
||||
"logical_chunk_indices": [
|
||||
int(first_chunk["chunk_index"]),
|
||||
int(second_chunk["chunk_index"]),
|
||||
],
|
||||
"terminal_merged": True,
|
||||
}
|
||||
)
|
||||
decode_groups.append(terminal_group)
|
||||
|
||||
plan = dict(plan)
|
||||
plan["decode_group_version"] = 1
|
||||
plan["logical_chunk_count"] = len(logical_chunks)
|
||||
plan["physical_decode_group_count"] = len(decode_groups)
|
||||
plan["decode_groups"] = decode_groups
|
||||
validate_assembly_plan(plan)
|
||||
return decode_entries, plan
|
||||
|
||||
+6
-2
@@ -390,7 +390,9 @@ def analyze_decoded_boundaries(
|
||||
"""Analyze every decoded chunk boundary without changing decoded content."""
|
||||
|
||||
plan = validate_assembly_plan(assembly_plan)
|
||||
chunks = list(plan["chunks"])
|
||||
chunks = list(
|
||||
plan["decode_groups"] if "decode_groups" in plan else plan["chunks"]
|
||||
)
|
||||
if len(images) != len(chunks):
|
||||
raise ValueError("decoded image count does not match the assembly plan")
|
||||
analyses: list[VideoBoundaryAnalysis] = []
|
||||
@@ -428,7 +430,9 @@ def correct_decoded_boundaries(
|
||||
"""Normalize qualified transient exposure shifts without changing motion."""
|
||||
|
||||
plan = validate_assembly_plan(assembly_plan)
|
||||
chunks = list(plan["chunks"])
|
||||
chunks = list(
|
||||
plan["decode_groups"] if "decode_groups" in plan else plan["chunks"]
|
||||
)
|
||||
if len(images) != len(chunks):
|
||||
raise ValueError("decoded image count does not match the assembly plan")
|
||||
corrected = list(images)
|
||||
|
||||
Reference in New Issue
Block a user