Complete V3.4 runtime contract synchronization
This commit is contained in:
@@ -1,6 +1,6 @@
|
||||
# ComfyUI-H3-Continuum 3.4.0
|
||||
|
||||
> **V3.4 hotfix notice:** The initial V3.4 repository package was incomplete because `v3/driving_nodes.py` was missing. We apologize for the issue. This has been fixed in commit `84c22ff`. If you installed V3.4 before this fix, run `git pull` or reinstall the node, then restart ComfyUI.
|
||||
> **V3.4 hotfix notice:** The initial V3.4 repository package was incomplete. Although the public V3.4 nodes were present after the first hotfix, their parent sampler and sequence runtime did not yet accept the new Driving Audio and Video Reference contracts. We apologize for the incomplete release. The complete V3.4 runtime has now been synchronized. If you installed V3.4 earlier, run `git pull` or reinstall the node, then restart ComfyUI.
|
||||
|
||||
Long-form MiniMax H3 video generation for ComfyUI with restartable chunks, persistent references, Driving Audio, Video Reference, seam correction, and optional Spectrum interoperability.
|
||||
|
||||
|
||||
@@ -0,0 +1,62 @@
|
||||
# V3.4 Driving Audio Runtime Validation
|
||||
|
||||
This file records the Windows real-generation evidence for the V3.4 Driving Audio path. The MP4 outputs retain their embedded ComfyUI workflow metadata; settings not repeated below should be recovered from that metadata rather than inferred.
|
||||
|
||||
## Test environment
|
||||
|
||||
- Windows 11, ComfyUI Core, NVIDIA GeForce RTX 5060 Ti 16 GB
|
||||
- H3 Continuum V3.4.0, I2VA + Continuation
|
||||
- 800 x 800, 24 fps, 5 seconds per chunk
|
||||
- Core Video/Audio VAE Decode followed by H3 Continuum Assemble
|
||||
- Fixed chunk seeds during comparisons
|
||||
- Driving Audio uses absolute-time guide slices while the original AUDIO stream is selected for final output
|
||||
- Generated audio and Audio Seam are bypassed for the final Driving Audio stream
|
||||
|
||||
## Source assets
|
||||
|
||||
- Short source: `Voice_Design_00002_.mp3`, approximately 4 seconds
|
||||
- Exact two-chunk source: `H3_V34_Driving_Audio_10s_Boundary_Test.wav`, 10 seconds
|
||||
- Long source: `H3_V34_Driving_Audio_15s_Long_Tail_Test.wav`, 15 seconds used with a 10-second output
|
||||
- Three-chunk source: `H3_V34_Driving_Audio_15s_Three_Chunk_Test.wav`, 15 seconds, 48 kHz mono
|
||||
- Three-chunk source structure: one spoken phrase in each 5-second interval with silence around the chunk boundaries
|
||||
|
||||
## Results
|
||||
|
||||
| Output | Purpose and relevant settings | Result | Time |
|
||||
|---|---|---|---:|
|
||||
| `MiniMax_H3_Continuum_V31_00091_.mp4` | 2 x 5 s; short source; existing exact-duration tail policy | Source audio was retained and the 10-second output completed without a custom short-audio rejection. | 14:22.85 |
|
||||
| `MiniMax_H3_Continuum_V31_00092_.mp4` | 2 x 5 s; exact 10-second boundary source | Boundary timing and original-stream selection passed. Source/output audio PSNR: 165.763 dB. | 14:23.90 |
|
||||
| `MiniMax_H3_Continuum_V31_00093_.mp4` | 2 x 5 s; 15-second source | Only the requested 10-second output interval was used; the later phrase was excluded naturally. Source/output audio PSNR: 165.731 dB. | 14:23.14 |
|
||||
| `MiniMax_H3_Continuum_V31_00094_.mp4` | 3 x 5 s; three-phrase source | All three absolute-time intervals and silence positions were retained. Source/output audio PSNR: 165.671 dB. | 22:33.29 |
|
||||
| `MiniMax_H3_Continuum_V31_00095_.mp4` | Run Storage initial generation | `0 reused / 3 generated`; stored output audio matched `00094` (`inf` dB comparison). | 22:42.63 |
|
||||
| `MiniMax_H3_Continuum_V31_00096_.mp4` | Immediate unchanged queue | Completed, but this run alone was not accepted as restart-resume evidence because ComfyUI execution caching could apply. | 22:42.63 |
|
||||
| `MiniMax_H3_Continuum_V31_00097_.mp4` | Restarted ComfyUI; unchanged Run Storage contract | `3 reused / 0 generated`, `resume=complete`; complete resume passed. | 01:36.47 |
|
||||
| `MiniMax_H3_Continuum_V31_00098_.mp4` | Third prompt changed only | `2 reused / 1 generated`; first regenerated chunk was Chunk 3; output audio remained identical (`inf` dB). | 09:47.09 |
|
||||
| `MiniMax_H3_Continuum_V31_00099_.mp4` | New portrait and three-section dialogue prompt; 3 x 5 s | User listening/video review passed. Original audio remained 15.000 s at 48 kHz mono; source/output PSNR was 165.671 dB. Silence intervals, mouth closure, face stability, and chunk-local dialogue timing were accepted at practical quality. | 23:05.38 |
|
||||
| `MiniMax_H3_Continuum_V31_00100_.mp4` | Same Driving Audio test with Spectrum enabled and Video/Audio Seam set to Auto | User review passed. Spectrum and Video Seam completed; final Driving Audio remained the original stream at 15.000 s, 48 kHz mono, PSNR 165.671 dB. Audio Seam diagnostics were produced, but the generated/seamed audio was bypassed for final output. | 15:34.54 reported by workflow; 953.16 s (15:53.16) external observation |
|
||||
| `MiniMax_H3_Continuum_V31_00101_.mp4` | Turbo LoRA 8-step profile using `res_multistep`; report unavailable | Runtime completed in 662.63 seconds. Recorded as the planned Turbo 8-step Driving Audio comparison and accepted as part of the completed Driving Audio gate. | 11:02.63 |
|
||||
| `MiniMax_H3_Continuum_V31_00102_.mp4` | Three-section city-pop prompt attempt | Generation and original Driving Audio output completed, but the prompt parser reported `Timeline-like syntax was not recognized` and used one Fixed prompt for all three chunks. This run is retained as parser-fallback evidence and is not accepted as a valid three-section visual-timeline comparison. | 11:24.89 |
|
||||
| `MiniMax_H3_Continuum_V31_00103_.mp4` | Generic audio-content-independent lip-sync prompt; Turbo LoRA 8-step; `euler` + `simple`; Spectrum disabled; 3 x 5 s | Timeline parsing returned three unique sections. The 40-second, 48 kHz stereo source `audio_ace_step_00003_.mp3` naturally supplied only the requested first 15 seconds. Final source/output PSNR was 169.031 dB left and 169.021 dB right. Original Driving Audio was selected and Audio Seam was bypassed. User listening review passed; a non-vocal passage was interpreted visually as a sound-effect-like event, which is accepted within native H3 behavior without an external Mel-band or dedicated lip-sync controller. | 10:58.09 |
|
||||
|
||||
## Acceptance summary
|
||||
|
||||
- Original waveform timing and final duration: PASS
|
||||
- Absolute-time slicing across 5-second chunks: PASS
|
||||
- Silence and later-phrase placement: PASS
|
||||
- No unintended generated dialogue in the final audio: PASS
|
||||
- Short source permissive execution: PASS
|
||||
- Longer-than-output source handling without a custom pre-trim policy: PASS
|
||||
- Run Storage complete reuse after restart: PASS
|
||||
- Run Storage partial regeneration from Chunk 3: PASS
|
||||
- Spectrum-enabled compatibility: PASS
|
||||
- Practical mouth movement and listening review: PASS
|
||||
- Strict phoneme-level lip-sync guarantee: not claimed
|
||||
|
||||
## Remaining cleanup
|
||||
|
||||
- Driving Audio currently makes final Audio Seam correction unnecessary. Skipping its analysis entirely would reduce confusing diagnostics and avoid unused processing; this is a cleanup item, not a correctness failure.
|
||||
- Native H3 audio-driven facial motion is not a strict phoneme-level lip-sync contract. External Mel-band or dedicated lip-sync control is outside the accepted V3.4 scope.
|
||||
|
||||
## Current verdict
|
||||
|
||||
**PASS.** Standard, Spectrum-enabled, Turbo 8-step, Run Storage, partial-regeneration, short-source, long-source, mono, and stereo tests establish that Driving Audio preserves the original final stream and maintains absolute timeline placement across Continuum chunks. Strict phoneme-level lip sync is not claimed.
|
||||
+41
-1
@@ -136,6 +136,16 @@ def automatic_project_key(prompt: Any, unique_id: Any) -> str:
|
||||
and str(audio_link[0]) in prompt
|
||||
):
|
||||
continue
|
||||
if current_id == node_id and name == "audio_vae":
|
||||
audio_link = inputs.get("driving_audio")
|
||||
if not (
|
||||
isinstance(audio_link, (list, tuple))
|
||||
and len(audio_link) == 2
|
||||
and isinstance(audio_link[0], (str, int))
|
||||
and isinstance(audio_link[1], int)
|
||||
and str(audio_link[0]) in prompt
|
||||
):
|
||||
continue
|
||||
value = inputs[name]
|
||||
if (
|
||||
isinstance(value, (list, tuple))
|
||||
@@ -557,6 +567,9 @@ def build_sampling_contract(
|
||||
upstream_graph_reasons: list[str] | None = None,
|
||||
reference_audio_contract: dict[str, Any] | None = None,
|
||||
reference_audio_vae: Any = None,
|
||||
driving_audio_contract: dict[str, Any] | None = None,
|
||||
driving_audio_vae: Any = None,
|
||||
reference_video_contract: dict[str, Any] | None = None,
|
||||
timeline_video_contract: dict[str, Any] | None = None,
|
||||
) -> tuple[dict[str, Any], bool, list[str]]:
|
||||
model_value, model_safe = _model_signature(model, model_fingerprint_value)
|
||||
@@ -576,6 +589,7 @@ def build_sampling_contract(
|
||||
)
|
||||
uses_video_vae = (
|
||||
conditioning_mode_uses_video_vae(conditioning_mode)
|
||||
or reference_video_contract is not None
|
||||
or timeline_video_contract is not None
|
||||
)
|
||||
if uses_video_vae:
|
||||
@@ -589,6 +603,12 @@ def build_sampling_contract(
|
||||
audio_vae_value, audio_vae_safe = _audio_vae_signature(reference_audio_vae)
|
||||
else:
|
||||
audio_vae_value, audio_vae_safe = None, True
|
||||
if driving_audio_contract is not None:
|
||||
driving_audio_vae_value, driving_audio_vae_safe = _audio_vae_signature(
|
||||
driving_audio_vae
|
||||
)
|
||||
else:
|
||||
driving_audio_vae_value, driving_audio_vae_safe = None, True
|
||||
if not torch.is_tensor(sigmas) or sigmas.ndim != 1:
|
||||
raise RunStorageError("Run Storage requires a one-dimensional SIGMAS tensor")
|
||||
chunks = int(prompt_plan["chunks"])
|
||||
@@ -615,6 +635,11 @@ def build_sampling_contract(
|
||||
"runtime": audio_vae_value,
|
||||
"graph_route": routes.get("reference_audio_vae"),
|
||||
}
|
||||
if driving_audio_contract is not None:
|
||||
driving_audio_vae_value = {
|
||||
"runtime": driving_audio_vae_value,
|
||||
"graph_route": routes.get("audio_vae"),
|
||||
}
|
||||
global_contract = {
|
||||
"sampling_contract_version": SAMPLING_CONTRACT_VERSION,
|
||||
"conditioning_mode": conditioning_mode,
|
||||
@@ -642,6 +667,11 @@ def build_sampling_contract(
|
||||
if reference_audio_contract is not None:
|
||||
global_contract["reference_audio"] = dict(reference_audio_contract)
|
||||
global_contract["reference_audio_vae"] = audio_vae_value
|
||||
if driving_audio_contract is not None:
|
||||
global_contract["driving_audio"] = dict(driving_audio_contract)
|
||||
global_contract["driving_audio_vae"] = driving_audio_vae_value
|
||||
if reference_video_contract is not None:
|
||||
global_contract["reference_video"] = dict(reference_video_contract)
|
||||
timeline_chunk_contracts = []
|
||||
if timeline_video_contract is not None:
|
||||
timeline_value = dict(timeline_video_contract)
|
||||
@@ -688,8 +718,10 @@ def build_sampling_contract(
|
||||
reasons.append("Video VAE contract is not completely observable")
|
||||
if not audio_vae_safe:
|
||||
reasons.append("Reference Audio VAE contract is not completely observable")
|
||||
if not driving_audio_vae_safe:
|
||||
reasons.append("Driving Audio VAE contract is not completely observable")
|
||||
graph_safe = bool(upstream_graph_safe) if graph_authoritative else True
|
||||
return contract, model_safe and clip_safe and video_vae_safe and audio_vae_safe and sampler_safe and graph_safe, reasons
|
||||
return contract, model_safe and clip_safe and video_vae_safe and audio_vae_safe and driving_audio_vae_safe and sampler_safe and graph_safe, reasons
|
||||
|
||||
|
||||
def revision_identity(contract: dict[str, Any]) -> tuple[str, str]:
|
||||
@@ -963,6 +995,9 @@ class RunStorageController:
|
||||
conditioning_mode: str | None = None,
|
||||
reference_audio_contract: dict[str, Any] | None = None,
|
||||
reference_audio_vae: Any = None,
|
||||
driving_audio_contract: dict[str, Any] | None = None,
|
||||
driving_audio_vae: Any = None,
|
||||
reference_video_contract: dict[str, Any] | None = None,
|
||||
timeline_video_contract: dict[str, Any] | None = None,
|
||||
) -> dict[str, Any] | None:
|
||||
if existing_session is not None:
|
||||
@@ -974,9 +1009,11 @@ class RunStorageController:
|
||||
self.sampler_node_id,
|
||||
require_video_vae=(
|
||||
conditioning_mode_uses_video_vae(str(conditioning_mode))
|
||||
or reference_video_contract is not None
|
||||
or timeline_video_contract is not None
|
||||
),
|
||||
require_reference_audio_vae=reference_audio_contract is not None,
|
||||
require_audio_vae=driving_audio_contract is not None,
|
||||
)
|
||||
contract, safe, reasons = build_sampling_contract(
|
||||
model=model, model_fingerprint_value=model_fingerprint_value,
|
||||
@@ -995,6 +1032,9 @@ class RunStorageController:
|
||||
upstream_graph_reasons=graph_reasons,
|
||||
reference_audio_contract=reference_audio_contract,
|
||||
reference_audio_vae=reference_audio_vae,
|
||||
driving_audio_contract=driving_audio_contract,
|
||||
driving_audio_vae=driving_audio_vae,
|
||||
reference_video_contract=reference_video_contract,
|
||||
timeline_video_contract=timeline_video_contract,
|
||||
)
|
||||
effective_nonce, nonce_decision = self._resolve_effective_nonce(
|
||||
|
||||
@@ -0,0 +1,46 @@
|
||||
import torch
|
||||
|
||||
from ComfyUI_H3_Continuum_Join.v2.context_diagnostics import (
|
||||
ContextDiagnosticsTracker,
|
||||
measure_video_context,
|
||||
)
|
||||
|
||||
|
||||
def test_measure_video_context_is_read_only():
|
||||
video = torch.linspace(-1.0, 1.0, 1 * 24 * 3 * 5 * 7).reshape(1, 24, 3, 5, 7)
|
||||
original = video.clone()
|
||||
stats = measure_video_context(video)
|
||||
assert stats.shape == tuple(video.shape)
|
||||
assert stats.std > 0.0
|
||||
assert stats.high_frequency_rms > 0.0
|
||||
assert stats.temporal_delta_rms > 0.0
|
||||
assert torch.equal(video, original)
|
||||
|
||||
|
||||
def test_constant_context_has_no_high_frequency_or_motion():
|
||||
stats = measure_video_context(torch.ones(1, 24, 2, 4, 4))
|
||||
assert stats.std == 0.0
|
||||
assert stats.high_frequency_rms == 0.0
|
||||
assert stats.temporal_delta_rms == 0.0
|
||||
assert stats.outlier_ratio == 0.0
|
||||
|
||||
|
||||
def test_tracker_reports_baseline_and_previous_ratios():
|
||||
tracker = ContextDiagnosticsTracker()
|
||||
first = torch.linspace(-1.0, 1.0, 1 * 24 * 2 * 4 * 4).reshape(1, 24, 2, 4, 4)
|
||||
first_report = tracker.observe(first, source_chunk=1, context_frames=22, reused=False)
|
||||
second_report = tracker.observe(first * 2.0, source_chunk=2, context_frames=22, reused=False)
|
||||
assert "hf_base=1.0000" in first_report
|
||||
assert "std_base=2.0000" in second_report
|
||||
assert "hf_prev=2.0000" in second_report
|
||||
assert "action=observe" in second_report
|
||||
|
||||
|
||||
def test_tracker_never_turns_diagnostic_failure_into_execution_error():
|
||||
report = ContextDiagnosticsTracker().observe(
|
||||
torch.zeros(1, 24, 2, 4),
|
||||
source_chunk=1,
|
||||
context_frames=22,
|
||||
reused=False,
|
||||
)
|
||||
assert "unavailable" in report
|
||||
@@ -115,19 +115,25 @@ def test_v217_facade_passes_only_sparse_override_clips(monkeypatch):
|
||||
assert not any(f"clip_{index}_prompt" in captured for index in (1, 3, 4))
|
||||
|
||||
|
||||
def test_v217_facade_rejects_sparse_override_outside_chunks(monkeypatch):
|
||||
monkeypatch.setattr(H3ContinuumSamplerV2, "run", lambda _self, **kwargs: None)
|
||||
def test_v217_facade_ignores_sparse_override_outside_chunks(monkeypatch):
|
||||
captured = {}
|
||||
|
||||
def fake_run(_self, **kwargs):
|
||||
captured.update(kwargs)
|
||||
return "images", "audio", "state", "session", "report"
|
||||
|
||||
monkeypatch.setattr(H3ContinuumSamplerV2, "run", fake_run)
|
||||
pack = H3ContinuumClipOverrides().build(
|
||||
PROMPT_FORMAT_OPTIONS[0], "[Clip 6]\noutside"
|
||||
)[0]
|
||||
with pytest.raises(PromptPlanError, match="outside"):
|
||||
H3ContinuumSampler().run(
|
||||
model="model", clip="clip", video_vae="video_vae", audio_vae="audio_vae",
|
||||
sampler="sampler", sigmas="sigmas", sequence_prompt="base",
|
||||
prompt_mode=PROMPT_FORMAT_OPTIONS[0], chunks=5, chunk_seconds=5.0,
|
||||
width=1344, height=768, continuity="Balanced - 22 frames", base_seed=123,
|
||||
seam_correction="Auto", prompt_overrides=pack,
|
||||
)
|
||||
H3ContinuumSampler().run(
|
||||
model="model", clip="clip", video_vae="video_vae", audio_vae="audio_vae",
|
||||
sampler="sampler", sigmas="sigmas", sequence_prompt="base",
|
||||
prompt_mode=PROMPT_FORMAT_OPTIONS[0], chunks=5, chunk_seconds=5.0,
|
||||
width=1344, height=768, continuity="Balanced - 22 frames", base_seed=123,
|
||||
seam_correction="Auto", prompt_overrides=pack,
|
||||
)
|
||||
assert not any(key.startswith("clip_") and key.endswith("_prompt") for key in captured)
|
||||
|
||||
|
||||
def test_v217_result_pack_expands_legacy_outputs():
|
||||
|
||||
+24
-23
@@ -127,27 +127,28 @@ def test_timeline_extra_sections_warn_and_are_ignored():
|
||||
("[0-5s]\n", "H3C-P005"),
|
||||
],
|
||||
)
|
||||
def test_invalid_timeline_reports_actionable_diagnostic(script, code):
|
||||
with pytest.raises(PromptPlanError) as captured:
|
||||
make_prompt_plan(
|
||||
mode=PROMPT_MODE_TIMELINE,
|
||||
script=script,
|
||||
chunks=2,
|
||||
chunk_seconds=5,
|
||||
)
|
||||
message = str(captured.value)
|
||||
assert code in message
|
||||
assert "Suggested fix:" in message
|
||||
def test_invalid_timeline_falls_back_with_actionable_diagnostic(script, code):
|
||||
plan = make_prompt_plan(
|
||||
mode=PROMPT_MODE_TIMELINE,
|
||||
script=script,
|
||||
chunks=2,
|
||||
chunk_seconds=5,
|
||||
)
|
||||
assert plan["mode"] == PROMPT_MODE_FIXED
|
||||
assert plan["prompts"] == [script.strip(), script.strip()]
|
||||
assert plan["diagnostics"][0]["code"] == "H3C-P100"
|
||||
assert code in plan["diagnostics"][0]["message"]
|
||||
|
||||
|
||||
def test_duplicate_chunk_section_is_an_error():
|
||||
with pytest.raises(PromptPlanError, match="H3C-P003"):
|
||||
make_prompt_plan(
|
||||
mode=PROMPT_MODE_TIMELINE,
|
||||
script="[Chunk 1]\none\n[Chunk 1]\ntwo",
|
||||
chunks=2,
|
||||
chunk_seconds=5,
|
||||
)
|
||||
def test_duplicate_chunk_section_falls_back_without_blocking():
|
||||
plan = make_prompt_plan(
|
||||
mode=PROMPT_MODE_TIMELINE,
|
||||
script="[Chunk 1]\none\n[Chunk 1]\ntwo",
|
||||
chunks=2,
|
||||
chunk_seconds=5,
|
||||
)
|
||||
assert plan["mode"] == PROMPT_MODE_FIXED
|
||||
assert any("H3C-P003" in item["message"] for item in plan["diagnostics"])
|
||||
|
||||
|
||||
def test_prompt_preflight_reports_explicit_chunk_sources():
|
||||
@@ -183,10 +184,10 @@ def test_external_prompt_with_same_text_keeps_the_same_hash():
|
||||
assert apply_prompt_overrides(plan, [None, "same"])["hashes"] == plan["hashes"]
|
||||
|
||||
|
||||
def test_external_prompt_rejects_empty_connected_text():
|
||||
def test_external_prompt_accepts_empty_connected_text():
|
||||
plan = make_prompt_plan(mode=PROMPT_MODE_FIXED, script="fallback", chunks=1, chunk_seconds=5)
|
||||
with pytest.raises(PromptPlanError):
|
||||
apply_prompt_overrides(plan, [" "])
|
||||
updated = apply_prompt_overrides(plan, [" "])
|
||||
assert updated["prompts"] == [""]
|
||||
|
||||
|
||||
def test_sequence_prompt_precedes_connected_plan_and_legacy_script():
|
||||
@@ -198,7 +199,7 @@ def test_sequence_prompt_precedes_connected_plan_and_legacy_script():
|
||||
def test_connected_plan_precedes_legacy_script_without_sequence_input():
|
||||
connected = make_prompt_plan(mode=PROMPT_FORMAT_FIXED, script="connected plan", chunks=2, chunk_seconds=5)
|
||||
resolved = build_sampler_prompt_plan(prompt_mode=PROMPT_FORMAT_AUTO, prompt_script="legacy", sequence_prompt=None, prompt_plan=connected, chunks=2, chunk_seconds=5)
|
||||
assert resolved is connected
|
||||
assert resolved == connected
|
||||
|
||||
|
||||
def test_sparse_overrides_replace_only_explicit_clips_and_rehash():
|
||||
|
||||
@@ -0,0 +1,298 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
import torch
|
||||
|
||||
from ComfyUI_H3_Continuum_Join.driving_audio import (
|
||||
DrivingAudioAssets,
|
||||
DrivingAudioSource,
|
||||
attach_driving_audio,
|
||||
encode_driving_audio,
|
||||
prepare_driving_audio_source,
|
||||
slice_driving_audio_latent,
|
||||
)
|
||||
from ComfyUI_H3_Continuum_Join.run_storage import automatic_project_key
|
||||
from ComfyUI_H3_Continuum_Join.reference_video import (
|
||||
REFERENCE_VIDEO_SIZE_BALANCED,
|
||||
REFERENCE_VIDEO_SIZE_EFFICIENT,
|
||||
REFERENCE_VIDEO_SIZE_MATCH_OUTPUT,
|
||||
REFERENCE_VIDEO_SIZE_OPTIONS,
|
||||
_resolved_size,
|
||||
)
|
||||
from ComfyUI_H3_Continuum_Join.v3.assembly import (
|
||||
H3ContinuumAssembleSeamExperimental,
|
||||
)
|
||||
from ComfyUI_H3_Continuum_Join.v3.driving_nodes import (
|
||||
H3ContinuumAssembleSeamV34,
|
||||
H3ContinuumSamplerV34,
|
||||
)
|
||||
|
||||
|
||||
def test_v34_video_reference_schema_uses_image_batch():
|
||||
schema = H3ContinuumSamplerV34.INPUT_TYPES()
|
||||
assert schema["optional"]["reference_video_1"][0] == "IMAGE"
|
||||
size_schema = schema["required"]["video_reference_size"]
|
||||
assert size_schema[0] == REFERENCE_VIDEO_SIZE_OPTIONS
|
||||
assert size_schema[1]["default"] == REFERENCE_VIDEO_SIZE_EFFICIENT
|
||||
|
||||
|
||||
def test_v34_requests_one_chunk_video_reference(monkeypatch):
|
||||
import ComfyUI_H3_Continuum_Join.reference_video as reference_video
|
||||
|
||||
captured = {}
|
||||
|
||||
class _Captured(RuntimeError):
|
||||
pass
|
||||
|
||||
def capture_target(
|
||||
_video,
|
||||
*,
|
||||
target_frames,
|
||||
output_width,
|
||||
output_height,
|
||||
size_mode,
|
||||
):
|
||||
captured["target_frames"] = target_frames
|
||||
captured["output_size"] = (output_width, output_height)
|
||||
captured["size_mode"] = size_mode
|
||||
raise _Captured
|
||||
|
||||
monkeypatch.setattr(
|
||||
reference_video,
|
||||
"prepare_reference_video_source",
|
||||
capture_target,
|
||||
)
|
||||
with pytest.raises(_Captured):
|
||||
H3ContinuumSamplerV34().run(
|
||||
reference_video_1=torch.zeros((243, 8, 8, 3)),
|
||||
driving_audio=None,
|
||||
audio_vae=None,
|
||||
chunks=2,
|
||||
chunk_seconds=5.0,
|
||||
width=800,
|
||||
height=800,
|
||||
video_reference_size=REFERENCE_VIDEO_SIZE_BALANCED,
|
||||
)
|
||||
assert captured["target_frames"] == 124
|
||||
assert captured["output_size"] == (800, 800)
|
||||
assert captured["size_mode"] == REFERENCE_VIDEO_SIZE_BALANCED
|
||||
|
||||
|
||||
def test_video_reference_size_modes_preserve_source_and_avoid_upscale():
|
||||
assert _resolved_size(
|
||||
768,
|
||||
768,
|
||||
output_width=800,
|
||||
output_height=800,
|
||||
size_mode=REFERENCE_VIDEO_SIZE_EFFICIENT,
|
||||
) == (640, 640)
|
||||
assert _resolved_size(
|
||||
768,
|
||||
768,
|
||||
output_width=800,
|
||||
output_height=800,
|
||||
size_mode=REFERENCE_VIDEO_SIZE_BALANCED,
|
||||
) == (768, 768)
|
||||
assert _resolved_size(
|
||||
768,
|
||||
768,
|
||||
output_width=800,
|
||||
output_height=800,
|
||||
size_mode=REFERENCE_VIDEO_SIZE_MATCH_OUTPUT,
|
||||
) == (768, 768)
|
||||
|
||||
|
||||
def test_video_reference_match_output_preserves_source_aspect_ratio():
|
||||
width, height = _resolved_size(
|
||||
1920,
|
||||
1080,
|
||||
output_width=1344,
|
||||
output_height=768,
|
||||
size_mode=REFERENCE_VIDEO_SIZE_MATCH_OUTPUT,
|
||||
)
|
||||
assert width % 32 == 0
|
||||
assert height % 32 == 0
|
||||
assert width <= 1920
|
||||
assert height <= 1080
|
||||
assert abs((width / height) - (1920 / 1080)) < 0.06
|
||||
|
||||
class _AudioVAE:
|
||||
audio_sample_rate = 10
|
||||
|
||||
def __init__(self):
|
||||
self.calls = []
|
||||
|
||||
def encode(self, waveform):
|
||||
self.calls.append(waveform)
|
||||
return torch.arange(400, dtype=torch.float32).reshape(1, 1, 1, 400)
|
||||
|
||||
|
||||
def _audio(seconds: int, *, tail_value: float = 0.0):
|
||||
waveform = torch.zeros((1, 2, seconds * 10), dtype=torch.float32)
|
||||
if seconds > 15:
|
||||
waveform[..., 150:] = tail_value
|
||||
return {"waveform": waveform, "sample_rate": 10}
|
||||
|
||||
|
||||
def test_v34_public_schema_replaces_reference_audio_with_driving_audio():
|
||||
schema = H3ContinuumSamplerV34.INPUT_TYPES()
|
||||
assert "driving_audio" in schema["optional"]
|
||||
assert "audio_vae" in schema["optional"]
|
||||
assert "reference_audio_1" not in schema["optional"]
|
||||
assert "reference_audio_vae" not in schema["optional"]
|
||||
|
||||
from ComfyUI_H3_Continuum_Join import nodes as root_nodes
|
||||
|
||||
assert root_nodes.NODE_DISPLAY_NAME_MAPPINGS["H3ContinuumSamplerV34"] == (
|
||||
"H3 Continuum Sampler V3.4"
|
||||
)
|
||||
assert root_nodes.NODE_DISPLAY_NAME_MAPPINGS[
|
||||
"H3ContinuumSamplerTimelineVideo"
|
||||
].startswith("[Legacy]")
|
||||
|
||||
|
||||
def test_unused_long_audio_tail_does_not_change_contract_or_output():
|
||||
vae = _AudioVAE()
|
||||
first = prepare_driving_audio_source(
|
||||
_audio(30, tail_value=1.0), vae, target_frames=360, fps=24
|
||||
)
|
||||
second = prepare_driving_audio_source(
|
||||
_audio(30, tail_value=9.0), vae, target_frames=360, fps=24
|
||||
)
|
||||
assert first.combined_hash == second.combined_hash
|
||||
assert first.source_audio["waveform"].shape[-1] == 150
|
||||
assert second.source_audio["waveform"].shape[-1] == 150
|
||||
|
||||
|
||||
def test_short_driving_audio_is_accepted_without_padding():
|
||||
source = prepare_driving_audio_source(
|
||||
_audio(4), _AudioVAE(), target_frames=360, fps=24
|
||||
)
|
||||
assert source.source_audio["waveform"].shape == (1, 2, 40)
|
||||
assert source.resampled_waveform.shape == (1, 2, 40)
|
||||
|
||||
|
||||
def test_encode_is_core_layout_and_absolute_slices_are_drift_free():
|
||||
vae = _AudioVAE()
|
||||
source = prepare_driving_audio_source(
|
||||
_audio(30), vae, target_frames=360, fps=24
|
||||
)
|
||||
assets = encode_driving_audio(source, vae)
|
||||
assert len(vae.calls) == 1
|
||||
assert vae.calls[0].shape == (1, 150, 2)
|
||||
|
||||
first = slice_driving_audio_latent(
|
||||
assets,
|
||||
cumulative_retained_before=0,
|
||||
total_frames=124,
|
||||
trim_frames=0,
|
||||
fps=24,
|
||||
)
|
||||
second = slice_driving_audio_latent(
|
||||
assets,
|
||||
cumulative_retained_before=124,
|
||||
total_frames=141,
|
||||
trim_frames=22,
|
||||
fps=24,
|
||||
)
|
||||
third = slice_driving_audio_latent(
|
||||
assets,
|
||||
cumulative_retained_before=243,
|
||||
total_frames=141,
|
||||
trim_frames=22,
|
||||
fps=24,
|
||||
)
|
||||
assert first.flatten()[0].item() == 0
|
||||
assert second.flatten()[0].item() == round(102 * 40 / 24)
|
||||
assert third.flatten()[0].item() == round(221 * 40 / 24)
|
||||
|
||||
|
||||
def test_driving_audio_uses_keyframe_payload_not_audio_reference_tokens():
|
||||
conditioning = [[torch.zeros((1, 2, 3)), {"minimax_ref_items": []}]]
|
||||
latent = torch.zeros((1, 32, 2, 10))
|
||||
output = attach_driving_audio(conditioning, latent)
|
||||
keyframe = output[0][1]["minimax_keyframes"][-1]
|
||||
assert keyframe == {"resolved_frame_index": 0, "audio_latent": latent}
|
||||
assert output[0][1]["minimax_ref_items"] == []
|
||||
|
||||
|
||||
def test_audio_vae_topology_is_ignored_when_driving_audio_is_disconnected():
|
||||
base = {
|
||||
"1": {"class_type": "H3ContinuumSamplerV34", "inputs": {}},
|
||||
"2": {"class_type": "VAELoader", "inputs": {}},
|
||||
}
|
||||
connected_vae_only = {
|
||||
**base,
|
||||
"1": {
|
||||
"class_type": "H3ContinuumSamplerV34",
|
||||
"inputs": {"audio_vae": ["2", 0]},
|
||||
},
|
||||
}
|
||||
assert automatic_project_key(base, "1") == automatic_project_key(
|
||||
connected_vae_only, "1"
|
||||
)
|
||||
|
||||
|
||||
def test_v34_assembler_selects_original_driving_audio_and_bypasses_seam(
|
||||
monkeypatch,
|
||||
):
|
||||
images = torch.zeros((10, 8, 8, 3))
|
||||
generated = {"waveform": torch.ones((1, 2, 20)), "sample_rate": 10}
|
||||
driving = {"waveform": torch.zeros((1, 2, 20)), "sample_rate": 10}
|
||||
|
||||
monkeypatch.setattr(
|
||||
H3ContinuumAssembleSeamExperimental,
|
||||
"assemble",
|
||||
lambda self, *args, **kwargs: (images, generated, "base report"),
|
||||
)
|
||||
output_images, output_audio, report = H3ContinuumAssembleSeamV34().assemble(
|
||||
images=[],
|
||||
audio=[],
|
||||
assembly_plan={"target_frames": 10},
|
||||
exact_total_duration=False,
|
||||
audio_seam="Auto",
|
||||
video_seam="Auto",
|
||||
diagnostics="Basic",
|
||||
driving_audio=driving,
|
||||
)
|
||||
assert output_images is images
|
||||
assert torch.equal(output_audio["waveform"], driving["waveform"])
|
||||
assert "Audio Seam bypassed" in report
|
||||
|
||||
|
||||
def test_v34_assembly_plan_audio_wins_when_direct_input_is_generated_audio(
|
||||
monkeypatch,
|
||||
):
|
||||
images = torch.zeros((10, 8, 8, 3))
|
||||
generated = {
|
||||
"waveform": torch.zeros((1, 2, 320000), dtype=torch.float32),
|
||||
"sample_rate": 32000,
|
||||
}
|
||||
original = {
|
||||
"waveform": torch.linspace(-1.0, 1.0, 480000, dtype=torch.float32)
|
||||
.reshape(1, 1, -1)
|
||||
.repeat(1, 2, 1),
|
||||
"sample_rate": 48000,
|
||||
}
|
||||
plan = {
|
||||
"target_frames": 10,
|
||||
"_h3_continuum_driving_audio_v1": original,
|
||||
}
|
||||
|
||||
monkeypatch.setattr(
|
||||
H3ContinuumAssembleSeamExperimental,
|
||||
"assemble",
|
||||
lambda self, *args, **kwargs: (images, generated, "base report"),
|
||||
)
|
||||
_, output_audio, report = H3ContinuumAssembleSeamV34().assemble(
|
||||
images=[],
|
||||
audio=[],
|
||||
assembly_plan=[plan],
|
||||
exact_total_duration=False,
|
||||
driving_audio=[generated],
|
||||
)
|
||||
|
||||
assert output_audio["sample_rate"] == 48000
|
||||
assert torch.equal(output_audio["waveform"], original["waveform"])
|
||||
assert output_audio["waveform"].data_ptr() != original["waveform"].data_ptr()
|
||||
assert "selected from assembly plan" in report
|
||||
@@ -65,7 +65,13 @@ def main() -> int:
|
||||
print(f"V2 prompt/session core self-test FAILED: {exc}")
|
||||
return 1
|
||||
|
||||
expected = {
|
||||
current_required = {
|
||||
"H3ContinuumSamplerV34",
|
||||
"H3ContinuumAssembleSeamV34",
|
||||
"H3ContinuumClipOverrides",
|
||||
"H3ContinuumResult",
|
||||
}
|
||||
legacy_required = {
|
||||
"H3ContinuumJoin",
|
||||
"H3ContinuumFinish",
|
||||
"H3ContinuumAssemble",
|
||||
@@ -77,18 +83,15 @@ def main() -> int:
|
||||
"H3ContinuumLoadSession",
|
||||
"H3ContinuumSessionInfo",
|
||||
"H3ContinuumSampler",
|
||||
"H3ContinuumClipOverrides",
|
||||
"H3ContinuumAdvanced",
|
||||
"H3ContinuumResult",
|
||||
"H3ContinuumSamplerProduction",
|
||||
"H3ContinuumSamplerTimelineVideo",
|
||||
"H3ContinuumSamplerV3",
|
||||
"H3ContinuumAdvancedV3",
|
||||
"H3ContinuumAssembleV3",
|
||||
"H3ContinuumAssembleSeamExperimental",
|
||||
"H3ContinuumSamplerV34",
|
||||
"H3ContinuumAssembleSeamV34",
|
||||
}
|
||||
expected = current_required | legacy_required
|
||||
actual = set(module.NODE_CLASS_MAPPINGS)
|
||||
if actual != expected:
|
||||
print(f"Node registration mismatch: expected {sorted(expected)}, got {sorted(actual)}")
|
||||
|
||||
@@ -0,0 +1,108 @@
|
||||
"""Read-only diagnostics for the latent video context passed between chunks."""
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
import math
|
||||
|
||||
import torch
|
||||
import torch.nn.functional as F
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class VideoContextStats:
|
||||
shape: tuple[int, ...]
|
||||
mean: float
|
||||
std: float
|
||||
low_frequency_rms: float
|
||||
high_frequency_rms: float
|
||||
temporal_delta_rms: float
|
||||
outlier_ratio: float
|
||||
|
||||
|
||||
def _rms(value: torch.Tensor) -> float:
|
||||
return float(torch.sqrt(torch.mean(value.square())).item()) if value.numel() else 0.0
|
||||
|
||||
|
||||
def measure_video_context(video: torch.Tensor) -> VideoContextStats:
|
||||
"""Measure a context tensor without modifying or retaining it."""
|
||||
if not torch.is_tensor(video) or video.ndim != 5:
|
||||
raise ValueError("video context must be a 5D tensor")
|
||||
value = video.detach().to(device="cpu", dtype=torch.float32)
|
||||
if not bool(torch.isfinite(value).all().item()):
|
||||
raise ValueError("video context contains NaN or Inf")
|
||||
|
||||
mean = float(value.mean().item())
|
||||
std = float(value.std(unbiased=False).item())
|
||||
low = F.avg_pool3d(
|
||||
value,
|
||||
kernel_size=(1, 3, 3),
|
||||
stride=1,
|
||||
padding=(0, 1, 1),
|
||||
count_include_pad=False,
|
||||
)
|
||||
high = value - low
|
||||
temporal = value[:, :, 1:] - value[:, :, :-1] if value.shape[2] > 1 else value[:, :, :0]
|
||||
if std > 0.0:
|
||||
outlier_ratio = float(((value - mean).abs() > 6.0 * std).float().mean().item())
|
||||
else:
|
||||
outlier_ratio = 0.0
|
||||
return VideoContextStats(
|
||||
shape=tuple(int(part) for part in video.shape),
|
||||
mean=mean,
|
||||
std=std,
|
||||
low_frequency_rms=_rms(low),
|
||||
high_frequency_rms=_rms(high),
|
||||
temporal_delta_rms=_rms(temporal),
|
||||
outlier_ratio=outlier_ratio,
|
||||
)
|
||||
|
||||
|
||||
def _ratio(value: float, reference: float) -> float:
|
||||
if reference == 0.0:
|
||||
return 1.0 if value == 0.0 else math.inf
|
||||
return value / reference
|
||||
|
||||
|
||||
def _format_ratio(value: float) -> str:
|
||||
return f"{value:.4f}" if math.isfinite(value) else "inf"
|
||||
|
||||
|
||||
class ContextDiagnosticsTracker:
|
||||
"""Track first-context and previous-context ratios for one sequence run."""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self._baseline: VideoContextStats | None = None
|
||||
self._previous: VideoContextStats | None = None
|
||||
|
||||
def observe(
|
||||
self,
|
||||
video: torch.Tensor,
|
||||
*,
|
||||
source_chunk: int,
|
||||
context_frames: int,
|
||||
reused: bool,
|
||||
) -> str:
|
||||
try:
|
||||
stats = measure_video_context(video)
|
||||
except Exception as exc:
|
||||
return f"context diagnostics source chunk {source_chunk}: unavailable ({exc})"
|
||||
|
||||
if self._baseline is None:
|
||||
self._baseline = stats
|
||||
previous = self._previous or stats
|
||||
baseline = self._baseline
|
||||
self._previous = stats
|
||||
return (
|
||||
f"context diagnostics source chunk {source_chunk}: frames={int(context_frames)}, "
|
||||
f"shape={stats.shape}, reused={'yes' if reused else 'no'}, "
|
||||
f"mean={stats.mean:.6f}, std={stats.std:.6f}, "
|
||||
f"hf={stats.high_frequency_rms:.6f}, motion={stats.temporal_delta_rms:.6f}, "
|
||||
f"outliers={stats.outlier_ratio:.6f}, "
|
||||
f"std_prev={_format_ratio(_ratio(stats.std, previous.std))}, "
|
||||
f"std_base={_format_ratio(_ratio(stats.std, baseline.std))}, "
|
||||
f"hf_prev={_format_ratio(_ratio(stats.high_frequency_rms, previous.high_frequency_rms))}, "
|
||||
f"hf_base={_format_ratio(_ratio(stats.high_frequency_rms, baseline.high_frequency_rms))}, "
|
||||
f"motion_prev={_format_ratio(_ratio(stats.temporal_delta_rms, previous.temporal_delta_rms))}, "
|
||||
f"motion_base={_format_ratio(_ratio(stats.temporal_delta_rms, baseline.temporal_delta_rms))}, "
|
||||
"action=observe"
|
||||
)
|
||||
@@ -307,6 +307,9 @@ class H3ContinuumSamplerV2:
|
||||
reference_assets=None,
|
||||
reference_audio_source=None,
|
||||
reference_audio_vae=None,
|
||||
driving_audio_source=None,
|
||||
driving_audio_vae=None,
|
||||
reference_video_source=None,
|
||||
timeline_video_source=None,
|
||||
**clip_prompt_inputs,
|
||||
):
|
||||
@@ -376,6 +379,9 @@ class H3ContinuumSamplerV2:
|
||||
reference_assets=reference_assets,
|
||||
reference_audio_source=reference_audio_source,
|
||||
reference_audio_vae=reference_audio_vae,
|
||||
driving_audio_source=driving_audio_source,
|
||||
driving_audio_vae=driving_audio_vae,
|
||||
reference_video_source=reference_video_source,
|
||||
timeline_video_source=timeline_video_source,
|
||||
)
|
||||
|
||||
|
||||
+1
-1
@@ -24,7 +24,7 @@ def _prompt_error(code,reason,*,line_number=None,source=None,suggested_fix):
|
||||
raise PromptPlanError("\n".join(lines))
|
||||
def prompt_hash(prompt:str)->str: return hashlib.sha256(prompt.encode("utf-8")).hexdigest()
|
||||
def _normalize_prompt(value:str,*,label:str)->str:
|
||||
return "" if value is None else str(value)
|
||||
return "" if value is None else str(value).strip()
|
||||
def _parse_json_list(script:str):
|
||||
stripped=script.lstrip()
|
||||
if not stripped.startswith("["): return None
|
||||
|
||||
+78
-20
@@ -16,11 +16,18 @@ from ..constants import (
|
||||
SEAM_CORRECTION_OPTIONS, V2_CONTINUITY_AUTO, normalize_diagnostics_mode,
|
||||
)
|
||||
from ..continuation import POLICY_REPLACE, prepare_conditioning
|
||||
from ..driving_audio import (
|
||||
attach_driving_audio,
|
||||
combine_driving_audio_identity,
|
||||
encode_driving_audio,
|
||||
slice_driving_audio_latent,
|
||||
)
|
||||
from ..model_patch import clone_model_for_chunk
|
||||
from ..state import assert_context_unchanged, context_fingerprint, make_plan, select_context, validate_state
|
||||
from ..temporal import align_frame_count_up, largest_context_capacity, make_extension_shape
|
||||
from ..version import PACKAGE_VERSION
|
||||
from .decoder import decode_sequence, decode_sequence_with_seam, enforce_total_frames
|
||||
from .context_diagnostics import ContextDiagnosticsTracker
|
||||
from .h3_builder import attach_keyframes, empty_h3_latent, encode_identity_latents, encode_prompt_conditioning, prepare_identity_assets
|
||||
from .motion import choose_context_frames
|
||||
from .prompts import prompt_plan_report, validate_prompt_plan
|
||||
@@ -33,6 +40,15 @@ from .session import (
|
||||
LOG=logging.getLogger("h3_continuum_join")
|
||||
class SequenceRuntimeError(RuntimeError): pass
|
||||
|
||||
def _record_context_diagnostics(*,tracker,reports,state,continuity,reused):
|
||||
if tracker is None: return
|
||||
try:
|
||||
context_frames,_,_=choose_context_frames(continuity,state)
|
||||
video_context,_,_=select_context(state,context_frames,include_audio=False)
|
||||
reports.append(tracker.observe(video_context,source_chunk=int(state["clip_index"]),context_frames=context_frames,reused=bool(reused)))
|
||||
except Exception as exc:
|
||||
reports.append(f"context diagnostics source chunk {state.get('clip_index','?')}: unavailable ({exc})")
|
||||
|
||||
def _check_decode_memory_budget(*,width,height,chunks,chunk_seconds):
|
||||
target_frames=max(1,int(round(chunks*chunk_seconds*FPS))); max_chunk_frames=max(5,int(round(chunk_seconds*FPS))+39)
|
||||
image_bytes=target_frames*width*height*3*4; transient_bytes=max_chunk_frames*width*height*3*4
|
||||
@@ -40,8 +56,6 @@ def _check_decode_memory_budget(*,width,height,chunks,chunk_seconds):
|
||||
try:
|
||||
import psutil
|
||||
available=int(psutil.virtual_memory().available); available_gib=available/1024**3
|
||||
if required_bytes>int(available*0.75):
|
||||
raise SequenceRuntimeError(f"estimated decoded output requires about {estimate_gib:.1f} GiB including one-chunk headroom, but only {available_gib:.1f} GiB RAM is currently available. Reduce chunks, chunk_seconds, or resolution before sampling.")
|
||||
except ImportError: pass
|
||||
return estimate_gib,available_gib
|
||||
|
||||
@@ -50,16 +64,21 @@ def _clone_entry_for_reuse(entry):
|
||||
|
||||
def _preserved_prefix(*,session,prompt_hashes,chunks,reroll_from_chunk,width,height,chunk_seconds,identity_hash):
|
||||
if session is None: return [],[]
|
||||
session=validate_session(session); notes=[]
|
||||
if int(session["width"])!=int(width) or int(session["height"])!=int(height): raise SessionValidationError(f"session is {session['width']}x{session['height']} but V2 is configured for {width}x{height}")
|
||||
if abs(float(session["chunk_seconds"])-float(chunk_seconds))>1e-6: raise SessionValidationError(f"session chunk_seconds={session['chunk_seconds']} does not match {chunk_seconds}")
|
||||
if str(session.get("identity_hash","none"))!=str(identity_hash): raise SessionValidationError("the current first-frame identity image does not match the saved session")
|
||||
notes=[]
|
||||
try: session=validate_session(session)
|
||||
except SessionValidationError as exc: return [],[f"saved session was ignored; generated a fresh run ({exc})"]
|
||||
if int(session["width"])!=int(width) or int(session["height"])!=int(height): return [],["saved session resolution differs; generated a fresh run"]
|
||||
if abs(float(session["chunk_seconds"])-float(chunk_seconds))>1e-6: return [],["saved session chunk duration differs; generated a fresh run"]
|
||||
if str(session.get("identity_hash","none"))!=str(identity_hash): return [],["saved session identity differs; generated a fresh run"]
|
||||
if reroll_from_chunk<0 or reroll_from_chunk>chunks: raise SessionValidationError("reroll_from_chunk must be 0 or a valid one-based chunk index")
|
||||
limit=min(len(session["chunks"]),chunks)
|
||||
if reroll_from_chunk>0: limit=min(limit,reroll_from_chunk-1)
|
||||
preserved=[]
|
||||
for index in range(limit):
|
||||
entry=validate_chunk_entry(session["chunks"][index])
|
||||
try: entry=validate_chunk_entry(session["chunks"][index])
|
||||
except SessionValidationError as exc:
|
||||
notes.append(f"session reuse stopped before chunk {index+1}: stored chunk was rejected ({exc})")
|
||||
break
|
||||
if entry["prompt_hash"]!=prompt_hashes[index]: notes.append(f"session reuse stopped before chunk {index+1}: prompt changed"); break
|
||||
preserved.append(_clone_entry_for_reuse(entry))
|
||||
if reroll_from_chunk==0 and len(preserved)==min(len(session["chunks"]),chunks):
|
||||
@@ -79,13 +98,17 @@ def _conditioning_cache(*,clip,prompts,assets,final_has_last_frame,reference_ass
|
||||
cache[key]=encode_prompt_conditioning(clip,prompt,first_image=assets.first_image,last_image=assets.last_image if include_last else None,reference_audio_assets=reference_audio_assets,timeline_video_assets=timeline_video_assets)
|
||||
return cache
|
||||
|
||||
def run_sequence(*,model:Any,clip:Any,video_vae:Any,audio_vae:Any,sampler:Any,sigmas:torch.Tensor,first_frame:torch.Tensor|None,last_frame:torch.Tensor|None,prompt_plan:dict[str,Any],width:int,height:int,continuity:str,base_seed:int,audio_continuity:bool,exact_total_duration:bool,diagnostics_mode:str,reroll_from_chunk:int,reroll_nonce:int,strict_compatibility:bool,debug:bool,seam_correction:str=SEAM_CORRECTION_OFF,enable_preview:bool=True,session:dict[str,Any]|None=None,initial_state:dict[str,Any]|None=None,latent_only:bool=False,reference_assets=None,reference_audio_source=None,reference_audio_vae=None,timeline_video_source=None):
|
||||
def run_sequence(*,model:Any,clip:Any,video_vae:Any,audio_vae:Any,sampler:Any,sigmas:torch.Tensor,first_frame:torch.Tensor|None,last_frame:torch.Tensor|None,prompt_plan:dict[str,Any],width:int,height:int,continuity:str,base_seed:int,audio_continuity:bool,exact_total_duration:bool,diagnostics_mode:str,reroll_from_chunk:int,reroll_nonce:int,strict_compatibility:bool,debug:bool,seam_correction:str=SEAM_CORRECTION_OFF,enable_preview:bool=True,session:dict[str,Any]|None=None,initial_state:dict[str,Any]|None=None,latent_only:bool=False,reference_assets=None,reference_audio_source=None,reference_audio_vae=None,driving_audio_source=None,driving_audio_vae=None,reference_video_source=None,timeline_video_source=None):
|
||||
from ..conditioning import detect_conditioning_mode, conditioning_mode_label
|
||||
from ..run_storage import get_active_run_storage
|
||||
storage_controller=get_active_run_storage()
|
||||
diagnostics_mode=normalize_diagnostics_mode(diagnostics_mode); plan=validate_prompt_plan(prompt_plan); chunks=int(plan["chunks"]); chunk_seconds=float(plan["chunk_seconds"]); prompts=list(plan["prompts"]); prompt_hashes=list(plan["hashes"]); width,height=int(width),int(height)
|
||||
# Legacy workflow input only. Runtime compatibility is advisory in V3.4.
|
||||
strict_compatibility=False
|
||||
if width<=0 or height<=0 or width%32 or height%32: raise SequenceRuntimeError("width and height must be positive multiples of 32")
|
||||
if session is not None and initial_state is not None: raise SequenceRuntimeError("connect either a session or an initial_state, not both")
|
||||
if session is not None and initial_state is not None:
|
||||
LOG.warning("Both session and initial_state were supplied; using the session and ignoring initial_state")
|
||||
initial_state=None
|
||||
if diagnostics_mode not in DIAGNOSTICS_OPTIONS: raise SequenceRuntimeError(f"unknown diagnostics mode: {diagnostics_mode!r}")
|
||||
if seam_correction not in SEAM_CORRECTION_OPTIONS: raise SequenceRuntimeError(f"unknown seam correction mode: {seam_correction!r}")
|
||||
if not 0<=int(reroll_from_chunk)<=chunks: raise SequenceRuntimeError("reroll_from_chunk must be 0 or a valid one-based chunk index")
|
||||
@@ -95,13 +118,17 @@ def run_sequence(*,model:Any,clip:Any,video_vae:Any,audio_vae:Any,sampler:Any,si
|
||||
reference_warning=""
|
||||
if reference_assets is not None:
|
||||
from ..reference import validate_reference_prompts
|
||||
if first_frame is not None or last_frame is not None: raise SequenceRuntimeError("Reference images cannot be combined with First Frame or Last Frame in V3.2")
|
||||
reference_warning=validate_reference_prompts(prompts,reference_assets.count)
|
||||
from ..reference_audio import (
|
||||
combine_reference_audio_identity,
|
||||
validate_reference_audio_prompts,
|
||||
)
|
||||
reference_audio_warning=validate_reference_audio_prompts(prompts,reference_audio_source)
|
||||
from ..reference_video import (
|
||||
combine_reference_video_identity,
|
||||
validate_reference_video_prompts,
|
||||
)
|
||||
reference_video_warning=validate_reference_video_prompts(prompts,reference_video_source)
|
||||
from ..timeline_video import (
|
||||
combine_timeline_video_identity,
|
||||
validate_timeline_video_prompts,
|
||||
@@ -110,15 +137,15 @@ def run_sequence(*,model:Any,clip:Any,video_vae:Any,audio_vae:Any,sampler:Any,si
|
||||
decode_estimate_gib=0.0; available_ram_gib=None
|
||||
if not latent_only: decode_estimate_gib,available_ram_gib=_check_decode_memory_budget(width=width,height=height,chunks=chunks,chunk_seconds=chunk_seconds)
|
||||
issues=check_comfy_h3_runtime()
|
||||
strict_compatibility=False
|
||||
if issues: LOG.warning("H3 runtime compatibility notes: %s","; ".join(issues))
|
||||
assets=prepare_identity_assets(video_vae,width=width,height=height,first_frame=first_frame,last_frame=last_frame,encode_latents=False)
|
||||
visual_identity_hash=reference_assets.combined_hash if reference_assets is not None else assets.identity_hash
|
||||
sequence_identity_hash=combine_reference_audio_identity(visual_identity_hash,reference_audio_source)
|
||||
sequence_identity_hash=combine_driving_audio_identity(sequence_identity_hash,driving_audio_source)
|
||||
sequence_identity_hash=combine_reference_video_identity(sequence_identity_hash,reference_video_source)
|
||||
sequence_identity_hash=combine_timeline_video_identity(sequence_identity_hash,timeline_video_source)
|
||||
current_model_fingerprint=model_fingerprint(model,extra_wrapper_keys=("h3_continuum_join.apply_model.v1",))
|
||||
if storage_controller is not None:
|
||||
stored_session=storage_controller.prepare(model=model,model_fingerprint_value=current_model_fingerprint,clip=clip,video_vae=video_vae,sampler=sampler,sigmas=sigmas,prompt_plan=plan,width=width,height=height,chunk_seconds=chunk_seconds,continuity=continuity,audio_continuity=audio_continuity,base_seed=base_seed,reroll_from_chunk=reroll_from_chunk,reroll_nonce=reroll_nonce,first_frame_hash=assets.first_frame_hash,last_frame_hash=assets.last_frame_hash,identity_hash=sequence_identity_hash,strict_compatibility=False,existing_session=session,reference_contract=reference_assets.contract if reference_assets is not None else None,conditioning_mode=conditioning_mode,reference_audio_contract=reference_audio_source.contract if reference_audio_source is not None else None,reference_audio_vae=reference_audio_vae,timeline_video_contract=timeline_video_source.contract if timeline_video_source is not None else None)
|
||||
stored_session=storage_controller.prepare(model=model,model_fingerprint_value=current_model_fingerprint,clip=clip,video_vae=video_vae,sampler=sampler,sigmas=sigmas,prompt_plan=plan,width=width,height=height,chunk_seconds=chunk_seconds,continuity=continuity,audio_continuity=audio_continuity,base_seed=base_seed,reroll_from_chunk=reroll_from_chunk,reroll_nonce=reroll_nonce,first_frame_hash=assets.first_frame_hash,last_frame_hash=assets.last_frame_hash,identity_hash=sequence_identity_hash,strict_compatibility=strict_compatibility,existing_session=session,reference_contract=reference_assets.contract if reference_assets is not None else None,conditioning_mode=conditioning_mode,reference_audio_contract=reference_audio_source.contract if reference_audio_source is not None else None,reference_audio_vae=reference_audio_vae,driving_audio_contract=driving_audio_source.contract if driving_audio_source is not None else None,driving_audio_vae=driving_audio_vae,reference_video_contract=reference_video_source.contract if reference_video_source is not None else None,timeline_video_contract=timeline_video_source.contract if timeline_video_source is not None else None)
|
||||
reroll_nonce=storage_controller.effective_reroll_nonce
|
||||
if stored_session is not None: session=stored_session
|
||||
accelerators=accelerator_summary(model)
|
||||
@@ -131,6 +158,11 @@ def run_sequence(*,model:Any,clip:Any,video_vae:Any,audio_vae:Any,sampler:Any,si
|
||||
if reference_audio_source is not None:
|
||||
reuse_notes.insert(0,"Reference Audio 1 conditioning: persistent across all chunks.")
|
||||
if reference_audio_warning: reuse_notes.append(reference_audio_warning)
|
||||
if driving_audio_source is not None:
|
||||
reuse_notes.insert(0,"Driving Audio: absolute-time guide slices enabled; original audio is preserved for final output.")
|
||||
if reference_video_source is not None:
|
||||
reuse_notes.insert(0,f"Video Reference conditioning: persistent across all chunks, 24 fps, resolved={reference_video_source.target_width}x{reference_video_source.target_height}, frames={reference_video_source.frame_count}.")
|
||||
if reference_video_warning: reuse_notes.append(reference_video_warning)
|
||||
if timeline_video_source is not None:
|
||||
reuse_notes.insert(0,f"Timeline Video conditioning: chunk-local slices, size={timeline_video_source.size_mode}, resolved={timeline_video_source.target_width}x{timeline_video_source.target_height}.")
|
||||
if timeline_video_warning: reuse_notes.append(timeline_video_warning)
|
||||
@@ -147,17 +179,38 @@ def run_sequence(*,model:Any,clip:Any,video_vae:Any,audio_vae:Any,sampler:Any,si
|
||||
if reference_audio_source is not None:
|
||||
from ..reference_audio import encode_reference_audio
|
||||
reference_audio_assets=encode_reference_audio(reference_audio_vae,reference_audio_source)
|
||||
driving_audio_assets=encode_driving_audio(driving_audio_source,driving_audio_vae)
|
||||
reference_video_assets=None
|
||||
if reference_video_source is not None:
|
||||
from ..reference_video import encode_reference_video
|
||||
reference_video_assets=encode_reference_video(video_vae,reference_video_source)
|
||||
if timeline_video_source is None:
|
||||
cache=_conditioning_cache(clip=clip,prompts=prompts,assets=assets,final_has_last_frame=last_frame is not None,reference_assets=reference_assets,reference_audio_assets=reference_audio_assets)
|
||||
cache=_conditioning_cache(clip=clip,prompts=prompts,assets=assets,final_has_last_frame=last_frame is not None,reference_assets=reference_assets,reference_audio_assets=reference_audio_assets,timeline_video_assets=reference_video_assets)
|
||||
entries=preserved[:]; previous_state=None
|
||||
if entries: previous_state=entry_to_state(entries[-1])
|
||||
if entries:
|
||||
try: previous_state=entry_to_state(entries[-1])
|
||||
except (SessionValidationError, ValueError) as exc:
|
||||
reuse_notes.append(f"saved continuation state was rejected; generated a fresh run ({exc})")
|
||||
entries=[]; previous_state=None
|
||||
elif initial_state is not None:
|
||||
previous_state=validate_state(initial_state)
|
||||
if int(previous_state["width"])!=width or int(previous_state["height"])!=height: raise SequenceRuntimeError(f"initial_state is {previous_state['width']}x{previous_state['height']} but V2 is configured for {width}x{height}")
|
||||
try:
|
||||
candidate=validate_state(initial_state)
|
||||
if int(candidate["width"])!=width or int(candidate["height"])!=height:
|
||||
reuse_notes.append("initial_state resolution differs; generated a fresh run")
|
||||
else: previous_state=candidate
|
||||
except ValueError as exc:
|
||||
reuse_notes.append(f"initial_state was rejected; generated a fresh run ({exc})")
|
||||
initial_frame_count=align_frame_count_up(int(round(chunk_seconds*FPS))); retained_frames=sum(int(entry["plan"]["net_frames"]) for entry in entries); sampling_reports=[]
|
||||
context_diagnostics=ContextDiagnosticsTracker() if bool(debug) else None
|
||||
if context_diagnostics is not None:
|
||||
if entries:
|
||||
for reused_entry in entries:
|
||||
_record_context_diagnostics(tracker=context_diagnostics,reports=sampling_reports,state=entry_to_state(reused_entry),continuity=continuity,reused=True)
|
||||
elif previous_state is not None:
|
||||
_record_context_diagnostics(tracker=context_diagnostics,reports=sampling_reports,state=previous_state,continuity=continuity,reused=True)
|
||||
for sequence_index in range(len(entries),chunks):
|
||||
prompt=prompts[sequence_index]; prompt_hash_value=prompt_hashes[sequence_index]; is_final=sequence_index==chunks-1; effective_reroll_nonce=int(reroll_nonce) if int(reroll_from_chunk)>0 and sequence_index+1>=int(reroll_from_chunk) else 0; seed=derive_chunk_seed(base_seed,sequence_index,effective_reroll_nonce); motion_score=0.0; video_context=None; audio_context=None; context_before=None
|
||||
timeline_video_assets=None
|
||||
timeline_video_assets=reference_video_assets
|
||||
chunk_cache=cache
|
||||
if timeline_video_source is not None:
|
||||
from ..timeline_video import encode_timeline_video_chunk
|
||||
@@ -172,20 +225,25 @@ def run_sequence(*,model:Any,clip:Any,video_vae:Any,audio_vae:Any,sampler:Any,si
|
||||
video_context,audio_context,grid_offset=select_context(previous_state,context_frames,include_audio=bool(audio_continuity)); context_before=context_fingerprint(video_context,audio_context)
|
||||
conditioning=prepare_conditioning(base_conditioning,video_context=video_context,audio_context=audio_context,audio_grid_offset=grid_offset,context_frames=context_frames,new_frame_count=shape.total_frames,first_frame_policy=POLICY_REPLACE,preserve_last_frame=True)
|
||||
clip_index=int(previous_state["clip_index"])+1; chunk_plan=make_plan(continuation=True,clip_index=clip_index,total_frames=shape.total_frames,trim_frames=context_frames,width=width,height=height,context_frames=context_frames,state_capacity_frames=largest_context_capacity(shape.net_new_frames),requested_extend_seconds=chunk_seconds,debug=debug)
|
||||
chunk_model=clone_model_for_chunk(model,strict=False,debug=bool(debug),chunk_index=clip_index,context_frames=context_frames if previous_state is not None else None)
|
||||
driving_audio_latent=slice_driving_audio_latent(driving_audio_assets,cumulative_retained_before=retained_frames,total_frames=int(chunk_plan["total_frames"]),trim_frames=int(chunk_plan["trim_frames"]),fps=FPS)
|
||||
conditioning=attach_driving_audio(conditioning,driving_audio_latent)
|
||||
chunk_model=clone_model_for_chunk(model,strict=bool(strict_compatibility),debug=bool(debug),chunk_index=clip_index,context_frames=context_frames if previous_state is not None else None)
|
||||
sampled=sample_chunk(model=chunk_model,conditioning=conditioning,latent=latent,sampler=sampler,sigmas=sigmas,seed=seed,enable_preview=bool(enable_preview))
|
||||
if context_before is not None and video_context is not None: assert_context_unchanged(video_context,audio_context,context_before)
|
||||
entry=make_chunk_entry(latent=sampled,plan=chunk_plan,prompt=prompt,prompt_hash=prompt_hash_value,seed=seed,context_frames=context_frames,motion_score=motion_score,reused=False); previous_state=entry_to_state(entry); entries.append(entry)
|
||||
_record_context_diagnostics(tracker=context_diagnostics,reports=sampling_reports,state=previous_state,continuity=continuity,reused=False)
|
||||
if storage_controller is not None: storage_controller.commit_chunk(entry, position=sequence_index)
|
||||
retained_frames+=int(chunk_plan["net_frames"])
|
||||
sampling_reports.append(f"chunk {sequence_index+1}/{chunks}: seed={seed}, frames={chunk_plan['total_frames']}, trim={chunk_plan['trim_frames']}, retained_total={retained_frames}, context={context_frames} ({reason}), motion={motion_score:.6f}, "+(f"interop=emitted actual_prefix={CONTINUUM_ACTUAL_PREFIX_STEPS} consumer=not_observable" if context_before is not None else "interop=not_emitted"))
|
||||
del sampled,latent,conditioning,chunk_model,chunk_cache
|
||||
if timeline_video_assets is not None: del timeline_video_assets
|
||||
if timeline_video_source is not None and timeline_video_assets is not None: del timeline_video_assets
|
||||
if len(entries)!=chunks: raise SequenceRuntimeError(f"internal sequence length mismatch: expected {chunks}, got {len(entries)}")
|
||||
if latent_only:
|
||||
last_state=entry_to_state(entries[-1]); parent_id=session.get("session_id") if session is not None else None
|
||||
settings={"continuity":continuity,"audio_continuity":bool(audio_continuity),"exact_total_duration":False,"prompt_mode":plan["mode"],"conditioning_mode":conditioning_mode,"base_seed":int(base_seed),"reroll_nonce":int(reroll_nonce),"diagnostics_mode":diagnostics_mode,"initial_state_source":initial_state is not None,"latent_first":True,"first_frame_hash":assets.first_frame_hash,"last_frame_hash":assets.last_frame_hash,"reference_contract":reference_assets.contract if reference_assets is not None else None}
|
||||
if reference_audio_source is not None: settings["reference_audio_contract"]=reference_audio_source.contract
|
||||
if driving_audio_source is not None: settings["driving_audio_contract"]=driving_audio_source.contract
|
||||
if reference_video_source is not None: settings["reference_video_contract"]=reference_video_source.contract
|
||||
if timeline_video_source is not None: settings["timeline_video_contract"]=timeline_video_source.contract
|
||||
new_session=make_session(chunks=entries,width=width,height=height,chunk_seconds=chunk_seconds,identity_hash=sequence_identity_hash,model_fingerprint_value=current_model_fingerprint,parent_session_id=parent_id,reroll_from_chunk=int(reroll_from_chunk),settings=settings)
|
||||
report_lines=[f"H3 Continuum V3 {PACKAGE_VERSION}",f"Conditioning mode: {conditioning_mode_label(conditioning_mode)}.",prompt_plan_report(plan),"Decode: external ComfyUI Core VAE nodes; full raw AV chunks retained.",accelerators,"Execution: conditioning precomputed; single call-local MODEL clone per chunk; no internal VAE decode.",*reuse_notes]
|
||||
|
||||
+2
-2
@@ -112,8 +112,8 @@ def validate_chunk_entry(entry: dict[str, Any]) -> dict[str, Any]:
|
||||
actual_frames = pixel_frames_for_latent_t(int(video.shape[2]))
|
||||
if actual_frames != int(plan["total_frames"]):
|
||||
raise SessionValidationError("session chunk video length does not match its plan")
|
||||
if not isinstance(entry.get("prompt"), str) or not entry["prompt"].strip():
|
||||
raise SessionValidationError("session chunk prompt is empty")
|
||||
if not isinstance(entry.get("prompt"), str):
|
||||
raise SessionValidationError("session chunk prompt is not a string")
|
||||
if not isinstance(entry.get("prompt_hash"), str) or len(entry["prompt_hash"]) != 64:
|
||||
raise SessionValidationError("session chunk prompt hash is invalid")
|
||||
if int(entry.get("seed", -1)) < 0:
|
||||
|
||||
+12
-4
@@ -252,6 +252,9 @@ class H3ContinuumSamplerV3:
|
||||
reference_assets=None,
|
||||
reference_audio_source=None,
|
||||
reference_audio_vae=None,
|
||||
driving_audio_source=None,
|
||||
driving_audio_vae=None,
|
||||
reference_video_source=None,
|
||||
timeline_video_source=None,
|
||||
):
|
||||
if prompt_overrides is not None and not isinstance(prompt_overrides, dict):
|
||||
@@ -328,6 +331,9 @@ class H3ContinuumSamplerV3:
|
||||
reference_assets=reference_assets,
|
||||
reference_audio_source=reference_audio_source,
|
||||
reference_audio_vae=reference_audio_vae,
|
||||
driving_audio_source=driving_audio_source,
|
||||
driving_audio_vae=driving_audio_vae,
|
||||
reference_video_source=reference_video_source,
|
||||
timeline_video_source=timeline_video_source,
|
||||
**clip_prompt_inputs,
|
||||
)
|
||||
@@ -620,6 +626,9 @@ class H3ContinuumSamplerProduction(H3ContinuumSamplerV3):
|
||||
reference_image_3=None,
|
||||
reference_audio_1=None,
|
||||
reference_audio_vae=None,
|
||||
driving_audio_source=None,
|
||||
driving_audio_vae=None,
|
||||
reference_video_source=None,
|
||||
timeline_video_source=None,
|
||||
):
|
||||
runtime_started_at = time.perf_counter()
|
||||
@@ -641,10 +650,6 @@ class H3ContinuumSamplerProduction(H3ContinuumSamplerV3):
|
||||
reference_audio_1,
|
||||
reference_audio_vae,
|
||||
)
|
||||
if reference_assets is not None and (first_frame is not None or last_frame is not None):
|
||||
raise ValueError(
|
||||
"Reference Images cannot be combined with First Frame or Last Frame"
|
||||
)
|
||||
def mark_runtime_start(assembly_plan):
|
||||
marked = dict(assembly_plan)
|
||||
marked["_runtime_started_at"] = runtime_started_at
|
||||
@@ -670,6 +675,9 @@ class H3ContinuumSamplerProduction(H3ContinuumSamplerV3):
|
||||
reference_assets=reference_assets,
|
||||
reference_audio_source=reference_audio_source,
|
||||
reference_audio_vae=reference_audio_vae,
|
||||
driving_audio_source=driving_audio_source,
|
||||
driving_audio_vae=driving_audio_vae,
|
||||
reference_video_source=reference_video_source,
|
||||
timeline_video_source=timeline_video_source,
|
||||
advanced={
|
||||
"audio_continuity": bool(audio_continuity),
|
||||
|
||||
Reference in New Issue
Block a user