Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
bfaa5fa733 | ||
|
|
9923a3e417 | ||
|
|
ea6fae7e56 | ||
|
|
72f9b07ed8 | ||
|
|
8d4232a142 |
@@ -0,0 +1,53 @@
|
||||
H3-LongVideos — Licence
|
||||
Copyright (c) 2026 Smite79. All rights reserved.
|
||||
|
||||
This licence applies to every version published on or after 2026-09-07.
|
||||
|
||||
WHAT YOU MAY DO
|
||||
|
||||
1. Download and use this software, in unmodified or modified form, for your
|
||||
own purposes, personal or commercial. Rendering with it, and whatever you
|
||||
render with it, is yours and is not covered by this licence.
|
||||
|
||||
2. Modify your own copy.
|
||||
|
||||
3. Submit changes back to the original project.
|
||||
|
||||
WHAT YOU MAY NOT DO WITHOUT WRITTEN PERMISSION
|
||||
|
||||
4. Redistribute this software, in whole or in part, modified or unmodified.
|
||||
That includes publishing it to any repository, registry, model hub, node
|
||||
manager, marketplace, or mirror; bundling it inside another package,
|
||||
product, image, or installer; and hosting it as a service.
|
||||
|
||||
5. Remove, alter, or obscure the copyright notice above, this licence, or the
|
||||
attribution in the source files — including where a permitted redistribution
|
||||
has been agreed.
|
||||
|
||||
6. Represent this software, or a derivative of it, as your own work.
|
||||
|
||||
ASKING
|
||||
|
||||
Permission for anything under 4 is granted case by case and is usually given
|
||||
for things like inclusion in a node manager. Ask via the project's GitHub
|
||||
issues at https://github.com/Smite79/MiniMax-H3-LongVideos.
|
||||
|
||||
EARLIER VERSIONS
|
||||
|
||||
Versions published before 2026-09-07 were released under Apache License 2.0.
|
||||
That grant is irrevocable for those versions: copies obtained under it stay
|
||||
under it, and this licence does not and cannot withdraw it retroactively. It
|
||||
governs this version and every version after it.
|
||||
|
||||
Apache 2.0 also required attribution, so a copy of an earlier version
|
||||
republished with the copyright notice stripped was already in breach of the
|
||||
licence it was taken under.
|
||||
|
||||
NO WARRANTY
|
||||
|
||||
This software is provided "as is", without warranty of any kind, express or
|
||||
implied, including but not limited to the warranties of merchantability,
|
||||
fitness for a particular purpose, and non-infringement. In no event shall the
|
||||
copyright holder be liable for any claim, damages, or other liability, whether
|
||||
in an action of contract, tort, or otherwise, arising from, out of, or in
|
||||
connection with the software or the use or other dealings in the software.
|
||||
@@ -33,35 +33,32 @@
|
||||
- Outputs: `plan`, `image1`..`image9`, `connected_images`
|
||||
- Reads back the nine optional images for a selected MiniMax H3 plan scene, for example by connecting the current `clip_index`.
|
||||
|
||||
- `Dumas H3 Long Videos (FL2VA + REF2VA)`
|
||||
- Inputs: H3 model stack, prompt socket, optional `first_frame`, optional `ref_1`..`ref_9`, plus the upstream long-video control surface for pacing, continuity, audio, overlays, and guards
|
||||
- Outputs: `images`, `audio`, `info`, `script`, `frames_per_shot`, `total_frames`, `shots`, `video_seconds`, `fps`, `fps_int`, `latent`, `soundscape`
|
||||
- First-pass Dumas port of the `MiniMax-H3-Longvideos` sampler, brought in as a local starting point for long-form H3 chaining work.
|
||||
- Keeps the upstream split-beats / handoff / ref-routing behavior close to source so future Dumas-specific improvements can be compared against a known baseline.
|
||||
- Full user guide: [`H3_LONG_VIDEOS_GUIDE.md`](./H3_LONG_VIDEOS_GUIDE.md)
|
||||
- Only the canonical `DumasH3LongVideos` node key is exposed now; the older FL2VA/REF2VA alias entries are no longer duplicated in the Add Node menu.
|
||||
- Prompt `<Picture N>` tags now map to the actual ref socket numbers you wire, even with gaps such as only `ref_2` and `ref_7` connected.
|
||||
- Character refs now contribute appearance and wardrobe context from the same structured object, while location refs contribute environment context from theirs.
|
||||
- The default ref2v bias is now stronger: `ref_mode` defaults to `auto ref2v` so untagged prompts condition every shot instead of only shot 1, and `ref_noise_aug` defaults to `0.95` rather than the upstream-literal `0.999`.
|
||||
- `Dumas H3 Latent Upscale Params` provides the optional pre-decode latent refinement stage for the long-video node.
|
||||
- Per-shot directives now support `continuity:`, `ref_mode:`, `ref_noise_aug:`, `anchor_add:`, `soundscape:`, and `music:` in addition to the existing timing and wardrobe directives.
|
||||
- `Dumas H3 Long Videos`
|
||||
- Inputs/outputs: the current upstream `MiniMax-H3-Longvideos` sampler surface, exposed under the existing `DumasH3LongVideos` key for saved Dumas workflows.
|
||||
- The local Dumas prompt-engineering fork has been removed from this node. Long Videos now wraps the upstream sampler/engine/runtime/audio/conditioning/shot-plan modules directly so it can track the source project again.
|
||||
- Upstream compatibility keys `H3LongVideos`, `H3LongVideosFL2VA`, `H3LongVideosV1`, and `H3LongVideosREF2VA` are also registered to the same class.
|
||||
- The old Dumas browser widget grouping script is disabled for this node because it targeted controls that no longer exist on the upstream sampler.
|
||||
- `handoff_frames` extends the upstream last-frame handoff: `1` keeps the current single keyframe behavior; higher values keep that final-frame keyframe and add earlier tail frames from the previous shot as claimed reference context for the next beat.
|
||||
- Upstream license text is included in [`H3_LONGVIDEOS_UPSTREAM_LICENSE.txt`](./H3_LONGVIDEOS_UPSTREAM_LICENSE.txt).
|
||||
|
||||
- `Dumas H3 Latent Upscale Params`
|
||||
- Inputs: `mode`, `model_name`, `method`, `width`, `height`, `device`, `precision`, `sampler_name`, `scheduler`, `steps`, `denoise`, `megapixels`, `tile_width`, `tile_height`, `overlap`, `fade_width`, `fade_height`, `overlap_mode`, `overlap_blend`, `tile_size_mode`, `grid_rows`, `grid_cols`, `spatial_w_overlap`, `spatial_h_overlap`, `min_tile_size`, `masked_area_noise`, `brightness_match`, `dynamic_fade`, `dynamic_fade_min`, `chunk_length`, `temporal_overlap`, `resize_conditioning`, `anchor_strength`
|
||||
- Output: `latent_upscale_param`
|
||||
- Bundles the optional latent-space upscaler settings used by `Dumas H3 Long Videos` before decode, so the main node can rebuild conditioning at the target size and run a short refinement pass with your chosen sampler, scheduler, step count, denoise, and the full upstream spatial split controls.
|
||||
- Legacy helper from the abandoned Dumas Long Videos fork. The current upstream-backed `Dumas H3 Long Videos` node does not consume this socket; it uses the upstream latent-upscale controls on the Long Videos node itself.
|
||||
|
||||
- `Dumas H3 Beat Prompt`
|
||||
- Inputs: authored through the custom front-end beat editor
|
||||
- Output: `prompt`
|
||||
- Builds one H3 prompt block per beat, with quick controls for per-shot timing, continuity, ref behavior, anchor additions, soundscape, and music while staying compatible with direct text editing.
|
||||
- Builds an upstream-compatible Long Videos prompt: optional scene paragraph, optional character sheet, then one blank-line-separated textbox per beat.
|
||||
- Per-beat helpers only emit upstream-supported state directives: `remove:` / `removed:` / `off:` and `add:` / `wear:` / `wearing:`.
|
||||
- Old Dumas-only beat directives such as `seconds:`, `continuity:`, `ref_mode:`, `ref_noise_aug:`, `anchor_add:`, `soundscape:`, and `music:` are stripped from the generated prompt so they are not sent to the upstream node as visible text.
|
||||
|
||||
- `Dumas H3 Prompt Curator`
|
||||
- Inputs: `action_prompt`, `anatomy_guard`, `subject_count_guard`, optional `anchor`, optional `soundscape`, optional `bgm`, optional `ref_1` through `ref_9`
|
||||
- Outputs: `prompt`, `ref_image_1` through `ref_image_9`, `reference_count`, `debug`, `anchor`, `sounds`, `bgm`, `original_ref_1` through `original_ref_9`
|
||||
- Outputs: `prompt`, `ref_image_1` through `ref_image_9`, `reference_count`, `debug`, `anchor`, `sounds`, `bgm`, `original_ref_1` through `original_ref_9`, `compiled_ref_description_1` through `compiled_ref_description_9`
|
||||
- Builds one standalone MiniMax H3 prompt from your final action text plus structured character/location references.
|
||||
- The action text can mention references by character/location name, alias, `<Picture N>`, or `<refN>`. Only mentioned references are emitted, and the output images are compacted/renumbered so skipped inputs do not leave gaps.
|
||||
- Extra component outputs expose the cleaned anchor, sounds, BGM, and each selected original reference image in compacted order.
|
||||
- Extra component outputs expose the cleaned anchor, sounds, BGM, and each selected original reference image plus its compiled reference description in compacted order.
|
||||
- Adds curated reference context, anatomy guard text, optional subject-count guard text, anchor/style text, `overall_soundscape:` text, and `background_music:` text while respecting MiniMax H3's reference-generation shape: one prompt plus up to nine reference images.
|
||||
|
||||
- `Dumas H3 Shot Length`
|
||||
|
||||
+55
-8
@@ -2,11 +2,46 @@ import json
|
||||
|
||||
|
||||
_DEFAULT_BEAT = "Describe this beat."
|
||||
_DEFAULT_STATE = {"beats": [{"text": _DEFAULT_BEAT}]}
|
||||
_DEFAULT_STATE = {"scene": "", "character_sheet": "", "beats": [{"text": _DEFAULT_BEAT}]}
|
||||
_LEGACY_DIRECTIVE_PREFIXES = (
|
||||
"seconds",
|
||||
"duration",
|
||||
"continuity",
|
||||
"ref_mode",
|
||||
"ref_noise_aug",
|
||||
"anchor_add",
|
||||
"overall_soundscape",
|
||||
"soundscape",
|
||||
"non_diegetic_music",
|
||||
"music",
|
||||
"wardrobe",
|
||||
"enter",
|
||||
"exit",
|
||||
)
|
||||
|
||||
|
||||
def _clone_default_state():
|
||||
return {"beats": [{"text": _DEFAULT_BEAT}]}
|
||||
return {
|
||||
"scene": "",
|
||||
"character_sheet": "",
|
||||
"beats": [{"text": _DEFAULT_BEAT}],
|
||||
}
|
||||
|
||||
|
||||
def _strip_legacy_directives(text):
|
||||
"""Remove directives from the abandoned Dumas Long Videos fork.
|
||||
|
||||
The upstream Long Videos node sends unknown field labels to the model as text,
|
||||
so this builder strips the old managed controls rather than emitting prompts
|
||||
that ask H3 to draw labels such as "seconds:" or "music:" in the frame.
|
||||
"""
|
||||
kept = []
|
||||
for line in str(text or "").splitlines():
|
||||
lowered = line.strip().lower()
|
||||
if any(lowered.startswith(f"{name}:") for name in _LEGACY_DIRECTIVE_PREFIXES):
|
||||
continue
|
||||
kept.append(line)
|
||||
return "\n".join(kept).strip()
|
||||
|
||||
|
||||
def _parse_beat_prompt_state(value):
|
||||
@@ -21,6 +56,8 @@ def _parse_beat_prompt_state(value):
|
||||
except Exception:
|
||||
return _clone_default_state()
|
||||
|
||||
scene = str(raw.get("scene") or "")
|
||||
character_sheet = str(raw.get("character_sheet") or "")
|
||||
beats = []
|
||||
for item in list(raw.get("beats") or []):
|
||||
if isinstance(item, dict):
|
||||
@@ -30,15 +67,25 @@ def _parse_beat_prompt_state(value):
|
||||
beats.append({"text": text})
|
||||
|
||||
if not beats:
|
||||
return _clone_default_state()
|
||||
return {"beats": beats}
|
||||
beats = [{"text": _DEFAULT_BEAT}]
|
||||
return {
|
||||
"scene": scene,
|
||||
"character_sheet": character_sheet,
|
||||
"beats": beats,
|
||||
}
|
||||
|
||||
|
||||
def _assemble_beat_prompt(state):
|
||||
parsed = _parse_beat_prompt_state(state)
|
||||
chunks = []
|
||||
scene = str(parsed.get("scene") or "").strip()
|
||||
if scene:
|
||||
chunks.append(scene)
|
||||
character_sheet = str(parsed.get("character_sheet") or "").strip()
|
||||
if character_sheet:
|
||||
chunks.append(character_sheet)
|
||||
for beat in parsed["beats"]:
|
||||
text = str(beat.get("text") or "").strip()
|
||||
text = _strip_legacy_directives(beat.get("text") or "")
|
||||
if text:
|
||||
chunks.append(text)
|
||||
return "\n\n".join(chunks)
|
||||
@@ -46,9 +93,9 @@ def _assemble_beat_prompt(state):
|
||||
|
||||
class DumasH3BeatPromptNode:
|
||||
DESCRIPTION = (
|
||||
"Build a MiniMax H3 prompt from one textbox per beat, with a front-end beat "
|
||||
"editor that can append directive examples and expose per-shot controls for "
|
||||
"timing, continuity, ref behavior, anchor additions, soundscape, and music."
|
||||
"Build an upstream MiniMax H3 Long Videos prompt: optional scene paragraph, "
|
||||
"optional character sheet, then one blank-line-separated textbox per beat. "
|
||||
"Per-beat helpers only emit directives the upstream node understands."
|
||||
)
|
||||
RETURN_TYPES = ("STRING",)
|
||||
RETURN_NAMES = ("prompt",)
|
||||
|
||||
+21
-7914
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,319 @@
|
||||
# H3-LongVideos -- https://github.com/Smite79/MiniMax-H3-LongVideos
|
||||
# Copyright (c) 2026 Smite79. All rights reserved.
|
||||
# Redistribution, in whole or in part, requires written permission.
|
||||
# This notice may not be removed or altered. See LICENSE.
|
||||
"""Audio policy shared by conditioning and soundtrack assembly."""
|
||||
|
||||
from dataclasses import dataclass
|
||||
|
||||
import torch
|
||||
import comfy.nested_tensor
|
||||
from h3_runtime import temporal_shape
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ShotAudio:
|
||||
speech: bool
|
||||
sounded: bool
|
||||
voiced_only: bool
|
||||
silence_enabled: bool
|
||||
lead_seconds: float
|
||||
latent_fps: int
|
||||
# The tail. Everything after the line's expected end is pinned the way the lead
|
||||
# pins everything before its start. All three default off, so a ShotAudio built
|
||||
# the old way -- six positional arguments -- behaves exactly the old way.
|
||||
line_seconds: float = 0.0 # planner's estimate of the spoken line
|
||||
tail_seconds: float = 0.0 # free audio kept after that estimate; 0 = no tail pin
|
||||
frame_count: int = 0 # the shot in pixel frames; the audio T comes from it
|
||||
|
||||
@property
|
||||
def pinned(self):
|
||||
return self.silence_enabled and not self.speech and not self.sounded
|
||||
|
||||
@property
|
||||
def lead_frames(self):
|
||||
if not self.speech or self.lead_seconds <= 0:
|
||||
return 0
|
||||
return round(self.lead_seconds * self.latent_fps)
|
||||
|
||||
@property
|
||||
def tail_frames(self):
|
||||
"""Audio latent frames pinned at the END of a dialogue shot.
|
||||
|
||||
The lead pins the opening so the line cannot start early; nothing pinned the
|
||||
close, and a 2s line in a 9s shot left 7s of open branch in a shot the model
|
||||
knows has a voice in it -- which is where speech carries on past the line, or
|
||||
doubles it. The free span is lead + the line's estimate + tail_seconds; the
|
||||
rest is held at encoded silence. The model chooses WHEN to speak, so the
|
||||
margin is the author's dial: a clipped word costs more than a second of babble.
|
||||
Off unless the shot speaks, the margin is set, and at least half a second would
|
||||
be pinned -- a sliver is not worth the risk of clipping."""
|
||||
if (not self.speech or self.tail_seconds <= 0 or self.line_seconds <= 0
|
||||
or self.frame_count <= 0):
|
||||
return 0
|
||||
total = temporal_shape(self.frame_count)[2]
|
||||
free = self.lead_frames + round((self.line_seconds + self.tail_seconds) * self.latent_fps)
|
||||
tail = total - free
|
||||
return tail if tail >= round(0.5 * self.latent_fps) else 0
|
||||
|
||||
|
||||
_SILENT_UNIT = {"lat": None, "key": None}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# NOTHING HERE IS SYNTHESISED ANY MORE. Removed on the report: "Just get rid of
|
||||
# the ambient sounds all together. They sound horrid. Go back to the model's
|
||||
# natural audio."
|
||||
#
|
||||
# What was here built the soundtrack's non-vocal half out of shaped noise: a room
|
||||
# tone from the scene's own wording (synth_ambient, over a table of recipes, with
|
||||
# plain_bed under it as a floor) and 21 foley recipes laid into the shots whose
|
||||
# audio branch is pinned to silence (foley_for, over _hits/_band/_room and later
|
||||
# _contact/_flow/_creak, timed off the picture's own movement for footsteps).
|
||||
#
|
||||
# It went in because a shot pinned to silence cannot get audio from the model at
|
||||
# all -- prompt text never opens a branch -- so auto_sound was writing sounds into
|
||||
# prompts that could not make them. That reasoning was sound and the thing it built
|
||||
# still did not pass: reported first as footsteps sounding like heartbeats and a
|
||||
# bathroom that tapped, and then, once both of those measured clean, as horrid
|
||||
# anyway. Synthesis that measures right and sounds wrong is the end of that road.
|
||||
#
|
||||
# So the audio is the model's, whole. H3 is a joint model and the audio branch is
|
||||
# where its sound comes from; the prompt still describes what a shot sounds like,
|
||||
# which is the half that was always doing the real work.
|
||||
#
|
||||
# The consequence, which is real and is reported in info rather than left to be
|
||||
# discovered: a shot with no line and no sound you wrote is pinned to silence and
|
||||
# is now SILENT. The pin is not a bug and is deliberately untouched -- it is what
|
||||
# stops a free branch filling itself with babble and a face lip-syncing to it.
|
||||
# Write the sound into the beat to open the branch on purpose, or wire a recording
|
||||
# to ambient_audio, which is played under the finished track and conditions
|
||||
# nothing. mix_ambient below is that path, and it is all that is left here.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _seamless_loop(x, n, sr):
|
||||
"""[C, M] -> [C, n], looped with a crossfade so the join does not click.
|
||||
|
||||
Plain tiling puts a discontinuity at every repeat, once per loop length. In a
|
||||
bed that is meant to sit under everything unnoticed, a regular click is the one
|
||||
thing that gets noticed -- the same objection that made the silence latent
|
||||
ping-pong its interior rather than tile it. Here the material is real audio
|
||||
being PLAYED rather than a latent being conditioned on, so it cannot be
|
||||
reversed: a room tone read backwards is fine, but footsteps are not. Crossfade
|
||||
instead, which works on both."""
|
||||
m = int(x.shape[-1])
|
||||
if m <= 0:
|
||||
return None
|
||||
if m >= n:
|
||||
return x[..., :n]
|
||||
fade = min(int(0.25 * sr), m // 4)
|
||||
if fade < 1:
|
||||
reps = -(-n // m)
|
||||
return x.repeat(1, reps)[..., :n]
|
||||
# OVERLAP-ADD the tail onto the head, and shorten the unit by the overlap. The
|
||||
# unit then runs x[m-fade] .. x[m-fade-1], so tiling it steps between samples
|
||||
# that were adjacent in the source and there is no discontinuity anywhere.
|
||||
#
|
||||
# Measured, because the obvious construction is wrong: appending the crossfade
|
||||
# to the END of a full-length unit leaves it finishing on x[fade-1] while the
|
||||
# next repeat starts on x[0], which are not adjacent -- a 2s tone that does not
|
||||
# divide evenly gave a 64x jump at the join, worse than plain tiling's 41x.
|
||||
t = torch.linspace(0.0, 1.0, fade, dtype=x.dtype, device=x.device)
|
||||
head = x[..., :fade] * t + x[..., m - fade:] * (1.0 - t)
|
||||
unit = torch.cat([head, x[..., fade:m - fade]], dim=-1)
|
||||
if int(unit.shape[-1]) < 1:
|
||||
reps = -(-n // m)
|
||||
return x.repeat(1, reps)[..., :n]
|
||||
reps = -(-n // int(unit.shape[-1]))
|
||||
return unit.repeat(1, reps)[..., :n]
|
||||
|
||||
|
||||
def mix_ambient(audio, sr, bed, level):
|
||||
"""Lay an ambient bed UNDER a finished soundtrack. -> (waveform, note).
|
||||
|
||||
The bed is PLAYED, not conditioned on: it is the file, at the level asked for,
|
||||
under whatever the model generated. That is the whole reason to do it here
|
||||
rather than in the sampler -- ambience needs no cooperation from a joint model,
|
||||
has nothing to lip-sync to, and so cannot put a voice in a wordless shot. The
|
||||
conditioning path can only steer the branch toward something bed-LIKE, and on a
|
||||
shot with a line it competes with the line.
|
||||
|
||||
Defensive throughout, like the silence latent: any failure returns the audio
|
||||
untouched with a note saying so, because a bed is a nicety and a render is not.
|
||||
"""
|
||||
try:
|
||||
if audio is None or bed is None or float(level or 0.0) <= 0.0:
|
||||
return audio, ""
|
||||
w = bed.get("waveform") if isinstance(bed, dict) else None
|
||||
if w is None or not int(getattr(w, "ndim", 0)):
|
||||
return audio, ("ambient_audio is wired but carries no waveform, so nothing "
|
||||
"was laid under the soundtrack")
|
||||
w = w[0] if w.dim() == 3 else w # [B, C, M] -> [C, M]
|
||||
if w.dim() != 2 or w.shape[-1] < 2:
|
||||
return audio, ("ambient_audio is too short to loop, so nothing was laid "
|
||||
"under the soundtrack")
|
||||
w = w.detach().to(dtype=audio.dtype, device=audio.device)
|
||||
b_sr = int((bed.get("sample_rate") if isinstance(bed, dict) else 0) or 0)
|
||||
# RESAMPLE, or the bed plays at the wrong speed and pitch. Linear is coarse
|
||||
# for music and inaudible on a room tone, which is what this input is for.
|
||||
resampled = ""
|
||||
if b_sr > 0 and b_sr != int(sr):
|
||||
want = max(2, int(round(w.shape[-1] * float(sr) / float(b_sr))))
|
||||
w = torch.nn.functional.interpolate(
|
||||
w.unsqueeze(0), size=want, mode="linear", align_corners=False)[0]
|
||||
resampled = f", resampled from {b_sr} Hz"
|
||||
ch = int(audio.shape[1])
|
||||
if int(w.shape[0]) != ch:
|
||||
w = (w.mean(dim=0, keepdim=True).repeat(ch, 1) if int(w.shape[0]) > ch
|
||||
else w[:1].repeat(ch, 1))
|
||||
n = int(audio.shape[-1])
|
||||
loop = _seamless_loop(w, n, int(sr))
|
||||
if loop is None:
|
||||
return audio, ""
|
||||
out = audio + loop.unsqueeze(0) * float(level)
|
||||
# NORMALISE rather than clip. Clipping a bed that pushed a loud line over
|
||||
# the top distorts the LINE, which is the thing worth keeping.
|
||||
peak = float(out.abs().max())
|
||||
gain = ""
|
||||
if peak > 1.0:
|
||||
out = out / peak
|
||||
gain = f", and the mix was scaled by {1.0 / peak:.2f} to stop it clipping"
|
||||
secs = w.shape[-1] / float(sr)
|
||||
return out, (f"an ambient bed was laid under the whole soundtrack at level "
|
||||
f"{float(level):.2f} -- {secs:.1f}s of audio{resampled}, looped "
|
||||
f"with a crossfade so the join does not click{gain}. It is your "
|
||||
f"file, played under what the model generated: it conditions "
|
||||
f"nothing, so it cannot put a voice in a wordless shot the way "
|
||||
f"an inferred bed did. Shots pinned to silence keep their silent "
|
||||
f"conditioning and get the bed on top, which is what makes a "
|
||||
f"wordless shot sound like a room instead of a mute")
|
||||
except Exception as exc:
|
||||
return audio, (f"the ambient bed could not be mixed ({type(exc).__name__}), so "
|
||||
f"the soundtrack is unchanged")
|
||||
|
||||
|
||||
_SILENCE_STATUS = {"asked": 0, "applied": 0, "why": ""}
|
||||
|
||||
|
||||
_SILENT_SECONDS = 2
|
||||
|
||||
|
||||
_SILENT_EDGE = 4
|
||||
|
||||
|
||||
def _silent_audio_latent(audio_vae, frame_count, fps):
|
||||
"""A keyframe audio latent of actual SILENCE, or None if it cannot be made.
|
||||
|
||||
H3 is a JOINT model: the mouth follows the audio branch. On a shot with no
|
||||
scripted line the branch is otherwise unconditioned, and an unconditioned audio
|
||||
branch invents a voice -- which the picture then lip-syncs to. The lips-closed
|
||||
sentence is arguing with a stream that has already decided someone is talking.
|
||||
|
||||
REBUILT 2026-09-05, from measurements against the real VAE rather than from
|
||||
reasoning. The previous version encoded one second, kept a SINGLE interior
|
||||
frame and repeated it, on the argument that silence is homogeneous. It is not,
|
||||
in latent space: encoded silence has genuine frame-to-frame variation (delta
|
||||
mean 0.002-0.004, max 0.021), and a repeated frame has a delta of exactly
|
||||
0.000000. That is a flat signal no encoder produces, and a model handed
|
||||
conditioning outside its own distribution has every reason to disregard it --
|
||||
which is an audio branch back to inventing a voice, with the report saying
|
||||
silence went on.
|
||||
|
||||
The fix that version was avoiding is real too: tiling the whole encoded second
|
||||
end to end leaves a 25x spike at each join (0.554 against 0.022), once per
|
||||
second, which is a metronome in the conditioning of a joint model.
|
||||
|
||||
So: encode two seconds, drop the padded ends, and PING-PONG the interior --
|
||||
forward, reversed, forward. Every join repeats a frame, so there is no seam,
|
||||
and the interior statistics are the encoder's own. Measured over a 9s shot:
|
||||
|
||||
one frame repeated peak 0.000686 delta mean 0.000000 max 0.000000
|
||||
whole 2s tiled peak 0.000314 delta mean 0.017451 max 0.554715
|
||||
interior ping-pong peak 0.000566 delta mean 0.002039 max 0.021159
|
||||
|
||||
where the encoder's own interior is mean 0.0021, max 0.0212. Decoded peak
|
||||
0.000566 on a +/-1.0 scale is about -65 dBFS: silence.
|
||||
|
||||
Everything here stays defensive. Shapes are CHECKED against what the layout
|
||||
expects rather than assumed, and any failure returns None so the shot falls
|
||||
back to an unconditioned branch instead of breaking the render -- the caller
|
||||
reports when that happens, so it is no longer a silent failure.
|
||||
"""
|
||||
try:
|
||||
sr = int(getattr(audio_vae, "audio_sample_rate", 0) or 0)
|
||||
if sr <= 0:
|
||||
return None
|
||||
_, _, want_t = temporal_shape(frame_count, fps)
|
||||
if want_t <= 0:
|
||||
return None
|
||||
key = (id(audio_vae), sr)
|
||||
block = _SILENT_UNIT.get("lat") if _SILENT_UNIT.get("key") == key else None
|
||||
if block is None:
|
||||
# CHANNELS LAST. comfy.sd.VAE.encode() does `pixel_samples.movedim(-1, 1)`
|
||||
# before handing off, so the audio VAE -- which wants [B, 2, L] -- must be
|
||||
# given [B, L, 2]. Passing [B, 2, L] raises inside the encoder, and an
|
||||
# early version did exactly that: swallowed by the guard below, so the
|
||||
# whole layer silently did nothing.
|
||||
#
|
||||
# Two seconds, encoded ONCE and cached. Encoding a full 15s shot instead
|
||||
# cost a VAE pass big enough to OOM mid-render on a 16GB card, where the
|
||||
# failure again degraded silently to no conditioning at all.
|
||||
enc = audio_vae.encode(torch.zeros((1, sr * _SILENT_SECONDS, 2)))
|
||||
if enc is None or enc.dim() != 4 or enc.shape[1] != 32:
|
||||
return None
|
||||
if enc.shape[-1] <= 2 * _SILENT_EDGE + 1:
|
||||
return None
|
||||
block = enc[..., _SILENT_EDGE:-_SILENT_EDGE].detach().to("cpu").clone()
|
||||
_SILENT_UNIT["lat"] = block
|
||||
_SILENT_UNIT["key"] = key
|
||||
n = block.shape[-1]
|
||||
if n < 1:
|
||||
return None
|
||||
# Forward, reversed, forward... Each join repeats a frame, so the seam that
|
||||
# plain tiling leaves is gone while the interior variation is the encoder's.
|
||||
pieces, have, i = [], 0, 0
|
||||
while have < want_t:
|
||||
piece = block if i % 2 == 0 else torch.flip(block, dims=[-1])
|
||||
pieces.append(piece)
|
||||
have += n
|
||||
i += 1
|
||||
out = torch.cat(pieces, dim=-1)[..., :want_t].clone()
|
||||
if out.shape[-1] != want_t:
|
||||
return None
|
||||
return out
|
||||
except Exception:
|
||||
return None # never fail a render for a nicety
|
||||
|
||||
|
||||
def _pin_audio_silence(latent, silence, lead_frames=None, tail_frames=0):
|
||||
"""Start target audio at encoded silence and preserve the requested span(s).
|
||||
|
||||
lead_frames None pins the whole shot. Otherwise the first lead_frames and the
|
||||
last tail_frames are held at silence and the span between is left to the model
|
||||
-- that is where the line goes. The tail is clipped to what the lead leaves, so
|
||||
the two can never overlap. Nothing pinned at all is a no-op, reported as False
|
||||
so the caller does not count it as applied."""
|
||||
try:
|
||||
video, audio = latent["samples"].unbind()
|
||||
silence = silence.to(device=audio.device, dtype=audio.dtype)
|
||||
if silence.shape != audio.shape:
|
||||
return False
|
||||
audio_mask = torch.ones_like(audio[:, :1])
|
||||
if lead_frames is None:
|
||||
audio_mask.zero_()
|
||||
else:
|
||||
t = audio.shape[-1]
|
||||
n = min(t, max(0, int(lead_frames)))
|
||||
m = min(t - n, max(0, int(tail_frames or 0)))
|
||||
if n <= 0 and m <= 0:
|
||||
return False
|
||||
if n > 0:
|
||||
audio_mask[..., :n] = 0
|
||||
if m > 0:
|
||||
audio_mask[..., t - m:] = 0
|
||||
latent["samples"] = comfy.nested_tensor.NestedTensor((video, silence))
|
||||
latent["noise_mask"] = comfy.nested_tensor.NestedTensor(
|
||||
(torch.ones_like(video[:, :1]), audio_mask))
|
||||
return True
|
||||
except Exception:
|
||||
return False
|
||||
@@ -0,0 +1,310 @@
|
||||
# H3-LongVideos -- https://github.com/Smite79/MiniMax-H3-LongVideos
|
||||
# Copyright (c) 2026 Smite79. All rights reserved.
|
||||
# Redistribution, in whole or in part, requires written permission.
|
||||
# This notice may not be removed or altered. See LICENSE.
|
||||
"""Decisions about which pictures may condition a shot."""
|
||||
|
||||
import torch
|
||||
import node_helpers
|
||||
from h3_runtime import (H3_FPS, AUDIO_LATENT_FPS, _empty_av_latent, _resize, ref_image_canvas,
|
||||
frame_levels)
|
||||
from h3_audio import _SILENCE_STATUS, _silent_audio_latent, _pin_audio_silence
|
||||
|
||||
|
||||
def may_carry_room(previous_cast, current_cast, tagged_names):
|
||||
"""A previous frame is safe as a reference only when it adds no subject."""
|
||||
previous = [name for name in (previous_cast or ()) if name]
|
||||
current = set(current_cast or ())
|
||||
tagged = set(tagged_names or ())
|
||||
return bool(previous) and all(name in current for name in previous) \
|
||||
and not any(name in tagged for name in previous)
|
||||
|
||||
|
||||
def may_carry_frame(previous_cast, current_cast, tagged_names):
|
||||
"""A previous frame of the SAME room is safe as a reference claimed with everyone in it.
|
||||
|
||||
Unlike may_carry_room, somebody this shot does not describe may be in it: the claim
|
||||
names them, and they are still in that room. Refused only for an empty frame, or
|
||||
one holding somebody whose own portrait also rides this shot -- two pictures of one
|
||||
person is how a second one gets drawn."""
|
||||
previous = [name for name in (previous_cast or ()) if name]
|
||||
current = set(current_cast or ())
|
||||
tagged = set(tagged_names or ())
|
||||
return bool(previous) and not any(name in tagged and name in current
|
||||
for name in previous)
|
||||
|
||||
|
||||
def recoverable_subject(cast, tagged_names, returning_names, captured):
|
||||
"""Return the sole safe recovered subject, or an empty string."""
|
||||
people = [name for name in (cast or ()) if name]
|
||||
if len(people) != 1:
|
||||
return ""
|
||||
name = people[0]
|
||||
return name if (name not in set(tagged_names or ())
|
||||
and name in set(returning_names or ())
|
||||
and captured.get(name) is not None) else ""
|
||||
|
||||
|
||||
KEYFRAME_SAFE_AUG = 0.99 # below this, a ref aug would soften the keyframe too
|
||||
|
||||
# What ONE boundary is allowed to claim it measured. Wider than any real per-pass drift,
|
||||
# narrow enough that a bad frame -- a flash, a cut to black, a frame the model lost --
|
||||
# cannot swing the estimate. The median across boundaries does the real rejecting.
|
||||
LEVEL_GAIN_CAP = 0.12 # in log-gain, so +-12.7% of contrast
|
||||
LEVEL_OFFSET_CAP = 0.05
|
||||
# The within-shot term is believed only when boundaries AGREE on its sign, and even then
|
||||
# only this far: within-shot change is often the author's (a light switched off), so it is
|
||||
# the half of the signal that cannot be trusted on its own.
|
||||
LEVEL_SHOT_GAIN_CAP = 0.015
|
||||
LEVEL_SHOT_OFFSET_CAP = 0.010
|
||||
LEVEL_AGREE = 2.0 / 3.0
|
||||
LEVEL_MIN_OBS = 3
|
||||
# What the correction may do to one handoff, whatever it measured. A cut should not carry
|
||||
# a visible grade step: shot N's last frame reaches the video uncorrected while N+1 is
|
||||
# sampled from a corrected keyframe, so an uncapped correction trades burn-in for a pop at
|
||||
# every join -- the same class of complaint, differently shaped.
|
||||
LEVEL_GAIN_LO, LEVEL_GAIN_HI = 0.80, 1.25
|
||||
LEVEL_OFFSET_BOUND = 0.02
|
||||
# Below this a frame is too flat for a contrast RATIO to mean anything.
|
||||
LEVEL_MIN_SIGMA = 0.01
|
||||
|
||||
|
||||
class HandoffLevels:
|
||||
"""Takes the grade the chain adds to itself back out of the handoff.
|
||||
|
||||
THE MEASUREMENT, which is the whole reason this needs no scene list. At every
|
||||
boundary the render holds two pictures that are SUPPOSED to be the same frame: K,
|
||||
the handoff it gave the shot, and R, frame one of what came back -- the model's own
|
||||
reproduction of K, from a keyframe labelled sigma 0.001. Nothing was asked to change
|
||||
between them, so everything separating them is the chain's own doing and none of it
|
||||
is the author's intent. That is the one difference in the loop that can be corrected
|
||||
without guessing at anybody's lighting, and R costs nothing to look at: it is the
|
||||
frame trim_seam throws away.
|
||||
|
||||
A beat that walks into a darker room moves K, and R follows it there. So the level is
|
||||
never anchored, never compared to shot 1, and never compared to a target -- only K
|
||||
against its own reproduction, boundary by boundary.
|
||||
|
||||
WHAT IT WILL NOT FIX. Clipping already baked into earlier shots, because the VAE
|
||||
clamps every decode and headroom spent is gone. Softening, which is a different
|
||||
measurement and a different cause. Anything spatial -- ghosting, local burn, identity
|
||||
drift. A tone curve with a knee in it, since this is affine per channel; the residual
|
||||
in the report is how that would show itself. The first boundary, which has nothing to
|
||||
measure yet. And a deliberate monotone move -- a film that dims every single beat --
|
||||
loses a bounded, reported fraction of itself."""
|
||||
|
||||
def __init__(self):
|
||||
self._bg, self._bo = [], [] # per boundary: K -> R, the chain's own drift
|
||||
self._sg, self._so = [], [] # per shot: R -> last frame, believed only on agreement
|
||||
self.applied = [] # (gain, offset) actually used, for the report
|
||||
|
||||
def observe(self, given, repro, last=None, pre_up_last=None):
|
||||
"""Record one boundary. given is the keyframe this shot got, repro is frame one
|
||||
of what it produced, last is its final frame, pre_up_last the handoff it hands on.
|
||||
|
||||
last/pre_up_last are how the pre-upscale handoff and the post-upscale output are
|
||||
put in the same frame of reference: their difference IS the pipeline's own offset,
|
||||
measured on one frame that went through both, so it can be subtracted from the
|
||||
K->R reading instead of being mistaken for drift. With latent_upscale off they are
|
||||
the same frame and the term is zero."""
|
||||
gm, gs = frame_levels(given)
|
||||
rm, rs = frame_levels(repro)
|
||||
if gm is None or rm is None:
|
||||
return False
|
||||
if float(gs.min()) < LEVEL_MIN_SIGMA or float(rs.min()) < LEVEL_MIN_SIGMA:
|
||||
return False
|
||||
ug = torch.zeros(3)
|
||||
uo = torch.zeros(3)
|
||||
lm, ls = frame_levels(last) if last is not None else (None, None)
|
||||
if pre_up_last is not None and lm is not None:
|
||||
pm, ps = frame_levels(pre_up_last)
|
||||
if pm is not None and float(ps.min()) >= LEVEL_MIN_SIGMA:
|
||||
ug = torch.log(ls / ps)
|
||||
uo = lm - pm
|
||||
self._bg.append((torch.log(rs / gs) - ug).clamp(-LEVEL_GAIN_CAP, LEVEL_GAIN_CAP))
|
||||
self._bo.append((rm - gm - uo).clamp(-LEVEL_OFFSET_CAP, LEVEL_OFFSET_CAP))
|
||||
if lm is not None and float(ls.min()) >= LEVEL_MIN_SIGMA:
|
||||
self._sg.append(torch.log(ls / rs))
|
||||
self._so.append(lm - rm)
|
||||
return True
|
||||
|
||||
def _agreed(self, rows, cap):
|
||||
"""The median of rows, but only per channel where at least LEVEL_AGREE of them
|
||||
share its sign. A within-shot change the boundaries disagree about is content, not
|
||||
drift, and content must not be corrected."""
|
||||
out = torch.zeros(3)
|
||||
if len(rows) < LEVEL_MIN_OBS:
|
||||
return out
|
||||
st = torch.stack(rows)
|
||||
med = st.median(dim=0).values
|
||||
agree = ((st * med.sign().unsqueeze(0)) > 0).float().mean(dim=0)
|
||||
keep = agree >= LEVEL_AGREE
|
||||
return torch.where(keep, med.clamp(-cap, cap), out)
|
||||
|
||||
def estimate(self):
|
||||
"""(gain_log, offset) the chain is drifting by per boundary, per channel."""
|
||||
if not self._bg:
|
||||
return None, None
|
||||
g = torch.stack(self._bg).median(dim=0).values + self._agreed(self._sg, LEVEL_SHOT_GAIN_CAP)
|
||||
o = torch.stack(self._bo).median(dim=0).values + self._agreed(self._so, LEVEL_SHOT_OFFSET_CAP)
|
||||
return g, o
|
||||
|
||||
def gains(self, strength):
|
||||
"""(gain, offset) as 3-vectors, or (None, None) when there is nothing worth doing.
|
||||
|
||||
Separate from note() because more than one frame leaves a shot -- the handoff,
|
||||
and any face captured for a return several shots later -- and they have to carry
|
||||
the SAME grade. A recovered face arriving at a different exposure from the shot
|
||||
around it would be a new bug of exactly the kind this is fixing."""
|
||||
g, o = self.estimate()
|
||||
if g is None or strength <= 0:
|
||||
return None, None
|
||||
gain = torch.exp(-float(strength) * g).clamp(LEVEL_GAIN_LO, LEVEL_GAIN_HI)
|
||||
off = (-float(strength) * o).clamp(-LEVEL_OFFSET_BOUND, LEVEL_OFFSET_BOUND)
|
||||
# The next thing this frame meets is an 8-bit quantisation, so a correction under
|
||||
# 1/255 would be erased on the way there. Claiming it would be worse than silence.
|
||||
if float((gain - 1.0).abs().max()) < 1e-3 and float(off.abs().max()) < 1.0 / 255.0:
|
||||
return None, None
|
||||
return gain, off
|
||||
|
||||
def note(self, gain, off):
|
||||
"""Record what was applied, and say it in one clause."""
|
||||
self.applied.append((gain.clone(), off.clone()))
|
||||
return (f"gain {'/'.join(f'{float(v):.3f}' for v in gain)} "
|
||||
f"level {'/'.join(f'{float(v):+.4f}' for v in off)}")
|
||||
|
||||
def _keyframe_latent(vae, hand_img):
|
||||
"""The keyframe latent for this shot: an ENCODE of the previous shot's last frame.
|
||||
|
||||
This was briefly an optimisation -- pass the previous shot's own latent straight
|
||||
through and skip a VAE round trip per boundary. It was wrong, and it degraded
|
||||
every shot after the first.
|
||||
|
||||
A keyframe is ONE pixel frame, and H3's grid puts that at 5f -> TWO latent
|
||||
frames. Slicing [:, :, -1:] off a finished shot hands over one. Worse, the video
|
||||
VAE is causal: the last latent of a 72-frame sequence encodes its temporal
|
||||
context, not a standalone opening frame, so even at the right count it does not
|
||||
mean what a keyframe means. The spatial-size guard could not see either problem.
|
||||
|
||||
The round trip is real but it is one lossy step on a correctly formed anchor,
|
||||
which beats a cheap malformed one."""
|
||||
return vae.encode(hand_img)
|
||||
|
||||
|
||||
def _build_ref_images(vae, images, gen_w, gen_h, mode="match"):
|
||||
"""(tokenizer items, DiT blocks) for a list of reference IMAGE tensors.
|
||||
|
||||
The tokenizer labels each one `<Picture N>:` itself, in the order given here --
|
||||
so the roster the prompt refers to is decided by input order, not by anything
|
||||
written in the prompt."""
|
||||
items, blocks = [], []
|
||||
for img in images:
|
||||
if img is None:
|
||||
continue
|
||||
h, w = int(img.shape[1]), int(img.shape[2])
|
||||
tw, th = ref_image_canvas(w, h, gen_w, gen_h, mode)
|
||||
resized = _resize(img[:1], tw, th, "disabled")
|
||||
items.append({"type": "image", "data": resized})
|
||||
blocks.append({"kind": "image", "latent_h": th // 16, "latent_w": tw // 16,
|
||||
"latent": vae.encode(resized)})
|
||||
return items, blocks
|
||||
|
||||
|
||||
def build_conditioning(clip, vae, audio_vae, prompt, width, height, length,
|
||||
handoff=None, refs=None,
|
||||
ref_noise_aug=0.999, silent=False, ref_image_size="match",
|
||||
handoff_as_ref=False, speech_lead_seconds=0.0, speech_tail_frames=0):
|
||||
"""Encode prompt, identity references, keyframe, and audio constraints for a shot."""
|
||||
latent, fc = _empty_av_latent(width, height, length, H3_FPS)
|
||||
refs = [r for r in (refs or []) if r is not None]
|
||||
|
||||
hand_img = None
|
||||
if handoff is not None:
|
||||
hand_img = _resize(handoff[:1], width, height, "disabled")
|
||||
|
||||
# REFERENCES AND THE KEYFRAME RIDE TOGETHER. This is the arrangement the node
|
||||
# had before I broke it, and the reason is in ComfyUI's own layout:
|
||||
#
|
||||
# model_base.py:2183-2191 cond_video_latents = keyframe latents THEN ref latents
|
||||
# model.py PackedLayout emits keyframe "cond" segments THEN ref "ref_img" ones
|
||||
#
|
||||
# The two orders agree, so both channels coexist. A shot takes its references AND
|
||||
# a real keyframe: the keyframe ANCHORS the first frame, which is what continuity
|
||||
# needs, while a reference only supplies identity. They are not alternatives.
|
||||
#
|
||||
# I had read "<Picture 1>" as MEANING the first frame on fl2va, and rearranged the
|
||||
# roster around that. It does not. Which image is the first frame is decided by
|
||||
# resolved_frame_index in minimax_keyframes, not by a label's number -- the labels
|
||||
# are only how the images are shown to the VLM, and what they have to line up with
|
||||
# is the <Picture N> tags in the prompt.
|
||||
#
|
||||
# So references come FIRST and keep slots 1..N, which is what a sheet line's
|
||||
# `Name: <Picture 1>, ...` points at, and the handoff is appended AFTER them where
|
||||
# it disturbs no numbering. It has to be in the list at all because
|
||||
# tokenize_with_weights is either/or: passing minimax_ref_items makes it ignore
|
||||
# `images` outright, so leaving the handoff out means the VLM is never shown where
|
||||
# the shot left off and re-imagines the scenery -- same place, new room.
|
||||
keyframe_ok = ref_noise_aug is None or float(ref_noise_aug) >= KEYFRAME_SAFE_AUG
|
||||
# One aug covers every visual condition row, references AND the keyframe. Below
|
||||
# KEYFRAME_SAFE_AUG the keyframe latent would be noised and labelled at the wrong
|
||||
# timestep, so the handoff stops being an anchor and rides as an extra reference
|
||||
# instead: weaker continuity, but nothing pretending to anchor while carrying noise.
|
||||
# ...or because the caller asked for it. A shot that introduces somebody already
|
||||
# in position wants the room this picture carries and NOT the first frame it
|
||||
# would force, and that is a demotion the aug knows nothing about.
|
||||
carry_as_ref = bool(hand_img is not None
|
||||
and (handoff_as_ref or (refs and not keyframe_ok)))
|
||||
|
||||
enc_refs = refs + ([hand_img] if carry_as_ref else [])
|
||||
items, blocks = ([], [])
|
||||
if enc_refs:
|
||||
items, blocks = _build_ref_images(vae, enc_refs, width, height, ref_image_size)
|
||||
if hand_img is not None and not carry_as_ref:
|
||||
items = items + [{"type": "image", "data": hand_img}]
|
||||
|
||||
if items:
|
||||
tokens = clip.tokenize(prompt, minimax_ref_items=items)
|
||||
else:
|
||||
tokens = clip.tokenize(prompt)
|
||||
cond = clip.encode_from_tokens_scheduled(tokens)
|
||||
|
||||
vals = {}
|
||||
if blocks:
|
||||
vals["minimax_refs"] = blocks
|
||||
# How CLEAN the references are shown. One aug covers every conditioning
|
||||
# latent, keyframe included -- which is why softening references below
|
||||
# KEYFRAME_SAFE_AUG would soften the anchor too.
|
||||
if ref_noise_aug is not None:
|
||||
vals["minimax_visual_cond_noise_aug"] = float(ref_noise_aug)
|
||||
|
||||
kfs = []
|
||||
if hand_img is not None and not carry_as_ref:
|
||||
kfs.append({"resolved_frame_index": 0,
|
||||
"latent": _keyframe_latent(vae, hand_img)})
|
||||
# Audio keyframes are extra conditioning rows in H3's PackedLayout. Pin the
|
||||
# generated target stream instead, so the joint model also sees a quiet mouth.
|
||||
# A dialogue shot pins its opening (the lead) and, past the line's estimated end,
|
||||
# its close (the tail); the span between is the model's.
|
||||
if silent or float(speech_lead_seconds or 0.0) > 0.0 or int(speech_tail_frames or 0) > 0:
|
||||
_SILENCE_STATUS["asked"] += 1
|
||||
if audio_vae is None:
|
||||
_SILENCE_STATUS["why"] = "no audio VAE is wired to the node"
|
||||
else:
|
||||
sil = _silent_audio_latent(audio_vae, fc, H3_FPS)
|
||||
if sil is None:
|
||||
_SILENCE_STATUS["why"] = ("the audio VAE would not encode a silent "
|
||||
"second -- the wrong VAE is on the "
|
||||
"audio_vae input")
|
||||
else:
|
||||
lead = None if silent else round(float(speech_lead_seconds) *
|
||||
AUDIO_LATENT_FPS)
|
||||
tail = 0 if silent else int(speech_tail_frames or 0)
|
||||
if _pin_audio_silence(latent, sil, lead, tail):
|
||||
_SILENCE_STATUS["applied"] += 1
|
||||
else:
|
||||
_SILENCE_STATUS["why"] = "the silent latent did not match the shot"
|
||||
if kfs:
|
||||
vals["minimax_keyframes"] = kfs
|
||||
if vals:
|
||||
cond = node_helpers.conditioning_set_values(cond, vals)
|
||||
return cond, latent, fc, carry_as_ref
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,739 @@
|
||||
# H3-LongVideos -- https://github.com/Smite79/MiniMax-H3-LongVideos
|
||||
# Copyright (c) 2026 Smite79. All rights reserved.
|
||||
# Redistribution, in whole or in part, requires written permission.
|
||||
# This notice may not be removed or altered. See LICENSE.
|
||||
"""Sampling, decoding, resizing, memory handling, and frame assembly."""
|
||||
|
||||
import math
|
||||
|
||||
import torch
|
||||
import nodes
|
||||
import comfy.utils
|
||||
import comfy.sample
|
||||
import comfy.samplers
|
||||
import comfy.nested_tensor
|
||||
import comfy.model_management as mm
|
||||
import latent_preview
|
||||
|
||||
|
||||
class FrameAccumulator:
|
||||
"""Build the final frame tensor once, retaining overflow only when necessary."""
|
||||
|
||||
def __init__(self, capacity, dtype, store_on_cpu):
|
||||
self.capacity = int(capacity)
|
||||
self.dtype = dtype
|
||||
self.store_on_cpu = bool(store_on_cpu)
|
||||
self.tensor = None
|
||||
self.used = 0
|
||||
self.overflow = []
|
||||
|
||||
def add(self, frames):
|
||||
count = int(frames.shape[0])
|
||||
if self.tensor is None and count:
|
||||
device = torch.device("cpu") if self.store_on_cpu else frames.device
|
||||
self.tensor = torch.empty(
|
||||
(max(count, self.capacity),) + tuple(frames.shape[1:]),
|
||||
dtype=self.dtype, device=device)
|
||||
if (not self.overflow and self.tensor is not None
|
||||
and self.used + count <= self.tensor.shape[0]):
|
||||
self.tensor[self.used:self.used + count].copy_(frames)
|
||||
self.used += count
|
||||
return
|
||||
self.overflow.append(frames.to("cpu", self.dtype, copy=True)
|
||||
if self.store_on_cpu else frames)
|
||||
|
||||
def release(self):
|
||||
"""Drop every tensor held, now, rather than whenever the collector gets to it.
|
||||
|
||||
On an interrupt the render unwinds through frames the collector tears down in
|
||||
its own order, and a large video buffer freed after the models it was sized
|
||||
against have already gone is a free the allocator cannot explain. Deliberately
|
||||
does NOT empty the cache: that is another CUDA call, and if the context is
|
||||
already in a sticky error state it is one more thing to abort inside."""
|
||||
self.tensor = None
|
||||
self.overflow = []
|
||||
self.used = 0
|
||||
|
||||
def finish(self):
|
||||
if not self.overflow:
|
||||
if self.tensor is None:
|
||||
return torch.cat(self.overflow, dim=0)
|
||||
if self.used == self.tensor.shape[0]:
|
||||
out = self.tensor
|
||||
else:
|
||||
# COMPACT, never a slice. A slice of a larger buffer keeps the WHOLE
|
||||
# buffer's storage alive, which is the retention this class exists to
|
||||
# prevent -- and test_the_chain_is_never_held_twice measures exactly
|
||||
# that, demanding no unused bytes behind the returned tensor.
|
||||
#
|
||||
# There is slack because the capacity is now an upper bound: it can no
|
||||
# longer assume trim_seam drops a frame at every seam, since a shot that
|
||||
# opens on no keyframe keeps its first frame. Over-allocating by at most
|
||||
# one frame per seam and compacting once is the bounded cost. The
|
||||
# alternative -- an exact guess that can be too small -- drops into the
|
||||
# overflow list, which with cleanup_between_shots off holds every shot's
|
||||
# decoded frames live on the GPU until the end of the run.
|
||||
out = torch.empty((self.used,) + tuple(self.tensor.shape[1:]),
|
||||
dtype=self.dtype, device=self.tensor.device)
|
||||
out.copy_(self.tensor[:self.used])
|
||||
self.tensor = None
|
||||
return out
|
||||
|
||||
extra = sum(int(piece.shape[0]) for piece in self.overflow)
|
||||
reference = self.tensor if self.tensor is not None else self.overflow[0]
|
||||
out = torch.empty((self.used + extra,) + tuple(reference.shape[1:]),
|
||||
dtype=self.dtype, device=reference.device)
|
||||
if self.tensor is not None and self.used:
|
||||
out[:self.used].copy_(self.tensor[:self.used])
|
||||
at = self.used
|
||||
while self.overflow:
|
||||
piece = self.overflow.pop(0)
|
||||
count = int(piece.shape[0])
|
||||
out[at:at + count].copy_(piece)
|
||||
at += count
|
||||
self.tensor = None
|
||||
return out
|
||||
|
||||
|
||||
H3_FPS = 24 # H3 renders 24 fps, always
|
||||
|
||||
|
||||
AUDIO_LATENT_FPS = 40 # audio latent frames per second
|
||||
|
||||
|
||||
AUTO_TILE_T = 8 # temporal chunk for a tiled decode
|
||||
|
||||
|
||||
MAX_FRAMES = 362 # H3's own ceiling (~15s)
|
||||
|
||||
|
||||
CANVAS_MULTIPLE = 32
|
||||
|
||||
|
||||
REF_IMAGE_SHORT_EDGE = 2048
|
||||
|
||||
|
||||
def align_frame_count(n):
|
||||
"""Up to the next valid H3 frame count. The grid is 17k+5."""
|
||||
n = max(5, int(n))
|
||||
while n % 17 != 5:
|
||||
n += 1
|
||||
return min(n, MAX_FRAMES)
|
||||
|
||||
|
||||
def video_latent_t(fc):
|
||||
return 2 if fc <= 5 else ((fc - 5) // 17) * 5 + 2
|
||||
|
||||
|
||||
def temporal_shape(length, fps=H3_FPS):
|
||||
"""(frame count, video latent frames, audio latent frames) for a shot.
|
||||
|
||||
`fps` is accepted but deliberately IGNORED: the audio latent has to line up
|
||||
with 24 fps video or the shot's sound is stretched against its picture."""
|
||||
fc = align_frame_count(length)
|
||||
return fc, video_latent_t(fc), round(fc / H3_FPS * AUDIO_LATENT_FPS)
|
||||
|
||||
|
||||
def ref_image_canvas(w, h, gen_w, gen_h, mode="match"):
|
||||
"""Pure: the (width, height) a reference image is encoded at.
|
||||
|
||||
'match' scales it (DOWN only, aspect kept) to the generation's pixel area, so a
|
||||
reference costs about as much as one frame of the shot. 'max' goes to the
|
||||
reference pipeline's 2048 short edge for the best identity fidelity, which on a
|
||||
long chain is several times slower because the rows are re-attended every step
|
||||
of every shot. Never upscales: a small reference stays small."""
|
||||
w, h = max(1, int(w)), max(1, int(h))
|
||||
if mode == "max":
|
||||
scale = min(1.0, REF_IMAGE_SHORT_EDGE / min(w, h))
|
||||
else:
|
||||
scale = min(1.0, math.sqrt((int(gen_w) * int(gen_h)) / float(w * h)))
|
||||
snap = lambda v: max(CANVAS_MULTIPLE, round(v * scale / CANVAS_MULTIPLE) * CANVAS_MULTIPLE)
|
||||
return snap(w), snap(h)
|
||||
|
||||
|
||||
def _resize(image, width, height, crop):
|
||||
s = image[..., :3].movedim(-1, 1)
|
||||
s = comfy.utils.common_upscale(s, width, height, "lanczos", crop)
|
||||
return s.movedim(1, -1)
|
||||
|
||||
|
||||
def _empty_av_latent(width, height, length, fps, batch_size=1):
|
||||
fc, lt, at = temporal_shape(length, fps)
|
||||
video = torch.zeros([batch_size, 24, lt, height // 16, width // 16], device=mm.intermediate_device())
|
||||
audio = torch.zeros([batch_size, 32, 2, at], device=mm.intermediate_device())
|
||||
return {"samples": comfy.nested_tensor.NestedTensor((video, audio))}, fc
|
||||
|
||||
|
||||
def _auto_tile_t(n_latent_frames, requested=None):
|
||||
"""Temporal tile for a tiled decode. An explicit value wins.
|
||||
|
||||
The decode_tile_frames widget is gone, so this is where the value comes from
|
||||
now. It has to come from somewhere: ComfyUI's decode_tiled_3d defaults tile_t
|
||||
to 999, i.e. SPATIAL tiles only, and expanding the whole clip's time axis at
|
||||
once is the single largest allocation in a run. A "tiled" decode that keeps the
|
||||
full temporal extent barely lowers the peak, so the OOM retry that switches
|
||||
tiling on was, without this, retrying with almost the same footprint."""
|
||||
if requested:
|
||||
return int(requested)
|
||||
n = int(n_latent_frames or 0)
|
||||
return AUTO_TILE_T if n > AUTO_TILE_T else None
|
||||
|
||||
|
||||
def _decode_video(vae, out_latent, tiled, free_first=None, tile_t=None, tile_xy=None,
|
||||
keep=()):
|
||||
"""Decode the video latent.
|
||||
|
||||
`free_first` is the diffusion model: sampling is finished, and the video VAE
|
||||
needs the room for THIS decode -- the free runs immediately before it, not to
|
||||
make room for the next shot. On a card where the DiT is most of the VRAM, the
|
||||
decode does not fit until it goes.
|
||||
|
||||
`keep` is what must NOT be evicted on the way. It was `keep_loaded=[]`, which
|
||||
unloaded every resident model -- including the video VAE, which ComfyUI then
|
||||
reloaded three lines later to run the decode. An evict-and-reload of the thing
|
||||
about to be used, once per shot, on every card. Peak VRAM is identical either
|
||||
way, since the VAE has to be resident to decode; the round trip was pure cost.
|
||||
|
||||
memory_required is ASKED FOR HONESTLY, which it was not. It was 1e30, and
|
||||
free_memory computes `memory_to_free = memory_required - get_free_memory(device)`
|
||||
(model_management.py:887), so 1e30 means "unload everything not in keep_loaded",
|
||||
every shot, in full -- skipping partially_unload entirely.
|
||||
|
||||
What that evicts is the DiT, three lines before the next shot needs it again. On
|
||||
a machine whose RAM is already full of finished frames there is nowhere for it to
|
||||
go but disk, so the reload is a read from the drive, once per shot. Reported as
|
||||
thrashing that slows the preload, and it is exactly that: the same weights being
|
||||
read back at every boundary.
|
||||
|
||||
The VAE knows what its own decode costs -- ComfyUI sizes it with
|
||||
memory_used_decode and uses that number everywhere else. Asked for that instead,
|
||||
a card with headroom frees NOTHING and the DiT simply stays. A card without
|
||||
headroom frees what it needs and no more, which is what partially_unload is for.
|
||||
1e30 remains the fallback for a VAE that cannot estimate itself."""
|
||||
latent = out_latent["samples"]
|
||||
if latent.is_nested:
|
||||
latent = latent.unbind()[0]
|
||||
if free_first is not None:
|
||||
try:
|
||||
mm.free_memory(_decode_headroom(vae, latent), mm.get_torch_device(),
|
||||
keep_loaded=_resident(keep or (vae,)))
|
||||
except Exception:
|
||||
pass
|
||||
# A VAE THAT ALREADY TILES DOES NOT NEED TO BE ASKED TO, AND ASKING COSTS 3x.
|
||||
#
|
||||
# MiniMaxH3VideoVAE.decode_tiled is, in full:
|
||||
#
|
||||
# def decode_tiled(self, z, **kwargs):
|
||||
# return self.decode(z)
|
||||
#
|
||||
# Every tile_t/overlap_t/tile_x/tile_y this function computes is discarded, so
|
||||
# the tiling the widget promises is not happening here -- the model tiles
|
||||
# internally either way (256px spatial, 17-frame temporal), which is why
|
||||
# comfy/sd.py sets handles_tiling on it.
|
||||
#
|
||||
# What the detour costs is the OUTPUT BUFFER. comfy's VAE.decode preallocates
|
||||
# ONE result at vae_output_dtype and hands it to the model as output_buffer=,
|
||||
# and MiniMaxH3VideoVAE.decode_temporal writes finalized chunks straight into
|
||||
# it. Going through decode_tiled instead reaches _decode_tiled_owned, which
|
||||
# calls the model with output_buffer=None -- so decode_temporal allocates its
|
||||
# own at torch.float32 -- and then makes an fp16 `copy=True` of that. Two
|
||||
# buffers, the larger of them at double width:
|
||||
#
|
||||
# tiled : fp32 2.60GB + fp16 copy 1.30GB = 3.90GB per shot
|
||||
# decode: one preallocated fp16 = 1.30GB per shot
|
||||
#
|
||||
# at 362 frames of 1056x608. Every shot, on the node's own default.
|
||||
#
|
||||
# So: when the VAE owns its tiling AND can be written into, the un-tiled call IS
|
||||
# the tiled one, minus the copies. Anything else keeps the old path -- this is a
|
||||
# detour around a detour, not a claim that tiling is useless.
|
||||
_owns_tiling = bool(getattr(vae, "handles_tiling", False) and getattr(
|
||||
getattr(vae, "first_stage_model", None), "comfy_has_chunked_io", False))
|
||||
if tiled and _owns_tiling:
|
||||
imgs = vae.decode(latent)
|
||||
elif tiled:
|
||||
# Temporal + spatial tiling. Without tile_t the VAE expands the WHOLE latent
|
||||
# clip at once, which on a 243-frame 1344x768 shot is the single largest
|
||||
# allocation in the run -- and on an unpruned checkpoint that is already
|
||||
# streaming, it is what tips the card over. Decoding in temporal chunks
|
||||
# trades a little speed for a much lower peak; None keeps ComfyUI's defaults.
|
||||
args = {}
|
||||
tile_t = _auto_tile_t(latent.shape[2] if latent.ndim >= 5 else 0, tile_t)
|
||||
if tile_t:
|
||||
args["tile_t"] = int(tile_t)
|
||||
args["overlap_t"] = max(1, int(tile_t) // 8)
|
||||
if tile_xy:
|
||||
args["tile_x"] = int(tile_xy)
|
||||
args["tile_y"] = int(tile_xy)
|
||||
try:
|
||||
imgs = vae.decode_tiled(latent, **args) if args else vae.decode_tiled(latent)
|
||||
except TypeError:
|
||||
imgs = vae.decode_tiled(latent) # older signature without tile_t
|
||||
else:
|
||||
imgs = vae.decode(latent)
|
||||
if len(imgs.shape) == 5:
|
||||
imgs = imgs.reshape(-1, imgs.shape[-3], imgs.shape[-2], imgs.shape[-1])
|
||||
return imgs
|
||||
|
||||
|
||||
def _decode_audio(audio_vae, out_latent):
|
||||
latent = out_latent["samples"]
|
||||
if latent.is_nested:
|
||||
latent = latent.unbind()[-1]
|
||||
audio = audio_vae.decode(latent).movedim(-1, 1)
|
||||
std = torch.std(audio, dim=[1, 2], keepdim=True) * 5.0
|
||||
std[std < 1.0] = 1.0
|
||||
audio = audio / std
|
||||
sr = getattr(audio_vae, "audio_sample_rate_output", getattr(audio_vae, "audio_sample_rate", 44100))
|
||||
return {"waveform": audio, "sample_rate": sr}
|
||||
|
||||
|
||||
def _is_oom(e):
|
||||
return isinstance(e, torch.cuda.OutOfMemoryError) or "out of memory" in str(e).lower()
|
||||
|
||||
|
||||
def _deep_cleanup():
|
||||
"""Release cached VRAM between shots so a long chain does not accumulate and OOM.
|
||||
|
||||
It unloads NOTHING. soft_empty_cache(force) ignores `force` in current ComfyUI
|
||||
(model_management.py:2050) -- the body only reaches empty_cache() and
|
||||
ipc_collect() -- so this drops cached blocks, not models. The `True` is kept
|
||||
only for older builds that read it; the older comment here claimed this took an
|
||||
unload_all_models path, and it does not."""
|
||||
try:
|
||||
mm.soft_empty_cache(True)
|
||||
except TypeError:
|
||||
mm.soft_empty_cache()
|
||||
try:
|
||||
if torch.cuda.is_available():
|
||||
torch.cuda.empty_cache()
|
||||
torch.cuda.ipc_collect()
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
DECODE_HEADROOM = 1.25 # over ComfyUI's own estimate, for working allocations
|
||||
|
||||
|
||||
SAMPLE_HEADROOM = 1.35 # likewise for sampling, which is the longer stretch
|
||||
|
||||
|
||||
def _decode_headroom(vae, latent):
|
||||
"""VRAM this decode actually needs, by the VAE's own estimate. 1e30 if unknown.
|
||||
|
||||
ComfyUI sizes every VAE with memory_used_decode and uses that number itself, so
|
||||
it is the honest figure to hand free_memory. The alternative -- and what was here
|
||||
-- is 1e30, which means "unload everything" and evicts the DiT before every
|
||||
decode, three lines before the next shot reloads it.
|
||||
|
||||
1e30 on failure rather than 0: a bad estimate that frees too little turns a slow
|
||||
render into an OOM, and a wrong guess should fall back to the behaviour that has
|
||||
been running, not to no freeing at all."""
|
||||
try:
|
||||
dtype = getattr(vae, "vae_dtype", None) or latent.dtype
|
||||
need = float(vae.memory_used_decode(tuple(latent.shape), dtype))
|
||||
if need > 0:
|
||||
return need * DECODE_HEADROOM
|
||||
except Exception:
|
||||
pass
|
||||
return 1e30
|
||||
|
||||
|
||||
def _resident(models):
|
||||
"""The LoadedModel entries ComfyUI currently holds for `models`.
|
||||
|
||||
That is the form free_memory's keep_loaded wants: it compares against the
|
||||
entries in current_loaded_models, not against the ModelPatcher objects a node
|
||||
is holding. Anything not matched is simply not kept, so a model that is not
|
||||
resident costs nothing here."""
|
||||
out = []
|
||||
for lm in list(getattr(mm, "current_loaded_models", [])):
|
||||
for m in models or ():
|
||||
if m is None:
|
||||
continue
|
||||
try:
|
||||
if lm.model is m or getattr(lm, "model", None) is getattr(m, "model", None):
|
||||
if lm not in out:
|
||||
out.append(lm)
|
||||
except Exception:
|
||||
pass
|
||||
return out
|
||||
|
||||
|
||||
def _image_out_dtype():
|
||||
"""The dtype ComfyUI itself hands between nodes on THIS install.
|
||||
|
||||
The join used to end in a hard-coded .float(), commented "back to what every
|
||||
downstream node expects". That was true when it was written and is not a
|
||||
constant: ComfyUI has --fp16-intermediates, and on an install running it the
|
||||
VAE's own decode already returns fp16 -- VAE.vae_output_dtype() IS
|
||||
model_management.intermediate_dtype() (comfy/sd.py) -- as do EmptyLatentImage
|
||||
and the rest of nodes.py. So on that install the node was taking frames the
|
||||
VAE handed it in fp16, widening them to fp32 nothing had asked for, and
|
||||
handing them to nodes whose own convention is fp16.
|
||||
|
||||
It is the largest thing this node holds, so the widening is not free: the
|
||||
2580-frame chain costed at the join is 9.3GB as fp16 and 18.5GB as fp32,
|
||||
against 44.6GB of staged weights on a 62GB machine -- which is the difference
|
||||
between the render finishing and the OOM killer taking the server. Reported as
|
||||
exactly that, twice.
|
||||
|
||||
Asked, not assumed, and never widened: whatever ComfyUI says it wants between
|
||||
nodes is what the chain is built in. An install with the flag off is told
|
||||
float32 and gets float32, byte for byte what it got before. Older builds have
|
||||
no intermediate_dtype at all, so the fallback is the old constant."""
|
||||
try:
|
||||
return mm.intermediate_dtype()
|
||||
except Exception:
|
||||
return torch.float32
|
||||
|
||||
|
||||
def _evict_all_but(keep_model, latent=None):
|
||||
"""Unload every model EXCEPT the diffusion model from the GPU.
|
||||
|
||||
This is the fix for VRAM ratcheting across a long chain. soft_empty_cache()
|
||||
only drops the CUDA allocator's cached blocks -- it does NOT unload models, so
|
||||
ComfyUI keeps the Qwen3-VL text encoder (~14.6GB) and both VAEs resident in
|
||||
current_loaded_models alongside the DiT. Each shot re-encodes the prompt
|
||||
(text encoder), encodes the handoff keyframe (video VAE), then samples (DiT),
|
||||
so all three compete for the card.
|
||||
|
||||
ComfyUI does free ahead of each load -- load_models_gpu() calls free_memory()
|
||||
for what it is about to need (model_management.py:975), so the weight path is
|
||||
not purely reactive. What it cannot size for is a long chain's ACTIVATIONS on
|
||||
a card where the DiT is most of the VRAM. Freeing explicitly, right after
|
||||
conditioning is built and before sampling, keeps only what the sampler needs.
|
||||
|
||||
ASKED FOR HONESTLY, and this is the expensive one. free_memory computes
|
||||
`memory_to_free = memory_required - get_free_memory(device)`, so 1e30 meant
|
||||
"unload everything but the DiT" on every shot, unconditionally -- on a 48GB card
|
||||
with room for all of it as readily as on a 16GB one. What it unloads is the
|
||||
~14.6GB text encoder and both VAEs, and the next shot re-encodes the prompt and
|
||||
the handoff keyframe, so all three come straight back. On a machine whose RAM is
|
||||
already full of finished frames they come back from DISK, once per shot, which is
|
||||
the thrashing this was reported as.
|
||||
|
||||
The DiT can size its own activations -- memory_required(shape) is what ComfyUI
|
||||
itself calls before a load -- so ask for that. A card with room frees nothing and
|
||||
keeps the encoder resident; a card without frees exactly as much as it must.
|
||||
1e30 stays the fallback, because a bad estimate that frees too little turns a
|
||||
slow render into an OOM."""
|
||||
need = 1e30
|
||||
try:
|
||||
if latent is not None:
|
||||
shape = latent["samples"].shape if isinstance(latent, dict) else latent.shape
|
||||
need = float(keep_model.model.memory_required(tuple(shape))) * SAMPLE_HEADROOM
|
||||
if not (need > 0):
|
||||
need = 1e30
|
||||
except Exception:
|
||||
need = 1e30
|
||||
try:
|
||||
mm.free_memory(need, mm.get_torch_device(),
|
||||
keep_loaded=_resident([keep_model]))
|
||||
except Exception:
|
||||
try:
|
||||
mm.soft_empty_cache(True)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
def _sample_on_sigmas(model, seed, cfg, sampler_name, positive, negative, latent, sigmas):
|
||||
"""common_ksampler, driven by an EXTERNAL sigma schedule.
|
||||
|
||||
common_ksampler derives its sigmas from (sampler_name, scheduler, steps, denoise)
|
||||
and takes no schedule argument, so a schedule computed anywhere else cannot
|
||||
reach it. Under PDD that is fatal rather than merely inconvenient: the heads
|
||||
accept only their nine trained boundaries, and re-deriving the grid from
|
||||
widgets means hitting it by coincidence and losing it again the moment a step
|
||||
count changes.
|
||||
|
||||
Mirrors nodes.common_ksampler's noise / mask / callback handling exactly -- the
|
||||
only substitution is comfy.sample.sample_custom for comfy.sample.sample."""
|
||||
latent_image = latent["samples"]
|
||||
latent_image = comfy.sample.fix_empty_latent_channels(
|
||||
model, latent_image,
|
||||
latent.get("downscale_ratio_spacial", None),
|
||||
latent.get("downscale_ratio_temporal", None))
|
||||
noise = comfy.sample.prepare_noise(latent_image, seed, latent.get("batch_index"))
|
||||
# `steps` here only sizes the progress bar -- the schedule is `sigmas`, whose
|
||||
# step count is one less than its length (the trailing 0.0 is an endpoint).
|
||||
callback = latent_preview.prepare_callback(model, max(len(sigmas) - 1, 1))
|
||||
samples = comfy.sample.sample_custom(
|
||||
model, noise, cfg, comfy.samplers.sampler_object(sampler_name), sigmas,
|
||||
positive, negative, latent_image,
|
||||
noise_mask=latent.get("noise_mask"), callback=callback,
|
||||
disable_pbar=not comfy.utils.PROGRESS_BAR_ENABLED, seed=seed)
|
||||
out = latent.copy()
|
||||
out.pop("downscale_ratio_spacial", None)
|
||||
out.pop("downscale_ratio_temporal", None)
|
||||
out["samples"] = samples
|
||||
return out
|
||||
|
||||
|
||||
RESIZE_CHUNK = 32
|
||||
|
||||
|
||||
def _stream_chunks(total):
|
||||
"""A collector that writes upscaled chunks into ONE destination as they land.
|
||||
|
||||
Both chunk loops in _upscale_frames used `out.append(...)` then
|
||||
`frames = torch.cat(out, dim=0)`. That is the shape the finished-chain join was
|
||||
rebuilt to stop, at a LARGER size: the list holds the whole upscaled chain and
|
||||
the cat allocates a second one, both live at the cat, and `out` is a local that
|
||||
is never cleared -- so it survives the cat, survives the trailing resize, and is
|
||||
still bound at the return. Meanwhile the CALLER's pre-upscale chain cannot be
|
||||
dropped either, because `part = frames[s:s+batch]` is a view into it.
|
||||
|
||||
At 2580 frames of 1056x608 that is 9.26GB per copy per doubling: 37GB x2 at 2x,
|
||||
and 148GB x2 with the RealESRGAN_x4plus that is sitting in models/upscale_models.
|
||||
Preallocating from the first chunk and copying into it removes exactly one of
|
||||
those two, and drops the list at the same time.
|
||||
|
||||
The destination is sized from the FIRST chunk, so the model's scale factor does
|
||||
not have to be known in advance, and the frame count is the caller's own -- an
|
||||
upscaler changes width and height, never the number of frames."""
|
||||
state = {"dst": None, "at": 0}
|
||||
|
||||
def put(piece):
|
||||
if state["dst"] is None:
|
||||
state["dst"] = torch.empty((int(total),) + tuple(piece.shape[1:]),
|
||||
dtype=piece.dtype, device=piece.device)
|
||||
k = int(piece.shape[0])
|
||||
end = min(state["at"] + k, state["dst"].shape[0])
|
||||
if end > state["at"]:
|
||||
state["dst"][state["at"]:end].copy_(piece[:end - state["at"]])
|
||||
state["at"] = end
|
||||
|
||||
def done():
|
||||
d, at = state["dst"], state["at"]
|
||||
if d is None:
|
||||
return None
|
||||
return d if at == d.shape[0] else d[:at]
|
||||
|
||||
return put, done
|
||||
|
||||
|
||||
def _resize_short_edge(frames, target, method="lanczos", chunk=0):
|
||||
"""Resize a [B,H,W,C] frame batch so its short edge == target (keeping aspect,
|
||||
snapped to /32). Plain high-quality resize -- enlarges, doesn't add detail.
|
||||
|
||||
IN CHUNKS, BECAUSE LANCZOS IS FOUR FULL-LENGTH COPIES. The whole chain went
|
||||
into one common_upscale call, and comfy.utils.lanczos is three successive list
|
||||
comprehensions over every frame at once:
|
||||
|
||||
images = [Image.fromarray(...) for image in samples] # N at source size
|
||||
images = [image.resize(...) for image in images] # N at target size
|
||||
images = [torch.from_numpy(np.array(im).astype(np.float32)/255.) ...]
|
||||
result = torch.stack(images)
|
||||
return result.to(samples.device, samples.dtype)
|
||||
|
||||
A comprehension builds the new list completely before rebinding the name, so at
|
||||
each rebind BOTH are live; then torch.stack allocates a full copy while its list
|
||||
still exists, and .to() allocates the result while the stack still exists. Note
|
||||
the astype(np.float32): the input is fp16 but the two largest transients are at
|
||||
DOUBLE its width. At 2580 frames to a 1080 short edge that peaked around 147GB
|
||||
to produce a 29GB result, and it fires on a DOWNSCALE too.
|
||||
|
||||
Chunked, the peak is the result plus one chunk's worth of that machinery. It is
|
||||
bit-identical: PIL resizes each frame independently, so per-chunk and per-chain
|
||||
give the same pixels. The early return for an already-correct size is kept, so
|
||||
the common no-op case still allocates nothing."""
|
||||
b, h, w, c = frames.shape
|
||||
if min(h, w) == target:
|
||||
return frames
|
||||
if h <= w:
|
||||
nh = target; nw = max(32, int(round(target * w / h / 32) * 32))
|
||||
else:
|
||||
nw = target; nh = max(32, int(round(target * h / w / 32) * 32))
|
||||
step = max(1, int(chunk) or RESIZE_CHUNK)
|
||||
out = torch.empty((b, nh, nw, c), dtype=frames.dtype, device=frames.device)
|
||||
for i in range(0, b, step):
|
||||
part = comfy.utils.common_upscale(
|
||||
frames[i:i + step].movedim(-1, 1), nw, nh, method, "disabled")
|
||||
out[i:i + step].copy_(part.movedim(1, -1))
|
||||
del part
|
||||
return out
|
||||
|
||||
|
||||
def _upscale_frames(frames, mode, model_name, target_short_edge, batch=4):
|
||||
"""Optional post-pass upscale of the finished frames (on CPU).
|
||||
mode 'model' : run a ComfyUI upscale model (Real-ESRGAN/UltraSharp class)
|
||||
via the registered loader+apply nodes, chunked with cleanup
|
||||
so 2000+ frames don't OOM; then fit to target short edge.
|
||||
mode 'rtx' : NVIDIA RTX Video Super Resolution (Tensor Cores; fastest,
|
||||
best quality for video -- needs Nvidia_RTX_Nodes_ComfyUI).
|
||||
mode 'lanczos' : plain high-quality resize to the target short edge.
|
||||
Any failure falls back to lanczos (or the raw frames), so it never breaks a
|
||||
render. Returns (frames, note). NOTE: this SHARPENS/ENLARGES; it does not
|
||||
reconstruct video detail the way a second-model (LTX 2.3) pass does."""
|
||||
if mode == "off" or frames is None or getattr(frames, "shape", [0])[0] == 0:
|
||||
return frames, ""
|
||||
note = ""
|
||||
if mode == "rtx":
|
||||
# NVIDIA RTX Video Super Resolution (Comfy-Org/Nvidia_RTX_Nodes_ComfyUI).
|
||||
# Runs on RTX Tensor Cores -- far faster than ESRGAN-class models and
|
||||
# generally cleaner on video, though like them it enhances/enlarges rather
|
||||
# than reconstructing detail (an LTX 2.3 re-generation does that).
|
||||
try:
|
||||
rtx = (_find_node(["rtx", "video", "super"]) or _find_node(["rtxvideosuperresolution"])
|
||||
or _find_node(["rtx", "upscale"]))
|
||||
if rtx is None:
|
||||
raise RuntimeError("RTX node not installed (Nvidia_RTX_Nodes_ComfyUI)")
|
||||
scale = 2
|
||||
if target_short_edge and int(target_short_edge) > 0:
|
||||
cur = min(frames.shape[1], frames.shape[2])
|
||||
if cur > 0:
|
||||
scale = max(1, min(4, int(round(int(target_short_edge) / cur))))
|
||||
_put, _done = _stream_chunks(frames.shape[0])
|
||||
n = frames.shape[0]
|
||||
step = max(1, int(batch))
|
||||
for st in range(0, n, step):
|
||||
part = frames[st:st + step]
|
||||
res = None
|
||||
for kw in ({"image": part, "scale": scale}, {"images": part, "scale": scale},
|
||||
{"image": part, "scale_factor": scale}, {"image": part}):
|
||||
try:
|
||||
res = _invoke_node(rtx, **kw); break
|
||||
except TypeError:
|
||||
continue
|
||||
if res is None:
|
||||
raise RuntimeError("RTX node signature not recognized")
|
||||
_put(res.detach().to("cpu"))
|
||||
del res, part
|
||||
_deep_cleanup()
|
||||
frames = _done()
|
||||
note = f"RTX Video Super Resolution x{scale}"
|
||||
if target_short_edge and int(target_short_edge) > 0:
|
||||
frames = _resize_short_edge(frames, int(target_short_edge))
|
||||
note += f"; fit to {int(target_short_edge)}px short edge"
|
||||
return frames, note
|
||||
except Exception as e:
|
||||
mode = "model"
|
||||
note = f"RTX upscale unavailable ({e}); fell back to model/lanczos"
|
||||
if mode == "model" and model_name and model_name != "none":
|
||||
try:
|
||||
loader = _find_node(["upscale", "model", "load"]) or _find_node(["loadupscalemodel"])
|
||||
applier = _find_node(["imageupscale", "model"]) or _find_node(["upscaleimageusingmodel"])
|
||||
if loader is None or applier is None:
|
||||
raise RuntimeError("upscale-model nodes not found")
|
||||
up_model = _invoke_node(loader, model_name=model_name)
|
||||
_put, _done = _stream_chunks(frames.shape[0])
|
||||
n = frames.shape[0]
|
||||
for s in range(0, n, max(1, int(batch))):
|
||||
part = frames[s:s + max(1, int(batch))]
|
||||
res = _invoke_node(applier, upscale_model=up_model, image=part)
|
||||
_put(res.detach().to("cpu"))
|
||||
del res, part
|
||||
_deep_cleanup()
|
||||
frames = _done()
|
||||
note = f"upscaled with {model_name}"
|
||||
except Exception as e:
|
||||
mode = "lanczos"
|
||||
note = f"model upscale unavailable ({e}); used lanczos"
|
||||
if target_short_edge and int(target_short_edge) > 0:
|
||||
try:
|
||||
frames = _resize_short_edge(frames, int(target_short_edge))
|
||||
note = (note + "; " if note else "") + f"fit to {int(target_short_edge)}px short edge"
|
||||
except Exception as e:
|
||||
note = (note + "; " if note else "") + f"resize failed ({e})"
|
||||
elif mode == "lanczos" and not note:
|
||||
note = "lanczos selected but no target set -> unchanged"
|
||||
return frames, note
|
||||
|
||||
|
||||
def _find_node(substrings):
|
||||
"""Find a registered node whose key contains all of `substrings` (lowercased)."""
|
||||
maps = getattr(nodes, "NODE_CLASS_MAPPINGS", {}) or {}
|
||||
for k, v in maps.items():
|
||||
kl = k.lower()
|
||||
if all(s in kl for s in substrings):
|
||||
return v
|
||||
return None
|
||||
|
||||
|
||||
def _invoke_node(cls, **kwargs):
|
||||
"""Call a registered ComfyUI node (V1 FUNCTION or V3 execute) with kwargs and
|
||||
return its first output. Used to reuse ComfyUI's own upscale-model loader/apply
|
||||
so we don't reimplement spandrel loading or tiled scaling."""
|
||||
inst = cls()
|
||||
fn = None
|
||||
if getattr(cls, "FUNCTION", None) and hasattr(inst, cls.FUNCTION):
|
||||
fn = getattr(inst, cls.FUNCTION)
|
||||
else:
|
||||
for cand in ("execute", "upscale", "load_model", "load"):
|
||||
if hasattr(inst, cand):
|
||||
fn = getattr(inst, cand); break
|
||||
if fn is None:
|
||||
raise RuntimeError("no callable entrypoint")
|
||||
out = fn(**kwargs)
|
||||
out = getattr(out, "result", out)
|
||||
return out[0] if isinstance(out, (tuple, list)) else out
|
||||
|
||||
|
||||
# --- THE GRADE THE CHAIN ADDS TO ITSELF -------------------------------------
|
||||
# Every shot boundary decodes a shot, hands its LAST frame over, and re-encodes that as
|
||||
# the next shot's keyframe. The distill reproduces the keyframe faithfully enough to
|
||||
# inherit whatever is already in it and SYNTHESISES frame 0 rather than copying it, so
|
||||
# its own bias lands on top: S_next = a*S + b, a near 1, b above 0. Linear at best,
|
||||
# geometric at worst, invisible shot to shot. And the VAE hard-clips every decode to
|
||||
# 0..1, which makes the expansion a RATCHET -- headroom spent is not recoverable, so it
|
||||
# shows as crushed blacks and blown highlights rather than merely as more contrast.
|
||||
#
|
||||
# These two are the measurement and the correction. Both work per colour channel,
|
||||
# because the clip is per channel: the VAE un-whitens with ImageNet stds before it
|
||||
# clamps, so the 0..1 rails sit at different distances in each channel and the blue
|
||||
# floor and red ceiling bite first. A single luma number would miss the colour half.
|
||||
LEVEL_POOL = 64 # cells per axis the level statistics are measured on
|
||||
|
||||
|
||||
def frame_levels(img):
|
||||
"""(mean, std) per colour channel for one frame, as 3-vectors, or (None, None).
|
||||
|
||||
Area-pooled to LEVEL_POOL first, so a pre-upscale frame and an upscaled one can be
|
||||
compared: pooling measures the PICTURE's levels rather than its resolution. Measured
|
||||
across a 2x resize, std agrees to 0.28% on picture-like content -- and to only 15%
|
||||
on pure noise, because pooling cannot preserve variance that lives entirely at the
|
||||
pixel scale. Real frames are the former, and whatever residual there is cancels
|
||||
anyway: the caller measures the same pipeline difference separately and subtracts it.
|
||||
|
||||
float32 throughout, deliberately: these frames are fp16 under
|
||||
--fp16-intermediates, and an fp16 mean accumulated over a 1344x768 frame biases
|
||||
badly enough to matter at the sizes being corrected here."""
|
||||
x = img
|
||||
if x.dim() == 4:
|
||||
x = x[0]
|
||||
if x.dim() != 3 or int(x.shape[-1]) < 3:
|
||||
return None, None
|
||||
if int(x.shape[0]) < 2 or int(x.shape[1]) < 2:
|
||||
return None, None
|
||||
x = x[..., :3].float().permute(2, 0, 1).unsqueeze(0)
|
||||
p = torch.nn.functional.adaptive_avg_pool2d(x, LEVEL_POOL)[0].reshape(3, -1)
|
||||
return p.mean(dim=1), p.std(dim=1)
|
||||
|
||||
|
||||
# The per-shot motion envelope lived here, measured so the built footsteps could be
|
||||
# timed off the picture. Nothing is built any more -- see the note at the top of
|
||||
# audio.py -- so there is nothing left to time, and a measurement with no reader is a
|
||||
# measurement that rots. Removed with the synthesiser it served.
|
||||
|
||||
|
||||
def apply_levels(img, gain, offset):
|
||||
"""Rescale a frame's contrast and level about its OWN per-channel mean.
|
||||
|
||||
The pivot is the frame's own mean and never a target. That is the whole reason this
|
||||
can run on any scene: a beat that walks into a darker room keeps its darkness,
|
||||
because nothing here knows or cares what the level is -- only how much the last
|
||||
boundary expanded it. Anchoring to shot 1 instead would cancel every deliberate
|
||||
lighting change in the film, which is the opposite failure.
|
||||
|
||||
Clamped into 0..1 because the next thing that happens to this frame is an 8-bit
|
||||
quantisation (comfy.utils.common_upscale goes through a uint8 PIL round trip even
|
||||
at the same size), so there is no headroom outside the range to borrow from."""
|
||||
x = img.float()
|
||||
c = min(3, int(x.shape[-1]))
|
||||
m = x[..., :c].reshape(-1, c).mean(dim=0)
|
||||
g = gain[:c].to(device=x.device, dtype=x.dtype)
|
||||
o = offset[:c].to(device=x.device, dtype=x.dtype)
|
||||
y = x.clone()
|
||||
y[..., :c] = ((x[..., :c] - m) * g + m + o).clamp(0.0, 1.0)
|
||||
return y.to(img.dtype)
|
||||
@@ -0,0 +1,109 @@
|
||||
# H3-LongVideos -- https://github.com/Smite79/MiniMax-H3-LongVideos
|
||||
# Copyright (c) 2026 Smite79. All rights reserved.
|
||||
# Redistribution, in whole or in part, requires written permission.
|
||||
# This notice may not be removed or altered. See LICENSE.
|
||||
"""Per-shot records passed from prompt planning to rendering."""
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
|
||||
@dataclass
|
||||
class Shot:
|
||||
prompt: str
|
||||
cast: list[str]
|
||||
speech: bool
|
||||
sounded: bool
|
||||
voiced_only: bool
|
||||
events: list[str]
|
||||
frame_count: int = 0
|
||||
refs: list[object] = field(default_factory=list)
|
||||
line_seconds: float = 0.0 # the planner's estimate of the spoken line, words / WORDS_PER_SEC
|
||||
|
||||
|
||||
@dataclass
|
||||
class ShotPlan:
|
||||
shots: list[Shot] = field(default_factory=list)
|
||||
|
||||
@property
|
||||
def prompts(self):
|
||||
return [shot.prompt for shot in self.shots]
|
||||
|
||||
def add(self, prompt, cast, speech, sounded, voiced_only, events):
|
||||
shot = Shot(prompt, list(cast or ()), bool(speech), bool(sounded),
|
||||
bool(voiced_only), list(events or ()))
|
||||
self.shots.append(shot)
|
||||
|
||||
def set_frame_counts(self, counts):
|
||||
counts = [int(n) for n in counts]
|
||||
if len(counts) != len(self.shots) or any(n <= 0 for n in counts):
|
||||
raise ValueError("frame counts must be positive and match the planned shots")
|
||||
for shot, count in zip(self.shots, counts):
|
||||
shot.frame_count = count
|
||||
|
||||
def validate(self):
|
||||
if any(shot.frame_count <= 0 for shot in self.shots):
|
||||
raise ValueError("each shot needs a positive frame count before rendering")
|
||||
return self
|
||||
|
||||
def __len__(self):
|
||||
return len(self.shots)
|
||||
|
||||
|
||||
@dataclass
|
||||
class PreparedVideo:
|
||||
"""Resolved inputs consumed by the render stage; model/tensor handles are shared."""
|
||||
_placed_shots: object
|
||||
_first_is_plate: object
|
||||
_returns: object
|
||||
_soft_landing: object
|
||||
_tagged_names: object
|
||||
ambient_audio: object
|
||||
ambient_level: float
|
||||
apply_model_sampling: bool
|
||||
audio_vae: object
|
||||
auto_sound: bool
|
||||
bared_shots: object
|
||||
cfg: float
|
||||
cleanup_between_shots: bool
|
||||
clip: object
|
||||
first_frame: object
|
||||
foley_level: float
|
||||
h: int
|
||||
latent_upscale: str
|
||||
latent_upscale_scale: float
|
||||
megapixels: float
|
||||
model: object
|
||||
moved_shots: object
|
||||
negative: object
|
||||
notes: list[str]
|
||||
plan: ShotPlan
|
||||
ref_noise_aug: float | None
|
||||
restart_after_removal: bool
|
||||
revealed_shots: object
|
||||
sampler_name: str
|
||||
scheduler: str
|
||||
seed: int
|
||||
shift_audio: float
|
||||
shift_video: float
|
||||
sigmas: object
|
||||
silence_nonspeech: bool
|
||||
speech_lead_seconds: float
|
||||
speech_tail_seconds: float
|
||||
hold_levels: float
|
||||
handoff_frames: int
|
||||
staging_shots: object
|
||||
steps: int
|
||||
stripped_shots: object
|
||||
cut_shots: object
|
||||
tiled_decode: bool
|
||||
trim_seam: bool
|
||||
upscale: str
|
||||
upscale_batch: int
|
||||
upscale_model: str
|
||||
upscale_target_short_edge: int
|
||||
vae: object
|
||||
w: int
|
||||
shot_rooms: object = None # {0-based shot: (room it opens in, room it ends in)}
|
||||
hardware_changed: object = None # 1-based shots that put hardware on or take it off
|
||||
shot_frames: object = None # {0-based shot: (who its frames show, who is still there at its end)}
|
||||
reentry_shots: object = None # {0-based shot: who walks in while the keyframe still has them}
|
||||
File diff suppressed because it is too large
Load Diff
+18
-2
@@ -1253,11 +1253,13 @@ def curate_h3_prompt(
|
||||
soundscape_text = _reference_text(soundscape)
|
||||
bgm_text = _reference_text(bgm)
|
||||
reference_description = ""
|
||||
individual_reference_descriptions = []
|
||||
if selected:
|
||||
reference_description = " ".join(
|
||||
individual_reference_descriptions = [
|
||||
_reference_context(ref, picture_number)
|
||||
for picture_number, (_slot, ref) in enumerate(selected, 1)
|
||||
)
|
||||
]
|
||||
reference_description = " ".join(individual_reference_descriptions)
|
||||
|
||||
prompt_parts = []
|
||||
_append_prompt_section(prompt_parts, "Scene anchor", anchor_text)
|
||||
@@ -1277,6 +1279,9 @@ def curate_h3_prompt(
|
||||
prompt = prompt[: _H3_PROMPT_MAX_CHARS - 3].rstrip() + "..."
|
||||
images = [_reference_image(ref) for _slot, ref in selected]
|
||||
images.extend([None] * (_H3_PROMPT_REF_SLOTS - len(images)))
|
||||
individual_reference_descriptions.extend(
|
||||
[""] * (_H3_PROMPT_REF_SLOTS - len(individual_reference_descriptions))
|
||||
)
|
||||
debug = (
|
||||
f"Selected {len(selected)} reference(s): "
|
||||
+ ", ".join(
|
||||
@@ -1295,6 +1300,7 @@ def curate_h3_prompt(
|
||||
soundscape_text,
|
||||
bgm_text,
|
||||
*images[:_H3_PROMPT_REF_SLOTS],
|
||||
*individual_reference_descriptions[:_H3_PROMPT_REF_SLOTS],
|
||||
)
|
||||
|
||||
|
||||
@@ -2687,6 +2693,7 @@ class DumasH3PromptCuratorNode:
|
||||
+ ("IMAGE",) * _H3_PROMPT_REF_SLOTS
|
||||
+ ("INT", "STRING", "STRING", "STRING", "STRING")
|
||||
+ ("IMAGE",) * _H3_PROMPT_REF_SLOTS
|
||||
+ ("STRING",) * _H3_PROMPT_REF_SLOTS
|
||||
)
|
||||
RETURN_NAMES = (
|
||||
"prompt",
|
||||
@@ -2713,6 +2720,15 @@ class DumasH3PromptCuratorNode:
|
||||
"original_ref_7",
|
||||
"original_ref_8",
|
||||
"original_ref_9",
|
||||
"compiled_ref_description_1",
|
||||
"compiled_ref_description_2",
|
||||
"compiled_ref_description_3",
|
||||
"compiled_ref_description_4",
|
||||
"compiled_ref_description_5",
|
||||
"compiled_ref_description_6",
|
||||
"compiled_ref_description_7",
|
||||
"compiled_ref_description_8",
|
||||
"compiled_ref_description_9",
|
||||
)
|
||||
FUNCTION = "curate_prompt"
|
||||
CATEGORY = "Dumas/MiniMax"
|
||||
|
||||
+74
-100
@@ -8,32 +8,15 @@ const DEFAULT_W = 520;
|
||||
const DEFAULT_H = 340;
|
||||
const DEFAULT_BEAT = "Describe this beat.";
|
||||
const STATE_PROPERTY = "dumas_h3_beat_prompt_state";
|
||||
const CONTINUITY_OPTIONS = ["", "soft carry", "hard cut", "keyframe carry", "handoff ref"];
|
||||
const REF_MODE_OPTIONS = ["", "auto ref2v", "where tagged", "first shot", "every shot", "every shot + handoff ref"];
|
||||
const MANAGED_DIRECTIVES = {
|
||||
seconds: ["seconds", "duration"],
|
||||
continuity: ["continuity"],
|
||||
ref_mode: ["ref_mode"],
|
||||
ref_noise_aug: ["ref_noise_aug"],
|
||||
anchor_add: ["anchor_add"],
|
||||
overall_soundscape: ["overall_soundscape", "soundscape"],
|
||||
non_diegetic_music: ["non_diegetic_music", "music"],
|
||||
remove: ["remove", "removed", "off"],
|
||||
add: ["add", "wear", "wearing"],
|
||||
};
|
||||
const DIRECTIVE_EXAMPLES = [
|
||||
["wardrobe set", "wardrobe: Maya = grey shorts, red jacket"],
|
||||
["wardrobe add", "wardrobe: Maya += red jacket"],
|
||||
["wardrobe remove", "wardrobe: Maya -= red jacket"],
|
||||
["seconds", "seconds: 8"],
|
||||
["exit", "exit: Maya"],
|
||||
["enter", "enter: Jon"],
|
||||
["continuity", "continuity: hard cut"],
|
||||
["ref_mode", "ref_mode: every shot"],
|
||||
["ref_noise_aug", "ref_noise_aug: 0.92"],
|
||||
["anchor_add", "anchor_add: harsh sodium-vapor spill, wet pavement, long-lens compression"],
|
||||
["overall_soundscape", "overall_soundscape: soft rain, distant traffic"],
|
||||
["non_diegetic_music", "non_diegetic_music: tense analog synth pulse"],
|
||||
["soundscape", "soundscape: fluorescent room tone, faint HVAC hum"],
|
||||
["music", "music: low ominous cello and sparse percussion"],
|
||||
["remove", "remove: red jacket"],
|
||||
["off", "off: steel collar"],
|
||||
["add", "add: white shirt underneath"],
|
||||
["wearing", "wearing: black coat"],
|
||||
];
|
||||
|
||||
function injectCSS() {
|
||||
@@ -183,7 +166,7 @@ function injectCSS() {
|
||||
}
|
||||
|
||||
function defaultState() {
|
||||
return { beats: [{ text: DEFAULT_BEAT }] };
|
||||
return { scene: "", character_sheet: "", beats: [{ text: DEFAULT_BEAT }] };
|
||||
}
|
||||
|
||||
function normalizeState(value) {
|
||||
@@ -200,7 +183,11 @@ function normalizeState(value) {
|
||||
const normalized = beats.map((beat) => ({
|
||||
text: typeof beat?.text === "string" ? beat.text : String(beat?.text || ""),
|
||||
}));
|
||||
return normalized.length ? { beats: normalized } : defaultState();
|
||||
return {
|
||||
scene: typeof parsed.scene === "string" ? parsed.scene : String(parsed.scene || ""),
|
||||
character_sheet: typeof parsed.character_sheet === "string" ? parsed.character_sheet : String(parsed.character_sheet || ""),
|
||||
beats: normalized.length ? normalized : [{ text: DEFAULT_BEAT }],
|
||||
};
|
||||
}
|
||||
|
||||
function readState(node) {
|
||||
@@ -321,6 +308,51 @@ function renderUI(node) {
|
||||
node._dh3bpRenderedState = JSON.stringify(state);
|
||||
ui.list.innerHTML = "";
|
||||
|
||||
const buildTopTextarea = ({ labelText, placeholder, value, onInput }) => {
|
||||
const card = document.createElement("div");
|
||||
card.className = "dh3bp-beat";
|
||||
|
||||
const label = document.createElement("div");
|
||||
label.className = "dh3bp-label";
|
||||
label.textContent = labelText;
|
||||
|
||||
const textarea = document.createElement("textarea");
|
||||
textarea.className = "dh3bp-text";
|
||||
textarea.placeholder = placeholder;
|
||||
textarea.value = value || "";
|
||||
textarea.addEventListener("input", () => {
|
||||
onInput(textarea.value);
|
||||
updateTextareaHeight(textarea);
|
||||
});
|
||||
textarea.addEventListener("keydown", stopCanvasKeyboard);
|
||||
|
||||
card.append(label, textarea);
|
||||
updateTextareaHeight(textarea);
|
||||
return card;
|
||||
};
|
||||
|
||||
ui.list.appendChild(buildTopTextarea({
|
||||
labelText: "Scene paragraph",
|
||||
placeholder: "Optional. Persistent location, lighting, camera, tone. Leave empty if you wire the Long Videos anchor input.",
|
||||
value: state.scene,
|
||||
onInput: (value) => {
|
||||
const next = readState(node);
|
||||
next.scene = value;
|
||||
writeState(node, next);
|
||||
},
|
||||
}));
|
||||
|
||||
ui.list.appendChild(buildTopTextarea({
|
||||
labelText: "Character sheet",
|
||||
placeholder: "Optional. One character per line, e.g. Maya: 27, she, silver hair, red jacket, the woman in <Picture 1>.",
|
||||
value: state.character_sheet,
|
||||
onInput: (value) => {
|
||||
const next = readState(node);
|
||||
next.character_sheet = value;
|
||||
writeState(node, next);
|
||||
},
|
||||
}));
|
||||
|
||||
state.beats.forEach((beat, index) => {
|
||||
const card = document.createElement("div");
|
||||
card.className = "dh3bp-beat";
|
||||
@@ -379,85 +411,27 @@ function renderUI(node) {
|
||||
return wrap;
|
||||
};
|
||||
|
||||
const secondsInput = document.createElement("input");
|
||||
secondsInput.className = "dh3bp-input";
|
||||
secondsInput.type = "text";
|
||||
secondsInput.placeholder = "8";
|
||||
secondsInput.value = readDirectiveValue(beat.text, MANAGED_DIRECTIVES.seconds);
|
||||
secondsInput.addEventListener("input", () => {
|
||||
applyTextUpdate(setDirectiveValue(textarea.value, "seconds", MANAGED_DIRECTIVES.seconds, secondsInput.value));
|
||||
const removeInput = document.createElement("input");
|
||||
removeInput.className = "dh3bp-input";
|
||||
removeInput.type = "text";
|
||||
removeInput.placeholder = "red jacket";
|
||||
removeInput.value = readDirectiveValue(beat.text, MANAGED_DIRECTIVES.remove);
|
||||
removeInput.addEventListener("input", () => {
|
||||
applyTextUpdate(setDirectiveValue(textarea.value, "remove", MANAGED_DIRECTIVES.remove, removeInput.value));
|
||||
});
|
||||
|
||||
const continuitySelect = document.createElement("select");
|
||||
continuitySelect.className = "dh3bp-select";
|
||||
CONTINUITY_OPTIONS.forEach((value) => {
|
||||
const option = document.createElement("option");
|
||||
option.value = value;
|
||||
option.textContent = value || "Default";
|
||||
continuitySelect.appendChild(option);
|
||||
});
|
||||
continuitySelect.value = readDirectiveValue(beat.text, MANAGED_DIRECTIVES.continuity);
|
||||
continuitySelect.addEventListener("change", () => {
|
||||
applyTextUpdate(setDirectiveValue(textarea.value, "continuity", MANAGED_DIRECTIVES.continuity, continuitySelect.value));
|
||||
});
|
||||
|
||||
const refModeSelect = document.createElement("select");
|
||||
refModeSelect.className = "dh3bp-select";
|
||||
REF_MODE_OPTIONS.forEach((value) => {
|
||||
const option = document.createElement("option");
|
||||
option.value = value;
|
||||
option.textContent = value || "Global";
|
||||
refModeSelect.appendChild(option);
|
||||
});
|
||||
refModeSelect.value = readDirectiveValue(beat.text, MANAGED_DIRECTIVES.ref_mode);
|
||||
refModeSelect.addEventListener("change", () => {
|
||||
applyTextUpdate(setDirectiveValue(textarea.value, "ref_mode", MANAGED_DIRECTIVES.ref_mode, refModeSelect.value));
|
||||
});
|
||||
|
||||
const refNoiseInput = document.createElement("input");
|
||||
refNoiseInput.className = "dh3bp-input";
|
||||
refNoiseInput.type = "text";
|
||||
refNoiseInput.placeholder = "0.95";
|
||||
refNoiseInput.value = readDirectiveValue(beat.text, MANAGED_DIRECTIVES.ref_noise_aug);
|
||||
refNoiseInput.addEventListener("input", () => {
|
||||
applyTextUpdate(setDirectiveValue(textarea.value, "ref_noise_aug", MANAGED_DIRECTIVES.ref_noise_aug, refNoiseInput.value));
|
||||
});
|
||||
|
||||
const anchorInput = document.createElement("input");
|
||||
anchorInput.className = "dh3bp-input";
|
||||
anchorInput.type = "text";
|
||||
anchorInput.placeholder = "extra per-shot style treatment";
|
||||
anchorInput.value = readDirectiveValue(beat.text, MANAGED_DIRECTIVES.anchor_add);
|
||||
anchorInput.addEventListener("input", () => {
|
||||
applyTextUpdate(setDirectiveValue(textarea.value, "anchor_add", MANAGED_DIRECTIVES.anchor_add, anchorInput.value));
|
||||
});
|
||||
|
||||
const soundscapeInput = document.createElement("input");
|
||||
soundscapeInput.className = "dh3bp-input";
|
||||
soundscapeInput.type = "text";
|
||||
soundscapeInput.placeholder = "faint traffic, loose sign rattle";
|
||||
soundscapeInput.value = readDirectiveValue(beat.text, MANAGED_DIRECTIVES.overall_soundscape);
|
||||
soundscapeInput.addEventListener("input", () => {
|
||||
applyTextUpdate(setDirectiveValue(textarea.value, "overall_soundscape", MANAGED_DIRECTIVES.overall_soundscape, soundscapeInput.value));
|
||||
});
|
||||
|
||||
const musicInput = document.createElement("input");
|
||||
musicInput.className = "dh3bp-input";
|
||||
musicInput.type = "text";
|
||||
musicInput.placeholder = "low pulsing synth tension";
|
||||
musicInput.value = readDirectiveValue(beat.text, MANAGED_DIRECTIVES.non_diegetic_music);
|
||||
musicInput.addEventListener("input", () => {
|
||||
applyTextUpdate(setDirectiveValue(textarea.value, "non_diegetic_music", MANAGED_DIRECTIVES.non_diegetic_music, musicInput.value));
|
||||
const addInput = document.createElement("input");
|
||||
addInput.className = "dh3bp-input";
|
||||
addInput.type = "text";
|
||||
addInput.placeholder = "white shirt underneath";
|
||||
addInput.value = readDirectiveValue(beat.text, MANAGED_DIRECTIVES.add);
|
||||
addInput.addEventListener("input", () => {
|
||||
applyTextUpdate(setDirectiveValue(textarea.value, "add", MANAGED_DIRECTIVES.add, addInput.value));
|
||||
});
|
||||
|
||||
controls.append(
|
||||
buildField({ labelText: "Seconds", input: secondsInput }),
|
||||
buildField({ labelText: "Continuity", input: continuitySelect }),
|
||||
buildField({ labelText: "Ref Mode", input: refModeSelect }),
|
||||
buildField({ labelText: "Ref Noise Aug", input: refNoiseInput }),
|
||||
buildField({ labelText: "Anchor Add", className: "dh3bp-control-wide", input: anchorInput }),
|
||||
buildField({ labelText: "Shot Soundscape", className: "dh3bp-control-wide", input: soundscapeInput }),
|
||||
buildField({ labelText: "Shot Music", className: "dh3bp-control-wide", input: musicInput }),
|
||||
buildField({ labelText: "Remove from memory", input: removeInput }),
|
||||
buildField({ labelText: "Add to memory", input: addInput }),
|
||||
);
|
||||
|
||||
const directives = document.createElement("div");
|
||||
@@ -501,7 +475,7 @@ function setupNode(node) {
|
||||
title.textContent = "Beat Prompt Builder";
|
||||
const subtitle = document.createElement("div");
|
||||
subtitle.className = "dh3bp-subtitle";
|
||||
subtitle.textContent = "One textbox per H3 beat, plus per-shot controls for timing, ref behavior, continuity, anchor adds, and audio directives.";
|
||||
subtitle.textContent = "Upstream Long Videos format: optional scene, optional character sheet, then one blank-line-separated beat per shot.";
|
||||
titleWrap.append(title, subtitle);
|
||||
|
||||
const addButton = document.createElement("button");
|
||||
|
||||
+5
-371
@@ -1,375 +1,9 @@
|
||||
import { app } from "/scripts/app.js";
|
||||
import { applyAdaptiveCanvasOnly } from "../shared/nodes2.mjs";
|
||||
|
||||
const COMFY_CLASS = "DumasH3LongVideos";
|
||||
const STATE_PROPERTY = "dumas_h3_longvideos_section_state";
|
||||
const DOM_WIDGET_NAME = "dumas_h3_longvideos_sections";
|
||||
const MIN_WIDTH = 520;
|
||||
const MIN_HEIGHT = 280;
|
||||
|
||||
const GROUPS = [
|
||||
{
|
||||
id: "prompt",
|
||||
label: "Prompt",
|
||||
defaultCollapsed: false,
|
||||
widgets: ["prompt", "resolution", "megapixels", "beat_split", "anchor_override", "shot_seconds", "plan_only", "fps"],
|
||||
},
|
||||
{
|
||||
id: "refs",
|
||||
label: "Refs",
|
||||
defaultCollapsed: true,
|
||||
widgets: ["ref_mode", "ref_image_size", "ref_noise_aug", "character_memory", "trim_seam", "vary_seed_per_shot", "handoff_offset"],
|
||||
},
|
||||
{
|
||||
id: "sampling",
|
||||
label: "Sampling",
|
||||
defaultCollapsed: true,
|
||||
widgets: [
|
||||
"steps", "cfg", "sampler_name", "scheduler", "seed",
|
||||
"apply_model_sampling", "shift_video", "shift_audio",
|
||||
"vram_headroom_gb", "allow_res_backoff",
|
||||
"decode_tile_frames", "decode_tile_size",
|
||||
],
|
||||
},
|
||||
{
|
||||
id: "audio",
|
||||
label: "Audio",
|
||||
defaultCollapsed: true,
|
||||
widgets: [
|
||||
"global_soundscape", "non_diegetic_music", "auto_soundscape",
|
||||
"auto_silence_nonspeech", "allow_nonspeech_vocals",
|
||||
"mute_nonspeech_audio", "mute_fade_ms",
|
||||
],
|
||||
},
|
||||
{
|
||||
id: "scene",
|
||||
label: "Scene Logic",
|
||||
defaultCollapsed: true,
|
||||
widgets: [
|
||||
"auto_wardrobe", "auto_props", "prevent_nudity", "exposed_terms",
|
||||
"anatomy_guard", "subject_count_guard", "lock_restraints",
|
||||
"contact_guard", "motion_guard", "solidity_guard",
|
||||
],
|
||||
},
|
||||
{
|
||||
id: "finish",
|
||||
label: "Upscale",
|
||||
defaultCollapsed: true,
|
||||
widgets: [
|
||||
"upscale", "upscale_model", "upscale_target_short_edge", "upscale_batch",
|
||||
],
|
||||
},
|
||||
{
|
||||
id: "overlay",
|
||||
label: "Overlays",
|
||||
defaultCollapsed: true,
|
||||
widgets: [
|
||||
"watermark_text", "watermark_position", "watermark_size", "watermark_opacity", "watermark_margin",
|
||||
"intro_text", "intro_position", "intro_seconds", "intro_fade", "intro_size",
|
||||
"overlay_font", "overlay_stroke",
|
||||
],
|
||||
},
|
||||
];
|
||||
|
||||
function injectCSS() {
|
||||
if (document.getElementById("dumas-h3lv-sections-css")) return;
|
||||
const style = document.createElement("style");
|
||||
style.id = "dumas-h3lv-sections-css";
|
||||
style.textContent = `
|
||||
.dh3lv-sections {
|
||||
box-sizing: border-box;
|
||||
width: 100%;
|
||||
padding: 8px 10px 6px;
|
||||
color: #e6e7eb;
|
||||
font: 12px/1.35 "Segoe UI", sans-serif;
|
||||
pointer-events: auto;
|
||||
background: linear-gradient(180deg, rgba(33, 36, 42, 0.96), rgba(22, 24, 29, 0.96));
|
||||
border-bottom: 1px solid rgba(255, 255, 255, 0.06);
|
||||
}
|
||||
.dh3lv-sections-head {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: space-between;
|
||||
gap: 8px;
|
||||
margin-bottom: 8px;
|
||||
}
|
||||
.dh3lv-sections-title {
|
||||
font-size: 11px;
|
||||
text-transform: uppercase;
|
||||
letter-spacing: 0.08em;
|
||||
color: #9da5b1;
|
||||
}
|
||||
.dh3lv-sections-actions {
|
||||
display: flex;
|
||||
gap: 6px;
|
||||
}
|
||||
.dh3lv-sections-list {
|
||||
display: flex;
|
||||
flex-wrap: wrap;
|
||||
gap: 6px;
|
||||
}
|
||||
.dh3lv-chip,
|
||||
.dh3lv-action {
|
||||
appearance: none;
|
||||
border: 1px solid #464d59;
|
||||
background: #262c35;
|
||||
color: #d7dce3;
|
||||
border-radius: 999px;
|
||||
padding: 5px 9px;
|
||||
cursor: pointer;
|
||||
font: inherit;
|
||||
line-height: 1.1;
|
||||
}
|
||||
.dh3lv-chip[data-open="true"] {
|
||||
background: #d96f2b;
|
||||
border-color: #f09358;
|
||||
color: #fff7f0;
|
||||
}
|
||||
.dh3lv-chip:hover,
|
||||
.dh3lv-action:hover {
|
||||
filter: brightness(1.06);
|
||||
}
|
||||
.dh3lv-count {
|
||||
opacity: 0.78;
|
||||
margin-left: 4px;
|
||||
font-size: 11px;
|
||||
}
|
||||
`;
|
||||
document.head.appendChild(style);
|
||||
}
|
||||
|
||||
function defaultState() {
|
||||
const state = {};
|
||||
for (const group of GROUPS) state[group.id] = !group.defaultCollapsed;
|
||||
return state;
|
||||
}
|
||||
|
||||
function parseState(value) {
|
||||
let parsed = value;
|
||||
if (typeof parsed === "string") {
|
||||
try {
|
||||
parsed = JSON.parse(parsed);
|
||||
} catch (_error) {
|
||||
parsed = null;
|
||||
}
|
||||
}
|
||||
const base = defaultState();
|
||||
if (!parsed || typeof parsed !== "object") return base;
|
||||
for (const group of GROUPS) {
|
||||
if (typeof parsed[group.id] === "boolean") base[group.id] = parsed[group.id];
|
||||
}
|
||||
return base;
|
||||
}
|
||||
|
||||
function readState(node) {
|
||||
return parseState(node.properties?.[STATE_PROPERTY] || node._dh3lvSectionState || "");
|
||||
}
|
||||
|
||||
function writeState(node, state) {
|
||||
const normalized = parseState(state);
|
||||
const serialized = JSON.stringify(normalized);
|
||||
node._dh3lvSectionState = serialized;
|
||||
node.properties = node.properties || {};
|
||||
node.properties[STATE_PROPERTY] = serialized;
|
||||
}
|
||||
|
||||
function findWidget(node, name) {
|
||||
return (node.widgets || []).find((widget) => widget?.name === name) || null;
|
||||
}
|
||||
|
||||
function isInteractiveTarget(target) {
|
||||
return !!target?.closest?.("button, input, textarea, select, label");
|
||||
}
|
||||
|
||||
function stopCanvasEvent(event) {
|
||||
if (isInteractiveTarget(event.target)) event.stopPropagation();
|
||||
}
|
||||
|
||||
function stopCanvasKeyboard(event) {
|
||||
if (isInteractiveTarget(event.target)) event.stopImmediatePropagation();
|
||||
}
|
||||
|
||||
function setWidgetHidden(widget, hidden) {
|
||||
if (!widget) return;
|
||||
if (!widget._dh3lvOriginal) {
|
||||
widget._dh3lvOriginal = {
|
||||
type: widget.type,
|
||||
computeSize: widget.computeSize,
|
||||
hidden: widget.hidden,
|
||||
};
|
||||
}
|
||||
if (hidden) {
|
||||
widget.type = "hidden";
|
||||
widget.hidden = true;
|
||||
widget.computeSize = () => [0, -4];
|
||||
return;
|
||||
}
|
||||
widget.type = widget._dh3lvOriginal.type;
|
||||
widget.hidden = !!widget._dh3lvOriginal.hidden;
|
||||
widget.computeSize = widget._dh3lvOriginal.computeSize;
|
||||
}
|
||||
|
||||
function applyVisibility(node) {
|
||||
const state = readState(node);
|
||||
for (const group of GROUPS) {
|
||||
for (const name of group.widgets) {
|
||||
const widget = findWidget(node, name);
|
||||
if (!widget || widget.name === DOM_WIDGET_NAME) continue;
|
||||
setWidgetHidden(widget, !state[group.id]);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function resizeNode(node) {
|
||||
requestAnimationFrame(() => {
|
||||
const size = node.computeSize?.();
|
||||
if (Array.isArray(size)) {
|
||||
node.size[0] = Math.max(MIN_WIDTH, size[0] || 0, node.size?.[0] || 0);
|
||||
node.size[1] = Math.max(MIN_HEIGHT, size[1] || 0);
|
||||
}
|
||||
node.setDirtyCanvas?.(true, true);
|
||||
});
|
||||
}
|
||||
|
||||
function renderToolbar(node) {
|
||||
const ui = node._dh3lvUI;
|
||||
if (!ui) return;
|
||||
const state = readState(node);
|
||||
ui.list.innerHTML = "";
|
||||
for (const group of GROUPS) {
|
||||
const button = document.createElement("button");
|
||||
button.type = "button";
|
||||
button.className = "dh3lv-chip";
|
||||
button.dataset.open = state[group.id] ? "true" : "false";
|
||||
button.textContent = state[group.id] ? `Hide ${group.label}` : `Show ${group.label}`;
|
||||
|
||||
const count = document.createElement("span");
|
||||
count.className = "dh3lv-count";
|
||||
count.textContent = String(group.widgets.filter((name) => findWidget(node, name)).length);
|
||||
button.appendChild(count);
|
||||
|
||||
button.addEventListener("click", () => {
|
||||
const next = readState(node);
|
||||
next[group.id] = !next[group.id];
|
||||
writeState(node, next);
|
||||
applyVisibility(node);
|
||||
renderToolbar(node);
|
||||
resizeNode(node);
|
||||
});
|
||||
ui.list.appendChild(button);
|
||||
}
|
||||
}
|
||||
|
||||
function setAll(node, open) {
|
||||
const next = {};
|
||||
for (const group of GROUPS) next[group.id] = !!open;
|
||||
writeState(node, next);
|
||||
applyVisibility(node);
|
||||
renderToolbar(node);
|
||||
resizeNode(node);
|
||||
}
|
||||
|
||||
function setupNode(node) {
|
||||
if (node._dh3lvUI) return;
|
||||
injectCSS();
|
||||
writeState(node, readState(node));
|
||||
|
||||
const root = document.createElement("div");
|
||||
root.className = "dh3lv-sections";
|
||||
|
||||
const head = document.createElement("div");
|
||||
head.className = "dh3lv-sections-head";
|
||||
|
||||
const title = document.createElement("div");
|
||||
title.className = "dh3lv-sections-title";
|
||||
title.textContent = "Sections";
|
||||
|
||||
const actions = document.createElement("div");
|
||||
actions.className = "dh3lv-sections-actions";
|
||||
|
||||
const expandAll = document.createElement("button");
|
||||
expandAll.type = "button";
|
||||
expandAll.className = "dh3lv-action";
|
||||
expandAll.textContent = "Expand All";
|
||||
expandAll.addEventListener("click", () => setAll(node, true));
|
||||
|
||||
const collapseAll = document.createElement("button");
|
||||
collapseAll.type = "button";
|
||||
collapseAll.className = "dh3lv-action";
|
||||
collapseAll.textContent = "Collapse Extras";
|
||||
collapseAll.addEventListener("click", () => {
|
||||
const next = defaultState();
|
||||
writeState(node, next);
|
||||
applyVisibility(node);
|
||||
renderToolbar(node);
|
||||
resizeNode(node);
|
||||
});
|
||||
|
||||
actions.append(expandAll, collapseAll);
|
||||
head.append(title, actions);
|
||||
|
||||
const list = document.createElement("div");
|
||||
list.className = "dh3lv-sections-list";
|
||||
root.append(head, list);
|
||||
root.addEventListener("pointerdown", stopCanvasEvent);
|
||||
root.addEventListener("mousedown", stopCanvasEvent);
|
||||
root.addEventListener("click", stopCanvasEvent);
|
||||
root.addEventListener("dblclick", stopCanvasEvent);
|
||||
root.addEventListener("keydown", stopCanvasKeyboard, true);
|
||||
|
||||
node._dh3lvUI = { root, list };
|
||||
const widget = node.addDOMWidget(DOM_WIDGET_NAME, "custom", root, {
|
||||
getValue: () => null,
|
||||
setValue: () => {},
|
||||
serialize: false,
|
||||
getMinHeight: () => 52,
|
||||
hideOnZoom: false,
|
||||
});
|
||||
applyAdaptiveCanvasOnly(widget);
|
||||
const widgets = node.widgets || [];
|
||||
const index = widgets.indexOf(widget);
|
||||
if (index > 0) {
|
||||
widgets.splice(index, 1);
|
||||
widgets.unshift(widget);
|
||||
}
|
||||
|
||||
applyVisibility(node);
|
||||
renderToolbar(node);
|
||||
resizeNode(node);
|
||||
}
|
||||
|
||||
// The Dumas Long Videos node now wraps the upstream MiniMax-H3-Longvideos
|
||||
// sampler directly. The old local frontend grouped Dumas-specific widgets that
|
||||
// no longer exist on the upstream node, so this extension intentionally does
|
||||
// nothing.
|
||||
app.registerExtension({
|
||||
name: "Dumas.H3LongVideosSections",
|
||||
async beforeRegisterNodeDef(nodeType, nodeData) {
|
||||
if (nodeData?.name !== COMFY_CLASS) return;
|
||||
|
||||
const originalOnNodeCreated = nodeType.prototype.onNodeCreated;
|
||||
nodeType.prototype.onNodeCreated = function onNodeCreated() {
|
||||
const result = originalOnNodeCreated?.apply(this, arguments);
|
||||
setupNode(this);
|
||||
return result;
|
||||
};
|
||||
|
||||
const originalConfigure = nodeType.prototype.onConfigure;
|
||||
nodeType.prototype.onConfigure = function onConfigure() {
|
||||
const result = originalConfigure?.apply(this, arguments);
|
||||
setupNode(this);
|
||||
writeState(this, readState(this));
|
||||
applyVisibility(this);
|
||||
renderToolbar(this);
|
||||
resizeNode(this);
|
||||
return result;
|
||||
};
|
||||
|
||||
const originalSerialize = nodeType.prototype.onSerialize;
|
||||
nodeType.prototype.onSerialize = function onSerialize(o) {
|
||||
writeState(this, readState(this));
|
||||
const result = originalSerialize?.apply(this, arguments);
|
||||
if (o && this.properties?.[STATE_PROPERTY]) {
|
||||
o.properties = o.properties || {};
|
||||
o.properties[STATE_PROPERTY] = this.properties[STATE_PROPERTY];
|
||||
}
|
||||
return result;
|
||||
};
|
||||
},
|
||||
name: "Dumas.H3LongVideos.UpstreamWrapper",
|
||||
});
|
||||
|
||||
@@ -26,6 +26,15 @@ const EXPECTED_OUTPUTS = [
|
||||
"original_ref_7",
|
||||
"original_ref_8",
|
||||
"original_ref_9",
|
||||
"compiled_ref_description_1",
|
||||
"compiled_ref_description_2",
|
||||
"compiled_ref_description_3",
|
||||
"compiled_ref_description_4",
|
||||
"compiled_ref_description_5",
|
||||
"compiled_ref_description_6",
|
||||
"compiled_ref_description_7",
|
||||
"compiled_ref_description_8",
|
||||
"compiled_ref_description_9",
|
||||
];
|
||||
const EXPECTED_NAMES = new Set(EXPECTED_OUTPUTS);
|
||||
|
||||
|
||||
@@ -11,35 +11,62 @@ class DumasH3BeatPromptTests(unittest.TestCase):
|
||||
state = self.module._parse_beat_prompt_state("not json")
|
||||
self.assertEqual(
|
||||
state,
|
||||
{"beats": [{"text": "Describe this beat."}]},
|
||||
{
|
||||
"scene": "",
|
||||
"character_sheet": "",
|
||||
"beats": [{"text": "Describe this beat."}],
|
||||
},
|
||||
)
|
||||
|
||||
def test_assemble_prompt_joins_beats_with_blank_lines(self):
|
||||
def test_assemble_prompt_outputs_upstream_sections(self):
|
||||
prompt = self.module._assemble_beat_prompt(
|
||||
{
|
||||
"scene": "A rainy kitchen at night.",
|
||||
"character_sheet": "Maya: 27, she, red jacket, silver hair.",
|
||||
"beats": [
|
||||
{"text": "A woman enters the room."},
|
||||
{"text": "wardrobe: Maya = red jacket\nShe sits at the table."},
|
||||
{"text": "Maya enters the room."},
|
||||
{"text": "remove: red jacket\nadd: white shirt underneath\nShe sits at the table."},
|
||||
{"text": " "},
|
||||
{"text": "music: low synth pulse"},
|
||||
]
|
||||
}
|
||||
)
|
||||
self.assertEqual(
|
||||
prompt,
|
||||
(
|
||||
"A woman enters the room.\n\n"
|
||||
"wardrobe: Maya = red jacket\nShe sits at the table.\n\n"
|
||||
"music: low synth pulse"
|
||||
"A rainy kitchen at night.\n\n"
|
||||
"Maya: 27, she, red jacket, silver hair.\n\n"
|
||||
"Maya enters the room.\n\n"
|
||||
"remove: red jacket\nadd: white shirt underneath\nShe sits at the table."
|
||||
),
|
||||
)
|
||||
|
||||
def test_assemble_prompt_strips_old_dumas_directives(self):
|
||||
prompt = self.module._assemble_beat_prompt(
|
||||
{
|
||||
"beats": [
|
||||
{
|
||||
"text": (
|
||||
"seconds: 8\n"
|
||||
"continuity: hard cut\n"
|
||||
"ref_mode: every shot\n"
|
||||
"soundscape: soft rain\n"
|
||||
"music: low synth\n"
|
||||
"Maya opens the cupboard.\n"
|
||||
"remove: red jacket"
|
||||
)
|
||||
},
|
||||
]
|
||||
}
|
||||
)
|
||||
|
||||
self.assertEqual(prompt, "Maya opens the cupboard.\nremove: red jacket")
|
||||
|
||||
def test_node_build_prompt_uses_hidden_state(self):
|
||||
node = self.module.DumasH3BeatPromptNode()
|
||||
result = node.build_prompt(
|
||||
'{"beats":[{"text":"Beat one"},{"text":"Beat two"}]}'
|
||||
'{"scene":"Scene","character_sheet":"Maya: 27, she","beats":[{"text":"Beat one"},{"text":"Beat two"}]}'
|
||||
)
|
||||
self.assertEqual(result, ("Beat one\n\nBeat two",))
|
||||
self.assertEqual(result, ("Scene\n\nMaya: 27, she\n\nBeat one\n\nBeat two",))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
+78
-1629
File diff suppressed because it is too large
Load Diff
@@ -505,6 +505,11 @@ class DumasImageNodeTests(unittest.TestCase):
|
||||
self.assertIs(result[15], dave_image)
|
||||
self.assertIs(result[16], cafe_image)
|
||||
self.assertIsNone(result[17])
|
||||
self.assertIn("<Picture 1> Dave", result[24])
|
||||
self.assertNotIn("<Picture 2> Coffee Shop", result[24])
|
||||
self.assertIn("<Picture 2> Coffee Shop", result[25])
|
||||
self.assertIn("Location context for <Picture 2> Coffee Shop", result[25])
|
||||
self.assertEqual(result[26], "")
|
||||
|
||||
def test_h3_prompt_curator_renumbers_explicit_reference_tags(self):
|
||||
node = self.image_nodes.DumasH3PromptCuratorNode()
|
||||
@@ -642,10 +647,14 @@ class DumasImageNodeTests(unittest.TestCase):
|
||||
|
||||
def test_h3_prompt_curator_uses_documented_reference_limits(self):
|
||||
node = self.image_nodes.DumasH3PromptCuratorNode()
|
||||
self.assertEqual(len(node.RETURN_TYPES), 24)
|
||||
self.assertEqual(len(node.RETURN_TYPES), 33)
|
||||
self.assertEqual(node.RETURN_NAMES[1:10], tuple(f"ref_image_{i}" for i in range(1, 10)))
|
||||
self.assertEqual(node.RETURN_NAMES[12:15], ("anchor", "sounds", "bgm"))
|
||||
self.assertEqual(node.RETURN_NAMES[15:24], tuple(f"original_ref_{i}" for i in range(1, 10)))
|
||||
self.assertEqual(
|
||||
node.RETURN_NAMES[24:33],
|
||||
tuple(f"compiled_ref_description_{i}" for i in range(1, 10)),
|
||||
)
|
||||
|
||||
def test_normalize_reference_upgrades_generic_summary_with_socket_picture_id(self):
|
||||
image = FakeTensorBatch()
|
||||
|
||||
Reference in New Issue
Block a user