Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
bfaa5fa733 | ||
|
|
9923a3e417 | ||
|
|
ea6fae7e56 | ||
|
|
72f9b07ed8 | ||
|
|
8d4232a142 | ||
|
|
d68d04254f |
@@ -0,0 +1,53 @@
|
|||||||
|
H3-LongVideos — Licence
|
||||||
|
Copyright (c) 2026 Smite79. All rights reserved.
|
||||||
|
|
||||||
|
This licence applies to every version published on or after 2026-09-07.
|
||||||
|
|
||||||
|
WHAT YOU MAY DO
|
||||||
|
|
||||||
|
1. Download and use this software, in unmodified or modified form, for your
|
||||||
|
own purposes, personal or commercial. Rendering with it, and whatever you
|
||||||
|
render with it, is yours and is not covered by this licence.
|
||||||
|
|
||||||
|
2. Modify your own copy.
|
||||||
|
|
||||||
|
3. Submit changes back to the original project.
|
||||||
|
|
||||||
|
WHAT YOU MAY NOT DO WITHOUT WRITTEN PERMISSION
|
||||||
|
|
||||||
|
4. Redistribute this software, in whole or in part, modified or unmodified.
|
||||||
|
That includes publishing it to any repository, registry, model hub, node
|
||||||
|
manager, marketplace, or mirror; bundling it inside another package,
|
||||||
|
product, image, or installer; and hosting it as a service.
|
||||||
|
|
||||||
|
5. Remove, alter, or obscure the copyright notice above, this licence, or the
|
||||||
|
attribution in the source files — including where a permitted redistribution
|
||||||
|
has been agreed.
|
||||||
|
|
||||||
|
6. Represent this software, or a derivative of it, as your own work.
|
||||||
|
|
||||||
|
ASKING
|
||||||
|
|
||||||
|
Permission for anything under 4 is granted case by case and is usually given
|
||||||
|
for things like inclusion in a node manager. Ask via the project's GitHub
|
||||||
|
issues at https://github.com/Smite79/MiniMax-H3-LongVideos.
|
||||||
|
|
||||||
|
EARLIER VERSIONS
|
||||||
|
|
||||||
|
Versions published before 2026-09-07 were released under Apache License 2.0.
|
||||||
|
That grant is irrevocable for those versions: copies obtained under it stay
|
||||||
|
under it, and this licence does not and cannot withdraw it retroactively. It
|
||||||
|
governs this version and every version after it.
|
||||||
|
|
||||||
|
Apache 2.0 also required attribution, so a copy of an earlier version
|
||||||
|
republished with the copyright notice stripped was already in breach of the
|
||||||
|
licence it was taken under.
|
||||||
|
|
||||||
|
NO WARRANTY
|
||||||
|
|
||||||
|
This software is provided "as is", without warranty of any kind, express or
|
||||||
|
implied, including but not limited to the warranties of merchantability,
|
||||||
|
fitness for a particular purpose, and non-infringement. In no event shall the
|
||||||
|
copyright holder be liable for any claim, damages, or other liability, whether
|
||||||
|
in an action of contract, tort, or otherwise, arising from, out of, or in
|
||||||
|
connection with the software or the use or other dealings in the software.
|
||||||
@@ -33,35 +33,32 @@
|
|||||||
- Outputs: `plan`, `image1`..`image9`, `connected_images`
|
- Outputs: `plan`, `image1`..`image9`, `connected_images`
|
||||||
- Reads back the nine optional images for a selected MiniMax H3 plan scene, for example by connecting the current `clip_index`.
|
- Reads back the nine optional images for a selected MiniMax H3 plan scene, for example by connecting the current `clip_index`.
|
||||||
|
|
||||||
- `Dumas H3 Long Videos (FL2VA + REF2VA)`
|
- `Dumas H3 Long Videos`
|
||||||
- Inputs: H3 model stack, prompt socket, optional `first_frame`, optional `ref_1`..`ref_9`, plus the upstream long-video control surface for pacing, continuity, audio, overlays, and guards
|
- Inputs/outputs: the current upstream `MiniMax-H3-Longvideos` sampler surface, exposed under the existing `DumasH3LongVideos` key for saved Dumas workflows.
|
||||||
- Outputs: `images`, `audio`, `info`, `script`, `frames_per_shot`, `total_frames`, `shots`, `video_seconds`, `fps`, `fps_int`, `latent`, `soundscape`
|
- The local Dumas prompt-engineering fork has been removed from this node. Long Videos now wraps the upstream sampler/engine/runtime/audio/conditioning/shot-plan modules directly so it can track the source project again.
|
||||||
- First-pass Dumas port of the `MiniMax-H3-Longvideos` sampler, brought in as a local starting point for long-form H3 chaining work.
|
- Upstream compatibility keys `H3LongVideos`, `H3LongVideosFL2VA`, `H3LongVideosV1`, and `H3LongVideosREF2VA` are also registered to the same class.
|
||||||
- Keeps the upstream split-beats / handoff / ref-routing behavior close to source so future Dumas-specific improvements can be compared against a known baseline.
|
- The old Dumas browser widget grouping script is disabled for this node because it targeted controls that no longer exist on the upstream sampler.
|
||||||
- Full user guide: [`H3_LONG_VIDEOS_GUIDE.md`](./H3_LONG_VIDEOS_GUIDE.md)
|
- `handoff_frames` extends the upstream last-frame handoff: `1` keeps the current single keyframe behavior; higher values keep that final-frame keyframe and add earlier tail frames from the previous shot as claimed reference context for the next beat.
|
||||||
- Only the canonical `DumasH3LongVideos` node key is exposed now; the older FL2VA/REF2VA alias entries are no longer duplicated in the Add Node menu.
|
- Upstream license text is included in [`H3_LONGVIDEOS_UPSTREAM_LICENSE.txt`](./H3_LONGVIDEOS_UPSTREAM_LICENSE.txt).
|
||||||
- Prompt `<Picture N>` tags now map to the actual ref socket numbers you wire, even with gaps such as only `ref_2` and `ref_7` connected.
|
|
||||||
- Character refs now contribute appearance and wardrobe context from the same structured object, while location refs contribute environment context from theirs.
|
|
||||||
- The default ref2v bias is now stronger: `ref_mode` defaults to `auto ref2v` so untagged prompts condition every shot instead of only shot 1, and `ref_noise_aug` defaults to `0.95` rather than the upstream-literal `0.999`.
|
|
||||||
- `Dumas H3 Latent Upscale Params` provides the optional pre-decode latent refinement stage for the long-video node.
|
|
||||||
- Per-shot directives now support `continuity:`, `ref_mode:`, `ref_noise_aug:`, `anchor_add:`, `soundscape:`, and `music:` in addition to the existing timing and wardrobe directives.
|
|
||||||
|
|
||||||
- `Dumas H3 Latent Upscale Params`
|
- `Dumas H3 Latent Upscale Params`
|
||||||
- Inputs: `mode`, `model_name`, `method`, `width`, `height`, `device`, `precision`, `sampler_name`, `scheduler`, `steps`, `denoise`, `megapixels`, `tile_width`, `tile_height`, `overlap`, `fade_width`, `fade_height`, `overlap_mode`, `overlap_blend`, `tile_size_mode`, `grid_rows`, `grid_cols`, `spatial_w_overlap`, `spatial_h_overlap`, `min_tile_size`, `masked_area_noise`, `brightness_match`, `dynamic_fade`, `dynamic_fade_min`, `chunk_length`, `temporal_overlap`, `resize_conditioning`, `anchor_strength`
|
- Inputs: `mode`, `model_name`, `method`, `width`, `height`, `device`, `precision`, `sampler_name`, `scheduler`, `steps`, `denoise`, `megapixels`, `tile_width`, `tile_height`, `overlap`, `fade_width`, `fade_height`, `overlap_mode`, `overlap_blend`, `tile_size_mode`, `grid_rows`, `grid_cols`, `spatial_w_overlap`, `spatial_h_overlap`, `min_tile_size`, `masked_area_noise`, `brightness_match`, `dynamic_fade`, `dynamic_fade_min`, `chunk_length`, `temporal_overlap`, `resize_conditioning`, `anchor_strength`
|
||||||
- Output: `latent_upscale_param`
|
- Output: `latent_upscale_param`
|
||||||
- Bundles the optional latent-space upscaler settings used by `Dumas H3 Long Videos` before decode, so the main node can rebuild conditioning at the target size and run a short refinement pass with your chosen sampler, scheduler, step count, denoise, and the full upstream spatial split controls.
|
- Legacy helper from the abandoned Dumas Long Videos fork. The current upstream-backed `Dumas H3 Long Videos` node does not consume this socket; it uses the upstream latent-upscale controls on the Long Videos node itself.
|
||||||
|
|
||||||
- `Dumas H3 Beat Prompt`
|
- `Dumas H3 Beat Prompt`
|
||||||
- Inputs: authored through the custom front-end beat editor
|
- Inputs: authored through the custom front-end beat editor
|
||||||
- Output: `prompt`
|
- Output: `prompt`
|
||||||
- Builds one H3 prompt block per beat, with quick controls for per-shot timing, continuity, ref behavior, anchor additions, soundscape, and music while staying compatible with direct text editing.
|
- Builds an upstream-compatible Long Videos prompt: optional scene paragraph, optional character sheet, then one blank-line-separated textbox per beat.
|
||||||
|
- Per-beat helpers only emit upstream-supported state directives: `remove:` / `removed:` / `off:` and `add:` / `wear:` / `wearing:`.
|
||||||
|
- Old Dumas-only beat directives such as `seconds:`, `continuity:`, `ref_mode:`, `ref_noise_aug:`, `anchor_add:`, `soundscape:`, and `music:` are stripped from the generated prompt so they are not sent to the upstream node as visible text.
|
||||||
|
|
||||||
- `Dumas H3 Prompt Curator`
|
- `Dumas H3 Prompt Curator`
|
||||||
- Inputs: `action_prompt`, `anatomy_guard`, `subject_count_guard`, optional `anchor`, optional `soundscape`, optional `bgm`, optional `ref_1` through `ref_9`
|
- Inputs: `action_prompt`, `anatomy_guard`, `subject_count_guard`, optional `anchor`, optional `soundscape`, optional `bgm`, optional `ref_1` through `ref_9`
|
||||||
- Outputs: `prompt`, `ref_image_1` through `ref_image_9`, `reference_count`, `debug`, `anchor`, `sounds`, `bgm`, `original_ref_1` through `original_ref_9`
|
- Outputs: `prompt`, `ref_image_1` through `ref_image_9`, `reference_count`, `debug`, `anchor`, `sounds`, `bgm`, `original_ref_1` through `original_ref_9`, `compiled_ref_description_1` through `compiled_ref_description_9`
|
||||||
- Builds one standalone MiniMax H3 prompt from your final action text plus structured character/location references.
|
- Builds one standalone MiniMax H3 prompt from your final action text plus structured character/location references.
|
||||||
- The action text can mention references by character/location name, alias, `<Picture N>`, or `<refN>`. Only mentioned references are emitted, and the output images are compacted/renumbered so skipped inputs do not leave gaps.
|
- The action text can mention references by character/location name, alias, `<Picture N>`, or `<refN>`. Only mentioned references are emitted, and the output images are compacted/renumbered so skipped inputs do not leave gaps.
|
||||||
- Extra component outputs expose the cleaned anchor, sounds, BGM, and each selected original reference image in compacted order.
|
- Extra component outputs expose the cleaned anchor, sounds, BGM, and each selected original reference image plus its compiled reference description in compacted order.
|
||||||
- Adds curated reference context, anatomy guard text, optional subject-count guard text, anchor/style text, `overall_soundscape:` text, and `background_music:` text while respecting MiniMax H3's reference-generation shape: one prompt plus up to nine reference images.
|
- Adds curated reference context, anatomy guard text, optional subject-count guard text, anchor/style text, `overall_soundscape:` text, and `background_music:` text while respecting MiniMax H3's reference-generation shape: one prompt plus up to nine reference images.
|
||||||
|
|
||||||
- `Dumas H3 Shot Length`
|
- `Dumas H3 Shot Length`
|
||||||
|
|||||||
+55
-8
@@ -2,11 +2,46 @@ import json
|
|||||||
|
|
||||||
|
|
||||||
_DEFAULT_BEAT = "Describe this beat."
|
_DEFAULT_BEAT = "Describe this beat."
|
||||||
_DEFAULT_STATE = {"beats": [{"text": _DEFAULT_BEAT}]}
|
_DEFAULT_STATE = {"scene": "", "character_sheet": "", "beats": [{"text": _DEFAULT_BEAT}]}
|
||||||
|
_LEGACY_DIRECTIVE_PREFIXES = (
|
||||||
|
"seconds",
|
||||||
|
"duration",
|
||||||
|
"continuity",
|
||||||
|
"ref_mode",
|
||||||
|
"ref_noise_aug",
|
||||||
|
"anchor_add",
|
||||||
|
"overall_soundscape",
|
||||||
|
"soundscape",
|
||||||
|
"non_diegetic_music",
|
||||||
|
"music",
|
||||||
|
"wardrobe",
|
||||||
|
"enter",
|
||||||
|
"exit",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def _clone_default_state():
|
def _clone_default_state():
|
||||||
return {"beats": [{"text": _DEFAULT_BEAT}]}
|
return {
|
||||||
|
"scene": "",
|
||||||
|
"character_sheet": "",
|
||||||
|
"beats": [{"text": _DEFAULT_BEAT}],
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _strip_legacy_directives(text):
|
||||||
|
"""Remove directives from the abandoned Dumas Long Videos fork.
|
||||||
|
|
||||||
|
The upstream Long Videos node sends unknown field labels to the model as text,
|
||||||
|
so this builder strips the old managed controls rather than emitting prompts
|
||||||
|
that ask H3 to draw labels such as "seconds:" or "music:" in the frame.
|
||||||
|
"""
|
||||||
|
kept = []
|
||||||
|
for line in str(text or "").splitlines():
|
||||||
|
lowered = line.strip().lower()
|
||||||
|
if any(lowered.startswith(f"{name}:") for name in _LEGACY_DIRECTIVE_PREFIXES):
|
||||||
|
continue
|
||||||
|
kept.append(line)
|
||||||
|
return "\n".join(kept).strip()
|
||||||
|
|
||||||
|
|
||||||
def _parse_beat_prompt_state(value):
|
def _parse_beat_prompt_state(value):
|
||||||
@@ -21,6 +56,8 @@ def _parse_beat_prompt_state(value):
|
|||||||
except Exception:
|
except Exception:
|
||||||
return _clone_default_state()
|
return _clone_default_state()
|
||||||
|
|
||||||
|
scene = str(raw.get("scene") or "")
|
||||||
|
character_sheet = str(raw.get("character_sheet") or "")
|
||||||
beats = []
|
beats = []
|
||||||
for item in list(raw.get("beats") or []):
|
for item in list(raw.get("beats") or []):
|
||||||
if isinstance(item, dict):
|
if isinstance(item, dict):
|
||||||
@@ -30,15 +67,25 @@ def _parse_beat_prompt_state(value):
|
|||||||
beats.append({"text": text})
|
beats.append({"text": text})
|
||||||
|
|
||||||
if not beats:
|
if not beats:
|
||||||
return _clone_default_state()
|
beats = [{"text": _DEFAULT_BEAT}]
|
||||||
return {"beats": beats}
|
return {
|
||||||
|
"scene": scene,
|
||||||
|
"character_sheet": character_sheet,
|
||||||
|
"beats": beats,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
def _assemble_beat_prompt(state):
|
def _assemble_beat_prompt(state):
|
||||||
parsed = _parse_beat_prompt_state(state)
|
parsed = _parse_beat_prompt_state(state)
|
||||||
chunks = []
|
chunks = []
|
||||||
|
scene = str(parsed.get("scene") or "").strip()
|
||||||
|
if scene:
|
||||||
|
chunks.append(scene)
|
||||||
|
character_sheet = str(parsed.get("character_sheet") or "").strip()
|
||||||
|
if character_sheet:
|
||||||
|
chunks.append(character_sheet)
|
||||||
for beat in parsed["beats"]:
|
for beat in parsed["beats"]:
|
||||||
text = str(beat.get("text") or "").strip()
|
text = _strip_legacy_directives(beat.get("text") or "")
|
||||||
if text:
|
if text:
|
||||||
chunks.append(text)
|
chunks.append(text)
|
||||||
return "\n\n".join(chunks)
|
return "\n\n".join(chunks)
|
||||||
@@ -46,9 +93,9 @@ def _assemble_beat_prompt(state):
|
|||||||
|
|
||||||
class DumasH3BeatPromptNode:
|
class DumasH3BeatPromptNode:
|
||||||
DESCRIPTION = (
|
DESCRIPTION = (
|
||||||
"Build a MiniMax H3 prompt from one textbox per beat, with a front-end beat "
|
"Build an upstream MiniMax H3 Long Videos prompt: optional scene paragraph, "
|
||||||
"editor that can append directive examples and expose per-shot controls for "
|
"optional character sheet, then one blank-line-separated textbox per beat. "
|
||||||
"timing, continuity, ref behavior, anchor additions, soundscape, and music."
|
"Per-beat helpers only emit directives the upstream node understands."
|
||||||
)
|
)
|
||||||
RETURN_TYPES = ("STRING",)
|
RETURN_TYPES = ("STRING",)
|
||||||
RETURN_NAMES = ("prompt",)
|
RETURN_NAMES = ("prompt",)
|
||||||
|
|||||||
+23
-7916
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,319 @@
|
|||||||
|
# H3-LongVideos -- https://github.com/Smite79/MiniMax-H3-LongVideos
|
||||||
|
# Copyright (c) 2026 Smite79. All rights reserved.
|
||||||
|
# Redistribution, in whole or in part, requires written permission.
|
||||||
|
# This notice may not be removed or altered. See LICENSE.
|
||||||
|
"""Audio policy shared by conditioning and soundtrack assembly."""
|
||||||
|
|
||||||
|
from dataclasses import dataclass
|
||||||
|
|
||||||
|
import torch
|
||||||
|
import comfy.nested_tensor
|
||||||
|
from h3_runtime import temporal_shape
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class ShotAudio:
|
||||||
|
speech: bool
|
||||||
|
sounded: bool
|
||||||
|
voiced_only: bool
|
||||||
|
silence_enabled: bool
|
||||||
|
lead_seconds: float
|
||||||
|
latent_fps: int
|
||||||
|
# The tail. Everything after the line's expected end is pinned the way the lead
|
||||||
|
# pins everything before its start. All three default off, so a ShotAudio built
|
||||||
|
# the old way -- six positional arguments -- behaves exactly the old way.
|
||||||
|
line_seconds: float = 0.0 # planner's estimate of the spoken line
|
||||||
|
tail_seconds: float = 0.0 # free audio kept after that estimate; 0 = no tail pin
|
||||||
|
frame_count: int = 0 # the shot in pixel frames; the audio T comes from it
|
||||||
|
|
||||||
|
@property
|
||||||
|
def pinned(self):
|
||||||
|
return self.silence_enabled and not self.speech and not self.sounded
|
||||||
|
|
||||||
|
@property
|
||||||
|
def lead_frames(self):
|
||||||
|
if not self.speech or self.lead_seconds <= 0:
|
||||||
|
return 0
|
||||||
|
return round(self.lead_seconds * self.latent_fps)
|
||||||
|
|
||||||
|
@property
|
||||||
|
def tail_frames(self):
|
||||||
|
"""Audio latent frames pinned at the END of a dialogue shot.
|
||||||
|
|
||||||
|
The lead pins the opening so the line cannot start early; nothing pinned the
|
||||||
|
close, and a 2s line in a 9s shot left 7s of open branch in a shot the model
|
||||||
|
knows has a voice in it -- which is where speech carries on past the line, or
|
||||||
|
doubles it. The free span is lead + the line's estimate + tail_seconds; the
|
||||||
|
rest is held at encoded silence. The model chooses WHEN to speak, so the
|
||||||
|
margin is the author's dial: a clipped word costs more than a second of babble.
|
||||||
|
Off unless the shot speaks, the margin is set, and at least half a second would
|
||||||
|
be pinned -- a sliver is not worth the risk of clipping."""
|
||||||
|
if (not self.speech or self.tail_seconds <= 0 or self.line_seconds <= 0
|
||||||
|
or self.frame_count <= 0):
|
||||||
|
return 0
|
||||||
|
total = temporal_shape(self.frame_count)[2]
|
||||||
|
free = self.lead_frames + round((self.line_seconds + self.tail_seconds) * self.latent_fps)
|
||||||
|
tail = total - free
|
||||||
|
return tail if tail >= round(0.5 * self.latent_fps) else 0
|
||||||
|
|
||||||
|
|
||||||
|
_SILENT_UNIT = {"lat": None, "key": None}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# NOTHING HERE IS SYNTHESISED ANY MORE. Removed on the report: "Just get rid of
|
||||||
|
# the ambient sounds all together. They sound horrid. Go back to the model's
|
||||||
|
# natural audio."
|
||||||
|
#
|
||||||
|
# What was here built the soundtrack's non-vocal half out of shaped noise: a room
|
||||||
|
# tone from the scene's own wording (synth_ambient, over a table of recipes, with
|
||||||
|
# plain_bed under it as a floor) and 21 foley recipes laid into the shots whose
|
||||||
|
# audio branch is pinned to silence (foley_for, over _hits/_band/_room and later
|
||||||
|
# _contact/_flow/_creak, timed off the picture's own movement for footsteps).
|
||||||
|
#
|
||||||
|
# It went in because a shot pinned to silence cannot get audio from the model at
|
||||||
|
# all -- prompt text never opens a branch -- so auto_sound was writing sounds into
|
||||||
|
# prompts that could not make them. That reasoning was sound and the thing it built
|
||||||
|
# still did not pass: reported first as footsteps sounding like heartbeats and a
|
||||||
|
# bathroom that tapped, and then, once both of those measured clean, as horrid
|
||||||
|
# anyway. Synthesis that measures right and sounds wrong is the end of that road.
|
||||||
|
#
|
||||||
|
# So the audio is the model's, whole. H3 is a joint model and the audio branch is
|
||||||
|
# where its sound comes from; the prompt still describes what a shot sounds like,
|
||||||
|
# which is the half that was always doing the real work.
|
||||||
|
#
|
||||||
|
# The consequence, which is real and is reported in info rather than left to be
|
||||||
|
# discovered: a shot with no line and no sound you wrote is pinned to silence and
|
||||||
|
# is now SILENT. The pin is not a bug and is deliberately untouched -- it is what
|
||||||
|
# stops a free branch filling itself with babble and a face lip-syncing to it.
|
||||||
|
# Write the sound into the beat to open the branch on purpose, or wire a recording
|
||||||
|
# to ambient_audio, which is played under the finished track and conditions
|
||||||
|
# nothing. mix_ambient below is that path, and it is all that is left here.
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def _seamless_loop(x, n, sr):
|
||||||
|
"""[C, M] -> [C, n], looped with a crossfade so the join does not click.
|
||||||
|
|
||||||
|
Plain tiling puts a discontinuity at every repeat, once per loop length. In a
|
||||||
|
bed that is meant to sit under everything unnoticed, a regular click is the one
|
||||||
|
thing that gets noticed -- the same objection that made the silence latent
|
||||||
|
ping-pong its interior rather than tile it. Here the material is real audio
|
||||||
|
being PLAYED rather than a latent being conditioned on, so it cannot be
|
||||||
|
reversed: a room tone read backwards is fine, but footsteps are not. Crossfade
|
||||||
|
instead, which works on both."""
|
||||||
|
m = int(x.shape[-1])
|
||||||
|
if m <= 0:
|
||||||
|
return None
|
||||||
|
if m >= n:
|
||||||
|
return x[..., :n]
|
||||||
|
fade = min(int(0.25 * sr), m // 4)
|
||||||
|
if fade < 1:
|
||||||
|
reps = -(-n // m)
|
||||||
|
return x.repeat(1, reps)[..., :n]
|
||||||
|
# OVERLAP-ADD the tail onto the head, and shorten the unit by the overlap. The
|
||||||
|
# unit then runs x[m-fade] .. x[m-fade-1], so tiling it steps between samples
|
||||||
|
# that were adjacent in the source and there is no discontinuity anywhere.
|
||||||
|
#
|
||||||
|
# Measured, because the obvious construction is wrong: appending the crossfade
|
||||||
|
# to the END of a full-length unit leaves it finishing on x[fade-1] while the
|
||||||
|
# next repeat starts on x[0], which are not adjacent -- a 2s tone that does not
|
||||||
|
# divide evenly gave a 64x jump at the join, worse than plain tiling's 41x.
|
||||||
|
t = torch.linspace(0.0, 1.0, fade, dtype=x.dtype, device=x.device)
|
||||||
|
head = x[..., :fade] * t + x[..., m - fade:] * (1.0 - t)
|
||||||
|
unit = torch.cat([head, x[..., fade:m - fade]], dim=-1)
|
||||||
|
if int(unit.shape[-1]) < 1:
|
||||||
|
reps = -(-n // m)
|
||||||
|
return x.repeat(1, reps)[..., :n]
|
||||||
|
reps = -(-n // int(unit.shape[-1]))
|
||||||
|
return unit.repeat(1, reps)[..., :n]
|
||||||
|
|
||||||
|
|
||||||
|
def mix_ambient(audio, sr, bed, level):
|
||||||
|
"""Lay an ambient bed UNDER a finished soundtrack. -> (waveform, note).
|
||||||
|
|
||||||
|
The bed is PLAYED, not conditioned on: it is the file, at the level asked for,
|
||||||
|
under whatever the model generated. That is the whole reason to do it here
|
||||||
|
rather than in the sampler -- ambience needs no cooperation from a joint model,
|
||||||
|
has nothing to lip-sync to, and so cannot put a voice in a wordless shot. The
|
||||||
|
conditioning path can only steer the branch toward something bed-LIKE, and on a
|
||||||
|
shot with a line it competes with the line.
|
||||||
|
|
||||||
|
Defensive throughout, like the silence latent: any failure returns the audio
|
||||||
|
untouched with a note saying so, because a bed is a nicety and a render is not.
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
if audio is None or bed is None or float(level or 0.0) <= 0.0:
|
||||||
|
return audio, ""
|
||||||
|
w = bed.get("waveform") if isinstance(bed, dict) else None
|
||||||
|
if w is None or not int(getattr(w, "ndim", 0)):
|
||||||
|
return audio, ("ambient_audio is wired but carries no waveform, so nothing "
|
||||||
|
"was laid under the soundtrack")
|
||||||
|
w = w[0] if w.dim() == 3 else w # [B, C, M] -> [C, M]
|
||||||
|
if w.dim() != 2 or w.shape[-1] < 2:
|
||||||
|
return audio, ("ambient_audio is too short to loop, so nothing was laid "
|
||||||
|
"under the soundtrack")
|
||||||
|
w = w.detach().to(dtype=audio.dtype, device=audio.device)
|
||||||
|
b_sr = int((bed.get("sample_rate") if isinstance(bed, dict) else 0) or 0)
|
||||||
|
# RESAMPLE, or the bed plays at the wrong speed and pitch. Linear is coarse
|
||||||
|
# for music and inaudible on a room tone, which is what this input is for.
|
||||||
|
resampled = ""
|
||||||
|
if b_sr > 0 and b_sr != int(sr):
|
||||||
|
want = max(2, int(round(w.shape[-1] * float(sr) / float(b_sr))))
|
||||||
|
w = torch.nn.functional.interpolate(
|
||||||
|
w.unsqueeze(0), size=want, mode="linear", align_corners=False)[0]
|
||||||
|
resampled = f", resampled from {b_sr} Hz"
|
||||||
|
ch = int(audio.shape[1])
|
||||||
|
if int(w.shape[0]) != ch:
|
||||||
|
w = (w.mean(dim=0, keepdim=True).repeat(ch, 1) if int(w.shape[0]) > ch
|
||||||
|
else w[:1].repeat(ch, 1))
|
||||||
|
n = int(audio.shape[-1])
|
||||||
|
loop = _seamless_loop(w, n, int(sr))
|
||||||
|
if loop is None:
|
||||||
|
return audio, ""
|
||||||
|
out = audio + loop.unsqueeze(0) * float(level)
|
||||||
|
# NORMALISE rather than clip. Clipping a bed that pushed a loud line over
|
||||||
|
# the top distorts the LINE, which is the thing worth keeping.
|
||||||
|
peak = float(out.abs().max())
|
||||||
|
gain = ""
|
||||||
|
if peak > 1.0:
|
||||||
|
out = out / peak
|
||||||
|
gain = f", and the mix was scaled by {1.0 / peak:.2f} to stop it clipping"
|
||||||
|
secs = w.shape[-1] / float(sr)
|
||||||
|
return out, (f"an ambient bed was laid under the whole soundtrack at level "
|
||||||
|
f"{float(level):.2f} -- {secs:.1f}s of audio{resampled}, looped "
|
||||||
|
f"with a crossfade so the join does not click{gain}. It is your "
|
||||||
|
f"file, played under what the model generated: it conditions "
|
||||||
|
f"nothing, so it cannot put a voice in a wordless shot the way "
|
||||||
|
f"an inferred bed did. Shots pinned to silence keep their silent "
|
||||||
|
f"conditioning and get the bed on top, which is what makes a "
|
||||||
|
f"wordless shot sound like a room instead of a mute")
|
||||||
|
except Exception as exc:
|
||||||
|
return audio, (f"the ambient bed could not be mixed ({type(exc).__name__}), so "
|
||||||
|
f"the soundtrack is unchanged")
|
||||||
|
|
||||||
|
|
||||||
|
_SILENCE_STATUS = {"asked": 0, "applied": 0, "why": ""}
|
||||||
|
|
||||||
|
|
||||||
|
_SILENT_SECONDS = 2
|
||||||
|
|
||||||
|
|
||||||
|
_SILENT_EDGE = 4
|
||||||
|
|
||||||
|
|
||||||
|
def _silent_audio_latent(audio_vae, frame_count, fps):
|
||||||
|
"""A keyframe audio latent of actual SILENCE, or None if it cannot be made.
|
||||||
|
|
||||||
|
H3 is a JOINT model: the mouth follows the audio branch. On a shot with no
|
||||||
|
scripted line the branch is otherwise unconditioned, and an unconditioned audio
|
||||||
|
branch invents a voice -- which the picture then lip-syncs to. The lips-closed
|
||||||
|
sentence is arguing with a stream that has already decided someone is talking.
|
||||||
|
|
||||||
|
REBUILT 2026-09-05, from measurements against the real VAE rather than from
|
||||||
|
reasoning. The previous version encoded one second, kept a SINGLE interior
|
||||||
|
frame and repeated it, on the argument that silence is homogeneous. It is not,
|
||||||
|
in latent space: encoded silence has genuine frame-to-frame variation (delta
|
||||||
|
mean 0.002-0.004, max 0.021), and a repeated frame has a delta of exactly
|
||||||
|
0.000000. That is a flat signal no encoder produces, and a model handed
|
||||||
|
conditioning outside its own distribution has every reason to disregard it --
|
||||||
|
which is an audio branch back to inventing a voice, with the report saying
|
||||||
|
silence went on.
|
||||||
|
|
||||||
|
The fix that version was avoiding is real too: tiling the whole encoded second
|
||||||
|
end to end leaves a 25x spike at each join (0.554 against 0.022), once per
|
||||||
|
second, which is a metronome in the conditioning of a joint model.
|
||||||
|
|
||||||
|
So: encode two seconds, drop the padded ends, and PING-PONG the interior --
|
||||||
|
forward, reversed, forward. Every join repeats a frame, so there is no seam,
|
||||||
|
and the interior statistics are the encoder's own. Measured over a 9s shot:
|
||||||
|
|
||||||
|
one frame repeated peak 0.000686 delta mean 0.000000 max 0.000000
|
||||||
|
whole 2s tiled peak 0.000314 delta mean 0.017451 max 0.554715
|
||||||
|
interior ping-pong peak 0.000566 delta mean 0.002039 max 0.021159
|
||||||
|
|
||||||
|
where the encoder's own interior is mean 0.0021, max 0.0212. Decoded peak
|
||||||
|
0.000566 on a +/-1.0 scale is about -65 dBFS: silence.
|
||||||
|
|
||||||
|
Everything here stays defensive. Shapes are CHECKED against what the layout
|
||||||
|
expects rather than assumed, and any failure returns None so the shot falls
|
||||||
|
back to an unconditioned branch instead of breaking the render -- the caller
|
||||||
|
reports when that happens, so it is no longer a silent failure.
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
sr = int(getattr(audio_vae, "audio_sample_rate", 0) or 0)
|
||||||
|
if sr <= 0:
|
||||||
|
return None
|
||||||
|
_, _, want_t = temporal_shape(frame_count, fps)
|
||||||
|
if want_t <= 0:
|
||||||
|
return None
|
||||||
|
key = (id(audio_vae), sr)
|
||||||
|
block = _SILENT_UNIT.get("lat") if _SILENT_UNIT.get("key") == key else None
|
||||||
|
if block is None:
|
||||||
|
# CHANNELS LAST. comfy.sd.VAE.encode() does `pixel_samples.movedim(-1, 1)`
|
||||||
|
# before handing off, so the audio VAE -- which wants [B, 2, L] -- must be
|
||||||
|
# given [B, L, 2]. Passing [B, 2, L] raises inside the encoder, and an
|
||||||
|
# early version did exactly that: swallowed by the guard below, so the
|
||||||
|
# whole layer silently did nothing.
|
||||||
|
#
|
||||||
|
# Two seconds, encoded ONCE and cached. Encoding a full 15s shot instead
|
||||||
|
# cost a VAE pass big enough to OOM mid-render on a 16GB card, where the
|
||||||
|
# failure again degraded silently to no conditioning at all.
|
||||||
|
enc = audio_vae.encode(torch.zeros((1, sr * _SILENT_SECONDS, 2)))
|
||||||
|
if enc is None or enc.dim() != 4 or enc.shape[1] != 32:
|
||||||
|
return None
|
||||||
|
if enc.shape[-1] <= 2 * _SILENT_EDGE + 1:
|
||||||
|
return None
|
||||||
|
block = enc[..., _SILENT_EDGE:-_SILENT_EDGE].detach().to("cpu").clone()
|
||||||
|
_SILENT_UNIT["lat"] = block
|
||||||
|
_SILENT_UNIT["key"] = key
|
||||||
|
n = block.shape[-1]
|
||||||
|
if n < 1:
|
||||||
|
return None
|
||||||
|
# Forward, reversed, forward... Each join repeats a frame, so the seam that
|
||||||
|
# plain tiling leaves is gone while the interior variation is the encoder's.
|
||||||
|
pieces, have, i = [], 0, 0
|
||||||
|
while have < want_t:
|
||||||
|
piece = block if i % 2 == 0 else torch.flip(block, dims=[-1])
|
||||||
|
pieces.append(piece)
|
||||||
|
have += n
|
||||||
|
i += 1
|
||||||
|
out = torch.cat(pieces, dim=-1)[..., :want_t].clone()
|
||||||
|
if out.shape[-1] != want_t:
|
||||||
|
return None
|
||||||
|
return out
|
||||||
|
except Exception:
|
||||||
|
return None # never fail a render for a nicety
|
||||||
|
|
||||||
|
|
||||||
|
def _pin_audio_silence(latent, silence, lead_frames=None, tail_frames=0):
|
||||||
|
"""Start target audio at encoded silence and preserve the requested span(s).
|
||||||
|
|
||||||
|
lead_frames None pins the whole shot. Otherwise the first lead_frames and the
|
||||||
|
last tail_frames are held at silence and the span between is left to the model
|
||||||
|
-- that is where the line goes. The tail is clipped to what the lead leaves, so
|
||||||
|
the two can never overlap. Nothing pinned at all is a no-op, reported as False
|
||||||
|
so the caller does not count it as applied."""
|
||||||
|
try:
|
||||||
|
video, audio = latent["samples"].unbind()
|
||||||
|
silence = silence.to(device=audio.device, dtype=audio.dtype)
|
||||||
|
if silence.shape != audio.shape:
|
||||||
|
return False
|
||||||
|
audio_mask = torch.ones_like(audio[:, :1])
|
||||||
|
if lead_frames is None:
|
||||||
|
audio_mask.zero_()
|
||||||
|
else:
|
||||||
|
t = audio.shape[-1]
|
||||||
|
n = min(t, max(0, int(lead_frames)))
|
||||||
|
m = min(t - n, max(0, int(tail_frames or 0)))
|
||||||
|
if n <= 0 and m <= 0:
|
||||||
|
return False
|
||||||
|
if n > 0:
|
||||||
|
audio_mask[..., :n] = 0
|
||||||
|
if m > 0:
|
||||||
|
audio_mask[..., t - m:] = 0
|
||||||
|
latent["samples"] = comfy.nested_tensor.NestedTensor((video, silence))
|
||||||
|
latent["noise_mask"] = comfy.nested_tensor.NestedTensor(
|
||||||
|
(torch.ones_like(video[:, :1]), audio_mask))
|
||||||
|
return True
|
||||||
|
except Exception:
|
||||||
|
return False
|
||||||
@@ -0,0 +1,310 @@
|
|||||||
|
# H3-LongVideos -- https://github.com/Smite79/MiniMax-H3-LongVideos
|
||||||
|
# Copyright (c) 2026 Smite79. All rights reserved.
|
||||||
|
# Redistribution, in whole or in part, requires written permission.
|
||||||
|
# This notice may not be removed or altered. See LICENSE.
|
||||||
|
"""Decisions about which pictures may condition a shot."""
|
||||||
|
|
||||||
|
import torch
|
||||||
|
import node_helpers
|
||||||
|
from h3_runtime import (H3_FPS, AUDIO_LATENT_FPS, _empty_av_latent, _resize, ref_image_canvas,
|
||||||
|
frame_levels)
|
||||||
|
from h3_audio import _SILENCE_STATUS, _silent_audio_latent, _pin_audio_silence
|
||||||
|
|
||||||
|
|
||||||
|
def may_carry_room(previous_cast, current_cast, tagged_names):
|
||||||
|
"""A previous frame is safe as a reference only when it adds no subject."""
|
||||||
|
previous = [name for name in (previous_cast or ()) if name]
|
||||||
|
current = set(current_cast or ())
|
||||||
|
tagged = set(tagged_names or ())
|
||||||
|
return bool(previous) and all(name in current for name in previous) \
|
||||||
|
and not any(name in tagged for name in previous)
|
||||||
|
|
||||||
|
|
||||||
|
def may_carry_frame(previous_cast, current_cast, tagged_names):
|
||||||
|
"""A previous frame of the SAME room is safe as a reference claimed with everyone in it.
|
||||||
|
|
||||||
|
Unlike may_carry_room, somebody this shot does not describe may be in it: the claim
|
||||||
|
names them, and they are still in that room. Refused only for an empty frame, or
|
||||||
|
one holding somebody whose own portrait also rides this shot -- two pictures of one
|
||||||
|
person is how a second one gets drawn."""
|
||||||
|
previous = [name for name in (previous_cast or ()) if name]
|
||||||
|
current = set(current_cast or ())
|
||||||
|
tagged = set(tagged_names or ())
|
||||||
|
return bool(previous) and not any(name in tagged and name in current
|
||||||
|
for name in previous)
|
||||||
|
|
||||||
|
|
||||||
|
def recoverable_subject(cast, tagged_names, returning_names, captured):
|
||||||
|
"""Return the sole safe recovered subject, or an empty string."""
|
||||||
|
people = [name for name in (cast or ()) if name]
|
||||||
|
if len(people) != 1:
|
||||||
|
return ""
|
||||||
|
name = people[0]
|
||||||
|
return name if (name not in set(tagged_names or ())
|
||||||
|
and name in set(returning_names or ())
|
||||||
|
and captured.get(name) is not None) else ""
|
||||||
|
|
||||||
|
|
||||||
|
KEYFRAME_SAFE_AUG = 0.99 # below this, a ref aug would soften the keyframe too
|
||||||
|
|
||||||
|
# What ONE boundary is allowed to claim it measured. Wider than any real per-pass drift,
|
||||||
|
# narrow enough that a bad frame -- a flash, a cut to black, a frame the model lost --
|
||||||
|
# cannot swing the estimate. The median across boundaries does the real rejecting.
|
||||||
|
LEVEL_GAIN_CAP = 0.12 # in log-gain, so +-12.7% of contrast
|
||||||
|
LEVEL_OFFSET_CAP = 0.05
|
||||||
|
# The within-shot term is believed only when boundaries AGREE on its sign, and even then
|
||||||
|
# only this far: within-shot change is often the author's (a light switched off), so it is
|
||||||
|
# the half of the signal that cannot be trusted on its own.
|
||||||
|
LEVEL_SHOT_GAIN_CAP = 0.015
|
||||||
|
LEVEL_SHOT_OFFSET_CAP = 0.010
|
||||||
|
LEVEL_AGREE = 2.0 / 3.0
|
||||||
|
LEVEL_MIN_OBS = 3
|
||||||
|
# What the correction may do to one handoff, whatever it measured. A cut should not carry
|
||||||
|
# a visible grade step: shot N's last frame reaches the video uncorrected while N+1 is
|
||||||
|
# sampled from a corrected keyframe, so an uncapped correction trades burn-in for a pop at
|
||||||
|
# every join -- the same class of complaint, differently shaped.
|
||||||
|
LEVEL_GAIN_LO, LEVEL_GAIN_HI = 0.80, 1.25
|
||||||
|
LEVEL_OFFSET_BOUND = 0.02
|
||||||
|
# Below this a frame is too flat for a contrast RATIO to mean anything.
|
||||||
|
LEVEL_MIN_SIGMA = 0.01
|
||||||
|
|
||||||
|
|
||||||
|
class HandoffLevels:
|
||||||
|
"""Takes the grade the chain adds to itself back out of the handoff.
|
||||||
|
|
||||||
|
THE MEASUREMENT, which is the whole reason this needs no scene list. At every
|
||||||
|
boundary the render holds two pictures that are SUPPOSED to be the same frame: K,
|
||||||
|
the handoff it gave the shot, and R, frame one of what came back -- the model's own
|
||||||
|
reproduction of K, from a keyframe labelled sigma 0.001. Nothing was asked to change
|
||||||
|
between them, so everything separating them is the chain's own doing and none of it
|
||||||
|
is the author's intent. That is the one difference in the loop that can be corrected
|
||||||
|
without guessing at anybody's lighting, and R costs nothing to look at: it is the
|
||||||
|
frame trim_seam throws away.
|
||||||
|
|
||||||
|
A beat that walks into a darker room moves K, and R follows it there. So the level is
|
||||||
|
never anchored, never compared to shot 1, and never compared to a target -- only K
|
||||||
|
against its own reproduction, boundary by boundary.
|
||||||
|
|
||||||
|
WHAT IT WILL NOT FIX. Clipping already baked into earlier shots, because the VAE
|
||||||
|
clamps every decode and headroom spent is gone. Softening, which is a different
|
||||||
|
measurement and a different cause. Anything spatial -- ghosting, local burn, identity
|
||||||
|
drift. A tone curve with a knee in it, since this is affine per channel; the residual
|
||||||
|
in the report is how that would show itself. The first boundary, which has nothing to
|
||||||
|
measure yet. And a deliberate monotone move -- a film that dims every single beat --
|
||||||
|
loses a bounded, reported fraction of itself."""
|
||||||
|
|
||||||
|
def __init__(self):
|
||||||
|
self._bg, self._bo = [], [] # per boundary: K -> R, the chain's own drift
|
||||||
|
self._sg, self._so = [], [] # per shot: R -> last frame, believed only on agreement
|
||||||
|
self.applied = [] # (gain, offset) actually used, for the report
|
||||||
|
|
||||||
|
def observe(self, given, repro, last=None, pre_up_last=None):
|
||||||
|
"""Record one boundary. given is the keyframe this shot got, repro is frame one
|
||||||
|
of what it produced, last is its final frame, pre_up_last the handoff it hands on.
|
||||||
|
|
||||||
|
last/pre_up_last are how the pre-upscale handoff and the post-upscale output are
|
||||||
|
put in the same frame of reference: their difference IS the pipeline's own offset,
|
||||||
|
measured on one frame that went through both, so it can be subtracted from the
|
||||||
|
K->R reading instead of being mistaken for drift. With latent_upscale off they are
|
||||||
|
the same frame and the term is zero."""
|
||||||
|
gm, gs = frame_levels(given)
|
||||||
|
rm, rs = frame_levels(repro)
|
||||||
|
if gm is None or rm is None:
|
||||||
|
return False
|
||||||
|
if float(gs.min()) < LEVEL_MIN_SIGMA or float(rs.min()) < LEVEL_MIN_SIGMA:
|
||||||
|
return False
|
||||||
|
ug = torch.zeros(3)
|
||||||
|
uo = torch.zeros(3)
|
||||||
|
lm, ls = frame_levels(last) if last is not None else (None, None)
|
||||||
|
if pre_up_last is not None and lm is not None:
|
||||||
|
pm, ps = frame_levels(pre_up_last)
|
||||||
|
if pm is not None and float(ps.min()) >= LEVEL_MIN_SIGMA:
|
||||||
|
ug = torch.log(ls / ps)
|
||||||
|
uo = lm - pm
|
||||||
|
self._bg.append((torch.log(rs / gs) - ug).clamp(-LEVEL_GAIN_CAP, LEVEL_GAIN_CAP))
|
||||||
|
self._bo.append((rm - gm - uo).clamp(-LEVEL_OFFSET_CAP, LEVEL_OFFSET_CAP))
|
||||||
|
if lm is not None and float(ls.min()) >= LEVEL_MIN_SIGMA:
|
||||||
|
self._sg.append(torch.log(ls / rs))
|
||||||
|
self._so.append(lm - rm)
|
||||||
|
return True
|
||||||
|
|
||||||
|
def _agreed(self, rows, cap):
|
||||||
|
"""The median of rows, but only per channel where at least LEVEL_AGREE of them
|
||||||
|
share its sign. A within-shot change the boundaries disagree about is content, not
|
||||||
|
drift, and content must not be corrected."""
|
||||||
|
out = torch.zeros(3)
|
||||||
|
if len(rows) < LEVEL_MIN_OBS:
|
||||||
|
return out
|
||||||
|
st = torch.stack(rows)
|
||||||
|
med = st.median(dim=0).values
|
||||||
|
agree = ((st * med.sign().unsqueeze(0)) > 0).float().mean(dim=0)
|
||||||
|
keep = agree >= LEVEL_AGREE
|
||||||
|
return torch.where(keep, med.clamp(-cap, cap), out)
|
||||||
|
|
||||||
|
def estimate(self):
|
||||||
|
"""(gain_log, offset) the chain is drifting by per boundary, per channel."""
|
||||||
|
if not self._bg:
|
||||||
|
return None, None
|
||||||
|
g = torch.stack(self._bg).median(dim=0).values + self._agreed(self._sg, LEVEL_SHOT_GAIN_CAP)
|
||||||
|
o = torch.stack(self._bo).median(dim=0).values + self._agreed(self._so, LEVEL_SHOT_OFFSET_CAP)
|
||||||
|
return g, o
|
||||||
|
|
||||||
|
def gains(self, strength):
|
||||||
|
"""(gain, offset) as 3-vectors, or (None, None) when there is nothing worth doing.
|
||||||
|
|
||||||
|
Separate from note() because more than one frame leaves a shot -- the handoff,
|
||||||
|
and any face captured for a return several shots later -- and they have to carry
|
||||||
|
the SAME grade. A recovered face arriving at a different exposure from the shot
|
||||||
|
around it would be a new bug of exactly the kind this is fixing."""
|
||||||
|
g, o = self.estimate()
|
||||||
|
if g is None or strength <= 0:
|
||||||
|
return None, None
|
||||||
|
gain = torch.exp(-float(strength) * g).clamp(LEVEL_GAIN_LO, LEVEL_GAIN_HI)
|
||||||
|
off = (-float(strength) * o).clamp(-LEVEL_OFFSET_BOUND, LEVEL_OFFSET_BOUND)
|
||||||
|
# The next thing this frame meets is an 8-bit quantisation, so a correction under
|
||||||
|
# 1/255 would be erased on the way there. Claiming it would be worse than silence.
|
||||||
|
if float((gain - 1.0).abs().max()) < 1e-3 and float(off.abs().max()) < 1.0 / 255.0:
|
||||||
|
return None, None
|
||||||
|
return gain, off
|
||||||
|
|
||||||
|
def note(self, gain, off):
|
||||||
|
"""Record what was applied, and say it in one clause."""
|
||||||
|
self.applied.append((gain.clone(), off.clone()))
|
||||||
|
return (f"gain {'/'.join(f'{float(v):.3f}' for v in gain)} "
|
||||||
|
f"level {'/'.join(f'{float(v):+.4f}' for v in off)}")
|
||||||
|
|
||||||
|
def _keyframe_latent(vae, hand_img):
|
||||||
|
"""The keyframe latent for this shot: an ENCODE of the previous shot's last frame.
|
||||||
|
|
||||||
|
This was briefly an optimisation -- pass the previous shot's own latent straight
|
||||||
|
through and skip a VAE round trip per boundary. It was wrong, and it degraded
|
||||||
|
every shot after the first.
|
||||||
|
|
||||||
|
A keyframe is ONE pixel frame, and H3's grid puts that at 5f -> TWO latent
|
||||||
|
frames. Slicing [:, :, -1:] off a finished shot hands over one. Worse, the video
|
||||||
|
VAE is causal: the last latent of a 72-frame sequence encodes its temporal
|
||||||
|
context, not a standalone opening frame, so even at the right count it does not
|
||||||
|
mean what a keyframe means. The spatial-size guard could not see either problem.
|
||||||
|
|
||||||
|
The round trip is real but it is one lossy step on a correctly formed anchor,
|
||||||
|
which beats a cheap malformed one."""
|
||||||
|
return vae.encode(hand_img)
|
||||||
|
|
||||||
|
|
||||||
|
def _build_ref_images(vae, images, gen_w, gen_h, mode="match"):
|
||||||
|
"""(tokenizer items, DiT blocks) for a list of reference IMAGE tensors.
|
||||||
|
|
||||||
|
The tokenizer labels each one `<Picture N>:` itself, in the order given here --
|
||||||
|
so the roster the prompt refers to is decided by input order, not by anything
|
||||||
|
written in the prompt."""
|
||||||
|
items, blocks = [], []
|
||||||
|
for img in images:
|
||||||
|
if img is None:
|
||||||
|
continue
|
||||||
|
h, w = int(img.shape[1]), int(img.shape[2])
|
||||||
|
tw, th = ref_image_canvas(w, h, gen_w, gen_h, mode)
|
||||||
|
resized = _resize(img[:1], tw, th, "disabled")
|
||||||
|
items.append({"type": "image", "data": resized})
|
||||||
|
blocks.append({"kind": "image", "latent_h": th // 16, "latent_w": tw // 16,
|
||||||
|
"latent": vae.encode(resized)})
|
||||||
|
return items, blocks
|
||||||
|
|
||||||
|
|
||||||
|
def build_conditioning(clip, vae, audio_vae, prompt, width, height, length,
|
||||||
|
handoff=None, refs=None,
|
||||||
|
ref_noise_aug=0.999, silent=False, ref_image_size="match",
|
||||||
|
handoff_as_ref=False, speech_lead_seconds=0.0, speech_tail_frames=0):
|
||||||
|
"""Encode prompt, identity references, keyframe, and audio constraints for a shot."""
|
||||||
|
latent, fc = _empty_av_latent(width, height, length, H3_FPS)
|
||||||
|
refs = [r for r in (refs or []) if r is not None]
|
||||||
|
|
||||||
|
hand_img = None
|
||||||
|
if handoff is not None:
|
||||||
|
hand_img = _resize(handoff[:1], width, height, "disabled")
|
||||||
|
|
||||||
|
# REFERENCES AND THE KEYFRAME RIDE TOGETHER. This is the arrangement the node
|
||||||
|
# had before I broke it, and the reason is in ComfyUI's own layout:
|
||||||
|
#
|
||||||
|
# model_base.py:2183-2191 cond_video_latents = keyframe latents THEN ref latents
|
||||||
|
# model.py PackedLayout emits keyframe "cond" segments THEN ref "ref_img" ones
|
||||||
|
#
|
||||||
|
# The two orders agree, so both channels coexist. A shot takes its references AND
|
||||||
|
# a real keyframe: the keyframe ANCHORS the first frame, which is what continuity
|
||||||
|
# needs, while a reference only supplies identity. They are not alternatives.
|
||||||
|
#
|
||||||
|
# I had read "<Picture 1>" as MEANING the first frame on fl2va, and rearranged the
|
||||||
|
# roster around that. It does not. Which image is the first frame is decided by
|
||||||
|
# resolved_frame_index in minimax_keyframes, not by a label's number -- the labels
|
||||||
|
# are only how the images are shown to the VLM, and what they have to line up with
|
||||||
|
# is the <Picture N> tags in the prompt.
|
||||||
|
#
|
||||||
|
# So references come FIRST and keep slots 1..N, which is what a sheet line's
|
||||||
|
# `Name: <Picture 1>, ...` points at, and the handoff is appended AFTER them where
|
||||||
|
# it disturbs no numbering. It has to be in the list at all because
|
||||||
|
# tokenize_with_weights is either/or: passing minimax_ref_items makes it ignore
|
||||||
|
# `images` outright, so leaving the handoff out means the VLM is never shown where
|
||||||
|
# the shot left off and re-imagines the scenery -- same place, new room.
|
||||||
|
keyframe_ok = ref_noise_aug is None or float(ref_noise_aug) >= KEYFRAME_SAFE_AUG
|
||||||
|
# One aug covers every visual condition row, references AND the keyframe. Below
|
||||||
|
# KEYFRAME_SAFE_AUG the keyframe latent would be noised and labelled at the wrong
|
||||||
|
# timestep, so the handoff stops being an anchor and rides as an extra reference
|
||||||
|
# instead: weaker continuity, but nothing pretending to anchor while carrying noise.
|
||||||
|
# ...or because the caller asked for it. A shot that introduces somebody already
|
||||||
|
# in position wants the room this picture carries and NOT the first frame it
|
||||||
|
# would force, and that is a demotion the aug knows nothing about.
|
||||||
|
carry_as_ref = bool(hand_img is not None
|
||||||
|
and (handoff_as_ref or (refs and not keyframe_ok)))
|
||||||
|
|
||||||
|
enc_refs = refs + ([hand_img] if carry_as_ref else [])
|
||||||
|
items, blocks = ([], [])
|
||||||
|
if enc_refs:
|
||||||
|
items, blocks = _build_ref_images(vae, enc_refs, width, height, ref_image_size)
|
||||||
|
if hand_img is not None and not carry_as_ref:
|
||||||
|
items = items + [{"type": "image", "data": hand_img}]
|
||||||
|
|
||||||
|
if items:
|
||||||
|
tokens = clip.tokenize(prompt, minimax_ref_items=items)
|
||||||
|
else:
|
||||||
|
tokens = clip.tokenize(prompt)
|
||||||
|
cond = clip.encode_from_tokens_scheduled(tokens)
|
||||||
|
|
||||||
|
vals = {}
|
||||||
|
if blocks:
|
||||||
|
vals["minimax_refs"] = blocks
|
||||||
|
# How CLEAN the references are shown. One aug covers every conditioning
|
||||||
|
# latent, keyframe included -- which is why softening references below
|
||||||
|
# KEYFRAME_SAFE_AUG would soften the anchor too.
|
||||||
|
if ref_noise_aug is not None:
|
||||||
|
vals["minimax_visual_cond_noise_aug"] = float(ref_noise_aug)
|
||||||
|
|
||||||
|
kfs = []
|
||||||
|
if hand_img is not None and not carry_as_ref:
|
||||||
|
kfs.append({"resolved_frame_index": 0,
|
||||||
|
"latent": _keyframe_latent(vae, hand_img)})
|
||||||
|
# Audio keyframes are extra conditioning rows in H3's PackedLayout. Pin the
|
||||||
|
# generated target stream instead, so the joint model also sees a quiet mouth.
|
||||||
|
# A dialogue shot pins its opening (the lead) and, past the line's estimated end,
|
||||||
|
# its close (the tail); the span between is the model's.
|
||||||
|
if silent or float(speech_lead_seconds or 0.0) > 0.0 or int(speech_tail_frames or 0) > 0:
|
||||||
|
_SILENCE_STATUS["asked"] += 1
|
||||||
|
if audio_vae is None:
|
||||||
|
_SILENCE_STATUS["why"] = "no audio VAE is wired to the node"
|
||||||
|
else:
|
||||||
|
sil = _silent_audio_latent(audio_vae, fc, H3_FPS)
|
||||||
|
if sil is None:
|
||||||
|
_SILENCE_STATUS["why"] = ("the audio VAE would not encode a silent "
|
||||||
|
"second -- the wrong VAE is on the "
|
||||||
|
"audio_vae input")
|
||||||
|
else:
|
||||||
|
lead = None if silent else round(float(speech_lead_seconds) *
|
||||||
|
AUDIO_LATENT_FPS)
|
||||||
|
tail = 0 if silent else int(speech_tail_frames or 0)
|
||||||
|
if _pin_audio_silence(latent, sil, lead, tail):
|
||||||
|
_SILENCE_STATUS["applied"] += 1
|
||||||
|
else:
|
||||||
|
_SILENCE_STATUS["why"] = "the silent latent did not match the shot"
|
||||||
|
if kfs:
|
||||||
|
vals["minimax_keyframes"] = kfs
|
||||||
|
if vals:
|
||||||
|
cond = node_helpers.conditioning_set_values(cond, vals)
|
||||||
|
return cond, latent, fc, carry_as_ref
|
||||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,739 @@
|
|||||||
|
# H3-LongVideos -- https://github.com/Smite79/MiniMax-H3-LongVideos
|
||||||
|
# Copyright (c) 2026 Smite79. All rights reserved.
|
||||||
|
# Redistribution, in whole or in part, requires written permission.
|
||||||
|
# This notice may not be removed or altered. See LICENSE.
|
||||||
|
"""Sampling, decoding, resizing, memory handling, and frame assembly."""
|
||||||
|
|
||||||
|
import math
|
||||||
|
|
||||||
|
import torch
|
||||||
|
import nodes
|
||||||
|
import comfy.utils
|
||||||
|
import comfy.sample
|
||||||
|
import comfy.samplers
|
||||||
|
import comfy.nested_tensor
|
||||||
|
import comfy.model_management as mm
|
||||||
|
import latent_preview
|
||||||
|
|
||||||
|
|
||||||
|
class FrameAccumulator:
|
||||||
|
"""Build the final frame tensor once, retaining overflow only when necessary."""
|
||||||
|
|
||||||
|
def __init__(self, capacity, dtype, store_on_cpu):
|
||||||
|
self.capacity = int(capacity)
|
||||||
|
self.dtype = dtype
|
||||||
|
self.store_on_cpu = bool(store_on_cpu)
|
||||||
|
self.tensor = None
|
||||||
|
self.used = 0
|
||||||
|
self.overflow = []
|
||||||
|
|
||||||
|
def add(self, frames):
|
||||||
|
count = int(frames.shape[0])
|
||||||
|
if self.tensor is None and count:
|
||||||
|
device = torch.device("cpu") if self.store_on_cpu else frames.device
|
||||||
|
self.tensor = torch.empty(
|
||||||
|
(max(count, self.capacity),) + tuple(frames.shape[1:]),
|
||||||
|
dtype=self.dtype, device=device)
|
||||||
|
if (not self.overflow and self.tensor is not None
|
||||||
|
and self.used + count <= self.tensor.shape[0]):
|
||||||
|
self.tensor[self.used:self.used + count].copy_(frames)
|
||||||
|
self.used += count
|
||||||
|
return
|
||||||
|
self.overflow.append(frames.to("cpu", self.dtype, copy=True)
|
||||||
|
if self.store_on_cpu else frames)
|
||||||
|
|
||||||
|
def release(self):
|
||||||
|
"""Drop every tensor held, now, rather than whenever the collector gets to it.
|
||||||
|
|
||||||
|
On an interrupt the render unwinds through frames the collector tears down in
|
||||||
|
its own order, and a large video buffer freed after the models it was sized
|
||||||
|
against have already gone is a free the allocator cannot explain. Deliberately
|
||||||
|
does NOT empty the cache: that is another CUDA call, and if the context is
|
||||||
|
already in a sticky error state it is one more thing to abort inside."""
|
||||||
|
self.tensor = None
|
||||||
|
self.overflow = []
|
||||||
|
self.used = 0
|
||||||
|
|
||||||
|
def finish(self):
|
||||||
|
if not self.overflow:
|
||||||
|
if self.tensor is None:
|
||||||
|
return torch.cat(self.overflow, dim=0)
|
||||||
|
if self.used == self.tensor.shape[0]:
|
||||||
|
out = self.tensor
|
||||||
|
else:
|
||||||
|
# COMPACT, never a slice. A slice of a larger buffer keeps the WHOLE
|
||||||
|
# buffer's storage alive, which is the retention this class exists to
|
||||||
|
# prevent -- and test_the_chain_is_never_held_twice measures exactly
|
||||||
|
# that, demanding no unused bytes behind the returned tensor.
|
||||||
|
#
|
||||||
|
# There is slack because the capacity is now an upper bound: it can no
|
||||||
|
# longer assume trim_seam drops a frame at every seam, since a shot that
|
||||||
|
# opens on no keyframe keeps its first frame. Over-allocating by at most
|
||||||
|
# one frame per seam and compacting once is the bounded cost. The
|
||||||
|
# alternative -- an exact guess that can be too small -- drops into the
|
||||||
|
# overflow list, which with cleanup_between_shots off holds every shot's
|
||||||
|
# decoded frames live on the GPU until the end of the run.
|
||||||
|
out = torch.empty((self.used,) + tuple(self.tensor.shape[1:]),
|
||||||
|
dtype=self.dtype, device=self.tensor.device)
|
||||||
|
out.copy_(self.tensor[:self.used])
|
||||||
|
self.tensor = None
|
||||||
|
return out
|
||||||
|
|
||||||
|
extra = sum(int(piece.shape[0]) for piece in self.overflow)
|
||||||
|
reference = self.tensor if self.tensor is not None else self.overflow[0]
|
||||||
|
out = torch.empty((self.used + extra,) + tuple(reference.shape[1:]),
|
||||||
|
dtype=self.dtype, device=reference.device)
|
||||||
|
if self.tensor is not None and self.used:
|
||||||
|
out[:self.used].copy_(self.tensor[:self.used])
|
||||||
|
at = self.used
|
||||||
|
while self.overflow:
|
||||||
|
piece = self.overflow.pop(0)
|
||||||
|
count = int(piece.shape[0])
|
||||||
|
out[at:at + count].copy_(piece)
|
||||||
|
at += count
|
||||||
|
self.tensor = None
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
H3_FPS = 24 # H3 renders 24 fps, always
|
||||||
|
|
||||||
|
|
||||||
|
AUDIO_LATENT_FPS = 40 # audio latent frames per second
|
||||||
|
|
||||||
|
|
||||||
|
AUTO_TILE_T = 8 # temporal chunk for a tiled decode
|
||||||
|
|
||||||
|
|
||||||
|
MAX_FRAMES = 362 # H3's own ceiling (~15s)
|
||||||
|
|
||||||
|
|
||||||
|
CANVAS_MULTIPLE = 32
|
||||||
|
|
||||||
|
|
||||||
|
REF_IMAGE_SHORT_EDGE = 2048
|
||||||
|
|
||||||
|
|
||||||
|
def align_frame_count(n):
|
||||||
|
"""Up to the next valid H3 frame count. The grid is 17k+5."""
|
||||||
|
n = max(5, int(n))
|
||||||
|
while n % 17 != 5:
|
||||||
|
n += 1
|
||||||
|
return min(n, MAX_FRAMES)
|
||||||
|
|
||||||
|
|
||||||
|
def video_latent_t(fc):
|
||||||
|
return 2 if fc <= 5 else ((fc - 5) // 17) * 5 + 2
|
||||||
|
|
||||||
|
|
||||||
|
def temporal_shape(length, fps=H3_FPS):
|
||||||
|
"""(frame count, video latent frames, audio latent frames) for a shot.
|
||||||
|
|
||||||
|
`fps` is accepted but deliberately IGNORED: the audio latent has to line up
|
||||||
|
with 24 fps video or the shot's sound is stretched against its picture."""
|
||||||
|
fc = align_frame_count(length)
|
||||||
|
return fc, video_latent_t(fc), round(fc / H3_FPS * AUDIO_LATENT_FPS)
|
||||||
|
|
||||||
|
|
||||||
|
def ref_image_canvas(w, h, gen_w, gen_h, mode="match"):
|
||||||
|
"""Pure: the (width, height) a reference image is encoded at.
|
||||||
|
|
||||||
|
'match' scales it (DOWN only, aspect kept) to the generation's pixel area, so a
|
||||||
|
reference costs about as much as one frame of the shot. 'max' goes to the
|
||||||
|
reference pipeline's 2048 short edge for the best identity fidelity, which on a
|
||||||
|
long chain is several times slower because the rows are re-attended every step
|
||||||
|
of every shot. Never upscales: a small reference stays small."""
|
||||||
|
w, h = max(1, int(w)), max(1, int(h))
|
||||||
|
if mode == "max":
|
||||||
|
scale = min(1.0, REF_IMAGE_SHORT_EDGE / min(w, h))
|
||||||
|
else:
|
||||||
|
scale = min(1.0, math.sqrt((int(gen_w) * int(gen_h)) / float(w * h)))
|
||||||
|
snap = lambda v: max(CANVAS_MULTIPLE, round(v * scale / CANVAS_MULTIPLE) * CANVAS_MULTIPLE)
|
||||||
|
return snap(w), snap(h)
|
||||||
|
|
||||||
|
|
||||||
|
def _resize(image, width, height, crop):
|
||||||
|
s = image[..., :3].movedim(-1, 1)
|
||||||
|
s = comfy.utils.common_upscale(s, width, height, "lanczos", crop)
|
||||||
|
return s.movedim(1, -1)
|
||||||
|
|
||||||
|
|
||||||
|
def _empty_av_latent(width, height, length, fps, batch_size=1):
|
||||||
|
fc, lt, at = temporal_shape(length, fps)
|
||||||
|
video = torch.zeros([batch_size, 24, lt, height // 16, width // 16], device=mm.intermediate_device())
|
||||||
|
audio = torch.zeros([batch_size, 32, 2, at], device=mm.intermediate_device())
|
||||||
|
return {"samples": comfy.nested_tensor.NestedTensor((video, audio))}, fc
|
||||||
|
|
||||||
|
|
||||||
|
def _auto_tile_t(n_latent_frames, requested=None):
|
||||||
|
"""Temporal tile for a tiled decode. An explicit value wins.
|
||||||
|
|
||||||
|
The decode_tile_frames widget is gone, so this is where the value comes from
|
||||||
|
now. It has to come from somewhere: ComfyUI's decode_tiled_3d defaults tile_t
|
||||||
|
to 999, i.e. SPATIAL tiles only, and expanding the whole clip's time axis at
|
||||||
|
once is the single largest allocation in a run. A "tiled" decode that keeps the
|
||||||
|
full temporal extent barely lowers the peak, so the OOM retry that switches
|
||||||
|
tiling on was, without this, retrying with almost the same footprint."""
|
||||||
|
if requested:
|
||||||
|
return int(requested)
|
||||||
|
n = int(n_latent_frames or 0)
|
||||||
|
return AUTO_TILE_T if n > AUTO_TILE_T else None
|
||||||
|
|
||||||
|
|
||||||
|
def _decode_video(vae, out_latent, tiled, free_first=None, tile_t=None, tile_xy=None,
|
||||||
|
keep=()):
|
||||||
|
"""Decode the video latent.
|
||||||
|
|
||||||
|
`free_first` is the diffusion model: sampling is finished, and the video VAE
|
||||||
|
needs the room for THIS decode -- the free runs immediately before it, not to
|
||||||
|
make room for the next shot. On a card where the DiT is most of the VRAM, the
|
||||||
|
decode does not fit until it goes.
|
||||||
|
|
||||||
|
`keep` is what must NOT be evicted on the way. It was `keep_loaded=[]`, which
|
||||||
|
unloaded every resident model -- including the video VAE, which ComfyUI then
|
||||||
|
reloaded three lines later to run the decode. An evict-and-reload of the thing
|
||||||
|
about to be used, once per shot, on every card. Peak VRAM is identical either
|
||||||
|
way, since the VAE has to be resident to decode; the round trip was pure cost.
|
||||||
|
|
||||||
|
memory_required is ASKED FOR HONESTLY, which it was not. It was 1e30, and
|
||||||
|
free_memory computes `memory_to_free = memory_required - get_free_memory(device)`
|
||||||
|
(model_management.py:887), so 1e30 means "unload everything not in keep_loaded",
|
||||||
|
every shot, in full -- skipping partially_unload entirely.
|
||||||
|
|
||||||
|
What that evicts is the DiT, three lines before the next shot needs it again. On
|
||||||
|
a machine whose RAM is already full of finished frames there is nowhere for it to
|
||||||
|
go but disk, so the reload is a read from the drive, once per shot. Reported as
|
||||||
|
thrashing that slows the preload, and it is exactly that: the same weights being
|
||||||
|
read back at every boundary.
|
||||||
|
|
||||||
|
The VAE knows what its own decode costs -- ComfyUI sizes it with
|
||||||
|
memory_used_decode and uses that number everywhere else. Asked for that instead,
|
||||||
|
a card with headroom frees NOTHING and the DiT simply stays. A card without
|
||||||
|
headroom frees what it needs and no more, which is what partially_unload is for.
|
||||||
|
1e30 remains the fallback for a VAE that cannot estimate itself."""
|
||||||
|
latent = out_latent["samples"]
|
||||||
|
if latent.is_nested:
|
||||||
|
latent = latent.unbind()[0]
|
||||||
|
if free_first is not None:
|
||||||
|
try:
|
||||||
|
mm.free_memory(_decode_headroom(vae, latent), mm.get_torch_device(),
|
||||||
|
keep_loaded=_resident(keep or (vae,)))
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
# A VAE THAT ALREADY TILES DOES NOT NEED TO BE ASKED TO, AND ASKING COSTS 3x.
|
||||||
|
#
|
||||||
|
# MiniMaxH3VideoVAE.decode_tiled is, in full:
|
||||||
|
#
|
||||||
|
# def decode_tiled(self, z, **kwargs):
|
||||||
|
# return self.decode(z)
|
||||||
|
#
|
||||||
|
# Every tile_t/overlap_t/tile_x/tile_y this function computes is discarded, so
|
||||||
|
# the tiling the widget promises is not happening here -- the model tiles
|
||||||
|
# internally either way (256px spatial, 17-frame temporal), which is why
|
||||||
|
# comfy/sd.py sets handles_tiling on it.
|
||||||
|
#
|
||||||
|
# What the detour costs is the OUTPUT BUFFER. comfy's VAE.decode preallocates
|
||||||
|
# ONE result at vae_output_dtype and hands it to the model as output_buffer=,
|
||||||
|
# and MiniMaxH3VideoVAE.decode_temporal writes finalized chunks straight into
|
||||||
|
# it. Going through decode_tiled instead reaches _decode_tiled_owned, which
|
||||||
|
# calls the model with output_buffer=None -- so decode_temporal allocates its
|
||||||
|
# own at torch.float32 -- and then makes an fp16 `copy=True` of that. Two
|
||||||
|
# buffers, the larger of them at double width:
|
||||||
|
#
|
||||||
|
# tiled : fp32 2.60GB + fp16 copy 1.30GB = 3.90GB per shot
|
||||||
|
# decode: one preallocated fp16 = 1.30GB per shot
|
||||||
|
#
|
||||||
|
# at 362 frames of 1056x608. Every shot, on the node's own default.
|
||||||
|
#
|
||||||
|
# So: when the VAE owns its tiling AND can be written into, the un-tiled call IS
|
||||||
|
# the tiled one, minus the copies. Anything else keeps the old path -- this is a
|
||||||
|
# detour around a detour, not a claim that tiling is useless.
|
||||||
|
_owns_tiling = bool(getattr(vae, "handles_tiling", False) and getattr(
|
||||||
|
getattr(vae, "first_stage_model", None), "comfy_has_chunked_io", False))
|
||||||
|
if tiled and _owns_tiling:
|
||||||
|
imgs = vae.decode(latent)
|
||||||
|
elif tiled:
|
||||||
|
# Temporal + spatial tiling. Without tile_t the VAE expands the WHOLE latent
|
||||||
|
# clip at once, which on a 243-frame 1344x768 shot is the single largest
|
||||||
|
# allocation in the run -- and on an unpruned checkpoint that is already
|
||||||
|
# streaming, it is what tips the card over. Decoding in temporal chunks
|
||||||
|
# trades a little speed for a much lower peak; None keeps ComfyUI's defaults.
|
||||||
|
args = {}
|
||||||
|
tile_t = _auto_tile_t(latent.shape[2] if latent.ndim >= 5 else 0, tile_t)
|
||||||
|
if tile_t:
|
||||||
|
args["tile_t"] = int(tile_t)
|
||||||
|
args["overlap_t"] = max(1, int(tile_t) // 8)
|
||||||
|
if tile_xy:
|
||||||
|
args["tile_x"] = int(tile_xy)
|
||||||
|
args["tile_y"] = int(tile_xy)
|
||||||
|
try:
|
||||||
|
imgs = vae.decode_tiled(latent, **args) if args else vae.decode_tiled(latent)
|
||||||
|
except TypeError:
|
||||||
|
imgs = vae.decode_tiled(latent) # older signature without tile_t
|
||||||
|
else:
|
||||||
|
imgs = vae.decode(latent)
|
||||||
|
if len(imgs.shape) == 5:
|
||||||
|
imgs = imgs.reshape(-1, imgs.shape[-3], imgs.shape[-2], imgs.shape[-1])
|
||||||
|
return imgs
|
||||||
|
|
||||||
|
|
||||||
|
def _decode_audio(audio_vae, out_latent):
|
||||||
|
latent = out_latent["samples"]
|
||||||
|
if latent.is_nested:
|
||||||
|
latent = latent.unbind()[-1]
|
||||||
|
audio = audio_vae.decode(latent).movedim(-1, 1)
|
||||||
|
std = torch.std(audio, dim=[1, 2], keepdim=True) * 5.0
|
||||||
|
std[std < 1.0] = 1.0
|
||||||
|
audio = audio / std
|
||||||
|
sr = getattr(audio_vae, "audio_sample_rate_output", getattr(audio_vae, "audio_sample_rate", 44100))
|
||||||
|
return {"waveform": audio, "sample_rate": sr}
|
||||||
|
|
||||||
|
|
||||||
|
def _is_oom(e):
|
||||||
|
return isinstance(e, torch.cuda.OutOfMemoryError) or "out of memory" in str(e).lower()
|
||||||
|
|
||||||
|
|
||||||
|
def _deep_cleanup():
|
||||||
|
"""Release cached VRAM between shots so a long chain does not accumulate and OOM.
|
||||||
|
|
||||||
|
It unloads NOTHING. soft_empty_cache(force) ignores `force` in current ComfyUI
|
||||||
|
(model_management.py:2050) -- the body only reaches empty_cache() and
|
||||||
|
ipc_collect() -- so this drops cached blocks, not models. The `True` is kept
|
||||||
|
only for older builds that read it; the older comment here claimed this took an
|
||||||
|
unload_all_models path, and it does not."""
|
||||||
|
try:
|
||||||
|
mm.soft_empty_cache(True)
|
||||||
|
except TypeError:
|
||||||
|
mm.soft_empty_cache()
|
||||||
|
try:
|
||||||
|
if torch.cuda.is_available():
|
||||||
|
torch.cuda.empty_cache()
|
||||||
|
torch.cuda.ipc_collect()
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
DECODE_HEADROOM = 1.25 # over ComfyUI's own estimate, for working allocations
|
||||||
|
|
||||||
|
|
||||||
|
SAMPLE_HEADROOM = 1.35 # likewise for sampling, which is the longer stretch
|
||||||
|
|
||||||
|
|
||||||
|
def _decode_headroom(vae, latent):
|
||||||
|
"""VRAM this decode actually needs, by the VAE's own estimate. 1e30 if unknown.
|
||||||
|
|
||||||
|
ComfyUI sizes every VAE with memory_used_decode and uses that number itself, so
|
||||||
|
it is the honest figure to hand free_memory. The alternative -- and what was here
|
||||||
|
-- is 1e30, which means "unload everything" and evicts the DiT before every
|
||||||
|
decode, three lines before the next shot reloads it.
|
||||||
|
|
||||||
|
1e30 on failure rather than 0: a bad estimate that frees too little turns a slow
|
||||||
|
render into an OOM, and a wrong guess should fall back to the behaviour that has
|
||||||
|
been running, not to no freeing at all."""
|
||||||
|
try:
|
||||||
|
dtype = getattr(vae, "vae_dtype", None) or latent.dtype
|
||||||
|
need = float(vae.memory_used_decode(tuple(latent.shape), dtype))
|
||||||
|
if need > 0:
|
||||||
|
return need * DECODE_HEADROOM
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
return 1e30
|
||||||
|
|
||||||
|
|
||||||
|
def _resident(models):
|
||||||
|
"""The LoadedModel entries ComfyUI currently holds for `models`.
|
||||||
|
|
||||||
|
That is the form free_memory's keep_loaded wants: it compares against the
|
||||||
|
entries in current_loaded_models, not against the ModelPatcher objects a node
|
||||||
|
is holding. Anything not matched is simply not kept, so a model that is not
|
||||||
|
resident costs nothing here."""
|
||||||
|
out = []
|
||||||
|
for lm in list(getattr(mm, "current_loaded_models", [])):
|
||||||
|
for m in models or ():
|
||||||
|
if m is None:
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
if lm.model is m or getattr(lm, "model", None) is getattr(m, "model", None):
|
||||||
|
if lm not in out:
|
||||||
|
out.append(lm)
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def _image_out_dtype():
|
||||||
|
"""The dtype ComfyUI itself hands between nodes on THIS install.
|
||||||
|
|
||||||
|
The join used to end in a hard-coded .float(), commented "back to what every
|
||||||
|
downstream node expects". That was true when it was written and is not a
|
||||||
|
constant: ComfyUI has --fp16-intermediates, and on an install running it the
|
||||||
|
VAE's own decode already returns fp16 -- VAE.vae_output_dtype() IS
|
||||||
|
model_management.intermediate_dtype() (comfy/sd.py) -- as do EmptyLatentImage
|
||||||
|
and the rest of nodes.py. So on that install the node was taking frames the
|
||||||
|
VAE handed it in fp16, widening them to fp32 nothing had asked for, and
|
||||||
|
handing them to nodes whose own convention is fp16.
|
||||||
|
|
||||||
|
It is the largest thing this node holds, so the widening is not free: the
|
||||||
|
2580-frame chain costed at the join is 9.3GB as fp16 and 18.5GB as fp32,
|
||||||
|
against 44.6GB of staged weights on a 62GB machine -- which is the difference
|
||||||
|
between the render finishing and the OOM killer taking the server. Reported as
|
||||||
|
exactly that, twice.
|
||||||
|
|
||||||
|
Asked, not assumed, and never widened: whatever ComfyUI says it wants between
|
||||||
|
nodes is what the chain is built in. An install with the flag off is told
|
||||||
|
float32 and gets float32, byte for byte what it got before. Older builds have
|
||||||
|
no intermediate_dtype at all, so the fallback is the old constant."""
|
||||||
|
try:
|
||||||
|
return mm.intermediate_dtype()
|
||||||
|
except Exception:
|
||||||
|
return torch.float32
|
||||||
|
|
||||||
|
|
||||||
|
def _evict_all_but(keep_model, latent=None):
|
||||||
|
"""Unload every model EXCEPT the diffusion model from the GPU.
|
||||||
|
|
||||||
|
This is the fix for VRAM ratcheting across a long chain. soft_empty_cache()
|
||||||
|
only drops the CUDA allocator's cached blocks -- it does NOT unload models, so
|
||||||
|
ComfyUI keeps the Qwen3-VL text encoder (~14.6GB) and both VAEs resident in
|
||||||
|
current_loaded_models alongside the DiT. Each shot re-encodes the prompt
|
||||||
|
(text encoder), encodes the handoff keyframe (video VAE), then samples (DiT),
|
||||||
|
so all three compete for the card.
|
||||||
|
|
||||||
|
ComfyUI does free ahead of each load -- load_models_gpu() calls free_memory()
|
||||||
|
for what it is about to need (model_management.py:975), so the weight path is
|
||||||
|
not purely reactive. What it cannot size for is a long chain's ACTIVATIONS on
|
||||||
|
a card where the DiT is most of the VRAM. Freeing explicitly, right after
|
||||||
|
conditioning is built and before sampling, keeps only what the sampler needs.
|
||||||
|
|
||||||
|
ASKED FOR HONESTLY, and this is the expensive one. free_memory computes
|
||||||
|
`memory_to_free = memory_required - get_free_memory(device)`, so 1e30 meant
|
||||||
|
"unload everything but the DiT" on every shot, unconditionally -- on a 48GB card
|
||||||
|
with room for all of it as readily as on a 16GB one. What it unloads is the
|
||||||
|
~14.6GB text encoder and both VAEs, and the next shot re-encodes the prompt and
|
||||||
|
the handoff keyframe, so all three come straight back. On a machine whose RAM is
|
||||||
|
already full of finished frames they come back from DISK, once per shot, which is
|
||||||
|
the thrashing this was reported as.
|
||||||
|
|
||||||
|
The DiT can size its own activations -- memory_required(shape) is what ComfyUI
|
||||||
|
itself calls before a load -- so ask for that. A card with room frees nothing and
|
||||||
|
keeps the encoder resident; a card without frees exactly as much as it must.
|
||||||
|
1e30 stays the fallback, because a bad estimate that frees too little turns a
|
||||||
|
slow render into an OOM."""
|
||||||
|
need = 1e30
|
||||||
|
try:
|
||||||
|
if latent is not None:
|
||||||
|
shape = latent["samples"].shape if isinstance(latent, dict) else latent.shape
|
||||||
|
need = float(keep_model.model.memory_required(tuple(shape))) * SAMPLE_HEADROOM
|
||||||
|
if not (need > 0):
|
||||||
|
need = 1e30
|
||||||
|
except Exception:
|
||||||
|
need = 1e30
|
||||||
|
try:
|
||||||
|
mm.free_memory(need, mm.get_torch_device(),
|
||||||
|
keep_loaded=_resident([keep_model]))
|
||||||
|
except Exception:
|
||||||
|
try:
|
||||||
|
mm.soft_empty_cache(True)
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
def _sample_on_sigmas(model, seed, cfg, sampler_name, positive, negative, latent, sigmas):
|
||||||
|
"""common_ksampler, driven by an EXTERNAL sigma schedule.
|
||||||
|
|
||||||
|
common_ksampler derives its sigmas from (sampler_name, scheduler, steps, denoise)
|
||||||
|
and takes no schedule argument, so a schedule computed anywhere else cannot
|
||||||
|
reach it. Under PDD that is fatal rather than merely inconvenient: the heads
|
||||||
|
accept only their nine trained boundaries, and re-deriving the grid from
|
||||||
|
widgets means hitting it by coincidence and losing it again the moment a step
|
||||||
|
count changes.
|
||||||
|
|
||||||
|
Mirrors nodes.common_ksampler's noise / mask / callback handling exactly -- the
|
||||||
|
only substitution is comfy.sample.sample_custom for comfy.sample.sample."""
|
||||||
|
latent_image = latent["samples"]
|
||||||
|
latent_image = comfy.sample.fix_empty_latent_channels(
|
||||||
|
model, latent_image,
|
||||||
|
latent.get("downscale_ratio_spacial", None),
|
||||||
|
latent.get("downscale_ratio_temporal", None))
|
||||||
|
noise = comfy.sample.prepare_noise(latent_image, seed, latent.get("batch_index"))
|
||||||
|
# `steps` here only sizes the progress bar -- the schedule is `sigmas`, whose
|
||||||
|
# step count is one less than its length (the trailing 0.0 is an endpoint).
|
||||||
|
callback = latent_preview.prepare_callback(model, max(len(sigmas) - 1, 1))
|
||||||
|
samples = comfy.sample.sample_custom(
|
||||||
|
model, noise, cfg, comfy.samplers.sampler_object(sampler_name), sigmas,
|
||||||
|
positive, negative, latent_image,
|
||||||
|
noise_mask=latent.get("noise_mask"), callback=callback,
|
||||||
|
disable_pbar=not comfy.utils.PROGRESS_BAR_ENABLED, seed=seed)
|
||||||
|
out = latent.copy()
|
||||||
|
out.pop("downscale_ratio_spacial", None)
|
||||||
|
out.pop("downscale_ratio_temporal", None)
|
||||||
|
out["samples"] = samples
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
RESIZE_CHUNK = 32
|
||||||
|
|
||||||
|
|
||||||
|
def _stream_chunks(total):
|
||||||
|
"""A collector that writes upscaled chunks into ONE destination as they land.
|
||||||
|
|
||||||
|
Both chunk loops in _upscale_frames used `out.append(...)` then
|
||||||
|
`frames = torch.cat(out, dim=0)`. That is the shape the finished-chain join was
|
||||||
|
rebuilt to stop, at a LARGER size: the list holds the whole upscaled chain and
|
||||||
|
the cat allocates a second one, both live at the cat, and `out` is a local that
|
||||||
|
is never cleared -- so it survives the cat, survives the trailing resize, and is
|
||||||
|
still bound at the return. Meanwhile the CALLER's pre-upscale chain cannot be
|
||||||
|
dropped either, because `part = frames[s:s+batch]` is a view into it.
|
||||||
|
|
||||||
|
At 2580 frames of 1056x608 that is 9.26GB per copy per doubling: 37GB x2 at 2x,
|
||||||
|
and 148GB x2 with the RealESRGAN_x4plus that is sitting in models/upscale_models.
|
||||||
|
Preallocating from the first chunk and copying into it removes exactly one of
|
||||||
|
those two, and drops the list at the same time.
|
||||||
|
|
||||||
|
The destination is sized from the FIRST chunk, so the model's scale factor does
|
||||||
|
not have to be known in advance, and the frame count is the caller's own -- an
|
||||||
|
upscaler changes width and height, never the number of frames."""
|
||||||
|
state = {"dst": None, "at": 0}
|
||||||
|
|
||||||
|
def put(piece):
|
||||||
|
if state["dst"] is None:
|
||||||
|
state["dst"] = torch.empty((int(total),) + tuple(piece.shape[1:]),
|
||||||
|
dtype=piece.dtype, device=piece.device)
|
||||||
|
k = int(piece.shape[0])
|
||||||
|
end = min(state["at"] + k, state["dst"].shape[0])
|
||||||
|
if end > state["at"]:
|
||||||
|
state["dst"][state["at"]:end].copy_(piece[:end - state["at"]])
|
||||||
|
state["at"] = end
|
||||||
|
|
||||||
|
def done():
|
||||||
|
d, at = state["dst"], state["at"]
|
||||||
|
if d is None:
|
||||||
|
return None
|
||||||
|
return d if at == d.shape[0] else d[:at]
|
||||||
|
|
||||||
|
return put, done
|
||||||
|
|
||||||
|
|
||||||
|
def _resize_short_edge(frames, target, method="lanczos", chunk=0):
|
||||||
|
"""Resize a [B,H,W,C] frame batch so its short edge == target (keeping aspect,
|
||||||
|
snapped to /32). Plain high-quality resize -- enlarges, doesn't add detail.
|
||||||
|
|
||||||
|
IN CHUNKS, BECAUSE LANCZOS IS FOUR FULL-LENGTH COPIES. The whole chain went
|
||||||
|
into one common_upscale call, and comfy.utils.lanczos is three successive list
|
||||||
|
comprehensions over every frame at once:
|
||||||
|
|
||||||
|
images = [Image.fromarray(...) for image in samples] # N at source size
|
||||||
|
images = [image.resize(...) for image in images] # N at target size
|
||||||
|
images = [torch.from_numpy(np.array(im).astype(np.float32)/255.) ...]
|
||||||
|
result = torch.stack(images)
|
||||||
|
return result.to(samples.device, samples.dtype)
|
||||||
|
|
||||||
|
A comprehension builds the new list completely before rebinding the name, so at
|
||||||
|
each rebind BOTH are live; then torch.stack allocates a full copy while its list
|
||||||
|
still exists, and .to() allocates the result while the stack still exists. Note
|
||||||
|
the astype(np.float32): the input is fp16 but the two largest transients are at
|
||||||
|
DOUBLE its width. At 2580 frames to a 1080 short edge that peaked around 147GB
|
||||||
|
to produce a 29GB result, and it fires on a DOWNSCALE too.
|
||||||
|
|
||||||
|
Chunked, the peak is the result plus one chunk's worth of that machinery. It is
|
||||||
|
bit-identical: PIL resizes each frame independently, so per-chunk and per-chain
|
||||||
|
give the same pixels. The early return for an already-correct size is kept, so
|
||||||
|
the common no-op case still allocates nothing."""
|
||||||
|
b, h, w, c = frames.shape
|
||||||
|
if min(h, w) == target:
|
||||||
|
return frames
|
||||||
|
if h <= w:
|
||||||
|
nh = target; nw = max(32, int(round(target * w / h / 32) * 32))
|
||||||
|
else:
|
||||||
|
nw = target; nh = max(32, int(round(target * h / w / 32) * 32))
|
||||||
|
step = max(1, int(chunk) or RESIZE_CHUNK)
|
||||||
|
out = torch.empty((b, nh, nw, c), dtype=frames.dtype, device=frames.device)
|
||||||
|
for i in range(0, b, step):
|
||||||
|
part = comfy.utils.common_upscale(
|
||||||
|
frames[i:i + step].movedim(-1, 1), nw, nh, method, "disabled")
|
||||||
|
out[i:i + step].copy_(part.movedim(1, -1))
|
||||||
|
del part
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def _upscale_frames(frames, mode, model_name, target_short_edge, batch=4):
|
||||||
|
"""Optional post-pass upscale of the finished frames (on CPU).
|
||||||
|
mode 'model' : run a ComfyUI upscale model (Real-ESRGAN/UltraSharp class)
|
||||||
|
via the registered loader+apply nodes, chunked with cleanup
|
||||||
|
so 2000+ frames don't OOM; then fit to target short edge.
|
||||||
|
mode 'rtx' : NVIDIA RTX Video Super Resolution (Tensor Cores; fastest,
|
||||||
|
best quality for video -- needs Nvidia_RTX_Nodes_ComfyUI).
|
||||||
|
mode 'lanczos' : plain high-quality resize to the target short edge.
|
||||||
|
Any failure falls back to lanczos (or the raw frames), so it never breaks a
|
||||||
|
render. Returns (frames, note). NOTE: this SHARPENS/ENLARGES; it does not
|
||||||
|
reconstruct video detail the way a second-model (LTX 2.3) pass does."""
|
||||||
|
if mode == "off" or frames is None or getattr(frames, "shape", [0])[0] == 0:
|
||||||
|
return frames, ""
|
||||||
|
note = ""
|
||||||
|
if mode == "rtx":
|
||||||
|
# NVIDIA RTX Video Super Resolution (Comfy-Org/Nvidia_RTX_Nodes_ComfyUI).
|
||||||
|
# Runs on RTX Tensor Cores -- far faster than ESRGAN-class models and
|
||||||
|
# generally cleaner on video, though like them it enhances/enlarges rather
|
||||||
|
# than reconstructing detail (an LTX 2.3 re-generation does that).
|
||||||
|
try:
|
||||||
|
rtx = (_find_node(["rtx", "video", "super"]) or _find_node(["rtxvideosuperresolution"])
|
||||||
|
or _find_node(["rtx", "upscale"]))
|
||||||
|
if rtx is None:
|
||||||
|
raise RuntimeError("RTX node not installed (Nvidia_RTX_Nodes_ComfyUI)")
|
||||||
|
scale = 2
|
||||||
|
if target_short_edge and int(target_short_edge) > 0:
|
||||||
|
cur = min(frames.shape[1], frames.shape[2])
|
||||||
|
if cur > 0:
|
||||||
|
scale = max(1, min(4, int(round(int(target_short_edge) / cur))))
|
||||||
|
_put, _done = _stream_chunks(frames.shape[0])
|
||||||
|
n = frames.shape[0]
|
||||||
|
step = max(1, int(batch))
|
||||||
|
for st in range(0, n, step):
|
||||||
|
part = frames[st:st + step]
|
||||||
|
res = None
|
||||||
|
for kw in ({"image": part, "scale": scale}, {"images": part, "scale": scale},
|
||||||
|
{"image": part, "scale_factor": scale}, {"image": part}):
|
||||||
|
try:
|
||||||
|
res = _invoke_node(rtx, **kw); break
|
||||||
|
except TypeError:
|
||||||
|
continue
|
||||||
|
if res is None:
|
||||||
|
raise RuntimeError("RTX node signature not recognized")
|
||||||
|
_put(res.detach().to("cpu"))
|
||||||
|
del res, part
|
||||||
|
_deep_cleanup()
|
||||||
|
frames = _done()
|
||||||
|
note = f"RTX Video Super Resolution x{scale}"
|
||||||
|
if target_short_edge and int(target_short_edge) > 0:
|
||||||
|
frames = _resize_short_edge(frames, int(target_short_edge))
|
||||||
|
note += f"; fit to {int(target_short_edge)}px short edge"
|
||||||
|
return frames, note
|
||||||
|
except Exception as e:
|
||||||
|
mode = "model"
|
||||||
|
note = f"RTX upscale unavailable ({e}); fell back to model/lanczos"
|
||||||
|
if mode == "model" and model_name and model_name != "none":
|
||||||
|
try:
|
||||||
|
loader = _find_node(["upscale", "model", "load"]) or _find_node(["loadupscalemodel"])
|
||||||
|
applier = _find_node(["imageupscale", "model"]) or _find_node(["upscaleimageusingmodel"])
|
||||||
|
if loader is None or applier is None:
|
||||||
|
raise RuntimeError("upscale-model nodes not found")
|
||||||
|
up_model = _invoke_node(loader, model_name=model_name)
|
||||||
|
_put, _done = _stream_chunks(frames.shape[0])
|
||||||
|
n = frames.shape[0]
|
||||||
|
for s in range(0, n, max(1, int(batch))):
|
||||||
|
part = frames[s:s + max(1, int(batch))]
|
||||||
|
res = _invoke_node(applier, upscale_model=up_model, image=part)
|
||||||
|
_put(res.detach().to("cpu"))
|
||||||
|
del res, part
|
||||||
|
_deep_cleanup()
|
||||||
|
frames = _done()
|
||||||
|
note = f"upscaled with {model_name}"
|
||||||
|
except Exception as e:
|
||||||
|
mode = "lanczos"
|
||||||
|
note = f"model upscale unavailable ({e}); used lanczos"
|
||||||
|
if target_short_edge and int(target_short_edge) > 0:
|
||||||
|
try:
|
||||||
|
frames = _resize_short_edge(frames, int(target_short_edge))
|
||||||
|
note = (note + "; " if note else "") + f"fit to {int(target_short_edge)}px short edge"
|
||||||
|
except Exception as e:
|
||||||
|
note = (note + "; " if note else "") + f"resize failed ({e})"
|
||||||
|
elif mode == "lanczos" and not note:
|
||||||
|
note = "lanczos selected but no target set -> unchanged"
|
||||||
|
return frames, note
|
||||||
|
|
||||||
|
|
||||||
|
def _find_node(substrings):
|
||||||
|
"""Find a registered node whose key contains all of `substrings` (lowercased)."""
|
||||||
|
maps = getattr(nodes, "NODE_CLASS_MAPPINGS", {}) or {}
|
||||||
|
for k, v in maps.items():
|
||||||
|
kl = k.lower()
|
||||||
|
if all(s in kl for s in substrings):
|
||||||
|
return v
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def _invoke_node(cls, **kwargs):
|
||||||
|
"""Call a registered ComfyUI node (V1 FUNCTION or V3 execute) with kwargs and
|
||||||
|
return its first output. Used to reuse ComfyUI's own upscale-model loader/apply
|
||||||
|
so we don't reimplement spandrel loading or tiled scaling."""
|
||||||
|
inst = cls()
|
||||||
|
fn = None
|
||||||
|
if getattr(cls, "FUNCTION", None) and hasattr(inst, cls.FUNCTION):
|
||||||
|
fn = getattr(inst, cls.FUNCTION)
|
||||||
|
else:
|
||||||
|
for cand in ("execute", "upscale", "load_model", "load"):
|
||||||
|
if hasattr(inst, cand):
|
||||||
|
fn = getattr(inst, cand); break
|
||||||
|
if fn is None:
|
||||||
|
raise RuntimeError("no callable entrypoint")
|
||||||
|
out = fn(**kwargs)
|
||||||
|
out = getattr(out, "result", out)
|
||||||
|
return out[0] if isinstance(out, (tuple, list)) else out
|
||||||
|
|
||||||
|
|
||||||
|
# --- THE GRADE THE CHAIN ADDS TO ITSELF -------------------------------------
|
||||||
|
# Every shot boundary decodes a shot, hands its LAST frame over, and re-encodes that as
|
||||||
|
# the next shot's keyframe. The distill reproduces the keyframe faithfully enough to
|
||||||
|
# inherit whatever is already in it and SYNTHESISES frame 0 rather than copying it, so
|
||||||
|
# its own bias lands on top: S_next = a*S + b, a near 1, b above 0. Linear at best,
|
||||||
|
# geometric at worst, invisible shot to shot. And the VAE hard-clips every decode to
|
||||||
|
# 0..1, which makes the expansion a RATCHET -- headroom spent is not recoverable, so it
|
||||||
|
# shows as crushed blacks and blown highlights rather than merely as more contrast.
|
||||||
|
#
|
||||||
|
# These two are the measurement and the correction. Both work per colour channel,
|
||||||
|
# because the clip is per channel: the VAE un-whitens with ImageNet stds before it
|
||||||
|
# clamps, so the 0..1 rails sit at different distances in each channel and the blue
|
||||||
|
# floor and red ceiling bite first. A single luma number would miss the colour half.
|
||||||
|
LEVEL_POOL = 64 # cells per axis the level statistics are measured on
|
||||||
|
|
||||||
|
|
||||||
|
def frame_levels(img):
|
||||||
|
"""(mean, std) per colour channel for one frame, as 3-vectors, or (None, None).
|
||||||
|
|
||||||
|
Area-pooled to LEVEL_POOL first, so a pre-upscale frame and an upscaled one can be
|
||||||
|
compared: pooling measures the PICTURE's levels rather than its resolution. Measured
|
||||||
|
across a 2x resize, std agrees to 0.28% on picture-like content -- and to only 15%
|
||||||
|
on pure noise, because pooling cannot preserve variance that lives entirely at the
|
||||||
|
pixel scale. Real frames are the former, and whatever residual there is cancels
|
||||||
|
anyway: the caller measures the same pipeline difference separately and subtracts it.
|
||||||
|
|
||||||
|
float32 throughout, deliberately: these frames are fp16 under
|
||||||
|
--fp16-intermediates, and an fp16 mean accumulated over a 1344x768 frame biases
|
||||||
|
badly enough to matter at the sizes being corrected here."""
|
||||||
|
x = img
|
||||||
|
if x.dim() == 4:
|
||||||
|
x = x[0]
|
||||||
|
if x.dim() != 3 or int(x.shape[-1]) < 3:
|
||||||
|
return None, None
|
||||||
|
if int(x.shape[0]) < 2 or int(x.shape[1]) < 2:
|
||||||
|
return None, None
|
||||||
|
x = x[..., :3].float().permute(2, 0, 1).unsqueeze(0)
|
||||||
|
p = torch.nn.functional.adaptive_avg_pool2d(x, LEVEL_POOL)[0].reshape(3, -1)
|
||||||
|
return p.mean(dim=1), p.std(dim=1)
|
||||||
|
|
||||||
|
|
||||||
|
# The per-shot motion envelope lived here, measured so the built footsteps could be
|
||||||
|
# timed off the picture. Nothing is built any more -- see the note at the top of
|
||||||
|
# audio.py -- so there is nothing left to time, and a measurement with no reader is a
|
||||||
|
# measurement that rots. Removed with the synthesiser it served.
|
||||||
|
|
||||||
|
|
||||||
|
def apply_levels(img, gain, offset):
|
||||||
|
"""Rescale a frame's contrast and level about its OWN per-channel mean.
|
||||||
|
|
||||||
|
The pivot is the frame's own mean and never a target. That is the whole reason this
|
||||||
|
can run on any scene: a beat that walks into a darker room keeps its darkness,
|
||||||
|
because nothing here knows or cares what the level is -- only how much the last
|
||||||
|
boundary expanded it. Anchoring to shot 1 instead would cancel every deliberate
|
||||||
|
lighting change in the film, which is the opposite failure.
|
||||||
|
|
||||||
|
Clamped into 0..1 because the next thing that happens to this frame is an 8-bit
|
||||||
|
quantisation (comfy.utils.common_upscale goes through a uint8 PIL round trip even
|
||||||
|
at the same size), so there is no headroom outside the range to borrow from."""
|
||||||
|
x = img.float()
|
||||||
|
c = min(3, int(x.shape[-1]))
|
||||||
|
m = x[..., :c].reshape(-1, c).mean(dim=0)
|
||||||
|
g = gain[:c].to(device=x.device, dtype=x.dtype)
|
||||||
|
o = offset[:c].to(device=x.device, dtype=x.dtype)
|
||||||
|
y = x.clone()
|
||||||
|
y[..., :c] = ((x[..., :c] - m) * g + m + o).clamp(0.0, 1.0)
|
||||||
|
return y.to(img.dtype)
|
||||||
@@ -0,0 +1,109 @@
|
|||||||
|
# H3-LongVideos -- https://github.com/Smite79/MiniMax-H3-LongVideos
|
||||||
|
# Copyright (c) 2026 Smite79. All rights reserved.
|
||||||
|
# Redistribution, in whole or in part, requires written permission.
|
||||||
|
# This notice may not be removed or altered. See LICENSE.
|
||||||
|
"""Per-shot records passed from prompt planning to rendering."""
|
||||||
|
|
||||||
|
from dataclasses import dataclass, field
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class Shot:
|
||||||
|
prompt: str
|
||||||
|
cast: list[str]
|
||||||
|
speech: bool
|
||||||
|
sounded: bool
|
||||||
|
voiced_only: bool
|
||||||
|
events: list[str]
|
||||||
|
frame_count: int = 0
|
||||||
|
refs: list[object] = field(default_factory=list)
|
||||||
|
line_seconds: float = 0.0 # the planner's estimate of the spoken line, words / WORDS_PER_SEC
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class ShotPlan:
|
||||||
|
shots: list[Shot] = field(default_factory=list)
|
||||||
|
|
||||||
|
@property
|
||||||
|
def prompts(self):
|
||||||
|
return [shot.prompt for shot in self.shots]
|
||||||
|
|
||||||
|
def add(self, prompt, cast, speech, sounded, voiced_only, events):
|
||||||
|
shot = Shot(prompt, list(cast or ()), bool(speech), bool(sounded),
|
||||||
|
bool(voiced_only), list(events or ()))
|
||||||
|
self.shots.append(shot)
|
||||||
|
|
||||||
|
def set_frame_counts(self, counts):
|
||||||
|
counts = [int(n) for n in counts]
|
||||||
|
if len(counts) != len(self.shots) or any(n <= 0 for n in counts):
|
||||||
|
raise ValueError("frame counts must be positive and match the planned shots")
|
||||||
|
for shot, count in zip(self.shots, counts):
|
||||||
|
shot.frame_count = count
|
||||||
|
|
||||||
|
def validate(self):
|
||||||
|
if any(shot.frame_count <= 0 for shot in self.shots):
|
||||||
|
raise ValueError("each shot needs a positive frame count before rendering")
|
||||||
|
return self
|
||||||
|
|
||||||
|
def __len__(self):
|
||||||
|
return len(self.shots)
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class PreparedVideo:
|
||||||
|
"""Resolved inputs consumed by the render stage; model/tensor handles are shared."""
|
||||||
|
_placed_shots: object
|
||||||
|
_first_is_plate: object
|
||||||
|
_returns: object
|
||||||
|
_soft_landing: object
|
||||||
|
_tagged_names: object
|
||||||
|
ambient_audio: object
|
||||||
|
ambient_level: float
|
||||||
|
apply_model_sampling: bool
|
||||||
|
audio_vae: object
|
||||||
|
auto_sound: bool
|
||||||
|
bared_shots: object
|
||||||
|
cfg: float
|
||||||
|
cleanup_between_shots: bool
|
||||||
|
clip: object
|
||||||
|
first_frame: object
|
||||||
|
foley_level: float
|
||||||
|
h: int
|
||||||
|
latent_upscale: str
|
||||||
|
latent_upscale_scale: float
|
||||||
|
megapixels: float
|
||||||
|
model: object
|
||||||
|
moved_shots: object
|
||||||
|
negative: object
|
||||||
|
notes: list[str]
|
||||||
|
plan: ShotPlan
|
||||||
|
ref_noise_aug: float | None
|
||||||
|
restart_after_removal: bool
|
||||||
|
revealed_shots: object
|
||||||
|
sampler_name: str
|
||||||
|
scheduler: str
|
||||||
|
seed: int
|
||||||
|
shift_audio: float
|
||||||
|
shift_video: float
|
||||||
|
sigmas: object
|
||||||
|
silence_nonspeech: bool
|
||||||
|
speech_lead_seconds: float
|
||||||
|
speech_tail_seconds: float
|
||||||
|
hold_levels: float
|
||||||
|
handoff_frames: int
|
||||||
|
staging_shots: object
|
||||||
|
steps: int
|
||||||
|
stripped_shots: object
|
||||||
|
cut_shots: object
|
||||||
|
tiled_decode: bool
|
||||||
|
trim_seam: bool
|
||||||
|
upscale: str
|
||||||
|
upscale_batch: int
|
||||||
|
upscale_model: str
|
||||||
|
upscale_target_short_edge: int
|
||||||
|
vae: object
|
||||||
|
w: int
|
||||||
|
shot_rooms: object = None # {0-based shot: (room it opens in, room it ends in)}
|
||||||
|
hardware_changed: object = None # 1-based shots that put hardware on or take it off
|
||||||
|
shot_frames: object = None # {0-based shot: (who its frames show, who is still there at its end)}
|
||||||
|
reentry_shots: object = None # {0-based shot: who walks in while the keyframe still has them}
|
||||||
File diff suppressed because it is too large
Load Diff
+18
-2
@@ -1253,11 +1253,13 @@ def curate_h3_prompt(
|
|||||||
soundscape_text = _reference_text(soundscape)
|
soundscape_text = _reference_text(soundscape)
|
||||||
bgm_text = _reference_text(bgm)
|
bgm_text = _reference_text(bgm)
|
||||||
reference_description = ""
|
reference_description = ""
|
||||||
|
individual_reference_descriptions = []
|
||||||
if selected:
|
if selected:
|
||||||
reference_description = " ".join(
|
individual_reference_descriptions = [
|
||||||
_reference_context(ref, picture_number)
|
_reference_context(ref, picture_number)
|
||||||
for picture_number, (_slot, ref) in enumerate(selected, 1)
|
for picture_number, (_slot, ref) in enumerate(selected, 1)
|
||||||
)
|
]
|
||||||
|
reference_description = " ".join(individual_reference_descriptions)
|
||||||
|
|
||||||
prompt_parts = []
|
prompt_parts = []
|
||||||
_append_prompt_section(prompt_parts, "Scene anchor", anchor_text)
|
_append_prompt_section(prompt_parts, "Scene anchor", anchor_text)
|
||||||
@@ -1277,6 +1279,9 @@ def curate_h3_prompt(
|
|||||||
prompt = prompt[: _H3_PROMPT_MAX_CHARS - 3].rstrip() + "..."
|
prompt = prompt[: _H3_PROMPT_MAX_CHARS - 3].rstrip() + "..."
|
||||||
images = [_reference_image(ref) for _slot, ref in selected]
|
images = [_reference_image(ref) for _slot, ref in selected]
|
||||||
images.extend([None] * (_H3_PROMPT_REF_SLOTS - len(images)))
|
images.extend([None] * (_H3_PROMPT_REF_SLOTS - len(images)))
|
||||||
|
individual_reference_descriptions.extend(
|
||||||
|
[""] * (_H3_PROMPT_REF_SLOTS - len(individual_reference_descriptions))
|
||||||
|
)
|
||||||
debug = (
|
debug = (
|
||||||
f"Selected {len(selected)} reference(s): "
|
f"Selected {len(selected)} reference(s): "
|
||||||
+ ", ".join(
|
+ ", ".join(
|
||||||
@@ -1295,6 +1300,7 @@ def curate_h3_prompt(
|
|||||||
soundscape_text,
|
soundscape_text,
|
||||||
bgm_text,
|
bgm_text,
|
||||||
*images[:_H3_PROMPT_REF_SLOTS],
|
*images[:_H3_PROMPT_REF_SLOTS],
|
||||||
|
*individual_reference_descriptions[:_H3_PROMPT_REF_SLOTS],
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
@@ -2687,6 +2693,7 @@ class DumasH3PromptCuratorNode:
|
|||||||
+ ("IMAGE",) * _H3_PROMPT_REF_SLOTS
|
+ ("IMAGE",) * _H3_PROMPT_REF_SLOTS
|
||||||
+ ("INT", "STRING", "STRING", "STRING", "STRING")
|
+ ("INT", "STRING", "STRING", "STRING", "STRING")
|
||||||
+ ("IMAGE",) * _H3_PROMPT_REF_SLOTS
|
+ ("IMAGE",) * _H3_PROMPT_REF_SLOTS
|
||||||
|
+ ("STRING",) * _H3_PROMPT_REF_SLOTS
|
||||||
)
|
)
|
||||||
RETURN_NAMES = (
|
RETURN_NAMES = (
|
||||||
"prompt",
|
"prompt",
|
||||||
@@ -2713,6 +2720,15 @@ class DumasH3PromptCuratorNode:
|
|||||||
"original_ref_7",
|
"original_ref_7",
|
||||||
"original_ref_8",
|
"original_ref_8",
|
||||||
"original_ref_9",
|
"original_ref_9",
|
||||||
|
"compiled_ref_description_1",
|
||||||
|
"compiled_ref_description_2",
|
||||||
|
"compiled_ref_description_3",
|
||||||
|
"compiled_ref_description_4",
|
||||||
|
"compiled_ref_description_5",
|
||||||
|
"compiled_ref_description_6",
|
||||||
|
"compiled_ref_description_7",
|
||||||
|
"compiled_ref_description_8",
|
||||||
|
"compiled_ref_description_9",
|
||||||
)
|
)
|
||||||
FUNCTION = "curate_prompt"
|
FUNCTION = "curate_prompt"
|
||||||
CATEGORY = "Dumas/MiniMax"
|
CATEGORY = "Dumas/MiniMax"
|
||||||
|
|||||||
+74
-100
@@ -8,32 +8,15 @@ const DEFAULT_W = 520;
|
|||||||
const DEFAULT_H = 340;
|
const DEFAULT_H = 340;
|
||||||
const DEFAULT_BEAT = "Describe this beat.";
|
const DEFAULT_BEAT = "Describe this beat.";
|
||||||
const STATE_PROPERTY = "dumas_h3_beat_prompt_state";
|
const STATE_PROPERTY = "dumas_h3_beat_prompt_state";
|
||||||
const CONTINUITY_OPTIONS = ["", "soft carry", "hard cut", "keyframe carry", "handoff ref"];
|
|
||||||
const REF_MODE_OPTIONS = ["", "auto ref2v", "where tagged", "first shot", "every shot", "every shot + handoff ref"];
|
|
||||||
const MANAGED_DIRECTIVES = {
|
const MANAGED_DIRECTIVES = {
|
||||||
seconds: ["seconds", "duration"],
|
remove: ["remove", "removed", "off"],
|
||||||
continuity: ["continuity"],
|
add: ["add", "wear", "wearing"],
|
||||||
ref_mode: ["ref_mode"],
|
|
||||||
ref_noise_aug: ["ref_noise_aug"],
|
|
||||||
anchor_add: ["anchor_add"],
|
|
||||||
overall_soundscape: ["overall_soundscape", "soundscape"],
|
|
||||||
non_diegetic_music: ["non_diegetic_music", "music"],
|
|
||||||
};
|
};
|
||||||
const DIRECTIVE_EXAMPLES = [
|
const DIRECTIVE_EXAMPLES = [
|
||||||
["wardrobe set", "wardrobe: Maya = grey shorts, red jacket"],
|
["remove", "remove: red jacket"],
|
||||||
["wardrobe add", "wardrobe: Maya += red jacket"],
|
["off", "off: steel collar"],
|
||||||
["wardrobe remove", "wardrobe: Maya -= red jacket"],
|
["add", "add: white shirt underneath"],
|
||||||
["seconds", "seconds: 8"],
|
["wearing", "wearing: black coat"],
|
||||||
["exit", "exit: Maya"],
|
|
||||||
["enter", "enter: Jon"],
|
|
||||||
["continuity", "continuity: hard cut"],
|
|
||||||
["ref_mode", "ref_mode: every shot"],
|
|
||||||
["ref_noise_aug", "ref_noise_aug: 0.92"],
|
|
||||||
["anchor_add", "anchor_add: harsh sodium-vapor spill, wet pavement, long-lens compression"],
|
|
||||||
["overall_soundscape", "overall_soundscape: soft rain, distant traffic"],
|
|
||||||
["non_diegetic_music", "non_diegetic_music: tense analog synth pulse"],
|
|
||||||
["soundscape", "soundscape: fluorescent room tone, faint HVAC hum"],
|
|
||||||
["music", "music: low ominous cello and sparse percussion"],
|
|
||||||
];
|
];
|
||||||
|
|
||||||
function injectCSS() {
|
function injectCSS() {
|
||||||
@@ -183,7 +166,7 @@ function injectCSS() {
|
|||||||
}
|
}
|
||||||
|
|
||||||
function defaultState() {
|
function defaultState() {
|
||||||
return { beats: [{ text: DEFAULT_BEAT }] };
|
return { scene: "", character_sheet: "", beats: [{ text: DEFAULT_BEAT }] };
|
||||||
}
|
}
|
||||||
|
|
||||||
function normalizeState(value) {
|
function normalizeState(value) {
|
||||||
@@ -200,7 +183,11 @@ function normalizeState(value) {
|
|||||||
const normalized = beats.map((beat) => ({
|
const normalized = beats.map((beat) => ({
|
||||||
text: typeof beat?.text === "string" ? beat.text : String(beat?.text || ""),
|
text: typeof beat?.text === "string" ? beat.text : String(beat?.text || ""),
|
||||||
}));
|
}));
|
||||||
return normalized.length ? { beats: normalized } : defaultState();
|
return {
|
||||||
|
scene: typeof parsed.scene === "string" ? parsed.scene : String(parsed.scene || ""),
|
||||||
|
character_sheet: typeof parsed.character_sheet === "string" ? parsed.character_sheet : String(parsed.character_sheet || ""),
|
||||||
|
beats: normalized.length ? normalized : [{ text: DEFAULT_BEAT }],
|
||||||
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
function readState(node) {
|
function readState(node) {
|
||||||
@@ -321,6 +308,51 @@ function renderUI(node) {
|
|||||||
node._dh3bpRenderedState = JSON.stringify(state);
|
node._dh3bpRenderedState = JSON.stringify(state);
|
||||||
ui.list.innerHTML = "";
|
ui.list.innerHTML = "";
|
||||||
|
|
||||||
|
const buildTopTextarea = ({ labelText, placeholder, value, onInput }) => {
|
||||||
|
const card = document.createElement("div");
|
||||||
|
card.className = "dh3bp-beat";
|
||||||
|
|
||||||
|
const label = document.createElement("div");
|
||||||
|
label.className = "dh3bp-label";
|
||||||
|
label.textContent = labelText;
|
||||||
|
|
||||||
|
const textarea = document.createElement("textarea");
|
||||||
|
textarea.className = "dh3bp-text";
|
||||||
|
textarea.placeholder = placeholder;
|
||||||
|
textarea.value = value || "";
|
||||||
|
textarea.addEventListener("input", () => {
|
||||||
|
onInput(textarea.value);
|
||||||
|
updateTextareaHeight(textarea);
|
||||||
|
});
|
||||||
|
textarea.addEventListener("keydown", stopCanvasKeyboard);
|
||||||
|
|
||||||
|
card.append(label, textarea);
|
||||||
|
updateTextareaHeight(textarea);
|
||||||
|
return card;
|
||||||
|
};
|
||||||
|
|
||||||
|
ui.list.appendChild(buildTopTextarea({
|
||||||
|
labelText: "Scene paragraph",
|
||||||
|
placeholder: "Optional. Persistent location, lighting, camera, tone. Leave empty if you wire the Long Videos anchor input.",
|
||||||
|
value: state.scene,
|
||||||
|
onInput: (value) => {
|
||||||
|
const next = readState(node);
|
||||||
|
next.scene = value;
|
||||||
|
writeState(node, next);
|
||||||
|
},
|
||||||
|
}));
|
||||||
|
|
||||||
|
ui.list.appendChild(buildTopTextarea({
|
||||||
|
labelText: "Character sheet",
|
||||||
|
placeholder: "Optional. One character per line, e.g. Maya: 27, she, silver hair, red jacket, the woman in <Picture 1>.",
|
||||||
|
value: state.character_sheet,
|
||||||
|
onInput: (value) => {
|
||||||
|
const next = readState(node);
|
||||||
|
next.character_sheet = value;
|
||||||
|
writeState(node, next);
|
||||||
|
},
|
||||||
|
}));
|
||||||
|
|
||||||
state.beats.forEach((beat, index) => {
|
state.beats.forEach((beat, index) => {
|
||||||
const card = document.createElement("div");
|
const card = document.createElement("div");
|
||||||
card.className = "dh3bp-beat";
|
card.className = "dh3bp-beat";
|
||||||
@@ -379,85 +411,27 @@ function renderUI(node) {
|
|||||||
return wrap;
|
return wrap;
|
||||||
};
|
};
|
||||||
|
|
||||||
const secondsInput = document.createElement("input");
|
const removeInput = document.createElement("input");
|
||||||
secondsInput.className = "dh3bp-input";
|
removeInput.className = "dh3bp-input";
|
||||||
secondsInput.type = "text";
|
removeInput.type = "text";
|
||||||
secondsInput.placeholder = "8";
|
removeInput.placeholder = "red jacket";
|
||||||
secondsInput.value = readDirectiveValue(beat.text, MANAGED_DIRECTIVES.seconds);
|
removeInput.value = readDirectiveValue(beat.text, MANAGED_DIRECTIVES.remove);
|
||||||
secondsInput.addEventListener("input", () => {
|
removeInput.addEventListener("input", () => {
|
||||||
applyTextUpdate(setDirectiveValue(textarea.value, "seconds", MANAGED_DIRECTIVES.seconds, secondsInput.value));
|
applyTextUpdate(setDirectiveValue(textarea.value, "remove", MANAGED_DIRECTIVES.remove, removeInput.value));
|
||||||
});
|
});
|
||||||
|
|
||||||
const continuitySelect = document.createElement("select");
|
const addInput = document.createElement("input");
|
||||||
continuitySelect.className = "dh3bp-select";
|
addInput.className = "dh3bp-input";
|
||||||
CONTINUITY_OPTIONS.forEach((value) => {
|
addInput.type = "text";
|
||||||
const option = document.createElement("option");
|
addInput.placeholder = "white shirt underneath";
|
||||||
option.value = value;
|
addInput.value = readDirectiveValue(beat.text, MANAGED_DIRECTIVES.add);
|
||||||
option.textContent = value || "Default";
|
addInput.addEventListener("input", () => {
|
||||||
continuitySelect.appendChild(option);
|
applyTextUpdate(setDirectiveValue(textarea.value, "add", MANAGED_DIRECTIVES.add, addInput.value));
|
||||||
});
|
|
||||||
continuitySelect.value = readDirectiveValue(beat.text, MANAGED_DIRECTIVES.continuity);
|
|
||||||
continuitySelect.addEventListener("change", () => {
|
|
||||||
applyTextUpdate(setDirectiveValue(textarea.value, "continuity", MANAGED_DIRECTIVES.continuity, continuitySelect.value));
|
|
||||||
});
|
|
||||||
|
|
||||||
const refModeSelect = document.createElement("select");
|
|
||||||
refModeSelect.className = "dh3bp-select";
|
|
||||||
REF_MODE_OPTIONS.forEach((value) => {
|
|
||||||
const option = document.createElement("option");
|
|
||||||
option.value = value;
|
|
||||||
option.textContent = value || "Global";
|
|
||||||
refModeSelect.appendChild(option);
|
|
||||||
});
|
|
||||||
refModeSelect.value = readDirectiveValue(beat.text, MANAGED_DIRECTIVES.ref_mode);
|
|
||||||
refModeSelect.addEventListener("change", () => {
|
|
||||||
applyTextUpdate(setDirectiveValue(textarea.value, "ref_mode", MANAGED_DIRECTIVES.ref_mode, refModeSelect.value));
|
|
||||||
});
|
|
||||||
|
|
||||||
const refNoiseInput = document.createElement("input");
|
|
||||||
refNoiseInput.className = "dh3bp-input";
|
|
||||||
refNoiseInput.type = "text";
|
|
||||||
refNoiseInput.placeholder = "0.95";
|
|
||||||
refNoiseInput.value = readDirectiveValue(beat.text, MANAGED_DIRECTIVES.ref_noise_aug);
|
|
||||||
refNoiseInput.addEventListener("input", () => {
|
|
||||||
applyTextUpdate(setDirectiveValue(textarea.value, "ref_noise_aug", MANAGED_DIRECTIVES.ref_noise_aug, refNoiseInput.value));
|
|
||||||
});
|
|
||||||
|
|
||||||
const anchorInput = document.createElement("input");
|
|
||||||
anchorInput.className = "dh3bp-input";
|
|
||||||
anchorInput.type = "text";
|
|
||||||
anchorInput.placeholder = "extra per-shot style treatment";
|
|
||||||
anchorInput.value = readDirectiveValue(beat.text, MANAGED_DIRECTIVES.anchor_add);
|
|
||||||
anchorInput.addEventListener("input", () => {
|
|
||||||
applyTextUpdate(setDirectiveValue(textarea.value, "anchor_add", MANAGED_DIRECTIVES.anchor_add, anchorInput.value));
|
|
||||||
});
|
|
||||||
|
|
||||||
const soundscapeInput = document.createElement("input");
|
|
||||||
soundscapeInput.className = "dh3bp-input";
|
|
||||||
soundscapeInput.type = "text";
|
|
||||||
soundscapeInput.placeholder = "faint traffic, loose sign rattle";
|
|
||||||
soundscapeInput.value = readDirectiveValue(beat.text, MANAGED_DIRECTIVES.overall_soundscape);
|
|
||||||
soundscapeInput.addEventListener("input", () => {
|
|
||||||
applyTextUpdate(setDirectiveValue(textarea.value, "overall_soundscape", MANAGED_DIRECTIVES.overall_soundscape, soundscapeInput.value));
|
|
||||||
});
|
|
||||||
|
|
||||||
const musicInput = document.createElement("input");
|
|
||||||
musicInput.className = "dh3bp-input";
|
|
||||||
musicInput.type = "text";
|
|
||||||
musicInput.placeholder = "low pulsing synth tension";
|
|
||||||
musicInput.value = readDirectiveValue(beat.text, MANAGED_DIRECTIVES.non_diegetic_music);
|
|
||||||
musicInput.addEventListener("input", () => {
|
|
||||||
applyTextUpdate(setDirectiveValue(textarea.value, "non_diegetic_music", MANAGED_DIRECTIVES.non_diegetic_music, musicInput.value));
|
|
||||||
});
|
});
|
||||||
|
|
||||||
controls.append(
|
controls.append(
|
||||||
buildField({ labelText: "Seconds", input: secondsInput }),
|
buildField({ labelText: "Remove from memory", input: removeInput }),
|
||||||
buildField({ labelText: "Continuity", input: continuitySelect }),
|
buildField({ labelText: "Add to memory", input: addInput }),
|
||||||
buildField({ labelText: "Ref Mode", input: refModeSelect }),
|
|
||||||
buildField({ labelText: "Ref Noise Aug", input: refNoiseInput }),
|
|
||||||
buildField({ labelText: "Anchor Add", className: "dh3bp-control-wide", input: anchorInput }),
|
|
||||||
buildField({ labelText: "Shot Soundscape", className: "dh3bp-control-wide", input: soundscapeInput }),
|
|
||||||
buildField({ labelText: "Shot Music", className: "dh3bp-control-wide", input: musicInput }),
|
|
||||||
);
|
);
|
||||||
|
|
||||||
const directives = document.createElement("div");
|
const directives = document.createElement("div");
|
||||||
@@ -501,7 +475,7 @@ function setupNode(node) {
|
|||||||
title.textContent = "Beat Prompt Builder";
|
title.textContent = "Beat Prompt Builder";
|
||||||
const subtitle = document.createElement("div");
|
const subtitle = document.createElement("div");
|
||||||
subtitle.className = "dh3bp-subtitle";
|
subtitle.className = "dh3bp-subtitle";
|
||||||
subtitle.textContent = "One textbox per H3 beat, plus per-shot controls for timing, ref behavior, continuity, anchor adds, and audio directives.";
|
subtitle.textContent = "Upstream Long Videos format: optional scene, optional character sheet, then one blank-line-separated beat per shot.";
|
||||||
titleWrap.append(title, subtitle);
|
titleWrap.append(title, subtitle);
|
||||||
|
|
||||||
const addButton = document.createElement("button");
|
const addButton = document.createElement("button");
|
||||||
|
|||||||
+5
-371
@@ -1,375 +1,9 @@
|
|||||||
import { app } from "/scripts/app.js";
|
import { app } from "/scripts/app.js";
|
||||||
import { applyAdaptiveCanvasOnly } from "../shared/nodes2.mjs";
|
|
||||||
|
|
||||||
const COMFY_CLASS = "DumasH3LongVideos";
|
|
||||||
const STATE_PROPERTY = "dumas_h3_longvideos_section_state";
|
|
||||||
const DOM_WIDGET_NAME = "dumas_h3_longvideos_sections";
|
|
||||||
const MIN_WIDTH = 520;
|
|
||||||
const MIN_HEIGHT = 280;
|
|
||||||
|
|
||||||
const GROUPS = [
|
|
||||||
{
|
|
||||||
id: "prompt",
|
|
||||||
label: "Prompt",
|
|
||||||
defaultCollapsed: false,
|
|
||||||
widgets: ["prompt", "resolution", "megapixels", "beat_split", "anchor_override", "shot_seconds", "plan_only", "fps"],
|
|
||||||
},
|
|
||||||
{
|
|
||||||
id: "refs",
|
|
||||||
label: "Refs",
|
|
||||||
defaultCollapsed: true,
|
|
||||||
widgets: ["ref_mode", "ref_image_size", "ref_noise_aug", "character_memory", "trim_seam", "vary_seed_per_shot", "handoff_offset"],
|
|
||||||
},
|
|
||||||
{
|
|
||||||
id: "sampling",
|
|
||||||
label: "Sampling",
|
|
||||||
defaultCollapsed: true,
|
|
||||||
widgets: [
|
|
||||||
"steps", "cfg", "sampler_name", "scheduler", "seed",
|
|
||||||
"apply_model_sampling", "shift_video", "shift_audio",
|
|
||||||
"vram_headroom_gb", "allow_res_backoff",
|
|
||||||
"decode_tile_frames", "decode_tile_size",
|
|
||||||
],
|
|
||||||
},
|
|
||||||
{
|
|
||||||
id: "audio",
|
|
||||||
label: "Audio",
|
|
||||||
defaultCollapsed: true,
|
|
||||||
widgets: [
|
|
||||||
"global_soundscape", "non_diegetic_music", "auto_soundscape",
|
|
||||||
"auto_silence_nonspeech", "allow_nonspeech_vocals",
|
|
||||||
"mute_nonspeech_audio", "mute_fade_ms",
|
|
||||||
],
|
|
||||||
},
|
|
||||||
{
|
|
||||||
id: "scene",
|
|
||||||
label: "Scene Logic",
|
|
||||||
defaultCollapsed: true,
|
|
||||||
widgets: [
|
|
||||||
"auto_wardrobe", "auto_props", "prevent_nudity", "exposed_terms",
|
|
||||||
"anatomy_guard", "subject_count_guard", "lock_restraints",
|
|
||||||
"contact_guard", "motion_guard", "solidity_guard",
|
|
||||||
],
|
|
||||||
},
|
|
||||||
{
|
|
||||||
id: "finish",
|
|
||||||
label: "Upscale",
|
|
||||||
defaultCollapsed: true,
|
|
||||||
widgets: [
|
|
||||||
"upscale", "upscale_model", "upscale_target_short_edge", "upscale_batch",
|
|
||||||
],
|
|
||||||
},
|
|
||||||
{
|
|
||||||
id: "overlay",
|
|
||||||
label: "Overlays",
|
|
||||||
defaultCollapsed: true,
|
|
||||||
widgets: [
|
|
||||||
"watermark_text", "watermark_position", "watermark_size", "watermark_opacity", "watermark_margin",
|
|
||||||
"intro_text", "intro_position", "intro_seconds", "intro_fade", "intro_size",
|
|
||||||
"overlay_font", "overlay_stroke",
|
|
||||||
],
|
|
||||||
},
|
|
||||||
];
|
|
||||||
|
|
||||||
function injectCSS() {
|
|
||||||
if (document.getElementById("dumas-h3lv-sections-css")) return;
|
|
||||||
const style = document.createElement("style");
|
|
||||||
style.id = "dumas-h3lv-sections-css";
|
|
||||||
style.textContent = `
|
|
||||||
.dh3lv-sections {
|
|
||||||
box-sizing: border-box;
|
|
||||||
width: 100%;
|
|
||||||
padding: 8px 10px 6px;
|
|
||||||
color: #e6e7eb;
|
|
||||||
font: 12px/1.35 "Segoe UI", sans-serif;
|
|
||||||
pointer-events: auto;
|
|
||||||
background: linear-gradient(180deg, rgba(33, 36, 42, 0.96), rgba(22, 24, 29, 0.96));
|
|
||||||
border-bottom: 1px solid rgba(255, 255, 255, 0.06);
|
|
||||||
}
|
|
||||||
.dh3lv-sections-head {
|
|
||||||
display: flex;
|
|
||||||
align-items: center;
|
|
||||||
justify-content: space-between;
|
|
||||||
gap: 8px;
|
|
||||||
margin-bottom: 8px;
|
|
||||||
}
|
|
||||||
.dh3lv-sections-title {
|
|
||||||
font-size: 11px;
|
|
||||||
text-transform: uppercase;
|
|
||||||
letter-spacing: 0.08em;
|
|
||||||
color: #9da5b1;
|
|
||||||
}
|
|
||||||
.dh3lv-sections-actions {
|
|
||||||
display: flex;
|
|
||||||
gap: 6px;
|
|
||||||
}
|
|
||||||
.dh3lv-sections-list {
|
|
||||||
display: flex;
|
|
||||||
flex-wrap: wrap;
|
|
||||||
gap: 6px;
|
|
||||||
}
|
|
||||||
.dh3lv-chip,
|
|
||||||
.dh3lv-action {
|
|
||||||
appearance: none;
|
|
||||||
border: 1px solid #464d59;
|
|
||||||
background: #262c35;
|
|
||||||
color: #d7dce3;
|
|
||||||
border-radius: 999px;
|
|
||||||
padding: 5px 9px;
|
|
||||||
cursor: pointer;
|
|
||||||
font: inherit;
|
|
||||||
line-height: 1.1;
|
|
||||||
}
|
|
||||||
.dh3lv-chip[data-open="true"] {
|
|
||||||
background: #d96f2b;
|
|
||||||
border-color: #f09358;
|
|
||||||
color: #fff7f0;
|
|
||||||
}
|
|
||||||
.dh3lv-chip:hover,
|
|
||||||
.dh3lv-action:hover {
|
|
||||||
filter: brightness(1.06);
|
|
||||||
}
|
|
||||||
.dh3lv-count {
|
|
||||||
opacity: 0.78;
|
|
||||||
margin-left: 4px;
|
|
||||||
font-size: 11px;
|
|
||||||
}
|
|
||||||
`;
|
|
||||||
document.head.appendChild(style);
|
|
||||||
}
|
|
||||||
|
|
||||||
function defaultState() {
|
|
||||||
const state = {};
|
|
||||||
for (const group of GROUPS) state[group.id] = !group.defaultCollapsed;
|
|
||||||
return state;
|
|
||||||
}
|
|
||||||
|
|
||||||
function parseState(value) {
|
|
||||||
let parsed = value;
|
|
||||||
if (typeof parsed === "string") {
|
|
||||||
try {
|
|
||||||
parsed = JSON.parse(parsed);
|
|
||||||
} catch (_error) {
|
|
||||||
parsed = null;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
const base = defaultState();
|
|
||||||
if (!parsed || typeof parsed !== "object") return base;
|
|
||||||
for (const group of GROUPS) {
|
|
||||||
if (typeof parsed[group.id] === "boolean") base[group.id] = parsed[group.id];
|
|
||||||
}
|
|
||||||
return base;
|
|
||||||
}
|
|
||||||
|
|
||||||
function readState(node) {
|
|
||||||
return parseState(node.properties?.[STATE_PROPERTY] || node._dh3lvSectionState || "");
|
|
||||||
}
|
|
||||||
|
|
||||||
function writeState(node, state) {
|
|
||||||
const normalized = parseState(state);
|
|
||||||
const serialized = JSON.stringify(normalized);
|
|
||||||
node._dh3lvSectionState = serialized;
|
|
||||||
node.properties = node.properties || {};
|
|
||||||
node.properties[STATE_PROPERTY] = serialized;
|
|
||||||
}
|
|
||||||
|
|
||||||
function findWidget(node, name) {
|
|
||||||
return (node.widgets || []).find((widget) => widget?.name === name) || null;
|
|
||||||
}
|
|
||||||
|
|
||||||
function isInteractiveTarget(target) {
|
|
||||||
return !!target?.closest?.("button, input, textarea, select, label");
|
|
||||||
}
|
|
||||||
|
|
||||||
function stopCanvasEvent(event) {
|
|
||||||
if (isInteractiveTarget(event.target)) event.stopPropagation();
|
|
||||||
}
|
|
||||||
|
|
||||||
function stopCanvasKeyboard(event) {
|
|
||||||
if (isInteractiveTarget(event.target)) event.stopImmediatePropagation();
|
|
||||||
}
|
|
||||||
|
|
||||||
function setWidgetHidden(widget, hidden) {
|
|
||||||
if (!widget) return;
|
|
||||||
if (!widget._dh3lvOriginal) {
|
|
||||||
widget._dh3lvOriginal = {
|
|
||||||
type: widget.type,
|
|
||||||
computeSize: widget.computeSize,
|
|
||||||
hidden: widget.hidden,
|
|
||||||
};
|
|
||||||
}
|
|
||||||
if (hidden) {
|
|
||||||
widget.type = "hidden";
|
|
||||||
widget.hidden = true;
|
|
||||||
widget.computeSize = () => [0, -4];
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
widget.type = widget._dh3lvOriginal.type;
|
|
||||||
widget.hidden = !!widget._dh3lvOriginal.hidden;
|
|
||||||
widget.computeSize = widget._dh3lvOriginal.computeSize;
|
|
||||||
}
|
|
||||||
|
|
||||||
function applyVisibility(node) {
|
|
||||||
const state = readState(node);
|
|
||||||
for (const group of GROUPS) {
|
|
||||||
for (const name of group.widgets) {
|
|
||||||
const widget = findWidget(node, name);
|
|
||||||
if (!widget || widget.name === DOM_WIDGET_NAME) continue;
|
|
||||||
setWidgetHidden(widget, !state[group.id]);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
function resizeNode(node) {
|
|
||||||
requestAnimationFrame(() => {
|
|
||||||
const size = node.computeSize?.();
|
|
||||||
if (Array.isArray(size)) {
|
|
||||||
node.size[0] = Math.max(MIN_WIDTH, size[0] || 0, node.size?.[0] || 0);
|
|
||||||
node.size[1] = Math.max(MIN_HEIGHT, size[1] || 0);
|
|
||||||
}
|
|
||||||
node.setDirtyCanvas?.(true, true);
|
|
||||||
});
|
|
||||||
}
|
|
||||||
|
|
||||||
function renderToolbar(node) {
|
|
||||||
const ui = node._dh3lvUI;
|
|
||||||
if (!ui) return;
|
|
||||||
const state = readState(node);
|
|
||||||
ui.list.innerHTML = "";
|
|
||||||
for (const group of GROUPS) {
|
|
||||||
const button = document.createElement("button");
|
|
||||||
button.type = "button";
|
|
||||||
button.className = "dh3lv-chip";
|
|
||||||
button.dataset.open = state[group.id] ? "true" : "false";
|
|
||||||
button.textContent = state[group.id] ? `Hide ${group.label}` : `Show ${group.label}`;
|
|
||||||
|
|
||||||
const count = document.createElement("span");
|
|
||||||
count.className = "dh3lv-count";
|
|
||||||
count.textContent = String(group.widgets.filter((name) => findWidget(node, name)).length);
|
|
||||||
button.appendChild(count);
|
|
||||||
|
|
||||||
button.addEventListener("click", () => {
|
|
||||||
const next = readState(node);
|
|
||||||
next[group.id] = !next[group.id];
|
|
||||||
writeState(node, next);
|
|
||||||
applyVisibility(node);
|
|
||||||
renderToolbar(node);
|
|
||||||
resizeNode(node);
|
|
||||||
});
|
|
||||||
ui.list.appendChild(button);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
function setAll(node, open) {
|
|
||||||
const next = {};
|
|
||||||
for (const group of GROUPS) next[group.id] = !!open;
|
|
||||||
writeState(node, next);
|
|
||||||
applyVisibility(node);
|
|
||||||
renderToolbar(node);
|
|
||||||
resizeNode(node);
|
|
||||||
}
|
|
||||||
|
|
||||||
function setupNode(node) {
|
|
||||||
if (node._dh3lvUI) return;
|
|
||||||
injectCSS();
|
|
||||||
writeState(node, readState(node));
|
|
||||||
|
|
||||||
const root = document.createElement("div");
|
|
||||||
root.className = "dh3lv-sections";
|
|
||||||
|
|
||||||
const head = document.createElement("div");
|
|
||||||
head.className = "dh3lv-sections-head";
|
|
||||||
|
|
||||||
const title = document.createElement("div");
|
|
||||||
title.className = "dh3lv-sections-title";
|
|
||||||
title.textContent = "Sections";
|
|
||||||
|
|
||||||
const actions = document.createElement("div");
|
|
||||||
actions.className = "dh3lv-sections-actions";
|
|
||||||
|
|
||||||
const expandAll = document.createElement("button");
|
|
||||||
expandAll.type = "button";
|
|
||||||
expandAll.className = "dh3lv-action";
|
|
||||||
expandAll.textContent = "Expand All";
|
|
||||||
expandAll.addEventListener("click", () => setAll(node, true));
|
|
||||||
|
|
||||||
const collapseAll = document.createElement("button");
|
|
||||||
collapseAll.type = "button";
|
|
||||||
collapseAll.className = "dh3lv-action";
|
|
||||||
collapseAll.textContent = "Collapse Extras";
|
|
||||||
collapseAll.addEventListener("click", () => {
|
|
||||||
const next = defaultState();
|
|
||||||
writeState(node, next);
|
|
||||||
applyVisibility(node);
|
|
||||||
renderToolbar(node);
|
|
||||||
resizeNode(node);
|
|
||||||
});
|
|
||||||
|
|
||||||
actions.append(expandAll, collapseAll);
|
|
||||||
head.append(title, actions);
|
|
||||||
|
|
||||||
const list = document.createElement("div");
|
|
||||||
list.className = "dh3lv-sections-list";
|
|
||||||
root.append(head, list);
|
|
||||||
root.addEventListener("pointerdown", stopCanvasEvent);
|
|
||||||
root.addEventListener("mousedown", stopCanvasEvent);
|
|
||||||
root.addEventListener("click", stopCanvasEvent);
|
|
||||||
root.addEventListener("dblclick", stopCanvasEvent);
|
|
||||||
root.addEventListener("keydown", stopCanvasKeyboard, true);
|
|
||||||
|
|
||||||
node._dh3lvUI = { root, list };
|
|
||||||
const widget = node.addDOMWidget(DOM_WIDGET_NAME, "custom", root, {
|
|
||||||
getValue: () => null,
|
|
||||||
setValue: () => {},
|
|
||||||
serialize: false,
|
|
||||||
getMinHeight: () => 52,
|
|
||||||
hideOnZoom: false,
|
|
||||||
});
|
|
||||||
applyAdaptiveCanvasOnly(widget);
|
|
||||||
const widgets = node.widgets || [];
|
|
||||||
const index = widgets.indexOf(widget);
|
|
||||||
if (index > 0) {
|
|
||||||
widgets.splice(index, 1);
|
|
||||||
widgets.unshift(widget);
|
|
||||||
}
|
|
||||||
|
|
||||||
applyVisibility(node);
|
|
||||||
renderToolbar(node);
|
|
||||||
resizeNode(node);
|
|
||||||
}
|
|
||||||
|
|
||||||
|
// The Dumas Long Videos node now wraps the upstream MiniMax-H3-Longvideos
|
||||||
|
// sampler directly. The old local frontend grouped Dumas-specific widgets that
|
||||||
|
// no longer exist on the upstream node, so this extension intentionally does
|
||||||
|
// nothing.
|
||||||
app.registerExtension({
|
app.registerExtension({
|
||||||
name: "Dumas.H3LongVideosSections",
|
name: "Dumas.H3LongVideos.UpstreamWrapper",
|
||||||
async beforeRegisterNodeDef(nodeType, nodeData) {
|
|
||||||
if (nodeData?.name !== COMFY_CLASS) return;
|
|
||||||
|
|
||||||
const originalOnNodeCreated = nodeType.prototype.onNodeCreated;
|
|
||||||
nodeType.prototype.onNodeCreated = function onNodeCreated() {
|
|
||||||
const result = originalOnNodeCreated?.apply(this, arguments);
|
|
||||||
setupNode(this);
|
|
||||||
return result;
|
|
||||||
};
|
|
||||||
|
|
||||||
const originalConfigure = nodeType.prototype.onConfigure;
|
|
||||||
nodeType.prototype.onConfigure = function onConfigure() {
|
|
||||||
const result = originalConfigure?.apply(this, arguments);
|
|
||||||
setupNode(this);
|
|
||||||
writeState(this, readState(this));
|
|
||||||
applyVisibility(this);
|
|
||||||
renderToolbar(this);
|
|
||||||
resizeNode(this);
|
|
||||||
return result;
|
|
||||||
};
|
|
||||||
|
|
||||||
const originalSerialize = nodeType.prototype.onSerialize;
|
|
||||||
nodeType.prototype.onSerialize = function onSerialize(o) {
|
|
||||||
writeState(this, readState(this));
|
|
||||||
const result = originalSerialize?.apply(this, arguments);
|
|
||||||
if (o && this.properties?.[STATE_PROPERTY]) {
|
|
||||||
o.properties = o.properties || {};
|
|
||||||
o.properties[STATE_PROPERTY] = this.properties[STATE_PROPERTY];
|
|
||||||
}
|
|
||||||
return result;
|
|
||||||
};
|
|
||||||
},
|
|
||||||
});
|
});
|
||||||
|
|||||||
@@ -0,0 +1,85 @@
|
|||||||
|
import { app } from "/scripts/app.js";
|
||||||
|
|
||||||
|
const NODE_NAME = "DumasH3PromptCurator";
|
||||||
|
const EXPECTED_OUTPUTS = [
|
||||||
|
"prompt",
|
||||||
|
"ref_image_1",
|
||||||
|
"ref_image_2",
|
||||||
|
"ref_image_3",
|
||||||
|
"ref_image_4",
|
||||||
|
"ref_image_5",
|
||||||
|
"ref_image_6",
|
||||||
|
"ref_image_7",
|
||||||
|
"ref_image_8",
|
||||||
|
"ref_image_9",
|
||||||
|
"reference_count",
|
||||||
|
"debug",
|
||||||
|
"anchor",
|
||||||
|
"sounds",
|
||||||
|
"bgm",
|
||||||
|
"original_ref_1",
|
||||||
|
"original_ref_2",
|
||||||
|
"original_ref_3",
|
||||||
|
"original_ref_4",
|
||||||
|
"original_ref_5",
|
||||||
|
"original_ref_6",
|
||||||
|
"original_ref_7",
|
||||||
|
"original_ref_8",
|
||||||
|
"original_ref_9",
|
||||||
|
"compiled_ref_description_1",
|
||||||
|
"compiled_ref_description_2",
|
||||||
|
"compiled_ref_description_3",
|
||||||
|
"compiled_ref_description_4",
|
||||||
|
"compiled_ref_description_5",
|
||||||
|
"compiled_ref_description_6",
|
||||||
|
"compiled_ref_description_7",
|
||||||
|
"compiled_ref_description_8",
|
||||||
|
"compiled_ref_description_9",
|
||||||
|
];
|
||||||
|
const EXPECTED_NAMES = new Set(EXPECTED_OUTPUTS);
|
||||||
|
|
||||||
|
function pruneStaleOutputs(node) {
|
||||||
|
if (!Array.isArray(node.outputs)) return;
|
||||||
|
|
||||||
|
const byName = new Map();
|
||||||
|
for (const output of node.outputs) {
|
||||||
|
if (!output?.name || !EXPECTED_NAMES.has(output.name) || byName.has(output.name)) continue;
|
||||||
|
byName.set(output.name, output);
|
||||||
|
}
|
||||||
|
|
||||||
|
const nextOutputs = [];
|
||||||
|
for (const name of EXPECTED_OUTPUTS) {
|
||||||
|
const existing = byName.get(name);
|
||||||
|
if (existing) {
|
||||||
|
nextOutputs.push(existing);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if (nextOutputs.length && nextOutputs.length !== node.outputs.length) {
|
||||||
|
node.outputs = nextOutputs;
|
||||||
|
node.size = node.computeSize?.() || node.size;
|
||||||
|
node.setDirtyCanvas?.(true, true);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
app.registerExtension({
|
||||||
|
name: "Dumas.H3PromptCuratorOutputs",
|
||||||
|
async beforeRegisterNodeDef(nodeType, nodeData) {
|
||||||
|
if (nodeData?.name !== NODE_NAME) return;
|
||||||
|
|
||||||
|
const originalOnNodeCreated = nodeType.prototype.onNodeCreated;
|
||||||
|
const originalOnConfigure = nodeType.prototype.onConfigure;
|
||||||
|
|
||||||
|
nodeType.prototype.onNodeCreated = function onNodeCreated() {
|
||||||
|
const created = originalOnNodeCreated?.apply(this, arguments);
|
||||||
|
pruneStaleOutputs(this);
|
||||||
|
return created;
|
||||||
|
};
|
||||||
|
|
||||||
|
nodeType.prototype.onConfigure = function onConfigure() {
|
||||||
|
const configured = originalOnConfigure?.apply(this, arguments);
|
||||||
|
pruneStaleOutputs(this);
|
||||||
|
return configured;
|
||||||
|
};
|
||||||
|
},
|
||||||
|
});
|
||||||
@@ -11,35 +11,62 @@ class DumasH3BeatPromptTests(unittest.TestCase):
|
|||||||
state = self.module._parse_beat_prompt_state("not json")
|
state = self.module._parse_beat_prompt_state("not json")
|
||||||
self.assertEqual(
|
self.assertEqual(
|
||||||
state,
|
state,
|
||||||
{"beats": [{"text": "Describe this beat."}]},
|
{
|
||||||
|
"scene": "",
|
||||||
|
"character_sheet": "",
|
||||||
|
"beats": [{"text": "Describe this beat."}],
|
||||||
|
},
|
||||||
)
|
)
|
||||||
|
|
||||||
def test_assemble_prompt_joins_beats_with_blank_lines(self):
|
def test_assemble_prompt_outputs_upstream_sections(self):
|
||||||
prompt = self.module._assemble_beat_prompt(
|
prompt = self.module._assemble_beat_prompt(
|
||||||
{
|
{
|
||||||
|
"scene": "A rainy kitchen at night.",
|
||||||
|
"character_sheet": "Maya: 27, she, red jacket, silver hair.",
|
||||||
"beats": [
|
"beats": [
|
||||||
{"text": "A woman enters the room."},
|
{"text": "Maya enters the room."},
|
||||||
{"text": "wardrobe: Maya = red jacket\nShe sits at the table."},
|
{"text": "remove: red jacket\nadd: white shirt underneath\nShe sits at the table."},
|
||||||
{"text": " "},
|
{"text": " "},
|
||||||
{"text": "music: low synth pulse"},
|
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
)
|
)
|
||||||
self.assertEqual(
|
self.assertEqual(
|
||||||
prompt,
|
prompt,
|
||||||
(
|
(
|
||||||
"A woman enters the room.\n\n"
|
"A rainy kitchen at night.\n\n"
|
||||||
"wardrobe: Maya = red jacket\nShe sits at the table.\n\n"
|
"Maya: 27, she, red jacket, silver hair.\n\n"
|
||||||
"music: low synth pulse"
|
"Maya enters the room.\n\n"
|
||||||
|
"remove: red jacket\nadd: white shirt underneath\nShe sits at the table."
|
||||||
),
|
),
|
||||||
)
|
)
|
||||||
|
|
||||||
|
def test_assemble_prompt_strips_old_dumas_directives(self):
|
||||||
|
prompt = self.module._assemble_beat_prompt(
|
||||||
|
{
|
||||||
|
"beats": [
|
||||||
|
{
|
||||||
|
"text": (
|
||||||
|
"seconds: 8\n"
|
||||||
|
"continuity: hard cut\n"
|
||||||
|
"ref_mode: every shot\n"
|
||||||
|
"soundscape: soft rain\n"
|
||||||
|
"music: low synth\n"
|
||||||
|
"Maya opens the cupboard.\n"
|
||||||
|
"remove: red jacket"
|
||||||
|
)
|
||||||
|
},
|
||||||
|
]
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
self.assertEqual(prompt, "Maya opens the cupboard.\nremove: red jacket")
|
||||||
|
|
||||||
def test_node_build_prompt_uses_hidden_state(self):
|
def test_node_build_prompt_uses_hidden_state(self):
|
||||||
node = self.module.DumasH3BeatPromptNode()
|
node = self.module.DumasH3BeatPromptNode()
|
||||||
result = node.build_prompt(
|
result = node.build_prompt(
|
||||||
'{"beats":[{"text":"Beat one"},{"text":"Beat two"}]}'
|
'{"scene":"Scene","character_sheet":"Maya: 27, she","beats":[{"text":"Beat one"},{"text":"Beat two"}]}'
|
||||||
)
|
)
|
||||||
self.assertEqual(result, ("Beat one\n\nBeat two",))
|
self.assertEqual(result, ("Scene\n\nMaya: 27, she\n\nBeat one\n\nBeat two",))
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
|
|||||||
+78
-1629
File diff suppressed because it is too large
Load Diff
@@ -505,6 +505,11 @@ class DumasImageNodeTests(unittest.TestCase):
|
|||||||
self.assertIs(result[15], dave_image)
|
self.assertIs(result[15], dave_image)
|
||||||
self.assertIs(result[16], cafe_image)
|
self.assertIs(result[16], cafe_image)
|
||||||
self.assertIsNone(result[17])
|
self.assertIsNone(result[17])
|
||||||
|
self.assertIn("<Picture 1> Dave", result[24])
|
||||||
|
self.assertNotIn("<Picture 2> Coffee Shop", result[24])
|
||||||
|
self.assertIn("<Picture 2> Coffee Shop", result[25])
|
||||||
|
self.assertIn("Location context for <Picture 2> Coffee Shop", result[25])
|
||||||
|
self.assertEqual(result[26], "")
|
||||||
|
|
||||||
def test_h3_prompt_curator_renumbers_explicit_reference_tags(self):
|
def test_h3_prompt_curator_renumbers_explicit_reference_tags(self):
|
||||||
node = self.image_nodes.DumasH3PromptCuratorNode()
|
node = self.image_nodes.DumasH3PromptCuratorNode()
|
||||||
@@ -642,10 +647,14 @@ class DumasImageNodeTests(unittest.TestCase):
|
|||||||
|
|
||||||
def test_h3_prompt_curator_uses_documented_reference_limits(self):
|
def test_h3_prompt_curator_uses_documented_reference_limits(self):
|
||||||
node = self.image_nodes.DumasH3PromptCuratorNode()
|
node = self.image_nodes.DumasH3PromptCuratorNode()
|
||||||
self.assertEqual(len(node.RETURN_TYPES), 24)
|
self.assertEqual(len(node.RETURN_TYPES), 33)
|
||||||
self.assertEqual(node.RETURN_NAMES[1:10], tuple(f"ref_image_{i}" for i in range(1, 10)))
|
self.assertEqual(node.RETURN_NAMES[1:10], tuple(f"ref_image_{i}" for i in range(1, 10)))
|
||||||
self.assertEqual(node.RETURN_NAMES[12:15], ("anchor", "sounds", "bgm"))
|
self.assertEqual(node.RETURN_NAMES[12:15], ("anchor", "sounds", "bgm"))
|
||||||
self.assertEqual(node.RETURN_NAMES[15:24], tuple(f"original_ref_{i}" for i in range(1, 10)))
|
self.assertEqual(node.RETURN_NAMES[15:24], tuple(f"original_ref_{i}" for i in range(1, 10)))
|
||||||
|
self.assertEqual(
|
||||||
|
node.RETURN_NAMES[24:33],
|
||||||
|
tuple(f"compiled_ref_description_{i}" for i in range(1, 10)),
|
||||||
|
)
|
||||||
|
|
||||||
def test_normalize_reference_upgrades_generic_summary_with_socket_picture_id(self):
|
def test_normalize_reference_upgrades_generic_summary_with_socket_picture_id(self):
|
||||||
image = FakeTensorBatch()
|
image = FakeTensorBatch()
|
||||||
|
|||||||
Reference in New Issue
Block a user