Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
bfaa5fa733 | ||
|
|
9923a3e417 | ||
|
|
ea6fae7e56 | ||
|
|
72f9b07ed8 | ||
|
|
8d4232a142 | ||
|
|
d68d04254f | ||
|
|
22398c5213 | ||
|
|
1a4cfbfd7e | ||
|
|
eac4e73b47 | ||
|
|
e973a0d785 | ||
|
|
388de1d837 |
@@ -0,0 +1,53 @@
|
|||||||
|
H3-LongVideos — Licence
|
||||||
|
Copyright (c) 2026 Smite79. All rights reserved.
|
||||||
|
|
||||||
|
This licence applies to every version published on or after 2026-09-07.
|
||||||
|
|
||||||
|
WHAT YOU MAY DO
|
||||||
|
|
||||||
|
1. Download and use this software, in unmodified or modified form, for your
|
||||||
|
own purposes, personal or commercial. Rendering with it, and whatever you
|
||||||
|
render with it, is yours and is not covered by this licence.
|
||||||
|
|
||||||
|
2. Modify your own copy.
|
||||||
|
|
||||||
|
3. Submit changes back to the original project.
|
||||||
|
|
||||||
|
WHAT YOU MAY NOT DO WITHOUT WRITTEN PERMISSION
|
||||||
|
|
||||||
|
4. Redistribute this software, in whole or in part, modified or unmodified.
|
||||||
|
That includes publishing it to any repository, registry, model hub, node
|
||||||
|
manager, marketplace, or mirror; bundling it inside another package,
|
||||||
|
product, image, or installer; and hosting it as a service.
|
||||||
|
|
||||||
|
5. Remove, alter, or obscure the copyright notice above, this licence, or the
|
||||||
|
attribution in the source files — including where a permitted redistribution
|
||||||
|
has been agreed.
|
||||||
|
|
||||||
|
6. Represent this software, or a derivative of it, as your own work.
|
||||||
|
|
||||||
|
ASKING
|
||||||
|
|
||||||
|
Permission for anything under 4 is granted case by case and is usually given
|
||||||
|
for things like inclusion in a node manager. Ask via the project's GitHub
|
||||||
|
issues at https://github.com/Smite79/MiniMax-H3-LongVideos.
|
||||||
|
|
||||||
|
EARLIER VERSIONS
|
||||||
|
|
||||||
|
Versions published before 2026-09-07 were released under Apache License 2.0.
|
||||||
|
That grant is irrevocable for those versions: copies obtained under it stay
|
||||||
|
under it, and this licence does not and cannot withdraw it retroactively. It
|
||||||
|
governs this version and every version after it.
|
||||||
|
|
||||||
|
Apache 2.0 also required attribution, so a copy of an earlier version
|
||||||
|
republished with the copyright notice stripped was already in breach of the
|
||||||
|
licence it was taken under.
|
||||||
|
|
||||||
|
NO WARRANTY
|
||||||
|
|
||||||
|
This software is provided "as is", without warranty of any kind, express or
|
||||||
|
implied, including but not limited to the warranties of merchantability,
|
||||||
|
fitness for a particular purpose, and non-infringement. In no event shall the
|
||||||
|
copyright holder be liable for any claim, damages, or other liability, whether
|
||||||
|
in an action of contract, tort, or otherwise, arising from, out of, or in
|
||||||
|
connection with the software or the use or other dealings in the software.
|
||||||
@@ -33,34 +33,32 @@
|
|||||||
- Outputs: `plan`, `image1`..`image9`, `connected_images`
|
- Outputs: `plan`, `image1`..`image9`, `connected_images`
|
||||||
- Reads back the nine optional images for a selected MiniMax H3 plan scene, for example by connecting the current `clip_index`.
|
- Reads back the nine optional images for a selected MiniMax H3 plan scene, for example by connecting the current `clip_index`.
|
||||||
|
|
||||||
- `Dumas H3 Long Videos (FL2VA + REF2VA)`
|
- `Dumas H3 Long Videos`
|
||||||
- Inputs: H3 model stack, prompt socket, optional `first_frame`, optional `ref_1`..`ref_9`, plus the upstream long-video control surface for pacing, continuity, audio, overlays, and guards
|
- Inputs/outputs: the current upstream `MiniMax-H3-Longvideos` sampler surface, exposed under the existing `DumasH3LongVideos` key for saved Dumas workflows.
|
||||||
- Outputs: `images`, `audio`, `info`, `script`, `frames_per_shot`, `total_frames`, `shots`, `video_seconds`, `fps`, `fps_int`, `latent`, `soundscape`
|
- The local Dumas prompt-engineering fork has been removed from this node. Long Videos now wraps the upstream sampler/engine/runtime/audio/conditioning/shot-plan modules directly so it can track the source project again.
|
||||||
- First-pass Dumas port of the `MiniMax-H3-Longvideos` sampler, brought in as a local starting point for long-form H3 chaining work.
|
- Upstream compatibility keys `H3LongVideos`, `H3LongVideosFL2VA`, `H3LongVideosV1`, and `H3LongVideosREF2VA` are also registered to the same class.
|
||||||
- Keeps the upstream split-beats / handoff / ref-routing behavior close to source so future Dumas-specific improvements can be compared against a known baseline.
|
- The old Dumas browser widget grouping script is disabled for this node because it targeted controls that no longer exist on the upstream sampler.
|
||||||
- Full user guide: [`H3_LONG_VIDEOS_GUIDE.md`](./H3_LONG_VIDEOS_GUIDE.md)
|
- `handoff_frames` extends the upstream last-frame handoff: `1` keeps the current single keyframe behavior; higher values keep that final-frame keyframe and add earlier tail frames from the previous shot as claimed reference context for the next beat.
|
||||||
- Only the canonical `DumasH3LongVideos` node key is exposed now; the older FL2VA/REF2VA alias entries are no longer duplicated in the Add Node menu.
|
- Upstream license text is included in [`H3_LONGVIDEOS_UPSTREAM_LICENSE.txt`](./H3_LONGVIDEOS_UPSTREAM_LICENSE.txt).
|
||||||
- Prompt `<Picture N>` tags now map to the actual ref socket numbers you wire, even with gaps such as only `ref_2` and `ref_7` connected.
|
|
||||||
- Character refs now contribute appearance and wardrobe context from the same structured object, while location refs contribute environment context from theirs.
|
|
||||||
- The default ref2v bias is now stronger: `ref_mode` defaults to `auto ref2v` so untagged prompts condition every shot instead of only shot 1, and `ref_noise_aug` defaults to `0.95` rather than the upstream-literal `0.999`.
|
|
||||||
- `Dumas H3 Latent Upscale Params` provides the optional pre-decode latent refinement stage for the long-video node.
|
|
||||||
- Per-shot directives now support `continuity:`, `ref_mode:`, `ref_noise_aug:`, `anchor_add:`, `soundscape:`, and `music:` in addition to the existing timing and wardrobe directives.
|
|
||||||
|
|
||||||
- `Dumas H3 Latent Upscale Params`
|
- `Dumas H3 Latent Upscale Params`
|
||||||
- Inputs: `mode`, `model_name`, `method`, `width`, `height`, `device`, `precision`, `sampler_name`, `scheduler`, `steps`, `denoise`, `megapixels`, `tile_width`, `tile_height`, `overlap`, `fade_width`, `fade_height`, `overlap_mode`, `overlap_blend`, `tile_size_mode`, `grid_rows`, `grid_cols`, `spatial_w_overlap`, `spatial_h_overlap`, `min_tile_size`, `masked_area_noise`, `brightness_match`, `dynamic_fade`, `dynamic_fade_min`, `chunk_length`, `temporal_overlap`, `resize_conditioning`, `anchor_strength`
|
- Inputs: `mode`, `model_name`, `method`, `width`, `height`, `device`, `precision`, `sampler_name`, `scheduler`, `steps`, `denoise`, `megapixels`, `tile_width`, `tile_height`, `overlap`, `fade_width`, `fade_height`, `overlap_mode`, `overlap_blend`, `tile_size_mode`, `grid_rows`, `grid_cols`, `spatial_w_overlap`, `spatial_h_overlap`, `min_tile_size`, `masked_area_noise`, `brightness_match`, `dynamic_fade`, `dynamic_fade_min`, `chunk_length`, `temporal_overlap`, `resize_conditioning`, `anchor_strength`
|
||||||
- Output: `latent_upscale_param`
|
- Output: `latent_upscale_param`
|
||||||
- Bundles the optional latent-space upscaler settings used by `Dumas H3 Long Videos` before decode, so the main node can rebuild conditioning at the target size and run a short refinement pass with your chosen sampler, scheduler, step count, denoise, and the full upstream spatial split controls.
|
- Legacy helper from the abandoned Dumas Long Videos fork. The current upstream-backed `Dumas H3 Long Videos` node does not consume this socket; it uses the upstream latent-upscale controls on the Long Videos node itself.
|
||||||
|
|
||||||
- `Dumas H3 Beat Prompt`
|
- `Dumas H3 Beat Prompt`
|
||||||
- Inputs: authored through the custom front-end beat editor
|
- Inputs: authored through the custom front-end beat editor
|
||||||
- Output: `prompt`
|
- Output: `prompt`
|
||||||
- Builds one H3 prompt block per beat, with quick controls for per-shot timing, continuity, ref behavior, anchor additions, soundscape, and music while staying compatible with direct text editing.
|
- Builds an upstream-compatible Long Videos prompt: optional scene paragraph, optional character sheet, then one blank-line-separated textbox per beat.
|
||||||
|
- Per-beat helpers only emit upstream-supported state directives: `remove:` / `removed:` / `off:` and `add:` / `wear:` / `wearing:`.
|
||||||
|
- Old Dumas-only beat directives such as `seconds:`, `continuity:`, `ref_mode:`, `ref_noise_aug:`, `anchor_add:`, `soundscape:`, and `music:` are stripped from the generated prompt so they are not sent to the upstream node as visible text.
|
||||||
|
|
||||||
- `Dumas H3 Prompt Curator`
|
- `Dumas H3 Prompt Curator`
|
||||||
- Inputs: `action_prompt`, `anatomy_guard`, `subject_count_guard`, optional `anchor`, optional `soundscape`, optional `bgm`, optional `ref_1` through `ref_9`
|
- Inputs: `action_prompt`, `anatomy_guard`, `subject_count_guard`, optional `anchor`, optional `soundscape`, optional `bgm`, optional `ref_1` through `ref_9`
|
||||||
- Outputs: `prompt`, `ref_image_1` through `ref_image_9`, `reference_count`, `debug`
|
- Outputs: `prompt`, `ref_image_1` through `ref_image_9`, `reference_count`, `debug`, `anchor`, `sounds`, `bgm`, `original_ref_1` through `original_ref_9`, `compiled_ref_description_1` through `compiled_ref_description_9`
|
||||||
- Builds one standalone MiniMax H3 prompt from your final action text plus structured character/location references.
|
- Builds one standalone MiniMax H3 prompt from your final action text plus structured character/location references.
|
||||||
- The action text can mention references by character/location name, alias, `<Picture N>`, or `<refN>`. Only mentioned references are emitted, and the output images are compacted/renumbered so skipped inputs do not leave gaps.
|
- The action text can mention references by character/location name, alias, `<Picture N>`, or `<refN>`. Only mentioned references are emitted, and the output images are compacted/renumbered so skipped inputs do not leave gaps.
|
||||||
|
- Extra component outputs expose the cleaned anchor, sounds, BGM, and each selected original reference image plus its compiled reference description in compacted order.
|
||||||
- Adds curated reference context, anatomy guard text, optional subject-count guard text, anchor/style text, `overall_soundscape:` text, and `background_music:` text while respecting MiniMax H3's reference-generation shape: one prompt plus up to nine reference images.
|
- Adds curated reference context, anatomy guard text, optional subject-count guard text, anchor/style text, `overall_soundscape:` text, and `background_music:` text while respecting MiniMax H3's reference-generation shape: one prompt plus up to nine reference images.
|
||||||
|
|
||||||
- `Dumas H3 Shot Length`
|
- `Dumas H3 Shot Length`
|
||||||
@@ -100,17 +98,17 @@
|
|||||||
- Output: `anchor`
|
- Output: `anchor`
|
||||||
- Offers a large preset dropdown of anchor-style titles such as cinematic action movie, comedy, found footage, 90s sitcom, mobile/cell phone captured, news broadcast, mockumentary, heist thriller, cyberpunk neon, nature documentary, courtroom drama, and more.
|
- Offers a large preset dropdown of anchor-style titles such as cinematic action movie, comedy, found footage, 90s sitcom, mobile/cell phone captured, news broadcast, mockumentary, heist thriller, cyberpunk neon, nature documentary, courtroom drama, and more.
|
||||||
- The preset wording is tuned for H3-safe persistent anchors: camera language, lighting, texture, production treatment, and tone, without naming characters or describing one-off actions.
|
- The preset wording is tuned for H3-safe persistent anchors: camera language, lighting, texture, production treatment, and tone, without naming characters or describing one-off actions.
|
||||||
- Selecting a preset fills the editable description field, and the edited multiline description is the `STRING` value passed downstream into H3 anchor sockets such as `anchor_override`.
|
- Selecting a preset fills the editable description field, and the edited multiline description is the `STRING` value passed downstream.
|
||||||
|
|
||||||
- `Dumas Soundscape Helper`
|
- `Dumas Soundscape Helper`
|
||||||
- Inputs: `soundscape`, `soundscape_description`
|
- Inputs: `soundscape`, `soundscape_description`
|
||||||
- Output: `soundscape`
|
- Output: `soundscape`
|
||||||
- Matching soundscape helper for standalone H3 prompts. Pick a preset such as quiet interior, rainy street, cafe, city night, forest, industrial, or silent, then edit the text that flows into `Dumas H3 Prompt Curator`.
|
- Matching soundscape helper for standalone H3 prompts. Pick a preset such as quiet interior, rainy street, cafe, city night, forest, industrial, or silent; the preset fills the editable textbox, and the edited text flows into `Dumas H3 Prompt Curator`.
|
||||||
|
|
||||||
- `Dumas Background Music Helper`
|
- `Dumas Background Music Helper`
|
||||||
- Inputs: `bgm`, `bgm_description`
|
- Inputs: `bgm`, `bgm_description`
|
||||||
- Output: `bgm`
|
- Output: `bgm`
|
||||||
- Matching BGM helper for standalone H3 prompts. Pick a preset such as subtle tension, cinematic suspense, emotional piano, dark ambient, hopeful orchestral, retro synth, action pulse, lo-fi, or no vocals, then edit the text that flows into `Dumas H3 Prompt Curator`.
|
- Matching BGM helper for standalone H3 prompts. Pick a preset such as subtle tension, cinematic suspense, emotional piano, dark ambient, hopeful orchestral, retro synth, action pulse, lo-fi, or no vocals; the preset fills the editable textbox, and the edited text flows into `Dumas H3 Prompt Curator`.
|
||||||
|
|
||||||
- `Dumas JSON String to Object`
|
- `Dumas JSON String to Object`
|
||||||
- Input: `json_string`
|
- Input: `json_string`
|
||||||
|
|||||||
+55
-8
@@ -2,11 +2,46 @@ import json
|
|||||||
|
|
||||||
|
|
||||||
_DEFAULT_BEAT = "Describe this beat."
|
_DEFAULT_BEAT = "Describe this beat."
|
||||||
_DEFAULT_STATE = {"beats": [{"text": _DEFAULT_BEAT}]}
|
_DEFAULT_STATE = {"scene": "", "character_sheet": "", "beats": [{"text": _DEFAULT_BEAT}]}
|
||||||
|
_LEGACY_DIRECTIVE_PREFIXES = (
|
||||||
|
"seconds",
|
||||||
|
"duration",
|
||||||
|
"continuity",
|
||||||
|
"ref_mode",
|
||||||
|
"ref_noise_aug",
|
||||||
|
"anchor_add",
|
||||||
|
"overall_soundscape",
|
||||||
|
"soundscape",
|
||||||
|
"non_diegetic_music",
|
||||||
|
"music",
|
||||||
|
"wardrobe",
|
||||||
|
"enter",
|
||||||
|
"exit",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def _clone_default_state():
|
def _clone_default_state():
|
||||||
return {"beats": [{"text": _DEFAULT_BEAT}]}
|
return {
|
||||||
|
"scene": "",
|
||||||
|
"character_sheet": "",
|
||||||
|
"beats": [{"text": _DEFAULT_BEAT}],
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _strip_legacy_directives(text):
|
||||||
|
"""Remove directives from the abandoned Dumas Long Videos fork.
|
||||||
|
|
||||||
|
The upstream Long Videos node sends unknown field labels to the model as text,
|
||||||
|
so this builder strips the old managed controls rather than emitting prompts
|
||||||
|
that ask H3 to draw labels such as "seconds:" or "music:" in the frame.
|
||||||
|
"""
|
||||||
|
kept = []
|
||||||
|
for line in str(text or "").splitlines():
|
||||||
|
lowered = line.strip().lower()
|
||||||
|
if any(lowered.startswith(f"{name}:") for name in _LEGACY_DIRECTIVE_PREFIXES):
|
||||||
|
continue
|
||||||
|
kept.append(line)
|
||||||
|
return "\n".join(kept).strip()
|
||||||
|
|
||||||
|
|
||||||
def _parse_beat_prompt_state(value):
|
def _parse_beat_prompt_state(value):
|
||||||
@@ -21,6 +56,8 @@ def _parse_beat_prompt_state(value):
|
|||||||
except Exception:
|
except Exception:
|
||||||
return _clone_default_state()
|
return _clone_default_state()
|
||||||
|
|
||||||
|
scene = str(raw.get("scene") or "")
|
||||||
|
character_sheet = str(raw.get("character_sheet") or "")
|
||||||
beats = []
|
beats = []
|
||||||
for item in list(raw.get("beats") or []):
|
for item in list(raw.get("beats") or []):
|
||||||
if isinstance(item, dict):
|
if isinstance(item, dict):
|
||||||
@@ -30,15 +67,25 @@ def _parse_beat_prompt_state(value):
|
|||||||
beats.append({"text": text})
|
beats.append({"text": text})
|
||||||
|
|
||||||
if not beats:
|
if not beats:
|
||||||
return _clone_default_state()
|
beats = [{"text": _DEFAULT_BEAT}]
|
||||||
return {"beats": beats}
|
return {
|
||||||
|
"scene": scene,
|
||||||
|
"character_sheet": character_sheet,
|
||||||
|
"beats": beats,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
def _assemble_beat_prompt(state):
|
def _assemble_beat_prompt(state):
|
||||||
parsed = _parse_beat_prompt_state(state)
|
parsed = _parse_beat_prompt_state(state)
|
||||||
chunks = []
|
chunks = []
|
||||||
|
scene = str(parsed.get("scene") or "").strip()
|
||||||
|
if scene:
|
||||||
|
chunks.append(scene)
|
||||||
|
character_sheet = str(parsed.get("character_sheet") or "").strip()
|
||||||
|
if character_sheet:
|
||||||
|
chunks.append(character_sheet)
|
||||||
for beat in parsed["beats"]:
|
for beat in parsed["beats"]:
|
||||||
text = str(beat.get("text") or "").strip()
|
text = _strip_legacy_directives(beat.get("text") or "")
|
||||||
if text:
|
if text:
|
||||||
chunks.append(text)
|
chunks.append(text)
|
||||||
return "\n\n".join(chunks)
|
return "\n\n".join(chunks)
|
||||||
@@ -46,9 +93,9 @@ def _assemble_beat_prompt(state):
|
|||||||
|
|
||||||
class DumasH3BeatPromptNode:
|
class DumasH3BeatPromptNode:
|
||||||
DESCRIPTION = (
|
DESCRIPTION = (
|
||||||
"Build a MiniMax H3 prompt from one textbox per beat, with a front-end beat "
|
"Build an upstream MiniMax H3 Long Videos prompt: optional scene paragraph, "
|
||||||
"editor that can append directive examples and expose per-shot controls for "
|
"optional character sheet, then one blank-line-separated textbox per beat. "
|
||||||
"timing, continuity, ref behavior, anchor additions, soundscape, and music."
|
"Per-beat helpers only emit directives the upstream node understands."
|
||||||
)
|
)
|
||||||
RETURN_TYPES = ("STRING",)
|
RETURN_TYPES = ("STRING",)
|
||||||
RETURN_NAMES = ("prompt",)
|
RETURN_NAMES = ("prompt",)
|
||||||
|
|||||||
+23
-7916
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,319 @@
|
|||||||
|
# H3-LongVideos -- https://github.com/Smite79/MiniMax-H3-LongVideos
|
||||||
|
# Copyright (c) 2026 Smite79. All rights reserved.
|
||||||
|
# Redistribution, in whole or in part, requires written permission.
|
||||||
|
# This notice may not be removed or altered. See LICENSE.
|
||||||
|
"""Audio policy shared by conditioning and soundtrack assembly."""
|
||||||
|
|
||||||
|
from dataclasses import dataclass
|
||||||
|
|
||||||
|
import torch
|
||||||
|
import comfy.nested_tensor
|
||||||
|
from h3_runtime import temporal_shape
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class ShotAudio:
|
||||||
|
speech: bool
|
||||||
|
sounded: bool
|
||||||
|
voiced_only: bool
|
||||||
|
silence_enabled: bool
|
||||||
|
lead_seconds: float
|
||||||
|
latent_fps: int
|
||||||
|
# The tail. Everything after the line's expected end is pinned the way the lead
|
||||||
|
# pins everything before its start. All three default off, so a ShotAudio built
|
||||||
|
# the old way -- six positional arguments -- behaves exactly the old way.
|
||||||
|
line_seconds: float = 0.0 # planner's estimate of the spoken line
|
||||||
|
tail_seconds: float = 0.0 # free audio kept after that estimate; 0 = no tail pin
|
||||||
|
frame_count: int = 0 # the shot in pixel frames; the audio T comes from it
|
||||||
|
|
||||||
|
@property
|
||||||
|
def pinned(self):
|
||||||
|
return self.silence_enabled and not self.speech and not self.sounded
|
||||||
|
|
||||||
|
@property
|
||||||
|
def lead_frames(self):
|
||||||
|
if not self.speech or self.lead_seconds <= 0:
|
||||||
|
return 0
|
||||||
|
return round(self.lead_seconds * self.latent_fps)
|
||||||
|
|
||||||
|
@property
|
||||||
|
def tail_frames(self):
|
||||||
|
"""Audio latent frames pinned at the END of a dialogue shot.
|
||||||
|
|
||||||
|
The lead pins the opening so the line cannot start early; nothing pinned the
|
||||||
|
close, and a 2s line in a 9s shot left 7s of open branch in a shot the model
|
||||||
|
knows has a voice in it -- which is where speech carries on past the line, or
|
||||||
|
doubles it. The free span is lead + the line's estimate + tail_seconds; the
|
||||||
|
rest is held at encoded silence. The model chooses WHEN to speak, so the
|
||||||
|
margin is the author's dial: a clipped word costs more than a second of babble.
|
||||||
|
Off unless the shot speaks, the margin is set, and at least half a second would
|
||||||
|
be pinned -- a sliver is not worth the risk of clipping."""
|
||||||
|
if (not self.speech or self.tail_seconds <= 0 or self.line_seconds <= 0
|
||||||
|
or self.frame_count <= 0):
|
||||||
|
return 0
|
||||||
|
total = temporal_shape(self.frame_count)[2]
|
||||||
|
free = self.lead_frames + round((self.line_seconds + self.tail_seconds) * self.latent_fps)
|
||||||
|
tail = total - free
|
||||||
|
return tail if tail >= round(0.5 * self.latent_fps) else 0
|
||||||
|
|
||||||
|
|
||||||
|
_SILENT_UNIT = {"lat": None, "key": None}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# NOTHING HERE IS SYNTHESISED ANY MORE. Removed on the report: "Just get rid of
|
||||||
|
# the ambient sounds all together. They sound horrid. Go back to the model's
|
||||||
|
# natural audio."
|
||||||
|
#
|
||||||
|
# What was here built the soundtrack's non-vocal half out of shaped noise: a room
|
||||||
|
# tone from the scene's own wording (synth_ambient, over a table of recipes, with
|
||||||
|
# plain_bed under it as a floor) and 21 foley recipes laid into the shots whose
|
||||||
|
# audio branch is pinned to silence (foley_for, over _hits/_band/_room and later
|
||||||
|
# _contact/_flow/_creak, timed off the picture's own movement for footsteps).
|
||||||
|
#
|
||||||
|
# It went in because a shot pinned to silence cannot get audio from the model at
|
||||||
|
# all -- prompt text never opens a branch -- so auto_sound was writing sounds into
|
||||||
|
# prompts that could not make them. That reasoning was sound and the thing it built
|
||||||
|
# still did not pass: reported first as footsteps sounding like heartbeats and a
|
||||||
|
# bathroom that tapped, and then, once both of those measured clean, as horrid
|
||||||
|
# anyway. Synthesis that measures right and sounds wrong is the end of that road.
|
||||||
|
#
|
||||||
|
# So the audio is the model's, whole. H3 is a joint model and the audio branch is
|
||||||
|
# where its sound comes from; the prompt still describes what a shot sounds like,
|
||||||
|
# which is the half that was always doing the real work.
|
||||||
|
#
|
||||||
|
# The consequence, which is real and is reported in info rather than left to be
|
||||||
|
# discovered: a shot with no line and no sound you wrote is pinned to silence and
|
||||||
|
# is now SILENT. The pin is not a bug and is deliberately untouched -- it is what
|
||||||
|
# stops a free branch filling itself with babble and a face lip-syncing to it.
|
||||||
|
# Write the sound into the beat to open the branch on purpose, or wire a recording
|
||||||
|
# to ambient_audio, which is played under the finished track and conditions
|
||||||
|
# nothing. mix_ambient below is that path, and it is all that is left here.
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def _seamless_loop(x, n, sr):
|
||||||
|
"""[C, M] -> [C, n], looped with a crossfade so the join does not click.
|
||||||
|
|
||||||
|
Plain tiling puts a discontinuity at every repeat, once per loop length. In a
|
||||||
|
bed that is meant to sit under everything unnoticed, a regular click is the one
|
||||||
|
thing that gets noticed -- the same objection that made the silence latent
|
||||||
|
ping-pong its interior rather than tile it. Here the material is real audio
|
||||||
|
being PLAYED rather than a latent being conditioned on, so it cannot be
|
||||||
|
reversed: a room tone read backwards is fine, but footsteps are not. Crossfade
|
||||||
|
instead, which works on both."""
|
||||||
|
m = int(x.shape[-1])
|
||||||
|
if m <= 0:
|
||||||
|
return None
|
||||||
|
if m >= n:
|
||||||
|
return x[..., :n]
|
||||||
|
fade = min(int(0.25 * sr), m // 4)
|
||||||
|
if fade < 1:
|
||||||
|
reps = -(-n // m)
|
||||||
|
return x.repeat(1, reps)[..., :n]
|
||||||
|
# OVERLAP-ADD the tail onto the head, and shorten the unit by the overlap. The
|
||||||
|
# unit then runs x[m-fade] .. x[m-fade-1], so tiling it steps between samples
|
||||||
|
# that were adjacent in the source and there is no discontinuity anywhere.
|
||||||
|
#
|
||||||
|
# Measured, because the obvious construction is wrong: appending the crossfade
|
||||||
|
# to the END of a full-length unit leaves it finishing on x[fade-1] while the
|
||||||
|
# next repeat starts on x[0], which are not adjacent -- a 2s tone that does not
|
||||||
|
# divide evenly gave a 64x jump at the join, worse than plain tiling's 41x.
|
||||||
|
t = torch.linspace(0.0, 1.0, fade, dtype=x.dtype, device=x.device)
|
||||||
|
head = x[..., :fade] * t + x[..., m - fade:] * (1.0 - t)
|
||||||
|
unit = torch.cat([head, x[..., fade:m - fade]], dim=-1)
|
||||||
|
if int(unit.shape[-1]) < 1:
|
||||||
|
reps = -(-n // m)
|
||||||
|
return x.repeat(1, reps)[..., :n]
|
||||||
|
reps = -(-n // int(unit.shape[-1]))
|
||||||
|
return unit.repeat(1, reps)[..., :n]
|
||||||
|
|
||||||
|
|
||||||
|
def mix_ambient(audio, sr, bed, level):
|
||||||
|
"""Lay an ambient bed UNDER a finished soundtrack. -> (waveform, note).
|
||||||
|
|
||||||
|
The bed is PLAYED, not conditioned on: it is the file, at the level asked for,
|
||||||
|
under whatever the model generated. That is the whole reason to do it here
|
||||||
|
rather than in the sampler -- ambience needs no cooperation from a joint model,
|
||||||
|
has nothing to lip-sync to, and so cannot put a voice in a wordless shot. The
|
||||||
|
conditioning path can only steer the branch toward something bed-LIKE, and on a
|
||||||
|
shot with a line it competes with the line.
|
||||||
|
|
||||||
|
Defensive throughout, like the silence latent: any failure returns the audio
|
||||||
|
untouched with a note saying so, because a bed is a nicety and a render is not.
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
if audio is None or bed is None or float(level or 0.0) <= 0.0:
|
||||||
|
return audio, ""
|
||||||
|
w = bed.get("waveform") if isinstance(bed, dict) else None
|
||||||
|
if w is None or not int(getattr(w, "ndim", 0)):
|
||||||
|
return audio, ("ambient_audio is wired but carries no waveform, so nothing "
|
||||||
|
"was laid under the soundtrack")
|
||||||
|
w = w[0] if w.dim() == 3 else w # [B, C, M] -> [C, M]
|
||||||
|
if w.dim() != 2 or w.shape[-1] < 2:
|
||||||
|
return audio, ("ambient_audio is too short to loop, so nothing was laid "
|
||||||
|
"under the soundtrack")
|
||||||
|
w = w.detach().to(dtype=audio.dtype, device=audio.device)
|
||||||
|
b_sr = int((bed.get("sample_rate") if isinstance(bed, dict) else 0) or 0)
|
||||||
|
# RESAMPLE, or the bed plays at the wrong speed and pitch. Linear is coarse
|
||||||
|
# for music and inaudible on a room tone, which is what this input is for.
|
||||||
|
resampled = ""
|
||||||
|
if b_sr > 0 and b_sr != int(sr):
|
||||||
|
want = max(2, int(round(w.shape[-1] * float(sr) / float(b_sr))))
|
||||||
|
w = torch.nn.functional.interpolate(
|
||||||
|
w.unsqueeze(0), size=want, mode="linear", align_corners=False)[0]
|
||||||
|
resampled = f", resampled from {b_sr} Hz"
|
||||||
|
ch = int(audio.shape[1])
|
||||||
|
if int(w.shape[0]) != ch:
|
||||||
|
w = (w.mean(dim=0, keepdim=True).repeat(ch, 1) if int(w.shape[0]) > ch
|
||||||
|
else w[:1].repeat(ch, 1))
|
||||||
|
n = int(audio.shape[-1])
|
||||||
|
loop = _seamless_loop(w, n, int(sr))
|
||||||
|
if loop is None:
|
||||||
|
return audio, ""
|
||||||
|
out = audio + loop.unsqueeze(0) * float(level)
|
||||||
|
# NORMALISE rather than clip. Clipping a bed that pushed a loud line over
|
||||||
|
# the top distorts the LINE, which is the thing worth keeping.
|
||||||
|
peak = float(out.abs().max())
|
||||||
|
gain = ""
|
||||||
|
if peak > 1.0:
|
||||||
|
out = out / peak
|
||||||
|
gain = f", and the mix was scaled by {1.0 / peak:.2f} to stop it clipping"
|
||||||
|
secs = w.shape[-1] / float(sr)
|
||||||
|
return out, (f"an ambient bed was laid under the whole soundtrack at level "
|
||||||
|
f"{float(level):.2f} -- {secs:.1f}s of audio{resampled}, looped "
|
||||||
|
f"with a crossfade so the join does not click{gain}. It is your "
|
||||||
|
f"file, played under what the model generated: it conditions "
|
||||||
|
f"nothing, so it cannot put a voice in a wordless shot the way "
|
||||||
|
f"an inferred bed did. Shots pinned to silence keep their silent "
|
||||||
|
f"conditioning and get the bed on top, which is what makes a "
|
||||||
|
f"wordless shot sound like a room instead of a mute")
|
||||||
|
except Exception as exc:
|
||||||
|
return audio, (f"the ambient bed could not be mixed ({type(exc).__name__}), so "
|
||||||
|
f"the soundtrack is unchanged")
|
||||||
|
|
||||||
|
|
||||||
|
_SILENCE_STATUS = {"asked": 0, "applied": 0, "why": ""}
|
||||||
|
|
||||||
|
|
||||||
|
_SILENT_SECONDS = 2
|
||||||
|
|
||||||
|
|
||||||
|
_SILENT_EDGE = 4
|
||||||
|
|
||||||
|
|
||||||
|
def _silent_audio_latent(audio_vae, frame_count, fps):
|
||||||
|
"""A keyframe audio latent of actual SILENCE, or None if it cannot be made.
|
||||||
|
|
||||||
|
H3 is a JOINT model: the mouth follows the audio branch. On a shot with no
|
||||||
|
scripted line the branch is otherwise unconditioned, and an unconditioned audio
|
||||||
|
branch invents a voice -- which the picture then lip-syncs to. The lips-closed
|
||||||
|
sentence is arguing with a stream that has already decided someone is talking.
|
||||||
|
|
||||||
|
REBUILT 2026-09-05, from measurements against the real VAE rather than from
|
||||||
|
reasoning. The previous version encoded one second, kept a SINGLE interior
|
||||||
|
frame and repeated it, on the argument that silence is homogeneous. It is not,
|
||||||
|
in latent space: encoded silence has genuine frame-to-frame variation (delta
|
||||||
|
mean 0.002-0.004, max 0.021), and a repeated frame has a delta of exactly
|
||||||
|
0.000000. That is a flat signal no encoder produces, and a model handed
|
||||||
|
conditioning outside its own distribution has every reason to disregard it --
|
||||||
|
which is an audio branch back to inventing a voice, with the report saying
|
||||||
|
silence went on.
|
||||||
|
|
||||||
|
The fix that version was avoiding is real too: tiling the whole encoded second
|
||||||
|
end to end leaves a 25x spike at each join (0.554 against 0.022), once per
|
||||||
|
second, which is a metronome in the conditioning of a joint model.
|
||||||
|
|
||||||
|
So: encode two seconds, drop the padded ends, and PING-PONG the interior --
|
||||||
|
forward, reversed, forward. Every join repeats a frame, so there is no seam,
|
||||||
|
and the interior statistics are the encoder's own. Measured over a 9s shot:
|
||||||
|
|
||||||
|
one frame repeated peak 0.000686 delta mean 0.000000 max 0.000000
|
||||||
|
whole 2s tiled peak 0.000314 delta mean 0.017451 max 0.554715
|
||||||
|
interior ping-pong peak 0.000566 delta mean 0.002039 max 0.021159
|
||||||
|
|
||||||
|
where the encoder's own interior is mean 0.0021, max 0.0212. Decoded peak
|
||||||
|
0.000566 on a +/-1.0 scale is about -65 dBFS: silence.
|
||||||
|
|
||||||
|
Everything here stays defensive. Shapes are CHECKED against what the layout
|
||||||
|
expects rather than assumed, and any failure returns None so the shot falls
|
||||||
|
back to an unconditioned branch instead of breaking the render -- the caller
|
||||||
|
reports when that happens, so it is no longer a silent failure.
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
sr = int(getattr(audio_vae, "audio_sample_rate", 0) or 0)
|
||||||
|
if sr <= 0:
|
||||||
|
return None
|
||||||
|
_, _, want_t = temporal_shape(frame_count, fps)
|
||||||
|
if want_t <= 0:
|
||||||
|
return None
|
||||||
|
key = (id(audio_vae), sr)
|
||||||
|
block = _SILENT_UNIT.get("lat") if _SILENT_UNIT.get("key") == key else None
|
||||||
|
if block is None:
|
||||||
|
# CHANNELS LAST. comfy.sd.VAE.encode() does `pixel_samples.movedim(-1, 1)`
|
||||||
|
# before handing off, so the audio VAE -- which wants [B, 2, L] -- must be
|
||||||
|
# given [B, L, 2]. Passing [B, 2, L] raises inside the encoder, and an
|
||||||
|
# early version did exactly that: swallowed by the guard below, so the
|
||||||
|
# whole layer silently did nothing.
|
||||||
|
#
|
||||||
|
# Two seconds, encoded ONCE and cached. Encoding a full 15s shot instead
|
||||||
|
# cost a VAE pass big enough to OOM mid-render on a 16GB card, where the
|
||||||
|
# failure again degraded silently to no conditioning at all.
|
||||||
|
enc = audio_vae.encode(torch.zeros((1, sr * _SILENT_SECONDS, 2)))
|
||||||
|
if enc is None or enc.dim() != 4 or enc.shape[1] != 32:
|
||||||
|
return None
|
||||||
|
if enc.shape[-1] <= 2 * _SILENT_EDGE + 1:
|
||||||
|
return None
|
||||||
|
block = enc[..., _SILENT_EDGE:-_SILENT_EDGE].detach().to("cpu").clone()
|
||||||
|
_SILENT_UNIT["lat"] = block
|
||||||
|
_SILENT_UNIT["key"] = key
|
||||||
|
n = block.shape[-1]
|
||||||
|
if n < 1:
|
||||||
|
return None
|
||||||
|
# Forward, reversed, forward... Each join repeats a frame, so the seam that
|
||||||
|
# plain tiling leaves is gone while the interior variation is the encoder's.
|
||||||
|
pieces, have, i = [], 0, 0
|
||||||
|
while have < want_t:
|
||||||
|
piece = block if i % 2 == 0 else torch.flip(block, dims=[-1])
|
||||||
|
pieces.append(piece)
|
||||||
|
have += n
|
||||||
|
i += 1
|
||||||
|
out = torch.cat(pieces, dim=-1)[..., :want_t].clone()
|
||||||
|
if out.shape[-1] != want_t:
|
||||||
|
return None
|
||||||
|
return out
|
||||||
|
except Exception:
|
||||||
|
return None # never fail a render for a nicety
|
||||||
|
|
||||||
|
|
||||||
|
def _pin_audio_silence(latent, silence, lead_frames=None, tail_frames=0):
|
||||||
|
"""Start target audio at encoded silence and preserve the requested span(s).
|
||||||
|
|
||||||
|
lead_frames None pins the whole shot. Otherwise the first lead_frames and the
|
||||||
|
last tail_frames are held at silence and the span between is left to the model
|
||||||
|
-- that is where the line goes. The tail is clipped to what the lead leaves, so
|
||||||
|
the two can never overlap. Nothing pinned at all is a no-op, reported as False
|
||||||
|
so the caller does not count it as applied."""
|
||||||
|
try:
|
||||||
|
video, audio = latent["samples"].unbind()
|
||||||
|
silence = silence.to(device=audio.device, dtype=audio.dtype)
|
||||||
|
if silence.shape != audio.shape:
|
||||||
|
return False
|
||||||
|
audio_mask = torch.ones_like(audio[:, :1])
|
||||||
|
if lead_frames is None:
|
||||||
|
audio_mask.zero_()
|
||||||
|
else:
|
||||||
|
t = audio.shape[-1]
|
||||||
|
n = min(t, max(0, int(lead_frames)))
|
||||||
|
m = min(t - n, max(0, int(tail_frames or 0)))
|
||||||
|
if n <= 0 and m <= 0:
|
||||||
|
return False
|
||||||
|
if n > 0:
|
||||||
|
audio_mask[..., :n] = 0
|
||||||
|
if m > 0:
|
||||||
|
audio_mask[..., t - m:] = 0
|
||||||
|
latent["samples"] = comfy.nested_tensor.NestedTensor((video, silence))
|
||||||
|
latent["noise_mask"] = comfy.nested_tensor.NestedTensor(
|
||||||
|
(torch.ones_like(video[:, :1]), audio_mask))
|
||||||
|
return True
|
||||||
|
except Exception:
|
||||||
|
return False
|
||||||
@@ -0,0 +1,310 @@
|
|||||||
|
# H3-LongVideos -- https://github.com/Smite79/MiniMax-H3-LongVideos
|
||||||
|
# Copyright (c) 2026 Smite79. All rights reserved.
|
||||||
|
# Redistribution, in whole or in part, requires written permission.
|
||||||
|
# This notice may not be removed or altered. See LICENSE.
|
||||||
|
"""Decisions about which pictures may condition a shot."""
|
||||||
|
|
||||||
|
import torch
|
||||||
|
import node_helpers
|
||||||
|
from h3_runtime import (H3_FPS, AUDIO_LATENT_FPS, _empty_av_latent, _resize, ref_image_canvas,
|
||||||
|
frame_levels)
|
||||||
|
from h3_audio import _SILENCE_STATUS, _silent_audio_latent, _pin_audio_silence
|
||||||
|
|
||||||
|
|
||||||
|
def may_carry_room(previous_cast, current_cast, tagged_names):
|
||||||
|
"""A previous frame is safe as a reference only when it adds no subject."""
|
||||||
|
previous = [name for name in (previous_cast or ()) if name]
|
||||||
|
current = set(current_cast or ())
|
||||||
|
tagged = set(tagged_names or ())
|
||||||
|
return bool(previous) and all(name in current for name in previous) \
|
||||||
|
and not any(name in tagged for name in previous)
|
||||||
|
|
||||||
|
|
||||||
|
def may_carry_frame(previous_cast, current_cast, tagged_names):
|
||||||
|
"""A previous frame of the SAME room is safe as a reference claimed with everyone in it.
|
||||||
|
|
||||||
|
Unlike may_carry_room, somebody this shot does not describe may be in it: the claim
|
||||||
|
names them, and they are still in that room. Refused only for an empty frame, or
|
||||||
|
one holding somebody whose own portrait also rides this shot -- two pictures of one
|
||||||
|
person is how a second one gets drawn."""
|
||||||
|
previous = [name for name in (previous_cast or ()) if name]
|
||||||
|
current = set(current_cast or ())
|
||||||
|
tagged = set(tagged_names or ())
|
||||||
|
return bool(previous) and not any(name in tagged and name in current
|
||||||
|
for name in previous)
|
||||||
|
|
||||||
|
|
||||||
|
def recoverable_subject(cast, tagged_names, returning_names, captured):
|
||||||
|
"""Return the sole safe recovered subject, or an empty string."""
|
||||||
|
people = [name for name in (cast or ()) if name]
|
||||||
|
if len(people) != 1:
|
||||||
|
return ""
|
||||||
|
name = people[0]
|
||||||
|
return name if (name not in set(tagged_names or ())
|
||||||
|
and name in set(returning_names or ())
|
||||||
|
and captured.get(name) is not None) else ""
|
||||||
|
|
||||||
|
|
||||||
|
KEYFRAME_SAFE_AUG = 0.99 # below this, a ref aug would soften the keyframe too
|
||||||
|
|
||||||
|
# What ONE boundary is allowed to claim it measured. Wider than any real per-pass drift,
|
||||||
|
# narrow enough that a bad frame -- a flash, a cut to black, a frame the model lost --
|
||||||
|
# cannot swing the estimate. The median across boundaries does the real rejecting.
|
||||||
|
LEVEL_GAIN_CAP = 0.12 # in log-gain, so +-12.7% of contrast
|
||||||
|
LEVEL_OFFSET_CAP = 0.05
|
||||||
|
# The within-shot term is believed only when boundaries AGREE on its sign, and even then
|
||||||
|
# only this far: within-shot change is often the author's (a light switched off), so it is
|
||||||
|
# the half of the signal that cannot be trusted on its own.
|
||||||
|
LEVEL_SHOT_GAIN_CAP = 0.015
|
||||||
|
LEVEL_SHOT_OFFSET_CAP = 0.010
|
||||||
|
LEVEL_AGREE = 2.0 / 3.0
|
||||||
|
LEVEL_MIN_OBS = 3
|
||||||
|
# What the correction may do to one handoff, whatever it measured. A cut should not carry
|
||||||
|
# a visible grade step: shot N's last frame reaches the video uncorrected while N+1 is
|
||||||
|
# sampled from a corrected keyframe, so an uncapped correction trades burn-in for a pop at
|
||||||
|
# every join -- the same class of complaint, differently shaped.
|
||||||
|
LEVEL_GAIN_LO, LEVEL_GAIN_HI = 0.80, 1.25
|
||||||
|
LEVEL_OFFSET_BOUND = 0.02
|
||||||
|
# Below this a frame is too flat for a contrast RATIO to mean anything.
|
||||||
|
LEVEL_MIN_SIGMA = 0.01
|
||||||
|
|
||||||
|
|
||||||
|
class HandoffLevels:
|
||||||
|
"""Takes the grade the chain adds to itself back out of the handoff.
|
||||||
|
|
||||||
|
THE MEASUREMENT, which is the whole reason this needs no scene list. At every
|
||||||
|
boundary the render holds two pictures that are SUPPOSED to be the same frame: K,
|
||||||
|
the handoff it gave the shot, and R, frame one of what came back -- the model's own
|
||||||
|
reproduction of K, from a keyframe labelled sigma 0.001. Nothing was asked to change
|
||||||
|
between them, so everything separating them is the chain's own doing and none of it
|
||||||
|
is the author's intent. That is the one difference in the loop that can be corrected
|
||||||
|
without guessing at anybody's lighting, and R costs nothing to look at: it is the
|
||||||
|
frame trim_seam throws away.
|
||||||
|
|
||||||
|
A beat that walks into a darker room moves K, and R follows it there. So the level is
|
||||||
|
never anchored, never compared to shot 1, and never compared to a target -- only K
|
||||||
|
against its own reproduction, boundary by boundary.
|
||||||
|
|
||||||
|
WHAT IT WILL NOT FIX. Clipping already baked into earlier shots, because the VAE
|
||||||
|
clamps every decode and headroom spent is gone. Softening, which is a different
|
||||||
|
measurement and a different cause. Anything spatial -- ghosting, local burn, identity
|
||||||
|
drift. A tone curve with a knee in it, since this is affine per channel; the residual
|
||||||
|
in the report is how that would show itself. The first boundary, which has nothing to
|
||||||
|
measure yet. And a deliberate monotone move -- a film that dims every single beat --
|
||||||
|
loses a bounded, reported fraction of itself."""
|
||||||
|
|
||||||
|
def __init__(self):
|
||||||
|
self._bg, self._bo = [], [] # per boundary: K -> R, the chain's own drift
|
||||||
|
self._sg, self._so = [], [] # per shot: R -> last frame, believed only on agreement
|
||||||
|
self.applied = [] # (gain, offset) actually used, for the report
|
||||||
|
|
||||||
|
def observe(self, given, repro, last=None, pre_up_last=None):
|
||||||
|
"""Record one boundary. given is the keyframe this shot got, repro is frame one
|
||||||
|
of what it produced, last is its final frame, pre_up_last the handoff it hands on.
|
||||||
|
|
||||||
|
last/pre_up_last are how the pre-upscale handoff and the post-upscale output are
|
||||||
|
put in the same frame of reference: their difference IS the pipeline's own offset,
|
||||||
|
measured on one frame that went through both, so it can be subtracted from the
|
||||||
|
K->R reading instead of being mistaken for drift. With latent_upscale off they are
|
||||||
|
the same frame and the term is zero."""
|
||||||
|
gm, gs = frame_levels(given)
|
||||||
|
rm, rs = frame_levels(repro)
|
||||||
|
if gm is None or rm is None:
|
||||||
|
return False
|
||||||
|
if float(gs.min()) < LEVEL_MIN_SIGMA or float(rs.min()) < LEVEL_MIN_SIGMA:
|
||||||
|
return False
|
||||||
|
ug = torch.zeros(3)
|
||||||
|
uo = torch.zeros(3)
|
||||||
|
lm, ls = frame_levels(last) if last is not None else (None, None)
|
||||||
|
if pre_up_last is not None and lm is not None:
|
||||||
|
pm, ps = frame_levels(pre_up_last)
|
||||||
|
if pm is not None and float(ps.min()) >= LEVEL_MIN_SIGMA:
|
||||||
|
ug = torch.log(ls / ps)
|
||||||
|
uo = lm - pm
|
||||||
|
self._bg.append((torch.log(rs / gs) - ug).clamp(-LEVEL_GAIN_CAP, LEVEL_GAIN_CAP))
|
||||||
|
self._bo.append((rm - gm - uo).clamp(-LEVEL_OFFSET_CAP, LEVEL_OFFSET_CAP))
|
||||||
|
if lm is not None and float(ls.min()) >= LEVEL_MIN_SIGMA:
|
||||||
|
self._sg.append(torch.log(ls / rs))
|
||||||
|
self._so.append(lm - rm)
|
||||||
|
return True
|
||||||
|
|
||||||
|
def _agreed(self, rows, cap):
|
||||||
|
"""The median of rows, but only per channel where at least LEVEL_AGREE of them
|
||||||
|
share its sign. A within-shot change the boundaries disagree about is content, not
|
||||||
|
drift, and content must not be corrected."""
|
||||||
|
out = torch.zeros(3)
|
||||||
|
if len(rows) < LEVEL_MIN_OBS:
|
||||||
|
return out
|
||||||
|
st = torch.stack(rows)
|
||||||
|
med = st.median(dim=0).values
|
||||||
|
agree = ((st * med.sign().unsqueeze(0)) > 0).float().mean(dim=0)
|
||||||
|
keep = agree >= LEVEL_AGREE
|
||||||
|
return torch.where(keep, med.clamp(-cap, cap), out)
|
||||||
|
|
||||||
|
def estimate(self):
|
||||||
|
"""(gain_log, offset) the chain is drifting by per boundary, per channel."""
|
||||||
|
if not self._bg:
|
||||||
|
return None, None
|
||||||
|
g = torch.stack(self._bg).median(dim=0).values + self._agreed(self._sg, LEVEL_SHOT_GAIN_CAP)
|
||||||
|
o = torch.stack(self._bo).median(dim=0).values + self._agreed(self._so, LEVEL_SHOT_OFFSET_CAP)
|
||||||
|
return g, o
|
||||||
|
|
||||||
|
def gains(self, strength):
|
||||||
|
"""(gain, offset) as 3-vectors, or (None, None) when there is nothing worth doing.
|
||||||
|
|
||||||
|
Separate from note() because more than one frame leaves a shot -- the handoff,
|
||||||
|
and any face captured for a return several shots later -- and they have to carry
|
||||||
|
the SAME grade. A recovered face arriving at a different exposure from the shot
|
||||||
|
around it would be a new bug of exactly the kind this is fixing."""
|
||||||
|
g, o = self.estimate()
|
||||||
|
if g is None or strength <= 0:
|
||||||
|
return None, None
|
||||||
|
gain = torch.exp(-float(strength) * g).clamp(LEVEL_GAIN_LO, LEVEL_GAIN_HI)
|
||||||
|
off = (-float(strength) * o).clamp(-LEVEL_OFFSET_BOUND, LEVEL_OFFSET_BOUND)
|
||||||
|
# The next thing this frame meets is an 8-bit quantisation, so a correction under
|
||||||
|
# 1/255 would be erased on the way there. Claiming it would be worse than silence.
|
||||||
|
if float((gain - 1.0).abs().max()) < 1e-3 and float(off.abs().max()) < 1.0 / 255.0:
|
||||||
|
return None, None
|
||||||
|
return gain, off
|
||||||
|
|
||||||
|
def note(self, gain, off):
|
||||||
|
"""Record what was applied, and say it in one clause."""
|
||||||
|
self.applied.append((gain.clone(), off.clone()))
|
||||||
|
return (f"gain {'/'.join(f'{float(v):.3f}' for v in gain)} "
|
||||||
|
f"level {'/'.join(f'{float(v):+.4f}' for v in off)}")
|
||||||
|
|
||||||
|
def _keyframe_latent(vae, hand_img):
|
||||||
|
"""The keyframe latent for this shot: an ENCODE of the previous shot's last frame.
|
||||||
|
|
||||||
|
This was briefly an optimisation -- pass the previous shot's own latent straight
|
||||||
|
through and skip a VAE round trip per boundary. It was wrong, and it degraded
|
||||||
|
every shot after the first.
|
||||||
|
|
||||||
|
A keyframe is ONE pixel frame, and H3's grid puts that at 5f -> TWO latent
|
||||||
|
frames. Slicing [:, :, -1:] off a finished shot hands over one. Worse, the video
|
||||||
|
VAE is causal: the last latent of a 72-frame sequence encodes its temporal
|
||||||
|
context, not a standalone opening frame, so even at the right count it does not
|
||||||
|
mean what a keyframe means. The spatial-size guard could not see either problem.
|
||||||
|
|
||||||
|
The round trip is real but it is one lossy step on a correctly formed anchor,
|
||||||
|
which beats a cheap malformed one."""
|
||||||
|
return vae.encode(hand_img)
|
||||||
|
|
||||||
|
|
||||||
|
def _build_ref_images(vae, images, gen_w, gen_h, mode="match"):
|
||||||
|
"""(tokenizer items, DiT blocks) for a list of reference IMAGE tensors.
|
||||||
|
|
||||||
|
The tokenizer labels each one `<Picture N>:` itself, in the order given here --
|
||||||
|
so the roster the prompt refers to is decided by input order, not by anything
|
||||||
|
written in the prompt."""
|
||||||
|
items, blocks = [], []
|
||||||
|
for img in images:
|
||||||
|
if img is None:
|
||||||
|
continue
|
||||||
|
h, w = int(img.shape[1]), int(img.shape[2])
|
||||||
|
tw, th = ref_image_canvas(w, h, gen_w, gen_h, mode)
|
||||||
|
resized = _resize(img[:1], tw, th, "disabled")
|
||||||
|
items.append({"type": "image", "data": resized})
|
||||||
|
blocks.append({"kind": "image", "latent_h": th // 16, "latent_w": tw // 16,
|
||||||
|
"latent": vae.encode(resized)})
|
||||||
|
return items, blocks
|
||||||
|
|
||||||
|
|
||||||
|
def build_conditioning(clip, vae, audio_vae, prompt, width, height, length,
|
||||||
|
handoff=None, refs=None,
|
||||||
|
ref_noise_aug=0.999, silent=False, ref_image_size="match",
|
||||||
|
handoff_as_ref=False, speech_lead_seconds=0.0, speech_tail_frames=0):
|
||||||
|
"""Encode prompt, identity references, keyframe, and audio constraints for a shot."""
|
||||||
|
latent, fc = _empty_av_latent(width, height, length, H3_FPS)
|
||||||
|
refs = [r for r in (refs or []) if r is not None]
|
||||||
|
|
||||||
|
hand_img = None
|
||||||
|
if handoff is not None:
|
||||||
|
hand_img = _resize(handoff[:1], width, height, "disabled")
|
||||||
|
|
||||||
|
# REFERENCES AND THE KEYFRAME RIDE TOGETHER. This is the arrangement the node
|
||||||
|
# had before I broke it, and the reason is in ComfyUI's own layout:
|
||||||
|
#
|
||||||
|
# model_base.py:2183-2191 cond_video_latents = keyframe latents THEN ref latents
|
||||||
|
# model.py PackedLayout emits keyframe "cond" segments THEN ref "ref_img" ones
|
||||||
|
#
|
||||||
|
# The two orders agree, so both channels coexist. A shot takes its references AND
|
||||||
|
# a real keyframe: the keyframe ANCHORS the first frame, which is what continuity
|
||||||
|
# needs, while a reference only supplies identity. They are not alternatives.
|
||||||
|
#
|
||||||
|
# I had read "<Picture 1>" as MEANING the first frame on fl2va, and rearranged the
|
||||||
|
# roster around that. It does not. Which image is the first frame is decided by
|
||||||
|
# resolved_frame_index in minimax_keyframes, not by a label's number -- the labels
|
||||||
|
# are only how the images are shown to the VLM, and what they have to line up with
|
||||||
|
# is the <Picture N> tags in the prompt.
|
||||||
|
#
|
||||||
|
# So references come FIRST and keep slots 1..N, which is what a sheet line's
|
||||||
|
# `Name: <Picture 1>, ...` points at, and the handoff is appended AFTER them where
|
||||||
|
# it disturbs no numbering. It has to be in the list at all because
|
||||||
|
# tokenize_with_weights is either/or: passing minimax_ref_items makes it ignore
|
||||||
|
# `images` outright, so leaving the handoff out means the VLM is never shown where
|
||||||
|
# the shot left off and re-imagines the scenery -- same place, new room.
|
||||||
|
keyframe_ok = ref_noise_aug is None or float(ref_noise_aug) >= KEYFRAME_SAFE_AUG
|
||||||
|
# One aug covers every visual condition row, references AND the keyframe. Below
|
||||||
|
# KEYFRAME_SAFE_AUG the keyframe latent would be noised and labelled at the wrong
|
||||||
|
# timestep, so the handoff stops being an anchor and rides as an extra reference
|
||||||
|
# instead: weaker continuity, but nothing pretending to anchor while carrying noise.
|
||||||
|
# ...or because the caller asked for it. A shot that introduces somebody already
|
||||||
|
# in position wants the room this picture carries and NOT the first frame it
|
||||||
|
# would force, and that is a demotion the aug knows nothing about.
|
||||||
|
carry_as_ref = bool(hand_img is not None
|
||||||
|
and (handoff_as_ref or (refs and not keyframe_ok)))
|
||||||
|
|
||||||
|
enc_refs = refs + ([hand_img] if carry_as_ref else [])
|
||||||
|
items, blocks = ([], [])
|
||||||
|
if enc_refs:
|
||||||
|
items, blocks = _build_ref_images(vae, enc_refs, width, height, ref_image_size)
|
||||||
|
if hand_img is not None and not carry_as_ref:
|
||||||
|
items = items + [{"type": "image", "data": hand_img}]
|
||||||
|
|
||||||
|
if items:
|
||||||
|
tokens = clip.tokenize(prompt, minimax_ref_items=items)
|
||||||
|
else:
|
||||||
|
tokens = clip.tokenize(prompt)
|
||||||
|
cond = clip.encode_from_tokens_scheduled(tokens)
|
||||||
|
|
||||||
|
vals = {}
|
||||||
|
if blocks:
|
||||||
|
vals["minimax_refs"] = blocks
|
||||||
|
# How CLEAN the references are shown. One aug covers every conditioning
|
||||||
|
# latent, keyframe included -- which is why softening references below
|
||||||
|
# KEYFRAME_SAFE_AUG would soften the anchor too.
|
||||||
|
if ref_noise_aug is not None:
|
||||||
|
vals["minimax_visual_cond_noise_aug"] = float(ref_noise_aug)
|
||||||
|
|
||||||
|
kfs = []
|
||||||
|
if hand_img is not None and not carry_as_ref:
|
||||||
|
kfs.append({"resolved_frame_index": 0,
|
||||||
|
"latent": _keyframe_latent(vae, hand_img)})
|
||||||
|
# Audio keyframes are extra conditioning rows in H3's PackedLayout. Pin the
|
||||||
|
# generated target stream instead, so the joint model also sees a quiet mouth.
|
||||||
|
# A dialogue shot pins its opening (the lead) and, past the line's estimated end,
|
||||||
|
# its close (the tail); the span between is the model's.
|
||||||
|
if silent or float(speech_lead_seconds or 0.0) > 0.0 or int(speech_tail_frames or 0) > 0:
|
||||||
|
_SILENCE_STATUS["asked"] += 1
|
||||||
|
if audio_vae is None:
|
||||||
|
_SILENCE_STATUS["why"] = "no audio VAE is wired to the node"
|
||||||
|
else:
|
||||||
|
sil = _silent_audio_latent(audio_vae, fc, H3_FPS)
|
||||||
|
if sil is None:
|
||||||
|
_SILENCE_STATUS["why"] = ("the audio VAE would not encode a silent "
|
||||||
|
"second -- the wrong VAE is on the "
|
||||||
|
"audio_vae input")
|
||||||
|
else:
|
||||||
|
lead = None if silent else round(float(speech_lead_seconds) *
|
||||||
|
AUDIO_LATENT_FPS)
|
||||||
|
tail = 0 if silent else int(speech_tail_frames or 0)
|
||||||
|
if _pin_audio_silence(latent, sil, lead, tail):
|
||||||
|
_SILENCE_STATUS["applied"] += 1
|
||||||
|
else:
|
||||||
|
_SILENCE_STATUS["why"] = "the silent latent did not match the shot"
|
||||||
|
if kfs:
|
||||||
|
vals["minimax_keyframes"] = kfs
|
||||||
|
if vals:
|
||||||
|
cond = node_helpers.conditioning_set_values(cond, vals)
|
||||||
|
return cond, latent, fc, carry_as_ref
|
||||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,739 @@
|
|||||||
|
# H3-LongVideos -- https://github.com/Smite79/MiniMax-H3-LongVideos
|
||||||
|
# Copyright (c) 2026 Smite79. All rights reserved.
|
||||||
|
# Redistribution, in whole or in part, requires written permission.
|
||||||
|
# This notice may not be removed or altered. See LICENSE.
|
||||||
|
"""Sampling, decoding, resizing, memory handling, and frame assembly."""
|
||||||
|
|
||||||
|
import math
|
||||||
|
|
||||||
|
import torch
|
||||||
|
import nodes
|
||||||
|
import comfy.utils
|
||||||
|
import comfy.sample
|
||||||
|
import comfy.samplers
|
||||||
|
import comfy.nested_tensor
|
||||||
|
import comfy.model_management as mm
|
||||||
|
import latent_preview
|
||||||
|
|
||||||
|
|
||||||
|
class FrameAccumulator:
|
||||||
|
"""Build the final frame tensor once, retaining overflow only when necessary."""
|
||||||
|
|
||||||
|
def __init__(self, capacity, dtype, store_on_cpu):
|
||||||
|
self.capacity = int(capacity)
|
||||||
|
self.dtype = dtype
|
||||||
|
self.store_on_cpu = bool(store_on_cpu)
|
||||||
|
self.tensor = None
|
||||||
|
self.used = 0
|
||||||
|
self.overflow = []
|
||||||
|
|
||||||
|
def add(self, frames):
|
||||||
|
count = int(frames.shape[0])
|
||||||
|
if self.tensor is None and count:
|
||||||
|
device = torch.device("cpu") if self.store_on_cpu else frames.device
|
||||||
|
self.tensor = torch.empty(
|
||||||
|
(max(count, self.capacity),) + tuple(frames.shape[1:]),
|
||||||
|
dtype=self.dtype, device=device)
|
||||||
|
if (not self.overflow and self.tensor is not None
|
||||||
|
and self.used + count <= self.tensor.shape[0]):
|
||||||
|
self.tensor[self.used:self.used + count].copy_(frames)
|
||||||
|
self.used += count
|
||||||
|
return
|
||||||
|
self.overflow.append(frames.to("cpu", self.dtype, copy=True)
|
||||||
|
if self.store_on_cpu else frames)
|
||||||
|
|
||||||
|
def release(self):
|
||||||
|
"""Drop every tensor held, now, rather than whenever the collector gets to it.
|
||||||
|
|
||||||
|
On an interrupt the render unwinds through frames the collector tears down in
|
||||||
|
its own order, and a large video buffer freed after the models it was sized
|
||||||
|
against have already gone is a free the allocator cannot explain. Deliberately
|
||||||
|
does NOT empty the cache: that is another CUDA call, and if the context is
|
||||||
|
already in a sticky error state it is one more thing to abort inside."""
|
||||||
|
self.tensor = None
|
||||||
|
self.overflow = []
|
||||||
|
self.used = 0
|
||||||
|
|
||||||
|
def finish(self):
|
||||||
|
if not self.overflow:
|
||||||
|
if self.tensor is None:
|
||||||
|
return torch.cat(self.overflow, dim=0)
|
||||||
|
if self.used == self.tensor.shape[0]:
|
||||||
|
out = self.tensor
|
||||||
|
else:
|
||||||
|
# COMPACT, never a slice. A slice of a larger buffer keeps the WHOLE
|
||||||
|
# buffer's storage alive, which is the retention this class exists to
|
||||||
|
# prevent -- and test_the_chain_is_never_held_twice measures exactly
|
||||||
|
# that, demanding no unused bytes behind the returned tensor.
|
||||||
|
#
|
||||||
|
# There is slack because the capacity is now an upper bound: it can no
|
||||||
|
# longer assume trim_seam drops a frame at every seam, since a shot that
|
||||||
|
# opens on no keyframe keeps its first frame. Over-allocating by at most
|
||||||
|
# one frame per seam and compacting once is the bounded cost. The
|
||||||
|
# alternative -- an exact guess that can be too small -- drops into the
|
||||||
|
# overflow list, which with cleanup_between_shots off holds every shot's
|
||||||
|
# decoded frames live on the GPU until the end of the run.
|
||||||
|
out = torch.empty((self.used,) + tuple(self.tensor.shape[1:]),
|
||||||
|
dtype=self.dtype, device=self.tensor.device)
|
||||||
|
out.copy_(self.tensor[:self.used])
|
||||||
|
self.tensor = None
|
||||||
|
return out
|
||||||
|
|
||||||
|
extra = sum(int(piece.shape[0]) for piece in self.overflow)
|
||||||
|
reference = self.tensor if self.tensor is not None else self.overflow[0]
|
||||||
|
out = torch.empty((self.used + extra,) + tuple(reference.shape[1:]),
|
||||||
|
dtype=self.dtype, device=reference.device)
|
||||||
|
if self.tensor is not None and self.used:
|
||||||
|
out[:self.used].copy_(self.tensor[:self.used])
|
||||||
|
at = self.used
|
||||||
|
while self.overflow:
|
||||||
|
piece = self.overflow.pop(0)
|
||||||
|
count = int(piece.shape[0])
|
||||||
|
out[at:at + count].copy_(piece)
|
||||||
|
at += count
|
||||||
|
self.tensor = None
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
H3_FPS = 24 # H3 renders 24 fps, always
|
||||||
|
|
||||||
|
|
||||||
|
AUDIO_LATENT_FPS = 40 # audio latent frames per second
|
||||||
|
|
||||||
|
|
||||||
|
AUTO_TILE_T = 8 # temporal chunk for a tiled decode
|
||||||
|
|
||||||
|
|
||||||
|
MAX_FRAMES = 362 # H3's own ceiling (~15s)
|
||||||
|
|
||||||
|
|
||||||
|
CANVAS_MULTIPLE = 32
|
||||||
|
|
||||||
|
|
||||||
|
REF_IMAGE_SHORT_EDGE = 2048
|
||||||
|
|
||||||
|
|
||||||
|
def align_frame_count(n):
|
||||||
|
"""Up to the next valid H3 frame count. The grid is 17k+5."""
|
||||||
|
n = max(5, int(n))
|
||||||
|
while n % 17 != 5:
|
||||||
|
n += 1
|
||||||
|
return min(n, MAX_FRAMES)
|
||||||
|
|
||||||
|
|
||||||
|
def video_latent_t(fc):
|
||||||
|
return 2 if fc <= 5 else ((fc - 5) // 17) * 5 + 2
|
||||||
|
|
||||||
|
|
||||||
|
def temporal_shape(length, fps=H3_FPS):
|
||||||
|
"""(frame count, video latent frames, audio latent frames) for a shot.
|
||||||
|
|
||||||
|
`fps` is accepted but deliberately IGNORED: the audio latent has to line up
|
||||||
|
with 24 fps video or the shot's sound is stretched against its picture."""
|
||||||
|
fc = align_frame_count(length)
|
||||||
|
return fc, video_latent_t(fc), round(fc / H3_FPS * AUDIO_LATENT_FPS)
|
||||||
|
|
||||||
|
|
||||||
|
def ref_image_canvas(w, h, gen_w, gen_h, mode="match"):
|
||||||
|
"""Pure: the (width, height) a reference image is encoded at.
|
||||||
|
|
||||||
|
'match' scales it (DOWN only, aspect kept) to the generation's pixel area, so a
|
||||||
|
reference costs about as much as one frame of the shot. 'max' goes to the
|
||||||
|
reference pipeline's 2048 short edge for the best identity fidelity, which on a
|
||||||
|
long chain is several times slower because the rows are re-attended every step
|
||||||
|
of every shot. Never upscales: a small reference stays small."""
|
||||||
|
w, h = max(1, int(w)), max(1, int(h))
|
||||||
|
if mode == "max":
|
||||||
|
scale = min(1.0, REF_IMAGE_SHORT_EDGE / min(w, h))
|
||||||
|
else:
|
||||||
|
scale = min(1.0, math.sqrt((int(gen_w) * int(gen_h)) / float(w * h)))
|
||||||
|
snap = lambda v: max(CANVAS_MULTIPLE, round(v * scale / CANVAS_MULTIPLE) * CANVAS_MULTIPLE)
|
||||||
|
return snap(w), snap(h)
|
||||||
|
|
||||||
|
|
||||||
|
def _resize(image, width, height, crop):
|
||||||
|
s = image[..., :3].movedim(-1, 1)
|
||||||
|
s = comfy.utils.common_upscale(s, width, height, "lanczos", crop)
|
||||||
|
return s.movedim(1, -1)
|
||||||
|
|
||||||
|
|
||||||
|
def _empty_av_latent(width, height, length, fps, batch_size=1):
|
||||||
|
fc, lt, at = temporal_shape(length, fps)
|
||||||
|
video = torch.zeros([batch_size, 24, lt, height // 16, width // 16], device=mm.intermediate_device())
|
||||||
|
audio = torch.zeros([batch_size, 32, 2, at], device=mm.intermediate_device())
|
||||||
|
return {"samples": comfy.nested_tensor.NestedTensor((video, audio))}, fc
|
||||||
|
|
||||||
|
|
||||||
|
def _auto_tile_t(n_latent_frames, requested=None):
|
||||||
|
"""Temporal tile for a tiled decode. An explicit value wins.
|
||||||
|
|
||||||
|
The decode_tile_frames widget is gone, so this is where the value comes from
|
||||||
|
now. It has to come from somewhere: ComfyUI's decode_tiled_3d defaults tile_t
|
||||||
|
to 999, i.e. SPATIAL tiles only, and expanding the whole clip's time axis at
|
||||||
|
once is the single largest allocation in a run. A "tiled" decode that keeps the
|
||||||
|
full temporal extent barely lowers the peak, so the OOM retry that switches
|
||||||
|
tiling on was, without this, retrying with almost the same footprint."""
|
||||||
|
if requested:
|
||||||
|
return int(requested)
|
||||||
|
n = int(n_latent_frames or 0)
|
||||||
|
return AUTO_TILE_T if n > AUTO_TILE_T else None
|
||||||
|
|
||||||
|
|
||||||
|
def _decode_video(vae, out_latent, tiled, free_first=None, tile_t=None, tile_xy=None,
|
||||||
|
keep=()):
|
||||||
|
"""Decode the video latent.
|
||||||
|
|
||||||
|
`free_first` is the diffusion model: sampling is finished, and the video VAE
|
||||||
|
needs the room for THIS decode -- the free runs immediately before it, not to
|
||||||
|
make room for the next shot. On a card where the DiT is most of the VRAM, the
|
||||||
|
decode does not fit until it goes.
|
||||||
|
|
||||||
|
`keep` is what must NOT be evicted on the way. It was `keep_loaded=[]`, which
|
||||||
|
unloaded every resident model -- including the video VAE, which ComfyUI then
|
||||||
|
reloaded three lines later to run the decode. An evict-and-reload of the thing
|
||||||
|
about to be used, once per shot, on every card. Peak VRAM is identical either
|
||||||
|
way, since the VAE has to be resident to decode; the round trip was pure cost.
|
||||||
|
|
||||||
|
memory_required is ASKED FOR HONESTLY, which it was not. It was 1e30, and
|
||||||
|
free_memory computes `memory_to_free = memory_required - get_free_memory(device)`
|
||||||
|
(model_management.py:887), so 1e30 means "unload everything not in keep_loaded",
|
||||||
|
every shot, in full -- skipping partially_unload entirely.
|
||||||
|
|
||||||
|
What that evicts is the DiT, three lines before the next shot needs it again. On
|
||||||
|
a machine whose RAM is already full of finished frames there is nowhere for it to
|
||||||
|
go but disk, so the reload is a read from the drive, once per shot. Reported as
|
||||||
|
thrashing that slows the preload, and it is exactly that: the same weights being
|
||||||
|
read back at every boundary.
|
||||||
|
|
||||||
|
The VAE knows what its own decode costs -- ComfyUI sizes it with
|
||||||
|
memory_used_decode and uses that number everywhere else. Asked for that instead,
|
||||||
|
a card with headroom frees NOTHING and the DiT simply stays. A card without
|
||||||
|
headroom frees what it needs and no more, which is what partially_unload is for.
|
||||||
|
1e30 remains the fallback for a VAE that cannot estimate itself."""
|
||||||
|
latent = out_latent["samples"]
|
||||||
|
if latent.is_nested:
|
||||||
|
latent = latent.unbind()[0]
|
||||||
|
if free_first is not None:
|
||||||
|
try:
|
||||||
|
mm.free_memory(_decode_headroom(vae, latent), mm.get_torch_device(),
|
||||||
|
keep_loaded=_resident(keep or (vae,)))
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
# A VAE THAT ALREADY TILES DOES NOT NEED TO BE ASKED TO, AND ASKING COSTS 3x.
|
||||||
|
#
|
||||||
|
# MiniMaxH3VideoVAE.decode_tiled is, in full:
|
||||||
|
#
|
||||||
|
# def decode_tiled(self, z, **kwargs):
|
||||||
|
# return self.decode(z)
|
||||||
|
#
|
||||||
|
# Every tile_t/overlap_t/tile_x/tile_y this function computes is discarded, so
|
||||||
|
# the tiling the widget promises is not happening here -- the model tiles
|
||||||
|
# internally either way (256px spatial, 17-frame temporal), which is why
|
||||||
|
# comfy/sd.py sets handles_tiling on it.
|
||||||
|
#
|
||||||
|
# What the detour costs is the OUTPUT BUFFER. comfy's VAE.decode preallocates
|
||||||
|
# ONE result at vae_output_dtype and hands it to the model as output_buffer=,
|
||||||
|
# and MiniMaxH3VideoVAE.decode_temporal writes finalized chunks straight into
|
||||||
|
# it. Going through decode_tiled instead reaches _decode_tiled_owned, which
|
||||||
|
# calls the model with output_buffer=None -- so decode_temporal allocates its
|
||||||
|
# own at torch.float32 -- and then makes an fp16 `copy=True` of that. Two
|
||||||
|
# buffers, the larger of them at double width:
|
||||||
|
#
|
||||||
|
# tiled : fp32 2.60GB + fp16 copy 1.30GB = 3.90GB per shot
|
||||||
|
# decode: one preallocated fp16 = 1.30GB per shot
|
||||||
|
#
|
||||||
|
# at 362 frames of 1056x608. Every shot, on the node's own default.
|
||||||
|
#
|
||||||
|
# So: when the VAE owns its tiling AND can be written into, the un-tiled call IS
|
||||||
|
# the tiled one, minus the copies. Anything else keeps the old path -- this is a
|
||||||
|
# detour around a detour, not a claim that tiling is useless.
|
||||||
|
_owns_tiling = bool(getattr(vae, "handles_tiling", False) and getattr(
|
||||||
|
getattr(vae, "first_stage_model", None), "comfy_has_chunked_io", False))
|
||||||
|
if tiled and _owns_tiling:
|
||||||
|
imgs = vae.decode(latent)
|
||||||
|
elif tiled:
|
||||||
|
# Temporal + spatial tiling. Without tile_t the VAE expands the WHOLE latent
|
||||||
|
# clip at once, which on a 243-frame 1344x768 shot is the single largest
|
||||||
|
# allocation in the run -- and on an unpruned checkpoint that is already
|
||||||
|
# streaming, it is what tips the card over. Decoding in temporal chunks
|
||||||
|
# trades a little speed for a much lower peak; None keeps ComfyUI's defaults.
|
||||||
|
args = {}
|
||||||
|
tile_t = _auto_tile_t(latent.shape[2] if latent.ndim >= 5 else 0, tile_t)
|
||||||
|
if tile_t:
|
||||||
|
args["tile_t"] = int(tile_t)
|
||||||
|
args["overlap_t"] = max(1, int(tile_t) // 8)
|
||||||
|
if tile_xy:
|
||||||
|
args["tile_x"] = int(tile_xy)
|
||||||
|
args["tile_y"] = int(tile_xy)
|
||||||
|
try:
|
||||||
|
imgs = vae.decode_tiled(latent, **args) if args else vae.decode_tiled(latent)
|
||||||
|
except TypeError:
|
||||||
|
imgs = vae.decode_tiled(latent) # older signature without tile_t
|
||||||
|
else:
|
||||||
|
imgs = vae.decode(latent)
|
||||||
|
if len(imgs.shape) == 5:
|
||||||
|
imgs = imgs.reshape(-1, imgs.shape[-3], imgs.shape[-2], imgs.shape[-1])
|
||||||
|
return imgs
|
||||||
|
|
||||||
|
|
||||||
|
def _decode_audio(audio_vae, out_latent):
|
||||||
|
latent = out_latent["samples"]
|
||||||
|
if latent.is_nested:
|
||||||
|
latent = latent.unbind()[-1]
|
||||||
|
audio = audio_vae.decode(latent).movedim(-1, 1)
|
||||||
|
std = torch.std(audio, dim=[1, 2], keepdim=True) * 5.0
|
||||||
|
std[std < 1.0] = 1.0
|
||||||
|
audio = audio / std
|
||||||
|
sr = getattr(audio_vae, "audio_sample_rate_output", getattr(audio_vae, "audio_sample_rate", 44100))
|
||||||
|
return {"waveform": audio, "sample_rate": sr}
|
||||||
|
|
||||||
|
|
||||||
|
def _is_oom(e):
|
||||||
|
return isinstance(e, torch.cuda.OutOfMemoryError) or "out of memory" in str(e).lower()
|
||||||
|
|
||||||
|
|
||||||
|
def _deep_cleanup():
|
||||||
|
"""Release cached VRAM between shots so a long chain does not accumulate and OOM.
|
||||||
|
|
||||||
|
It unloads NOTHING. soft_empty_cache(force) ignores `force` in current ComfyUI
|
||||||
|
(model_management.py:2050) -- the body only reaches empty_cache() and
|
||||||
|
ipc_collect() -- so this drops cached blocks, not models. The `True` is kept
|
||||||
|
only for older builds that read it; the older comment here claimed this took an
|
||||||
|
unload_all_models path, and it does not."""
|
||||||
|
try:
|
||||||
|
mm.soft_empty_cache(True)
|
||||||
|
except TypeError:
|
||||||
|
mm.soft_empty_cache()
|
||||||
|
try:
|
||||||
|
if torch.cuda.is_available():
|
||||||
|
torch.cuda.empty_cache()
|
||||||
|
torch.cuda.ipc_collect()
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
DECODE_HEADROOM = 1.25 # over ComfyUI's own estimate, for working allocations
|
||||||
|
|
||||||
|
|
||||||
|
SAMPLE_HEADROOM = 1.35 # likewise for sampling, which is the longer stretch
|
||||||
|
|
||||||
|
|
||||||
|
def _decode_headroom(vae, latent):
|
||||||
|
"""VRAM this decode actually needs, by the VAE's own estimate. 1e30 if unknown.
|
||||||
|
|
||||||
|
ComfyUI sizes every VAE with memory_used_decode and uses that number itself, so
|
||||||
|
it is the honest figure to hand free_memory. The alternative -- and what was here
|
||||||
|
-- is 1e30, which means "unload everything" and evicts the DiT before every
|
||||||
|
decode, three lines before the next shot reloads it.
|
||||||
|
|
||||||
|
1e30 on failure rather than 0: a bad estimate that frees too little turns a slow
|
||||||
|
render into an OOM, and a wrong guess should fall back to the behaviour that has
|
||||||
|
been running, not to no freeing at all."""
|
||||||
|
try:
|
||||||
|
dtype = getattr(vae, "vae_dtype", None) or latent.dtype
|
||||||
|
need = float(vae.memory_used_decode(tuple(latent.shape), dtype))
|
||||||
|
if need > 0:
|
||||||
|
return need * DECODE_HEADROOM
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
return 1e30
|
||||||
|
|
||||||
|
|
||||||
|
def _resident(models):
|
||||||
|
"""The LoadedModel entries ComfyUI currently holds for `models`.
|
||||||
|
|
||||||
|
That is the form free_memory's keep_loaded wants: it compares against the
|
||||||
|
entries in current_loaded_models, not against the ModelPatcher objects a node
|
||||||
|
is holding. Anything not matched is simply not kept, so a model that is not
|
||||||
|
resident costs nothing here."""
|
||||||
|
out = []
|
||||||
|
for lm in list(getattr(mm, "current_loaded_models", [])):
|
||||||
|
for m in models or ():
|
||||||
|
if m is None:
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
if lm.model is m or getattr(lm, "model", None) is getattr(m, "model", None):
|
||||||
|
if lm not in out:
|
||||||
|
out.append(lm)
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def _image_out_dtype():
|
||||||
|
"""The dtype ComfyUI itself hands between nodes on THIS install.
|
||||||
|
|
||||||
|
The join used to end in a hard-coded .float(), commented "back to what every
|
||||||
|
downstream node expects". That was true when it was written and is not a
|
||||||
|
constant: ComfyUI has --fp16-intermediates, and on an install running it the
|
||||||
|
VAE's own decode already returns fp16 -- VAE.vae_output_dtype() IS
|
||||||
|
model_management.intermediate_dtype() (comfy/sd.py) -- as do EmptyLatentImage
|
||||||
|
and the rest of nodes.py. So on that install the node was taking frames the
|
||||||
|
VAE handed it in fp16, widening them to fp32 nothing had asked for, and
|
||||||
|
handing them to nodes whose own convention is fp16.
|
||||||
|
|
||||||
|
It is the largest thing this node holds, so the widening is not free: the
|
||||||
|
2580-frame chain costed at the join is 9.3GB as fp16 and 18.5GB as fp32,
|
||||||
|
against 44.6GB of staged weights on a 62GB machine -- which is the difference
|
||||||
|
between the render finishing and the OOM killer taking the server. Reported as
|
||||||
|
exactly that, twice.
|
||||||
|
|
||||||
|
Asked, not assumed, and never widened: whatever ComfyUI says it wants between
|
||||||
|
nodes is what the chain is built in. An install with the flag off is told
|
||||||
|
float32 and gets float32, byte for byte what it got before. Older builds have
|
||||||
|
no intermediate_dtype at all, so the fallback is the old constant."""
|
||||||
|
try:
|
||||||
|
return mm.intermediate_dtype()
|
||||||
|
except Exception:
|
||||||
|
return torch.float32
|
||||||
|
|
||||||
|
|
||||||
|
def _evict_all_but(keep_model, latent=None):
|
||||||
|
"""Unload every model EXCEPT the diffusion model from the GPU.
|
||||||
|
|
||||||
|
This is the fix for VRAM ratcheting across a long chain. soft_empty_cache()
|
||||||
|
only drops the CUDA allocator's cached blocks -- it does NOT unload models, so
|
||||||
|
ComfyUI keeps the Qwen3-VL text encoder (~14.6GB) and both VAEs resident in
|
||||||
|
current_loaded_models alongside the DiT. Each shot re-encodes the prompt
|
||||||
|
(text encoder), encodes the handoff keyframe (video VAE), then samples (DiT),
|
||||||
|
so all three compete for the card.
|
||||||
|
|
||||||
|
ComfyUI does free ahead of each load -- load_models_gpu() calls free_memory()
|
||||||
|
for what it is about to need (model_management.py:975), so the weight path is
|
||||||
|
not purely reactive. What it cannot size for is a long chain's ACTIVATIONS on
|
||||||
|
a card where the DiT is most of the VRAM. Freeing explicitly, right after
|
||||||
|
conditioning is built and before sampling, keeps only what the sampler needs.
|
||||||
|
|
||||||
|
ASKED FOR HONESTLY, and this is the expensive one. free_memory computes
|
||||||
|
`memory_to_free = memory_required - get_free_memory(device)`, so 1e30 meant
|
||||||
|
"unload everything but the DiT" on every shot, unconditionally -- on a 48GB card
|
||||||
|
with room for all of it as readily as on a 16GB one. What it unloads is the
|
||||||
|
~14.6GB text encoder and both VAEs, and the next shot re-encodes the prompt and
|
||||||
|
the handoff keyframe, so all three come straight back. On a machine whose RAM is
|
||||||
|
already full of finished frames they come back from DISK, once per shot, which is
|
||||||
|
the thrashing this was reported as.
|
||||||
|
|
||||||
|
The DiT can size its own activations -- memory_required(shape) is what ComfyUI
|
||||||
|
itself calls before a load -- so ask for that. A card with room frees nothing and
|
||||||
|
keeps the encoder resident; a card without frees exactly as much as it must.
|
||||||
|
1e30 stays the fallback, because a bad estimate that frees too little turns a
|
||||||
|
slow render into an OOM."""
|
||||||
|
need = 1e30
|
||||||
|
try:
|
||||||
|
if latent is not None:
|
||||||
|
shape = latent["samples"].shape if isinstance(latent, dict) else latent.shape
|
||||||
|
need = float(keep_model.model.memory_required(tuple(shape))) * SAMPLE_HEADROOM
|
||||||
|
if not (need > 0):
|
||||||
|
need = 1e30
|
||||||
|
except Exception:
|
||||||
|
need = 1e30
|
||||||
|
try:
|
||||||
|
mm.free_memory(need, mm.get_torch_device(),
|
||||||
|
keep_loaded=_resident([keep_model]))
|
||||||
|
except Exception:
|
||||||
|
try:
|
||||||
|
mm.soft_empty_cache(True)
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
def _sample_on_sigmas(model, seed, cfg, sampler_name, positive, negative, latent, sigmas):
|
||||||
|
"""common_ksampler, driven by an EXTERNAL sigma schedule.
|
||||||
|
|
||||||
|
common_ksampler derives its sigmas from (sampler_name, scheduler, steps, denoise)
|
||||||
|
and takes no schedule argument, so a schedule computed anywhere else cannot
|
||||||
|
reach it. Under PDD that is fatal rather than merely inconvenient: the heads
|
||||||
|
accept only their nine trained boundaries, and re-deriving the grid from
|
||||||
|
widgets means hitting it by coincidence and losing it again the moment a step
|
||||||
|
count changes.
|
||||||
|
|
||||||
|
Mirrors nodes.common_ksampler's noise / mask / callback handling exactly -- the
|
||||||
|
only substitution is comfy.sample.sample_custom for comfy.sample.sample."""
|
||||||
|
latent_image = latent["samples"]
|
||||||
|
latent_image = comfy.sample.fix_empty_latent_channels(
|
||||||
|
model, latent_image,
|
||||||
|
latent.get("downscale_ratio_spacial", None),
|
||||||
|
latent.get("downscale_ratio_temporal", None))
|
||||||
|
noise = comfy.sample.prepare_noise(latent_image, seed, latent.get("batch_index"))
|
||||||
|
# `steps` here only sizes the progress bar -- the schedule is `sigmas`, whose
|
||||||
|
# step count is one less than its length (the trailing 0.0 is an endpoint).
|
||||||
|
callback = latent_preview.prepare_callback(model, max(len(sigmas) - 1, 1))
|
||||||
|
samples = comfy.sample.sample_custom(
|
||||||
|
model, noise, cfg, comfy.samplers.sampler_object(sampler_name), sigmas,
|
||||||
|
positive, negative, latent_image,
|
||||||
|
noise_mask=latent.get("noise_mask"), callback=callback,
|
||||||
|
disable_pbar=not comfy.utils.PROGRESS_BAR_ENABLED, seed=seed)
|
||||||
|
out = latent.copy()
|
||||||
|
out.pop("downscale_ratio_spacial", None)
|
||||||
|
out.pop("downscale_ratio_temporal", None)
|
||||||
|
out["samples"] = samples
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
RESIZE_CHUNK = 32
|
||||||
|
|
||||||
|
|
||||||
|
def _stream_chunks(total):
|
||||||
|
"""A collector that writes upscaled chunks into ONE destination as they land.
|
||||||
|
|
||||||
|
Both chunk loops in _upscale_frames used `out.append(...)` then
|
||||||
|
`frames = torch.cat(out, dim=0)`. That is the shape the finished-chain join was
|
||||||
|
rebuilt to stop, at a LARGER size: the list holds the whole upscaled chain and
|
||||||
|
the cat allocates a second one, both live at the cat, and `out` is a local that
|
||||||
|
is never cleared -- so it survives the cat, survives the trailing resize, and is
|
||||||
|
still bound at the return. Meanwhile the CALLER's pre-upscale chain cannot be
|
||||||
|
dropped either, because `part = frames[s:s+batch]` is a view into it.
|
||||||
|
|
||||||
|
At 2580 frames of 1056x608 that is 9.26GB per copy per doubling: 37GB x2 at 2x,
|
||||||
|
and 148GB x2 with the RealESRGAN_x4plus that is sitting in models/upscale_models.
|
||||||
|
Preallocating from the first chunk and copying into it removes exactly one of
|
||||||
|
those two, and drops the list at the same time.
|
||||||
|
|
||||||
|
The destination is sized from the FIRST chunk, so the model's scale factor does
|
||||||
|
not have to be known in advance, and the frame count is the caller's own -- an
|
||||||
|
upscaler changes width and height, never the number of frames."""
|
||||||
|
state = {"dst": None, "at": 0}
|
||||||
|
|
||||||
|
def put(piece):
|
||||||
|
if state["dst"] is None:
|
||||||
|
state["dst"] = torch.empty((int(total),) + tuple(piece.shape[1:]),
|
||||||
|
dtype=piece.dtype, device=piece.device)
|
||||||
|
k = int(piece.shape[0])
|
||||||
|
end = min(state["at"] + k, state["dst"].shape[0])
|
||||||
|
if end > state["at"]:
|
||||||
|
state["dst"][state["at"]:end].copy_(piece[:end - state["at"]])
|
||||||
|
state["at"] = end
|
||||||
|
|
||||||
|
def done():
|
||||||
|
d, at = state["dst"], state["at"]
|
||||||
|
if d is None:
|
||||||
|
return None
|
||||||
|
return d if at == d.shape[0] else d[:at]
|
||||||
|
|
||||||
|
return put, done
|
||||||
|
|
||||||
|
|
||||||
|
def _resize_short_edge(frames, target, method="lanczos", chunk=0):
|
||||||
|
"""Resize a [B,H,W,C] frame batch so its short edge == target (keeping aspect,
|
||||||
|
snapped to /32). Plain high-quality resize -- enlarges, doesn't add detail.
|
||||||
|
|
||||||
|
IN CHUNKS, BECAUSE LANCZOS IS FOUR FULL-LENGTH COPIES. The whole chain went
|
||||||
|
into one common_upscale call, and comfy.utils.lanczos is three successive list
|
||||||
|
comprehensions over every frame at once:
|
||||||
|
|
||||||
|
images = [Image.fromarray(...) for image in samples] # N at source size
|
||||||
|
images = [image.resize(...) for image in images] # N at target size
|
||||||
|
images = [torch.from_numpy(np.array(im).astype(np.float32)/255.) ...]
|
||||||
|
result = torch.stack(images)
|
||||||
|
return result.to(samples.device, samples.dtype)
|
||||||
|
|
||||||
|
A comprehension builds the new list completely before rebinding the name, so at
|
||||||
|
each rebind BOTH are live; then torch.stack allocates a full copy while its list
|
||||||
|
still exists, and .to() allocates the result while the stack still exists. Note
|
||||||
|
the astype(np.float32): the input is fp16 but the two largest transients are at
|
||||||
|
DOUBLE its width. At 2580 frames to a 1080 short edge that peaked around 147GB
|
||||||
|
to produce a 29GB result, and it fires on a DOWNSCALE too.
|
||||||
|
|
||||||
|
Chunked, the peak is the result plus one chunk's worth of that machinery. It is
|
||||||
|
bit-identical: PIL resizes each frame independently, so per-chunk and per-chain
|
||||||
|
give the same pixels. The early return for an already-correct size is kept, so
|
||||||
|
the common no-op case still allocates nothing."""
|
||||||
|
b, h, w, c = frames.shape
|
||||||
|
if min(h, w) == target:
|
||||||
|
return frames
|
||||||
|
if h <= w:
|
||||||
|
nh = target; nw = max(32, int(round(target * w / h / 32) * 32))
|
||||||
|
else:
|
||||||
|
nw = target; nh = max(32, int(round(target * h / w / 32) * 32))
|
||||||
|
step = max(1, int(chunk) or RESIZE_CHUNK)
|
||||||
|
out = torch.empty((b, nh, nw, c), dtype=frames.dtype, device=frames.device)
|
||||||
|
for i in range(0, b, step):
|
||||||
|
part = comfy.utils.common_upscale(
|
||||||
|
frames[i:i + step].movedim(-1, 1), nw, nh, method, "disabled")
|
||||||
|
out[i:i + step].copy_(part.movedim(1, -1))
|
||||||
|
del part
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def _upscale_frames(frames, mode, model_name, target_short_edge, batch=4):
|
||||||
|
"""Optional post-pass upscale of the finished frames (on CPU).
|
||||||
|
mode 'model' : run a ComfyUI upscale model (Real-ESRGAN/UltraSharp class)
|
||||||
|
via the registered loader+apply nodes, chunked with cleanup
|
||||||
|
so 2000+ frames don't OOM; then fit to target short edge.
|
||||||
|
mode 'rtx' : NVIDIA RTX Video Super Resolution (Tensor Cores; fastest,
|
||||||
|
best quality for video -- needs Nvidia_RTX_Nodes_ComfyUI).
|
||||||
|
mode 'lanczos' : plain high-quality resize to the target short edge.
|
||||||
|
Any failure falls back to lanczos (or the raw frames), so it never breaks a
|
||||||
|
render. Returns (frames, note). NOTE: this SHARPENS/ENLARGES; it does not
|
||||||
|
reconstruct video detail the way a second-model (LTX 2.3) pass does."""
|
||||||
|
if mode == "off" or frames is None or getattr(frames, "shape", [0])[0] == 0:
|
||||||
|
return frames, ""
|
||||||
|
note = ""
|
||||||
|
if mode == "rtx":
|
||||||
|
# NVIDIA RTX Video Super Resolution (Comfy-Org/Nvidia_RTX_Nodes_ComfyUI).
|
||||||
|
# Runs on RTX Tensor Cores -- far faster than ESRGAN-class models and
|
||||||
|
# generally cleaner on video, though like them it enhances/enlarges rather
|
||||||
|
# than reconstructing detail (an LTX 2.3 re-generation does that).
|
||||||
|
try:
|
||||||
|
rtx = (_find_node(["rtx", "video", "super"]) or _find_node(["rtxvideosuperresolution"])
|
||||||
|
or _find_node(["rtx", "upscale"]))
|
||||||
|
if rtx is None:
|
||||||
|
raise RuntimeError("RTX node not installed (Nvidia_RTX_Nodes_ComfyUI)")
|
||||||
|
scale = 2
|
||||||
|
if target_short_edge and int(target_short_edge) > 0:
|
||||||
|
cur = min(frames.shape[1], frames.shape[2])
|
||||||
|
if cur > 0:
|
||||||
|
scale = max(1, min(4, int(round(int(target_short_edge) / cur))))
|
||||||
|
_put, _done = _stream_chunks(frames.shape[0])
|
||||||
|
n = frames.shape[0]
|
||||||
|
step = max(1, int(batch))
|
||||||
|
for st in range(0, n, step):
|
||||||
|
part = frames[st:st + step]
|
||||||
|
res = None
|
||||||
|
for kw in ({"image": part, "scale": scale}, {"images": part, "scale": scale},
|
||||||
|
{"image": part, "scale_factor": scale}, {"image": part}):
|
||||||
|
try:
|
||||||
|
res = _invoke_node(rtx, **kw); break
|
||||||
|
except TypeError:
|
||||||
|
continue
|
||||||
|
if res is None:
|
||||||
|
raise RuntimeError("RTX node signature not recognized")
|
||||||
|
_put(res.detach().to("cpu"))
|
||||||
|
del res, part
|
||||||
|
_deep_cleanup()
|
||||||
|
frames = _done()
|
||||||
|
note = f"RTX Video Super Resolution x{scale}"
|
||||||
|
if target_short_edge and int(target_short_edge) > 0:
|
||||||
|
frames = _resize_short_edge(frames, int(target_short_edge))
|
||||||
|
note += f"; fit to {int(target_short_edge)}px short edge"
|
||||||
|
return frames, note
|
||||||
|
except Exception as e:
|
||||||
|
mode = "model"
|
||||||
|
note = f"RTX upscale unavailable ({e}); fell back to model/lanczos"
|
||||||
|
if mode == "model" and model_name and model_name != "none":
|
||||||
|
try:
|
||||||
|
loader = _find_node(["upscale", "model", "load"]) or _find_node(["loadupscalemodel"])
|
||||||
|
applier = _find_node(["imageupscale", "model"]) or _find_node(["upscaleimageusingmodel"])
|
||||||
|
if loader is None or applier is None:
|
||||||
|
raise RuntimeError("upscale-model nodes not found")
|
||||||
|
up_model = _invoke_node(loader, model_name=model_name)
|
||||||
|
_put, _done = _stream_chunks(frames.shape[0])
|
||||||
|
n = frames.shape[0]
|
||||||
|
for s in range(0, n, max(1, int(batch))):
|
||||||
|
part = frames[s:s + max(1, int(batch))]
|
||||||
|
res = _invoke_node(applier, upscale_model=up_model, image=part)
|
||||||
|
_put(res.detach().to("cpu"))
|
||||||
|
del res, part
|
||||||
|
_deep_cleanup()
|
||||||
|
frames = _done()
|
||||||
|
note = f"upscaled with {model_name}"
|
||||||
|
except Exception as e:
|
||||||
|
mode = "lanczos"
|
||||||
|
note = f"model upscale unavailable ({e}); used lanczos"
|
||||||
|
if target_short_edge and int(target_short_edge) > 0:
|
||||||
|
try:
|
||||||
|
frames = _resize_short_edge(frames, int(target_short_edge))
|
||||||
|
note = (note + "; " if note else "") + f"fit to {int(target_short_edge)}px short edge"
|
||||||
|
except Exception as e:
|
||||||
|
note = (note + "; " if note else "") + f"resize failed ({e})"
|
||||||
|
elif mode == "lanczos" and not note:
|
||||||
|
note = "lanczos selected but no target set -> unchanged"
|
||||||
|
return frames, note
|
||||||
|
|
||||||
|
|
||||||
|
def _find_node(substrings):
|
||||||
|
"""Find a registered node whose key contains all of `substrings` (lowercased)."""
|
||||||
|
maps = getattr(nodes, "NODE_CLASS_MAPPINGS", {}) or {}
|
||||||
|
for k, v in maps.items():
|
||||||
|
kl = k.lower()
|
||||||
|
if all(s in kl for s in substrings):
|
||||||
|
return v
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def _invoke_node(cls, **kwargs):
|
||||||
|
"""Call a registered ComfyUI node (V1 FUNCTION or V3 execute) with kwargs and
|
||||||
|
return its first output. Used to reuse ComfyUI's own upscale-model loader/apply
|
||||||
|
so we don't reimplement spandrel loading or tiled scaling."""
|
||||||
|
inst = cls()
|
||||||
|
fn = None
|
||||||
|
if getattr(cls, "FUNCTION", None) and hasattr(inst, cls.FUNCTION):
|
||||||
|
fn = getattr(inst, cls.FUNCTION)
|
||||||
|
else:
|
||||||
|
for cand in ("execute", "upscale", "load_model", "load"):
|
||||||
|
if hasattr(inst, cand):
|
||||||
|
fn = getattr(inst, cand); break
|
||||||
|
if fn is None:
|
||||||
|
raise RuntimeError("no callable entrypoint")
|
||||||
|
out = fn(**kwargs)
|
||||||
|
out = getattr(out, "result", out)
|
||||||
|
return out[0] if isinstance(out, (tuple, list)) else out
|
||||||
|
|
||||||
|
|
||||||
|
# --- THE GRADE THE CHAIN ADDS TO ITSELF -------------------------------------
|
||||||
|
# Every shot boundary decodes a shot, hands its LAST frame over, and re-encodes that as
|
||||||
|
# the next shot's keyframe. The distill reproduces the keyframe faithfully enough to
|
||||||
|
# inherit whatever is already in it and SYNTHESISES frame 0 rather than copying it, so
|
||||||
|
# its own bias lands on top: S_next = a*S + b, a near 1, b above 0. Linear at best,
|
||||||
|
# geometric at worst, invisible shot to shot. And the VAE hard-clips every decode to
|
||||||
|
# 0..1, which makes the expansion a RATCHET -- headroom spent is not recoverable, so it
|
||||||
|
# shows as crushed blacks and blown highlights rather than merely as more contrast.
|
||||||
|
#
|
||||||
|
# These two are the measurement and the correction. Both work per colour channel,
|
||||||
|
# because the clip is per channel: the VAE un-whitens with ImageNet stds before it
|
||||||
|
# clamps, so the 0..1 rails sit at different distances in each channel and the blue
|
||||||
|
# floor and red ceiling bite first. A single luma number would miss the colour half.
|
||||||
|
LEVEL_POOL = 64 # cells per axis the level statistics are measured on
|
||||||
|
|
||||||
|
|
||||||
|
def frame_levels(img):
|
||||||
|
"""(mean, std) per colour channel for one frame, as 3-vectors, or (None, None).
|
||||||
|
|
||||||
|
Area-pooled to LEVEL_POOL first, so a pre-upscale frame and an upscaled one can be
|
||||||
|
compared: pooling measures the PICTURE's levels rather than its resolution. Measured
|
||||||
|
across a 2x resize, std agrees to 0.28% on picture-like content -- and to only 15%
|
||||||
|
on pure noise, because pooling cannot preserve variance that lives entirely at the
|
||||||
|
pixel scale. Real frames are the former, and whatever residual there is cancels
|
||||||
|
anyway: the caller measures the same pipeline difference separately and subtracts it.
|
||||||
|
|
||||||
|
float32 throughout, deliberately: these frames are fp16 under
|
||||||
|
--fp16-intermediates, and an fp16 mean accumulated over a 1344x768 frame biases
|
||||||
|
badly enough to matter at the sizes being corrected here."""
|
||||||
|
x = img
|
||||||
|
if x.dim() == 4:
|
||||||
|
x = x[0]
|
||||||
|
if x.dim() != 3 or int(x.shape[-1]) < 3:
|
||||||
|
return None, None
|
||||||
|
if int(x.shape[0]) < 2 or int(x.shape[1]) < 2:
|
||||||
|
return None, None
|
||||||
|
x = x[..., :3].float().permute(2, 0, 1).unsqueeze(0)
|
||||||
|
p = torch.nn.functional.adaptive_avg_pool2d(x, LEVEL_POOL)[0].reshape(3, -1)
|
||||||
|
return p.mean(dim=1), p.std(dim=1)
|
||||||
|
|
||||||
|
|
||||||
|
# The per-shot motion envelope lived here, measured so the built footsteps could be
|
||||||
|
# timed off the picture. Nothing is built any more -- see the note at the top of
|
||||||
|
# audio.py -- so there is nothing left to time, and a measurement with no reader is a
|
||||||
|
# measurement that rots. Removed with the synthesiser it served.
|
||||||
|
|
||||||
|
|
||||||
|
def apply_levels(img, gain, offset):
|
||||||
|
"""Rescale a frame's contrast and level about its OWN per-channel mean.
|
||||||
|
|
||||||
|
The pivot is the frame's own mean and never a target. That is the whole reason this
|
||||||
|
can run on any scene: a beat that walks into a darker room keeps its darkness,
|
||||||
|
because nothing here knows or cares what the level is -- only how much the last
|
||||||
|
boundary expanded it. Anchoring to shot 1 instead would cancel every deliberate
|
||||||
|
lighting change in the film, which is the opposite failure.
|
||||||
|
|
||||||
|
Clamped into 0..1 because the next thing that happens to this frame is an 8-bit
|
||||||
|
quantisation (comfy.utils.common_upscale goes through a uint8 PIL round trip even
|
||||||
|
at the same size), so there is no headroom outside the range to borrow from."""
|
||||||
|
x = img.float()
|
||||||
|
c = min(3, int(x.shape[-1]))
|
||||||
|
m = x[..., :c].reshape(-1, c).mean(dim=0)
|
||||||
|
g = gain[:c].to(device=x.device, dtype=x.dtype)
|
||||||
|
o = offset[:c].to(device=x.device, dtype=x.dtype)
|
||||||
|
y = x.clone()
|
||||||
|
y[..., :c] = ((x[..., :c] - m) * g + m + o).clamp(0.0, 1.0)
|
||||||
|
return y.to(img.dtype)
|
||||||
@@ -0,0 +1,109 @@
|
|||||||
|
# H3-LongVideos -- https://github.com/Smite79/MiniMax-H3-LongVideos
|
||||||
|
# Copyright (c) 2026 Smite79. All rights reserved.
|
||||||
|
# Redistribution, in whole or in part, requires written permission.
|
||||||
|
# This notice may not be removed or altered. See LICENSE.
|
||||||
|
"""Per-shot records passed from prompt planning to rendering."""
|
||||||
|
|
||||||
|
from dataclasses import dataclass, field
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class Shot:
|
||||||
|
prompt: str
|
||||||
|
cast: list[str]
|
||||||
|
speech: bool
|
||||||
|
sounded: bool
|
||||||
|
voiced_only: bool
|
||||||
|
events: list[str]
|
||||||
|
frame_count: int = 0
|
||||||
|
refs: list[object] = field(default_factory=list)
|
||||||
|
line_seconds: float = 0.0 # the planner's estimate of the spoken line, words / WORDS_PER_SEC
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class ShotPlan:
|
||||||
|
shots: list[Shot] = field(default_factory=list)
|
||||||
|
|
||||||
|
@property
|
||||||
|
def prompts(self):
|
||||||
|
return [shot.prompt for shot in self.shots]
|
||||||
|
|
||||||
|
def add(self, prompt, cast, speech, sounded, voiced_only, events):
|
||||||
|
shot = Shot(prompt, list(cast or ()), bool(speech), bool(sounded),
|
||||||
|
bool(voiced_only), list(events or ()))
|
||||||
|
self.shots.append(shot)
|
||||||
|
|
||||||
|
def set_frame_counts(self, counts):
|
||||||
|
counts = [int(n) for n in counts]
|
||||||
|
if len(counts) != len(self.shots) or any(n <= 0 for n in counts):
|
||||||
|
raise ValueError("frame counts must be positive and match the planned shots")
|
||||||
|
for shot, count in zip(self.shots, counts):
|
||||||
|
shot.frame_count = count
|
||||||
|
|
||||||
|
def validate(self):
|
||||||
|
if any(shot.frame_count <= 0 for shot in self.shots):
|
||||||
|
raise ValueError("each shot needs a positive frame count before rendering")
|
||||||
|
return self
|
||||||
|
|
||||||
|
def __len__(self):
|
||||||
|
return len(self.shots)
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class PreparedVideo:
|
||||||
|
"""Resolved inputs consumed by the render stage; model/tensor handles are shared."""
|
||||||
|
_placed_shots: object
|
||||||
|
_first_is_plate: object
|
||||||
|
_returns: object
|
||||||
|
_soft_landing: object
|
||||||
|
_tagged_names: object
|
||||||
|
ambient_audio: object
|
||||||
|
ambient_level: float
|
||||||
|
apply_model_sampling: bool
|
||||||
|
audio_vae: object
|
||||||
|
auto_sound: bool
|
||||||
|
bared_shots: object
|
||||||
|
cfg: float
|
||||||
|
cleanup_between_shots: bool
|
||||||
|
clip: object
|
||||||
|
first_frame: object
|
||||||
|
foley_level: float
|
||||||
|
h: int
|
||||||
|
latent_upscale: str
|
||||||
|
latent_upscale_scale: float
|
||||||
|
megapixels: float
|
||||||
|
model: object
|
||||||
|
moved_shots: object
|
||||||
|
negative: object
|
||||||
|
notes: list[str]
|
||||||
|
plan: ShotPlan
|
||||||
|
ref_noise_aug: float | None
|
||||||
|
restart_after_removal: bool
|
||||||
|
revealed_shots: object
|
||||||
|
sampler_name: str
|
||||||
|
scheduler: str
|
||||||
|
seed: int
|
||||||
|
shift_audio: float
|
||||||
|
shift_video: float
|
||||||
|
sigmas: object
|
||||||
|
silence_nonspeech: bool
|
||||||
|
speech_lead_seconds: float
|
||||||
|
speech_tail_seconds: float
|
||||||
|
hold_levels: float
|
||||||
|
handoff_frames: int
|
||||||
|
staging_shots: object
|
||||||
|
steps: int
|
||||||
|
stripped_shots: object
|
||||||
|
cut_shots: object
|
||||||
|
tiled_decode: bool
|
||||||
|
trim_seam: bool
|
||||||
|
upscale: str
|
||||||
|
upscale_batch: int
|
||||||
|
upscale_model: str
|
||||||
|
upscale_target_short_edge: int
|
||||||
|
vae: object
|
||||||
|
w: int
|
||||||
|
shot_rooms: object = None # {0-based shot: (room it opens in, room it ends in)}
|
||||||
|
hardware_changed: object = None # 1-based shots that put hardware on or take it off
|
||||||
|
shot_frames: object = None # {0-based shot: (who its frames show, who is still there at its end)}
|
||||||
|
reentry_shots: object = None # {0-based shot: who walks in while the keyframe still has them}
|
||||||
File diff suppressed because it is too large
Load Diff
+57
-8
@@ -1249,12 +1249,21 @@ def curate_h3_prompt(
|
|||||||
selected = _selected_prompt_refs(action_prompt, normalized_refs)
|
selected = _selected_prompt_refs(action_prompt, normalized_refs)
|
||||||
picture_map = {slot_number: index for index, (slot_number, _ref) in enumerate(selected, 1)}
|
picture_map = {slot_number: index for index, (slot_number, _ref) in enumerate(selected, 1)}
|
||||||
action = _replace_reference_tags(action_prompt, picture_map)
|
action = _replace_reference_tags(action_prompt, picture_map)
|
||||||
|
anchor_text = _reference_text(anchor)
|
||||||
|
soundscape_text = _reference_text(soundscape)
|
||||||
|
bgm_text = _reference_text(bgm)
|
||||||
|
reference_description = ""
|
||||||
|
individual_reference_descriptions = []
|
||||||
|
if selected:
|
||||||
|
individual_reference_descriptions = [
|
||||||
|
_reference_context(ref, picture_number)
|
||||||
|
for picture_number, (_slot, ref) in enumerate(selected, 1)
|
||||||
|
]
|
||||||
|
reference_description = " ".join(individual_reference_descriptions)
|
||||||
|
|
||||||
prompt_parts = []
|
prompt_parts = []
|
||||||
_append_prompt_section(prompt_parts, "Scene anchor", anchor)
|
_append_prompt_section(prompt_parts, "Scene anchor", anchor_text)
|
||||||
if selected:
|
_append_prompt_section(prompt_parts, "Reference context", reference_description)
|
||||||
contexts = [_reference_context(ref, picture_number) for picture_number, (_slot, ref) in enumerate(selected, 1)]
|
|
||||||
_append_prompt_section(prompt_parts, "Reference context", " ".join(contexts))
|
|
||||||
_append_prompt_section(prompt_parts, "Action", action)
|
_append_prompt_section(prompt_parts, "Action", action)
|
||||||
if anatomy_guard == "on" or (anatomy_guard == "auto" and any(ref.get("kind") == "character" for _slot, ref in selected)):
|
if anatomy_guard == "on" or (anatomy_guard == "auto" and any(ref.get("kind") == "character" for _slot, ref in selected)):
|
||||||
_append_prompt_section(prompt_parts, "Anatomy guard", _ANATOMY_GUARD_TEXT)
|
_append_prompt_section(prompt_parts, "Anatomy guard", _ANATOMY_GUARD_TEXT)
|
||||||
@@ -1262,14 +1271,17 @@ def curate_h3_prompt(
|
|||||||
subject_count_guard == "auto" and any(ref.get("kind") == "character" for _slot, ref in selected)
|
subject_count_guard == "auto" and any(ref.get("kind") == "character" for _slot, ref in selected)
|
||||||
):
|
):
|
||||||
_append_prompt_section(prompt_parts, "Subject count guard", _subject_count_guard_text(selected))
|
_append_prompt_section(prompt_parts, "Subject count guard", _subject_count_guard_text(selected))
|
||||||
_append_prompt_section(prompt_parts, "overall_soundscape", soundscape)
|
_append_prompt_section(prompt_parts, "overall_soundscape", soundscape_text)
|
||||||
_append_prompt_section(prompt_parts, "background_music", bgm)
|
_append_prompt_section(prompt_parts, "background_music", bgm_text)
|
||||||
|
|
||||||
prompt = "\n\n".join(prompt_parts).strip()
|
prompt = "\n\n".join(prompt_parts).strip()
|
||||||
if len(prompt) > _H3_PROMPT_MAX_CHARS:
|
if len(prompt) > _H3_PROMPT_MAX_CHARS:
|
||||||
prompt = prompt[: _H3_PROMPT_MAX_CHARS - 3].rstrip() + "..."
|
prompt = prompt[: _H3_PROMPT_MAX_CHARS - 3].rstrip() + "..."
|
||||||
images = [_reference_image(ref) for _slot, ref in selected]
|
images = [_reference_image(ref) for _slot, ref in selected]
|
||||||
images.extend([None] * (_H3_PROMPT_REF_SLOTS - len(images)))
|
images.extend([None] * (_H3_PROMPT_REF_SLOTS - len(images)))
|
||||||
|
individual_reference_descriptions.extend(
|
||||||
|
[""] * (_H3_PROMPT_REF_SLOTS - len(individual_reference_descriptions))
|
||||||
|
)
|
||||||
debug = (
|
debug = (
|
||||||
f"Selected {len(selected)} reference(s): "
|
f"Selected {len(selected)} reference(s): "
|
||||||
+ ", ".join(
|
+ ", ".join(
|
||||||
@@ -1279,7 +1291,17 @@ def curate_h3_prompt(
|
|||||||
if selected
|
if selected
|
||||||
else "Selected 0 references."
|
else "Selected 0 references."
|
||||||
)
|
)
|
||||||
return (prompt, *images[:_H3_PROMPT_REF_SLOTS], len(selected), debug)
|
return (
|
||||||
|
prompt,
|
||||||
|
*images[:_H3_PROMPT_REF_SLOTS],
|
||||||
|
len(selected),
|
||||||
|
debug,
|
||||||
|
anchor_text,
|
||||||
|
soundscape_text,
|
||||||
|
bgm_text,
|
||||||
|
*images[:_H3_PROMPT_REF_SLOTS],
|
||||||
|
*individual_reference_descriptions[:_H3_PROMPT_REF_SLOTS],
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def _parse_positive_int(value):
|
def _parse_positive_int(value):
|
||||||
@@ -2666,7 +2688,13 @@ class DumasH3PromptCuratorNode:
|
|||||||
"compacted so only mentioned names, aliases, or explicit <Picture N>/<refN> "
|
"compacted so only mentioned names, aliases, or explicit <Picture N>/<refN> "
|
||||||
"tags are sent onward."
|
"tags are sent onward."
|
||||||
)
|
)
|
||||||
RETURN_TYPES = ("STRING",) + ("IMAGE",) * _H3_PROMPT_REF_SLOTS + ("INT", "STRING")
|
RETURN_TYPES = (
|
||||||
|
("STRING",)
|
||||||
|
+ ("IMAGE",) * _H3_PROMPT_REF_SLOTS
|
||||||
|
+ ("INT", "STRING", "STRING", "STRING", "STRING")
|
||||||
|
+ ("IMAGE",) * _H3_PROMPT_REF_SLOTS
|
||||||
|
+ ("STRING",) * _H3_PROMPT_REF_SLOTS
|
||||||
|
)
|
||||||
RETURN_NAMES = (
|
RETURN_NAMES = (
|
||||||
"prompt",
|
"prompt",
|
||||||
"ref_image_1",
|
"ref_image_1",
|
||||||
@@ -2680,6 +2708,27 @@ class DumasH3PromptCuratorNode:
|
|||||||
"ref_image_9",
|
"ref_image_9",
|
||||||
"reference_count",
|
"reference_count",
|
||||||
"debug",
|
"debug",
|
||||||
|
"anchor",
|
||||||
|
"sounds",
|
||||||
|
"bgm",
|
||||||
|
"original_ref_1",
|
||||||
|
"original_ref_2",
|
||||||
|
"original_ref_3",
|
||||||
|
"original_ref_4",
|
||||||
|
"original_ref_5",
|
||||||
|
"original_ref_6",
|
||||||
|
"original_ref_7",
|
||||||
|
"original_ref_8",
|
||||||
|
"original_ref_9",
|
||||||
|
"compiled_ref_description_1",
|
||||||
|
"compiled_ref_description_2",
|
||||||
|
"compiled_ref_description_3",
|
||||||
|
"compiled_ref_description_4",
|
||||||
|
"compiled_ref_description_5",
|
||||||
|
"compiled_ref_description_6",
|
||||||
|
"compiled_ref_description_7",
|
||||||
|
"compiled_ref_description_8",
|
||||||
|
"compiled_ref_description_9",
|
||||||
)
|
)
|
||||||
FUNCTION = "curate_prompt"
|
FUNCTION = "curate_prompt"
|
||||||
CATEGORY = "Dumas/MiniMax"
|
CATEGORY = "Dumas/MiniMax"
|
||||||
|
|||||||
@@ -3,7 +3,7 @@ import { app } from "/scripts/app.js";
|
|||||||
const NODE_NAME = "DumasAnchorStyle";
|
const NODE_NAME = "DumasAnchorStyle";
|
||||||
const STYLE_INPUT = "anchor_style";
|
const STYLE_INPUT = "anchor_style";
|
||||||
const DESCRIPTION_INPUT = "style_description";
|
const DESCRIPTION_INPUT = "style_description";
|
||||||
const H3_NOTE = " Keep this anchor focused on persistent camera language, lighting, texture, environment treatment, and tone; do not name characters or describe one-off actions.";
|
const H3_NOTE = "";
|
||||||
const PRESETS = {
|
const PRESETS = {
|
||||||
"cinematic action movie": "Big-screen action cinema with assertive visual storytelling: dynamic camera placement, strong forward momentum, crisp geography, muscular lighting contrast, practical atmosphere, and a sense of physical consequence. Favor heroic framing, controlled handheld energy or motivated tracking moves, dramatic silhouettes, tasteful lens flares, impact-driven pacing, and polished studio spectacle without drifting into comic-book unreality unless the shot explicitly asks for it." + H3_NOTE,
|
"cinematic action movie": "Big-screen action cinema with assertive visual storytelling: dynamic camera placement, strong forward momentum, crisp geography, muscular lighting contrast, practical atmosphere, and a sense of physical consequence. Favor heroic framing, controlled handheld energy or motivated tracking moves, dramatic silhouettes, tasteful lens flares, impact-driven pacing, and polished studio spectacle without drifting into comic-book unreality unless the shot explicitly asks for it." + H3_NOTE,
|
||||||
"comedy": "Play the scene for comedic readability and timing: clear staging, expressive performances, slightly heightened reactions, clean eyelines, and visual beats that leave room for the joke to land. Use bright approachable lighting, grounded but playful production design, readable framing, and a tone that feels observant, awkward, or absurd without becoming broad parody unless the action supports it." + H3_NOTE,
|
"comedy": "Play the scene for comedic readability and timing: clear staging, expressive performances, slightly heightened reactions, clean eyelines, and visual beats that leave room for the joke to land. Use bright approachable lighting, grounded but playful production design, readable framing, and a tone that feels observant, awkward, or absurd without becoming broad parody unless the action supports it." + H3_NOTE,
|
||||||
@@ -51,28 +51,72 @@ const PRESETS = {
|
|||||||
"fantasy adventure": "Rousing fantasy-adventure language: scenic scale, adventurous clarity, tactile costume-and-prop detail, and camera movement that feels exploratory rather than oppressive. Favor storybook geography, weathered materials, golden or stormy atmosphere, and a tone of peril, wonder, and forward motion." + H3_NOTE,
|
"fantasy adventure": "Rousing fantasy-adventure language: scenic scale, adventurous clarity, tactile costume-and-prop detail, and camera movement that feels exploratory rather than oppressive. Favor storybook geography, weathered materials, golden or stormy atmosphere, and a tone of peril, wonder, and forward motion." + H3_NOTE,
|
||||||
};
|
};
|
||||||
|
|
||||||
|
const SOUNDSCAPE_PRESETS = {
|
||||||
|
"quiet interior": "quiet indoor room tone, faint ventilation and distant household ambience",
|
||||||
|
"rainy street": "steady rain, wet pavement, distant traffic hum",
|
||||||
|
"cafe": "low room tone, faint glassware, cutlery, and muted conversation",
|
||||||
|
"city night": "distant traffic hum, occasional horn, night air",
|
||||||
|
"forest": "wind in leaves, distant birds, soft natural ambience",
|
||||||
|
"industrial": "large interior reverb, distant metal ticks, low machine hum",
|
||||||
|
"silent": "no dialogue, no vocals, only the natural ambient bed of the scene",
|
||||||
|
"custom": "",
|
||||||
|
};
|
||||||
|
|
||||||
|
const BGM_PRESETS = {
|
||||||
|
"none": "",
|
||||||
|
"subtle tension": "low, restrained tension bed with sparse pulses and no vocals",
|
||||||
|
"cinematic suspense": "cinematic suspense score with muted strings, low drones, and controlled rising pressure",
|
||||||
|
"emotional piano": "soft emotional piano underscoring with gentle space and no vocals",
|
||||||
|
"dark ambient": "dark ambient music bed with deep drones, distant texture, and slow unease",
|
||||||
|
"hopeful orchestral": "hopeful orchestral underscore with warm strings, gentle brass, and restrained lift",
|
||||||
|
"retro synth": "retro synth score with analog pulses, warm pads, and steady momentum",
|
||||||
|
"action pulse": "driving action pulse with percussion, rhythmic bass, and urgent forward motion",
|
||||||
|
"lo-fi": "soft lo-fi instrumental bed with mellow rhythm and warm tape texture",
|
||||||
|
"no vocals": "instrumental background music only, no singing, no lyrics, no vocal hooks",
|
||||||
|
"custom": "",
|
||||||
|
};
|
||||||
|
|
||||||
|
const NODE_CONFIGS = {
|
||||||
|
[NODE_NAME]: {
|
||||||
|
presetInput: STYLE_INPUT,
|
||||||
|
descriptionInput: DESCRIPTION_INPUT,
|
||||||
|
presets: PRESETS,
|
||||||
|
},
|
||||||
|
DumasSoundscapeHelper: {
|
||||||
|
presetInput: "soundscape",
|
||||||
|
descriptionInput: "soundscape_description",
|
||||||
|
presets: SOUNDSCAPE_PRESETS,
|
||||||
|
},
|
||||||
|
DumasBackgroundMusicHelper: {
|
||||||
|
presetInput: "bgm",
|
||||||
|
descriptionInput: "bgm_description",
|
||||||
|
presets: BGM_PRESETS,
|
||||||
|
},
|
||||||
|
};
|
||||||
|
|
||||||
function findWidget(node, name) {
|
function findWidget(node, name) {
|
||||||
return (node.widgets || []).find((widget) => widget?.name === name) || null;
|
return (node.widgets || []).find((widget) => widget?.name === name) || null;
|
||||||
}
|
}
|
||||||
|
|
||||||
app.registerExtension({
|
app.registerExtension({
|
||||||
name: "Dumas.AnchorStyle",
|
name: "Dumas.PresetTextHelpers",
|
||||||
async beforeRegisterNodeDef(nodeType, nodeData) {
|
async beforeRegisterNodeDef(nodeType, nodeData) {
|
||||||
if (nodeData?.name !== NODE_NAME) return;
|
const config = NODE_CONFIGS[nodeData?.name];
|
||||||
|
if (!config) return;
|
||||||
|
|
||||||
const originalOnNodeCreated = nodeType.prototype.onNodeCreated;
|
const originalOnNodeCreated = nodeType.prototype.onNodeCreated;
|
||||||
|
|
||||||
nodeType.prototype.onNodeCreated = function onNodeCreated() {
|
nodeType.prototype.onNodeCreated = function onNodeCreated() {
|
||||||
const created = originalOnNodeCreated?.apply(this, arguments);
|
const created = originalOnNodeCreated?.apply(this, arguments);
|
||||||
const styleWidget = findWidget(this, STYLE_INPUT);
|
const styleWidget = findWidget(this, config.presetInput);
|
||||||
const descriptionWidget = findWidget(this, DESCRIPTION_INPUT);
|
const descriptionWidget = findWidget(this, config.descriptionInput);
|
||||||
if (!styleWidget || !descriptionWidget) return created;
|
if (!styleWidget || !descriptionWidget) return created;
|
||||||
|
|
||||||
const originalCallback = styleWidget.callback;
|
const originalCallback = styleWidget.callback;
|
||||||
styleWidget.callback = (...args) => {
|
styleWidget.callback = (...args) => {
|
||||||
const selected = String(styleWidget.value || "");
|
const selected = String(styleWidget.value || "");
|
||||||
if (Object.hasOwn(PRESETS, selected)) {
|
if (Object.hasOwn(config.presets, selected)) {
|
||||||
descriptionWidget.value = PRESETS[selected];
|
descriptionWidget.value = config.presets[selected];
|
||||||
descriptionWidget.inputEl?.dispatchEvent(new Event("input", { bubbles: true }));
|
descriptionWidget.inputEl?.dispatchEvent(new Event("input", { bubbles: true }));
|
||||||
}
|
}
|
||||||
this.setDirtyCanvas?.(true, true);
|
this.setDirtyCanvas?.(true, true);
|
||||||
|
|||||||
+74
-100
@@ -8,32 +8,15 @@ const DEFAULT_W = 520;
|
|||||||
const DEFAULT_H = 340;
|
const DEFAULT_H = 340;
|
||||||
const DEFAULT_BEAT = "Describe this beat.";
|
const DEFAULT_BEAT = "Describe this beat.";
|
||||||
const STATE_PROPERTY = "dumas_h3_beat_prompt_state";
|
const STATE_PROPERTY = "dumas_h3_beat_prompt_state";
|
||||||
const CONTINUITY_OPTIONS = ["", "soft carry", "hard cut", "keyframe carry", "handoff ref"];
|
|
||||||
const REF_MODE_OPTIONS = ["", "auto ref2v", "where tagged", "first shot", "every shot", "every shot + handoff ref"];
|
|
||||||
const MANAGED_DIRECTIVES = {
|
const MANAGED_DIRECTIVES = {
|
||||||
seconds: ["seconds", "duration"],
|
remove: ["remove", "removed", "off"],
|
||||||
continuity: ["continuity"],
|
add: ["add", "wear", "wearing"],
|
||||||
ref_mode: ["ref_mode"],
|
|
||||||
ref_noise_aug: ["ref_noise_aug"],
|
|
||||||
anchor_add: ["anchor_add"],
|
|
||||||
overall_soundscape: ["overall_soundscape", "soundscape"],
|
|
||||||
non_diegetic_music: ["non_diegetic_music", "music"],
|
|
||||||
};
|
};
|
||||||
const DIRECTIVE_EXAMPLES = [
|
const DIRECTIVE_EXAMPLES = [
|
||||||
["wardrobe set", "wardrobe: Maya = grey shorts, red jacket"],
|
["remove", "remove: red jacket"],
|
||||||
["wardrobe add", "wardrobe: Maya += red jacket"],
|
["off", "off: steel collar"],
|
||||||
["wardrobe remove", "wardrobe: Maya -= red jacket"],
|
["add", "add: white shirt underneath"],
|
||||||
["seconds", "seconds: 8"],
|
["wearing", "wearing: black coat"],
|
||||||
["exit", "exit: Maya"],
|
|
||||||
["enter", "enter: Jon"],
|
|
||||||
["continuity", "continuity: hard cut"],
|
|
||||||
["ref_mode", "ref_mode: every shot"],
|
|
||||||
["ref_noise_aug", "ref_noise_aug: 0.92"],
|
|
||||||
["anchor_add", "anchor_add: harsh sodium-vapor spill, wet pavement, long-lens compression"],
|
|
||||||
["overall_soundscape", "overall_soundscape: soft rain, distant traffic"],
|
|
||||||
["non_diegetic_music", "non_diegetic_music: tense analog synth pulse"],
|
|
||||||
["soundscape", "soundscape: fluorescent room tone, faint HVAC hum"],
|
|
||||||
["music", "music: low ominous cello and sparse percussion"],
|
|
||||||
];
|
];
|
||||||
|
|
||||||
function injectCSS() {
|
function injectCSS() {
|
||||||
@@ -183,7 +166,7 @@ function injectCSS() {
|
|||||||
}
|
}
|
||||||
|
|
||||||
function defaultState() {
|
function defaultState() {
|
||||||
return { beats: [{ text: DEFAULT_BEAT }] };
|
return { scene: "", character_sheet: "", beats: [{ text: DEFAULT_BEAT }] };
|
||||||
}
|
}
|
||||||
|
|
||||||
function normalizeState(value) {
|
function normalizeState(value) {
|
||||||
@@ -200,7 +183,11 @@ function normalizeState(value) {
|
|||||||
const normalized = beats.map((beat) => ({
|
const normalized = beats.map((beat) => ({
|
||||||
text: typeof beat?.text === "string" ? beat.text : String(beat?.text || ""),
|
text: typeof beat?.text === "string" ? beat.text : String(beat?.text || ""),
|
||||||
}));
|
}));
|
||||||
return normalized.length ? { beats: normalized } : defaultState();
|
return {
|
||||||
|
scene: typeof parsed.scene === "string" ? parsed.scene : String(parsed.scene || ""),
|
||||||
|
character_sheet: typeof parsed.character_sheet === "string" ? parsed.character_sheet : String(parsed.character_sheet || ""),
|
||||||
|
beats: normalized.length ? normalized : [{ text: DEFAULT_BEAT }],
|
||||||
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
function readState(node) {
|
function readState(node) {
|
||||||
@@ -321,6 +308,51 @@ function renderUI(node) {
|
|||||||
node._dh3bpRenderedState = JSON.stringify(state);
|
node._dh3bpRenderedState = JSON.stringify(state);
|
||||||
ui.list.innerHTML = "";
|
ui.list.innerHTML = "";
|
||||||
|
|
||||||
|
const buildTopTextarea = ({ labelText, placeholder, value, onInput }) => {
|
||||||
|
const card = document.createElement("div");
|
||||||
|
card.className = "dh3bp-beat";
|
||||||
|
|
||||||
|
const label = document.createElement("div");
|
||||||
|
label.className = "dh3bp-label";
|
||||||
|
label.textContent = labelText;
|
||||||
|
|
||||||
|
const textarea = document.createElement("textarea");
|
||||||
|
textarea.className = "dh3bp-text";
|
||||||
|
textarea.placeholder = placeholder;
|
||||||
|
textarea.value = value || "";
|
||||||
|
textarea.addEventListener("input", () => {
|
||||||
|
onInput(textarea.value);
|
||||||
|
updateTextareaHeight(textarea);
|
||||||
|
});
|
||||||
|
textarea.addEventListener("keydown", stopCanvasKeyboard);
|
||||||
|
|
||||||
|
card.append(label, textarea);
|
||||||
|
updateTextareaHeight(textarea);
|
||||||
|
return card;
|
||||||
|
};
|
||||||
|
|
||||||
|
ui.list.appendChild(buildTopTextarea({
|
||||||
|
labelText: "Scene paragraph",
|
||||||
|
placeholder: "Optional. Persistent location, lighting, camera, tone. Leave empty if you wire the Long Videos anchor input.",
|
||||||
|
value: state.scene,
|
||||||
|
onInput: (value) => {
|
||||||
|
const next = readState(node);
|
||||||
|
next.scene = value;
|
||||||
|
writeState(node, next);
|
||||||
|
},
|
||||||
|
}));
|
||||||
|
|
||||||
|
ui.list.appendChild(buildTopTextarea({
|
||||||
|
labelText: "Character sheet",
|
||||||
|
placeholder: "Optional. One character per line, e.g. Maya: 27, she, silver hair, red jacket, the woman in <Picture 1>.",
|
||||||
|
value: state.character_sheet,
|
||||||
|
onInput: (value) => {
|
||||||
|
const next = readState(node);
|
||||||
|
next.character_sheet = value;
|
||||||
|
writeState(node, next);
|
||||||
|
},
|
||||||
|
}));
|
||||||
|
|
||||||
state.beats.forEach((beat, index) => {
|
state.beats.forEach((beat, index) => {
|
||||||
const card = document.createElement("div");
|
const card = document.createElement("div");
|
||||||
card.className = "dh3bp-beat";
|
card.className = "dh3bp-beat";
|
||||||
@@ -379,85 +411,27 @@ function renderUI(node) {
|
|||||||
return wrap;
|
return wrap;
|
||||||
};
|
};
|
||||||
|
|
||||||
const secondsInput = document.createElement("input");
|
const removeInput = document.createElement("input");
|
||||||
secondsInput.className = "dh3bp-input";
|
removeInput.className = "dh3bp-input";
|
||||||
secondsInput.type = "text";
|
removeInput.type = "text";
|
||||||
secondsInput.placeholder = "8";
|
removeInput.placeholder = "red jacket";
|
||||||
secondsInput.value = readDirectiveValue(beat.text, MANAGED_DIRECTIVES.seconds);
|
removeInput.value = readDirectiveValue(beat.text, MANAGED_DIRECTIVES.remove);
|
||||||
secondsInput.addEventListener("input", () => {
|
removeInput.addEventListener("input", () => {
|
||||||
applyTextUpdate(setDirectiveValue(textarea.value, "seconds", MANAGED_DIRECTIVES.seconds, secondsInput.value));
|
applyTextUpdate(setDirectiveValue(textarea.value, "remove", MANAGED_DIRECTIVES.remove, removeInput.value));
|
||||||
});
|
});
|
||||||
|
|
||||||
const continuitySelect = document.createElement("select");
|
const addInput = document.createElement("input");
|
||||||
continuitySelect.className = "dh3bp-select";
|
addInput.className = "dh3bp-input";
|
||||||
CONTINUITY_OPTIONS.forEach((value) => {
|
addInput.type = "text";
|
||||||
const option = document.createElement("option");
|
addInput.placeholder = "white shirt underneath";
|
||||||
option.value = value;
|
addInput.value = readDirectiveValue(beat.text, MANAGED_DIRECTIVES.add);
|
||||||
option.textContent = value || "Default";
|
addInput.addEventListener("input", () => {
|
||||||
continuitySelect.appendChild(option);
|
applyTextUpdate(setDirectiveValue(textarea.value, "add", MANAGED_DIRECTIVES.add, addInput.value));
|
||||||
});
|
|
||||||
continuitySelect.value = readDirectiveValue(beat.text, MANAGED_DIRECTIVES.continuity);
|
|
||||||
continuitySelect.addEventListener("change", () => {
|
|
||||||
applyTextUpdate(setDirectiveValue(textarea.value, "continuity", MANAGED_DIRECTIVES.continuity, continuitySelect.value));
|
|
||||||
});
|
|
||||||
|
|
||||||
const refModeSelect = document.createElement("select");
|
|
||||||
refModeSelect.className = "dh3bp-select";
|
|
||||||
REF_MODE_OPTIONS.forEach((value) => {
|
|
||||||
const option = document.createElement("option");
|
|
||||||
option.value = value;
|
|
||||||
option.textContent = value || "Global";
|
|
||||||
refModeSelect.appendChild(option);
|
|
||||||
});
|
|
||||||
refModeSelect.value = readDirectiveValue(beat.text, MANAGED_DIRECTIVES.ref_mode);
|
|
||||||
refModeSelect.addEventListener("change", () => {
|
|
||||||
applyTextUpdate(setDirectiveValue(textarea.value, "ref_mode", MANAGED_DIRECTIVES.ref_mode, refModeSelect.value));
|
|
||||||
});
|
|
||||||
|
|
||||||
const refNoiseInput = document.createElement("input");
|
|
||||||
refNoiseInput.className = "dh3bp-input";
|
|
||||||
refNoiseInput.type = "text";
|
|
||||||
refNoiseInput.placeholder = "0.95";
|
|
||||||
refNoiseInput.value = readDirectiveValue(beat.text, MANAGED_DIRECTIVES.ref_noise_aug);
|
|
||||||
refNoiseInput.addEventListener("input", () => {
|
|
||||||
applyTextUpdate(setDirectiveValue(textarea.value, "ref_noise_aug", MANAGED_DIRECTIVES.ref_noise_aug, refNoiseInput.value));
|
|
||||||
});
|
|
||||||
|
|
||||||
const anchorInput = document.createElement("input");
|
|
||||||
anchorInput.className = "dh3bp-input";
|
|
||||||
anchorInput.type = "text";
|
|
||||||
anchorInput.placeholder = "extra per-shot style treatment";
|
|
||||||
anchorInput.value = readDirectiveValue(beat.text, MANAGED_DIRECTIVES.anchor_add);
|
|
||||||
anchorInput.addEventListener("input", () => {
|
|
||||||
applyTextUpdate(setDirectiveValue(textarea.value, "anchor_add", MANAGED_DIRECTIVES.anchor_add, anchorInput.value));
|
|
||||||
});
|
|
||||||
|
|
||||||
const soundscapeInput = document.createElement("input");
|
|
||||||
soundscapeInput.className = "dh3bp-input";
|
|
||||||
soundscapeInput.type = "text";
|
|
||||||
soundscapeInput.placeholder = "faint traffic, loose sign rattle";
|
|
||||||
soundscapeInput.value = readDirectiveValue(beat.text, MANAGED_DIRECTIVES.overall_soundscape);
|
|
||||||
soundscapeInput.addEventListener("input", () => {
|
|
||||||
applyTextUpdate(setDirectiveValue(textarea.value, "overall_soundscape", MANAGED_DIRECTIVES.overall_soundscape, soundscapeInput.value));
|
|
||||||
});
|
|
||||||
|
|
||||||
const musicInput = document.createElement("input");
|
|
||||||
musicInput.className = "dh3bp-input";
|
|
||||||
musicInput.type = "text";
|
|
||||||
musicInput.placeholder = "low pulsing synth tension";
|
|
||||||
musicInput.value = readDirectiveValue(beat.text, MANAGED_DIRECTIVES.non_diegetic_music);
|
|
||||||
musicInput.addEventListener("input", () => {
|
|
||||||
applyTextUpdate(setDirectiveValue(textarea.value, "non_diegetic_music", MANAGED_DIRECTIVES.non_diegetic_music, musicInput.value));
|
|
||||||
});
|
});
|
||||||
|
|
||||||
controls.append(
|
controls.append(
|
||||||
buildField({ labelText: "Seconds", input: secondsInput }),
|
buildField({ labelText: "Remove from memory", input: removeInput }),
|
||||||
buildField({ labelText: "Continuity", input: continuitySelect }),
|
buildField({ labelText: "Add to memory", input: addInput }),
|
||||||
buildField({ labelText: "Ref Mode", input: refModeSelect }),
|
|
||||||
buildField({ labelText: "Ref Noise Aug", input: refNoiseInput }),
|
|
||||||
buildField({ labelText: "Anchor Add", className: "dh3bp-control-wide", input: anchorInput }),
|
|
||||||
buildField({ labelText: "Shot Soundscape", className: "dh3bp-control-wide", input: soundscapeInput }),
|
|
||||||
buildField({ labelText: "Shot Music", className: "dh3bp-control-wide", input: musicInput }),
|
|
||||||
);
|
);
|
||||||
|
|
||||||
const directives = document.createElement("div");
|
const directives = document.createElement("div");
|
||||||
@@ -501,7 +475,7 @@ function setupNode(node) {
|
|||||||
title.textContent = "Beat Prompt Builder";
|
title.textContent = "Beat Prompt Builder";
|
||||||
const subtitle = document.createElement("div");
|
const subtitle = document.createElement("div");
|
||||||
subtitle.className = "dh3bp-subtitle";
|
subtitle.className = "dh3bp-subtitle";
|
||||||
subtitle.textContent = "One textbox per H3 beat, plus per-shot controls for timing, ref behavior, continuity, anchor adds, and audio directives.";
|
subtitle.textContent = "Upstream Long Videos format: optional scene, optional character sheet, then one blank-line-separated beat per shot.";
|
||||||
titleWrap.append(title, subtitle);
|
titleWrap.append(title, subtitle);
|
||||||
|
|
||||||
const addButton = document.createElement("button");
|
const addButton = document.createElement("button");
|
||||||
|
|||||||
+5
-371
@@ -1,375 +1,9 @@
|
|||||||
import { app } from "/scripts/app.js";
|
import { app } from "/scripts/app.js";
|
||||||
import { applyAdaptiveCanvasOnly } from "../shared/nodes2.mjs";
|
|
||||||
|
|
||||||
const COMFY_CLASS = "DumasH3LongVideos";
|
|
||||||
const STATE_PROPERTY = "dumas_h3_longvideos_section_state";
|
|
||||||
const DOM_WIDGET_NAME = "dumas_h3_longvideos_sections";
|
|
||||||
const MIN_WIDTH = 520;
|
|
||||||
const MIN_HEIGHT = 280;
|
|
||||||
|
|
||||||
const GROUPS = [
|
|
||||||
{
|
|
||||||
id: "prompt",
|
|
||||||
label: "Prompt",
|
|
||||||
defaultCollapsed: false,
|
|
||||||
widgets: ["prompt", "resolution", "megapixels", "beat_split", "anchor_override", "shot_seconds", "plan_only", "fps"],
|
|
||||||
},
|
|
||||||
{
|
|
||||||
id: "refs",
|
|
||||||
label: "Refs",
|
|
||||||
defaultCollapsed: true,
|
|
||||||
widgets: ["ref_mode", "ref_image_size", "ref_noise_aug", "character_memory", "trim_seam", "vary_seed_per_shot", "handoff_offset"],
|
|
||||||
},
|
|
||||||
{
|
|
||||||
id: "sampling",
|
|
||||||
label: "Sampling",
|
|
||||||
defaultCollapsed: true,
|
|
||||||
widgets: [
|
|
||||||
"steps", "cfg", "sampler_name", "scheduler", "seed",
|
|
||||||
"apply_model_sampling", "shift_video", "shift_audio",
|
|
||||||
"vram_headroom_gb", "allow_res_backoff",
|
|
||||||
"decode_tile_frames", "decode_tile_size",
|
|
||||||
],
|
|
||||||
},
|
|
||||||
{
|
|
||||||
id: "audio",
|
|
||||||
label: "Audio",
|
|
||||||
defaultCollapsed: true,
|
|
||||||
widgets: [
|
|
||||||
"global_soundscape", "non_diegetic_music", "auto_soundscape",
|
|
||||||
"auto_silence_nonspeech", "allow_nonspeech_vocals",
|
|
||||||
"mute_nonspeech_audio", "mute_fade_ms",
|
|
||||||
],
|
|
||||||
},
|
|
||||||
{
|
|
||||||
id: "scene",
|
|
||||||
label: "Scene Logic",
|
|
||||||
defaultCollapsed: true,
|
|
||||||
widgets: [
|
|
||||||
"auto_wardrobe", "auto_props", "prevent_nudity", "exposed_terms",
|
|
||||||
"anatomy_guard", "subject_count_guard", "lock_restraints",
|
|
||||||
"contact_guard", "motion_guard", "solidity_guard",
|
|
||||||
],
|
|
||||||
},
|
|
||||||
{
|
|
||||||
id: "finish",
|
|
||||||
label: "Upscale",
|
|
||||||
defaultCollapsed: true,
|
|
||||||
widgets: [
|
|
||||||
"upscale", "upscale_model", "upscale_target_short_edge", "upscale_batch",
|
|
||||||
],
|
|
||||||
},
|
|
||||||
{
|
|
||||||
id: "overlay",
|
|
||||||
label: "Overlays",
|
|
||||||
defaultCollapsed: true,
|
|
||||||
widgets: [
|
|
||||||
"watermark_text", "watermark_position", "watermark_size", "watermark_opacity", "watermark_margin",
|
|
||||||
"intro_text", "intro_position", "intro_seconds", "intro_fade", "intro_size",
|
|
||||||
"overlay_font", "overlay_stroke",
|
|
||||||
],
|
|
||||||
},
|
|
||||||
];
|
|
||||||
|
|
||||||
function injectCSS() {
|
|
||||||
if (document.getElementById("dumas-h3lv-sections-css")) return;
|
|
||||||
const style = document.createElement("style");
|
|
||||||
style.id = "dumas-h3lv-sections-css";
|
|
||||||
style.textContent = `
|
|
||||||
.dh3lv-sections {
|
|
||||||
box-sizing: border-box;
|
|
||||||
width: 100%;
|
|
||||||
padding: 8px 10px 6px;
|
|
||||||
color: #e6e7eb;
|
|
||||||
font: 12px/1.35 "Segoe UI", sans-serif;
|
|
||||||
pointer-events: auto;
|
|
||||||
background: linear-gradient(180deg, rgba(33, 36, 42, 0.96), rgba(22, 24, 29, 0.96));
|
|
||||||
border-bottom: 1px solid rgba(255, 255, 255, 0.06);
|
|
||||||
}
|
|
||||||
.dh3lv-sections-head {
|
|
||||||
display: flex;
|
|
||||||
align-items: center;
|
|
||||||
justify-content: space-between;
|
|
||||||
gap: 8px;
|
|
||||||
margin-bottom: 8px;
|
|
||||||
}
|
|
||||||
.dh3lv-sections-title {
|
|
||||||
font-size: 11px;
|
|
||||||
text-transform: uppercase;
|
|
||||||
letter-spacing: 0.08em;
|
|
||||||
color: #9da5b1;
|
|
||||||
}
|
|
||||||
.dh3lv-sections-actions {
|
|
||||||
display: flex;
|
|
||||||
gap: 6px;
|
|
||||||
}
|
|
||||||
.dh3lv-sections-list {
|
|
||||||
display: flex;
|
|
||||||
flex-wrap: wrap;
|
|
||||||
gap: 6px;
|
|
||||||
}
|
|
||||||
.dh3lv-chip,
|
|
||||||
.dh3lv-action {
|
|
||||||
appearance: none;
|
|
||||||
border: 1px solid #464d59;
|
|
||||||
background: #262c35;
|
|
||||||
color: #d7dce3;
|
|
||||||
border-radius: 999px;
|
|
||||||
padding: 5px 9px;
|
|
||||||
cursor: pointer;
|
|
||||||
font: inherit;
|
|
||||||
line-height: 1.1;
|
|
||||||
}
|
|
||||||
.dh3lv-chip[data-open="true"] {
|
|
||||||
background: #d96f2b;
|
|
||||||
border-color: #f09358;
|
|
||||||
color: #fff7f0;
|
|
||||||
}
|
|
||||||
.dh3lv-chip:hover,
|
|
||||||
.dh3lv-action:hover {
|
|
||||||
filter: brightness(1.06);
|
|
||||||
}
|
|
||||||
.dh3lv-count {
|
|
||||||
opacity: 0.78;
|
|
||||||
margin-left: 4px;
|
|
||||||
font-size: 11px;
|
|
||||||
}
|
|
||||||
`;
|
|
||||||
document.head.appendChild(style);
|
|
||||||
}
|
|
||||||
|
|
||||||
function defaultState() {
|
|
||||||
const state = {};
|
|
||||||
for (const group of GROUPS) state[group.id] = !group.defaultCollapsed;
|
|
||||||
return state;
|
|
||||||
}
|
|
||||||
|
|
||||||
function parseState(value) {
|
|
||||||
let parsed = value;
|
|
||||||
if (typeof parsed === "string") {
|
|
||||||
try {
|
|
||||||
parsed = JSON.parse(parsed);
|
|
||||||
} catch (_error) {
|
|
||||||
parsed = null;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
const base = defaultState();
|
|
||||||
if (!parsed || typeof parsed !== "object") return base;
|
|
||||||
for (const group of GROUPS) {
|
|
||||||
if (typeof parsed[group.id] === "boolean") base[group.id] = parsed[group.id];
|
|
||||||
}
|
|
||||||
return base;
|
|
||||||
}
|
|
||||||
|
|
||||||
function readState(node) {
|
|
||||||
return parseState(node.properties?.[STATE_PROPERTY] || node._dh3lvSectionState || "");
|
|
||||||
}
|
|
||||||
|
|
||||||
function writeState(node, state) {
|
|
||||||
const normalized = parseState(state);
|
|
||||||
const serialized = JSON.stringify(normalized);
|
|
||||||
node._dh3lvSectionState = serialized;
|
|
||||||
node.properties = node.properties || {};
|
|
||||||
node.properties[STATE_PROPERTY] = serialized;
|
|
||||||
}
|
|
||||||
|
|
||||||
function findWidget(node, name) {
|
|
||||||
return (node.widgets || []).find((widget) => widget?.name === name) || null;
|
|
||||||
}
|
|
||||||
|
|
||||||
function isInteractiveTarget(target) {
|
|
||||||
return !!target?.closest?.("button, input, textarea, select, label");
|
|
||||||
}
|
|
||||||
|
|
||||||
function stopCanvasEvent(event) {
|
|
||||||
if (isInteractiveTarget(event.target)) event.stopPropagation();
|
|
||||||
}
|
|
||||||
|
|
||||||
function stopCanvasKeyboard(event) {
|
|
||||||
if (isInteractiveTarget(event.target)) event.stopImmediatePropagation();
|
|
||||||
}
|
|
||||||
|
|
||||||
function setWidgetHidden(widget, hidden) {
|
|
||||||
if (!widget) return;
|
|
||||||
if (!widget._dh3lvOriginal) {
|
|
||||||
widget._dh3lvOriginal = {
|
|
||||||
type: widget.type,
|
|
||||||
computeSize: widget.computeSize,
|
|
||||||
hidden: widget.hidden,
|
|
||||||
};
|
|
||||||
}
|
|
||||||
if (hidden) {
|
|
||||||
widget.type = "hidden";
|
|
||||||
widget.hidden = true;
|
|
||||||
widget.computeSize = () => [0, -4];
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
widget.type = widget._dh3lvOriginal.type;
|
|
||||||
widget.hidden = !!widget._dh3lvOriginal.hidden;
|
|
||||||
widget.computeSize = widget._dh3lvOriginal.computeSize;
|
|
||||||
}
|
|
||||||
|
|
||||||
function applyVisibility(node) {
|
|
||||||
const state = readState(node);
|
|
||||||
for (const group of GROUPS) {
|
|
||||||
for (const name of group.widgets) {
|
|
||||||
const widget = findWidget(node, name);
|
|
||||||
if (!widget || widget.name === DOM_WIDGET_NAME) continue;
|
|
||||||
setWidgetHidden(widget, !state[group.id]);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
function resizeNode(node) {
|
|
||||||
requestAnimationFrame(() => {
|
|
||||||
const size = node.computeSize?.();
|
|
||||||
if (Array.isArray(size)) {
|
|
||||||
node.size[0] = Math.max(MIN_WIDTH, size[0] || 0, node.size?.[0] || 0);
|
|
||||||
node.size[1] = Math.max(MIN_HEIGHT, size[1] || 0);
|
|
||||||
}
|
|
||||||
node.setDirtyCanvas?.(true, true);
|
|
||||||
});
|
|
||||||
}
|
|
||||||
|
|
||||||
function renderToolbar(node) {
|
|
||||||
const ui = node._dh3lvUI;
|
|
||||||
if (!ui) return;
|
|
||||||
const state = readState(node);
|
|
||||||
ui.list.innerHTML = "";
|
|
||||||
for (const group of GROUPS) {
|
|
||||||
const button = document.createElement("button");
|
|
||||||
button.type = "button";
|
|
||||||
button.className = "dh3lv-chip";
|
|
||||||
button.dataset.open = state[group.id] ? "true" : "false";
|
|
||||||
button.textContent = state[group.id] ? `Hide ${group.label}` : `Show ${group.label}`;
|
|
||||||
|
|
||||||
const count = document.createElement("span");
|
|
||||||
count.className = "dh3lv-count";
|
|
||||||
count.textContent = String(group.widgets.filter((name) => findWidget(node, name)).length);
|
|
||||||
button.appendChild(count);
|
|
||||||
|
|
||||||
button.addEventListener("click", () => {
|
|
||||||
const next = readState(node);
|
|
||||||
next[group.id] = !next[group.id];
|
|
||||||
writeState(node, next);
|
|
||||||
applyVisibility(node);
|
|
||||||
renderToolbar(node);
|
|
||||||
resizeNode(node);
|
|
||||||
});
|
|
||||||
ui.list.appendChild(button);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
function setAll(node, open) {
|
|
||||||
const next = {};
|
|
||||||
for (const group of GROUPS) next[group.id] = !!open;
|
|
||||||
writeState(node, next);
|
|
||||||
applyVisibility(node);
|
|
||||||
renderToolbar(node);
|
|
||||||
resizeNode(node);
|
|
||||||
}
|
|
||||||
|
|
||||||
function setupNode(node) {
|
|
||||||
if (node._dh3lvUI) return;
|
|
||||||
injectCSS();
|
|
||||||
writeState(node, readState(node));
|
|
||||||
|
|
||||||
const root = document.createElement("div");
|
|
||||||
root.className = "dh3lv-sections";
|
|
||||||
|
|
||||||
const head = document.createElement("div");
|
|
||||||
head.className = "dh3lv-sections-head";
|
|
||||||
|
|
||||||
const title = document.createElement("div");
|
|
||||||
title.className = "dh3lv-sections-title";
|
|
||||||
title.textContent = "Sections";
|
|
||||||
|
|
||||||
const actions = document.createElement("div");
|
|
||||||
actions.className = "dh3lv-sections-actions";
|
|
||||||
|
|
||||||
const expandAll = document.createElement("button");
|
|
||||||
expandAll.type = "button";
|
|
||||||
expandAll.className = "dh3lv-action";
|
|
||||||
expandAll.textContent = "Expand All";
|
|
||||||
expandAll.addEventListener("click", () => setAll(node, true));
|
|
||||||
|
|
||||||
const collapseAll = document.createElement("button");
|
|
||||||
collapseAll.type = "button";
|
|
||||||
collapseAll.className = "dh3lv-action";
|
|
||||||
collapseAll.textContent = "Collapse Extras";
|
|
||||||
collapseAll.addEventListener("click", () => {
|
|
||||||
const next = defaultState();
|
|
||||||
writeState(node, next);
|
|
||||||
applyVisibility(node);
|
|
||||||
renderToolbar(node);
|
|
||||||
resizeNode(node);
|
|
||||||
});
|
|
||||||
|
|
||||||
actions.append(expandAll, collapseAll);
|
|
||||||
head.append(title, actions);
|
|
||||||
|
|
||||||
const list = document.createElement("div");
|
|
||||||
list.className = "dh3lv-sections-list";
|
|
||||||
root.append(head, list);
|
|
||||||
root.addEventListener("pointerdown", stopCanvasEvent);
|
|
||||||
root.addEventListener("mousedown", stopCanvasEvent);
|
|
||||||
root.addEventListener("click", stopCanvasEvent);
|
|
||||||
root.addEventListener("dblclick", stopCanvasEvent);
|
|
||||||
root.addEventListener("keydown", stopCanvasKeyboard, true);
|
|
||||||
|
|
||||||
node._dh3lvUI = { root, list };
|
|
||||||
const widget = node.addDOMWidget(DOM_WIDGET_NAME, "custom", root, {
|
|
||||||
getValue: () => null,
|
|
||||||
setValue: () => {},
|
|
||||||
serialize: false,
|
|
||||||
getMinHeight: () => 52,
|
|
||||||
hideOnZoom: false,
|
|
||||||
});
|
|
||||||
applyAdaptiveCanvasOnly(widget);
|
|
||||||
const widgets = node.widgets || [];
|
|
||||||
const index = widgets.indexOf(widget);
|
|
||||||
if (index > 0) {
|
|
||||||
widgets.splice(index, 1);
|
|
||||||
widgets.unshift(widget);
|
|
||||||
}
|
|
||||||
|
|
||||||
applyVisibility(node);
|
|
||||||
renderToolbar(node);
|
|
||||||
resizeNode(node);
|
|
||||||
}
|
|
||||||
|
|
||||||
|
// The Dumas Long Videos node now wraps the upstream MiniMax-H3-Longvideos
|
||||||
|
// sampler directly. The old local frontend grouped Dumas-specific widgets that
|
||||||
|
// no longer exist on the upstream node, so this extension intentionally does
|
||||||
|
// nothing.
|
||||||
app.registerExtension({
|
app.registerExtension({
|
||||||
name: "Dumas.H3LongVideosSections",
|
name: "Dumas.H3LongVideos.UpstreamWrapper",
|
||||||
async beforeRegisterNodeDef(nodeType, nodeData) {
|
|
||||||
if (nodeData?.name !== COMFY_CLASS) return;
|
|
||||||
|
|
||||||
const originalOnNodeCreated = nodeType.prototype.onNodeCreated;
|
|
||||||
nodeType.prototype.onNodeCreated = function onNodeCreated() {
|
|
||||||
const result = originalOnNodeCreated?.apply(this, arguments);
|
|
||||||
setupNode(this);
|
|
||||||
return result;
|
|
||||||
};
|
|
||||||
|
|
||||||
const originalConfigure = nodeType.prototype.onConfigure;
|
|
||||||
nodeType.prototype.onConfigure = function onConfigure() {
|
|
||||||
const result = originalConfigure?.apply(this, arguments);
|
|
||||||
setupNode(this);
|
|
||||||
writeState(this, readState(this));
|
|
||||||
applyVisibility(this);
|
|
||||||
renderToolbar(this);
|
|
||||||
resizeNode(this);
|
|
||||||
return result;
|
|
||||||
};
|
|
||||||
|
|
||||||
const originalSerialize = nodeType.prototype.onSerialize;
|
|
||||||
nodeType.prototype.onSerialize = function onSerialize(o) {
|
|
||||||
writeState(this, readState(this));
|
|
||||||
const result = originalSerialize?.apply(this, arguments);
|
|
||||||
if (o && this.properties?.[STATE_PROPERTY]) {
|
|
||||||
o.properties = o.properties || {};
|
|
||||||
o.properties[STATE_PROPERTY] = this.properties[STATE_PROPERTY];
|
|
||||||
}
|
|
||||||
return result;
|
|
||||||
};
|
|
||||||
},
|
|
||||||
});
|
});
|
||||||
|
|||||||
@@ -0,0 +1,85 @@
|
|||||||
|
import { app } from "/scripts/app.js";
|
||||||
|
|
||||||
|
const NODE_NAME = "DumasH3PromptCurator";
|
||||||
|
const EXPECTED_OUTPUTS = [
|
||||||
|
"prompt",
|
||||||
|
"ref_image_1",
|
||||||
|
"ref_image_2",
|
||||||
|
"ref_image_3",
|
||||||
|
"ref_image_4",
|
||||||
|
"ref_image_5",
|
||||||
|
"ref_image_6",
|
||||||
|
"ref_image_7",
|
||||||
|
"ref_image_8",
|
||||||
|
"ref_image_9",
|
||||||
|
"reference_count",
|
||||||
|
"debug",
|
||||||
|
"anchor",
|
||||||
|
"sounds",
|
||||||
|
"bgm",
|
||||||
|
"original_ref_1",
|
||||||
|
"original_ref_2",
|
||||||
|
"original_ref_3",
|
||||||
|
"original_ref_4",
|
||||||
|
"original_ref_5",
|
||||||
|
"original_ref_6",
|
||||||
|
"original_ref_7",
|
||||||
|
"original_ref_8",
|
||||||
|
"original_ref_9",
|
||||||
|
"compiled_ref_description_1",
|
||||||
|
"compiled_ref_description_2",
|
||||||
|
"compiled_ref_description_3",
|
||||||
|
"compiled_ref_description_4",
|
||||||
|
"compiled_ref_description_5",
|
||||||
|
"compiled_ref_description_6",
|
||||||
|
"compiled_ref_description_7",
|
||||||
|
"compiled_ref_description_8",
|
||||||
|
"compiled_ref_description_9",
|
||||||
|
];
|
||||||
|
const EXPECTED_NAMES = new Set(EXPECTED_OUTPUTS);
|
||||||
|
|
||||||
|
function pruneStaleOutputs(node) {
|
||||||
|
if (!Array.isArray(node.outputs)) return;
|
||||||
|
|
||||||
|
const byName = new Map();
|
||||||
|
for (const output of node.outputs) {
|
||||||
|
if (!output?.name || !EXPECTED_NAMES.has(output.name) || byName.has(output.name)) continue;
|
||||||
|
byName.set(output.name, output);
|
||||||
|
}
|
||||||
|
|
||||||
|
const nextOutputs = [];
|
||||||
|
for (const name of EXPECTED_OUTPUTS) {
|
||||||
|
const existing = byName.get(name);
|
||||||
|
if (existing) {
|
||||||
|
nextOutputs.push(existing);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if (nextOutputs.length && nextOutputs.length !== node.outputs.length) {
|
||||||
|
node.outputs = nextOutputs;
|
||||||
|
node.size = node.computeSize?.() || node.size;
|
||||||
|
node.setDirtyCanvas?.(true, true);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
app.registerExtension({
|
||||||
|
name: "Dumas.H3PromptCuratorOutputs",
|
||||||
|
async beforeRegisterNodeDef(nodeType, nodeData) {
|
||||||
|
if (nodeData?.name !== NODE_NAME) return;
|
||||||
|
|
||||||
|
const originalOnNodeCreated = nodeType.prototype.onNodeCreated;
|
||||||
|
const originalOnConfigure = nodeType.prototype.onConfigure;
|
||||||
|
|
||||||
|
nodeType.prototype.onNodeCreated = function onNodeCreated() {
|
||||||
|
const created = originalOnNodeCreated?.apply(this, arguments);
|
||||||
|
pruneStaleOutputs(this);
|
||||||
|
return created;
|
||||||
|
};
|
||||||
|
|
||||||
|
nodeType.prototype.onConfigure = function onConfigure() {
|
||||||
|
const configured = originalOnConfigure?.apply(this, arguments);
|
||||||
|
pruneStaleOutputs(this);
|
||||||
|
return configured;
|
||||||
|
};
|
||||||
|
},
|
||||||
|
});
|
||||||
@@ -11,35 +11,62 @@ class DumasH3BeatPromptTests(unittest.TestCase):
|
|||||||
state = self.module._parse_beat_prompt_state("not json")
|
state = self.module._parse_beat_prompt_state("not json")
|
||||||
self.assertEqual(
|
self.assertEqual(
|
||||||
state,
|
state,
|
||||||
{"beats": [{"text": "Describe this beat."}]},
|
{
|
||||||
|
"scene": "",
|
||||||
|
"character_sheet": "",
|
||||||
|
"beats": [{"text": "Describe this beat."}],
|
||||||
|
},
|
||||||
)
|
)
|
||||||
|
|
||||||
def test_assemble_prompt_joins_beats_with_blank_lines(self):
|
def test_assemble_prompt_outputs_upstream_sections(self):
|
||||||
prompt = self.module._assemble_beat_prompt(
|
prompt = self.module._assemble_beat_prompt(
|
||||||
{
|
{
|
||||||
|
"scene": "A rainy kitchen at night.",
|
||||||
|
"character_sheet": "Maya: 27, she, red jacket, silver hair.",
|
||||||
"beats": [
|
"beats": [
|
||||||
{"text": "A woman enters the room."},
|
{"text": "Maya enters the room."},
|
||||||
{"text": "wardrobe: Maya = red jacket\nShe sits at the table."},
|
{"text": "remove: red jacket\nadd: white shirt underneath\nShe sits at the table."},
|
||||||
{"text": " "},
|
{"text": " "},
|
||||||
{"text": "music: low synth pulse"},
|
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
)
|
)
|
||||||
self.assertEqual(
|
self.assertEqual(
|
||||||
prompt,
|
prompt,
|
||||||
(
|
(
|
||||||
"A woman enters the room.\n\n"
|
"A rainy kitchen at night.\n\n"
|
||||||
"wardrobe: Maya = red jacket\nShe sits at the table.\n\n"
|
"Maya: 27, she, red jacket, silver hair.\n\n"
|
||||||
"music: low synth pulse"
|
"Maya enters the room.\n\n"
|
||||||
|
"remove: red jacket\nadd: white shirt underneath\nShe sits at the table."
|
||||||
),
|
),
|
||||||
)
|
)
|
||||||
|
|
||||||
|
def test_assemble_prompt_strips_old_dumas_directives(self):
|
||||||
|
prompt = self.module._assemble_beat_prompt(
|
||||||
|
{
|
||||||
|
"beats": [
|
||||||
|
{
|
||||||
|
"text": (
|
||||||
|
"seconds: 8\n"
|
||||||
|
"continuity: hard cut\n"
|
||||||
|
"ref_mode: every shot\n"
|
||||||
|
"soundscape: soft rain\n"
|
||||||
|
"music: low synth\n"
|
||||||
|
"Maya opens the cupboard.\n"
|
||||||
|
"remove: red jacket"
|
||||||
|
)
|
||||||
|
},
|
||||||
|
]
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
self.assertEqual(prompt, "Maya opens the cupboard.\nremove: red jacket")
|
||||||
|
|
||||||
def test_node_build_prompt_uses_hidden_state(self):
|
def test_node_build_prompt_uses_hidden_state(self):
|
||||||
node = self.module.DumasH3BeatPromptNode()
|
node = self.module.DumasH3BeatPromptNode()
|
||||||
result = node.build_prompt(
|
result = node.build_prompt(
|
||||||
'{"beats":[{"text":"Beat one"},{"text":"Beat two"}]}'
|
'{"scene":"Scene","character_sheet":"Maya: 27, she","beats":[{"text":"Beat one"},{"text":"Beat two"}]}'
|
||||||
)
|
)
|
||||||
self.assertEqual(result, ("Beat one\n\nBeat two",))
|
self.assertEqual(result, ("Scene\n\nMaya: 27, she\n\nBeat one\n\nBeat two",))
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
|
|||||||
+78
-1629
File diff suppressed because it is too large
Load Diff
@@ -499,6 +499,17 @@ class DumasImageNodeTests(unittest.TestCase):
|
|||||||
self.assertIsNone(result[3])
|
self.assertIsNone(result[3])
|
||||||
self.assertEqual(result[10], 2)
|
self.assertEqual(result[10], 2)
|
||||||
self.assertIn("input 3-><Picture 2> Coffee Shop", result[11])
|
self.assertIn("input 3-><Picture 2> Coffee Shop", result[11])
|
||||||
|
self.assertEqual(result[12], "grounded handheld thriller")
|
||||||
|
self.assertEqual(result[13], "steady rain")
|
||||||
|
self.assertEqual(result[14], "low suspense music")
|
||||||
|
self.assertIs(result[15], dave_image)
|
||||||
|
self.assertIs(result[16], cafe_image)
|
||||||
|
self.assertIsNone(result[17])
|
||||||
|
self.assertIn("<Picture 1> Dave", result[24])
|
||||||
|
self.assertNotIn("<Picture 2> Coffee Shop", result[24])
|
||||||
|
self.assertIn("<Picture 2> Coffee Shop", result[25])
|
||||||
|
self.assertIn("Location context for <Picture 2> Coffee Shop", result[25])
|
||||||
|
self.assertEqual(result[26], "")
|
||||||
|
|
||||||
def test_h3_prompt_curator_renumbers_explicit_reference_tags(self):
|
def test_h3_prompt_curator_renumbers_explicit_reference_tags(self):
|
||||||
node = self.image_nodes.DumasH3PromptCuratorNode()
|
node = self.image_nodes.DumasH3PromptCuratorNode()
|
||||||
@@ -636,8 +647,14 @@ class DumasImageNodeTests(unittest.TestCase):
|
|||||||
|
|
||||||
def test_h3_prompt_curator_uses_documented_reference_limits(self):
|
def test_h3_prompt_curator_uses_documented_reference_limits(self):
|
||||||
node = self.image_nodes.DumasH3PromptCuratorNode()
|
node = self.image_nodes.DumasH3PromptCuratorNode()
|
||||||
self.assertEqual(len(node.RETURN_TYPES), 12)
|
self.assertEqual(len(node.RETURN_TYPES), 33)
|
||||||
self.assertEqual(node.RETURN_NAMES[1:10], tuple(f"ref_image_{i}" for i in range(1, 10)))
|
self.assertEqual(node.RETURN_NAMES[1:10], tuple(f"ref_image_{i}" for i in range(1, 10)))
|
||||||
|
self.assertEqual(node.RETURN_NAMES[12:15], ("anchor", "sounds", "bgm"))
|
||||||
|
self.assertEqual(node.RETURN_NAMES[15:24], tuple(f"original_ref_{i}" for i in range(1, 10)))
|
||||||
|
self.assertEqual(
|
||||||
|
node.RETURN_NAMES[24:33],
|
||||||
|
tuple(f"compiled_ref_description_{i}" for i in range(1, 10)),
|
||||||
|
)
|
||||||
|
|
||||||
def test_normalize_reference_upgrades_generic_summary_with_socket_picture_id(self):
|
def test_normalize_reference_upgrades_generic_summary_with_socket_picture_id(self):
|
||||||
image = FakeTensorBatch()
|
image = FakeTensorBatch()
|
||||||
|
|||||||
Reference in New Issue
Block a user