Method

← Meridian · Inference · Studio

Meridian synthesizes a new observation of an existing event. It separates which source moment is shown from which camera observes it, then uses geometry to make that choice visible to a video model. The geometry supplies a spatial constraint; the model supplies the appearance of the completed shot, including regions the source camera did not see.

This is geometry-guided video re-camera, not a persistent 4D reconstruction or an action-conditioned simulator. A new view is a generated interpretation of the recorded event, not evidence of what an unobserved camera would actually have captured.

Overview. Source time selects both appearance and geometry; the authored camera makes a projected reference. Both references condition the video model. Real example: Spring (2019), © Blender Foundation, CC BY 4.0; input retimed, view projected and generated. Full figure, attribution and provenance.

1. Choose a source timeline

For each output frame t, a source-frame map s(t) selects the image and geometry to use:

Timeline	Source-frame selection
Preserve the input's pace	Advance one source frame per output frame.
Hold a moment	Repeat one source frame while the target camera can keep moving.
Slow motion or accelerated action	Retime the input to a constant 24 fps before reconstruction, then advance through that export normally.

Both video references follow the same selected timeline. Meridian is not asked to invent a different action speed from an unchanged reference. See Timing for frame-index semantics, freeze windows, and FFmpeg recipes.

The CLI constructs s(t) from --start, --frames, and optionally --freeze. The studio constructs it from keyframes: source indices interpolate linearly and are rounded to integers. Studio source keys must be non-decreasing; easing affects the camera path, not the source-frame mapping.

2. Reconstruct the source span

The input is resized and letterboxed into a 1280 × 1280 square, then downsampled to 512 × 512 for VGGT-Omega. One model call processes the selected source span jointly, returning per-frame depth, confidence, camera extrinsics, and intrinsics. Per-frame outputs do not mean independent single-frame inference. Changing the reconstruction span can change estimates for frames shared by both spans.

Before unprojection, the implementation removes:

Non-finite depth or confidence, and confidence values at or below 1e-5.
Depth discontinuities whose 3 × 3 local range exceeds 30% of the depth magnitude.
The lowest-confidence 2% of the remaining candidates in each frame.

Depth and validity are upsampled to the letterboxed input resolution. A pixel is retained only when the interpolated validity exceeds 0.999, limiting points introduced across rejected boundaries. Source RGB supplies the point colors. There is no fused mesh, persistent scene optimization, or cross-frame point-cloud accumulation in this stage.

Coordinates and scale

Geometry has a reconstruction-relative scale, not calibrated meters. Camera translations use zm, a median scene depth. Choosing a distant background as the depth reference makes the same numerical move much larger than choosing the subject.

CLI: a camera offset is applied in each selected source camera's local coordinates: C_target(t) = C_source(s(t)) @ delta(t). By default, zm comes from valid depths in the first selected frame; for --freeze, it comes from the held frame. --pivot fx,fy restricts the depth measurement to a neighborhood of a pixel in the cropped image. --pivot-lock also places the orbit center at the corresponding 3D point.
Studio: all keys share the coordinate frame of the source camera at start: x right, y down, z forward. Positions and look-at points are expressed in units of zm. The API measures zm around a chosen pixel at pivot_frame, falling back to valid picture depths when too few local points remain. The current page uses the picture center at start as this scale reference; a key's aims at control changes its look-at point, not the scale reference.

The CLI's source-relative trajectory and the studio's shared-frame trajectory are different ways of authoring a camera. Similar-looking controls need not produce identical paths on a moving-camera clip.

3. Render a geometric reference

At each output time, the selected source frame's colored point cloud is projected through the target camera and its lens. A z-buffer resolves visibility; each point splats onto a 3 × 3 pixel neighborhood. Uncovered pixels are filled with RGB (128, 128, 128).

The current implementation rasterizes at the output canvas, then downsamples the result to the 480-class conditioning canvas. There is no geometric inpainting before generation. Coverage is computed for diagnostics, but no coverage mask is fed to the transformer.

The studio's what the model sees preview and the saved render.mp4 show this downsampled reference. They are encoded video previews, not lossless copies of the in-memory conditioning pixels. The magenta-hole view is a diagnostic visualization only; the model receives the grey-hole version.

4. Condition the video transformer

MiniMax-H3's VAE encodes two references:

<Video 1> — source: the selected source images, at the 480 class.
<Video 2> — geometry: the rendered target view, at the same conditioning class.

The target is generated at the 768 class. Here “class” means an aspect-ratio bucket, not a fixed width or height. For a 16:9 input, the reference canvas is 832 × 480 and the output is 1344 × 768; square inputs use 640 × 640 and 1024 × 1024 respectively. The nearest bucket is chosen by log aspect ratio, with a centered crop inside the letterbox.

source span ──► joint VGGT-Omega reconstruction ──► per-frame geometry
      │                                                   │
      │                     source-frame map + camera path│
      │                                                   ▼
      │                                       z-buffered point splat
      │                                                   │
      ▼                                                   ▼
source reference, 480 class                    view reference, 480 class
      └────────────────────────┬──────────────────────────┘
                               ▼
              VAE → packed reference tokens + fixed text
                               ▼
               finetuned H3 + distilled LoRA → VAE decode
                               ▼
                     new shot, 768 class, 24 fps

The references are concatenated as tokens, not added as channels. recam/h3.py uses Diffusers' MiniMaxH3Ref2VAPrepareLayoutStep.build_ref2va_packed_sequence to create the reference layout, position IDs, and modality tags. The transformer architecture is unchanged.

Reference video rows receive the upstream conditioning-noise convention 0.999 × latent + 0.001 × noise and stay fixed during denoising. Target video rows begin as random noise. The source is also VAE-encoded at target resolution to establish the target latent shape; its values are not used to initialize the target rows.

Fixed text and the audio branch

assets/prompt.txt describes the two-reference editing task: retain the source event and complete the geometry reference's grey holes. Its embeddings are precomputed for each supported output length, so inference does not load Qwen3-VL. Editing the text file alone does not change inference; the shipped embeddings are what the model reads. An embedding-generation script is not included in this release.

The packed layout retains H3's audio branch. Cached silence latents supply its shape and length; the current code initializes audio rows with noise, denoises them, and discards the result. Meridian does not generate or preserve a soundtrack through that branch. The CLI's optional audio file is instead made by muxing the source soundtrack after video generation.

5. Sample the new shot

Mode	CLI settings	Transformer evaluations
Fast adapter, default	`--steps 4 --flow-shift 3` with the LoRA loaded	3
Teacher	`--no-lora --steps 50 --flow-shift 12`	49

The H3 scheduler counts the terminal zero-noise point in --steps; that endpoint does not require another model evaluation. The adapter must be loaded on Meridian's finetuned transformer, not the unmodified MiniMax-H3 checkpoint. Forward counts do not equal end-to-end speedups: geometry, VAE work, and file writing still take time.

Training overview

Training provenance, augmentations, and distillation design have moved to Training and distillation.

Implementation map

Source	What to read
`recam/geometry.py`	`reconstruct`, `warp`, and `render_hw`: geometry filtering, projection, and visibility.
`recam/path.py`	`plan_path` and `hermite`: keyframe interpolation, time mapping, and zero-roll look-at cameras.
`recam/h3.py`	`bucket`, `pack`, and `denoise`: canvases, reference conditioning, and the scheduler.
`inference/sample.py`	CLI time windows, parametric camera moves, diagnostics, and output files.
`service/app.py`	`prepare`, `geo`, and `do_render`: cached reconstruction and resident inference.

For weight provenance and modification notices, see MODIFICATIONS.md.