docs/studio.md

Studio: author a new shot

← Meridian · Installation · API · CLI

The self-hosted studio lets you place cameras in a reconstructed scene, inspect the geometry reference, and generate a take without writing a command for every path revision.

What updates in real time?

After reconstruction and point-cloud loading, the browser's 3D view updates interactively as you move cameras, change their aim or look through a key. This is the real-time authoring preview—not real-time generative video.

The full-path geometric-reference video is refreshed by the server after a valid edit. It uses GPU warping and video encoding, and may wait behind reconstruction or generation on the same service. The final generated take is a separate job started with Render this take. No fixed preview latency or frame-rate guarantee is implied.

Start the service

After installation, run from the release directory:

CARD=0 bash service/run.sh --host 127.0.0.1 --port 8412

Open http://127.0.0.1:8412 after the terminal prints ready. Startup loads VGGT-Omega, the VAE, the finetuned teacher, and the student adapter onto one GPU; the previously reported B200 cold-start time is about 95 seconds. The active shell must have the VGGT-Omega environment variables set.

If you downloaded the VAE locally, append --model-dir ../MiniMax-H3.

Keep the service private. The program defaults to 0.0.0.0 when --host is omitted; the command above deliberately binds to loopback. The service has no authentication, per-user isolation, upload quota, or bounded durable job queue. For access to a remote machine, use an SSH tunnel or a protected deployment with authentication, resource limits, and the safeguards required by the model license. Do not expose this development service directly to the internet.

From clip to take

1. Choose a clip and source window

Upload an MP4/MOV/WebM or choose one of the sample clips. The service normalizes it to H.264, 24 fps, and an aspect-preserving frame bounded by 1280 × 1280, with rotation baked in. Unlike the CLI, it performs this normalization automatically.

The current page requires at least 73 normalized input frames. It detects candidate hard cuts and prepares an initial window of at most 124 source frames, stopping before the next detected cut. Cut detection is heuristic; split a clip manually if a cut is missed or a flash is mistaken for one. Moving the window's start reconstructs the new span and resets the keys.

Select the take length separately. The page offers 73, 124, 175, or 243 output frames; the CLI exposes all eight supported model lengths. A longer take does not automatically mean a longer source window or additional captured action.

2. Start from a camera move

Use a template: orbit, push in, slide, crane, freeze + orbit, or the clip's own camera. Templates replace the existing keys; Ctrl+Z undoes an edit.

The source-camera template initializes editable endpoints; it is not an exact replay of every estimated source pose. Likewise, a template translated into a few keys is an editable approximation of its underlying parametric move. Inspect the resulting reference rather than assuming it matches a CLI command exactly.

3. Refine the keys

Each key chooses a camera and a time:

Control	Meaning
Camera position	Where to observe the scene from. Drag a camera in the 3D view.
aims at	The key's look-at point. Use pick in 3D or centre. This does not change the reconstruction's depth scale.
clip frame it shows	Source frame, indexed in the normalized uploaded clip.
output frame it lands on	Position in the generated take. The first and last keys anchor its endpoints.
lens	Horizontal field of view, converted to a multiplier over the selected source frame's estimated focal length.

Click a key's row or thumbnail to look through its camera; use back to the overview to see the whole path. While looking through a key, drag to aim, Shift-drag to translate, and scroll to dolly. Add a key at the preview frame to refine a segment. Camera roll is fixed to zero.

Go beyond a preset

A path can combine several stages: push forward → turn toward a detail → slide right → retreat. Add a key at each change of intention, then set its position and look-at point in the shared 3D scene. Position and aim interpolate along cubic curves; the source-frame map is interpolated separately. This is different from ramping yaw, translation and dolly together in one CLI sweep.

To let the camera travel while an instant holds, assign the same clip frame it shows to two or more keys at different output frames. Resume with a later source frame. Inspect every segment and the joins: several keys do not guarantee adequate geometry, subject visibility or generated continuity. The walkthrough plan includes a concrete timing sketch, not scene-independent camera coordinates or an already-generated demonstration.

4. Inspect the reference

After a valid edit, the studio re-warps the path before enabling generation:

• what the model sees: the grey-hole geometry reference at conditioning resolution.
• where the pixels are missing: the same view with missing regions highlighted in magenta.

The reference is generated by the same geometry path used for inference. Its browser playback is a compressed visualization, not a pixel-exact copy of the tensor. Coherent framing and stable surfaces matter more than a low missing-pixel percentage. Pay special attention to faces, thin structures, subject silhouettes, and new surfaces revealed by the camera.

5. Render and compare

Choose Render this take after checking the path. The service performs VAE encoding, three student forwards, decoding, and video writing; progress appears on the render screen. Generation recomputes the warp rather than reading the preview MP4 back into the model.

The take screen shows the selected source timeline, geometry reference, and generated shot in sync. Download out.mp4 or the three-up grid.mp4. Service outputs have no soundtrack; the CLI's out_audio.mp4 muxing workflow is not part of the studio. Camera archives (cams.npz) are CLI-only.

Source time and output time

The keyframe representation is {pos, look, src, t, ease, focal}. src is an absolute source index; t is an output index. Between two keys, the source rate is:

rate = (next.src - current.src) / (next.t - current.t)

• Rate 1: preserve the uploaded video's pace, including any slow motion or speed-up already baked into that video.
• Rate 0: hold one instant while the camera may move.
• Other positive rates: interpolate the source indices and round to frames, duplicating or skipping them. The UI warns outside holds and approximately 1:1 playback; this is not a motion interpolation system.
• Negative rates: rejected. Source keys must never run backward.

For slow motion or speed-ups, use the 24 fps source-retiming workflow first, then author a 1:1 path over that export. A warning about a keyframe segment does not mean pre-retimed footage is unsupported.

Check long takes carefully. The page prepares at most 124 source frames. If you stretch that entire source window across a 243-frame take with two endpoints, the source advances at roughly half speed; it does not play 124 frames normally and then automatically hold. To preserve pace, place an explicit key where live motion ends, followed by a hold, or use the CLI with enough pre-retimed input frames. Changing take length or choosing a template can change these rates.

Preview checks

The page enables rendering only after the latest valid warp passes these checks:

Check	Current threshold
Clearance proxy	ahead >= -0.1, in pivot-depth units.
Difference from source camera	Translation moved > 0.004, key orientation change turned > 0.5°, or focal change zoomed > 0.01.

ahead is the minimum over time of the fifth-percentile target-camera depth for valid points in the central source region. It helps flag fly-throughs; it is not a complete collision test or a guarantee that the camera stays outside every surface. moved measures departure from the source camera, not whether the target camera travels over time. A different but stationary view can pass.

The missing-pixel percentage is informative, not a gate. A pure focal change may pass the motion check while still being ignored by the model.

These clearance and camera-change checks live in the browser, not /render. API clients must inspect previews and validate their own requests; calling /render bypasses the page's checks. The API also does not enforce the page's cut-aware 124-frame window policy.

Memory and lifecycle

One process owns one GPU. A lock serializes GPU work, including preparation, previews, and generation; multiple requests do not yield concurrent GPU inference. Render requests start background threads that can wait on the lock, but there is no bounded queue, cancellation API, or durable job scheduler. Run one worker, not multiple Uvicorn workers that each load a model copy.

Option	Default	Purpose
--host, --port	0.0.0.0, 8412	Listen address; use loopback unless the deployment is protected.
--work	<release>/work	Uploads, preview files, and generated takes.
--samples	<release>/examples/media	Sample MP4s listed on the first screen.
--max-clips	8	Maximum number of decoded clips in the in-memory clip cache.
--max-prep	8	Maximum number of prepared source spans in the geometry cache.
--ckpt, --lora	Release directories	Teacher and student adapter. The service always loads an adapter.
--model-dir	MiniMaxAI/MiniMax-H3	VAE location.
--vggt-repo, --vggt	Environment-based	Geometry code and checkpoint.
--steps, --flow-shift	4, 3	Keep these at the student sampling settings for this release.

Prepared spans retain tensors on the GPU, and cache limits count entries, not bytes. Memory can grow as you explore different windows. Lower --max-prep, use shorter windows, or restart to release old sessions when operating near the memory limit. Reported single-take peaks do not bound a long-running service with many cached spans.

service/run.sh restarts the process only after exit code 3, used for a poisoned CUDA context. Other exits stop the wrapper. Clip, preparation, and job registries are in memory: a restart loses the live session even if files remain on disk. Re-upload/select the clip and prepare it again. Evicted clips or prepared spans can similarly invalidate older browser tabs.

Generated files are not automatically expired. Monitor <work>/clips, <work>/warp, and <work>/takes; stop the service before manually removing data still referenced by an active session. Keep uploaded footage private and use material you have permission to process.

Troubleshooting the studio

Symptom	Next step
Render is disabled	Wait for the latest warp, check key order and source direction, then inspect the clearance and camera-change messages.
The take unexpectedly slows down	Compare source and output indices, especially after choosing 175/243 frames or applying a template.
Preparation fails near a cut	Move to a continuous span with at least two source frames, or trim and upload the shot separately.
An old tab starts failing	Its cached clip or span may have been evicted, or the service restarted. Select the clip again.
Previews stop while a take renders	GPU work is serialized; there is no separate preview GPU.
Memory rises over a session	Reduce the prepared-span cache or restart; source-window length and cached tensors matter as well as output length.
Page reports a GPU restart	Watch the terminal for ready, then start a new session. Previous job IDs will not be restored.

See Installation for dependencies and API for programmatic use.