docs/studio.md

# Studio: author a new shot

[← Meridian](../README.md) · [Installation](installation.md) · [API](api.md) · [CLI](inference.md)

The self-hosted studio lets you place cameras in a reconstructed scene, inspect the geometry reference,
and generate a take without writing a command for every path revision.

## What updates in real time?

After reconstruction and point-cloud loading, the **browser's 3D view updates interactively** as you
move cameras, change their aim or look through a key. This is the real-time authoring preview—not
real-time generative video.

The **full-path geometric-reference video** is refreshed by the server after a valid edit. It uses
GPU warping and video encoding, and may wait behind reconstruction or generation on the same service.
The **final generated take** is a separate job started with **Render this take**. No fixed preview
latency or frame-rate guarantee is implied.

## Start the service

After [installation](installation.md), run from the release directory:

```bash
CARD=0 bash service/run.sh --host 127.0.0.1 --port 8412
```

Open `http://127.0.0.1:8412` after the terminal prints `ready`. Startup loads VGGT-Omega, the VAE,
the finetuned teacher, and the student adapter onto one GPU; the previously reported B200 cold-start
time is about 95 seconds. The active shell must have the VGGT-Omega environment variables set.

If you downloaded the VAE locally, append `--model-dir ../MiniMax-H3`.

**Keep the service private.** The program defaults to `0.0.0.0` when `--host` is omitted; the command
above deliberately binds to loopback. The service has no authentication, per-user isolation, upload
quota, or bounded durable job queue. For access to a remote machine, use an SSH tunnel or a protected
deployment with authentication, resource limits, and the safeguards required by the model license.
Do not expose this development service directly to the internet.

## From clip to take

### 1. Choose a clip and source window

Upload an MP4/MOV/WebM or choose one of the sample clips. The service normalizes it to H.264,
24 fps, and an aspect-preserving frame bounded by 1280 × 1280, with rotation baked in. Unlike the
CLI, it performs this normalization automatically.

The current page requires at least **73 normalized input frames**. It detects candidate hard cuts and
prepares an initial window of at most **124 source frames**, stopping before the next detected cut.
Cut detection is heuristic; split a clip manually if a cut is missed or a flash is mistaken for one.
Moving the window's start reconstructs the new span and **resets the keys**.

Select the take length separately. The page offers **73, 124, 175, or 243 output frames**; the CLI
exposes all eight supported model lengths. A longer take does not automatically mean a longer source
window or additional captured action.

### 2. Start from a camera move

Use a template: **orbit**, **push in**, **slide**, **crane**, **freeze + orbit**, or **the clip's own
camera**. Templates replace the existing keys; **Ctrl+Z** undoes an edit.

The source-camera template initializes editable endpoints; it is not an exact replay of every
estimated source pose. Likewise, a template translated into a few keys is an editable approximation
of its underlying parametric move. Inspect the resulting reference rather than assuming it matches
a CLI command exactly.

### 3. Refine the keys

Each key chooses a camera and a time:

| Control | Meaning |
|---|---|
| Camera position | Where to observe the scene from. Drag a camera in the 3D view. |
| **aims at** | The key's look-at point. Use **pick in 3D** or **centre**. This does not change the reconstruction's depth scale. |
| **clip frame it shows** | Source frame, indexed in the normalized uploaded clip. |
| **output frame it lands on** | Position in the generated take. The first and last keys anchor its endpoints. |
| **lens** | Horizontal field of view, converted to a multiplier over the selected source frame's estimated focal length. |

Click a key's row or thumbnail to look through its camera; use **back to the overview** to see the
whole path. While looking through a key, drag to aim, Shift-drag to translate, and scroll to dolly.
Add a key at the preview frame to refine a segment. Camera roll is fixed to zero.

### Go beyond a preset

A path can combine several stages: **push forward → turn toward a detail → slide right → retreat**.
Add a key at each change of intention, then set its position and look-at point in the shared 3D scene.
Position and aim interpolate along cubic curves; the source-frame map is interpolated separately.
This is different from ramping yaw, translation and dolly together in one CLI sweep.

To let the camera travel while an instant holds, assign the same **clip frame it shows** to two or
more keys at different output frames. Resume with a later source frame. Inspect every segment and
the joins: several keys do not guarantee adequate geometry, subject visibility or generated continuity.
The [walkthrough plan](studio_walkthrough.md#source-and-camera-design) includes a concrete timing
sketch, not scene-independent camera coordinates or an already-generated demonstration.

### 4. Inspect the reference

After a valid edit, the studio re-warps the path before enabling generation:

- **what the model sees:** the grey-hole geometry reference at conditioning resolution.
- **where the pixels are missing:** the same view with missing regions highlighted in magenta.

The reference is generated by the same geometry path used for inference. Its browser playback is a
compressed visualization, not a pixel-exact copy of the tensor. Coherent framing and stable surfaces
matter more than a low missing-pixel percentage. Pay special attention to faces, thin structures,
subject silhouettes, and new surfaces revealed by the camera.

### 5. Render and compare

Choose **Render this take** after checking the path. The service performs VAE encoding, three student
forwards, decoding, and video writing; progress appears on the render screen. Generation recomputes
the warp rather than reading the preview MP4 back into the model.

The take screen shows the selected source timeline, geometry reference, and generated shot in sync.
Download `out.mp4` or the three-up `grid.mp4`. Service outputs have no soundtrack; the CLI's
`out_audio.mp4` muxing workflow is not part of the studio. Camera archives (`cams.npz`) are CLI-only.

## Source time and output time

The keyframe representation is `{pos, look, src, t, ease, focal}`. `src` is an absolute source index;
`t` is an output index. Between two keys, the source rate is:

```text
rate = (next.src - current.src) / (next.t - current.t)
```

- **Rate 1:** preserve the uploaded video's pace, including any slow motion or speed-up already
  baked into that video.
- **Rate 0:** hold one instant while the camera may move.
- **Other positive rates:** interpolate the source indices and round to frames, duplicating or
  skipping them. The UI warns outside holds and approximately 1:1 playback; this is not a motion
  interpolation system.
- **Negative rates:** rejected. Source keys must never run backward.

For slow motion or speed-ups, use the [24 fps source-retiming workflow](inference.md#slow-motion-and-speed-ups)
first, then author a 1:1 path over that export. A warning about a keyframe segment does not mean
pre-retimed footage is unsupported.

**Check long takes carefully.** The page prepares at most 124 source frames. If you stretch that
entire source window across a 243-frame take with two endpoints, the source advances at roughly
half speed; it does not play 124 frames normally and then automatically hold. To preserve pace,
place an explicit key where live motion ends, followed by a hold, or use the CLI with enough
pre-retimed input frames. Changing take length or choosing a template can change these rates.

## Preview checks

The page enables rendering only after the latest valid warp passes these checks:

| Check | Current threshold |
|---|---|
| Clearance proxy | `ahead >= -0.1`, in pivot-depth units. |
| Difference from source camera | Translation `moved > 0.004`, key orientation change `turned > 0.5°`, or focal change `zoomed > 0.01`. |

`ahead` is the minimum over time of the fifth-percentile target-camera depth for valid points in
the central source region. It helps flag fly-throughs; it is **not** a complete collision test or a
guarantee that the camera stays outside every surface. `moved` measures departure from the source
camera, not whether the target camera travels over time. A different but stationary view can pass.

The missing-pixel percentage is informative, not a gate. A pure focal change may pass the motion
check while still being ignored by the model.

**These clearance and camera-change checks live in the browser, not `/render`.** API clients must
inspect previews and validate their own requests; calling `/render` bypasses the page's checks.
The API also does not enforce the page's cut-aware 124-frame window policy.

## Memory and lifecycle

One process owns one GPU. A lock serializes GPU work, including preparation, previews, and generation;
multiple requests do not yield concurrent GPU inference. Render requests start background threads
that can wait on the lock, but there is no bounded queue, cancellation API, or durable job scheduler.
Run **one worker**, not multiple Uvicorn workers that each load a model copy.

| Option | Default | Purpose |
|---|---|---|
| `--host`, `--port` | `0.0.0.0`, `8412` | Listen address; use loopback unless the deployment is protected. |
| `--work` | `<release>/work` | Uploads, preview files, and generated takes. |
| `--samples` | `<release>/examples/media` | Sample MP4s listed on the first screen. |
| `--max-clips` | `8` | Maximum number of decoded clips in the in-memory clip cache. |
| `--max-prep` | `8` | Maximum number of prepared source spans in the geometry cache. |
| `--ckpt`, `--lora` | Release directories | Teacher and student adapter. The service always loads an adapter. |
| `--model-dir` | `MiniMaxAI/MiniMax-H3` | VAE location. |
| `--vggt-repo`, `--vggt` | Environment-based | Geometry code and checkpoint. |
| `--steps`, `--flow-shift` | `4`, `3` | Keep these at the student sampling settings for this release. |

Prepared spans retain tensors on the GPU, and cache limits count **entries**, not bytes. Memory can
grow as you explore different windows. Lower `--max-prep`, use shorter windows, or restart to release
old sessions when operating near the memory limit. Reported single-take peaks do not bound a
long-running service with many cached spans.

`service/run.sh` restarts the process only after exit code `3`, used for a poisoned CUDA context.
Other exits stop the wrapper. Clip, preparation, and job registries are in memory: a restart loses
the live session even if files remain on disk. Re-upload/select the clip and prepare it again.
Evicted clips or prepared spans can similarly invalidate older browser tabs.

Generated files are not automatically expired. Monitor `<work>/clips`, `<work>/warp`, and
`<work>/takes`; stop the service before manually removing data still referenced by an active session.
Keep uploaded footage private and use material you have permission to process.

## Troubleshooting the studio

| Symptom | Next step |
|---|---|
| Render is disabled | Wait for the latest warp, check key order and source direction, then inspect the clearance and camera-change messages. |
| The take unexpectedly slows down | Compare source and output indices, especially after choosing 175/243 frames or applying a template. |
| Preparation fails near a cut | Move to a continuous span with at least two source frames, or trim and upload the shot separately. |
| An old tab starts failing | Its cached clip or span may have been evicted, or the service restarted. Select the clip again. |
| Previews stop while a take renders | GPU work is serialized; there is no separate preview GPU. |
| Memory rises over a session | Reduce the prepared-span cache or restart; source-window length and cached tensors matter as well as output length. |
| Page reports a GPU restart | Watch the terminal for `ready`, then start a new session. Previous job IDs will not be restored. |

See [Installation](installation.md#setup-problems) for dependencies and [API](api.md) for programmatic use.