Hugging Face's logo

MuScriptor
/
muscriptor-small

Safetensors
PyTorch
muscriptor
music
music-transcription
automatic-music-transcription
amt
audio-to-midi
midi
music-information-retrieval
transformer
Model card Files
xet
 Community
muscriptor-small / README.md
adefossez's picture
adefossez
Update README.md
8c127f6 verified
preview code
|
Raw
History Blame Contribute Delete
10.4 kB
metadata 
license: cc-by-nc-4.0
library_name: muscriptor
extra_gated_prompt: >-
  MuScriptor is the result of a research collaboration between Mirelo and Kyutai
  whose purpose is to transcribe audio to MIDI/music sheet. It is provided
  primarily for research purposes under the CC BY-NC 4.0 licence supplemented by
  the below specific conditions of use.

  Specific conditions of use: MuScriptor and any generated content by MuScriptor
  are provided as is without any warranty of any kind, including but not limited
  to any warranty of non-infringement. Use of MuScriptor and its output must
  comply with all applicable laws and must not result in, involve, or facilitate
  any illegal or unauthorized activity. Prohibited uses include, without
  limitation, inputting music files and transcribing them to MIDI/music sheet
  without having all the necessary rights, including intellectual property
  rights, under applicable laws. Accordingly, users of MuScriptor undertake and
  warrant to have all the necessary rights, including intellectual property
  rights, in connection with their use of MuScriptor and its output. We disclaim
  all liability for any non-compliant use and users of MuScriptor shall
  indemnify, defend, and hold harmless Mirelo and Kyutai from and against any
  and all claims, damages, losses, liabilities, and expenses (including
  reasonable attorneys' fees) incurred by Mirelo and/or Kyutai arising out of or
  resulting from their failure to comply with the terms of the CC BY-NC 4.0
  licence and/or these specific conditions of use.
extra_gated_fields:
  Company or university if applicable: text
  I am a:
    type: select
    options:
      - Musician
      - AI Researcher
      - Other
tags:
  - music
  - music-transcription
  - automatic-music-transcription
  - amt
  - audio-to-midi
  - midi
  - music-information-retrieval
  - transformer
  - pytorch

MuScriptor — small (≈100M)

MuScriptor is an open-weight model for general-purpose, multi-instrument automatic music transcription (AMT): it converts a music recording (any genre, multiple simultaneous instruments) into a stream of notes played. This repository hosts the small variant (≈100M parameters), the fastest and most lightweight checkpoint.

muscriptor-small is the smallest/fastest option, suitable for lower-resource settings. For higher quality use muscriptor-medium (≈300M, good trade-off) or muscriptor-large (≈1.3B, best quality).

Table of contents

Quickstart

Install the muscriptor package (it uses huggingface_hub to fetch weights automatically):

pip install git+https://github.com/muscriptor/muscriptor.git
# TODO (PyPI release forthcoming: pip install muscriptor)

Python 

from pathlib import Path
from muscriptor import TranscriptionModel

# "small" resolves to hf://MuScriptor/muscriptor-small and downloads on first use.
model = TranscriptionModel.load_model("small")

# Get a MIDI file directly:
Path("out.mid").write_bytes(model.transcribe_to_midi("audio.wav"))

# Or stream note events as they are transcribed:
for event in model.transcribe("audio.wav"):
    print(event)  # NoteStartEvent / NoteEndEvent / ProgressEvent

load_model accepts a size keyword ("small"/"medium"/"large"), a local .safetensors path, or an hf:// / https:// URL. Weights loaded by size keyword (or any hf:// URL) are cached in the standard Hugging Face cache (~/.cache/huggingface/hub, configurable via HF_HOME); weights fetched from a plain http(s):// URL are cached under ~/.cache/muscriptor/. Input audio can be WAV or any format libsndfile reads (mp3, flac, ogg, m4a, …); it is resampled to 16 kHz mono internally.

CLI 

muscriptor transcribe --model small audio.wav -o out.mid

Model description

MuScriptor performs transcription by autoregressively predicting a MIDI-like token sequence given the mel-spectrogram of a short audio segment, following the sequence-to-sequence AMT paradigm (cf. MT3). It deliberately avoids complex architectural tweaks in favor of a simple, decoder-only Transformer.

  • Architecture: decoder-only Transformer (this variant: dim=768, num_heads=12, num_layers=14).
  • Input: raw waveform (16 kHz, mono) of a 5-second segment → mel-spectrogram (STFT n_fft=2048, hop 160 → 100 Hz frame rate, 512 mel bins). The spectrogram is projected to the model dimension and used as a prefix condition.
  • Output tokenization: MT3-like note events; the 128 MIDI programs are mapped to 36 instrument subgroups using the MT3_FULL_PLUS taxonomy. Decoding is greedy (argmax) by default, with optional classifier-free guidance (CFG).
  • Inference: audio is processed in 5-second chunks; note events are emitted in temporal order. Optional instrument conditioning stabilizes predictions across chunk boundaries and lets you restrict/customize the transcription (see below).

Note on the representation: the tokenizer recovers onset/offset timing, pitch, and instrument, but not velocity. It also cannot represent two notes of the same pitch and instrument sounding at the same time. Drums are onset-only.

Model variants

Repo Params dim heads layers Notes
muscriptor-small ≈100M 768 12 14 this model · smallest / fastest
muscriptor-medium ≈300M 1024 16 24 good trade-off
muscriptor-large ≈1.3B 1536 24 48 best quality

All variants share the same input pipeline, tokenizer, and training recipe; they differ only in latent dimension, attention heads, and depth.

Intended uses & limitations

Intended uses

  • General-purpose transcription of real, multi-instrument music across genres (classical → heavy metal) into MIDI.
  • A building block for music information retrieval (chord/key recognition), musicological analysis, generative-modeling data pipelines, and tools for musicians.
  • Lower-resource / latency-sensitive settings where the larger variants are too heavy.

Out of scope / use with care

  • Not a substitute for a hand-annotated score; expect errors, especially on dense mixes, unusual timbres, and heavily processed audio.
  • Velocity/dynamics are not produced (see note above).
  • Onset/offset precision is lower for some styles (e.g. choral music), and exact offsets are inherently harder than onsets.
  • Being the smallest variant, it trades transcription accuracy for speed and footprint relative to medium/large.

Limitations & biases

  • Training data skews toward pop and Western classical music, and the instrument distribution is long-tailed (piano/guitar/bass/drums are most frequent). Rare instruments and underrepresented genres may be transcribed less reliably.
  • The fixed MT3_FULL_PLUS 36-group instrument taxonomy limits instrument granularity.
  • Simultaneous same-pitch/same-instrument notes cannot be represented by the tokenizer.

Instrument conditioning

The model can be told which instrument groups are present in the track. Supplying the correct set improves quantitative scores and produces more coherent instrument assignments across segments.

from muscriptor.tokenizer.mt3 import MT3_FULL_PLUS_GROUP_NAMES

# `instrument_group` is a space-separated string of MT3_FULL_PLUS group IDs.
# Convert readable group names to IDs:
names = ["acoustic_piano", "acoustic_guitar", "acoustic_bass"]
instrument_group = " ".join(str(MT3_FULL_PLUS_GROUP_NAMES[n]) for n in names)  # -> "0 4 7"

# Only expect piano, acoustic guitar and bass in this track:
model.transcribe_to_midi("audio.wav", instrument_group=instrument_group)

muscriptor transcribe --model small --instruments "acoustic_piano,acoustic_guitar,acoustic_bass" audio.wav -o out.mid
muscriptor list-instruments   # show all available group names

Evaluation

Metrics are instrument-agnostic F1 scores computed with mir_eval on D_Test, the authors' held-out test set of 372 multi-instrument tracks.

Model-size comparison (F1 ↑; from the paper's scaling study, models trained on D_Real only, CFG = 2):

Variant Params Onset Frame Offset Drums Multi
muscriptor-small 100M 51.2 67.2 38.7 41.5 38.2
muscriptor-medium 300M 52.4 68.0 40.3 42.0 39.7
muscriptor-large 1.3B 53.2 68.7 41.0 42.5 40.5

These numbers come from the model-size ablation, which trains on real audio only. The released checkpoints additionally use synthetic pre-training and RL post-training, which improve real-world quality substantially beyond these figures. See muscriptor-large and the paper for per-dataset results.

Citation 

@inproceedings{muscriptor2026,
  title     = {MuScriptor: An Open Model for Multi-Instrument Music Transcription},
  author    = {Rouard, Simon and Krause, Michael and Roebel, Axel and
               Simon-Gabriel, Carl-Johann and D{\'e}fossez, Alexandre},
  year      = {2026},
  note      = {Kyutai, Mirelo AI, IRCAM}
}

License

Code released under the MIT License. Weights released under CC-BY-NC.