Muose-50M-Decision

Muose-50M-Decision is a 50.76M parameter decision model built on the custom Muose transformer architecture.

Architecture

• 50,760,960 total parameters
• 8 transformer layers
• 640 hidden dimension
• 10 query attention heads
• 2 KV heads
• Grouped Query Attention
• RoPE
• RMSNorm
• SwiGLU
• Custom 24,000-token Byte-Level BPE tokenizer
• Scalar decision scoring head

Muose evaluates candidate criteria against a state and instruction, then converts their scores into a probability distribution.

Example

STATE:

A user repeatedly posts advertisements.

INSTRUCTIONS:

Which moderation action best matches the situation?

CRITERIA:

• keep: Normal useful discussion
• remove: Unsolicited promotional spam
• review: Content requiring human review

Muose scores each criterion independently.

Loading

Install:

pip install torch safetensors tokenizers

Basic loading:

import json

from safetensors.torch import load_model
from tokenizers import ByteLevelBPETokenizer

from model import Muose, MuoseConfig
from decision_model import MuoseDecision


with open("config.json", "r") as f:
    release_config = json.load(f)


config = MuoseConfig(
    **release_config["base_config"]
)

base = Muose(config)

model = MuoseDecision(base)

load_model(
    model,
    "model.safetensors",
)

model.eval()

tokenizer = ByteLevelBPETokenizer(
    "vocab.json",
    "merges.txt",
)

Training

Base pretraining:

• FineWeb-Edu
• Approximately 500 million training tokens
• Trained from random initialization

Decision fine-tuning:

• BANKING77

Evaluation

BANKING77 official test:

• 6-way accuracy: 96.07%
• 20-way accuracy: 90.08%
• 77-way accuracy: 79.94%

Custom out-of-domain personal test suite:

• 16 / 20 correct
• 80.0%

Limitations

This is an experimental small decision model.

The model was primarily decision-tuned using banking intent data and may perform substantially worse on out-of-domain decisions.

Output score distributions are not calibrated probabilities of correctness.

Muose may make incorrect decisions with high confidence.

License

Muose-50M-Decision is released under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International license.

CC BY-NC-SA 4.0

Commercial use is not permitted under this license.

For separate commercial licensing, contact the model owner.