Instructions to use MarsupialAI/Monstral-123B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use MarsupialAI/Monstral-123B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="MarsupialAI/Monstral-123B")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("MarsupialAI/Monstral-123B")
model = AutoModelForCausalLM.from_pretrained("MarsupialAI/Monstral-123B")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use MarsupialAI/Monstral-123B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "MarsupialAI/Monstral-123B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "MarsupialAI/Monstral-123B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/MarsupialAI/Monstral-123B

SGLang

How to use MarsupialAI/Monstral-123B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "MarsupialAI/Monstral-123B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "MarsupialAI/Monstral-123B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "MarsupialAI/Monstral-123B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "MarsupialAI/Monstral-123B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use MarsupialAI/Monstral-123B with Docker Model Runner:
```
docker model run hf.co/MarsupialAI/Monstral-123B
```
Browse Quantizations to use this model in llama.cpp, Ollama, LM Studio, or any compatible app.

Monstral 123B

A Mistral-Large merge

This model is a slerp merge of Behemoth and Magnum V4. The intention was to moisten up Behemoth a bit and give it some of that Claude flavor, but without being nearly as thirsty as Magnum. I feel it succeeds in both areas.

Mergefuel:

TheDrummer/Behemoth-123B-v1
anthracite-org/magnum-v4-123b

See recipe.txt for full details.

This model is uncensored and perfectly capable of generating objectionable material. It is far less likely to return NSFW content for SFW prompts than Magnum V4, but you should still exercise caution. This model has been observed exhibiting toxic/aggressive behavior in certain RP scenarios, even when the character card isn't explicitely demanding it. As with any LLM, no factual claims made by the model should be taken at face value. You know that boilerplate safety disclaimer that most professional models have? Assume this has it too. This model is for entertainment purposes only.

GGUFs: https://huggingface.co/MarsupialAI/Monstral-123B_iMat_GGUF

EXL2: https://huggingface.co/MarsupialAI/Monstral-123B_4.0bpw_EXL2

Prompt Format

Mistral or Metharme

Gross Groveling

If you like this merge, Drummer would like me to beg you for donations on his behalf. Personally I think he just spends it all on drugs and extremely weird pornography, but on the off chance some of it goes to finetuning costs, this is the link: https://ko-fi.com/thedrummer

Downloads last month: 7

Safetensors

Model size

123B params

Tensor type

F16

Model tree for MarsupialAI/Monstral-123B

Quantizations

5 models

Collection including MarsupialAI/Monstral-123B

Merges

Collection

Stuff I've cooked myself, for better or for worse. • 14 items • Updated Jun 18, 2025 • 3