Image-Text-to-Text
Transformers
Safetensors
mage_vl
multimodal
vision-language-model
mage-vl
video-understanding
streaming
conversational
custom_code
Instructions to use microsoft/Mage-VL with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use microsoft/Mage-VL with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="microsoft/Mage-VL", trust_remote_code=True) messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# Load model directly from transformers import AutoModelForImageTextToText model = AutoModelForImageTextToText.from_pretrained("microsoft/Mage-VL", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use microsoft/Mage-VL with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "microsoft/Mage-VL" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "microsoft/Mage-VL", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/microsoft/Mage-VL
- SGLang
How to use microsoft/Mage-VL with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "microsoft/Mage-VL" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "microsoft/Mage-VL", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "microsoft/Mage-VL" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "microsoft/Mage-VL", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use microsoft/Mage-VL with Docker Model Runner:
docker model run hf.co/microsoft/Mage-VL
docs: add install-free Transformers usage section
#2
by kcz358 - opened
README.md
CHANGED
|
@@ -219,6 +219,84 @@ pip install "transformers>=5.7" accelerate pillow torch torchvision \
|
|
| 219 |
|
| 220 |
Codec-based video inference also requires `ffmpeg` and `ffprobe` on `PATH`.
|
| 221 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 222 |
### Examples
|
| 223 |
|
| 224 |
Two sample inputs ship with this repository:
|
|
|
|
| 219 |
|
| 220 |
Codec-based video inference also requires `ffmpeg` and `ffprobe` on `PATH`.
|
| 221 |
|
| 222 |
+
### Using 🤗 Transformers directly
|
| 223 |
+
|
| 224 |
+
Mage-VL ships its modeling, processing, and chat-template code inside this repository, so image and frame-sampled video inference need **no script download and no extra package** beyond `transformers`:
|
| 225 |
+
|
| 226 |
+
```python
|
| 227 |
+
import torch
|
| 228 |
+
from PIL import Image
|
| 229 |
+
from transformers import AutoModelForCausalLM, AutoProcessor
|
| 230 |
+
|
| 231 |
+
model_id = "microsoft/Mage-VL"
|
| 232 |
+
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
|
| 233 |
+
model = AutoModelForCausalLM.from_pretrained(
|
| 234 |
+
model_id, trust_remote_code=True, torch_dtype="auto", device_map="auto"
|
| 235 |
+
).eval()
|
| 236 |
+
|
| 237 |
+
messages = [{"role": "user", "content": [
|
| 238 |
+
{"type": "image"},
|
| 239 |
+
{"type": "text", "text": "Describe this image in detail."},
|
| 240 |
+
]}]
|
| 241 |
+
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
|
| 242 |
+
|
| 243 |
+
inputs = processor(
|
| 244 |
+
text=[text], images=[Image.open("examples/dog.jpg").convert("RGB")], return_tensors="pt"
|
| 245 |
+
)
|
| 246 |
+
inputs = {k: (v.to(model.device) if hasattr(v, "to") else v) for k, v in inputs.items()}
|
| 247 |
+
inputs["pixel_values"] = inputs["pixel_values"].to(model.dtype)
|
| 248 |
+
|
| 249 |
+
with torch.inference_mode():
|
| 250 |
+
output = model.generate(**inputs, max_new_tokens=256, do_sample=False)
|
| 251 |
+
print(processor.tokenizer.decode(
|
| 252 |
+
output[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True
|
| 253 |
+
).strip())
|
| 254 |
+
```
|
| 255 |
+
|
| 256 |
+
> The image depicts a dog sitting on a patterned rug. The dog appears to be a
|
| 257 |
+
> medium-sized breed with a thick, fluffy coat. Its fur is primarily white with
|
| 258 |
+
> patches of black and brown. The dog's ears are perked up, and it has a calm and
|
| 259 |
+
> attentive expression. [...]
|
| 260 |
+
|
| 261 |
+
For video, use `{"type": "video"}` in the message and pass a list of PIL frames as `videos=[frames]`:
|
| 262 |
+
|
| 263 |
+
```python
|
| 264 |
+
import cv2
|
| 265 |
+
import numpy as np
|
| 266 |
+
from PIL import Image
|
| 267 |
+
|
| 268 |
+
def sample_video(path, num_frames=32):
|
| 269 |
+
capture = cv2.VideoCapture(path)
|
| 270 |
+
total = int(capture.get(cv2.CAP_PROP_FRAME_COUNT))
|
| 271 |
+
frames = []
|
| 272 |
+
for index in np.linspace(0, total - 1, min(num_frames, total), dtype=int):
|
| 273 |
+
capture.set(cv2.CAP_PROP_POS_FRAMES, int(index))
|
| 274 |
+
ok, frame = capture.read()
|
| 275 |
+
if ok:
|
| 276 |
+
frames.append(Image.fromarray(cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)))
|
| 277 |
+
capture.release()
|
| 278 |
+
return frames
|
| 279 |
+
|
| 280 |
+
messages = [{"role": "user", "content": [
|
| 281 |
+
{"type": "video"},
|
| 282 |
+
{"type": "text", "text": "Describe this video."},
|
| 283 |
+
]}]
|
| 284 |
+
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
|
| 285 |
+
|
| 286 |
+
inputs = processor(
|
| 287 |
+
text=[text],
|
| 288 |
+
videos=[sample_video("examples/soccer-broadcast.mp4", 32)],
|
| 289 |
+
return_tensors="pt",
|
| 290 |
+
padding=True,
|
| 291 |
+
)
|
| 292 |
+
```
|
| 293 |
+
|
| 294 |
+
> The video opens with a man in a black polo shirt, sporting a short haircut,
|
| 295 |
+
> standing in a stadium. He is holding a yellow microphone with the BBC Sport
|
| 296 |
+
> logo on it. The background reveals a large crowd of spectators. [...]
|
| 297 |
+
|
| 298 |
+
Add `attn_implementation="sdpa"` to `from_pretrained` if `flash-attn` is not installed. Codec-based video (H.264/HEVC, DCVC-RT) and streaming commentary use [`inference.py`](inference.py) below.
|
| 299 |
+
|
| 300 |
### Examples
|
| 301 |
|
| 302 |
Two sample inputs ship with this repository:
|