docs: add install-free Transformers usage section

#2
by kcz358 - opened
Files changed (1) hide show
  1. README.md +78 -0
README.md CHANGED
@@ -219,6 +219,84 @@ pip install "transformers>=5.7" accelerate pillow torch torchvision \
219
 
220
  Codec-based video inference also requires `ffmpeg` and `ffprobe` on `PATH`.
221
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
222
  ### Examples
223
 
224
  Two sample inputs ship with this repository:
 
219
 
220
  Codec-based video inference also requires `ffmpeg` and `ffprobe` on `PATH`.
221
 
222
+ ### Using 🤗 Transformers directly
223
+
224
+ Mage-VL ships its modeling, processing, and chat-template code inside this repository, so image and frame-sampled video inference need **no script download and no extra package** beyond `transformers`:
225
+
226
+ ```python
227
+ import torch
228
+ from PIL import Image
229
+ from transformers import AutoModelForCausalLM, AutoProcessor
230
+
231
+ model_id = "microsoft/Mage-VL"
232
+ processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
233
+ model = AutoModelForCausalLM.from_pretrained(
234
+ model_id, trust_remote_code=True, torch_dtype="auto", device_map="auto"
235
+ ).eval()
236
+
237
+ messages = [{"role": "user", "content": [
238
+ {"type": "image"},
239
+ {"type": "text", "text": "Describe this image in detail."},
240
+ ]}]
241
+ text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
242
+
243
+ inputs = processor(
244
+ text=[text], images=[Image.open("examples/dog.jpg").convert("RGB")], return_tensors="pt"
245
+ )
246
+ inputs = {k: (v.to(model.device) if hasattr(v, "to") else v) for k, v in inputs.items()}
247
+ inputs["pixel_values"] = inputs["pixel_values"].to(model.dtype)
248
+
249
+ with torch.inference_mode():
250
+ output = model.generate(**inputs, max_new_tokens=256, do_sample=False)
251
+ print(processor.tokenizer.decode(
252
+ output[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True
253
+ ).strip())
254
+ ```
255
+
256
+ > The image depicts a dog sitting on a patterned rug. The dog appears to be a
257
+ > medium-sized breed with a thick, fluffy coat. Its fur is primarily white with
258
+ > patches of black and brown. The dog's ears are perked up, and it has a calm and
259
+ > attentive expression. [...]
260
+
261
+ For video, use `{"type": "video"}` in the message and pass a list of PIL frames as `videos=[frames]`:
262
+
263
+ ```python
264
+ import cv2
265
+ import numpy as np
266
+ from PIL import Image
267
+
268
+ def sample_video(path, num_frames=32):
269
+ capture = cv2.VideoCapture(path)
270
+ total = int(capture.get(cv2.CAP_PROP_FRAME_COUNT))
271
+ frames = []
272
+ for index in np.linspace(0, total - 1, min(num_frames, total), dtype=int):
273
+ capture.set(cv2.CAP_PROP_POS_FRAMES, int(index))
274
+ ok, frame = capture.read()
275
+ if ok:
276
+ frames.append(Image.fromarray(cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)))
277
+ capture.release()
278
+ return frames
279
+
280
+ messages = [{"role": "user", "content": [
281
+ {"type": "video"},
282
+ {"type": "text", "text": "Describe this video."},
283
+ ]}]
284
+ text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
285
+
286
+ inputs = processor(
287
+ text=[text],
288
+ videos=[sample_video("examples/soccer-broadcast.mp4", 32)],
289
+ return_tensors="pt",
290
+ padding=True,
291
+ )
292
+ ```
293
+
294
+ > The video opens with a man in a black polo shirt, sporting a short haircut,
295
+ > standing in a stadium. He is holding a yellow microphone with the BBC Sport
296
+ > logo on it. The background reveals a large crowd of spectators. [...]
297
+
298
+ Add `attn_implementation="sdpa"` to `from_pretrained` if `flash-attn` is not installed. Codec-based video (H.264/HEVC, DCVC-RT) and streaming commentary use [`inference.py`](inference.py) below.
299
+
300
  ### Examples
301
 
302
  Two sample inputs ship with this repository: