Instructions to use wangkanai/qwen2.5-vl-3b-instruct with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use wangkanai/qwen2.5-vl-3b-instruct with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="wangkanai/qwen2.5-vl-3b-instruct")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("wangkanai/qwen2.5-vl-3b-instruct", dtype="auto")

llama-cpp-python

How to use wangkanai/qwen2.5-vl-3b-instruct with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="wangkanai/qwen2.5-vl-3b-instruct",
	filename="qwen2.5-vl-3b-instruct-abliterated-f16.gguf",
)

llm.create_chat_completion(
	messages = [
		{
			"role": "user",
			"content": [
				{
					"type": "text",
					"text": "Describe this image in one sentence."
				},
				{
					"type": "image_url",
					"image_url": {
						"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
					}
				}
			]
		}
	]
)

Notebooks
Google Colab
Kaggle
Local Apps

llama.cpp

How to use wangkanai/qwen2.5-vl-3b-instruct with llama.cpp:

Install from brew

brew install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf wangkanai/qwen2.5-vl-3b-instruct:F16
# Run inference directly in the terminal:
llama-cli -hf wangkanai/qwen2.5-vl-3b-instruct:F16

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf wangkanai/qwen2.5-vl-3b-instruct:F16
# Run inference directly in the terminal:
llama-cli -hf wangkanai/qwen2.5-vl-3b-instruct:F16

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf wangkanai/qwen2.5-vl-3b-instruct:F16
# Run inference directly in the terminal:
./llama-cli -hf wangkanai/qwen2.5-vl-3b-instruct:F16

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf wangkanai/qwen2.5-vl-3b-instruct:F16
# Run inference directly in the terminal:
./build/bin/llama-cli -hf wangkanai/qwen2.5-vl-3b-instruct:F16

Use Docker

docker model run hf.co/wangkanai/qwen2.5-vl-3b-instruct:F16

LM Studio
Jan

vLLM

How to use wangkanai/qwen2.5-vl-3b-instruct with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "wangkanai/qwen2.5-vl-3b-instruct"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "wangkanai/qwen2.5-vl-3b-instruct",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/wangkanai/qwen2.5-vl-3b-instruct:F16

SGLang

How to use wangkanai/qwen2.5-vl-3b-instruct with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "wangkanai/qwen2.5-vl-3b-instruct" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "wangkanai/qwen2.5-vl-3b-instruct",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "wangkanai/qwen2.5-vl-3b-instruct" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "wangkanai/qwen2.5-vl-3b-instruct",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Ollama
How to use wangkanai/qwen2.5-vl-3b-instruct with Ollama:
```
ollama run hf.co/wangkanai/qwen2.5-vl-3b-instruct:F16
```

Unsloth Studio new

How to use wangkanai/qwen2.5-vl-3b-instruct with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for wangkanai/qwen2.5-vl-3b-instruct to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for wangkanai/qwen2.5-vl-3b-instruct to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for wangkanai/qwen2.5-vl-3b-instruct to start chatting

Pi new

How to use wangkanai/qwen2.5-vl-3b-instruct with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama-server -hf wangkanai/qwen2.5-vl-3b-instruct:F16

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "wangkanai/qwen2.5-vl-3b-instruct:F16"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent new

How to use wangkanai/qwen2.5-vl-3b-instruct with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama-server -hf wangkanai/qwen2.5-vl-3b-instruct:F16

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default wangkanai/qwen2.5-vl-3b-instruct:F16

Run Hermes

hermes

Docker Model Runner
How to use wangkanai/qwen2.5-vl-3b-instruct with Docker Model Runner:
```
docker model run hf.co/wangkanai/qwen2.5-vl-3b-instruct:F16
```

Lemonade

How to use wangkanai/qwen2.5-vl-3b-instruct with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull wangkanai/qwen2.5-vl-3b-instruct:F16

Run and chat with the model

lemonade run user.qwen2.5-vl-3b-instruct-F16

List all available models

lemonade list

Qwen2.5-VL-3B-Instruct (Abliterated)

A vision-language model based on Qwen2.5-VL-3B-Instruct with safety filters removed ("abliterated"). This multimodal model can process both images and text to generate text responses, making it suitable for visual question answering, image captioning, and multimodal reasoning tasks.

Model Description

Qwen2.5-VL-3B-Instruct is a 3 billion parameter vision-language model from the Qwen family. This abliterated version has had its safety guardrails removed, allowing for more flexible and uncensored responses while maintaining the model's core capabilities in understanding and reasoning about visual content.

Key Capabilities:

Visual question answering
Image captioning and description
Multimodal reasoning and analysis
Document understanding and OCR
Chart and diagram interpretation
Scene understanding and spatial reasoning

Note: The "abliterated" designation means this model has reduced content filtering compared to the original release. Use responsibly and in accordance with applicable laws and regulations.

Repository Contents

This repository contains multiple model format variants optimized for different use cases:

File	Format	Size	Precision	Use Case
`qwen2.5-vl-3b-instruct-abliterated.safetensors`	SafeTensors	7.0 GB	FP32/BF16	Full precision, PyTorch/Transformers
`qwen2.5-vl-3b-instruct-abliterated-f16.gguf`	GGUF	5.76 GB	FP16	llama.cpp, high quality
`qwen2.5-vl-3b-instruct-abliterated-q5-k-m.gguf`	GGUF	2.07 GB	Q5_K_M	llama.cpp, balanced quality/size
`qwen2.5-vl-3b-instruct-abliterated-q4-k-m.gguf`	GGUF	1.80 GB	Q4_K_M	llama.cpp, maximum efficiency

Total Repository Size: ~17 GB (all variants)

Hardware Requirements

SafeTensors Format (.safetensors)

VRAM: 8-10 GB (FP16), 14-16 GB (FP32)
RAM: 16 GB minimum
Disk Space: 7.0 GB
Recommended GPU: NVIDIA RTX 3060 (12GB) or higher

GGUF Format (FP16)

VRAM: 6-8 GB
RAM: 12 GB minimum
Disk Space: 5.76 GB
Recommended: NVIDIA RTX 3060, AMD RX 6700 XT

GGUF Format (Q5_K_M)

VRAM: 3-4 GB
RAM: 8 GB minimum
Disk Space: 2.07 GB
Recommended: NVIDIA GTX 1660, RTX 3050

GGUF Format (Q4_K_M)

VRAM: 2-3 GB
RAM: 8 GB minimum
Disk Space: 1.80 GB
Recommended: NVIDIA GTX 1650, integrated GPUs

Usage Examples

Using with Transformers (SafeTensors)

from transformers import Qwen2VLForConditionalGeneration, AutoTokenizer, AutoProcessor
from PIL import Image
import torch

# Load model and processor
model_path = "E:/huggingface/qwen2.5-vl-3b-instruct/qwen2.5-vl-3b-instruct-abliterated.safetensors"
model = Qwen2VLForConditionalGeneration.from_pretrained(
    model_path,
    torch_dtype=torch.float16,
    device_map="auto"
)
processor = AutoProcessor.from_pretrained("Qwen/Qwen2-VL-3B-Instruct")

# Load and process image
image = Image.open("your_image.jpg")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image"},
            {"type": "text", "text": "Describe this image in detail."}
        ]
    }
]

# Prepare inputs
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = processor(text=[text], images=[image], return_tensors="pt").to(model.device)

# Generate response
with torch.inference_mode():
    generated_ids = model.generate(
        **inputs,
        max_new_tokens=512,
        temperature=0.7,
        top_p=0.9
    )

output = processor.batch_decode(
    generated_ids,
    skip_special_tokens=True,
    clean_up_tokenization_spaces=True
)[0]

print(output)

Using with llama.cpp (GGUF)

# Using FP16 version
llama-cli \
  --model "E:/huggingface/qwen2.5-vl-3b-instruct/qwen2.5-vl-3b-instruct-abliterated-f16.gguf" \
  --image "your_image.jpg" \
  --prompt "Describe this image" \
  --ctx-size 4096 \
  --n-predict 512

# Using Q5_K_M quantized version (recommended balance)
llama-cli \
  --model "E:/huggingface/qwen2.5-vl-3b-instruct/qwen2.5-vl-3b-instruct-abliterated-q5-k-m.gguf" \
  --image "your_image.jpg" \
  --prompt "What objects can you see in this image?" \
  --ctx-size 4096 \
  --n-predict 512 \
  --threads 8

# Using Q4_K_M quantized version (maximum efficiency)
llama-cli \
  --model "E:/huggingface/qwen2.5-vl-3b-instruct/qwen2.5-vl-3b-instruct-abliterated-q4-k-m.gguf" \
  --image "your_image.jpg" \
  --prompt "Analyze this image" \
  --ctx-size 4096 \
  --n-predict 512

Using with Python llama-cpp-python

from llama_cpp import Llama
from llama_cpp.llama_chat_format import Llava15ChatHandler

# Initialize with FP16
chat_handler = Llava15ChatHandler(clip_model_path="path/to/clip/model")
llm = Llama(
    model_path="E:/huggingface/qwen2.5-vl-3b-instruct/qwen2.5-vl-3b-instruct-abliterated-f16.gguf",
    chat_handler=chat_handler,
    n_ctx=4096,
    n_gpu_layers=-1,  # Use GPU acceleration
    verbose=False
)

# Generate response
response = llm.create_chat_completion(
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "image_url", "image_url": {"url": "file://path/to/image.jpg"}},
                {"type": "text", "text": "What's in this image?"}
            ]
        }
    ],
    max_tokens=512,
    temperature=0.7
)

print(response['choices'][0]['message']['content'])

Model Specifications

Specification	Details
Architecture	Qwen2-VL (Vision-Language Transformer)
Parameters	~3 billion
Base Model	Qwen2.5-VL-3B-Instruct
Vision Encoder	ViT-based visual encoder
Context Length	4096 tokens (text)
Languages	Primarily English, supports multilingual
Modifications	Abliterated (safety filters removed)
Formats Available	SafeTensors, GGUF (FP16, Q5_K_M, Q4_K_M)

Performance Tips and Optimization

For SafeTensors Format

Use torch.float16 or torch.bfloat16 for inference to reduce memory usage
Enable device_map="auto" for automatic GPU memory management
Use Flash Attention 2 if available: model.config.use_flash_attention_2 = True
Batch processing: Process multiple images in batches for better throughput

For GGUF Format

FP16: Best quality, use when VRAM allows (6-8 GB)
Q5_K_M: Recommended balance of quality and efficiency (3-4 GB VRAM)
Q4_K_M: Maximum efficiency for resource-constrained systems (2-3 GB VRAM)
Adjust --threads parameter based on CPU core count
Use --n-gpu-layers -1 to offload all layers to GPU when possible

Image Preprocessing

Resize images to reasonable dimensions (e.g., 1024x1024 max) before processing
Supported formats: JPEG, PNG, WebP, BMP
Use clear, well-lit images for best results
Higher resolution images require more VRAM

Generation Parameters

Temperature: 0.7-0.9 for creative descriptions, 0.1-0.3 for factual analysis
Top-p: 0.9-0.95 for diverse outputs, 0.7-0.8 for focused responses
Max tokens: 256-512 for descriptions, 1024+ for detailed analysis

Quantization Information

GGUF Quantization Schemes

Quantization	Description	Quality Loss	Memory Savings
FP16	Half precision, no quantization	~0%	~50% vs FP32
Q5_K_M	5-bit quantization, medium variant	<5%	~75% vs FP32
Q4_K_M	4-bit quantization, medium variant	5-10%	~80% vs FP32

Recommendation: Q5_K_M offers the best balance for most use cases, with minimal quality loss and significant memory savings.

License

This model is released under the Apache 2.0 License.

Copyright 2024 Qwen Team

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

Important Note: This is an abliterated (uncensored) version with safety filters removed. Users are responsible for ensuring their use complies with applicable laws, regulations, and ethical guidelines.

Citation

If you use this model in your research or applications, please cite:

@article{qwen2vl2024,
  title={Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution},
  author={Qwen Team},
  journal={arXiv preprint arXiv:2409.12191},
  year={2024}
}

Related Resources

Original Qwen2-VL: https://huggingface.co/Qwen/Qwen2-VL-3B-Instruct
Qwen Documentation: https://qwen.readthedocs.io/
Model Repository: https://github.com/QwenLM/Qwen2-VL
Transformers Documentation: https://huggingface.co/docs/transformers
llama.cpp: https://github.com/ggerganov/llama.cpp

Acknowledgments

Original model developed by the Qwen Team at Alibaba Cloud
Abliteration process removes safety guardrails while preserving model capabilities
GGUF quantization enables efficient deployment on consumer hardware