Instructions to use QuantTrio/Qwen3.5-27B-AWQ with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use QuantTrio/Qwen3.5-27B-AWQ with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="QuantTrio/Qwen3.5-27B-AWQ")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForImageTextToText

processor = AutoProcessor.from_pretrained("QuantTrio/Qwen3.5-27B-AWQ")
model = AutoModelForImageTextToText.from_pretrained("QuantTrio/Qwen3.5-27B-AWQ")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use QuantTrio/Qwen3.5-27B-AWQ with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "QuantTrio/Qwen3.5-27B-AWQ"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "QuantTrio/Qwen3.5-27B-AWQ",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/QuantTrio/Qwen3.5-27B-AWQ

SGLang

How to use QuantTrio/Qwen3.5-27B-AWQ with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "QuantTrio/Qwen3.5-27B-AWQ" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "QuantTrio/Qwen3.5-27B-AWQ",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "QuantTrio/Qwen3.5-27B-AWQ" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "QuantTrio/Qwen3.5-27B-AWQ",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use QuantTrio/Qwen3.5-27B-AWQ with Docker Model Runner:
```
docker model run hf.co/QuantTrio/Qwen3.5-27B-AWQ
```

My personal vLLM launch cmd on my old personal 2x3090 workstation

by tclf90 - opened Mar 2

Discussion

tclf90

QuantTrio org Mar 2

•

edited Mar 2

vllm serve \
    ...path.../tclf90/Qwen3.5-35B-A3B-AWQ \
    --served-model-name Qwen3.5-35B-A3B-AWQ \
    --swap-space 4 \
    --max-num-seqs 4 \
    --enable-prefix-caching \
    --max_num_batched_tokens 2112 \
    --kv-cache-dtype fp8_e4m3 \
    --tensor-parallel-size 2 \
    --gpu-memory-utilization 0.9 \
    --max-model-len 262144 \
    --enable-auto-tool-choice \
    --tool-call-parser qwen3_coder \
    --reasoning-parser qwen3 \
    --no-enforce-eager \
    --compilation-config.mode 3 \
    --compilation-config.cudagraph_mode FULL_AND_PIECEWISE \
    --compilation-config.cudagraph_capture_sizes [1,2,4,8,16,24,32,40,48,56,64] \
    --compilation-config.max_cudagraph_capture_size 64 \
    --compilation-config.use_inductor_graph_partition true \
    --trust-remote-code \
    --host localhost \
    --port 8000

roughly 110t/s for single request (without mtp)
(mtp is slower on this machine, not sure why, so i turned it off for now)

dwaynedu

Mar 3

Stuck on vllm startup. Running vllm nightly, 1xH20

tclf90

QuantTrio org Mar 3

Stuck on vllm startup. Running vllm nightly, 1xH20

These logs LGTM.
I think it takes time... kind of...

bastoker

Mar 9

•

edited Mar 11

vllm serve \
    ...path.../tclf90/Qwen3.5-35B-A3B-AWQ \
    --served-model-name Qwen3.5-35B-A3B-AWQ \
    --swap-space 4 \
    --max-num-seqs 4 \
    --enable-prefix-caching \
    --max_num_batched_tokens 2112 \
    --kv-cache-dtype fp8_e4m3 \
    --tensor-parallel-size 2 \
    --gpu-memory-utilization 0.9 \
    --max-model-len 262144 \
    --enable-auto-tool-choice \
    --tool-call-parser qwen3_coder \
    --reasoning-parser qwen3 \
    --no-enforce-eager \
    --compilation-config.mode 3 \
    --compilation-config.cudagraph_mode FULL_AND_PIECEWISE \
    --compilation-config.cudagraph_capture_sizes [1,2,4,8,16,24,32,40,48,56,64] \
    --compilation-config.max_cudagraph_capture_size 64 \
    --compilation-config.use_inductor_graph_partition true \
    --trust-remote-code \
    --host localhost \
    --port 8000

roughly 110t/s for single request (without mtp)
(mtp is slower on this machine, not sure why, so i turned it off for now)

Thanks for the config, for a single request I get ~55 tps on dual 5060Ti 16GB with MTP and ~35 tps without:

    -v /media/bas/images/.cache:/root/.cache \
    --network=host \
    --ipc=host \
    vllm/vllm-openai:nightly \
    "QuantTrio/Qwen3.5-27B-AWQ" \
    --port 8000 --host 0.0.0.0 \
    --trust-remote-code --tensor-parallel 2 --max-model-len $((1024*128)) --max-num-seqs 1 --gpu-memory-utilization 0.87 \
    --kv-cache-dtype fp8_e4m3 \
    --max-num-batched-tokens 2112 \
    --language-model-only \
    --performance-mode interactivity \
    --enable-auto-tool-choice \
    --tool-call-parser qwen3_coder \
    --reasoning-parser qwen3 \
    --enable-prefix-caching \
    --speculative-config '{"method":"qwen3_next_mtp","num_speculative_tokens":2}' \
    --compilation-config.mode 3 \
    --compilation-config.cudagraph_mode FULL_AND_PIECEWISE \
    --compilation-config.cudagraph_capture_sizes [1,2,4,8,16,24,32] \
    --compilation-config.max_cudagraph_capture_size 32 \
    --compilation-config.use_inductor_graph_partition true

docker image: https://hub.docker.com/layers/vllm/vllm-openai/nightly/images - vllm image id when i pulled was sha256:364d579a2bc60dd4ad5c2cabf5d79d45b979d7867a424ce38ecd28f158c81ad4

mancub

Mar 15

•

edited Mar 17

roughly 110t/s for single request (without mtp)
(mtp is slower on this machine, not sure why, so i turned it off for now)

I have a vllm nightly setup in Debian via Proxmox using 2x3090 with P2P driver and I can barely get 65t/s. This is using your configuration above + no thinking.

Are your 3090s NVLink-ed?

My other hardware is Epyc 7J13 on a H12SSL-NT with 256GB RAM...it can't be my hardware?!

rumcs

Apr 4

roughly 110t/s for single request (without mtp)
(mtp is slower on this machine, not sure why, so i turned it off for now)

I have a vllm nightly setup in Debian via Proxmox using 2x3090 with P2P driver and I can barely get 65t/s. This is using your configuration above + no thinking.

Are your 3090s NVLink-ed?

My other hardware is Epyc 7J13 on a H12SSL-NT with 256GB RAM...it can't be my hardware?!

I think the 110 tps is actually for 35b a3b, not this 27b.

mancub

Apr 4

I think the 110 tps is actually for 35b a3b, not this 27b.

Yeah it would seem so...and I was hoping for some magic 😀

Forcewithme

Apr 8

for a single request I get ~55 tps on dual 5060Ti 16GB with MTP and ~35 tps without
Hi @bastoker , how long is the prefill context?

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment