Instructions to use qingcheng-ai/QWQ-32B-FP8 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use qingcheng-ai/QWQ-32B-FP8 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="qingcheng-ai/QWQ-32B-FP8")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("qingcheng-ai/QWQ-32B-FP8")
model = AutoModelForCausalLM.from_pretrained("qingcheng-ai/QWQ-32B-FP8")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use qingcheng-ai/QWQ-32B-FP8 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "qingcheng-ai/QWQ-32B-FP8"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "qingcheng-ai/QWQ-32B-FP8",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/qingcheng-ai/QWQ-32B-FP8

SGLang

How to use qingcheng-ai/QWQ-32B-FP8 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "qingcheng-ai/QWQ-32B-FP8" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "qingcheng-ai/QWQ-32B-FP8",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "qingcheng-ai/QWQ-32B-FP8" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "qingcheng-ai/QWQ-32B-FP8",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use qingcheng-ai/QWQ-32B-FP8 with Docker Model Runner:
```
docker model run hf.co/qingcheng-ai/QWQ-32B-FP8
```

Model Overview

Description

FP8 Quantized QwQ-32B.

Evaluation

The test results in the following table are based on the MMLU benchmark.

In order to speed up the test, we prevent the model from generating too long thought chains, so the score may be different from that with longer thought chain.

In our experiment, the accuracy of the FP8 quantized version is almost the same as the BF16 version, and it can be used for faster inference.

Data Format	MMLU Score
BF16 Official	61.2
FP8 Quantized	61.2
Q8_0 (INT8)	59.1
AWQ (INT4)	53.4

Quickstart

We recommend using the Chitu inference framework(https://github.com/thu-pacman/chitu) to run this model.

Here provides a simple command to show you how to run QwQ-32B-FP8.

torchrun --nproc_per_node 1 \
    --master_port=22525 \
    -m chitu \
    serve.port=21002 \
    infer.cache_type=paged \
    infer.pp_size=1 \
    infer.tp_size=1 \
    models=QwQ-32B-FP8 \
    models.ckpt_dir="your model path" \
    models.tokenizer_path="your model path" \
    dtype=float16 \
    infer.do_load=True \
    infer.max_reqs=1 \
    scheduler.prefill_first.num_tasks=100 \
    infer.max_seq_len=4096 \
    request.max_new_tokens=100 \
    infer.use_cuda_graph=True

Contact

solution@qingcheng.ai

Downloads last month: 8

Safetensors

Model size

33B params

Tensor type

F32

F16

F8_E4M3

Model tree for qingcheng-ai/QWQ-32B-FP8

Base model

Qwen/Qwen2.5-32B

Finetuned

Qwen/QwQ-32B

Quantized

(162)

this model