Instructions to use qingcheng-ai/Qwen3-8B-fp4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use qingcheng-ai/Qwen3-8B-fp4 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="qingcheng-ai/Qwen3-8B-fp4")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("qingcheng-ai/Qwen3-8B-fp4")
model = AutoModelForCausalLM.from_pretrained("qingcheng-ai/Qwen3-8B-fp4")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use qingcheng-ai/Qwen3-8B-fp4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "qingcheng-ai/Qwen3-8B-fp4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "qingcheng-ai/Qwen3-8B-fp4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/qingcheng-ai/Qwen3-8B-fp4

SGLang

How to use qingcheng-ai/Qwen3-8B-fp4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "qingcheng-ai/Qwen3-8B-fp4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "qingcheng-ai/Qwen3-8B-fp4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "qingcheng-ai/Qwen3-8B-fp4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "qingcheng-ai/Qwen3-8B-fp4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use qingcheng-ai/Qwen3-8B-fp4 with Docker Model Runner:
```
docker model run hf.co/qingcheng-ai/Qwen3-8B-fp4
```

Evaluation

The test results in the following table are based on the MMLU benchmark.

In order to speed up the test, we prevent the model from generating too long thought chains, so the score may be different from that with longer thought chain.

In our experiment, the accuracy of the FP4 quantized version is almost the same as the BF16 version, and it can be used for faster inference.

Data Format	MMLU Score
BF16 Official	79.92
FP4 Quantized	79.50

Quickstart

We recommend using the Chitu inference framework(https://github.com/thu-pacman/chitu) to run this model. Here provides a simple command to show you how to run Qwen3-8B-fp4.

torchrun --nproc_per_node 1 \
    --master_port=22525 \
    -m chitu \
    serve.port=21002 \
    infer.cache_type=paged \
    infer.pp_size=1 \
    infer.tp_size=1 \
    models=Qwen3-8B-fp4 \
    models.ckpt_dir="your model path" \
    models.tokenizer_path="your model path" \
    dtype=float16 \
    infer.do_load=True \
    infer.max_reqs=1 \
    scheduler.prefill_first.num_tasks=100 \
    infer.max_seq_len=4096 \
    request.max_new_tokens=100 \
    infer.use_cuda_graph=True

Contact

solution@qingcheng.ai

Downloads last month: 3

Safetensors

Model size

5B params

Tensor type

F16

F8_E4M3

Model tree for qingcheng-ai/Qwen3-8B-fp4

Base model

Qwen/Qwen3-8B-Base

Finetuned

Qwen/Qwen3-8B

Quantized

(287)

this model