Instructions to use tiny-random/deepseek-v4-bf16 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use tiny-random/deepseek-v4-bf16 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="tiny-random/deepseek-v4-bf16")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("tiny-random/deepseek-v4-bf16")
model = AutoModelForCausalLM.from_pretrained("tiny-random/deepseek-v4-bf16", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use tiny-random/deepseek-v4-bf16 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "tiny-random/deepseek-v4-bf16"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tiny-random/deepseek-v4-bf16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/tiny-random/deepseek-v4-bf16

SGLang

How to use tiny-random/deepseek-v4-bf16 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "tiny-random/deepseek-v4-bf16" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tiny-random/deepseek-v4-bf16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "tiny-random/deepseek-v4-bf16" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tiny-random/deepseek-v4-bf16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use tiny-random/deepseek-v4-bf16 with Docker Model Runner:
```
docker model run hf.co/tiny-random/deepseek-v4-bf16
```

hc_head.hc_base parameter name mismatch

by hnhyzz - opened 1 day ago

Discussion

hnhyzz

1 day ago

I encountered this error when serving the model using vllm with AMD cards.
The parameter name hc_head.hc_base is unexpected. Should it be hc_head_base?

(EngineCore pid=936) ERROR 08-01 02:59:59 [core.py:1349] File "/workspace/vllm/model_executor/models/utils.py", line 358, in _load_module
(EngineCore pid=936) ERROR 08-01 02:59:59 [core.py:1349] yield from self._load_module(
(EngineCore pid=936) ERROR 08-01 02:59:59 [core.py:1349] File "/workspace/vllm/model_executor/models/utils.py", line 331, in _load_module
(EngineCore pid=936) ERROR 08-01 02:59:59 [core.py:1349] loaded_params = module_load_weights(weights)
(EngineCore pid=936) ERROR 08-01 02:59:59 [core.py:1349] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=936) ERROR 08-01 02:59:59 [core.py:1349] File "/workspace/vllm/models/deepseek_v4/amd/model.py", line 860, in load_weights
(EngineCore pid=936) ERROR 08-01 02:59:59 [core.py:1349] param = params_dict[name]
(EngineCore pid=936) ERROR 08-01 02:59:59 [core.py:1349] ~~~~~~~~~~~^^^^^^
(EngineCore pid=936) ERROR 08-01 02:59:59 [core.py:1349] KeyError: 'hc_head.hc_base'

yujiepan

tiny-random org about 22 hours ago

Hello, I think the HF implementation's naming is incompatible with vLLM (see https://github.com/huggingface/transformers/blob/3717b9cda226af7377da1944f395af0eac6e2b51/src/transformers/models/deepseek_v4/modeling_deepseek_v4.py#L957).

	HF	vLLM	Deepseek official release
naming	hc_head.hc_base	hc_head_base	hc_head_base
bf16 support	yes	no (correct me if wrong)	no

For this bf16 model, i only tested on HF framework (see example usage) and thus followed HF's naming.

Maybe try tiny-random/deepseek-v4-fp4-fp8, which is tested on sglang (and should also work on vLLM）

yujiepan changed discussion status to closed about 22 hours ago

yujiepan changed discussion status to open about 22 hours ago

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment