Instructions to use PygmalionAI/pygmalion-6b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use PygmalionAI/pygmalion-6b with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="PygmalionAI/pygmalion-6b")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("PygmalionAI/pygmalion-6b")
model = AutoModelForCausalLM.from_pretrained("PygmalionAI/pygmalion-6b")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use PygmalionAI/pygmalion-6b with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "PygmalionAI/pygmalion-6b"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "PygmalionAI/pygmalion-6b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/PygmalionAI/pygmalion-6b

SGLang

How to use PygmalionAI/pygmalion-6b with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "PygmalionAI/pygmalion-6b" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "PygmalionAI/pygmalion-6b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "PygmalionAI/pygmalion-6b" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "PygmalionAI/pygmalion-6b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use PygmalionAI/pygmalion-6b with Docker Model Runner:
```
docker model run hf.co/PygmalionAI/pygmalion-6b
```

Is there any plan for a Pygmalion model based on OPT

#11

by Conanak99 - opened Feb 22, 2023

Discussion

Conanak99

Feb 22, 2023

•

edited Feb 22, 2023

Recently, with the help of Flexgen, we can offload OPT model with limited GPU memory https://github.com/FMInference/FlexGen

Some experiments show that we can offload OPT-6.7B and OPT-13B with just ~2GB VRAM.

Unfortunately, Pygmalion is based on GPT model. Is there any plan for training another Pygmalion model based on OPT? This will help people with low GPU run the model locally, and we can also run bigger Pygmalion model on Colab with 16GB limit.

11b

Pygmalion org Feb 23, 2023

Not at the moment, since I don't like how restrictive OPT's license is.

However, I am keeping an eye on the project since the developers plan to support other model architectures (at least according to their roadmap). If I get the compute resources to train bigger models, they will likely be based on NeoX, so feel free leave a thumbs up for NeoX support on the FlexGen repo: https://github.com/FMInference/FlexGen/issues/9

mirrorcult

Mar 9, 2023

possibility of a model based on LLaMA 7B/13B? though I imagine the same restrictiveness applies there

mirrorcult

Mar 9, 2023

but would ofc be much more cost efficient to fine tune than neoX-20B (and much more creative)

11b

Pygmalion org Mar 10, 2023

possibility of a model based on LLaMA 7B/13B? though I imagine the same restrictiveness applies there

Yep, the CEO of HuggingFace himself has asked people to not upload any LLaMA models until further notice. The claims about the model's performance are very exciting though, so if Meta allows distribution of fine-tunes I do plan on trying it.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment