Instructions to use openaccess-ai-collective/wizard-mega-13b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use openaccess-ai-collective/wizard-mega-13b with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="openaccess-ai-collective/wizard-mega-13b")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("openaccess-ai-collective/wizard-mega-13b")
model = AutoModelForCausalLM.from_pretrained("openaccess-ai-collective/wizard-mega-13b")

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use openaccess-ai-collective/wizard-mega-13b with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "openaccess-ai-collective/wizard-mega-13b"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "openaccess-ai-collective/wizard-mega-13b",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/openaccess-ai-collective/wizard-mega-13b

SGLang

How to use openaccess-ai-collective/wizard-mega-13b with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "openaccess-ai-collective/wizard-mega-13b" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "openaccess-ai-collective/wizard-mega-13b",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "openaccess-ai-collective/wizard-mega-13b" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "openaccess-ai-collective/wizard-mega-13b",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use openaccess-ai-collective/wizard-mega-13b with Docker Model Runner:
```
docker model run hf.co/openaccess-ai-collective/wizard-mega-13b
```

Prompt format contradiction

by 2EyeGuy - opened May 18, 2023

Discussion

2EyeGuy

May 18, 2023

•

edited May 18, 2023

You give two examples of the following prompt format:
### Instruction: <question>\n\n### Assistant: <answer>\n\n

But your hugging face space uses this version of the Alpaca prompt format:
### Instruction: \n<question>\n\n### Response:\n<answer>\n\n

And this version of the Vicuna 1.1 prompt format:
USER: <question>\nASSISTANT: <answer>\n

But reading the axolotl source code and looking at your config file, shows that you trained two of the models with a different version of the Vicuna 1.1 instruction format, and one of the models with the Alpaca instruction format. Which is a weird way of doing it.

Two of the datasets:
USER: <question> ASSISTANT: <answer></s>

The other dataset:
Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\n<question>\n\n### Response:\n<answer>

So what format should I actually use?

EDIT: fixed my mistake about the hugging face space

winglian

Open Access AI Collective org May 19, 2023

My goal is to give the model enough examples that it is capable of handling any prompt format and could infer the meaning.

Aillian

Oct 17, 2023

have you figured it out?

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment