Instructions to use OpenAssistant/llama2-13b-orca-8k-3319 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use OpenAssistant/llama2-13b-orca-8k-3319 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="OpenAssistant/llama2-13b-orca-8k-3319")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("OpenAssistant/llama2-13b-orca-8k-3319")
model = AutoModelForCausalLM.from_pretrained("OpenAssistant/llama2-13b-orca-8k-3319")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use OpenAssistant/llama2-13b-orca-8k-3319 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "OpenAssistant/llama2-13b-orca-8k-3319"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "OpenAssistant/llama2-13b-orca-8k-3319",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/OpenAssistant/llama2-13b-orca-8k-3319

SGLang

How to use OpenAssistant/llama2-13b-orca-8k-3319 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "OpenAssistant/llama2-13b-orca-8k-3319" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "OpenAssistant/llama2-13b-orca-8k-3319",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "OpenAssistant/llama2-13b-orca-8k-3319" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "OpenAssistant/llama2-13b-orca-8k-3319",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use OpenAssistant/llama2-13b-orca-8k-3319 with Docker Model Runner:
```
docker model run hf.co/OpenAssistant/llama2-13b-orca-8k-3319
```

llama2-13b-orca-8k-3319

Commit History

Update config.json (#7)

b02bf24

andreaskoepf

jordiclive commited on Jul 27, 2023

corrrect link to meta-llama/Llama-2-13b

a7d6df4

andreaskoepf commited on Jul 25, 2023

added sampling report

160f58e

andreaskoepf commited on Jul 25, 2023

add system prompt examples

b3c75ee

andreaskoepf commited on Jul 24, 2023

remove emtpy system prompt example

2096b54

andreaskoepf commited on Jul 24, 2023

update prompt info

fd1ead2

andreaskoepf commited on Jul 24, 2023

set max_position_embeddings and max_length to 8192

f0b152f

andreaskoepf commited on Jul 24, 2023

add usage example written by Jordi

ced264d

andreaskoepf commited on Jul 24, 2023

add text generation prompts

3d4a919

andreaskoepf commited on Jul 24, 2023

move dataset composition info

41d97e2

andreaskoepf commited on Jul 24, 2023

Update README.md (#1)

bd00b25

andreaskoepf

jordiclive commited on Jul 24, 2023

add llama2 license & use guide

4263228

Andreas Köpf commited on Jul 24, 2023

Update README.md

9517467

andreaskoepf commited on Jul 24, 2023

Update README.md

cfd15e0

andreaskoepf commited on Jul 24, 2023

Update README.md

ccfe9f2

andreaskoepf commited on Jul 24, 2023

Update README.md

0e3cc6b

andreaskoepf commited on Jul 24, 2023

Update README.md

c4a9aad

andreaskoepf commited on Jul 24, 2023

Update README.md

8a82537

andreaskoepf commited on Jul 24, 2023

Update README.md

7f844fd

andreaskoepf commited on Jul 24, 2023

Update README.md

776288c

andreaskoepf commited on Jul 24, 2023

Update README.md

ec69d63

andreaskoepf commited on Jul 24, 2023

Update README.md

8808e0f

andreaskoepf commited on Jul 24, 2023

Update README.md

5a7e696

andreaskoepf commited on Jul 24, 2023

Upload tokenizer

77def0d

andreaskoepf commited on Jul 24, 2023

Upload LlamaForCausalLM

cdedd68

andreaskoepf commited on Jul 24, 2023

Update README.md

72f506c

andreaskoepf commited on Jul 24, 2023

initial commit

a95cfcd

andreaskoepf commited on Jul 24, 2023

Commit History

Update config.json (#7) b02bf24

corrrect link to meta-llama/Llama-2-13b a7d6df4

added sampling report 160f58e

add system prompt examples b3c75ee

remove emtpy system prompt example 2096b54

update prompt info fd1ead2

set max_position_embeddings and max_length to 8192 f0b152f

add usage example written by Jordi ced264d

add text generation prompts 3d4a919

move dataset composition info 41d97e2

Update README.md (#1) bd00b25

add llama2 license & use guide 4263228

Update README.md 9517467

Update README.md cfd15e0

Update README.md ccfe9f2

Update README.md 0e3cc6b

Update README.md c4a9aad

Update README.md 8a82537

Update README.md 7f844fd

Update README.md 776288c

Update README.md ec69d63

Update README.md 8808e0f

Update README.md 5a7e696

Upload tokenizer 77def0d

Upload LlamaForCausalLM cdedd68

Update README.md 72f506c

initial commit a95cfcd

Update config.json (#7)

b02bf24

corrrect link to meta-llama/Llama-2-13b

a7d6df4

added sampling report

160f58e

add system prompt examples

b3c75ee

remove emtpy system prompt example

2096b54

update prompt info

fd1ead2

set max_position_embeddings and max_length to 8192

f0b152f

add usage example written by Jordi

ced264d

add text generation prompts

3d4a919

move dataset composition info

41d97e2

Update README.md (#1)

bd00b25

add llama2 license & use guide

4263228

Update README.md

9517467

Update README.md

cfd15e0

Update README.md

ccfe9f2

Update README.md

0e3cc6b

Update README.md

c4a9aad

Update README.md

8a82537

Update README.md

7f844fd

Update README.md

776288c

Update README.md

ec69d63

Update README.md

8808e0f

Update README.md

5a7e696

Upload tokenizer

77def0d

Upload LlamaForCausalLM

cdedd68

Update README.md

72f506c

initial commit

a95cfcd