Instructions to use BEE-spoke-data/smol_llama-220M-open_instruct with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use BEE-spoke-data/smol_llama-220M-open_instruct with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="BEE-spoke-data/smol_llama-220M-open_instruct")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("BEE-spoke-data/smol_llama-220M-open_instruct")
model = AutoModelForCausalLM.from_pretrained("BEE-spoke-data/smol_llama-220M-open_instruct")

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use BEE-spoke-data/smol_llama-220M-open_instruct with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "BEE-spoke-data/smol_llama-220M-open_instruct"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "BEE-spoke-data/smol_llama-220M-open_instruct",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/BEE-spoke-data/smol_llama-220M-open_instruct

SGLang

How to use BEE-spoke-data/smol_llama-220M-open_instruct with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "BEE-spoke-data/smol_llama-220M-open_instruct" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "BEE-spoke-data/smol_llama-220M-open_instruct",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "BEE-spoke-data/smol_llama-220M-open_instruct" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "BEE-spoke-data/smol_llama-220M-open_instruct",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use BEE-spoke-data/smol_llama-220M-open_instruct with Docker Model Runner:
```
docker model run hf.co/BEE-spoke-data/smol_llama-220M-open_instruct
```
Browse Quantizations to use this model in llama.cpp, Ollama, LM Studio, or any compatible app.

BEE-spoke-data/smol_llama-220M-open_instruct

Please note that this is an experiment, and the model has limitations because it is smol.

prompt format is alpaca.

Below is an instruction that describes a task, paired with an input that
provides further context. Write a response that appropriately completes
the request.  

### Instruction:  

How can I increase my meme production/output? Currently, I only create them in ancient babylonian which is time consuming.  

### Response:

This was not trained using a separate 'inputs' field (as VMware/open-instruct doesn't use one).

Example

Output on the text above ^. The inference API is set to sample with low temp so you should see (at least slightly) different generations each time.

Note that the inference API parameters used here are an initial educated guess, and may be updated over time:

inference:
  parameters:
    do_sample: true
    renormalize_logits: true
    temperature: 0.25
    top_p: 0.95
    top_k: 50
    min_new_tokens: 2
    max_new_tokens: 96
    repetition_penalty: 1.04
    no_repeat_ngram_size: 6
    epsilon_cutoff: 0.0006

Feel free to experiment with the parameters using the model in Python and let us know if you have improved results with other params!

Data

This was trained on VMware/open-instruct so do whatever you want, provided it falls under the base apache-2.0 license :)

Open LLM Leaderboard Evaluation Results

Detailed results can be found here

Metric	Value
Avg.	29.19
AI2 Reasoning Challenge (25-Shot)	25.00
HellaSwag (10-Shot)	29.71
MMLU (5-Shot)	26.11
TruthfulQA (0-shot)	44.06
Winogrande (5-shot)	50.28
GSM8k (5-shot)	0.00