Instructions to use ryandt/MusingCaterpillar with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use ryandt/MusingCaterpillar with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="ryandt/MusingCaterpillar")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("ryandt/MusingCaterpillar")
model = AutoModelForCausalLM.from_pretrained("ryandt/MusingCaterpillar", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use ryandt/MusingCaterpillar with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ryandt/MusingCaterpillar"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ryandt/MusingCaterpillar",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/ryandt/MusingCaterpillar

SGLang

How to use ryandt/MusingCaterpillar with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "ryandt/MusingCaterpillar" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ryandt/MusingCaterpillar",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "ryandt/MusingCaterpillar" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ryandt/MusingCaterpillar",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use ryandt/MusingCaterpillar with Docker Model Runner:
```
docker model run hf.co/ryandt/MusingCaterpillar
```

This is not a 7B. It's a ~9B. Please label appropriately.

by georgewalker - opened Jan 10, 2024

Discussion

georgewalker

Jan 10, 2024

Like several of the top '7B' models on the leaderboard, this is actually a (roughly) 9B, downstream of https://huggingface.co/zyh3826/GML-Mistral-merged-v1, a merge that combined the first 32 layers (ie all of them) of one Mistral-7B finetune with the last 8 layers of another Mistral finetune, creating a model that is about 9B parameters.

It is helpful to label model sizes appropriately. Better would be if Huggingface labeled models based on their file size and bpw, instead of allowing for these sorts of mistakes to occur and proliferate, as one mislabeled model begets others derived from it.

ryandt

Owner Jan 11, 2024

Apologies for this. I will admit I'm new to this all. I don't remember declaring the size anywhere, and on the model card it says it is 8.99B.

In fact, on the LLM leaderboard submission page (as well as the model card page), I don't see where I would have set it to 7B.

If you let me know where I can correct this, I'm happy to do so.
Thanks

mrfakename

Jan 24, 2024

Yeah AFAIK you can't set it manually they try to auto-detect it. 9B is closer to 7 than 13 so its set to 7

ryandt

Owner Jan 24, 2024

Maybe I'm stupid, but I'm seeing it set to 8.99B:

where are you guys seeing this?

mrfakename

Jan 24, 2024

Sorry, on the Open LLM Leaderboard. It says 8.99B in SafeTensors

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment