Instructions to use Zoyd/afrizalha_Kancil-V1-llama3-fp16-4_25bpw_exl2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Zoyd/afrizalha_Kancil-V1-llama3-fp16-4_25bpw_exl2 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Zoyd/afrizalha_Kancil-V1-llama3-fp16-4_25bpw_exl2")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Zoyd/afrizalha_Kancil-V1-llama3-fp16-4_25bpw_exl2")
model = AutoModelForCausalLM.from_pretrained("Zoyd/afrizalha_Kancil-V1-llama3-fp16-4_25bpw_exl2")

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use Zoyd/afrizalha_Kancil-V1-llama3-fp16-4_25bpw_exl2 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Zoyd/afrizalha_Kancil-V1-llama3-fp16-4_25bpw_exl2"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Zoyd/afrizalha_Kancil-V1-llama3-fp16-4_25bpw_exl2",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/Zoyd/afrizalha_Kancil-V1-llama3-fp16-4_25bpw_exl2

SGLang

How to use Zoyd/afrizalha_Kancil-V1-llama3-fp16-4_25bpw_exl2 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Zoyd/afrizalha_Kancil-V1-llama3-fp16-4_25bpw_exl2" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Zoyd/afrizalha_Kancil-V1-llama3-fp16-4_25bpw_exl2",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Zoyd/afrizalha_Kancil-V1-llama3-fp16-4_25bpw_exl2" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Zoyd/afrizalha_Kancil-V1-llama3-fp16-4_25bpw_exl2",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Unsloth Studio new

How to use Zoyd/afrizalha_Kancil-V1-llama3-fp16-4_25bpw_exl2 with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for Zoyd/afrizalha_Kancil-V1-llama3-fp16-4_25bpw_exl2 to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for Zoyd/afrizalha_Kancil-V1-llama3-fp16-4_25bpw_exl2 to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for Zoyd/afrizalha_Kancil-V1-llama3-fp16-4_25bpw_exl2 to start chatting

Load model with FastModel

pip install unsloth
from unsloth import FastModel
model, tokenizer = FastModel.from_pretrained(
    model_name="Zoyd/afrizalha_Kancil-V1-llama3-fp16-4_25bpw_exl2",
    max_seq_length=2048,
)

Docker Model Runner
How to use Zoyd/afrizalha_Kancil-V1-llama3-fp16-4_25bpw_exl2 with Docker Model Runner:
```
docker model run hf.co/Zoyd/afrizalha_Kancil-V1-llama3-fp16-4_25bpw_exl2
```

Configuration Parsing Warning:In config.json: "quantization_config.bits" must be an integer

Exllamav2 quant (exl2 / 4.25 bpw) made with ExLlamaV2 v0.1.3

Other EXL2 quants:

Quant	Model Size	lm_head
2.2	3250 MB	6
2.5	3478 MB	6
3.0	3895 MB	6
3.5	4311 MB	6
3.75	4518 MB	6
4.0	4727 MB	6
4.25	4935 MB	6
5.0	5559 MB	6
6.0	6493 MB	8
6.5	6912 MB	8
8.0	8116 MB	8

Document Title

Introducing the Kancil family of open models

Kancil is a fine-tuned version of Llama 3 8B using synthetic QA dataset generated with Llama 3 70B. Version zero of Kancil is the first generative Indonesian LLM gain functional instruction performance using solely synthetic data.

❕Go straight to the colab demo❕

Beta preview

Selamat datang!

I am ultra-overjoyed to introduce you... the 🦌 Kancil! It's a fine-tuned version of Llama 3 8B with the Tumpeng, an instruction dataset of 14.8 million words. Both the model and dataset is openly available in Huggingface.

📚 The dataset was synthetically generated from Llama 3 70B. A big problem with existing Indonesian instruction dataset is they're in reality not-very-good-translations of English datasets. Llama 3 70B can generate fluent Indonesian! (with minor caveats 😔)

🦚 This follows previous efforts for collection of open, fine-tuned Indonesian models, like Merak and Cendol. However, Kancil solely leverages synthetic data in a very creative way, which makes it a very unique contribution!

Version 1.0

This is the second working prototype, Kancil V1. ✨ Training

2.2x Dataset word count
2x lora parameters
Rank-stabilized lora
2x fun

✨ New features

Multi-turn conversation (beta; optimized for curhat/personal advice 😂)
Better text generation (full or outline writing; optimized for essays)
QA from text (copy paste to prompt and ask a question about it)
Making slogans

This model was fine-tuned with QLoRA using the amazing Unsloth framework! It was built on top of unsloth/llama-3-8b-bnb-4bit and subsequently merged with the adapter.

Uses

This model is developed with research purposes for researchers or general AI hobbyists. However, it has one big application: You can have lots of fun with it!

Out-of-Scope Use

This is a research preview model with minimal safety curation. Do not use this model for commercial or practical applications.

You are also not allowed to use this model without having fun.

Getting started

As mentioned, this model was trained with Unsloth. Please use its code for better experience.

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

# Available versions
KancilV1 = "catinthebag/Kancil-V1-llama3-fp16"

# Load the model
tokenizer = AutoTokenizer.from_pretrained("catinthebag/Kancil-V1-llama3-fp16")
model = AutoModelForCausalLM.from_pretrained("catinthebag/Kancil-V1-llama3-fp16")
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model.to(device)

# This model was trained on this specific prompt template. Changing it might lead to performance degradations.
prompt_template = """<|user|>
{prompt}

<|assistant|>
{response}"""

# Start generating!
inputs = tokenizer(
[
prompt_template.format(
        prompt="""Bagaimana cara memberi tahu orang tua kalau saya ditolak universitas favorit saya?""",
        response="",)
], return_tensors = "pt").to("cuda")

outputs = model.generate(**inputs, max_new_tokens = 600, temperature=.3, use_cache = True)
print(tokenizer.batch_decode(outputs)[0].replace('\\n', '\n'))

Note: There is an issue with the dataset where the newline characters are interpreted as literal strings. Very sorry about this! 😔 Please keep the .replace() method to fix newline errors.

Acknowledgments

Developed by: Afrizal Hasbi Azizy
License: Llama 3 Community License Agreement

Downloads last month: 1

Zoyd
/

afrizalha_Kancil-V1-llama3-fp16-4_25bpw_exl2