Instructions to use ai21labs/AI21-Jamba-Mini-1.5 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use ai21labs/AI21-Jamba-Mini-1.5 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="ai21labs/AI21-Jamba-Mini-1.5")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("ai21labs/AI21-Jamba-Mini-1.5")
model = AutoModelForCausalLM.from_pretrained("ai21labs/AI21-Jamba-Mini-1.5")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use ai21labs/AI21-Jamba-Mini-1.5 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ai21labs/AI21-Jamba-Mini-1.5"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ai21labs/AI21-Jamba-Mini-1.5",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/ai21labs/AI21-Jamba-Mini-1.5

SGLang

How to use ai21labs/AI21-Jamba-Mini-1.5 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "ai21labs/AI21-Jamba-Mini-1.5" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ai21labs/AI21-Jamba-Mini-1.5",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "ai21labs/AI21-Jamba-Mini-1.5" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ai21labs/AI21-Jamba-Mini-1.5",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use ai21labs/AI21-Jamba-Mini-1.5 with Docker Model Runner:
```
docker model run hf.co/ai21labs/AI21-Jamba-Mini-1.5
```

Evaluation using RULER

#12

by coranholmes - opened Sep 24, 2024

Discussion

coranholmes

Sep 24, 2024

Hi, I am trying to evaluate my sft model trained on Jamba-1.5-mini. Would you please share a bit more details about how you use RULER repo to evaluate your model? I use the container provided by the author and got the following error:

usage: prepare.py [-h] --save_dir SAVE_DIR [--benchmark BENCHMARK] --task TASK [--subset SUBSET] --tokenizer_path TOKENIZER_PATH [--tokenizer_type TOKENIZER_TYPE] --max_seq_length MAX_SEQ_LENGTH
                  [--num_samples NUM_SAMPLES] [--random_seed RANDOM_SEED] [--model_template_type MODEL_TEMPLATE_TYPE] [--remove_newline_tab] [--chunk_idx CHUNK_IDX] [--chunk_amount CHUNK_AMOUNT]
prepare.py: error: argument --tokenizer_type: expected one argument
[NeMo W 2024-09-24 15:27:02 nemo_logging:349] /usr/local/lib/python3.10/dist-packages/pydub/utils.py:170: RuntimeWarning: Couldn't find ffmpeg or avconv - defaulting to ffmpeg, but may not work
      warn("Couldn't find ffmpeg or avconv - defaulting to ffmpeg, but may not work", RuntimeWarning)

Predict niah_single_1
from benchmark_root/jamba_13k_v1/synthetic/4096/data/niah_single_1/validation.jsonl
to benchmark_root/jamba_13k_v1/synthetic/4096/pred/niah_single_1.jsonl
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/nemo/collections/asr/parts/utils/manifest_utils.py", line 476, in read_manifest
    f = open(manifest.get(), 'r', encoding='utf-8')
FileNotFoundError: [Errno 2] No such file or directory: 'benchmark_root/jamba_13k_v1/synthetic/4096/data/niah_single_1/validation.jsonl'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/mnt/afs/xxx/Codes/RULER/scripts/pred/call_api.py", line 333, in <module>
    main()
  File "/mnt/afs/xxx/Codes/RULER/scripts/pred/call_api.py", line 238, in main
    data = read_manifest(task_file)
  File "/usr/local/lib/python3.10/dist-packages/nemo/collections/asr/parts/utils/manifest_utils.py", line 478, in read_manifest
    raise Exception(f"Manifest file could not be opened: {manifest}")
Exception: Manifest file could not be opened: <class 'nemo.utils.data_utils.DataStoreObject'>: store_path=benchmark_root/jamba_13k_v1/synthetic/4096/data/niah_single_1/validation.jsonl, local_path=benchmark_root/jamba_13k_v1/synthetic/4096/data/niah_single_1/validation.jsonl
usage: prepare.py [-h] --save_dir SAVE_DIR [--benchmark BENCHMARK] --task TASK [--subset SUBSET] --tokenizer_path TOKENIZER_PATH [--tokenizer_type TOKENIZER_TYPE] --max_seq_length MAX_SEQ_LENGTH
                  [--num_samples NUM_SAMPLES] [--random_seed RANDOM_SEED] [--model_template_type MODEL_TEMPLATE_TYPE] [--remove_newline_tab] [--chunk_idx CHUNK_IDX] [--chunk_amount CHUNK_AMOUNT]
prepare.py: error: argument --tokenizer_type: expected one argument
[NeMo W 2024-09-24 15:27:10 nemo_logging:349] /usr/local/lib/python3.10/dist-packages/pydub/utils.py:170: RuntimeWarning: Couldn't find ffmpeg or avconv - defaulting to ffmpeg, but may not work
      warn("Couldn't find ffmpeg or avconv - defaulting to ffmpeg, but may not work", RuntimeWarning)

michael-go

AI21 org Oct 7, 2024

Hi,
Seems that you managed to resolve the --tokenizer_type error here: https://github.com/hsiehjackson/RULER/issues/69 ?

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment