Instructions to use mlx-community/aya-vision-32b-6bit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use mlx-community/aya-vision-32b-6bit with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="mlx-community/aya-vision-32b-6bit")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForImageTextToText

processor = AutoProcessor.from_pretrained("mlx-community/aya-vision-32b-6bit")
model = AutoModelForImageTextToText.from_pretrained("mlx-community/aya-vision-32b-6bit")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

MLX

How to use mlx-community/aya-vision-32b-6bit with MLX:

# Make sure mlx-vlm is installed
# pip install --upgrade mlx-vlm

from mlx_vlm import load, generate
from mlx_vlm.prompt_utils import apply_chat_template
from mlx_vlm.utils import load_config

# Load the model
model, processor = load("mlx-community/aya-vision-32b-6bit")
config = load_config("mlx-community/aya-vision-32b-6bit")

# Prepare input
image = ["http://images.cocodataset.org/val2017/000000039769.jpg"]
prompt = "Describe this image."

# Apply chat template
formatted_prompt = apply_chat_template(
    processor, config, prompt, num_images=1
)

# Generate output
output = generate(model, processor, formatted_prompt, image)
print(output)

Notebooks
Google Colab
Kaggle
Local Apps
LM Studio

vLLM

How to use mlx-community/aya-vision-32b-6bit with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "mlx-community/aya-vision-32b-6bit"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "mlx-community/aya-vision-32b-6bit",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/mlx-community/aya-vision-32b-6bit

SGLang

How to use mlx-community/aya-vision-32b-6bit with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "mlx-community/aya-vision-32b-6bit" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "mlx-community/aya-vision-32b-6bit",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "mlx-community/aya-vision-32b-6bit" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "mlx-community/aya-vision-32b-6bit",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use mlx-community/aya-vision-32b-6bit with Docker Model Runner:
```
docker model run hf.co/mlx-community/aya-vision-32b-6bit
```

aya-vision-32b-6bit / tokenizer_config.json

prince-canuma

Upload folder using huggingface_hub

f21cf1d verified about 1 year ago

raw

history blame contribute delete

9.87 kB

	{
	"add_bos_token": true,
	"add_eos_token": false,
	"add_prefix_space": false,
	"added_tokens_decoder": {
	"0": {
	"content": "<PAD>",
	"lstrip": false,
	"normalized": false,
	"rstrip": false,
	"single_word": false,
	"special": true
	},
	"1": {
	"content": "<UNK>",
	"lstrip": false,
	"normalized": false,
	"rstrip": false,
	"single_word": false,
	"special": true
	},
	"2": {
	"content": "<CLS>",
	"lstrip": false,
	"normalized": false,
	"rstrip": false,
	"single_word": false,
	"special": true
	},
	"3": {
	"content": "<SEP>",
	"lstrip": false,
	"normalized": false,
	"rstrip": false,
	"single_word": false,
	"special": true
	},
	"4": {
	"content": "<MASK_TOKEN>",
	"lstrip": false,
	"normalized": false,
	"rstrip": false,
	"single_word": false,
	"special": true
	},
	"5": {
	"content": "<BOS_TOKEN>",
	"lstrip": false,
	"normalized": false,
	"rstrip": false,
	"single_word": false,
	"special": true
	},
	"6": {
	"content": "<EOS_TOKEN>",
	"lstrip": false,
	"normalized": false,
	"rstrip": false,
	"single_word": false,
	"special": true
	},
	"7": {
	"content": "<EOP_TOKEN>",
	"lstrip": false,
	"normalized": false,
	"rstrip": false,
	"single_word": false,
	"special": true
	},
	"255000": {
	"content": "<\|START_OF_TURN_TOKEN\|>",
	"lstrip": false,
	"normalized": false,
	"rstrip": false,
	"single_word": false,
	"special": false
	},
	"255001": {
	"content": "<\|END_OF_TURN_TOKEN\|>",
	"lstrip": false,
	"normalized": false,
	"rstrip": false,
	"single_word": false,
	"special": true
	},
	"255002": {
	"content": "<\|YES_TOKEN\|>",
	"lstrip": false,
	"normalized": false,
	"rstrip": false,
	"single_word": false,
	"special": false
	},
	"255003": {
	"content": "<\|NO_TOKEN\|>",
	"lstrip": false,
	"normalized": false,
	"rstrip": false,
	"single_word": false,
	"special": false
	},
	"255004": {
	"content": "<\|GOOD_TOKEN\|>",
	"lstrip": false,
	"normalized": false,
	"rstrip": false,
	"single_word": false,
	"special": false
	},
	"255005": {
	"content": "<\|BAD_TOKEN\|>",
	"lstrip": false,
	"normalized": false,
	"rstrip": false,
	"single_word": false,
	"special": false
	},
	"255006": {
	"content": "<\|USER_TOKEN\|>",
	"lstrip": false,
	"normalized": false,
	"rstrip": false,
	"single_word": false,
	"special": false
	},
	"255007": {
	"content": "<\|CHATBOT_TOKEN\|>",
	"lstrip": false,
	"normalized": false,
	"rstrip": false,
	"single_word": false,
	"special": false
	},
	"255008": {
	"content": "<\|SYSTEM_TOKEN\|>",
	"lstrip": false,
	"normalized": false,
	"rstrip": false,
	"single_word": false,
	"special": false
	},
	"255009": {
	"content": "<\|USER_0_TOKEN\|>",
	"lstrip": false,
	"normalized": false,
	"rstrip": false,
	"single_word": false,
	"special": false
	},
	"255010": {
	"content": "<\|USER_1_TOKEN\|>",
	"lstrip": false,
	"normalized": false,
	"rstrip": false,
	"single_word": false,
	"special": false
	},
	"255011": {
	"content": "<\|USER_2_TOKEN\|>",
	"lstrip": false,
	"normalized": false,
	"rstrip": false,
	"single_word": false,
	"special": false
	},
	"255012": {
	"content": "<\|USER_3_TOKEN\|>",
	"lstrip": false,
	"normalized": false,
	"rstrip": false,
	"single_word": false,
	"special": false
	},
	"255013": {
	"content": "<\|USER_4_TOKEN\|>",
	"lstrip": false,
	"normalized": false,
	"rstrip": false,
	"single_word": false,
	"special": false
	},
	"255014": {
	"content": "<\|USER_5_TOKEN\|>",
	"lstrip": false,
	"normalized": false,
	"rstrip": false,
	"single_word": false,
	"special": false
	},
	"255015": {
	"content": "<\|USER_6_TOKEN\|>",
	"lstrip": false,
	"normalized": false,
	"rstrip": false,
	"single_word": false,
	"special": false
	},
	"255016": {
	"content": "<\|USER_7_TOKEN\|>",
	"lstrip": false,
	"normalized": false,
	"rstrip": false,
	"single_word": false,
	"special": false
	},
	"255017": {
	"content": "<\|USER_8_TOKEN\|>",
	"lstrip": false,
	"normalized": false,
	"rstrip": false,
	"single_word": false,
	"special": false
	},
	"255018": {
	"content": "<\|USER_9_TOKEN\|>",
	"lstrip": false,
	"normalized": false,
	"rstrip": false,
	"single_word": false,
	"special": false
	},
	"255019": {
	"content": "<\|START_OF_IMG\|>",
	"lstrip": false,
	"normalized": false,
	"rstrip": false,
	"single_word": false,
	"special": false
	},
	"255020": {
	"content": "<\|END_OF_IMG\|>",
	"lstrip": false,
	"normalized": false,
	"rstrip": false,
	"single_word": false,
	"special": false
	},
	"255021": {
	"content": "<\|IMG_LINE_BREAK\|>",
	"lstrip": false,
	"normalized": false,
	"rstrip": false,
	"single_word": false,
	"special": false
	},
	"255022": {
	"content": "<\|IMG_PATCH\|>",
	"lstrip": false,
	"normalized": false,
	"rstrip": false,
	"single_word": false,
	"special": false
	},
	"255023": {
	"content": "<\|EXTRA_0_TOKEN\|>",
	"lstrip": false,
	"normalized": false,
	"rstrip": false,
	"single_word": false,
	"special": false
	},
	"255024": {
	"content": "<\|EXTRA_1_TOKEN\|>",
	"lstrip": false,
	"normalized": false,
	"rstrip": false,
	"single_word": false,
	"special": false
	},
	"255025": {
	"content": "<\|EXTRA_2_TOKEN\|>",
	"lstrip": false,
	"normalized": false,
	"rstrip": false,
	"single_word": false,
	"special": false
	},
	"255026": {
	"content": "<\|EXTRA_3_TOKEN\|>",
	"lstrip": false,
	"normalized": false,
	"rstrip": false,
	"single_word": false,
	"special": false
	},
	"255027": {
	"content": "<\|EXTRA_4_TOKEN\|>",
	"lstrip": false,
	"normalized": false,
	"rstrip": false,
	"single_word": false,
	"special": false
	},
	"255028": {
	"content": "<\|EXTRA_5_TOKEN\|>",
	"lstrip": false,
	"normalized": false,
	"rstrip": false,
	"single_word": false,
	"special": false
	},
	"255029": {
	"content": "<\|EXTRA_6_TOKEN\|>",
	"lstrip": false,
	"normalized": false,
	"rstrip": false,
	"single_word": false,
	"special": false
	},
	"255030": {
	"content": "<\|EXTRA_7_TOKEN\|>",
	"lstrip": false,
	"normalized": false,
	"rstrip": false,
	"single_word": false,
	"special": false
	},
	"255031": {
	"content": "<\|EXTRA_8_TOKEN\|>",
	"lstrip": false,
	"normalized": false,
	"rstrip": false,
	"single_word": false,
	"special": false
	},
	"255032": {
	"content": "<\|EXTRA_9_TOKEN\|>",
	"lstrip": false,
	"normalized": false,
	"rstrip": false,
	"single_word": false,
	"special": false
	}
	},
	"bos_token": "<BOS_TOKEN>",
	"chat_template": [
	{
	"name": "default",
	"template": "{{ bos_token }}<\|START_OF_TURN_TOKEN\|><\|SYSTEM_TOKEN\|>You are Aya Vision, a brilliant, sophisticated, AI-assistant chatbot trained to assist human users by providing thorough responses. You are a large vision language model built by the Cohere For AI. You are capable of interpreting images, including describing them, answering questions about their contents, extracting textual information, and analyzing visual context.<\|END_OF_TURN_TOKEN\|>\n{%- for message in messages -%}\n <\|START_OF_TURN_TOKEN\|>{{ message.role \| replace(\"user\", \"<\|USER_TOKEN\|>\") \| replace(\"assistant\", \"<\|CHATBOT_TOKEN\|><\|START_RESPONSE\|>\") \| replace(\"system\", \"<\|SYSTEM_TOKEN\|>\") }}\n {%- if message.content is defined -%}\n {%- if message.content is string -%}\n{{ message.content }}\n {%- else -%}\n {%- for item in message.content \| selectattr('type', 'equalto', 'image') -%}\n<image>\n {%- endfor -%}\n {%- for item in message.content \| selectattr('type', 'equalto', 'text') -%}\n{{ item.text }}\n {%- endfor -%}\n {%- endif -%}\n {%- elif message.message is defined -%}\n {%- if message.message is string -%}\n{{ message.message }}\n {%- else -%}\n {%- for item in message.message \| selectattr('type', 'equalto', 'image') -%}\n<image>\n {%- endfor -%}\n {%- for item in message.message \| selectattr('type', 'equalto', 'text') -%}\n{{ item.text }}\n {%- endfor -%}\n {%- endif -%}\n {%- endif -%}\n {%- if message.role == \"assistant\" -%}\n<\|END_RESPONSE\|>\n {%- endif -%}\n<\|END_OF_TURN_TOKEN\|>\n{%- endfor -%}\n{%- if add_generation_prompt -%}\n<\|START_OF_TURN_TOKEN\|><\|CHATBOT_TOKEN\|>\n{%- endif -%}\n"
	}
	],
	"clean_up_tokenization_spaces": false,
	"eos_token": "<\|END_OF_TURN_TOKEN\|>",
	"extra_special_tokens": {},
	"legacy": true,
	"merges_file": null,
	"model_max_length": 1000000000000000019884624838656,
	"pad_token": "<PAD>",
	"processor_class": "AyaVisionProcessor",
	"sp_model_kwargs": {},
	"spaces_between_special_tokens": false,
	"tokenizer_class": "CohereTokenizer",
	"unk_token": null,
	"use_default_system_prompt": false,
	"vocab_file": null
	}