Marin tokenizer with tool delimiters
Derived from marin-community/marin-tokenizer at revision a5ca45f2feb6c959bd87b81689aa7279b5bdcaa2. It preserves the Marin chat template and vocabulary size.
The following reserved IDs are assigned to chat markers:
| Token | ID |
|---|---|
| `< | start_think |
| `< | end_think |
<tool_call> |
128005 |
</tool_call> |
128011 |
<tool_call> and </tool_call> each encode as one token.
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support