AI-Agent-Security ByteLevel BPE Tokenizer
This repository contains a fast ByteLevel BPE tokenizer trained on a focused AI-agent-security corpus. It is a tokenizer artifact, not a trained language model.
Tokenizer details
| Property | Value |
|---|---|
| Updated | 21 July 2026 |
| Training corpus | AI-agent-security paper titles and abstracts |
| Training corpus size | 85,724 characters |
| Algorithm | Hugging Face tokenizers ByteLevel BPE |
| Target vocabulary size | 8,192 (2**13) |
| Actual vocabulary size | 7,027 |
| Special tokens | <unk>, <pad>, <bos>, <eos> |
| Prefix-space behavior | add_prefix_space=False |
The actual vocabulary is smaller than the requested target because training stopped after the corpus could not provide additional BPE merges.
Usage
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained(
"berkbirkan/aiagent-security-bpe-tokenizer"
)
text = "Prompt injection attacks target tool-using LLM agents."
token_ids = tokenizer.encode(text, add_special_tokens=False)
print(tokenizer.convert_ids_to_tokens(token_ids))
print(token_ids)
print(tokenizer.decode(token_ids))
Focused benchmark — 21 July 2026
The tokenizer was compared with Qwen2.5, Gemma 2, DeepSeek-V3, and SecureBERT. All counts exclude automatically added special tokens. Vocabulary sizes are not matched, so the benchmark measures practical segmentation rather than an algorithm-only comparison.
Sentence token counts
| Sentence | This tokenizer | Qwen2.5 | Gemma 2 | DeepSeek-V3 | SecureBERT |
|---|---|---|---|---|---|
| Prompt injection attacks target tool-using LLM agents and autonomous coding agents. | 16 | 14 | 15 | 15 | 17 |
| Agent red-teaming reveals vulnerabilities in skill recommendation systems. | 11 | 11 | 12 | 11 | 12 |
| Security guardrails reduce prompt injection risks in AI agent workflows. | 11 | 12 | 12 | 12 | 13 |
| Secret scanner agents extract credentials and access context from unstructured documents. | 13 | 13 | 12 | 12 | 12 |
Training-corpus compression
| Tokenizer | Total tokens | Characters/token |
|---|---|---|
| This tokenizer | 15,103 | 5.676 |
| Qwen2.5 | 16,522 | 5.188 |
| Gemma 2 | 16,521 | 5.189 |
| DeepSeek-V3 | 16,093 | 5.327 |
| SecureBERT | 16,027 | 5.349 |
The tokenizer produces the lowest token count on its own training corpus. This is expected for a domain-trained tokenizer and does not imply the same ranking on general-purpose or out-of-domain text.
Domain-term fragmentation
| Term | This tokenizer | Qwen2.5 | Gemma 2 | DeepSeek-V3 | SecureBERT |
|---|---|---|---|---|---|
| prompt injection | 4 | 2 | 2 | 3 | 3 |
| LLM agents | 2 | 3 | 3 | 3 | 3 |
| tool-using agents | 4 | 3 | 4 | 4 | 4 |
| autonomous agents | 3 | 3 | 2 | 3 | 2 |
| coding agents | 3 | 2 | 2 | 2 | 2 |
| agent red-teaming | 4 | 4 | 5 | 4 | 5 |
| skill recommendation | 3 | 2 | 2 | 2 | 2 |
| security guardrails | 3 | 3 | 3 | 3 | 4 |
| secret scanner | 4 | 2 | 2 | 2 | 2 |
| vulnerability detection | 4 | 4 | 3 | 4 | 2 |
| agentic workflows | 3 | 3 | 3 | 3 | 3 |
| supply-chain attacks | 6 | 3 | 4 | 4 | 5 |
Training approach
The tokenizer backend uses tokenizers.models.BPE, a ByteLevel pre-tokenizer,
a ByteLevel decoder, and the complete ByteLevel initial alphabet. It is wrapped
with PreTrainedTokenizerFast for direct use through Transformers and
AutoTokenizer.
Limitations
- The training corpus is small and narrowly focused on AI-agent security.
- The tokenizer is primarily English-oriented because of its training data.
- Results on the training corpus should not be interpreted as a general-purpose tokenizer ranking.
- BOS and EOS tokens are registered but are not automatically inserted by the current post-processing template.
- This repository contains no model weights and cannot generate text by itself.
Related project
Training code, corpus-building steps, reproducible benchmark outputs, and charts are available in the security-bpe-tokenizer GitHub repository.