EvoLenTokenizer/base-200k

base_5120

This model is a BERT-based DNA language model trained with the standard byte-pair encoding (BPE) tokenizer with vocabulary size 5,120. It is the baseline control model from the paper EvoLen: Evolution-Guided Tokenization for DNA Language Model (COLM 2026).

• Paper: EvoLen: Evolution-Guided Tokenization for DNA Language Model
• Code: https://github.com/HN020719/EvoLen

Model description

This model uses a BERT-base architecture (12 layers, hidden size 768) with a vocabulary of 5,120 tokens. It was pretrained on human genome sequences using masked language modeling. The tokenizer is the standard BPE baseline described in the EvoLen repository.

Intended uses & limitations

This model is primarily intended as a baseline for evaluating the EvoLen tokenizer in DNA language model benchmarks. It can be fine-tuned for downstream genomic tasks, but it is not specifically designed for production use.

Training and evaluation data

The model was pretrained on the human genome (hg38). Training sequences were tokenized into 512-token windows and a held-out evaluation set was used to report the following results.

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 4e-05
• train_batch_size: 96
• eval_batch_size: 96
• seed: 42
• optimizer: Adam with betas=(0.9,0.98) and epsilon=1e-06
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 10000
• training_steps: 200000

Training results

• Loss: 5.0263
• Accuracy: 0.2365
• Perplexity: 152.36

Framework versions

• Transformers 4.35.2
• Pytorch 2.8.0+cu128
• Datasets 4.4.1
• Tokenizers 0.15.2