Instructions to use EvoLenTokenizer/base-200k with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use EvoLenTokenizer/base-200k with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("feature-extraction", model="EvoLenTokenizer/base-200k")# Load model directly from transformers import AutoTokenizer, AutoModelForMaskedLM tokenizer = AutoTokenizer.from_pretrained("EvoLenTokenizer/base-200k") model = AutoModelForMaskedLM.from_pretrained("EvoLenTokenizer/base-200k", device_map="auto") - Notebooks
- Google Colab
- Kaggle
Improve model card with metadata and links
#1
by nielsr HF Staff - opened
This PR adds the missing license, library name, pipeline tag, and links to the paper and code repository, making it easier for users to discover and use the model. It also clarifies that base_5120 is the standard BPE 5120 baseline from the EvoLen paper.
nancyH changed pull request status to merged