Tokenizer Eval

non-profit

Activity Feed Request to join this org

AI & ML interests

Tokenizers

Recent Activity

christopher updated a dataset about 2 hours ago

tokenizer-eval/universal-unigram

christopher published a dataset about 2 hours ago

tokenizer-eval/universal-unigram

christopher updated a Space about 3 hours ago

tokenizer-eval/README

View all activity

updated a dataset about 2 hours ago

tokenizer-eval/universal-unigram

Viewer • Updated about 2 hours ago • 17.9M

published a dataset about 2 hours ago

tokenizer-eval/universal-unigram

Viewer • Updated about 2 hours ago • 17.9M

updated a Space about 3 hours ago

README

published a Space about 3 hours ago

README

updated a dataset about 3 hours ago

tokenizer-eval/universal-wordpiece

Viewer • Updated about 3 hours ago • 11.3M

published a dataset about 3 hours ago

tokenizer-eval/universal-wordpiece

Viewer • Updated about 3 hours ago • 11.3M

updated a dataset about 3 hours ago

tokenizer-eval/universal-wordlevel

Viewer • Updated about 3 hours ago • 3.73M

published a dataset about 3 hours ago

tokenizer-eval/universal-wordlevel

Viewer • Updated about 3 hours ago • 3.73M

in tokenizer-eval/new-tokenizer-metadata 18 days ago

[bot] Conversion to Parquet

#1 opened 19 days ago by

parquet-converter

updated 2 datasets about 2 months ago

tokenizer-eval/tokenizer-data

Viewer • Updated Feb 26 • 24.8k • 23

tokenizer-eval/tokenizer-metadata

Viewer • Updated Feb 26 • 424k • 29

authored a paper 4 months ago

Bolmo: Byteifying the Next Generation of Language Models

Paper • 2512.15586 • Published Dec 17, 2025 • 17

submitted a paper to Daily Papers 4 months ago

Bolmo: Byteifying the Next Generation of Language Models

Paper • 2512.15586 • Published Dec 17, 2025 • 17

authored a paper 5 months ago

Economies of Open Intelligence: Tracing Power & Participation in the Model Ecosystem

Paper • 2512.03073 • Published Nov 27, 2025 • 7

authored a paper 6 months ago

Global PIQA: Evaluating Physical Commonsense Reasoning Across 100+ Languages and Cultures

Paper • 2510.24081 • Published Oct 28, 2025 • 22

authored a paper 6 months ago

The German Commons - 154 Billion Tokens of Openly Licensed Text for German Language Models

Paper • 2510.13996 • Published Oct 15, 2025 • 9

posted an update 6 months ago

Post

713

Something very cool is cooking at

1 reply

·

authored a paper 11 months ago

RewardBench 2: Advancing Reward Model Evaluation

Paper • 2506.01937 • Published Jun 2, 2025 • 7

authored a paper 12 months ago

Fishing for Magikarp: Automatically Detecting Under-trained Tokens in Large Language Models

Paper • 2405.05417 • Published May 8, 2024 • 1

authored a paper about 1 year ago

Command A: An Enterprise-Ready Large Language Model

Paper • 2504.00698 • Published Apr 1, 2025 • 29