TokSuite

community

AI & ML interests

Tokenization, Robustness, LLMs

Recent Activity

gsaltintas updated a collection 3 days ago

TokSuite Text-Matched Models

gsaltintas updated a Space 7 days ago

toksuite/quick-tokenizer-accuracy

gsaltintas updated a dataset 9 days ago

toksuite/toksuite_chinese

View all activity

Papers

TokSuite: Measuring the Impact of Tokenizer Choice on Language Model Behavior

View all Papers

toksuite 's models 20

toksuite/google-gemma-2-2b

Text Generation • 2B • Updated Dec 25, 2025 • 162

toksuite/meta-llama-Llama-3.2-1B

Text Generation • 2B • Updated Dec 25, 2025 • 171

toksuite/CohereLabs-aya-expanse-8b

Text Generation • 2B • Updated Dec 25, 2025 • 129

toksuite/tiktoken-gpt-4o

Text Generation • 2B • Updated Dec 25, 2025 • 663

toksuite/common-pile-comma-v0.1

Text Generation • 2B • Updated Dec 25, 2025 • 236

toksuite/microsoft-Phi-3-mini-4k-instruct

Text Generation • 1B • Updated Dec 25, 2025 • 124

toksuite/google-bert-bert-base-multilingual-cased

Text Generation • 2B • Updated Dec 25, 2025 • 213

toksuite/Qwen-Qwen3-8B

Text Generation • 2B • Updated Dec 25, 2025 • 107

toksuite/tokenmonster-englishcode-32000-consistent-v1

Text Generation • 1B • Updated Dec 25, 2025 • 144

toksuite/mistralai-tekken

Text Generation • 2B • Updated Dec 25, 2025 • 294

toksuite/gpt2

Text Generation • 1B • Updated Dec 25, 2025 • 162

toksuite/google-byt5-small

Text Generation • 1B • Updated Dec 25, 2025 • 238

toksuite/bigscience-bloom

Text Generation • 2B • Updated Dec 25, 2025 • 299

toksuite/facebook-xglm-564M

Text Generation • 2B • Updated Dec 25, 2025 • 174

toksuite/meta-llama-Llama-3.2-7B

8B • Updated Dec 22, 2025 • 44

toksuite/supertoken_models-llama_google-gemma-2-2b-textmatched

2B • Updated Dec 20, 2025 • 1

toksuite/common-pile-comma-v0.1-textmatched

2B • Updated Dec 20, 2025 • 34

toksuite/Qwen-Qwen3-8B-textmatched

2B • Updated Dec 20, 2025 • 36

toksuite/meta-llama-Llama-3.2-1B-textmatched

2B • Updated Dec 20, 2025 • 35

toksuite/meta-llama-Llama-3.2-300M

0.6B • Updated Dec 20, 2025 • 4