toktok-rs
crates.ioFast, exact BPE tokenizer for OpenAI encodings and Qwen3.5 (cl100k_base, o200k_base, o200k_harmony, qwen3.5) — tiktoken-identical ids, plus a token-count bound for TypeSafe's Jev, no dependencies
Latest v0.3.0 · Oct 21, 2026
Downloads
No download history yet.
Linked repositories
| ## | RepositoryRepository | ConnectionConnection |
|---|---|---|
| 1 | vectorize-io/toktok | Source package · Manifest |
Recent releases
| Version | Published |
|---|---|
| 0.3.0 | Oct 21, 2026 |
| 0.2.0 | Oct 19, 2026 |
| 0.1.4 | Sep 2, 2026 |
| 0.1.3 | Sep 2, 2026 |
| 0.1.2 | Sep 2, 2026 |