tokenizers
crates.ioFastest implementation of the most used tokenizers in AI.
Latest v1.0.0-rc.3 · Oct 19, 2026
Downloads
No download history yet.
Linked repositories
| ## | RepositoryRepository | ConnectionConnection |
|---|---|---|
| 1 | huggingface/tokenizers | Source package · Manifest |
| 2 | paradedb/paradedb | Source package · Manifest |
Recent releases
| Version | Published |
|---|---|
| 1.0.0-rc.3 | Oct 19, 2026 |
| 1.0.0-rc.2 | Sep 21, 2026 |
| 0.23.2 | Sep 3, 2026 |
| 0.23.1 | Apr 27, 2026 |
| 0.22.2 | Dec 2, 2025 |