tiktoken
crates.ioA high-performance pure-Rust implementation of OpenAI's tiktoken BPE tokenizer
Latest v4.1.2 · Aug 13, 2026
Downloads
No download history yet.
Linked repositories
| ## | RepositoryRepository | ConnectionConnection |
|---|---|---|
| 1 | goliajp/rust-tiktoken | Registry metadata |
| 2 | openai/tiktoken | Source package · Manifest |
Recent releases
| Version | Published |
|---|---|
| 4.1.2 | Aug 13, 2026 |
| 4.1.1 | Aug 13, 2026 |
| 4.0.1 | Aug 13, 2026 |
| 4.0.0 | Aug 13, 2026 |
| 3.8.3 | Aug 10, 2026 |