tokenizers

Why Is Everyone Choosing Rust? Rust keeps appearing behind tools developers use every day. What makes it worth choosing—and when is it worth the effort? Through visual examples, we explain how Rust handles memory, what ownership and borrowing mean, and why these ideas matter for reliable software. We also explore its role in Python tools, AI infrastructure, and production systems. In this video: - Memory safety, ownership, and borrowing explained simply - Performance and parallel processing—without the hype - Rust inside tools like uv, Ruff, Polars, and Hugging Face Tokenizers - Lessons from GitHub’s Copilot runtime and Google’s Android work - When to choose Rust, keep your current language, or replace just one component Rust won’t make every program faster or prevent every bug. Understanding where it helps is more useful than following the trend. Sources and further reading: 1. Rust ownership: doc.rust-lang.org 2. Python parallel execution: docs.python.org 3. Astral tools: docs.astral.sh 4. Polars: pola-rs/polars 5. Hugging Face Tokenizers: huggingface/tokenizers 6. GitHub’s Rust migration: github.blog 7. Rust in Android: blog.google Visual models and example workflows are simplified for teaching. Reported production results apply to the environments described by their sources. #Rust #Programming #SoftwareEngineering

Rust Didn't Win Because It's Fast GitHub rewrote Copilot's runtime in Rust. OpenAI's Codex is now almost all Rust. Bun, the runtime Claude Code ships on, moved 535,496 lines of Zig to Rust in 11 days. But it isn't about speed: C has been fast for half a century. We fly inside a real laptop, down to the CPU die and the memory cells, to show what Rust actually changed, and why AI coding agents love it. 0:00 Everyone is rewriting in Rust 0:49 This is Sidequest Explained 1:00 Inside the machine: instructions, bytes, stack and heap 1:53 Way one: freeing memory by hand (C and C++) 2:23 An elevator crash started Rust 2:40 Way two: the garbage collector (Discord's spikes) 3:12 Way three: ownership 3:36 Checked before it runs, and Android's results 4:07 Close to the metal: 28 bytes vs 8 4:21 Copilot: 5.25 s to 292 ms 4:41 Why AI agents love the Rust compiler 5:11 Is Python dead? What it means for you Sources GitHub Blog, migrating the Copilot runtime to Rust (Sep 2026): github.blog Bun, Rewriting Bun in Rust (Jul 2026): bun.sh OpenAI Codex repository: openai/codex Codex native rewrite discussion: codex#1174 Grok Build repository: xai-org/grok-build Dennis Ritchie, The Development of the C Language: bell-labs.com Microsoft MSRC, 70% of security bugs are memory safety (2019): microsoft.com Chromium memory safety: chromium.org MIT Technology Review, Rust's origin (2023): technologyreview.com Rust 1.0 announcement: blog.rust-lang.org Discord, Why Discord is switching from Go to Rust (2020): discord.com The Rust Book, ownership and borrowing: doc.rust-lang.org Google, memory-safe languages in Android 13 (2022): security.googleblog.com Google, Rust in Android (Nov 2025): blog.google Python sys.getsizeof / CPython ints: docs.python.org Hugging Face tokenizers: huggingface/tokenizers OpenAI tiktoken: openai/tiktoken pydantic-core: pydantic/pydantic-core uv by Astral: astral-sh/uv Numbers are the sources' own. The Copilot timings are client + session + one-turn overhead in the opt-in in-process mode, not counting the model or network. Android's 1000× is memory-safety vulnerability density. Code on screen is an illustration, not a run. Credits Die micrographs: Fritzchens Fritz (CC0). Memory-cell micrograph (Micron MT4C1024) and SEM plate (1886VE10): ZeptoBars, CC BY 3.0. Office and hardware photos: Pexels. HDRIs and materials: Poly Haven, ambientCG (CC0). Music: "Lofi Study" by FASSounds (Pixabay). Rust, GitHub, Copilot, OpenAI, Codex, Bun, Claude, xAI, Grok, Microsoft, Chrome, Android, Discord, Python, Hugging Face, Astral, pydantic and other names and logos belong to their owners, used for identification and commentary; Sidequest is not affiliated with any of them. Narration: an AI clone of my own voice. Animation: original, built in three.js. #rust #programming #ai

@tulkenss yes @art_zucker worked on the added token matcher: Tokens bucketed by first byte, SIMD to flag where a bucket could start, then an 8-byte prefix compare and a hash lookup to confirm github.com/huggingface/tokenizers/blob/fb55b11bf6… I think I'll do a blogpost about it, too!

93% of the most downlaoded models on the Hub already give the same output that 0.23 on main 🔥 The 0% on the card is one import error Try rc3 with pip install --pre tokenizers and tell us what breaks 🤗 github.com/huggingface/tokenizers/issues

Why AI Infrastructure Is Moving to Rust The model isn't written in Rust. The GPU code isn't Rust. Nobody is rewriting PyTorch in Rust. So why did GitHub just rebuild Copilot's runtime in Rust, and why is Rust inside the AI tools you already install? This video follows one prompt, "Summarize this contract in three bullet points", from the moment you press Enter to the first word that comes back, through a server pictured as a restaurant kitchen at dinner rush: - Python is the head chef: it calls the orders, and the GPU does the maths - Tokens: why the model never reads your letters - Batching, and why every new word is another trip to the GPU - Rush hour: why the CPU work around the model becomes the wait - The global interpreter lock (GIL): one cook at a time - Garbage collection pauses and the slowest requests - Why C and C++ were the old fix, and why memory-safety bugs made it risky - Rust's ownership: memory cleaned at a known point, mistakes caught before the program runs - PyO3: Rust you import from Python like any other module - Where Rust already runs: tokenizers, tiktoken, pydantic-core, safetensors, TGI, SGLang - Two fair objections: free-threaded Python 3.14, and what Rust costs Sources: - GitHub, "Migrating the GitHub Copilot runtime to Rust, using Copilot": github.blog - vLLM V1 architecture: vllm.ai - vLLM v0.6.0 performance update: vllm.ai - Chromium, memory safety: chromium.org - Discord, "Why Discord is switching from Go to Rust": discord.com - What's New in Python 3.14 (free-threaded Python): docs.python.org - Hugging Face tokenizers: huggingface/tokenizers - OpenAI tiktoken: openai/tiktoken - PyO3: pyo3.rs 00:00 Where Rust isn't 00:42 Follow one prompt 01:09 The chef doesn't cook 01:44 Tokens 02:16 One word per trip 02:54 Count the stations 03:20 Rush hour 04:01 The GIL 04:46 Garbage collection pauses 05:28 The old fix: C++ 06:13 Ownership 06:51 Every cook at once 07:18 Rust inside Python 07:44 The real stack 08:30 Two fair objections 09:04 The first word

tokenizers v1.0.0-rc.3 is out! 🕸️ WebAssembly bindings for local AI: tokenize in the browser, no server needed 📦 Python wheels up to 8x smaller than 0.23 npm i tokenizers-wasm pip install --pre tokenizers Release notes: github.com/huggingface/tokenizers/releases/tag/v1…

3 AM. The AI Server Crashed Again. Then Came Rust. #Shorts 3 AM. A model server crashes. Again. A memory bug in the C++ layer, Python threads stuck on the GIL, installs that crawl. (The 3 AM team is an illustrative scene, not a real company or reported incident.) That's the kind of pain behind a real trend: a growing number of AI tools are built in Rust. • Hugging Face tokenizers and safetensors, Polars, Pydantic v2's core and OpenAI's tiktoken have Rust cores. • Astral says uv is 10–100x faster than pip (vendor claim). • OpenAI rewrote its Codex CLI in Rust (announced May 2025) for a zero-dependency install, native sandboxing and lower memory use. • Microsoft (2019: ~70% of its CVEs) and Chromium (~70% of high-severity bugs) found most serious security bugs were memory-safety issues. The catch: Python still rules training and research, GPU kernels are mostly C++/CUDA, and Rust has a learning curve. It's not "everyone moving to Rust". It's Python on top, Rust underneath. Sources: huggingface/tokenizers huggingface/safetensors pola-rs/polars pydantic/pydantic openai/tiktoken docs.astral.sh codex#1174 msrc.microsoft.com chromium.org docs.python.org Narrated story; not affiliated with the Rust Foundation or any project shown. GASK Education Cloud · AI News Subscribe to @GASKEducationCloud for short, factual AI news and lessons.

Why AI Infrastructure Is Moving to Rust ? GitHub just rewrote its Copilot agent runtime in Rust - and you are probably running Rust inside your Python AI code already. So is Rust replacing Python in AI? No: Python stays the language you drive AI with, while Rust replaces the C and C++ engine underneath and the servers around it. In this video: why Python only gives the orders, the memory bug C++ lets through and Rust refuses to compile, four reasons Rust fits AI systems (no garbage collector, real parallel threads, memory layout control, PyO3), real Rust projects in the AI stack, the honest downsides, and who should actually learn Rust. The C++ and Rust snippets are our own examples; the PyO3 snippet is the official example from the PyO3 README (Apache-2.0 / MIT). Copilot performance figures are GitHub's own measurements, excluding model and network time. Chapters: 0:00 Intro 0:13 GitHub's Rust rewrite 0:48 Rust you already run 1:20 Who really does the math 1:42 Two different jobs 2:00 The problem with C++ 2:42 What Rust does instead 3:07 Why Rust fits AI systems 4:13 Rust inside Python 4:35 Rust around AI 4:59 The catch 5:23 Should you learn Rust? Stock footage: Video by Atakan Ozkan on Pexels; Video by Nothing Ahead on Pexels. Sources (checked 2026-10-17): github.blog huggingface/tokenizers safetensors/safetensors openai/tiktoken github.com/pydantic/pydantic/tree/main/pydantic-c… pydantic.dev pola-rs/polars astral-sh/uv astral-sh/ruff qdrant/qdrant lancedb/lancedb huggingface/candle tracel-ai/burn PyO3/pyo3 PyO3/maturin chromium.org microsoft.com docs.python.org doc.rust-lang.org doc.rust-lang.org docs.pytorch.org huggingface.co docs.lancedb.com docs.python.org raw.githubusercontent.com Independent explainers of AI creative tools. Not affiliated with, endorsed by, or sponsored by any company or tool shown. Prices and features change fast - always check the official sites. Narration uses an AI-generated voice. Script and facts are written and checked by the creator. #rust #python #ai #AItools #AIart #AIvideo

Hugging Face Tokenizers is a text tokenization library, written in Rust. Turn text into the token IDs language models consume, using pretrained tokenizers from Rust, Python, or Node.js. 11k+ stars huggingface/tokenizers

@tulkenss Yes, mostly a fast path for ASCII chars! Plus some other tricks, here's the PR if you're interested: tokenizers#2508