Reduce API surface for BLAKE2 to match OpenSSL semantics
This is an updated version of the original post which reduces the scope of the work on BLAKE2.
I originally wanted to remove non-standard parameters to BLAKE2 to make the module more maintainable in the future as well as SIMD support, but I am planning to only remove SIMD support for now.
I still think it's wrong to use tree-hashing with BLAKE2 since BLAKE3 exists and should be used instead. Nonetheless, after looking at the deprecation PRs (gh-160374 and gh-160375) and the long period this would require to actually remove the code, I don't think it's really worth it. It would at least reduce the impact to downstream users almost to zero. Instead, I suggest that we soft-deprecate the interface for tree hashing in the docs and suggest BLAKE3 instead.
----
Original post:
I want to reduce BLAKE2 in hashlib to a minimal API matching OpenSSL. More precisely, BLAKE2 is kept without SIMD and matches the API of OpenSSL's BLAKE2 (digest, MAC): data and digest_size for the hash, and salt/person with a key. Tree hashing, salt/person without a key, and HMAC-BLAKE2 will be deprecated in 3.16 and removed in 3.21. If necessary, the deprecation period will be extended to a few more releases but I really want to get rid of the annoying API surface.
BLAKE2 is already a keyed hash function, it doesn't make sense to use HMAC on top of it, despite some libraries using the HMAC-BLAKE2 HACL* sources for that (see gh-138114).
In an ideal world, I would remove BLAKE2 and recommend BLAKE3 or SHA-based alternatives. Note that currently (and likely in the future), there are no plans to add BLAKE3 to hashlib (see gh-83479). For a fast secure hash, BLAKE3 is available as the blake3 package. It is maintained by the BLAKE3 authors, has a hashlib-like API, supports keyed hashing and key derivation, and has wheels for all major platforms.
Maintenance pain
BLAKE2 has been in hashlib since 3.6, first as a custom module with our own implementation, then as a wrapper around the vendored HACL* sources (support for HACL* HMAC was added in 3.14, together with support for BLAKE2 despite numerous shortcomings on some platforms).
BLAKE2 has the largest API in hashlib (key, salt, person and seven tree-hashing parameters). Most of its maintenance cost comes from the SIMD code: it needs per-platform compiler flags and CPU detection as well as dedicated code paths for the vectorized versions, is only enabled on (some) x86-64, and usually breaks on non-Tier-1 platforms:
- gh-123748: macOS universal2 builds failed (SIMD256 compiles for x86-64 but not ARM64);
- gh-124304: SIMD had to be disabled on older Android versions;
- gh-126458: SIMD support wrongly detected under WASI;
- gh-128134: missing aligned_alloc check on older macOS;
- gh-130213: older toolchains failed without the SIMD flags;
- gh-133042: duplicate symbols when cross-compiling for Emscripten (still unfixed);
- gh-138114: HMAC-BLAKE2 does not properly use the SIMD implementations (still unfixed).
The module also needed several correctness fixes (gh-152851, gh-155830, gh-155835).
Impact
GitHub code search (excluding forks, CPython copies and vendored site-packages) finds about 34,000 Python files using hashlib.blake2b, 4,800 using hashlib.blake2s, and 300 using hashlib.new("blake2b") or hashlib.new("blake2s").
62 of the 500 most-starred active Python repositories on GitHub use BLAKE2, e.g. PyTorch, Transformers, Home Assistant, Sentry, Streamlit, Gradio and Airflow.
Of 1,753 sampled call sites (1,420 repositories), 1,649 (1,336 repositories) only use digest_size, 59 (51 repositories) use key, and 45 (38 repositories) use salt or person without a key. None use tree hashing. Most of them only need a fast hash, for which SHA-2, SHA-3 or BLAKE3 also work. This includes:
- projects that store digests, such as Home Assistant (cache filenames), Mathesar (database identifiers) and Onyx (search-index IDs);
- projects that depend on the exact BLAKE2 output, such as networkx (graph hashes), LangChain (indexing), apprise and a minisign verifier (BLAKE2b-512 signatures), copyparty (client checksums);
- projects that use keyed BLAKE2 such as agno (MinHash) and openai-python (Bedrock helper); or
- projects such as PyPI or Poetry for checksums, Warehouse for storage and publishing (though it looks like a legacy API), or python-tuf.
One popular project affected by this change would be ComfyUI (which uses person without a key). In particular, I don't expect too much of downstream breakage (considering the long deprecation period) but I would appreciate if anyone against this proposal could explain why SIMD BLAKE2 should be kept (note that I already plan to remove SIMD support in 3.16 without a deprecation notice as it created too many issues, while exposing the OpenSSL implementation).
Note that projects can rely on nacl.hash.blake2b (PyNaCl does not offer a 2s version) or PyCryptodome for blake2b and blake2s. AFAIK, there is no PyPI packages providing the tree-hashing feature but the latter is neither standardized by an RFC nor used in the wild.
----
> AI disclosure: Opus 5.5 was used for collecting and classifying data.
4 likes · 1 comment
Mentions
Coming from W&B? Switching is easy 👇 📖 Docs: huggingface.co 🤖 LeRobot: huggingface/lerobot 🎯 Trackio: gradio-app/trackio
Gradio 10月28日的新版修了批量请求的运行记录：同一批里的每个用户，都能在自己的历史里找到那次运行。 用它搭模型演示页，排查结果时能少翻一层。注意：官方同时明确，定时器和页面加载触发的运行不存历史，别把它当完整审计日志。 gradio-app/gradio
44K ⭐ AI Demos In Five Minutes (Gradio) Gradio is an open source tool that turns one Python function into a working web app: input box, button, and result, shareable as a single link. gradio-app/gradio This is a discovery based on the official GitHub page and public documentation, not a hands-on test. The link is in the description.
44K ⭐ Demo AI Dalam Lima Menit (Gradio) Gradio adalah alat open sors untuk membuat web app AI dari satu fungsi Python. Halaman web, kolom input, tombol, dan hasil — semuanya otomatis, lalu bisa dibagikan lewat link. gradio-app/gradio Video ini merupakan discovery berdasarkan halaman GitHub resmi dan dokumentasi publik, bukan hasil pengujian langsung. Link-nya ada di deskripsi.
已写好 summarize(text) 的 Python 用户，可用 Gradio 加网页输入框。 安装：pip install gradio import gradio as gr gr.Interface(summarize,"text","text").launch() 需 Python 3.10+；示例只接界面，摘要模型与费用另配。先在本机验证。 项目：gradio-app/gradio
Reduce API surface for BLAKE2 to match OpenSSL semantics This is an updated version of the original post which reduces the scope of the work on BLAKE2. I originally wanted to remove non-standard parameters to BLAKE2 to make the module more maintainable in the future as well as SIMD support, but I am planning to only remove SIMD support for now. I still think it's wrong to use tree-hashing with BLAKE2 since BLAKE3 exists and should be used instead. Nonetheless, after looking at the deprecation PRs (gh-160374 and gh-160375) and the long period this would require to actually remove the code, I don't think it's really worth it. It would at least reduce the impact to downstream users almost to zero. Instead, I suggest that we soft-deprecate the interface for tree hashing in the docs and suggest BLAKE3 instead. ---- Original post: I want to reduce BLAKE2 in hashlib to a minimal API matching OpenSSL. More precisely, BLAKE2 is kept without SIMD and matches the API of OpenSSL's BLAKE2 (digest, MAC): data and digest_size for the hash, and salt/person with a key. Tree hashing, salt/person without a key, and HMAC-BLAKE2 will be deprecated in 3.16 and removed in 3.21. If necessary, the deprecation period will be extended to a few more releases but I really want to get rid of the annoying API surface. BLAKE2 is already a keyed hash function, it doesn't make sense to use HMAC on top of it, despite some libraries using the HMAC-BLAKE2 HACL* sources for that (see gh-138114). In an ideal world, I would remove BLAKE2 and recommend BLAKE3 or SHA-based alternatives. Note that currently (and likely in the future), there are no plans to add BLAKE3 to hashlib (see gh-83479). For a fast secure hash, BLAKE3 is available as the blake3 package. It is maintained by the BLAKE3 authors, has a hashlib-like API, supports keyed hashing and key derivation, and has wheels for all major platforms. Maintenance pain BLAKE2 has been in hashlib since 3.6, first as a custom module with our own implementation, then as a wrapper around the vendored HACL* sources (support for HACL* HMAC was added in 3.14, together with support for BLAKE2 despite numerous shortcomings on some platforms). BLAKE2 has the largest API in hashlib (key, salt, person and seven tree-hashing parameters). Most of its maintenance cost comes from the SIMD code: it needs per-platform compiler flags and CPU detection as well as dedicated code paths for the vectorized versions, is only enabled on (some) x86-64, and usually breaks on non-Tier-1 platforms: - gh-123748: macOS universal2 builds failed (SIMD256 compiles for x86-64 but not ARM64); - gh-124304: SIMD had to be disabled on older Android versions; - gh-126458: SIMD support wrongly detected under WASI; - gh-128134: missing aligned_alloc check on older macOS; - gh-130213: older toolchains failed without the SIMD flags; - gh-133042: duplicate symbols when cross-compiling for Emscripten (still unfixed); - gh-138114: HMAC-BLAKE2 does not properly use the SIMD implementations (still unfixed). The module also needed several correctness fixes (gh-152851, gh-155830, gh-155835). Impact GitHub code search (excluding forks, CPython copies and vendored site-packages) finds about 34,000 Python files using hashlib.blake2b, 4,800 using hashlib.blake2s, and 300 using hashlib.new("blake2b") or hashlib.new("blake2s"). 62 of the 500 most-starred active Python repositories on GitHub use BLAKE2, e.g. PyTorch, Transformers, Home Assistant, Sentry, Streamlit, Gradio and Airflow. Of 1,753 sampled call sites (1,420 repositories), 1,649 (1,336 repositories) only use digest_size, 59 (51 repositories) use key, and 45 (38 repositories) use salt or person without a key. None use tree hashing. Most of them only need a fast hash, for which SHA-2, SHA-3 or BLAKE3 also work. This includes: - projects that store digests, such as Home Assistant (cache filenames), Mathesar (database identifiers) and Onyx (search-index IDs); - projects that depend on the exact BLAKE2 output, such as networkx (graph hashes), LangChain (indexing), apprise and a minisign verifier (BLAKE2b-512 signatures), copyparty (client checksums); - projects that use keyed BLAKE2 such as agno (MinHash) and openai-python (Bedrock helper); or - projects such as PyPI or Poetry for checksums, Warehouse for storage and publishing (though it looks like a legacy API), or python-tuf. One popular project affected by this change would be ComfyUI (which uses person without a key). In particular, I don't expect too much of downstream breakage (considering the long deprecation period) but I would appreciate if anyone against this proposal could explain why SIMD BLAKE2 should be kept (note that I already plan to remove SIMD support in 3.16 without a deprecation notice as it created too many issues, while exposing the OpenSSL implementation). Note that projects can rely on nacl.hash.blake2b (PyNaCl does not offer a 2s version) or PyCryptodome for blake2b and blake2s. AFAIK, there is no PyPI packages providing the tree-hashing feature but the latter is neither standardized by an RFC nor used in the wild. ---- > AI disclosure: Opus 5.5 was used for collecting and classifying data.
模型已经跑通，怎么快速给别人一个可操作的 Demo？ Gradio 用几行 Python 就能把模型或函数变成网页，支持文本、图片等输入。适合 AI 课程展示和原型验证；公开分享前检查数据和调用成本。 gradio-app/gradio #Python #AI开发
너라면 동료에게 요약과 이미지 분류 중 어떤 기능부터 시험하게 할래? 공식 저장소 gradio-app/gradio 이런 AI 도구 활용법은 여기서도 나눠👇 discord.gg