caronc

Reduce API surface for BLAKE2 to match OpenSSL semantics This is an updated version of the original post which reduces the scope of the work on BLAKE2. I originally wanted to remove non-standard parameters to BLAKE2 to make the module more maintainable in the future as well as SIMD support, but I am planning to only remove SIMD support for now. I still think it's wrong to use tree-hashing with BLAKE2 since BLAKE3 exists and should be used instead. Nonetheless, after looking at the deprecation PRs (gh-160374 and gh-160375) and the long period this would require to actually remove the code, I don't think it's really worth it. It would at least reduce the impact to downstream users almost to zero. Instead, I suggest that we soft-deprecate the interface for tree hashing in the docs and suggest BLAKE3 instead. ---- Original post: I want to reduce BLAKE2 in hashlib to a minimal API matching OpenSSL. More precisely, BLAKE2 is kept without SIMD and matches the API of OpenSSL's BLAKE2 (digest, MAC): data and digest_size for the hash, and salt/person with a key. Tree hashing, salt/person without a key, and HMAC-BLAKE2 will be deprecated in 3.16 and removed in 3.21. If necessary, the deprecation period will be extended to a few more releases but I really want to get rid of the annoying API surface. BLAKE2 is already a keyed hash function, it doesn't make sense to use HMAC on top of it, despite some libraries using the HMAC-BLAKE2 HACL* sources for that (see gh-138114). In an ideal world, I would remove BLAKE2 and recommend BLAKE3 or SHA-based alternatives. Note that currently (and likely in the future), there are no plans to add BLAKE3 to hashlib (see gh-83479). For a fast secure hash, BLAKE3 is available as the blake3 package. It is maintained by the BLAKE3 authors, has a hashlib-like API, supports keyed hashing and key derivation, and has wheels for all major platforms. Maintenance pain BLAKE2 has been in hashlib since 3.6, first as a custom module with our own implementation, then as a wrapper around the vendored HACL* sources (support for HACL* HMAC was added in 3.14, together with support for BLAKE2 despite numerous shortcomings on some platforms). BLAKE2 has the largest API in hashlib (key, salt, person and seven tree-hashing parameters). Most of its maintenance cost comes from the SIMD code: it needs per-platform compiler flags and CPU detection as well as dedicated code paths for the vectorized versions, is only enabled on (some) x86-64, and usually breaks on non-Tier-1 platforms: - gh-123748: macOS universal2 builds failed (SIMD256 compiles for x86-64 but not ARM64); - gh-124304: SIMD had to be disabled on older Android versions; - gh-126458: SIMD support wrongly detected under WASI; - gh-128134: missing aligned_alloc check on older macOS; - gh-130213: older toolchains failed without the SIMD flags; - gh-133042: duplicate symbols when cross-compiling for Emscripten (still unfixed); - gh-138114: HMAC-BLAKE2 does not properly use the SIMD implementations (still unfixed). The module also needed several correctness fixes (gh-152851, gh-155830, gh-155835). Impact GitHub code search (excluding forks, CPython copies and vendored site-packages) finds about 34,000 Python files using hashlib.blake2b, 4,800 using hashlib.blake2s, and 300 using hashlib.new("blake2b") or hashlib.new("blake2s"). 62 of the 500 most-starred active Python repositories on GitHub use BLAKE2, e.g. PyTorch, Transformers, Home Assistant, Sentry, Streamlit, Gradio and Airflow. Of 1,753 sampled call sites (1,420 repositories), 1,649 (1,336 repositories) only use digest_size, 59 (51 repositories) use key, and 45 (38 repositories) use salt or person without a key. None use tree hashing. Most of them only need a fast hash, for which SHA-2, SHA-3 or BLAKE3 also work. This includes: - projects that store digests, such as Home Assistant (cache filenames), Mathesar (database identifiers) and Onyx (search-index IDs); - projects that depend on the exact BLAKE2 output, such as networkx (graph hashes), LangChain (indexing), apprise and a minisign verifier (BLAKE2b-512 signatures), copyparty (client checksums); - projects that use keyed BLAKE2 such as agno (MinHash) and openai-python (Bedrock helper); or - projects such as PyPI or Poetry for checksums, Warehouse for storage and publishing (though it looks like a legacy API), or python-tuf. One popular project affected by this change would be ComfyUI (which uses person without a key). In particular, I don't expect too much of downstream breakage (considering the long deprecation period) but I would appreciate if anyone against this proposal could explain why SIMD BLAKE2 should be kept (note that I already plan to remove SIMD support in 3.16 without a deprecation notice as it created too many issues, while exposing the OpenSSL implementation). Note that projects can rely on nacl.hash.blake2b (PyNaCl does not offer a 2s version) or PyCryptodome for blake2b and blake2s. AFAIK, there is no PyPI packages providing the tree-hashing feature but the latter is neither standardized by an RFC nor used in the wild. ---- > AI disclosure: Opus 5.5 was used for collecting and classifying data.

Tired of coding separate alerts for every app? Apprise sends messages to Discord, Telegram, Slack and more from one Python library or CLI. Great for server notifications. caronc/apprise #Notifications #Python #fo