python-certifi

eMudhra CA Compromise: Two Roots to Remove, and Dropping Comodo AAA Broke 1 of 16,125 Chains --- title: eMudhra CA Compromise: Two Roots to Remove, and Dropping Comodo AAA Broke 1 of 16,125 Chains published: true date: 2026-11-03 09:00:00 UTC tags: Security,TLS,Certificates,Docker canonical_url: devops-daily.com --- Firefox and Chrome stopped trusting the six compromised eMudhra TLS intermediate CAs within hours of the report on 17 October 2026. Your containers, JVMs, virtualenvs and Go binaries did not get that update. Each of them reads its own copy of a trust store, and that copy decides whether a forged certificate for any hostname is accepted. We collected 34 trust stores from package files, published bundles and source trees, and verified the public test chains for the compromised keys against each one. Then we connected real clients to the public test hosts, and checked what Debian's fix does to the chains that 16,348 of the top 20,000 sites send. Three results matter if you run anything outside a browser: - Two roots, not one. Each compromised TLS key chains to emSign ECC Root CA - G3 and, through a Sectigo cross-sign, to AAA Certificate Services. Before the fix, Debian 11, 12 and 13, Ubuntu 24.04 as released, Alpine 3.18 and OpenJDK's cacerts held both. There, removing only the emSign root leaves the Sectigo path open. - The root list predicts everything. In 560 OpenSSL verifications, the two roots decided every result. The fixed Go, Node.js, Bun and CPython releases block the keys in code. Node.js 24.22.1 and Go 1.26.10 rejected them even with the old Debian 12 bundle. - Removing AAA broke one chain in our test. Of 16,125 served chains that verified before Debian's fix, one (jisc.ac.uk) failed after it with OpenSSL 3. GnuTLS and Go also failed on that site. 262 more fail only for a client that never looks for another path. TL;DR - Update ca-certificates and rebuild images from a fresh base. A digest pinned before 20 October keeps the old bundle. - Debian 11, Alpine 3.18 and 3.19 and Node.js 20 get no fix. Their newest releases still trust the keys. - On Debian and Ubuntu, delete /etc/ssl/certs/java/cacerts and run update-ca-certificates --fresh; the Java keystore is not rebuilt on its own. - Upgrade to runtimes that block the keys in code: Go 1.26.10 or 1.27.3 (then rebuild your Go binaries), Node.js 22.23.4, 24.22.1 or 26.12.1, Bun 1.4.4, Python 3.11.18, 3.12.16, 3.13.18, 3.14.10 or 3.15.2. - If you run a server, do not end your chain at a certificate issued by AAA Certificate Services. - Run scan-tree.py on an unpacked image, a virtualenv or a JDK to find what still trusts the keys. Prerequisites - Container images, CI runners or servers that make outbound TLS connections - Python 3.9 or later and the OpenSSL 3 command line tool, to run the companion scripts - docker (or any tool that can export an image filesystem) if you want to scan images What happened, in short On 17 October, a researcher reported to the Apple, Chrome, Microsoft and Mozilla root programs that the private keys of eight eMudhra intermediate CAs could be computed from their public certificates. Mozilla confirmed it and had Firefox blocking the TLS intermediates through OneCRL by 22:59 UTC the same day. Six of the eight CAs issue TLS certificates; two are code signing CAs (Mozilla bug 2083449). The keys are not theoretical. Between 20 and 21 October, people other than eMudhra logged 107,107 certificates signed with six of the keys to Certificate Transparency. 106,050 of them are proof-of-concept certificates for google.com (comment 44). Almost none of them carry a revocation URL. Browsers get blocks through OneCRL, CRLSets and Apple's and Microsoft's trust updates. Almost everything else reads a file. We used the public chains and test hosts of the incident tracker, which also lists the patched versions. | Date (UTC) | What changed | | --- | --- | | 17 to 19 Oct | Firefox (OneCRL), Chrome (CRLSets) and Apple block the certificates; eMudhra and Sectigo revoke them | | 20 Oct | Go 1.26.10 and 1.27.3; Debian 20250419~deb12u2 and 20250419+deb13u1; Ubuntu 20260816ubuntu0.*; certifi 2026.10.19; curl CA bundle 2026-10-20 | | 21 Oct | Node.js 22.23.4, 24.22.1 and 26.12.1 | | 22 Oct | Bun 1.4.4 | | 24 to 27 Oct | OpenJDK removes four roots from its source tree (JDK-8395690); no OpenJDK update release yet | | 26 Oct | Python 3.15.2, 3.14.10, 3.13.18, 3.12.16 and 3.11.18 | One key, two certificates, two roots Each of the six TLS keys appears in two intermediate certificates with the same subject and the same public key. One was issued by emSign Root TLS CA - G3, which is cross-signed by emSign ECC Root CA - G3. The other was cross-signed by Sectigo's AAA Certificate Services (the old Comodo root, called Comodo AAA Services root in Mozilla's data). A leaf signed with the key verifies through either one. 1. Forged leaf any hostname 2. EM DV TLS CA - G3B issued by eMudhra 3. EM DV TLS CA - G3B cross-signed by Sectigo 4. emSign Root TLS CA - G3 cross-signed 5. AAA Certificate Services Comodo, 2004 6. emSign ECC Root CA - G3 removed 20 Oct Connections: - Forged leaf -> EM DV TLS CA - G3B - Forged leaf -> EM DV TLS CA - G3B - EM DV TLS CA - G3B -> emSign Root TLS CA - G3 - EM DV TLS CA - G3B -> AAA Certificate Services - emSign Root TLS CA - G3 -> emSign ECC Root CA - G3 Mozilla stopped trusting AAA Certificate Services for TLS in April 2025. Its root data (NSS builtins) marks the root as trusted for server authentication in version 2.74. Version 2.76, which came with NSS 3.111, does not (aaa-trust-history.txt reads it from the Firefox source). Debian's 20250419 package is built from 2.74, so Debian 11, 12 and 13 hold AAA, and so do Ubuntu 24.04's release-pocket package and Alpine 3.18. Before the fix, bundles built from later data held only the emSign root. These include Ubuntu's June 2026 update, Alpine 3.19, certifi, curl, Node.js, Bun, RHEL and Temurin. Debian's own August 2026 unstable package, built from 2.90, has no AAA either. So the cause is the age of the data, not Debian's packaging. That is why Debian removed two roots and the others removed one. Finding 1: the root list decides it Every script, recorded result and the scanner are in the companion repository. Each stage writes a file the next stage reads, and every number below comes from one of those files. The-DevOps-Daily/emudhra-trust-store-check on GitHub We took each trust store from a fixed file (a package, a published bundle or a source tree), so we know exactly which version we tested. The repository records the URL and SHA-256 of every download. For each store, a script first predicted the result: it accepts the emSign path if it holds emSign ECC Root CA - G3, and the Sectigo path if it holds AAA Certificate Services. Then openssl verify checked the 16 public test chains against each store, with the clock fixed at 30 October 00:00 UTC. There are eight chains per path, including leaves never logged to CT and chains through a rollover certificate. The result: 560 verifications, 0 that differed from the prediction. A store accepts all eight chains of a path, or none. | Trust store | Before the fix | Fixed in | | --- | --- | --- | | Debian 11 bullseye | both paths (20250419~deb12u1~deb11u1, the newest) | no fix, LTS ended 31 August 2026 | | Debian 12 bookworm | both paths (20250419~deb12u1) | 20250419~deb12u2 | | Debian 13 trixie | both paths (20250419) | 20250419+deb13u1 | | Ubuntu 24.04 | both paths (20240203, release pocket); emSign path (20260601~24.04.1) | 20260816ubuntu0.24.04.1 | | Ubuntu 22.04 | emSign path (20260601~22.04.1) | 20260816ubuntu0.22.04.1 | | Alpine 3.18, 3.19 (EOL) | both paths (3.18), emSign path (3.19) | no fix in either branch | | Alpine 3.22 | not tested | 20261020-r0 (also in 3.20, 3.21, 3.23, 3.24 and edge) | | RHEL 9 family (p11-kit, parsed) | emSign path (2025.2.80_v9.0.305-91.el9) | 2026.2.92_v9.0.319-90.3.el9_8 | | Python certifi | emSign path (2025.8.3, 2026.7.22) | 2026.10.18 | | curl CA bundle | emSign path (2026-09-25) | 2026-10-20 | | Node.js bundled roots | emSign path (20.20.2, 22.23.3, 24.22.0) | 22.23.4, 24.22.1 | | Bun bundled roots | emSign path (1.4.3) | 1.4.4 | | OpenJDK source cacerts | both paths (jdk21u before JDK-8395690) | in source since 27 October; no update release yet | | Eclipse Temurin JRE cacerts | emSign path (21.0.12.1+1; 11.0.32.1+1, the newest 11 build) | 21.0.13+9 holds neither root | Three details from the table are easy to miss: - Temurin is not OpenJDK. OpenJDK's own cacerts holds both roots, and OpenJDK tells you to delete four of them by hand. Temurin builds its cacerts from Mozilla's data, so its 23 October release already holds neither. A JDK from another vendor can be either; check the file, not the version. - certifi over-removed first. certifi 2026.10.18 removed every emSign root, including emSign Root CA - G1, which Mozilla still trusts. Users of the InCommon certificate service, which many US universities use, reported breakage in issue #439, and 2026.10.19 put the unaffected roots back. In our scan below, 47 of the served chains end at emSign Root CA - G1. - Removing one root is not enough everywhere. We built one synthetic store, Debian 12's old bundle minus only emSign ECC Root CA - G3. It rejected all eight emSign-path chains and accepted all eight Sectigo-path chains. That is what you get if you apply the one-root workaround, which is right for Alpine 3.19 and later, to a Debian image. the Sectigo-path chain against three Debian 12 bundles # old bundle: holds both roots $ openssl verify -no-CApath -no-CAstore -attime 1793318400 -CAfile debian-12-before.pem -untrusted aaa-chain.pem aaa-chain.pem aaa-chain.pem: OK # old bundle minus emSign ECC Root CA - G3 only $ openssl verify -no-CApath -no-CAstore -attime 1793318400 -CAfile minus-emsign.pem -untrusted aaa-chain.pem aaa-chain.pem aaa-chain.pem: OK # 20250419~deb12u2: both roots removed $ openssl verify -no-CApath -no-CAstore -attime 1793318400 -CAfile debian-12-fixed.pem -untrusted aaa-chain.pem aaa-chain.pem C = NL, O = eMudhra BV, CN = EM DV TLS CA - G3B error 20 at 1 depth lookup: unable to get local issuer certificate error aaa-chain.pem: verification failed -no-CApath -no-CAstore keeps the machine's own /etc/ssl/certs out of the test, and -attime makes the result repeatable. This command does not check revocation. The 16 intermediate certificates were revoked on 18 and 19 October. In the next section, our client commands did not turn on CRL or OCSP checking either, and with the old bundles they accepted the two public test chains. Finding 2: a blocklist in the runtime beats a stale store Four runtimes did not wait for trust stores. They are Go 1.26.10 and 1.27.3 (go.dev), Node.js 22.23.4, 24.22.1 and 26.12.1, Bun 1.4.4, and the October 26 CPython releases. Each one compiles a list of the eight compromised public keys into its TLS code. It rejects any chain that contains one of them. Bun's release notes mention only a root update; the list is in blocked_spkis.cpp. Our first stage downloads the four source files from the fix commits. All four hold the same eight SHA-256 hashes of the SubjectPublicKeyInfo. The lists match the key, not a certificate. So they cover both certificates of every key, and the rollover chains. To see the difference with real clients, we connected curl, Python, Node.js and Go on a Debian 12 machine to two of the public test hosts: one serves the emSign path, the other the Sectigo path. Each client got a trust store from the matrix above. For Go, we built the same small HTTPS client with Go 1.26.9 and 1.26.10. All 24 results matched the prediction: | Client | Trust store | emSign path | Sectigo path | | --- | --- | --- | --- | | curl 7.88.1 (OpenSSL 3.0) | Debian 12 20250419~deb12u1 | accepted | accepted | | curl 7.88.1 | Ubuntu 24.04 20260601~24.04.1 | accepted | rejected | | curl 7.88.1 | Debian 12 20250419~deb12u2 | rejected | rejected | | Python 3.11.2 (Debian 12) | Debian 12 20250419~deb12u1 | accepted | accepted | | Python 3.11.2 | certifi 2026.7.22 | accepted | rejected | | Python 3.11.2 | certifi 2026.10.19 | rejected | rejected | | Node.js 24.22.0 | bundled roots | accepted | rejected | | Node.js 24.22.0 | --use-openssl-ca, Debian 12 20250419~deb12u1 | accepted | accepted | | Node.js 24.22.1 | bundled roots | rejected | rejected | | Node.js 24.22.1 | --use-openssl-ca, Debian 12 20250419~deb12u1 | rejected | rejected | | Go 1.26.9 | SSL_CERT_FILE, Debian 12 20250419~deb12u1 | accepted | accepted | | Go 1.26.10 | SSL_CERT_FILE, Debian 12 20250419~deb12u1 | rejected | rejected | With the old Debian bundle, curl, Python, Node.js 24.22.0 and Go 1.26.9 accept a chain signed with a compromised key on both paths. Node.js 24.22.1 and Go 1.26.10 with the same bundle refuse both. Go 1.26.10 reports x509: no valid chains built, and Node.js reports CERT_REVOKED: same stale bundle, two Node.js versions (30 October, 11:25 UTC) # debian-12-before.pem is ca-certificates 20250419~deb12u1; get.js is in tools/ $ SSL_CERT_FILE=debian-12-before.pem SSL_CERT_DIR=/nonexistent ./node-24.22.0 --use-openssl-ca get.js dv-g3b-aaa-emudhra.we-have-a-problem.com ACCEPT HTTP 200 $ SSL_CERT_FILE=debian-12-before.pem SSL_CERT_DIR=/nonexistent ./node-24.22.1 --use-openssl-ca get.js dv-g3b-aaa-emudhra.we-have-a-problem.com reject CERT_REVOKED: certificate revoked Debian 12's own Python (3.11.2-6+deb12u9 on our machine) has no denylist, and OpenJDK fixed Java only by removing roots from cacerts. Both are only as safe as the bundle they read. On macOS, Apple says that the system curl, git and python3 check /etc/ssl/cert.pem and accept these chains until a future update. Finding 3: what removing AAA breaks Removing AAA Certificate Services closes the Sectigo path, but other chains still use this 2004 root, and Apple and the incident tracker both warn that removing it can break sites. We measured how many. We connected once to each of the top 20,000 domains of the Tranco list PYK6J, or to www. if the bare name failed. All connections came from one IPv4 address on 30 October between 11:11 and 11:36 UTC, and we saved the chain each server sent. 16,348 answered. Then a small C program on OpenSSL 3.0.22 verified every chain against Debian 12's bundle before and after the fix. Debian 13's bundles hold the same certificates. The program verified each chain in three ways: - modern : the OpenSSL default since 1.1.0, which looks in the trust store first at each step - nofirst : trust store first turned off, alternative chain search on - strict : both turned off, so the verifier follows only the chain the server sent Of the 16,348 chains, 16,125 verified before the fix in modern mode. 1,296 contain a Sectigo, Comodo, USERTrust or AAA certificate, and 283 include a certificate that AAA issued. After the fix: | Mode | Chains that verified before the fix | Anchored at AAA before the fix | Broken by the fix | | --- | --- | --- | --- | | modern | 16,125 | 1 | 1 | | nofirst | 16,124 | 262 | 1 | | strict | 16,111 | 262 | 262 | No chain was anchored at emSign ECC Root CA - G3, so every break comes from AAA. Most of the 262 servers send an old copy of a current root, such as USERTrust RSA Certification Authority, cross-signed by AAA. A modern verifier finds the self-signed copy of that root in the store first and never reaches AAA. A verifier that follows only the served chain gets to AAA, does not find it, and fails: Chains that break only for a client that follows the served chain | | Value | | --- | --- | | USERTrust RSA Certification Authority | 193 | | SSL.com TLS ECC or RSA Root CA 2022 | 31 | | Sectigo Public Server Authentication Root R46 | 23 | | USERTrust ECC Certification Authority | 12 | | COMODO RSA Certification Authority | 2 | | SSL.com TLS Issuing ECC CA R2 | 1 | _262 of the 16,111 chains that verified in strict mode before the fix, Tranco PYK6J top 20,000, scanned 30 October 2026. Grouped by the certificate issued by AAA Certificate Services that the server sends. In modern mode, OpenSSL 3.0 accepts all but one of them after the fix._ The one real break is jisc.ac.uk, and it is not a Sectigo certificate. The server sends its leaf and SSL.com TLS Issuing ECC CA R2, cross-signed by AAA, and nothing else. The same key also has a certificate from SSL.com TLS ECC Root CA 2022 (crt.sh 8505503579), which Debian holds. The server does not send it, and OpenSSL does not fetch missing intermediates. So on our Debian 12 machine after the fix, curl fails: jisc.ac.uk before and after Debian removed AAA # Debian 12, ca-certificates 20250419~deb12u2, 30 October 11:44 UTC $ curl -sS -o /dev/null --max-time 20 -w '%{http_code}\n' https://jisc.ac.uk/ curl: (60) SSL certificate problem: unable to get local issuer certificate More details here: https://curl.se/docs/sslcerts.html curl failed to verify the legitimacy of the server and therefore could not establish a secure connection to it. To learn more about this situation and how to fix it, please visit the web page mentioned above. 000 $ curl -sS -o /dev/null --max-time 20 -w '%{http_code}\n' --cacert stores/debian-12-before.pem --capath /nonexistent https://jisc.ac.uk/ 301 To check this with a second TLS stack, we ran git ls-remote with both bundles. On Debian 12, git 2.39.5 uses GnuTLS 3.7.9. The targets were jisc.ac.uk and a seeded random sample of 20 of the other 261 strict-mode breaks. GnuTLS rejected jisc.ac.uk after the fix and accepted the 20 others. Our Go client, built with Go 1.26.9 and 1.26.10, also failed on jisc.ac.uk with the fixed bundle (x509: certificate signed by unknown authority). So in our tests the AAA removal costs little: one chain in 16,125 for OpenSSL 3, and GnuTLS and Go agreed on the chains we gave them. The risk is in two places. The first is servers that send only an AAA cross-sign of an intermediate, like jisc.ac.uk; the fix is to send the intermediate that the current root issued. The second is clients that cannot search for another path, which we modelled but did not identify. Where the old roots hide Updating the host does not update what your services read. Check these first: 1. Images built before 20 October , and every image pinned by digest. The bundle is a file in a layer. Docker's own note puts it plainly: an image "keeps trusting them until it is rebuilt". 2. Bases and runtimes that will not get a fix. debian:bullseye and everything built on it, alpine:3.18 and alpine:3.19: their newest ca-certificates packages still trust both paths or the emSign path. Node.js 20, end of life since 30 April 2026, still bundles the emSign root and has no blocklist. 3. The Java keystore on Debian and Ubuntu. ca-certificates-java does not rebuild /etc/ssl/certs/java/cacerts when ca-certificates changes, so OpenJDK packages keep the old roots after the update (OpenJDK guidance). 4. Bundles copied by hand. A COPY --from=build /etc/ssl/certs/ca-certificates.crt into a scratch or distroless image, a certifi pin in a lockfile, or SSL_CERT_FILE, REQUESTS_CA_BUNDLE, PIP_CERT and NODE_EXTRA_CA_CERTS pointing at a file from last year. 5. Static Go binaries built with Go older than 1.26.10. The blocklist is compiled in, so only a rebuild adds it. go version -m ./binary prints the Go version that built it. 6. Runtimes with their own roots. Node.js before 22.23.4 or 24.22.1 and Bun before 1.4.4 use bundled roots by default, not the system store. To find them, the companion repository has scan-tree.py. It reads a directory tree and reports bundles, keystores and binaries that hold either root, runtimes that carry the blocklist, and Go programs and CPython ssl modules that do not. This is its real output on a test tree with six files: the old Debian 12 bundle, Node.js 24.22.0 and 24.22.1, a fixed certifi bundle (correctly silent), the ssl module of Debian 12's Python, and Temurin 11's cacerts: TRUSTS fixture/etc/ssl/certs/ca-certificates.crt CA bundle with 150 certificates holds emSign ECC Root CA - G3 and AAA Certificate Services TRUSTS fixture/usr/local/bin/node embeds emSign ECC Root CA - G3 in its bundled roots TRUSTS* fixture/usr/lib/python3.11/lib-dynload/_ssl.cpython-311-aarch64-linux-gnu.so CPython ssl module without the eMudhra blocklist: it accepts whatever its trust store accepts TRUSTS fixture/usr/lib/jvm/temurin-11/lib/security/cacerts CA bundle with 118 certificates holds emSign ECC Root CA - G3 ok fixture/opt/node-24.22.1/bin/node holds all 8 eMudhra blocklist keys (a byte match, not a runtime test) 4 finding(s) that still trust a chain through the compromised keys. TRUSTS*: safe only if every trust store the program reads at run time is fixed. No finding is not proof: only the bundle, keystore and binary formats listed in --help are checked. TRUSTS* means the program has no blocklist of its own, so it is only as safe as the bundle it reads. An ok is a byte match for all eight keys, not a test of the running program. For a container image, export the filesystem and scan it: mkdir rootfs && docker export "$(docker create IMAGE)" | tar -xf - -C rootfs python3 scan-tree.py rootfs Then fix what it finds: # Debian and Ubuntu images: upgrade the bundle, then rebuild the Java keystore apt-get update && apt-get install --only-upgrade -y ca-certificates rm -f /etc/ssl/certs/java/cacerts && update-ca-certificates --fresh # Alpine 3.20 and later (3.18 and 3.19 get no fix: move to a newer base) apk upgrade --no-cache ca-certificates ca-certificates-bundle # Check what an image actually has dpkg-query -W ca-certificates # fixed: 20250419~deb12u2, 20250419+deb13u1, 20260816ubuntu0.* apk info -v ca-certificates-bundle # fixed: 20261020-r0 rpm -q ca-certificates # RHEL 9 fixed: 2026.2.92_v9.0.319-90.3.el9_8 What we could not conclude - How many systems are still exposed. The matrix says what each version does, not how many copies of each version are running. We picked old versions on purpose. - Other JDKs and platform verifiers. We did not check the cacerts of Amazon Corretto, Azul Zulu, Microsoft's build, Oracle JDK or distribution packages. We did not test Windows, macOS or Android, and relied on the vendors for those. - RHEL's real extraction. We did not run trust extract; we parsed the p11-kit file and kept certificates trusted for server authentication. Our fixed RHEL 9 store has 118 certificates, the same count as the curl bundle of 20 October, which is a cross-check, not proof. - Revocation. We did not test clients with CRL or OCSP checks turned on. The leaves on the test hosts have no revocation pointers. The eMudhra CRL endpoint also refused some User-Agents (comment 44), so we would not count on it. - AAA beyond the top 20,000. One IPv4 address, one morning, the bare name or www. only. Internal services, other host names, mail servers and the long tail can differ, and a CDN can send a different chain to a different network. - Which real clients behave like "strict" mode. We modelled them with two OpenSSL flags. OpenSSL 3.0 in modern mode handled 261 of the 262 chains, and GnuTLS 3.7.9 the 20 we sampled. We did not test Java, rustls, Windows, macOS or embedded TLS libraries on them, or Go beyond jisc.ac.uk. Summary - Each compromised key chains to two roots. Stores built from Mozilla data older than April 2025, including Debian 11, 12 and 13, need both emSign ECC Root CA - G3 and AAA Certificate Services removed. - The roots in a store predicted all 560 of our verifications. Check the file, not the version string; scan-tree.py does that for a whole image. - The fixed Go, Node.js, Bun and Python releases block the keys whatever the store says. Debian 11, Alpine 3.18 and 3.19 and Node.js 20 get no fix. - In our OpenSSL test, removing AAA broke 1 of 16,125 served chains (jisc.ac.uk). GnuTLS and Go failed on it too, and the fix for it is on the server. _Originally published at devops-daily.com._

Thank you for the explanation. For everyone: here's a relevant quote from the certifi repo to show just how old the problem is: > The root has been present in certifi since 2019, the fact that the intermediates weren't issued until 2025 doesn't change the fact that if you have a copy of certifi from 2020, it's got the root and therefore you must upgrade. python-certifi#439

Python緊急更新を時系列で解説｜pip・仮想環境に残る対応 2026年10月24日10:06 JST時点。Pythonの証明書問題への対応を、10月20日の注意喚起、21日の緊急版、既存の仮想環境・コンテナの確認まで時系列で解説します。 「Python本体を更新したのに、なぜpipやvenvも確認するの？」に、証明書への信頼が保存される場所をたどって答えます。ずんだもんと四国めたんの対話と、独自の図解で整理しました。 対象は今回のCA証明書問題に関するPython周辺の対応です。過去の標準ライブラリの脆弱性とは別件。配布元独自の修正、OS・アプリのCA設定は個別確認が必要です。PR160042は情報基準時点でOpen。追加防御の提案を配布済みとは扱いません。攻撃可能性と侵害成功、修正版公開と利用環境への適用を分けています。日付は告知・提案の日付で、非公開の発見日は推測していません。時間軸の間隔は模式的です。 本番環境の監査や証明書検証の無効化は行っていません。更新方法は下記公式の最新案内を照合してください。 チャプター 00:00 結論：本体の更新だけでは終わらない 00:26 10月20日：証明書への信頼を見直す 01:07 なぜOSの更新だけでは足りないのか 01:49 10月21日：同梱pipを更新した緊急版 02:48 更新する通信をどう信用する？ 03:25 追加防御は、配布済みとは限らない 04:02 現在：自分の環境まで更新を届ける 04:50 コンテナと更新後の動作確認 05:25 別の脆弱性と混同しない 05:43 まとめ：発表から自分の環境まで 公式出典 discuss.python.org blog.python.org github.com/certifi/python-certifi/security/adviso… pip.pypa.io docs.python.org cpython#160040 cpython#160042 blog.python.org 音声：VOICEVOX:四国めたん / VOICEVOX:ずんだもん 立ち絵：坂本アヒル様（四国めたん・ずんだもん） 図解・構成：How It Really Works? 独自制作 文字：Noto Sans CJK / DejaVu Sans Mono #Python #脆弱性 #セキュリティ [en-US] A Chronological Guide to the Urgent Python Update: Addressing Remaining Issues with pip and Virtu... As of October 24, 2026, 10:06 JST. This document provides a chronological explanation of the response to the Python certificate issue, from the October 20th alert to the October 21st emergency update and the verification of existing virtual environments and containers. It answers the question, "Why check pip and venv even though I updated the Python core?" by tracing where trust in certificates is stored. This is organized with a dialogue between Zundamon and Shikoku Metan, and original diagrams. This document focuses on the response to the current CA certificate issue in Python. It is separate from past standard library vulnerabilities. Distributor-specific fixes and OS/application CA settings require individual verification. PR160042 is open as of the information standard. Additional defense proposals are not treated as distributed. Attack potential and successful breaches, and the release of fixes and their application to usage environments are separated. Dates are the announcement/proposal dates; the discovery date of non-public information is not estimated. The time intervals are schematic. We have not performed audits of the production environment or disabled certificate verification. Please refer to the latest official announcement below for update instructions. Chapter 00:00 Conclusion: Updating the main program alone is not enough 00:26 October 20: Reconsidering trust in certificates 01:07 Why OS updates alone are not enough 01:49 October 21: Emergency version with updated bundled pip 02:48 How to trust the communication for updates? 03:25 Additional defenses may not be distributed 04:02 Currently: Delivering updates to your environment 04:50 Checking container and post-update operation 05:25 Do not confuse with other vulnerabilities 05:43 Summary: From the announcement to my own environment Official sources discuss.python.org blog.python.org github.com/certifi/python-certifi/security/adviso… c7w-9gw2 pip.pypa.io docs.python.org cpython#160040 cpython#160042 blog.python.org Voice: VOICEVOX: Shikoku Metan / VOICEVOX: Zundamon Character Illustration: Sakamoto Ahiru (Shikoku Metan, Zundamon) Diagrams & Composition: How It Really Works? Original Production Text: Noto Sans CJK / DejaVu Sans Mono #Python #Vulnerability #Security

PHP fixes, Python trust updates, NIST governance drafts PHP fixes, Python trust updates, NIST governance drafts Coverage: 2026-10-16T18:00:00Z through 2026-10-23T18:00:00Z (research cutoff). Sources retain their original publication dates; plans, research and availability are qualified in the episode. CHAPTERS 0:00 PHP: patch and investigate 3:46 The Foundation’s account of a delayed decision 5:15 Python: check the trust chain 8:08 NIST: two initial public drafts 10:16 Act on the fixes; review the decision process PRIMARY SOURCES (numbered references; publisher / publication date) [1] PHP Group (2026-10-21) php.net [2] New Zealand National Cyber Security Centre (2026-10-22) ncsc.govt.nz [3] New Zealand National Cyber Security Centre (2026-10-22) ncsc.govt.nz [4] The PHP Foundation (2026-10-21) thephp.foundation [5] US National Institute of Standards and Technology (2026-10-21) csrc.nist.gov [6] US National Institute of Standards and Technology (2026-10-22) csrc.nist.gov [7] certifi maintainers (2026-10-20) github.com/certifi/python-certifi/security/adviso… [8] Python Software Foundation security announcement / Seth Larson (2026-10-20) discuss.python.org [9] Python release team / Hugo van Kemenade (2026-10-21) discuss.python.org [10] Python Software Foundation CNA (2026-10-22) cve.org [11] CPython maintainers (2026-10-21) cpython#160098 [12] CPython maintainers (2026-10-22) cpython#160340 [13] CPython maintainers (2026-10-22) cpython#160339 [14] CPython maintainers (2026-10-22) cpython#160341 [15] CPython maintainers (2026-10-22) cpython#160342 [16] CPython maintainers (2026-10-22) cpython#160343 SERIES Cybersecurity, Privacy & Governance youtube.com VISUAL CREDITS (numbered source references) Sources [2], [3]: NCSC New Zealand · Original explanation Sources [2], [1], [3], [4], [6]: Original explanation Sources [2]: NCSC New Zealand · 22 Oct 2026 © Crown / NCSC New Zealand, headline text CC BY 4.0; visual presentation used only under this finite editorial assessment. Sources [2], [1], [5], [6]: Primary-source title excerpt Sources [2], [1]: narrowly assessed editorial source excerpt Sources [1]: PHP Group · 21 Oct 2026 © The PHP Group; finite news excerpt, no reusable licence. PHP Group · Original explanation PHP Group / PHP Foundation · Original explanation Sources [4]: The PHP Foundation · Original explanation Sources [7]: certifi maintainers · Original explanation Sources [7], [17], [18], [19], [20]: Original explanation · conceptual Sources [17]: PSF security announcement · Original explanation Editorial analysis from PSF advisories · Original explanation Sources [18]: Python release team / PSF · Original explanation Sources [19]: PSF security advisory / CNA · Original explanation Sources [20]: CPython maintainers / release team · Original explanation Sources [5], [6]: NIST · initial public draft National Institute of Standards and Technology; unmarked web information reproduced with credit under its official Use of NIST information policy. No agency logo or endorsement. Sources [6]: NIST · Original explanation VISUAL SOURCE INDEX [17] discuss.python.org [18] discuss.python.org [19] mail.python.org [20] api.github.com Nehkai's appearance and voice are synthetic. This is original AI-assisted reporting and commentary, directed by Setheus Darkhour. We are independent of the organisations we cover; this episode has no paid sponsor. Visuals: the channel's approved synthetic presenter, original explanatory diagrams and credited story-specific source media. Narrow source excerpts carry their documented editorial assessments where indicated. Theme: Sidequest themesong, supplied by the channel owner with commercial-use rights. Diagrams are explanatory; archival footage is labelled. youtube.com

Breach Week Dropped 10 'Patch Now' Posts. A Local Gemma Boiled Them Down to 2 Lines, and Code Checked Every Number It Wrote. This is a submission for the Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady On Oct 21 a post went viral: "the internet broke today", ten incidents in one list. I checked every one against its source. Eight are real, one number was off by about 2,000x, and one never had a source beyond a forum. So I built TwoToe, a sloth that does breach-week triage on my laptop at 3 a.m., or the minute I log on if the laptop was asleep. Code scans the machine and decides what is true. A local Gemma 4 E4B only phrases it. A validator with no AI in it rejects any version, CVE, number or vendor name the model wrote that code did not establish for that finding. In the first eval round it caught 141 of 405 drafts doing exactly that. By round four, gemma4:e4b had 0 of 80 drafts rejected at a 9.0 s median on a 4 GB laptop GPU, and across all four rounds 0 ungrounded tokens reached a final brief. On my own machine the nightly scan covers 9,246 files in 1.3 s. The model's only job is to write two lines and a short checklist each morning, so you can stop reading breach threads and go back to bed. > TL;DR > - What: TwoToe scans your projects and machine overnight against this week's advisories and leaves you two lines and a checklist of at most 7 items: what to do, why, where it was found, how sure it is, and the vendor's link. Then it stays quiet until tomorrow night. > - The model result: four eval rounds on 24 synthetic fixtures, 3 local models, 5 runs each. Round 1: the validator rejected 141 of 405 drafts for an ungrounded token, and caught all of them. Round 4: gemma4:e4b 0 of 80 rejected, 9.0 s median, 19 tokens/s on an RTX 3050 Laptop GPU (4 GB). The matcher found 20 of 20 expected findings with 0 false positives. > - Cost: no API, no account, no key. The model runs in Ollama on 127.0.0.1. The only outside request is the CISA KEV feed. > - Durability: the nightly run is a Temporal workflow. I killed the worker with taskkill /F 8 s after the model had answered: 3 activities were already done and kept, 1 was retried, 0 ran twice, and the model's finished answer came from cache instead of a second model call. > - Honesty: that 0% is partly copying. Gemma kept the template's action word for word in 79 of 95 items. And the validator checks tokens, not meaning: "you are not affected" would pass. Both are in the tables below. > - Everything is open: code, fixtures, all four eval reports, the redacted agent session and the scripts behind every image are in T-RexBytes/twotoe. MIT licensed. > 3 findings in 30 seconds > > 1. The model copies more than it writes. In round 4 Gemma passed the validator every time, mostly by keeping the template's words. Its most personal touch is a line-2 tail, "take a deep breath.", in 46 of 80 briefs. > 2. Every fix was right about the symptom and wrong about the cause. In round 3 I hid the long advisory summary from the model. 87 drafts still contained it word for word. The leak was mine: the allowed-token list carried the whole sentence. > 3. The tool's first real run flagged the tool. TwoToe warned that it was itself running on an unpatched Node (24.11.0; the fix is 24.22.1) that still trusted the eMudhra root. winget's Node package was unpatched too. > Nothing private was sent anywhere. TwoToe reads .env key names, never values. Every fixture .env carries a planted canary value, and it appeared in 0 outputs in every eval round. Indicator files are hashed, never opened. All network goes through an allowlist: loopback, www.cisa.gov, and Sentry only if you opt in. The brief prints that ledger. Every screenshot in this post is a synthetic fixture. --- !TwoToe morning card for Friday, October 23, written by the local model gemma4:e4b: "4 things need you today: update Plex Media Server to 1.43.3.10861 or newer; rebuild env with litellm 1.106.2 and rotate keys; +2 more in the checklist." Below it, the checklist with evidence rows, the network ledger and run stats Why this matters: ten incidents, and the shortcut was backwards > "...may have accessed encrypted environment variables and, for some of them, obtained the decrypted contents." That's Railway on Oct 21, describing what an unauthorized third party did. "May have" and "for some of them" are doing a lot of work in that sentence, and the viral post shortened it to "env vars decrypted". It was one of ten items in a post titled "the internet broke today". Three of the ten (Railway, OpenCode, LiteLLM) were about leaked credentials. Behind the ten sit 16 source pages. The question I had was smaller: which of these hit my machine, and what do I do before breakfast? The shortcut was to paste my lockfiles and .env names into a cloud chatbot. In a week about keys leaking from someone else's cloud, that meant sending a list of my keys to someone else's cloud. So first I checked the list: | # | Viral claim | Verdict | What the sources say | | --- | --- | --- | --- | | 1 | eMudhra: 8 keys, 60M+ certs at risk | Inflated | Keys trivially recoverable; about 30,000 certs need migration; 60M appears in no source; no forged certs reported | | 2 | SBI: 3,897 customers' data on the dark web | Confirmed | Regulation 30 filing, Oct 21 | | 3 | Rajasthan Police, 70,000 records | Unverified | Only a forum claim. Deliberately not in TwoToe's pack | | 4 | PHP "php2shell", CVSS 9.2, patch or go offline | Partly | Two CVEs, CVSS v4.0 9.2; CISA's offline advice is for 8.1 and older or systems that can't update; not in KEV | | 5 | GitLab CVSS 10.0 unauthenticated RCE | Confirmed | CVE-2026-110264, fixed in 7 branches on Oct 21 | | 6 | Node.js Brotli RCE + TLS bypass, 10 CVEs | Confirmed | 10 CVEs, 4 High; fixed in 22.23.4, 24.22.1, 26.12.1 | | 7 | Railway breached, env vars decrypted | Confirmed (wording) | "may have" accessed, "some" decrypted, a subset of users | | 8 | OpenCode provider keys exposed | Confirmed, wrong date | Disclosed Oct 17, not Oct 21; 1.4% of Console users | | 9 | LiteLLM 1.107.0 on PyPI was malware | Confirmed | 1.106.2 is the last safe release | | 10 | Plex2Shell exploited, KEV deadline Oct 23 | Partly | In KEV since Oct 20; two CVEs due Oct 23, two due Nov 3 | "About 2,000x": 60,000,000 / 30,000. Every fact in TwoToe's advisory pack links to the page it came from and carries the date it was last checked. All of them were re-fetched on Oct 23. {% details Sources (all fetched Oct 23) %} - Railway: blog.railway.com - Node.js: nodejs.org - PHP: php.net and cisa.gov - GitLab: docs.gitlab.com and docs.gitlab.com - Plex: forums.plex.tv and cisa.gov - LiteLLM: docs.litellm.ai and wiz.io - eMudhra: bugzilla.mozilla.org, scotthelme.co.uk and github.com/certifi/python-certifi/security/adviso… - OpenCode: thehackernews.com and infosecurity-magazine.com - SBI: internetfreedom.in - CISA KEV feed: cisa.gov {% enddetails %} What's new here | Existing option | What it gives you | What it can't tell you | | --- | --- | --- | | Reading every advisory | Every fact, in full | Which ones apply to this machine, and which key to rotate first | | osv-scanner, npm audit, pip-audit | Known-bad versions in a lockfile | Breaches with no package to scan (Railway env vars, OpenCode console keys), CA trust (eMudhra), installed apps (Plex). Most of them also send your package list to an online database by default | | gitleaks, trufflehog | Secret values found in files | They read the values. In a credential-leak week, the triage tool shouldn't hold your secrets | | Asking a cloud chatbot | A decent summary | Whether its version numbers are right, and it needs your key names in someone else's cloud | | TwoToe | Two lines and a checklist, made on your machine, with every version, CVE and number checked by code | It checks tokens, not meaning. The pack covers one week's incidents plus CISA KEV | What I Built TwoToe turns breach week into two lines you can read before coffee. You install it once. At 3 a.m. a Temporal schedule scans your project folders and machine (lockfiles, Python envs, runtimes, installed apps, trust stores, .env key names, indicator files), matches the result against this week's advisory pack, and has a local Gemma write the brief. This is a real run on the kitchen-sink fixture, where Plex, LiteLLM, Railway and Node are all vulnerable: PS twotoe> pnpm twotoe brief --fixture eval/fixtures/f22-kitchen-sink twotoe Fri Oct 23, 07:00 4 things need you today: update plex to 1.43.3.10861; rebuild your litellm env on 1.106.2; +2 more in the checklist. Everything else: you are doing great. Checked 14 things in 32.8s, 0 secrets left this machine. [ ] 1 Update Plex Media Server to 1.43.3.10861 or... definite Plex Media Server [ ] 2 Rebuild the env with litellm 1.106.2 and ro... definite .venv [ ] 3 Roll every env var, API key, and registry c... likely railway.toml, .env (RAILWAY_TOKEN) [ ] 4 Upgrade Node from 24.22.0 to 24.22.1 definite node --version card: twotoe card model: gemma4:e4b (llm) validator: 0 rejections Line 1 is what to do. Line 2 is everything else. Its numbers ("Checked 14 things", "0 secrets left this machine") are computed by code from the scan and the network ledger, never written by the model. Each checklist item says what to do, why, where it was found, how sure TwoToe is (definite, likely, possible), and links the vendor advisory. A possible item has to start with "Confirm", and the validator enforces that. You read it, you fix two things, and you close the terminal. There's no feed, no streak, and no dashboard asking you to come back. The number I care about is minutes not spent reading breach threads. !The morning card at phone width: Friday, October 23, a sleeping sloth, "Good morning.", line 1 in large red type, then the checklist, most urgent first The morning card (twotoe card) serves the same brief on 127.0.0.1 only, with the evidence rows, the network ledger and run stats. It remembers your ticks in the browser and nothing else. Lighthouse accessibility is 100 in light and dark mode. !The full morning card in dark mode for the kitchen-sink fixture: four checklist items with evidence and vendor links, a network ledger with one 7.8 KB POST to 127.0.0.1 under "0 secrets left this machine.", and run stats: 15 checks, 40.9 s, 0 validator rejections Demo Everything runs locally. There's no .env, no API key and no account. You need Node 24.22.1 or newer (pnpm refuses anything older), Ollama, and the Temporal CLI only for the nightly run and the crash test. git clone https://github.com/T-RexBytes/twotoe; cd twotoe corepack enable --install-directory (npm config get prefix) pnpm pnpm install ollama pull gemma4:e4b pnpm twotoe brief --no-llm --fixture eval/fixtures/f22-kitchen-sink # no model: the template writes it pnpm twotoe brief --fixture eval/fixtures/f22-kitchen-sink # Gemma writes it, the validator checks it pnpm twotoe ask "why does the Node one matter?" --fixture eval/fixtures/f22-kitchen-sink pnpm --filter @twotoe/card build; pnpm twotoe card # morning card on 127.0.0.1:4777 pnpm tsx packages/worker/scripts/crash-test.ts # kills the worker mid-run, shows it resume pnpm twotoe brief --save # your own machine !The brief command typed in Windows PowerShell, the model wait sped up and captioned, then the real two lines and checklist There's no hosted demo on purpose: the real card never leaves 127.0.0.1. The card ships with three example briefs, each a real run on a synthetic fixture. Here are the other two, a Railway user and a clean night: !Morning card for the Railway user fixture: one thing needs you today, roll the env vars and keys stored in Railway !Morning card for the clean machine fixture: "Nothing needs you today." Code {% github T-RexBytes/twotoe %} packages/core holds the scanners, the matcher, the advisory pack and the egress guard. packages/writer holds the prompt, the validator and the template. packages/worker is the Temporal workflow, packages/cli the commands, apps/card the morning card, and eval/ the 24 fixtures and the harness. Every number in this post comes from eval/RESULTS*.md, docs/post/build-log.md or a screenshot in docs/post/images/. 422 tests pass locally and on GitHub CI, with 94% line coverage in core and 98% in writer. MIT licensed. How I Built It 03:00 Temporal schedule (catch-up 18 h, overlap skip) | collectInventory scanners: lockfiles, Python envs, runtimes, apps, trust stores, | .env key NAMES, indicator files (SHA-256 only) loadPack this week's advisories, each with source URLs and verifiedAt | matchAdvisories pure functions, no AI -> findings + allowedTokens | writeBriefWithValidation Gemma 4 E4B (Ollama, 127.0.0.1) writes JSON | validator (no AI) checks every version, CVE, number, vendor name | rejected -> one retry with the reasons -> still rejected -> template assembleBrief line-2 numbers and the network ledger, computed by code | persistAndNotify brief saved; card on 127.0.0.1:4777 The rule behind all of it: code owns the facts, the model only phrases them, and code checks what the model said. The first wall: code decides what's true The scanners collect facts and nothing else. Each one is built around what it must never do: | Scanner | What it reads | What it never does | | --- | --- | --- | | .env files, compose environment: | Key names, like RAILWAY_TOKEN | Hold a value. The parser is a character state machine that only remembers which quote is open, so KEY= lines inside a quoted PEM are never reported | | Indicator files (cook.pth, cook.py, sitecustomize.py) | Size and SHA-256 of the raw bytes | Open them as text, import them, or run them. A test spies on every file-reading function to prove it | | Python envs, Node lockfiles, Dockerfiles | Package versions, like litellm 1.107.0 | Run pip, npm or pnpm | | Runtimes and installed apps | node --version, Plex's version | Use a shell. execFile only, 5 s timeout, empty environment, empty temp folder | | Trust stores | Whether the eMudhra root is still trusted | Change anything | The matcher turns that inventory into findings. Each finding carries its evidence and an allowedTokens list: the exact versions, CVE IDs, vendor names and numbers the model may say about that finding. This is the Node finding from the run above, trimmed: { "id": "nodejs-2026-10-21:22619b27", "confidence": "definite", "severity": "high", "evidence": [{ "detector": "runtimeVersion", "location": "node --version", "detail": "node 24.22.0 installed" }], "allowedTokens": ["Node.js", "Node", "Brotli", "TLS", "24.22.0", "24.22.1", "CVE-2026-111497", "..."], "vars": { "found": "24.22.0", "fixed": "24.22.1" } } The model isn't needed to decide anything. brief --no-llm writes the full brief from templates. Priority is plain arithmetic: severity, minus KEV and due-date bonuses, minus confidence. Ties go to leaked credentials, because a leaked key is being used now and a patch gap is only being scanned for. !The same kitchen-sink brief written by templates alone, with no model call The second wall: a validator with no AI in it Gemma 4 E4B gets the findings, not the scan, and returns JSON: the line-1 actions, a short line-2 tail, one checklist item per finding, and FYI lines. The validator pulls every risky token out of that text and checks it against the allowed set for that finding only, so a real version borrowed from another finding fails too. This is the per-item check from packages/writer/src/validate.ts, trimmed: for (const item of draft.checklist) { const f = byId.get(item.findingId); if (!f || !expectedIds.has(f.id)) continue; const action = item.action.trim(); const why = item.why.trim(); const allowed = allowedSet([f]); // this finding's tokens, nobody else's for (const tok of ungrounded(`${action} ${why}`, allowed)) reasons.push(`ungrounded token "${tok}" in ${f.id}`); for (const w of badHype(`${action} ${why}`, [f], kevOf)) reasons.push(`hype word "${w}" not allowed in ${f.id}`); if (f.confidence === "possible" && !/^Confirm\b/.test(action)) reasons.push(`possible finding ${f.id} must start with "Confirm"`); if (why.length > MAX_WHY) reasons.push(`why over ${MAX_WHY} characters in ${f.id}`); } The rest of the rules: - Hype words are earned. "critical" passes only when a finding's severity is critical, and "actively exploited" only when CISA KEV lists it. - The line-2 tail claims nothing. No digits, no versions, no vendor names, 60 characters at most. - Rejected once: the model gets one retry with the reasons. Rejected twice: the template writes the failing parts, and the brief says so (llm-fallback). - Cloud model tags are refused. Ollama has gemma4:cloud, so a typo in the config could otherwise send findings off the machine. A round-1 draft from qwen3:4b shows why this matters. The fixture's only finding is OpenCode, a possible, so line 1 should have no action at all: DRAFT line1Actions: ["bump Node to 24.22.1"] VALIDATOR line1Actions has 1 item(s), expected 0 ungrounded token "24.22.1" in line1Actions ungrounded token "Node" in line1Actions why over 140 characters in opencode-console-2026-10-17:fb74fe33 FINAL line 1: "Nothing needs you today." The OpenCode "Confirm..." item kept, its why from the template "Bump Node to 24.22.1" came from my own prompt's few-shot example. The model read a brief about OpenCode and told me to upgrade Node, because my example did. Follow-up questions go through the same wall. twotoe ask answers only from tonight's findings: !twotoe ask "why does the Node one matter?" on the kitchen-sink fixture: a two-sentence answer about the Brotli and TLS fixes and the upgrade from 24.22.0 to 24.22.1, based on finding nodejs-2026-10-21, validator passed Results !Line chart, "How often the validator said no": share of model drafts rejected per eval round. gemma4:e2b falls from 96% to 25%, gemma4:e4b from 43% to 0%, qwen3:4b sits near 60% for three rounds, then drops to 0% in v4. Ungrounded tokens in any final brief: 0 in every round | Writer (round 4) | Drafts rejected | Ungrounded tokens before the validator | Ungrounded tokens in final briefs | Briefs needing the template | Median / p90 writer time | Tokens/s | | --- | --- | --- | --- | --- | --- | --- | | Template only (no model) | 0 of 24 | 0 | 0 | n/a | about 0 s | n/a | | gemma4:e4b (default) | 0 of 80 | 0 | 0 | 0 of 80 | 9.0 s / 13.4 s | 19.0 | | gemma4:e2b | 23 of 94 (24.5%) | 0 | 0 | 9 of 80 | 2.9 s / 6.5 s | 65.0 | | qwen3:4b | 0 of 80 | 0 | 0 | 0 of 80 | 12.6 s / 22.7 s | 12.5 | _{24 synthetic fixtures, 16 with findings, 5 runs per model, so 80 briefs per model. Prompt version 2026-10-22.3, RTX 3050 Laptop GPU (4 GB). The kitchen-sink fixture end to end with e4b: 26.9 s median for 4 findings.} Did the fixes work, or did I loosen the check? Same fixtures, same validator, four rounds: | Round | What changed | Drafts with an ungrounded token (before the validator) | gemma4:e4b drafts rejected | e4b briefs needing the template | | --- | --- | --- | --- | --- | | v1 | First prompt | 141 of 405 | 43% (56 of 129) | 7 of 80 | | v2 | Exact counts in the prompt, a fake few-shot example | 0 of 384 | 48% | 14 of 80 (worse) | | v3 | The model sees only a summary that fits 140 characters | 0 | 26% | 10 of 80 | | v4 | Allowed-token leak fixed | 0 | 0% (0 of 80) | 0 of 80 | That first row is the whole project in one line. 141 drafts contained a version, CVE, number or product name that code had not established for that spot. Not one reached a brief. Some were inventions, like fake finding ids or a Node version in an OpenCode brief. Most of gemma4:e2b's were real facts in the wrong section. The matcher, which has no model in it, found 20 of 20 expected findings with 0 false positives and 0 misses, and put the kitchen-sink items in the right order. Read the tables with these in mind: - 0% rejected is mostly copying, not writing. gemma4:e4b kept the template's action word for word in 79 of 95 checklist items and reworded the "why" in about two thirds. qwen3:4b copied more: 85 of 95 actions, 63 of 95 "why" lines, and "go back to sleep." in 51 of 80 briefs. The validator didn't get looser; the models got closer to the facts they were handed. Whether that reads better than the template is still open: a blind A/B kit is in eval/blind/, and I haven't run it with readers yet. - gemma4:e2b is 3x faster and still the only model rejected. Its 23 rejected drafts are all formatting: line-1 actions over 60 characters, FYI items in the checklist, FYI lines over 160. - The validator checks tokens, not meaning. A grounded version used in the wrong direction, a negation ("you are not affected") or a confident tone on a possible finding would all pass. That's listed in the repo as a known gap. - I wrote the matcher and the ground truth. So 20 of 20 is weaker evidence than it looks. kev-recent has no fixture: 0 / 0 / 0 there means "not exercised", not "perfect". - Five runs with one seed are one sample. Seed 42 at temperature 0.2 gave byte-identical briefs, so the eval varies the seed per run. {% details Evaluation protocol, definitions and numbers %} | Quantity | Definition | | --- | --- | | Fixture | A synthetic home folder with hand-written ground truth in expected.json. 24 of them, 16 with findings. Every one has a canary value in a real .env | | Draft | One model response. A brief takes one draft, or two when the validator rejects the first | | Ungrounded token | A version, CVE, number or vendor name in the model's text that isn't in that finding's allowedTokens | | Drafts rejected | Drafts with at least one validator reason: grounding, hype word, the Confirm rule, counts or length caps | | Briefs needing the template | The second draft was rejected too, so the template wrote the failing parts (llm-fallback) | | Matcher true positive | Both the advisory and the confidence tier match the ground truth | | Runs | 5 per model. Run r uses seed 41 + r (run 1 is the production seed, 42). Temperature 0.2, thinking off, model warmed up before timing | | Hardware | AMD Ryzen 7 7445HS, 15.3 GB RAM, NVIDIA RTX 3050 Laptop GPU (4 GB), Windows 11, Ollama 0.40.8 | | Wall time | v1 64.7 min, v2 56.1 min, v4 39.1 min. v3 was stopped for low memory at 239 of 240 briefs and rebuilt from the per-brief artifacts; the report names the missing brief | | Canary and egress | 0 canary hits in every round. Egress during every eval: 127.0.0.1 only | Ground truth in eval/fixtures/*/expected.json was never edited to make a number pass. {% enddetails %} Making the nightly run survive with Temporal A 3 a.m. job on a laptop has two enemies: the laptop being asleep, and the job dying halfway through a model call (42 s in the Sentry trace below). Each step is a Temporal activity: - The schedule lives in Temporal: 03:00 local, overlap SKIP, catch-up window 18 h. My spec said 6 h. With 6 h, opening the laptop after 9 a.m. silently skips the night's brief. 18 h still runs it the same day and can never overlap the next night. - Temporal can't wake a laptop, and I don't pretend it can. The catch-up window runs the missed brief a minute after you log on. A Task Scheduler mode (sleep --no-temporal) sets WakeToRun, for machines with wake timers allowed. - The inventory and pack move between activities as on-disk cache references, so env key names never enter Temporal's history. - Model responses are cached per request inside the activity. A retry after a crash reuses a finished answer. A validator retry is a new request, so it doesn't count as a duplicate. - A model that's down gets no second draft. If Ollama refuses the connection, the brief skips the validator retry, falls back to the template right away and records why. Retrying a refused connection would only add 90 s. Crash run, with the real model. Real Temporal dev server, real gemma4:e4b, then taskkill /T /F on the worker after the model had answered. The test holds the activity in a 20 s pause after the answer arrives, so the kill lands while that answer exists only in the activity's cache. | Time | What happened | | --- | --- | | t+0 s | Workflow twotoe-crash-2026-10-23T14-49-43-674Z starts on worker 1 | | t+53.2 s | The model has answered. The activity enters the 20 s pause | | t+61.2 s | taskkill /PID 16884 /T /F: worker 1 killed 8 s into the pause, with 3 activities already done | | next | Worker 2 (pid 23388) starts as a fresh process and waits out the 30 s heartbeat timeout | | t+89.9 s | Workflow completed: llm mode, 4 checklist items. writeBriefWithValidation ran attempt 2 and took the model's answer from cache | Afterwards: 0 activities executed twice, 1 model call and 1 reuse from cache, 0 model requests computed twice, and 0 canary hits in the cache, logs or briefs. !Crash test summary in PowerShell: workflow completed in llm mode with 4 checklist items in 89.9 s; worker 1 killed 8 s into a 20 s delay inside writeBriefWithValidation; 3 activities completed before the crash; 1 retried; 0 executed twice; model calls and reuses 1 and 1; 0 model requests computed twice; 0 canary hits Here's Temporal's own timeline of another crash run that afternoon. collectInventory, loadPack and matchAdvisories finished before the kill. The long 2 • writeBriefWithValidation bar is the killed activity waiting out the dead worker and finishing on attempt 2: !Temporal Web UI, workflow twotoe-crash-2026-10-23T14-41-35-547Z completed in 1 m 6 s: collectInventory, loadPack and matchAdvisories finish early, a long "2 • writeBriefWithValidation" retry bar spans the gap, then assembleBrief and persistAndNotify. The workflow input shows a relative .twotoe-demo config path !The same timeline during a real crash run, sped up about 8x Look at the workflow input in that screenshot: a relative .twotoe-demo path. Until Oct 23 that box showed my real user folder. That story is the second item in the next section. Watching the agent with Sentry Telemetry is off by default, and the Sentry SDK isn't even loaded unless you turn it on. When you do, each run is one trace: twotoe.brief -> scan -> match -> invoke_agent TwoToe -> chat gemma4:e4b -> execute_tool groundingValidator, and again for each retry. In this trace the first validator call is red and says why: !Sentry Agent Activity for TwoToe on gemma4:e4b, 1.12 min: the first groundingValidator call is red, 2.51 ms, and its output reads "forced rejection for the demo trace (draft was otherwise checked as usual)"; the second chat call and the second validator call pass What the trace showed me is that the check is free. The validator took 2.51 ms. The model call it guards took 42.43 s (2.9K tokens in, 472 out), and the retry took 24.88 s. The whole run is 9 spans and 1.12 minutes. The rejection is data on the span (twotoe.validator.rejections: 1), next to the model's token counts: !Validator span attributes in Sentry: twotoe.attempt 1, twotoe.validator.rejections 1 !The chat gemma4:e4b span in Sentry: model name, 2.9K input tokens, 472 output tokens, and the draft itself (demo mode on a fixture only) Honesty note: the first rejection in this trace was forced (--force-reject-first, fixtures only) so the retry would show, and the validator's output on that span says so. Sentry didn't find a bug for me; the bugs below came from Temporal UI screenshots and tests. Prompts and responses are attached only in demo mode on fixtures. The trace upload is the only request besides loopback, and it shows up in the brief's own network ledger. What went wrong - My model warm-up didn't warm anything. twotoe doctor loaded Gemma with Ollama's default context size. The brief asks for 8,192, so Ollama reloaded the model on the first brief: 45 s before the fix, 29.5 s after. A screenshot said 45 s next to an eval median of 26.9 s, and that's how I noticed. - Temporal's history held my home path and my PC's name. The schedule passed the absolute config path as input, activity results carried absolute evidence paths, and the worker identity was pid@hostname. All of it shows in the Temporal UI and stays in temporal.db. The test meant to catch this could never fail: it searched JSON.stringify(history), but payloads are bytes. The fixed test decodes every payload, and it's what found the config path. - The crash test failed once, and the bug was in the test. It required exactly one model call. That run legitimately needed two, because the validator rejected the first draft. It now checks that no model request is computed twice. - Three crash runs, three kill times, one finish time. All three first runs finished at 88.4 s, despite kills at 52.5, 64.9 and 63.7 s. It looked like a bug. It was the SDK throttling heartbeats to every 24 s (0.8 x the 30 s timeout), so recovery is "last heartbeat that got through + 30 s + 10 s backoff". My code says "heartbeat every 10 s". The server sees 24. - My "hidden" nightly tasks weren't hidden. Windows Terminal ignores -WindowStyle Hidden, so two windows opened at log on. And the launcher died on the worker's first stderr line, because PowerShell 5.1 turns redirected stderr into errors. Fixed with OS-level redirection and conhost.exe --headless. - Nested egress guards left the process unguarded. Installing a second guard replaced the first, and uninstalling it removed both. No shipped command nests guards, but one of my screenshot scripts did, and its ledger came back empty. Guards now stack, and an inner guard can only narrow the allowlist. - My spec's own mockup said "Thu Oct 23". Oct 23, 2026 is a Friday. The test caught it because the code computes the weekday. Why Does Open Innovation Matter? A triage tool that needs your credentials in someone else's cloud is incident number eleven. Railway, OpenCode and LiteLLM were all credential incidents. TwoToe's model runs on 127.0.0.1, and the brief proves it with a network ledger built from every request. Open weights let me measure instead of hope. Each eval round is 240 briefs, and the timed rounds took 39 to 65 minutes on a laptop GPU, with no API bill and no rate limit. Four rounds is how I found the leak in my own allowed-token list. I could afford the fourth. It works when everything else is down. No account, no key. --offline skips even the KEV refresh. If your cloud keys get revoked in the middle of an incident, triage still runs. Swapping models is one setting. TWOTOE_MODEL, the only env var TwoToe reads, switches between any Ollama model. Tags containing cloud or -remote are refused. The facts are open too. Every advisory in the pack has its source links and a verifiedAt date. Anything I couldn't confirm is listed in docs/VERIFY.md instead of shipped. The honest caveat: the advisory pack is hand-curated for one week. CISA KEV refreshes itself; next week's Railway won't. And if you turn Sentry on, that part is a cloud service: it gets span names, timings and token counts, never paths, key names or prompts outside demo mode. My Agent Session I built TwoToe with Claude Code, running natively on Windows (PowerShell, not WSL), from Oct 22 to Oct 23: 41 of my messages and 1,099 tool calls. It worked one phase at a time from docs/SPEC.md, with a hard stop at every gate for my "go", under a CLAUDE.md of hard rules: never read env values, never open indicator files, never run package managers, nothing on the network outside the allowlist, and never edit expected.json to make a metric pass. My first message ended: "never invent advisory facts, never weaken a hard rule to make a test pass, and tell me plainly when something failed." The whole session is in the repo, redacted (home paths, user and host names, emails, Sentry DSN fragments, keys): the conversation and every tool call. Three moments worth opening: - Gate 0, before any code: it re-fetched every advisory and checked my own spec against the sources. 9 items were wrong, unsupported or still marked VERIFY, from GitLab's fix versions to the eMudhra "60M". The pack has the corrected facts. The two primary sources it couldn't read (SBI's filing returned HTTP 403) are named, with the secondary sources used instead. - Gate 6c, the third eval: free RAM dropped to 2.3 GB of 15.3 GB, with Ollama holding about 6 GB, so the run was stopped at 239 of 240 briefs. The report was rebuilt from the per-brief artifacts and names the one missing brief instead of hiding it. - Phase 8, ask: my spec suggested Mastra for the follow-up command. The agent found that the Mastra package sends usage data to PostHog by default, which breaks the network rule, so ask runs on the same local model and validator instead. The Lazy Payoff Here's the breach week I didn't have to do by hand: open sixteen source pages, find the fixed versions, run node -v, check every Python venv for litellm, grep every .env for Railway keys, look up whether my CA bundles still trust a root I'd never heard of, and do it all again next week. TwoToe's nightly scan of my real machine covers 9,246 files in 1.3 seconds and runs 49 checks before I've opened a terminal. Then I read two lines. The first real run found an unpatched Node and an eMudhra root still trusted by Node, by Git for Windows' CA bundle and by certifi in 1 of 3 Python environments. After the Node upgrade, the same scan shows Node no longer trusts the root, and the other two are next on the checklist. And when I killed it mid-thought, it finished anyway, without asking the model twice. Prize Categories - Best Use of Gemma: Gemma 4 E4B (gemma4:e4b, Ollama, 100% on a 4 GB RTX 3050 laptop GPU) is the writer. I ran a bake-off against gemma4:e2b and qwen3:4b over four eval rounds, 24 fixtures, 5 runs each: e4b went from 43% of drafts rejected to 0 of 80, at a 9.0 s median, with 0 ungrounded tokens in any final brief. e2b is 3x faster but still 24.5% rejected, and qwen3:4b is 40% slower. I've shown where the 0% flatters it. - Best Use of Temporal: a Temporal Schedule (03:00, 18 h catch-up, overlap skip) and a six-activity workflow, with the model's answer cached per request inside the activity. I killed the worker with taskkill /F after the model had answered: 3 activities kept, 1 retried, 0 executed twice, and the answer came from cache, all visible in Temporal's own timeline. Temporal's UI is also how I found my home path in the history, and that's fixed. - Best Use of Sentry Agent Tracing: one trace per run, with invoke_agent, chat and execute_tool spans carrying gen_ai.* model names, token counts and tokens/s, and every validator verdict as its own span with the rejection reasons as data. The trace shows a 2.51 ms check guarding a 42.43 s model call. It's opt-in only and scrubbed of paths, key names and the canary, and the forced demo rejection is labelled on the span. Built by @t-rexbytes, @soumyadeepdey & @subarnamaity with Claude Code. What's the scariest "patch NOW" post you read this month that turned out to be about software you don't even run? Leave it in the comments. If it involves a CA you'd never heard of, I may need it for next week's pack.

US unis were amongst the first folk complaining about the certifi Python library zapping the eMudra CAs from their cacerts. python-certifi#439

Communiqué eMudhra : emudhra.com Google Chrome : blog.google Mozilla Firefox : blog.mozilla.org certifi (Python) : python-certifi#439

Eight CA private keys under 256: checking every EC key in CCADB with the Python standard library On 17 October, someone told the browser root programs they could recover the private keys of eight intermediate CA certificates issued by eMudhra. Mozilla's incident report, Bugzilla 2083449, explains how that was possible in one line: > The 8 impacted keys all used integers less than 256. A P-384 private key should be a random number about 384 bits long. These were one byte. Every signature they made was valid and every chain built on them validated. Anyone could find the keys by trying 255 numbers. I wanted to see that myself without installing anything, so I wrote a check that uses only the Python standard library. It reads the public key out of every certificate in CCADB's public reports, compares it with the public keys for private keys 1 to 65,535, and matches SPKI hashes against the list Go shipped on 19 October. Then I pointed a second script at the trust store on the machine I ran it on, and found the distrusted root still in it. Everything below was run on 22 October 2026. What the incident report says From the report and its comments, attributed to whoever said it: - The report reached Mozilla at 20:06 UTC on 17 October with proof of key control for all eight. Mozilla says it recovered the keys independently by 21:48 UTC. - Six are TLS intermediates trusted by Chrome, Mozilla, Microsoft and Apple. Two are code-signing intermediates trusted by Microsoft. - eMudhra revoked all eight on 18 October with the reason cACompromise. Sectigo revoked its cross-signatures of the same keys on 19 October. - eMudhra says the keys came from one key ceremony on 30 April 2025, only one of the eight was in production, and no end-entity certificates were issued from any of them. Its preliminary assessment points at a flaw in the key-generation script during a wrap/unwrap test that departed from its usual ceremony. It says it is still assessing. - On 20 October, Mozilla wrote that "multiple individuals have independently recovered the private keys" and removed trust in the whole emSign ECC Root CA - G3. Chrome said it would take a similar path. Go's fix landed a day earlier. Go 1.26.10 and 1.27.3 add a distrustedSPKIHashes map of eight SHA-256 hashes to crypto/x509 and drop any chain containing one of them. The commit names no CA. My reading of why Go needed its own list comes from verify.go itself. The doc comment on Verify says "this function doesn't do any revocation checking", and on Linux the roots come from the OS bundle (root_linux.go tries /etc/ssl/certs/ca-certificates.crt first). With no CRL check and no control over the OS bundle, a list of bad keys is the one thing Go can enforce everywhere, and the same commit adds the check to the macOS and Windows verifier paths too. Finding a small private key without any discrete-log maths An EC public key is the point d·G, where d is the private key and G is the curve's generator. Recovering d from a properly random key is the hard problem the whole scheme rests on. Recovering it when d is below 256 is a loop: compute G, 2G, 3G and so on, and compare each one with the public key. Here is the curve arithmetic for P-256 and P-384, in affine coordinates with Python's built-in modular inverse. The self-test matters more than it looks. A typo in the generator or the prime would make the table silently useless, and checking that G is on the curve and that n·G is the point at infinity catches a typo in any of the four constants. import csv, hashlib, re, sys from base64 import b64decode # NIST P-256 and P-384: field prime, curve b (a = -3 on both), generator, order CURVES = { "P-256": dict( oid=bytes.fromhex("2a8648ce3d030107"), p=2**256 - 2**224 + 2**192 + 2**96 - 1, b=0x5AC635D8AA3A93E7B3EBBD55769886BC651D06B0CC53B0F63BCE3C3E27D2604B, g=(0x6B17D1F2E12C4247F8BCE6E563A440F277037D812DEB33A0F4A13945D898C296, 0x4FE342E2FE1A7F9B8EE7EB4A7C0F9E162BCE33576B315ECECBB6406837BF51F5), n=0xFFFFFFFF00000000FFFFFFFFFFFFFFFFBCE6FAADA7179E84F3B9CAC2FC632551), "P-384": dict( oid=bytes.fromhex("2b81040022"), p=2**384 - 2**128 - 2**96 + 2**32 - 1, b=0xB3312FA7E23EE7E4988E056BE3F82D19181D9C6EFE8141120314088F5013875AC656398D8A2ED19D2A85C8EDD3EC2AEF, g=(0xAA87CA22BE8B05378EB1C71EF320AD746E1D3B628BA79B9859F741E082542A385502F25DBF55296C3A545E3872760AB7, 0x3617DE4A96262C6F5D9E98BF9292DC29F8F41DBD289A147CE9DA3113B5F0B8C00A60B1CE1D7E819D7A431D7C90EA0E5F), n=0xFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFC7634D81F4372DDF581A0DB248B0A77AECEC196ACCC52973), } def ec_add(c, P, Q): """Affine point addition on y^2 = x^3 - 3x + b. None is the point at infinity.""" p = c["p"] if P is None: return Q if Q is None: return P if P[0] == Q[0] and (P[1] + Q[1]) % p == 0: return None if P == Q: lam = (3 * P[0] * P[0] - 3) * pow(2 * P[1], -1, p) % p else: lam = (Q[1] - P[1]) * pow(Q[0] - P[0], -1, p) % p x = (lam * lam - P[0] - Q[0]) % p return x, (lam * (P[0] - x) - P[1]) % p def ec_mul(c, k, P): R = None while k: if k & 1: R = ec_add(c, R, P) P = ec_add(c, P, P) k >>= 1 return R def self_test(c): x, y = c["g"] assert (y * y - (x**3 - 3 * x + c["b"])) % c["p"] == 0, "G is not on the curve" assert ec_mul(c, c["n"], c["g"]) is None, "n*G is not the point at infinity" def small_key_table(c, limit): """Public points for private keys 1..limit-1, keyed by point.""" table, P = {}, None for d in range(1, limit): P = ec_add(c, P, c["g"]) table[P] = d return table Pulling the public key out of a certificate Go hashes RawSubjectPublicKeyInfo, the complete DER-encoded SubjectPublicKeyInfo, so the script has to cut out exactly those bytes. A certificate is a SEQUENCE containing the to-be-signed SEQUENCE, and the SPKI is the sixth field of that once the optional [0] version tag is skipped. A minimal DER reader is enough: def tlv(buf, i): """Read one DER tag-length-value. Returns (tag, value_start, value_end).""" tag, length = buf[i], buf[i + 1] i += 2 if length & 0x80: n = length & 0x7F length = int.from_bytes(buf[i:i + n], "big") i += n return tag, i, i + length def children(buf, start, end): out = [] while start < end: tag, vs, ve = tlv(buf, start) out.append((tag, start, vs, ve)) start = ve return out def spki_der(cert): """The raw SubjectPublicKeyInfo bytes, the thing Go hashes.""" _, vs, ve = tlv(cert, 0) _, tvs, tve = tlv(cert, vs) # tbsCertificate fields = children(cert, tvs, tve) if fields[0][0] == 0xA0: # explicit [0] version fields = fields[1:] _, start, _, end = fields[5] # serial, sigAlg, issuer, validity, subject, spki return cert[start:end] def ec_public_point(spki): """(curve name, (x, y)) for an EC key, or (None, None) for anything else.""" _, vs, ve = tlv(spki, 0) (_, _, avs, ave), (_, _, bvs, bve) = children(spki, vs, ve) params = children(spki, avs, ave) if len(params) < 2: return None, None curve_oid = spki[params[1][2]:params[1][3]] point = spki[bvs + 1:bve] # skip the unused-bits byte for name, c in CURVES.items(): if c["oid"] == curve_oid and point[0] == 4: half = (len(point) - 1) // 2 return name, (int.from_bytes(point[1:1 + half], "big"), int.from_bytes(point[1 + half:], "big")) return None, None def pem_to_der(text): m = re.search(r"-----BEGIN CERTIFICATE-----(.*?)-----END CERTIFICATE-----", text, re.S) return b64decode("".join(m.group(1).split())) if m else None Running it over CCADB The last part holds Go's eight hashes, copied from verify.go at tag go1.27.3, and walks a CCADB CSV. The three CCADB reports below are Mozilla's revoked intermediates, all intermediates, and included roots, each with a PEM column. GO_DISTRUSTED = { # crypto/x509/verify.go at go1.27.3, lines 1417-1426 "ebed6475c2c2d80fcb199f3fe8dd6e6b3ce60b77eb15d4e42c3dec0c6f5b5105", "49dd353dc2b919a3770645117d0e60c16115e7d1bc259eefe734b00ea434fe5f", "e0254c7fcdc265e2f0f8a428dabb75ab4ae039b4872d62f5ee57f0877d5a8a41", "5c0660a2b164910d8ff589c18e14552e9e0ec237dcefe2cb736d41a4811f81a0", "43bffa19d6ef651451f6841e019931e96f6f2eaeff28ab4b6f1ce4576de2717b", "63ecefee431d49523bc1136c4e72d0206a477086caf606cd5cf025686c109cf8", "ec4ffa1ad0ac88081314b7aaffef6a91f97cd57291bb5ba686c5415f5eb4f6b6", "92c063d67e8f14db71d288e11266ffdaf69cbf115f7bab14f3bc2f68d25e0e30", } LIMIT = 2**16 # private keys 1..65535: covers "below 256" with room to spare def scan(path, tables): csv.field_size_limit(sys.maxsize) rows = list(csv.DictReader(open(path, encoding="utf-8"))) pem_col = next(k for k in rows[0] if "PEM" in k) name_col = next(k for k in ("Certificate Subject Common Name", "Common Name or Certificate Name") if k in rows[0]) counts = {"rows": len(rows), "P-256": 0, "P-384": 0, "go_match": 0} hits = [] for r in rows: der = pem_to_der(r[pem_col]) if der is None: continue spki = spki_der(der) spki_hash = hashlib.sha256(spki).hexdigest() in_go = spki_hash in GO_DISTRUSTED counts["go_match"] += in_go curve, point = ec_public_point(spki) if curve is None: continue counts[curve] += 1 d = tables[curve].get(point) if d is not None or in_go: hits.append((r[name_col], r.get("Certificate Issuer Common Name", ""), curve, d, spki_hash[:16], in_go)) return counts, hits if __name__ == "__main__": tables = {} for name, c in CURVES.items(): self_test(c) tables[name] = small_key_table(c, LIMIT) print(f"self-test passed; tables hold private keys 1..{LIMIT - 1} on P-256 and P-384") for path in sys.argv[1:]: counts, hits = scan(path, tables) print(f"\n{path.split('/')[-1]}: {counts['rows']} rows, " f"{counts['P-256']} P-256 keys, {counts['P-384']} P-384 keys, " f"{counts['go_match']} certificates whose SPKI hash is in Go's list") for subj, issuer, curve, d, h, in_go in sorted(hits): size = f"{d.bit_length()}-bit key" if d else "not small" print(f" {subj:<20} issuer={issuer:<26} {curve} {size:<10} spki={h}... in_go={in_go}") curl -s -o ccadb.csv 'https://ccadb.my.salesforce-sites.com/mozilla/PublicIntermediateCertsRevokedWithPEMCSV' curl -s -O 'https://ccadb.my.salesforce-sites.com/mozilla/PublicAllIntermediateCertsWithPEMCSV' curl -s -O 'https://ccadb.my.salesforce-sites.com/mozilla/IncludedCACertificateReportPEMCSV' mv PublicAllIntermediateCertsWithPEMCSV PublicAllIntermediateCertsWithPEMCSV.csv mv IncludedCACertificateReportPEMCSV IncludedCACertificateReportPEMCSV.csv python3 tinykey_check.py ccadb.csv PublicAllIntermediateCertsWithPEMCSV.csv IncludedCACertificateReportPEMCSV.csv self-test passed; tables hold private keys 1..65535 on P-256 and P-384 ccadb.csv: 2977 rows, 114 P-256 keys, 243 P-384 keys, 16 certificates whose SPKI hash is in Go's list EM DV TLS CA - G3B issuer=AAA Certificate Services P-384 8-bit key spki=ebed6475c2c2d80f... in_go=True EM DV TLS CA - G3B issuer=emSign Root TLS CA - G3 P-384 8-bit key spki=ebed6475c2c2d80f... in_go=True EM DV TLS CA - G3C issuer=AAA Certificate Services P-384 8-bit key spki=49dd353dc2b919a3... in_go=True EM DV TLS CA - G3C issuer=emSign Root TLS CA - G3 P-384 8-bit key spki=49dd353dc2b919a3... in_go=True EM EV TLS CA - G3B issuer=AAA Certificate Services P-384 7-bit key spki=e0254c7fcdc265e2... in_go=True EM EV TLS CA - G3B issuer=emSign Root TLS CA - G3 P-384 7-bit key spki=e0254c7fcdc265e2... in_go=True EM EV TLS CA - G3C issuer=AAA Certificate Services P-384 6-bit key spki=5c0660a2b164910d... in_go=True EM EV TLS CA - G3C issuer=emSign Root TLS CA - G3 P-384 6-bit key spki=5c0660a2b164910d... in_go=True EM OV CS CA - G3B issuer=AAA Certificate Services P-384 7-bit key spki=ec4ffa1ad0ac8808... in_go=True EM OV CS CA - G3B issuer=emSign Root CS CA - G3 P-384 7-bit key spki=ec4ffa1ad0ac8808... in_go=True EM OV CS CA - G3C issuer=AAA Certificate Services P-384 8-bit key spki=92c063d67e8f14db... in_go=True EM OV CS CA - G3C issuer=emSign Root CS CA - G3 P-384 8-bit key spki=92c063d67e8f14db... in_go=True EM OV TLS CA - G3B issuer=AAA Certificate Services P-384 8-bit key spki=43bffa19d6ef6514... in_go=True EM OV TLS CA - G3B issuer=emSign Root TLS CA - G3 P-384 8-bit key spki=43bffa19d6ef6514... in_go=True EM OV TLS CA - G3C issuer=AAA Certificate Services P-384 3-bit key spki=63ecefee431d4952... in_go=True EM OV TLS CA - G3C issuer=emSign Root TLS CA - G3 P-384 3-bit key spki=63ecefee431d4952... in_go=True PublicAllIntermediateCertsWithPEMCSV.csv: 2517 rows, 420 P-256 keys, 374 P-384 keys, 0 certificates whose SPKI hash is in Go's list IncludedCACertificateReportPEMCSV.csv: 169 rows, 3 P-256 keys, 48 P-384 keys, 0 certificates whose SPKI hash is in Go's list The whole run took about four seconds. Sixteen revoked certificates carry the eight keys, because each key was certified twice: once by eMudhra's own emSign root and once by Sectigo's AAA Certificate Services cross-sign. All eight of Go's hashes are there, and every one of the eight keys is between 3 and 8 bits long. I print the bit length rather than the number. The smallest is EM OV TLS CA - G3C, whose private key is 4. That value is already public: mitmproxy's main branch has a new --emudhra option, listed under "Unreleased" in its changelog, that lets the proxy sign the certificates it generates with this key. I parsed the PEM from that commit with openssl pkey, and it is 47 zero bytes followed by 04, with an SPKI hash of 63ecefee..., the sixth entry in Go's map. The other seven values are just as easy to print. I left them out because there is no reason to copy them around. The other two reports are the control group. Among the 794 EC keys in CCADB's all-intermediates report and the 51 EC roots in Mozilla's included-roots report, nothing has a private key under 65,536. The revoked list also holds a second batch of eMudhra cross-signs that Sectigo revoked on 21 October with the reason cessationOfOperation, and none of those keys is small either. This test catches one specific failure: a key that is a small integer. A key drawn from a broken random number generator would pass it. The part that applies to your machine Revoking an intermediate only protects clients that check revocation. OpenSSL's verify does not by default, and neither does Python's ssl module. Here is the G3C intermediate from the CCADB CSV, checked against the system bundle on the Ubuntu 24.04 container I ran all of this in. The certificate linking emSign Root TLS CA - G3 to emSign ECC Root CA - G3 is not in either CCADB intermediate report I downloaded, so I took it from the same mitmproxy commit. BUNDLE=/etc/ssl/certs/ca-certificates.crt # 1. Path validation only (openssl verify does not check CRLs by default) openssl verify -CAfile "$BUNDLE" -untrusted emsign-root-tls-g3-cross.pem em-ov-tls-g3c.pem # 2. Same chain, with the issuer's CRL checked curl -s --retry 3 -A Mozilla/5.0 -o roottlscag3.crl 'http://crl.emsign.com/?roottlscag3.crl' openssl crl -inform DER -in roottlscag3.crl -out roottlscag3.pem openssl verify -CAfile "$BUNDLE" -untrusted emsign-root-tls-g3-cross.pem \ -crl_check -CRLfile roottlscag3.pem em-ov-tls-g3c.pem em-ov-tls-g3c.pem: OK C = US, O = eMudhra Inc, CN = EM OV TLS CA - G3C error 23 at 0 depth lookup: certificate revoked error em-ov-tls-g3c.pem: verification failed Path validation alone says OK. The CRL says revoked. When I ran this, the CRL host answered curl's default user agent with a 403, so the script sends a different one. Python's default context on the same container: import ssl ctx = ssl.create_default_context() crl_flags = ssl.VERIFY_CRL_CHECK_LEAF | ssl.VERIFY_CRL_CHECK_CHAIN print("CRL checking on by default:", bool(ctx.verify_flags & crl_flags)) for cert in ctx.get_ca_certs(): name = dict(rdn[0] for rdn in cert["subject"]).get("commonName", "") if name.startswith("emSign"): print("trusted root:", name) CRL checking on by default: False trusted root: emSign Root CA - C1 trusted root: emSign ECC Root CA - C3 trusted root: emSign Root CA - G1 trusted root: emSign ECC Root CA - G3 So on this container, a Python client with default settings trusted the root above the revoked intermediates and did not look at the CRL. The fix for that is the trust bundle, and different bundles took different routes. This script checks a bundle by the root's SPKI hash rather than its name: import hashlib, re, sys from tinykey_check import pem_to_der, spki_der # SPKI SHA-256 of emSign ECC Root CA - G3, as listed in rustls/webpki-roots PR 133 G3_ROOT = "8d417db2dd8bf5e3084d1e3f196d583849d81bdd4c00c70b9d39369e96b8c782" for path in sys.argv[1:]: pems = re.findall(r"-----BEGIN CERTIFICATE-----.*?-----END CERTIFICATE-----", open(path).read(), re.S) hashes = {hashlib.sha256(spki_der(pem_to_der(p))).hexdigest() for p in pems} print(f"{path}: {len(pems)} certificates, emSign ECC Root CA - G3 present: {G3_ROOT in hashes}") for v in 2026.07.22 2026.10.18 2026.10.19; do curl -s -o certifi-$v.pem https://raw.githubusercontent.com/certifi/python-certifi/$v/certifi/cacert.pem done python3 bundle_check.py /etc/ssl/certs/ca-certificates.crt certifi-2026.07.22.pem certifi-2026.10.18.pem certifi-2026.10.19.pem /etc/ssl/certs/ca-certificates.crt: 121 certificates, emSign ECC Root CA - G3 present: True certifi-2026.07.22.pem: 121 certificates, emSign ECC Root CA - G3 present: True certifi-2026.10.18.pem: 117 certificates, emSign ECC Root CA - G3 present: False certifi-2026.10.19.pem: 120 certificates, emSign ECC Root CA - G3 present: False Three different answers: - Ubuntu published ca-certificates 20260816ubuntu0.24.04.1 to noble-security on 20 October. Its changelog blacklists emSign ECC Root CA - G3 "due to intermediate cert compromise". My container still had 20260601~24.04.1 because nobody had upgraded it since. The fix existed and it was not on the machine. - certifi 2026.10.18 removed all four emSign roots. Universities whose services chain to other emSign roots reported breakage in certifi issue 439, and 2026.10.19 put three back and kept G3 out. A maintainer explained in that thread why the first release went wider than the intermediates: "there's no facility for revoking an intermediate only." - rustls webpki-roots 1.0.10 (PR 133) removed only G3 and printed its SPKI hash, 8d417db2...c782. That is the same hash bundle_check.py computes from Ubuntu's copy of the root, which is how I know the script is matching the right certificate. What I am keeping from this A root bundle is the only lever most clients have, and a machine can hold several copies: the OS package, a pinned certifi in a lockfile, a container image built before 20 October, and Go binaries built with a toolchain older than 1.26.10 or 1.27.3. Each needs its own update or rebuild. The SPKI-hash check above takes a few seconds per bundle. The small-key test is cheap enough to run on any EC public key you generate yourself, as a smoke test. It would have flagged these eight on the day of the ceremony. It says nothing about keys that are bad in subtler ways. And a valid signature tells you nothing about how the key behind it was made. Every chain here validated, and that was the problem. Written with AI assistance from my own notes and test results. I checked every fact and command before publishing.

Certificate authority keys: eMudhra's browser and runtime trust gap The eMudhra certificate authority incident has an awkward developer-facing cleanup problem. Browser vendors can block a compromised certificate chain while a deployed application keeps reading an older CA bundle. Eight intermediate signing keys were compromised, and your browser's update history cannot tell you which trust material a containerized service has loaded. The place to close the incident is the deployed client's actual validation path. {% embed youtube.com %} TL;DR - Mozilla's incident report identifies eight compromised intermediate signing keys: six for TLS and two for code signing. A compromised intermediate key gives its holder signing authority under that certificate. - Mozilla independently recovered the affected keys. The public report describes flawed key generation and says the certificates date to April 2025. Actual misuse against other clients remains unknown. - The report describes no certificate transparency entries for those intermediates during the period. Its browser-exploitation conclusion is scoped to clients enforcing certificate transparency. - Browser blocking, CA revocation and application trust-store updates are separate mechanisms. Check the CA bundle and validation configuration used by the running application. - Primary release evidence confirms certifi 2026.10.19 and Node 24.22.1 for the branch discussed in the episode. A custom trust bundle or another release branch needs its own check. What happened to the certificate authority keys? The public incident page was published on October 20. It says the original reporter, Wayne, demonstrated control of eight intermediate CA private keys by signing a fixed message with each one. The message was “We have a problem here - Wayne”. The root-program teams received the report on October 17. Mozilla's initial incident description provides independent confirmation. Mozilla verified the proof of key control and independently recovered the impacted keys without knowing Wayne's method. The report now says the keys were generated incorrectly and could be recovered from the certificates. That supports a specific conclusion: the signing keys were recoverable, and outsiders demonstrated control. It gives us no basis to invent a separate server break-in, an attacker list or a set of affected customer accounts. The incident page says actual misuse against clients outside the browser ecosystem remains unknown. Urgency can coexist with an honest account of what the evidence establishes. The eight-key figure also needs its categories. Mozilla identifies six TLS intermediates and two code-signing intermediates. The episode focuses on the TLS-client cleanup; those two categories do different jobs. The number is a count of compromised intermediate signing keys. Root certificates sit at a different layer of a client's trust decision. How an intermediate CA fits into TLS validation A TLS server presents a certificate chain. A simplified view has the server certificate, an intermediate certificate and a root that the client recognizes as trusted. Real deployments can include more intermediate levels and cross-signs. The client follows the presented chain and applies its validation rules, including the trust material available in its own environment. The intermediate's private key authorizes signatures under that intermediate certificate. Someone controlling that key can create a mathematically valid signature. The usefulness of the signature depends on the client's trust in the signer and on the other checks that the client performs. This explains why certificate verification has several moving parts. Chain building and cryptographic verification establish relationships between certificates. Trust policy decides which authorities the client accepts. Revocation or specific distrust information can withdraw acceptance after a problem is discovered. Certificate transparency adds another requirement in clients that enforce it. Those checks have different inputs and different update paths. A root bundle can be current for ordinary chain building while an intermediate-specific blocking rule arrives through a browser's emergency distribution mechanism. Another program can have a bundled store with a different maintenance schedule. Treating the word “trust” as one global switch hides the work your application actually performs. What certificate transparency tells us about exposure The public incident report dates the affected certificates to April 2025. It reports no certificate transparency entries for those intermediates during the period and concludes that CT-enforcing clients such as the named major browsers could not have been attacked through this route. That qualification belongs next to the alarming headline. A demonstrated ability to sign certificates under an intermediate is evidence of compromised authority. A successful impersonation also depends on the client's rules. For a browser enforcing transparency, a forged chain would have to satisfy that additional requirement. The same incident page points out that many other clients do not enforce certificate transparency. It leaves actual exploitation against those clients unknown. I would keep those two statements together in an incident ticket: the extra browser requirement narrows the exposure claim; other clients require their own assessment. This article relies on the report and Mozilla's confirmation. I have not independently surveyed CT logs, recovered a private key or tested every client mentioned here. There is no exploit demonstration in the episode. The distinction matters because a source's scoped conclusion can turn into a much broader claim when somebody abbreviates it for a dashboard. Certificate revocation and the runtime trust-store gap The report records Firefox and Chrome blocking action on October 17, Apple action on October 19 and Microsoft action on October 20. It also says the CA revoked the compromised intermediates. Other clients can require software updates because they do not consume the revocation information needed to reject those chains. Publishing a revocation changes the information available to validators. It helps a client only when that client receives and applies the information. A program that validates against a static bundle needs a mitigation compatible with its actual validation mechanism. Mozilla's incident thread describes emergency removal of trust in emSign ECC Root CA - G3 after discussion with downstream maintainers. That protects clients which lack revocation handling or a mechanism to withdraw trust in a specific intermediate. Removing a root from a store changes the acceptance of a chain; it does not, by itself, establish compromise of that root's private key. Even browser delivery needed checking. In comment 29, Mozilla says some Android profiles were failing to apply the updated blocklist at startup. It reports a fix shipped in Firefox 158.0.2. Vendor remediation dates describe vendor actions. Your phone still has an installed version and local state, which is why keeping the browser current remains part of the advice. What certifi and Node actually released For Python software using certifi, the 2026.10.19 release gives a concrete maintained package to inspect. Its release commit is titled “Restore unaffected emSign roots (#439) (#440)”. The diff restores unaffected roots after an earlier removal. That correction is useful operational evidence. Maintainers are trying to reject the dangerous chain while preserving legitimate services. A blanket instruction to delete every certificate carrying an eMudhra or emSign name would discard the distinctions recorded in the release history. Use the maintained update appropriate to the actual dependency and configuration. For Node, 24.22.1 shipped on October 21 as a security release. Its notable changes include “crypto: update root certificates to NSS 3.131.1”. The linked root-update PR is the primary implementation evidence for the bundled-root change. That is a verified release for the Node branch named here. It is not an inventory of every Node process you operate. Other branches need their own release advice, and a process using a custom or system CA configuration needs that configuration examined. Likewise, updating one Python environment cannot establish the dependency state of every virtual environment or container image. Check the deployed process, then test the right thing I would begin with the application that makes the connection. Identify the running environment and the trust material it actually uses. Record whether it relies on an operating-system store, a runtime bundle, a dependency such as certifi or a custom CA file. This is investigation guidance, rather than a claim that one configuration applies to every framework. Apply the relevant maintained update, rebuild the deployment when the bundle is baked into an image, and restart processes where they need to load changed trust material. Verify the deployed state afterwards. An updated workstation and an old production container can happily coexist while making different certificate decisions. The public incident page supplies test hosts and offline chains for checking clients. Keep certificate-validation failures separate from transport failures. A network timeout can come from routing, DNS, a proxy or an unavailable host. It does not prove that a validator rejected the compromised intermediate. The page diagnostic in the research browser included timeouts, so the episode makes no experimental “this browser is safe” claim from that observation. Preserve certificate verification while following the maintainers' documented checks. A successful investigation should leave you with the client's real environment, its trust configuration and a meaningful validation result. “The browser update installed” leaves the server-side question unanswered. Also in this episode: Tor and an SSH identity warning A researcher account published October 20 describes 333 compromised Tor exit relays and an earlier-October SSH interception campaign. Its roughly ten-percent figure counts relays with different capacities. It does not measure intercepted traffic share. A Tor network-health operator notice confirms reports of SSH interception on compromised hosts. The researcher noticed a changed server-identity warning. Verify such a change through a trusted channel before accepting it and logging in. Routing and endpoint authentication have separate jobs. Verdict: NEEDS REVIEW I stamped the eMudhra story NEEDS REVIEW because I want the deployed client's trust checked before the incident gets closed. Browser remediation is useful evidence for that browser. The application owns another validation decision, and somebody needs to verify it. FAQ Were eight root private keys compromised? Mozilla identifies eight intermediate signing keys, with six TLS intermediates and two code-signing intermediates. Emergency root distrust is a mitigation decision at a different layer. Does updating Chrome update certifi inside my container? A separately bundled application dependency has its own deployment path. Inspect the running environment and the bundle the client actually uses. Does a failed connection prove the chain is blocked? A certificate-validation error is meaningful for that check. A timeout alone leaves the certificate decision unobserved. Was the Tor incident happening when the article was published? The researcher describes interception beginning October 2 and relays flagged or shut down over the following days. The write-up was published October 20. Sources - Public eMudhra incident report - Mozilla incident 2083449 - certifi 2026.10.19 release - certifi restoration commit - Node 24.22.1 security release - Node root-certificate update PR - KABA's Tor researcher report - Tor operator notice --- *This article expands on an episode of The Daily Diff, a five-minute daily video on what shipped and what broke in tech. Watch the episode · Subscribe on YouTube · the written diff lands in your inbox every morning at thedailydiff.dev.* Video transcript and primary sources

@lynxluna @BukanYahya Tapi mostly di India si market-share dia, malesnya package client yang punya list trust CA harus manually update: github.com/certifi/python-certifi/commit/662df63d…