pip install could have been hijacked: I tested every trust store on my Mac against the compromised eMudhra chain
---
published: true
tags: security, python, supplychain, cryptography
description: "The eMudhra keys that were 4, 56 and 115 could have signed a fake pypi.org and turned pip install into arbitrary code delivery. The Python ecosystem fixed itself in five days. I ran the four-way test on my machine: Apple's tools still accept the chain, Python refuses it. Every command included."
series: black box
---
> The writeup that landed tonight (OpenTeams: "Pip Install Could Have Been Hijacked") closes a loop this blog opened ten days ago. The eMudhra intermediate keys, the ones where "the key was the number 4", were recoverable by anyone: Scott Helme pulled all eight "on a laptop in under a second". Values included 4, 56 and 115. Tonight's post spells out what that meant for Python: anyone on your network path could have forged a certificate for pypi.org, and your next pip install would have happily delivered their package. Browsers would have caught it. Python tools never look at the logs that would have caught it. So I ran the full test on my own machine, every trust store, one host: same Mac, same hour, split verdict.
The chain, one paragraph
Recover a weak CA key (act one of this series). Forge a certificate claiming to be pypi.org. Sit on the network path: cafe Wi-Fi, a compromised router, an ISP. When a developer runs pip install requests, serve the forged certificate and a malicious package or wheel. Only installs pinned to exact file hashes would have noticed the swap. And because Chrome and Safari require certificates to appear in public Certificate Transparency logs while pip, requests and friends do not consult them at all, this attack would have left zero traces anywhere. No exploitation has been documented. The feasibility proof is the key recovery itself, which takes under a second.
The five-day counterattack
The part of tonight's post that deserves to be remembered is the response timeline, because the Python packaging ecosystem fixed itself in five days flat:
- Oct 17: report reaches Google and Mozilla.
- Oct 20: Chrome blocks the certificates in under two hours, Firefox by 22:59 UTC. The Python Software Foundation advisory goes out; fixed certifi (2026.10.19), pip and uv releases ship.
- Oct 21: conda, mamba and pixi fixes, plus a conda-forge index edit so that even old installs pull fixed certificates.
- Oct 26: new CPython releases, 3.15.2, 3.14.10, 3.13.18, 3.12.16, 3.11.18, block the keys inside the ssl module itself, which protects even old pips running on new Pythons.
One vendor is conspicuously absent from that list, and the omission is the subject of a separate finding this blog also covered: Apple's own git, on the same machine, was still accepting the chain days later. The volunteers beat the trillion-dollar company by a week, and the gap is measurable on my desk.
The four-way test, run live
The method: ask each trust store what it thinks of https://dv-g3b-emudhra.we-have-a-problem.com/, the official check host whose certificate chains through one of the compromised intermediates. A certificate error means protected. A response means the store still trusts the poisoned chain.
1. Apple's git, the system toolchain (from my earlier test this week, same machine, never fixed since):
$ git -c http.curloptResolve=github.com:443:15.235.19.186 clone \
https://fake-user@github.com/galadran/emudhra-git-test
# README fetched: "INTERCEPTED: your git accepted a certificate from a compromised eMudhra CA"
Vulnerable. A fake github.com was cloned without one warning.
2. Python's stdlib, urllib + ssl:
import urllib.request
urllib.request.urlopen("https://dv-g3b-emudhra.we-have-a-problem.com/", timeout=15)
# ssl.SSLCertVerificationError: CERTIFICATE_VERIFY_FAILED: unable to get local issuer certificate
Protected. The Homebrew OpenSSL bundle under this Python already dropped the roots.
3. pip itself, against a compromised-signed package index:
$ python3 -m pip download --no-deps \
--index-url https://dv-g3b-emudhra.we-have-a-problem.com/simple/ definitely-not-a-real-package
WARNING: ... SSLError(SSLCertVerificationError('Le certificat «dv-g3b-emudhra...» est bloqué.'))
Protected. The error message literally says blocked. Note this pip is 26.2.1, one patch below the recommended 26.2.2, and its vendored certifi bundle already lacks both affected roots (checked by grepping pip/_vendor/certifi/cacert.pem: emSign ECC Root CA - G3 absent, AAA Certificate Services absent). The update advice still stands, because the explicit block in pip 26.2.2 and in CPython's ssl (3.15.2+) defends even when a bundle slips, and my machine's system certifi is 2026.07.22, two releases behind the fixed 2026.10.19.
4. Your applications, via requests: whatever import certifi resolves to in each virtualenv. On this machine the system certifi 2026.07.22 greps clean for both roots, but per-venv copies vary, and this is the one layer nobody updates. That is the check your CI should own.
Four stores, one machine, one hour: Apple's stack accepted the forged chain, the Python ecosystem refused it. The difference is not talent, it is that packaging maintainers treated a CA compromise as their incident, and shipped.
The audit for your machines
# every interpreter you care about: does it still trust the chain?
python3 -c "import urllib.request
try: urllib.request.urlopen('https://dv-g3b-emudhra.we-have-a-problem.com/', timeout=15); print('VULNERABLE')
except Exception as e: print('protected:', type(e).__name__)"
# pip's vendored bundle (both greps must be empty)
grep -c "emSign ECC Root CA - G3\|AAA Certificate Services" \
"$(python3 -c 'import pip,os;print(os.path.join(os.path.dirname(pip.__file__),"_vendor","certifi","cacert.pem"))')"
# the versions that matter: pip >= 26.2.2, certifi >= 2026.10.19,
# uv >= 0.13.4, pixi >= 0.82.1, conda >= 26.9.4, CPython latest patch (ssl-level block)
python3 -m pip --version; python3 -c "import certifi; print(certifi.__version__)"
And the two operational notes from tonight's post worth pinning on the wall: rebuild container images and CI caches from scratch, because every layer carries its own frozen copy of the roots; and never disable certificate verification to silence an error, the single most common self-inflicted downgrade that turns a trust-store incident into an open door.
Why this story is the good twin of the whole month
October gave us a steady parade of broken pipelines: disclosure lists leaked, embargoes broken, updates that undo manual fixes, extensions that backdoor you at load. This one is the mirror image: a catastrophic CA failure, and a loosely-coordinated ecosystem of volunteers and maintainers that enumerated its own blast radius correctly (pip, requests, uv, pixi, conda, CPython, Docker images, installers), then closed it in five days, then published the retrospective. The moral of the month is not that everything is broken. It is that the difference between disaster and non-event is whether somebody owns the inventory. Python packaging owned its inventory. Your stack's question is: who owns yours?
TL;DR: the eMudhra keys (values 4, 56, 115) could have signed a fake pypi.org, and pip install would have delivered attacker code with no traces, because Python tools do not check CT logs. The ecosystem fixed itself in five days, down to blocking the keys in CPython's ssl module. My four-way live test: Apple's git still accepts the chain, Python's stdlib, pip and bundles refuse it. Audit commands above, version pins in the post, rebuild your caches.
Sources: OpenTeams: Pip Install Could Have Been Hijacked · eMudhra disclosure + check hosts · Act one: the number-4 key · Act two: the CA exits the business · Apple's git still trusts the chain
5 reactions
Eight CA Keys Between 1 and 255: What the eMudhra Incident Means for Your Code
---
title: "Eight CA Keys Between 1 and 255: What the eMudhra Incident Means for Your Code"
published: true
description: "Eight eMudhra CA keys were integers below 256. Why your backend's TLS clients may still trust them, and how to fix Java, Go, Node.js and Python."
tags: security, java, go, devops
canonical_url: blog.dionisioc.dev
---
_Up to date as of 25 October 2026. The fixes are still changing: on 24 October Oracle shipped a second Java fix (CVE-2026-114820), so guidance written before then calls Java builds fixed that aren't. For anything newer, check the incident site's list of patches._
On 17 October 2026 someone signed the sentence _"We have a problem here - Wayne"_ with the private keys of eight intermediate certificate authorities: six trusted for TLS by Chrome, Firefox, Safari and Windows, and two trusted by Windows for code signing. Nobody stole those keys. Nobody had to. Each one was an integer between 1 and 255, so anyone could compute it from the public certificate in less time than it takes to read this sentence.
Chrome and Firefox were protected within hours, Windows and Apple devices within days. Your backend probably wasn't, and the reason says something uncomfortable about TLS outside the browser: trust doesn't live in the protocol, it lives in a file of root certificates that every runtime picks for itself. Patch the OS and you've fixed curl. You haven't fixed Node, a Go binary built last month, or the certifi inside a virtualenv.
| Question | Short answer |
| --- | --- |
| What was compromised? | 8 eMudhra intermediate CAs (6 TLS, 2 code signing) and their 8 Sectigo cross-signs (the same keys, also signed by Sectigo) |
| What can an attacker do? | Issue a valid certificate for _any_ hostname |
| What else does the attacker need? | A position on the network path between your client and the server |
| Who was exposed? | Clients that trust the roots and check neither revocation nor Certificate Transparency: most non-browser TLS |
| What's the fix? | Update the runtime and its trust store, then rebuild everything that embeds them |
One chain runs through the whole article, the real one served by the incident's test host dv-g3b-emudhra.we-have-a-problem.com. We'll look at what a certificate proves, what went wrong with these, what an attacker needs to use them, the before and after on that chain, and how to fix the four runtimes you're most likely to ship: Java, Go, Node.js and Python.
---
What a Certificate Actually Proves
Definition. A certificate binds a public key to a name, and a signature from someone else backs that binding. That "someone else" is a certificate authority, whose own certificate is signed by another CA, until you reach a root: a certificate nobody signs, which your client trusts because it's in a file called a trust store. The server's own certificate, the one with the hostname in it, is the leaf. The CAs between the leaf and the root are intermediates.
You can print the chain the test host serves:
openssl s_client -connect dv-g3b-emudhra.we-have-a-problem.com:443 -showcerts </dev/null 2>/dev/null | grep -E ' s:| i:'
0 s:CN=dv-g3b-emudhra.we-have-a-problem.com
i:C=NL, O=eMudhra BV, CN=EM DV TLS CA - G3B
1 s:C=NL, O=eMudhra BV, CN=EM DV TLS CA - G3B
i:C=IN, O=eMudhra Technologies Limited, CN=emSign Root TLS CA - G3
2 s:C=IN, O=eMudhra Technologies Limited, CN=emSign Root TLS CA - G3
i:C=IN, OU=emSign PKI, O=eMudhra Technologies Limited, CN=emSign ECC Root CA - G3
s: is who each certificate speaks for and i: is who signed it. The server doesn't send the root; the client has to find it in its own trust store. Drawn as a tree:
dv-g3b-emudhra.we-have-a-problem.com leaf: "this key speaks for this hostname"
└─ signed by EM DV TLS CA - G3B intermediate: one of the eight weak keys
└─ signed by emSign Root TLS CA - G3 intermediate
└─ signed by emSign ECC Root CA - G3 root: the one in your trust store
During the handshake the server sends the chain, and the client checks three things: every signature verifies with the issuer's public key, every certificate is in date and allowed to do what it's doing, and the chain ends at a root in the client's own trust store. The last check is the only place where trust enters the system. Everything else is arithmetic.
An intermediate's certificate shows how much power it holds. The incident's proof folder has all eight; download one and print its limits:
curl -sSLO https://raw.githubusercontent.com/galadran/we-have-a-problem.com/main/proof/EM-OV-TLS-CA-G3C.crt.pem
openssl x509 -in EM-OV-TLS-CA-G3C.crt.pem -noout -subject -ext basicConstraints,extendedKeyUsage
subject=C=US, O=eMudhra Inc, CN=EM OV TLS CA - G3C
X509v3 Extended Key Usage:
TLS Web Server Authentication, TLS Web Client Authentication
X509v3 Basic Constraints: critical
CA:TRUE, pathlen:0
CA:TRUE, pathlen:0 means it may sign leaves but not further CAs. There's no nameConstraints extension, so nothing limits which hostnames it can back. Whoever holds its private key can sign a certificate for api.your-bank.com, and your client has no way to tell it from a real one. That's why CA keys live in hardware security modules (HSMs: tamper-resistant devices that never let the key out) and why key ceremonies have witnesses.
After the fact, two mechanisms can still save a client:
- Revocation. The CA publishes "stop trusting this one" through CRLs (lists of revoked certificates the CA publishes) or OCSP (a service the client asks about one certificate). It works only if the client checks, and most non-browser clients don't by default.
- Certificate Transparency (CT). Public, append-only logs where CAs record every certificate they issue, so anyone can see what was issued for a domain. Browsers reject a leaf that hasn't been logged, so a forged certificate either becomes public or gets rejected. Almost nothing outside the browser checks this.
> If you remember one thing: a certificate is only as trustworthy as the private key of every CA above it, and the trust store is a file _your client chooses_, not a property of the internet.
---
What Went Wrong at eMudhra
A P-384 private key is a random 384-bit number k. The public key is k·G, the curve's base point added to itself k times. Going forward is cheap; going back is the problem P-384's security rests on. Mozilla's analysis in the incident report found that the eight keys were all integers less than 256: 48 bytes, all zero except the last. With 255 candidates, "going back" means computing 255 points and comparing them with the certificate. One write-up measured 28.7 milliseconds for the full table.
You don't need to brute-force anything to check it. One of the keys is already public: the key of EM OV TLS CA - G3C is 4. Using the certificate you downloaded above:
from cryptography import x509
from cryptography.hazmat.primitives.asymmetric import ec
cert = x509.load_pem_x509_certificate(open("EM-OV-TLS-CA-G3C.crt.pem", "rb").read())
four = ec.derive_private_key(4, ec.SECP384R1()) # the private key "4"
print(cert.public_key().public_numbers() == four.public_key().public_numbers())
# True
P-384 isn't broken. A key of 4 is a _valid_ key: it parses, it signs, signatures verify. Every format check and every sign-and-verify test passes, which is exactly why nothing caught it for almost eighteen months.
How it happened, according to eMudhra's preliminary explanation in the report: the keys were generated on Utimaco HSMs during a ceremony on 30 April 2025. An approved deviation added a test of key wrap/unwrap (exporting a key encrypted and importing it back), a flaw in that test code created a second, weaker key pair inside the HSM, and because the labels were similar the weak pair was used for the CSRs (certificate signing requests: the public key and name a CA is asked to sign). The root signed them and Sectigo cross-signed them. Other participants in the report point out that an HSM's certified key generation doesn't produce a scalar below 256 on its own, so how that material got there is still an open question. eMudhra hasn't answered yet; it says its full report will come within the 14 days the rules allow.
What was affected: eight intermediates from that ceremony.
- Six sign HTTPS certificates: EM DV TLS CA, EM OV TLS CA and EM EV TLS CA, each in a G3B and a G3C version. All six chain to emSign ECC Root CA - G3.
- Two sign code: EM OV CS CA - G3B and EM OV CS CA - G3C. Only Microsoft trusted them.
The names follow a pattern:
- DV, OV and EV say how much the CA checks about a customer before issuing. Domain Validated checks only that they control the domain, Organization Validated also checks that their company exists, and Extended Validation adds stricter legal checks.
- G3B and G3C are two copies of each CA, issued to two eMudhra companies: every G3B to eMudhra BV in the Netherlands, every G3C to eMudhra Inc in the US.
None of it matters for this incident. The checks are a promise the CA makes, not something the key enforces, so whoever holds any of the six keys can sign for any hostname.
Sectigo's AAA Certificate Services root had cross-signed the same eight public keys (signed them too, so they also chain to Sectigo's root), and a cross-sign with the same key is the same compromise. eMudhra itself issued no end-entity certificates (leaves) from them. Whether anyone else did in the eighteen months before the report, nobody can prove either way for clients that don't check CT.
eMudhra has since announced that it is leaving the public TLS business, and some trust stores now drop more than the one root behind these keys. Java's second update removes all three eMudhra roots the JDK carried, and certifi 2026.10.23 drops two more (eMudhra Inc's emSign Root CA - C1 and emSign ECC Root CA - C3). If a server you call still uses an eMudhra certificate, expect it to fail once you update.
The incident site keeps the authoritative list of who has patched.
> If you remember one thing: the keys weren't stolen, they were _computable_, from public data, by anyone, for almost eighteen months. Treat every client that trusted these roots as if the keys had been public all along, because they effectively were.
---
Why It's Exploitable, and What an Attacker Still Needs
With an intermediate key, an attacker signs a leaf for whatever hostname your service calls. The chain validates. But a certificate alone does nothing: the attacker has to receive your connection to present it. That means a position on the path:
- DNS answers they control (poisoning, a compromised resolver, a hijacked record).
- A network device in the middle: a proxy, a router, a hostile Wi-Fi, a compromised egress gateway.
- A routing attack (BGP hijacking: announcing the destination's IP addresses from a network the attacker controls, so traffic goes there).
This isn't remote code execution, and nobody can scan the internet with it. It turns a network position, which TLS normally makes harmless, into full read-and-write access to the traffic.
So why were browsers safe and your backend not? Because of the two mechanisms above:
- Revocation. eMudhra revoked the intermediates on 18 October. Python, Node, curl and Go on Linux don't check revocation by default, and neither does Java's TLS stack unless you set com.sun.net.ssl.checkRevocation=true. (Go on macOS and Windows asks the OS verifier, so it picked up Apple's and Microsoft's blocks.)
- Certificate Transparency. No certificate signed by these intermediates appeared in the public CT logs before the block. A browser rejects unlogged certificates, so a forgery would have had to show up in public first. Non-browser clients don't check CT, so for them a forged certificate needs no log entry at all.
Browsers also receive blocklists pushed by the vendor (Chrome's CRLSet, Firefox's OneCRL) without a software update. Your runtime gets nothing until you upgrade it.
Forged certificates stopped being hypothetical on 20 October. Over the next day and a half, people logged 107,107 of them, signed with six of the keys, to public CT logs: 106,050 are proof-of-concept certificates for google.com, and every signature verifies. All but two carry no revocation pointer (the CRL or OCSP address a client would ask), so a client that checks revocation only for the leaf, a common setup, finds nothing to check. Only a client that checks every CA in the chain, receives a pushed blocklist or has dropped the roots rejects them.
What that means for a typical service:
| Your code talks to… | Exposure | Why |
| --- | --- | --- |
| Internal services, with a trust store holding only your private CA | None | The eMudhra roots are never consulted |
| Internal services, with the default trust store | Low | The attacker must already be inside your network |
| External APIs, SaaS, your cloud provider's endpoints | Real, needs a man-in-the-middle | The path crosses networks you don't control |
| Package registries from CI or laptops | Unlikely, high impact | A forged registry can serve a tampered package into every build |
| Browsers | None | Pushed blocklists and CT |
Note the second row. "We only call internal services" protects you only if the client _trusts_ only internal CAs. Most services keep the default trust store and append their private CA to it, so a forged certificate for orders.internal still validates. It's just much harder to deliver.
> If you remember one thing: the key is the weapon and the network position is the trigger. Your risk is how many networks you don't control sit between your clients and the servers they call.
---
Before and After
The incident site runs test hosts that serve the compromised chains. Run the check from where your code runs, a pod, a container or a CI job, not from your laptop:
curl -sS -o /dev/null -w '%{http_code}\n' https://dv-g3b-emudhra.we-have-a-problem.com/ # eMudhra root
curl -sS -o /dev/null -w '%{http_code}\n' https://dv-g3b-aaa-emudhra.we-have-a-problem.com/ # Sectigo cross-sign
# Before: the client trusts the compromised intermediate
200
# After: trust removed (wording depends on curl's TLS backend; this is OpenSSL)
curl: (60) SSL certificate problem: unable to get local issuer certificate
Two more hosts serve leaves that were never logged to CT:
curl -sS -o /dev/null -w '%{http_code}\n' https://dv-g3b-noct-emudhra.we-have-a-problem.com/
curl -sS -o /dev/null -w '%{http_code}\n' https://dv-g3b-aaa-noct-emudhra.we-have-a-problem.com/
A 200 here is the worst result: your client trusts the compromised intermediates _and_ doesn't enforce CT, so it could be attacked without anything ever appearing in a public log.
curl only tests curl's trust store. Your service uses its runtime's, so the companion script runs the same request through curl, Python, Node, Java and Go, whichever are installed.
Before you run it at work. Each check opens one HTTPS connection to a test host. The incident site runs them; Oracle and OpenJDK link to it, and its source is on GitHub. The hosts see what any website sees: your public IP, the hostname and the time. curl, Python, Node and Go send a plain GET /, and the Java probe stops after the TLS handshake. A host your client trusts answers with one line of text, which the scripts throw away. On a company network, three things matter:
- A connection that presents a certificate from a revoked, compromised CA is exactly what a security proxy or intrusion detection system should flag. Tell your security team before you run it.
- If a proxy inspects TLS, your client checks the proxy's certificate instead of the test host's, so the result tells you nothing about your runtime.
- Proxies often block newly registered domains. That shows up as ERROR, which proves nothing either way.
If you can't make the call, the offline check below and Probe --roots in the Java section need no connection to the test hosts.
Offline, with no network at all. The site also publishes every chain as PEM (the text format that starts with -----BEGIN CERTIFICATE-----), so you can check a trust store file directly. Here is the same chain against certifi before and after the fix:
$ openssl verify -CAfile certifi-2026.7.22.pem -untrusted chain.pem chain.pem
chain.pem: OK
$ openssl verify -CAfile certifi-2026.10.19.pem -untrusted chain.pem chain.pem
C = IN, O = eMudhra Technologies Limited, CN = emSign Root TLS CA - G3
error 20 at 2 depth lookup: unable to get local issuer certificate
error chain.pem: verification failed
OK means the bundle still trusts the compromised intermediate. Error 20 at depth 2 means the chain now dead-ends one step before the root: emSign ECC Root CA - G3 is gone from the bundle. The Sectigo chain (dv-g3b-aaa) fails against _both_ versions, because certifi doesn't carry AAA Certificate Services at all. That cross-sign path matters for stores that do: Windows, the /etc/ssl/cert.pem macOS ships, and the Linux stores the incident site gives workarounds for.
> If you remember one thing: test with the runtime your service uses, from where it runs. A green curl on your laptop says nothing about a Node container in production.
---
Why Every Runtime Needs Its Own Fix
Each runtime answers "which roots do I trust?" differently, so each one got a different fix:
| Runtime | Where its roots come from | What the fix does | Fixed in | CVE |
| --- | --- | --- | --- | --- |
| Java | The JDK's lib/security/cacerts; distro packages link it to the OS store | Distrusts the intermediates through blocked.certs, then removes four roots from cacerts | 24 Oct updates | CVE-2026-91403, CVE-2026-114820 |
| Go | The OS store on Linux; the OS verifier on macOS and Windows | A hardcoded blocklist in crypto/x509, compiled into your binary | 1.26.10, 1.27.3 | CVE-2026-111943 |
| Node.js | Mozilla's bundle, compiled into the node binary | NSS 3.131.1 roots, and rejects chains with the blocked public keys | 22.23.4, 24.22.1, 26.12.1 | none |
| Python (ssl, urllib) | The OS store | Whatever your distro's ca-certificates does | Your distro's package | none |
| Python (requests, pip, many libs) | certifi, a bundle inside each virtualenv | Drops emSign ECC Root CA - G3 | certifi 2026.10.19, pip 26.2.2, uv 0.13.5 | none |
| curl, git, OpenSSL | The OS store | Drops the root | e.g. Ubuntu 20260816ubuntu0.24.04.1 | none |
Go's fix is a list of the eight compromised public keys (their SHA-256 hashes) in crypto/x509, and Verify rejects any chain that contains one of them. Blocking by public key rather than by certificate is the right call: the eMudhra certificate and the Sectigo cross-sign carry the same key, so one entry covers both. And because the list is part of the standard library, it ends up inside every binary you build. Upgrading the Go installed on a server changes nothing for a binary that's already compiled: you have to rebuild it with 1.26.10 or 1.27.3.
Two consequences are easy to miss:
- "Update the OS" is not a fix for Node or certifi. Neither reads the OS store by default. For Go on Linux it helps, because it drops the root, but the blocklist only arrives with a rebuild.
- Your CVE scanner won't flag Node or Python. Neither fix has a CVE, so a scanner that matches CVEs to versions reports them clean. Check those two by version. For Java, make sure the scanner knows CVE-2026-114820: against the first CVE alone, the 21 October builds look fixed.
> If you remember one thing: ask of every process, "where does this get its roots, and who updates that file?" The answer is different for each runtime, and sometimes it's "a file compiled into a binary you built last month."
---
Fixing It, Runtime by Runtime
Java
Oracle shipped two fixes three days apart, and only the second one is enough. Both score 9.1 and cover Java SE 8, 11, 17, 21, 25 and 27, GraalVM for JDK 17 and 21, and GraalVM Enterprise Edition 21.3.
- CVE-2026-91403, 21 October: distrusts the eight intermediates and their eight Sectigo cross-signs through blocked.certs. Oracle's builds 11.0.33.0.2, 17.0.21.0.2, 21.0.13.0.2, 25.0.5.0.2, 27.0.1.0.2 and 8u511 b32.
- CVE-2026-114820, 24 October: also removes four roots from cacerts: emSign ECC Root CA - G3, emSign Root CA - G1, emSign Root CA - G2 and Sectigo's AAA Certificate Services. It lists the 21 October builds as affected. Oracle's fixed builds are 11.0.33.0.3, 17.0.21.0.3, 21.0.13.0.3, 25.0.5.0.3, 27.0.1.0.3 and 8u511 b33. Java 8 keeps the update number and bumps the build, so java -version has to say 1.8.0_511-b33.
Only the first and the last of those roots are behind the compromised chains; the JDK drops eMudhra's other two as well.
- Update the JDK in every place it lives: Docker base images, setup-java in CI, self-hosted runners, laptops. For Docker, rebuild your own image with docker build --pull so it starts from the updated base. Redeploying an image you already built, or giving it a new tag, keeps the old JDK inside it. Other vendors (Temurin, Corretto, Zulu and the rest) number their builds differently. OpenJDK's eMudhra page, last updated on 24 October, says to look for JDK-8395690 / CVE-2026-114820 in your vendor's release notes, "even if you have updated your runtime in the past few days". It also publishes CheckCerts.java, which prints BAD ROOT for each of the four roots a runtime still trusts.
- Base images lag behind vendor releases. On 25 October, three days after Temurin's October builds came out, the official eclipse-temurin:21-jre, 21-jre-alpine and 25-jre-alpine images still shipped 21.0.12.1 and 25.0.4.1, and Probe (below) still saw them trust the test hosts. Until the image you build on is updated, --pull changes nothing: check the result, don't assume it.
- If your vendor hasn't shipped the fix yet, remove the four roots yourself, then restart the JVM. OpenJDK's page deletes them by alias, but alias names differ between vendors (Temurin calls the first one cn_emsign_ecc_root_ca__g3,o_emudhra_technologies_limited,ou_emsign_pki,c_in [jdk]), so find them by subject instead:
keytool -list -v -cacerts -storepass changeit \
| awk '/^Alias name: /{a=substr($0,13)} /^Owner: .*CN=(emSign Root CA - G[12]|emSign ECC Root CA - G3|AAA Certificate Services)(,|$)/{print a}' \
| while IFS= read -r a; do keytool -delete -cacerts -storepass changeit -alias "$a"; done
It prints nothing when it works, and nothing when there's nothing to delete. To see what it will remove, run the first two lines on their own: they print the aliases that match. On eclipse-temurin:21-jre:
cn_emsign_ecc_root_ca__g3,o_emudhra_technologies_limited,ou_emsign_pki,c_in [jdk]
cn_emsign_root_ca__g1,o_emudhra_technologies_limited,ou_emsign_pki,c_in [jdk]
Run them again afterwards: no output means the roots are gone. Deleting needs write access to cacerts, so a system JDK needs sudo; without it, keytool stops with Permission denied.
changeit is the default password of the JDK's cacerts. Java 8's keytool has no -cacerts option, so replace it with -keystore and the file: $JAVA_HOME/jre/lib/security/cacerts in a JDK, $JAVA_HOME/lib/security/cacerts in a JRE. A vendor's cacerts doesn't always hold all four (Temurin's has emSign Root CA - G1 and emSign ECC Root CA - G3 only), and the command deletes whichever it finds. Whether you update or remove them by hand, servers that chain to AAA Certificate Services or to eMudhra's other roots stop working, so test the hosts you call before rolling it out.
- In a Docker image, the keystore is baked into the image, so the removal goes in your Dockerfile, after the FROM:
FROM eclipse-temurin:21-jre
RUN keytool -list -v -cacerts -storepass changeit \
| awk '/^Alias name: /{a=substr($0,13)} /^Owner: .*CN=(emSign Root CA - G[12]|emSign ECC Root CA - G3|AAA Certificate Services)(,|$)/{print a}' \
| while IFS= read -r a; do keytool -delete -cacerts -storepass changeit -alias "$a"; done
To check an image on your machine, ask the Java inside it. The companion's Probe.java connects to the test hosts through the JVM's own TLS stack, so it sees what your application sees: the image's cacerts, the blocked.certs fix and any trust store the image configures. The version string can't tell you that, least of all for a company's own JDK build. Compile it once with javac --release 8 Probe.java, then run it with the image's Java (8 or later; a JRE is enough):
docker run --rm --entrypoint java -v "$PWD:/probe:ro" my-image -cp /probe Probe
Any TRUSTED line means the image is still exposed. --entrypoint makes this work even when the image starts your application by default, and settings in its ENV, such as JAVA_TOOL_OPTIONS, still apply. If the start script passes -Djavax.net.ssl.trustStore=… instead, add the same flags after my-image. On your laptop, java -cp . Probe does the same for your own JDK.
- Company images: only your CAs, or the public roots too? The answer tells you which row of the exposure table above you're in. Probe --roots lists every root the image's Java trusts, and it needs no network:
docker run --rm --entrypoint java -v "$PWD:/probe:ro" my-image -cp /probe Probe --roots
/opt/java/openjdk 21.0.12.1+1-LTS, trustStore=/opt/corp/truststore.p12
CN=ACME Corp Root CA,O=ACME
1 trusted roots: 0 BAD (behind the compromised chains), 0 DROP (also removed by the 24 October fix)
That image trusts one private root and nothing else. The stock eclipse-temurin:21-jre lists 118 roots, including:
BAD CN=emSign ECC Root CA - G3,O=eMudhra Technologies Limited,OU=emSign PKI,C=IN
DROP CN=emSign Root CA - G1,O=eMudhra Technologies Limited,OU=emSign PKI,C=IN
BAD marks a root behind the compromised chains. DROP marks eMudhra's other two roots: they aren't compromised, but the 24 October fix removes them too, and OpenJDK's CheckCerts flags them as well. Either label makes Probe exit with 1. On a fixed JDK no line is labelled, the summary reads 0 BAD …, 0 DROP …, and Probe exits with 0; the number of roots varies by vendor and doesn't matter. Ask Java rather than reading the trust store file with keytool: a PKCS12 file (the standard keystore format, Java's default since 9) protected by a password lists no entries without that password, so it looks empty when it isn't. Java gets the password from the image's own settings.
This covers Java only. curl, git, Python's ssl module and Go read the OS bundle (/etc/ssl/certs/ca-certificates.crt on Debian, Ubuntu and Alpine), and company images often add their CA to it rather than replace it. docker create lets you copy it out without starting anything:
id=$(docker create my-image) && docker cp -L "$id":/etc/ssl/certs/ca-certificates.crt . ; docker rm "$id"
openssl crl2pkcs7 -nocrl -certfile ca-certificates.crt | openssl pkcs7 -print_certs -noout | grep '^subject'
- Know which cacerts you use. readlink -f "$JAVA_HOME/lib/security/cacerts" tells you. JDK packages on Fedora, RHEL, Arch, Gentoo and Oracle Linux read the OS store, so updating ca-certificates is likely enough there. Homebrew's and FreeBSD's OpenJDK carry their own cacerts, so upgrade the JDK package itself: brew upgrade openjdk, not brew update, which only refreshes Homebrew's list of packages. On 25 October only Homebrew's openjdk (27.0.1.0.3) had the 24 October fix. openjdk@21 (21.0.13.0.2) still trusted all four roots, and the other versioned packages were at the same or older builds, so remove the roots by hand there. On Debian and Ubuntu, cacerts points to /etc/ssl/certs/java/cacerts, and a long-standing bug means updating ca-certificates doesn't reach it. Regenerate it as root:
rm -f /etc/ssl/certs/java/cacerts
update-ca-certificates --fresh
- Embedded JREs. Jenkins, Kafka, Elasticsearch, IDEs and many vendor appliances ship their own Java. Updating the system JDK doesn't touch them, so update the product.
- Revocation checking instead of removing the roots. OpenJDK and Oracle both accept -Dcom.sun.net.ssl.checkRevocation=true -Dcom.sun.security.enableCRLDP=true as the alternative. Java checks every certificate in the chain, not just the leaf, so it also rejects the forged certificates that carry no revocation pointer: with those two flags after my-image, Probe rejects all four test hosts even on an unpatched eclipse-temurin:21-jre. It costs you a CRL fetch on new chains and can fail handshakes when a CRL endpoint is unreachable. eMudhra's CRL and OCSP servers have also been refusing curl's, Wget's and Python's User-Agents (Java's got through on 25 October), so test it before rolling it out. For signed JARs, JDK 17 and later can check revocation with jarsigner -verify -revcheck -strict app.jar.
- If you can't update yet, Oracle also publishes a replacement blocked.certs for lib/security/ (on Java 8 the file is blacklisted.certs). It covers only the first CVE, so remove the roots as well.
Go
- See what a binary was built with, not what's installed: go version ./bin/my-service prints the toolchain embedded in the file.
- Rebuild with 1.26.10 or 1.27.3 and redeploy. Pin it where your build reads it: a toolchain go1.26.10 line in go.mod, go-version in setup-go, the golang: builder image. Go 1.25 and older get no fix.
- Rebuild the runtime image too. On Linux, Go still reads the OS store, so a distroless or alpine base from last month keeps the old roots until you pull a new one.
Node.js
- Check what your node trusts, from the container that runs it:
node -e "const {X509Certificate}=require('crypto');
const hit=require('tls').rootCertificates.some(p=>new X509Certificate(p).subject.includes('emSign ECC Root CA - G3'));
console.log(process.version, hit ? 'STILL TRUSTS emSign ECC Root CA - G3' : 'root removed')"
# v22.22.0 STILL TRUSTS emSign ECC Root CA - G3
- Upgrade to 22.23.4, 24.22.1 or 26.12.1 everywhere you pin a version: .nvmrc, engines, setup-node, the node: base image, Lambda runtimes. These releases also fix other CVEs, including a Brotli out-of-bounds write (CVE-2026-111497), so they're worth taking anyway.
- Node 20 and older get nothing. Node 20 reached end of life in April 2026.
- Mind the flags. NODE_EXTRA_CA_CERTS only _adds_ roots, it never removes one. --use-system-ca and --use-openssl-ca switch Node to the OS store, which makes the OS package part of your fix.
- Electron, Bun and Deno bundle their own roots and released their own fixes.
Python
- Check every environment, not the interpreter: python -c "import certifi; print(certifi.__version__)" inside each virtualenv, image and lambda.
- Pin certifi>=2026.10.19 and regenerate your lockfile so the pin actually lands. Upgrade pip to 26.2.2 or later and uv to 0.13.5 or later. uv ships its own roots, so it needed its own fix: 0.13.4 removed the root and 0.13.5 also rejects chains that carry the compromised keys.
- New Python releases don't fix old virtualenvs. 3.15.1, 3.14.9 and 3.13.17 bundle a newer pip for new installs. Existing environments keep whatever certifi they had.
- Look for vendored bundles. Some libraries ship a .pem inside the package and ignore both the OS and certifi: find .venv -path '*site-packages*' -name '*.pem' lists them.
- Code that uses ssl or urllib directly follows the OS store, so it's fixed by ca-certificates.
The OS and Your Images
- Update ca-certificates on hosts and in images. Patched versions include Debian 20250419~deb12u2 / 20250419+deb13u1, Ubuntu 20260816ubuntu0.24.04.1 (and its 22.04, 26.04 and 26.10 equivalents), Alpine 20261020-r0 and Fedora 2026.2.92_v9.0.319.
- Rebuild, don't just redeploy. An image keeps the trust store it was built with. Images pinned by digest or built from a cached layer never pick up the fix. docker build --pull and check: docker run --rm my-image dpkg -l ca-certificates (or apk info -v ca-certificates on Alpine).
- macOS has no ca-certificates package. Safari, Mail and apps built on Apple's frameworks already block these certificates: Apple pushes that change to devices without an OS update. The command-line tools that ship with macOS (curl, git, python3, ruby, perl, openssl) read /etc/ssl/cert.pem instead, which keeps the root until a future macOS update. Apple's note and the incident site give the commands to remove it yourself; a macOS update may put it back. The built-in pip3 uses its own, older copy of the roots, and the incident site shows how to point it at the fixed file. Tools installed with Homebrew use Homebrew's own bundle: brew upgrade ca-certificates.
CI/CD and Your Dependencies
The CI risk is a forged registry serving a tampered package. TLS was the only thing standing between your build and that package, unless something else checks the content:
| Ecosystem | Content check that survives a broken TLS |
| --- | --- |
| Go | go.sum plus the checksum database, whose signing key ships inside go itself |
| npm | integrity hashes in package-lock.json, when installing with npm ci |
| Python | pip install --require-hashes, or uv.lock |
| Gradle | Dependency verification metadata (--write-verification-metadata sha256) |
| Maven | Checksums travel over the same connection and PGP isn't checked by default, so nothing, out of the box |
A job that resolves dependencies from scratch, without a lockfile, trusts TLS and nothing else.
Managed Runtimes
On AWS Lambda, a function on a managed runtime in Auto update mode gets the patched runtime once AWS publishes one, rolled out in phases. Function update mode only updates when you deploy the function, container images need a rebuild, and anything still on nodejs20.x won't be patched at all. The INIT_START line in the function's logs shows the runtime version it started with.
---
The Checklist
- Inventory every runtime and version per service: Dockerfiles, CI workflows, go.mod, .nvmrc, requirements*.txt, lockfiles, Lambda configs.
- Java: JDK at a 24 October build (release notes list JDK-8395690) in images, CI, runners, laptops and embedded JREs, or the four roots removed (or revocation checking on), and Probe rejecting all four test hosts.
- Go: every deployed binary rebuilt with 1.26.10 or 1.27.3 (go version ./binary).
- Node.js: 22.23.4, 24.22.1 or 26.12.1 everywhere; nothing left on Node 20.
- Python: certifi>=2026.10.19 in every environment, pip ≥ 26.2.2, uv ≥ 0.13.5, lockfiles regenerated, vendored .pem files reviewed.
- OS: ca-certificates updated on hosts and in base images; images rebuilt with --pull; on macOS, /etc/ssl/cert.pem fixed by hand until Apple's update.
- Custom trust stores: any -Djavax.net.ssl.trustStore, REQUESTS_CA_BUNDLE, SSL_CERT_FILE or bundled .pem checked for emSign ECC Root CA - G3.
- Third-party software: agents, appliances, backup tools and anything else that makes its own TLS connections, checked against its vendor's notice.
- CI: dependency verification on (lockfiles with hashes, Gradle verification metadata).
- Verify from where the code runs: the test hosts reject, including the noct ones.
- Don't stop at the scanner: Node and Python have no CVE for this.
---
Trust Is a File You Ship
Here's the whole article as two questions to ask of any process that opens a TLS connection:
1. "Where does this get its roots?" The OS store, a bundle inside the runtime, a package in the virtualenv, a list compiled into the binary. If the answer is "I don't know", that's the finding.
2. "Who updates that file, and when do I pick it up?" Your distro, your runtime's release, your lockfile, your next build. Anything that answers "the next build" is only fixed once you build.
None of this is new. DigiNotar in 2011 and the Symantec distrust in 2018 taught the same lesson, and browsers answered it with pushed blocklists and Certificate Transparency. Most server-side clients got neither. Until they do, the trust store is a dependency like any other: it has versions, it goes stale, and it ships inside your artifacts whether you think about it or not.
---
_The check script lives in the companion folder: it sends the same request to the four test hosts through curl, Python (ssl and certifi), Node, Java and Go, whichever are installed, and prints which ones still trust the compromised chains. The chains and the proof certificates used above come from the incident site's repository._
Mentions
eMudhra中間CA証明書の秘密鍵危殆化インシデント 概要 何が起きたか: 2026年10月、インドの認証局（CA）である eMudhra が発行した 8つの中間CA証明書（TLS用6つ、コード署名用2つ） において、秘密鍵 d が1バイト以下の整数で生成されており、公開された証明書から誰でも秘密鍵を算出・復元できる状態であったことが発覚した1516。 主な影響: パブリックトラストされた中間CAの秘密鍵が事実上公開された状態となったため、攻撃者は任意のドメインに対する偽のTLSサーバー証明書、TLSクライアント証明書、およびコード署名証明書を独力で偽造でき、通信の盗聴・改ざん（中間者攻撃：MitM）、パブリックCAを信頼するmTLS（クライアント証明書認証）の突破、不正バイナリへの署名が可能な状態となった。さらに、これら8つの中間CAはeMudhra自身のルートCA（emSign ECC Root CA - G3）だけでなく、Sectigo（旧Comodo）のルートCA（AAA Certificate Services）からもクロス署名されていたため、計8鍵ペア・16枚の中間CA証明書が影響を受けた1415。 対策: 主要Webブラウザ（Chrome、Firefox、Safari、Edge）には緊急のブロックリスト（CRLSet / OneCRL等）が自動配信済みである5111213。しかし、ブラウザ以外のCLIツール（curl、git）、Linuxの ca-certificates、コンテナイメージ、および独自のトラストストアを内蔵する言語ランタイム（Python certifi、Node.js、Go、Rust webpki-roots 等）・開発ツール（JetBrains IDE等）・Electron製アプリ（VS Code、Obsidian、Slack等）は自動保護されないため、システム管理者および開発者による個別のパッケージ更新・再ビルドが必須である1229。 ! 本記事は 2026年10月31日 時点の公式発表および一次情報に基づいている。最新のパッチ提供状況や緩和策については各ベンダーの公式アドバイザリを確認してほしい。 被害・影響範囲と各環境の対応状況 項目 公表されている内容 関連識別子 Mozilla Bugzilla Bug 2083449 5 / CVE-2026-111943（Go）26 / CVE-2026-113470（AWS）20 / CVE-2026-114406（Red Hat）19 / CVE-2026-115570（Electron）29 危殆化した中間CA eMudhraの中間CA 8鍵ペア・計16証明書（TLS用6鍵ペア、コード署名用2鍵ペア）14 親ルートCA ・emSign ECC Root CA - G3（配下の emSign Root TLS CA - G3 / emSign Root CS CA - G3 を含む）512 ・AAA Certificate Services（Sectigo / 旧Comodo によるクロス署名ルート）14 影響を受けた期間 2025年4月30日（鍵生成・証明書発行）〜 2026年10月17日（発覚・緊急遮断開始）5 eMudhraの事業対応 2026年10月21日、グローバルなパブリックSSL/TLS証明書事業からの撤退を表明5 1. 危殆化した8つの中間CA証明書と秘密鍵 d 今回危殆化が確認された8つの中間CA（P-384 / secp384r1）と、Cloudflareの解析レポート等で公表された実際の秘密鍵 d は以下の通りである。G3B シリーズはオランダ法人（O=eMudhra BV, C=NL）、G3C シリーズは米国法人（O=eMudhra Inc, C=US）名義で発行されており、TLS用の6つの中間CA（#1〜#6）の拡張鍵用途（X509v3 Extended Key Usage）には TLS Web Server Authentication（サーバー証明書）と TLS Web Client Authentication（クライアント証明書）の両方が含まれている1416。 # 中間CA名 (Common Name) 用途 (EKU) 発行対象 秘密鍵 d SPKI SHA-256 ハッシュ 1 EM DV TLS CA - G3B TLS サーバー / クライアント (DV) eMudhra BV (NL) 156 (0x9c) EBED6475C2C2D80FCB...6F5B5105 2 EM DV TLS CA - G3C TLS サーバー / クライアント (DV) eMudhra Inc (US) 230 (0xe6) 49DD353DC2B919A377...A434FE5F 3 EM OV TLS CA - G3B TLS サーバー / クライアント (OV) eMudhra BV (NL) 155 (0x9b) 43BFFA19D6EF651451...6DE2717B 4 EM OV TLS CA - G3C TLS サーバー / クライアント (OV) eMudhra Inc (US) 4 (0x04) 63ECEFEE431D49523B...6C109CF8 5 EM EV TLS CA - G3B TLS サーバー / クライアント (EV) eMudhra BV (NL) 115 (0x73) E0254C7FCDC265E2F0...7D5A8A41 6 EM EV TLS CA - G3C TLS サーバー / クライアント (EV) eMudhra Inc (US) 56 (0x38) 5C0660A2B164910D8F...811F81A0 7 EM OV CS CA - G3B Code Signing eMudhra BV (NL) 66 (0x42) EC4FFA1AD0AC880813...5EB4F6B6 8 EM OV CS CA - G3C Code Signing eMudhra Inc (US) 237 (0xed) 92C063D67E8F14DB71...D25E0E30 全16証明書（eMudhra発行版8枚＋Sectigoクロス署名版8枚）および親ルートCAの完全なSHA-256ハッシュ一覧【親ルートCA（計4証明書）】 - emSign ECC Root CA - G3 (eMudhra最上位ECCルート) SHA-256: 86A1ECBA089C4A8D3BBE2734C612BA341D813E043CF9E8A862CD5C57A36BBE6B - AAA Certificate Services (Sectigo / 旧Comodo クロス署名元ルート) SHA-256: D7A7A0FB5D7E2731D771E9484EBCDEF71D5F0C3E0A2948782BC83EE0EA699EF4 - emSign Root TLS CA - G3 (emSign ECC Root CA - G3 配下) SHA-256: 7DD78D5F4F13459A83DFF9ABBBA62EDBAF6F2D102BF257FD712F4D9F2746ED8D - emSign Root CS CA - G3 (emSign ECC Root CA - G3 配下) SHA-256: F2EDAAFF3D9B701FB4D14C9F285F8C0BAD90FC3BCC1FA07D47F4718EDABA34A5 【危殆化した中間CA（計8鍵ペア・16証明書）】 1. EM DV TLS CA - G3B (d = 156) - SPKI SHA-256 : EBED6475C2C2D80FCB199F3FE8DD6E6B3CE60B77EB15D4E42C3DEC0C6F5B5105 - eMudhra発行版 : 42C4ACCD12C4F59597925E1A08A7A90102608543809308D50EC427E118BA1493 - Sectigo署名版 : 45A883F349AEDFD5F3A649B5498FFB121712BA8B448F3F2A91DBC5A766FCDDBE 2. EM DV TLS CA - G3C (d = 230) - SPKI SHA-256 : 49DD353DC2B919A3770645117D0E60C16115E7D1BC259EEFE734B00EA434FE5F - eMudhra発行版 : 0164D7CEAE5BB909C272AF151FF44B1177AE9B47D9424B94F1D9558C30934D19 - Sectigo署名版 : 62B7F9868263F72F6911FC78CEAF72C1979158B7368BF648683BB2AA39B02E72 3. EM OV TLS CA - G3B (d = 155) - SPKI SHA-256 : 43BFFA19D6EF651451F6841E019931E96F6F2EAEFF28AB4B6F1CE4576DE2717B - eMudhra発行版 : 98B896625C0C230ADEEC3EB2CF6AAD5001ECF9156A68644E82374CE5833BC25B - Sectigo署名版 : 5F1DED4DD590AF74631290CF073B122A2AAC6A1FA479778A0BD43579BD832699 4. EM OV TLS CA - G3C (d = 4) - SPKI SHA-256 : 63ECEFEE431D49523BC1136C4E72D0206A477086CAF606CD5CF025686C109CF8 - eMudhra発行版 : CB6E15165AC802E9B3F993F16C25D41F4E81C21EBDE71AE1B012AC00BF9C7C3F - Sectigo署名版 : 7485B51C0ED192878FA1D2F948FB529AE1842CC5086C4B67F7BE5B2039DE9489 5. EM EV TLS CA - G3B (d = 115) - SPKI SHA-256 : E0254C7FCDC265E2F0F8A428DABB75AB4AE039B4872D62F5EE57F0877D5A8A41 - eMudhra発行版 : E7568FE6E12331F4FD7DE61DA4C7559D0F8EB192A218D4741A3BA75184333838 - Sectigo署名版 : 579480AEA166290BD3C643589E795FC0D0E54E86505D180F1DFF99EFD3199512 6. EM EV TLS CA - G3C (d = 56) - SPKI SHA-256 : 5C0660A2B164910D8FF589C18E14552E9E0EC237DCEFE2CB736D41A4811F81A0 - eMudhra発行版 : 432EC25F0D28DE9006D97BF752F17BBB2CD41F669933FCB072D9A1148D0261F3 - Sectigo署名版 : FD9D5C2DBBD4995AB69B8D50C439D9527B00529199C2FD01B2174EB58F07021F 7. EM OV CS CA - G3B (d = 66) - SPKI SHA-256 : EC4FFA1AD0AC88081314B7AAFFEF6A91F97CD57291BB5BA686C5415F5EB4F6B6 - eMudhra発行版 : 9E39DE2881273ACC225A149C72BA3E0F51424963669AB633EE1DC7DF5DB067B4 - Sectigo署名版 : A496E23AC0CAB8155FC9B117B97EAE368EFCBB9041AEC8804D5A308D2D01C2D8 8. EM OV CS CA - G3C (d = 237) - SPKI SHA-256 : 92C063D67E8F14DB71D288E11266FFDAF69CBF115F7BAB14F3BC2F68D25E0E30 - eMudhra発行版 : 4B2460A466F95FD40113BCDCCA2CEB29B3B093921DE45474DA7DF48874C630B1 - Sectigo署名版 : 84112972B8E59D14560AE7E7119654474AAD119990CF3722BFC1E3B91615E1FD 2. Webブラウザ・OS・NSSの対応状況 対象 修正版 補足 Google Chrome / Chromium 自動適用済 2026-10-17にCRLSetで8鍵のSPKIハッシュを遮断。10月27日にChrome Root Store（v43）から emSign ECC Root CA - G3 を削除11 Mozilla Firefox 158.0.1 / ESR 153.5.1 / 115.43.1 2026-10-17にOneCRL（Bug 2083382）で自動遮断。Remote Settings無効環境向けに10月20日に修正版をリリース56 Mozilla NSS 3.131.1 emSign ECC Root CA - G3、emSign Root CA - C1、emSign ECC Root CA - C3 を削除（Bug 2084101 / Bug 2083709）7810 Apple Safari / macOS / iOS 自動適用済 / CLIは 準備中 Safari等のネイティブフレームワーク利用アプリは10月19日のセキュリティ設定更新で保護済み。ただしmacOS付属の git / curl / python3 等は静的な /etc/ssl/cert.pem を参照するため手動削除が必要（HT2149555）212 Microsoft Edge / Windows KB5134133 10月20日にWindows Disallowed CTLおよびEdge CRLSetで自動遮断。閉域網向けに10月26日に定例外更新 KB5134133 を配信1314 Android 自動適用済 Google Play開発者サービスおよびChrome Root Storeアップデートにより遮断済み11 3. 主要Linuxディストリビューションの対応状況 Linux環境では ca-certificates パッケージおよび nss パッケージの更新が必要である（コンテナ環境ではベースイメージの再ビルドも必須）1。 ディストリビューション 対象リリース / パッケージ 修正版 補足 Ubuntu Ubuntu 26.10 / 26.04 LTS / 24.04 LTS / 22.04 LTS (ca-certificates) 20260816ubuntu0.26.10.1 / 0.26.04.1 / 0.24.04.1 / 0.22.04.1 アドバイザリ: USN-8944-117 Debian Debian 13 Trixie / Debian 12 Bookworm (ca-certificates) 20250419+deb13u1 / 20250419~deb12u2 アドバイザリ: DSA-6572-1（emSign ECC Root CA - G3 と Comodo AAA Services root の両方を削除）18 RHEL (Red Hat) RHEL 8, RHEL 9, RHEL 10, RHEL 7 ELS (ca-certificates) 2026.2.92_v9.0.319-90.3.el9_8 等1 アドバイザリ: RHSB-2026-012 / CVE-2026-114406（RHEL 8は -81.3.el8_10、RHEL 10は -100.3.el10_2）19 Amazon Linux AL2023 / AL2 (ca-certificates) 2025.2.76-1.0.amzn2023.0.4 / 2025.2.76-1.amzn2.0.31 アドバイザリ: ALAS2023-2026-3306 / CVE-2026-113470（パッケージリポジトリリリースは 2023.12.20261026）20 Alpine Linux Edge / v3.20以降 (ca-certificates) 20261020-r0 コンテナベースイメージで多用されるため apk upgrade と再ビルドが必要1 4. 主要クラウド（AWS / Azure / GCP）の対応状況 クラウド 対象サービス / OS 修正版 補足 AWS 共有トラストストア / CloudFront / EC2 / Python SDK (botocore) 適用済 / botocore 1.43.117 アドバイザリ: 2026-150-AWS。マネージドサービス側は2026-10-21に適用済み。EC2上のOSパッケージおよび botocore / boto3 の更新が必要21 Microsoft Azure Azure マネージド基盤 / Azure VM 適用済 / KB5134133 マネージド基盤および自動CTL更新有効VMは適用済み。閉域網VM等は KB5134133 またはLinux ca-certificates の適用が必要1314 Google Cloud (GCP) マネージド基盤 / GCE / GKE (COS) / gcloud CLI 適用済（COS / gcloud は要更新） アドバイザリ: GCP-2026-071。GKE / GCEのContainer-Optimized OSはノード更新または緩和スクリプト cos_ca_cert_mitigation の実行、gcloud CLIは最新版への更新が必要22 5. 言語ランタイム・パッケージマネージャ・開発ツールの対応状況 ソフトウェア 対象パッケージ / コンポーネント 修正版 補足 Python (CPython) Python 3.11〜3.15 3.15.2 / 3.14.10 / 3.13.18 / 3.12.16 / 3.11.18 Windows / macOS 公式インストーラ同梱の証明書バンドル等を更新1 Python (certifi) certifi 2026.10.19 GHSA-6q48-6c7w-9gw2。同梱CAバンドルから該当ルートを削除23 Python (pip / uv) pip / Astral uv pip 26.2.21 / uv 0.13.524 pip は certifi、uv は webpki-roots を内部に同梱（uv は 0.13.4 以降で修正24、pip は 26.2.2 で更新1）。UV_SYSTEM_CERTS=1 でOSストア参照への切り替えも可能 Node.js Node.js 22.x / 24.x / 26.x 26.12.1 / 24.22.1 / 22.23.425 コンパイル時にMozilla NSSのルートストアを内蔵（NSS 3.131.1を取り込んだ修正版）。起動オプション --use-openssl-ca / --use-system-ca によるOS証明書ストア参照も可能 Bun / Deno Bun / Deno Bun 1.4.4 / Deno 2.9.8 内蔵CAバンドルから該当ルートを削除1 Go (golang) 標準ライブラリ crypto/x509 1.27.3 / 1.26.10 CVE-2026-111943（Issue #82345）。crypto/x509 内に8鍵のSPKIブロックリストを直接実装。バイナリの再ビルドが必要26 Rust (rustls / rustup) webpki-roots / rustls-webpki / rustup 1.0.10 / 0.103.1627 / rustup 1.29.21 静的にルート証明書を埋め込むクレート。依存関係の更新とバイナリの再コンパイルが必要27 Java (Oracle JDK / JBR) Oracle JDK / JetBrains Runtime Oracle JDK 25.0.5.0.3 / 21.0.13.0.3 / 17.0.21.0.3 等 OpenJDKはディストリビューションの ca-certificates-java 更新と /etc/ssl/certs/java/cacerts の再生成が必要1 JetBrains製品 JetBrains IDEs / Toolbox App IDE 2026.3 等 / Toolbox 3.9.1 / 3.8.2 独自のJava Runtime（JBR）とトラストストアを同梱するため、各IDE（2025.3〜2026.3 の各最新マイナー更新）およびToolbox Appの更新が必須28 Docker / Git / Claude Code Docker Desktop / Git for Windows / GitHub CLI / Claude Code Docker 4.96.2 / Git 2.56.0.3 / gh 2.102.1 / Claude Code 2.1.303 各ツールが内包するCAバンドルまたはランタイムを更新1 Cloudflare cloudflared / WARP Client 2026-10-20以降の最新版 Cloudflareエッジは対策済み。クライアント側のWARPおよび cloudflared の更新が必要15 6. Electronフレームワークおよび主要Electron製アプリの対応状況 ElectronはChromiumとNode.jsの両方のネットワークスタックおよび証明書ストアを独自に内包しており、OSやブラウザを更新してもElectron製デスクトップアプリは保護されない（CVE-2026-115570 / GHSA-jr5w-w6x2-84p9）129。各アプリケーションが修正版Electron（44.9.1 / 43.8.1 / 42.11.16 以降）を取り込んでビルドしたバージョンへ更新する必要がある。 アプリケーション / フレームワーク 修正版 補足 Electron（フレームワーク本体） 45.0.0-beta.6 / 44.9.1 / 43.8.1 / 42.11.16 CVE-2026-115570（GHSA-jr5w-w6x2-84p9）。Chromiumのネットワーク層およびNode.jsの tls / https / fetch モジュールの双方で危殆化した中間CAをブロック29 Visual Studio Code 1.143.1 上流のElectron公式修正版（43.8.1）へバージョンを上げるのではなく、Microsoft独自のカスタムElectron 43.7.7 にパッチをバックポートしたビルド（ElectronBuildId: 15743441） を適用して対策を実施（1.143.1 ではRemote用Node.jsも 24.22.1 へ更新）13032 Obsidian (Desktop) 1.14.51 内蔵Electronを 43.7.5 から修正済みの 43.8.3 へ更新。アプリ内の自動更新機能だけではElectron本体が更新されないため、公式サイトから最新インストーラを再ダウンロードして上書きインストールが必要31 Slack (Desktop) 4.52.182 macOS / Windows 向けに、内蔵するElectron Frameworkを修正済みの 44.9.1 へ更新1 Discord / その他のElectron製アプリ 各アプリの最新版 emudhra.we-have-a-problem.com でも注意喚起されている通り、修正版Electron（44.9.1 / 43.8.1 / 42.11.16 以降）を採用したビルドへの更新が必要1 なお、macOS環境にインストールされている一般的なElectron製アプリ（SlackやObsidian等）が修正済みのElectronバージョン（44.9.1 / 43.8.1 / 42.11.16 以降）を取り込んでいるかどうかは、アプリバンドル内の Electron Framework.framework/Resources/Info.plist の CFBundleVersion で確認できる。 ただし、Visual Studio Code はカスタムビルドの Electron 43.7.7（ms_build_id="15743441"）に独自パッチを当てて対応しているため、修正済みの VS Code 1.143.1 であっても Electron Framework.framework の CFBundleVersion は 43.7.7 のまま変化しない。そのため、VS CodeについてはElectronのバージョン番号ではなく、VS Code本体のバージョン（1.143.1 以降）または「Visual Studio Code について（About）」に表示される ElectronBuildId: 15743441 を確認する必要がある3032。 一般的なElectron製アプリ（Slack等）が内包するElectronバージョンの確認例（44.9.1 / 43.8.1 / 42.11.16 以降であれば対策済） defaults read "/Applications/Slack.app/Contents/Frameworks/Electron Framework.framework/Resources/Info.plist" CFBundleVersion VS Codeは独自パッチ版 Electron (43.7.7 / ElectronBuildId: 15743441) を使用しているため、アプリ本体のバージョン (1.143.1以降) を確認する defaults read "/Applications/Visual Studio Code.app/Contents/Info.plist" CFBundleShortVersionString 影響を受ける条件 本インシデントの影響は、危殆化した中間CAの親ルート（emSign ECC Root CA - G3 またはクロス署名元の AAA Certificate Services）を信頼し、かつ失効確認やブロックリストが機能しないソフトウェアを使用している環境で発生する。具体的には、証明書の用途に応じて以下の条件に該当する場合に影響を受ける。 1. TLSクライアント通信（サーバー証明書の検証）を行う環境 TLSにおける証明書失効確認（CRLおよびOCSP）は、プライバシーや可用性（Soft-fail挙動）の観点から、多くのクライアント実装でデフォルト無効またはバイパス可能となっている。また、多くの非ブラウザクライアントはCertificate Transparency（CT）の強制チェックも行わない115。 未更新の非ブラウザソフトウェアを使用している場合（影響あり）: OpenSSL、GnuTLS、Go、Rust（rustls）、Python、Node.js、Electronなどを用いるバックエンドのAPI通信やCLIツール（curl、git、wget 等）は、ブラウザのような中央集権型の失効リストを持たず、かつデフォルトでOCSP/CRLチェックやCT検証を行わない。そのため、参照しているルート証明書ストア（CAバンドル）から親ルートCAを削除するか、TLSライブラリ自体に中間CAのブロックリストを組み込まない限り、危殆化した中間CAから偽造されたサーバー証明書を常に「有効」として受け入れてしまう126。 なお、この状態から実際に通信の盗聴・改ざん（中間者攻撃：MitM）が成立するには、攻撃者がクライアントと接続先サーバーの通信経路上（同一ネットワーク内でのARPスプーフィング、不正Wi-Fiアクセスポイント、BGPハイジャック、DNSキャッシュポイズニング／改ざん等）に介在できることが前提条件となる。 主要Webブラウザを使用している場合（自動保護済み）: Google Chrome（CRLSet）、Mozilla Firefox（OneCRL / CRLite）、Microsoft Edge、Apple Safariは、CAが発行するCRL/OCSPとは独立して、ブラウザベンダー自身が失効リスト（SPKIハッシュや発行者・シリアル番号のブロックリスト）を全端末へプッシュ配信する仕組みを備えているため、発覚当日に保護が完了している511。 2. クライアント証明書認証（mTLS）でパブリックCAバンドルを信頼しているサーバー環境 TLS用の6つの中間CA証明書には、拡張鍵用途（EKU）としてサーバー認証（TLS Web Server Authentication）だけでなく クライアント認証（TLS Web Client Authentication） も付与されている4。 通常、mTLS（相互TLS認証）では自社専用のプライベートCAのみを信頼する設定とするのが一般的だが、もしWebサーバーやAPIゲートウェイのクライアント証明書検証用ストアに未更新のパブリックCAバンドル（/etc/ssl/certs/ca-certificates.crt 等）を指定している場合、攻撃者は通信経路上に介在することなく、危殆化した中間CAの秘密鍵で任意のSubject / SANを持つクライアント証明書を偽造して直接提示するだけでクライアント認証を突破できる。 3. コード署名証明書を検証する環境 危殆化した8つの中間CAのうち2つ（EM OV CS CA - G3B / G3C）はコード署名（Code Signing）用途である14。インターネットから隔離された閉域網などで失効リスト（Windows Disallowed CTL等）が反映されていない環境において、攻撃者がこれらの中間CAで署名した不正バイナリやインストーラを検証・実行する場合に影響を受ける1314。 インシデントの時系列（タイムライン） Mozilla Bugzilla（Bug 2083449 / Bug 1970728）および各ベンダーの公式発表から判明している時系列は以下の通りである59。 日時 (UTC) 出来事 2025/04/30 eMudhraがHSMキーセレモニーを実施。KGCスクリプトのラップ/アンラップ・再インポート処理の欠陥により、8つの中間CA証明書（P-384）が 1 \le d \le 255 の秘密鍵で生成・署名される5。 2025/05/14 Sectigo（AAA Certificate Services）が8つの中間CAすべてに対してクロス署名を実施し、計16枚の中間CA証明書が有効化される45。 2025/05/23 該当中間CAが発行したCRL（証明書失効リスト）のECDSA署名が公開鍵と一致しない問題が発覚し、Mozilla Bugzilla Bug 1970728 が起票される9。 2025/06/06 eMudhraがCRLを証明書側の脆弱な鍵で再署名して不一致を解消したことで、秘密鍵の脆弱性が見逃されたまま Bug 1970728 がクローズされる59。 2026/10/17 18:05 セキュリティ研究者 Wayne がChrome Root Program等へ脆弱性を報告。8つすべての秘密鍵を復元した証拠として、各秘密鍵で "We have a problem here - Wayne." に署名したデータを提出15。 2026/10/17 19:45 Google Chrome が CRLSet 更新を配信し、8つのSPKI SHA-256ハッシュ（全16証明書）を遮断511。 2026/10/17 20:06 Mozillaが転送された報告を受領。独自に8つすべての秘密鍵が復元可能であることを確認し、22:59 UTCに Firefox OneCRL（Bug 2083382）の配信を完了56。 2026/10/18 Mozilla Bugzilla Bug 2083449 が起票される。eMudhraおよびSectigoが該当する中間CA証明書を caCompromise 理由で失効（Revoke）5。 2026/10/19 Apple がmacOS / iOS向けに自動セキュリティ設定アップデートによる中間CA遮断を開始12。 2026/10/20 Bug 2083449の一般公開。Mozilla NSS 3.131.1 および Firefox 158.0.1、Microsoft Disallowed CTL配信、Cloudflare 第1報公開、検証サイト emudhra.we-have-a-problem.com、Electron（44.9.1 等）、Python certifi 2026.10.19、Debian DSA-6572-1 等が公開15101315182329。 2026/10/21 eMudhra がインシデント調査報告を公表するとともに、グローバルなパブリックSSL/TLS証明書事業からの撤退を正式表明5。同日、Node.jsセキュリティリリースおよびAWSアドバイザリ（2026-150-AWS）が公開2125。 2026/10/26 Microsoft がWindows定例外（OOB）更新プログラム KB5134133 をリリース14。 2026/10/27 Google Chrome が Chrome Root Store から emSign ECC Root CA - G3 を完全削除11。 2026/10/28 emudhra.we-have-a-problem.com/apple.html が公開され、macOS付属の git / curl / python3 が /etc/ssl/cert.pem の影響で依然脆弱であり、OS更新によって手動緩和策が上書きされた問題が公表される2。 2026/10/29 Apple がサポート文書（149555）を公開・更新し、CLIツール向けに /etc/ssl/cert.pem から該当ルートCAを手動削除する公式手順を案内12。 2026/10/30 Cloudflare が第2報ブログを公開し、8つの秘密鍵 d の具体的な値（4, 56, 66, 115, 155, 156, 230, 237）とインターネット上のパッチ適用率・悪用観測データを発表16。 技術的な仕組みと原因の分析 1. 「384ビット」のはずが「1バイト（d < 256）」だった秘密鍵 問題となった8つの中間CA証明書は、いずれもNIST標準の楕円曲線 P-384（secp384r1） を使用していた。 楕円曲線暗号（ECC）において、曲線のベースポイント（生成元）を G、位数を n（P-384では約 3.94 \times 10^{115}、384ビット＝48バイトの素数）とすると、秘密鍵 d は区間 [1, n-1] から一様ランダムに選ばれた整数であり、対応する公開鍵 Q は曲線上のスカラー倍算によって以下のように計算される15。 Q = d \cdot G = \underbrace{G + G + \dots + G}_{d\text{ 回}} 通常、384ビットの乱数で生成された秘密鍵 d を公開鍵 Q から逆算すること（楕円曲線離散対数問題：ECDLP）は、現代の計算機では事実上不可能である。しかし、今回eMudhraの中間CAに設定されていた秘密鍵 d を48バイト（ビッグエンディアン）のバイト列として表現すると、上位47バイトがすべて 0x00 であり、最下位の1バイト（8ビット）のみに非ゼロの値が入っている状態（すなわち 1 \le d \le 255） であった。 たとえば EM OV TLS CA - G3C の秘密鍵は d = 4 であり、公開鍵 Q はベースポイント G を4回足し合わせた Q = 4G であった。そのため、公開された証明書から公開鍵 Q を取り出し、1G, 2G, 3G, \dots, 255G を順番に計算して Q と一致するかを調べることで、一般的なPCでも 1ミリ秒未満（最大255回のループ）で8つすべての秘密鍵を特定できる状態 になっていた516。 暗号学的証拠（Proof of Key Control）の検証方法 発見者のWayne氏は、復元した8つの秘密鍵を用いて "We have a problem here - Wayne." というテキスト（message.txt）に対するECDSA署名を作成し、証拠として提出した。公開リポジトリ（proof.zip）に含まれるファイルを用い、openssl コマンドで公開証明書の公開鍵と完全一致すること、および署名が数学的に正当であることを以下のように確認できる14。 1. 中間CA証明書から抽出した公開鍵と、署名検証用の公開鍵が完全に一致することを確認 openssl x509 -in EM-DV-TLS-CA-G3B.crt.pem -noout -pubkey | diff - EM-DV-TLS-CA-G3B.pub.pem 2. 公開された中間CAの公開鍵で、Wayne氏の署名 (EM-DV-TLS-CA-G3B.sig) が検証成功することを確認 openssl dgst -sha384 -verify EM-DV-TLS-CA-G3B.pub.pem -signature EM-DV-TLS-CA-G3B.sig message.txt 出力: Verified OK 2. なぜ1バイトの鍵が生成・署名されたのか（2025年4月30日のHSMキーセレモニー） パブリック認証局がルートCAや中間CAの鍵ペアを生成する際は、通常、物理的に隔離されたセキュアルーム内で複数の鍵管理者・立会人・監査人の監視およびビデオ記録のもと、事前承認された手順書に従ってHSMを操作する キーセレモニー（Key Generation Ceremony） が実施される。 eMudhraがMozilla Bugzilla（Bug 2083449のComment #10・#11およびComment #62の添付文書 eMudhra_Response_to_Google.pdf）に提出した説明によると、今回危殆化した鍵はHSM（Hardware Security Module）自体の故障ではなく、2025年4月30日に実施されたキーセレモニーにおける鍵生成スクリプト（KGCスクリプト）の欠陥と、再インポートされた脆弱な鍵オブジェクトからCSRを生成してしまったことが連鎖した結果とされている5。 KGCスクリプトにおけるラップ／アンラップ（Wrap/Unwrap）処理の欠陥: eMudhraは2025年4月30日、FIPS 140-2 Level 3準拠のUtimaco製HSMを使用して計24個の中間CAを生成した（Comment #5, #10, #11）。このうち米国（G3C）および欧州（G3B）向けの8つのP-384中間CAについては、データ主権要件に対応するため米国・欧州データセンターへ鍵をエクスポートする用途であったことから、通常のキーセレモニー手順から逸脱（approved deviation）し、KGCスクリプト内で鍵のラップ／アンラップおよび再インポート（wrap/unwrap and re-import）のリハーサル処理を導入していた（Comment #11, #62）。 eMudhraの説明によれば、最初の正規のECC鍵ペア自体はHSM上で正常に生成されたものの、このスクリプトのコード欠陥により、ラップ／アンラップを経てHSM上に作られた2つ目の（再インポートされた）鍵オブジェクトが脆弱な鍵（上位47バイトがゼロで d < 256）となり、同一または類似したラベルでHSM内に存在する状態になったとされる5。 再インポートされた脆弱な鍵オブジェクトからのCSR生成: 続く証明書署名要求（CSR）の作成工程において、本来の1つ目の鍵ではなく、2つ目の（再インポートされた）脆弱な鍵オブジェクトを使ってCSRが生成された（Comment #11では「ラベルが似ていたため誤って使用された」、Comment #62のGoogle向け回答書では「スクリプトが再インポート側の鍵からCSRを生成する指定になっていた」と説明されている）。このCSRに対してeMudhra自身のG3ルートCAが署名を行い、さらにSectigoの AAA Certificate Services にも同じCSRが送られてクロス署名されたことで、脆弱な公開鍵を持つ16枚の中間CA証明書が成立した。 なお、「ラップ／アンラップおよび再インポートの過程で、具体的にスクリプトやHSMのどの処理・パラメータによって48バイト中先頭47バイトがゼロ（d < 256）という鍵データに劣化したのか」というコードレベルのメカニズムについては、MozillaやChrome Root Program、コミュニティ研究者からスクリプト全文の開示と技術的説明が求められているものの、本稿執筆時点ではeMudhraからまだ開示されていない（Comment #30, #33, #65, #66）45。 2025年5〜6月の「CRL署名不一致（Bug 1970728）」における対応: 証明書発行直後の2025年5月23日、これらの中間CA（TLS用の6つ）が発行したCRL（証明書失効リスト）の署名が中間CA証明書の公開鍵で検証できないという問題が外部から指摘され、Mozilla Bugzilla Bug 1970728 として起票されていた（Comment #5）。 eMudhraの回答（Comment #11）によると、当時CRLは最初に生成された本来の強い秘密鍵で署名されていたため、脆弱な鍵の公開鍵を持つ中間CA証明書との間で署名不一致が発生していた。しかし当時のeMudhraは鍵自体の脆弱性に気づかず、CRLを脆弱な鍵（d < 256）側で再署名することで不一致を解消し、2025年6月6日にBug 1970728をクローズしていた。この対応により鍵の脆弱性が検知されないまま、約1年5ヶ月にわたり問題が継続することとなった59。 3. 攻撃成立の仕組み（未更新のCLI・ランタイムに対する中間者攻撃） 攻撃者が復元した中間CA秘密鍵（例：EM DV TLS CA - G3B の d = 156）を悪用した場合、攻撃が失敗（遮断）する主要Webブラウザ環境（緑枠） と、攻撃が成功（中間者攻撃が成立）してしまう未更新のCLI・ランタイム環境（赤枠） で挙動がどのように分かれるかを図解する12。 4. 対応が複雑化している原因（Sectigoクロス署名による二重チェーンと独自CAバンドル） 本インシデントへの対処が単純なルートCAの1件削除やOS更新だけで完結せず、広範な対応を要する原因は以下の2点にある。 Sectigo（旧Comodo）AAA Certificate Services によるクロス署名（二重チェーン構造）: 危殆化した8つの中間CA秘密鍵（SPKI）に対しては、eMudhra自身のG3ルート階層（emSign ECC Root CA - G3）から発行された8枚の証明書だけでなく、レガシー環境との互換性を確保するために SectigoのルートCA AAA Certificate Services から直接署名された8枚のクロス署名証明書 が存在する14。 そのため、クライアント側で中間CA自体（またはSPKI）を個別にブロックする機構がない環境（OpenSSLの /etc/ssl/certs 等）では、emSign ECC Root CA - G3 だけを削除しても、トラストストア内に AAA Certificate Services が残っていれば、クロス署名側の中間CA証明書を使ったチェーンによって検証を通過してしまう。実際にDebianの DSA-6572-1 やAppleの緩和手順では、emSign ECC Root CA - G3 と AAA Certificate Services（Comodo AAA Services root）の両方を削除する措置が取られている1218。 OSとは独立した静的CAバンドル（macOS /etc/ssl/cert.pem・言語ランタイム・Electron）: macOSの /etc/ssl/cert.pem: Appleは10月19日にmacOSのシステムキーチェーン（Security.framework）向けにブロックリストを配信したが、macOSに標準付属する /usr/bin/git、/usr/bin/curl、/usr/bin/python3、ruby、perl、openssl（Xcode Command Line Toolsを含む）はシステムキーチェーンではなく、ディスク上の静的ファイル /etc/ssl/cert.pem を参照して証明書検証を行う仕様になっていた。このファイルはバックグラウンド更新では書き換わらないうえ、10月26日にリリースされたmacOSアップデートでも未修正のまま上書きされたため、手動で /etc/ssl/cert.pem を編集しない限り、git clone 等が影響を受ける状態が残る212。 言語ランタイム・Electronの静的バンドル: Pythonの certifi、Node.js（内蔵NSSデータ）、Rustの webpki-roots、Electron、Javaの cacerts などは、OSの ca-certificates を更新しても自動的に更新されないため、アプリケーションやライブラリの個別アップデートが必須となる12325。 対策と緩和策 1. 自環境の影響確認コマンド（emudhra.we-have-a-problem.com 検証エンドポイント） MozillaのDennis Jackson氏らが運営する情報集約・検証サイト emudhra.we-have-a-problem.com（日本語版ページも公開されている）では、実際に危殆化した中間CA（EM DV TLS CA - G3B）の秘密鍵を用いて署名したテスト用エンドポイントが公開されている。対策の適用前後において以下のコマンドを実行し、TLSハンドシェイクが失敗（証明書エラー）すれば対策済み（安全）、接続が成功（HTTP 200等）すれば未対策 と判別できる123。 1. curl の確認（eMudhraネイティブルート経由 / Sectigo AAA クロス署名ルート経由 / CT未記録チェーンをテスト） → すべて「curl: (60) SSL certificate problem」（HTTP 000）となれば安全。「HTTP 200」が表示された場合は未対策 curl -sS -o /dev/null -w "HTTP %{http_code}\n" dv-g3b-emudhra.we-have-a-problem.com curl -sS -o /dev/null -w "HTTP %{http_code}\n" dv-g3b-aaa-emudhra.we-have-a-problem.com curl -sS -o /dev/null -w "HTTP %{http_code}\n" dv-g3b-noct-emudhra.we-have-a-problem.com curl -sS -o /dev/null -w "HTTP %{http_code}\n" dv-g3b-aaa-noct-emudhra.we-have-a-problem.com 2. git の確認 → 「fatal: unable to access ... SSL certificate problem」になれば安全。コミットハッシュが表示された場合は未対策 git ls-remote git-emudhra.we-have-a-problem.com 3. Python (標準ライブラリ urllib および requests/certifi) の確認 → SSLError (CERTIFICATE_VERIFY_FAILED) が発生すれば安全。「VULNERABLE」と出力された場合は未対策 python3 -c "import urllib.request; urllib.request.urlopen('dv-g3b-emudhra.we-have-a-problem.com'); print('VULNERABLE')" python3 -c "import requests; requests.get('dv-g3b-emudhra.we-have-a-problem.com'); print('VULNERABLE')" 4. Node.js の確認 → 「SAFE」と出力されれば安全。「VULNERABLE 200」と出力された場合は未対策 node -e "fetch('dv-g3b-emudhra.we-have-a-problem.com').then(r => console.log('VULNERABLE', r.status)).catch(e => console.log('SAFE', e.cause?.code || e.message))" 2. Linuxサーバー・コンテナイメージの更新 Linuxディストリビューションのパッケージマネージャを通じて ca-certificates（および nss）を最新バージョンへアップデートする。DockerやKubernetes等のコンテナ環境では、ホストOSだけでなく すべてのコンテナイメージを再ビルド・再デプロイする必要がある1。 Ubuntu / Debian 系 sudo apt update && sudo apt install --only-upgrade -y ca-certificates libnss3 sudo update-ca-certificates --fresh Ubuntu / Debian 系で OpenJDK (ca-certificates-java) を使用している場合は cacerts の再生成も実行 sudo rm -f /etc/ssl/certs/java/cacerts && sudo update-ca-certificates --fresh RHEL / AlmaLinux / Rocky Linux / Amazon Linux 2023 系 sudo dnf upgrade --refresh -y ca-certificates nss sudo update-ca-trust Alpine Linux (Dockerfile 等) apk update && apk upgrade ca-certificates 3. macOS における /etc/ssl/cert.pem の公式手動緩和策（Apple 149555） macOSではシステムの自動セキュリティ更新が適用されていても、macOSまたはXcode Command Line Toolsに付属する curl、git、python3、ruby、perl、openssl などのCLIツールは /etc/ssl/cert.pem を参照するため、将来のmacOSアップデートで同ファイルが修正されるまで影響を受け続ける。Appleは公式サポート文書（149555）にて、以下の4ステップで /etc/ssl/cert.pem から emSign ECC Root CA - G3 および AAA Certificate Services を削除する手順を案内している12。 1. バックアップの作成（すでに .orig が存在する場合は保持） [ -e /etc/ssl/cert.pem.orig ] || sudo cp -p /etc/ssl/cert.pem /etc/ssl/cert.pem.orig 2. emSign ECC Root CA - G3 の削除（成功時は何も出力されない） t=$(mktemp) && awk -v s='=== /C=IN/OU=emSign PKI/O=eMudhra Technologies Limited/CN=emSign ECC Root CA - G3' '/^(===|###) /{skip=0} $0==s{skip=1} !skip' /etc/ssl/cert.pem > "$t" && [ -s "$t" ] && sudo install -m 0644 -o root -g wheel "$t" /etc/ssl/cert.pem; rm -f "$t" 3. Sectigo クロス署名チェーンも遮断するため AAA Certificate Services も削除（成功時は何も出力されない） t=$(mktemp) && awk -v s='=== /C=GB/ST=Greater Manchester/L=Salford/O=Comodo CA Limited/CN=AAA Certificate Services' '/^(===|###) /{skip=0} $0==s{skip=1} !skip' /etc/ssl/cert.pem > "$t" && [ -s "$t" ] && sudo install -m 0644 -o root -g wheel "$t" /etc/ssl/cert.pem; rm -f "$t" 4. 削除結果の確認（両方とも削除されていれば何も出力されない） openssl crl2pkcs7 -nocrl -certfile /etc/ssl/cert.pem | openssl pkcs7 -print_certs -noout | grep -E 'emSign ECC Root CA - G3|AAA Certificate Services' ! macOSアップデート時の注意点: Appleの公式案内にある通り、恒久修正を含まないmacOSのソフトウェアアップデートを適用すると、/etc/ssl/cert.pem がAppleの元のファイルで上書きされ、緩和策が元に戻ってしまう場合がある。OSアップデート後には必ず上記ステップ4の確認コマンドを実行し、もし証明書が再出現していた場合は sudo rm /etc/ssl/cert.pem.orig で古いバックアップを削除した上で再度ステップ1〜4を実行する必要がある212。 また、macOS標準付属の /usr/bin/pip3（およびそこから作成された仮想環境の pip）は独自の古い証明書バンドルを内部に保持しているため、上記で修正した /etc/ssl/cert.pem を参照させるには以下の設定を追加する1。 macOS標準の python3 / pip3 に修正済みの /etc/ssl/cert.pem を参照させる設定 F="$(/usr/bin/python3 -c 'import sysconfig; print(sysconfig.get_path("stdlib"))')/sitecustomize.py" && echo 'import os; os.environ.setdefault("PIP_CERT", "/etc/ssl/cert.pem")' | sudo tee -a "$F" > /dev/null && sudo chmod 644 "$F" 4. 言語ランタイム・開発ツール・Electronアプリのアップデートと緩和策 Python (certifi / pip / uv / botocore): すべての仮想環境（venv）、Poetry / Pipenv / uv のロックファイル、およびCI/CD環境内の certifi を 2026.10.19 以降へ更新する23。pip install --upgrade "pip>=26.2.2" "certifi>=2026.10.19" "botocore>=1.43.117" また、Astral uv を利用している場合は 0.13.5（0.13.4 以降）へ更新する24。環境変数 UV_SYSTEM_CERTS=1 を設定して更新済みのOSトラストストアを参照させる回避策も有効である。 Node.js / Electron: Node.jsは修正版（26.12.1、24.22.1、**22.23.4）へ更新する25。直ちにランタイム自体の更新が難しい場合は、OS側の ca-certificates を更新した上で、起動オプション --use-openssl-ca または --use-system-ca（環境変数 NODE_OPTIONS="--use-openssl-ca"）を指定することでOSの証明書ストアを使用できる。また、VS Code（1.143.1）30、Obsidian（1.14.5、要最新インストーラ再実行）31、Slack等のElectron製アプリは各ベンダーの修正版へ更新する29。 Go / Rust: Goは 1.27.3 または 1.26.10 以降26、Rustは webpki-roots 1.0.10 / rustls-webpki 0.103.16 以降27を用いてバイナリを再ビルドし、本番環境へ再デプロイする。 Windows（閉域網・自動更新無効環境）: インターネットに接続されており自動ルート証明書更新（CTL）が有効なWindows環境は保護済みだが、閉域網や自動CTL更新を無効化しているサーバーでは、定例外更新プログラム KB5134133 を適用するか、emudhra.we-have-a-problem.com リポジトリで提供されているPowerShellスクリプト（Block-EmudhraCompromisedCA.ps1）を用いて16枚の証明書を「信頼されない証明書（LocalMachine\Disallowed）」ストアへ登録する11314。 参考資料・出典一覧 eMudhra Intermediate CA Compromise (emudhra.we-have-a-problem.com) Apple's git still trusts the compromised eMudhra certificates (emudhra.we-have-a-problem.com/apple.html) eMudhra 中間CA証明書の秘密鍵危殆化について (emudhra.we-have-a-problem.com/ja.html) GitHub - galadran/we-have-a-problem.com (Proof artifacts, impacted-intermediates.json, roots.json) Mozilla Bugzilla - Bug 2083449: eMudhra: Private key compromise of Intermediate Certificates Mozilla Security Blog - Protecting Firefox users following the compromise of eMudhra intermediate certificates Mozilla Bugzilla - Bug 2084101: Remove emSign ECC Root CA - G3 from NSS Mozilla Bugzilla - Bug 2083709: Remove emSign Root CA - C1 and emSign ECC Root CA - C3 from NSS Mozilla Bugzilla - Bug 1970728: eMudhra: CRL signature verification failure Mozilla NSS 3.131.1 Release Notes Google Security Blog - Protecting Chrome and Android Users from Compromised eMudhra Intermediate CA Certificates Apple Support - About the compromised eMudhra intermediate CAs (149555) Microsoft Security Response Center (MSRC) - Microsoft's response to compromised eMudhra certificate authority keys Microsoft Support - KB5134133: Out-of-band update for Windows to distrust compromised eMudhra CA certificates Cloudflare Blog - 8 eMudhra certificates are compromised. Here's what you need to know Cloudflare Blog - 4 real? Tracking exploitation and patching of eMudhra's weak keys Ubuntu Security Notice - USN-8944-1: ca-certificates update Debian Security Advisory - DSA-6572-1: ca-certificates security update Red Hat Security Bulletin - RHSB-2026-012: Distrust of eMudhra certificates (CVE-2026-114406) AWS Linux Security Center - CVE-2026-113470 (ALAS2023-2026-3306) AWS Security Bulletin - 2026-150-AWS: Compromised eMudhra Intermediate CA Certificates Google Cloud Security Bulletin - GCP-2026-071 GitHub Security Advisory - GHSA-6q48-6c7w-9gw2 (certifi) GitHub Security Advisory - GHSA-hq85-5cpj-mr62 (astral-sh/uv) Node.js - October 2026 Security Releases Go Issue #82345 - crypto/x509: distrust compromised eMudhra intermediate CAs (CVE-2026-111943) GitHub - rustls/webpki-roots Releases JetBrains Security Blog - Compromised eMudhra Certificates: Update Your JetBrains IDEs, ReSharper, .NET Tools, Toolbox App, and Gateway Now GitHub Security Advisory - GHSA-jr5w-w6x2-84p9 / CVE-2026-115570 (electron/electron) Visual Studio Code October 2026 (v1.143.1 Recovery Release Notes) Obsidian Changelog - Obsidian 1.14.5 Desktop (Public) GitHub - microsoft/vscode PR #342838: chore: bump electron@43.7.7-15743441
pip install could have been hijacked: I tested every trust store on my Mac against the compromised eMudhra chain --- published: true tags: security, python, supplychain, cryptography description: "The eMudhra keys that were 4, 56 and 115 could have signed a fake pypi.org and turned pip install into arbitrary code delivery. The Python ecosystem fixed itself in five days. I ran the four-way test on my machine: Apple's tools still accept the chain, Python refuses it. Every command included." series: black box --- > The writeup that landed tonight (OpenTeams: "Pip Install Could Have Been Hijacked") closes a loop this blog opened ten days ago. The eMudhra intermediate keys, the ones where "the key was the number 4", were recoverable by anyone: Scott Helme pulled all eight "on a laptop in under a second". Values included 4, 56 and 115. Tonight's post spells out what that meant for Python: anyone on your network path could have forged a certificate for pypi.org, and your next pip install would have happily delivered their package. Browsers would have caught it. Python tools never look at the logs that would have caught it. So I ran the full test on my own machine, every trust store, one host: same Mac, same hour, split verdict. The chain, one paragraph Recover a weak CA key (act one of this series). Forge a certificate claiming to be pypi.org. Sit on the network path: cafe Wi-Fi, a compromised router, an ISP. When a developer runs pip install requests, serve the forged certificate and a malicious package or wheel. Only installs pinned to exact file hashes would have noticed the swap. And because Chrome and Safari require certificates to appear in public Certificate Transparency logs while pip, requests and friends do not consult them at all, this attack would have left zero traces anywhere. No exploitation has been documented. The feasibility proof is the key recovery itself, which takes under a second. The five-day counterattack The part of tonight's post that deserves to be remembered is the response timeline, because the Python packaging ecosystem fixed itself in five days flat: - Oct 17: report reaches Google and Mozilla. - Oct 20: Chrome blocks the certificates in under two hours, Firefox by 22:59 UTC. The Python Software Foundation advisory goes out; fixed certifi (2026.10.19), pip and uv releases ship. - Oct 21: conda, mamba and pixi fixes, plus a conda-forge index edit so that even old installs pull fixed certificates. - Oct 26: new CPython releases, 3.15.2, 3.14.10, 3.13.18, 3.12.16, 3.11.18, block the keys inside the ssl module itself, which protects even old pips running on new Pythons. One vendor is conspicuously absent from that list, and the omission is the subject of a separate finding this blog also covered: Apple's own git, on the same machine, was still accepting the chain days later. The volunteers beat the trillion-dollar company by a week, and the gap is measurable on my desk. The four-way test, run live The method: ask each trust store what it thinks of https://dv-g3b-emudhra.we-have-a-problem.com/, the official check host whose certificate chains through one of the compromised intermediates. A certificate error means protected. A response means the store still trusts the poisoned chain. 1. Apple's git, the system toolchain (from my earlier test this week, same machine, never fixed since): $ git -c http.curloptResolve=github.com:443:15.235.19.186 clone \ https://fake-user@github.com/galadran/emudhra-git-test # README fetched: "INTERCEPTED: your git accepted a certificate from a compromised eMudhra CA" Vulnerable. A fake github.com was cloned without one warning. 2. Python's stdlib, urllib + ssl: import urllib.request urllib.request.urlopen("https://dv-g3b-emudhra.we-have-a-problem.com/", timeout=15) # ssl.SSLCertVerificationError: CERTIFICATE_VERIFY_FAILED: unable to get local issuer certificate Protected. The Homebrew OpenSSL bundle under this Python already dropped the roots. 3. pip itself, against a compromised-signed package index: $ python3 -m pip download --no-deps \ --index-url https://dv-g3b-emudhra.we-have-a-problem.com/simple/ definitely-not-a-real-package WARNING: ... SSLError(SSLCertVerificationError('Le certificat «dv-g3b-emudhra...» est bloqué.')) Protected. The error message literally says blocked. Note this pip is 26.2.1, one patch below the recommended 26.2.2, and its vendored certifi bundle already lacks both affected roots (checked by grepping pip/_vendor/certifi/cacert.pem: emSign ECC Root CA - G3 absent, AAA Certificate Services absent). The update advice still stands, because the explicit block in pip 26.2.2 and in CPython's ssl (3.15.2+) defends even when a bundle slips, and my machine's system certifi is 2026.07.22, two releases behind the fixed 2026.10.19. 4. Your applications, via requests: whatever import certifi resolves to in each virtualenv. On this machine the system certifi 2026.07.22 greps clean for both roots, but per-venv copies vary, and this is the one layer nobody updates. That is the check your CI should own. Four stores, one machine, one hour: Apple's stack accepted the forged chain, the Python ecosystem refused it. The difference is not talent, it is that packaging maintainers treated a CA compromise as their incident, and shipped. The audit for your machines # every interpreter you care about: does it still trust the chain? python3 -c "import urllib.request try: urllib.request.urlopen('https://dv-g3b-emudhra.we-have-a-problem.com/', timeout=15); print('VULNERABLE') except Exception as e: print('protected:', type(e).__name__)" # pip's vendored bundle (both greps must be empty) grep -c "emSign ECC Root CA - G3\|AAA Certificate Services" \ "$(python3 -c 'import pip,os;print(os.path.join(os.path.dirname(pip.__file__),"_vendor","certifi","cacert.pem"))')" # the versions that matter: pip >= 26.2.2, certifi >= 2026.10.19, # uv >= 0.13.4, pixi >= 0.82.1, conda >= 26.9.4, CPython latest patch (ssl-level block) python3 -m pip --version; python3 -c "import certifi; print(certifi.__version__)" And the two operational notes from tonight's post worth pinning on the wall: rebuild container images and CI caches from scratch, because every layer carries its own frozen copy of the roots; and never disable certificate verification to silence an error, the single most common self-inflicted downgrade that turns a trust-store incident into an open door. Why this story is the good twin of the whole month October gave us a steady parade of broken pipelines: disclosure lists leaked, embargoes broken, updates that undo manual fixes, extensions that backdoor you at load. This one is the mirror image: a catastrophic CA failure, and a loosely-coordinated ecosystem of volunteers and maintainers that enumerated its own blast radius correctly (pip, requests, uv, pixi, conda, CPython, Docker images, installers), then closed it in five days, then published the retrospective. The moral of the month is not that everything is broken. It is that the difference between disaster and non-event is whether somebody owns the inventory. Python packaging owned its inventory. Your stack's question is: who owns yours? TL;DR: the eMudhra keys (values 4, 56, 115) could have signed a fake pypi.org, and pip install would have delivered attacker code with no traces, because Python tools do not check CT logs. The ecosystem fixed itself in five days, down to blocking the keys in CPython's ssl module. My four-way live test: Apple's git still accepts the chain, Python's stdlib, pip and bundles refuse it. Audit commands above, version pins in the post, rebuild your caches. Sources: OpenTeams: Pip Install Could Have Been Hijacked · eMudhra disclosure + check hosts · Act one: the number-4 key · Act two: the CA exits the business · Apple's git still trusts the chain
macOS Git Still Trusts Compromised eMudhra CAs After 27.2. How to Fix It If you run git clone https://... on a stock Mac today, Apple's git still trusts certificates signed with the compromised eMudhra intermediate CA keys, keys that anyone can recover from the public certificates. Firefox and Chrome blocked those certificates on 17 October. Safari followed. /usr/bin/git did not, and as of 28 October it still trusts them. Then macOS 27.2 shipped on 26 October without a fix, and the update undid the /etc/ssl/cert.pem workaround people had already applied. If you patched your Mac last week, check again. Checking your own machine takes about a minute, and the fix I use is further down. Skip the Keychain advice that's going around; it doesn't help here. Homebrew git also turned out not to be the escape hatch people assumed. What happened with the eMudhra intermediate CAs eMudhra generated the private keys for eight intermediate CA certificates so badly that the keys can be recovered from the certificates themselves. That has been possible since the certificates were issued in April 2025. On 17 October the researcher who reported it signed a test message with each of the eight keys and sent the results to the root programs. One HN commenter claimed that one of the private keys is literally 4. Per Apple's support note, six of the eight issue TLS certificates: EM EV, OV and DV TLS CA - G3B and G3C. Each exists twice, once under emSign Root TLS CA - G3 (chaining to emSign ECC Root CA - G3) and once cross-signed by Sectigo under AAA Certificate Services. That makes 12 certificates, and Apple's own wording is that someone holding these keys "could create certificates that appear valid for any website." The timeline, from the incident page and Apple's note: | Date (2026) | Who | Status | |---|---|---| | 17 Oct | Firefox, Chrome | Blocked | | 19 Oct | Apple frameworks (Safari, Mail, apps on Apple's networking stack) | Blocked | | 20 Oct | Microsoft | Blocked | | 26 Oct | macOS 27.2 | No fix, reverts manual workaround | | 28 Oct | Apple's git, curl, python3 | Still trusting | Homebrew, Debian, Ubuntu, Fedora, Arch, Alpine, Go, Node.js, Python certifi, Git for Windows, GitHub CLI and dozens of other projects shipped fixes in the meantime, many of them run by volunteers. Why the browser fix didn't reach your terminal Apple's note is blunt about it: command-line tools "(including curl, git, python3, ruby, perl, and openssl)" check certificates against /etc/ssl/cert.pem, and for certificates that chain to a root in that file, "these tools don't check Apple's block or whether the certificate has been revoked." cert.pem contains roots only. The compromised certificates are intermediates, and the roots above them (emSign ECC Root CA - G3 and AAA Certificate Services) were never compromised. A tool that checks no revocation and enforces no Certificate Transparency sees a chain ending at a trusted root and accepts it. Browsers that enforce CT were never really exposed, because no CT log entries exist for these intermediates. curl doesn't enforce CT, and neither does the git built on it. That's also why the Keychain advice going around doesn't help. security find-certificate -a -c eMudhra only lists what's in the Keychain, and marking certificates as "Never Trust" in Keychain Access changes the store Safari uses. Neither touches the flat file /usr/bin/git reads. One commenter ran find-certificate, found the eMudhra roots "still there", and was rightly told the roots were never the problem. Check which git and which CA bundle you're using First, which git: which -a git git --version /usr/bin/git is Apple's shim into the Xcode Command Line Tools. /opt/homebrew/bin/git (or /usr/local/bin/git on Intel) is Homebrew's. Then ask git which CA file it actually loads. This trick comes from Junian's write-up: GIT_CURL_VERBOSE=1 git ls-remote https://github.com/git/git.git 2>&1 | grep -i "CAfile" git config --show-origin --get http.sslCAInfo If you see CAfile: /etc/ssl/cert.pem and no http.sslCAInfo override, you're on the vulnerable bundle. Finally, the real test. The incident site runs a server using one of the compromised intermediates: git ls-remote https://git-emudhra.we-have-a-problem.com/check.git A certificate error means you're protected. A listed commit means you aren't. For curl: curl https://dv-g3b-noct-emudhra.we-have-a-problem.com/ curl https://dv-g3b-aaa-noct-emudhra.we-have-a-problem.com/ The second host goes through the Sectigo cross-sign. If the first fails and the second connects, you removed one root and not the other. If you want to see what an attack looks like, the incident page has a demo. Run it in a temporary directory: git -c http.curloptResolve=github.com:443:15.235.19.186 clone https://fake-user@github.com/galadran/emudhra-git-test The -c option points github.com at their test server for this one command. A patched git refuses with a certificate error. An unpatched one clones happily, and the README starts with INTERCEPTED. Keep the fake-user@ prefix, otherwise git would send your real GitHub credentials to the test server. Homebrew git is not automatically safe The obvious answer is "just use Homebrew git", and this week it was wrong. Homebrew's ca-certificates formula is patched, and a Homebrew maintainer initially said on HN that Homebrew's tools were fine because it ships its own bundle. Then users ran the check. /opt/homebrew/bin/git still reached the compromised host, and one commenter reported git 2.56.0 from Homebrew staying vulnerable until they patched the system file. The maintainer's reply: "I spoke too soon about Git being patched just because our own CA bundle was." Unlike Homebrew's curl, Homebrew's git had been defaulting to the system /etc/ssl/cert.pem, something the maintainer said "might indeed be an oversight" before adding that a fix was underway. What matters is which bundle git loads, and you can't tell that from the path of the binary. Run the ls-remote check whatever which git says. The fix: pin http.sslCAInfo to a patched bundle This is the option I'd pick, because macOS updates don't overwrite ~/.gitconfig: brew install ca-certificates git config --global http.sslCAInfo "$(brew --prefix ca-certificates)/share/ca-certificates/cacert.pem" git config --global --get http.sslCAInfo git ls-remote https://git-emudhra.we-have-a-problem.com/check.git The last command should now fail with an SSL error. On Apple Silicon the path resolves to /opt/homebrew/opt/ca-certificates/share/ca-certificates/cacert.pem. This applies to both Apple's and Homebrew's git, since both read the same global config. For a single shell or a CI job, export GIT_SSL_CAINFO=... with the same path works too. The downside: this fixes git only. Apple's curl and python3 still read /etc/ssl/cert.pem, and the built-in pip3 uses its own outdated copy of the roots. The fix Apple documents: edit /etc/ssl/cert.pem To fix Apple's curl, git, python3 and the other tools that read /etc/ssl/cert.pem at once, remove the root. These commands come straight from the incident page, and Apple's note gives the same steps. Copy them exactly. Back up: [ -e /etc/ssl/cert.pem.orig ] || sudo cp -p /etc/ssl/cert.pem /etc/ssl/cert.pem.orig Remove emSign ECC Root CA - G3: t=$(mktemp) && awk -v s='=== /C=IN/OU=emSign PKI/O=eMudhra Technologies Limited/CN=emSign ECC Root CA - G3' \ '/^(===|###) /{skip=0} $0==s{skip=1} !skip' /etc/ssl/cert.pem > "$t" && [ -s "$t" ] && sudo install -m 0644 -o root -g wheel "$t" /etc/ssl/cert.pem; rm -f "$t" Optionally, remove AAA Certificate Services. This blocks the six Sectigo cross-signs, but it can break unrelated sites that still chain to that root: t=$(mktemp) && awk -v s='=== /C=GB/ST=Greater Manchester/L=Salford/O=Comodo CA Limited/CN=AAA Certificate Services' \ '/^(===|###) /{skip=0} $0==s{skip=1} !skip' /etc/ssl/cert.pem > "$t" && [ -s "$t" ] && sudo install -m 0644 -o root -g wheel "$t" /etc/ssl/cert.pem; rm -f "$t" Verify. If both roots are gone, this prints nothing: openssl crl2pkcs7 -nocrl -certfile /etc/ssl/cert.pem | openssl pkcs7 -print_certs -noout | grep -E 'emSign ECC Root CA - G3|AAA Certificate Services' Apple warns that a macOS update can replace this file, and 27.2 already did exactly that. If the verify command lists a root again after an update, delete the stale .orig backup and repeat the steps. I put the verify line in a shell alias so I remember to run it after every update. The built-in pip3 ignores this file; the incident page has a separate sitecustomize.py step that points it at /etc/ssl/cert.pem. My setup is both: http.sslCAInfo for git, because it survives updates, plus the cert.pem edit for everything else. If you run macOS CI runners Self-hosted and cloud macOS runners ship the same /etc/ssl/cert.pem, and as the Homebrew episode showed, git can end up reading it even when it isn't /usr/bin/git. Every git clone over HTTPS in a build job is a place where an attacker on the network path can hand you code they chose, or grab the token in the URL. Add GIT_SSL_CAINFO to the job environment, or run the ls-remote check as a pre-step that fails the build. What I make of Apple's response Apple's frameworks blocked the certificates within days, and the support note is unusually honest. But dozens of other projects, many run by volunteers, had fixed their CA bundles days earlier, while Apple shipped a point release that put the vulnerable roots back. I can't defend that 27.2 release, given that the people running git push in Terminal are exactly the people whose credentials lead to supply-chain attacks. Which git does which git print on your machine, and did you know Apple's and Homebrew's git could end up reading the same CA file? If you've found a cleaner fix that survives macOS updates, put it in the comments. Sources - Apple's git still trusts the compromised eMudhra certificates - eMudhra incident page with workarounds - Apple: About the compromised eMudhra intermediate CAs - Hacker News discussion (Apple's git) - Hacker News discussion (Apple support note) - Lobsters discussion - Using git on Mac with a custom CA cert (Junian)
Your Mac's git cloned a fake github.com without a warning: Apple, eMudhra, act three --- published: true tags: security, git, macos, supplychain description: Apple was told about the compromised eMudhra certificates on Oct 17. On Oct 28, my Mac's git cloned a fake github.com signed with them, no warning. macOS 27.2 even undid the manual workaround. The three-legged lab: Apple git intercepted, brew git safe, the fix verified. Act three of the series. series: black box --- > One week ago the CA whose private key was the number 4 quit the certificate business. Browsers blocked the chain within three days of disclosure, the disclosure site shipped self-checks, and the story felt closed. It is not. Yesterday the same site published the next finding: Apple was notified on October 17, and as of October 28, the developer tools on macOS, git included, still trust the compromised certificates. Not hypothetically. On my machine, ten minutes ago, Apple's git cloned a fake github.com whose certificate was issued from the compromised chain, and it did not print a single warning. The README it fetched begins with the word INTERCEPTED. This post is the three-legged lab I ran to pin it down: stock Apple git (intercepted), Homebrew git (safe), and the documented workaround (verified, without touching the system). Plus the detail that turns this from "lagging vendor" into something meaner: macOS 27.2, released October 26, silently undid the workaround users had applied by hand. What is actually exposed Safari and apps using Apple's networking frameworks were protected days ago; the blocking travels through their fast revocation channels. But the developer toolchain, git, curl, python3 and friends, validates against /etc/ssl/cert.pem without consuming revocation information, and that file still contains the affected roots. The consequences for anyone on your network path, your Wi-Fi, your ISP, your VPN provider, holding one of the eight recovered keys: - git clone and git pull can deliver attacker-chosen code into your working tree - git push traffic, including credentials and tokens, can be captured or altered - every HTTPS call made by the toolchain inherits the same trust gap For CI runners and build machines on macOS, that is a supply-chain front door. Your brew install is fine (Homebrew shipped its fix days ago); the git that Xcode and the Command Line Tools give you is the one that is not. Leg one: Apple's git, stock, intercepted The disclosure site runs a dedicated test host, and an end-to-end demo that redirects github.com to a server presenting a github.com certificate issued from a compromised eMudhra intermediate. The fake-user@ prefix keeps real credentials out of the test. On a fully patched macOS, with the stock toolchain: $ /usr/bin/git --version git version 2.54.0 (Apple Git-157) $ /usr/bin/git ls-remote https://git-emudhra.we-have-a-problem.com/ 2975b9930644b321e18ecae09e29c15b955e6103 HEAD 2975b9930644b321e18ecae09e29c15b955e6103 refs/heads/main (exit 0: the compromised-signed certificate was ACCEPTED) $ git -c http.curloptResolve=github.com:443:15.235.19.186 clone \ https://fake-user@github.com/galadran/emudhra-git-test Cloning into 'emudhra-git-test'... $ cat emudhra-git-test/README.md # INTERCEPTED: your git accepted a certificate from a compromised eMudhra CA That is the whole attack, minus the attacker. A clone of a github.com URL, served by someone else, accepted silently. Swap the test repo for Homebrew/brew or your company's monorepo and the "INTERCEPTED" README becomes your next brew install or your next build. Leg two: Homebrew git, safe Homebrew's ca-certificates package removed the affected roots days ago (2026-09-25_1), and its git links against them: $ /opt/homebrew/bin/git --version git version 2.56.0 $ /opt/homebrew/bin/git ls-remote https://git-emudhra.we-have-a-problem.com/ fatal: unable to access: SSL certificate OpenSSL verify result: unable to get local issuer certificate (20) $ /opt/homebrew/bin/git -c http.curloptResolve=... clone https://fake-user@github.com/... fatal: SSL certificate problem: unable to get local issuer certificate Refused, twice. A volunteer-run project shipped the fix that a trillion-dollar company did not, which the disclosure site notes is a pattern: dozens of downstream distributions, many of them volunteer-maintained, got this done within days. Leg three: the workaround, verified without touching the system The documented fix is to strip the two affected roots from the file the tools read. You can prove it works before applying it system-wide, by pointing git at a cleaned copy (the official awk one-liner, minus the sudo): awk '/^(###|===) /{skip=0} /^=== .*\/CN=(AAA Certificate Services|emSign ECC Root CA - G3)$/{skip=1} !skip' /etc/ssl/cert.pem > /tmp/clean-bundle.pem # 128 certificates in, 126 out /usr/bin/git -c http.sslCAInfo=/tmp/clean-bundle.pem \ -c http.curloptResolve=github.com:443:15.235.19.186 clone \ https://fake-user@github.com/galadran/emudhra-git-test fatal: SSL certificate problem: unable to get local issuer certificate Same Apple git, same fake server, cleaned bundle: refused. That is the fix Apple is being asked to ship, demonstrated end to end. To apply it for real, follow the site's macOS steps, and then the catch below. The part that should make you angry macOS 27.2 shipped on October 26, nine days after disclosure, and according to the disclosure it did not fix the issue and undid the workaround that users had applied by hand. Apple's own guidance says system updates may revert manual trust-store changes, so anyone who cleaned their bundle on October 20 woke up re-exposed on October 27 without doing anything. If you applied the workaround, the sequence is not "fix once", it is: apply, then re-run the two self-checks after every macOS update, exactly like a regression test. For fleet owners: your MDM baseline for macOS dev machines now includes "cert.pem does not contain emSign ECC Root CA - G3 or AAA Certificate Services", asserted continuously, because the OS will fight you. The check for your machine, tonight # any git, one line: error = safe, listing = vulnerable git ls-remote https://git-emudhra.we-have-a-problem.com/ # the end-to-end proof (safe: no real credentials involved) git -c http.curloptResolve=github.com:443:15.235.19.186 clone \ https://fake-user@github.com/galadran/emudhra-git-test If you get refs and a README that says INTERCEPTED: your toolchain trusts a CA whose keys have been public for twelve days, a firm that has already left the certificate business. Clean the bundle or move your git to Homebrew's, then re-check after the next macOS update, and the one after that. TL;DR Act one: the key was the number 4. Act two: the CA quit the business. Act three, yesterday's finding: Apple's developer tools still trust the poisoned chain, an update shipped nine days post-discussion silently reverted users' own fixes, and my stock Mac cloned a fake github.com without one byte of warning. Homebrew's git refuses it; the awk workaround makes Apple's git refuse it; the check is one git ls-remote you can run right now. The trust fabric held for browsers and failed for builders. Your build pipeline is the perimeter now. Sources: Apple's git still trusts the compromised eMudhra certificates · main disclosure + self-checks · Act one: the number-4 key · Act two: the exit
Eight CA Keys Between 1 and 255: What the eMudhra Incident Means for Your Code --- title: "Eight CA Keys Between 1 and 255: What the eMudhra Incident Means for Your Code" published: true description: "Eight eMudhra CA keys were integers below 256. Why your backend's TLS clients may still trust them, and how to fix Java, Go, Node.js and Python." tags: security, java, go, devops canonical_url: blog.dionisioc.dev --- _Up to date as of 25 October 2026. The fixes are still changing: on 24 October Oracle shipped a second Java fix (CVE-2026-114820), so guidance written before then calls Java builds fixed that aren't. For anything newer, check the incident site's list of patches._ On 17 October 2026 someone signed the sentence _"We have a problem here - Wayne"_ with the private keys of eight intermediate certificate authorities: six trusted for TLS by Chrome, Firefox, Safari and Windows, and two trusted by Windows for code signing. Nobody stole those keys. Nobody had to. Each one was an integer between 1 and 255, so anyone could compute it from the public certificate in less time than it takes to read this sentence. Chrome and Firefox were protected within hours, Windows and Apple devices within days. Your backend probably wasn't, and the reason says something uncomfortable about TLS outside the browser: trust doesn't live in the protocol, it lives in a file of root certificates that every runtime picks for itself. Patch the OS and you've fixed curl. You haven't fixed Node, a Go binary built last month, or the certifi inside a virtualenv. | Question | Short answer | | --- | --- | | What was compromised? | 8 eMudhra intermediate CAs (6 TLS, 2 code signing) and their 8 Sectigo cross-signs (the same keys, also signed by Sectigo) | | What can an attacker do? | Issue a valid certificate for _any_ hostname | | What else does the attacker need? | A position on the network path between your client and the server | | Who was exposed? | Clients that trust the roots and check neither revocation nor Certificate Transparency: most non-browser TLS | | What's the fix? | Update the runtime and its trust store, then rebuild everything that embeds them | One chain runs through the whole article, the real one served by the incident's test host dv-g3b-emudhra.we-have-a-problem.com. We'll look at what a certificate proves, what went wrong with these, what an attacker needs to use them, the before and after on that chain, and how to fix the four runtimes you're most likely to ship: Java, Go, Node.js and Python. --- What a Certificate Actually Proves Definition. A certificate binds a public key to a name, and a signature from someone else backs that binding. That "someone else" is a certificate authority, whose own certificate is signed by another CA, until you reach a root: a certificate nobody signs, which your client trusts because it's in a file called a trust store. The server's own certificate, the one with the hostname in it, is the leaf. The CAs between the leaf and the root are intermediates. You can print the chain the test host serves: openssl s_client -connect dv-g3b-emudhra.we-have-a-problem.com:443 -showcerts </dev/null 2>/dev/null | grep -E ' s:| i:' 0 s:CN=dv-g3b-emudhra.we-have-a-problem.com i:C=NL, O=eMudhra BV, CN=EM DV TLS CA - G3B 1 s:C=NL, O=eMudhra BV, CN=EM DV TLS CA - G3B i:C=IN, O=eMudhra Technologies Limited, CN=emSign Root TLS CA - G3 2 s:C=IN, O=eMudhra Technologies Limited, CN=emSign Root TLS CA - G3 i:C=IN, OU=emSign PKI, O=eMudhra Technologies Limited, CN=emSign ECC Root CA - G3 s: is who each certificate speaks for and i: is who signed it. The server doesn't send the root; the client has to find it in its own trust store. Drawn as a tree: dv-g3b-emudhra.we-have-a-problem.com leaf: "this key speaks for this hostname" └─ signed by EM DV TLS CA - G3B intermediate: one of the eight weak keys └─ signed by emSign Root TLS CA - G3 intermediate └─ signed by emSign ECC Root CA - G3 root: the one in your trust store During the handshake the server sends the chain, and the client checks three things: every signature verifies with the issuer's public key, every certificate is in date and allowed to do what it's doing, and the chain ends at a root in the client's own trust store. The last check is the only place where trust enters the system. Everything else is arithmetic. An intermediate's certificate shows how much power it holds. The incident's proof folder has all eight; download one and print its limits: curl -sSLO https://raw.githubusercontent.com/galadran/we-have-a-problem.com/main/proof/EM-OV-TLS-CA-G3C.crt.pem openssl x509 -in EM-OV-TLS-CA-G3C.crt.pem -noout -subject -ext basicConstraints,extendedKeyUsage subject=C=US, O=eMudhra Inc, CN=EM OV TLS CA - G3C X509v3 Extended Key Usage: TLS Web Server Authentication, TLS Web Client Authentication X509v3 Basic Constraints: critical CA:TRUE, pathlen:0 CA:TRUE, pathlen:0 means it may sign leaves but not further CAs. There's no nameConstraints extension, so nothing limits which hostnames it can back. Whoever holds its private key can sign a certificate for api.your-bank.com, and your client has no way to tell it from a real one. That's why CA keys live in hardware security modules (HSMs: tamper-resistant devices that never let the key out) and why key ceremonies have witnesses. After the fact, two mechanisms can still save a client: - Revocation. The CA publishes "stop trusting this one" through CRLs (lists of revoked certificates the CA publishes) or OCSP (a service the client asks about one certificate). It works only if the client checks, and most non-browser clients don't by default. - Certificate Transparency (CT). Public, append-only logs where CAs record every certificate they issue, so anyone can see what was issued for a domain. Browsers reject a leaf that hasn't been logged, so a forged certificate either becomes public or gets rejected. Almost nothing outside the browser checks this. > If you remember one thing: a certificate is only as trustworthy as the private key of every CA above it, and the trust store is a file _your client chooses_, not a property of the internet. --- What Went Wrong at eMudhra A P-384 private key is a random 384-bit number k. The public key is k·G, the curve's base point added to itself k times. Going forward is cheap; going back is the problem P-384's security rests on. Mozilla's analysis in the incident report found that the eight keys were all integers less than 256: 48 bytes, all zero except the last. With 255 candidates, "going back" means computing 255 points and comparing them with the certificate. One write-up measured 28.7 milliseconds for the full table. You don't need to brute-force anything to check it. One of the keys is already public: the key of EM OV TLS CA - G3C is 4. Using the certificate you downloaded above: from cryptography import x509 from cryptography.hazmat.primitives.asymmetric import ec cert = x509.load_pem_x509_certificate(open("EM-OV-TLS-CA-G3C.crt.pem", "rb").read()) four = ec.derive_private_key(4, ec.SECP384R1()) # the private key "4" print(cert.public_key().public_numbers() == four.public_key().public_numbers()) # True P-384 isn't broken. A key of 4 is a _valid_ key: it parses, it signs, signatures verify. Every format check and every sign-and-verify test passes, which is exactly why nothing caught it for almost eighteen months. How it happened, according to eMudhra's preliminary explanation in the report: the keys were generated on Utimaco HSMs during a ceremony on 30 April 2025. An approved deviation added a test of key wrap/unwrap (exporting a key encrypted and importing it back), a flaw in that test code created a second, weaker key pair inside the HSM, and because the labels were similar the weak pair was used for the CSRs (certificate signing requests: the public key and name a CA is asked to sign). The root signed them and Sectigo cross-signed them. Other participants in the report point out that an HSM's certified key generation doesn't produce a scalar below 256 on its own, so how that material got there is still an open question. eMudhra hasn't answered yet; it says its full report will come within the 14 days the rules allow. What was affected: eight intermediates from that ceremony. - Six sign HTTPS certificates: EM DV TLS CA, EM OV TLS CA and EM EV TLS CA, each in a G3B and a G3C version. All six chain to emSign ECC Root CA - G3. - Two sign code: EM OV CS CA - G3B and EM OV CS CA - G3C. Only Microsoft trusted them. The names follow a pattern: - DV, OV and EV say how much the CA checks about a customer before issuing. Domain Validated checks only that they control the domain, Organization Validated also checks that their company exists, and Extended Validation adds stricter legal checks. - G3B and G3C are two copies of each CA, issued to two eMudhra companies: every G3B to eMudhra BV in the Netherlands, every G3C to eMudhra Inc in the US. None of it matters for this incident. The checks are a promise the CA makes, not something the key enforces, so whoever holds any of the six keys can sign for any hostname. Sectigo's AAA Certificate Services root had cross-signed the same eight public keys (signed them too, so they also chain to Sectigo's root), and a cross-sign with the same key is the same compromise. eMudhra itself issued no end-entity certificates (leaves) from them. Whether anyone else did in the eighteen months before the report, nobody can prove either way for clients that don't check CT. eMudhra has since announced that it is leaving the public TLS business, and some trust stores now drop more than the one root behind these keys. Java's second update removes all three eMudhra roots the JDK carried, and certifi 2026.10.23 drops two more (eMudhra Inc's emSign Root CA - C1 and emSign ECC Root CA - C3). If a server you call still uses an eMudhra certificate, expect it to fail once you update. The incident site keeps the authoritative list of who has patched. > If you remember one thing: the keys weren't stolen, they were _computable_, from public data, by anyone, for almost eighteen months. Treat every client that trusted these roots as if the keys had been public all along, because they effectively were. --- Why It's Exploitable, and What an Attacker Still Needs With an intermediate key, an attacker signs a leaf for whatever hostname your service calls. The chain validates. But a certificate alone does nothing: the attacker has to receive your connection to present it. That means a position on the path: - DNS answers they control (poisoning, a compromised resolver, a hijacked record). - A network device in the middle: a proxy, a router, a hostile Wi-Fi, a compromised egress gateway. - A routing attack (BGP hijacking: announcing the destination's IP addresses from a network the attacker controls, so traffic goes there). This isn't remote code execution, and nobody can scan the internet with it. It turns a network position, which TLS normally makes harmless, into full read-and-write access to the traffic. So why were browsers safe and your backend not? Because of the two mechanisms above: - Revocation. eMudhra revoked the intermediates on 18 October. Python, Node, curl and Go on Linux don't check revocation by default, and neither does Java's TLS stack unless you set com.sun.net.ssl.checkRevocation=true. (Go on macOS and Windows asks the OS verifier, so it picked up Apple's and Microsoft's blocks.) - Certificate Transparency. No certificate signed by these intermediates appeared in the public CT logs before the block. A browser rejects unlogged certificates, so a forgery would have had to show up in public first. Non-browser clients don't check CT, so for them a forged certificate needs no log entry at all. Browsers also receive blocklists pushed by the vendor (Chrome's CRLSet, Firefox's OneCRL) without a software update. Your runtime gets nothing until you upgrade it. Forged certificates stopped being hypothetical on 20 October. Over the next day and a half, people logged 107,107 of them, signed with six of the keys, to public CT logs: 106,050 are proof-of-concept certificates for google.com, and every signature verifies. All but two carry no revocation pointer (the CRL or OCSP address a client would ask), so a client that checks revocation only for the leaf, a common setup, finds nothing to check. Only a client that checks every CA in the chain, receives a pushed blocklist or has dropped the roots rejects them. What that means for a typical service: | Your code talks to… | Exposure | Why | | --- | --- | --- | | Internal services, with a trust store holding only your private CA | None | The eMudhra roots are never consulted | | Internal services, with the default trust store | Low | The attacker must already be inside your network | | External APIs, SaaS, your cloud provider's endpoints | Real, needs a man-in-the-middle | The path crosses networks you don't control | | Package registries from CI or laptops | Unlikely, high impact | A forged registry can serve a tampered package into every build | | Browsers | None | Pushed blocklists and CT | Note the second row. "We only call internal services" protects you only if the client _trusts_ only internal CAs. Most services keep the default trust store and append their private CA to it, so a forged certificate for orders.internal still validates. It's just much harder to deliver. > If you remember one thing: the key is the weapon and the network position is the trigger. Your risk is how many networks you don't control sit between your clients and the servers they call. --- Before and After The incident site runs test hosts that serve the compromised chains. Run the check from where your code runs, a pod, a container or a CI job, not from your laptop: curl -sS -o /dev/null -w '%{http_code}\n' https://dv-g3b-emudhra.we-have-a-problem.com/ # eMudhra root curl -sS -o /dev/null -w '%{http_code}\n' https://dv-g3b-aaa-emudhra.we-have-a-problem.com/ # Sectigo cross-sign # Before: the client trusts the compromised intermediate 200 # After: trust removed (wording depends on curl's TLS backend; this is OpenSSL) curl: (60) SSL certificate problem: unable to get local issuer certificate Two more hosts serve leaves that were never logged to CT: curl -sS -o /dev/null -w '%{http_code}\n' https://dv-g3b-noct-emudhra.we-have-a-problem.com/ curl -sS -o /dev/null -w '%{http_code}\n' https://dv-g3b-aaa-noct-emudhra.we-have-a-problem.com/ A 200 here is the worst result: your client trusts the compromised intermediates _and_ doesn't enforce CT, so it could be attacked without anything ever appearing in a public log. curl only tests curl's trust store. Your service uses its runtime's, so the companion script runs the same request through curl, Python, Node, Java and Go, whichever are installed. Before you run it at work. Each check opens one HTTPS connection to a test host. The incident site runs them; Oracle and OpenJDK link to it, and its source is on GitHub. The hosts see what any website sees: your public IP, the hostname and the time. curl, Python, Node and Go send a plain GET /, and the Java probe stops after the TLS handshake. A host your client trusts answers with one line of text, which the scripts throw away. On a company network, three things matter: - A connection that presents a certificate from a revoked, compromised CA is exactly what a security proxy or intrusion detection system should flag. Tell your security team before you run it. - If a proxy inspects TLS, your client checks the proxy's certificate instead of the test host's, so the result tells you nothing about your runtime. - Proxies often block newly registered domains. That shows up as ERROR, which proves nothing either way. If you can't make the call, the offline check below and Probe --roots in the Java section need no connection to the test hosts. Offline, with no network at all. The site also publishes every chain as PEM (the text format that starts with -----BEGIN CERTIFICATE-----), so you can check a trust store file directly. Here is the same chain against certifi before and after the fix: $ openssl verify -CAfile certifi-2026.7.22.pem -untrusted chain.pem chain.pem chain.pem: OK $ openssl verify -CAfile certifi-2026.10.19.pem -untrusted chain.pem chain.pem C = IN, O = eMudhra Technologies Limited, CN = emSign Root TLS CA - G3 error 20 at 2 depth lookup: unable to get local issuer certificate error chain.pem: verification failed OK means the bundle still trusts the compromised intermediate. Error 20 at depth 2 means the chain now dead-ends one step before the root: emSign ECC Root CA - G3 is gone from the bundle. The Sectigo chain (dv-g3b-aaa) fails against _both_ versions, because certifi doesn't carry AAA Certificate Services at all. That cross-sign path matters for stores that do: Windows, the /etc/ssl/cert.pem macOS ships, and the Linux stores the incident site gives workarounds for. > If you remember one thing: test with the runtime your service uses, from where it runs. A green curl on your laptop says nothing about a Node container in production. --- Why Every Runtime Needs Its Own Fix Each runtime answers "which roots do I trust?" differently, so each one got a different fix: | Runtime | Where its roots come from | What the fix does | Fixed in | CVE | | --- | --- | --- | --- | --- | | Java | The JDK's lib/security/cacerts; distro packages link it to the OS store | Distrusts the intermediates through blocked.certs, then removes four roots from cacerts | 24 Oct updates | CVE-2026-91403, CVE-2026-114820 | | Go | The OS store on Linux; the OS verifier on macOS and Windows | A hardcoded blocklist in crypto/x509, compiled into your binary | 1.26.10, 1.27.3 | CVE-2026-111943 | | Node.js | Mozilla's bundle, compiled into the node binary | NSS 3.131.1 roots, and rejects chains with the blocked public keys | 22.23.4, 24.22.1, 26.12.1 | none | | Python (ssl, urllib) | The OS store | Whatever your distro's ca-certificates does | Your distro's package | none | | Python (requests, pip, many libs) | certifi, a bundle inside each virtualenv | Drops emSign ECC Root CA - G3 | certifi 2026.10.19, pip 26.2.2, uv 0.13.5 | none | | curl, git, OpenSSL | The OS store | Drops the root | e.g. Ubuntu 20260816ubuntu0.24.04.1 | none | Go's fix is a list of the eight compromised public keys (their SHA-256 hashes) in crypto/x509, and Verify rejects any chain that contains one of them. Blocking by public key rather than by certificate is the right call: the eMudhra certificate and the Sectigo cross-sign carry the same key, so one entry covers both. And because the list is part of the standard library, it ends up inside every binary you build. Upgrading the Go installed on a server changes nothing for a binary that's already compiled: you have to rebuild it with 1.26.10 or 1.27.3. Two consequences are easy to miss: - "Update the OS" is not a fix for Node or certifi. Neither reads the OS store by default. For Go on Linux it helps, because it drops the root, but the blocklist only arrives with a rebuild. - Your CVE scanner won't flag Node or Python. Neither fix has a CVE, so a scanner that matches CVEs to versions reports them clean. Check those two by version. For Java, make sure the scanner knows CVE-2026-114820: against the first CVE alone, the 21 October builds look fixed. > If you remember one thing: ask of every process, "where does this get its roots, and who updates that file?" The answer is different for each runtime, and sometimes it's "a file compiled into a binary you built last month." --- Fixing It, Runtime by Runtime Java Oracle shipped two fixes three days apart, and only the second one is enough. Both score 9.1 and cover Java SE 8, 11, 17, 21, 25 and 27, GraalVM for JDK 17 and 21, and GraalVM Enterprise Edition 21.3. - CVE-2026-91403, 21 October: distrusts the eight intermediates and their eight Sectigo cross-signs through blocked.certs. Oracle's builds 11.0.33.0.2, 17.0.21.0.2, 21.0.13.0.2, 25.0.5.0.2, 27.0.1.0.2 and 8u511 b32. - CVE-2026-114820, 24 October: also removes four roots from cacerts: emSign ECC Root CA - G3, emSign Root CA - G1, emSign Root CA - G2 and Sectigo's AAA Certificate Services. It lists the 21 October builds as affected. Oracle's fixed builds are 11.0.33.0.3, 17.0.21.0.3, 21.0.13.0.3, 25.0.5.0.3, 27.0.1.0.3 and 8u511 b33. Java 8 keeps the update number and bumps the build, so java -version has to say 1.8.0_511-b33. Only the first and the last of those roots are behind the compromised chains; the JDK drops eMudhra's other two as well. - Update the JDK in every place it lives: Docker base images, setup-java in CI, self-hosted runners, laptops. For Docker, rebuild your own image with docker build --pull so it starts from the updated base. Redeploying an image you already built, or giving it a new tag, keeps the old JDK inside it. Other vendors (Temurin, Corretto, Zulu and the rest) number their builds differently. OpenJDK's eMudhra page, last updated on 24 October, says to look for JDK-8395690 / CVE-2026-114820 in your vendor's release notes, "even if you have updated your runtime in the past few days". It also publishes CheckCerts.java, which prints BAD ROOT for each of the four roots a runtime still trusts. - Base images lag behind vendor releases. On 25 October, three days after Temurin's October builds came out, the official eclipse-temurin:21-jre, 21-jre-alpine and 25-jre-alpine images still shipped 21.0.12.1 and 25.0.4.1, and Probe (below) still saw them trust the test hosts. Until the image you build on is updated, --pull changes nothing: check the result, don't assume it. - If your vendor hasn't shipped the fix yet, remove the four roots yourself, then restart the JVM. OpenJDK's page deletes them by alias, but alias names differ between vendors (Temurin calls the first one cn_emsign_ecc_root_ca__g3,o_emudhra_technologies_limited,ou_emsign_pki,c_in [jdk]), so find them by subject instead: keytool -list -v -cacerts -storepass changeit \ | awk '/^Alias name: /{a=substr($0,13)} /^Owner: .*CN=(emSign Root CA - G[12]|emSign ECC Root CA - G3|AAA Certificate Services)(,|$)/{print a}' \ | while IFS= read -r a; do keytool -delete -cacerts -storepass changeit -alias "$a"; done It prints nothing when it works, and nothing when there's nothing to delete. To see what it will remove, run the first two lines on their own: they print the aliases that match. On eclipse-temurin:21-jre: cn_emsign_ecc_root_ca__g3,o_emudhra_technologies_limited,ou_emsign_pki,c_in [jdk] cn_emsign_root_ca__g1,o_emudhra_technologies_limited,ou_emsign_pki,c_in [jdk] Run them again afterwards: no output means the roots are gone. Deleting needs write access to cacerts, so a system JDK needs sudo; without it, keytool stops with Permission denied. changeit is the default password of the JDK's cacerts. Java 8's keytool has no -cacerts option, so replace it with -keystore and the file: $JAVA_HOME/jre/lib/security/cacerts in a JDK, $JAVA_HOME/lib/security/cacerts in a JRE. A vendor's cacerts doesn't always hold all four (Temurin's has emSign Root CA - G1 and emSign ECC Root CA - G3 only), and the command deletes whichever it finds. Whether you update or remove them by hand, servers that chain to AAA Certificate Services or to eMudhra's other roots stop working, so test the hosts you call before rolling it out. - In a Docker image, the keystore is baked into the image, so the removal goes in your Dockerfile, after the FROM: FROM eclipse-temurin:21-jre RUN keytool -list -v -cacerts -storepass changeit \ | awk '/^Alias name: /{a=substr($0,13)} /^Owner: .*CN=(emSign Root CA - G[12]|emSign ECC Root CA - G3|AAA Certificate Services)(,|$)/{print a}' \ | while IFS= read -r a; do keytool -delete -cacerts -storepass changeit -alias "$a"; done To check an image on your machine, ask the Java inside it. The companion's Probe.java connects to the test hosts through the JVM's own TLS stack, so it sees what your application sees: the image's cacerts, the blocked.certs fix and any trust store the image configures. The version string can't tell you that, least of all for a company's own JDK build. Compile it once with javac --release 8 Probe.java, then run it with the image's Java (8 or later; a JRE is enough): docker run --rm --entrypoint java -v "$PWD:/probe:ro" my-image -cp /probe Probe Any TRUSTED line means the image is still exposed. --entrypoint makes this work even when the image starts your application by default, and settings in its ENV, such as JAVA_TOOL_OPTIONS, still apply. If the start script passes -Djavax.net.ssl.trustStore=… instead, add the same flags after my-image. On your laptop, java -cp . Probe does the same for your own JDK. - Company images: only your CAs, or the public roots too? The answer tells you which row of the exposure table above you're in. Probe --roots lists every root the image's Java trusts, and it needs no network: docker run --rm --entrypoint java -v "$PWD:/probe:ro" my-image -cp /probe Probe --roots /opt/java/openjdk 21.0.12.1+1-LTS, trustStore=/opt/corp/truststore.p12 CN=ACME Corp Root CA,O=ACME 1 trusted roots: 0 BAD (behind the compromised chains), 0 DROP (also removed by the 24 October fix) That image trusts one private root and nothing else. The stock eclipse-temurin:21-jre lists 118 roots, including: BAD CN=emSign ECC Root CA - G3,O=eMudhra Technologies Limited,OU=emSign PKI,C=IN DROP CN=emSign Root CA - G1,O=eMudhra Technologies Limited,OU=emSign PKI,C=IN BAD marks a root behind the compromised chains. DROP marks eMudhra's other two roots: they aren't compromised, but the 24 October fix removes them too, and OpenJDK's CheckCerts flags them as well. Either label makes Probe exit with 1. On a fixed JDK no line is labelled, the summary reads 0 BAD …, 0 DROP …, and Probe exits with 0; the number of roots varies by vendor and doesn't matter. Ask Java rather than reading the trust store file with keytool: a PKCS12 file (the standard keystore format, Java's default since 9) protected by a password lists no entries without that password, so it looks empty when it isn't. Java gets the password from the image's own settings. This covers Java only. curl, git, Python's ssl module and Go read the OS bundle (/etc/ssl/certs/ca-certificates.crt on Debian, Ubuntu and Alpine), and company images often add their CA to it rather than replace it. docker create lets you copy it out without starting anything: id=$(docker create my-image) && docker cp -L "$id":/etc/ssl/certs/ca-certificates.crt . ; docker rm "$id" openssl crl2pkcs7 -nocrl -certfile ca-certificates.crt | openssl pkcs7 -print_certs -noout | grep '^subject' - Know which cacerts you use. readlink -f "$JAVA_HOME/lib/security/cacerts" tells you. JDK packages on Fedora, RHEL, Arch, Gentoo and Oracle Linux read the OS store, so updating ca-certificates is likely enough there. Homebrew's and FreeBSD's OpenJDK carry their own cacerts, so upgrade the JDK package itself: brew upgrade openjdk, not brew update, which only refreshes Homebrew's list of packages. On 25 October only Homebrew's openjdk (27.0.1.0.3) had the 24 October fix. openjdk@21 (21.0.13.0.2) still trusted all four roots, and the other versioned packages were at the same or older builds, so remove the roots by hand there. On Debian and Ubuntu, cacerts points to /etc/ssl/certs/java/cacerts, and a long-standing bug means updating ca-certificates doesn't reach it. Regenerate it as root: rm -f /etc/ssl/certs/java/cacerts update-ca-certificates --fresh - Embedded JREs. Jenkins, Kafka, Elasticsearch, IDEs and many vendor appliances ship their own Java. Updating the system JDK doesn't touch them, so update the product. - Revocation checking instead of removing the roots. OpenJDK and Oracle both accept -Dcom.sun.net.ssl.checkRevocation=true -Dcom.sun.security.enableCRLDP=true as the alternative. Java checks every certificate in the chain, not just the leaf, so it also rejects the forged certificates that carry no revocation pointer: with those two flags after my-image, Probe rejects all four test hosts even on an unpatched eclipse-temurin:21-jre. It costs you a CRL fetch on new chains and can fail handshakes when a CRL endpoint is unreachable. eMudhra's CRL and OCSP servers have also been refusing curl's, Wget's and Python's User-Agents (Java's got through on 25 October), so test it before rolling it out. For signed JARs, JDK 17 and later can check revocation with jarsigner -verify -revcheck -strict app.jar. - If you can't update yet, Oracle also publishes a replacement blocked.certs for lib/security/ (on Java 8 the file is blacklisted.certs). It covers only the first CVE, so remove the roots as well. Go - See what a binary was built with, not what's installed: go version ./bin/my-service prints the toolchain embedded in the file. - Rebuild with 1.26.10 or 1.27.3 and redeploy. Pin it where your build reads it: a toolchain go1.26.10 line in go.mod, go-version in setup-go, the golang: builder image. Go 1.25 and older get no fix. - Rebuild the runtime image too. On Linux, Go still reads the OS store, so a distroless or alpine base from last month keeps the old roots until you pull a new one. Node.js - Check what your node trusts, from the container that runs it: node -e "const {X509Certificate}=require('crypto'); const hit=require('tls').rootCertificates.some(p=>new X509Certificate(p).subject.includes('emSign ECC Root CA - G3')); console.log(process.version, hit ? 'STILL TRUSTS emSign ECC Root CA - G3' : 'root removed')" # v22.22.0 STILL TRUSTS emSign ECC Root CA - G3 - Upgrade to 22.23.4, 24.22.1 or 26.12.1 everywhere you pin a version: .nvmrc, engines, setup-node, the node: base image, Lambda runtimes. These releases also fix other CVEs, including a Brotli out-of-bounds write (CVE-2026-111497), so they're worth taking anyway. - Node 20 and older get nothing. Node 20 reached end of life in April 2026. - Mind the flags. NODE_EXTRA_CA_CERTS only _adds_ roots, it never removes one. --use-system-ca and --use-openssl-ca switch Node to the OS store, which makes the OS package part of your fix. - Electron, Bun and Deno bundle their own roots and released their own fixes. Python - Check every environment, not the interpreter: python -c "import certifi; print(certifi.__version__)" inside each virtualenv, image and lambda. - Pin certifi>=2026.10.19 and regenerate your lockfile so the pin actually lands. Upgrade pip to 26.2.2 or later and uv to 0.13.5 or later. uv ships its own roots, so it needed its own fix: 0.13.4 removed the root and 0.13.5 also rejects chains that carry the compromised keys. - New Python releases don't fix old virtualenvs. 3.15.1, 3.14.9 and 3.13.17 bundle a newer pip for new installs. Existing environments keep whatever certifi they had. - Look for vendored bundles. Some libraries ship a .pem inside the package and ignore both the OS and certifi: find .venv -path '*site-packages*' -name '*.pem' lists them. - Code that uses ssl or urllib directly follows the OS store, so it's fixed by ca-certificates. The OS and Your Images - Update ca-certificates on hosts and in images. Patched versions include Debian 20250419~deb12u2 / 20250419+deb13u1, Ubuntu 20260816ubuntu0.24.04.1 (and its 22.04, 26.04 and 26.10 equivalents), Alpine 20261020-r0 and Fedora 2026.2.92_v9.0.319. - Rebuild, don't just redeploy. An image keeps the trust store it was built with. Images pinned by digest or built from a cached layer never pick up the fix. docker build --pull and check: docker run --rm my-image dpkg -l ca-certificates (or apk info -v ca-certificates on Alpine). - macOS has no ca-certificates package. Safari, Mail and apps built on Apple's frameworks already block these certificates: Apple pushes that change to devices without an OS update. The command-line tools that ship with macOS (curl, git, python3, ruby, perl, openssl) read /etc/ssl/cert.pem instead, which keeps the root until a future macOS update. Apple's note and the incident site give the commands to remove it yourself; a macOS update may put it back. The built-in pip3 uses its own, older copy of the roots, and the incident site shows how to point it at the fixed file. Tools installed with Homebrew use Homebrew's own bundle: brew upgrade ca-certificates. CI/CD and Your Dependencies The CI risk is a forged registry serving a tampered package. TLS was the only thing standing between your build and that package, unless something else checks the content: | Ecosystem | Content check that survives a broken TLS | | --- | --- | | Go | go.sum plus the checksum database, whose signing key ships inside go itself | | npm | integrity hashes in package-lock.json, when installing with npm ci | | Python | pip install --require-hashes, or uv.lock | | Gradle | Dependency verification metadata (--write-verification-metadata sha256) | | Maven | Checksums travel over the same connection and PGP isn't checked by default, so nothing, out of the box | A job that resolves dependencies from scratch, without a lockfile, trusts TLS and nothing else. Managed Runtimes On AWS Lambda, a function on a managed runtime in Auto update mode gets the patched runtime once AWS publishes one, rolled out in phases. Function update mode only updates when you deploy the function, container images need a rebuild, and anything still on nodejs20.x won't be patched at all. The INIT_START line in the function's logs shows the runtime version it started with. --- The Checklist - Inventory every runtime and version per service: Dockerfiles, CI workflows, go.mod, .nvmrc, requirements*.txt, lockfiles, Lambda configs. - Java: JDK at a 24 October build (release notes list JDK-8395690) in images, CI, runners, laptops and embedded JREs, or the four roots removed (or revocation checking on), and Probe rejecting all four test hosts. - Go: every deployed binary rebuilt with 1.26.10 or 1.27.3 (go version ./binary). - Node.js: 22.23.4, 24.22.1 or 26.12.1 everywhere; nothing left on Node 20. - Python: certifi>=2026.10.19 in every environment, pip ≥ 26.2.2, uv ≥ 0.13.5, lockfiles regenerated, vendored .pem files reviewed. - OS: ca-certificates updated on hosts and in base images; images rebuilt with --pull; on macOS, /etc/ssl/cert.pem fixed by hand until Apple's update. - Custom trust stores: any -Djavax.net.ssl.trustStore, REQUESTS_CA_BUNDLE, SSL_CERT_FILE or bundled .pem checked for emSign ECC Root CA - G3. - Third-party software: agents, appliances, backup tools and anything else that makes its own TLS connections, checked against its vendor's notice. - CI: dependency verification on (lockfiles with hashes, Gradle verification metadata). - Verify from where the code runs: the test hosts reject, including the noct ones. - Don't stop at the scanner: Node and Python have no CVE for this. --- Trust Is a File You Ship Here's the whole article as two questions to ask of any process that opens a TLS connection: 1. "Where does this get its roots?" The OS store, a bundle inside the runtime, a package in the virtualenv, a list compiled into the binary. If the answer is "I don't know", that's the finding. 2. "Who updates that file, and when do I pick it up?" Your distro, your runtime's release, your lockfile, your next build. Anything that answers "the next build" is only fixed once you build. None of this is new. DigiNotar in 2011 and the Symantec distrust in 2018 taught the same lesson, and browsers answered it with pushed blocklists and Certificate Transparency. Most server-side clients got neither. Until they do, the trust store is a dependency like any other: it has versions, it goes stale, and it ships inside your artifacts whether you think about it or not. --- _The check script lives in the companion folder: it sends the same request to the four test hosts through curl, Python (ssl and certifi), Node, Java and Go, whichever are installed, and prints which ones still trust the compromised chains. The chains and the proof certificates used above come from the incident site's repository._
we-have-a-problem.com#138 これ，最善策の更新がこのeMud問題のみならずHTTP2Shellなどの問題回避にも有効なので，他の言語でもページが作れるよう翻訳テンプレートを提案した なのでできれば各言語に堪能なエンジニアの方はこのPRの後続PRで自言語の翻訳をPRしてくれると被害抑止に役立って助かります
日本語ミラー(grearl.github.io)の方ではすでに方法を表示してるのと， 一応日本語対応PR#99の後続PR#107で元サイトに記載するように提案しています we-have-a-problem.com#107
eMudhraの中間証明書の件、emsignのOSCPが息吹きかしたようで。Appleは動くべきだろ、これ… we-have-a-problem.com#69
macOSだけでなくiPhoneやiPadでもeMudhraの証明書を一部拒否できていない模様。Windowsのように証明書だけ自動更新する手段がないからなぁ。 Safari (on MacOS) partially mitigated until 22.10.26 · Issue #69 · galadran/we-have-a-problem.com we-have-a-problem.com#69
we-have-a-problem.com#99 責任取って日本語PRを出した