@harisheoran @SarvamAI incident fixed at root
• broke: task accepted updates after the parent run was terminal (Completed/Failed)
• cause: missing lifecycle check in UpdateTasksBulk
• fix: PR #15008 ->adding authorization guards to enforce terminal-run immutability
pipelines#15008 (2/2)
9 views
@harisheoran @SarvamAI Toil killed • Toil:
manual triage & validation across kubeflow/pipelines
• built: agentic issue analysis & quality triage workflow in CI (PRs #13372, #14022, #14035)
• saved: automated triage gatekeeping before maintainer review cycles
code: pipelines#13372 (1/2)
24 views
CVE-2026-118759: CVE-2026-118759: Cross-Tenant Pod Log Exposure via Authorization Bypass in Kubeflow Pipelines
CVE-2026-118759: Cross-Tenant Pod Log Exposure via Authorization Bypass in Kubeflow Pipelines
> Vulnerability ID: CVE-2026-118759
> CVSS Score: 7.5
> Published: 2026-10-30
Prior to version 2.18.0, the Kubeflow Pipelines API server contains a missing authorization and confused deputy vulnerability in its run log streaming endpoint. The HTTP handler for GET /apis/v1alpha1/runs/{run_id}/nodes/{node_id}/log fails to propagate incoming request headers to gRPC metadata and omits caller authorization checks against the requested run. Furthermore, the backend accepts the caller-supplied node_id parameter directly as a Kubernetes pod name without verifying that the pod belongs to the specified pipeline run. An attacker with network access to the API server who knows a valid run identifier can read live or archived execution logs from any pod containing a main container within the run's Kubernetes namespace.
TL;DR
Kubeflow Pipelines prior to 2.18.0 fails to authorize requests to the run log streaming endpoint (GET /apis/v1alpha1/runs/{run_id}/nodes/{node_id}/log) and does not verify pod-to-run ownership, allowing unauthorized callers to stream execution logs from arbitrary pods across tenant namespaces using the API server's service account permissions.
---
Technical Details
- Vulnerability Type: Missing Authorization (CWE-862) / Exposure of Sensitive Information (CWE-200)
- Attack Vector: Network (HTTP GET to /apis/v1alpha1/runs/{run_id}/nodes/{node_id}/log)
- CVSS v3.1 Score: 7.5 High (CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:N/A:N)
- EPSS Score: 0.34% (27.01st percentile)
- Security Impact: Cross-tenant read access to live and archived Kubernetes pod execution logs
- Exploit Status: No known public exploits in the wild
- CISA KEV Status: Not Listed
- Fixed Version: Kubeflow Pipelines 2.18.0
Affected Systems
- Kubeflow Pipelines (kubeflow/pipelines)
- kfp-api-server
- Kubeflow Pipelines: < 2.18.0 (Fixed in: 2.18.0)
Code Analysis
Commit: be4723e
Enforce authorization with dedicated readLog RBAC verb and verify pod-to-run ownership via label check in RunLogServer and ResourceManager.
--- a/backend/src/apiserver/common/const.go
+++ b/backend/src/apiserver/common/const.go
@@ -41,6 +41,7 @@ const (
RbacResourceVerbUnarchive = "unarchive"
RbacResourceVerbReportMetrics = "reportMetrics"
RbacResourceVerbReadArtifact = "readArtifact"
+ RbacResourceVerbReadLog = "readLog"
RbacResourceVerbReport = "report"
RbacResourceVerbUse = "use"
)
Mitigation Strategies
- Upgrade Kubeflow Pipelines API server and deployment manifests to version 2.18.0 or later.
- Enforce Kubernetes NetworkPolicies to restrict direct pod-to-service access to the ml-pipeline API server on port 8888.
- Deploy Istio AuthorizationPolicy or API gateway rules to block or restrict access to /apis/v1alpha1/runs//nodes//log.
- Scope the ml-pipeline-api-server Kubernetes ServiceAccount permissions to prevent reading pod logs in non-pipeline administrative namespaces.
Remediation Steps:
1. Update the Kubeflow Pipelines API server container image and SDK components to release 2.18.0.
2. Apply the updated multi-user Kustomize RBAC manifests (manifests/kustomize/base/installs/multi-user/view-edit-cluster-roles.yaml) so that legitimate roles include the new 'readLog' verb on 'runs' resources.
3. Verify that unauthenticated or cross-tenant requests to GET /apis/v1alpha1/runs/{run_id}/nodes/{node_id}/log return HTTP 403 Forbidden.
4. Audit Kubernetes API server logs for historical 'pods/log' requests issued by system:serviceaccount:kubeflow:ml-pipeline-api-server targeting pods without the 'pipelines.kubeflow.org/run_id' label.
References
- GitHub Security Advisory GHSA-wg5q-x4j9-qm54
- Kubeflow Pipelines Fix Pull Request #13684
- Kubeflow Pipelines Fix Commit be4723e33727c6dad71a95cba61e8f61562ab20f
- Kubeflow Pipelines Release 2.18.0
- National Vulnerability Database (NVD) Record for CVE-2026-118759
- CVE.org Record for CVE-2026-118759
- cvelistV5 JSON Record for CVE-2026-118759
- MITRE ATT&CK Mapping
- FIRST EPSS Scoring
- CISA Known Exploited Vulnerabilities Catalog
---
Read the full report for CVE-2026-118759 on our website for more details including interactive diagrams and full exploit analysis.
CVE-2026-118761: CVE-2026-118761: Pre-Authorization Decompression Bomb Denial of Service in Kubeflow Pipelines API Server
CVE-2026-118761: Pre-Authorization Decompression Bomb Denial of Service in Kubeflow Pipelines API Server
> Vulnerability ID: CVE-2026-118761
> CVSS Score: 7.5
> Published: 2026-10-30
A high-severity Denial of Service (DoS) vulnerability exists in Kubeflow Pipelines prior to version 2.18.0 due to improper handling of highly compressed archive files (CWE-409). The API server decompresses uploaded ZIP and gzip-compressed TAR archives directly into memory without bounding the expanded byte stream or archive traversal metadata. Because decompression occurs before namespace-level authorization in multipart upload handlers, an unauthenticated or unauthorized attacker with network access to the API server can exhaust container heap memory and trigger Out-Of-Memory (OOM) terminations.
TL;DR
Kubeflow Pipelines prior to 2.18.0 fails to bound uncompressed output streams and TAR PAX metadata during pipeline archive uploads. Because multipart upload handlers process archives prior to authorization checks, remote callers can submit small compression bombs to trigger an Out-Of-Memory (OOM) crash in the API server.
---
⚠️ Exploit Status: POC
Technical Details
- CVE / Advisory ID: CVE-2026-118761 / GHSA-8fmv-6rg5-5924
- Weakness Class: CWE-409: Improper Handling of Highly Compressed Data (Data Amplification)
- CVSS v3.1 Score: 7.5 High (CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H)
- Attack Vector: Network (Pre-authorization multipart HTTP archive upload)
- EPSS Score: 0.50% (42.47th Percentile)
- Primary Impact: Denial of Service via Container Out-Of-Memory (OOM) Termination
- Affected Versions: Kubeflow Pipelines < 2.18.0
- Patched Version: Kubeflow Pipelines 2.18.0
Affected Systems
- Kubeflow Pipelines API Server (ml-pipeline-apiserver)
- Kubeflow Pipelines Persistence Agent (ml-pipeline-persistenceagent)
- Kubeflow Pipelines: < 2.18.0 (Fixed in: 2.18.0)
Code Analysis
Commit: 1121d9a
Enforce uncompressed file size limits using io.LimitReader in loadFile, decompressPipelineTarball, and decompressPipelineZip (PR #14171).
Commit: ba9ae72
Prevent tarball traversal DoS via ArchiveTraversalBudget, +1 sentinel checks before io.EOF, saturating arithmetic helpers, and persistence agent response bounds (PR #14186).
Exploit Details
- GitHub Pull Request #14171 (Unit Test Fixtures): Official regression tests demonstrating oversized ZIP and TAR.GZ archive generation and memory limit verification.
- GitHub Pull Request #14186 (PAX Header Bomb Fixtures): Official regression test suite including createPreTargetPaxBomb to reproduce TAR PAX extended header traversal exhaustion.
Mitigation Strategies
- Upgrade Kubeflow Pipelines to version 2.18.0 or later to enforce bounded decompression and TAR traversal budgets.
- Implement service mesh (Istio AuthorizationPolicy) or API gateway authentication rules to block unauthenticated access to pipeline upload endpoints.
- Apply Kubernetes NetworkPolicies to restrict direct pod-to-pod and external network access to the ml-pipeline-apiserver service.
- Configure strict HTTP request body size limits on the Ingress controller or reverse proxy fronting the Kubeflow API server.
- Define explicit memory limits (resources.limits.memory) on the ml-pipeline-apiserver and persistenceagent deployments to isolate OOM events from the host node.
Remediation Steps:
1. Verify the currently deployed Kubeflow Pipelines version by inspecting the container image tags on the ml-pipeline and ml-pipeline-persistenceagent deployments.
2. Update the Kubeflow Pipelines manifests or Helm charts to reference release version 2.18.0 (github.com/kubeflow/pipelines/releases/tag/2.18.0).
3. Apply the updated manifests to the cluster and verify that the ml-pipeline-apiserver and ml-pipeline-persistenceagent pods roll out cleanly.
4. Audit Istio AuthorizationPolicies or Ingress configurations to ensure that /apis/v1beta1/pipelines/upload and /apis/v2beta1/pipelines/upload cannot be reached by unauthenticated clients.
5. Monitor cluster events for OOMKilled terminations using: kubectl get events -n kubeflow --field-selector reason=OOMKilled
References
- GitHub Security Advisory GHSA-8fmv-6rg5-5924
- CVE-2026-118761 Record on CVE.org
- NVD Vulnerability Detail for CVE-2026-118761
- Fix Commit: Bounded Reads & File Limits (PR #14171)
- Fix Commit: Tar Traversal Budget & Overflow Protection (PR #14186)
- Kubeflow Pipelines Pull Request #14171
- Kubeflow Pipelines Pull Request #14186
- Kubeflow Pipelines Release 2.18.0
---
Read the full report for CVE-2026-118761 on our website for more details including interactive diagrams and full exploit analysis.
If anyone wants to take a look 👇
pipelines#12686
3 likes · 28 views
Mentions
@harisheoran @SarvamAI incident fixed at root • broke: task accepted updates after the parent run was terminal (Completed/Failed) • cause: missing lifecycle check in UpdateTasksBulk • fix: PR #15008 ->adding authorization guards to enforce terminal-run immutability pipelines#15008 (2/2)
@harisheoran @SarvamAI Toil killed • Toil: manual triage & validation across kubeflow/pipelines • built: agentic issue analysis & quality triage workflow in CI (PRs #13372, #14022, #14035) • saved: automated triage gatekeeping before maintainer review cycles code: pipelines#13372 (1/2)
CVE-2026-118759: CVE-2026-118759: Cross-Tenant Pod Log Exposure via Authorization Bypass in Kubeflow Pipelines CVE-2026-118759: Cross-Tenant Pod Log Exposure via Authorization Bypass in Kubeflow Pipelines > Vulnerability ID: CVE-2026-118759 > CVSS Score: 7.5 > Published: 2026-10-30 Prior to version 2.18.0, the Kubeflow Pipelines API server contains a missing authorization and confused deputy vulnerability in its run log streaming endpoint. The HTTP handler for GET /apis/v1alpha1/runs/{run_id}/nodes/{node_id}/log fails to propagate incoming request headers to gRPC metadata and omits caller authorization checks against the requested run. Furthermore, the backend accepts the caller-supplied node_id parameter directly as a Kubernetes pod name without verifying that the pod belongs to the specified pipeline run. An attacker with network access to the API server who knows a valid run identifier can read live or archived execution logs from any pod containing a main container within the run's Kubernetes namespace. TL;DR Kubeflow Pipelines prior to 2.18.0 fails to authorize requests to the run log streaming endpoint (GET /apis/v1alpha1/runs/{run_id}/nodes/{node_id}/log) and does not verify pod-to-run ownership, allowing unauthorized callers to stream execution logs from arbitrary pods across tenant namespaces using the API server's service account permissions. --- Technical Details - Vulnerability Type: Missing Authorization (CWE-862) / Exposure of Sensitive Information (CWE-200) - Attack Vector: Network (HTTP GET to /apis/v1alpha1/runs/{run_id}/nodes/{node_id}/log) - CVSS v3.1 Score: 7.5 High (CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:N/A:N) - EPSS Score: 0.34% (27.01st percentile) - Security Impact: Cross-tenant read access to live and archived Kubernetes pod execution logs - Exploit Status: No known public exploits in the wild - CISA KEV Status: Not Listed - Fixed Version: Kubeflow Pipelines 2.18.0 Affected Systems - Kubeflow Pipelines (kubeflow/pipelines) - kfp-api-server - Kubeflow Pipelines: < 2.18.0 (Fixed in: 2.18.0) Code Analysis Commit: be4723e Enforce authorization with dedicated readLog RBAC verb and verify pod-to-run ownership via label check in RunLogServer and ResourceManager. --- a/backend/src/apiserver/common/const.go +++ b/backend/src/apiserver/common/const.go @@ -41,6 +41,7 @@ const ( RbacResourceVerbUnarchive = "unarchive" RbacResourceVerbReportMetrics = "reportMetrics" RbacResourceVerbReadArtifact = "readArtifact" + RbacResourceVerbReadLog = "readLog" RbacResourceVerbReport = "report" RbacResourceVerbUse = "use" ) Mitigation Strategies - Upgrade Kubeflow Pipelines API server and deployment manifests to version 2.18.0 or later. - Enforce Kubernetes NetworkPolicies to restrict direct pod-to-service access to the ml-pipeline API server on port 8888. - Deploy Istio AuthorizationPolicy or API gateway rules to block or restrict access to /apis/v1alpha1/runs//nodes//log. - Scope the ml-pipeline-api-server Kubernetes ServiceAccount permissions to prevent reading pod logs in non-pipeline administrative namespaces. Remediation Steps: 1. Update the Kubeflow Pipelines API server container image and SDK components to release 2.18.0. 2. Apply the updated multi-user Kustomize RBAC manifests (manifests/kustomize/base/installs/multi-user/view-edit-cluster-roles.yaml) so that legitimate roles include the new 'readLog' verb on 'runs' resources. 3. Verify that unauthenticated or cross-tenant requests to GET /apis/v1alpha1/runs/{run_id}/nodes/{node_id}/log return HTTP 403 Forbidden. 4. Audit Kubernetes API server logs for historical 'pods/log' requests issued by system:serviceaccount:kubeflow:ml-pipeline-api-server targeting pods without the 'pipelines.kubeflow.org/run_id' label. References - GitHub Security Advisory GHSA-wg5q-x4j9-qm54 - Kubeflow Pipelines Fix Pull Request #13684 - Kubeflow Pipelines Fix Commit be4723e33727c6dad71a95cba61e8f61562ab20f - Kubeflow Pipelines Release 2.18.0 - National Vulnerability Database (NVD) Record for CVE-2026-118759 - CVE.org Record for CVE-2026-118759 - cvelistV5 JSON Record for CVE-2026-118759 - MITRE ATT&CK Mapping - FIRST EPSS Scoring - CISA Known Exploited Vulnerabilities Catalog --- Read the full report for CVE-2026-118759 on our website for more details including interactive diagrams and full exploit analysis.
CVE-2026-118761: CVE-2026-118761: Pre-Authorization Decompression Bomb Denial of Service in Kubeflow Pipelines API Server CVE-2026-118761: Pre-Authorization Decompression Bomb Denial of Service in Kubeflow Pipelines API Server > Vulnerability ID: CVE-2026-118761 > CVSS Score: 7.5 > Published: 2026-10-30 A high-severity Denial of Service (DoS) vulnerability exists in Kubeflow Pipelines prior to version 2.18.0 due to improper handling of highly compressed archive files (CWE-409). The API server decompresses uploaded ZIP and gzip-compressed TAR archives directly into memory without bounding the expanded byte stream or archive traversal metadata. Because decompression occurs before namespace-level authorization in multipart upload handlers, an unauthenticated or unauthorized attacker with network access to the API server can exhaust container heap memory and trigger Out-Of-Memory (OOM) terminations. TL;DR Kubeflow Pipelines prior to 2.18.0 fails to bound uncompressed output streams and TAR PAX metadata during pipeline archive uploads. Because multipart upload handlers process archives prior to authorization checks, remote callers can submit small compression bombs to trigger an Out-Of-Memory (OOM) crash in the API server. --- ⚠️ Exploit Status: POC Technical Details - CVE / Advisory ID: CVE-2026-118761 / GHSA-8fmv-6rg5-5924 - Weakness Class: CWE-409: Improper Handling of Highly Compressed Data (Data Amplification) - CVSS v3.1 Score: 7.5 High (CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H) - Attack Vector: Network (Pre-authorization multipart HTTP archive upload) - EPSS Score: 0.50% (42.47th Percentile) - Primary Impact: Denial of Service via Container Out-Of-Memory (OOM) Termination - Affected Versions: Kubeflow Pipelines < 2.18.0 - Patched Version: Kubeflow Pipelines 2.18.0 Affected Systems - Kubeflow Pipelines API Server (ml-pipeline-apiserver) - Kubeflow Pipelines Persistence Agent (ml-pipeline-persistenceagent) - Kubeflow Pipelines: < 2.18.0 (Fixed in: 2.18.0) Code Analysis Commit: 1121d9a Enforce uncompressed file size limits using io.LimitReader in loadFile, decompressPipelineTarball, and decompressPipelineZip (PR #14171). Commit: ba9ae72 Prevent tarball traversal DoS via ArchiveTraversalBudget, +1 sentinel checks before io.EOF, saturating arithmetic helpers, and persistence agent response bounds (PR #14186). Exploit Details - GitHub Pull Request #14171 (Unit Test Fixtures): Official regression tests demonstrating oversized ZIP and TAR.GZ archive generation and memory limit verification. - GitHub Pull Request #14186 (PAX Header Bomb Fixtures): Official regression test suite including createPreTargetPaxBomb to reproduce TAR PAX extended header traversal exhaustion. Mitigation Strategies - Upgrade Kubeflow Pipelines to version 2.18.0 or later to enforce bounded decompression and TAR traversal budgets. - Implement service mesh (Istio AuthorizationPolicy) or API gateway authentication rules to block unauthenticated access to pipeline upload endpoints. - Apply Kubernetes NetworkPolicies to restrict direct pod-to-pod and external network access to the ml-pipeline-apiserver service. - Configure strict HTTP request body size limits on the Ingress controller or reverse proxy fronting the Kubeflow API server. - Define explicit memory limits (resources.limits.memory) on the ml-pipeline-apiserver and persistenceagent deployments to isolate OOM events from the host node. Remediation Steps: 1. Verify the currently deployed Kubeflow Pipelines version by inspecting the container image tags on the ml-pipeline and ml-pipeline-persistenceagent deployments. 2. Update the Kubeflow Pipelines manifests or Helm charts to reference release version 2.18.0 (github.com/kubeflow/pipelines/releases/tag/2.18.0). 3. Apply the updated manifests to the cluster and verify that the ml-pipeline-apiserver and ml-pipeline-persistenceagent pods roll out cleanly. 4. Audit Istio AuthorizationPolicies or Ingress configurations to ensure that /apis/v1beta1/pipelines/upload and /apis/v2beta1/pipelines/upload cannot be reached by unauthenticated clients. 5. Monitor cluster events for OOMKilled terminations using: kubectl get events -n kubeflow --field-selector reason=OOMKilled References - GitHub Security Advisory GHSA-8fmv-6rg5-5924 - CVE-2026-118761 Record on CVE.org - NVD Vulnerability Detail for CVE-2026-118761 - Fix Commit: Bounded Reads & File Limits (PR #14171) - Fix Commit: Tar Traversal Budget & Overflow Protection (PR #14186) - Kubeflow Pipelines Pull Request #14171 - Kubeflow Pipelines Pull Request #14186 - Kubeflow Pipelines Release 2.18.0 --- Read the full report for CVE-2026-118761 on our website for more details including interactive diagrams and full exploit analysis.
If anyone wants to take a look 👇 pipelines#12686