cilium

Cilium Upgrade Fails Due to Init Container Timeout: Solution to Resolve CrashLoopBackOff Issue Introduction Upgrading Cilium, a critical networking and security solution for Kubernetes, often introduces complex challenges. Recently, users upgrading from Cilium 1.18.16 to 1.19.10 or 1.20.4 have encountered a persistent issue: the Cilium agent enters an Init:CrashLoopBackOff state. This failure originates from the config init container, which is unable to establish a connection to the Kubernetes API server due to an I/O timeout. While the worker node can successfully reach the API server's ClusterIP using tools like curl, Cilium’s initialization process fails to do so, exposing a specific connectivity issue within the containerized environment. This discrepancy highlights a critical interaction between Cilium’s initialization logic and Kubernetes API server accessibility during upgrades. The implications are severe: prolonged cluster downtime, potential misapplication of network policies, and increased operational burden for administrators. This issue is particularly pressing for users migrating to Cilium 1.19 or 1.20. While a workaround exists, it circumvents the problem without resolving the underlying API connectivity failure, necessitating further investigation. This article dissects the technical mechanisms driving the issue, the causal relationships between components, and the practical steps to mitigate immediate disruptions, ensuring smoother operations for Kubernetes clusters dependent on Cilium. The Causal Chain: Root Causes and Mechanisms The issue originates in the config init container, which is responsible for fetching Cilium’s configuration by querying the Kubernetes API server. During the upgrade process, this container executes the command cilium-dbg build-config, which relies on API communication to retrieve essential configuration data. The failure occurs when the container’s network namespace cannot establish a TCP connection to the API server within the configured timeout period, despite the host network (accessible via curl) functioning correctly. This indicates a namespace-specific connectivity issue, likely exacerbated by transient network policies or incomplete service discovery during the upgrade. The causal sequence is as follows: - Trigger: Cilium upgrade initiates the config init container to fetch configuration. - Mechanism: The container’s network namespace fails to resolve or connect to the API server’s ClusterIP due to incomplete service proxy configuration or transient network policy enforcement. - Outcome: The cilium-dbg build-config command times out, causing the container to crash and triggering a CrashLoopBackOff state in the Cilium agent. The presence of CiliumNodeConfig may compound this issue by introducing additional API dependencies or configuration conflicts during the upgrade. However, the precise interaction between CiliumNodeConfig and the connectivity failure requires further analysis to establish causality. The Workaround: Bypassing the Init Container A community-developed workaround circumvents the issue by eliminating the config init container and configuring Cilium to read its settings directly from a ConfigMap. While effective in restoring Cilium functionality, this approach does not address the root cause of the API connectivity failure. Implementation steps are as follows: 1. Verify CiliumNodeConfig Usage: Check for existing CiliumNodeConfig resources: kubectl get ciliumnodeconfigs.cilium.io -A 2. Modify Helm Configuration: Add the following to your Helm values file (e.g., my-values.yaml) to disable the init container and specify the ConfigMap source: daemon: configSources: "config-map:cilium-config" 3. Apply the Upgrade: Execute the Helm upgrade command with the modified configuration: helm upgrade cilium cilium/cilium \--version 1.20.4 \-n kube-system \-f my-values.yaml This configuration removes the config init container, allowing Cilium to start without API connectivity issues. However, it disables CiliumNodeConfig as a configuration source, which may impact clusters relying on per-node overrides. Administrators must weigh the trade-offs before implementing this workaround. Practical Considerations and Risks While the workaround provides immediate operational relief, it does not resolve the underlying API connectivity issue. By bypassing the config init container, the CrashLoopBackOff state is avoided, but the root cause remains unaddressed. This leaves clusters vulnerable to recurrence during future upgrades or configuration changes. Key risks include: - Unaddressed connectivity issues manifesting as inconsistent network policy enforcement or delayed pod initialization. - Increased complexity in troubleshooting due to masked symptoms. - Dependency on a non-standard configuration that deviates from Cilium’s recommended deployment model. Administrators should monitor clusters closely and treat this workaround as a temporary measure until a permanent fix is released by the Cilium development team. Conclusion The Init:CrashLoopBackOff issue during Cilium upgrades represents a significant operational challenge for Kubernetes administrators, stemming from a namespace-specific API connectivity failure in the config init container. While the community-driven workaround restores functionality by bypassing the problematic container, it does not address the underlying cause. Understanding the technical mechanisms and trade-offs of this issue is critical for effective management of Cilium upgrades and ensuring Kubernetes cluster stability. For affected users, this workaround can mitigate immediate downtime, but vigilance is required to monitor for residual connectivity issues. Long-term resolution depends on a comprehensive fix from the Cilium project, which will provide a more robust and sustainable solution. Administrators are encouraged to stay informed about ongoing investigations and contribute to the community’s efforts in resolving this issue. Problem Analysis The Cilium upgrade issue manifesting as Init:CrashLoopBackOff during transitions from version 1.18.16 to 1.19.10 or 1.20.4 originates from a critical failure in the config init container. This container is responsible for initializing Cilium’s configuration by querying the Kubernetes API server. The failure occurs due to an I/O timeout when the container attempts to establish a TCP connection to the API server. The error message explicitly states: "Unable to contact k8s api-server: Get '10.96.0.1': dial tcp 10.96.0.1:443: i/o timeout" While the host network functions correctly—as evidenced by successful curl requests to the API ClusterIP—the container’s isolated network namespace fails to resolve or connect to the API server’s ClusterIP. This discrepancy underscores a namespace-specific connectivity issue, attributable to the following mechanisms: - Incomplete Service Proxy Configuration: During the upgrade, kube-proxy’s IPVS or iptables rules may not fully propagate or configure correctly, preventing the container from resolving the API server’s IP. - Transient Network Policies: Misapplied or temporary Cilium network policies during the upgrade can block the container’s outbound traffic to the API server. The causal chain is as follows: 1. Trigger: The upgrade initiates the config init container, which executes cilium-dbg build-config, requiring API server access to fetch Cilium’s configuration. 2. Failure Mechanism: The container’s network namespace lacks the necessary routes, DNS resolution, or service proxy rules to establish a TCP connection to the API server within the timeout window, causing cilium-dbg build-config to fail. 3. Consequence: The container crashes, resulting in the Init:CrashLoopBackOff state, which halts the Cilium agent’s initialization and prolongs cluster downtime. Scenarios Where This Issue Occurs This issue is not isolated but arises in multiple scenarios, each driven by distinct mechanisms: | | | | --- | --- | | Scenario | Mechanism | | 1. Upgrading to Cilium 1.19/1.20 | The config init container fails due to incomplete kube-proxy configuration during the upgrade, preventing API server connectivity. | | 2. Using CiliumNodeConfig Overrides | Per-node configurations introduce additional API dependencies, exacerbating connectivity issues during upgrades. | | 3. Transient Network Policies | Misapplied or temporary Cilium network policies block the container’s outbound traffic to the API server. | | 4. Incomplete Kube-Proxy Configuration | IPVS or iptables rules fail to propagate fully, preventing the container from resolving the API server’s ClusterIP. | | 5. Network Namespace Isolation | The container’s network namespace lacks necessary routes or DNS resolution for the API server, isolating it from cluster networking. | | 6. Resource Contention During Upgrade | High CPU or memory usage during the upgrade delays API server responses, triggering timeouts in the config init container. | Risks and Implications The proposed workaround—bypassing the config init container and configuring Cilium to read from a ConfigMap—provides immediate relief but introduces significant risks: - Masked Symptoms: The root cause of API connectivity failure remains unaddressed, leaving clusters vulnerable to recurrence. - Disabled CiliumNodeConfig: Per-node configuration overrides are disabled, reducing flexibility in cluster management. - Operational Overhead: Administrators must manually verify and apply the workaround, increasing the risk of misconfiguration. The underlying risk mechanisms are: 1. Dependency Shift: Relying on a ConfigMap instead of the API server shifts the failure point but does not resolve the network namespace issue. 2. Configuration Drift: Disabling CiliumNodeConfig may lead to configuration drift if per-node overrides were previously used. 3. Delayed Root Cause Resolution: Masking symptoms postpones the identification and resolution of the underlying connectivity problem. A permanent fix from the Cilium development team is critical to address the root cause and ensure stable, reliable upgrades. Troubleshooting Cilium Upgrade Failures Due to Kubernetes API Timeout Upgrading Cilium from version 1.18.16 to 1.19.10 or 1.20.4 may trigger a critical failure where the config init container fails to establish a TCP connection to the Kubernetes API server's ClusterIP. This failure occurs within the container's network namespace, resulting in an I/O timeout and causing the Cilium agent to enter Init:CrashLoopBackOff. The host network remains functional, isolating the issue to the container's network configuration. Below is a structured guide to diagnose, apply a temporary workaround, and understand the underlying mechanisms. Step 1: Verify the Issue Confirm the failure by inspecting the logs of the config init container: - Run: kubectl describe pod <cilium-pod-name> -n kube-system - Identify the error: Unable to contact k8s api-server: Get "https://10.96.0.1:443/api/v1/namespaces/kube-system": dial tcp 10.96.0.1:443: i/o timeout Step 2: Validate Host Network Connectivity Ensure the host network can reach the Kubernetes API server, confirming the issue is namespace-specific: - Run: curl -k https://10.96.0.1:443/api/v1/namespaces/kube-system - A successful response indicates the issue is confined to the container's network namespace, likely due to misconfigured service proxy rules or network policies. Step 3: Assess CiliumNodeConfig Usage Determine if CiliumNodeConfig is in use, as it may introduce additional API dependencies that exacerbate the issue: - Run: kubectl get ciliumnodeconfigs.cilium.io -A - If no output, proceed to the workaround. If in use, evaluate the risk of disabling it (see Risks of Workaround below). Step 4: Implement the Workaround Bypass the config init container by configuring Cilium to read settings directly from the ConfigMap: - Add the following to your Helm values file (e.g., my-values.yaml): daemon: configSources: "config-map:cilium-config" - Apply the configuration using Helm: helm upgrade cilium cilium/cilium --version 1.20.4 -n kube-system -f my-values.yaml Step 5: Confirm Resolution Verify that the Cilium agents start successfully: - Run: kubectl get pods -n kube-system | grep cilium - Ensure all Cilium pods are in the Running state. Risks of the Workaround While this workaround resolves the immediate issue, it introduces significant risks: - Masked Root Cause: The underlying API connectivity failure remains unaddressed, increasing the likelihood of recurrence. - Disabled CiliumNodeConfig: Eliminates per-node configuration overrides, reducing cluster management flexibility. - Operational Overhead: Manual intervention heightens the risk of misconfiguration during future upgrades. Mechanism: The workaround redirects configuration retrieval from the config init container to the ConfigMap, circumventing the namespace-specific connectivity issue. However, it does not rectify the container's network namespace failure, leaving the cluster susceptible to similar issues. Permanent Fix Required The Cilium development team must address the root cause of the API connectivity failure to ensure stable upgrades. Track progress in the Cilium issue: Cilium Issue #44464. Best Practices and Prevention The Cilium upgrade issue resulting in Init:CrashLoopBackOff stems from Kubernetes API timeouts during the initialization process. While a workaround exists, it circumvents the symptom rather than resolving the underlying connectivity problem. Below are evidence-based strategies to mitigate risks during Cilium upgrades, focusing on pre-upgrade validation, proactive monitoring, and robust configuration practices. Pre-Upgrade Validation - Verify Kubernetes API Connectivity: Prior to initiating the upgrade, confirm that all nodes can establish connections to the Kubernetes API server. Utilize kubectl or curl to test connectivity to the API server's ClusterIP (10.96.0.1:443). This ensures the Cilium init container's network namespace can resolve and connect to the API server, preventing I/O timeouts. Mechanism: Incomplete service proxy rules (e.g., kube-proxy IPVS/iptables) or misconfigured network policies can obstruct API access. Pre-upgrade testing identifies these issues before they trigger failures. - Audit CiliumNodeConfig Overrides: Inspect active CiliumNodeConfig resources using kubectl get ciliumnodeconfigs.cilium.io -A. Evaluate whether these configurations introduce additional API dependencies or conflicts that could exacerbate upgrade issues. Mechanism: CiliumNodeConfig increases API interaction complexity, potentially delaying or blocking the config init container's initialization process due to transient connectivity issues. - Review Network Policies: Examine Cilium network policies for rules that may inadvertently block outbound traffic to the Kubernetes API server during upgrades. Temporarily disable or adjust policies to ensure uninterrupted API access. Mechanism: Transient policies applied during upgrades can block the config init container's TCP connection attempts, triggering timeouts. Proactive Monitoring Strategies - Monitor Init Container Logs: During upgrades, actively monitor the logs of the config init container for I/O timeout errors. Leverage kubectl logs or centralized logging solutions to detect issues promptly. Mechanism: Early detection of timeout errors enables immediate intervention, minimizing downtime and operational impact. - Track API Server Performance: Implement monitoring for Kubernetes API server latency and response times. Elevated latency or resource contention can delay API responses, triggering timeouts in the init container. Mechanism: Resource-constrained API servers may fail to respond within the init container's timeout window, leading to crashes. - Establish CrashLoopBackOff Alerts: Configure alerts for pods entering CrashLoopBackOff states during upgrades. This facilitates rapid response to prevent prolonged cluster unavailability. Mechanism: Timely alerts enable administrators to apply workarounds or investigate root causes before the issue escalates. Robust Configuration Practices - Utilize ConfigMap for Configuration: Configure Cilium to read settings from a ConfigMap instead of relying on the config init container. Add daemon: configSources: "config-map:cilium-config" to your Helm values file. Mechanism: Bypassing the init container eliminates the risk of namespace-specific connectivity issues during upgrades. - Limit CiliumNodeConfig Usage: Avoid using CiliumNodeConfig unless strictly necessary. Per-node overrides increase complexity and API dependencies, elevating the risk of upgrade failures. Mechanism: Reducing API interactions during upgrades lowers the likelihood of transient connectivity issues. - Test Upgrades in Staging Environments: Always test Cilium upgrades in a staging environment that mirrors production. This identifies potential issues without impacting live workloads. Mechanism: Staging environments replicate production configurations, exposing upgrade-related problems before they affect critical systems. Workaround Risks and Limitations The workaround of using ConfigMap instead of the config init container mitigates immediate symptoms but introduces risks: - Masked Root Cause: The underlying API connectivity failure remains unresolved, leaving clusters susceptible to recurrence. Mechanism: The workaround shifts the failure point without addressing the namespace-specific connectivity issue. - Reduced Configuration Flexibility: Disabling CiliumNodeConfig diminishes cluster management flexibility and risks configuration drift if overrides were previously used. Mechanism: Loss of per-node configurations can lead to misaligned settings across nodes, causing policy or performance inconsistencies. - Increased Operational Complexity: Manual application of the workaround heightens the risk of misconfiguration, particularly in large or complex clusters. Mechanism: Human error during Helm configuration changes can introduce unintended side effects or inconsistencies. Long-Term Resolution While the workaround provides temporary relief, a permanent fix from the Cilium development team is essential. Monitor Cilium Issue #44464 for updates on root cause resolution. In the interim, adopt the above practices to minimize upgrade risks and maintain system stability. Mechanism: Addressing the root cause ensures stable upgrades by resolving namespace-specific connectivity issues, eliminating the need for temporary workarounds.

🐝 Tetragon v1.8.0 is live! 🐝 This is Tetragon's largest release with 1,260 commits (a 70% increase) across 566 PRs (a 123% increase) from 54 contributors. Learn more in the release notes: github.com/cilium/tetragon/releases/tag/v1.8.0

CalicoとCiliumの通信経路を実測する：Service・VXLAN・Policy・既存TCP やったこと 代表的なCNIであるCalicoとCiliumを取り上げ、kube-proxyありのiptables構成と、kube-proxyなしのeBPF構成の通信経路を比較した。 同じPod・Service・NetworkPolicyの構成で動かしたところ、HTTP接続やDNS、readinessによる接続先の切り替え、通信の許可・拒否といった基本動作に差は見られなかった。 一方、Pod内やNode上でパケットを採取して確認すると、Serviceの宛先がbackend宛に変換される場所や、接続状態を保持する仕組みに明確な違いがあった。 CNIとは CNI（Container Network Interface）は、KubernetesがPodのネットワークをセットアップする際に呼び出す共通インターフェース規格である。CNIプラグインを通じてネットワークインターフェースやIPアドレスを設定し、Podをネットワークにつなぐ。CalicoやCiliumはCNIに対応したネットワークソフトウェアで、Pod間通信やNetworkPolicyによるアクセス制御も担う。 kube-proxyの役割 PodにIPアドレスが付いて通信できるようになっても、ServiceのClusterIPをどのbackend Podへ振り分けるかという処理が必要になる。このServiceの転送ルールを各Nodeに設定するのがkube-proxyである。ServiceとEndpointSliceの変化を監視し、接続先の追加・削除などをルールに反映する。 今回のiptablesモードでは、kube-proxyがカーネルにルールを設定し、実際のbackend選択と宛先変換はLinuxカーネルが行う。通信のたびにkube-proxyプロセスがパケットを受け取り、中継する構成ではない。Pod同士をつなぐネットワークと、その上でServiceの宛先を選ぶ処理を分けると、Calicoとkube-proxyが担う役割も見えやすくなる。 eBPFでServiceを処理する構成 eBPFはLinuxカーネルのパケット処理やソケット操作の途中でプログラムを動かせる仕組みである。CNIのような接続規格やkube-proxyのようなコンポーネントを指す名前ではなく、ネットワーク機能の実装にも使われる技術である。 CiliumはeBPFを使って、Pod間通信やNetworkPolicyに加え、Serviceの振り分けも実装する。このService機能でkube-proxyを置き換えるのがkube-proxy replacementである。今回有効にしたsocket-LBでは、アプリのconnect()を処理する段階でbackendを選び、ソケットの宛先を書き換える。アプリは同じClusterIPへ接続していても、宛先が変わる場所はiptables構成と異なる。 CalicoとCiliumの特徴と今回の検証構成 CalicoはBGPによるルーティングや柔軟なNetworkPolicy、CiliumはeBPFによる高度な通信制御やHubbleによる可視化が主な特徴として挙げられる。Calicoにもkube-proxyを置き換えるeBPFモードがあるが、本記事ではCalicoのiptables構成と、Ciliumのsocket-LBを有効にしたeBPF構成を比較した。以降のCalicoおよびCiliumの記述は、この検証構成を指す。 役割 今回のCalico構成 今回のCilium構成 PodのIP・ネットワーク接続 Calico Cilium NetworkPolicyの適用 Calicoが設定したiptablesルール CiliumのeBPF Serviceのbackend選択と宛先変換 kube-proxyが設定したiptablesルール CiliumのeBPF（socket-LB） 検証結果 アプリから見た基本動作は同じだったが、ServiceからPodへの宛先変換は異なる場所で処理されていた。 主な共通点と違いは次の4点である。 Serviceの宛先変換：CalicoはPodを出た後にNode上で変換する。一方、Ciliumはsocket接続時（connectシステムコール時）に変換するため、Podのeth0には最初からbackend宛のSYNパケットが現れる。 別Nodeへの転送：どちらもVXLANによるカプセル化を使用していた。ただし、外側UDPのポート番号とVXLANのVNIの扱いに違いがあった。 NetworkPolicy：どちらもパケットが宛先Podへ届く前に破棄された。CalicoではiptablesのDROPカウンタ、Ciliumではendpointの拒否イベントとして観測できた。 readinessによる切り替え：NotReadyになったPodは新規接続の振り分け先から除外されたが、すでに確立していた既存TCP接続は切断されず、同じPodと通信を維持できた。 最も大きな違いは黄色いひし形で示した宛先変換の位置にある。図は別Nodeのbackendに接続する場合を描いている。 ※図の凡例：青はPod・アプリ、黄色いひし形は変換やPolicyの判定処理、灰色は転送経路、赤はパケットの拒否を表す。 Container内のアプリプロセスはuser spaceで動く。Pod内のContainerは同じnetwork namespaceを使い、IPアドレスとポート空間を共有する。アプリのsocketやPodのeth0はこのnamespaceに属する。socket-LBのcgroup hookもiptablesもカーネル内の処理である。図ではkindのContainerとして動く論理Nodeと、その中の処理の区分を示した。これらのNodeは同じLinuxカーネル上で動く。 比較した構成 検証環境はcontrol-plane 1台、worker 2台のクラスタ構成とした。clientと同じworkerノードにbackend Podを1台、別workerノードにもう1台配置している。Serviceへの新規接続とPod IPへの直接接続を比較し、readiness probeの状態変化やNetworkPolicyの適用前後における挙動を観測した。 項目 Calico Cilium バージョン v3.33.0 v1.20.2 Serviceの実装 kube-proxyのiptablesモード kube-proxy replacement、Full socket-LB Node間の転送 VXLAN、BGP無効 VXLAN 共通条件 Kubernetes v1.35.0、IPv4、Pod MTU 1400 同左 検証詳細 Calicoの場合 ServiceのClusterIPへ接続すると、client Podのeth0だけでなく、Node側のvethインターフェース上でもClusterIP宛のSYNパケットが観測された。パケットの宛先がbackendのPod IPへ変換されるのは、Nodeに入った後である。 別Node上で動作するbackend Podが選択された通信経路を追跡した結果は、以下のとおりである（IPアドレスは役割名に置き換えている）。 client Pod eth0 client:33706 → Service:8080 SYN seq=433715408 Nodeのhost veth client:33706 → Service:8080 SYN seq=433715408 vxlan.calico client:33706 → backend:8080 SYN seq=433715408 backend Pod eth0 client:33706 → backend:8080 SYN seq=433715408 送信元ポートとSYNシーケンス番号は一貫しており、宛先IPアドレスのみが途中で書き換わっていることがわかる。これは、Node上でkube-proxyが生成したiptablesチェーン（KUBE-SERVICES → KUBE-SVC-* → KUBE-SEP-* → DNAT）のルールによって処理されているためである。実際に新規接続を8回試行したところ、該当Serviceのエントリポイントにあるカウンタも正確に8パケット分増加していた。 なお、ここでkube-proxyが担っている役割は、カーネルにiptablesルールをプロビジョニングすることである。通信が発生するたびにパケットがkube-proxyプロセスへ渡されるわけではなく、Linuxカーネル自体がiptablesルールに従ってbackendを選択し、DNAT（宛先NAT）を実行する。[1] また、この宛先変換のマッピング情報はconntrack（接続追跡テーブル）に保持される。今回の検証でも、original方向にclient → Service、reply方向にbackend → clientという対応関係が記録されていた。この接続状態が管理されているおかげで、Service宛てに接続したクライアントと、実際に応答を返すbackend Podとの間で正常にセッションが維持される。 Ciliumの場合 一方、CiliumではクライアントアプリがClusterIP宛てに接続を要求していても、client Podのeth0でキャプチャされたSYNパケットは最初からbackendのPod IP宛てになっていた。新規に試行した8回の接続すべてにおいて、この挙動が確認できた。 これは、Ciliumのsocket-LB機能が、アプリケーションがconnect()システムコールを発行した時点でServiceのbackendを選択し、ソケットの宛先アドレス自体を書き換えるためである。TCPパケットが生成されるよりも前の段階で宛先が書き換わるため、パケットがPodを出た後にNode上でServiceのDNATを行う必要がなくなる。[2] 今回の通常設定ではソケットトレースログが間引かれるため、一時的にレートリミットを解除して別の8接続を採取した。各ソケット（socket cookie）において、以下のように変換前後の状態を確認できた。 接続先の変換 Service:8080 → backend:8080 アプリに見せる宛先 backend:8080 → Service:8080 2行目は、アプリがgetpeername()で接続相手の情報を参照した際に返される逆変換（reverse translation）の処理である。ネットワーク上ではbackend Podと直接通信しつつも、アプリケーション側にはあくまで「Serviceに接続している」と見せかける仕組みになっている。実際のパケット送受信はbackend宛てのBPF conntrackテーブルで追跡され、Serviceとの対応関係はBPFのsocket reverse NATテーブルで管理されていた。 なお、socketトレースとpcap（パケットキャプチャ）の突き合わせは、clientのcgroup、逐次実行した各接続のタイムスタンプ、および応答したbackend Podの組み合わせによって確認した。socket cookieはパケットヘッダには含まれず、トレース側には送信元ポートが出力されないため、これらを直接の結合キーにすることはできない点に注意が必要である。 このアーキテクチャの違いは障害調査やトラブルシューティング時のパケット解析（tcpdump）にも大きく影響する。Cilium環境のPod内でパケットをキャプチャしてClusterIP宛ての通信が見当たらなくても、Serviceが使われていないわけではない。socket-LBが有効な構成では、Pod内であっても追うべき宛先はbackendのIPアドレスになる。 別Nodeへの転送は、両方ともVXLANを採用 Serviceの宛先変換が行われる場所は異なっていたものの、今回の検証構成におけるNode間のパケット転送は、CalicoとCiliumのどちらもVXLANによるオーバーレイネットワークを用いていた。同一Node内のPod同士の直接通信ではVXLANヘッダは付与されないが、別Node上のPod宛て通信では、内側のPod間TCPパケットが外側のNode間UDPパケットでカプセル化される。 内側のTCPパケットにおける送信元・宛先IP、ポート番号、SYNシーケンス番号を追跡したところ、送信元NodeのNIC、トンネルデバイス、宛先NodeのNIC、そして受信側Podのeth0まで、一連の通信を正確に追うことができた。トンネルデバイス名としては、Calicoがvxlan.calico、Ciliumがcilium_vxlanを使用していた。 特に興味深いのはVXLANヘッダに含まれるVNI（VXLAN Network Identifier）の使われ方である。Calicoでは固定値の4096が設定されていたのに対し、Ciliumではclientからbackendへの往路が8639、復路が40262となっており、それぞれ送信元Podに割り当てられたSecurity Identityの値と一致していた。Ciliumはトンネルでパケットをカプセル化して転送する際、VNIフィールドを利用して送信元Podのセキュリティ属性（どのNetworkPolicyグループに属するか）を宛先Nodeへ伝達している。[3] NetworkPolicyの動作：どちらも宛先Podの手前で拒否 別Node上で動作するbackend Podに対してIngressの隔離NetworkPolicyを適用し、client PodからPod IPへ直接接続を試行した。その結果、どちらの構成でもSYNパケットは宛先Nodeのeth0までは到達したものの、backend Podのeth0には届かずに破棄された。 今回の検証では1接続の最初のSYNパケットと再送パケットを含めて計3パケットを観測した。Calicoでは対象Pod向けのiptablesチェーンにおいてDROPカウンタが3パケット（180バイト）増加し、Ciliumでは対象endpointのイベントログに3件のPolicy deniedが記録された。その後、送信元Podを許可するルールを追加すると、どちらの構成でもパケットがbackend Podへ到達し、HTTP 200の正常応答が得られるようになった。 「通信が拒否された」という最終的な結果は同じであっても、トラブルシューティング時に確認すべき調査ポイントは異なる。CalicoであればiptablesチェーンのDROPカウンタを確認し、CiliumであればeBPFエンドポイントやPolicyのイベントログをパケットトレースと突き合わせることで、パケットが宛先Nodeに到達した後に破棄された要因を正確に特定できる。 readiness probeによる制御：新規接続の振り分け先変更と既存TCPの継続 通信確立中のbackend PodをNotReady状態に遷移させ、選択状態の収束を確認してから新規TCP接続を試すと、もう一方のReadyなPodへ切り替わっていた。しかしその間も、すでに確立していた既存のTCP接続は切断されることなく、元のPodから継続してHTTP 200応答を受け取り続けた。 readiness probeの失敗は、Pod上のアプリケーションのTCPソケットを強制クローズする処理ではない。今回の検証用backend Podは、NotReady状態であってもリクエストを受け取れば正常に応答できる状態を保っていた。つまり、Serviceで「新規接続の振り分け対象から外れること」と、「既存の接続済みソケットで通信を維持できること」は完全に両立する。 観測した状態 Calico Cilium NotReady後の新規8接続 すべて残ったReadyなPodへ すべて残ったReadyなPodへ 既存TCP 同じ接続で52応答 同じ接続で55応答 保持していた接続状態 Serviceとbackendを結ぶconntrack backend宛のBPF conntrackとsocket reverse NAT 既存のTCP接続についてはPodを削除せずにclientと同一Node上のbackendとの間で約1分間にわたり観測を継続した（クライアント側での自動再接続は行わずに1本のTCPセッションを維持）。その後、readiness状態を回復させると、両構成ともに新規接続の振り分け先として2つのbackend Podが再び選択されるようになった。なお、通常のClusterIP Serviceに対するDNS名前解決の結果は、この一連の動作中も一貫して同一のClusterIPアドレスを返し続けていた。 まとめ CalicoとCiliumのどちらを採用した場合でも、アプリケーションから見たKubernetesの基本動作（Service連携やアクセス制御）は共通していた。しかし、パケットレベルの挙動に着目すると、Podから送出されるパケットの宛先アドレスには決定的な違いが存在していた。Calico（iptables構成）ではパケットがPodを出た後にNodeのカーネル内でServiceからbackendへの変換が行われるのに対し、Cilium（socket-LB構成）ではソケットのconnect()処理時にあらかじめ変換が完了している。 このアーキテクチャの違いを理解しておくことは、パケットキャプチャ（tcpdump）を用いた障害解析やネットワーク設計において極めて重要である。Calicoでは「どの観測点の間でClusterIPからPod IPへ宛先が書き換わるか」を追う必要がある一方、Ciliumでは最初からbackend宛てのパケットとsocket-LBの状態を突き合わせて確認しなければならない。一見同じように機能するKubernetesクラスタであっても、基盤となるCNIの実装特性に応じた適切な観測ポイントを見極めることが肝要である。 余談: なぜkube-proxyを利用しない構成が広がっているのか kube-proxyを用いないいわゆる「kube-proxy-less」な構成は、主要クラウド各社のマネージドKubernetesサービスでも標準採用が進んでいる。2026年10月時点の各社公式ドキュメントに基づくと、現状は以下のように整理できる。 クラウドの構成 ネットワークの実装 kube-proxy GKE Autopilotの新規クラスタ Ciliumを使うDataplane V2が既定 使用しない AKS Automatic Azure CNI Overlay powered by Ciliumが既定 使用しない EKSのEC2ノード構成（Auto Modeを除く） Amazon VPC CNIとkube-proxy add-on 使用する Google CloudのGKE Dataplane V2では、Serviceのパケット転送およびNetworkPolicyの評価をeBPFで処理し、高度なポリシーログ出力も提供している。Microsoft AzureのAKSでも、Azure CNIとCiliumのデータプレーンを統合することで、Service転送やセキュリティポリシーの適用、さらに拡張アドオンを通じた詳細な可視化を実現している。[4, 5] 一方、AWSのEKSにおける標準的なEC2ノード構成では、Amazon VPC CNIがPodネットワークのプロビジョニングを行い、Serviceのロードバランシングはkube-proxyアドオンが担当する。NetworkPolicyの適用エンジンと、Serviceの負荷分散にkube-proxyを用いるかどうかは独立した選択肢となっている。[6, 7] なお、Calicoにもkube-proxyを置き換えてeBPFでServiceを処理するモードが存在し、kube-proxy自体にも従来のiptablesに代わるnftablesモードなどのパフォーマンス改善が進められている。そのため、単に「CalicoかCiliumか」という製品の比較にとどまらず、「どのデータプレーン層でServiceルーティングを実装するか」という設計視点が重要となる。[8, 9] Ciliumをフル活用する構成の大きな利点は、Serviceのbackend検索やNetworkPolicyの評価、通信の可観測性をすべて同一のeBPFデータプレーンで一元処理できる点にある。特にsocket-LBを適用した場合、ソケット接続の開始時点でbackendが確定するため、後段のネットワークスタックにおけるService DNAT処理を丸ごと省略できる。今回の検証においてPodのeth0で観測された宛先の違いは、まさにこのアーキテクチャ設計の違いがパケットレベルに現れた好例といえる。[10] AIの利用について 本記事の作成にはAIを利用している。 参考文献 kubernetes/pkg/proxy/iptables/proxier.go at v1.35.0 · kubernetes/kubernetes · GitHub cilium/Documentation/network/kubernetes/kubeproxy-free.rst at v1.20.2 · cilium/cilium · GitHub cilium/Documentation/network/concepts/routing.rst at v1.20.2 · cilium/cilium · GitHub GKE Dataplane V2  |  GKE networking  |  Google Cloud Documentation Configure Azure CNI Powered by Cilium in Azure Kubernetes Service (AKS) - Azure Kubernetes Service | Microsoft Learn Assign IPs to Pods with the Amazon VPC CNI - Amazon EKS Manage kube-proxy in Amazon EKS clusters - Amazon EKS About Calico eBPF | Calico Documentation Virtual IPs and Service Proxies | Kubernetes Kubernetes Without kube-proxy — Cilium 1.20.4 documentation

How Cilium Cut Restart Times by Reusing Compiled BPF Objects How Cilium Cut Restart Times by Reusing Compiled BPF Objects If you’ve ever worked with eBPF programs in production, you know that compiling them can be a time-consuming bottleneck, especially during restarts. In Cilium, a popular networking and security project, the agent used to recompile all BPF programs from scratch on every restart. This led to delays stretching into minutes and even caused missed regeneration deadlines in continuous integration (CI) environments. In this article, we’ll explore the root cause of this inefficiency, the technical details of the caching solution introduced, and the broader engineering lesson about build artifact reuse. --- The Problem: Unnecessary Recompilation on Every Restart Cilium’s agent manages a set of BPF programs that it compiles using clang. Before the change, each time the agent restarted, it triggered a full recompilation of all BPF programs. This was painfully slow — in some cases, endpoint template compilations took up to 553 seconds. Why did this happen? Because the agent had no way to detect if the BPF source code or its build environment had changed since the last compilation. Without such detection, it always assumed it needed to recompile everything. This behavior caused two main issues: - Long startup delays: Developers and CI pipelines had to wait for lengthy recompilations even if nothing changed. - Missed deadlines: In CI, where quick feedback loops matter, these delays led to missed regeneration deadlines and slowed down testing. --- Root Cause: Lack of Input Hashing and Cache The core reason for the redundant recompilation was the absence of a mechanism to track build inputs and reuse previously compiled artifacts. The agent’s compilation process depends on multiple inputs: - BPF source files - Clang compiler configuration and environment - Go build ID (which can affect generated headers) - Generated headers and other build artifacts Before this change, none of these inputs were hashed or checked for differences. The compilation was always triggered unconditionally, causing expensive clang invocations even if the inputs were identical. --- The Solution: Input Hashing and Caching Compiled Objects The fix introduced a robust caching mechanism keyed by a hash that uniquely identifies the compilation inputs. Here’s how it works in detail: 1. Hashing all relevant inputs: The system computes a single hash that combines the contents of all BPF source files, the clang configuration and environment variables, the Go build ID, and any generated headers. This produces a unique fingerprint representing the exact compilation context. 2. Cache lookup on agent restart: When the agent starts, it calculates this hash and checks if compiled BPF objects for this hash already exist on disk. 3. Reuse or recompile: - If a matching cached object is found, the agent loads it directly, skipping the expensive compilation. - If not, it proceeds with a normal compilation and then caches the resulting objects under the new hash. This caching logic is primarily implemented in pkg/datapath/loader/cache.go. To ensure correctness, new tests were added in cache_test.go and hash_test.go verifying that the hashing accurately reflects input changes and that cached objects are reused appropriately. This approach ensures that recompilation only happens when something truly changes, drastically reducing restart times. --- Takeaway: Hashing Inputs to Cache Build Artifacts Saves Time and Resources Build systems and runtime components often depend on multiple inputs that rarely change between runs. Without tracking these inputs, systems waste time recompiling or rebuilding unnecessarily. By hashing all relevant inputs and caching build artifacts keyed by this hash, you can: - Avoid redundant expensive work - Speed up restarts and deployments - Reduce CPU and memory usage during builds - Improve developer and CI pipeline productivity In Cilium’s case, this change turned restart times from minutes down to seconds, enabling faster iteration and more reliable CI. If you’re building systems that compile code or generate artifacts dynamically, consider implementing a similar input hashing and caching mechanism. --- This improvement came from the pull request titled "loader: Reuse compiled BPF objects across agent restarts" in the cilium/cilium repository. Read the full breakdown on nossletter.tech. --- Originally published on Nossletter.

Tetragon: eBPF-based Security Observability and Runtime Enforcement (5.1k stars) What it does All text below is quoted directly from the project README. >eBPF-based Security Observability and Runtime Enforcement Getting started: - Process execution events - System call activity - I/O activity including network & file access - Process lifecycle: generating process_exec and process_exit events by default, enabling full process lifecycle observability. - Generic tracing: generating process_kprobe, process_tracepoint and process_uprobe events for more advanced and custom use cases. - network observability - credentials monitoring - privileged execution >When used in a Kubernetes environment, Tetragon is Kubernetes-aware - that is, it understands Kubernetes identities such as namespaces, pods and so on - so that security event detection can be configured in relation to individual workloads. Repository cilium/tetragon Repository stats |Metric|Value| |:-|:-| |Stars|5,133| |Forks|638| |Last release|v1.7.1 (2026-08-25)| |Languages|C 61.3%, Go 36.9%, C++ 0.4%| |Contributors|200| |AI contributor accounts (Claude / Codex / Cursor / Copilot)|None| |AI co-authored commits|None| Main contributor |Field|Value| |:-|:-| |GitHub|@olsajiri| |Commits to this repo|1,346 (top contributor)| |Followers|42| ────────────────────────────── 📌 If you want to browse more repos like this, we've organized them by category and rankings here: bestgithubrepositories.com 🗂️ We're also archiving repos where possible, so useful projects don't disappear when they're removed from GitHub.

Potential Bug: GKE anetd MTU shrinking leads to network failure (Dataplane V2) Cluster Info: - Cluster note: GKE Dataplane V2 with WireGuard inter-node encryption (UDP 51871) - Versions: Control plane and all live pools 1.35.8-gke.1796000, channel REGULAR (changed to STABLE during this incident) - Anetd Image: cni-writer v0.1.40-gke.6, cilium v1.18.7-gke1.35-gke.39 I have also posted on discuss.google.dev but wanted to get more insights from everyone here that I've been following. Please let me know if this is not the right place. Summary: My team just recovered from a downtime started from a GKE control plane upgrade last night. Our observation so far points to a suspected bug somewhere in anetd cni-writer. I'm hoping for anyone who may have experienced something similar and can give pointers to the right direction. - GKE's anetd cni-writer v0.1.40-gke.6 lowers a node's MTU by 80 bytes on every anetd restart. - Pods created earlier keep the larger MTU, and their full-size packets vanish. - We recover by recreating pods that predate anetd restart, then to be on the safer side, we replace every node to reset the MTU to baseline. - Current measures my team is taking: monitor anetd pods for any restart/delete. Every node MTU is fresh at 1460, so one anetd restart drops that node MTU to 1380 while its running pods keep 1460. We observe that mismatch to be the outage. - Logs from our apps:App -> RabbitMQ: Broker closed AMQP connections for missed heartbeats; unhandled ECONNRESET exits the processApp -> Valkey: Valkey replies dropped; ioredis waited for the kernel retry limit, then failed every queued command So, the suspected bug, step-by-step: 1. A fresh node starts at MTU 1460: NIC, CNI configuration, host-side pod interfaces, and pod eth0. WireGuard cilium_wg0 is 1380, leaving 80 bytes for encryption. 2. Every anetd pod start reruns its cni-writer init container. Control-plane upgrades roll anetd on every node, including pools with node auto-upgrade off. 3. cni-writer reads cilium_wg0 (1380) as the base MTU and writes it as the CNI pod MTU. 4. Host-side pod interfaces shrink to 1380. Running pods keep eth0 at 1460 and send packets the host side drops, with no ICMP back. Large replies stall; clients time out. 5. Cilium subtracts 80 from the smaller base: WireGuard becomes 1300. The next restart copies 1300, and so on. Logs from cni-writer anetd pods: # anetd-9j66s on 3fc1dd1c-6nrh, second anetd start on this node I1025 19:18:02.916461 cni_writer.go:378] Checking priority interface "cilium_wg0" for MTU. I1025 19:18:02.916725 cni_writer.go:383] Using MTU 1380 obtained from priority interface cilium_wg0 I1025 19:18:02.916738 cni_writer.go:518] Successfully determined and cached MTU: 1380 (from cilium_wg0) # anetd-cdmvs on 521c6876-gk4s I1025 19:04:35.214705 cni_writer.go:383] Using MTU 1220 obtained from priority interface cilium_wg0 # anetd-l8mf6 on 56b8fd8a-mdxw, control-plane upgrade roll I1025 17:23:25.846768 cni_writer.go:383] Using MTU 1300 obtained from priority interface cilium_wg0 --- Update, I also ran an LLM to check upstream repos for clues, here goes: - GoogleCloudPlatform/netd PR #199 (commit ac67942e, 2023-02-27) - Prefers cilium_wg0 over the default NIC when writing pod MTU. - Matches the policy in our cni-writer logs; legacy shell source, not proven to be our binary. - GoogleCloudPlatform/netd PR #202 (commit c50a56a0, 2023-02-28) - Refactors MTU selection into a loop, keeping WireGuard first. - The policy was kept, not reverted. - cilium/cilium commit e99bee54 (2019-05-27) - CNI configuration MTU overrides the agent's configured MTU. - Why --mtu=1460 alone does not hold the base. - cilium/cilium commit 6ae08fdc (2021-05-05) - WireGuard MTU is device MTU minus WireguardOverhead, 80 bytes. - Normal accounting becomes a loop when the base is copied from WireGuard. - GoogleCloudPlatform/netd commit 38ff1139 (2025-01-22) - Skips legacy CNI writing on unified-CNI nodes, handing it to anetd. - Marks the boundary; does not date the defect. - GoogleCloudPlatform/netd issue #33 (closed 2025-12-04) - Asks to move the shell CNI generator to Go; closed as 'CNI writer is there now'. - The Go writer exists; no public source or build mapping. cilium-agent startup: + mtu=1460 + exec /usr/bin/cilium-agent --k8s-kubeconfig-path=/var/lib/kubelet/kubeconfig --config-dir=/tmp/cilium/config-map --mtu=1460 {"level":"info","msg":"Overwriting MTU based on CNI configuration","module":"agent.datapath.mtu","mtu":1460} {"level":"info","msg":"Using configured MTU","module":"agent.datapath.mtu","mtu":1460} Reconstructed loop: | Startup | Writer's input | CNI base MTU | WireGuard MTU | | ---------------------- | ----------------------------------- | ------------ | ------------- | | Fresh node | Node NIC, no WireGuard interface yet | 1460 | 1380 | | First affected restart | Existing cilium_wg0 at 1380 | 1380 | 1300 | | Second affected restart | Existing cilium_wg0 at 1300 | 1300 | 1220 | - The writer reuses an encapsulation-adjusted MTU as the unadjusted base. Cilium accepts it and subtracts the overhead again. The observed 80-byte steps match. - We found no public cni_writer.go for v0.1.40-gke.6 and no build-to-commit mapping. The deployed Cilium revision b8f2f41d06 does not resolve publicly; upstream v1.18.7 is commit 2adae214. - The public netd MTU test disables the Cilium plugin and mocks NICs, so it never exercises the restart loop. - No public issue confirms this ratchet or a fixed GKE version.

Hubble answers 1,103 times by title, five times as an application, and 37,258 times in page bodies Hubble answers 1,103 times by title, five times as an application, and 37,258 times in page bodies A product name that also belongs to a telescope creates a measurement problem no query can fix. Hubble is the observability layer that ships with Cilium, and it is also the name of a space telescope, a film, and several unrelated services. A set of ZoomEye measurements taken on 26 October 2026 shows how far apart the three standard counts can land, and why the smallest number here is the one that describes the software. The three counts Each query ran against ZoomEye with sub_type="all" and pagesize=1, so every total describes matched assets rather than retrieved records. | Query | Matches | | --- | ---: | | app="Hubble" | 5 | | title="Hubble" | 1,103 | | http.body="Hubble" | 37,258 | | port=12000 | 2,681,323 | The ordering is inverted from the usual pattern. Normally the fingerprint outnumbers the title, and the title outnumbers nothing in particular. Here the fingerprint is essentially empty at five, while the body count runs to five digits. Why the fingerprint is nearly empty An application fingerprint is not a string match. It is a rule that recognises a product from characteristics of its response, and a rule has to be written before it can match. Cilium's Hubble exposes a gRPC service and, in its UI form, a small web interface that a cluster operator enables deliberately. Neither surface is common on the public internet, and neither has the stable response shape that fingerprint authors tend to target. Five matches is therefore a statement about the fingerprint set and about the population, in that order. It is not a count of Hubble deployments, which are mostly internal to Kubernetes clusters and reachable through port forwarding or an internal service address rather than a public one. What the 37,258 body matches actually contain The body count is where naming ambiguity dominates. The string "Hubble" appears in documentation about Cilium, in astronomy sites, in marketing pages, in blog indexes, and in any response that mentions the telescope. A body match records that the word was present; it records nothing about the responding system. port=12000 illustrates the same trap in numeric form. The port is used by Hubble's UI in some configurations and by a long list of unrelated services in others. A count of 2.68 million for a bare port query is a measurement of the port's popularity, not of the product. Any published number derived from it would be indefensible, and that is the reason the query is listed here next to its true meaning rather than used as a headline. What Hubble is worth measuring The value of the tool is in what it sees. Hubble collects flow-level data between workloads: source and destination identity, verdicts from network policy, DNS queries, and HTTP status codes for the requests it can parse. That dataset answers the question an incident responder asks first — which workload talked to which other workload, and was it allowed. The dataset is also a map of internal topology. Service names, namespaces, pod identities, and the destinations that workloads actually reach appear in flow records. A reachable Hubble interface is not a metrics endpoint; it is a live view of cluster east-west traffic. Cilium's documentation positions Hubble as an observability component with its own UI and CLI, and the security-relevant detail is that the flow data is only as sensitive as the cluster is interesting. In a cluster that holds customer data, a readable flow record describes where that data lives and how it moves. The realistic exposure and the controls The realistic exposure for Hubble is not a public port. It is an operator who enabled the UI and exposed it through the same ingress as application traffic, or a debugging session that left a port forward running on a jump host. Both are configuration decisions with short-term convenience and no obvious failure signal. The controls are correspondingly mundane. Keep the UI on an internal service address and reach it through an authenticated proxy. Treat the flow data as production telemetry with the retention and access rules that implies. And check the network policy verdicts that Hubble reports, because a cluster where every flow is allowed has an observability tool that documents its own lack of segmentation. What these numbers support The honest statement is narrow. On 26 October 2026, ZoomEye matched five assets against its Hubble application fingerprint, 1,103 pages with the product name in the title, and 37,258 responses containing the string somewhere in the body. The first number reflects a small public population and an even smaller fingerprint set. The last two reflect a word that many unrelated systems use. That distinction is the whole methodological point. A count is only useful when the reader knows what the query tested, and a product with a common name produces the widest gap between a number and its meaning. References - Cilium documentation, Hubble observability: docs.cilium.io - Hubble source repository: cilium/hubble - ZoomEye search platform: zoomeye.ai Measurement note Queries: app="Hubble", title="Hubble", http.body="Hubble", port=12000. Executed 26 October 2026, sub_type="all", pagesize=1, global scope. Counts describe matching assets, not confirmed vulnerabilities or exploitation.

Kustomize JSON patch add: fields a Helm chart does not template A chart can only set the fields its templates write. When a Kubernetes field you need is missing from them, the value you add to values.yaml is silently ignored, unless the chart has a values schema that rejects it. This post sets nine such fields from open issues on the rendered manifests, first with a Kustomize patch over helm template, then with Kapitan's mutate generator. Most of them are RFC 6902 JSON Patch operations, and mutate's json6902 takes the same operations as Kustomize, so the two versions of each patch read almost the same. Each patch was rendered from the chart version named in its section with Helm 4.3.1, Kustomize 5.8.3, and Kapitan 0.36.3 with mutate 0.1.0. In every case the Kustomize output and the mutate output set the same value and changed nothing else in the render. The mutate output was installed on kind clusters on Kubernetes 1.37.0 with three worker nodes, and each Kustomize output's patched object passed a server-side dry run there. Charts from HTTP repositories matched the sha256 their index.yaml lists. The guide Patch a Helm chart compares the places such a patch can live. The class for each chart pins its version, like this one for Vault, and the target adds mutate as a second compile entry on the chart's output_path, as the guide shows: parameters: kapitan: dependencies: - type: helm source: https://helm.releases.hashicorp.com chart_name: vault version: 0.34.1 output_path: charts/vault compile: - input_type: helm input_paths: [charts/vault] output_path: vault helm_values_files: [values/vault.yaml] helm_params: name: vault namespace: vault Vault Helm chart: Service appProtocol {#vault-service-appprotocol} hashicorp/vault-helm#1108, opened in April 2025 and at 9 thumbs up, asks for appProtocol: https on the Vault Service. Without it, Gateway API implementations such as GKE's Gateway controller talk plain HTTP to a Vault that serves HTTPS, according to the issue and the GKE documentation it links. No maintainer has replied. Chart 0.34.1, released on 13 August 2026, writes the ports in server-service.yaml without it: ports: - name: {{ include "vault.scheme" . }} port: {{ .Values.server.service.port }} targetPort: {{ .Values.server.service.targetPort }} {{- if and (.Values.server.service.nodePort) (eq (.Values.server.service.type | toString) "NodePort") }} nodePort: {{ .Values.server.service.nodePort }} {{- end }} - name: https-internal port: 8201 targetPort: 8201 The port takes its name from the scheme, https once global.tlsDisable is false, which this values file sets: global: tlsDisable: false resources: - helm.yaml patches: - target: kind: Service name: vault patch: |- - op: test path: /spec/ports/0/name value: https - op: add path: /spec/ports/0/appProtocol value: https parameters: mutations: vault-app-protocol: match: {apiVersion: v1, kind: Service, name: vault} json6902: - {op: test, path: /spec/ports/0/name, value: https} - {op: add, path: /spec/ports/0/appProtocol, value: https} test makes both patches refuse to mark a plain HTTP port as https. With the chart's default tlsDisable: true the first port is named http, and both stopped: Error: testing value /spec/ports/0/name failed: test failed mutate: mutation vault-app-protocol: vault/templates/server-service.yaml: json6902 test of '/spec/ports/0/name' failed The Kustomize target name: vault matched the Service vault only, not vault-internal or vault-agent-injector-svc. The test applied the chart's Services to kind, without the Vault server, and the stored Service carries the field: https 8200 appProtocol=https CloudNativePG cluster chart: topologySpreadConstraints {#cloudnative-pg-topology-spread-constraints} cloudnative-pg/charts#817, opened in February 2026, asks for topologySpreadConstraints on the Cluster that the cluster chart renders. The CloudNativePG operator supports the field, but the chart's cluster.yaml only templates affinity. Pull request #818, opened the same day, adds it. It has had no review and now conflicts with the chart's main branch. In its thread, a user pointed out that the label cnpg.io/cluster alone also matches pooler and Job pods, and suggested adding cnpg.io/podRole: instance to the selector. On node-local storage that selector has a catch. CloudNativePG creates each instance's volume through an initdb or join Job, whose pod gets the instance's constraints but not the label cnpg.io/podRole, so the constraint does not count it. kind's local-path volumes are bound to the node of the first pod that mounts them. In a run with that selector, the join Job of the third instance landed on the node of the second, the volume followed it, and the third instance stayed Pending: 0/4 nodes are available: 1 node(s) didn't match pod topology spread constraints, 1 node(s) had untolerated taint(s), 2 node(s) didn't match PersistentVolume's node affinity. preemption: 0/4 nodes are available: 1 No preemption victims found for incoming pod, 3 Preemption is not helpful for scheduling. A selector that counts the Job pods too and leaves out only poolers spread all three instances. Until a chart release has the value, chart 0.9.0 takes this patch: resources: - helm.yaml patches: - target: group: postgresql.cnpg.io kind: Cluster name: pg-cluster patch: |- - op: add path: /spec/topologySpreadConstraints value: - maxSkew: 1 topologyKey: kubernetes.io/hostname whenUnsatisfiable: DoNotSchedule labelSelector: matchLabels: cnpg.io/cluster: pg-cluster matchExpressions: - key: cnpg.io/podRole operator: NotIn values: [pooler] parameters: mutations: cluster-topology-spread: match: {apiVersion: postgresql.cnpg.io/v1, kind: Cluster, name: pg-cluster} json6902: - op: add path: /spec/topologySpreadConstraints value: - maxSkew: 1 topologyKey: kubernetes.io/hostname whenUnsatisfiable: DoNotSchedule labelSelector: matchLabels: cnpg.io/cluster: pg-cluster matchExpressions: - key: cnpg.io/podRole operator: NotIn values: [pooler] With operator chart 0.29.1 on the cluster and the release named pg, the Cluster pg-cluster became Ready on a fresh cluster. Its three instance pods carry the constraint and run on three different workers: pg-cluster-1 node=set-fields-worker2 ready=true spread=kubernetes.io/hostname pg-cluster-2 node=set-fields-worker ready=true spread=kubernetes.io/hostname pg-cluster-3 node=set-fields-worker3 ready=true spread=kubernetes.io/hostname Harbor Helm chart: annotations on the core Deployment {#harbor-core-deployment-annotations} goharbor/harbor-helm#2221, opened in August 2025, asks for annotations and labels on the components' Deployments, beyond the pod annotations the chart takes. One commenter needs an annotation on harbor-core for Reloader. A collaborator suggested a Helm hook, which the reporter cannot use with Argo CD. Three pull requests for the values were closed unmerged by the stale bot. In chart 1.19.4, released on 16 October 2026, core-dpl.yaml writes no annotations into the Deployment's metadata, and core.podAnnotations, whose comment in values.yaml reads "Additional deployment annotations", goes to the pod template only: metadata: name: {{ template "harbor.core" . }} namespace: {{ .Release.Namespace | quote }} labels: {{ include "harbor.labels" . | indent 4 }} component: core app.kubernetes.io/component: core An annotation is a key in a map, which a strategic merge patch and mutate's merge add without touching other keys: resources: - helm.yaml patches: - patch: |- apiVersion: apps/v1 kind: Deployment metadata: name: harbor-core namespace: harbor annotations: reloader.stakater.com/auto: "true" parameters: mutations: harbor-core-annotations: match: {apiVersion: apps/v1, kind: Deployment, name: harbor-core} merge: metadata: annotations: reloader.stakater.com/auto: "true" Harbor generates secrets on every render, so two renders differ in their Secrets and in the Deployments' checksum annotations. The comparison of the renders left those out. On kind, every Harbor Deployment became Available, and the stored harbor-core carries the annotation: harbor-core annotations: {"deployment.kubernetes.io/revision":"1","reloader.stakater.com/auto":"true"} ECK operator chart: ServiceMonitor labels and interval {#eck-operator-servicemonitor} elastic/cloud-on-k8s#7717, opened in April 2024, asks for labels and the interval on the operator's ServiceMonitor, because a Prometheus from kube-prometheus-stack selects ServiceMonitors by its release label. A commenter confirmed in August 2026 that 3.5.0 ignores serviceMonitor.labels and keeps a second ServiceMonitor by hand. The chart has no values schema, so the key is silently dropped. In 3.5.0, released on 4 August 2026, service-monitor.yaml renders the ServiceMonitor only with metrics in secure mode, with the chart's own labels and a fixed interval: {{- if and .Values.config.metrics.secureMode.enabled .Values.serviceMonitor.enabled }} apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: {{ include "eck-operator.fullname" . }} namespace: {{ ternary .Values.serviceMonitor.namespace .Release.Namespace (not (and (.Values.serviceMonitor) (empty .Values.serviceMonitor.namespace))) }} labels: {{- include "eck-operator.labels" . | nindent 4 }} spec: namespaceSelector: matchNames: - {{ .Release.Namespace }} selector: matchLabels: app.kubernetes.io/name: {{ include "eck-operator.name" . }}-metrics-service app.kubernetes.io/instance: {{ .Release.Name }} endpoints: - port: https path: /metrics scheme: https interval: 30s These values turn the metrics endpoint and secure mode on: config: metrics: port: "8080" secureMode: enabled: true resources: - helm.yaml patches: - target: group: monitoring.coreos.com kind: ServiceMonitor name: elastic-operator patch: |- - op: add path: /metadata/labels/release value: kube-prometheus-stack - op: test path: /spec/endpoints/0/interval value: 30s - op: replace path: /spec/endpoints/0/interval value: 60s parameters: mutations: eck-servicemonitor: match: {apiVersion: monitoring.coreos.com/v1, kind: ServiceMonitor, name: elastic-operator} json6902: - {op: add, path: /metadata/labels/release, value: kube-prometheus-stack} - {op: test, path: /spec/endpoints/0/interval, value: 30s} - {op: replace, path: /spec/endpoints/0/interval, value: 60s} add on /metadata/labels/release adds one key to the existing labels. The test of 30s stops the patch if a release changes the interval. On kind, with the ServiceMonitor CRD of kube-prometheus-stack 92.11.0, the operator became ready and the stored ServiceMonitor has both values: release=kube-prometheus-stack interval=60s Temporal Helm chart: Service trafficDistribution {#temporal-service-trafficdistribution} temporalio/helm-charts#897, opened in April 2026, asks for spec.trafficDistribution on the server Services, at least the frontend. Its author sets the field today with a separate server-side apply after each install. No maintainer has replied. Chart 1.7.0, released on 12 September 2026, writes type, ports and selector in server-service.yaml and has no values schema, so server.frontend.service.trafficDistribution is ignored. The chart needs a database to render, and this values file names one that the test never contacted: server: config: persistence: datastores: default: sql: pluginName: postgres12 databaseName: temporal connectAddr: db.example.internal:5432 visibility: sql: pluginName: postgres12 databaseName: temporal_visibility connectAddr: db.example.internal:5432 resources: - helm.yaml patches: - target: kind: Service name: temporal-frontend patch: |- - op: add path: /spec/trafficDistribution value: PreferSameZone parameters: mutations: temporal-traffic-distribution: match: {apiVersion: v1, kind: Service, name: temporal-frontend} json6902: - {op: add, path: /spec/trafficDistribution, value: PreferSameZone} PreferClose, the value in the issue, is the older name for PreferSameZone, and the Kubernetes documentation lists it as deprecated. The field is generally available since Kubernetes 1.33. The value PreferSameZone is enabled by default from 1.34, so a 1.33 cluster needs PreferClose. Without a database the Temporal servers cannot start, so the test applied the chart's Services only, and the API server stored the field: temporal-frontend trafficDistribution: PreferSameZone MetalLB Helm chart: the speaker without host networking {#metallb-speaker-hostnetwork} metallb/metallb#3005, opened in April 2026, asks for hostNetwork: false on the speaker DaemonSet. The reporter runs FRR on every node next to MetalLB, and FRR does not accept iBGP peering with an address in its own network namespace, so the speaker has to run in the pod network and peer with the node's FRR. Another user peers the speakers with Calico the same way. The maintainers asked for the use case. Pull request #3099 adds a speaker.hostNetwork value, and on it the maintainer fedepaol wrote "I think it makes sense to document this, mentioning that l2 won't be able to work." It is not merged. Chart 0.16.1 hard-codes the field in speaker.yaml: hostNetwork: true By default the chart runs BGP in a separate frr-k8s DaemonSet, which keeps host networking whatever the speaker does. The test therefore uses MetalLB's native mode, in which the speaker itself speaks BGP: frrk8s: enabled: false speaker: frr: enabled: false Flipping hostNetwork alone is not enough in 0.16.1. The speaker serves its health endpoint on 127.0.0.1 inside its own network namespace (speaker/main.go), and the chart's probes ask for 127.0.0.1, which the kubelet resolves in the node's network namespace. With only the field changed, no speaker became ready on kind, and every probe was refused: Readiness probe failed: Get "http://127.0.0.1:17472/readyz": dial tcp 127.0.0.1:17472: connect: connection refused Liveness probe failed: Get "http://127.0.0.1:17472/healthz": dial tcp 127.0.0.1:17472: connect: connection refused Pull request #3099 changes the field only, so it would leave the same probes in place. The patch below also removes both probes. The speaker then has no health check, and the kubelet restarts it only when the process exits: resources: - helm.yaml patches: - target: kind: DaemonSet name: metallb-speaker patch: |- - op: test path: /spec/template/spec/hostNetwork value: true - op: replace path: /spec/template/spec/hostNetwork value: false - op: test path: /spec/template/spec/containers/0/name value: speaker - op: remove path: /spec/template/spec/containers/0/livenessProbe - op: remove path: /spec/template/spec/containers/0/readinessProbe parameters: mutations: metallb-speaker-pod-network: match: {apiVersion: apps/v1, kind: DaemonSet, name: metallb-speaker} json6902: - {op: test, path: /spec/template/spec/hostNetwork, value: true} - {op: replace, path: /spec/template/spec/hostNetwork, value: false} - {op: test, path: /spec/template/spec/containers/0/name, value: speaker} - {op: remove, path: /spec/template/spec/containers/0/livenessProbe} - {op: remove, path: /spec/template/spec/containers/0/readinessProbe} On kind, a speaker ran on each of the four nodes with its own pod IP. Without a readiness probe, ready means that the container runs: metallb-speaker-6nsjd hostNetwork=false pod IP is the node IP: no ready=true restarts=0 metallb-speaker-6rr28 hostNetwork=false pod IP is the node IP: no ready=true restarts=0 metallb-speaker-8z7rr hostNetwork=false pod IP is the node IP: no ready=true restarts=0 metallb-speaker-hc622 hostNetwork=false pod IP is the node IP: no ready=true restarts=0 L2 mode answers ARP requests for the service address on the node's interface, which a speaker in the pod network does not have. With an L2 pool from the kind network and an nginx LoadBalancer Service, the test compared both settings from the host: speaker in the pod network: the LoadBalancer address from the host: not reachable speaker in the host network (plain chart): the LoadBalancer address from the host: reachable The test did not set up a BGP peer, so it shows that the speakers run in the pod network, not that they announce routes over BGP from there. Consul Helm chart: tolerations on the hook Jobs {#consul-hook-job-tolerations} hashicorp/consul-k8s#4469, opened in January 2025 and at 7 thumbs up, asks for tolerations and a nodeSelector on the Job that generates the gossip encryption key. When every node carries a taint, that Job and tls-init stay Pending, and the thread names a third Job, tls-init-cleanup. The workaround offered there is to helm pull the chart and edit its templates, which a commenter who installs from HashiCorp's repository cannot do. Pull request #4625, opened in June 2025, adds the values and has had no review. In chart 2.0.4, released on 11 September 2026, none of the three Jobs sets either field in its pod spec, and 2.1.0-rc2 does not either. All three are Helm hooks, as tls-init-job.yaml shows for tls-init: annotations: "helm.sh/hook": pre-install,pre-upgrade "helm.sh/hook-weight": "1" "helm.sh/hook-delete-policy": hook-succeeded,before-hook-creation The gossip key Job runs at the same hooks, tls-init-cleanup at pre-delete. These values turn on TLS and the gossip key, and global.name gives the Jobs the names from the issue: global: name: consul tls: enabled: true gossipEncryption: autoGenerate: true The test tainted and labelled all three workers with example.com/purpose=infra and set the chart's own tolerations and nodeSelector values for the server, the connect injector and the webhook cert manager. global.acls.tolerations reaches the consul-gateway-resources Job, which a commenter also mentions, and the two gateway cleanup Jobs, even with ACLs off. Those three have no nodeSelector value. With the plain chart, both pre-install Jobs stayed Pending: consul-tls-init-fh7nh: 0/4 nodes are available: 4 node(s) had untolerated taint(s). preemption: 0/4 nodes are available: 4 Preemption is not helpful for scheduling. The three Jobs carry app: consul whatever the release is called, and component with their own name, so one Kustomize target with a set-based selector matches all three: resources: - helm.yaml patches: - target: group: batch kind: Job labelSelector: app=consul,component in (tls-init,tls-init-cleanup,gossip-encryption-autogenerate) patch: |- - op: add path: /spec/template/spec/tolerations value: - key: example.com/purpose operator: Equal value: infra effect: NoSchedule - op: add path: /spec/template/spec/nodeSelector value: example.com/purpose: infra mutate's match takes one value per key, so each Job gets a mutation of its own, and the three share the operations through an inventory key: parameters: hook_scheduling: - op: add path: /spec/template/spec/tolerations value: - {key: example.com/purpose, operator: Equal, value: infra, effect: NoSchedule} - op: add path: /spec/template/spec/nodeSelector value: example.com/purpose: infra mutations: consul-tls-init: match: {apiVersion: batch/v1, kind: Job, labels: {app: consul, component: tls-init}} json6902: ${hook_scheduling} consul-tls-init-cleanup: match: {apiVersion: batch/v1, kind: Job, labels: {app: consul, component: tls-init-cleanup}} json6902: ${hook_scheduling} consul-gossip-encryption-autogenerate: match: {apiVersion: batch/v1, kind: Job, labels: {app: consul, component: gossip-encryption-autogenerate}} json6902: ${hook_scheduling} Without a cluster to ask, helm template renders the chart's two PodDisruptionBudgets as policy/v1beta1, which Kubernetes 1.25 removed, and the first server-side dry run refused them. Both renders therefore pass --api-versions policy/v1/PodDisruptionBudget, which Kapitan takes as api_versions in helm_params. Helm hooks with the patched Jobs kubectl apply of the whole render would also start tls-init-cleanup, which deletes the Secrets consul-ca-cert, consul-ca-key and consul-server-cert. The test applied the mutate output in the order of helm install: the pre-install hooks, the templates, then the post-install Job. The server and the connect injector became ready. The pre-delete hooks, applied last as helm uninstall would run them, removed the three Secrets. Every Job completed on a tainted worker: consul-tls-init node=cac-consul-worker2 taint=example.com/purpose=infra:NoSchedule complete=True consul-gossip-encryption-autogenerate node=cac-consul-worker taint=example.com/purpose=infra:NoSchedule complete=True consul-tls-init-cleanup node=cac-consul-worker2 taint=example.com/purpose=infra:NoSchedule complete=True For helm install itself, Helm 4 passes hooks to the post-renderer together with the templates. Helm 3 never did. With Helm 3.22.1 and the same Kustomize patch as a post-renderer script, helm template printed the three Jobs without the fields. A Helm 4 post-renderer is a plugin, and this one runs the Kustomize patch from the directory it is given: name: kustomize-pr version: 0.1.0 type: postrenderer/v1 apiVersion: v1 runtime: subprocess runtimeConfig: platformCommand: - command: "${HELM_PLUGIN_DIR}/kustomize.sh" #!/usr/bin/env bash # stdin: Helm's render, hooks included. $1: the directory with kustomization.yaml. set -euo pipefail tmp=$(mktemp -d) trap 'rm -rf "$tmp"' EXIT cp "$1/kustomization.yaml" "$tmp/" cat > "$tmp/helm.yaml" kustomize build "$tmp" helm plugin install ./kustomize-pr helm install consul consul --repo https://helm.releases.hashicorp.com --version 2.0.4 \ -n consul -f consul-values.yaml --post-renderer kustomize-pr --post-renderer-args "$PWD" --wait The plugin's output equalled the Kustomize build. With Helm 4.3.1 on kind, the release installed, and helm uninstall ran tls-init-cleanup, each Job on a tainted worker: Successfully assigned consul/consul-gossip-encryption-autogenerate-jdm6k to cac-consul-worker Successfully assigned consul/consul-tls-init-qt7ql to cac-consul-worker3 Successfully assigned consul/consul-tls-init-cleanup-wkt87 to cac-consul-worker APISIX ingress controller chart: pod labels {#apisix-ingress-controller-pod-labels} apache/apisix-helm-chart#978, opened in June 2026 and at 11 thumbs up, asks for a podLabels value in the apisix-ingress-controller chart, so that the controller's pods can carry labels for observability, scheduling or policies. Nobody has commented. The apisix chart in the same repository got podLabels through pull request #965, merged on 30 September 2026. The controller chart takes deployment.podAnnotations, but in 1.4.0, released on 17 September 2026, deployment.yaml gives the pod template the selector labels only: labels: {{- include "apisix-ingress-controller-manager.selectorLabels" . | nindent 8 }} The chart's labelsOverride replaces these selector labels, so a label set there also lands in the Deployment's spec.selector and in the selectors of both Services. Over the installed chart, the API server refused the change: The Deployment "apisix-ingress-controller" is invalid: spec.selector: Invalid value: {"matchLabels":{"app.kubernetes.io/instance":"apisix-ingress-controller","app.kubernetes.io/name":"apisix-ingress-controller","team":"platform"}}: field is immutable add on a path below the pod template's labels sets one label there and leaves the selector alone: resources: - helm.yaml patches: - target: group: apps kind: Deployment name: apisix-ingress-controller patch: |- - op: add path: /spec/template/metadata/labels/team value: platform parameters: mutations: apisix-pod-labels: match: {apiVersion: apps/v1, kind: Deployment, name: apisix-ingress-controller} json6902: - {op: add, path: /spec/template/metadata/labels/team, value: platform} A key with a slash such as example.com/team goes into the path as example.com~1team, and both tools then set the label example.com/team. The chart generates its webhook certificate on every render, so the comparison of the renders left out that Secret and the webhook configuration's caBundle. helm template renders the chart's CRDs only with --include-crds, which Kapitan's helm input passes by default. On kind, the controller became ready, and its pod carries the label: apisix-ingress-controller-85b5bb8fb4-9m4kc labels: {"app.kubernetes.io/instance":"apisix-ingress-controller","app.kubernetes.io/name":"apisix-ingress-controller","pod-template-hash":"85b5bb8fb4","team":"platform"} Cilium Helm chart: ipFamilyPolicy on cilium-ingress {#cilium-ingress-ipfamilypolicy} cilium/cilium#27003, opened in July 2023, asks for ipFamilyPolicy and ipFamilies on the Service of Cilium's ingress controller. Without them the Service gets the cluster's primary family only, so on a dual-stack cluster the Ingresses answer on IPv4 or on IPv6, not on both. The maintainer youngnick wrote in July 2023 that LB Service customisation was planned for that cycle, and pinned the issue in December 2023. Chart 1.20.3, released on 15 October 2026, renders the Service with its ports, type, loadBalancerClass, allocateLoadBalancerNodePorts, loadBalancerIP and externalTrafficPolicy, and 1.21.0-pre.3 renders the same. The chart has this Service in shared mode only. In dedicated mode the operator creates a Service per Ingress, which a patch of the chart does not reach. The ingress controller needs Cilium's kube-proxy replacement, so the test ran Cilium as the CNI of a dual-stack kind cluster without kube-proxy: kubeProxyReplacement: true k8sServiceHost: cac-cilium-control-plane k8sServicePort: 6443 ipam: mode: kubernetes ipv6: enabled: true ingressController: enabled: true loadbalancerMode: shared resources: - helm.yaml patches: - target: kind: Service name: cilium-ingress patch: |- - op: add path: /spec/ipFamilyPolicy value: RequireDualStack - op: add path: /spec/ipFamilies value: [IPv4, IPv6] parameters: mutations: cilium-ingress-dual-stack: match: {apiVersion: v1, kind: Service, name: cilium-ingress} json6902: - {op: add, path: /spec/ipFamilyPolicy, value: RequireDualStack} - {op: add, path: /spec/ipFamilies, value: [IPv4, IPv6]} The chart generates the Hubble certificates on every render, so the comparison of the renders left out those Secrets. The plain chart gave the Service one IPv4 cluster IP. The mutate output, applied over it, changed the existing Service in place, and 60 seconds later the operator had not reverted it: ipFamilyPolicy=SingleStack ipFamilies=["IPv4"] clusterIPs=["10.96.179.251"] loadBalancer=198.51.100.0 ipFamilyPolicy=RequireDualStack ipFamilies=["IPv4","IPv6"] clusterIPs=["10.96.179.251","fd00:10:96::b12c"] loadBalancer=198.51.100.0 2001:db8:: LB-IPAM had a pool with one block of each family from the documentation ranges and gave the Service an address of each. An Ingress for an echo backend then answered on all four addresses from a node, with curl in the node's network namespace: 10.96.179.251 from node: 200, from pod: 000 fd00:10:96::b12c from node: 200, from pod: 200 198.51.100.0 from node: 200, from pod: 000 2001:db8:: from node: 200, from pod: 200 From a pod, requests over IPv4 timed out on this test host, with the plain chart as well and on an IPv4-only kind cluster too. The patch does not cause it, and the test does not cover that path. ipFamilies with both families needs a dual-stack cluster, whatever the policy. On an IPv4-only kind cluster the server-side dry run refused the patched Service: * spec.ipFamilyPolicy: Invalid value: "RequireDualStack": this cluster is not configured for dual-stack services * spec.ipFamilies[1]: Invalid value: "IPv6": not configured on this cluster With PreferDualStack it still refused ipFamilies[1]. PreferDualStack without ipFamilies passed there and got IPv4 only. Originally published at kadet-hub.org.

network policyの判断を、IP addressだけでなくidentityにも。 CiliumはeBPF-based dataplaneを持つnetworking、observability、security solution。identity-based security modelでL3-L7 policyを適用し、network addressingから分離できます。 cilium/cilium

32 bytes verified, 40 didn't: parsing HTTP inside the eBPF verifier's budget --- title: "32 bytes verified, 40 didn't: parsing HTTP inside the eBPF verifier's budget" published: true description: "A classic eBPF loop verified over 32 bytes and failed at 40. bpf_loop() and open-coded iterators parsed all 256 bytes in a few hundred verifier instructions." tags: ebpf, linux, go, observability series: Building a zero-instrumentation eBPF agent --- My eBPF agent traces HTTPS traffic without touching the application: uprobes on OpenSSL's SSL_write/SSL_read and on Go's crypto/tls copy the plaintext out of the process before encryption, and the agent turns it into OpenTelemetry spans. The kernel side copies a bounded prefix of each payload — 256 bytes — into a ring buffer, and all the HTTP parsing happens in userspace. The obvious question is why. Parsing in the kernel would mean shipping less data and filtering earlier. "The verifier won't allow it" is the usual answer, but I wanted a number, not folklore. So I wrote the in-kernel parser five different ways and recorded exactly what the verifier said about each. The answer turned out to be more interesting than "no". Repo: sAchin-680/eBPF-observability-agent The setup Every attempt is a real uprobe program that reads up to 256 bytes of the buffer passed to SSL_write and tries to find Host: headers. None is ever attached — they exist to be loaded, and the interesting ones are the loads that fail. Each program is loaded on its own with cilium/ebpf, because loading them together would stop at the first rejection and hide the rest. The test records the outcome, the time, and — new for this post — the verifier's own count of instructions it processed. I originally ran this on kernel 6.8. For this post I re-ran everything on 6.18. Every accept/reject outcome was identical. !Terminal recording on kernel 6.18: the classic per-header parse is accepted at 32 bytes and rejected from 40 bytes up, while the bpf_loop and bpf_for versions are accepted at 256 bytes, then both pass every correctness check Attempt 1: scan until the data says stop The natural way to find the end of a header block: while (data[i] != '\0') { if (data[i] == '\n') lines++; i++; } Rejected: invalid read from stack R3 off=0 size=1 What's interesting is how it fails. Since kernel 5.3 the verifier accepts bounded loops by simulating every iteration. It did exactly that here: it followed i all the way to the end of the 256-byte buffer and refused the read that walks off it. It isn't objecting to the loop. It found the overrun. Attempt 2: bound it by the caller's length for (int i = 0; i < num; i++) { ... } Rejected the same way. The verifier's register state shows why: R6_r=Pscalar(smin=umin=250, smax=umax=0x7fffffff, ...) invalid read from stack R2 off=0 size=1 umax=0x7fffffff. The length comes from userspace, so the verifier knows nothing about it and has to assume two billion. A bound the verifier can't see is no bound at all. Attempt 3: bound it by the buffer for (int i = 0; i < BUF; i++) { if (data[i] == '\n') lines++; } Accepted. 22 instructions, and the verifier processed 6,389 instructions to prove it safe — walking all 256 iterations. But it only counts line breaks. Real parsing needs per-line work. Attempt 4: actually parse — and hit the wall The smallest thing that resembles parsing headers: at each \r\n, check whether the line that just ended starts with Host:, then remember where the next line begins. for (int i = 0; i + 1 < size; i++) { if (data[i] != '\r' || data[i + 1] != '\n') continue; if (line_start + 5 < size && data[line_start] == 'H' && data[line_start + 1] == 'o' && data[line_start + 2] == 's' && data[line_start + 3] == 't' && data[line_start + 4] == ':') matches++; line_start = i + 2; } I compiled the identical code at different buffer sizes. Every version is 76 instructions — the compiler doesn't unroll it. The difference is entirely in how much work the verifier does: | Buffer | Outcome | Instructions processed by the verifier | | ---: | :--- | ---: | | 32 bytes | accepted | 158,806 | | 40 bytes | rejected, E2BIG | over 1,000,000 | | 48 bytes | rejected, E2BIG | over 1,000,000 | | 56 bytes | rejected, E2BIG | over 1,000,000 | | 64 bytes | rejected, E2BIG | over 1,000,000 | | 256 bytes | rejected, E2BIG | over 1,000,000 | E2BIG ("argument list too long") is what the verifier returns when it exhausts its budget of one million processed instructions. 32 bytes uses 16% of the budget. 40 bytes exceeds it. A 25% bigger buffer costs more than six times the verification work. A Host header alone is often longer than 32 bytes, so this isn't a tuning problem — this shape of parser simply can't cover a real request. Why it explodes The verifier walks every iteration and every branch. Normally it keeps the cost down by pruning: when it reaches an instruction in a state equivalent to one it has already proven safe, it stops exploring. line_start defeats that. It's used as a memory offset, so the verifier must track its exact value — and every path through the data sets it differently, depending on where the \r\n pairs fell. States that differ only in line_start can't be merged, so the number of live paths multiplies with each byte. Counting line breaks in attempt 3 has no such variable, which is why it stays cheap over 256 bytes. !Diagram: in the classic loop the verifier explores a tree of paths where each line break sets line_start to a different value, so paths multiply; with bpf_loop the callback is verified once for any index, costing 149 instructions for 256 bytes An aside: asking for the log changed the answer On the 6.8 machine, requesting the verifier's log for the 64-byte program changed the reported error from E2BIG to EINVAL, and verification went from 0.3 s to 5.7 s. Worse: the loader retries with a progressively bigger log buffer while the log is truncated, and no buffer was ever big enough. The test process reached 5.3 GB resident on a 5.9 GB machine and was OOM-killed. Verification itself took 280 ms. Only the log was unbounded. The experiment now loads the size sweep with logging disabled and asks only for the verifier's statistics, which are a few lines long. Attempt 5: the two features built for exactly this My original conclusion was "in-kernel parsing is ruled out." That conclusion was only about the classic loop. Newer kernels have two features designed for loops the verifier can't afford to walk: - bpf_loop() (kernel 5.17): a helper that calls a callback N times. The verifier checks the callback once, with the index known only to be in [0, N). - Open-coded iterators (bpf_for, kernel 6.4): an ordinary-looking loop that the verifier checks by converging on a stable state, rather than by walking every iteration. Here's the same parse with bpf_loop(), over the full 256 bytes: static long parse_step(__u64 index, void *arg) { struct parse_ctx *c = arg; __u64 i = index; __u64 ls; if (i >= BUF - 1) return 1; if (c->data[i] != '\r' || c->data[i + 1] != '\n') return 0; ls = c->line_start; if (ls < BUF - 5 && c->data[ls] == 'H' && c->data[ls + 1] == 'o' && c->data[ls + 2] == 's' && c->data[ls + 3] == 't' && c->data[ls + 4] == ':') c->matches++; c->line_start = i + 2; return 0; } /* in the uprobe: */ bpf_loop(BUF - 1, parse_step, &c, 0); First, the verifier and the compiler disagreed My first versions of both were rejected instantly. Not because of the budget — because of how I'd written the bounds checks: invalid unbounded variable-offset read from stack R5 invalid variable-offset read from stack R2 var_off=(0x0; 0x1ff) off=-256 size=1 Two classic mistakes: 1. if (i + 1 >= BUF) let the compiler test a copy of the index, so the register actually used as the offset was never bounded in the verifier's eyes. Testing i >= BUF - 1 directly bounds the register that's used. 2. if (line_start + 5 < BUF) looks like it proves line_start < 251. In 32-bit arithmetic it doesn't: the addition could wrap, so the verifier learns nothing about line_start from it. line_start < BUF - 5 says the same thing in a form it can use. The agent's production capture code already follows the same rule — it masks the copy length with n &= (MAX_DATA - 1) rather than relying on an earlier if, because the mask states the bound in a form the verifier accepts on every path. Bounds have to be written for the verifier, not just for the reader. Then: 256 bytes, a few hundred instructions | Version | Buffer | Outcome | Instructions processed | | :--- | ---: | :--- | ---: | | classic bounded loop | 32 bytes | accepted | 158,806 | | classic bounded loop | 40 bytes | rejected | over 1,000,000 | | bpf_loop() | 256 bytes | accepted | 149 | | bpf_for iterator | 256 bytes | accepted | 351 | !Log-scale bar chart of verifier instructions processed: classic loop 32 bytes 158,806 accepted; classic loop 40 and 256 bytes over the 1,000,000 budget; bpf_loop 256 bytes 149; bpf_for 256 bytes 351 Eight times the buffer of the largest classic loop that verifies, for 0.1–0.2% of the verification work. Loading isn't the same as being right, so I also checked they compute the correct answer. Uprobe programs can't be test-run directly, so the identical parse is wrapped in a BPF_PROG_TYPE_SYSCALL program and run on real requests with BPF_PROG_TEST_RUN: run_bpf_loop one Host header len= 50 matches=1 want=1 ok run_bpf_loop no Host header len= 51 matches=0 want=0 ok run_bpf_loop two Host headers len= 42 matches=2 want=2 ok run_bpf_loop Host past byte 200 len=224 matches=1 want=1 ok run_bpf_for one Host header len= 50 matches=1 want=1 ok run_bpf_for no Host header len= 51 matches=0 want=0 ok run_bpf_for two Host headers len= 42 matches=2 want=2 ok run_bpf_for Host past byte 200 len=224 matches=1 want=1 ok That last case — a Host header past byte 200 — is one the classic loop couldn't even be verified over. So why does the agent still parse in userspace? The verifier is no longer the deciding reason on a modern kernel. These are: The kernel floor. The agent ships as one CO-RE binary, verified by checksum on kernels 5.15 and 6.8. bpf_loop() needs 5.17 and iterators need 6.4. Ubuntu 22.04 runs 5.15. Using either would mean dropping supported kernels or maintaining two kernel-side parsers. Where the work runs. A uprobe handler runs synchronously in the traced application's own thread, before SSL_write continues. Copying 256 bytes and returning keeps the cost on the application's request path as small as possible. Parsing in userspace moves that work onto the agent's own threads. What the kernel can see. A header block can be split across several SSL_write calls, and the kernel program sees one call at a time. Userspace sees the whole stream and can correlate requests with responses. What I can test. The userspace parser is ordinary Go with ordinary unit tests. The kernel parser needs root, a specific kernel, and a test harness just to run. The cost of this design is explicit: the captured prefix is 256 bytes, so headers beyond it are truncated — and truncation is reported on every event rather than hidden. If the agent ever drops 5.15, an in-kernel pre-filter with bpf_loop() — say, only capture requests to certain hosts — becomes a reasonable option rather than an impossible one. That's a different conclusion from the one I started with. What the evidence does and doesn't establish | Evidence | Establishes | Does not establish | | --- | --- | --- | | Classic loop: 32 bytes accepted, 40+ rejected | This parser shape exceeds the 1M-instruction budget above 32 bytes, on 6.8 and 6.18 | That every classic-loop parser hits the same wall at the same size | | bpf_loop / bpf_for accepted at 256 bytes | These two loop forms verify cheaply for this parse on 6.18 | Behavior on kernels I haven't tested between 5.17/6.4 and 6.18 | | Test-run results | The parse returns the right count for these inputs | That it handles every valid HTTP/1.1 request | | Verifier statistics | How much work the verifier did for these programs on this kernel | Run-time cost of the programs once attached | What I'd take from this 1. Measure the constraint before designing around it. "The verifier won't allow it" was true for one loop shape on one kind of kernel, and false for another. 2. Watch the verifier's instruction count, not just pass/fail. 158,806 of 1,000,000 at 32 bytes was the warning that 40 bytes would fail. 3. Write bounds for the verifier. Check the register you index with, avoid sums that can wrap, and mask when you can. 4. Re-test old conclusions when the platform moves. I'd written "in-kernel parsing is ruled out" into the project's docs. On a 6.x kernel, that's no longer true. Reproduce it sudo go test ./test/verifier/ -v The classic-loop experiment is in test/verifier/headers.bpf.c, and the write-up with the raw verifier output is in docs/verifier/. The bpf_loop/bpf_for versions, the statistics test and the test-run checks are in test/verifier/extra.bpf.c. {% github sAchin-680/eBPF-observability-agent %} Next in this series: why benchmarking the agent against an idle machine made it look faster than running no agent at all — and the control arm that fixed it. --- I'm Sachin, an SRE / platform engineer working on observability, eBPF and Kubernetes — currently open to new roles. LinkedIn · GitHub