kedacore

Kustomize patch env variable with valueFrom in a Helm chart Many charts take extra environment variables as a map of names to values and render each entry as name and value. A variable that comes from a Secret, a ConfigMap or the downward API needs valueFrom, which such a map cannot express. This post adds those entries to the rendered Deployments instead, merged into the container by its name, first with a Kustomize strategic merge patch over helm template, then with Kapitan's mutate generator. The two cases come from open issues against the KEDA HTTP add-on chart and the Crossplane chart. Each patch was rendered from the chart version named in its section with Helm 4.3.1, Kustomize 5.8.3, and Kapitan 0.36.3 with mutate 0.1.0, and the result was installed on a kind cluster on Kubernetes 1.37.0. Both charts come from HTTP chart repositories, and each downloaded package matched the sha256 its repository's index.yaml lists. The guide Patch a Helm chart compares the places such a patch can live. KEDA HTTP add-on chart: env with valueFrom {#keda-http-add-on-env} kedacore/charts#761 asked in June 2025 for extra environment variables on the HTTP add-on's interceptor, scaler and operator. Pull request #777, merged in October 2025, added extraEnvs to each, as a map of names to values. In July 2026 a commenter pointed out that a map cannot set variables from ConfigMaps, Secrets or the downward API, gave HOST_IP from status.hostIP as the example, and proposed an env list. Nobody has replied. Chart 0.16.0 renders the map in interceptor/deployment.yaml: {{- range $key, $value := .Values.interceptor.extraEnvs }} - name: {{ $key }} value: {{ $value | quote }} Given the commenter's entry as a map, the chart quotes the map as a string, and the container gets that text as the value: keda-add-ons-http: the value as an env entry: [{'name': 'HOST_IP', 'value': 'map[valueFrom:map[fieldRef:map[fieldPath:status.hostIP]]]'}] KEDA's core chart 2.21.0 has an env list for the operator, the metrics adapter and the webhooks. The add-on chart has none. Kustomize over helm template helm template http-add-on keda-add-ons-http --repo https://kedacore.github.io/charts \ --version 0.16.0 --namespace keda > helm.yaml kustomize build . with this kustomization.yaml next to helm.yaml: resources: - helm.yaml patches: - patch: |- apiVersion: apps/v1 kind: Deployment metadata: name: keda-add-ons-http-interceptor namespace: keda spec: template: spec: containers: - name: keda-add-ons-http-interceptor env: - name: HOST_IP valueFrom: fieldRef: fieldPath: status.hostIP Kustomize knows the schema of a Deployment, so the strategic merge finds the container by name and the entry by its name within env, and keeps the chart's other entries. It puts the new entry first in the list. Apart from that entry, and from fields whose value is null, which Kustomize drops, the output equalled the plain render. Kapitan's mutate The class pins the chart by version: parameters: kapitan: dependencies: - type: helm source: https://kedacore.github.io/charts chart_name: keda-add-ons-http version: 0.16.0 output_path: charts/keda-add-ons-http compile: - input_type: helm input_paths: [charts/keda-add-ons-http] output_path: keda-add-ons-http helm_params: name: http-add-on namespace: keda merge alone would replace the containers list with the one entry of the patch. merge_lists merges containers and env by name instead, at any depth: parameters: mutations: interceptor-host-ip: match: {apiVersion: apps/v1, kind: Deployment, name: keda-add-ons-http-interceptor} merge_lists: {containers: name, env: name} merge: spec: template: spec: containers: - name: keda-add-ons-http-interceptor env: - name: HOST_IP valueFrom: fieldRef: {fieldPath: status.hostIP} The target adds mutate as a second compile entry on the chart's output_path, as the guide shows. mutate appends the entry at the end of env, and nothing else in the render changed. On kind The add-on needs KEDA itself, so the test installed KEDA's chart 2.21.0 first. All six Deployments became Available, the interceptor Pod became Ready, and the live Pod carries the entry, with the apiVersion the API server fills in: {"name":"HOST_IP","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"status.hostIP"}}} The Kustomize output passed a server-side dry run against the same cluster. Crossplane Helm chart: env from a Secret {#crossplane-env-secret} crossplane/crossplane#7792, opened in September 2026, needs HTTP_PROXY for every Crossplane pod from a corporate egress proxy whose URL carries a password. extraEnvVarsCrossplane takes literals only, so the password ends up in plain text in the values repository, which the reporter's scanner flags. Another user offered to implement valueFrom support. No pull request or maintainer reply exists yet. Chart 2.4.2 renders the map in deployment.yaml: {{- range $key, $value := .Values.extraEnvVarsCrossplane }} - name: {{ $key | replace "." "_" }} value: {{ $value | quote }} With the issue's entry as a map, the chart renders it as a string like the KEDA chart does: crossplane: the value as an env entry: [{'name': 'HTTP_PROXY', 'value': 'map[valueFrom:map[secretKeyRef:map[key:url name:http-proxy]]]'}] Kustomize over helm template helm template crossplane crossplane --repo https://charts.crossplane.io/stable \ --version 2.4.2 --namespace crossplane-system > helm.yaml kustomize build . with this kustomization.yaml: resources: - helm.yaml patches: - patch: |- apiVersion: apps/v1 kind: Deployment metadata: name: crossplane namespace: crossplane-system spec: template: spec: containers: - name: crossplane env: - name: HTTP_PROXY valueFrom: secretKeyRef: name: http-proxy key: url The Deployments crossplane and crossplane-rbac-manager both name their container crossplane, so the patch selects the Deployment by name. As with the KEDA chart, the entry went first and nothing else changed. Kapitan's mutate The class pins chart 2.4.2 like the KEDA one, and the mutation has the same shape: parameters: mutations: crossplane-http-proxy: match: {apiVersion: apps/v1, kind: Deployment, name: crossplane} merge_lists: {containers: name, env: name} merge: spec: template: spec: containers: - name: crossplane env: - name: HTTP_PROXY valueFrom: secretKeyRef: {name: http-proxy, key: url} On kind The Secret http-proxy did not exist when the mutated output was applied, so the kubelet could not start the container: secret "http-proxy" not found Once the Secret existed, with an http:// URL as url, both Deployments became Available, and the live Pod carries the reference: {"name":"HTTP_PROXY","valueFrom":{"secretKeyRef":{"key":"url","name":"http-proxy"}}} The proxy in the Secret does not exist, and Crossplane still reached the API server: Go programs use HTTP_PROXY for http:// URLs only, and the API server is reached over HTTPS. The Kustomize output passed a server-side dry run. When the container name does not match Both tools merge by the name they are given and add a new list entry when no container has it. With the KEDA patch changed to the name interceptor, the mutate compile and the Kustomize build both passed, and each Deployment then had a second container with nothing but a name and env. The API server refused both, mutate's output, where the extra container comes second, and Kustomize's, where it comes first: The Deployment "keda-add-ons-http-interceptor" is invalid: spec.template.spec.containers[1].image: Required value The Deployment "keda-add-ons-http-interceptor" is invalid: spec.template.spec.containers[0].image: Required value mutate's expects counts the matched documents, here the one Deployment, so it does not catch a container name that a chart release changes. A server-side dry run in CI does. Originally published at kadet-hub.org.

KEDA: Kubernetes-based Event Driven Autoscaling (10.6k stars) What it does All text below is quoted directly from the project README. >Kubernetes-based Event Driven Autoscaling >KEDA allows for fine-grained autoscaling (including to/from zero) for event driven Kubernetes workloads. KEDA serves as a Kubernetes Metrics Server and allows users to define autoscaling rules using a dedicated Kubernetes custom resource definition. >KEDA can run on both the cloud and the edge, integrates natively with Kubernetes components such as the Horizontal Pod Autoscaler, and has no external dependencies. >We are a Cloud Native Computing Foundation (CNCF) graduated project. Getting started: - QuickStart - RabbitMQ and Go - QuickStart - Azure Functions and Queues - QuickStart - Azure Functions and Kafka on Openshift 4 - QuickStart - Azure Storage Queue with ScaledJob >There are many ways to deploy KEDA including Helm, Operator Hub and YAML files. >If interested in contributing or participating in the direction of KEDA, you can join our community meetings! Just want to learn or chat about KEDA? Feel free to join the conversation in #KEDA on the Kubernetes Slack! >We use GitHub issues to build our backlog, a complete overview of all open items and our planning. Repository kedacore/keda Repository stats |Metric|Value| |:-|:-| |Stars|10,619| |Forks|1,547| |Last release|v2.21.0 (2026-09-23)| |Languages|Go 99.4%| |Contributors|546| |AI contributor accounts (Claude / Codex / Cursor / Copilot)|Copilot (21 commits), claude (9)| |AI co-authored commits|Copilot: 25, Claude: 9| Main contributor |Field|Value| |:-|:-| |Name|Zbynek Roubalik| |GitHub|@zroubalik| |Commits to this repo|518 (top contributor)| |Bio|Founder & CTO @ Kedify | KEDA Maintainer| |Company / Location|Kedify / Czech Republic| |Website|kedify.io| |X / Twitter|@zroubalik| |Followers|130| |Public repos|94| |On GitHub since|2011-04-13|

Your Readiness Probe Is Lying to You A readiness probe is a promise. It tells Kubernetes "route traffic to me, I can handle it." Most of the time the promise is kept because the probe checks the thing that actually matters. The failures worth writing about are the ones where the probe keeps saying Ready while the component has quietly lost the one connection it needs to do any useful work. The traffic keeps arriving. None of it can be served. Nothing looks broken on the dashboard. I hit exactly this in KEDA, the CNCF event-driven autoscaler, and the fix merged as kedacore/keda #8226. The bug is a good teaching case on its own. The part I did not expect was the second bug hiding inside the fix, which a maintainer caught in review. > TL;DR > - KEDA's Metrics Server reported Ready even when its gRPC connection to the keda-operator was down. > - In that state the HPA silently stops receiving scale signals, while the replica still looks healthy. > - The fix makes the readiness probe observe the real gRPC connection state and report NotReady when the connection is unusable. > - A naive version of that fix has a trap: once a replica goes NotReady it stops receiving RPCs, so an Idle connection never gets woken and the replica can stay NotReady forever. > - The merged fix nudges an Idle connection to reconnect from inside the readiness check itself. The setup: who talks to whom KEDA has two moving parts here. The keda-operator does the actual work of talking to scalers and computing metrics. The Metrics Server (a custom metrics API server) is what the Kubernetes HPA queries when it wants to know "how many of these should be running." The Metrics Server does not compute metrics itself. It forwards the request to the operator over a gRPC connection and relays the answer. That gRPC connection is the Metrics Server's entire reason to exist. If it is down, the Metrics Server is a shell. It can accept an HPA query and has nothing to answer it with. The lie The Metrics Server's readiness probe did not check that connection. It reported Ready as long as the process was up and its HTTP server was listening. So when the gRPC connection to the operator dropped, the following happened, all at once and all invisible: the replica stayed Ready, Kubernetes kept it in the APIService endpoints, the HPA kept sending it metric queries, and every one of those queries failed or hung because there was no live operator connection behind them. The autoscaler stops scaling, and the one signal you would look at, pod readiness, is green. This is the specific shape of outage that is worst to debug. A crash is loud. A readiness probe that passes while the component cannot serve is silent, and it points you away from the real cause, because the first thing anyone checks is "are the pods ready," and they are. The fix, version one The fix is conceptually simple. Make readiness reflect the dependency. gRPC already tracks connection state through grpc.ClientConn.GetState(), which returns one of Idle, Connecting, Ready, TransientFailure, or Shutdown. Crucially, GetState() is a pure observer. It reports the current state and never initiates a connection, so it is safe to call from a probe with no side effects. So version one added a small accessor on the metrics-service client that returned the connection state, and wired a readyz check in the adapter that reported NotReady unless the connection was Ready. When the operator connection drops, the probe fails, Kubernetes pulls the replica from the APIService endpoints, and the HPA stops sending it queries it cannot answer. When the connection recovers, the probe passes again. That is the behavior you want. The second bug, caught in review Here is the part I like. A maintainer looked at version one and pointed out a failure mode that only exists because the fix works. Think through what happens after the probe correctly reports NotReady. The replica gets removed from the APIService endpoints. Now no RPCs flow through that client at all, because nothing is routed to it. And gRPC, by default, parks a connection in the Idle state after a period with no RPCs (GRPC_CLIENT_IDLE_TIMEOUT_MS, 30 minutes by default). An idle gRPC connection does not reconnect on its own. It waits for someone to use it. So the naive fix can deadlock the recovery. The connection goes bad, the probe reports NotReady, the replica stops getting RPCs, the connection drifts to Idle, and nothing ever wakes it, because the only thing that would wake it is an RPC, and RPCs only resume once the probe reports Ready again, which requires the connection to be awake. The replica can sit NotReady permanently, needing a manual restart to recover. The fix for a silent hang would have introduced a silent hang. The resolution is to make the readiness check active, not passive, in exactly one narrow case. When the accessor observes an Idle connection, it triggers a non-blocking reconnect before returning: // GetConnectionState returns the current connectivity state of the underlying // gRPC connection to the KEDA metrics service. // // When the connection is Idle it also triggers a non-blocking reconnection // attempt (moving it towards Connecting) before returning. Once a non-Ready // state removes the replica from the APIService endpoints, no RPCs flow through // this client, so nothing else would ever wake an Idle connection. func (c *GrpcClient) GetConnectionState() connectivity.State { state := c.connection.GetState() if state == connectivity.Idle { c.connection.Connect() } return state } Connect() is non-blocking and a no-op when the connection is not Idle, so it is safe to call from a probe. The probe still returns the pre-nudge state (so a just-nudged Idle connection reports NotReady that cycle), but the connection is now moving toward Connecting and then Ready on its own. The replica recovers without a human. The probe went from a pure observer to an observer-with-one-side-effect, and that single side effect is what closes the recovery loop. Reproduce it yourself The regression test does not need a cluster or a running operator. It exercises the connection-state logic directly with an in-process gRPC client, so you can run it from a clean checkout: git clone --depth 20 https://github.com/kedacore/keda.git cd keda go test ./pkg/metricsservice/... -run 'TestGetConnectionState' -v Two tests cover the two behaviors above. One asserts that a freshly created client (gRPC does not dial until first use) is not Ready, so a probe built on it correctly reports NotReady for a replica that has not connected yet. The other asserts the review fix: calling the accessor on an Idle connection moves it off Idle within a couple of seconds without any RPC being issued, so a sidelined replica can recover on its own. > Expected: both subtests pass. (If you are on a network that cannot reach the Go module proxy, the clone's large dependency tree may need go clean -modcache or a working GOPROXY first.) How to adopt the pattern The KEDA-specific takeaway is small: upgrade to a release that includes #8226 and your Metrics Server will stop advertising readiness while blind to the operator. The portable takeaway is the one worth internalizing. A readiness probe should check the dependency that the component cannot work without, not just that the process is up. For a forwarding service, that dependency is the downstream connection. And when you make a probe gate traffic on a connection's health, check whether removing traffic from an unhealthy replica also removes the thing that would let it recover. If recovery depends on activity, and your probe stops activity, you need the probe itself to drive recovery, carefully and without blocking. Check the real dependency, and make sure the healthy path back exists. A probe that only ever reports the process is alive is answering a question nobody asked. The question Kubernetes is actually asking is "can you serve." Answer that one. --- If you run forwarding or adapter services behind readiness probes, it is worth auditing what they actually check. The gap between "process is up" and "can serve" is where the quiet outages live.