gateway-api

ingress-nginx 退役之後：在臨時叢集實測 Traefik 的 ingress-nginx 相容模式與 Gateway API > 一句話結論：用 ingress-nginx 的小叢集，不必急著重寫所有 Ingress。實測下來，Traefik 的 ingress-nginx 相容模式可以原封不動吃下常見的 annotation；想一步到位改 Gateway API 也行，但轉址狀態碼、IP 白名單這類細節要自己補。這篇是評估，不是遷移紀錄。 遇到的問題 單機叢集常見的配置是：k3s 的內建 Traefik 關掉，自己用 Helm 裝 ingress-nginx，所有服務掛 ingressClassName: nginx，再用幾個 nginx.ingress.kubernetes.io/... annotation 處理 HTTPS 轉址、逾時、IP 白名單。 這個專案已經退役了。官方 repo 的說明寫得很直接： - 2026 年 3 月之後，不再有新版本、bug 修正，也不再修補新發現的安全漏洞。 - 已經部署的不會壞，Helm chart 和映像也還在。 - 沒有安裝過的人，建議直接找一個 Gateway API 的實作。 也就是說現有的叢集不會馬上出事，但每一天都在累積「沒人修」的風險。接下來有兩種做法： 1. 換 controller，Ingress 不動。 讓別的 controller 認得 nginx 的 annotation。 2. 換 API。 改寫成 Gateway API 的 Gateway 與 HTTPRoute。 這篇不下結論要選哪個，而是在隔離的臨時叢集把兩條路都實際跑一遍，記下哪些順利、哪些要注意。 測試環境 整個實驗用一個臨時的 k3d 單節點叢集，不佔主機的埠，測完就刪。 | 項目 | 版本 | |---|---| | k3s（k3d 預設） | v1.31.5 | | Traefik（Helm chart 42.2.0） | v3.7.16 | | Gateway API CRD（standard） | v1.7.0 | | cert-manager | v1.21.3 | | ingress2gateway | v1.2.0 | 測試用的後端是 traefik/whoami，它會回傳請求的標頭，還有 ?wait=25s 這種可以讓後端故意慢回應的參數，以及 /echo 的 WebSocket 端點，很適合拿來測逾時。 要先講清楚的限制： - 叢集裡用自簽憑證，沒有真的向 Let's Encrypt 申請。 - 從叢集內的 Pod 發請求，沒有經過 NodePort 或 LoadBalancer，所以「來源 IP 會不會被改掉」沒有驗證。 - 單節點，沒做效能比較。 手把手步驟 前置：k3s 內建的 Traefik 不能直接用 先查了 k3s 自帶的 Traefik。k3d 預設的 k3s v1.31.5，內建的是 Traefik 2.11.18。 Traefik 的 ingress-nginx 相容 provider，官方文件寫明需要 v3.6.2 以上。所以 v1.31.5 附的版本沒有這個功能（用 traefik --help 也確實只看到 kubernetesgateway，而且還需要 --experimental.kubernetesgateway）。 比較新的 k3s 會附 Traefik 3.7.x：查 k3s v1.37.1 的內建 manifest，chart 是 41.4.0，對應 Traefik v3.7.12。但舊版 k3s 的叢集，得自己用 Helm 裝新版 Traefik。 下面的實驗都是這樣做：建叢集時停用內建 Traefik，再用 Helm 裝。 k3d cluster create demo --servers 1 --agents 0 --no-lb \ --k3s-arg "--disable=traefik@server:0" 路線一：讓 Traefik 直接吃 nginx 的 Ingress Helm values 只要打開 provider： providers: kubernetesIngressNGINX: enabled: true # 預設 ingressClass 是 nginx 原本的 Ingress 完全不用改。測試用的是三種最常見的寫法，原封不動貼進去： # 1. HTTPS 轉址 metadata: annotations: nginx.ingress.kubernetes.io/force-ssl-redirect: "true" nginx.ingress.kubernetes.io/ssl-redirect: "true" # 2. 長時間逾時 nginx.ingress.kubernetes.io/proxy-read-timeout: "3600" nginx.ingress.kubernetes.io/proxy-send-timeout: "3600" # 3. IP 白名單 nginx.ingress.kubernetes.io/whitelist-source-range: "192.0.2.0/24" 實測結果： | 項目 | 結果 | |---|---| | HTTP → HTTPS 轉址 | 308 Permanent Redirect，Location 正確帶回原本的路徑，和 ingress-nginx 的預設一致 | | whitelist-source-range | 名單外的來源收到 403 Forbidden | | proxy-read-timeout | 有生效。為了方便驗證，改成 5 秒、後端 25 秒才回應，結果在 5.0 秒收到 504；改成 60 秒就正常通過 | | backend-protocol: "HTTPS" | 後端用自簽憑證的 HTTPS，收到 200；拿掉 annotation 就變成 400 | | WebSocket | 透過這個 Ingress 連線、閒置 30 秒後仍可收發 | 這個 provider 可以和 Gateway API 的 provider 同時開著。Traefik 官方文件沒有說明兩者能不能並存，所以實測了：同一個 Traefik 裡，Ingress 和 HTTPRoute 都正常回應。 有一個小地方：Ingress 的 ADDRESS 欄位是空的。這個 provider 預設沒有回填 status.loadBalancer，chart 有 publishService 這個選項，我沒有測它有沒有用。依賴 Ingress status 的工具（例如自動建立 DNS 記錄的）要留意。 另外，如果打算讓 ingress-nginx 和 Traefik 並行一段時間再切換，官方文件提醒兩個 controller 會互相覆寫 Ingress 的 status，路由不受影響，但監看 status 的工具會被干擾。這一點我沒有實測。 路線二：改寫成 Gateway API 先裝 Gateway API 的標準 CRD，再裝 Traefik 並打開 Gateway provider： kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.7.0/standard-install.yaml providers: kubernetesGateway: enabled: true gateway: enabled: false # 自己建 Gateway Traefik 的 chart 輸出訊息建議安裝 v1.5.1，我裝的是 v1.7.0，GatewayClass 仍然正常被接受。 Gateway 的 listener 埠必須對應 Traefik 的 entrypoint（預設是 HTTP 8000、HTTPS 8443），不是對外的 80 和 443： apiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: {name: main, namespace: traefik} spec: gatewayClassName: traefik listeners: - name: web port: 8000 protocol: HTTP allowedRoutes: {namespaces: {from: All}} - name: websecure port: 8443 protocol: HTTPS tls: mode: Terminate certificateRefs: [{name: demo-tls}] allowedRoutes: {namespaces: {from: All}} HTTP 轉 HTTPS apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: {name: app-redirect, namespace: demo} spec: parentRefs: [{name: main, namespace: traefik, sectionName: web}] hostnames: [app.example.test] rules: - filters: - type: RequestRedirect requestRedirect: {scheme: https, statusCode: 308} 這一段有三個發現： - 不要寫 port。 一開始寫了 port: 443，Location 變成 https://app.example.test:443/x。拿掉之後才是乾淨的 https://app.example.test/x。 - statusCode 預設是 302。 沒寫的話，原本 ingress-nginx 的 308 會變成臨時轉址。 - 308 可以用，但屬於 Extended。 CRD 的說明是「301 與 302 是 Core，其餘是 Extended」。實測 Traefik 3.7.16 接受並回傳 308 Permanent Redirect。換別的 Gateway 實作就不一定支援，要看該實作的文件。 IP 白名單 Gateway API 標準沒有 IP 白名單的欄位。Traefik 的做法是建一個 Middleware，再用 ExtensionRef 掛到 HTTPRoute： apiVersion: traefik.io/v1alpha1 kind: Middleware metadata: {name: only-office, namespace: demo} spec: ipAllowList: sourceRange: ["192.0.2.0/24"] --- apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: {name: internal, namespace: demo} spec: parentRefs: [{name: main, namespace: traefik, sectionName: websecure}] hostnames: [internal.example.test] rules: - filters: - type: ExtensionRef extensionRef: {group: traefik.io, kind: Middleware, name: only-office} backendRefs: [{name: whoami, port: 80}] 名單外的來源收到 403 Forbidden，把 sourceRange 換成 Pod 網段就通過。這是 Traefik 專屬的寫法，換 Gateway 實作就得重寫。 cert-manager 整合 憑證的位置從 Ingress 的 annotation，搬到 Gateway 的 listener。cert-manager 要打開 Gateway 支援： config: apiVersion: controller.config.cert-manager.io/v1alpha1 kind: ControllerConfiguration enableGatewayAPI: true Gateway 加上 issuer 的 annotation，listener 指向一個「還不存在」的 Secret： metadata: annotations: cert-manager.io/cluster-issuer: selfsigned spec: listeners: - name: auto hostname: auto.example.test port: 8443 protocol: HTTPS tls: mode: Terminate certificateRefs: [{name: auto-tls}] 大約 12 秒後，cert-manager 自動建出 Certificate（READY=True）和 auto-tls 這個 Secret，Traefik 開始用它提供 HTTPS，請求回 200。 這裡用的是自簽的 issuer。真正用 Let's Encrypt 時，DNS-01 或 HTTP-01 的流程這次沒有驗證。 長連線與逾時 原本 annotation 裡的 proxy-read-timeout: 3600，主要是為了 WebSocket 和長時間回應。到了 Traefik，要分三種情況看。 預設值。 不調任何逾時： - WebSocket 連上之後閒置 400 秒，再送一次訊息，仍然收到回應。 - 後端 100 秒才回應的 HTTP 請求，等到 100.1 秒成功收到 200。 3600 秒本身沒有實際等待測試，只能說預設值下，閒置幾百秒不會被切斷。 調得很短的 entrypoint 逾時。 為了確認哪些設定會影響，把 HTTPS entrypoint 的 readTimeout、writeTimeout、idleTimeout 都設成 10 秒： ports: websecure: transport: respondingTimeouts: readTimeout: 10s writeTimeout: 10s idleTimeout: 10s - WebSocket 閒置 30 秒，仍然正常。升級成 WebSocket 之後，連線不受這幾個逾時影響。 - 一般 HTTP 請求，後端 25 秒才回應，在第 10 秒被切斷（客戶端看到連線被中斷，沒有收到 HTTP 狀態）。 也就是說，要讓「長時間才回應」的請求（例如串流輸出）不被切掉，要看的是 entrypoint 的 writeTimeout。預設值下 100 秒的回應沒有被切斷（前面實測過），所以一般不用動，但一旦被調過就會是問題。 nginx annotation 不能蓋過 entrypoint。 在 entrypoint 設了 10 秒的情況下，即使 Ingress 寫了 proxy-read-timeout: "3600"，後端 25 秒的請求還是在 10 秒被切斷。annotation 控制的是後端那一段的逾時，不會改寫 entrypoint 的設定。 ingress2gateway 轉出來的東西 官方的轉換工具 ingress2gateway 可以把現有的 Ingress 轉成 Gateway API。拿上面的三個 Ingress 餵給 v1.2.0： ingress2gateway print --input-file ing.yaml --providers ingress-nginx 轉換器會印出警告，值得逐條看： - force-ssl-redirect 被標為 Unsupported annotation。不過因為 Ingress 有 TLS，工具仍然自動產生了 308 的 HTTP → HTTPS 轉址路由，結果是對的，警告看起來只是沒有逐個 annotation 處理。 - whitelist-source-range 轉換失敗：IP-based authorization is not supported。白名單會直接掉，轉完的路由是沒有限制的。 - 逾時 annotation 會做「best-effort」轉成 timeouts.request，但數字不對。 最後一點值得單獨講。同樣的 annotation，轉出來的結果： | proxy-read-timeout（秒） | 轉出的 timeouts.request | |---|---| | 60 | 10m0s（應該是 1 分鐘） | | 100 | 16m40s（應該是 1 分 40 秒） | | 3600 | 10h0m0s（應該是 1 小時） | 每一筆都是預期的 10 倍。從原始碼看，解析 annotation 的那一步是正確的（秒數轉成 Duration），所以問題出在後面哪一步，這裡沒有追到，也沒找到對應的 issue。這是 v1.2.0 的觀察，之後版本可能已經修掉，重點是轉換結果一定要人工逐項核對，不要直接 apply。 踩到的坑 整理一下實測中遇到的： 1. k3s 內建的 Traefik 可能太舊。 v1.31.5 附的是 2.11.18，沒有 ingress-nginx 相容 provider，要自己用 Helm 裝 3.6.2 以上。 2. Gateway 的 listener 埠是 entrypoint 的埠。 寫成 80、443 會對不上。 3. RequestRedirect 不要寫 port，也別忘了 statusCode。 前者會讓網址多出 :443，後者預設是 302，不是 ingress-nginx 的 308。 4. IP 白名單在 Gateway API 沒有標準寫法。 得靠實作專屬的 Middleware，轉換工具也不會幫你處理。 5. 轉換工具的輸出要核對。 至少逾時數字要自己重算。 為什麼這樣評估 如果現有叢集的 Ingress 很多，而且多半只用到轉址、逾時、白名單這幾個 annotation，路線一的成本最低：換掉 controller 就好，YAML 一行不改，風險也最小。缺點是 annotation 的相容只涵蓋「一部分」。Traefik 官方文件也寫了某些 annotation 的行為有差異，所以還是要逐項驗證。 路線二是長期比較乾淨的方向，因為 Kubernetes 官方文件說 Ingress API 已經凍結，並建議改用 Gateway。但每個服務都得重寫，白名單、逾時這類能力要換成實作專屬的寫法，等於把鎖定換了一個地方。 所以比較務實的順序是：先用路線一把 controller 換掉，擺脫退役的專案；再依需求，一個服務一個服務搬到 Gateway API。這篇只是把可行性驗過一遍，哪一天真的動手，仍然要在自己的叢集重新驗證，尤其是真實憑證、來源 IP 與多節點的情境。 延伸閱讀 - Ingress NGINX 退役說明（官方 repo） - Kubernetes：Ingress（含 API 已凍結的說明） - Traefik：從 ingress-nginx 遷移 - Traefik：Kubernetes Ingress NGINX provider - ingress2gateway - cert-manager：Gateway API 支援

Gateway API in Production: Replacing Ingress NGINX Without Breaking Your TLS Certificates Ingress NGINX is in maintenance mode, and the Kubernetes project has been nudging everyone toward Gateway API for a while now. The networking spec graduated to GA (v1) for GatewayClass, Gateway and HTTPRoute, the ecosystem has converged, and your next cluster upgrade is a good excuse to stop postponing the migration. The problem is that "replace your ingress controller" is a sentence that hides a lot of risk: a single typo in a TLS listener and every browser in production gets a certificate error. This article walks through a migration strategy that never requires a big-bang cutover: run Gateway API next to your existing Ingress NGINX, move one hostname at a time, shift traffic by weight, and validate the TLS chain at every step. We'll use Envoy Gateway as the controller and cert-manager for certificates, but the pattern applies to any conformant implementation (Istio, Cilium, Traefik, NGINX Gateway Fabric). Context: why this migration is different Swapping an ingress controller is not like swapping a Deployment image. Three things make it risky: 1. TLS termination moves. Your certificates live in Secrets referenced by Ingress objects. The new data plane has to reference the same Secrets (or new ones), possibly from a different namespace, under a stricter cross-namespace model. 2. The public IP changes. A new controller means a new LoadBalancer Service, a new external IP, and a DNS change. DNS changes are not atomic — you cannot roll them back in 30 seconds. 3. Annotations don't translate. Rewrites, CORS, rate limits, auth, timeouts — everything encoded in nginx.ingress.kubernetes.io/* annotations becomes either native Gateway API fields, filters, or implementation-specific policies. The good news: Gateway API was explicitly designed for coexistence. An Ingress and a Gateway can serve the same hostname from the same cluster, and nothing in the spec forbids two controllers from watching different resources. That's the lever we're going to pull. Target architecture The model splits responsibilities across three object types, which maps nicely onto real org boundaries: - GatewayClass — cluster-scoped, owned by the platform team. Points at a controller. - Gateway — the listener (ports, protocols, TLS certs, allowed routes). Lives in an infrastructure namespace. - HTTPRoute — owned by application teams, in application namespaces, attached to a Gateway via parentRefs. - ReferenceGrant — the explicit opt-in that allows a cross-namespace reference (Gateway → Secret, HTTPRoute → Service). Concretely: one shared Gateway in gateway-infra, TLS Secrets managed by cert-manager, and each team shipping its own HTTPRoute in its own namespace. Step 1 — Install the CRDs and the controller On a vanilla Kubernetes cluster, install the standard channel CRDs first: kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.1/standard-install.yaml kubectl get crd | grep gateway.networking.k8s.io > OpenShift note: since 4.19, the Ingress Operator can manage the Gateway API CRDs itself. Check for ownership before applying anything, otherwise the operator will fight you: > ```bash > oc get crd gateways.gateway.networking.k8s.io \ > -o jsonpath='{.metadata.labels}{"\n"}{.metadata.annotations}{"\n"}' > ``` > If the CRDs are operator-managed, skip the kubectl apply and only install your controller. Envoy Gateway's data plane runs unprivileged on ports 10080/10443, so the restricted-v2 SCC is sufficient — no SCC patching required. Install Envoy Gateway: helm install eg oci://docker.io/envoyproxy/gateway-helm \ --version v1.2.6 \ -n envoy-gateway-system --create-namespace \ --set config.envoyGateway.provider.kubernetes.rateLimitDeployment.container.resources.requests.cpu=100m kubectl -n envoy-gateway-system rollout status deploy/envoy-gateway Then declare the GatewayClass: apiVersion: gateway.networking.k8s.io/v1 kind: GatewayClass metadata: name: eg-external spec: controllerName: gateway.envoyproxy.io/gatewayclass-controller kubectl get gatewayclass eg-external \ -o jsonpath='{.status.conditions[?(@.type=="Accepted")].status}{"\n"}' # expected: True Step 2 — Wire cert-manager to Gateway API cert-manager ships a "gateway shim" that watches Gateway objects for the cert-manager.io/cluster-issuer annotation and creates Certificate resources from the listeners. It's off by default: helm upgrade --install cert-manager jetstack/cert-manager \ -n cert-manager --create-namespace \ --version v1.16.2 \ --set crds.enabled=true \ --set config.apiVersion="controller.config.cert-manager.io/v1alpha1" \ --set config.kind="ControllerConfiguration" \ --set config.enableGatewayAPI=true For HTTP-01 solving, cert-manager can create a temporary HTTPRoute instead of an Ingress: apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: letsencrypt-prod spec: acme: server: https://acme-v02.api.letsencrypt.org/directory email: platform@example.com privateKeySecretRef: name: letsencrypt-prod-account-key solvers: - http01: gatewayHTTPRoute: parentRefs: - name: apps-gw namespace: gateway-infra kind: Gateway Critical detail during coexistence: if your existing Ingress NGINX still serves the hostname and holds the DNS record, HTTP-01 challenges will land on NGINX, not on the Gateway. Two safe options: keep issuing certificates through the existing Ingress solver until DNS is switched, or — far better — use DNS-01. DNS-01 is independent of which data plane owns the IP, which removes an entire class of migration failures. Step 3 — The Gateway, reusing the existing certificate Here's the trick that avoids re-issuing certs on day one: point the new Gateway at the same Secret your Ingress already uses. No new ACME order, no rate-limit risk, identical chain. apiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: name: apps-gw namespace: gateway-infra annotations: cert-manager.io/cluster-issuer: letsencrypt-prod spec: gatewayClassName: eg-external listeners: - name: http protocol: HTTP port: 80 hostname: "*.example.com" allowedRoutes: namespaces: from: Selector selector: matchLabels: gateway-access: "true" - name: https-shop protocol: HTTPS port: 443 hostname: shop.example.com tls: mode: Terminate certificateRefs: - kind: Secret name: shop-example-com-tls namespace: shop allowedRoutes: namespaces: from: Selector selector: matchLabels: gateway-access: "true" Two things to notice. First, allowedRoutes uses a label selector, not All — namespaces must explicitly opt in. Second, the certificate Secret lives in the shop namespace while the Gateway lives in gateway-infra. That cross-namespace reference is denied by default. You need a ReferenceGrant in the namespace that owns the Secret: apiVersion: gateway.networking.k8s.io/v1beta1 kind: ReferenceGrant metadata: name: allow-gw-to-read-shop-tls namespace: shop spec: from: - group: gateway.networking.k8s.io kind: Gateway namespace: gateway-infra to: - group: "" kind: Secret name: shop-example-com-tls Note the name field in to: — grant access to that one Secret, not to every Secret in the namespace. Verify the listener is actually programmed. This is the single most useful command of the whole migration: kubectl -n gateway-infra get gateway apps-gw \ -o jsonpath='{range .status.listeners[*]}{.name}{"\t"}{range .conditions[*]}{.type}={.status}{" "}{end}{"\n"}{end}' A ResolvedRefs=False with reason RefNotPermitted means your ReferenceGrant is missing or mismatched. Step 4 — The HTTPRoute apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: shop namespace: shop spec: parentRefs: - name: apps-gw namespace: gateway-infra sectionName: https-shop hostnames: - shop.example.com rules: - matches: - path: type: PathPrefix value: /api filters: - type: RequestHeaderModifier requestHeaderModifier: set: - name: X-Forwarded-Proto value: https backendRefs: - name: shop-api port: 8080 - matches: - path: type: PathPrefix value: / backendRefs: - name: shop-frontend port: 8080 Don't forget the namespace label required by allowedRoutes: kubectl label namespace shop gateway-access=true If a rule needs to target a Service in another namespace (say payments), add a ReferenceGrant in payments with from.kind: HTTPRoute. Same mechanism, same least-privilege rule: name the target Service explicitly. Step 5 — Validate TLS before touching DNS At this point both data planes serve shop.example.com, but only NGINX has the DNS record. Test the Gateway directly by resolving the hostname yourself: GW_IP=$(kubectl -n envoy-gateway-system get svc \ -l gateway.envoyproxy.io/owning-gateway-name=apps-gw \ -o jsonpath='{.items[0].status.loadBalancer.ingress[0].ip}') curl -sSv --resolve shop.example.com:443:$GW_IP https://shop.example.com/healthz Then compare the certificate served by each data plane. Identical fingerprints mean zero client-visible change: fp() { openssl s_client -connect "$1:443" -servername shop.example.com </dev/null 2>/dev/null \ | openssl x509 -noout -fingerprint -sha256; } NGINX_IP=$(dig +short shop.example.com | head -1) echo "nginx: $(fp $NGINX_IP)" echo "gateway: $(fp $GW_IP)" Also check the full chain and SAN list — a frequent failure mode is a Gateway serving only the leaf certificate because the Secret was built by hand: openssl s_client -connect $GW_IP:443 -servername shop.example.com -showcerts </dev/null 2>/dev/null \ | grep -E 's:|i:' openssl s_client -connect $GW_IP:443 -servername shop.example.com </dev/null 2>/dev/null \ | openssl x509 -noout -dates -ext subjectAltName And make sure an unknown SNI doesn't silently fall through to the wrong certificate: openssl s_client -connect $GW_IP:443 -servername notmydomain.example.com </dev/null 2>&1 | head -20 Step 6 — Shift traffic by weight, without DNS DNS-based cutovers are coarse: TTLs lie, resolvers cache, and rollback takes minutes. A much tighter loop is to make Ingress NGINX itself forward a percentage of traffic to the Gateway, using its canary annotations plus an ExternalName Service. apiVersion: v1 kind: Service metadata: name: gw-bridge namespace: shop spec: type: ExternalName externalName: envoy-gateway-infra-apps-gw-7f9c2d11.envoy-gateway-system.svc.cluster.local ports: - name: https port: 443 Get the real Service name first: kubectl -n envoy-gateway-system get svc \ -l gateway.envoyproxy.io/owning-gateway-name=apps-gw -o name Then a canary Ingress on the same hostname: apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: shop-canary-to-gateway namespace: shop annotations: nginx.ingress.kubernetes.io/canary: "true" nginx.ingress.kubernetes.io/canary-weight: "5" nginx.ingress.kubernetes.io/backend-protocol: "HTTPS" nginx.ingress.kubernetes.io/upstream-vhost: "shop.example.com" nginx.ingress.kubernetes.io/proxy-ssl-server-name: "on" nginx.ingress.kubernetes.io/proxy-ssl-name: "shop.example.com" spec: ingressClassName: nginx tls: - hosts: [shop.example.com] secretName: shop-example-com-tls rules: - host: shop.example.com http: paths: - path: / pathType: Prefix backend: service: name: gw-bridge port: number: 443 Now you have a real dial. Ramp it with a single patch, watch your SLOs, and roll back in seconds: for w in 5 25 50 100; do kubectl -n shop annotate ingress shop-canary-to-gateway \ nginx.ingress.kubernetes.io/canary-weight="$w" --overwrite sleep 600 done # instant rollback kubectl -n shop annotate ingress shop-canary-to-gateway \ nginx.ingress.kubernetes.io/canary-weight="0" --overwrite You can also pin the canary to a header (nginx.ingress.kubernetes.io/canary-by-header: x-use-gateway) so QA can target the new path deterministically before any percentage of real users is affected. Once 100% has been stable for a few days, switch DNS to the Gateway's IP, wait out the TTL, then delete the canary Ingress and the legacy Ingress. The same weighting primitive exists natively inside Gateway API for backend-level canaries: backendRefs: - name: shop-frontend port: 8080 weight: 90 - name: shop-frontend-next port: 8080 weight: 10 Security best practices Never use allowedRoutes.namespaces.from: All on an internet-facing Gateway. Any namespace in the cluster could then attach a route to your public listener and claim a hostname. Use Same or a label Selector, and treat the label as a privileged operation controlled by the platform team. Scope every ReferenceGrant. Always set to[].name. A grant without a name allows every Secret (or every Service) of that kind in the namespace — including, potentially, the Secret holding a wildcard certificate. Enforce TLS floor and cipher policy. Gateway API leaves this to implementation-specific policies. With Envoy Gateway: apiVersion: gateway.envoyproxy.io/v1alpha1 kind: ClientTrafficPolicy metadata: name: tls-hardening namespace: gateway-infra spec: targetRefs: - group: gateway.networking.k8s.io kind: Gateway name: apps-gw tls: minVersion: "1.2" maxVersion: "1.3" ciphers: - ECDHE-ECDSA-AES256-GCM-SHA384 - ECDHE-RSA-AES256-GCM-SHA384 - ECDHE-ECDSA-CHACHA20-POLY1305 headers: xForwardedClientCert: mode: Sanitize Redirect HTTP to HTTPS at the route level, so a misconfigured app can never be served in clear text: apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: https-redirect namespace: shop spec: parentRefs: - name: apps-gw namespace: gateway-infra sectionName: http hostnames: ["shop.example.com"] rules: - filters: - type: RequestRedirect requestRedirect: scheme: https statusCode: 301 Encrypt the last hop too. If your backend speaks TLS, use BackendTLSPolicy with explicit CA validation and hostname verification rather than disabling verification: apiVersion: gateway.networking.k8s.io/v1alpha3 kind: BackendTLSPolicy metadata: name: shop-api-upstream-tls namespace: shop spec: targetRefs: - group: "" kind: Service name: shop-api sectionName: "8443" validation: caCertificateRefs: - group: "" kind: ConfigMap name: internal-ca hostname: shop-api.shop.svc.cluster.local Lock down the data plane with NetworkPolicy. The Envoy pods should accept traffic from the LoadBalancer and reach only the namespaces they route to. On OpenShift, confirm the proxy runs under restricted-v2 and not a relaxed SCC: oc -n envoy-gateway-system get pod -l app.kubernetes.io/name=envoy \ -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.annotations.openshift\.io/scc}{"\n"}{end}' Audit certificate ownership after the switch. When you finally delete the legacy Ingress, make sure cert-manager still owns the Certificate object — deleting an Ingress with the cert-manager.io/cluster-issuer annotation can garbage-collect the Certificate and, depending on your setup, the Secret with it. Create a standalone Certificate resource explicitly rather than relying on annotation-driven shims for anything production-critical: apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: shop-example-com namespace: shop spec: secretName: shop-example-com-tls duration: 2160h renewBefore: 720h privateKey: algorithm: ECDSA size: 256 rotationPolicy: Always dnsNames: - shop.example.com issuerRef: name: letsencrypt-prod kind: ClusterIssuer Alert on expiry independently. Scrape certmanager_certificate_expiration_timestamp_seconds and alert at 14 days, plus a black-box probe that checks the certificate actually served on port 443. A valid Secret that the Gateway failed to load is a silent outage waiting for renewal day. Conclusion Migrating off Ingress NGINX doesn't have to be a maintenance-window event. The key takeaways: - Coexist first. Gateway API and Ingress can serve the same hostname simultaneously. Build the Gateway, validate it out-of-band with curl --resolve, and only then send it traffic. - Reuse the existing TLS Secret for the first cutover. Comparing SHA-256 fingerprints between the old and new data plane is a 10-second check that eliminates the scariest failure mode. - ReferenceGrant is a security boundary, not a formality. Scope it to a named Secret or Service, and keep allowedRoutes on a label selector. - Shift traffic with weights, not DNS. The NGINX canary + ExternalName bridge gives you a percentage dial with sub-second rollback. Move DNS only once 100% has been stable for days. - Prefer DNS-01 during the transition, so certificate issuance never depends on which data plane currently owns the public IP. - Harden explicitly. TLS floor, HTTP→HTTPS redirect, upstream TLS validation and NetworkPolicy are all policy objects you have to write — nothing is secure by default just because it's new. Do one low-traffic hostname end to end first. The second one takes an afternoon, and by the fifth you'll be deleting Ingress objects without thinking about it.

Kubernetes in 2026: What It Is, How It Works, and a Production Baseline You Can Apply Today Kubernetes is the system most teams now use to run containers in production. In the CNCF annual survey announced on January 20, 2026, 82% of container users reported running Kubernetes in production, up from 66% in 2023. Operating it well is now a core engineering skill, and this post teaches it in three layers. The first layer explains what Kubernetes is, the problem it solves, and the objects you work with every day. The second shows how a cluster is built and what happens between typing kubectl apply and a running container. The third is a production baseline for a stateless HTTP service: seven manifests, each explained, plus the 2026 changes that can break a cluster you have not touched in a year. Kubernetes 1.37 shipped on August 26, 2026, and the project now supports three minor versions at once. Every feature below carries its maturity level and the version that set it. Where a claim comes from a release note, a blog post or a survey, it links to that source. Where the baseline depends on something your cluster must provide, such as a Gateway API controller or a CNI plugin that enforces NetworkPolicy, the text says so. What Kubernetes is, and the problem it solves In the project's own words, Kubernetes is a portable, extensible, open source platform for managing containerized workloads and services, with both declarative configuration and automation. The name comes from the Greek word for helmsman or pilot, and K8s abbreviates it by counting the eight letters between the K and the s. Google open sourced the project in 2014, and it combines over 15 years of Google's experience running production workloads at scale with ideas from the community. To see why it exists, follow how deployment changed. Teams first ran applications on physical servers, where one application could take most of a machine's resources and starve the others. Virtual machines fixed that by isolating applications, but each VM runs a full operating system. Containers relax that isolation and share the host operating system, so they are lighter and portable across clouds and Linux distributions. Containers solve packaging, not operations. In production something must restart a container that dies, spread containers across many machines, send traffic only to the healthy ones, and roll out new versions without downtime. The overview puts it plainly: if a container goes down, another container needs to start, and it would be easier if a system handled that. Kubernetes is that system. The overview lists what you get from it, summarized here. | Capability | What it does | | --- | --- | | Service discovery and load balancing | Exposes a container by DNS name or IP address and spreads traffic across copies | | Storage orchestration | Mounts the storage system you choose, from local disks to public cloud storage | | Automated rollouts and rollbacks | Moves the actual state to the state you describe, at a controlled rate | | Automatic bin packing | Places containers on nodes using the CPU and memory you declare for each | | Self-healing | Restarts failed containers and withholds traffic until a container is ready | | Secret and configuration management | Updates secrets and configuration without rebuilding images | | Batch execution | Runs batch and CI workloads and replaces failed containers if you want | | Horizontal scaling | Scales up or down by command, in a UI, or automatically on CPU usage | It also helps to know what Kubernetes leaves out. It is not an all-inclusive platform as a service: it does not build your code, run your CI/CD pipeline, or ship databases, message buses and caches as built-in services, and it does not choose your logging or monitoring tools. It is also not orchestration in the strict sense of running step A, then B, then C. It is a set of independent control processes that keep driving the current state toward the state you declared. The core idea: you declare, controllers converge Everything in Kubernetes is an object, a persistent record in the cluster. The objects documentation calls an object a record of intent: once you create it, the system works constantly to make it exist. Almost every object has a spec, which is the state you want, and a status, which Kubernetes fills in with the state it observes. Take a Deployment whose spec asks for three replicas. Kubernetes reads the spec and starts three instances. If one fails, the status no longer matches the spec, and Kubernetes starts a replacement. The component doing this work is a controller, which runs a control loop, a non-terminating loop that regulates the state of a system. The docs compare it to a thermostat that switches equipment on and off to bring a room toward the temperature you set. The practical consequence is that you rarely issue instructions such as "start this container". You change the declared state, and controllers work out the steps. That is why this post is built from YAML files you apply, not scripts you run. The objects you use every day A handful of object types cover most applications. The table lists them in the order you meet them. | Object | What it is | How you use it | | --- | --- | --- | | Pod | The smallest deployable unit: one or more tightly coupled containers with shared storage and network | The unit that actually runs, normally created through a workload resource and not by hand | | ReplicaSet | Keeps a stable set of replica Pods running | Rarely touched directly, because a Deployment manages it for you | | Deployment | Declarative updates for Pods and ReplicaSets, usually for apps that keep no state | Running and updating a stateless service | | Service | A stable network endpoint for a changing set of Pods | Giving clients one name and address to call | | ConfigMap | Non-confidential configuration stored as key-value pairs | Settings consumed as environment variables, arguments or files | | Secret | A small amount of sensitive data such as a password, token or key | Credentials kept out of images and Pod specs | | Namespace | A scope that isolates groups of resources inside one cluster | Separating teams or projects, since names must be unique only within a namespace | | Node | A worker machine, virtual or physical, that runs Pods | The capacity underneath everything, managed for you by the control plane | Other workload resources cover other shapes of work. A StatefulSet gives each Pod a sticky identity for applications that need persistent storage or a stable network identity. A DaemonSet runs node-local facilities such as a networking helper. A Job runs a one-off task to completion. Pods are ephemeral. The Service documentation says you should not expect any individual Pod to be reliable or durable, because Pods are created and destroyed to match the desired state. Each Pod gets its own IP address, but the set of healthy Pods keeps changing. That is the reason Services exist: clients need one stable name in front of Pods that come and go. One caution about Secrets. By default they are stored unencrypted in etcd, and anyone with API access can read them, as can anyone allowed to create a Pod in the namespace. The Secret documentation tells you to enable encryption at rest and to restrict access with RBAC. A ConfigMap provides no secrecy at all. Here is a minimal Deployment, the object you will write most often. # hello.yaml apiVersion: apps/v1 kind: Deployment metadata: name: hello-node spec: replicas: 2 selector: matchLabels: app: hello-node template: metadata: labels: app: hello-node spec: containers: - name: agnhost image: registry.k8s.io/e2e-test-images/agnhost:2.53 command: ["/agnhost", "netexec", "--http-port=8080"] ports: - containerPort: 8080 Read it from the top. apiVersion and kind name the API and the object type, and metadata.name names the object. Everything under spec is desired state: two replicas of a Pod built from the template. The selector tells the Deployment which Pods it owns by matching the template's labels, and the Deployment documentation says the API rejects a Deployment whose two do not match. Labels are plain key-value pairs that Services, Deployments and policies use to find Pods. How a cluster is built A cluster has a control plane and one or more worker nodes. The components overview says the control plane components manage the overall state of the cluster, and the node components run on every node, maintaining running Pods and providing the runtime environment. | Component | Runs on | Job | | --- | --- | --- | | kube-apiserver | Control plane | Serves the Kubernetes HTTP API as the front end of the control plane, and scales horizontally by running more instances | | etcd | Control plane | Consistent, highly available key-value store for all cluster data, which you need a backup plan for | | kube-scheduler | Control plane | Watches for newly created Pods with no node and selects a node for each | | kube-controller-manager | Control plane | Runs the controller processes, compiled into one binary and run as one process | | cloud-controller-manager | Control plane, optional | Integrates with the underlying cloud provider | | kubelet | Every node | Makes sure the containers described in PodSpecs are running and healthy, and ignores containers Kubernetes did not create | | kube-proxy | Every node, optional | Maintains the network rules that implement Services | | Container runtime | Every node | The software that actually runs containers | !Mermaid Diagram Every arrow runs through the API server. The Kubernetes API documentation says users, the different parts of your cluster and external components all communicate with one another through it. The scheduler, the controllers and the kubelets never call each other. Each watches the API for work and writes its results back, so a component that restarts can read the current state again. Three consequences shape every manifest in this post. The scheduler reads requests, so wrong requests mean wrong placement. The kubelet enforces limits, so wrong limits mean CPU throttling or out-of-memory kills. Controllers retry forever, so a bad rollout never fixes itself unless a probe tells the truth. What happens when you run kubectl apply Follow one command through the system. The CLI turns your file into API calls, as the objects documentation describes, and from there the work passes between components without any of them calling another directly. !Mermaid Diagram A chain of hand-offs turns that file into running containers. The API server accepts and stores the Deployment. The Deployment controller creates a ReplicaSet, and the ReplicaSet creates the Pods, which the Deployment documentation describes as happening in the background. The scheduler sees Pods without a node and picks one for each. The kubelet on that node starts the containers through the runtime and reports status back. Try it on a local cluster You need a cluster and kubectl. The learning environment page lists kind, which runs cluster nodes as Docker containers, and minikube, which runs a single-node cluster on your machine, among other options. Install one of them and kubectl, then confirm that kubectl get nodes shows a node in the Ready state. kubectl apply -f hello.yaml kubectl get deployments kubectl get pods kubectl scale deployment/hello-node --replicas=4 kubectl get pods kubectl delete pod <one-of-the-pod-names> kubectl get pods kubectl get events kubectl port-forward deployment/hello-node 8080:8080 Run them in order. The first creates the Deployment and, through it, two Pods. Scaling to four changes the desired state, and Kubernetes starts two more. Deleting a Pod does not lower the count: the ReplicaSet notices the gap and creates a replacement, the behavior the docs list as replica replacement under self-healing. kubectl get events shows the hand-offs from the previous section as they happened. The port-forward command tunnels local port 8080 to a Pod, so you can open http://localhost:8080 and see the test image echo your request back, the same image the official Hello Minikube tutorial uses. It does not return until you stop it with Ctrl+C. When you are done, run kubectl delete -f hello.yaml to remove the Deployment and its Pods. Where Kubernetes stands in October 2026 That demo skipped everything production needs: pinned versions, resource limits, probes, security controls and a safe way to take traffic. The rest of this post adds them, starting with the versions you can choose from. The project maintains release branches for the three most recent minor releases and supports each patch series for roughly fourteen months: twelve in the standard period, then two in maintenance mode, when new releases cover only CVE fixes, dependency issues and critical core component issues. As of October 10, 2026, the releases page lists the following. | Minor | Latest patch listed | End of life | | --- | --- | --- | | 1.37 | 1.37.1 (September 15, 2026) | October 28, 2027 | | 1.36 | 1.36.5 | June 28, 2027 | | 1.35 | 1.35.9 | February 28, 2027 | | 1.34 | 1.34.12 | October 27, 2026 | The next patch releases for all four lines are scheduled for October 13, 2026, so the patch numbers above will be stale within days. The fourth row, 1.34, is in maintenance mode, which began on August 27, 2026, and its end of life is under three weeks away. Plan the move to 1.35 or later now, and upgrade one minor version at a time, because kubeadm does not support skipping minor versions. The last two releases were large. Kubernetes 1.36 (Haru), released April 22, 2026, had 70 enhancements: 18 Stable, 25 Beta and 25 Alpha. Kubernetes 1.37 (Garhwal) had 67: 16 Stable, 23 Beta, 27 Alpha and one deprecation or removal. You do not need most of them. You need the handful that change how you write manifests. The baseline, one manifest at a time The target is one service called api in a namespace called shop, behind a shared Gateway, with three replicas spread across zones. Replace the names, the host, the image digest and the GatewayClass. Everything else is meant to be applied as written. !Mermaid Diagram Save each block below as its own file, in order, using the file name in the first comment line. 1. Namespace: enforce Pod Security Admission # 00-namespace.yaml apiVersion: v1 kind: Namespace metadata: name: shop labels: pod-security.kubernetes.io/enforce: restricted pod-security.kubernetes.io/enforce-version: latest pod-security.kubernetes.io/warn: restricted pod-security.kubernetes.io/audit: restricted shared-gateway-access: "true" Pod Security Admission has been stable since 1.25. The restricted profile rejects Pods that run as root, allow privilege escalation, keep default Linux capabilities or lack a seccomp profile. Setting enforce, warn and audit together means you see violations in three places. The last label is used by the Gateway in step 4. 2. Deployment # 01-deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: api namespace: shop labels: app.kubernetes.io/name: api spec: # replicas is omitted on purpose: the HPA in step 6 owns it revisionHistoryLimit: 3 strategy: type: RollingUpdate rollingUpdate: maxUnavailable: 0 maxSurge: 1 selector: matchLabels: app.kubernetes.io/name: api template: metadata: labels: app.kubernetes.io/name: api spec: hostUsers: false automountServiceAccountToken: false terminationGracePeriodSeconds: 45 securityContext: runAsNonRoot: true runAsUser: 10001 runAsGroup: 10001 seccompProfile: type: RuntimeDefault topologySpreadConstraints: - maxSkew: 1 topologyKey: topology.kubernetes.io/zone whenUnsatisfiable: ScheduleAnyway labelSelector: matchLabels: app.kubernetes.io/name: api containers: - name: api # Pin by digest. Replace this placeholder with your real image digest. image: registry.example.com/shop/api@sha256:0000000000000000000000000000000000000000000000000000000000000000 ports: - name: http containerPort: 8080 resources: requests: cpu: 250m memory: 256Mi limits: memory: 256Mi resizePolicy: - resourceName: cpu restartPolicy: NotRequired - resourceName: memory restartPolicy: RestartContainer securityContext: allowPrivilegeEscalation: false readOnlyRootFilesystem: true capabilities: drop: ["ALL"] startupProbe: httpGet: path: /healthz port: http periodSeconds: 2 failureThreshold: 30 readinessProbe: httpGet: path: /readyz port: http periodSeconds: 5 failureThreshold: 2 livenessProbe: httpGet: path: /healthz port: http periodSeconds: 10 failureThreshold: 3 lifecycle: preStop: sleep: seconds: 10 volumeMounts: - name: tmp mountPath: /tmp volumes: - name: tmp emptyDir: sizeLimit: 64Mi Four settings carry most of the weight. hostUsers: false puts the Pod in its own user namespace, so root inside the container maps to an unprivileged user on the node. User namespaces reached GA in 1.36 and are Linux-only. The concept page lists the node requirements: Linux 6.3 or later for idmap mounts on tmpfs, runc 1.2 or later or crun 1.9 or later, and containerd 2.0 or later or CRI-O 1.25 or later. Check your nodes before relying on the setting. maxUnavailable: 0 with maxSurge: 1 makes a rollout add one new Pod and wait for it to be Ready before removing an old one. The topology spread constraint keeps replicas balanced across zones. ScheduleAnyway means a zone outage does not leave Pods Pending. Switch to DoNotSchedule only if imbalance is worse than reduced capacity for your service. The three probes answer three different questions. The startup probe allows a slow boot of up to 60 seconds (30 failures at 2 second intervals) before the other probes begin. Readiness decides whether the Pod receives traffic, and should check dependencies you cannot serve without. Liveness should check only that the process is not wedged. Never make liveness depend on a database, or a database outage becomes a restart storm. The preStop sleep and the 45 second grace period solve a race. Endpoint removal and SIGTERM happen at about the same time, so for a few seconds a terminating Pod can still receive requests. Sleeping 10 seconds before SIGTERM gives proxies time to stop sending traffic. Your application must still handle SIGTERM by finishing in-flight requests inside the remaining 35 seconds. The sleep hook action is Stable since 1.34, when the PodLifecycleSleepAction feature gate became locked on, so every supported minor has it. The lifecycle hooks page lists it beside the exec and HTTP handlers. The kubelet runs it, so your image needs no sleep binary. Memory request equals memory limit, and there is no CPU limit. Memory cannot be compressed, so an equal limit makes OOM behavior predictable. CPU can be throttled, and a CPU limit often throttles bursts the node could have served. The Pod still gets a CPU guarantee through its request. Revisit this choice if you run a multi-tenant cluster that needs hard caps. 3. Service # 02-service.yaml apiVersion: v1 kind: Service metadata: name: api namespace: shop spec: type: ClusterIP selector: app.kubernetes.io/name: api ports: - name: http port: 80 targetPort: http The Service gives the Pods one stable virtual IP and a DNS name, api.shop.svc. It is ClusterIP on purpose. Traffic from outside the cluster enters through the Gateway, not through a Service of type LoadBalancer per application, and not through the deprecated externalIPs field covered later. 4. Gateway and HTTPRoute # 03-gateway-and-route.yaml apiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: name: public namespace: gateway-system spec: gatewayClassName: example-gateway-class listeners: - name: https protocol: HTTPS port: 443 hostname: api.example.com tls: mode: Terminate certificateRefs: - kind: Secret name: api-example-com-tls allowedRoutes: namespaces: from: Selector selector: matchLabels: shared-gateway-access: "true" --- apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: api namespace: shop spec: parentRefs: - name: public namespace: gateway-system sectionName: https hostnames: - api.example.com rules: - matches: - path: type: PathPrefix value: / backendRefs: - name: api port: 80 Gateway API splits ownership. The platform team owns the Gateway and its listeners, and application teams own the routes that attach to it. Gateway and HTTPRoute have been GA since Gateway API v1.0. Gateway API v1.6.0, released June 30, 2026, also moved TCPRoute and UDPRoute to the Standard channel at API version v1. Gateway API is not built into the API server. You install its CRDs and a controller that implements your GatewayClass, and example-gateway-class is a placeholder for that controller's class name. The allowedRoutes selector is why step 1 labeled the shop namespace: a route from an unlabeled namespace is not allowed to attach. The manifest assumes two objects already exist: the gateway-system namespace, and a kubernetes.io/tls Secret named api-example-com-tls inside it. Per the Gateway CRD at v1.6.0, a certificate reference into another namespace is invalid unless a ReferenceGrant allows it, and the listener then reports ResolvedRefs as False with the reason RefNotPermitted. Create both before you apply this file. 5. PodDisruptionBudget # 04-pdb.yaml apiVersion: policy/v1 kind: PodDisruptionBudget metadata: name: api namespace: shop spec: maxUnavailable: 1 selector: matchLabels: app.kubernetes.io/name: api A PodDisruptionBudget limits voluntary evictions, such as a node drain during an upgrade. With it, a drain can take one api Pod at a time and must wait for the replacement to be Ready. It does not protect against crashes or node failures. Prefer maxUnavailable over minAvailable, so the budget keeps working when the HPA changes the replica count. 6. HorizontalPodAutoscaler # 05-hpa.yaml apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata: name: api namespace: shop spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: api minReplicas: 3 maxReplicas: 12 metrics: - type: Resource resource: name: cpu target: type: Utilization averageUtilization: 70 behavior: scaleDown: stabilizationWindowSeconds: 300 policies: - type: Percent value: 25 periodSeconds: 60 Leaving spec.replicas out of the Deployment makes the HPA the only writer of the replica count. If both write, every kubectl apply resets it. On the first apply the Deployment starts at one replica until the HPA raises it to minReplicas, so apply both in the same command. CPU utilization is measured against the request, which is why requests must be honest. Resource metrics come from the metrics.k8s.io API, usually served by metrics-server. That API graduated to Stable in 1.37 after nearly nine years in Beta. The scale-down policy above removes at most 25% of replicas per minute, and only after five minutes of consistently lower demand, which avoids flapping on spiky traffic. 7. NetworkPolicy # 06-networkpolicy.yaml apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: default-deny namespace: shop spec: podSelector: {} policyTypes: - Ingress - Egress --- apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: allow-gateway-to-api namespace: shop spec: podSelector: matchLabels: app.kubernetes.io/name: api policyTypes: - Ingress ingress: - from: - namespaceSelector: matchLabels: kubernetes.io/metadata.name: gateway-system ports: - protocol: TCP port: 8080 --- apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: allow-dns-egress namespace: shop spec: podSelector: {} policyTypes: - Egress egress: - to: - namespaceSelector: matchLabels: kubernetes.io/metadata.name: kube-system ports: - protocol: UDP port: 53 - protocol: TCP port: 53 NetworkPolicy objects are enforced only if your CNI plugin implements them. On a plugin that does not, these manifests are accepted and silently do nothing, so test with a throwaway Pod that should be blocked. Default-deny egress also blocks your database and every external API, so add one allow rule per destination before you ship. Some Gateway controllers run their data plane in a namespace other than gateway-system; adjust the selector to match. Apply and verify kubectl apply --dry-run=server -f 00-namespace.yaml kubectl apply -f 00-namespace.yaml kubectl apply --dry-run=server -f 01-deployment.yaml -f 02-service.yaml -f 04-pdb.yaml -f 05-hpa.yaml -f 06-networkpolicy.yaml kubectl apply -f 01-deployment.yaml -f 02-service.yaml -f 04-pdb.yaml -f 05-hpa.yaml -f 06-networkpolicy.yaml kubectl apply -f 03-gateway-and-route.yaml kubectl -n shop rollout status deploy/api kubectl -n shop get deploy,hpa,pdb,pods -o wide kubectl -n shop get httproute api -o jsonpath='{.status.parents[*].conditions[*].type}{"\n"}' The server-side dry run catches schema errors and admission rejections, including Pod Security violations, without creating anything. The last command prints the route's status condition types; look for Accepted and ResolvedRefs, and read each condition's status field when either is missing. Resize a running Pod without restarting it In-place Pod resize reached Stable in 1.35, after starting as Alpha in 1.27. A container's resources are now the desired state, the Pod status shows what is actually applied, and you change the desired values through the Pod's resize subresource. Per the task documentation, the server must be at 1.33 or later and kubectl at 1.32 or later. kubectl -n shop patch pod <pod-name> --subresource resize --patch \ '{"spec":{"containers":[{"name":"api","resources":{"requests":{"cpu":"500m"}}}]}}' kubectl -n shop get pod <pod-name> \ -o jsonpath='{.status.containerStatuses[0].resources}{"\n"}' The resizePolicy in step 2 decides the cost. With CPU set to NotRequired, the container keeps running while the kubelet applies the change. With memory set to RestartContainer, a memory change restarts that container, because many runtimes cannot change their heap bounds on the fly. If one patch changes both, the container restarts, as the documentation states. A resize edits one Pod. The Deployment template still holds the old values, so replacement Pods start with them. Use resize for emergency relief and for autoscalers, since the Vertical Pod Autoscaler's InPlaceOrRecreate mode, which the same post reports as Beta and built on this feature, and change the template for anything lasting. Scale a worker to zero HorizontalPodAutoscaler scale to zero reached Beta in 1.37 and is on by default: the HPAScaleToZero feature gate is enabled on both the kube-apiserver and the kube-controller-manager. It works only with an object metric or an external metric. CPU and memory come from running Pods, and at zero replicas there is nothing left to measure, so there would be no signal to scale back up. # queue-worker-hpa.yaml apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata: name: queue-worker namespace: shop spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: queue-worker minReplicas: 0 maxReplicas: 10 metrics: - type: External external: metric: name: queue_consumer_lag selector: matchLabels: name: worker_tasks target: type: AverageValue averageValue: "30" This HPA asks for one replica per 30 queued tasks and drops to zero when the queue is empty. The external metric must be served through the External Metrics API by an adapter such as Prometheus Adapter, and you can confirm Kubernetes can read it with kubectl get --raw before creating the HPA. The target type matters. The release post's example uses Value, but in the controller source at v1.37.1 a Value target multiplies the metric-to-target ratio by the number of ready Pods, so a queue that stays long keeps multiplying the replica count. AverageValue computes the queue length divided by 30, rounded up, and the HPA walkthrough uses it for its own queue example. Start the Deployment with at least one replica. The release post says manually setting a Deployment to zero has always paused autoscaling, and a workload at zero is resumed only if the HPA itself scaled it down and recorded the ScaledToZero condition. Do not use this for the api service. The release notes are explicit that Kubernetes Services do not buffer requests while no Pods are ready, so request-driven HTTP workloads need a separate buffering layer. Scale to zero fits work that can wait in a durable queue, where a cold start costs seconds of latency and nobody is waiting on a socket. What to fix before you upgrade Four 2026 changes can bite a cluster that has been quiet. This script covers the two you can detect from the API. The other two need a look at your nodes and your release notes. #!/usr/bin/env bash # preflight.sh: run with cluster-admin credentials set -euo pipefail echo "== 1. Is ingress-nginx still running?" kubectl get pods --all-namespaces --selector app.kubernetes.io/name=ingress-nginx echo "== 2. Services that still set spec.externalIPs" kubectl get svc --all-namespaces -o json \ | jq -r '.items[] | select((.spec.externalIPs // []) | length > 0) | "\(.metadata.namespace)/\(.metadata.name)"' echo "== 3. Client and server versions" kubectl version Ingress NGINX is retired The Kubernetes project said best-effort maintenance of Ingress NGINX would continue until March 2026. After that there would be no releases, no bug fixes and no security updates, although existing deployments keep working. That date has passed, and the GitHub repository now shows it as archived on March 24, 2026 and read-only. A statement from the Steering and Security Response Committees, citing internal Datadog research, put its use at about 50% of cloud native environments. The statement is blunt about the cost of waiting: staying on a retired controller leaves you and your users vulnerable to attack, and none of the alternatives is a direct drop-in replacement. Install a Gateway API controller next to Ingress NGINX, translate your routes (the project's ingress2gateway tool reached 1.0 in March 2026), test, then move traffic over gradually. Review the translated output by hand. The ingress2gateway 1.0 post shows the tool warning that it does not support the configuration-snippet annotation, and the retirement notice names the "snippets" annotations, which add arbitrary NGINX configuration directives, as an option now considered a serious security flaw. Rebuild what each snippet did with Gateway API features instead of copying it across. Service externalIPs is deprecated The .spec.externalIPs field was formally deprecated in 1.36 because it assumes every user in the cluster is trusted, which enables the attacks described in CVE-2020-8554. A future minor release will drop the behavior from kube-proxy and require conformant implementations not to support it. If the preflight script prints anything, plan a replacement. The same post calls a hand-assigned LoadBalancer IP the easiest option and also the worst, because users who may set it can still replicate the CVE. It lists two better routes: a load balancer controller such as MetalLB, or a Gateway whose address you set in .spec.addresses. As a precaution, enable the DenyServiceExternalIPs admission controller so nobody adds new uses. cgroup v1 nodes will not start Support for cgroup v2 has been stable since 1.25, and cgroup v1 went into maintenance mode in 1.31. Starting with 1.35, the kubelet option failCgroupV1 defaults to true, so the kubelet refuses to start on a cgroup v1 node unless you set it to false, a temporary override while removal is tracked in KEP-5573. On each node, stat -fc %T /sys/fs/cgroup prints cgroup2fs for v2 and tmpfs for v1. Fix every v1 node before the upgrade, not during it. SELinux volume labeling changes In 1.37 the SELinuxMount and SELinuxChangePolicy features reached Stable and are enabled by default. Volumes then mount with one SELinux context instead of being relabeled recursively, but only for CSI drivers that opt in with .spec.seLinuxMount: true. Pods with different SELinux labels that share a volume on one node can therefore fail to start. The release notes advise setting .spec.seLinuxChangePolicy to Recursive on a Pod to keep the old behavior, and they say clusters without SELinux see no effect. If your nodes enforce SELinux, as on the RHEL family, read the project's post on SELinux volume label changes and their implications in 1.37 before upgrading, and test any workload where Pods share a volume. Where AI workloads fit The same CNCF survey found that 66% of organizations hosting generative AI models use Kubernetes for some or all of their inference. It also found that only 7% of organizations deploy models daily and that 44% do not yet run AI or ML workloads on Kubernetes. Most teams are early, which makes the platform choices you make now easy to get right and expensive to redo. The baseline in this post is the right foundation for an inference service, with one change: GPUs and other devices should be requested through Dynamic Resource Allocation, not through the old device-plugin counters. DRA reached GA in 1.34, and in 1.37 the ResourceClaim .status.devices field graduated to Stable, which lets drivers report device-specific status. Read the DRA concept pages before designing GPU scheduling. A seven-day rollout plan | Day | Do this | Done when | | --- | --- | --- | | 1 | Run preflight.sh, check node cgroup versions, list your supported-version dates | You know which of the four 2026 changes apply to you | | 2 | Apply steps 1 to 3 and 5 to 6 in a staging namespace | rollout status succeeds and the HPA reports metrics | | 3 | Install a Gateway API controller in staging, create gateway-system and the TLS Secret, then apply step 4 | The HTTPRoute shows Accepted and ResolvedRefs | | 4 | Apply step 7 and add one egress rule per real dependency | A blocked test Pod fails and the app still works | | 5 | Enable hostUsers: false on one workload | The Pod starts and passes its probes | | 6 | Drain a node and watch the PodDisruptionBudget | Drain completes with at most one api Pod down at a time | | 7 | Upgrade staging one minor version, then production | Both clusters pass preflight.sh on the new version | Closing notes None of this requires a new tool. Every manifest above is plain Kubernetes, and every feature labeled Stable is safe to depend on. Features labeled Beta, such as HPA scale to zero, are on by default in 1.37, but they can still change, so pin your cluster version and read the release notes before each upgrade. Managed Kubernetes services often lag the upstream release by weeks or months. Run kubectl version and check your provider's release calendar before assuming a 1.37 feature is available to you. The upstream support dates in this post apply to the community project, and your provider may set its own. --- Published via ZyVOP — Write once in Markdown, auto-backup to GitHub, and syndicate to Dev.to, Medium & Hashnode in 1 click.