controller-runtime

klog, zap, logrus などで stderr に吐かれるログのレベルを Datadog で良い感じにする こんにちは、@key60228 です。 新しく立ち上げた GKE Cluster に Datadog Agent を入れてログを有効化してみたところ、とんでもない数のエラーログが吐かれていました。 まだ何も特別なことをしていないのに…？？？と不思議に思いログの詳細を見てみたところ、特に問題なさそうなログがほとんどでした。 結論としては、各コンポーネントが吐くログが Datadog が解釈できる形式で構造化されておらず、エラーレベルとして判定されてしまっていることが原因でした。 というわけでこの記事では各コンポーネント、ロガーがどんな構造で出力していて、どうすれば正しいログレベルで表示できるのかについてまとめていければと思います。 ソリューションを求めている方へ Datadog のログパイプラインで加工してあげてください。 Terraform pipelines.tf resource "datadog_logs_custom_pipeline" "container_log_level" { name = "container-log-level" description = "stderr 由来で error になったコンテナログの status を、本文の klog / zap / logfmt のレベルから付け直す。" is_enabled = true filter { query = "kube_cluster_name:*" } processor { grok_parser { name = "klog / zap / logfmt からレベルを取り出す" is_enabled = true source = "message" samples = [ "I0911 07:22:25.408586 1 cni_writer.go:930] Successfully synced \"gke-xxxxx-xxxxx-nod-xxxxxxxx-xxxx\"", # klog (GKE cni-writer) "E0911 07:10:07.225293 1 base_controller.go:161] \"controller failed to sync\" err=\"...\" key=\"image-sync\"", # klog (OCM work-agent) "2026-09-11T07:19:45Z\tINFO\tcontrollers.Spoke\trequeueing\t{\"request\": {\"name\":\"xxx\",\"namespace\":\"fleetconfig-system\"}, \"after\": \"1m0s\"}", # zap console、タブ区切り (fleetconfig-controller) "2026-09-11T04:39:22.159Z error logging/logging.go:16 NodeCollector: Failed to export metrics {\"kind\": \"receiver\"}", # zap console、スペース区切り (gke-metrics-agent) "time=2026-09-11T06:13:28Z level=info msg=Reconciling subsys=sfc reasons=[...]", # logfmt / logrus (cilium) ] grok { support_rules = "" match_rules = file("${path.module}/grok/container_log_level.grok") } } } processor { lookup_processor { name = "klog の severity 1 文字をレベル語に直す" is_enabled = true source = "klog.severity" target = "level" lookup_table = [ "I,info", "W,warning", "E,error", "F,critical", ] } } processor { lookup_processor { name = "zap のレベルを status remapper が解釈できる語に直す" is_enabled = true source = "zap.level" target = "level" lookup_table = [ "DEBUG,debug", "INFO,info", "WARN,warning", "ERROR,error", "DPANIC,critical", "PANIC,critical", "FATAL,critical", ] } } processor { lookup_processor { name = "logrus のレベルを status remapper が解釈できる語に直す" is_enabled = true source = "logfmt.level" target = "level" lookup_table = [ "trace,debug", "debug,debug", "info,info", "warning,warning", "error,error", "fatal,critical", "panic,critical", ] } } processor { status_remapper { name = "level を status にする" is_enabled = true sources = ["level"] } } } grok/container_log_level.grok klog %{regex("[IWEF]"):klog.severity}%{regex("\d{4} \d{2}:\d{2}:\d{2}\.\d+")}\s+%{integer:klog.thread_id}\s+%{regex("[^\]]+"):klog.location}]\s?%{data:klog.message} zap %{regex("\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(\.\d+)?(Z|[+-]\d{2}:\d{2})"):zap.timestamp}[\t ]+%{regex("(DEBUG|INFO|WARN|ERROR|DPANIC|PANIC|FATAL|debug|info|warn|error|dpanic|panic|fatal)"):zap.level:uppercase}[\t ]+%{data:zap.message} logfmt_with_time time=%{notSpace:logfmt.time}\s+level=%{word:logfmt.level:lowercase}\s+%{data:logfmt.message} logfmt level=%{word:logfmt.level:lowercase}\s+%{data:logfmt.message} Datadog Log Management のログレベルの仕組み Datadog のログでは status という予約済み属性が Log Explorer で表示されるログレベルに対応しています。1 Datadog Agent を使っている場合、この status は次の3つのタイミングで決まります。 Datadog Agent がログを送る時 Datadog が JSON ログをパースする時 ログパイプラインの Status Remapper を通る時 ① Datadog Agent がログを送る時 Kubernetes の Datadog Agent (各ノードの DaemonSet) では stdout に出たものを info、stderr に出たものを error と判定して Datadog に送っているようです。 github.com/DataDog/datadog-agent/blob/f85c4980d2b… ② Datadog が JSON ログをパースする時 次に、ログが JSON 形式であれば、Datadog は取り込み時の前処理で status、severity、level などの属性を探し、見つかった値で status を上書きします。2 {"level":"info","msg":"..."} のようなログは、stderr に書かれていても info として扱われます。 ③ ログパイプラインの Status Remapper を通る時 最後に、ログパイプライン に Status Remapper を置くと、任意の属性の値で status を上書きできます。 Status Remapper は指定された属性の値を先頭の文字列で以下の通り判定してくれます。3 値 status 0 から 7 の整数 Syslog の規格に従う emerg または f で始まる文字列 (大文字小文字を区別しない) emerg (0) a で始まる文字列 (大文字小文字を区別しない) alert (1) c で始まる文字列 (大文字小文字を区別しない) critical (2) err で始まる文字列 (大文字小文字を区別しない) error (3) w で始まる文字列 (大文字小文字を区別しない) warning (4) n で始まる文字列 (大文字小文字を区別しない) notice (5) i で始まる文字列 (大文字小文字を区別しない) info (6) d、t、v、trace、verbose で始まる文字列 (大文字小文字を区別しない) debug (7) o または s で始まる文字列、または OK か Success に一致する文字列 (大文字小文字を区別しない) OK 上記以外 info (6) つまり、プレーンテキストの本文にレベルを埋め込んでいるログは、②の前処理では拾われず、③のパイプラインを用意しない限り①のストリームだけで status が決まります。 今回の自分の環境のエラーログは、まさにこのパターンでした。 対策と各コンポーネント/ロガーの仕組み 根本的には各コンポーネントが JSON でログを吐くか、レベルに応じて stdout/stderr を使い分けていればこういった問題は起きません。 しかし GKE が管理するシステムコンポーネントは出力形式をこちら側では変えられず、OSS のコンポーネントも全てが出力形式を選べるわけではありません。 また、Datadog Agent でストリームと status の mapping を変更する方法も現時点ではなさそうです。 となるとメッセージ本文からレベルを取り出し、Status Remapper で status 属性を更新するログパイプラインを用意するのが良さそうです。 今回の自分の環境で問題になったログは、出力元のロガーで分けると klog、zap、logrus の3種類でした。 これらのロガーで、どのようにレベルを抽出すれば良いか見ていきます。 klog klog は glog の fork で、Kubernetes 本体も使用しているロガーです。 kubernetes/klog 今回の環境では、GKE のシステムコンポーネント (cni-writer、gke-metadata-server、proxy-agent etc.) や cert-manager などが klog かつ stderr でログを吐いていました。 klog の構造は以下のようになっており、先頭の L が severity を指しています。 Lmmdd hh:mm:ss.uuuuuu threadid file:line] msg github.com/kubernetes/klog/blob/ae3961e316a310e71… severity は I が INFO、W が WARNING、E が ERROR、F が FATAL を指します。 github.com/kubernetes/klog/blob/ae3961e316a310e71… 例えば proxy-agent の次のログは INFO ですが、stderr に書かれているため Datadog 上では ERROR になっていました。 そこで、まずは Grok Parser で属性をパースしてあげます。 klog %{regex("[IWEF]"):klog.severity}%{regex("\d{4} \d{2}:\d{2}:\d{2}\.\d+")}\s+%{integer:klog.thread_id}\s+%{regex("[^\]]+"):klog.location}]\s?%{data:klog.message} 上記ルールで IWEF の値を klog.severity に書き出しましたが、klog の場合はこの severity をそのまま Status Remapper に渡すことはできません。 I と W は期待通り info と warning になり、F は emergency になりますが、E は何にも当てはまらず、info になってしまいます。 そこで、Lookup Processor で klog.severity を調整してから level 属性に入れます。 (ついでに F も critical に調整しています) I,info W,warning E,error F,critical その後 Status Remapper を挟んであげることで、proxy-agent の INFO ログは info として表示されるようになりました。 zap zap は Uber が開発しているロガーで、Kubernetes 周りでは controller-runtime を使ったコントローラーの多くが zap を使っています。 uber-go/zap 今回の環境では、gke-metrics-agent や 別記事 で導入していた Open Cluster Management の fleetconfig-controller が zap でログを吐いていました。 zap には JSON で出力する JSON encoder と、人が読みやすいテキストで出力する console encoder があります。 github.com/uber-go/zap/blob/4892335e05f14bce8a98a… console encoder はデフォルトでは、タイムスタンプ、レベル、ロガー名、呼び出し元、メッセージ、追加フィールド (JSON) を区切り文字でつないで 1 行にします。 TIMESTAMP LEVEL LOGGER CALLER MESSAGE {FIELDS} zap の console encoder の出力は、次のルールで分解します。 zap %{regex("\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(\.\d+)?(Z|[+-]\d{2}:\d{2})"):zap.timestamp}[\t ]+%{regex("(DEBUG|INFO|WARN|ERROR|DPANIC|PANIC|FATAL|debug|info|warn|error|dpanic|panic|fatal)"):zap.level:uppercase}[\t ]+%{data:zap.message} 同じ console encoder でも、出力の細かいところはコンポーネントによって異なっているため、区切り文字はタブと半角スペース、レベルは大文字と小文字の両方に対応しています。 zap のレベルは DEBUG、INFO、WARN、ERROR、DPANIC、PANIC、FATAL とあり、今のままだと DPANIC が debug に、PANIC が info になってしまいます。 ついでに FATAL は klog に合わせて critical として扱うよう、Lookup Processor で調整してあげます。 github.com/uber-go/zap/blob/4892335e05f14bce8a98a… "DEBUG,debug", "INFO,info", "WARN,warning", "ERROR,error", "DPANIC,critical", "PANIC,critical", "FATAL,critical", なお、controller-runtime の zap のオプションをコマンドライン引数に結び付けているコントローラーであれば、--zap-encoder=json を渡して JSON encoder に切り替えられます。 JSON encoder の出力には level 属性が含まれるので、前処理で status に反映され、パイプラインは要りません。 logrus logrus もメジャーな構造化ロガーで、今回の環境では Zitadel が logrus でログを出していました。 sirupsen/logrus 出力形式は TextFormatter と JSONFormatter から選べ、TextFormatter は端末以外に出力するとき key=value を並べた logfmt 風の形式になります。 time="2026-10-13T05:58:18Z" level=INFO msg="structured logger configured" caller="..." error="..." レベルは level= の後ろに info、warning、error のような語で書かれます。 スクリーンショットのログは level=INFO ですが、Datadog 上では ERROR になっていました。 logrus の TextFormatter の出力は、次の 2 つのルールで分解します。 logfmt_with_time time=%{notSpace:logfmt.time}\s+level=%{word:logfmt.level:lowercase}\s+%{data:logfmt.message} logfmt level=%{word:logfmt.level:lowercase}\s+%{data:logfmt.message} TextFormatter は設定によってタイムスタンプを出さないことがあるため、time= で始まる行と level= で始まる行の両方に対応しています。 今回のログのように時刻を time="..." と引用符で囲む場合も、引用符ごと notSpace で logfmt.time に取り込まれるので問題ありません。 logrus のレベルは以下の通りなので、zap 同様に Lookup Processor で調整してあげましょう。 github.com/sirupsen/logrus/blob/6ef0748d1177dadae… "trace,debug", "debug,debug", "info,info", "warning,warning", "error,error", "fatal,critical", "panic,critical", これでログレベルが正しく表現されるようになりました。 ! この記事の本筋とズレるので本文では言及しませんでしたが、ZITADEL はオプションでログフォーマットを JSON 化できます。4 まとめ パイプラインを適用したあとは、status:error のログがほとんどなくなりました。 これで万事解決…とは思いつつ、klog、zap、logrus とは異なる出力形式を持つロガーを採用しているコンポーネントが出てきたらその都度対応しないといけない…？ 🤔 GKE とか各 KaaS のシステムコンポーネント以外は大概ログの format などをこっちで指定できるから特に問題にならないのかな〜〜とは思いつつ、こんな複雑なことをしないといけないのはちょっと不思議な気がするな〜〜 5 もっと良い方法あるよって方、もし居たら教えてください〜〜 最後に AI Shiftではエンジニアの採用に力を入れています！少しでも興味を持っていただけましたら、カジュアル面談でお話しませんか？（オンライン・19時以降の面談も可能です！） 【面談フォームはこちら】 hrmos.co 脚注 属性とエイリアス設定 - Datadog Docs ↩︎ Pipelines (Preprocessing) - Datadog Docs ↩︎ Log Status Remapper - Datadog Docs ↩︎ ZITADEL Production Setup - ZITADEL Docs ↩︎ 「こんな複雑なことをしないといけないのはおかしい」というソフトウェアエンジニアの勘 - valid,invalid ↩︎

Why did Reconcile() run again? Easy event tracing in controller-runtime !Gopher is exploring Kubernetes events If you build Kubernetes operators in Go using controller-runtime (Kubebuilder or Operator SDK), you probably know this pain. You update one small field in your Custom Resource, look at the logs, and see that Reconcile() was called 4 or 5 times in a row. Why? Was it your Spec update? A Status update from a child Deployment? An external ConfigMap sync? Or just a standard requeue? By default, controller-runtime gives you a very simple method signature: func (r *MyReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) Look at ctrl.Request - it contains only Namespace and Name. The actual Kubernetes events that triggered this execution are lost inside the internal workqueue. Why does this happen? (Workqueue Deduplication) To protect the Kubernetes API Server from too many requests, controller-runtime uses a deduplicating workqueue. If 5 events happen almost at the same time for default/my-cr (for example, Spec change + Deployment status update + Pod status change), the workqueue compresses them into a single Reconcile() call. This is great for cluster performance, but it makes debugging really hard: - You don't see which resource actually triggered the call. - You don't see what changed (Spec, Status, or Metadata). - It's hard to trace child resources (Deployments, Services) back to the main CR loop. My solution: whyreconcile To solve this problem for my own projects, I built whyreconcile - a simple, lightweight Go package for controller-runtime. It doesn't change your business logic. Instead, it hooks into event handlers, compares old and new object states (shallow diff), saves the cause metadata, and prints a clear summary right before your Reconcile() starts. How it looks in your logs: reconcile causes controller=my-controller target=default/my-cr causes=3 summary=PrimaryResourceSpecUpdate:1, SecondaryResourceStatusOrMetadata:2 sources=2 In just one log line, you immediately see: - Target: Which resource is being reconciled (default/my-cr). - Causes count: 3 events happened since the last run. - Summary: 1 Spec update on the Primary CR and 2 Status/Metadata updates on secondary resources. How it works under the hood The idea is very straightforward: 1. Catch events: - For Primary resources (For), it intercepts events using predicates and shallow diffs. - For Secondary/External resources (Watches), it uses a custom TypedEventHandler to map child objects (like Deployments) to the parent CR. 2. Store causes: As events flow in, whyreconcile saves cause info in a thread-safe in-memory store under the target resource key (Namespace/Name). 3. Print before Reconcile: When controller-runtime takes a request from the workqueue, whyreconcile extracts all stored causes for this key and prints them before calling your original Reconcile() code. Quick Setup (takes 2 minutes) You don't need to change anything inside your Reconcile() function. Just update SetupWithManager: 1. Install go get https://github.com/arsenalzp/whyreconcile 2. Wrap your setup code import ( "https://github.com/arsenalzp/whyreconcile" appsv1 "k8s.io/api/apps/v1" ctrl "sigs.k8s.io/controller-runtime" ) func (r *MyReconciler) SetupWithManager(mgr ctrl.Manager) error { // 1. Create analyzer analyzer := whyreconcile.NewAnalyzer() return ctrl.NewControllerManagedBy(mgr). // 2. Wrap Primary resource For(&v1alpha1.MyCR{}, analyzer.NewPrimaryOpts()). // 3. Wrap Secondary resource (e.g., child Deployment) Watches( &appsv1.Deployment{}, analyzer.NewSecondaryHandler( "child-deployment", handler.EnqueueRequestForOwner( mgr.GetScheme(), mgr.GetRESTMapper(), &v1alpha1.MyCR{}, ), ), ). // 4. Wrap your Reconciler Complete(analyzer.WrapReconcile(r)) } And that's it! Next time you run your operator locally or in a dev cluster, you will see exactly why each Reconcile() was triggered. Try it out The project is open-source and free to use: 📁 GitHub: arsenalzp/whyreconcile If you write Kubernetes operators, give it a try in your dev environment! I'd love to hear your feedback or ideas in the comments.

Kubernetes Controllers: The Cache-Staleness Bug at Scale > Key takeaways > - Structure reconcilers as ordered, independently testable steps, each reporting its own status condition, over one monolithic Reconcile function. > - Informer caches lag the API server by tens of milliseconds at scale; a stale read inside a MergeFrom patch can clobber a condition you set 49ms earlier. > - Fix status patching with an uncached reader inside RetryOnConflict; Kubernetes v1.36 (April 2026) now lets controllers detect and skip behind-cache reconciles. > - Even a logging change is a production change: a duplicate log key once dropped every log line from the busiest controller in an environment. Everything below comes from operating a major European automotive manufacturer's internal Kubernetes platform, running a fleet of custom controllers on managed GKE. Names and identifiers are generic on purpose. The failure modes are exact. What is the step-based reconciler pattern? The step-based reconciler pattern splits a controller's reconcile loop into an ordered sequence of small, composable steps. Each step reports its own status condition, instead of one monolithic function doing everything. In practice, this keeps provisioning logic table-testable and makes partial progress visible. Across our controller suite, it turned a 400-line reconcile body into a readable chain. It is the same reconciliation discipline behind the [Kubernetes as a Service control plane](/en/blog/build-kubernetes-as-a-service/) we operate, where dozens of resource types share one contract. A monolithic Reconcile becomes unreadable fast. Picture one custom resource that has to provision networking, IAM, namespaces, quotas, and DNS. Have you ever tried to unit-test a function like that? So we organize each controller as controller.go plus steps/creation/ and steps/deletion/ packages. Every step embeds a shared BaseStep. It holds the controller-runtime client, an uncached reader, the scheme, a logger, and an event recorder. Each step also exposes explicit Ready and Reconciling conditions. An Environment resource, for example, runs a chain like this: EnsureFinalizer, ResolveClusterConfigRef, ComputeTargetNamespaceName, EnsureNamespace, ApplyResourceQuotas, ApplyLimitRanges, ApplyServiceAccounts, ApplyRoleBindings, ApplyNetworkPolicies, DeleteFinalizer. Each step is small, ordered, and independently testable. As a result, partial-failure states show up as per-step conditions. You can read them straight from kubectl get print columns. The single most valuable primitive we built was not the step interface itself. It was the shared status-patching helper underneath it. Every step calls one function to update conditions. So that function centralizes the conflict-safe read-then-patch dance in exactly one place. When we later found a cache bug, we fixed it once, not in fifteen call sites. The shared conflict-safe status patch A status helper re-fetches the latest object, mutates it, and patches it with client.MergeFrom inside a retry.RetryOnConflict loop. So it avoids two reconciling steps racing each other's condition writes. And the generation-aware comparison means unchanged state does not re-patch. That keeps the API server and your audit log quiet. func (r *StepRuntime) UpdateStatusFields(ctx context.Context, obj client.Object, mutate func(client.Object) error) error { return retry.RetryOnConflict(retry.DefaultRetry, func() error { // Read through the UNCACHED reader, not the informer cache. fresh := obj.DeepCopyObject().(client.Object) if err := r.UncachedClient.Get(ctx, client.ObjectKeyFromObject(obj), fresh); err != nil { return err } base := fresh.DeepCopyObject().(client.Object) if err := mutate(fresh); err != nil { return err // may be ErrNoOp; caller short-circuits } return r.Client.Status().Patch(ctx, fresh, client.MergeFrom(base)) }) } Two small details carry weight. The ErrNoOp sentinel lets a step's mutate function say "nothing changed", distinct from a real error. So callers short-circuit cleanly. And event correlation only emits a Kubernetes event when the visible condition state actually changed, not on every generation-only refresh. Without that guard, a controller reconciling every 20 seconds floods the event stream with duplicates. > Gotcha: The uncached read in the snippet above was not the original design. It was a live-incident fix. The first version read through the informer cache, and that is exactly where the next section starts. Why does a fresh status condition get overwritten? A fresh status condition gets overwritten when a step reads the object through the informer cache instead of the API server. The cache can lag by tens of milliseconds within a single reconcile. That stale read then rides along in a MergeFrom patch and clobbers a condition another step set moments earlier (controller-runtime #741, 2021). Here is the incident, exactly as it bit us. A routine release introduced new condition types on the StandardCluster resource, replacing an older readiness condition with a more specific one. A fresh reconcile did three things in sequence. First, MarkReconciling patched Ready=False at T=0. Second, the next step, seeing no prior value for its new condition type, re-read the object to update status. Third, the informer cache still returned the pre-restart object carrying a stale Ready=True. That stale Ready=True traveled inside the same MergeFrom patch. So it overwrote the Ready=False written 49 milliseconds earlier. Net effect: clusters never converged to Ready. The maddening part? Running the controller locally fixed it every time. A local run used a fresh client that bypassed the shared cache. It looked non-deterministic. It was not. It was a read-your-own-write violation hiding in the cache. > Gotcha: If a bug disappears when you run the controller locally but persists in-cluster, suspect informer-cache staleness before you suspect a logic error. Local runs often bypass the shared cache path that production hits. Most articles explain the reconcile loop as a clean read-diff-write cycle and stop there. They never connect the informer cache to a correctness bug. The cache is usually framed as a pure performance optimization. But at scale it becomes a correctness hazard, much like a [stale eBPF program on GKE Dataplane V2](/en/blog/gke-dataplane-v2-stale-ebpf-networkpolicy/) can keep enforcing a NetworkPolicy you already deleted. "Read" and "write" no longer target the same view of the world. The upstream root cause is a DeltaFIFO backlog plus read-write mutex contention in the informer cache. Under load, that produces stale reads (kubernetes/kubernetes #130767, 2025). The fix in our code was to read through an uncached reader inside the retry loop, exactly as the earlier snippet shows. Upstream, the fix arrived in Kubernetes v1.36 in April 2026. Controllers can now check the cache's resourceVersion. Then they skip reconciliation when the cache is behind their own writes. That closes the read-your-own-write gap at the framework level (kubernetes.io blog, 2026). When should OperationResultUpdated requeue? OperationResultUpdated should almost never mean "requeue and come back later." It means something changed, and you must still fall through to the readiness check before deciding to requeue. Treating Updated as terminal produced an infinite roughly 20-second reconcile loop across seven step files in our service-exposure controller. The shape of the bug is worth memorizing, because it is easy to reintroduce. A step's mutate function toggled an annotation on every reconcile. For example, a fast-reconcile-interval hint derived from current readiness. So the annotation changed each pass. As a result, controllerutil.CreateOrPatch returned OperationResultUpdated almost every time. Several steps then treated Updated as "done for now, requeue". So they never reached the check for whether the underlying cloud resource is actually ready. switch result { case controllerutil.OperationResultCreated, controllerutil.OperationResultUpdated, controllerutil.OperationResultNone: // All three fall through to the SAME readiness check. if !cloudResourceReady(obj) { return ctrl.Result{RequeueAfter: 20 * time.Second}, nil } return ctrl.Result{}, nil // ready: stop requeuing } We fixed this in seven sibling files. We collapsed the Created, Updated, and unchanged cases into one readiness check. Then we requeue only when the resource is still not ready. If you review a new step that wraps CreateOrPatch, look for the anti-pattern case OperationResultUpdated: return ctrl.Result{RequeueAfter: ...}, nil with no readiness check ahead of it. That single line is a self-perpetuating loop waiting to happen. How do you handle read-after-create NotFound races? Handle a read-after-create NotFound by retrying the transient error a few times, not by widening a cache window. The object exists on the API server but is not yet visible to your very next Get. This is an API-server-side visibility gap, the mirror image of client-side cache staleness, so the fixes point in opposite directions. We hit this when a workspace-creation step patched owner references onto namespaces right after creating them. Occasionally the next Get returned NotFound, even though the namespace clearly existed. The instinct is to assume a longer cache-consistency wait would help. But the gap here is eventual server-side visibility, not a stale client cache. So a longer wait does nothing. Instead, the correct pattern is a short bounded retry on the transient NotFound. Treat it as expected, not as step failure. > Gotcha: A stale-cache bug and a read-after-create bug feel identical from a stack trace. One is fixed by reading past the cache; the other by retrying the read. Diagnose which side of the client-server boundary you are on before you reach for a fix. How do you scale controllers with environment-based sharding? Environment-based sharding partitions reconciliation by a business dimension: deployment environment. It uses controller-runtime predicates, rather than splitting the CRD, API, or codebase. One controller instance reconciling every environment couples their fate. So a reconcile storm in dev can degrade ope. Sharding decouples them, while keeping a single schema. The mechanism is a release topology, not a code fork. So we deploy one Helm chart as several releases of the same version. The same fleet-shaped coordination shows up when you promote add-ons across many clusters, which we handle with [Sveltos-driven fleet add-on promotion](/en/blog/kubernetes-fleet-addon-promotion-sveltos/). A common release runs environment-agnostic singletons like the platform API and tenant controller. One release per environment runs only the environment-scoped controllers. Each gets a CLI argument declaring which single environment it owns. Each environment-scoped controller composes its existing predicates with one extra environment-matching predicate. That predicate admits an object into the work queue only when its environment matches the shard's. And it derives that environment from the existing source of truth, like a label or namespace. Filtering happens before objects enter the queue. So out-of-scope objects never trigger a reconcile, and the existing logic stays untouched. The design's real cost is not code. It is an operational invariant that lives outside the type system: the union of environment releases must cover every environment exactly once, no overlap and no gaps. The type checker cannot enforce this. Your release automation must. We also give each shard an independent leader-election identifier so releases never contend for the same lock. A missing environment argument falls back to non-sharded mode, which makes rollout incremental and reversible. Why is even a logging change a production change? A logging change is a production change. Why? Logs flow through parsers and agents that fail in ways your unit tests never see. In March 2026, two upgrades on our control plane both traced back to the same mistake: code duplicating behavior the framework already provided. And one of them silently deleted logs from the busiest controller in an environment. The first incident was an apimachinery upgrade. A dependency bump removed a generated ProtoMessage() method from core meta types. Our generated proto file embedded those Kubernetes types directly. So the first time the gRPC server tried to marshal any response, the protobuf runtime panicked at lazy descriptor init. That took down the entire process. Because the panic fires at file-descriptor init rather than per field, one bad request crashed everything. The fix moved the HTTP gateway from endpoint-based registration to in-process registration. That bypassed the binary codec entirely, by reusing an existing JSON marshaler. The second incident is the one I retell most. We added a reconcileID correlation field to all eight controllers with a .WithValues("controller", ..., "reconcileID", uuid.New()) call at the top of every Reconcile. Two days later, every log line from the busiest controller in one environment had silently vanished from Cloud Logging. The root cause was a duplicate JSON key. Would your test suite have caught that? Ours did not. Controller-runtime already injects controller, name, namespace, and reconcileID into the context logger automatically. So our added call re-added the same keys. That produced JSON objects with duplicate keys. GKE's fluentbit log agent parses those logs via msgpack. And msgpack's Go implementation cannot hash a duplicate-key object. So it dropped the entire line, rather than erroring loudly. One environment escaped: its busiest controller used a logger variant that replaced keys instead of appending them. The fix was simple. We removed the redundant call, because the framework already carried every field we were trying to add. > Gotcha: Before you add fields to a structured logger, check what your framework already injects. Duplicate keys are valid JSON to your program and poison to your log pipeline. What keeps controllers reliable at scale Custom controllers fail in ways the tutorials rarely mention. And almost all of those failures live at one boundary: between your reconcile logic and the framework's caching. So structure reconcilers as ordered, independently testable steps. That keeps each unit of provisioning logic small and partial progress visible. Then centralize status patching in one conflict-safe helper. Read through an uncached reader in that path, so a stale cache read cannot overwrite a condition you just set. Watch the framework, not just your code. OperationResultUpdated is not a signal to requeue. It is a signal to re-check readiness. A read-after-create NotFound wants a retry, not a wider cache window. And a logging change can be as destructive as a schema change. So if you run controllers at scale, adopt the Kubernetes v1.36 cache-staleness detection early. And treat every framework upgrade as the production change it is.