An Empty resources: {} Makes Kubernetes 1.34 to 1.36 Pods BestEffort
---
title: An Empty resources: {} Makes Kubernetes 1.34 to 1.36 Pods BestEffort
published: true
date: 2026-11-04 15:00:00 UTC
tags: Kubernetes,QoS,GitLabRunner,k3s
canonical_url: devops-daily.com
---
On Kubernetes 1.34, 1.35 and 1.36, one empty block in a pod spec changes how the node treats the pod. If the pod sets spec.resources: {} at pod level, the kubelet gives it QoS class BestEffort. This happens even when every container sets requests and limits, whenever the PodLevelResources feature gate is on, which is the default from 1.34. GitLab Runner's Kubernetes executor builds its job pods with that empty block. Its source does this from 18.10.0 to 19.4.3, unless the runner config sets pod-level values.
We ran the same pods on k3s v1.34.12, v1.35.9, v1.36.5 and v1.37.1. On the three older versions, the empty block moved the pod into the BestEffort cgroup slice. The kubelet set its oom_score_adj to 1000, the highest value the kernel accepts. Next to a busy neighbour, a pod that requested half a CPU got 0.06 to 0.08 cores instead of 0.57 to 0.58. A namespace quota that blocks BestEffort pods rejected it. On 1.37.1 the same pods kept their Burstable and Guaranteed classes.
One thing did not change, and the research we started from got it wrong. It said these pods are first in line for kubelet eviction under memory pressure. They are not. The kubelet's eviction ranking compares memory usage with the memory request. For an empty block, that request still comes from the containers. The QoS class is not part of the ranking, and the kubelet evicted our pods in the same order on every version.
TL;DR
- On k3s v1.34.12, v1.35.9 and v1.36.5, a pod with container requests and limits plus spec.resources: {} reported qosClass: BestEffort. The Guaranteed variant did too. On v1.37.1 they reported Burstable and Guaranteed. With the new PodLevelResourcesFixKubeletQOSClass feature gate turned off, 1.37.1 went back to BestEffort.
- The node settings followed the class. oom_score_adj went from 935 (Burstable) and -997 (Guaranteed) to 1000, and the pod cgroup moved to kubepods-besteffort.slice with cpu.weight 1 and no pod-level memory.max. The container-level limits stayed in place.
- A neighbour pod kept both vCPUs busy. Next to it, the BestEffort-classified pod got a median of 0.061 to 0.083 cores on the three affected versions. The same pod without the empty block got 0.573 to 0.580 (so 10.6% to 14.3%). On 1.37.1 the two were equal.
- A ResourceQuota with scope BestEffort and pods: "0" rejected the pod on 1.34 to 1.36 and admitted it on 1.37.
- The kubelet's memory-pressure eviction order was the same on all four versions, in all 12 repetitions: the true BestEffort pod first, then by memory usage relative to the container request. The empty block did not move a pod forward.
- If you run GitLab Runner on Kubernetes, upgrade to 19.5.0 or later. As of 2 November 2026 the latest Kubernetes patch releases for 1.34, 1.35 and 1.36 still have the bug.
Prerequisites
- A cluster on Kubernetes 1.34, 1.35 or 1.36 (kubectl version), or a machine where you can run the lab
- kubectl and jq
- If you run GitLab Runner with the Kubernetes executor: its version (gitlab-runner --version)
- To run the lab: a DigitalOcean token (referral link); one s-2vcpu-4gb droplet for about an hour costs a few cents
Where the empty block comes from
Pod-level resources (KEP-2837) let you set CPU and memory for the pod as a whole, not only per container. The feature gate PodLevelResources went alpha in 1.32 and beta, on by default, in 1.34. Our Kubernetes 1.36 post covers the in-place resize of those values.
The QoS calculation has a shortcut for them. This is ComputePodQOS at v1.34.0, v1.35.0 and v1.36.0, and still at the latest patches, v1.34.13, v1.35.10 and v1.36.6:
// pkg/apis/core/v1/helper/qos/qos.go, v1.36.6
// When pod-level resources are specified, we use them to determine QoS class.
if utilfeature.DefaultFeatureGate.Enabled(features.PodLevelResources) &&
pod.Spec.Resources != nil {
An empty block is not nil. It has no requests and no limits, so the function returns BestEffort. The loop over the containers never runs. The issue that reported this, #135082 from November 2025, says that almost everywhere else Kubernetes does not differentiate between nil and empty pod-level resources. The helper for that, IsPodLevelResourcesSet, treats {} as not set.
The fix is PR #137150, merged on 15 July 2026 for 1.37. It adds the condition IsPodLevelResourcesSet(pod) behind a new gate, PodLevelResourcesFixKubeletQOSClass, beta and on by default from 1.37. It was not backported. The patch releases of 27 October 2026, v1.34.13, v1.35.10 and v1.36.6, still have the old condition. The rest of that release is in our Kubernetes 1.37 review; PodLevelResources itself stays beta in 1.37.
GitLab Runner added pod-level resources to its job pods in 18.10.0 (MR !5922). At every tag we read (18.10.0, 19.3.5 and 19.4.3), the helper that builds the block returns a non-nil object:
// executors/kubernetes/kubernetes.go, GitLab Runner v19.4.3
func (s *executor) podResourcesReference() *api.ResourceRequirements {
resources := api.ResourceRequirements{}
if s.configurationOverwrites != nil {
resources.Limits = s.configurationOverwrites.podLimits
resources.Requests = s.configurationOverwrites.podRequests
}
return &resources
}
If the runner config sets none of pod_cpu_request, pod_memory_request, pod_cpu_limit and pod_memory_limit, both maps are empty. The pod then goes to the API server with resources: {}. The MR that fixed it, !7346, says the same: job pods "were showing qosClass: BestEffort instead of the expected Burstable", and after the fix .spec.resources "should now be empty/absent (was {})". It merged on 1 October 2026 and shipped in 19.5.0. The 19.4.3 and 19.3.5 tags of 28 October 2026 still have the old function. The class matters only when the runner config gives the job containers requests or limits. Without them, the pods were BestEffort anyway.
The lab
The lab runs on one DigitalOcean droplet: s-2vcpu-4gb in fra1, Ubuntu 24.04, cgroup v2. A cloud firewall allows SSH from one address and nothing else. The Kubernetes API port is never exposed, and kubectl runs on the node over SSH. For each version, a script removes k3s, installs the pinned release, and runs five stages. The kubelet gets an explicit hard eviction threshold, memory.available<2Gi. It also gets a 30 second pressure transition period instead of the default 5 minutes, so repetitions do not wait for the node condition to clear.
1. pod spec resources: {} + container requests
2. ComputePodQOS sees a block: BestEffort
3. IsPodLevelResourcesSet sees {}: not set
4. oom_score_adj 1000
5. cgroup slice besteffort, cpu.weight 1
6. ResourceQuota scope BestEffort
7. eviction ranking container requests
Connections:
- pod spec -> ComputePodQOS
- pod spec -> IsPodLevelResourcesSet
- ComputePodQOS -> oom_score_adj
- ComputePodQOS -> cgroup slice
- ComputePodQOS -> ResourceQuota scope
- IsPodLevelResourcesSet -> eviction ranking
The pods:
| Pod | Pod-level block | Container requests | Container limits |
| --- | --- | --- | --- |
| be | none | none | none |
| ctl-burstable | none | 100m CPU, 256Mi | 500m CPU, 512Mi |
| emp-burstable | {} | 100m CPU, 256Mi | 500m CPU, 512Mi |
| ctl-guaranteed | none | 200m CPU, 256Mi | 200m CPU, 256Mi |
| emp-guaranteed | {} | 200m CPU, 256Mi | 200m CPU, 256Mi |
| plr-burstable | requests 100m/256Mi, limits 500m/512Mi | none | none |
plr-burstable is the control for the feature itself: a real pod-level block works as documented on every version. The empty block is the only problem.
We wrote down the claim and what would disprove it before the droplet existed, and committed it first (CLAIM.md). Every recorded file carries a hash of the manifests and node scripts, and the summary script refuses to mix files with different hashes.
The-DevOps-Daily/k8s-empty-pod-resources on GitHub
Finding 1: the class flips on 1.34 to 1.36
The API server stored the empty block exactly as written. On v1.34.12, v1.35.9 and v1.36.5 the kubelet then reported BestEffort for both emp-* pods, while their controls kept Burstable and Guaranteed. On v1.37.1 the emp-* pods got the same class as their controls:
results/demo-k3s-v1.36.5.txt and results/demo-k3s-v1.37.1.txt
# k3s v1.36.5+k3s1
# ... kubectl apply and wait omitted; the whole session is in the file
$ kubectl get pods -n demo -o custom-columns='NAME:.metadata.name,POD-LEVEL:.spec.resources,QOS:.status.qosClass'
NAME POD-LEVEL QOS
ctl-burstable <none> Burstable
ctl-guaranteed <none> Guaranteed
emp-burstable map[] BestEffort
emp-guaranteed map[] BestEffort
# k3s v1.37.1+k3s1, same manifests
# ... kubectl apply and wait omitted
$ kubectl get pods -n demo -o custom-columns='NAME:.metadata.name,POD-LEVEL:.spec.resources,QOS:.status.qosClass'
NAME POD-LEVEL QOS
ctl-burstable <none> Burstable
ctl-guaranteed <none> Guaranteed
emp-burstable map[] Burstable
emp-guaranteed map[] Guaranteed
The same result on every version, from results/summary.md:
| Pod | v1.34.12 | v1.35.9 | v1.36.5 | v1.37.1 | v1.37.1, gate off |
| --- | --- | --- | --- | --- | --- |
| ctl-burstable | Burstable | Burstable | Burstable | Burstable | Burstable |
| emp-burstable | BestEffort | BestEffort | BestEffort | Burstable | BestEffort |
| ctl-guaranteed | Guaranteed | Guaranteed | Guaranteed | Guaranteed | Guaranteed |
| emp-guaranteed | BestEffort | BestEffort | BestEffort | Guaranteed | BestEffort |
| plr-burstable | Burstable | Burstable | Burstable | Burstable | Burstable |
| be | BestEffort | BestEffort | BestEffort | BestEffort | BestEffort |
plr-burstable, with real pod-level requests and limits, was Burstable everywhere. Pod-level resources work. Only the empty block breaks the class.
To check that the 1.37 gate is the fix and not something else in the release, we reinstalled v1.37.1 with PodLevelResourcesFixKubeletQOSClass=false on both the API server and the kubelet. Both emp-* pods went back to BestEffort, with oom_score_adj 1000. The API server and kubelet metrics reported the gate as BETA, on by default and off in that run.
Finding 2: the kernel sees a BestEffort pod
QoS is not only a label. The kubelet turns it into numbers that the kernel uses. We read them from /proc and /sys/fs/cgroup on the node, for the main process of each container:
oom_score_adj of the main container process
| | Value | Series |
| --- | --- | --- |
| ctl-burstable, every version | 935 | no pod-level block |
| emp-burstable, 1.34 to 1.36 | 1000 | resources: {}, 1.34 to 1.36 |
| emp-burstable, 1.37.1 | 935 | resources: {}, 1.37.1 |
| ctl-guaranteed, every version | -997 | no pod-level block |
| emp-guaranteed, 1.34 to 1.36 | 1000 | resources: {}, 1.34 to 1.36 |
| emp-guaranteed, 1.37.1 | -997 | resources: {}, 1.37.1 |
_Read from /proc//oom_score_adj on the node. The 1.34.12, 1.35.9 and 1.36.5 values are identical. The kernel adds this value to a process's badness score when it picks one to kill. Source: results/summary.md._
The kernel adds oom_score_adj to each process's badness score before it picks a process to kill, and it accepts values from -1000 to +1000 (proc.rst). A Burstable container gets 1000 - 1000 * request / node memory, which is 935 for a 256Mi request on this node. A Guaranteed container gets -997. The empty block gave both 1000, the same as a pod with no resources at all. oom_score, the current OOM killer score the kernel shows for the process, went from 1290 (Burstable) and 4 (Guaranteed) to 1334.
The cgroup changed too. On v1.36.5:
| | ctl-burstable | emp-burstable | ctl-guaranteed | emp-guaranteed |
| --- | --- | --- | --- | --- |
| QoS slice | kubepods-burstable.slice | kubepods-besteffort.slice | kubepods.slice | kubepods-besteffort.slice |
| pod cpu.weight | 4 | 1 | 8 | 1 |
| pod cpu.max | 0.5 CPU | max (none) | 0.2 CPU | max (none) |
| pod memory.max | 512 MiB | max (none) | 256 MiB | max (none) |
| container cpu.max | 0.5 CPU | 0.5 CPU | 0.2 CPU | 0.2 CPU |
| container memory.max | 512 MiB | 512 MiB | 256 MiB | 256 MiB |
| oom_score | 1290 | 1334 | 4 | 1334 |
The container cgroup still has its own limits, so emp-burstable cannot use more than 512 MiB or half a CPU. What it loses is everything the kubelet sets at pod and QoS level. The pod cgroup has no memory limit and no CPU quota of its own. It sits in kubepods-besteffort.slice, where the kubelet sets cpu.weight to 1 for the whole slice.
Finding 3: half a CPU requested, 0.06 to 0.08 cores received
cpu.weight 1 matters only when the CPU is busy. To see how much, we ran cpu-neighbour, a Burstable pod that requests 1 CPU and runs two busy processes, so both vCPUs are always taken. Next to it, cpu-ctl and cpu-emp took turns, three times each, alternating which went first. Both request 500m CPU with a 1 CPU limit and run one busy process; cpu-emp adds the empty block. Each pod reads its own cgroup's usage_usec over 20 seconds after a 5 second warm-up.
CPU used next to a busy neighbour, median of 3 runs
| | Value | Series |
| --- | --- | --- |
| v1.34.12 cpu-ctl | 0.579 cores | cpu-ctl (no block) |
| v1.34.12 cpu-emp | 0.083 cores | cpu-emp (resources: {}) |
| v1.35.9 cpu-ctl | 0.58 cores | cpu-ctl (no block) |
| v1.35.9 cpu-emp | 0.07 cores | cpu-emp (resources: {}) |
| v1.36.5 cpu-ctl | 0.573 cores | cpu-ctl (no block) |
| v1.36.5 cpu-emp | 0.061 cores | cpu-emp (resources: {}) |
| v1.37.1 cpu-ctl | 0.576 cores | cpu-ctl (no block) |
| v1.37.1 cpu-emp | 0.577 cores | cpu-emp (resources: {}) |
_2 vCPU droplet. Both pods request 500m with a 1 CPU limit and run one busy process; the neighbour requests 1 CPU and runs two. Each pod measured its own cgroup usage_usec over 20 s. Source: results/summary.md._
On the three affected versions, cpu-emp got a median of 0.083, 0.070 and 0.061 cores, against 0.579, 0.580 and 0.573 for cpu-ctl. That is 14.3%, 12.1% and 10.6% of the control's CPU, or 7.0 to 9.4 times less. On v1.37.1 the two were the same: 0.576 and 0.577 cores.
The weights explain the direction. cpu-ctl's pod cgroup has cpu.weight 20 for its 500m request and sits in the burstable slice next to the neighbour. The kubelet sets the burstable slice's weight from the sum of the CPU requests of the Burstable pods in it, which gave 63 here. cpu-emp's pod cgroup has cpu.weight 1 and sits in the besteffort slice. The kubelet always sets that slice to the minimum, cpu.weight 1. The 500m request of cpu-emp still sets a weight on its container cgroup, but that only counts against other containers in the same pod. On a busy node like this one, a CI build in the BestEffort slice competes with cpu.weight 1 at that level and can get far less CPU than Burstable or Guaranteed pods. We did not run a build.
The runs were short and sequential. Each pod ran three times in a fixed alternating order, and we did not record the neighbour's own usage or other load on the host. On the affected versions, each of the nine cpu-emp runs got less CPU than every cpu-ctl run, including the runs where cpu-emp went first. The exact percentages still belong to this setup.
Finding 4: a "no BestEffort" quota rejects the pod
Some clusters forbid BestEffort pods with a quota like this one, so every workload must declare resources:
apiVersion: v1
kind: ResourceQuota
metadata:
name: no-besteffort
spec:
hard:
pods: '0'
scopes: ['BestEffort']
The API server applies the scope at create time, before any kubelet sees the pod, and it calls the same GetPodQOS function. With no status yet, that function computes the class from the spec. On 1.34 to 1.36 it rejected emp-burstable, a pod whose container sets requests and limits:
quota.py on k3s v1.36.5
$ grep -A3 ' quota$' results/k3s-v1.36.5/run.log
2026-11-02T10:33:20Z quota
be admitted=False Error from server (Forbidden): error when creating "/root/k8s-empty-pod-resources/manifests/quota/be.yaml": pods "be" is forbidden: exceeded quota: no-besteffort, requested: pods=1, used: pods=0, limited: pods=0
ctl-burstable admitted=True pod/ctl-burstable created
emp-burstable admitted=False Error from server (Forbidden): error when creating "/root/k8s-empty-pod-resources/manifests/quota/emp-burstable.yaml": pods "emp-burstable" is forbidden: exceeded quota: no-besteffort, requested: pods=1, used: pods=0, limited: pods=0
On v1.37.1 the API server admitted emp-burstable, and still rejected be. We did not run a runner. But a job pod from 18.10.0 to 19.4.x carries the same empty block, so in a namespace with this quota it should fail the same way on 1.34 to 1.36. The fix discussion names the reverse risk: a pod "could theoretically bypass restricted BestEffort quotas by being classified as Guaranteed during admission, while still behaving as BestEffort at the node level". That can happen when the API server has the fix and the kubelet does not.
Finding 5: the eviction order did not change
The five eviction pods each allocate a fixed amount of memory and then sleep. They are sized so that the two possible rules predict different orders:
| Pod | Memory request | Allocates | Working set before the hog | Usage against request |
| --- | --- | --- | --- | --- |
| be | none | 50 MiB | 54 MiB | above (request is 0) |
| ctl | 256Mi | 200 MiB | 204 MiB | about 52 MiB below |
| gua-ctl | 256Mi (= limit) | 150 MiB | 154 MiB | about 102 MiB below |
| emp + {} | 256Mi | 100 MiB | 104 MiB | about 152 MiB below |
| gua-emp + {} | 256Mi (= limit) | 50 MiB | 54 MiB | about 202 MiB below |
If the kubelet ranked by QoS class, emp and gua-emp would go out together with be, before ctl and gua-ctl. If it ranks by container requests, be goes first because any usage exceeds a request of zero, and the rest follow in order of usage minus request: ctl, gua-ctl, emp, gua-emp.
A host process outside every pod, hog.py, then holds MemAvailable at 1000 MiB. That is below the 2Gi threshold even after all five pods are gone, so the kubelet keeps evicting until nothing evictable is left. The kubelet cannot evict the hog. The pods differ in size on purpose, to separate the two models, so this is not a matched pair. CoreDNS also ran on the node. The kubelet ranked it last in every ranking it logged. In 5 of the 12 logs the window ran long enough to show the next step: the kubelet logged cannot evict a critical pod once only CoreDNS was left.
Every repetition on every version gave the request order. This is the kubelet's own ranking on v1.36.5, while it reported emp and gua-emp as BestEffort:
k3s v1.36.5, repetition 1
# the class the kubelet reported for each pod
$ jq -c 'select(.rep == 1) | .qos' results/k3s-v1.36.5/40-pressure.jsonl
{"be":"BestEffort","ctl":"Burstable","emp":"BestEffort","gua-ctl":"Guaranteed","gua-emp":"BestEffort"}
# the kubelet's ranking before each eviction; sed keeps the journald timestamp and the message, and drops the host name, the service PID, the severity letter, the kubelet's own timestamp, the second PID and the source file
$ grep -E 'ranked for eviction|evicted successfully' results/k3s-v1.36.5/40-kubelet-rep1.log | sed -E 's/^([^]+) .*eviction_manager.go:[0-9]+\] /\1 /'
2026-11-02T10:38:17.816980+00:00 "Eviction manager: pods ranked for eviction" pods=["evict-check/be","evict-check/ctl","evict-check/gua-ctl","evict-check/emp","evict-check/gua-emp","kube-system/coredns-7cfb7bc9c7-svb26"]
2026-11-02T10:38:20.108268+00:00 "Eviction manager: pod is evicted successfully" pod="evict-check/be"
2026-11-02T10:38:21.127785+00:00 "Eviction manager: pods ranked for eviction" pods=["evict-check/ctl","evict-check/gua-ctl","evict-check/emp","evict-check/gua-emp","kube-system/coredns-7cfb7bc9c7-svb26"]
2026-11-02T10:38:23.555889+00:00 "Eviction manager: pod is evicted successfully" pod="evict-check/ctl"
2026-11-02T10:38:24.579473+00:00 "Eviction manager: pods ranked for eviction" pods=["evict-check/gua-ctl","evict-check/emp","evict-check/gua-emp","kube-system/coredns-7cfb7bc9c7-svb26"]
2026-11-02T10:38:27.242735+00:00 "Eviction manager: pod is evicted successfully" pod="evict-check/gua-ctl"
2026-11-02T10:38:28.259762+00:00 "Eviction manager: pods ranked for eviction" pods=["evict-check/emp","evict-check/gua-emp","kube-system/coredns-7cfb7bc9c7-svb26"]
2026-11-02T10:38:30.555840+00:00 "Eviction manager: pod is evicted successfully" pod="evict-check/emp"
2026-11-02T10:38:31.572048+00:00 "Eviction manager: pods ranked for eviction" pods=["evict-check/gua-emp","kube-system/coredns-7cfb7bc9c7-svb26"]
2026-11-02T10:38:34.078890+00:00 "Eviction manager: pod is evicted successfully" pod="evict-check/gua-emp"
The source explains it. rankMemoryPressure sorts by three keys. First, whether usage exceeds the memory request. Then pod priority. Then usage minus the memory request. The first and third keys get the request from GetResourceRequestQuantity. For an empty block, that helper asks IsPodLevelResourcesSet, gets false, and uses the container requests. Priority does not read requests, and the QoS class is not an input at all. If the kubelet had used a request of zero for emp, emp would have been in the first group with be. It came fourth.
When each pod was evicted, 12 repetitions
| Series | Samples | Min | Median | p95 | Max |
| --- | --- | --- | --- | --- | --- |
| be | 12 | 5.6s | 8.350000000000001s | 11.5s | 11.5s |
| ctl | 12 | 9.1s | 11.95s | 15.3s | 15.3s |
| gua-ctl | 12 | 12.7s | 16s | 20.9s | 20.9s |
| emp | 12 | 16.1s | 20.15s | 24.7s | 24.7s |
| gua-emp | 12 | 19.5s | 23.6s | 29.9s | 29.9s |
_Seconds after the host hog started, from the kubelet's log. 3 repetitions each on k3s v1.34.12, v1.35.9, v1.36.5 and v1.37.1. Every repetition evicted the pods in the same order. Source: results/summary.md._
In all 12 repetitions the kubelet started to reclaim 3.0 to 9.2 s after the hog started, and evicted the first pod 2.3 to 3.3 s later. After that it evicted one pod at a time. 41 of the 48 gaps between evictions were 4.0 s or less (median 3.6 s, longest 8.2 s). The last pod was gone 19.5 to 29.9 s after the hog started. It does not wait 10 seconds between evictions: after each one it waits for the pod to be cleaned up and then checks again. Its own available figure at the first eviction was 931 to 1212 MiB, well under the 2Gi threshold and far from empty. The kernel logged no OOM kill in any repetition, so the order is the kubelet's alone.
The two mechanisms are easy to mix up. The kubelet's eviction manager checks every 10 seconds by default when it has nothing to evict, and it ignores the class. The kernel OOM killer acts when memory runs out before the kubelet frees any. It adds oom_score_adj to each process's badness score, and the empty block gives the pod the same 1000 as a pod with no resources at all (Finding 2). We did not trigger a kernel OOM, so we did not watch the kernel make that choice.
What we got wrong before measuring
The research note we started from had the core right and two details wrong.
- Right: Kubernetes 1.34 to 1.36 compute BestEffort for an empty pod-level block. GitLab Runner 18.10 to 19.4 build their job pods with that block (from the source and the fix MR). On 1.34 to 1.36 the pods get oom_score_adj 1000. 1.37 fixes the class while its new gate is on, which is the default.
- Wrong: it called #137150 the upstream issue. That is the fix PR. The issue is #135082.
- Wrong, and it matters: it said these pods are first in line for kubelet eviction under memory pressure. Finding 5 shows they are not.
- Missing: the 1.37 fix is a beta feature gate, not a plain code change, and nothing was backported to the 1.34, 1.35 or 1.36 patch releases.
We got one prediction wrong ourselves. CLAIM.md said the BestEffort-classified pod would get under 10% of the control's CPU. It got 10.6% to 14.3%. Our falsification line was 50%, so the claim stands, but the number we wrote down was too pessimistic, and we report it as a miss.
We also kept a one-repetition smoke run on v1.36.5 (results/_smoke-k3s-v1.36.5/). It used the same code stamp and gave the same results, but the summary does not count it.
What to do
1. Find the pods. This query lists every pod with an empty pod-level block and the class it got:
2. GitLab Runner 18.10.0 to 19.4.x: upgrade to 19.5.0 or later. The job pods then go out without a pod-level block, and the class comes from the containers again.
3. If you cannot upgrade the runner yet, give the block real values. Set pod_cpu_request and pod_memory_request (and the _limit keys if you want limits) under [runners.kubernetes]. The QoS class then comes from those values, like plr-burstable above. This is a behaviour change, not only a workaround: the pod cgroup then takes its limits from the pod-level values too, as it did for plr-burstable.
4. Kubernetes 1.37 fixes it by default. The fix sits behind a beta gate, so check that nobody turned PodLevelResourcesFixKubeletQOSClass off.
5. Check your own templates. A Helm chart that renders a pod-level block from a value that defaults to {} produces the same bug. Render the key only when it has content:
An empty resources: {} inside a container is harmless. Only the pod-level one under spec changes the class.
What we could not conclude
- k3s is one distribution. We ran upstream Kubernetes code as k3s packages it. A managed service can turn the PodLevelResources gate off or carry its own patches. Run the jq query on your own cluster.
- No GitLab instance and no runner ran here. The runner part rests on its source at each tag and on the MR description. The emp-* pods reproduce the field the runner sends, not a full job pod with helper and service containers.
- We did not trigger a kernel OOM kill. The oom_score values are the numbers the kernel uses to choose a victim, read from /proc. We did not watch it choose.
- The eviction timing is ours. It depends on the 2Gi threshold, the hog's 1000 MiB target and how fast it allocates. The order is the finding, not the seconds.
- The CPU gap depends on the neighbours. One neighbour with a 1 CPU request on a 2 vCPU node gave the numbers above. More Burstable or Guaranteed load should make the BestEffort slice's share smaller. On an idle node the weights do not matter, because they only divide contended CPU time; we did not measure that case.
- One node, one admission check. We did not test scheduling across nodes, or any admission rule other than the one quota. The gate-off run on 1.37.1 covered only the class, the kernel-facing numbers and the quota, not CPU or eviction.
- Other consumers of the class were not tested. From the source, the kubelet also rejects new BestEffort pods while the node has the MemoryPressure condition. The static CPU manager gives exclusive CPUs only to Guaranteed pods. We did not measure either.
Summary
On Kubernetes 1.34, 1.35 and 1.36, with the PodLevelResources gate at its default (on), spec.resources: {} makes a pod BestEffort no matter what its containers request. On our node that meant oom_score_adj 1000 instead of 935 or -997, and a place in the BestEffort slice with cpu.weight 1 and no pod-level memory or CPU limit. Next to a busy neighbour the pod got about 7 to 9 times less CPU, and a quota that forbids BestEffort pods rejected it. The empty block did not change the kubelet's eviction order. The ranking uses memory requests, and for an empty block those still come from the containers: 12 of 12 repetitions evicted our pods in exactly the order the requests predict. Kubernetes 1.37 fixes the class behind a beta gate that is on by default, and GitLab Runner 19.5.0 stops sending the block. Run the jq query above on your clusters, and upgrade whichever side you can first.
_Originally published at devops-daily.com._
10 reactions
Stop Writing Bash Scripts for Everything! Build Your Own DevOps CLI Tool With Go
Imagine having one command that checks your servers, monitors Docker containers, tests API endpoints, and reports infrastructure health.
Instead of maintaining 20 different scripts, you build one powerful CLI tool.
Sounds cool?
That's exactly where Go becomes interesting for DevOps engineers.
Bash is amazing for automation. Python is excellent for scripting. But when you want to build portable, compiled command-line applications that are easy to distribute, Go deserves your attention.
Think about tools like:
- Docker
- kubectl
- Terraform
- Helm
- Hugo
Go is used to build all of these.
So why not build your own?
Let's create a practical DevOps CLI tool from scratch. 🚀
---
🤔 1. Why Should DevOps Engineers Learn Go?
Imagine you're managing multiple servers.
You regularly need to:
- Check whether servers are reachable.
- Monitor API availability.
- Inspect Docker containers.
- Check application health.
- Automate repetitive operations.
- Generate reports.
You could write separate Bash or Python scripts for every task.
But as your automation grows, maintaining those scripts can become difficult.
Go offers some useful advantages:
| Feature | Why it matters |
|---|---|
| Compiled binaries | Easy application distribution |
| Static typing | Catch many errors during development |
| Goroutines | Run concurrent tasks |
| Standard library | Build HTTP clients and servers |
| Cross-compilation | Build executables for different platforms |
| Fast startup | Useful for command-line applications |
The idea isn't to replace Bash or Python.
It's to choose the right tool for the job.
⚡ 2. What Exactly Is a CLI Tool?
CLI stands for Command-Line Interface.
You've probably used these commands:
docker ps
kubectl get pods
terraform plan
git status
These are examples of command-line interfaces.
A CLI tool allows users to interact with software through commands and arguments.
Imagine building your own tool called devopsctl.
Its commands could look like this:
devopsctl server check
devopsctl docker status
devopsctl api health
devopsctl system info
Instead of manually remembering multiple scripts, you have one organized command-line application.
Our goal is to build a small version of this idea.
---
🛠️ 3. Set Up Your Go Environment
First, install Go from:
go.dev
Verify the installation:
go version
Create your project:
mkdir devopsctl
cd devopsctl
Initialize a Go module:
go mod init devopsctl
Create a file:
touch main.go
Your project structure:
devopsctl/
│
├── main.go
├── go.mod
└── README.md
Now let's start building.
🌐 4. Build Your First Server Health Checker
One common DevOps task is checking whether a service is reachable.
Let's build a command that checks multiple URLs concurrently.
Create main.go:
package main
import (
"fmt"
"net/http"
"sync"
"time"
)
func checkServer(url string, wg *sync.WaitGroup) {
defer wg.Done()
client := http.Client{
Timeout: 5 * time.Second,
}
start := time.Now()
response, err := client.Get(url)
if err != nil {
fmt.Printf("[DOWN] %s: %v\n", url, err)
return
}
defer response.Body.Close()
latency := time.Since(start)
fmt.Printf(
"[UP] %s | Status: %s | Latency: %v\n",
url,
response.Status,
latency,
)
}
func main() {
servers := []string{
"https://google.com",
"https://github.com",
"https://go.dev",
}
var wg sync.WaitGroup
for _, server := range servers {
wg.Add(1)
go checkServer(server, &wg)
}
wg.Wait()
fmt.Println("Health checks completed!")
}
Run it:
go run main.go
Example output:
[UP] https://google.com | Status: 200 OK | Latency: 120ms
[UP] https://github.com | Status: 200 OK | Latency: 180ms
[UP] https://go.dev | Status: 200 OK | Latency: 95ms
Health checks completed!
Your first Go-based DevOps utility is ready.
What did we learn?
- net/http handles HTTP requests.
- sync.WaitGroup coordinates concurrent tasks.
- go starts a goroutine.
- time measures request duration.
- Error handling helps identify unavailable services.
One important detail: this checks whether an HTTP request succeeds at the transport level. A service returning HTTP 500 is still reachable, but its application response indicates an error. A production health checker should evaluate HTTP status codes and application-specific health endpoints.
---
🐳 5. Add Docker Container Monitoring
Now let's make the project more interesting.
What if your tool could inspect Docker containers?
Docker exposes a command-line interface that you can call from Go.
For a simple learning example:
package main
import (
"fmt"
"os/exec"
)
func checkDocker() {
cmd := exec.Command("docker", "ps")
output, err := cmd.CombinedOutput()
if err != nil {
fmt.Println("Docker command failed:", err)
return
}
fmt.Println(string(output))
}
func main() {
fmt.Println("Checking Docker containers...")
checkDocker()
}
Run:
go run main.go
This executes:
docker ps
and displays the result.
For a more advanced production tool, consider using the Docker Engine API or official Go SDK rather than parsing command output.
Also, external command execution should use fixed executable paths or carefully validated arguments. Never pass untrusted user input directly into a shell command.
---
📊 6. Turn It Into a Real CLI Application
So far, we've written individual functions.
But a professional CLI should support commands and arguments.
For example:
devopsctl health
devopsctl docker
devopsctl version
Go provides the flag package in its standard library.
Here's a simple example:
package main
import (
"flag"
"fmt"
)
func main() {
health := flag.Bool(
"health",
false,
"Check application health",
)
version := flag.Bool(
"version",
false,
"Show application version",
)
flag.Parse()
if *health {
fmt.Println("Application is healthy!")
return
}
if *version {
fmt.Println("devopsctl version 1.0.0")
return
}
flag.Usage()
}
Run:
go run main.go --health
Output:
Application is healthy!
Check the version:
go run main.go --version
For larger applications, explore Cobra, a popular Go library for building command-line applications.
spf13/cobra
---
🚀 7. Compile Your CLI Into a Single Executable
Here's one of the most satisfying parts of Go.
Build your application:
go build -o devopsctl
Run it:
./devopsctl --health
You can also cross-compile for different operating systems and architectures.
For example, build a Linux AMD64 executable from another supported development environment:
GOOS=linux GOARCH=amd64 go build -o devopsctl-linux
Build for Windows:
GOOS=windows GOARCH=amd64 go build -o devopsctl.exe
Build for macOS on Apple Silicon:
GOOS=darwin GOARCH=arm64 go build -o devopsctl-macos
This makes it convenient to distribute your CLI without requiring users to install a separate Go runtime.
Remember that cross-compilation can depend on native dependencies and build configuration. The examples above are simplest for pure-Go projects.
---
☸️ 8. Take It Further: Build a Kubernetes CLI
Now imagine extending devopsctl with Kubernetes functionality.
You could support commands such as:
devopsctl k8s pods
devopsctl k8s deployments
devopsctl k8s nodes
Go has a powerful Kubernetes ecosystem.
Explore:
- client-go — Kubernetes API client library.
- controller-runtime — Libraries for controllers and operators.
- Kubebuilder — Framework for building Kubernetes APIs and controllers.
Official resources:
kubernetes.io
kubernetes/client-go
With these tools, you can build applications that interact with Kubernetes clusters programmatically.
For example, a Kubernetes monitoring CLI could:
1. Connect to a cluster using kubeconfig.
2. List Pods across namespaces.
3. Identify Pods that are not Ready.
4. Display restart counts.
5. Retrieve selected Pod logs.
6. Generate a troubleshooting report.
That is a much more advanced project than simply running kubectl get pods.
---
🧪 9. Build This Portfolio Project: DevOpsCTL
Here's the project I'd recommend building and publishing on GitHub.
<text>Project: DevOpsCTL — Your Personal DevOps Command-Line Assistant</text>
Core features
| Module | Functionality |
|---|---|
| Server | Check endpoint availability |
| Docker | List and inspect containers |
| Kubernetes | List Pods and Deployments |
| System | Display basic system information |
| API | Check HTTP status and latency |
| Reports | Export health-check results as JSON |
Suggested architecture
DEVOPSCTL
|
+----------+----------+
| | |
v v v
Server Docker Kubernetes
Module Module Module
| | |
v v v
HTTP Docker K8s API
Client SDK Client
| | |
+----------+----------+
|
v
CLI Output
|
v
JSON Reports
Suggested repository structure
devopsctl/
│
├── cmd/
│ └── root.go
│
├── internal/
│ ├── server/
│ ├── docker/
│ ├── kubernetes/
│ └── system/
│
├── main.go
├── go.mod
├── Dockerfile
├── README.md
└── LICENSE
Advanced features to add later
- Prometheus metrics integration.
- YAML configuration support.
- Concurrent health checks with configurable limits.
- Colorful terminal output.
- Unit and integration tests.
- GitHub Actions CI pipeline.
- Release binaries for Linux, macOS, and Windows.
Don't build everything at once. Start with the health checker, then add one module at a time.
---
🗓️ 10. Your 7-Day Go Developer Challenge
Want to actually learn Go? Follow this challenge.
| Day | Task |
|---|---|
| 1 | Learn Go syntax, variables, and functions |
| 2 | Learn structs, slices, maps, and interfaces |
| 3 | Practice goroutines and channels |
| 4 | Build a REST API |
| 5 | Build a server health checker |
| 6 | Integrate Docker or Kubernetes |
| 7 | Publish your first Go CLI project on GitHub |
At the end of seven days, you won't be a Go expert.
But you'll have something much more valuable: a working project and practical experience.
---
🎯 Final Thoughts
Go is not just another programming language to add to your resume.
It's a way to move from writing small automation scripts toward building real infrastructure tools.
You can use it to create:
- DevOps automation utilities.
- Cloud-native applications.
- REST APIs.
- Kubernetes operators.
- Monitoring systems.
- Distributed services.
- Command-line applications.
And the best part?
You don't need to master the entire language before building something useful.
Start small. Build something. Break it. Debug it. Improve it.
Don't just learn tools built by engineers. Start building tools that other engineers can use.
👇 Let's discuss!
If you could build your own DevOps CLI tool, which feature would you implement first?
1. Docker monitoring
2. Kubernetes troubleshooting
3. AWS resource management
4. Server health monitoring
5. CI/CD automation
Share your ideas in the comments!
---
📚 Continue Your Go Learning Journey
Want a structured resource to learn Go?
🐹 Mastering Go — Complete Guide
You can also explore:
- 🐳 Docker Mastery
- ☸️ CKA Complete Study Guide
- 🏗️ Terraform Associate (003) Exam Crash Course
- 🚀 DevOps Complete Pack
Learn Go. Automate intelligently. Build the tools of tomorrow.
5 reactions
LWKD: Week Ending October 25, 2026
Last Week in Kubernetes Development
Week Ending October 25, 2026
Developer News
SIG ContribEx has updated the repo creation guidelines and the WG governance document to clarify how Working Groups relate to code ownership and repository creation, giving future WG proposals a clearer path from charter to code.
SIG Architecture has added notes on future conformance in the developer guide, outlining how upcoming conformance work (including the newly formed WG Workload Conformance) will relate to existing conformance definitions.
SIG ContribEx is coordinating KubeCon EU 2027 Project Pavilion and PR participation for SIGs, WGs, Committees, and Subprojects; interested groups should submit the registration form by November 25.
The Maintainer Summit NA 2026 unconference is accepting topic ideas; sessions are open community-led discussions and need only a topic, a note-taker, and a facilitator attending in Salt Lake City.
The SIG Scheduling leadership change proposed last week has taken effect after lazy consensus closed on October 26: Heba Elayoty is now a SIG Chair, Matt Matejczyk is a SIG Tech Lead, and Kensei Nakada has stepped down from Chair while remaining a Tech Lead.
Release Schedule
Next Deadline: Code Freeze, November 16th (AoE)
November patch releases are targeted for November 10, with cherry-pick PRs due November 6.
Featured PRs
143463: KEP-5710: Log error when PostFilter plugin does not implement PodGroupPostFilterPlugin
jdzikowski added a scheduler initialization check for plugins that implement PostFilter but not PodGroupPostFilterPlugin, as part of KEP-5710. With the GenericWorkload feature gate enabled, the scheduler now logs an error when an enabled plugin will be skipped during scheduling for a PodGroup or CompositePodGroup. Previously, this behavior could go unnoticed, potentially leaving workload-aware scheduling integrations without the expected post-filtering behavior. The check also fulfills one of the KEP's graduation criteria for General Availability.
143387: KEP-5491: Adds integration and e2e test for List Type Attributes
everpeace added integration and end-to-end tests for list-typed matchAttribute and distinctAttribute constraints in Dynamic Resource Allocation (DRA), as specified by KEP-5491. The tests exercise allocation against a real API server and scheduler, covering intersecting and disjoint attribute lists, scalar-to-list matching, and distinctness constraints. These cases help verify that device allocation behaves correctly across different attribute combinations and provide coverage required by the KEP's test plan for beta promotion in Kubernetes v1.38.
KEP of the Week
KEP-6268: Stall and Resume Slow Watch Streams
A slow watch client can fill its kube-apiserver buffer, delaying other watchers before the server force-closes its stream. KEP-6268 instead marks that watcher unsynced, catches it up from shared watch-cache history, and resumes it on the same connection. If the needed events have aged out, the server sends 410 Gone. Clients and the API need no changes.
@nami-ant and @rpb-ant authored the KEP for SIG API Machinery. They revised the design in September to share catch-up work across watchers, following etcd's model. The KEP merged October 1, while the kube-apiserver implementation remains under review behind the off-by-default WatchCacheStallResume gate. KEP-6268 is implementable and targets Alpha in v1.38; Beta and Stable milestones are TBD.
Other Merges
- Fix for a kubelet bug that set pod sandbox CPU and memory limits to the partial sum of container limits when only some containers declared them
- kube-scheduler's preemption disruption metric gains a victim label; PreemptionVictim gains a Type() method and PreemptionCandidate.NumPodGroupDisruptions() is replaced by NumDisruptionsByVictimType()
- MutatingAdmissionWebhook config gains a staticOrder field to control manifest vs. API-registered webhook ordering; requires the ManifestBasedAdmissionControlConfig gate and must be removed before downgrading to v1.37 or earlier
- A new evictionrequest-controller creates Evictions from EvictionRequests and coordinates external responders
- Add cgroup_mount_mode to the CRI API to support writable cgroups
- kubectl selector commands like create pdb --selector now accept !=
- client-go adds DropManagedFields to omit managedFields from API responses, behind the new alpha ManagedFieldsOptOutClient gate
- kubelet adds the alpha kubelet_device_plugin_devices metric, breaking device counts down by health state
- Watch cache keeps the last snapshot behind the compaction window, avoiding etcd fallback for Exact LIST
- Workload-aware scheduling: Pod scheduling status now names its parent PodGroup and PlacementFeasible counts are clearer, stricter CompositePodGroup validation, recursive status propagation, nodes with empty topology label values skipped as placement domains, multiple placement plugins intersected, and TopologyAwareWorkloadScheduling promoted to Beta (still disabled by default)
Subprojects and Dependency Updates
- etcd has released v3.7.3, v3.6.16, and v3.5.35, including multiple security patches, as well as fixing a few v3.7 regressions.
- vertical-pod-autoscaler v1.8.1 and vertical-pod-autoscaler v1.7.3 were both released. They are built with Go 1.26.10, blocking the compromised eMudhra intermediate CAs.
- etcd v3.7.3: New patch release, see CHANGELOG; also v3.6.16, v3.5.35
- coredns v1.15.0: Fixes CVE-2026-56865, CVE-2026-56864, CVE-2026-33818, CVE-2026-97032, CVE-2026-78659, CVE-2026-97031, and CVE-2026-111943; adds DNS-over-QUIC forwarding support, opt-in JSON query logging, and authenticated RFC 2136 dynamic DNS updates
- prometheus v3.15.1: Builds with Go 1.27.3 and bumps golang.org/x/net to v0.60.0 to fix HTTP/2 vulnerabilities; also v3.13.5
- vsphere-csi-driver v3.8.0: Adds support for Kubernetes 1.36, upgrades csi-proxy to v2.0.0-alpha.2, and bumps Go dependencies to address CVEs
Shoutouts
- No shoutouts this week. Want to thank someone for special efforts to improve Kubernetes? Tag them in the #shoutouts channel.
LWKD is a publication of Kubernetes Project Contributor Communications. Read it on the web, and contribute or send corrections to the GitHub repository.
Powered by Buttondown, the easiest way to start and grow your newsletter.
updated the repo creation guidelines: community#9232
WG governance document: community#9228
added notes on future conformance: community#9227
November 16th (AoE): github.com/kubernetes/sig-release/tree/master/rel…
143463: KEP-5710: Log error when PostFilter plugin does not implement PodGroupPostFilterPlugin: kubernetes#143463
jdzikowski: github.com/jdzikowski
KEP-5710: enhancements#5710
143387: KEP-5491: Adds integration and e2e test for List Type Attributes: kubernetes#143387
everpeace: github.com/everpeace
KEP-5491: enhancements#5491
KEP-6268: Stall and Resume Slow Watch Streams: github.com/kubernetes/enhancements/blob/master/ke…
@nami-ant: github.com/nami-ant
@rpb-ant: github.com/rpb-ant
KEP merged October 1: enhancements#6269
kube-apiserver implementation: kubernetes#141450
Beta and Stable milestones are TBD: enhancements#6268
kubelet bug: kubernetes#141014
gains a victim label: kubernetes#142955
staticOrder field: kubernetes#143160
evictionrequest-controller: kubernetes#137133
cgroup_mount_mode to the CRI API: kubernetes#139824
create pdb --selector: kubernetes#142216
DropManagedFields: kubernetes#142695
kubelet_device_plugin_devices: kubernetes#143223
keeps the last snapshot behind the compaction window: kubernetes#143325
now names its parent PodGroup: kubernetes#143184
PlacementFeasible counts: kubernetes#143183
CompositePodGroup validation: kubernetes#142830
recursive status propagation: kubernetes#141860
empty topology label values skipped: kubernetes#140308
multiple placement plugins intersected: kubernetes#139565
promoted to Beta: kubernetes#140295
vertical-pod-autoscaler v1.8.1: github.com/kubernetes/autoscaler/releases/tag/ver…
vertical-pod-autoscaler v1.7.3: github.com/kubernetes/autoscaler/releases/tag/ver…
etcd v3.7.3: github.com/etcd-io/etcd/releases/tag/v3.7.3
v3.6.16: github.com/etcd-io/etcd/releases/tag/v3.6.16
v3.5.35: github.com/etcd-io/etcd/releases/tag/v3.5.35
coredns v1.15.0: github.com/coredns/coredns/releases/tag/v1.15.0
prometheus v3.15.1: github.com/prometheus/prometheus/releases/tag/v3.…
v3.13.5: github.com/prometheus/prometheus/releases/tag/v3.…
vsphere-csi-driver v3.8.0: github.com/kubernetes-sigs/vsphere-csi-driver/rel…
to the GitHub repository: kubernetes-sigs/lwkd
klog, zap, logrus などで stderr に吐かれるログのレベルを Datadog で良い感じにする
こんにちは、@key60228 です。
新しく立ち上げた GKE Cluster に Datadog Agent を入れてログを有効化してみたところ、とんでもない数のエラーログが吐かれていました。
まだ何も特別なことをしていないのに…？？？と不思議に思いログの詳細を見てみたところ、特に問題なさそうなログがほとんどでした。
結論としては、各コンポーネントが吐くログが Datadog が解釈できる形式で構造化されておらず、エラーレベルとして判定されてしまっていることが原因でした。
というわけでこの記事では各コンポーネント、ロガーがどんな構造で出力していて、どうすれば正しいログレベルで表示できるのかについてまとめていければと思います。
ソリューションを求めている方へ
Datadog のログパイプラインで加工してあげてください。
Terraform
pipelines.tf
resource "datadog_logs_custom_pipeline" "container_log_level" {
name = "container-log-level"
description = "stderr 由来で error になったコンテナログの status を、本文の klog / zap / logfmt のレベルから付け直す。"
is_enabled = true
filter {
query = "kube_cluster_name:*"
}
processor {
grok_parser {
name = "klog / zap / logfmt からレベルを取り出す"
is_enabled = true
source = "message"
samples = [
"I0911 07:22:25.408586 1 cni_writer.go:930] Successfully synced \"gke-xxxxx-xxxxx-nod-xxxxxxxx-xxxx\"", # klog (GKE cni-writer)
"E0911 07:10:07.225293 1 base_controller.go:161] \"controller failed to sync\" err=\"...\" key=\"image-sync\"", # klog (OCM work-agent)
"2026-09-11T07:19:45Z\tINFO\tcontrollers.Spoke\trequeueing\t{\"request\": {\"name\":\"xxx\",\"namespace\":\"fleetconfig-system\"}, \"after\": \"1m0s\"}", # zap console、タブ区切り (fleetconfig-controller)
"2026-09-11T04:39:22.159Z error logging/logging.go:16 NodeCollector: Failed to export metrics {\"kind\": \"receiver\"}", # zap console、スペース区切り (gke-metrics-agent)
"time=2026-09-11T06:13:28Z level=info msg=Reconciling subsys=sfc reasons=[...]", # logfmt / logrus (cilium)
]
grok {
support_rules = ""
match_rules = file("${path.module}/grok/container_log_level.grok")
}
}
}
processor {
lookup_processor {
name = "klog の severity 1 文字をレベル語に直す"
is_enabled = true
source = "klog.severity"
target = "level"
lookup_table = [
"I,info",
"W,warning",
"E,error",
"F,critical",
]
}
}
processor {
lookup_processor {
name = "zap のレベルを status remapper が解釈できる語に直す"
is_enabled = true
source = "zap.level"
target = "level"
lookup_table = [
"DEBUG,debug",
"INFO,info",
"WARN,warning",
"ERROR,error",
"DPANIC,critical",
"PANIC,critical",
"FATAL,critical",
]
}
}
processor {
lookup_processor {
name = "logrus のレベルを status remapper が解釈できる語に直す"
is_enabled = true
source = "logfmt.level"
target = "level"
lookup_table = [
"trace,debug",
"debug,debug",
"info,info",
"warning,warning",
"error,error",
"fatal,critical",
"panic,critical",
]
}
}
processor {
status_remapper {
name = "level を status にする"
is_enabled = true
sources = ["level"]
}
}
}
grok/container_log_level.grok
klog %{regex("[IWEF]"):klog.severity}%{regex("\d{4} \d{2}:\d{2}:\d{2}\.\d+")}\s+%{integer:klog.thread_id}\s+%{regex("[^\]]+"):klog.location}]\s?%{data:klog.message}
zap %{regex("\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(\.\d+)?(Z|[+-]\d{2}:\d{2})"):zap.timestamp}[\t ]+%{regex("(DEBUG|INFO|WARN|ERROR|DPANIC|PANIC|FATAL|debug|info|warn|error|dpanic|panic|fatal)"):zap.level:uppercase}[\t ]+%{data:zap.message}
logfmt_with_time time=%{notSpace:logfmt.time}\s+level=%{word:logfmt.level:lowercase}\s+%{data:logfmt.message}
logfmt level=%{word:logfmt.level:lowercase}\s+%{data:logfmt.message}
Datadog Log Management のログレベルの仕組み
Datadog のログでは status という予約済み属性が Log Explorer で表示されるログレベルに対応しています。1
Datadog Agent を使っている場合、この status は次の3つのタイミングで決まります。
Datadog Agent がログを送る時
Datadog が JSON ログをパースする時
ログパイプラインの Status Remapper を通る時
① Datadog Agent がログを送る時
Kubernetes の Datadog Agent (各ノードの DaemonSet) では stdout に出たものを info、stderr に出たものを error と判定して Datadog に送っているようです。
github.com/DataDog/datadog-agent/blob/f85c4980d2b…
② Datadog が JSON ログをパースする時
次に、ログが JSON 形式であれば、Datadog は取り込み時の前処理で status、severity、level などの属性を探し、見つかった値で status を上書きします。2
{"level":"info","msg":"..."} のようなログは、stderr に書かれていても info として扱われます。
③ ログパイプラインの Status Remapper を通る時
最後に、ログパイプライン に Status Remapper を置くと、任意の属性の値で status を上書きできます。
Status Remapper は指定された属性の値を先頭の文字列で以下の通り判定してくれます。3
値
status
0 から 7 の整数
Syslog の規格に従う
emerg または f で始まる文字列 (大文字小文字を区別しない)
emerg (0)
a で始まる文字列 (大文字小文字を区別しない)
alert (1)
c で始まる文字列 (大文字小文字を区別しない)
critical (2)
err で始まる文字列 (大文字小文字を区別しない)
error (3)
w で始まる文字列 (大文字小文字を区別しない)
warning (4)
n で始まる文字列 (大文字小文字を区別しない)
notice (5)
i で始まる文字列 (大文字小文字を区別しない)
info (6)
d、t、v、trace、verbose で始まる文字列 (大文字小文字を区別しない)
debug (7)
o または s で始まる文字列、または OK か Success に一致する文字列 (大文字小文字を区別しない)
OK
上記以外
info (6)
つまり、プレーンテキストの本文にレベルを埋め込んでいるログは、②の前処理では拾われず、③のパイプラインを用意しない限り①のストリームだけで status が決まります。
今回の自分の環境のエラーログは、まさにこのパターンでした。
対策と各コンポーネント/ロガーの仕組み
根本的には各コンポーネントが JSON でログを吐くか、レベルに応じて stdout/stderr を使い分けていればこういった問題は起きません。
しかし GKE が管理するシステムコンポーネントは出力形式をこちら側では変えられず、OSS のコンポーネントも全てが出力形式を選べるわけではありません。
また、Datadog Agent でストリームと status の mapping を変更する方法も現時点ではなさそうです。
となるとメッセージ本文からレベルを取り出し、Status Remapper で status 属性を更新するログパイプラインを用意するのが良さそうです。
今回の自分の環境で問題になったログは、出力元のロガーで分けると klog、zap、logrus の3種類でした。
これらのロガーで、どのようにレベルを抽出すれば良いか見ていきます。
klog
klog は glog の fork で、Kubernetes 本体も使用しているロガーです。
kubernetes/klog
今回の環境では、GKE のシステムコンポーネント (cni-writer、gke-metadata-server、proxy-agent etc.) や cert-manager などが klog かつ stderr でログを吐いていました。
klog の構造は以下のようになっており、先頭の L が severity を指しています。
Lmmdd hh:mm:ss.uuuuuu threadid file:line] msg
github.com/kubernetes/klog/blob/ae3961e316a310e71…
severity は I が INFO、W が WARNING、E が ERROR、F が FATAL を指します。
github.com/kubernetes/klog/blob/ae3961e316a310e71…
例えば proxy-agent の次のログは INFO ですが、stderr に書かれているため Datadog 上では ERROR になっていました。
そこで、まずは Grok Parser で属性をパースしてあげます。
klog %{regex("[IWEF]"):klog.severity}%{regex("\d{4} \d{2}:\d{2}:\d{2}\.\d+")}\s+%{integer:klog.thread_id}\s+%{regex("[^\]]+"):klog.location}]\s?%{data:klog.message}
上記ルールで IWEF の値を klog.severity に書き出しましたが、klog の場合はこの severity をそのまま Status Remapper に渡すことはできません。
I と W は期待通り info と warning になり、F は emergency になりますが、E は何にも当てはまらず、info になってしまいます。
そこで、Lookup Processor で klog.severity を調整してから level 属性に入れます。 (ついでに F も critical に調整しています)
I,info
W,warning
E,error
F,critical
その後 Status Remapper を挟んであげることで、proxy-agent の INFO ログは info として表示されるようになりました。
zap
zap は Uber が開発しているロガーで、Kubernetes 周りでは controller-runtime を使ったコントローラーの多くが zap を使っています。
uber-go/zap
今回の環境では、gke-metrics-agent や 別記事 で導入していた Open Cluster Management の fleetconfig-controller が zap でログを吐いていました。
zap には JSON で出力する JSON encoder と、人が読みやすいテキストで出力する console encoder があります。
github.com/uber-go/zap/blob/4892335e05f14bce8a98a…
console encoder はデフォルトでは、タイムスタンプ、レベル、ロガー名、呼び出し元、メッセージ、追加フィールド (JSON) を区切り文字でつないで 1 行にします。
TIMESTAMP LEVEL LOGGER CALLER MESSAGE {FIELDS}
zap の console encoder の出力は、次のルールで分解します。
zap %{regex("\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(\.\d+)?(Z|[+-]\d{2}:\d{2})"):zap.timestamp}[\t ]+%{regex("(DEBUG|INFO|WARN|ERROR|DPANIC|PANIC|FATAL|debug|info|warn|error|dpanic|panic|fatal)"):zap.level:uppercase}[\t ]+%{data:zap.message}
同じ console encoder でも、出力の細かいところはコンポーネントによって異なっているため、区切り文字はタブと半角スペース、レベルは大文字と小文字の両方に対応しています。
zap のレベルは DEBUG、INFO、WARN、ERROR、DPANIC、PANIC、FATAL とあり、今のままだと DPANIC が debug に、PANIC が info になってしまいます。
ついでに FATAL は klog に合わせて critical として扱うよう、Lookup Processor で調整してあげます。
github.com/uber-go/zap/blob/4892335e05f14bce8a98a…
"DEBUG,debug",
"INFO,info",
"WARN,warning",
"ERROR,error",
"DPANIC,critical",
"PANIC,critical",
"FATAL,critical",
なお、controller-runtime の zap のオプションをコマンドライン引数に結び付けているコントローラーであれば、--zap-encoder=json を渡して JSON encoder に切り替えられます。
JSON encoder の出力には level 属性が含まれるので、前処理で status に反映され、パイプラインは要りません。
logrus
logrus もメジャーな構造化ロガーで、今回の環境では Zitadel が logrus でログを出していました。
sirupsen/logrus
出力形式は TextFormatter と JSONFormatter から選べ、TextFormatter は端末以外に出力するとき key=value を並べた logfmt 風の形式になります。
time="2026-10-13T05:58:18Z" level=INFO msg="structured logger configured" caller="..." error="..."
レベルは level= の後ろに info、warning、error のような語で書かれます。
スクリーンショットのログは level=INFO ですが、Datadog 上では ERROR になっていました。
logrus の TextFormatter の出力は、次の 2 つのルールで分解します。
logfmt_with_time time=%{notSpace:logfmt.time}\s+level=%{word:logfmt.level:lowercase}\s+%{data:logfmt.message}
logfmt level=%{word:logfmt.level:lowercase}\s+%{data:logfmt.message}
TextFormatter は設定によってタイムスタンプを出さないことがあるため、time= で始まる行と level= で始まる行の両方に対応しています。
今回のログのように時刻を time="..." と引用符で囲む場合も、引用符ごと notSpace で logfmt.time に取り込まれるので問題ありません。
logrus のレベルは以下の通りなので、zap 同様に Lookup Processor で調整してあげましょう。
github.com/sirupsen/logrus/blob/6ef0748d1177dadae…
"trace,debug",
"debug,debug",
"info,info",
"warning,warning",
"error,error",
"fatal,critical",
"panic,critical",
これでログレベルが正しく表現されるようになりました。
!
この記事の本筋とズレるので本文では言及しませんでしたが、ZITADEL はオプションでログフォーマットを JSON 化できます。4
まとめ
パイプラインを適用したあとは、status:error のログがほとんどなくなりました。
これで万事解決…とは思いつつ、klog、zap、logrus とは異なる出力形式を持つロガーを採用しているコンポーネントが出てきたらその都度対応しないといけない…？ 🤔
GKE とか各 KaaS のシステムコンポーネント以外は大概ログの format などをこっちで指定できるから特に問題にならないのかな〜〜とは思いつつ、こんな複雑なことをしないといけないのはちょっと不思議な気がするな〜〜 5
もっと良い方法あるよって方、もし居たら教えてください〜〜
最後に
AI Shiftではエンジニアの採用に力を入れています！少しでも興味を持っていただけましたら、カジュアル面談でお話しませんか？（オンライン・19時以降の面談も可能です！）
【面談フォームはこちら】
hrmos.co
脚注
属性とエイリアス設定 - Datadog Docs ↩︎
Pipelines (Preprocessing) - Datadog Docs ↩︎
Log Status Remapper - Datadog Docs ↩︎
ZITADEL Production Setup - ZITADEL Docs ↩︎
「こんな複雑なことをしないといけないのはおかしい」というソフトウェアエンジニアの勘 - valid,invalid ↩︎
Mentions
CVE-2026-116472: ingress-nginx mirror-target self-DOS -
CVE-2026-116472: ingress-nginx mirror-target self-DOS - kubernetes#143852
CVE-2026-116472: ingress-nginx mirror-target self-DOS -
An Empty resources: {} Makes Kubernetes 1.34 to 1.36 Pods BestEffort --- title: An Empty resources: {} Makes Kubernetes 1.34 to 1.36 Pods BestEffort published: true date: 2026-11-04 15:00:00 UTC tags: Kubernetes,QoS,GitLabRunner,k3s canonical_url: devops-daily.com --- On Kubernetes 1.34, 1.35 and 1.36, one empty block in a pod spec changes how the node treats the pod. If the pod sets spec.resources: {} at pod level, the kubelet gives it QoS class BestEffort. This happens even when every container sets requests and limits, whenever the PodLevelResources feature gate is on, which is the default from 1.34. GitLab Runner's Kubernetes executor builds its job pods with that empty block. Its source does this from 18.10.0 to 19.4.3, unless the runner config sets pod-level values. We ran the same pods on k3s v1.34.12, v1.35.9, v1.36.5 and v1.37.1. On the three older versions, the empty block moved the pod into the BestEffort cgroup slice. The kubelet set its oom_score_adj to 1000, the highest value the kernel accepts. Next to a busy neighbour, a pod that requested half a CPU got 0.06 to 0.08 cores instead of 0.57 to 0.58. A namespace quota that blocks BestEffort pods rejected it. On 1.37.1 the same pods kept their Burstable and Guaranteed classes. One thing did not change, and the research we started from got it wrong. It said these pods are first in line for kubelet eviction under memory pressure. They are not. The kubelet's eviction ranking compares memory usage with the memory request. For an empty block, that request still comes from the containers. The QoS class is not part of the ranking, and the kubelet evicted our pods in the same order on every version. TL;DR - On k3s v1.34.12, v1.35.9 and v1.36.5, a pod with container requests and limits plus spec.resources: {} reported qosClass: BestEffort. The Guaranteed variant did too. On v1.37.1 they reported Burstable and Guaranteed. With the new PodLevelResourcesFixKubeletQOSClass feature gate turned off, 1.37.1 went back to BestEffort. - The node settings followed the class. oom_score_adj went from 935 (Burstable) and -997 (Guaranteed) to 1000, and the pod cgroup moved to kubepods-besteffort.slice with cpu.weight 1 and no pod-level memory.max. The container-level limits stayed in place. - A neighbour pod kept both vCPUs busy. Next to it, the BestEffort-classified pod got a median of 0.061 to 0.083 cores on the three affected versions. The same pod without the empty block got 0.573 to 0.580 (so 10.6% to 14.3%). On 1.37.1 the two were equal. - A ResourceQuota with scope BestEffort and pods: "0" rejected the pod on 1.34 to 1.36 and admitted it on 1.37. - The kubelet's memory-pressure eviction order was the same on all four versions, in all 12 repetitions: the true BestEffort pod first, then by memory usage relative to the container request. The empty block did not move a pod forward. - If you run GitLab Runner on Kubernetes, upgrade to 19.5.0 or later. As of 2 November 2026 the latest Kubernetes patch releases for 1.34, 1.35 and 1.36 still have the bug. Prerequisites - A cluster on Kubernetes 1.34, 1.35 or 1.36 (kubectl version), or a machine where you can run the lab - kubectl and jq - If you run GitLab Runner with the Kubernetes executor: its version (gitlab-runner --version) - To run the lab: a DigitalOcean token (referral link); one s-2vcpu-4gb droplet for about an hour costs a few cents Where the empty block comes from Pod-level resources (KEP-2837) let you set CPU and memory for the pod as a whole, not only per container. The feature gate PodLevelResources went alpha in 1.32 and beta, on by default, in 1.34. Our Kubernetes 1.36 post covers the in-place resize of those values. The QoS calculation has a shortcut for them. This is ComputePodQOS at v1.34.0, v1.35.0 and v1.36.0, and still at the latest patches, v1.34.13, v1.35.10 and v1.36.6: // pkg/apis/core/v1/helper/qos/qos.go, v1.36.6 // When pod-level resources are specified, we use them to determine QoS class. if utilfeature.DefaultFeatureGate.Enabled(features.PodLevelResources) && pod.Spec.Resources != nil { An empty block is not nil. It has no requests and no limits, so the function returns BestEffort. The loop over the containers never runs. The issue that reported this, #135082 from November 2025, says that almost everywhere else Kubernetes does not differentiate between nil and empty pod-level resources. The helper for that, IsPodLevelResourcesSet, treats {} as not set. The fix is PR #137150, merged on 15 July 2026 for 1.37. It adds the condition IsPodLevelResourcesSet(pod) behind a new gate, PodLevelResourcesFixKubeletQOSClass, beta and on by default from 1.37. It was not backported. The patch releases of 27 October 2026, v1.34.13, v1.35.10 and v1.36.6, still have the old condition. The rest of that release is in our Kubernetes 1.37 review; PodLevelResources itself stays beta in 1.37. GitLab Runner added pod-level resources to its job pods in 18.10.0 (MR !5922). At every tag we read (18.10.0, 19.3.5 and 19.4.3), the helper that builds the block returns a non-nil object: // executors/kubernetes/kubernetes.go, GitLab Runner v19.4.3 func (s *executor) podResourcesReference() *api.ResourceRequirements { resources := api.ResourceRequirements{} if s.configurationOverwrites != nil { resources.Limits = s.configurationOverwrites.podLimits resources.Requests = s.configurationOverwrites.podRequests } return &resources } If the runner config sets none of pod_cpu_request, pod_memory_request, pod_cpu_limit and pod_memory_limit, both maps are empty. The pod then goes to the API server with resources: {}. The MR that fixed it, !7346, says the same: job pods "were showing qosClass: BestEffort instead of the expected Burstable", and after the fix .spec.resources "should now be empty/absent (was {})". It merged on 1 October 2026 and shipped in 19.5.0. The 19.4.3 and 19.3.5 tags of 28 October 2026 still have the old function. The class matters only when the runner config gives the job containers requests or limits. Without them, the pods were BestEffort anyway. The lab The lab runs on one DigitalOcean droplet: s-2vcpu-4gb in fra1, Ubuntu 24.04, cgroup v2. A cloud firewall allows SSH from one address and nothing else. The Kubernetes API port is never exposed, and kubectl runs on the node over SSH. For each version, a script removes k3s, installs the pinned release, and runs five stages. The kubelet gets an explicit hard eviction threshold, memory.available<2Gi. It also gets a 30 second pressure transition period instead of the default 5 minutes, so repetitions do not wait for the node condition to clear. 1. pod spec resources: {} + container requests 2. ComputePodQOS sees a block: BestEffort 3. IsPodLevelResourcesSet sees {}: not set 4. oom_score_adj 1000 5. cgroup slice besteffort, cpu.weight 1 6. ResourceQuota scope BestEffort 7. eviction ranking container requests Connections: - pod spec -> ComputePodQOS - pod spec -> IsPodLevelResourcesSet - ComputePodQOS -> oom_score_adj - ComputePodQOS -> cgroup slice - ComputePodQOS -> ResourceQuota scope - IsPodLevelResourcesSet -> eviction ranking The pods: | Pod | Pod-level block | Container requests | Container limits | | --- | --- | --- | --- | | be | none | none | none | | ctl-burstable | none | 100m CPU, 256Mi | 500m CPU, 512Mi | | emp-burstable | {} | 100m CPU, 256Mi | 500m CPU, 512Mi | | ctl-guaranteed | none | 200m CPU, 256Mi | 200m CPU, 256Mi | | emp-guaranteed | {} | 200m CPU, 256Mi | 200m CPU, 256Mi | | plr-burstable | requests 100m/256Mi, limits 500m/512Mi | none | none | plr-burstable is the control for the feature itself: a real pod-level block works as documented on every version. The empty block is the only problem. We wrote down the claim and what would disprove it before the droplet existed, and committed it first (CLAIM.md). Every recorded file carries a hash of the manifests and node scripts, and the summary script refuses to mix files with different hashes. The-DevOps-Daily/k8s-empty-pod-resources on GitHub Finding 1: the class flips on 1.34 to 1.36 The API server stored the empty block exactly as written. On v1.34.12, v1.35.9 and v1.36.5 the kubelet then reported BestEffort for both emp-* pods, while their controls kept Burstable and Guaranteed. On v1.37.1 the emp-* pods got the same class as their controls: results/demo-k3s-v1.36.5.txt and results/demo-k3s-v1.37.1.txt # k3s v1.36.5+k3s1 # ... kubectl apply and wait omitted; the whole session is in the file $ kubectl get pods -n demo -o custom-columns='NAME:.metadata.name,POD-LEVEL:.spec.resources,QOS:.status.qosClass' NAME POD-LEVEL QOS ctl-burstable <none> Burstable ctl-guaranteed <none> Guaranteed emp-burstable map[] BestEffort emp-guaranteed map[] BestEffort # k3s v1.37.1+k3s1, same manifests # ... kubectl apply and wait omitted $ kubectl get pods -n demo -o custom-columns='NAME:.metadata.name,POD-LEVEL:.spec.resources,QOS:.status.qosClass' NAME POD-LEVEL QOS ctl-burstable <none> Burstable ctl-guaranteed <none> Guaranteed emp-burstable map[] Burstable emp-guaranteed map[] Guaranteed The same result on every version, from results/summary.md: | Pod | v1.34.12 | v1.35.9 | v1.36.5 | v1.37.1 | v1.37.1, gate off | | --- | --- | --- | --- | --- | --- | | ctl-burstable | Burstable | Burstable | Burstable | Burstable | Burstable | | emp-burstable | BestEffort | BestEffort | BestEffort | Burstable | BestEffort | | ctl-guaranteed | Guaranteed | Guaranteed | Guaranteed | Guaranteed | Guaranteed | | emp-guaranteed | BestEffort | BestEffort | BestEffort | Guaranteed | BestEffort | | plr-burstable | Burstable | Burstable | Burstable | Burstable | Burstable | | be | BestEffort | BestEffort | BestEffort | BestEffort | BestEffort | plr-burstable, with real pod-level requests and limits, was Burstable everywhere. Pod-level resources work. Only the empty block breaks the class. To check that the 1.37 gate is the fix and not something else in the release, we reinstalled v1.37.1 with PodLevelResourcesFixKubeletQOSClass=false on both the API server and the kubelet. Both emp-* pods went back to BestEffort, with oom_score_adj 1000. The API server and kubelet metrics reported the gate as BETA, on by default and off in that run. Finding 2: the kernel sees a BestEffort pod QoS is not only a label. The kubelet turns it into numbers that the kernel uses. We read them from /proc and /sys/fs/cgroup on the node, for the main process of each container: oom_score_adj of the main container process | | Value | Series | | --- | --- | --- | | ctl-burstable, every version | 935 | no pod-level block | | emp-burstable, 1.34 to 1.36 | 1000 | resources: {}, 1.34 to 1.36 | | emp-burstable, 1.37.1 | 935 | resources: {}, 1.37.1 | | ctl-guaranteed, every version | -997 | no pod-level block | | emp-guaranteed, 1.34 to 1.36 | 1000 | resources: {}, 1.34 to 1.36 | | emp-guaranteed, 1.37.1 | -997 | resources: {}, 1.37.1 | _Read from /proc//oom_score_adj on the node. The 1.34.12, 1.35.9 and 1.36.5 values are identical. The kernel adds this value to a process's badness score when it picks one to kill. Source: results/summary.md._ The kernel adds oom_score_adj to each process's badness score before it picks a process to kill, and it accepts values from -1000 to +1000 (proc.rst). A Burstable container gets 1000 - 1000 * request / node memory, which is 935 for a 256Mi request on this node. A Guaranteed container gets -997. The empty block gave both 1000, the same as a pod with no resources at all. oom_score, the current OOM killer score the kernel shows for the process, went from 1290 (Burstable) and 4 (Guaranteed) to 1334. The cgroup changed too. On v1.36.5: | | ctl-burstable | emp-burstable | ctl-guaranteed | emp-guaranteed | | --- | --- | --- | --- | --- | | QoS slice | kubepods-burstable.slice | kubepods-besteffort.slice | kubepods.slice | kubepods-besteffort.slice | | pod cpu.weight | 4 | 1 | 8 | 1 | | pod cpu.max | 0.5 CPU | max (none) | 0.2 CPU | max (none) | | pod memory.max | 512 MiB | max (none) | 256 MiB | max (none) | | container cpu.max | 0.5 CPU | 0.5 CPU | 0.2 CPU | 0.2 CPU | | container memory.max | 512 MiB | 512 MiB | 256 MiB | 256 MiB | | oom_score | 1290 | 1334 | 4 | 1334 | The container cgroup still has its own limits, so emp-burstable cannot use more than 512 MiB or half a CPU. What it loses is everything the kubelet sets at pod and QoS level. The pod cgroup has no memory limit and no CPU quota of its own. It sits in kubepods-besteffort.slice, where the kubelet sets cpu.weight to 1 for the whole slice. Finding 3: half a CPU requested, 0.06 to 0.08 cores received cpu.weight 1 matters only when the CPU is busy. To see how much, we ran cpu-neighbour, a Burstable pod that requests 1 CPU and runs two busy processes, so both vCPUs are always taken. Next to it, cpu-ctl and cpu-emp took turns, three times each, alternating which went first. Both request 500m CPU with a 1 CPU limit and run one busy process; cpu-emp adds the empty block. Each pod reads its own cgroup's usage_usec over 20 seconds after a 5 second warm-up. CPU used next to a busy neighbour, median of 3 runs | | Value | Series | | --- | --- | --- | | v1.34.12 cpu-ctl | 0.579 cores | cpu-ctl (no block) | | v1.34.12 cpu-emp | 0.083 cores | cpu-emp (resources: {}) | | v1.35.9 cpu-ctl | 0.58 cores | cpu-ctl (no block) | | v1.35.9 cpu-emp | 0.07 cores | cpu-emp (resources: {}) | | v1.36.5 cpu-ctl | 0.573 cores | cpu-ctl (no block) | | v1.36.5 cpu-emp | 0.061 cores | cpu-emp (resources: {}) | | v1.37.1 cpu-ctl | 0.576 cores | cpu-ctl (no block) | | v1.37.1 cpu-emp | 0.577 cores | cpu-emp (resources: {}) | _2 vCPU droplet. Both pods request 500m with a 1 CPU limit and run one busy process; the neighbour requests 1 CPU and runs two. Each pod measured its own cgroup usage_usec over 20 s. Source: results/summary.md._ On the three affected versions, cpu-emp got a median of 0.083, 0.070 and 0.061 cores, against 0.579, 0.580 and 0.573 for cpu-ctl. That is 14.3%, 12.1% and 10.6% of the control's CPU, or 7.0 to 9.4 times less. On v1.37.1 the two were the same: 0.576 and 0.577 cores. The weights explain the direction. cpu-ctl's pod cgroup has cpu.weight 20 for its 500m request and sits in the burstable slice next to the neighbour. The kubelet sets the burstable slice's weight from the sum of the CPU requests of the Burstable pods in it, which gave 63 here. cpu-emp's pod cgroup has cpu.weight 1 and sits in the besteffort slice. The kubelet always sets that slice to the minimum, cpu.weight 1. The 500m request of cpu-emp still sets a weight on its container cgroup, but that only counts against other containers in the same pod. On a busy node like this one, a CI build in the BestEffort slice competes with cpu.weight 1 at that level and can get far less CPU than Burstable or Guaranteed pods. We did not run a build. The runs were short and sequential. Each pod ran three times in a fixed alternating order, and we did not record the neighbour's own usage or other load on the host. On the affected versions, each of the nine cpu-emp runs got less CPU than every cpu-ctl run, including the runs where cpu-emp went first. The exact percentages still belong to this setup. Finding 4: a "no BestEffort" quota rejects the pod Some clusters forbid BestEffort pods with a quota like this one, so every workload must declare resources: apiVersion: v1 kind: ResourceQuota metadata: name: no-besteffort spec: hard: pods: '0' scopes: ['BestEffort'] The API server applies the scope at create time, before any kubelet sees the pod, and it calls the same GetPodQOS function. With no status yet, that function computes the class from the spec. On 1.34 to 1.36 it rejected emp-burstable, a pod whose container sets requests and limits: quota.py on k3s v1.36.5 $ grep -A3 ' quota$' results/k3s-v1.36.5/run.log 2026-11-02T10:33:20Z quota be admitted=False Error from server (Forbidden): error when creating "/root/k8s-empty-pod-resources/manifests/quota/be.yaml": pods "be" is forbidden: exceeded quota: no-besteffort, requested: pods=1, used: pods=0, limited: pods=0 ctl-burstable admitted=True pod/ctl-burstable created emp-burstable admitted=False Error from server (Forbidden): error when creating "/root/k8s-empty-pod-resources/manifests/quota/emp-burstable.yaml": pods "emp-burstable" is forbidden: exceeded quota: no-besteffort, requested: pods=1, used: pods=0, limited: pods=0 On v1.37.1 the API server admitted emp-burstable, and still rejected be. We did not run a runner. But a job pod from 18.10.0 to 19.4.x carries the same empty block, so in a namespace with this quota it should fail the same way on 1.34 to 1.36. The fix discussion names the reverse risk: a pod "could theoretically bypass restricted BestEffort quotas by being classified as Guaranteed during admission, while still behaving as BestEffort at the node level". That can happen when the API server has the fix and the kubelet does not. Finding 5: the eviction order did not change The five eviction pods each allocate a fixed amount of memory and then sleep. They are sized so that the two possible rules predict different orders: | Pod | Memory request | Allocates | Working set before the hog | Usage against request | | --- | --- | --- | --- | --- | | be | none | 50 MiB | 54 MiB | above (request is 0) | | ctl | 256Mi | 200 MiB | 204 MiB | about 52 MiB below | | gua-ctl | 256Mi (= limit) | 150 MiB | 154 MiB | about 102 MiB below | | emp + {} | 256Mi | 100 MiB | 104 MiB | about 152 MiB below | | gua-emp + {} | 256Mi (= limit) | 50 MiB | 54 MiB | about 202 MiB below | If the kubelet ranked by QoS class, emp and gua-emp would go out together with be, before ctl and gua-ctl. If it ranks by container requests, be goes first because any usage exceeds a request of zero, and the rest follow in order of usage minus request: ctl, gua-ctl, emp, gua-emp. A host process outside every pod, hog.py, then holds MemAvailable at 1000 MiB. That is below the 2Gi threshold even after all five pods are gone, so the kubelet keeps evicting until nothing evictable is left. The kubelet cannot evict the hog. The pods differ in size on purpose, to separate the two models, so this is not a matched pair. CoreDNS also ran on the node. The kubelet ranked it last in every ranking it logged. In 5 of the 12 logs the window ran long enough to show the next step: the kubelet logged cannot evict a critical pod once only CoreDNS was left. Every repetition on every version gave the request order. This is the kubelet's own ranking on v1.36.5, while it reported emp and gua-emp as BestEffort: k3s v1.36.5, repetition 1 # the class the kubelet reported for each pod $ jq -c 'select(.rep == 1) | .qos' results/k3s-v1.36.5/40-pressure.jsonl {"be":"BestEffort","ctl":"Burstable","emp":"BestEffort","gua-ctl":"Guaranteed","gua-emp":"BestEffort"} # the kubelet's ranking before each eviction; sed keeps the journald timestamp and the message, and drops the host name, the service PID, the severity letter, the kubelet's own timestamp, the second PID and the source file $ grep -E 'ranked for eviction|evicted successfully' results/k3s-v1.36.5/40-kubelet-rep1.log | sed -E 's/^([^]+) .*eviction_manager.go:[0-9]+\] /\1 /' 2026-11-02T10:38:17.816980+00:00 "Eviction manager: pods ranked for eviction" pods=["evict-check/be","evict-check/ctl","evict-check/gua-ctl","evict-check/emp","evict-check/gua-emp","kube-system/coredns-7cfb7bc9c7-svb26"] 2026-11-02T10:38:20.108268+00:00 "Eviction manager: pod is evicted successfully" pod="evict-check/be" 2026-11-02T10:38:21.127785+00:00 "Eviction manager: pods ranked for eviction" pods=["evict-check/ctl","evict-check/gua-ctl","evict-check/emp","evict-check/gua-emp","kube-system/coredns-7cfb7bc9c7-svb26"] 2026-11-02T10:38:23.555889+00:00 "Eviction manager: pod is evicted successfully" pod="evict-check/ctl" 2026-11-02T10:38:24.579473+00:00 "Eviction manager: pods ranked for eviction" pods=["evict-check/gua-ctl","evict-check/emp","evict-check/gua-emp","kube-system/coredns-7cfb7bc9c7-svb26"] 2026-11-02T10:38:27.242735+00:00 "Eviction manager: pod is evicted successfully" pod="evict-check/gua-ctl" 2026-11-02T10:38:28.259762+00:00 "Eviction manager: pods ranked for eviction" pods=["evict-check/emp","evict-check/gua-emp","kube-system/coredns-7cfb7bc9c7-svb26"] 2026-11-02T10:38:30.555840+00:00 "Eviction manager: pod is evicted successfully" pod="evict-check/emp" 2026-11-02T10:38:31.572048+00:00 "Eviction manager: pods ranked for eviction" pods=["evict-check/gua-emp","kube-system/coredns-7cfb7bc9c7-svb26"] 2026-11-02T10:38:34.078890+00:00 "Eviction manager: pod is evicted successfully" pod="evict-check/gua-emp" The source explains it. rankMemoryPressure sorts by three keys. First, whether usage exceeds the memory request. Then pod priority. Then usage minus the memory request. The first and third keys get the request from GetResourceRequestQuantity. For an empty block, that helper asks IsPodLevelResourcesSet, gets false, and uses the container requests. Priority does not read requests, and the QoS class is not an input at all. If the kubelet had used a request of zero for emp, emp would have been in the first group with be. It came fourth. When each pod was evicted, 12 repetitions | Series | Samples | Min | Median | p95 | Max | | --- | --- | --- | --- | --- | --- | | be | 12 | 5.6s | 8.350000000000001s | 11.5s | 11.5s | | ctl | 12 | 9.1s | 11.95s | 15.3s | 15.3s | | gua-ctl | 12 | 12.7s | 16s | 20.9s | 20.9s | | emp | 12 | 16.1s | 20.15s | 24.7s | 24.7s | | gua-emp | 12 | 19.5s | 23.6s | 29.9s | 29.9s | _Seconds after the host hog started, from the kubelet's log. 3 repetitions each on k3s v1.34.12, v1.35.9, v1.36.5 and v1.37.1. Every repetition evicted the pods in the same order. Source: results/summary.md._ In all 12 repetitions the kubelet started to reclaim 3.0 to 9.2 s after the hog started, and evicted the first pod 2.3 to 3.3 s later. After that it evicted one pod at a time. 41 of the 48 gaps between evictions were 4.0 s or less (median 3.6 s, longest 8.2 s). The last pod was gone 19.5 to 29.9 s after the hog started. It does not wait 10 seconds between evictions: after each one it waits for the pod to be cleaned up and then checks again. Its own available figure at the first eviction was 931 to 1212 MiB, well under the 2Gi threshold and far from empty. The kernel logged no OOM kill in any repetition, so the order is the kubelet's alone. The two mechanisms are easy to mix up. The kubelet's eviction manager checks every 10 seconds by default when it has nothing to evict, and it ignores the class. The kernel OOM killer acts when memory runs out before the kubelet frees any. It adds oom_score_adj to each process's badness score, and the empty block gives the pod the same 1000 as a pod with no resources at all (Finding 2). We did not trigger a kernel OOM, so we did not watch the kernel make that choice. What we got wrong before measuring The research note we started from had the core right and two details wrong. - Right: Kubernetes 1.34 to 1.36 compute BestEffort for an empty pod-level block. GitLab Runner 18.10 to 19.4 build their job pods with that block (from the source and the fix MR). On 1.34 to 1.36 the pods get oom_score_adj 1000. 1.37 fixes the class while its new gate is on, which is the default. - Wrong: it called #137150 the upstream issue. That is the fix PR. The issue is #135082. - Wrong, and it matters: it said these pods are first in line for kubelet eviction under memory pressure. Finding 5 shows they are not. - Missing: the 1.37 fix is a beta feature gate, not a plain code change, and nothing was backported to the 1.34, 1.35 or 1.36 patch releases. We got one prediction wrong ourselves. CLAIM.md said the BestEffort-classified pod would get under 10% of the control's CPU. It got 10.6% to 14.3%. Our falsification line was 50%, so the claim stands, but the number we wrote down was too pessimistic, and we report it as a miss. We also kept a one-repetition smoke run on v1.36.5 (results/_smoke-k3s-v1.36.5/). It used the same code stamp and gave the same results, but the summary does not count it. What to do 1. Find the pods. This query lists every pod with an empty pod-level block and the class it got: 2. GitLab Runner 18.10.0 to 19.4.x: upgrade to 19.5.0 or later. The job pods then go out without a pod-level block, and the class comes from the containers again. 3. If you cannot upgrade the runner yet, give the block real values. Set pod_cpu_request and pod_memory_request (and the _limit keys if you want limits) under [runners.kubernetes]. The QoS class then comes from those values, like plr-burstable above. This is a behaviour change, not only a workaround: the pod cgroup then takes its limits from the pod-level values too, as it did for plr-burstable. 4. Kubernetes 1.37 fixes it by default. The fix sits behind a beta gate, so check that nobody turned PodLevelResourcesFixKubeletQOSClass off. 5. Check your own templates. A Helm chart that renders a pod-level block from a value that defaults to {} produces the same bug. Render the key only when it has content: An empty resources: {} inside a container is harmless. Only the pod-level one under spec changes the class. What we could not conclude - k3s is one distribution. We ran upstream Kubernetes code as k3s packages it. A managed service can turn the PodLevelResources gate off or carry its own patches. Run the jq query on your own cluster. - No GitLab instance and no runner ran here. The runner part rests on its source at each tag and on the MR description. The emp-* pods reproduce the field the runner sends, not a full job pod with helper and service containers. - We did not trigger a kernel OOM kill. The oom_score values are the numbers the kernel uses to choose a victim, read from /proc. We did not watch it choose. - The eviction timing is ours. It depends on the 2Gi threshold, the hog's 1000 MiB target and how fast it allocates. The order is the finding, not the seconds. - The CPU gap depends on the neighbours. One neighbour with a 1 CPU request on a 2 vCPU node gave the numbers above. More Burstable or Guaranteed load should make the BestEffort slice's share smaller. On an idle node the weights do not matter, because they only divide contended CPU time; we did not measure that case. - One node, one admission check. We did not test scheduling across nodes, or any admission rule other than the one quota. The gate-off run on 1.37.1 covered only the class, the kernel-facing numbers and the quota, not CPU or eviction. - Other consumers of the class were not tested. From the source, the kubelet also rejects new BestEffort pods while the node has the MemoryPressure condition. The static CPU manager gives exclusive CPUs only to Guaranteed pods. We did not measure either. Summary On Kubernetes 1.34, 1.35 and 1.36, with the PodLevelResources gate at its default (on), spec.resources: {} makes a pod BestEffort no matter what its containers request. On our node that meant oom_score_adj 1000 instead of 935 or -997, and a place in the BestEffort slice with cpu.weight 1 and no pod-level memory or CPU limit. Next to a busy neighbour the pod got about 7 to 9 times less CPU, and a quota that forbids BestEffort pods rejected it. The empty block did not change the kubelet's eviction order. The ranking uses memory requests, and for an empty block those still come from the containers: 12 of 12 repetitions evicted our pods in exactly the order the requests predict. Kubernetes 1.37 fixes the class behind a beta gate that is on by default, and GitLab Runner 19.5.0 stops sending the block. Run the jq query above on your clusters, and upgrade whichever side you can first. _Originally published at devops-daily.com._
CalicoとCiliumの通信経路を実測する：Service・VXLAN・Policy・既存TCP やったこと 代表的なCNIであるCalicoとCiliumを取り上げ、kube-proxyありのiptables構成と、kube-proxyなしのeBPF構成の通信経路を比較した。 同じPod・Service・NetworkPolicyの構成で動かしたところ、HTTP接続やDNS、readinessによる接続先の切り替え、通信の許可・拒否といった基本動作に差は見られなかった。 一方、Pod内やNode上でパケットを採取して確認すると、Serviceの宛先がbackend宛に変換される場所や、接続状態を保持する仕組みに明確な違いがあった。 CNIとは CNI（Container Network Interface）は、KubernetesがPodのネットワークをセットアップする際に呼び出す共通インターフェース規格である。CNIプラグインを通じてネットワークインターフェースやIPアドレスを設定し、Podをネットワークにつなぐ。CalicoやCiliumはCNIに対応したネットワークソフトウェアで、Pod間通信やNetworkPolicyによるアクセス制御も担う。 kube-proxyの役割 PodにIPアドレスが付いて通信できるようになっても、ServiceのClusterIPをどのbackend Podへ振り分けるかという処理が必要になる。このServiceの転送ルールを各Nodeに設定するのがkube-proxyである。ServiceとEndpointSliceの変化を監視し、接続先の追加・削除などをルールに反映する。 今回のiptablesモードでは、kube-proxyがカーネルにルールを設定し、実際のbackend選択と宛先変換はLinuxカーネルが行う。通信のたびにkube-proxyプロセスがパケットを受け取り、中継する構成ではない。Pod同士をつなぐネットワークと、その上でServiceの宛先を選ぶ処理を分けると、Calicoとkube-proxyが担う役割も見えやすくなる。 eBPFでServiceを処理する構成 eBPFはLinuxカーネルのパケット処理やソケット操作の途中でプログラムを動かせる仕組みである。CNIのような接続規格やkube-proxyのようなコンポーネントを指す名前ではなく、ネットワーク機能の実装にも使われる技術である。 CiliumはeBPFを使って、Pod間通信やNetworkPolicyに加え、Serviceの振り分けも実装する。このService機能でkube-proxyを置き換えるのがkube-proxy replacementである。今回有効にしたsocket-LBでは、アプリのconnect()を処理する段階でbackendを選び、ソケットの宛先を書き換える。アプリは同じClusterIPへ接続していても、宛先が変わる場所はiptables構成と異なる。 CalicoとCiliumの特徴と今回の検証構成 CalicoはBGPによるルーティングや柔軟なNetworkPolicy、CiliumはeBPFによる高度な通信制御やHubbleによる可視化が主な特徴として挙げられる。Calicoにもkube-proxyを置き換えるeBPFモードがあるが、本記事ではCalicoのiptables構成と、Ciliumのsocket-LBを有効にしたeBPF構成を比較した。以降のCalicoおよびCiliumの記述は、この検証構成を指す。 役割 今回のCalico構成 今回のCilium構成 PodのIP・ネットワーク接続 Calico Cilium NetworkPolicyの適用 Calicoが設定したiptablesルール CiliumのeBPF Serviceのbackend選択と宛先変換 kube-proxyが設定したiptablesルール CiliumのeBPF（socket-LB） 検証結果 アプリから見た基本動作は同じだったが、ServiceからPodへの宛先変換は異なる場所で処理されていた。 主な共通点と違いは次の4点である。 Serviceの宛先変換：CalicoはPodを出た後にNode上で変換する。一方、Ciliumはsocket接続時（connectシステムコール時）に変換するため、Podのeth0には最初からbackend宛のSYNパケットが現れる。 別Nodeへの転送：どちらもVXLANによるカプセル化を使用していた。ただし、外側UDPのポート番号とVXLANのVNIの扱いに違いがあった。 NetworkPolicy：どちらもパケットが宛先Podへ届く前に破棄された。CalicoではiptablesのDROPカウンタ、Ciliumではendpointの拒否イベントとして観測できた。 readinessによる切り替え：NotReadyになったPodは新規接続の振り分け先から除外されたが、すでに確立していた既存TCP接続は切断されず、同じPodと通信を維持できた。 最も大きな違いは黄色いひし形で示した宛先変換の位置にある。図は別Nodeのbackendに接続する場合を描いている。 ※図の凡例：青はPod・アプリ、黄色いひし形は変換やPolicyの判定処理、灰色は転送経路、赤はパケットの拒否を表す。 Container内のアプリプロセスはuser spaceで動く。Pod内のContainerは同じnetwork namespaceを使い、IPアドレスとポート空間を共有する。アプリのsocketやPodのeth0はこのnamespaceに属する。socket-LBのcgroup hookもiptablesもカーネル内の処理である。図ではkindのContainerとして動く論理Nodeと、その中の処理の区分を示した。これらのNodeは同じLinuxカーネル上で動く。 比較した構成 検証環境はcontrol-plane 1台、worker 2台のクラスタ構成とした。clientと同じworkerノードにbackend Podを1台、別workerノードにもう1台配置している。Serviceへの新規接続とPod IPへの直接接続を比較し、readiness probeの状態変化やNetworkPolicyの適用前後における挙動を観測した。 項目 Calico Cilium バージョン v3.33.0 v1.20.2 Serviceの実装 kube-proxyのiptablesモード kube-proxy replacement、Full socket-LB Node間の転送 VXLAN、BGP無効 VXLAN 共通条件 Kubernetes v1.35.0、IPv4、Pod MTU 1400 同左 検証詳細 Calicoの場合 ServiceのClusterIPへ接続すると、client Podのeth0だけでなく、Node側のvethインターフェース上でもClusterIP宛のSYNパケットが観測された。パケットの宛先がbackendのPod IPへ変換されるのは、Nodeに入った後である。 別Node上で動作するbackend Podが選択された通信経路を追跡した結果は、以下のとおりである（IPアドレスは役割名に置き換えている）。 client Pod eth0 client:33706 → Service:8080 SYN seq=433715408 Nodeのhost veth client:33706 → Service:8080 SYN seq=433715408 vxlan.calico client:33706 → backend:8080 SYN seq=433715408 backend Pod eth0 client:33706 → backend:8080 SYN seq=433715408 送信元ポートとSYNシーケンス番号は一貫しており、宛先IPアドレスのみが途中で書き換わっていることがわかる。これは、Node上でkube-proxyが生成したiptablesチェーン（KUBE-SERVICES → KUBE-SVC-* → KUBE-SEP-* → DNAT）のルールによって処理されているためである。実際に新規接続を8回試行したところ、該当Serviceのエントリポイントにあるカウンタも正確に8パケット分増加していた。 なお、ここでkube-proxyが担っている役割は、カーネルにiptablesルールをプロビジョニングすることである。通信が発生するたびにパケットがkube-proxyプロセスへ渡されるわけではなく、Linuxカーネル自体がiptablesルールに従ってbackendを選択し、DNAT（宛先NAT）を実行する。[1] また、この宛先変換のマッピング情報はconntrack（接続追跡テーブル）に保持される。今回の検証でも、original方向にclient → Service、reply方向にbackend → clientという対応関係が記録されていた。この接続状態が管理されているおかげで、Service宛てに接続したクライアントと、実際に応答を返すbackend Podとの間で正常にセッションが維持される。 Ciliumの場合 一方、CiliumではクライアントアプリがClusterIP宛てに接続を要求していても、client Podのeth0でキャプチャされたSYNパケットは最初からbackendのPod IP宛てになっていた。新規に試行した8回の接続すべてにおいて、この挙動が確認できた。 これは、Ciliumのsocket-LB機能が、アプリケーションがconnect()システムコールを発行した時点でServiceのbackendを選択し、ソケットの宛先アドレス自体を書き換えるためである。TCPパケットが生成されるよりも前の段階で宛先が書き換わるため、パケットがPodを出た後にNode上でServiceのDNATを行う必要がなくなる。[2] 今回の通常設定ではソケットトレースログが間引かれるため、一時的にレートリミットを解除して別の8接続を採取した。各ソケット（socket cookie）において、以下のように変換前後の状態を確認できた。 接続先の変換 Service:8080 → backend:8080 アプリに見せる宛先 backend:8080 → Service:8080 2行目は、アプリがgetpeername()で接続相手の情報を参照した際に返される逆変換（reverse translation）の処理である。ネットワーク上ではbackend Podと直接通信しつつも、アプリケーション側にはあくまで「Serviceに接続している」と見せかける仕組みになっている。実際のパケット送受信はbackend宛てのBPF conntrackテーブルで追跡され、Serviceとの対応関係はBPFのsocket reverse NATテーブルで管理されていた。 なお、socketトレースとpcap（パケットキャプチャ）の突き合わせは、clientのcgroup、逐次実行した各接続のタイムスタンプ、および応答したbackend Podの組み合わせによって確認した。socket cookieはパケットヘッダには含まれず、トレース側には送信元ポートが出力されないため、これらを直接の結合キーにすることはできない点に注意が必要である。 このアーキテクチャの違いは障害調査やトラブルシューティング時のパケット解析（tcpdump）にも大きく影響する。Cilium環境のPod内でパケットをキャプチャしてClusterIP宛ての通信が見当たらなくても、Serviceが使われていないわけではない。socket-LBが有効な構成では、Pod内であっても追うべき宛先はbackendのIPアドレスになる。 別Nodeへの転送は、両方ともVXLANを採用 Serviceの宛先変換が行われる場所は異なっていたものの、今回の検証構成におけるNode間のパケット転送は、CalicoとCiliumのどちらもVXLANによるオーバーレイネットワークを用いていた。同一Node内のPod同士の直接通信ではVXLANヘッダは付与されないが、別Node上のPod宛て通信では、内側のPod間TCPパケットが外側のNode間UDPパケットでカプセル化される。 内側のTCPパケットにおける送信元・宛先IP、ポート番号、SYNシーケンス番号を追跡したところ、送信元NodeのNIC、トンネルデバイス、宛先NodeのNIC、そして受信側Podのeth0まで、一連の通信を正確に追うことができた。トンネルデバイス名としては、Calicoがvxlan.calico、Ciliumがcilium_vxlanを使用していた。 特に興味深いのはVXLANヘッダに含まれるVNI（VXLAN Network Identifier）の使われ方である。Calicoでは固定値の4096が設定されていたのに対し、Ciliumではclientからbackendへの往路が8639、復路が40262となっており、それぞれ送信元Podに割り当てられたSecurity Identityの値と一致していた。Ciliumはトンネルでパケットをカプセル化して転送する際、VNIフィールドを利用して送信元Podのセキュリティ属性（どのNetworkPolicyグループに属するか）を宛先Nodeへ伝達している。[3] NetworkPolicyの動作：どちらも宛先Podの手前で拒否 別Node上で動作するbackend Podに対してIngressの隔離NetworkPolicyを適用し、client PodからPod IPへ直接接続を試行した。その結果、どちらの構成でもSYNパケットは宛先Nodeのeth0までは到達したものの、backend Podのeth0には届かずに破棄された。 今回の検証では1接続の最初のSYNパケットと再送パケットを含めて計3パケットを観測した。Calicoでは対象Pod向けのiptablesチェーンにおいてDROPカウンタが3パケット（180バイト）増加し、Ciliumでは対象endpointのイベントログに3件のPolicy deniedが記録された。その後、送信元Podを許可するルールを追加すると、どちらの構成でもパケットがbackend Podへ到達し、HTTP 200の正常応答が得られるようになった。 「通信が拒否された」という最終的な結果は同じであっても、トラブルシューティング時に確認すべき調査ポイントは異なる。CalicoであればiptablesチェーンのDROPカウンタを確認し、CiliumであればeBPFエンドポイントやPolicyのイベントログをパケットトレースと突き合わせることで、パケットが宛先Nodeに到達した後に破棄された要因を正確に特定できる。 readiness probeによる制御：新規接続の振り分け先変更と既存TCPの継続 通信確立中のbackend PodをNotReady状態に遷移させ、選択状態の収束を確認してから新規TCP接続を試すと、もう一方のReadyなPodへ切り替わっていた。しかしその間も、すでに確立していた既存のTCP接続は切断されることなく、元のPodから継続してHTTP 200応答を受け取り続けた。 readiness probeの失敗は、Pod上のアプリケーションのTCPソケットを強制クローズする処理ではない。今回の検証用backend Podは、NotReady状態であってもリクエストを受け取れば正常に応答できる状態を保っていた。つまり、Serviceで「新規接続の振り分け対象から外れること」と、「既存の接続済みソケットで通信を維持できること」は完全に両立する。 観測した状態 Calico Cilium NotReady後の新規8接続 すべて残ったReadyなPodへ すべて残ったReadyなPodへ 既存TCP 同じ接続で52応答 同じ接続で55応答 保持していた接続状態 Serviceとbackendを結ぶconntrack backend宛のBPF conntrackとsocket reverse NAT 既存のTCP接続についてはPodを削除せずにclientと同一Node上のbackendとの間で約1分間にわたり観測を継続した（クライアント側での自動再接続は行わずに1本のTCPセッションを維持）。その後、readiness状態を回復させると、両構成ともに新規接続の振り分け先として2つのbackend Podが再び選択されるようになった。なお、通常のClusterIP Serviceに対するDNS名前解決の結果は、この一連の動作中も一貫して同一のClusterIPアドレスを返し続けていた。 まとめ CalicoとCiliumのどちらを採用した場合でも、アプリケーションから見たKubernetesの基本動作（Service連携やアクセス制御）は共通していた。しかし、パケットレベルの挙動に着目すると、Podから送出されるパケットの宛先アドレスには決定的な違いが存在していた。Calico（iptables構成）ではパケットがPodを出た後にNodeのカーネル内でServiceからbackendへの変換が行われるのに対し、Cilium（socket-LB構成）ではソケットのconnect()処理時にあらかじめ変換が完了している。 このアーキテクチャの違いを理解しておくことは、パケットキャプチャ（tcpdump）を用いた障害解析やネットワーク設計において極めて重要である。Calicoでは「どの観測点の間でClusterIPからPod IPへ宛先が書き換わるか」を追う必要がある一方、Ciliumでは最初からbackend宛てのパケットとsocket-LBの状態を突き合わせて確認しなければならない。一見同じように機能するKubernetesクラスタであっても、基盤となるCNIの実装特性に応じた適切な観測ポイントを見極めることが肝要である。 余談: なぜkube-proxyを利用しない構成が広がっているのか kube-proxyを用いないいわゆる「kube-proxy-less」な構成は、主要クラウド各社のマネージドKubernetesサービスでも標準採用が進んでいる。2026年10月時点の各社公式ドキュメントに基づくと、現状は以下のように整理できる。 クラウドの構成 ネットワークの実装 kube-proxy GKE Autopilotの新規クラスタ Ciliumを使うDataplane V2が既定 使用しない AKS Automatic Azure CNI Overlay powered by Ciliumが既定 使用しない EKSのEC2ノード構成（Auto Modeを除く） Amazon VPC CNIとkube-proxy add-on 使用する Google CloudのGKE Dataplane V2では、Serviceのパケット転送およびNetworkPolicyの評価をeBPFで処理し、高度なポリシーログ出力も提供している。Microsoft AzureのAKSでも、Azure CNIとCiliumのデータプレーンを統合することで、Service転送やセキュリティポリシーの適用、さらに拡張アドオンを通じた詳細な可視化を実現している。[4, 5] 一方、AWSのEKSにおける標準的なEC2ノード構成では、Amazon VPC CNIがPodネットワークのプロビジョニングを行い、Serviceのロードバランシングはkube-proxyアドオンが担当する。NetworkPolicyの適用エンジンと、Serviceの負荷分散にkube-proxyを用いるかどうかは独立した選択肢となっている。[6, 7] なお、Calicoにもkube-proxyを置き換えてeBPFでServiceを処理するモードが存在し、kube-proxy自体にも従来のiptablesに代わるnftablesモードなどのパフォーマンス改善が進められている。そのため、単に「CalicoかCiliumか」という製品の比較にとどまらず、「どのデータプレーン層でServiceルーティングを実装するか」という設計視点が重要となる。[8, 9] Ciliumをフル活用する構成の大きな利点は、Serviceのbackend検索やNetworkPolicyの評価、通信の可観測性をすべて同一のeBPFデータプレーンで一元処理できる点にある。特にsocket-LBを適用した場合、ソケット接続の開始時点でbackendが確定するため、後段のネットワークスタックにおけるService DNAT処理を丸ごと省略できる。今回の検証においてPodのeth0で観測された宛先の違いは、まさにこのアーキテクチャ設計の違いがパケットレベルに現れた好例といえる。[10] AIの利用について 本記事の作成にはAIを利用している。 参考文献 kubernetes/pkg/proxy/iptables/proxier.go at v1.35.0 · kubernetes/kubernetes · GitHub cilium/Documentation/network/kubernetes/kubeproxy-free.rst at v1.20.2 · cilium/cilium · GitHub cilium/Documentation/network/concepts/routing.rst at v1.20.2 · cilium/cilium · GitHub GKE Dataplane V2 | GKE networking | Google Cloud Documentation Configure Azure CNI Powered by Cilium in Azure Kubernetes Service (AKS) - Azure Kubernetes Service | Microsoft Learn Assign IPs to Pods with the Amazon VPC CNI - Amazon EKS Manage kube-proxy in Amazon EKS clusters - Amazon EKS About Calico eBPF | Calico Documentation Virtual IPs and Service Proxies | Kubernetes Kubernetes Without kube-proxy — Cilium 1.20.4 documentation
Stop Writing Bash Scripts for Everything! Build Your Own DevOps CLI Tool With Go Imagine having one command that checks your servers, monitors Docker containers, tests API endpoints, and reports infrastructure health. Instead of maintaining 20 different scripts, you build one powerful CLI tool. Sounds cool? That's exactly where Go becomes interesting for DevOps engineers. Bash is amazing for automation. Python is excellent for scripting. But when you want to build portable, compiled command-line applications that are easy to distribute, Go deserves your attention. Think about tools like: - Docker - kubectl - Terraform - Helm - Hugo Go is used to build all of these. So why not build your own? Let's create a practical DevOps CLI tool from scratch. 🚀 --- 🤔 1. Why Should DevOps Engineers Learn Go? Imagine you're managing multiple servers. You regularly need to: - Check whether servers are reachable. - Monitor API availability. - Inspect Docker containers. - Check application health. - Automate repetitive operations. - Generate reports. You could write separate Bash or Python scripts for every task. But as your automation grows, maintaining those scripts can become difficult. Go offers some useful advantages: | Feature | Why it matters | |---|---| | Compiled binaries | Easy application distribution | | Static typing | Catch many errors during development | | Goroutines | Run concurrent tasks | | Standard library | Build HTTP clients and servers | | Cross-compilation | Build executables for different platforms | | Fast startup | Useful for command-line applications | The idea isn't to replace Bash or Python. It's to choose the right tool for the job. ⚡ 2. What Exactly Is a CLI Tool? CLI stands for Command-Line Interface. You've probably used these commands: docker ps kubectl get pods terraform plan git status These are examples of command-line interfaces. A CLI tool allows users to interact with software through commands and arguments. Imagine building your own tool called devopsctl. Its commands could look like this: devopsctl server check devopsctl docker status devopsctl api health devopsctl system info Instead of manually remembering multiple scripts, you have one organized command-line application. Our goal is to build a small version of this idea. --- 🛠️ 3. Set Up Your Go Environment First, install Go from: go.dev Verify the installation: go version Create your project: mkdir devopsctl cd devopsctl Initialize a Go module: go mod init devopsctl Create a file: touch main.go Your project structure: devopsctl/ │ ├── main.go ├── go.mod └── README.md Now let's start building. 🌐 4. Build Your First Server Health Checker One common DevOps task is checking whether a service is reachable. Let's build a command that checks multiple URLs concurrently. Create main.go: package main import ( "fmt" "net/http" "sync" "time" ) func checkServer(url string, wg *sync.WaitGroup) { defer wg.Done() client := http.Client{ Timeout: 5 * time.Second, } start := time.Now() response, err := client.Get(url) if err != nil { fmt.Printf("[DOWN] %s: %v\n", url, err) return } defer response.Body.Close() latency := time.Since(start) fmt.Printf( "[UP] %s | Status: %s | Latency: %v\n", url, response.Status, latency, ) } func main() { servers := []string{ "https://google.com", "https://github.com", "https://go.dev", } var wg sync.WaitGroup for _, server := range servers { wg.Add(1) go checkServer(server, &wg) } wg.Wait() fmt.Println("Health checks completed!") } Run it: go run main.go Example output: [UP] https://google.com | Status: 200 OK | Latency: 120ms [UP] https://github.com | Status: 200 OK | Latency: 180ms [UP] https://go.dev | Status: 200 OK | Latency: 95ms Health checks completed! Your first Go-based DevOps utility is ready. What did we learn? - net/http handles HTTP requests. - sync.WaitGroup coordinates concurrent tasks. - go starts a goroutine. - time measures request duration. - Error handling helps identify unavailable services. One important detail: this checks whether an HTTP request succeeds at the transport level. A service returning HTTP 500 is still reachable, but its application response indicates an error. A production health checker should evaluate HTTP status codes and application-specific health endpoints. --- 🐳 5. Add Docker Container Monitoring Now let's make the project more interesting. What if your tool could inspect Docker containers? Docker exposes a command-line interface that you can call from Go. For a simple learning example: package main import ( "fmt" "os/exec" ) func checkDocker() { cmd := exec.Command("docker", "ps") output, err := cmd.CombinedOutput() if err != nil { fmt.Println("Docker command failed:", err) return } fmt.Println(string(output)) } func main() { fmt.Println("Checking Docker containers...") checkDocker() } Run: go run main.go This executes: docker ps and displays the result. For a more advanced production tool, consider using the Docker Engine API or official Go SDK rather than parsing command output. Also, external command execution should use fixed executable paths or carefully validated arguments. Never pass untrusted user input directly into a shell command. --- 📊 6. Turn It Into a Real CLI Application So far, we've written individual functions. But a professional CLI should support commands and arguments. For example: devopsctl health devopsctl docker devopsctl version Go provides the flag package in its standard library. Here's a simple example: package main import ( "flag" "fmt" ) func main() { health := flag.Bool( "health", false, "Check application health", ) version := flag.Bool( "version", false, "Show application version", ) flag.Parse() if *health { fmt.Println("Application is healthy!") return } if *version { fmt.Println("devopsctl version 1.0.0") return } flag.Usage() } Run: go run main.go --health Output: Application is healthy! Check the version: go run main.go --version For larger applications, explore Cobra, a popular Go library for building command-line applications. spf13/cobra --- 🚀 7. Compile Your CLI Into a Single Executable Here's one of the most satisfying parts of Go. Build your application: go build -o devopsctl Run it: ./devopsctl --health You can also cross-compile for different operating systems and architectures. For example, build a Linux AMD64 executable from another supported development environment: GOOS=linux GOARCH=amd64 go build -o devopsctl-linux Build for Windows: GOOS=windows GOARCH=amd64 go build -o devopsctl.exe Build for macOS on Apple Silicon: GOOS=darwin GOARCH=arm64 go build -o devopsctl-macos This makes it convenient to distribute your CLI without requiring users to install a separate Go runtime. Remember that cross-compilation can depend on native dependencies and build configuration. The examples above are simplest for pure-Go projects. --- ☸️ 8. Take It Further: Build a Kubernetes CLI Now imagine extending devopsctl with Kubernetes functionality. You could support commands such as: devopsctl k8s pods devopsctl k8s deployments devopsctl k8s nodes Go has a powerful Kubernetes ecosystem. Explore: - client-go — Kubernetes API client library. - controller-runtime — Libraries for controllers and operators. - Kubebuilder — Framework for building Kubernetes APIs and controllers. Official resources: kubernetes.io kubernetes/client-go With these tools, you can build applications that interact with Kubernetes clusters programmatically. For example, a Kubernetes monitoring CLI could: 1. Connect to a cluster using kubeconfig. 2. List Pods across namespaces. 3. Identify Pods that are not Ready. 4. Display restart counts. 5. Retrieve selected Pod logs. 6. Generate a troubleshooting report. That is a much more advanced project than simply running kubectl get pods. --- 🧪 9. Build This Portfolio Project: DevOpsCTL Here's the project I'd recommend building and publishing on GitHub. <text>Project: DevOpsCTL — Your Personal DevOps Command-Line Assistant</text> Core features | Module | Functionality | |---|---| | Server | Check endpoint availability | | Docker | List and inspect containers | | Kubernetes | List Pods and Deployments | | System | Display basic system information | | API | Check HTTP status and latency | | Reports | Export health-check results as JSON | Suggested architecture DEVOPSCTL | +----------+----------+ | | | v v v Server Docker Kubernetes Module Module Module | | | v v v HTTP Docker K8s API Client SDK Client | | | +----------+----------+ | v CLI Output | v JSON Reports Suggested repository structure devopsctl/ │ ├── cmd/ │ └── root.go │ ├── internal/ │ ├── server/ │ ├── docker/ │ ├── kubernetes/ │ └── system/ │ ├── main.go ├── go.mod ├── Dockerfile ├── README.md └── LICENSE Advanced features to add later - Prometheus metrics integration. - YAML configuration support. - Concurrent health checks with configurable limits. - Colorful terminal output. - Unit and integration tests. - GitHub Actions CI pipeline. - Release binaries for Linux, macOS, and Windows. Don't build everything at once. Start with the health checker, then add one module at a time. --- 🗓️ 10. Your 7-Day Go Developer Challenge Want to actually learn Go? Follow this challenge. | Day | Task | |---|---| | 1 | Learn Go syntax, variables, and functions | | 2 | Learn structs, slices, maps, and interfaces | | 3 | Practice goroutines and channels | | 4 | Build a REST API | | 5 | Build a server health checker | | 6 | Integrate Docker or Kubernetes | | 7 | Publish your first Go CLI project on GitHub | At the end of seven days, you won't be a Go expert. But you'll have something much more valuable: a working project and practical experience. --- 🎯 Final Thoughts Go is not just another programming language to add to your resume. It's a way to move from writing small automation scripts toward building real infrastructure tools. You can use it to create: - DevOps automation utilities. - Cloud-native applications. - REST APIs. - Kubernetes operators. - Monitoring systems. - Distributed services. - Command-line applications. And the best part? You don't need to master the entire language before building something useful. Start small. Build something. Break it. Debug it. Improve it. Don't just learn tools built by engineers. Start building tools that other engineers can use. 👇 Let's discuss! If you could build your own DevOps CLI tool, which feature would you implement first? 1. Docker monitoring 2. Kubernetes troubleshooting 3. AWS resource management 4. Server health monitoring 5. CI/CD automation Share your ideas in the comments! --- 📚 Continue Your Go Learning Journey Want a structured resource to learn Go? 🐹 Mastering Go — Complete Guide You can also explore: - 🐳 Docker Mastery - ☸️ CKA Complete Study Guide - 🏗️ Terraform Associate (003) Exam Crash Course - 🚀 DevOps Complete Pack Learn Go. Automate intelligently. Build the tools of tomorrow.
LWKD: Week Ending October 25, 2026 Last Week in Kubernetes Development Week Ending October 25, 2026 Developer News SIG ContribEx has updated the repo creation guidelines and the WG governance document to clarify how Working Groups relate to code ownership and repository creation, giving future WG proposals a clearer path from charter to code. SIG Architecture has added notes on future conformance in the developer guide, outlining how upcoming conformance work (including the newly formed WG Workload Conformance) will relate to existing conformance definitions. SIG ContribEx is coordinating KubeCon EU 2027 Project Pavilion and PR participation for SIGs, WGs, Committees, and Subprojects; interested groups should submit the registration form by November 25. The Maintainer Summit NA 2026 unconference is accepting topic ideas; sessions are open community-led discussions and need only a topic, a note-taker, and a facilitator attending in Salt Lake City. The SIG Scheduling leadership change proposed last week has taken effect after lazy consensus closed on October 26: Heba Elayoty is now a SIG Chair, Matt Matejczyk is a SIG Tech Lead, and Kensei Nakada has stepped down from Chair while remaining a Tech Lead. Release Schedule Next Deadline: Code Freeze, November 16th (AoE) November patch releases are targeted for November 10, with cherry-pick PRs due November 6. Featured PRs 143463: KEP-5710: Log error when PostFilter plugin does not implement PodGroupPostFilterPlugin jdzikowski added a scheduler initialization check for plugins that implement PostFilter but not PodGroupPostFilterPlugin, as part of KEP-5710. With the GenericWorkload feature gate enabled, the scheduler now logs an error when an enabled plugin will be skipped during scheduling for a PodGroup or CompositePodGroup. Previously, this behavior could go unnoticed, potentially leaving workload-aware scheduling integrations without the expected post-filtering behavior. The check also fulfills one of the KEP's graduation criteria for General Availability. 143387: KEP-5491: Adds integration and e2e test for List Type Attributes everpeace added integration and end-to-end tests for list-typed matchAttribute and distinctAttribute constraints in Dynamic Resource Allocation (DRA), as specified by KEP-5491. The tests exercise allocation against a real API server and scheduler, covering intersecting and disjoint attribute lists, scalar-to-list matching, and distinctness constraints. These cases help verify that device allocation behaves correctly across different attribute combinations and provide coverage required by the KEP's test plan for beta promotion in Kubernetes v1.38. KEP of the Week KEP-6268: Stall and Resume Slow Watch Streams A slow watch client can fill its kube-apiserver buffer, delaying other watchers before the server force-closes its stream. KEP-6268 instead marks that watcher unsynced, catches it up from shared watch-cache history, and resumes it on the same connection. If the needed events have aged out, the server sends 410 Gone. Clients and the API need no changes. @nami-ant and @rpb-ant authored the KEP for SIG API Machinery. They revised the design in September to share catch-up work across watchers, following etcd's model. The KEP merged October 1, while the kube-apiserver implementation remains under review behind the off-by-default WatchCacheStallResume gate. KEP-6268 is implementable and targets Alpha in v1.38; Beta and Stable milestones are TBD. Other Merges - Fix for a kubelet bug that set pod sandbox CPU and memory limits to the partial sum of container limits when only some containers declared them - kube-scheduler's preemption disruption metric gains a victim label; PreemptionVictim gains a Type() method and PreemptionCandidate.NumPodGroupDisruptions() is replaced by NumDisruptionsByVictimType() - MutatingAdmissionWebhook config gains a staticOrder field to control manifest vs. API-registered webhook ordering; requires the ManifestBasedAdmissionControlConfig gate and must be removed before downgrading to v1.37 or earlier - A new evictionrequest-controller creates Evictions from EvictionRequests and coordinates external responders - Add cgroup_mount_mode to the CRI API to support writable cgroups - kubectl selector commands like create pdb --selector now accept != - client-go adds DropManagedFields to omit managedFields from API responses, behind the new alpha ManagedFieldsOptOutClient gate - kubelet adds the alpha kubelet_device_plugin_devices metric, breaking device counts down by health state - Watch cache keeps the last snapshot behind the compaction window, avoiding etcd fallback for Exact LIST - Workload-aware scheduling: Pod scheduling status now names its parent PodGroup and PlacementFeasible counts are clearer, stricter CompositePodGroup validation, recursive status propagation, nodes with empty topology label values skipped as placement domains, multiple placement plugins intersected, and TopologyAwareWorkloadScheduling promoted to Beta (still disabled by default) Subprojects and Dependency Updates - etcd has released v3.7.3, v3.6.16, and v3.5.35, including multiple security patches, as well as fixing a few v3.7 regressions. - vertical-pod-autoscaler v1.8.1 and vertical-pod-autoscaler v1.7.3 were both released. They are built with Go 1.26.10, blocking the compromised eMudhra intermediate CAs. - etcd v3.7.3: New patch release, see CHANGELOG; also v3.6.16, v3.5.35 - coredns v1.15.0: Fixes CVE-2026-56865, CVE-2026-56864, CVE-2026-33818, CVE-2026-97032, CVE-2026-78659, CVE-2026-97031, and CVE-2026-111943; adds DNS-over-QUIC forwarding support, opt-in JSON query logging, and authenticated RFC 2136 dynamic DNS updates - prometheus v3.15.1: Builds with Go 1.27.3 and bumps golang.org/x/net to v0.60.0 to fix HTTP/2 vulnerabilities; also v3.13.5 - vsphere-csi-driver v3.8.0: Adds support for Kubernetes 1.36, upgrades csi-proxy to v2.0.0-alpha.2, and bumps Go dependencies to address CVEs Shoutouts - No shoutouts this week. Want to thank someone for special efforts to improve Kubernetes? Tag them in the #shoutouts channel. LWKD is a publication of Kubernetes Project Contributor Communications. Read it on the web, and contribute or send corrections to the GitHub repository. Powered by Buttondown, the easiest way to start and grow your newsletter. updated the repo creation guidelines: community#9232 WG governance document: community#9228 added notes on future conformance: community#9227 November 16th (AoE): github.com/kubernetes/sig-release/tree/master/rel… 143463: KEP-5710: Log error when PostFilter plugin does not implement PodGroupPostFilterPlugin: kubernetes#143463 jdzikowski: github.com/jdzikowski KEP-5710: enhancements#5710 143387: KEP-5491: Adds integration and e2e test for List Type Attributes: kubernetes#143387 everpeace: github.com/everpeace KEP-5491: enhancements#5491 KEP-6268: Stall and Resume Slow Watch Streams: github.com/kubernetes/enhancements/blob/master/ke… @nami-ant: github.com/nami-ant @rpb-ant: github.com/rpb-ant KEP merged October 1: enhancements#6269 kube-apiserver implementation: kubernetes#141450 Beta and Stable milestones are TBD: enhancements#6268 kubelet bug: kubernetes#141014 gains a victim label: kubernetes#142955 staticOrder field: kubernetes#143160 evictionrequest-controller: kubernetes#137133 cgroup_mount_mode to the CRI API: kubernetes#139824 create pdb --selector: kubernetes#142216 DropManagedFields: kubernetes#142695 kubelet_device_plugin_devices: kubernetes#143223 keeps the last snapshot behind the compaction window: kubernetes#143325 now names its parent PodGroup: kubernetes#143184 PlacementFeasible counts: kubernetes#143183 CompositePodGroup validation: kubernetes#142830 recursive status propagation: kubernetes#141860 empty topology label values skipped: kubernetes#140308 multiple placement plugins intersected: kubernetes#139565 promoted to Beta: kubernetes#140295 vertical-pod-autoscaler v1.8.1: github.com/kubernetes/autoscaler/releases/tag/ver… vertical-pod-autoscaler v1.7.3: github.com/kubernetes/autoscaler/releases/tag/ver… etcd v3.7.3: github.com/etcd-io/etcd/releases/tag/v3.7.3 v3.6.16: github.com/etcd-io/etcd/releases/tag/v3.6.16 v3.5.35: github.com/etcd-io/etcd/releases/tag/v3.5.35 coredns v1.15.0: github.com/coredns/coredns/releases/tag/v1.15.0 prometheus v3.15.1: github.com/prometheus/prometheus/releases/tag/v3.… v3.13.5: github.com/prometheus/prometheus/releases/tag/v3.… vsphere-csi-driver v3.8.0: github.com/kubernetes-sigs/vsphere-csi-driver/rel… to the GitHub repository: kubernetes-sigs/lwkd
kubecost 3.x を GKE に入れてみた — 入れる前に知っておきたかった 9 つのこと はじめに Kubernetes のクラスタは、誰がいくら使っているのかが見えにくいです。 クラウドの請求書には「ノード 3 台分」としか書かれておらず、その上で動くアプリごとの内訳は出てきません。 それを見えるようにするツールが kubecost です。検証用の GKE クラスタに入れてみました。 入れること自体は Helm で 1 コマンドですが、既定のまま入れると困る点がいくつもありました。 この記事は、実際に入れて確かめた「入れる前に知っておきたかったこと」のまとめです。 対象は、Kubernetes を触り始めて「requests って何となく書いている」くらいの方です。requestsは後述で説明します。 確かめた環境 項目 内容 クラスタ GKE（検証用）。手元の確認には minikube も使用 kubecost Helm chart kubecost/kubecost 3.2.4、無償版（Foundations） 見る方法 まずは kubectl port-forward。その後 Kong 越しの公開を準備 まず、何が見えるのか namespace ごとのコスト Monitor/Allocatiomsにnamespace（クラスタの中の区画）ごとに、CPU・メモリ・ディスク・ロードバランサーのコストが並びます。 注目してほしいのは idle の行です。これは「ノードとして借りているのに、どの Pod にも割り当てられていない分」です。 このクラスタでは合計 $16.83 のうち $8.71 が idle でした。半分以上が、誰も使っていない分の料金です。 何にお金がかかっているか Monitor/Assetsでクラウドの資源の内訳を見ることができます。ほとんどがノード代だと分かります。 どこを削れるか Savingsでは削減の候補が金額つきで並びます。一番大きいのは「Right-size container requests」、つまりコンテナの requests の見直しでした。 ! requests とは requests は「このコンテナはこれだけ CPU とメモリを使う予定です」という申告です。 Kubernetes はこの申告をもとに Pod をノードに詰めます。申告が実際より大きいと、使っていない分までノードを押さえてしまい、その分の料金を払うことになります。 コンテナごとに、今の申告（current）と、実際の使用量から計算した推奨値（rec）が出ます。 この画面は記事の最後でもう一度使います。 ! なお、入れてまだ 2 日目のスクリーンショットなので、数字は粗いです。 入れる前に知っておきたかった 9 つのこと 1. 2.x と 3.x は別物。検索で出てくる情報の多くは 2.x 最初につまずいたのがここです。 2.x 3.x Helm chart の名前 kubecost/cost-analyzer kubecost/kubecost Prometheus 必須 不要 データの置き場 Prometheus ClickHouse（同梱） 2.x の公式ドキュメントには、Prometheus は省略できず、無効にするとコストが 0 になる、と書かれています。 検証用クラスタにはPrometheusサーバがない（外部送信のみ）ため、同時構築が必要な2.xではなく3.xを選びました。 ! chart の名前もリポジトリの URL も違うので、2.x の記事の手順をそのまま 3.x に使うことはできません。 また、2.x の最新である 2.9 は 3.x への移行準備用の版で、新規に利用する場合は適していません（helm template がエラーで止まります）。 2. 既定のまま入れると、強い権限を持つ部品が一緒に入る chart を既定値で展開すると 8 つの部品が出てきます。そのうち 2 つは、入れる前に権限を確認した方がよいものでした。 部品 持っている権限 何のための部品か networkCosts ノードのネットワークを直接使う特権コンテナ。全 namespace の Secret を読める 通信量ごとのコスト clusterController クラスタの資源を作成・変更・削除できる 推奨値を自動で適用する Secret には、データベースのパスワードなどが入っています。コストを見るツールに、そこまでの権限を渡す必要はありませんでした。 この 2 つを無効にすると、残る権限は読み取りだけになります。 networkCosts: enabled: false clusterController: enabled: false ! 「無効にすると requests の推奨値が出なくなるのでは」と最初は思いましたが、出ました。 clusterController は推奨値を適用するための部品で、表示するだけなら要りません。 3. ディスクを既定で 161Gi 要求する。消しても残る デフォルトのままだと、永続ディスクを合計 161Gi を必要とします。検証用には大きすぎるので、8Gi まで削りました。 ! もう 1 つ、helm uninstall してもディスク（PVC）が消えません。消さずに残す印が chart 側で付いているためです。 試しに入れて消す、を繰り返すと、使われないディスクが課金されたまま残ります。消すときは PVC も手で消します。 kubectl get pvc -n kubecost 4. 既定では、利用状況を外部に送信する デフォルトでは、利用状況の送信とログの収集が有効です。社内のクラスタに入れるなら、先に決めておく項目です。 telemetry: enabled: false reporting: logCollection: false 無効にした状態でブラウザの通信を見たところ、画面からの通信はすべて kubecost 自身に向いていました。 5. Spot を使っているなら、ラベルを教えないと高く見積もられる Spot（クラウドの余った計算資源を安く借りる仕組み。GKE では Spot VM）のノードは、通常より大幅に安いです。 kubecost は「このノードは Spot か」をノードのラベルで判断します。どのラベルを見るかを教えないと、通常価格で計算され、コストがずっと高く出ます。 GKE の場合はこうです。 finopsagent: agent: kubecost: spotLabel: cloud.google.com/gke-spot spotLabelValue: "true" このラベルはクラウドによって違います。同じクラウドでも、ノードの増やし方（Karpenter を使うかどうか）で変わります。 6. 手元のクラスタで数字が出ても、正しいとは限らない minikube（手元の PC で動かす Kubernetes）でも、コストの数字は出ます。 ただし、これは実際の料金ではなく、kubecost が持っている仮の単価で計算したものです。 ! 「手元で数字が出た」は「動いた」の確認にはなりますが、「金額が合っている」かの確認にはなりません。 金額の確認は、実際のクラウドの上で、請求と見比べて行います。 7. 無償版でできないことは、API が教えてくれる 無償版（Foundations）には制限があります。どこまで使えるかは、実際に叩くのが確実でした。 やったこと 結果 30 日ぶんのデータを要求 拒否。上限は 15 日（360 時間） ResourceQuota の推奨値 拒否（有償版が必要） コンテナの requests の推奨値 使える kubecost 自身のログイン機能（OIDC） 使えない（有償版が必要） 最後の行が次の項目につながります。 8. アラートは「壊れたときだけ」届く kubecost 自身の状態を見張って Slack に知らせる機能（health アラート）があります。 設定に「1 時間ごと」とあったので、1 時間ごとに通知が来ると思っていましたが、来ませんでした。 1時間ごとに毎回通知が来るわけではなく、通知は『異常があった時だけ』です。 通知の経路が生きているかは、Alerts 画面の「Test Alert」ボタンで確かめます。 ! 「通知が来ない」を「壊れている」と読んでも、「正常」と読んでも間違えます。確かめる手段を別に持っておくのが大事でした。 9. 画面にログイン機能が無い。公開するなら手前で守る ! 無償版の kubecost には、ユーザーもロールもありません。URL に届いた人は全員、すべての画面と API を使えます。 port-forward で自分だけが見るなら問題ありませんが、チームで見るために公開するなら、手前に門番が要ります。 私たちは API ゲートウェイの Kong を手前に置き、4 種類の守りを重ねる設計にしました。(これから実装していく予定です！） 守り 見るもの IP 制限 どこから来たか ログイン（Keycloak） 誰か ロールの検査 管理者か 特定パスの拒否 入れた人にも見せないもの 公開前に確かめたところ、穴が 2 つ見つかりました。1 つ目（IP 制限が 3 回に 1 回外れる）は前の記事に書きました。ここでは 2 つ目を書きます。 拒否したはずのパスが %2F ですり抜けた kubecost の API の中に、Slack の webhook の URL を返すものがあります（/model/v3/alerts 以下）。 webhook の URL は、知っていれば誰でもそのチャンネルに投稿できる合鍵です。管理者にも見せる必要はないので、このパスだけ Kong で拒否しました。 rules: - matches: - path: {type: PathPrefix, value: /model/v3/alerts} 叩くと 403 が返りました。ここで終わりにするところでしたが、形を変えて試しました。 リクエストのパス 結果 /model/v3/alerts/global/slack 403 //model//v3/alerts/global/slack 403 /model/x/../v3/alerts/global/slack 403 /model%2Fv3/alerts/global/slack 200。webhook が返る /model/x%2F..%2Fv3/alerts/global/slack 200。webhook が返る %2F は / を別の書き方にしたもの（URL エンコード）です。 原因は、同じ文字列を 2 者が別の読み方をしていたことでした。 Kong は、照合の前にパスを整えます（// をまとめる、../ を畳む）。ただし %2F は / に戻しません。URL の決まりでは、エンコードされた / は「区切りではない、ただの文字」を意味するからで、Kong の動きは仕様どおりです kubecost の画面側の nginx は、%2F を / に戻してから中継先を決めます どちらも単体では間違っていません。検査する者と使う者で読み方がずれていたことが穴でした。 直し方は、%2F を含むパスをまとめて拒否する条件を足すことでした。 rules: - matches: - path: {type: PathPrefix, value: /model/v3/alerts} - path: {type: RegularExpression, value: "/.%2[Ff]."} これで全部 403 になり、通常の画面は壊れませんでした。ここまでは手元の minikube（Kong 3.9）で確かめた内容です。 ! 正常な形で 403 が出るのは「設定が書けた」の確認です。「塞いだ」の確認は、形を変えた入力でやります。 「ほかに漏れる API は無い」の確かめ方 webhook を返す API がほかに無いかも確かめました。kubecost のバイナリからパスらしい文字列を 32,329 本取り出し、全部叩きました。 このとき、本物の webhook は使いません。検査のログに本物の秘密が残るからです。 かといって何も設定せずに叩くと、0 件でも「漏れない」のか「返すものが無いだけ」なのか区別できません。 そこで、一目で偽物と分かる文字列を webhook として設定し、先に「漏れると分かっているパス」で検出できることを確かめてから総当たりしました。 CANARY=CANARYWEBHOOK0001 先に確認: 漏れると分かっているパスで見つからなければ、検査の方が壊れている curl -s "${BASE}/v3/alerts/global/slack" | grep -q "${CANARY}" \ || { echo "検査が機能していない" >&2; exit 1; } while read -r p; do curl -s "${BASE}${p}" | grep -q "${CANARY}" && echo "LEAK ${p}" done < candidates.txt 返したのは 2 本だけで、どちらも拒否済みのパスの下でした。 ! 副産物として、Helm の設定値を丸ごと返す API も見つかりました。 設定値に秘密を直接書いていたら、そこから読めていたことになります。秘密は設定値に書かず Secret で渡す、という決まりの理由が 1 つ増えました。 見えた。では、どう直すか 最後に、requests の推奨値の画面に戻ります。 このクラスタでは、申告が多すぎるものと、足りないものの両方がありました。 コンテナ メモリの申告（current） 推奨値（rec） 状態 データベース層のアプリ 4Gi 568Mi 多すぎる kubecost の収集役 512Mi 244Mi 多すぎる Kong 256Mi 1310Mi 足りない Keycloak 512Mi 819Mi 足りない 多すぎる申告はお金の無駄になり、足りない申告は、ノードが混んだときに Pod が追い出される原因になります。 コストを見るつもりで入れたツールが、安定性の問題も見つけてくれました。 ただし、kubecost が出すのは推奨値の表示までです。 そして、この数字を人が読んで設定ファイルを書き換える運用は、アプリが増えると続きません。使用量は日々変わるからです。 そこで次に試したいのが、Kubernetes の 2 つの自動調整の仕組みです。 仕組み 何を変えるか 向いている場面 VPA（Vertical Pod Autoscaler） 1 つの Pod の requests を、実際の使用量に合わせて上げ下げする 申告が実態とずれているとき HPA（Horizontal Pod Autoscaler） Pod の数を、負荷に合わせて増減する 時間帯で負荷が変わるとき VPA を入れれば、多すぎる申告が自動で下がり、その分ノードを減らせるかもしれません。足りない申告も自動で上がります。 HPA を入れれば、負荷の低い時間帯は Pod を減らしておけます。 どちらも、kubecost が見せてくれた「ズレ」を、人の手を借りずに直し続けてくれる仕組みです。 ! 本当にコストが下がるのかは、まだ試していません。 次は VPA と HPA を入れて、kubecost の数字が入れる前と後でどう変わるかを見てみるつもりです。 まとめ kubecost は 2.x と 3.x で構成が違う。Prometheus が無いクラスタなら 3.x 既定のまま入れない。権限の強い部品、大きなディスク、外部への送信を先に確認する Spot のラベルを教えないと、コストが高く出る 手元で数字が出ても、金額が正しい証拠にはならない 無償版にはログイン機能が無い。公開するなら手前で守り、形を変えた入力で確かめる 推奨値は表示まで。VPA と HPA を入れればコストも下げられるかもしれないので、次に試す 参考文献 Kubecost Core Architecture Overview（3.x） — 3.x の部品の構成 Kubecost Core Architecture Overview（2.x） — 2.x は Prometheus が必須であること kubecost/kubecost（Helm chart） — 3.x の chart と既定値 OpenCost — kubecost のコスト計算の元になっているオープンソース Resource Management for Pods and Containers（Kubernetes） — requests と limits の意味 HTTPRoute（Gateway API） — パスの照合方法（PathPrefix / RegularExpression） Route（Kong Gateway） — Kong のルートの照合とパスの扱い Request Termination プラグイン（Kong） — その場で決まった応答を返すプラグイン ngx_http_core_module の location（nginx） — 照合は %XX をデコードした後のパスで行われること RFC 3986 Uniform Resource Identifier — 予約文字と、エンコードされた予約文字の扱い（2.2 節、6.2.2 節） Autoscaling Workloads（Kubernetes） — VPA と HPA の全体像 Horizontal Pod Autoscaling（Kubernetes） Vertical Pod Autoscaler（kubernetes/autoscaler） Vertical Pod Autoscaling（GKE） 執筆者紹介 / 会社情報 執筆者 Higu BREXA SOLVIA SES アサイン先 株式会社Scalar 最近は Terraform・AWS・GCP・Kubernetes を勉強しています。わからないことだらけですが少しずつ整理しながら書いています！ 会社情報 Scalar, Inc は、「データマネジメントの未来を創る」を掲げ、データベースミドルウェアの研究開発・販売を行う日本発のグローバルスタートアップ企業です！複数の異なるデータベース（RDB、NoSQL など）を仮想的に統合し、トランザクション処理や分析クエリを実現するデータベースミドルウェアである ScalarDB、分散台帳技術（DLT）を用いたソフトウェアで、データの改ざん検知を行う ScalarDL を主な製品としています。是非下記の弊社サイトもチェック📝をお願いします！ 🔧 Product 🧑💻 開発者ブログ 🏢 コーポレートサイト
@trq212 so two things: 1) @bgrant0607, one of the OG kubernetes architects, wrote about drift/forking in K8s, and while code is different i think some concepts that went into the design of the kustomize CLI are at play here - github.com/kubernetes/design-proposals-archive/bl… 2) i think @0xblacklight 's work on a "design-iterated-loop" skill and his talk at AIE captures maybe the shape of "how can we drive agents to keep our fork up to date w/ upstream while preserving our changes" youtube.com (also a very k8s/control-loop-pilled approach)
klog, zap, logrus などで stderr に吐かれるログのレベルを Datadog で良い感じにする こんにちは、@key60228 です。 新しく立ち上げた GKE Cluster に Datadog Agent を入れてログを有効化してみたところ、とんでもない数のエラーログが吐かれていました。 まだ何も特別なことをしていないのに…？？？と不思議に思いログの詳細を見てみたところ、特に問題なさそうなログがほとんどでした。 結論としては、各コンポーネントが吐くログが Datadog が解釈できる形式で構造化されておらず、エラーレベルとして判定されてしまっていることが原因でした。 というわけでこの記事では各コンポーネント、ロガーがどんな構造で出力していて、どうすれば正しいログレベルで表示できるのかについてまとめていければと思います。 ソリューションを求めている方へ Datadog のログパイプラインで加工してあげてください。 Terraform pipelines.tf resource "datadog_logs_custom_pipeline" "container_log_level" { name = "container-log-level" description = "stderr 由来で error になったコンテナログの status を、本文の klog / zap / logfmt のレベルから付け直す。" is_enabled = true filter { query = "kube_cluster_name:*" } processor { grok_parser { name = "klog / zap / logfmt からレベルを取り出す" is_enabled = true source = "message" samples = [ "I0911 07:22:25.408586 1 cni_writer.go:930] Successfully synced \"gke-xxxxx-xxxxx-nod-xxxxxxxx-xxxx\"", # klog (GKE cni-writer) "E0911 07:10:07.225293 1 base_controller.go:161] \"controller failed to sync\" err=\"...\" key=\"image-sync\"", # klog (OCM work-agent) "2026-09-11T07:19:45Z\tINFO\tcontrollers.Spoke\trequeueing\t{\"request\": {\"name\":\"xxx\",\"namespace\":\"fleetconfig-system\"}, \"after\": \"1m0s\"}", # zap console、タブ区切り (fleetconfig-controller) "2026-09-11T04:39:22.159Z error logging/logging.go:16 NodeCollector: Failed to export metrics {\"kind\": \"receiver\"}", # zap console、スペース区切り (gke-metrics-agent) "time=2026-09-11T06:13:28Z level=info msg=Reconciling subsys=sfc reasons=[...]", # logfmt / logrus (cilium) ] grok { support_rules = "" match_rules = file("${path.module}/grok/container_log_level.grok") } } } processor { lookup_processor { name = "klog の severity 1 文字をレベル語に直す" is_enabled = true source = "klog.severity" target = "level" lookup_table = [ "I,info", "W,warning", "E,error", "F,critical", ] } } processor { lookup_processor { name = "zap のレベルを status remapper が解釈できる語に直す" is_enabled = true source = "zap.level" target = "level" lookup_table = [ "DEBUG,debug", "INFO,info", "WARN,warning", "ERROR,error", "DPANIC,critical", "PANIC,critical", "FATAL,critical", ] } } processor { lookup_processor { name = "logrus のレベルを status remapper が解釈できる語に直す" is_enabled = true source = "logfmt.level" target = "level" lookup_table = [ "trace,debug", "debug,debug", "info,info", "warning,warning", "error,error", "fatal,critical", "panic,critical", ] } } processor { status_remapper { name = "level を status にする" is_enabled = true sources = ["level"] } } } grok/container_log_level.grok klog %{regex("[IWEF]"):klog.severity}%{regex("\d{4} \d{2}:\d{2}:\d{2}\.\d+")}\s+%{integer:klog.thread_id}\s+%{regex("[^\]]+"):klog.location}]\s?%{data:klog.message} zap %{regex("\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(\.\d+)?(Z|[+-]\d{2}:\d{2})"):zap.timestamp}[\t ]+%{regex("(DEBUG|INFO|WARN|ERROR|DPANIC|PANIC|FATAL|debug|info|warn|error|dpanic|panic|fatal)"):zap.level:uppercase}[\t ]+%{data:zap.message} logfmt_with_time time=%{notSpace:logfmt.time}\s+level=%{word:logfmt.level:lowercase}\s+%{data:logfmt.message} logfmt level=%{word:logfmt.level:lowercase}\s+%{data:logfmt.message} Datadog Log Management のログレベルの仕組み Datadog のログでは status という予約済み属性が Log Explorer で表示されるログレベルに対応しています。1 Datadog Agent を使っている場合、この status は次の3つのタイミングで決まります。 Datadog Agent がログを送る時 Datadog が JSON ログをパースする時 ログパイプラインの Status Remapper を通る時 ① Datadog Agent がログを送る時 Kubernetes の Datadog Agent (各ノードの DaemonSet) では stdout に出たものを info、stderr に出たものを error と判定して Datadog に送っているようです。 github.com/DataDog/datadog-agent/blob/f85c4980d2b… ② Datadog が JSON ログをパースする時 次に、ログが JSON 形式であれば、Datadog は取り込み時の前処理で status、severity、level などの属性を探し、見つかった値で status を上書きします。2 {"level":"info","msg":"..."} のようなログは、stderr に書かれていても info として扱われます。 ③ ログパイプラインの Status Remapper を通る時 最後に、ログパイプライン に Status Remapper を置くと、任意の属性の値で status を上書きできます。 Status Remapper は指定された属性の値を先頭の文字列で以下の通り判定してくれます。3 値 status 0 から 7 の整数 Syslog の規格に従う emerg または f で始まる文字列 (大文字小文字を区別しない) emerg (0) a で始まる文字列 (大文字小文字を区別しない) alert (1) c で始まる文字列 (大文字小文字を区別しない) critical (2) err で始まる文字列 (大文字小文字を区別しない) error (3) w で始まる文字列 (大文字小文字を区別しない) warning (4) n で始まる文字列 (大文字小文字を区別しない) notice (5) i で始まる文字列 (大文字小文字を区別しない) info (6) d、t、v、trace、verbose で始まる文字列 (大文字小文字を区別しない) debug (7) o または s で始まる文字列、または OK か Success に一致する文字列 (大文字小文字を区別しない) OK 上記以外 info (6) つまり、プレーンテキストの本文にレベルを埋め込んでいるログは、②の前処理では拾われず、③のパイプラインを用意しない限り①のストリームだけで status が決まります。 今回の自分の環境のエラーログは、まさにこのパターンでした。 対策と各コンポーネント/ロガーの仕組み 根本的には各コンポーネントが JSON でログを吐くか、レベルに応じて stdout/stderr を使い分けていればこういった問題は起きません。 しかし GKE が管理するシステムコンポーネントは出力形式をこちら側では変えられず、OSS のコンポーネントも全てが出力形式を選べるわけではありません。 また、Datadog Agent でストリームと status の mapping を変更する方法も現時点ではなさそうです。 となるとメッセージ本文からレベルを取り出し、Status Remapper で status 属性を更新するログパイプラインを用意するのが良さそうです。 今回の自分の環境で問題になったログは、出力元のロガーで分けると klog、zap、logrus の3種類でした。 これらのロガーで、どのようにレベルを抽出すれば良いか見ていきます。 klog klog は glog の fork で、Kubernetes 本体も使用しているロガーです。 kubernetes/klog 今回の環境では、GKE のシステムコンポーネント (cni-writer、gke-metadata-server、proxy-agent etc.) や cert-manager などが klog かつ stderr でログを吐いていました。 klog の構造は以下のようになっており、先頭の L が severity を指しています。 Lmmdd hh:mm:ss.uuuuuu threadid file:line] msg github.com/kubernetes/klog/blob/ae3961e316a310e71… severity は I が INFO、W が WARNING、E が ERROR、F が FATAL を指します。 github.com/kubernetes/klog/blob/ae3961e316a310e71… 例えば proxy-agent の次のログは INFO ですが、stderr に書かれているため Datadog 上では ERROR になっていました。 そこで、まずは Grok Parser で属性をパースしてあげます。 klog %{regex("[IWEF]"):klog.severity}%{regex("\d{4} \d{2}:\d{2}:\d{2}\.\d+")}\s+%{integer:klog.thread_id}\s+%{regex("[^\]]+"):klog.location}]\s?%{data:klog.message} 上記ルールで IWEF の値を klog.severity に書き出しましたが、klog の場合はこの severity をそのまま Status Remapper に渡すことはできません。 I と W は期待通り info と warning になり、F は emergency になりますが、E は何にも当てはまらず、info になってしまいます。 そこで、Lookup Processor で klog.severity を調整してから level 属性に入れます。 (ついでに F も critical に調整しています) I,info W,warning E,error F,critical その後 Status Remapper を挟んであげることで、proxy-agent の INFO ログは info として表示されるようになりました。 zap zap は Uber が開発しているロガーで、Kubernetes 周りでは controller-runtime を使ったコントローラーの多くが zap を使っています。 uber-go/zap 今回の環境では、gke-metrics-agent や 別記事 で導入していた Open Cluster Management の fleetconfig-controller が zap でログを吐いていました。 zap には JSON で出力する JSON encoder と、人が読みやすいテキストで出力する console encoder があります。 github.com/uber-go/zap/blob/4892335e05f14bce8a98a… console encoder はデフォルトでは、タイムスタンプ、レベル、ロガー名、呼び出し元、メッセージ、追加フィールド (JSON) を区切り文字でつないで 1 行にします。 TIMESTAMP LEVEL LOGGER CALLER MESSAGE {FIELDS} zap の console encoder の出力は、次のルールで分解します。 zap %{regex("\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(\.\d+)?(Z|[+-]\d{2}:\d{2})"):zap.timestamp}[\t ]+%{regex("(DEBUG|INFO|WARN|ERROR|DPANIC|PANIC|FATAL|debug|info|warn|error|dpanic|panic|fatal)"):zap.level:uppercase}[\t ]+%{data:zap.message} 同じ console encoder でも、出力の細かいところはコンポーネントによって異なっているため、区切り文字はタブと半角スペース、レベルは大文字と小文字の両方に対応しています。 zap のレベルは DEBUG、INFO、WARN、ERROR、DPANIC、PANIC、FATAL とあり、今のままだと DPANIC が debug に、PANIC が info になってしまいます。 ついでに FATAL は klog に合わせて critical として扱うよう、Lookup Processor で調整してあげます。 github.com/uber-go/zap/blob/4892335e05f14bce8a98a… "DEBUG,debug", "INFO,info", "WARN,warning", "ERROR,error", "DPANIC,critical", "PANIC,critical", "FATAL,critical", なお、controller-runtime の zap のオプションをコマンドライン引数に結び付けているコントローラーであれば、--zap-encoder=json を渡して JSON encoder に切り替えられます。 JSON encoder の出力には level 属性が含まれるので、前処理で status に反映され、パイプラインは要りません。 logrus logrus もメジャーな構造化ロガーで、今回の環境では Zitadel が logrus でログを出していました。 sirupsen/logrus 出力形式は TextFormatter と JSONFormatter から選べ、TextFormatter は端末以外に出力するとき key=value を並べた logfmt 風の形式になります。 time="2026-10-13T05:58:18Z" level=INFO msg="structured logger configured" caller="..." error="..." レベルは level= の後ろに info、warning、error のような語で書かれます。 スクリーンショットのログは level=INFO ですが、Datadog 上では ERROR になっていました。 logrus の TextFormatter の出力は、次の 2 つのルールで分解します。 logfmt_with_time time=%{notSpace:logfmt.time}\s+level=%{word:logfmt.level:lowercase}\s+%{data:logfmt.message} logfmt level=%{word:logfmt.level:lowercase}\s+%{data:logfmt.message} TextFormatter は設定によってタイムスタンプを出さないことがあるため、time= で始まる行と level= で始まる行の両方に対応しています。 今回のログのように時刻を time="..." と引用符で囲む場合も、引用符ごと notSpace で logfmt.time に取り込まれるので問題ありません。 logrus のレベルは以下の通りなので、zap 同様に Lookup Processor で調整してあげましょう。 github.com/sirupsen/logrus/blob/6ef0748d1177dadae… "trace,debug", "debug,debug", "info,info", "warning,warning", "error,error", "fatal,critical", "panic,critical", これでログレベルが正しく表現されるようになりました。 ! この記事の本筋とズレるので本文では言及しませんでしたが、ZITADEL はオプションでログフォーマットを JSON 化できます。4 まとめ パイプラインを適用したあとは、status:error のログがほとんどなくなりました。 これで万事解決…とは思いつつ、klog、zap、logrus とは異なる出力形式を持つロガーを採用しているコンポーネントが出てきたらその都度対応しないといけない…？ 🤔 GKE とか各 KaaS のシステムコンポーネント以外は大概ログの format などをこっちで指定できるから特に問題にならないのかな〜〜とは思いつつ、こんな複雑なことをしないといけないのはちょっと不思議な気がするな〜〜 5 もっと良い方法あるよって方、もし居たら教えてください〜〜 最後に AI Shiftではエンジニアの採用に力を入れています！少しでも興味を持っていただけましたら、カジュアル面談でお話しませんか？（オンライン・19時以降の面談も可能です！） 【面談フォームはこちら】 hrmos.co 脚注 属性とエイリアス設定 - Datadog Docs ↩︎ Pipelines (Preprocessing) - Datadog Docs ↩︎ Log Status Remapper - Datadog Docs ↩︎ ZITADEL Production Setup - ZITADEL Docs ↩︎ 「こんな複雑なことをしないといけないのはおかしい」というソフトウェアエンジニアの勘 - valid,invalid ↩︎