Four Kubernetes node bugs I found by reading CI failures to the end
---
title: "Four Kubernetes node bugs I found by reading CI failures to the end"
published: true
description: "A flake nobody had root-caused, a test that hung nodes for hours, a 3.9 GB image that never finished pulling, and a kubelet memory regression. How each one went from a red CI run to a patch."
tags: kubernetes, go, sre, opensource
---
Kubernetes' node tests fail a lot. Most of the time someone re-runs them, they go green, and everyone moves on.
Over the past week I picked four failures in SIG Node territory and refused to move on — I kept reading the logs until the timeline explained the failure. Three were CI flakes. The fourth was a second, separate bug hiding inside a reported kubelet 1.37 memory regression. In every case, the bug wasn't where the error message pointed.
Status, to be upfront: as of October 10, 2026, all four of my fixes are open pull requests, triaged or awaiting review, not merged. This post is about the investigations — which is the part I'd want to read anyway.
How I read a failing CI run
Every Kubernetes CI run on Prow keeps its artifacts: the build log, the JUnit results, the kubelet and containerd logs from the node, and for node tests, the VM's serial console. TestGrid shows the history, so you can tell a one-off from a pattern.
My loop for each flake:
1. Find the pattern in TestGrid. Which job, how often, and — most useful — which similar jobs don't fail.
2. Build a timeline across logs. Test log, kubelet log and serial console on one clock, to the second.
3. Explain the difference. If only one job fails, something about that job's environment is the clue.
4. Reproduce before fixing. Either in a unit test that fails on master, or on a machine set up like the CI node.
Bug 1: the image record that kept disappearing
PR: kubernetes/kubernetes#142801 · fixes #140942
The symptom
Tests for KubeletEnsureSecretPulledImages — the feature that makes the kubelet re-check credentials before reusing a private image — kept failing in one job, ci-kubernetes-node-swap-ubuntu-serial: 3 of the last 10 runs, 6 of 20 over a longer window. A test helper waits up to five seconds for the kubelet to write an ImagePulledRecord for an image, and the record never appeared.
This had already been through two issues. One was closed by a PR that only added debug logging. The working theory in the open issue was that the kubelet doesn't write a fresh record when an image is already cached from an earlier test.
The timeline
From the failing run (prow job 2080994746846679040):
!Timeline of the failing run: at 14:49:41 the test pod pulls the pause image and the record is written; at 14:49:42 the test restarts the kubelet and image GC runs at 85% disk usage, exactly the high threshold, and the post-GC hook deletes the record; at 14:49:47 the test times out waiting for a record that no longer exists
The record was written. Then it was deleted one second later. The cached-image theory was wrong.
Why only this job
That job's image filesystem sat at about 85% for the whole run — exactly the kubelet's default high threshold for image garbage collection. Every kubelet restart in that job triggered a GC pass. Other jobs never got there.
The root cause
After a GC pass, the kubelet calls post-GC hooks with the list of images that "remain". The ImagePullManager's hook, PruneUnknownRecords, deletes every pulled-record for an image that isn't in that list.
The list was built inside freeSpace(), from the images it evaluated. But images in use by a running container, and pinned images, are filtered out before freeSpace() ever sees them. And freeSpace() stops early once it has freed enough bytes, so images after that point were never added either:
// imagesInEvictionOrder, simplified
for image, record := range im.imageRecords {
if isImageUsed(image, imagesInUse) {
continue // in-use images never reach freeSpace()...
}
if record.pinned {
continue // ...and neither do pinned ones
}
images = append(images, evictionInfo{id: image, imageRecord: *record})
}
So the list of "remaining" images was really "the eviction candidates GC looked at and kept". The image the test pod was running wasn't on it, and its record got pruned.
It fails safe — a later pod just has to re-verify credentials — but it's wrong, and it's invisible: the deletions weren't logged.
The fix
Pass the hooks every image the GC manager still tracks, not just the candidates it evaluated:
// remainingImages returns the IDs of all images still known after a GC pass.
// This must include in-use and pinned images, not just the eviction candidates
// that survived: post-GC hooks treat anything missing from the list as gone.
func (im *realImageGCManager) remainingImages() []string {
im.imageRecordsLock.Lock()
defer im.imageRecordsLock.Unlock()
// The same image can be tracked once per runtime handler.
imageIDs := sets.New[string]()
for imageKey := range im.imageRecords {
imageIDs.Insert(getImageIDFromTuple(imageKey))
}
return sets.List(imageIDs)
}
Both GarbageCollect and DeleteUnusedImages now call im.runPostGCHooks(ctx, im.remainingImages(), freeTime), and pruned records are logged at V(4). The new TestPostGCHooksReceiveAllRemainingImages fails on master — the hook receives an empty list — and passes with the change.
Status (Oct 10, 2026): open, awaiting triage and review from SIG Node.
Bug 2: the swap test that hung nodes for hours
PR: kubernetes/kubernetes#142850 · related to #138226
The symptom
A swap e2e test, "should be able to use more than the node memory capacity", sometimes passed, sometimes failed, and sometimes produced a run that ended in exit status 255 with no JUnit results at all — the node had stopped responding.
The numbers didn't add up
The test allocates memory equal to the node's full capacity — 8 GiB on the CI node, which also has 1 GiB of swap. The system needs some of that RAM too, so part of the pod's allocation can't fit in memory and has to go to swap. In my reproduction below, when the pod was allowed to finish, about 761 MiB of it ended up in swap.
Under LimitedSwap, a container's swap limit is proportional to its memory request:
> swap limit = (memory request ÷ node memory) × node swap
| Request | Swap limit | Against ~761 MiB of overflow |
| --- | ---: | :--- |
| 30% (before) | ~307 MiB | too small |
| 90% (after) | ~921 MiB | fits |
So in every run, the pod hit its swap limit with memory still left to place. What happened next varied — which is why the test flaked instead of failing every time:
- The OOM killer stepped in quickly. The run recovered, and sometimes the test passed.
- It didn't. The kernel tried to reclaim memory elsewhere. On this image system.slice and user.slice can't swap, so it thrashed the page cache instead — and the node hung for hours. Those are the exit status 255 runs with no JUnit results.
The serial console of one failed run showed exactly this: the pod pinned at its swap limit, with roughly 700 MB of swap still free on the node.
Reproducing it
On a VM set up like the CI node (8 GiB RAM, 1 GiB swap, Ubuntu 24.04, the same slice swap settings): at 30% the pod hit its swap limit and was OOM-killed. At 90% it finished allocating with about 761 MiB in swap, with no OOM and no stalls.
The fix
Three changes, all test-only:
1. Request 90% of memory, so the pod's swap limit can hold the overflow.
2. Lower the memory.available hard-eviction threshold to 100 Mi for these tests. Under node-level pressure, swap only starts a few seconds before the default 250 Mi threshold would evict the pod — before the test can see any swap usage.
3. Delete the pod with DeleteSync, so it's gone before the kubelet restarts in AfterEach. Nothing kills the pod while the kubelet is down.
Status (Oct 10, 2026): open, triaged as important-longterm, with the swap tests' original author asked to review.
Bug 3: the 3.9 GB image that never finished pulling
PR: kubernetes/kubernetes#142802 · related to #141200
The symptom
The GPU sanity test (nvidia-smi plus a CUDA demo) timed out after 30 minutes with the pod stuck in ErrImagePull. Even passing runs spent up to 24 minutes pulling the image.
The containerd log
cancel pulling image docker.io/nvidia/cuda:12.5.0-devel-ubuntu22.04 because of no progress in 5m0s
stop pulling image ...: active requests=1, bytes read=3918316892
failed to extract layer (... sha256:496b16dd...) to overlayfs as "extract-...": context canceled
The image is about 3.9 GB, and its largest layer is about 2.5 GB. The log shows 3.9 GB had been read before the pull was cancelled during layer extraction. So the expensive part wasn't the network — it was unpacking the big layer, which reported no "progress" to containerd's five-minute watchdog. The cancelled pull didn't reuse what it had already fetched, so each kubelet retry started the download again. That's containerd/containerd#13909, with a proposed fix upstream.
The fix
Use a smaller image. On amd64, the test only needs nvidia-smi (which comes from the driver) and the cuda-demo-suite-12-5 apt package, whose binaries link the CUDA runtime statically. The ~90 MB nvidia/cuda:12.5.0-base-ubuntu22.04 image has the same CUDA apt repository, keyring and environment. Other architectures build samples with nvcc, so they keep the devel image, and the pod is pinned to the GPU node's architecture so the image always matches the script's uname -m branch.
I left arm64 alone because I couldn't test it on arm64 GPU nodes. It can still hit the containerd behavior.
Status (Oct 10, 2026): open, triaged as important-longterm, awaiting /ok-to-test and review.
Bug 4: a second bug inside the 1.37 kubelet memory regression
PR: google/cadvisor#3946 · issue kubernetes/kubernetes#142874
This one isn't a flake. A SIG Node contributor reported 20–30% higher kubelet memory in Kubernetes 1.37, traced to cAdvisor — the library the kubelet uses to collect container stats — and wrote the main fix. My part is a separate, smaller bug I found while reproducing it. To keep the credit straight:
| | What it fixes | Author |
| --- | --- | --- |
| google/cadvisor#3944 | The reported cause: changing counters that stop housekeeping from ever backing off | the reporter |
| google/cadvisor#3946 | A back-off reset bug that makes the interval oscillate even when stats are unchanged | me |
The reported cause
cAdvisor polls each container's stats on a "housekeeping" interval. When stats haven't changed, it backs off — doubling the interval up to a maximum. When they change, it drops back to the baseline.
MemoryQoS, default-on in 1.37, sets memory.high on containers. The kernel then reclaims memory continuously, which keeps advancing counters in memory.stat. cAdvisor compared whole stats structs, so they never looked unchanged, and housekeeping never backed off. The reporter's fix, google/cadvisor#3944, excludes those counters from the comparison.
The second bug underneath
I reproduced the regression and confirmed #3944 fixes the counters. But while simulating it, I found the back-off had a bug of its own:
if stats[0].StatsEq(stats[1]) && (cd.housekeepingInterval < cd.maxHousekeepingInterval) {
cd.housekeepingInterval *= 2
// ... cap at max ...
} else if cd.housekeepingInterval != *HousekeepingInterval {
// Lower interval back to the baseline.
cd.housekeepingInterval = *HousekeepingInterval
}
Once the interval reaches the maximum, interval < max is false — so even with unchanged stats, execution falls into the else branch and resets to the baseline. Instead of staying backed off, the interval cycles: with cAdvisor's defaults, 1s → 2s → 4s → … → 1m → 1s; with the kubelet's settings, 10s → 15s → 10s → 15s.
!Chart of the housekeeping interval over successive ticks with unchanged stats: the original code climbs to the maximum and then drops back to the baseline in a repeating sawtooth, while the fixed code climbs to the maximum and stays there
The fix separates the two conditions, so only changed stats reset the interval:
if stats[0].StatsEq(stats[1]) {
if cd.housekeepingInterval < cd.maxHousekeepingInterval {
cd.housekeepingInterval *= 2
// ... cap at max ...
}
} else if cd.housekeepingInterval != *HousekeepingInterval {
cd.housekeepingInterval = *HousekeepingInterval
}
The new TestNextHousekeepingIntervalDynamic steps through unchanged and changed samples. Run against the original code, it fails exactly where the sawtooth starts:
!Terminal recording: the new test fails on the original cAdvisor code, expecting 4s but getting 1s at step 3 and 2s at step 4, then passes with the fix
In my simulation of a 10-minute run with the kubelet's settings and MemoryQoS-style advancing counters, a container got 60 housekeeping ticks on the current code, 48 with #3944, and 41 with both fixes. Most of the improvement comes from #3944; mine removes the oscillation that remains. Neither patch alone gets the kubelet to back off as intended, and the simulation measures housekeeping ticks, not the production memory figure in the report.
Status (Oct 10, 2026): #3946 is open; the issue is assigned to me, and the reporter called the patch a good one.
What the four had in common
Each bug lived at an edge that normal runs never reach:
- A list that silently meant "some of everything". "Remaining images" excluded in-use and pinned images, and nothing downstream knew.
- A number that couldn't work. A swap limit smaller than the overflow it had to hold.
- A timeout that measured the wrong thing. A "no progress" watchdog applied to an unpack that reports no progress — and the fetched data not reused.
- An else branch that fired at the boundary. Correct below the maximum, wrong exactly at it.
And each one was explained by a difference: the one job at 85% disk, the one test whose request didn't match its allocation, the one image that was 40 times bigger than it needed to be, the one comparison that never settled.
Flakes aren't noise. They're bugs that only show up under one environment's conditions. The job that fails is telling you which conditions.
What the evidence does and doesn't establish
| Evidence | Establishes | Does not establish |
| --- | --- | --- |
| Prow timeline for bug 1 | The record was written, then pruned by GC one second later, in that run | That every historical failure of these tests had this cause |
| TestPostGCHooksReceiveAllRemainingImages | The hook now receives in-use and pinned images | That the fix has been reviewed or accepted |
| VM reproduction for bug 2 | 30% fails and 90% holds on a CI-like machine | Behavior on every node image the swap tests run on |
| containerd log for bug 3 | 3.9 GB had been read before the pull was cancelled during layer extraction | That every layer had fully downloaded, or that the smaller image never hits the same timeout |
| cAdvisor test, failing on original and passing with the fix | The back-off reset bug exists and the change removes it | The production memory savings |
| 60 / 48 / 41 tick simulation | The relative effect of #3944 and #3946 in my model | The 20–30% memory figure, which comes from the reporter's environment |
Check each one yourself
Bug 1 — kubelet image GC. Check out the PR branch and run the new unit test. Without the fix, the hook receives an empty list:
go test ./pkg/kubelet/images/ -run TestPostGCHooksReceiveAllRemainingImages -v
Bug 2 — swap test. This is a node e2e test, so it needs a Linux node with swap enabled. The PR describes the VM I used (8 GiB RAM, 1 GiB swap, Ubuntu 24.04). From a Kubernetes checkout:
make test-e2e-node FOCUS="should be able to use more than the node memory capacity"
Bug 3 — GPU image. The full test needs a GPU node; in CI it's the pull-kubernetes-e2e-gce-device-plugin-gpu job. The size difference behind the fix is easy to see anywhere crictl runs:
crictl pull nvidia/cuda:12.5.0-devel-ubuntu22.04 # ~3.9 GB, ~2.5 GB largest layer
crictl pull nvidia/cuda:12.5.0-base-ubuntu22.04 # ~90 MB
The cancellation itself is containerd's image_pull_progress_timeout (five minutes by default) firing during extraction; containerd/containerd#13909 has the details.
Bug 4 — cAdvisor back-off.
git clone https://github.com/google/cadvisor && cd cadvisor/lib
# add TestNextHousekeepingIntervalDynamic from the PR, then:
go test ./manager/ -run TestNextHousekeepingIntervalDynamic -v
On the original code it fails at step 3; with the fix applied, it passes.
---
I'm Sachin, an SRE / platform engineer working on Kubernetes, observability and distributed systems — currently open to new roles. LinkedIn · GitHub
1 reaction
Kubernetes Memory on cgroup v2: Debugging OOM Kills and Evictions
A container is OOMKilled, but kubectl top never showed it reaching its limit. Or a Pod is evicted while your node dashboard still shows free memory. In both cases, the first question is which memory value the decision actually used.
This guide separates cgroup-limit OOM, node-side OOM and kubelet eviction, then shows how to inspect the evidence and review memory requests and limits. It assumes familiarity with Pod resources and basic PromQL.
TL;DR
- The working set is a heuristic, not a measurement of unreclaimable memory. It subtracts inactive_file from usage but still includes reclaimable active page cache and slab.
- Distinguish memcg OOM from global OOM using kernel logs. A container can be below its own limit when an ancestor cgroup reaches its limit or a node-side allocation fails. OOMKilled and SystemOOM alone cannot establish the cause.
- Eviction uses the kubelet's memory.available, not Linux MemAvailable. Check the kubelet's statistics and configured thresholds.
- Review requests and limits separately. Requests influence placement and victim selection. To address a cgroup limit, reduce usage or adjust the limit after checking node capacity.
- Check Memory QoS configuration as well as its feature gate. In v1.37, the gate defaults to enabled, but request-based protection and memory.high remain disabled by default.
This article targets cgroup v2, Kubernetes v1.37.1 and containerd. See [Components and versions](#components-and-versions) for exact versions. The kind examples use a v1.37.0 node image.
Use the [investigation workflow](#investigation-workflow) for an incident, the [metric map](#quick-reference-contradictions-explained-and-a-metric-map) to understand a dashboard, or [requests and limits](#what-to-do-about-it) for prevention. The [kind examples](#hands-on-verification-with-kind) explore reclaim and Memory QoS in more detail.
Identify the failure before choosing a metric
| Observation | Mechanism | Evidence to check |
|:---|:---|:---|
| Container termination reason is OOMKilled | The kernel killed a process; this alone does not distinguish the OOM source | Kernel logs, container and ancestor cgroup limits, node allocation constraints |
| Pod is Evicted with The node was low on resource: memory. | The kubelet responded to node memory pressure | Kubelet memory.available, configured thresholds and eviction message |
| Exit code is 137 without an OOM reason | The process received SIGKILL; the exit code alone does not establish OOM | Termination details, events and kernel logs |
memcg OOM means an allocation cannot fit within a cgroup limit after reclaim. Global OOM means a node-side allocation cannot be satisfied after reclaim, potentially within NUMA or cpuset constraints. A container below its own limit can be affected by either an ancestor limit or a node-side failure.
Investigating OOM and eviction
Investigation workflow
Start with the observed termination or eviction, then compare metrics from the same time. Scrapes can miss the peak; kernel logs are stronger evidence of the OOM source than a sampled usage graph.
1. When a Pod was OOMKilled
- Check whether container_memory_usage_bytes is close to the limit. memcg OOM is evaluated against this value
- Compare with container_memory_rss to tell whether anonymous pages are the main cause. An OOM can occur even when RSS is low
- If the gap between the working set or usage and RSS is large, page cache, shmem or kernel memory may be pushing usage up
- If OOMKilled occurs despite headroom under the limit, suspect global OOM. Correlate SystemOOM with kernel logs and inspect ancestor limits. A missing event does not establish memcg OOM
- If you see no spike in the metrics, suspect a surge shorter than the scrape interval. See [My working set is below the limit, but the container was OOMKilled](#my-working-set-is-below-the-limit-but-the-container-was-oomkilled) for what to check
2. When a Pod was evicted
- If the message is The node was low on resource: memory., node-wide memory pressure is the cause
- Use the [node-utilization query](#why-node-exporter-and-kubelet-usage-disagree) to check the kubelet-view node memory utilization. The node-exporter-based utilization alone is not enough to judge
- Check whether a Pod with a small (or unset) memory request relative to actual usage is on the same node. Compare each Pod's container_memory_working_set_bytes (cAdvisor) against kube_pod_resource_request{resource="memory"} (kube-scheduler). Note that Pods with large requests make the scheduler keep placement density low, so they are unlikely to be the cause of pressure
3. To prevent recurrence
- Set a memory request informed by workload history to avoid excessive placement density. PromQL for checking is collected in [Align memory request with actual usage](#align-memory-request-with-actual-usage)
- For containers where memcg OOM occurs, also review the limit
- Control placement with Pod anti-affinity or topology spread constraints so that Pods with large memory usage do not gather on the same node
Telling which path occurred
Using Kubernetes information
Start by narrowing down with what you can check using kubectl.
| Where to look | What it tells you |
|:---|:---|
| Last State in kubectl describe pod, or lastState.terminated in kubectl get pod -o yaml | That an OOM kill happened (reason: OOMKilled). It cannot distinguish the path |
| Events in kubectl describe node <node name> | Use SystemOOM as a clue and verify the source in kernel logs. If there is no record, that does not make it a memcg OOM; it just means you cannot tell |
There is no OOM-specific event on the Pod side. You only see things like BackOff accompanying the restart.
Check it as follows. SystemOOM is an event attached to the Node, and since Node is not a namespaced object, the event itself is recorded in the default namespace.
# 1. Find which node the OOMKilled Pod is on
kubectl get pod -n <namespace> <Pod name> -o wide
Output:
NAME READY STATUS RESTARTS AGE IP NODE
myapp-... 1/1 Running 5 (2m ago) 1h 10.0.0.1 node-1
# 2. Check whether that node's events contain SystemOOM
kubectl describe node node-1
Output:
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning SystemOOM 3m kubelet System OOM encountered, victim process: java, pid: 12345
# To list only events
kubectl get events -n default \
--field-selector reason=SystemOOM,involvedObject.name=node-1
If SystemOOM exists, inspect kernel messages from the same time: constraint, oom_memcg, task_memcg and the killed PID. A parsing failure can also produce this event. Match the event and kernel log to the container's termination time (lastState.terminated.finishedAt).
Not finding SystemOOM is not evidence of memcg OOM. The same state results from event TTL expiry, a kubelet restart, or a missed event.
The event retention period is set by kube-apiserver's --event-ttl, and the default is 1 hour (options.go). So an old SystemOOM may no longer be available through kubectl. If you forward events to an external logging platform, check there.
On the node, inspect the kernel log:
sudo journalctl -k -b -o short-iso \
| rg 'oom-kill:|Out of memory:|Memory cgroup out of memory:|Killed process'
CONSTRAINT_MEMCG and oom_memcg identify the limiting cgroup; task_memcg identifies the victim's cgroup. If these differ, inspect the Pod and ancestor limits. CONSTRAINT_CPUSET and CONSTRAINT_MEMORY_POLICY call for checking NUMA and cpuset restrictions. Align timestamps with the container's finishedAt.
If logs are unavailable, use the following as supporting evidence rather than proof:
- Whether container_memory_usage_bytes was close to the limit at that time (if so, memcg OOM is likely)
- Whether [the kubelet-view node memory utilization](#why-node-exporter-and-kubelet-usage-disagree) was high at that time (if so, global OOM is likely)
Why OOMKilled cannot distinguish the paths
When a task exits with code 137, the container runtime (containerd) checks oom_kill in the cgroup's memory.events, and if it is 1 or more, it updates the termination reason to OOMKilled (events.go). The check is oomMetricsEventOccurred() in the same file.
switch v := taskMetricsAny.(type) {
case *cg1.Metrics:
return v.GetMemoryOomControl().GetOomKill() > 0, nil
case *cg2.Metrics:
return v.GetMemoryEvents().GetOomKill() > 0, nil
The definition of oom_kill in memory.events is "The number of processes belonging to this cgroup killed by any kind of OOM killer". So a process chosen and killed by the kernel in a global OOM is counted too, and both paths produce OOMKilled.
{% details How the kubelet parser produces SystemOOM %}
When SystemOOM is recorded
SystemOOM is an event the kubelet records on the Node object. The kubelet parses OOM messages in the kernel log and records it only when the cgroup where the OOM happened is judged to be the root (/) (oom_watcher_linux.go).
for event := range outStream {
// Count every OOM kill per container to back the
// container_oom_events_total metric.
recordOOMKill(event.ContainerName)
if event.VictimContainerName == recordEventContainerName {
...
ow.recorder.WithLogger(logger).Eventf(ref, v1.EventTypeWarning, systemOOMEvent, "%s", eventMsg)
}
}
The usual association with global OOM comes from the combination of the kernel log format and the parser. For an OOM caused by a memcg limit, the kernel prints oom_memcg=<cgroup path>, whereas for a global OOM it prints ,global_oom instead (mem_cgroup_print_oom_context() in mm/memcontrol.c).
if (memcg) {
pr_cont(",oom_memcg=");
pr_cont_cgroup_path(memcg->css.cgroup);
} else
pr_cont(",global_oom");
The modern-format regular expression in the kubelet's forked parser matches the format containing oom_memcg= (oomparser_linux.go).
containerRegexp = regexp.MustCompile(`oom-kill:constraint=(.*),nodemask=(.*),cpuset=(.*),mems_allowed=(.*),oom_memcg=(.*),task_memcg=(.*),task=(.*),pid=(.*),uid=(.*)`)
So no cgroup can be extracted from a global OOM message, and the initial value / of OomInstance is left as is (StreamOoms in oomparser_linux.go).
oom := &OomInstance{
ContainerName: "/",
VictimContainerName: "/",
TimeOfDeath: msg.Timestamp,
}
In other words, the presence of SystemOOM is not "the result of directly determining the path" but "the result of whether the kernel log could be parsed". Still, in practice it is enough to understand that a recorded event is a reason to inspect node-side OOM, while neither presence nor absence establishes the source.
{% enddetails %}
How it appears in metrics
Metrics do not distinguish the path with a label. The kernel's __oom_kill_process() records the same event regardless of the path.
| Metric | Emitted by | memcg OOM | global OOM |
|:---|:---|:---|:---|
| node_vmstat_oom_kill | node-exporter (oom_kill in /proc/vmstat) | Increases | Increases |
| container_oom_events_total | kubelet (result of kernel log parsing) | The series of the container the killed process belonged to increases | The id="/" series increases, not a container series |
| oom_kill in cgroup v2 memory.events | (a cgroup file, not a metric; containerd reads it) | Increases | Increases |
Only container_oom_events_total behaves differently because, as described earlier, the parser cannot extract a cgroup in a global OOM and treats it as the root (/). This is only a result of whether the log could be parsed, so do not use it to determine the path; confirm the source from constraint and oom_memcg in the kernel log.
> Note: Although container_oom_events_total has container_ in its name, in v1.37 it is counted by the kubelet, not cAdvisor (recordOOMKill() in oom_counter.go). The meaning of the value does not change, but you have to look elsewhere when reading the code.
What to look at after narrowing down
- If you judge it to be a global OOM, inspect node-side memory pressure and any NUMA or cpuset constraints. Compare the memory requests and actual usage of the Pods placed on the same node, and check whether any Pod has too small a request
- If you judge it to be a memcg OOM, review that container's memory limit and its actual usage. If the container seems to have headroom under its limit, it may have reached the limit of the Pod as a whole or of /kubepods.slice
- If you cannot decide, keep both possibilities open and check both the container's limit and usage and the node's memory utilization
{% details Kernel OOM behavior and victim selection %}
The two OOM kill paths
There are two paths by which a container's process gets OOM killed. Both are SIGKILLs from the kernel's OOM killer, but the conditions under which they occur and the scope they affect differ. As [defined in the terminology notes](#how-this-article-names-the-two-oom-kill-paths), memcg OOM enforces a configured limit and global OOM protects the node, so their purposes differ at the root.
| Path | Condition | What gets killed | Node event |
|:---|:---|:---|:---|
| memcg OOM | memory.max exceeded for a container, Pod or /kubepods.slice | Processes inside the cgroup that hit its limit | Not recorded |
| global OOM | A node-side allocation cannot be satisfied after reclaim, possibly within NUMA or cpuset constraints | Chosen from eligible processes under the allocation constraints | SystemOOM is recorded when the kubelet parses the log and treats the triggering cgroup as / |
Even if your container is under its limit, it can be OOM killed because of another Pod on the same node. That is global OOM.
SystemOOM is a kubelet-generated clue to investigate, not a definitive classification of the OOM source. Events expire according to the API server's --event-ttl (one hour by default), so a missing event does not rule out global OOM.
OOM kill from exceeding a cgroup limit (memcg OOM)
resources.limits.memory is set as the container cgroup's memory.max. When a cgroup's memory.current reaches memory.max, this is what happens. As described above, limits are set not only on containers but also on Pods and /kubepods.slice, and the same flow applies whichever level reaches its limit.
1. The kernel tries to reclaim memory inside that cgroup. Page cache is basically reclaimed here.
2. If usage still does not fit under memory.max after reclaim, the cgroup's OOM killer SIGKILLs processes in that cgroup.
3. With cgroup v2, the kubelet defaults to singleProcessOOMKill: false and sets memory.oom.group=1 for the container. A memcg OOM therefore normally kills the container's processes together. With singleProcessOOMKill: true, a child process can be killed while the container keeps running. Container restarts follow restartPolicy. See the implementation.
The implementation is try_charge_memcg() in mm/memcontrol.c. It is a goto loop returning to the retry: label, and proceeds in this order.
| Step | Location | What happens |
|:---|:---|:---|
| 1. Try to charge | L2177-L2178 | If page_counter_try_charge() succeeds, the allocation completes and it returns |
| 2. Reclaim | L2211-L2216 | On failure it reclaims with try_to_free_mem_cgroup_pages(), and if mem_cgroup_margin() satisfies the request, goes back to 1 |
| 3. Count the retry | L2244-L2245 | If reclaim is not enough, it decrements nr_retries and goes back to 1. The initial value is MAX_RECLAIM_RETRIES (= 16; mm/internal.h) |
| 4. Invoke the OOM killer | L2259-L2264 | Only after the retries are used up does it call mem_cgroup_oom() |
MAX_RECLAIM_RETRIES is a retry budget, not a guarantee of exactly 16 reclaim attempts. Allocation flags and reclaim progress change the control flow. The check in step 2 looks like this in code.
nr_reclaimed = try_to_free_mem_cgroup_pages(mem_over_limit, nr_pages,
gfp_mask, reclaim_options, NULL);
psi_memstall_leave(&pflags);
if (mem_cgroup_margin(mem_over_limit) >= nr_pages)
goto retry;
The kernel documentation for memory.max says the same.
> Memory usage hard limit. ... If a cgroup's memory usage reaches this limit and can't be reduced, the OOM killer is invoked in the cgroup.
> Warning: memory.current reaching memory.max does not by itself cause an OOM. A cgroup sitting at its limit with page cache is a normal state. For the same reason, the working set reaching memory.max is not an OOM condition either. Only inactive_file is subtracted from the working set, and reclaimable memory remains in the rest:
>
> - active_file: demoted to inactive under pressure and then reclaimed
> - slab_reclaimable (dentry / inode caches): in memcg reclaim too, shrink_slab() is called after shrink_lruvec() (shrink_node_memcgs() in mm/vmscan.c)
>
> So there is no threshold of "OOM once either value reaches the limit". The actual condition is when the charge fails and reclaim cannot make room. What cannot be reclaimed is mainly anonymous pages (without swap) and shmem, so that is where to look when narrowing down.
That said, what the kernel compares against memory.max in a memcg OOM is the whole of memory.current, and it is not decided by any single metric. memory.current includes, besides anonymous pages, kernel memory, shmem (tmpfs and shared memory), and page cache that cannot be reclaimed, or cannot be reclaimed right away.
container_memory_rss is an indicator for determining whether anonymous pages are the main cause. If cgroup RSS is close to the limit, anonymous pages are a likely major contributor, but RSS being below the limit is no evidence that an OOM will not occur. When the gap between RSS and the working set is large, check container_memory_usage_bytes (the value of memory.current) together with the breakdown in memory.stat on the node: anon, file, shmem, slab and so on.
OOM kill from node memory exhaustion (global OOM)
When a node-side allocation cannot be satisfied after reclaim, the kernel evaluates eligible processes using oom_badness(). cpusets, NUMA policies and oom_score_adj=-1000 can restrict the candidates. A container need not exceed its own limit to be selected.
There is no guaranteed ordering between kubelet eviction and the kernel's OOM killer. The kubelet evicts Pods to protect the node, and its selection puts "Pods whose usage exceeds their memory request" first (rankMemoryPressure() in helpers.go).
// rankMemoryPressure orders the input pods for eviction in response to memory pressure.
// It ranks by whether or not the pod's usage exceeds its requests, then by priority, and
// finally by memory usage above requests.
func rankMemoryPressure(pods []*v1.Pod, stats statsFunc) {
orderedBy(exceedMemoryRequests(stats), priority, memory(stats)).Sort(pods)
}
A Pod with a zero memory request and positive usage belongs to the "exceeds request" group. Within that group, priority and then usage above request determine the order, so it is not necessarily evicted first. If eviction acts in time, the result is Evicted, not OOMKilled.
Global OOM happens first when eviction cannot reclaim memory as fast as memory grows. There are several reasons it cannot keep up.
| Factor | Value | Effect |
|:---|:---|:---|
| Evaluation interval | 10 seconds (evictionMonitoringPeriod) | Detection can be up to 10 seconds late |
| Pods evicted per evaluation | Only one ("we kill at most a single pod during each eviction interval" in eviction_manager.go) | At most one Pod is evicted per synchronization; notifications and cleanup waits mean this is not a fixed rate |
| Waiting for cleanup | Up to 30 seconds (podCleanupTimeout) | The next evaluation may be delayed further |
| Default threshold | memory.available<100Mi | Extremely thin on large nodes |
| How memory.available is computed | From the root cgroup's anon + file | Contains no kernel memory, so it [overestimates free memory](#values-the-kubelet-uses-for-eviction-decisions) |
The official Kubernetes documentation also says (Node out of memory behavior):
> If the node experiences an out of memory (OOM) event prior to the kubelet being able to reclaim memory, the node depends on the oom_killer to respond.
It is more likely when applications that allocate a lot of memory right at startup, or in response to a sudden surge in requests, share a node.
> Warning: Eviction protects the node; it does not protect individual Pods from OOM. Containers without limits have no memory.max backstop at all, so eviction is merely the last line of defense.
>
> Moreover, rankMemoryPressure() looks at Pod priority before usage size. If the offending Pod has high priority, low-priority Pods with small usage get evicted first, and global OOM can occur without the pressure being resolved. The proper countermeasure is to set limits and requests rather than relying on eviction.
Which process gets killed
The kernel evaluates eligible processes using resident pages, swap, page tables and oom_score_adj; the adjustment alone does not establish a fixed kill order. The kubelet sets oom_score_adj (a bonus added to oom_score) according to the container's QoS class, so the QoS class and the memory request setting directly become how likely a container is to be killed. You can check the QoS class with kubectl get pod <Pod name> -o jsonpath='{.status.qosClass}', and how it is determined is described in Pod Quality of Service Classes.
| Condition | oom_score_adj | Likelihood of being killed |
|:---|:---|:---|
| system-node-critical (evaluated before the QoS class) | -997 | Least likely |
| Guaranteed | -997 | Least likely |
| Burstable | 1000 - 1000 × memory request / node memory capacity (clamped to 3–999) | The smaller the request, the more likely |
| BestEffort | 1000 | Most likely |
The implementation is GetContainerOOMScoreAdjust() in Kubernetes' policy.go. system-node-critical is evaluated before the QoS class decision.
if types.IsNodeCriticalPod(pod) {
// Only node critical pod should be the last to get killed.
return guaranteedOOMScoreAdj
}
The formula above is the container-level baseline. Pod-level requests and restartable init containers can introduce adjustments. An omitted memory request can also default from the limit, so inspect the applied Pod spec.
The Burstable clamp is implemented with a lower bound of 1000 + guaranteedOOMScoreAdj (= 3) and an upper bound of besteffortOOMScoreAdj - 1 (= 999). A Pod whose request is set smaller than its actual usage is more likely to be targeted in both eviction and global OOM.
> Warning: PriorityClass works differently for eviction and for global OOM. It is a misreading to think that raising priority protects you from both.
>
> | PriorityClass | Eviction | global OOM (oom_score_adj) |
> |:---|:---|:---|
> | User defined (up to 1000000000) | The higher the value, the later it is evicted | Determined by QoS class and request |
> | system-cluster-critical | Exempt from node-pressure eviction | Determined by QoS class and request (not protected) |
> | system-node-critical | Exempt from node-pressure eviction | Fixed at -997 |
> | static / mirror Pod | Exempt from node-pressure eviction | Determined by QoS class and request |
>
> The three exempt rows identify critical Pods for which IsCriticalPod() is true; the kubelet checks static and mirror Pods as well as system-critical priority classes.
>
> What oom_score_adj protects is only system-node-critical, for which IsNodeCriticalPod() is true. If a system-cluster-critical Pod is BestEffort, its oom_score_adj is 1000, and it is more likely to be selected in a global OOM. "I raised the priority but it was still OOMKilled" comes from this path.
> Warning: Neither an OOM kill nor node-pressure eviction directly deletes the Pod API object. What happens next depends on restartPolicy.
>
> - Always or OnFailure: the container is recreated on the same node. The Pod stays on the same node, so if the cause is not resolved, OOM kills will happen again
> - Never: the container is not recreated and the Pod remains Failed. If a controller such as a Job creates a successor Pod, it may be placed on a different node
>
> If OOMKilled repeats even though there is headroom under the limit, suspect this path.
> Note: Because the OOM kill frees memory, the pressure may already be gone when the kubelet next evaluates free memory. In that case the node may get no MemoryPressure condition and no node.kubernetes.io/memory-pressure:NoSchedule taint, and scheduling of new Pods does not stop. Conversely, if the pressure continues, a later evaluation sets the condition and taint, and eviction also occurs. Note that it is decided by memory.available at evaluation time, not by whether an OOM kill happened.
{% enddetails %}
Checking with PSI (Pressure Stall Information)
This is an auxiliary indicator for the case where things are slow although usage has not reached the limit. It shows how long processes were actually stalled waiting for memory reclaim.
{% details How to read PSI and PromQL %}
You can obtain the time a container was stalled waiting for memory allocation as PSI. Even when usage has not reached the limit, you can detect being made to wait by memory reclaim.
| Metric | Emitted by | Description |
|:---|:---|:---|
| container_pressure_memory_waiting_seconds_total | cAdvisor (embedded in kubelet) | Time some processes were waiting for memory (total of some in memory.pressure) |
| container_pressure_memory_stalled_seconds_total | cAdvisor (embedded in kubelet) | Time all processes were stalled waiting for memory (total of full in memory.pressure) |
Note: These metrics are controlled by the KubeletPSI feature gate, which went GA in v1.36 and is enabled by default (it cannot be disabled). The kernel must support PSI, though, and the kubelet decides this by whether cpu.pressure exists at the cgroup root (cadvisor_linux.go). If the metrics do not appear, check whether the kernel was built with CONFIG_PSI=y and whether psi=0 is set as a kernel parameter.
memory.pressure also has avg10 / avg60 / avg300, "the average over the last N seconds (%)", but only the cumulative value (total) is exposed as a Prometheus metric. So use rate() to see a ratio.
The unit is seconds, so the result of rate() is "how many seconds of waiting per second", a ratio from 0 to 1. Multiply by 100 to get a percentage directly.
<!-- PromQL uses Python fencing for syntax highlighting only. -->
# Per Pod, the share of time (%) all processes were stalled waiting for memory
100 * max by (namespace, pod) (
rate(container_pressure_memory_stalled_seconds_total{container="", image="", pod!=""}[5m])
)
# Per Pod, the share of time (%) some processes were waiting for memory
100 * max by (namespace, pod) (
rate(container_pressure_memory_waiting_seconds_total{container="", image="", pod!=""}[5m])
)
When there is plenty of memory, neither increases, so rate() stays at 0. Under pressure, waiting (some) rises first, and if it gets worse stalled (full) follows. waiting counts even when only some processes waited, so to judge severity, use stalled. It is the time all processes in the container were stopped, so if it shows a value, throughput is directly affected.
To see memory pressure on the whole node, use the root cgroup series.
# Share of time (%) the whole node was completely stalled waiting for memory
100 * max by (node) (
rate(container_pressure_memory_stalled_seconds_total{id="/"}[5m])
* on(instance) group_left(node) kubelet_node_name
)
Containers that are struggling although they are below their limit can be found by combining PSI with utilization. container_spec_memory_limit_bytes is the container limit exposed by cAdvisor.
# Containers whose usage is under 80 % of the limit but that are stalled waiting for memory
(
100 * rate(container_pressure_memory_stalled_seconds_total{container!="", image!=""}[5m]) > 1
)
and
(
container_memory_usage_bytes{container!="", image!=""}
/ container_spec_memory_limit_bytes{container!="", image!=""} < 0.8
)
and keeps only series whose label sets match exactly on both sides. A container with no limit has container_spec_memory_limit_bytes of 0, so the division gives +Inf, which does not satisfy < 0.8, and it is excluded automatically.
Here is an example alert. It catches stalled that keeps occurring.
- alert: ContainerMemoryStalled
# Fires when the 5-minute average of fully stalled time is 5 % or more for 10 minutes
expr: |
100 * max by (namespace, pod) (
rate(container_pressure_memory_stalled_seconds_total{container="", image="", pod!=""}[5m])
) > 5
for: 10m
labels:
severity: warning
annotations:
summary: "A Pod is stalled waiting for memory allocation (add the namespace and pod labels with your own template)"
description: "It may be made to wait by reclaim even though it has not reached its memory limit. Check the gap between usage and RSS and the amount of page cache."
The right threshold of 5 % depends on the environment. First look at the distribution of values without an alert to understand the normal level, then decide.
For details on PSI see Understand Pressure Stall Information (PSI) Metrics.
{% enddetails %}
Quick reference: contradictions explained and a metric map
The working set, cgroup usage, process RSS and node availability answer different questions. Use this map to identify the value behind the dashboard before comparing it with a limit or threshold. For a non-root cgroup, working set is max(memory.current - inactive_file, 0). It still includes some reclaimable memory.
| Scope | Metric | Emitted by | Backing value in cgroup v2 / /proc | Page cache handling | Main use |
|:---|:---|:---|:---|:---|:---|
| Container | container_memory_working_set_bytes | cAdvisor (embedded in kubelet) | max(memory.current − inactive_file, 0) | Active pages count as in use | kubectl top pod, usage visualization |
| Container | container_memory_rss | cAdvisor (embedded in kubelet) | anon in memory.stat | Not included (anonymous pages only) | Understanding what the app allocated |
| Container | container_memory_usage_bytes | cAdvisor (embedded in kubelet) | memory.current | Everything included | What memcg OOM is evaluated against |
| Node | container_memory_working_set_bytes{id="/"} | cAdvisor (embedded in kubelet) | anon + file − inactive_file in the root cgroup's memory.stat (see below) | Active pages count as in use | Eviction decisions, kubectl top node |
| Node | node_memory_MemAvailable_bytes | node-exporter | MemAvailable in /proc/meminfo | Reclaimable pages count as free | Visualizing node memory utilization |
| Node | kube_node_status_capacity{resource="memory"} | kube-state-metrics | (status.capacity of the Node object) | (capacity, not usage) | Denominator when converting to utilization |
Looking at the same node, utilization computed from node_memory_MemAvailable_bytes does not match the utilization the kubelet uses for eviction decisions. With workloads that do a lot of file I/O or logging, the difference can reach several GiB.
Different emitters also mean different labels on the metrics. cAdvisor metrics carry no label identifying the node, so when you compute per-node utilization you need to align labels with kubelet_node_name, as shown later.
Kubernetes memory usage and cgroup v2
Why the numbers differ
A process RSS measurement includes its resident mappings, including shared and file-backed pages. Cgroup accounting charges memory to a group, while working set subtracts one category of file cache from usage. An application's heap metric covers a different subset again.
Reclaimability is a separate question: clean file cache can often be reclaimed, dirty pages may need writeback, and anonymous memory normally needs swap or explicit release by the process. Calling free does not necessarily return allocator-held memory to the OS.
These differences explain why heap size, process RSS, cgroup RSS and working set need not match. The following sections connect those values to cgroup v2 accounting.
Memory breakdown in cgroup v2
Container memory usage is managed in a per-container cgroup. In cgroup v2, memory.current is the current usage and memory.stat is its breakdown.
# Example values in a container's cgroup (checked on the node)
cat /sys/fs/cgroup/<container cgroup>/memory.current
Output:
5368709120
cat /sys/fs/cgroup/<container cgroup>/memory.stat
Output:
anon 2147483648 # anonymous pages (heap, stack: memory not backed by a file)
file 3087007744 # page cache (including tmpfs and shared memory)
shmem 268435456 # the part of file that is tmpfs or shared memory
...
inactive_anon 134217728 # anon LRU: not accessed for a while
active_anon 2013265920 # anon LRU: recently accessed
inactive_file 1073741824 # file LRU: not accessed for a while
active_file 1744830464 # file LRU: recently accessed
unevictable 268435456 # pages that cannot be reclaimed (here, the tmpfs part)
...
The following relationships hold in this example. Note that inactive_file + active_file does not equal file; it equals file - shmem. The next section explains why.
anon + file + kernel memory = 5368709120 (memory.current)
inactive_file + active_file = 2818572288 = file - shmem
inactive_anon + active_anon = 2147483648 = anon (shmem is not in here)
unevictable = 268435456 = shmem (tmpfs without swap)
The main fields of memory.stat are:
anon (anonymous pages)
- Memory not backed by a file. Regions the application allocated with malloc and similar belong here.
- Unlike page cache, the kernel cannot reclaim it independently of the process. With swap it can write pages out and free up space, but on nodes that do not use swap it is memory the kernel cannot reclaim under pressure. When swap is unavailable, the kernel sets scan_balance = SCAN_FILE and excludes the anon LRU from scanning (get_scan_count() in mm/vmscan.c).
- On the other hand, this value drops if the process itself returns memory to the OS with munmap or madvise(MADV_DONTNEED). Note that calling free does not reduce it unless the allocator or language runtime returns the memory to the OS.
file (page cache)
- Contents of files held in memory by the kernel during reads and writes. The application did not allocate it explicitly.
- When memory runs short the kernel discards (reclaims) it automatically, so it is basically reclaimable memory.
- Cache of files on disk is split into active_file (recently accessed) and inactive_file (not accessed for a while). Reclaim takes inactive_file first.
- active_file + inactive_file does not equal file. These are totals of the LRU lists, and as described below shmem is on the anon LRU rather than the file LRU. Besides shmem, pages that fall off the LRU, such as mlocked pages, also show up as a difference.
- It includes not only the cache of files on disk but also shmem (tmpfs and shared memory), because Linux treats these as page cache too. shmem is written out to swap rather than to a file, so on nodes without swap it cannot be reclaimed. The breakdown is visible as shmem in memory.stat.
> Warning: shmem is counted in file, but it is not on the file LRU. The kernel classifies swap-backed pages as anon, and tmpfs pages are among them. folio_is_file_lru() in mm_inline.h is defined as "0 if @folio is a normal anonymous folio, a tmpfs folio or otherwise ram or swap backed folio".
>
> The counters they are charged to are different, though. A tmpfs page is added to both NR_FILE_PAGES (= file) and NR_SHMEM (= shmem) (mm/shmem.c):
>
> ```c
> __lruvec_stat_mod_folio(folio, NR_FILE_PAGES, nr);
> __lruvec_stat_mod_folio(folio, NR_SHMEM, nr);
> ```
>
> Furthermore, for a Kubernetes emptyDir with medium: Memory, the kubelet mounts tmpfs with the noswap option when the node supports it (generateTmpfsMountOptions() in empty_dir.go). The kernel makes the inode of a noswap tmpfs unevictable (shmem_get_inode() in mm/shmem.c), so these pages sit on the unevictable LRU, not the anon LRU.
>
> ```c
> if (sbinfo->noswap)
> mapping_set_unevictable(inode->i_mapping);
> ```
>
> So shmem is accounted as follows:
>
> | mount | LRU | Where it shows up in memory.stat |
> |:---|:---|:---|
> | emptyDir with medium: Memory (noswap) | unevictable | unevictable |
> | Any other tmpfs / shared memory | anon LRU | inactive_anon / active_anon |
>
> In neither case does it enter inactive_file / active_file. That matters in two ways later:
>
> - inactive_file + active_file generally corresponds to file - shmem; unevictable pages and asynchronous statistics can cause differences
> - Only inactive_file is subtracted from the working set, so shmem stays in the working set in full
With workloads that write a lot of logs or read and write large files, the page cache can grow to several GiB even though the application did not intend that.
> Note: memory.current includes memory the kernel itself uses (slab, socket buffers and so on) in addition to anon and file. So adding anon and file alone does not give memory.current.
How cgroup v2 memory controls map to Pod settings
cgroup v2 memory control interface
cgroup v2 offers four levels of control over memory usage. memory.min and memory.low are protections ("do not reclaim up to here"), and memory.high and memory.max are limits ("do not allow use beyond here").
| File | Type | Behavior |
|:---|:---|:---|
| memory.min | Protection (hard) | Protects memory within the effective boundary determined by ancestors and usage. It does not preallocate physical memory |
| memory.low | Protection (soft) | Best-effort avoidance of reclaim up to this value. It is reclaimed if nothing else can be |
| memory.high | Limit (soft) | Exceeding it applies reclaim pressure and throttling. Crossing memory.high does not itself invoke the OOM killer; memory.max and global OOM still apply |
| memory.max | Limit (hard) | Exceeding it triggers reclaim, and if usage still does not fit, the cgroup's OOM killer runs |
Other files that report state:
- memory.current: current usage (what container_memory_usage_bytes is backed by)
- memory.stat: usage breakdown
- memory.events: counts of low, high, max, oom, oom_kill and oom_group_kill
- memory.swap.max: swap limit
- memory.pressure: PSI for waiting on memory allocation
For exact definitions of each file see the kernel's Memory Interface Files, and for how Kubernetes treats cgroup v2 see About cgroup v2.
How Pod settings are reflected
Pod settings are reflected into cgroup v2 as follows.
| Pod setting | Target in cgroup v2 |
|:---|:---|
| resources.limits.memory | memory.max |
| resources.requests.memory | Not reflected (with the default kubelet configuration in v1.37; you can reflect it by changing settings → [KEP-2570](#kep-2570-support-memory-qos-with-cgroups-v2)) |
| resources.limits.cpu | cpu.max |
| resources.requests.cpu | cpu.weight |
cgroup v2 is a directory tree. Its top is the root cgroup (/sys/fs/cgroup), where any process that is in no child cgroup belongs, so it contains not only Pods but every process on the node, such as the kubelet and containerd. Kubernetes creates the hierarchy below it: "all Pods on the node → Pod → container".
# With the systemd cgroup driver (the kubeadm default; names differ with the cgroupfs driver, see below)
/sys/fs/cgroup ← root cgroup (id="/")
│ contains every process on the node
│ the value used for eviction decisions
│
├── system.slice/ kubelet, containerd, sshd, etc.
│ ├── kubelet.service (processes that are not Pods)
│ └── containerd.service
│
└── kubepods.slice/ ← all Pods on the node
│ (id="/kubepods.slice")
├── kubepods-pod<UID>.slice/ ← Pod (Guaranteed)
│ ├── cri-containerd-<ID>.scope ← container
│ └── cri-containerd-<pause ID>.scope ← pause container
│
├── kubepods-burstable.slice/ (intermediate slice per QoS class)
│ └── kubepods-burstable-pod<UID>.slice/ ← Pod (Burstable)
│ └── cri-containerd-<ID>.scope ← container
│
└── kubepods-besteffort.slice/
└── kubepods-besteffort-pod<UID>.slice/ ← Pod (BestEffort)
└── cri-containerd-<ID>.scope ← container
When filtering metrics, the root cgroup is id="/", all Pods on the node are id="/kubepods.slice", a Pod is container="", image="", pod!="", and a container is container!="", image!="". What the kubelet uses for eviction decisions is the working set of the topmost root cgroup ([see below](#values-the-kubelet-uses-for-eviction-decisions)). That is a different level from memcg OOM caused by a container or Pod exceeding its limit.
Memory limits are set not only on containers but also on higher levels.
| Level | cgroup | Memory limit |
|:---|:---|:---|
| Whole node (root cgroup) | / | Not set (the root has no memory.max → [see below](#values-the-kubelet-uses-for-eviction-decisions)) |
| All Pods on the node | /kubepods.slice | Capacity - kube-reserved - system-reserved when Pod allocatable enforcement is enabled |
| Pod | Differs by QoS class (see below) | The Pod's effective limit, including Pod-level spec.resources.limits.memory when configured |
| Container | cri-containerd-<ID>.scope under the Pod's cgroup | That container's limits.memory |
The examples here use container-level resources and omit Pod-level resources. Without a Pod-level limit, the required container limits must be present for a finite Pod limit to be set. A Pod's effective limit is not simply the sum of its regular containers' limits.memory. If there are init containers, their maximum is also considered, and spec.overhead is added if set. For the calculation see Resource sharing within containers and Pod Overhead.
So even if a container has not reached its own limit, an OOM kill can occur when the Pod as a whole or /kubepods.slice hits its limit.
> Note: The paths above assume the kubelet uses the systemd cgroup driver. With the cgroupfs driver, paths look like /kubepods/burstable/pod<UID>/<container ID>, with no .slice or .scope and no cri-containerd- prefix on the container.
>
> The kubelet's own default is cgroupfs, but kubeadm sets systemd when the value is empty (defaults.go, kubelet.go). In v1.37, a supported runtime's CRI RuntimeConfig response takes precedence through driver auto-detection. Check the effective driver with:
>
> ```bash
> $ kubectl get --raw "/api/v1/nodes/<node name>/proxy/configz" \
| jq '.kubeletconfig.cgroupDriver'
> "systemd"
> ```
>
> The id label of cAdvisor metrics also changes with the cgroup driver, so when filtering in PromQL, check the value in your own cluster with this query:
>
> ```bash
> group by (id) (container_memory_working_set_bytes{id=~"/kubepods|/kubepods.slice"})
> ```
>
> A Pod's cgroup path depends on its QoS class. As the tree above shows, Burstable and BestEffort go through an intermediate slice per QoS class.
>
> In the <UID> part, systemd escaping replaces the hyphens of the Pod UID with underscores (example: 6983351e-4ede-... → pod6983351e_4ede_...). The UID of a static Pod (mirror Pod) is a hash with no hyphens, so it looks like pod1ff70b81885169da4e9e32c9530b77c9.slice.
>
> The QoS class and UID of a Pod can be read from the Pod resource, and once you have those two you can assemble the cgroup path.
>
> ```bash
> $ kubectl get pod -n <namespace> <Pod name> -o jsonpath='{.status.qosClass}{"\n"}{.metadata.uid}{"\n"}'
> Burstable
> 6983351e-4ede-4611-8bab-b986614af18d
>
> # To list them
> $ kubectl get pods -n <namespace> -o custom-columns='NAME:.metadata.name,QOS:.status.qosClass'
> ```
Container metrics
The cAdvisor built into the kubelet exposes the cgroup values above as Prometheus metrics. They are served from the kubelet's /metrics/cadvisor endpoint, and you do not need to run a separate cAdvisor Pod or DaemonSet.
| Metric | Emitted by | Backing value in cgroup v2 | Description |
|:---|:---|:---|:---|
| container_memory_usage_bytes | cAdvisor (embedded in kubelet) | memory.current | Usage including anonymous pages, page cache and kernel memory. The value memcg OOM is evaluated against |
| container_memory_working_set_bytes | cAdvisor (embedded in kubelet) | memory.current - inactive_file | Usage minus inactive_file, clamped to zero. A heuristic that still includes some reclaimable memory |
| container_memory_rss | cAdvisor (embedded in kubelet) | anon in memory.stat | Amount of anonymous pages. Unlike true RSS, it does not include mmaped file pages (see below) |
| container_memory_cache | cAdvisor (embedded in kubelet) | file in memory.stat | Page cache size. Includes shmem (tmpfs and shared memory) |
| container_memory_total_active_file_bytes | cAdvisor (embedded in kubelet) | active_file in memory.stat | Active page cache |
| container_memory_total_inactive_file_bytes | cAdvisor (embedded in kubelet) | inactive_file in memory.stat | Inactive page cache |
| container_memory_mapped_file | cAdvisor (embedded in kubelet) | file_mapped in memory.stat | Amount of mmaped files |
| `cont
2 comments
Mentions
Four Kubernetes node bugs I found by reading CI failures to the end --- title: "Four Kubernetes node bugs I found by reading CI failures to the end" published: true description: "A flake nobody had root-caused, a test that hung nodes for hours, a 3.9 GB image that never finished pulling, and a kubelet memory regression. How each one went from a red CI run to a patch." tags: kubernetes, go, sre, opensource --- Kubernetes' node tests fail a lot. Most of the time someone re-runs them, they go green, and everyone moves on. Over the past week I picked four failures in SIG Node territory and refused to move on — I kept reading the logs until the timeline explained the failure. Three were CI flakes. The fourth was a second, separate bug hiding inside a reported kubelet 1.37 memory regression. In every case, the bug wasn't where the error message pointed. Status, to be upfront: as of October 10, 2026, all four of my fixes are open pull requests, triaged or awaiting review, not merged. This post is about the investigations — which is the part I'd want to read anyway. How I read a failing CI run Every Kubernetes CI run on Prow keeps its artifacts: the build log, the JUnit results, the kubelet and containerd logs from the node, and for node tests, the VM's serial console. TestGrid shows the history, so you can tell a one-off from a pattern. My loop for each flake: 1. Find the pattern in TestGrid. Which job, how often, and — most useful — which similar jobs don't fail. 2. Build a timeline across logs. Test log, kubelet log and serial console on one clock, to the second. 3. Explain the difference. If only one job fails, something about that job's environment is the clue. 4. Reproduce before fixing. Either in a unit test that fails on master, or on a machine set up like the CI node. Bug 1: the image record that kept disappearing PR: kubernetes/kubernetes#142801 · fixes #140942 The symptom Tests for KubeletEnsureSecretPulledImages — the feature that makes the kubelet re-check credentials before reusing a private image — kept failing in one job, ci-kubernetes-node-swap-ubuntu-serial: 3 of the last 10 runs, 6 of 20 over a longer window. A test helper waits up to five seconds for the kubelet to write an ImagePulledRecord for an image, and the record never appeared. This had already been through two issues. One was closed by a PR that only added debug logging. The working theory in the open issue was that the kubelet doesn't write a fresh record when an image is already cached from an earlier test. The timeline From the failing run (prow job 2080994746846679040): !Timeline of the failing run: at 14:49:41 the test pod pulls the pause image and the record is written; at 14:49:42 the test restarts the kubelet and image GC runs at 85% disk usage, exactly the high threshold, and the post-GC hook deletes the record; at 14:49:47 the test times out waiting for a record that no longer exists The record was written. Then it was deleted one second later. The cached-image theory was wrong. Why only this job That job's image filesystem sat at about 85% for the whole run — exactly the kubelet's default high threshold for image garbage collection. Every kubelet restart in that job triggered a GC pass. Other jobs never got there. The root cause After a GC pass, the kubelet calls post-GC hooks with the list of images that "remain". The ImagePullManager's hook, PruneUnknownRecords, deletes every pulled-record for an image that isn't in that list. The list was built inside freeSpace(), from the images it evaluated. But images in use by a running container, and pinned images, are filtered out before freeSpace() ever sees them. And freeSpace() stops early once it has freed enough bytes, so images after that point were never added either: // imagesInEvictionOrder, simplified for image, record := range im.imageRecords { if isImageUsed(image, imagesInUse) { continue // in-use images never reach freeSpace()... } if record.pinned { continue // ...and neither do pinned ones } images = append(images, evictionInfo{id: image, imageRecord: *record}) } So the list of "remaining" images was really "the eviction candidates GC looked at and kept". The image the test pod was running wasn't on it, and its record got pruned. It fails safe — a later pod just has to re-verify credentials — but it's wrong, and it's invisible: the deletions weren't logged. The fix Pass the hooks every image the GC manager still tracks, not just the candidates it evaluated: // remainingImages returns the IDs of all images still known after a GC pass. // This must include in-use and pinned images, not just the eviction candidates // that survived: post-GC hooks treat anything missing from the list as gone. func (im *realImageGCManager) remainingImages() []string { im.imageRecordsLock.Lock() defer im.imageRecordsLock.Unlock() // The same image can be tracked once per runtime handler. imageIDs := sets.New[string]() for imageKey := range im.imageRecords { imageIDs.Insert(getImageIDFromTuple(imageKey)) } return sets.List(imageIDs) } Both GarbageCollect and DeleteUnusedImages now call im.runPostGCHooks(ctx, im.remainingImages(), freeTime), and pruned records are logged at V(4). The new TestPostGCHooksReceiveAllRemainingImages fails on master — the hook receives an empty list — and passes with the change. Status (Oct 10, 2026): open, awaiting triage and review from SIG Node. Bug 2: the swap test that hung nodes for hours PR: kubernetes/kubernetes#142850 · related to #138226 The symptom A swap e2e test, "should be able to use more than the node memory capacity", sometimes passed, sometimes failed, and sometimes produced a run that ended in exit status 255 with no JUnit results at all — the node had stopped responding. The numbers didn't add up The test allocates memory equal to the node's full capacity — 8 GiB on the CI node, which also has 1 GiB of swap. The system needs some of that RAM too, so part of the pod's allocation can't fit in memory and has to go to swap. In my reproduction below, when the pod was allowed to finish, about 761 MiB of it ended up in swap. Under LimitedSwap, a container's swap limit is proportional to its memory request: > swap limit = (memory request ÷ node memory) × node swap | Request | Swap limit | Against ~761 MiB of overflow | | --- | ---: | :--- | | 30% (before) | ~307 MiB | too small | | 90% (after) | ~921 MiB | fits | So in every run, the pod hit its swap limit with memory still left to place. What happened next varied — which is why the test flaked instead of failing every time: - The OOM killer stepped in quickly. The run recovered, and sometimes the test passed. - It didn't. The kernel tried to reclaim memory elsewhere. On this image system.slice and user.slice can't swap, so it thrashed the page cache instead — and the node hung for hours. Those are the exit status 255 runs with no JUnit results. The serial console of one failed run showed exactly this: the pod pinned at its swap limit, with roughly 700 MB of swap still free on the node. Reproducing it On a VM set up like the CI node (8 GiB RAM, 1 GiB swap, Ubuntu 24.04, the same slice swap settings): at 30% the pod hit its swap limit and was OOM-killed. At 90% it finished allocating with about 761 MiB in swap, with no OOM and no stalls. The fix Three changes, all test-only: 1. Request 90% of memory, so the pod's swap limit can hold the overflow. 2. Lower the memory.available hard-eviction threshold to 100 Mi for these tests. Under node-level pressure, swap only starts a few seconds before the default 250 Mi threshold would evict the pod — before the test can see any swap usage. 3. Delete the pod with DeleteSync, so it's gone before the kubelet restarts in AfterEach. Nothing kills the pod while the kubelet is down. Status (Oct 10, 2026): open, triaged as important-longterm, with the swap tests' original author asked to review. Bug 3: the 3.9 GB image that never finished pulling PR: kubernetes/kubernetes#142802 · related to #141200 The symptom The GPU sanity test (nvidia-smi plus a CUDA demo) timed out after 30 minutes with the pod stuck in ErrImagePull. Even passing runs spent up to 24 minutes pulling the image. The containerd log cancel pulling image docker.io/nvidia/cuda:12.5.0-devel-ubuntu22.04 because of no progress in 5m0s stop pulling image ...: active requests=1, bytes read=3918316892 failed to extract layer (... sha256:496b16dd...) to overlayfs as "extract-...": context canceled The image is about 3.9 GB, and its largest layer is about 2.5 GB. The log shows 3.9 GB had been read before the pull was cancelled during layer extraction. So the expensive part wasn't the network — it was unpacking the big layer, which reported no "progress" to containerd's five-minute watchdog. The cancelled pull didn't reuse what it had already fetched, so each kubelet retry started the download again. That's containerd/containerd#13909, with a proposed fix upstream. The fix Use a smaller image. On amd64, the test only needs nvidia-smi (which comes from the driver) and the cuda-demo-suite-12-5 apt package, whose binaries link the CUDA runtime statically. The ~90 MB nvidia/cuda:12.5.0-base-ubuntu22.04 image has the same CUDA apt repository, keyring and environment. Other architectures build samples with nvcc, so they keep the devel image, and the pod is pinned to the GPU node's architecture so the image always matches the script's uname -m branch. I left arm64 alone because I couldn't test it on arm64 GPU nodes. It can still hit the containerd behavior. Status (Oct 10, 2026): open, triaged as important-longterm, awaiting /ok-to-test and review. Bug 4: a second bug inside the 1.37 kubelet memory regression PR: google/cadvisor#3946 · issue kubernetes/kubernetes#142874 This one isn't a flake. A SIG Node contributor reported 20–30% higher kubelet memory in Kubernetes 1.37, traced to cAdvisor — the library the kubelet uses to collect container stats — and wrote the main fix. My part is a separate, smaller bug I found while reproducing it. To keep the credit straight: | | What it fixes | Author | | --- | --- | --- | | google/cadvisor#3944 | The reported cause: changing counters that stop housekeeping from ever backing off | the reporter | | google/cadvisor#3946 | A back-off reset bug that makes the interval oscillate even when stats are unchanged | me | The reported cause cAdvisor polls each container's stats on a "housekeeping" interval. When stats haven't changed, it backs off — doubling the interval up to a maximum. When they change, it drops back to the baseline. MemoryQoS, default-on in 1.37, sets memory.high on containers. The kernel then reclaims memory continuously, which keeps advancing counters in memory.stat. cAdvisor compared whole stats structs, so they never looked unchanged, and housekeeping never backed off. The reporter's fix, google/cadvisor#3944, excludes those counters from the comparison. The second bug underneath I reproduced the regression and confirmed #3944 fixes the counters. But while simulating it, I found the back-off had a bug of its own: if stats[0].StatsEq(stats[1]) && (cd.housekeepingInterval < cd.maxHousekeepingInterval) { cd.housekeepingInterval *= 2 // ... cap at max ... } else if cd.housekeepingInterval != *HousekeepingInterval { // Lower interval back to the baseline. cd.housekeepingInterval = *HousekeepingInterval } Once the interval reaches the maximum, interval < max is false — so even with unchanged stats, execution falls into the else branch and resets to the baseline. Instead of staying backed off, the interval cycles: with cAdvisor's defaults, 1s → 2s → 4s → … → 1m → 1s; with the kubelet's settings, 10s → 15s → 10s → 15s. !Chart of the housekeeping interval over successive ticks with unchanged stats: the original code climbs to the maximum and then drops back to the baseline in a repeating sawtooth, while the fixed code climbs to the maximum and stays there The fix separates the two conditions, so only changed stats reset the interval: if stats[0].StatsEq(stats[1]) { if cd.housekeepingInterval < cd.maxHousekeepingInterval { cd.housekeepingInterval *= 2 // ... cap at max ... } } else if cd.housekeepingInterval != *HousekeepingInterval { cd.housekeepingInterval = *HousekeepingInterval } The new TestNextHousekeepingIntervalDynamic steps through unchanged and changed samples. Run against the original code, it fails exactly where the sawtooth starts: !Terminal recording: the new test fails on the original cAdvisor code, expecting 4s but getting 1s at step 3 and 2s at step 4, then passes with the fix In my simulation of a 10-minute run with the kubelet's settings and MemoryQoS-style advancing counters, a container got 60 housekeeping ticks on the current code, 48 with #3944, and 41 with both fixes. Most of the improvement comes from #3944; mine removes the oscillation that remains. Neither patch alone gets the kubelet to back off as intended, and the simulation measures housekeeping ticks, not the production memory figure in the report. Status (Oct 10, 2026): #3946 is open; the issue is assigned to me, and the reporter called the patch a good one. What the four had in common Each bug lived at an edge that normal runs never reach: - A list that silently meant "some of everything". "Remaining images" excluded in-use and pinned images, and nothing downstream knew. - A number that couldn't work. A swap limit smaller than the overflow it had to hold. - A timeout that measured the wrong thing. A "no progress" watchdog applied to an unpack that reports no progress — and the fetched data not reused. - An else branch that fired at the boundary. Correct below the maximum, wrong exactly at it. And each one was explained by a difference: the one job at 85% disk, the one test whose request didn't match its allocation, the one image that was 40 times bigger than it needed to be, the one comparison that never settled. Flakes aren't noise. They're bugs that only show up under one environment's conditions. The job that fails is telling you which conditions. What the evidence does and doesn't establish | Evidence | Establishes | Does not establish | | --- | --- | --- | | Prow timeline for bug 1 | The record was written, then pruned by GC one second later, in that run | That every historical failure of these tests had this cause | | TestPostGCHooksReceiveAllRemainingImages | The hook now receives in-use and pinned images | That the fix has been reviewed or accepted | | VM reproduction for bug 2 | 30% fails and 90% holds on a CI-like machine | Behavior on every node image the swap tests run on | | containerd log for bug 3 | 3.9 GB had been read before the pull was cancelled during layer extraction | That every layer had fully downloaded, or that the smaller image never hits the same timeout | | cAdvisor test, failing on original and passing with the fix | The back-off reset bug exists and the change removes it | The production memory savings | | 60 / 48 / 41 tick simulation | The relative effect of #3944 and #3946 in my model | The 20–30% memory figure, which comes from the reporter's environment | Check each one yourself Bug 1 — kubelet image GC. Check out the PR branch and run the new unit test. Without the fix, the hook receives an empty list: go test ./pkg/kubelet/images/ -run TestPostGCHooksReceiveAllRemainingImages -v Bug 2 — swap test. This is a node e2e test, so it needs a Linux node with swap enabled. The PR describes the VM I used (8 GiB RAM, 1 GiB swap, Ubuntu 24.04). From a Kubernetes checkout: make test-e2e-node FOCUS="should be able to use more than the node memory capacity" Bug 3 — GPU image. The full test needs a GPU node; in CI it's the pull-kubernetes-e2e-gce-device-plugin-gpu job. The size difference behind the fix is easy to see anywhere crictl runs: crictl pull nvidia/cuda:12.5.0-devel-ubuntu22.04 # ~3.9 GB, ~2.5 GB largest layer crictl pull nvidia/cuda:12.5.0-base-ubuntu22.04 # ~90 MB The cancellation itself is containerd's image_pull_progress_timeout (five minutes by default) firing during extraction; containerd/containerd#13909 has the details. Bug 4 — cAdvisor back-off. git clone https://github.com/google/cadvisor && cd cadvisor/lib # add TestNextHousekeepingIntervalDynamic from the PR, then: go test ./manager/ -run TestNextHousekeepingIntervalDynamic -v On the original code it fails at step 3; with the fix applied, it passes. --- I'm Sachin, an SRE / platform engineer working on Kubernetes, observability and distributed systems — currently open to new roles. LinkedIn · GitHub
cAdvisor: 134,520 pages name it, 13,948 answer on 8080, and no fingerprint exists to check against cAdvisor: 134,520 pages name it, 13,948 answer on 8080, and no fingerprint exists to check against cAdvisor is the container resource monitor that ships next to Kubernetes and gets installed by hand everywhere else. Its README is short and direct: it runs "in the background" with a web interface on port 8080, and it exposes resource usage and performance characteristics for the containers on the host. The measurements Four queries on 2026-10-26, one result page each, totals only: - title="cAdvisor" returned 134,520 matching assets. - title=="cAdvisor" returned 0 matching assets. - app="cAdvisor" returned 0 matching assets. - title="cAdvisor" && port="8080" returned 13,948 matching assets. The pair of zeros is the finding that shapes everything else. cAdvisor is a service with no navigable product page: it answers an HTTP API and serves a small interface, and its title is not exactly the string a strict comparison expects. A fingerprint database has nothing to classify either, so both the strict title and the application query return nothing while a loose title match finds 134,520 assets. What the monitor knows cAdvisor's own documentation describes its data model: for each container it keeps resource isolation parameters, historical resource usage, histograms, and network statistics, exported per container and machine-wide. On a normal host that means container names, image names and digests, mounted volume paths, labels, and the process tree in whatever form it exposes. None of it is exotic, and all of it is inventory. A list of image names and volume paths on a host is exactly what someone needs in order to know where the data lives, and the labels usually carry the deployment's own naming scheme. There is a second detail in the runtime options: profiling is available through the web interface at host:port/debug/pprof/, controlled by a flag documented as --profiling=false. A deployment that leaves the default alone is a deployment whose debug endpoint is reachable wherever the monitor is. Debug endpoints have a long history of being more revealing than the service they sit beside. Why 13,948 is the number to use The 10 percent intersection matters for a practical reason. A loose title match can pick up documentation pages, dashboards that mention cAdvisor, and monitoring consoles that reference it. The port-scoped query removes most of those, because a documentation page that mentions cAdvisor is not usually served on port 8080 by the same host that runs the monitor. So 13,948 assets are the ones where a page named the product and the host answered on the port the product uses. That is a smaller and much more defensible population: hosts where a container monitor is listening on a conventional, well-known port. Whether those listeners are reachable beyond the scanner's vantage point is a separate question, and the index cannot answer it. What the number supports is prevalence and convention, not exploitability. The operational question cAdvisor is almost always installed for a legitimate reason: somebody needed container metrics and Prometheus was pointed at the endpoint. That makes it a component with an owner inside the platform team and no owner in the security review, which is the pattern behind most of the exposure in observability tooling. Four checks belong in the review. Does the listener bind to a private interface, or to all interfaces? A Prometheus scrape comes from one or two addresses, so an any-address binding buys nothing but exposure. Is there authentication in front of it? cAdvisor is generally deployed without its own authentication because it is considered internal. Internal means the binding, not the intent. Is the profiling endpoint enabled on an internal-only port or on the same listener as the metrics? The second arrangement puts a debug interface on the same path space as the scrape endpoint. Does the monitor see containers it should not? A host that runs both customer workloads and platform services gives one monitor visibility into both, and the metrics carry names. Detection and inventory The inventory record should say: host, binding address, port, whether profiling is enabled, and which scrape source is authorised. That is five fields, and it converts the asset from "some monitoring thing" into a component with a policy. For detection, the useful signals are a new client address hitting the metrics endpoint and a request to the debug path. Both are ordinary HTTP logs on the listener, and both are usually dropped because nobody thought the port was worth logging. Method and limits Queries ran on 2026-10-26 with sub_type=all and pagesize=1; only totals were reported. The zero application fingerprint is a statement about classification, not about absence, and this report treats it that way. No attempt was made to infer versions or authentication state from the index, because the index does not carry them. The one conclusion worth repeating is that an asset can be extremely common and completely unclassified. A tool without a title and without a fingerprint is invisible to a review that only asks what software has a signature. References - ZoomEye observation, title="cAdvisor", recorded 2026-10-26: zoomeye.ai - ZoomEye observation, title="cAdvisor" && port="8080": zoomeye.ai - cAdvisor source repository and README: google/cadvisor - cAdvisor runtime options, including the profiling flag and default port: github.com/google/cadvisor/blob/master/docs/runti…
K8s upgrade drops a path; fixed scripts read other values After a Kubernetes version upgrade replaces the node image, procedures and scripts that read container CPU and memory values from under /sys/fs/cgroup stop working. $ kubectl exec <pod> -- cat /sys/fs/cgroup/cpu/cpu.stat cat: can't open '/sys/fs/cgroup/cpu/cpu.stat': No such file or directory command terminated with exit code 1 Running ls inside the container shows no cpu/ and no memory/. cpu.stat and memory.stat, which used to live in separate directories, are listed directly. The node's /sys/fs/cgroup looks the same. $ kubectl exec <pod> -- ls /sys/fs/cgroup cgroup.controllers cgroup.events ... cpu.stat ... memory.current memory.events ... memory.stat ... The layout with one directory per controller is cgroup v1, and this single flat layout is cgroup v2. The switch from v1 to v2 happened when the node image changed. Removing cpu/ from the script's path opens a file with the same name. $ kubectl exec <pod> -- cat /sys/fs/cgroup/cpu.stat usage_usec 768869 user_usec 684571 system_usec 84297 nice_usec 0 core_sched.force_idle_usec 0 nr_periods 38 nr_throttled 37 throttled_usec 1592698 nr_bursts 0 burst_usec 0 The throttled_time line from the old cpu.stat is missing, and there is a throttled_usec line in its place. A script that collected throttled_time finds no matching line and returns an empty result without raising an error. Even once the files open, some values read the v1 way come out different. This article covers three of them: CPU throttle time, memory usage, and OOM count. It is written for people who read the files under /sys/fs/cgroup with their own scripts or runbooks. If you see these values through a monitoring agent such as cAdvisor, the agent handles v2 itself, and that case is outside the scope here. An explanation of v2 itself and the migration steps are on the Kubernetes blog. Three reads that change Throttle time throttled_time, missing from the cpu.stat above, became throttled_usec. It is the total time the cgroup was stopped for hitting its CPU limit, and its unit changed from nanoseconds to microseconds.[^cpu-stat] If you change only the key name and keep the /1e9 that converts nanoseconds to seconds, you get a value 1/1000 of the real one. [^cpu-stat]: The cpu.stat entry in the kernel's cgroup v2 documentation. $ kubectl exec <pod> -- awk '$1=="throttled_usec"{printf "%.6f\n", $2/1e9}' /sys/fs/cgroup/cpu.stat 0.001593 With /1e6 instead, it comes out to 1.592698 seconds. nr_throttled and nr_periods, which give the fraction of periods that hit the limit, keep the same names and meanings as in v1, so the division works unchanged. Memory usage In v1, memory usage was calculated with this formula. memory.usage_in_bytes − total_inactive_file (a line in memory.stat) When a process reads or writes a file, the kernel keeps the contents in memory as page cache, and the cgroup's usage includes it. The part of the page cache that hasn't been used recently is inactive_file, which the kernel reclaims as soon as memory runs short. Usage minus inactive_file is the part that can't be released right away: the working set. The kubelet exposes this value as container_memory_working_set_bytes, and the memory column of kubectl top pod shows the same value.[^working-set] [^working-set]: The memory entry in the Kubernetes Resource metrics pipeline documentation. In v2, the formula becomes:[^cadvisor] memory.current − inactive_file (a line in memory.stat) [^cadvisor]: cAdvisor's handler.go subtracts total_inactive_file for v1 and inactive_file for v2. The memory-available-cgroupv2.sh example in the Kubernetes documentation writes the v2 formula the same way. v2's memory.stat has no lines starting with total_. The aggregate lines from v1 appear in v2 under their names with total_ removed. $ kubectl exec <pod> -- cat /sys/fs/cgroup/memory.current 108158976 $ kubectl exec <pod> -- grep ^inactive_file /sys/fs/cgroup/memory.stat inactive_file 104906752 The difference, about 3 MiB, is the working set. The corrected value matches container_memory_working_set_bytes from the kubelet. $ kubectl get --raw "/api/v1/nodes/<node>/proxy/metrics/resource" | grep ^container_memory_working_set_bytes OOM count In v1, a container's OOM count came from oom_kill in memory.oom_control.[^v1-oom] In v2, the same oom_kill is in memory.events, a file that counts the memory events in that cgroup by type. However, oom_kill in the container's memory.events reads 0 even after the container has restarted due to OOM. The count is kept in the memory.events of the Pod's slice, one level above the container. The rest of this section covers why it reads 0, how to find the Pod's slice, and how to read the slice's memory.events, in that order. [^v1-oom]: The memory.oom_control entry in the kernel's cgroup v1 memory documentation. oom_kill is the number of processes in that cgroup killed by OOM. Why the container's oom_kill reads 0 In a Pod that restarted due to OOM, the container's memory.events reads like this. $ kubectl exec <pod> -- grep oom /sys/fs/cgroup/memory.events oom 0 oom_kill 0 oom_group_kill 0 On v2 nodes, the kubelet sets the container's memory.oom.group to 1.[^oom-group] When OOM occurs in this cgroup, the kernel kills every process in the container, the container is recreated along with its cgroup, and the counters start again from 0. [^oom-group]: In the kubelet's kuberuntime_container_linux.go, memory.oom.group is set to 1 on v2 nodes unless singleProcessOOMKill is true. This was added in kubernetes/kubernetes#117793. To check whether your nodes behave this way, see whether the container's /sys/fs/cgroup/memory.oom.group is 1. If the kubelet's singleProcessOOMKill is true, it stays 0. Finding the Pod's slice The Pod's slice is the cgroup that groups all of the Pod's containers, and its memory.events adds up the counts of the containers under it. The slice survives container restarts, so the count survives too. /sys/fs/cgroup/ └── kubepods.slice/ └── kubepods-burstable.slice/ └── kubepods-burstable-pod<uid>.slice/ ← Pod's slice └── cri-containerd-<container-id>.scope/ ← container's scope To locate the slice, search for the scope by container ID. The container ID is the part of the Pod's containerID after containerd://. When you enter the node with kubectl debug node, the node's files appear under /host. $ kubectl get pod <pod> -o jsonpath='{.status.containerStatuses[*].containerID}' containerd://<container-id> $ kubectl debug node/<node> -it --image=busybox # find /host/sys/fs/cgroup -type d -name 'cri-containerd-<container-id>.scope' The parent of the directory it finds is the Pod's slice. cri-containerd- is the prefix used when the runtime is containerd. If you can log in to the node as root, drop /host from the path. Reading the slice's memory.events In the slice's memory.events, oom_group_kill is the OOM count, and it matches restartCount. oom_group_kill is the number of times all processes in the cgroup were killed together because of memory.oom.group.[^memory-events] For a Pod with a restartCount of 3, it looks like this. [^memory-events]: The memory.events entry in the kernel's cgroup v2 documentation. # grep oom <slice>/memory.events oom 3 oom_kill 9 oom_group_kill 3 oom_kill goes up each time the kernel kills a process, so a single OOM raises it by more than one. In this Pod, three OOMs brought it to 9.
Kubernetes 更新でパスが消え、直すと別の値を読む Kubernetes のバージョンを上げてノードイメージが替わったあと、コンテナの CPU やメモリの値を /sys/fs/cgroup の下から取っていた手順やスクリプトが止まる。 $ kubectl exec <pod> -- cat /sys/fs/cgroup/cpu/cpu.stat cat: can't open '/sys/fs/cgroup/cpu/cpu.stat': No such file or directory command terminated with exit code 1 コンテナの中で ls を打つと、cpu/ も memory/ もなく、ディレクトリに分かれていたはずの cpu.stat や memory.stat が直に並んでいる。ノードの /sys/fs/cgroup でも同じだ。 $ kubectl exec <pod> -- ls /sys/fs/cgroup cgroup.controllers cgroup.events （略） cpu.stat （略） memory.current memory.events （略） memory.stat （略） コントローラごとにディレクトリが分かれていたのが cgroup v1、1つにまとまったこの並びが cgroup v2 だ。ノードイメージが替わったときに、v1 から v2 に切り替わっている。 スクリプトのパスから cpu/ を外すと、同じ名前のファイルが開く。 $ kubectl exec <pod> -- cat /sys/fs/cgroup/cpu.stat usage_usec 768869 user_usec 684571 system_usec 84297 nice_usec 0 core_sched.force_idle_usec 0 nr_periods 38 nr_throttled 37 throttled_usec 1592698 nr_bursts 0 burst_usec 0 これまでの cpu.stat にあった throttled_time の行がない。代わりに throttled_usec がある。throttled_time を集計していたスクリプトは、行が見つからないまま、エラーも出さずに空を返す。 ファイルが開くようになったあとも、v1 と同じ読み方で別の値が出るところがある。この記事では、そのうち CPU の throttle の時間、メモリの使用量、OOM の回数を扱う。 この記事は、/sys/fs/cgroup のファイルを自前のスクリプトや手順書で読んでいる人向けに書く。cAdvisor などの監視エージェント経由で見ているなら、エージェントが v2 を読み分けるので、この記事の範囲の外だ。v2 そのものの説明と移行の手順は Kubernetes 公式ブログにある。 読み方が変わる3か所 throttle の時間 冒頭の cpu.stat で消えていた throttled_time は、throttled_usec になった。CPU の上限に当たって止められていた時間の合計で、単位はナノ秒からマイクロ秒に変わっている1。キー名だけ書き換えて、ナノ秒を秒にする /1e9 を残すと、1000 分の 1 の値が出る。 $ kubectl exec <pod> -- awk '$1=="throttled_usec"{printf "%.6f\n", $2/1e9}' /sys/fs/cgroup/cpu.stat 0.001593 /1e6 に直すと 1.592698 秒になる。 上限に当たった周期の割合を出す nr_throttled と nr_periods は、名前も意味も v1 と同じで、割り算の式はそのまま使える。 メモリの使用量 メモリの使用量は、v1 ではこの式で出していた。 memory.usage_in_bytes − total_inactive_file（memory.stat の行） ファイルを読み書きすると、カーネルはその内容を page cache としてメモリに残す。cgroup の使用量にはこの分も入る。そのうち最近使われていない分が inactive_file で、メモリが足りなくなればすぐ回収される。使用量から inactive_file を差し引いた、すぐには手放せない分が working set だ。kubelet はこの値を container_memory_working_set_bytes として出していて、kubectl top pod のメモリの列もこの値だ2。 v2 の式はこうなる3。 memory.current − inactive_file（memory.stat の行） v2 の memory.stat には total_ で始まる行が1つもなく、v1 の集計の行は、v2 では total_ を外した名前になっている。 $ kubectl exec <pod> -- cat /sys/fs/cgroup/memory.current 108158976 $ kubectl exec <pod> -- grep ^inactive_file /sys/fs/cgroup/memory.stat inactive_file 104906752 引き算の結果の約 3 MiB が working set だ。 直した値は、kubelet が出す container_memory_working_set_bytes と同じになる。 $ kubectl get --raw "/api/v1/nodes/<node>/proxy/metrics/resource" | grep ^container_memory_working_set_bytes OOM の回数 v1 では、コンテナの OOM の回数を memory.oom_control の oom_kill で数えていた4。v2 では同じ oom_kill が memory.events にある。memory.events は、その cgroup でメモリについて起きた出来事を種類ごとに数えたファイルだ。ただ、コンテナの memory.events の oom_kill は、OOM で再起動したあとに読んでも 0 になる。回数が残るのは、コンテナの1つ上にある Pod の slice の memory.events だ。 以下では、0 になる理由、Pod の slice の探し方、slice の memory.events の読み方の順に書く。 コンテナの oom_kill が 0 になる理由 OOM で再起動した Pod で、コンテナの memory.events を読むとこうなる。 $ kubectl exec <pod> -- grep oom /sys/fs/cgroup/memory.events oom 0 oom_kill 0 oom_group_kill 0 v2 のノードでは、kubelet がコンテナの memory.oom.group を 1 にする5。この cgroup で OOM が起きると、カーネルはコンテナのプロセスを全部強制終了し、コンテナは cgroup ごと作り直されて、カウンタも 0 から数え直す。 自分のノードがこうなるかは、コンテナの /sys/fs/cgroup/memory.oom.group が 1 かどうかで分かる。kubelet の singleProcessOOMKill が true なら 0 のままだ。 Pod の slice を探す Pod の slice は、Pod のコンテナ全部をまとめた cgroup で、memory.events には配下のコンテナの分が合計される。コンテナが再起動しても slice は残るので、回数も残る。 /sys/fs/cgroup/ └── kubepods.slice/ └── kubepods-burstable.slice/ └── kubepods-burstable-pod<uid>.slice/ ← Pod の slice └── cri-containerd-<コンテナ ID>.scope/ ← コンテナの scope slice の場所は、コンテナ ID で scope を探して調べる。コンテナ ID は、Pod の containerID の containerd:// のあとの部分だ。kubectl debug node でノードに入ると、ノードのファイルは /host の下に見える。 $ kubectl get pod <pod> -o jsonpath='{.status.containerStatuses[*].containerID}' containerd://<コンテナ ID> $ kubectl debug node/<node> -it --image=busybox find /host/sys/fs/cgroup -type d -name 'cri-containerd-<コンテナ ID>.scope' 見つかったディレクトリの1つ上が Pod の slice だ。cri-containerd- はランタイムが containerd の場合の名前になる。ノードに root で入れるなら、パスの /host を外す。 slice の memory.events を読む slice の memory.events では、oom_group_kill が OOM の回数で、restartCount と一致する。oom_group_kill は、memory.oom.group によって cgroup のプロセスをまとめて強制終了した回数だ6。restartCount が 3 の Pod では、次のようになる。 grep oom <slice>/memory.events oom 3 oom_kill 9 oom_group_kill 3 oom_kill は、カーネルがプロセスを強制終了するたびに増えるので、1回の OOM で複数増える。この Pod では、3回の OOM で 9 になった。 脚注 カーネルの cgroup-v2 文書の cpu.stat の項。 ↩︎ Kubernetes の Resource metrics pipeline の memory の項。 ↩︎ cAdvisor の handler.go で、v1 は total_inactive_file、v2 は inactive_file を差し引いている。Kubernetes の文書にある memory-available-cgroupv2.sh も、v2 の式を同じ形で書いている。 ↩︎ カーネルの cgroup-v1 の memory 文書の memory.oom_control の項。oom_kill は、その cgroup で OOM により強制終了されたプロセスの数。 ↩︎ kubelet の kuberuntime_container_linux.go で、v2 のノードでは singleProcessOOMKill が true でない限り memory.oom.group を 1 にしている。この処理は kubernetes/kubernetes#117793 で入った。 ↩︎ カーネルの cgroup-v2 文書の memory.events の項。 ↩︎
Kubernetes Memory on cgroup v2: Debugging OOM Kills and Evictions A container is OOMKilled, but kubectl top never showed it reaching its limit. Or a Pod is evicted while your node dashboard still shows free memory. In both cases, the first question is which memory value the decision actually used. This guide separates cgroup-limit OOM, node-side OOM and kubelet eviction, then shows how to inspect the evidence and review memory requests and limits. It assumes familiarity with Pod resources and basic PromQL. TL;DR - The working set is a heuristic, not a measurement of unreclaimable memory. It subtracts inactive_file from usage but still includes reclaimable active page cache and slab. - Distinguish memcg OOM from global OOM using kernel logs. A container can be below its own limit when an ancestor cgroup reaches its limit or a node-side allocation fails. OOMKilled and SystemOOM alone cannot establish the cause. - Eviction uses the kubelet's memory.available, not Linux MemAvailable. Check the kubelet's statistics and configured thresholds. - Review requests and limits separately. Requests influence placement and victim selection. To address a cgroup limit, reduce usage or adjust the limit after checking node capacity. - Check Memory QoS configuration as well as its feature gate. In v1.37, the gate defaults to enabled, but request-based protection and memory.high remain disabled by default. This article targets cgroup v2, Kubernetes v1.37.1 and containerd. See [Components and versions](#components-and-versions) for exact versions. The kind examples use a v1.37.0 node image. Use the [investigation workflow](#investigation-workflow) for an incident, the [metric map](#quick-reference-contradictions-explained-and-a-metric-map) to understand a dashboard, or [requests and limits](#what-to-do-about-it) for prevention. The [kind examples](#hands-on-verification-with-kind) explore reclaim and Memory QoS in more detail. Identify the failure before choosing a metric | Observation | Mechanism | Evidence to check | |:---|:---|:---| | Container termination reason is OOMKilled | The kernel killed a process; this alone does not distinguish the OOM source | Kernel logs, container and ancestor cgroup limits, node allocation constraints | | Pod is Evicted with The node was low on resource: memory. | The kubelet responded to node memory pressure | Kubelet memory.available, configured thresholds and eviction message | | Exit code is 137 without an OOM reason | The process received SIGKILL; the exit code alone does not establish OOM | Termination details, events and kernel logs | memcg OOM means an allocation cannot fit within a cgroup limit after reclaim. Global OOM means a node-side allocation cannot be satisfied after reclaim, potentially within NUMA or cpuset constraints. A container below its own limit can be affected by either an ancestor limit or a node-side failure. Investigating OOM and eviction Investigation workflow Start with the observed termination or eviction, then compare metrics from the same time. Scrapes can miss the peak; kernel logs are stronger evidence of the OOM source than a sampled usage graph. 1. When a Pod was OOMKilled - Check whether container_memory_usage_bytes is close to the limit. memcg OOM is evaluated against this value - Compare with container_memory_rss to tell whether anonymous pages are the main cause. An OOM can occur even when RSS is low - If the gap between the working set or usage and RSS is large, page cache, shmem or kernel memory may be pushing usage up - If OOMKilled occurs despite headroom under the limit, suspect global OOM. Correlate SystemOOM with kernel logs and inspect ancestor limits. A missing event does not establish memcg OOM - If you see no spike in the metrics, suspect a surge shorter than the scrape interval. See [My working set is below the limit, but the container was OOMKilled](#my-working-set-is-below-the-limit-but-the-container-was-oomkilled) for what to check 2. When a Pod was evicted - If the message is The node was low on resource: memory., node-wide memory pressure is the cause - Use the [node-utilization query](#why-node-exporter-and-kubelet-usage-disagree) to check the kubelet-view node memory utilization. The node-exporter-based utilization alone is not enough to judge - Check whether a Pod with a small (or unset) memory request relative to actual usage is on the same node. Compare each Pod's container_memory_working_set_bytes (cAdvisor) against kube_pod_resource_request{resource="memory"} (kube-scheduler). Note that Pods with large requests make the scheduler keep placement density low, so they are unlikely to be the cause of pressure 3. To prevent recurrence - Set a memory request informed by workload history to avoid excessive placement density. PromQL for checking is collected in [Align memory request with actual usage](#align-memory-request-with-actual-usage) - For containers where memcg OOM occurs, also review the limit - Control placement with Pod anti-affinity or topology spread constraints so that Pods with large memory usage do not gather on the same node Telling which path occurred Using Kubernetes information Start by narrowing down with what you can check using kubectl. | Where to look | What it tells you | |:---|:---| | Last State in kubectl describe pod, or lastState.terminated in kubectl get pod -o yaml | That an OOM kill happened (reason: OOMKilled). It cannot distinguish the path | | Events in kubectl describe node <node name> | Use SystemOOM as a clue and verify the source in kernel logs. If there is no record, that does not make it a memcg OOM; it just means you cannot tell | There is no OOM-specific event on the Pod side. You only see things like BackOff accompanying the restart. Check it as follows. SystemOOM is an event attached to the Node, and since Node is not a namespaced object, the event itself is recorded in the default namespace. # 1. Find which node the OOMKilled Pod is on kubectl get pod -n <namespace> <Pod name> -o wide Output: NAME READY STATUS RESTARTS AGE IP NODE myapp-... 1/1 Running 5 (2m ago) 1h 10.0.0.1 node-1 # 2. Check whether that node's events contain SystemOOM kubectl describe node node-1 Output: ... Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning SystemOOM 3m kubelet System OOM encountered, victim process: java, pid: 12345 # To list only events kubectl get events -n default \ --field-selector reason=SystemOOM,involvedObject.name=node-1 If SystemOOM exists, inspect kernel messages from the same time: constraint, oom_memcg, task_memcg and the killed PID. A parsing failure can also produce this event. Match the event and kernel log to the container's termination time (lastState.terminated.finishedAt). Not finding SystemOOM is not evidence of memcg OOM. The same state results from event TTL expiry, a kubelet restart, or a missed event. The event retention period is set by kube-apiserver's --event-ttl, and the default is 1 hour (options.go). So an old SystemOOM may no longer be available through kubectl. If you forward events to an external logging platform, check there. On the node, inspect the kernel log: sudo journalctl -k -b -o short-iso \ | rg 'oom-kill:|Out of memory:|Memory cgroup out of memory:|Killed process' CONSTRAINT_MEMCG and oom_memcg identify the limiting cgroup; task_memcg identifies the victim's cgroup. If these differ, inspect the Pod and ancestor limits. CONSTRAINT_CPUSET and CONSTRAINT_MEMORY_POLICY call for checking NUMA and cpuset restrictions. Align timestamps with the container's finishedAt. If logs are unavailable, use the following as supporting evidence rather than proof: - Whether container_memory_usage_bytes was close to the limit at that time (if so, memcg OOM is likely) - Whether [the kubelet-view node memory utilization](#why-node-exporter-and-kubelet-usage-disagree) was high at that time (if so, global OOM is likely) Why OOMKilled cannot distinguish the paths When a task exits with code 137, the container runtime (containerd) checks oom_kill in the cgroup's memory.events, and if it is 1 or more, it updates the termination reason to OOMKilled (events.go). The check is oomMetricsEventOccurred() in the same file. switch v := taskMetricsAny.(type) { case *cg1.Metrics: return v.GetMemoryOomControl().GetOomKill() > 0, nil case *cg2.Metrics: return v.GetMemoryEvents().GetOomKill() > 0, nil The definition of oom_kill in memory.events is "The number of processes belonging to this cgroup killed by any kind of OOM killer". So a process chosen and killed by the kernel in a global OOM is counted too, and both paths produce OOMKilled. {% details How the kubelet parser produces SystemOOM %} When SystemOOM is recorded SystemOOM is an event the kubelet records on the Node object. The kubelet parses OOM messages in the kernel log and records it only when the cgroup where the OOM happened is judged to be the root (/) (oom_watcher_linux.go). for event := range outStream { // Count every OOM kill per container to back the // container_oom_events_total metric. recordOOMKill(event.ContainerName) if event.VictimContainerName == recordEventContainerName { ... ow.recorder.WithLogger(logger).Eventf(ref, v1.EventTypeWarning, systemOOMEvent, "%s", eventMsg) } } The usual association with global OOM comes from the combination of the kernel log format and the parser. For an OOM caused by a memcg limit, the kernel prints oom_memcg=<cgroup path>, whereas for a global OOM it prints ,global_oom instead (mem_cgroup_print_oom_context() in mm/memcontrol.c). if (memcg) { pr_cont(",oom_memcg="); pr_cont_cgroup_path(memcg->css.cgroup); } else pr_cont(",global_oom"); The modern-format regular expression in the kubelet's forked parser matches the format containing oom_memcg= (oomparser_linux.go). containerRegexp = regexp.MustCompile(`oom-kill:constraint=(.*),nodemask=(.*),cpuset=(.*),mems_allowed=(.*),oom_memcg=(.*),task_memcg=(.*),task=(.*),pid=(.*),uid=(.*)`) So no cgroup can be extracted from a global OOM message, and the initial value / of OomInstance is left as is (StreamOoms in oomparser_linux.go). oom := &OomInstance{ ContainerName: "/", VictimContainerName: "/", TimeOfDeath: msg.Timestamp, } In other words, the presence of SystemOOM is not "the result of directly determining the path" but "the result of whether the kernel log could be parsed". Still, in practice it is enough to understand that a recorded event is a reason to inspect node-side OOM, while neither presence nor absence establishes the source. {% enddetails %} How it appears in metrics Metrics do not distinguish the path with a label. The kernel's __oom_kill_process() records the same event regardless of the path. | Metric | Emitted by | memcg OOM | global OOM | |:---|:---|:---|:---| | node_vmstat_oom_kill | node-exporter (oom_kill in /proc/vmstat) | Increases | Increases | | container_oom_events_total | kubelet (result of kernel log parsing) | The series of the container the killed process belonged to increases | The id="/" series increases, not a container series | | oom_kill in cgroup v2 memory.events | (a cgroup file, not a metric; containerd reads it) | Increases | Increases | Only container_oom_events_total behaves differently because, as described earlier, the parser cannot extract a cgroup in a global OOM and treats it as the root (/). This is only a result of whether the log could be parsed, so do not use it to determine the path; confirm the source from constraint and oom_memcg in the kernel log. > Note: Although container_oom_events_total has container_ in its name, in v1.37 it is counted by the kubelet, not cAdvisor (recordOOMKill() in oom_counter.go). The meaning of the value does not change, but you have to look elsewhere when reading the code. What to look at after narrowing down - If you judge it to be a global OOM, inspect node-side memory pressure and any NUMA or cpuset constraints. Compare the memory requests and actual usage of the Pods placed on the same node, and check whether any Pod has too small a request - If you judge it to be a memcg OOM, review that container's memory limit and its actual usage. If the container seems to have headroom under its limit, it may have reached the limit of the Pod as a whole or of /kubepods.slice - If you cannot decide, keep both possibilities open and check both the container's limit and usage and the node's memory utilization {% details Kernel OOM behavior and victim selection %} The two OOM kill paths There are two paths by which a container's process gets OOM killed. Both are SIGKILLs from the kernel's OOM killer, but the conditions under which they occur and the scope they affect differ. As [defined in the terminology notes](#how-this-article-names-the-two-oom-kill-paths), memcg OOM enforces a configured limit and global OOM protects the node, so their purposes differ at the root. | Path | Condition | What gets killed | Node event | |:---|:---|:---|:---| | memcg OOM | memory.max exceeded for a container, Pod or /kubepods.slice | Processes inside the cgroup that hit its limit | Not recorded | | global OOM | A node-side allocation cannot be satisfied after reclaim, possibly within NUMA or cpuset constraints | Chosen from eligible processes under the allocation constraints | SystemOOM is recorded when the kubelet parses the log and treats the triggering cgroup as / | Even if your container is under its limit, it can be OOM killed because of another Pod on the same node. That is global OOM. SystemOOM is a kubelet-generated clue to investigate, not a definitive classification of the OOM source. Events expire according to the API server's --event-ttl (one hour by default), so a missing event does not rule out global OOM. OOM kill from exceeding a cgroup limit (memcg OOM) resources.limits.memory is set as the container cgroup's memory.max. When a cgroup's memory.current reaches memory.max, this is what happens. As described above, limits are set not only on containers but also on Pods and /kubepods.slice, and the same flow applies whichever level reaches its limit. 1. The kernel tries to reclaim memory inside that cgroup. Page cache is basically reclaimed here. 2. If usage still does not fit under memory.max after reclaim, the cgroup's OOM killer SIGKILLs processes in that cgroup. 3. With cgroup v2, the kubelet defaults to singleProcessOOMKill: false and sets memory.oom.group=1 for the container. A memcg OOM therefore normally kills the container's processes together. With singleProcessOOMKill: true, a child process can be killed while the container keeps running. Container restarts follow restartPolicy. See the implementation. The implementation is try_charge_memcg() in mm/memcontrol.c. It is a goto loop returning to the retry: label, and proceeds in this order. | Step | Location | What happens | |:---|:---|:---| | 1. Try to charge | L2177-L2178 | If page_counter_try_charge() succeeds, the allocation completes and it returns | | 2. Reclaim | L2211-L2216 | On failure it reclaims with try_to_free_mem_cgroup_pages(), and if mem_cgroup_margin() satisfies the request, goes back to 1 | | 3. Count the retry | L2244-L2245 | If reclaim is not enough, it decrements nr_retries and goes back to 1. The initial value is MAX_RECLAIM_RETRIES (= 16; mm/internal.h) | | 4. Invoke the OOM killer | L2259-L2264 | Only after the retries are used up does it call mem_cgroup_oom() | MAX_RECLAIM_RETRIES is a retry budget, not a guarantee of exactly 16 reclaim attempts. Allocation flags and reclaim progress change the control flow. The check in step 2 looks like this in code. nr_reclaimed = try_to_free_mem_cgroup_pages(mem_over_limit, nr_pages, gfp_mask, reclaim_options, NULL); psi_memstall_leave(&pflags); if (mem_cgroup_margin(mem_over_limit) >= nr_pages) goto retry; The kernel documentation for memory.max says the same. > Memory usage hard limit. ... If a cgroup's memory usage reaches this limit and can't be reduced, the OOM killer is invoked in the cgroup. > Warning: memory.current reaching memory.max does not by itself cause an OOM. A cgroup sitting at its limit with page cache is a normal state. For the same reason, the working set reaching memory.max is not an OOM condition either. Only inactive_file is subtracted from the working set, and reclaimable memory remains in the rest: > > - active_file: demoted to inactive under pressure and then reclaimed > - slab_reclaimable (dentry / inode caches): in memcg reclaim too, shrink_slab() is called after shrink_lruvec() (shrink_node_memcgs() in mm/vmscan.c) > > So there is no threshold of "OOM once either value reaches the limit". The actual condition is when the charge fails and reclaim cannot make room. What cannot be reclaimed is mainly anonymous pages (without swap) and shmem, so that is where to look when narrowing down. That said, what the kernel compares against memory.max in a memcg OOM is the whole of memory.current, and it is not decided by any single metric. memory.current includes, besides anonymous pages, kernel memory, shmem (tmpfs and shared memory), and page cache that cannot be reclaimed, or cannot be reclaimed right away. container_memory_rss is an indicator for determining whether anonymous pages are the main cause. If cgroup RSS is close to the limit, anonymous pages are a likely major contributor, but RSS being below the limit is no evidence that an OOM will not occur. When the gap between RSS and the working set is large, check container_memory_usage_bytes (the value of memory.current) together with the breakdown in memory.stat on the node: anon, file, shmem, slab and so on. OOM kill from node memory exhaustion (global OOM) When a node-side allocation cannot be satisfied after reclaim, the kernel evaluates eligible processes using oom_badness(). cpusets, NUMA policies and oom_score_adj=-1000 can restrict the candidates. A container need not exceed its own limit to be selected. There is no guaranteed ordering between kubelet eviction and the kernel's OOM killer. The kubelet evicts Pods to protect the node, and its selection puts "Pods whose usage exceeds their memory request" first (rankMemoryPressure() in helpers.go). // rankMemoryPressure orders the input pods for eviction in response to memory pressure. // It ranks by whether or not the pod's usage exceeds its requests, then by priority, and // finally by memory usage above requests. func rankMemoryPressure(pods []*v1.Pod, stats statsFunc) { orderedBy(exceedMemoryRequests(stats), priority, memory(stats)).Sort(pods) } A Pod with a zero memory request and positive usage belongs to the "exceeds request" group. Within that group, priority and then usage above request determine the order, so it is not necessarily evicted first. If eviction acts in time, the result is Evicted, not OOMKilled. Global OOM happens first when eviction cannot reclaim memory as fast as memory grows. There are several reasons it cannot keep up. | Factor | Value | Effect | |:---|:---|:---| | Evaluation interval | 10 seconds (evictionMonitoringPeriod) | Detection can be up to 10 seconds late | | Pods evicted per evaluation | Only one ("we kill at most a single pod during each eviction interval" in eviction_manager.go) | At most one Pod is evicted per synchronization; notifications and cleanup waits mean this is not a fixed rate | | Waiting for cleanup | Up to 30 seconds (podCleanupTimeout) | The next evaluation may be delayed further | | Default threshold | memory.available<100Mi | Extremely thin on large nodes | | How memory.available is computed | From the root cgroup's anon + file | Contains no kernel memory, so it [overestimates free memory](#values-the-kubelet-uses-for-eviction-decisions) | The official Kubernetes documentation also says (Node out of memory behavior): > If the node experiences an out of memory (OOM) event prior to the kubelet being able to reclaim memory, the node depends on the oom_killer to respond. It is more likely when applications that allocate a lot of memory right at startup, or in response to a sudden surge in requests, share a node. > Warning: Eviction protects the node; it does not protect individual Pods from OOM. Containers without limits have no memory.max backstop at all, so eviction is merely the last line of defense. > > Moreover, rankMemoryPressure() looks at Pod priority before usage size. If the offending Pod has high priority, low-priority Pods with small usage get evicted first, and global OOM can occur without the pressure being resolved. The proper countermeasure is to set limits and requests rather than relying on eviction. Which process gets killed The kernel evaluates eligible processes using resident pages, swap, page tables and oom_score_adj; the adjustment alone does not establish a fixed kill order. The kubelet sets oom_score_adj (a bonus added to oom_score) according to the container's QoS class, so the QoS class and the memory request setting directly become how likely a container is to be killed. You can check the QoS class with kubectl get pod <Pod name> -o jsonpath='{.status.qosClass}', and how it is determined is described in Pod Quality of Service Classes. | Condition | oom_score_adj | Likelihood of being killed | |:---|:---|:---| | system-node-critical (evaluated before the QoS class) | -997 | Least likely | | Guaranteed | -997 | Least likely | | Burstable | 1000 - 1000 × memory request / node memory capacity (clamped to 3–999) | The smaller the request, the more likely | | BestEffort | 1000 | Most likely | The implementation is GetContainerOOMScoreAdjust() in Kubernetes' policy.go. system-node-critical is evaluated before the QoS class decision. if types.IsNodeCriticalPod(pod) { // Only node critical pod should be the last to get killed. return guaranteedOOMScoreAdj } The formula above is the container-level baseline. Pod-level requests and restartable init containers can introduce adjustments. An omitted memory request can also default from the limit, so inspect the applied Pod spec. The Burstable clamp is implemented with a lower bound of 1000 + guaranteedOOMScoreAdj (= 3) and an upper bound of besteffortOOMScoreAdj - 1 (= 999). A Pod whose request is set smaller than its actual usage is more likely to be targeted in both eviction and global OOM. > Warning: PriorityClass works differently for eviction and for global OOM. It is a misreading to think that raising priority protects you from both. > > | PriorityClass | Eviction | global OOM (oom_score_adj) | > |:---|:---|:---| > | User defined (up to 1000000000) | The higher the value, the later it is evicted | Determined by QoS class and request | > | system-cluster-critical | Exempt from node-pressure eviction | Determined by QoS class and request (not protected) | > | system-node-critical | Exempt from node-pressure eviction | Fixed at -997 | > | static / mirror Pod | Exempt from node-pressure eviction | Determined by QoS class and request | > > The three exempt rows identify critical Pods for which IsCriticalPod() is true; the kubelet checks static and mirror Pods as well as system-critical priority classes. > > What oom_score_adj protects is only system-node-critical, for which IsNodeCriticalPod() is true. If a system-cluster-critical Pod is BestEffort, its oom_score_adj is 1000, and it is more likely to be selected in a global OOM. "I raised the priority but it was still OOMKilled" comes from this path. > Warning: Neither an OOM kill nor node-pressure eviction directly deletes the Pod API object. What happens next depends on restartPolicy. > > - Always or OnFailure: the container is recreated on the same node. The Pod stays on the same node, so if the cause is not resolved, OOM kills will happen again > - Never: the container is not recreated and the Pod remains Failed. If a controller such as a Job creates a successor Pod, it may be placed on a different node > > If OOMKilled repeats even though there is headroom under the limit, suspect this path. > Note: Because the OOM kill frees memory, the pressure may already be gone when the kubelet next evaluates free memory. In that case the node may get no MemoryPressure condition and no node.kubernetes.io/memory-pressure:NoSchedule taint, and scheduling of new Pods does not stop. Conversely, if the pressure continues, a later evaluation sets the condition and taint, and eviction also occurs. Note that it is decided by memory.available at evaluation time, not by whether an OOM kill happened. {% enddetails %} Checking with PSI (Pressure Stall Information) This is an auxiliary indicator for the case where things are slow although usage has not reached the limit. It shows how long processes were actually stalled waiting for memory reclaim. {% details How to read PSI and PromQL %} You can obtain the time a container was stalled waiting for memory allocation as PSI. Even when usage has not reached the limit, you can detect being made to wait by memory reclaim. | Metric | Emitted by | Description | |:---|:---|:---| | container_pressure_memory_waiting_seconds_total | cAdvisor (embedded in kubelet) | Time some processes were waiting for memory (total of some in memory.pressure) | | container_pressure_memory_stalled_seconds_total | cAdvisor (embedded in kubelet) | Time all processes were stalled waiting for memory (total of full in memory.pressure) | Note: These metrics are controlled by the KubeletPSI feature gate, which went GA in v1.36 and is enabled by default (it cannot be disabled). The kernel must support PSI, though, and the kubelet decides this by whether cpu.pressure exists at the cgroup root (cadvisor_linux.go). If the metrics do not appear, check whether the kernel was built with CONFIG_PSI=y and whether psi=0 is set as a kernel parameter. memory.pressure also has avg10 / avg60 / avg300, "the average over the last N seconds (%)", but only the cumulative value (total) is exposed as a Prometheus metric. So use rate() to see a ratio. The unit is seconds, so the result of rate() is "how many seconds of waiting per second", a ratio from 0 to 1. Multiply by 100 to get a percentage directly. <!-- PromQL uses Python fencing for syntax highlighting only. --> # Per Pod, the share of time (%) all processes were stalled waiting for memory 100 * max by (namespace, pod) ( rate(container_pressure_memory_stalled_seconds_total{container="", image="", pod!=""}[5m]) ) # Per Pod, the share of time (%) some processes were waiting for memory 100 * max by (namespace, pod) ( rate(container_pressure_memory_waiting_seconds_total{container="", image="", pod!=""}[5m]) ) When there is plenty of memory, neither increases, so rate() stays at 0. Under pressure, waiting (some) rises first, and if it gets worse stalled (full) follows. waiting counts even when only some processes waited, so to judge severity, use stalled. It is the time all processes in the container were stopped, so if it shows a value, throughput is directly affected. To see memory pressure on the whole node, use the root cgroup series. # Share of time (%) the whole node was completely stalled waiting for memory 100 * max by (node) ( rate(container_pressure_memory_stalled_seconds_total{id="/"}[5m]) * on(instance) group_left(node) kubelet_node_name ) Containers that are struggling although they are below their limit can be found by combining PSI with utilization. container_spec_memory_limit_bytes is the container limit exposed by cAdvisor. # Containers whose usage is under 80 % of the limit but that are stalled waiting for memory ( 100 * rate(container_pressure_memory_stalled_seconds_total{container!="", image!=""}[5m]) > 1 ) and ( container_memory_usage_bytes{container!="", image!=""} / container_spec_memory_limit_bytes{container!="", image!=""} < 0.8 ) and keeps only series whose label sets match exactly on both sides. A container with no limit has container_spec_memory_limit_bytes of 0, so the division gives +Inf, which does not satisfy < 0.8, and it is excluded automatically. Here is an example alert. It catches stalled that keeps occurring. - alert: ContainerMemoryStalled # Fires when the 5-minute average of fully stalled time is 5 % or more for 10 minutes expr: | 100 * max by (namespace, pod) ( rate(container_pressure_memory_stalled_seconds_total{container="", image="", pod!=""}[5m]) ) > 5 for: 10m labels: severity: warning annotations: summary: "A Pod is stalled waiting for memory allocation (add the namespace and pod labels with your own template)" description: "It may be made to wait by reclaim even though it has not reached its memory limit. Check the gap between usage and RSS and the amount of page cache." The right threshold of 5 % depends on the environment. First look at the distribution of values without an alert to understand the normal level, then decide. For details on PSI see Understand Pressure Stall Information (PSI) Metrics. {% enddetails %} Quick reference: contradictions explained and a metric map The working set, cgroup usage, process RSS and node availability answer different questions. Use this map to identify the value behind the dashboard before comparing it with a limit or threshold. For a non-root cgroup, working set is max(memory.current - inactive_file, 0). It still includes some reclaimable memory. | Scope | Metric | Emitted by | Backing value in cgroup v2 / /proc | Page cache handling | Main use | |:---|:---|:---|:---|:---|:---| | Container | container_memory_working_set_bytes | cAdvisor (embedded in kubelet) | max(memory.current − inactive_file, 0) | Active pages count as in use | kubectl top pod, usage visualization | | Container | container_memory_rss | cAdvisor (embedded in kubelet) | anon in memory.stat | Not included (anonymous pages only) | Understanding what the app allocated | | Container | container_memory_usage_bytes | cAdvisor (embedded in kubelet) | memory.current | Everything included | What memcg OOM is evaluated against | | Node | container_memory_working_set_bytes{id="/"} | cAdvisor (embedded in kubelet) | anon + file − inactive_file in the root cgroup's memory.stat (see below) | Active pages count as in use | Eviction decisions, kubectl top node | | Node | node_memory_MemAvailable_bytes | node-exporter | MemAvailable in /proc/meminfo | Reclaimable pages count as free | Visualizing node memory utilization | | Node | kube_node_status_capacity{resource="memory"} | kube-state-metrics | (status.capacity of the Node object) | (capacity, not usage) | Denominator when converting to utilization | Looking at the same node, utilization computed from node_memory_MemAvailable_bytes does not match the utilization the kubelet uses for eviction decisions. With workloads that do a lot of file I/O or logging, the difference can reach several GiB. Different emitters also mean different labels on the metrics. cAdvisor metrics carry no label identifying the node, so when you compute per-node utilization you need to align labels with kubelet_node_name, as shown later. Kubernetes memory usage and cgroup v2 Why the numbers differ A process RSS measurement includes its resident mappings, including shared and file-backed pages. Cgroup accounting charges memory to a group, while working set subtracts one category of file cache from usage. An application's heap metric covers a different subset again. Reclaimability is a separate question: clean file cache can often be reclaimed, dirty pages may need writeback, and anonymous memory normally needs swap or explicit release by the process. Calling free does not necessarily return allocator-held memory to the OS. These differences explain why heap size, process RSS, cgroup RSS and working set need not match. The following sections connect those values to cgroup v2 accounting. Memory breakdown in cgroup v2 Container memory usage is managed in a per-container cgroup. In cgroup v2, memory.current is the current usage and memory.stat is its breakdown. # Example values in a container's cgroup (checked on the node) cat /sys/fs/cgroup/<container cgroup>/memory.current Output: 5368709120 cat /sys/fs/cgroup/<container cgroup>/memory.stat Output: anon 2147483648 # anonymous pages (heap, stack: memory not backed by a file) file 3087007744 # page cache (including tmpfs and shared memory) shmem 268435456 # the part of file that is tmpfs or shared memory ... inactive_anon 134217728 # anon LRU: not accessed for a while active_anon 2013265920 # anon LRU: recently accessed inactive_file 1073741824 # file LRU: not accessed for a while active_file 1744830464 # file LRU: recently accessed unevictable 268435456 # pages that cannot be reclaimed (here, the tmpfs part) ... The following relationships hold in this example. Note that inactive_file + active_file does not equal file; it equals file - shmem. The next section explains why. anon + file + kernel memory = 5368709120 (memory.current) inactive_file + active_file = 2818572288 = file - shmem inactive_anon + active_anon = 2147483648 = anon (shmem is not in here) unevictable = 268435456 = shmem (tmpfs without swap) The main fields of memory.stat are: anon (anonymous pages) - Memory not backed by a file. Regions the application allocated with malloc and similar belong here. - Unlike page cache, the kernel cannot reclaim it independently of the process. With swap it can write pages out and free up space, but on nodes that do not use swap it is memory the kernel cannot reclaim under pressure. When swap is unavailable, the kernel sets scan_balance = SCAN_FILE and excludes the anon LRU from scanning (get_scan_count() in mm/vmscan.c). - On the other hand, this value drops if the process itself returns memory to the OS with munmap or madvise(MADV_DONTNEED). Note that calling free does not reduce it unless the allocator or language runtime returns the memory to the OS. file (page cache) - Contents of files held in memory by the kernel during reads and writes. The application did not allocate it explicitly. - When memory runs short the kernel discards (reclaims) it automatically, so it is basically reclaimable memory. - Cache of files on disk is split into active_file (recently accessed) and inactive_file (not accessed for a while). Reclaim takes inactive_file first. - active_file + inactive_file does not equal file. These are totals of the LRU lists, and as described below shmem is on the anon LRU rather than the file LRU. Besides shmem, pages that fall off the LRU, such as mlocked pages, also show up as a difference. - It includes not only the cache of files on disk but also shmem (tmpfs and shared memory), because Linux treats these as page cache too. shmem is written out to swap rather than to a file, so on nodes without swap it cannot be reclaimed. The breakdown is visible as shmem in memory.stat. > Warning: shmem is counted in file, but it is not on the file LRU. The kernel classifies swap-backed pages as anon, and tmpfs pages are among them. folio_is_file_lru() in mm_inline.h is defined as "0 if @folio is a normal anonymous folio, a tmpfs folio or otherwise ram or swap backed folio". > > The counters they are charged to are different, though. A tmpfs page is added to both NR_FILE_PAGES (= file) and NR_SHMEM (= shmem) (mm/shmem.c): > > ```c > __lruvec_stat_mod_folio(folio, NR_FILE_PAGES, nr); > __lruvec_stat_mod_folio(folio, NR_SHMEM, nr); > ``` > > Furthermore, for a Kubernetes emptyDir with medium: Memory, the kubelet mounts tmpfs with the noswap option when the node supports it (generateTmpfsMountOptions() in empty_dir.go). The kernel makes the inode of a noswap tmpfs unevictable (shmem_get_inode() in mm/shmem.c), so these pages sit on the unevictable LRU, not the anon LRU. > > ```c > if (sbinfo->noswap) > mapping_set_unevictable(inode->i_mapping); > ``` > > So shmem is accounted as follows: > > | mount | LRU | Where it shows up in memory.stat | > |:---|:---|:---| > | emptyDir with medium: Memory (noswap) | unevictable | unevictable | > | Any other tmpfs / shared memory | anon LRU | inactive_anon / active_anon | > > In neither case does it enter inactive_file / active_file. That matters in two ways later: > > - inactive_file + active_file generally corresponds to file - shmem; unevictable pages and asynchronous statistics can cause differences > - Only inactive_file is subtracted from the working set, so shmem stays in the working set in full With workloads that write a lot of logs or read and write large files, the page cache can grow to several GiB even though the application did not intend that. > Note: memory.current includes memory the kernel itself uses (slab, socket buffers and so on) in addition to anon and file. So adding anon and file alone does not give memory.current. How cgroup v2 memory controls map to Pod settings cgroup v2 memory control interface cgroup v2 offers four levels of control over memory usage. memory.min and memory.low are protections ("do not reclaim up to here"), and memory.high and memory.max are limits ("do not allow use beyond here"). | File | Type | Behavior | |:---|:---|:---| | memory.min | Protection (hard) | Protects memory within the effective boundary determined by ancestors and usage. It does not preallocate physical memory | | memory.low | Protection (soft) | Best-effort avoidance of reclaim up to this value. It is reclaimed if nothing else can be | | memory.high | Limit (soft) | Exceeding it applies reclaim pressure and throttling. Crossing memory.high does not itself invoke the OOM killer; memory.max and global OOM still apply | | memory.max | Limit (hard) | Exceeding it triggers reclaim, and if usage still does not fit, the cgroup's OOM killer runs | Other files that report state: - memory.current: current usage (what container_memory_usage_bytes is backed by) - memory.stat: usage breakdown - memory.events: counts of low, high, max, oom, oom_kill and oom_group_kill - memory.swap.max: swap limit - memory.pressure: PSI for waiting on memory allocation For exact definitions of each file see the kernel's Memory Interface Files, and for how Kubernetes treats cgroup v2 see About cgroup v2. How Pod settings are reflected Pod settings are reflected into cgroup v2 as follows. | Pod setting | Target in cgroup v2 | |:---|:---| | resources.limits.memory | memory.max | | resources.requests.memory | Not reflected (with the default kubelet configuration in v1.37; you can reflect it by changing settings → [KEP-2570](#kep-2570-support-memory-qos-with-cgroups-v2)) | | resources.limits.cpu | cpu.max | | resources.requests.cpu | cpu.weight | cgroup v2 is a directory tree. Its top is the root cgroup (/sys/fs/cgroup), where any process that is in no child cgroup belongs, so it contains not only Pods but every process on the node, such as the kubelet and containerd. Kubernetes creates the hierarchy below it: "all Pods on the node → Pod → container". # With the systemd cgroup driver (the kubeadm default; names differ with the cgroupfs driver, see below) /sys/fs/cgroup ← root cgroup (id="/") │ contains every process on the node │ the value used for eviction decisions │ ├── system.slice/ kubelet, containerd, sshd, etc. │ ├── kubelet.service (processes that are not Pods) │ └── containerd.service │ └── kubepods.slice/ ← all Pods on the node │ (id="/kubepods.slice") ├── kubepods-pod<UID>.slice/ ← Pod (Guaranteed) │ ├── cri-containerd-<ID>.scope ← container │ └── cri-containerd-<pause ID>.scope ← pause container │ ├── kubepods-burstable.slice/ (intermediate slice per QoS class) │ └── kubepods-burstable-pod<UID>.slice/ ← Pod (Burstable) │ └── cri-containerd-<ID>.scope ← container │ └── kubepods-besteffort.slice/ └── kubepods-besteffort-pod<UID>.slice/ ← Pod (BestEffort) └── cri-containerd-<ID>.scope ← container When filtering metrics, the root cgroup is id="/", all Pods on the node are id="/kubepods.slice", a Pod is container="", image="", pod!="", and a container is container!="", image!="". What the kubelet uses for eviction decisions is the working set of the topmost root cgroup ([see below](#values-the-kubelet-uses-for-eviction-decisions)). That is a different level from memcg OOM caused by a container or Pod exceeding its limit. Memory limits are set not only on containers but also on higher levels. | Level | cgroup | Memory limit | |:---|:---|:---| | Whole node (root cgroup) | / | Not set (the root has no memory.max → [see below](#values-the-kubelet-uses-for-eviction-decisions)) | | All Pods on the node | /kubepods.slice | Capacity - kube-reserved - system-reserved when Pod allocatable enforcement is enabled | | Pod | Differs by QoS class (see below) | The Pod's effective limit, including Pod-level spec.resources.limits.memory when configured | | Container | cri-containerd-<ID>.scope under the Pod's cgroup | That container's limits.memory | The examples here use container-level resources and omit Pod-level resources. Without a Pod-level limit, the required container limits must be present for a finite Pod limit to be set. A Pod's effective limit is not simply the sum of its regular containers' limits.memory. If there are init containers, their maximum is also considered, and spec.overhead is added if set. For the calculation see Resource sharing within containers and Pod Overhead. So even if a container has not reached its own limit, an OOM kill can occur when the Pod as a whole or /kubepods.slice hits its limit. > Note: The paths above assume the kubelet uses the systemd cgroup driver. With the cgroupfs driver, paths look like /kubepods/burstable/pod<UID>/<container ID>, with no .slice or .scope and no cri-containerd- prefix on the container. > > The kubelet's own default is cgroupfs, but kubeadm sets systemd when the value is empty (defaults.go, kubelet.go). In v1.37, a supported runtime's CRI RuntimeConfig response takes precedence through driver auto-detection. Check the effective driver with: > > ```bash > $ kubectl get --raw "/api/v1/nodes/<node name>/proxy/configz" \ | jq '.kubeletconfig.cgroupDriver' > "systemd" > ``` > > The id label of cAdvisor metrics also changes with the cgroup driver, so when filtering in PromQL, check the value in your own cluster with this query: > > ```bash > group by (id) (container_memory_working_set_bytes{id=~"/kubepods|/kubepods.slice"}) > ``` > > A Pod's cgroup path depends on its QoS class. As the tree above shows, Burstable and BestEffort go through an intermediate slice per QoS class. > > In the <UID> part, systemd escaping replaces the hyphens of the Pod UID with underscores (example: 6983351e-4ede-... → pod6983351e_4ede_...). The UID of a static Pod (mirror Pod) is a hash with no hyphens, so it looks like pod1ff70b81885169da4e9e32c9530b77c9.slice. > > The QoS class and UID of a Pod can be read from the Pod resource, and once you have those two you can assemble the cgroup path. > > ```bash > $ kubectl get pod -n <namespace> <Pod name> -o jsonpath='{.status.qosClass}{"\n"}{.metadata.uid}{"\n"}' > Burstable > 6983351e-4ede-4611-8bab-b986614af18d > > # To list them > $ kubectl get pods -n <namespace> -o custom-columns='NAME:.metadata.name,QOS:.status.qosClass' > ``` Container metrics The cAdvisor built into the kubelet exposes the cgroup values above as Prometheus metrics. They are served from the kubelet's /metrics/cadvisor endpoint, and you do not need to run a separate cAdvisor Pod or DaemonSet. | Metric | Emitted by | Backing value in cgroup v2 | Description | |:---|:---|:---|:---| | container_memory_usage_bytes | cAdvisor (embedded in kubelet) | memory.current | Usage including anonymous pages, page cache and kernel memory. The value memcg OOM is evaluated against | | container_memory_working_set_bytes | cAdvisor (embedded in kubelet) | memory.current - inactive_file | Usage minus inactive_file, clamped to zero. A heuristic that still includes some reclaimable memory | | container_memory_rss | cAdvisor (embedded in kubelet) | anon in memory.stat | Amount of anonymous pages. Unlike true RSS, it does not include mmaped file pages (see below) | | container_memory_cache | cAdvisor (embedded in kubelet) | file in memory.stat | Page cache size. Includes shmem (tmpfs and shared memory) | | container_memory_total_active_file_bytes | cAdvisor (embedded in kubelet) | active_file in memory.stat | Active page cache | | container_memory_total_inactive_file_bytes | cAdvisor (embedded in kubelet) | inactive_file in memory.stat | Inactive page cache | | container_memory_mapped_file | cAdvisor (embedded in kubelet) | file_mapped in memory.stat | Amount of mmaped files | | `cont