Kubernetes cost optimization for startups
---
id: 4621832
title: Kubernetes cost optimization for startups
published: true
description: 'The 2026 playbook to cut Kubernetes bills: right-sizing, spot, idle sleep, quotas, image pulls, NAT routing, and the change that compounds.'
tags: 'kubernetes, aws, devops, cost'
canonical_url: 'ownkube.io'
cover_image: 'ownkube.io'
---
What surprises people first about a startup's Kubernetes bill is how predictable the leaks are. Across the small-team AWS and GCP bills we audit, the same seven patterns drive 70 to 90% of the waste, and they're all fixable in a single quarter.
This post is the consolidated playbook. It covers the seven patterns, the realistic savings on each, and the structural change that keeps them from drifting back: making the platform carry the defaults.
Skim answer:
- The seven highest-leverage patterns: right-sizing, spot capacity, idle environment sleep, namespace quotas, image-pull traffic reduction, NAT-routing audits, and storage class right-sizing.
- Combined impact: typically cuts a small-team Kubernetes bill by 40 to 65%.
- Timeline: all seven are fixable in a single quarter.
Why startup K8s bills are usually 2x what they should be
The math behind the typical waste:
- Most teams set resource requests at 2x to 4x what the workload actually uses, "to be safe". Cluster autoscaler then provisions nodes for the inflated request rather than the real usage.
- Idle preview environments and staging clusters run 24/7 even though they're used 30 hours a week.
- Container images get pulled from public registries through NAT gateways, racking up data-processing fees nobody sees.
- EBS volumes are provisioned at default gp2 sizes that exceed real I/O needs by an order of magnitude.
Each of these is a small, steady bleed. Add up enough of them and the cluster bill is double what it should be.
Pattern 1: Right-size every workload
This is the largest lever. Most workloads we see have resource requests set during initial deployment and never revisited. A typical web pod might have a 1 vCPU / 1 GB RAM request and a 95th-percentile usage of 0.18 vCPU / 240 MB RAM, so the other 80% of the request is reservation paid for and unused.
To fix it, run Vertical Pod Autoscaler in recommendation mode for one to two weeks, then apply the recommendations. For long-tail workloads with bursty traffic, use VPA in Auto mode with sensible min/max bounds.
Expect to save 25 to 45% of cluster compute. It's the biggest line-item improvement on every audit.
Pattern 2: Spot capacity for the workloads that tolerate it
Stateless web pods, queue consumers, build runners, batch jobs, and preview environments are excellent candidates for spot. Database primaries, control plane nodes, and single-replica stateful services should stay on-demand. We covered the full pattern in AWS spot instances in production.
To fix it, use mixed instance pools with an on-demand base and spot for the rest. Karpenter or Cluster Autoscaler with mixed ASGs handles the orchestration. Set PodDisruptionBudgets and 30 to 120 second graceful shutdown windows.
Expect 50 to 70% off the spot-eligible portion of the compute bill. For a typical small-team workload, that's roughly 30 to 50% off the total cluster compute.
Pattern 3: Sleep idle environments
Staging, preview, and developer sandbox environments sit idle most of the week. A preview environment created on Monday for a PR that merges Friday runs for ~120 hours while being used for maybe 4. The other 116 hours are waste.
To fix it, add an idle-detection loop that scales preview deployments to zero after N hours of no traffic, then scales back up on the next request. Tools like KEDA, scaler-controller, or platform-layer features handle this.
Expect to save 50 to 75% of preview environment cost. For teams with 5 to 20 active previews at any time, this can be the second-biggest single-pattern saving.
Pattern 4: Namespace quotas and request governance
Without ResourceQuota on each namespace, any service team can ship a deployment that requests more than it needs. Quotas force the conversation: "you want 8 vCPU for this worker, defend it."
To fix it, pin a ResourceQuota per namespace to a realistic budget, add a LimitRange to set sensible default requests for pods without explicit requests, and revisit the quotas in a monthly review.
Expect 10 to 20% in indirect savings through culture change. The bigger win is preventing future drift.
Pattern 5: Pull images from a private registry
Most teams pull container images from Docker Hub, GitHub Container Registry, or Quay through a NAT gateway. Each pull is a few hundred MB. With a busy CI fleet plus rolling production deploys, that's tens of GB per day routed through NAT at $0.045 per GB.
To fix it, mirror your base images to a private Amazon ECR registry (or GCP Artifact Registry), enable the ECR VPC interface endpoint, and configure your image pull policy to use the mirror. Image pulls then stay on the AWS backbone at near-zero per-GB cost.
Expect to save $50 to $400 per month for a typical small team, and more for orgs that deploy often.
Pattern 6: Audit NAT routing
NAT gateway data-processing fees are the most under-noticed cost on AWS bills. We covered the full pattern in our NAT gateway cost guide. The short version:
- Enable S3 and DynamoDB gateway endpoints (free) on every VPC.
- Add interface endpoints for high-volume AWS services (Secrets Manager, STS, CloudWatch Logs).
- Dual-stack your VPC and route IPv6 traffic through a free egress-only IGW.
- Reduce NAT topology from 3 AZs to 2 if availability tolerates.
Expect to save $200 to $1,500 per month depending on cluster traffic.
Pattern 7: Storage class right-sizing
EBS gp3 has replaced gp2 as the sensible default for most cluster volumes. It's cheaper and faster, and you pay only for the IOPS you provision (on gp2, IOPS scale with volume size).
To fix it, migrate from gp2 to gp3 across your cluster and right-size volume capacity (most workloads provision 100 GB when they use 12 GB). For workloads with very low I/O, consider sc1 or st1 for cold data.
Expect to save 20 to 35% on EBS line items.
The structural change that compounds them all
Each of the seven patterns above is a one-time engineering project, and implementing them all takes 2 to 4 weeks of focused work for a small team. The harder question is what stops the drift from coming back six months later.
Without an explicit owner, nothing does. Request inflation creeps back in, new services get deployed without quotas, new environments forget the sleep schedule, new container pulls go through NAT, and the cluster bill returns to its old shape within two quarters.
The structural fix is to have the platform carry these decisions instead of leaving them to tribal memory. On Ownkube, running inside your own cloud account, the platform ships with the primitives that keep these patterns from drifting:
- Managed Postgres, so database sizing and backups aren't a manual task per environment.
- Horizontal autoscaling on every workload, so requests track real traffic instead of a number set at first deploy and never revisited.
- Built-in logs and metrics, so a right-sizing pass is a five-minute look at a dashboard instead of a new observability rollout.
- One-command rollbacks, so a bad change costs you a rollback instead of an incident.
You still own the architectural decisions. Ownkube keeps the primitives in place so the drift has fewer places to hide.
A worked example
Take a typical 2026 SaaS running Kubernetes on AWS: 1 cluster, ~24 vCPU production fleet, 8 vCPU staging, 10 active preview environments, RDS Postgres, ElastiCache.
Before optimization, it pays ~$2,400/month cluster compute + $180 NAT + $90 EBS = $2,670/month. Here is what each pattern saves:
| Pattern | Saving |
|---|---|
| Right-sizing (Pattern 1) | -$840 |
| Spot capacity (Pattern 2) | -$520 |
| Idle environment sleep (Pattern 3) | -$280 |
| Namespace quotas + governance (Pattern 4) | -$120 |
| Private image registry (Pattern 5) | -$140 |
| NAT routing audit (Pattern 6) | -$110 |
| Storage class right-sizing (Pattern 7) | -$30 |
| Total saved | -$2,040 |
After all seven, the bill is ~$630/month, about a 76% reduction on the cluster bill.
These numbers come from the workload defined above, so your savings will differ with yours.
Decision checklist
Before you start, confirm:
- [ ] Do you have a way to measure current per-workload resource usage (Prometheus, CloudWatch, Datadog)?
- [ ] Is your cluster on Kubernetes 1.27+ (so VPA and Karpenter work cleanly)?
- [ ] Do you have at least one engineer with a couple of weeks of focused time?
- [ ] Is anyone going to own ongoing cost vigilance after the initial pass, or do you need that on autopilot?
If you ticked all four, you're set up to do the work in-house. If you ticked three or fewer, consider a platform layer that runs these patterns by default.
Closing
Kubernetes cost optimization at a small startup comes down to seven well-understood patterns plus one structural change to stop the drift from coming back: making the platform itself carry the defaults instead of leaving them to memory.
If you'd rather skip the initial work and start with a platform that ships those defaults, Ownkube runs inside your own AWS account. Starter is free forever on one instance, and Production is $5 per vCPU plus $1 per GiB RAM per month once you scale. Deploy my first app.
Kubernetes Memory on cgroup v2: Debugging OOM Kills and Evictions
A container is OOMKilled, but kubectl top never showed it reaching its limit. Or a Pod is evicted while your node dashboard still shows free memory. In both cases, the first question is which memory value the decision actually used.
This guide separates cgroup-limit OOM, node-side OOM and kubelet eviction, then shows how to inspect the evidence and review memory requests and limits. It assumes familiarity with Pod resources and basic PromQL.
TL;DR
- The working set is a heuristic, not a measurement of unreclaimable memory. It subtracts inactive_file from usage but still includes reclaimable active page cache and slab.
- Distinguish memcg OOM from global OOM using kernel logs. A container can be below its own limit when an ancestor cgroup reaches its limit or a node-side allocation fails. OOMKilled and SystemOOM alone cannot establish the cause.
- Eviction uses the kubelet's memory.available, not Linux MemAvailable. Check the kubelet's statistics and configured thresholds.
- Review requests and limits separately. Requests influence placement and victim selection. To address a cgroup limit, reduce usage or adjust the limit after checking node capacity.
- Check Memory QoS configuration as well as its feature gate. In v1.37, the gate defaults to enabled, but request-based protection and memory.high remain disabled by default.
This article targets cgroup v2, Kubernetes v1.37.1 and containerd. See [Components and versions](#components-and-versions) for exact versions. The kind examples use a v1.37.0 node image.
Use the [investigation workflow](#investigation-workflow) for an incident, the [metric map](#quick-reference-contradictions-explained-and-a-metric-map) to understand a dashboard, or [requests and limits](#what-to-do-about-it) for prevention. The [kind examples](#hands-on-verification-with-kind) explore reclaim and Memory QoS in more detail.
Identify the failure before choosing a metric
| Observation | Mechanism | Evidence to check |
|:---|:---|:---|
| Container termination reason is OOMKilled | The kernel killed a process; this alone does not distinguish the OOM source | Kernel logs, container and ancestor cgroup limits, node allocation constraints |
| Pod is Evicted with The node was low on resource: memory. | The kubelet responded to node memory pressure | Kubelet memory.available, configured thresholds and eviction message |
| Exit code is 137 without an OOM reason | The process received SIGKILL; the exit code alone does not establish OOM | Termination details, events and kernel logs |
memcg OOM means an allocation cannot fit within a cgroup limit after reclaim. Global OOM means a node-side allocation cannot be satisfied after reclaim, potentially within NUMA or cpuset constraints. A container below its own limit can be affected by either an ancestor limit or a node-side failure.
Investigating OOM and eviction
Investigation workflow
Start with the observed termination or eviction, then compare metrics from the same time. Scrapes can miss the peak; kernel logs are stronger evidence of the OOM source than a sampled usage graph.
1. When a Pod was OOMKilled
- Check whether container_memory_usage_bytes is close to the limit. memcg OOM is evaluated against this value
- Compare with container_memory_rss to tell whether anonymous pages are the main cause. An OOM can occur even when RSS is low
- If the gap between the working set or usage and RSS is large, page cache, shmem or kernel memory may be pushing usage up
- If OOMKilled occurs despite headroom under the limit, suspect global OOM. Correlate SystemOOM with kernel logs and inspect ancestor limits. A missing event does not establish memcg OOM
- If you see no spike in the metrics, suspect a surge shorter than the scrape interval. See [My working set is below the limit, but the container was OOMKilled](#my-working-set-is-below-the-limit-but-the-container-was-oomkilled) for what to check
2. When a Pod was evicted
- If the message is The node was low on resource: memory., node-wide memory pressure is the cause
- Use the [node-utilization query](#why-node-exporter-and-kubelet-usage-disagree) to check the kubelet-view node memory utilization. The node-exporter-based utilization alone is not enough to judge
- Check whether a Pod with a small (or unset) memory request relative to actual usage is on the same node. Compare each Pod's container_memory_working_set_bytes (cAdvisor) against kube_pod_resource_request{resource="memory"} (kube-scheduler). Note that Pods with large requests make the scheduler keep placement density low, so they are unlikely to be the cause of pressure
3. To prevent recurrence
- Set a memory request informed by workload history to avoid excessive placement density. PromQL for checking is collected in [Align memory request with actual usage](#align-memory-request-with-actual-usage)
- For containers where memcg OOM occurs, also review the limit
- Control placement with Pod anti-affinity or topology spread constraints so that Pods with large memory usage do not gather on the same node
Telling which path occurred
Using Kubernetes information
Start by narrowing down with what you can check using kubectl.
| Where to look | What it tells you |
|:---|:---|
| Last State in kubectl describe pod, or lastState.terminated in kubectl get pod -o yaml | That an OOM kill happened (reason: OOMKilled). It cannot distinguish the path |
| Events in kubectl describe node <node name> | Use SystemOOM as a clue and verify the source in kernel logs. If there is no record, that does not make it a memcg OOM; it just means you cannot tell |
There is no OOM-specific event on the Pod side. You only see things like BackOff accompanying the restart.
Check it as follows. SystemOOM is an event attached to the Node, and since Node is not a namespaced object, the event itself is recorded in the default namespace.
# 1. Find which node the OOMKilled Pod is on
kubectl get pod -n <namespace> <Pod name> -o wide
Output:
NAME READY STATUS RESTARTS AGE IP NODE
myapp-... 1/1 Running 5 (2m ago) 1h 10.0.0.1 node-1
# 2. Check whether that node's events contain SystemOOM
kubectl describe node node-1
Output:
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning SystemOOM 3m kubelet System OOM encountered, victim process: java, pid: 12345
# To list only events
kubectl get events -n default \
--field-selector reason=SystemOOM,involvedObject.name=node-1
If SystemOOM exists, inspect kernel messages from the same time: constraint, oom_memcg, task_memcg and the killed PID. A parsing failure can also produce this event. Match the event and kernel log to the container's termination time (lastState.terminated.finishedAt).
Not finding SystemOOM is not evidence of memcg OOM. The same state results from event TTL expiry, a kubelet restart, or a missed event.
The event retention period is set by kube-apiserver's --event-ttl, and the default is 1 hour (options.go). So an old SystemOOM may no longer be available through kubectl. If you forward events to an external logging platform, check there.
On the node, inspect the kernel log:
sudo journalctl -k -b -o short-iso \
| rg 'oom-kill:|Out of memory:|Memory cgroup out of memory:|Killed process'
CONSTRAINT_MEMCG and oom_memcg identify the limiting cgroup; task_memcg identifies the victim's cgroup. If these differ, inspect the Pod and ancestor limits. CONSTRAINT_CPUSET and CONSTRAINT_MEMORY_POLICY call for checking NUMA and cpuset restrictions. Align timestamps with the container's finishedAt.
If logs are unavailable, use the following as supporting evidence rather than proof:
- Whether container_memory_usage_bytes was close to the limit at that time (if so, memcg OOM is likely)
- Whether [the kubelet-view node memory utilization](#why-node-exporter-and-kubelet-usage-disagree) was high at that time (if so, global OOM is likely)
Why OOMKilled cannot distinguish the paths
When a task exits with code 137, the container runtime (containerd) checks oom_kill in the cgroup's memory.events, and if it is 1 or more, it updates the termination reason to OOMKilled (events.go). The check is oomMetricsEventOccurred() in the same file.
switch v := taskMetricsAny.(type) {
case *cg1.Metrics:
return v.GetMemoryOomControl().GetOomKill() > 0, nil
case *cg2.Metrics:
return v.GetMemoryEvents().GetOomKill() > 0, nil
The definition of oom_kill in memory.events is "The number of processes belonging to this cgroup killed by any kind of OOM killer". So a process chosen and killed by the kernel in a global OOM is counted too, and both paths produce OOMKilled.
{% details How the kubelet parser produces SystemOOM %}
When SystemOOM is recorded
SystemOOM is an event the kubelet records on the Node object. The kubelet parses OOM messages in the kernel log and records it only when the cgroup where the OOM happened is judged to be the root (/) (oom_watcher_linux.go).
for event := range outStream {
// Count every OOM kill per container to back the
// container_oom_events_total metric.
recordOOMKill(event.ContainerName)
if event.VictimContainerName == recordEventContainerName {
...
ow.recorder.WithLogger(logger).Eventf(ref, v1.EventTypeWarning, systemOOMEvent, "%s", eventMsg)
}
}
The usual association with global OOM comes from the combination of the kernel log format and the parser. For an OOM caused by a memcg limit, the kernel prints oom_memcg=<cgroup path>, whereas for a global OOM it prints ,global_oom instead (mem_cgroup_print_oom_context() in mm/memcontrol.c).
if (memcg) {
pr_cont(",oom_memcg=");
pr_cont_cgroup_path(memcg->css.cgroup);
} else
pr_cont(",global_oom");
The modern-format regular expression in the kubelet's forked parser matches the format containing oom_memcg= (oomparser_linux.go).
containerRegexp = regexp.MustCompile(`oom-kill:constraint=(.*),nodemask=(.*),cpuset=(.*),mems_allowed=(.*),oom_memcg=(.*),task_memcg=(.*),task=(.*),pid=(.*),uid=(.*)`)
So no cgroup can be extracted from a global OOM message, and the initial value / of OomInstance is left as is (StreamOoms in oomparser_linux.go).
oom := &OomInstance{
ContainerName: "/",
VictimContainerName: "/",
TimeOfDeath: msg.Timestamp,
}
In other words, the presence of SystemOOM is not "the result of directly determining the path" but "the result of whether the kernel log could be parsed". Still, in practice it is enough to understand that a recorded event is a reason to inspect node-side OOM, while neither presence nor absence establishes the source.
{% enddetails %}
How it appears in metrics
Metrics do not distinguish the path with a label. The kernel's __oom_kill_process() records the same event regardless of the path.
| Metric | Emitted by | memcg OOM | global OOM |
|:---|:---|:---|:---|
| node_vmstat_oom_kill | node-exporter (oom_kill in /proc/vmstat) | Increases | Increases |
| container_oom_events_total | kubelet (result of kernel log parsing) | The series of the container the killed process belonged to increases | The id="/" series increases, not a container series |
| oom_kill in cgroup v2 memory.events | (a cgroup file, not a metric; containerd reads it) | Increases | Increases |
Only container_oom_events_total behaves differently because, as described earlier, the parser cannot extract a cgroup in a global OOM and treats it as the root (/). This is only a result of whether the log could be parsed, so do not use it to determine the path; confirm the source from constraint and oom_memcg in the kernel log.
> Note: Although container_oom_events_total has container_ in its name, in v1.37 it is counted by the kubelet, not cAdvisor (recordOOMKill() in oom_counter.go). The meaning of the value does not change, but you have to look elsewhere when reading the code.
What to look at after narrowing down
- If you judge it to be a global OOM, inspect node-side memory pressure and any NUMA or cpuset constraints. Compare the memory requests and actual usage of the Pods placed on the same node, and check whether any Pod has too small a request
- If you judge it to be a memcg OOM, review that container's memory limit and its actual usage. If the container seems to have headroom under its limit, it may have reached the limit of the Pod as a whole or of /kubepods.slice
- If you cannot decide, keep both possibilities open and check both the container's limit and usage and the node's memory utilization
{% details Kernel OOM behavior and victim selection %}
The two OOM kill paths
There are two paths by which a container's process gets OOM killed. Both are SIGKILLs from the kernel's OOM killer, but the conditions under which they occur and the scope they affect differ. As [defined in the terminology notes](#how-this-article-names-the-two-oom-kill-paths), memcg OOM enforces a configured limit and global OOM protects the node, so their purposes differ at the root.
| Path | Condition | What gets killed | Node event |
|:---|:---|:---|:---|
| memcg OOM | memory.max exceeded for a container, Pod or /kubepods.slice | Processes inside the cgroup that hit its limit | Not recorded |
| global OOM | A node-side allocation cannot be satisfied after reclaim, possibly within NUMA or cpuset constraints | Chosen from eligible processes under the allocation constraints | SystemOOM is recorded when the kubelet parses the log and treats the triggering cgroup as / |
Even if your container is under its limit, it can be OOM killed because of another Pod on the same node. That is global OOM.
SystemOOM is a kubelet-generated clue to investigate, not a definitive classification of the OOM source. Events expire according to the API server's --event-ttl (one hour by default), so a missing event does not rule out global OOM.
OOM kill from exceeding a cgroup limit (memcg OOM)
resources.limits.memory is set as the container cgroup's memory.max. When a cgroup's memory.current reaches memory.max, this is what happens. As described above, limits are set not only on containers but also on Pods and /kubepods.slice, and the same flow applies whichever level reaches its limit.
1. The kernel tries to reclaim memory inside that cgroup. Page cache is basically reclaimed here.
2. If usage still does not fit under memory.max after reclaim, the cgroup's OOM killer SIGKILLs processes in that cgroup.
3. With cgroup v2, the kubelet defaults to singleProcessOOMKill: false and sets memory.oom.group=1 for the container. A memcg OOM therefore normally kills the container's processes together. With singleProcessOOMKill: true, a child process can be killed while the container keeps running. Container restarts follow restartPolicy. See the implementation.
The implementation is try_charge_memcg() in mm/memcontrol.c. It is a goto loop returning to the retry: label, and proceeds in this order.
| Step | Location | What happens |
|:---|:---|:---|
| 1. Try to charge | L2177-L2178 | If page_counter_try_charge() succeeds, the allocation completes and it returns |
| 2. Reclaim | L2211-L2216 | On failure it reclaims with try_to_free_mem_cgroup_pages(), and if mem_cgroup_margin() satisfies the request, goes back to 1 |
| 3. Count the retry | L2244-L2245 | If reclaim is not enough, it decrements nr_retries and goes back to 1. The initial value is MAX_RECLAIM_RETRIES (= 16; mm/internal.h) |
| 4. Invoke the OOM killer | L2259-L2264 | Only after the retries are used up does it call mem_cgroup_oom() |
MAX_RECLAIM_RETRIES is a retry budget, not a guarantee of exactly 16 reclaim attempts. Allocation flags and reclaim progress change the control flow. The check in step 2 looks like this in code.
nr_reclaimed = try_to_free_mem_cgroup_pages(mem_over_limit, nr_pages,
gfp_mask, reclaim_options, NULL);
psi_memstall_leave(&pflags);
if (mem_cgroup_margin(mem_over_limit) >= nr_pages)
goto retry;
The kernel documentation for memory.max says the same.
> Memory usage hard limit. ... If a cgroup's memory usage reaches this limit and can't be reduced, the OOM killer is invoked in the cgroup.
> Warning: memory.current reaching memory.max does not by itself cause an OOM. A cgroup sitting at its limit with page cache is a normal state. For the same reason, the working set reaching memory.max is not an OOM condition either. Only inactive_file is subtracted from the working set, and reclaimable memory remains in the rest:
>
> - active_file: demoted to inactive under pressure and then reclaimed
> - slab_reclaimable (dentry / inode caches): in memcg reclaim too, shrink_slab() is called after shrink_lruvec() (shrink_node_memcgs() in mm/vmscan.c)
>
> So there is no threshold of "OOM once either value reaches the limit". The actual condition is when the charge fails and reclaim cannot make room. What cannot be reclaimed is mainly anonymous pages (without swap) and shmem, so that is where to look when narrowing down.
That said, what the kernel compares against memory.max in a memcg OOM is the whole of memory.current, and it is not decided by any single metric. memory.current includes, besides anonymous pages, kernel memory, shmem (tmpfs and shared memory), and page cache that cannot be reclaimed, or cannot be reclaimed right away.
container_memory_rss is an indicator for determining whether anonymous pages are the main cause. If cgroup RSS is close to the limit, anonymous pages are a likely major contributor, but RSS being below the limit is no evidence that an OOM will not occur. When the gap between RSS and the working set is large, check container_memory_usage_bytes (the value of memory.current) together with the breakdown in memory.stat on the node: anon, file, shmem, slab and so on.
OOM kill from node memory exhaustion (global OOM)
When a node-side allocation cannot be satisfied after reclaim, the kernel evaluates eligible processes using oom_badness(). cpusets, NUMA policies and oom_score_adj=-1000 can restrict the candidates. A container need not exceed its own limit to be selected.
There is no guaranteed ordering between kubelet eviction and the kernel's OOM killer. The kubelet evicts Pods to protect the node, and its selection puts "Pods whose usage exceeds their memory request" first (rankMemoryPressure() in helpers.go).
// rankMemoryPressure orders the input pods for eviction in response to memory pressure.
// It ranks by whether or not the pod's usage exceeds its requests, then by priority, and
// finally by memory usage above requests.
func rankMemoryPressure(pods []*v1.Pod, stats statsFunc) {
orderedBy(exceedMemoryRequests(stats), priority, memory(stats)).Sort(pods)
}
A Pod with a zero memory request and positive usage belongs to the "exceeds request" group. Within that group, priority and then usage above request determine the order, so it is not necessarily evicted first. If eviction acts in time, the result is Evicted, not OOMKilled.
Global OOM happens first when eviction cannot reclaim memory as fast as memory grows. There are several reasons it cannot keep up.
| Factor | Value | Effect |
|:---|:---|:---|
| Evaluation interval | 10 seconds (evictionMonitoringPeriod) | Detection can be up to 10 seconds late |
| Pods evicted per evaluation | Only one ("we kill at most a single pod during each eviction interval" in eviction_manager.go) | At most one Pod is evicted per synchronization; notifications and cleanup waits mean this is not a fixed rate |
| Waiting for cleanup | Up to 30 seconds (podCleanupTimeout) | The next evaluation may be delayed further |
| Default threshold | memory.available<100Mi | Extremely thin on large nodes |
| How memory.available is computed | From the root cgroup's anon + file | Contains no kernel memory, so it [overestimates free memory](#values-the-kubelet-uses-for-eviction-decisions) |
The official Kubernetes documentation also says (Node out of memory behavior):
> If the node experiences an out of memory (OOM) event prior to the kubelet being able to reclaim memory, the node depends on the oom_killer to respond.
It is more likely when applications that allocate a lot of memory right at startup, or in response to a sudden surge in requests, share a node.
> Warning: Eviction protects the node; it does not protect individual Pods from OOM. Containers without limits have no memory.max backstop at all, so eviction is merely the last line of defense.
>
> Moreover, rankMemoryPressure() looks at Pod priority before usage size. If the offending Pod has high priority, low-priority Pods with small usage get evicted first, and global OOM can occur without the pressure being resolved. The proper countermeasure is to set limits and requests rather than relying on eviction.
Which process gets killed
The kernel evaluates eligible processes using resident pages, swap, page tables and oom_score_adj; the adjustment alone does not establish a fixed kill order. The kubelet sets oom_score_adj (a bonus added to oom_score) according to the container's QoS class, so the QoS class and the memory request setting directly become how likely a container is to be killed. You can check the QoS class with kubectl get pod <Pod name> -o jsonpath='{.status.qosClass}', and how it is determined is described in Pod Quality of Service Classes.
| Condition | oom_score_adj | Likelihood of being killed |
|:---|:---|:---|
| system-node-critical (evaluated before the QoS class) | -997 | Least likely |
| Guaranteed | -997 | Least likely |
| Burstable | 1000 - 1000 × memory request / node memory capacity (clamped to 3–999) | The smaller the request, the more likely |
| BestEffort | 1000 | Most likely |
The implementation is GetContainerOOMScoreAdjust() in Kubernetes' policy.go. system-node-critical is evaluated before the QoS class decision.
if types.IsNodeCriticalPod(pod) {
// Only node critical pod should be the last to get killed.
return guaranteedOOMScoreAdj
}
The formula above is the container-level baseline. Pod-level requests and restartable init containers can introduce adjustments. An omitted memory request can also default from the limit, so inspect the applied Pod spec.
The Burstable clamp is implemented with a lower bound of 1000 + guaranteedOOMScoreAdj (= 3) and an upper bound of besteffortOOMScoreAdj - 1 (= 999). A Pod whose request is set smaller than its actual usage is more likely to be targeted in both eviction and global OOM.
> Warning: PriorityClass works differently for eviction and for global OOM. It is a misreading to think that raising priority protects you from both.
>
> | PriorityClass | Eviction | global OOM (oom_score_adj) |
> |:---|:---|:---|
> | User defined (up to 1000000000) | The higher the value, the later it is evicted | Determined by QoS class and request |
> | system-cluster-critical | Exempt from node-pressure eviction | Determined by QoS class and request (not protected) |
> | system-node-critical | Exempt from node-pressure eviction | Fixed at -997 |
> | static / mirror Pod | Exempt from node-pressure eviction | Determined by QoS class and request |
>
> The three exempt rows identify critical Pods for which IsCriticalPod() is true; the kubelet checks static and mirror Pods as well as system-critical priority classes.
>
> What oom_score_adj protects is only system-node-critical, for which IsNodeCriticalPod() is true. If a system-cluster-critical Pod is BestEffort, its oom_score_adj is 1000, and it is more likely to be selected in a global OOM. "I raised the priority but it was still OOMKilled" comes from this path.
> Warning: Neither an OOM kill nor node-pressure eviction directly deletes the Pod API object. What happens next depends on restartPolicy.
>
> - Always or OnFailure: the container is recreated on the same node. The Pod stays on the same node, so if the cause is not resolved, OOM kills will happen again
> - Never: the container is not recreated and the Pod remains Failed. If a controller such as a Job creates a successor Pod, it may be placed on a different node
>
> If OOMKilled repeats even though there is headroom under the limit, suspect this path.
> Note: Because the OOM kill frees memory, the pressure may already be gone when the kubelet next evaluates free memory. In that case the node may get no MemoryPressure condition and no node.kubernetes.io/memory-pressure:NoSchedule taint, and scheduling of new Pods does not stop. Conversely, if the pressure continues, a later evaluation sets the condition and taint, and eviction also occurs. Note that it is decided by memory.available at evaluation time, not by whether an OOM kill happened.
{% enddetails %}
Checking with PSI (Pressure Stall Information)
This is an auxiliary indicator for the case where things are slow although usage has not reached the limit. It shows how long processes were actually stalled waiting for memory reclaim.
{% details How to read PSI and PromQL %}
You can obtain the time a container was stalled waiting for memory allocation as PSI. Even when usage has not reached the limit, you can detect being made to wait by memory reclaim.
| Metric | Emitted by | Description |
|:---|:---|:---|
| container_pressure_memory_waiting_seconds_total | cAdvisor (embedded in kubelet) | Time some processes were waiting for memory (total of some in memory.pressure) |
| container_pressure_memory_stalled_seconds_total | cAdvisor (embedded in kubelet) | Time all processes were stalled waiting for memory (total of full in memory.pressure) |
Note: These metrics are controlled by the KubeletPSI feature gate, which went GA in v1.36 and is enabled by default (it cannot be disabled). The kernel must support PSI, though, and the kubelet decides this by whether cpu.pressure exists at the cgroup root (cadvisor_linux.go). If the metrics do not appear, check whether the kernel was built with CONFIG_PSI=y and whether psi=0 is set as a kernel parameter.
memory.pressure also has avg10 / avg60 / avg300, "the average over the last N seconds (%)", but only the cumulative value (total) is exposed as a Prometheus metric. So use rate() to see a ratio.
The unit is seconds, so the result of rate() is "how many seconds of waiting per second", a ratio from 0 to 1. Multiply by 100 to get a percentage directly.
<!-- PromQL uses Python fencing for syntax highlighting only. -->
# Per Pod, the share of time (%) all processes were stalled waiting for memory
100 * max by (namespace, pod) (
rate(container_pressure_memory_stalled_seconds_total{container="", image="", pod!=""}[5m])
)
# Per Pod, the share of time (%) some processes were waiting for memory
100 * max by (namespace, pod) (
rate(container_pressure_memory_waiting_seconds_total{container="", image="", pod!=""}[5m])
)
When there is plenty of memory, neither increases, so rate() stays at 0. Under pressure, waiting (some) rises first, and if it gets worse stalled (full) follows. waiting counts even when only some processes waited, so to judge severity, use stalled. It is the time all processes in the container were stopped, so if it shows a value, throughput is directly affected.
To see memory pressure on the whole node, use the root cgroup series.
# Share of time (%) the whole node was completely stalled waiting for memory
100 * max by (node) (
rate(container_pressure_memory_stalled_seconds_total{id="/"}[5m])
* on(instance) group_left(node) kubelet_node_name
)
Containers that are struggling although they are below their limit can be found by combining PSI with utilization. container_spec_memory_limit_bytes is the container limit exposed by cAdvisor.
# Containers whose usage is under 80 % of the limit but that are stalled waiting for memory
(
100 * rate(container_pressure_memory_stalled_seconds_total{container!="", image!=""}[5m]) > 1
)
and
(
container_memory_usage_bytes{container!="", image!=""}
/ container_spec_memory_limit_bytes{container!="", image!=""} < 0.8
)
and keeps only series whose label sets match exactly on both sides. A container with no limit has container_spec_memory_limit_bytes of 0, so the division gives +Inf, which does not satisfy < 0.8, and it is excluded automatically.
Here is an example alert. It catches stalled that keeps occurring.
- alert: ContainerMemoryStalled
# Fires when the 5-minute average of fully stalled time is 5 % or more for 10 minutes
expr: |
100 * max by (namespace, pod) (
rate(container_pressure_memory_stalled_seconds_total{container="", image="", pod!=""}[5m])
) > 5
for: 10m
labels:
severity: warning
annotations:
summary: "A Pod is stalled waiting for memory allocation (add the namespace and pod labels with your own template)"
description: "It may be made to wait by reclaim even though it has not reached its memory limit. Check the gap between usage and RSS and the amount of page cache."
The right threshold of 5 % depends on the environment. First look at the distribution of values without an alert to understand the normal level, then decide.
For details on PSI see Understand Pressure Stall Information (PSI) Metrics.
{% enddetails %}
Quick reference: contradictions explained and a metric map
The working set, cgroup usage, process RSS and node availability answer different questions. Use this map to identify the value behind the dashboard before comparing it with a limit or threshold. For a non-root cgroup, working set is max(memory.current - inactive_file, 0). It still includes some reclaimable memory.
| Scope | Metric | Emitted by | Backing value in cgroup v2 / /proc | Page cache handling | Main use |
|:---|:---|:---|:---|:---|:---|
| Container | container_memory_working_set_bytes | cAdvisor (embedded in kubelet) | max(memory.current − inactive_file, 0) | Active pages count as in use | kubectl top pod, usage visualization |
| Container | container_memory_rss | cAdvisor (embedded in kubelet) | anon in memory.stat | Not included (anonymous pages only) | Understanding what the app allocated |
| Container | container_memory_usage_bytes | cAdvisor (embedded in kubelet) | memory.current | Everything included | What memcg OOM is evaluated against |
| Node | container_memory_working_set_bytes{id="/"} | cAdvisor (embedded in kubelet) | anon + file − inactive_file in the root cgroup's memory.stat (see below) | Active pages count as in use | Eviction decisions, kubectl top node |
| Node | node_memory_MemAvailable_bytes | node-exporter | MemAvailable in /proc/meminfo | Reclaimable pages count as free | Visualizing node memory utilization |
| Node | kube_node_status_capacity{resource="memory"} | kube-state-metrics | (status.capacity of the Node object) | (capacity, not usage) | Denominator when converting to utilization |
Looking at the same node, utilization computed from node_memory_MemAvailable_bytes does not match the utilization the kubelet uses for eviction decisions. With workloads that do a lot of file I/O or logging, the difference can reach several GiB.
Different emitters also mean different labels on the metrics. cAdvisor metrics carry no label identifying the node, so when you compute per-node utilization you need to align labels with kubelet_node_name, as shown later.
Kubernetes memory usage and cgroup v2
Why the numbers differ
A process RSS measurement includes its resident mappings, including shared and file-backed pages. Cgroup accounting charges memory to a group, while working set subtracts one category of file cache from usage. An application's heap metric covers a different subset again.
Reclaimability is a separate question: clean file cache can often be reclaimed, dirty pages may need writeback, and anonymous memory normally needs swap or explicit release by the process. Calling free does not necessarily return allocator-held memory to the OS.
These differences explain why heap size, process RSS, cgroup RSS and working set need not match. The following sections connect those values to cgroup v2 accounting.
Memory breakdown in cgroup v2
Container memory usage is managed in a per-container cgroup. In cgroup v2, memory.current is the current usage and memory.stat is its breakdown.
# Example values in a container's cgroup (checked on the node)
cat /sys/fs/cgroup/<container cgroup>/memory.current
Output:
5368709120
cat /sys/fs/cgroup/<container cgroup>/memory.stat
Output:
anon 2147483648 # anonymous pages (heap, stack: memory not backed by a file)
file 3087007744 # page cache (including tmpfs and shared memory)
shmem 268435456 # the part of file that is tmpfs or shared memory
...
inactive_anon 134217728 # anon LRU: not accessed for a while
active_anon 2013265920 # anon LRU: recently accessed
inactive_file 1073741824 # file LRU: not accessed for a while
active_file 1744830464 # file LRU: recently accessed
unevictable 268435456 # pages that cannot be reclaimed (here, the tmpfs part)
...
The following relationships hold in this example. Note that inactive_file + active_file does not equal file; it equals file - shmem. The next section explains why.
anon + file + kernel memory = 5368709120 (memory.current)
inactive_file + active_file = 2818572288 = file - shmem
inactive_anon + active_anon = 2147483648 = anon (shmem is not in here)
unevictable = 268435456 = shmem (tmpfs without swap)
The main fields of memory.stat are:
anon (anonymous pages)
- Memory not backed by a file. Regions the application allocated with malloc and similar belong here.
- Unlike page cache, the kernel cannot reclaim it independently of the process. With swap it can write pages out and free up space, but on nodes that do not use swap it is memory the kernel cannot reclaim under pressure. When swap is unavailable, the kernel sets scan_balance = SCAN_FILE and excludes the anon LRU from scanning (get_scan_count() in mm/vmscan.c).
- On the other hand, this value drops if the process itself returns memory to the OS with munmap or madvise(MADV_DONTNEED). Note that calling free does not reduce it unless the allocator or language runtime returns the memory to the OS.
file (page cache)
- Contents of files held in memory by the kernel during reads and writes. The application did not allocate it explicitly.
- When memory runs short the kernel discards (reclaims) it automatically, so it is basically reclaimable memory.
- Cache of files on disk is split into active_file (recently accessed) and inactive_file (not accessed for a while). Reclaim takes inactive_file first.
- active_file + inactive_file does not equal file. These are totals of the LRU lists, and as described below shmem is on the anon LRU rather than the file LRU. Besides shmem, pages that fall off the LRU, such as mlocked pages, also show up as a difference.
- It includes not only the cache of files on disk but also shmem (tmpfs and shared memory), because Linux treats these as page cache too. shmem is written out to swap rather than to a file, so on nodes without swap it cannot be reclaimed. The breakdown is visible as shmem in memory.stat.
> Warning: shmem is counted in file, but it is not on the file LRU. The kernel classifies swap-backed pages as anon, and tmpfs pages are among them. folio_is_file_lru() in mm_inline.h is defined as "0 if @folio is a normal anonymous folio, a tmpfs folio or otherwise ram or swap backed folio".
>
> The counters they are charged to are different, though. A tmpfs page is added to both NR_FILE_PAGES (= file) and NR_SHMEM (= shmem) (mm/shmem.c):
>
> ```c
> __lruvec_stat_mod_folio(folio, NR_FILE_PAGES, nr);
> __lruvec_stat_mod_folio(folio, NR_SHMEM, nr);
> ```
>
> Furthermore, for a Kubernetes emptyDir with medium: Memory, the kubelet mounts tmpfs with the noswap option when the node supports it (generateTmpfsMountOptions() in empty_dir.go). The kernel makes the inode of a noswap tmpfs unevictable (shmem_get_inode() in mm/shmem.c), so these pages sit on the unevictable LRU, not the anon LRU.
>
> ```c
> if (sbinfo->noswap)
> mapping_set_unevictable(inode->i_mapping);
> ```
>
> So shmem is accounted as follows:
>
> | mount | LRU | Where it shows up in memory.stat |
> |:---|:---|:---|
> | emptyDir with medium: Memory (noswap) | unevictable | unevictable |
> | Any other tmpfs / shared memory | anon LRU | inactive_anon / active_anon |
>
> In neither case does it enter inactive_file / active_file. That matters in two ways later:
>
> - inactive_file + active_file generally corresponds to file - shmem; unevictable pages and asynchronous statistics can cause differences
> - Only inactive_file is subtracted from the working set, so shmem stays in the working set in full
With workloads that write a lot of logs or read and write large files, the page cache can grow to several GiB even though the application did not intend that.
> Note: memory.current includes memory the kernel itself uses (slab, socket buffers and so on) in addition to anon and file. So adding anon and file alone does not give memory.current.
How cgroup v2 memory controls map to Pod settings
cgroup v2 memory control interface
cgroup v2 offers four levels of control over memory usage. memory.min and memory.low are protections ("do not reclaim up to here"), and memory.high and memory.max are limits ("do not allow use beyond here").
| File | Type | Behavior |
|:---|:---|:---|
| memory.min | Protection (hard) | Protects memory within the effective boundary determined by ancestors and usage. It does not preallocate physical memory |
| memory.low | Protection (soft) | Best-effort avoidance of reclaim up to this value. It is reclaimed if nothing else can be |
| memory.high | Limit (soft) | Exceeding it applies reclaim pressure and throttling. Crossing memory.high does not itself invoke the OOM killer; memory.max and global OOM still apply |
| memory.max | Limit (hard) | Exceeding it triggers reclaim, and if usage still does not fit, the cgroup's OOM killer runs |
Other files that report state:
- memory.current: current usage (what container_memory_usage_bytes is backed by)
- memory.stat: usage breakdown
- memory.events: counts of low, high, max, oom, oom_kill and oom_group_kill
- memory.swap.max: swap limit
- memory.pressure: PSI for waiting on memory allocation
For exact definitions of each file see the kernel's Memory Interface Files, and for how Kubernetes treats cgroup v2 see About cgroup v2.
How Pod settings are reflected
Pod settings are reflected into cgroup v2 as follows.
| Pod setting | Target in cgroup v2 |
|:---|:---|
| resources.limits.memory | memory.max |
| resources.requests.memory | Not reflected (with the default kubelet configuration in v1.37; you can reflect it by changing settings → [KEP-2570](#kep-2570-support-memory-qos-with-cgroups-v2)) |
| resources.limits.cpu | cpu.max |
| resources.requests.cpu | cpu.weight |
cgroup v2 is a directory tree. Its top is the root cgroup (/sys/fs/cgroup), where any process that is in no child cgroup belongs, so it contains not only Pods but every process on the node, such as the kubelet and containerd. Kubernetes creates the hierarchy below it: "all Pods on the node → Pod → container".
# With the systemd cgroup driver (the kubeadm default; names differ with the cgroupfs driver, see below)
/sys/fs/cgroup ← root cgroup (id="/")
│ contains every process on the node
│ the value used for eviction decisions
│
├── system.slice/ kubelet, containerd, sshd, etc.
│ ├── kubelet.service (processes that are not Pods)
│ └── containerd.service
│
└── kubepods.slice/ ← all Pods on the node
│ (id="/kubepods.slice")
├── kubepods-pod<UID>.slice/ ← Pod (Guaranteed)
│ ├── cri-containerd-<ID>.scope ← container
│ └── cri-containerd-<pause ID>.scope ← pause container
│
├── kubepods-burstable.slice/ (intermediate slice per QoS class)
│ └── kubepods-burstable-pod<UID>.slice/ ← Pod (Burstable)
│ └── cri-containerd-<ID>.scope ← container
│
└── kubepods-besteffort.slice/
└── kubepods-besteffort-pod<UID>.slice/ ← Pod (BestEffort)
└── cri-containerd-<ID>.scope ← container
When filtering metrics, the root cgroup is id="/", all Pods on the node are id="/kubepods.slice", a Pod is container="", image="", pod!="", and a container is container!="", image!="". What the kubelet uses for eviction decisions is the working set of the topmost root cgroup ([see below](#values-the-kubelet-uses-for-eviction-decisions)). That is a different level from memcg OOM caused by a container or Pod exceeding its limit.
Memory limits are set not only on containers but also on higher levels.
| Level | cgroup | Memory limit |
|:---|:---|:---|
| Whole node (root cgroup) | / | Not set (the root has no memory.max → [see below](#values-the-kubelet-uses-for-eviction-decisions)) |
| All Pods on the node | /kubepods.slice | Capacity - kube-reserved - system-reserved when Pod allocatable enforcement is enabled |
| Pod | Differs by QoS class (see below) | The Pod's effective limit, including Pod-level spec.resources.limits.memory when configured |
| Container | cri-containerd-<ID>.scope under the Pod's cgroup | That container's limits.memory |
The examples here use container-level resources and omit Pod-level resources. Without a Pod-level limit, the required container limits must be present for a finite Pod limit to be set. A Pod's effective limit is not simply the sum of its regular containers' limits.memory. If there are init containers, their maximum is also considered, and spec.overhead is added if set. For the calculation see Resource sharing within containers and Pod Overhead.
So even if a container has not reached its own limit, an OOM kill can occur when the Pod as a whole or /kubepods.slice hits its limit.
> Note: The paths above assume the kubelet uses the systemd cgroup driver. With the cgroupfs driver, paths look like /kubepods/burstable/pod<UID>/<container ID>, with no .slice or .scope and no cri-containerd- prefix on the container.
>
> The kubelet's own default is cgroupfs, but kubeadm sets systemd when the value is empty (defaults.go, kubelet.go). In v1.37, a supported runtime's CRI RuntimeConfig response takes precedence through driver auto-detection. Check the effective driver with:
>
> ```bash
> $ kubectl get --raw "/api/v1/nodes/<node name>/proxy/configz" \
| jq '.kubeletconfig.cgroupDriver'
> "systemd"
> ```
>
> The id label of cAdvisor metrics also changes with the cgroup driver, so when filtering in PromQL, check the value in your own cluster with this query:
>
> ```bash
> group by (id) (container_memory_working_set_bytes{id=~"/kubepods|/kubepods.slice"})
> ```
>
> A Pod's cgroup path depends on its QoS class. As the tree above shows, Burstable and BestEffort go through an intermediate slice per QoS class.
>
> In the <UID> part, systemd escaping replaces the hyphens of the Pod UID with underscores (example: 6983351e-4ede-... → pod6983351e_4ede_...). The UID of a static Pod (mirror Pod) is a hash with no hyphens, so it looks like pod1ff70b81885169da4e9e32c9530b77c9.slice.
>
> The QoS class and UID of a Pod can be read from the Pod resource, and once you have those two you can assemble the cgroup path.
>
> ```bash
> $ kubectl get pod -n <namespace> <Pod name> -o jsonpath='{.status.qosClass}{"\n"}{.metadata.uid}{"\n"}'
> Burstable
> 6983351e-4ede-4611-8bab-b986614af18d
>
> # To list them
> $ kubectl get pods -n <namespace> -o custom-columns='NAME:.metadata.name,QOS:.status.qosClass'
> ```
Container metrics
The cAdvisor built into the kubelet exposes the cgroup values above as Prometheus metrics. They are served from the kubelet's /metrics/cadvisor endpoint, and you do not need to run a separate cAdvisor Pod or DaemonSet.
| Metric | Emitted by | Backing value in cgroup v2 | Description |
|:---|:---|:---|:---|
| container_memory_usage_bytes | cAdvisor (embedded in kubelet) | memory.current | Usage including anonymous pages, page cache and kernel memory. The value memcg OOM is evaluated against |
| container_memory_working_set_bytes | cAdvisor (embedded in kubelet) | memory.current - inactive_file | Usage minus inactive_file, clamped to zero. A heuristic that still includes some reclaimable memory |
| container_memory_rss | cAdvisor (embedded in kubelet) | anon in memory.stat | Amount of anonymous pages. Unlike true RSS, it does not include mmaped file pages (see below) |
| container_memory_cache | cAdvisor (embedded in kubelet) | file in memory.stat | Page cache size. Includes shmem (tmpfs and shared memory) |
| container_memory_total_active_file_bytes | cAdvisor (embedded in kubelet) | active_file in memory.stat | Active page cache |
| container_memory_total_inactive_file_bytes | cAdvisor (embedded in kubelet) | inactive_file in memory.stat | Inactive page cache |
| container_memory_mapped_file | cAdvisor (embedded in kubelet) | file_mapped in memory.stat | Amount of mmaped files |
| `cont
2 comments
Mentions
kubecost 3.x を GKE に入れてみた — 入れる前に知っておきたかった 9 つのこと はじめに Kubernetes のクラスタは、誰がいくら使っているのかが見えにくいです。 クラウドの請求書には「ノード 3 台分」としか書かれておらず、その上で動くアプリごとの内訳は出てきません。 それを見えるようにするツールが kubecost です。検証用の GKE クラスタに入れてみました。 入れること自体は Helm で 1 コマンドですが、既定のまま入れると困る点がいくつもありました。 この記事は、実際に入れて確かめた「入れる前に知っておきたかったこと」のまとめです。 対象は、Kubernetes を触り始めて「requests って何となく書いている」くらいの方です。requestsは後述で説明します。 確かめた環境 項目 内容 クラスタ GKE（検証用）。手元の確認には minikube も使用 kubecost Helm chart kubecost/kubecost 3.2.4、無償版（Foundations） 見る方法 まずは kubectl port-forward。その後 Kong 越しの公開を準備 まず、何が見えるのか namespace ごとのコスト Monitor/Allocatiomsにnamespace（クラスタの中の区画）ごとに、CPU・メモリ・ディスク・ロードバランサーのコストが並びます。 注目してほしいのは idle の行です。これは「ノードとして借りているのに、どの Pod にも割り当てられていない分」です。 このクラスタでは合計 $16.83 のうち $8.71 が idle でした。半分以上が、誰も使っていない分の料金です。 何にお金がかかっているか Monitor/Assetsでクラウドの資源の内訳を見ることができます。ほとんどがノード代だと分かります。 どこを削れるか Savingsでは削減の候補が金額つきで並びます。一番大きいのは「Right-size container requests」、つまりコンテナの requests の見直しでした。 ! requests とは requests は「このコンテナはこれだけ CPU とメモリを使う予定です」という申告です。 Kubernetes はこの申告をもとに Pod をノードに詰めます。申告が実際より大きいと、使っていない分までノードを押さえてしまい、その分の料金を払うことになります。 コンテナごとに、今の申告（current）と、実際の使用量から計算した推奨値（rec）が出ます。 この画面は記事の最後でもう一度使います。 ! なお、入れてまだ 2 日目のスクリーンショットなので、数字は粗いです。 入れる前に知っておきたかった 9 つのこと 1. 2.x と 3.x は別物。検索で出てくる情報の多くは 2.x 最初につまずいたのがここです。 2.x 3.x Helm chart の名前 kubecost/cost-analyzer kubecost/kubecost Prometheus 必須 不要 データの置き場 Prometheus ClickHouse（同梱） 2.x の公式ドキュメントには、Prometheus は省略できず、無効にするとコストが 0 になる、と書かれています。 検証用クラスタにはPrometheusサーバがない（外部送信のみ）ため、同時構築が必要な2.xではなく3.xを選びました。 ! chart の名前もリポジトリの URL も違うので、2.x の記事の手順をそのまま 3.x に使うことはできません。 また、2.x の最新である 2.9 は 3.x への移行準備用の版で、新規に利用する場合は適していません（helm template がエラーで止まります）。 2. 既定のまま入れると、強い権限を持つ部品が一緒に入る chart を既定値で展開すると 8 つの部品が出てきます。そのうち 2 つは、入れる前に権限を確認した方がよいものでした。 部品 持っている権限 何のための部品か networkCosts ノードのネットワークを直接使う特権コンテナ。全 namespace の Secret を読める 通信量ごとのコスト clusterController クラスタの資源を作成・変更・削除できる 推奨値を自動で適用する Secret には、データベースのパスワードなどが入っています。コストを見るツールに、そこまでの権限を渡す必要はありませんでした。 この 2 つを無効にすると、残る権限は読み取りだけになります。 networkCosts: enabled: false clusterController: enabled: false ! 「無効にすると requests の推奨値が出なくなるのでは」と最初は思いましたが、出ました。 clusterController は推奨値を適用するための部品で、表示するだけなら要りません。 3. ディスクを既定で 161Gi 要求する。消しても残る デフォルトのままだと、永続ディスクを合計 161Gi を必要とします。検証用には大きすぎるので、8Gi まで削りました。 ! もう 1 つ、helm uninstall してもディスク（PVC）が消えません。消さずに残す印が chart 側で付いているためです。 試しに入れて消す、を繰り返すと、使われないディスクが課金されたまま残ります。消すときは PVC も手で消します。 kubectl get pvc -n kubecost 4. 既定では、利用状況を外部に送信する デフォルトでは、利用状況の送信とログの収集が有効です。社内のクラスタに入れるなら、先に決めておく項目です。 telemetry: enabled: false reporting: logCollection: false 無効にした状態でブラウザの通信を見たところ、画面からの通信はすべて kubecost 自身に向いていました。 5. Spot を使っているなら、ラベルを教えないと高く見積もられる Spot（クラウドの余った計算資源を安く借りる仕組み。GKE では Spot VM）のノードは、通常より大幅に安いです。 kubecost は「このノードは Spot か」をノードのラベルで判断します。どのラベルを見るかを教えないと、通常価格で計算され、コストがずっと高く出ます。 GKE の場合はこうです。 finopsagent: agent: kubecost: spotLabel: cloud.google.com/gke-spot spotLabelValue: "true" このラベルはクラウドによって違います。同じクラウドでも、ノードの増やし方（Karpenter を使うかどうか）で変わります。 6. 手元のクラスタで数字が出ても、正しいとは限らない minikube（手元の PC で動かす Kubernetes）でも、コストの数字は出ます。 ただし、これは実際の料金ではなく、kubecost が持っている仮の単価で計算したものです。 ! 「手元で数字が出た」は「動いた」の確認にはなりますが、「金額が合っている」かの確認にはなりません。 金額の確認は、実際のクラウドの上で、請求と見比べて行います。 7. 無償版でできないことは、API が教えてくれる 無償版（Foundations）には制限があります。どこまで使えるかは、実際に叩くのが確実でした。 やったこと 結果 30 日ぶんのデータを要求 拒否。上限は 15 日（360 時間） ResourceQuota の推奨値 拒否（有償版が必要） コンテナの requests の推奨値 使える kubecost 自身のログイン機能（OIDC） 使えない（有償版が必要） 最後の行が次の項目につながります。 8. アラートは「壊れたときだけ」届く kubecost 自身の状態を見張って Slack に知らせる機能（health アラート）があります。 設定に「1 時間ごと」とあったので、1 時間ごとに通知が来ると思っていましたが、来ませんでした。 1時間ごとに毎回通知が来るわけではなく、通知は『異常があった時だけ』です。 通知の経路が生きているかは、Alerts 画面の「Test Alert」ボタンで確かめます。 ! 「通知が来ない」を「壊れている」と読んでも、「正常」と読んでも間違えます。確かめる手段を別に持っておくのが大事でした。 9. 画面にログイン機能が無い。公開するなら手前で守る ! 無償版の kubecost には、ユーザーもロールもありません。URL に届いた人は全員、すべての画面と API を使えます。 port-forward で自分だけが見るなら問題ありませんが、チームで見るために公開するなら、手前に門番が要ります。 私たちは API ゲートウェイの Kong を手前に置き、4 種類の守りを重ねる設計にしました。(これから実装していく予定です！） 守り 見るもの IP 制限 どこから来たか ログイン（Keycloak） 誰か ロールの検査 管理者か 特定パスの拒否 入れた人にも見せないもの 公開前に確かめたところ、穴が 2 つ見つかりました。1 つ目（IP 制限が 3 回に 1 回外れる）は前の記事に書きました。ここでは 2 つ目を書きます。 拒否したはずのパスが %2F ですり抜けた kubecost の API の中に、Slack の webhook の URL を返すものがあります（/model/v3/alerts 以下）。 webhook の URL は、知っていれば誰でもそのチャンネルに投稿できる合鍵です。管理者にも見せる必要はないので、このパスだけ Kong で拒否しました。 rules: - matches: - path: {type: PathPrefix, value: /model/v3/alerts} 叩くと 403 が返りました。ここで終わりにするところでしたが、形を変えて試しました。 リクエストのパス 結果 /model/v3/alerts/global/slack 403 //model//v3/alerts/global/slack 403 /model/x/../v3/alerts/global/slack 403 /model%2Fv3/alerts/global/slack 200。webhook が返る /model/x%2F..%2Fv3/alerts/global/slack 200。webhook が返る %2F は / を別の書き方にしたもの（URL エンコード）です。 原因は、同じ文字列を 2 者が別の読み方をしていたことでした。 Kong は、照合の前にパスを整えます（// をまとめる、../ を畳む）。ただし %2F は / に戻しません。URL の決まりでは、エンコードされた / は「区切りではない、ただの文字」を意味するからで、Kong の動きは仕様どおりです kubecost の画面側の nginx は、%2F を / に戻してから中継先を決めます どちらも単体では間違っていません。検査する者と使う者で読み方がずれていたことが穴でした。 直し方は、%2F を含むパスをまとめて拒否する条件を足すことでした。 rules: - matches: - path: {type: PathPrefix, value: /model/v3/alerts} - path: {type: RegularExpression, value: "/.%2[Ff]."} これで全部 403 になり、通常の画面は壊れませんでした。ここまでは手元の minikube（Kong 3.9）で確かめた内容です。 ! 正常な形で 403 が出るのは「設定が書けた」の確認です。「塞いだ」の確認は、形を変えた入力でやります。 「ほかに漏れる API は無い」の確かめ方 webhook を返す API がほかに無いかも確かめました。kubecost のバイナリからパスらしい文字列を 32,329 本取り出し、全部叩きました。 このとき、本物の webhook は使いません。検査のログに本物の秘密が残るからです。 かといって何も設定せずに叩くと、0 件でも「漏れない」のか「返すものが無いだけ」なのか区別できません。 そこで、一目で偽物と分かる文字列を webhook として設定し、先に「漏れると分かっているパス」で検出できることを確かめてから総当たりしました。 CANARY=CANARYWEBHOOK0001 先に確認: 漏れると分かっているパスで見つからなければ、検査の方が壊れている curl -s "${BASE}/v3/alerts/global/slack" | grep -q "${CANARY}" \ || { echo "検査が機能していない" >&2; exit 1; } while read -r p; do curl -s "${BASE}${p}" | grep -q "${CANARY}" && echo "LEAK ${p}" done < candidates.txt 返したのは 2 本だけで、どちらも拒否済みのパスの下でした。 ! 副産物として、Helm の設定値を丸ごと返す API も見つかりました。 設定値に秘密を直接書いていたら、そこから読めていたことになります。秘密は設定値に書かず Secret で渡す、という決まりの理由が 1 つ増えました。 見えた。では、どう直すか 最後に、requests の推奨値の画面に戻ります。 このクラスタでは、申告が多すぎるものと、足りないものの両方がありました。 コンテナ メモリの申告（current） 推奨値（rec） 状態 データベース層のアプリ 4Gi 568Mi 多すぎる kubecost の収集役 512Mi 244Mi 多すぎる Kong 256Mi 1310Mi 足りない Keycloak 512Mi 819Mi 足りない 多すぎる申告はお金の無駄になり、足りない申告は、ノードが混んだときに Pod が追い出される原因になります。 コストを見るつもりで入れたツールが、安定性の問題も見つけてくれました。 ただし、kubecost が出すのは推奨値の表示までです。 そして、この数字を人が読んで設定ファイルを書き換える運用は、アプリが増えると続きません。使用量は日々変わるからです。 そこで次に試したいのが、Kubernetes の 2 つの自動調整の仕組みです。 仕組み 何を変えるか 向いている場面 VPA（Vertical Pod Autoscaler） 1 つの Pod の requests を、実際の使用量に合わせて上げ下げする 申告が実態とずれているとき HPA（Horizontal Pod Autoscaler） Pod の数を、負荷に合わせて増減する 時間帯で負荷が変わるとき VPA を入れれば、多すぎる申告が自動で下がり、その分ノードを減らせるかもしれません。足りない申告も自動で上がります。 HPA を入れれば、負荷の低い時間帯は Pod を減らしておけます。 どちらも、kubecost が見せてくれた「ズレ」を、人の手を借りずに直し続けてくれる仕組みです。 ! 本当にコストが下がるのかは、まだ試していません。 次は VPA と HPA を入れて、kubecost の数字が入れる前と後でどう変わるかを見てみるつもりです。 まとめ kubecost は 2.x と 3.x で構成が違う。Prometheus が無いクラスタなら 3.x 既定のまま入れない。権限の強い部品、大きなディスク、外部への送信を先に確認する Spot のラベルを教えないと、コストが高く出る 手元で数字が出ても、金額が正しい証拠にはならない 無償版にはログイン機能が無い。公開するなら手前で守り、形を変えた入力で確かめる 推奨値は表示まで。VPA と HPA を入れればコストも下げられるかもしれないので、次に試す 参考文献 Kubecost Core Architecture Overview（3.x） — 3.x の部品の構成 Kubecost Core Architecture Overview（2.x） — 2.x は Prometheus が必須であること kubecost/kubecost（Helm chart） — 3.x の chart と既定値 OpenCost — kubecost のコスト計算の元になっているオープンソース Resource Management for Pods and Containers（Kubernetes） — requests と limits の意味 HTTPRoute（Gateway API） — パスの照合方法（PathPrefix / RegularExpression） Route（Kong Gateway） — Kong のルートの照合とパスの扱い Request Termination プラグイン（Kong） — その場で決まった応答を返すプラグイン ngx_http_core_module の location（nginx） — 照合は %XX をデコードした後のパスで行われること RFC 3986 Uniform Resource Identifier — 予約文字と、エンコードされた予約文字の扱い（2.2 節、6.2.2 節） Autoscaling Workloads（Kubernetes） — VPA と HPA の全体像 Horizontal Pod Autoscaling（Kubernetes） Vertical Pod Autoscaler（kubernetes/autoscaler） Vertical Pod Autoscaling（GKE） 執筆者紹介 / 会社情報 執筆者 Higu BREXA SOLVIA SES アサイン先 株式会社Scalar 最近は Terraform・AWS・GCP・Kubernetes を勉強しています。わからないことだらけですが少しずつ整理しながら書いています！ 会社情報 Scalar, Inc は、「データマネジメントの未来を創る」を掲げ、データベースミドルウェアの研究開発・販売を行う日本発のグローバルスタートアップ企業です！複数の異なるデータベース（RDB、NoSQL など）を仮想的に統合し、トランザクション処理や分析クエリを実現するデータベースミドルウェアである ScalarDB、分散台帳技術（DLT）を用いたソフトウェアで、データの改ざん検知を行う ScalarDL を主な製品としています。是非下記の弊社サイトもチェック📝をお願いします！ 🔧 Product 🧑💻 開発者ブログ 🏢 コーポレートサイト
Kubernetes cost optimization for startups --- id: 4621832 title: Kubernetes cost optimization for startups published: true description: 'The 2026 playbook to cut Kubernetes bills: right-sizing, spot, idle sleep, quotas, image pulls, NAT routing, and the change that compounds.' tags: 'kubernetes, aws, devops, cost' canonical_url: 'ownkube.io' cover_image: 'ownkube.io' --- What surprises people first about a startup's Kubernetes bill is how predictable the leaks are. Across the small-team AWS and GCP bills we audit, the same seven patterns drive 70 to 90% of the waste, and they're all fixable in a single quarter. This post is the consolidated playbook. It covers the seven patterns, the realistic savings on each, and the structural change that keeps them from drifting back: making the platform carry the defaults. Skim answer: - The seven highest-leverage patterns: right-sizing, spot capacity, idle environment sleep, namespace quotas, image-pull traffic reduction, NAT-routing audits, and storage class right-sizing. - Combined impact: typically cuts a small-team Kubernetes bill by 40 to 65%. - Timeline: all seven are fixable in a single quarter. Why startup K8s bills are usually 2x what they should be The math behind the typical waste: - Most teams set resource requests at 2x to 4x what the workload actually uses, "to be safe". Cluster autoscaler then provisions nodes for the inflated request rather than the real usage. - Idle preview environments and staging clusters run 24/7 even though they're used 30 hours a week. - Container images get pulled from public registries through NAT gateways, racking up data-processing fees nobody sees. - EBS volumes are provisioned at default gp2 sizes that exceed real I/O needs by an order of magnitude. Each of these is a small, steady bleed. Add up enough of them and the cluster bill is double what it should be. Pattern 1: Right-size every workload This is the largest lever. Most workloads we see have resource requests set during initial deployment and never revisited. A typical web pod might have a 1 vCPU / 1 GB RAM request and a 95th-percentile usage of 0.18 vCPU / 240 MB RAM, so the other 80% of the request is reservation paid for and unused. To fix it, run Vertical Pod Autoscaler in recommendation mode for one to two weeks, then apply the recommendations. For long-tail workloads with bursty traffic, use VPA in Auto mode with sensible min/max bounds. Expect to save 25 to 45% of cluster compute. It's the biggest line-item improvement on every audit. Pattern 2: Spot capacity for the workloads that tolerate it Stateless web pods, queue consumers, build runners, batch jobs, and preview environments are excellent candidates for spot. Database primaries, control plane nodes, and single-replica stateful services should stay on-demand. We covered the full pattern in AWS spot instances in production. To fix it, use mixed instance pools with an on-demand base and spot for the rest. Karpenter or Cluster Autoscaler with mixed ASGs handles the orchestration. Set PodDisruptionBudgets and 30 to 120 second graceful shutdown windows. Expect 50 to 70% off the spot-eligible portion of the compute bill. For a typical small-team workload, that's roughly 30 to 50% off the total cluster compute. Pattern 3: Sleep idle environments Staging, preview, and developer sandbox environments sit idle most of the week. A preview environment created on Monday for a PR that merges Friday runs for ~120 hours while being used for maybe 4. The other 116 hours are waste. To fix it, add an idle-detection loop that scales preview deployments to zero after N hours of no traffic, then scales back up on the next request. Tools like KEDA, scaler-controller, or platform-layer features handle this. Expect to save 50 to 75% of preview environment cost. For teams with 5 to 20 active previews at any time, this can be the second-biggest single-pattern saving. Pattern 4: Namespace quotas and request governance Without ResourceQuota on each namespace, any service team can ship a deployment that requests more than it needs. Quotas force the conversation: "you want 8 vCPU for this worker, defend it." To fix it, pin a ResourceQuota per namespace to a realistic budget, add a LimitRange to set sensible default requests for pods without explicit requests, and revisit the quotas in a monthly review. Expect 10 to 20% in indirect savings through culture change. The bigger win is preventing future drift. Pattern 5: Pull images from a private registry Most teams pull container images from Docker Hub, GitHub Container Registry, or Quay through a NAT gateway. Each pull is a few hundred MB. With a busy CI fleet plus rolling production deploys, that's tens of GB per day routed through NAT at $0.045 per GB. To fix it, mirror your base images to a private Amazon ECR registry (or GCP Artifact Registry), enable the ECR VPC interface endpoint, and configure your image pull policy to use the mirror. Image pulls then stay on the AWS backbone at near-zero per-GB cost. Expect to save $50 to $400 per month for a typical small team, and more for orgs that deploy often. Pattern 6: Audit NAT routing NAT gateway data-processing fees are the most under-noticed cost on AWS bills. We covered the full pattern in our NAT gateway cost guide. The short version: - Enable S3 and DynamoDB gateway endpoints (free) on every VPC. - Add interface endpoints for high-volume AWS services (Secrets Manager, STS, CloudWatch Logs). - Dual-stack your VPC and route IPv6 traffic through a free egress-only IGW. - Reduce NAT topology from 3 AZs to 2 if availability tolerates. Expect to save $200 to $1,500 per month depending on cluster traffic. Pattern 7: Storage class right-sizing EBS gp3 has replaced gp2 as the sensible default for most cluster volumes. It's cheaper and faster, and you pay only for the IOPS you provision (on gp2, IOPS scale with volume size). To fix it, migrate from gp2 to gp3 across your cluster and right-size volume capacity (most workloads provision 100 GB when they use 12 GB). For workloads with very low I/O, consider sc1 or st1 for cold data. Expect to save 20 to 35% on EBS line items. The structural change that compounds them all Each of the seven patterns above is a one-time engineering project, and implementing them all takes 2 to 4 weeks of focused work for a small team. The harder question is what stops the drift from coming back six months later. Without an explicit owner, nothing does. Request inflation creeps back in, new services get deployed without quotas, new environments forget the sleep schedule, new container pulls go through NAT, and the cluster bill returns to its old shape within two quarters. The structural fix is to have the platform carry these decisions instead of leaving them to tribal memory. On Ownkube, running inside your own cloud account, the platform ships with the primitives that keep these patterns from drifting: - Managed Postgres, so database sizing and backups aren't a manual task per environment. - Horizontal autoscaling on every workload, so requests track real traffic instead of a number set at first deploy and never revisited. - Built-in logs and metrics, so a right-sizing pass is a five-minute look at a dashboard instead of a new observability rollout. - One-command rollbacks, so a bad change costs you a rollback instead of an incident. You still own the architectural decisions. Ownkube keeps the primitives in place so the drift has fewer places to hide. A worked example Take a typical 2026 SaaS running Kubernetes on AWS: 1 cluster, ~24 vCPU production fleet, 8 vCPU staging, 10 active preview environments, RDS Postgres, ElastiCache. Before optimization, it pays ~$2,400/month cluster compute + $180 NAT + $90 EBS = $2,670/month. Here is what each pattern saves: | Pattern | Saving | |---|---| | Right-sizing (Pattern 1) | -$840 | | Spot capacity (Pattern 2) | -$520 | | Idle environment sleep (Pattern 3) | -$280 | | Namespace quotas + governance (Pattern 4) | -$120 | | Private image registry (Pattern 5) | -$140 | | NAT routing audit (Pattern 6) | -$110 | | Storage class right-sizing (Pattern 7) | -$30 | | Total saved | -$2,040 | After all seven, the bill is ~$630/month, about a 76% reduction on the cluster bill. These numbers come from the workload defined above, so your savings will differ with yours. Decision checklist Before you start, confirm: - [ ] Do you have a way to measure current per-workload resource usage (Prometheus, CloudWatch, Datadog)? - [ ] Is your cluster on Kubernetes 1.27+ (so VPA and Karpenter work cleanly)? - [ ] Do you have at least one engineer with a couple of weeks of focused time? - [ ] Is anyone going to own ongoing cost vigilance after the initial pass, or do you need that on autopilot? If you ticked all four, you're set up to do the work in-house. If you ticked three or fewer, consider a platform layer that runs these patterns by default. Closing Kubernetes cost optimization at a small startup comes down to seven well-understood patterns plus one structural change to stop the drift from coming back: making the platform itself carry the defaults instead of leaving them to memory. If you'd rather skip the initial work and start with a platform that ships those defaults, Ownkube runs inside your own AWS account. Starter is free forever on one instance, and Production is $5 per vCPU plus $1 per GiB RAM per month once you scale. Deploy my first app.
Kubernetes Memory on cgroup v2: Debugging OOM Kills and Evictions A container is OOMKilled, but kubectl top never showed it reaching its limit. Or a Pod is evicted while your node dashboard still shows free memory. In both cases, the first question is which memory value the decision actually used. This guide separates cgroup-limit OOM, node-side OOM and kubelet eviction, then shows how to inspect the evidence and review memory requests and limits. It assumes familiarity with Pod resources and basic PromQL. TL;DR - The working set is a heuristic, not a measurement of unreclaimable memory. It subtracts inactive_file from usage but still includes reclaimable active page cache and slab. - Distinguish memcg OOM from global OOM using kernel logs. A container can be below its own limit when an ancestor cgroup reaches its limit or a node-side allocation fails. OOMKilled and SystemOOM alone cannot establish the cause. - Eviction uses the kubelet's memory.available, not Linux MemAvailable. Check the kubelet's statistics and configured thresholds. - Review requests and limits separately. Requests influence placement and victim selection. To address a cgroup limit, reduce usage or adjust the limit after checking node capacity. - Check Memory QoS configuration as well as its feature gate. In v1.37, the gate defaults to enabled, but request-based protection and memory.high remain disabled by default. This article targets cgroup v2, Kubernetes v1.37.1 and containerd. See [Components and versions](#components-and-versions) for exact versions. The kind examples use a v1.37.0 node image. Use the [investigation workflow](#investigation-workflow) for an incident, the [metric map](#quick-reference-contradictions-explained-and-a-metric-map) to understand a dashboard, or [requests and limits](#what-to-do-about-it) for prevention. The [kind examples](#hands-on-verification-with-kind) explore reclaim and Memory QoS in more detail. Identify the failure before choosing a metric | Observation | Mechanism | Evidence to check | |:---|:---|:---| | Container termination reason is OOMKilled | The kernel killed a process; this alone does not distinguish the OOM source | Kernel logs, container and ancestor cgroup limits, node allocation constraints | | Pod is Evicted with The node was low on resource: memory. | The kubelet responded to node memory pressure | Kubelet memory.available, configured thresholds and eviction message | | Exit code is 137 without an OOM reason | The process received SIGKILL; the exit code alone does not establish OOM | Termination details, events and kernel logs | memcg OOM means an allocation cannot fit within a cgroup limit after reclaim. Global OOM means a node-side allocation cannot be satisfied after reclaim, potentially within NUMA or cpuset constraints. A container below its own limit can be affected by either an ancestor limit or a node-side failure. Investigating OOM and eviction Investigation workflow Start with the observed termination or eviction, then compare metrics from the same time. Scrapes can miss the peak; kernel logs are stronger evidence of the OOM source than a sampled usage graph. 1. When a Pod was OOMKilled - Check whether container_memory_usage_bytes is close to the limit. memcg OOM is evaluated against this value - Compare with container_memory_rss to tell whether anonymous pages are the main cause. An OOM can occur even when RSS is low - If the gap between the working set or usage and RSS is large, page cache, shmem or kernel memory may be pushing usage up - If OOMKilled occurs despite headroom under the limit, suspect global OOM. Correlate SystemOOM with kernel logs and inspect ancestor limits. A missing event does not establish memcg OOM - If you see no spike in the metrics, suspect a surge shorter than the scrape interval. See [My working set is below the limit, but the container was OOMKilled](#my-working-set-is-below-the-limit-but-the-container-was-oomkilled) for what to check 2. When a Pod was evicted - If the message is The node was low on resource: memory., node-wide memory pressure is the cause - Use the [node-utilization query](#why-node-exporter-and-kubelet-usage-disagree) to check the kubelet-view node memory utilization. The node-exporter-based utilization alone is not enough to judge - Check whether a Pod with a small (or unset) memory request relative to actual usage is on the same node. Compare each Pod's container_memory_working_set_bytes (cAdvisor) against kube_pod_resource_request{resource="memory"} (kube-scheduler). Note that Pods with large requests make the scheduler keep placement density low, so they are unlikely to be the cause of pressure 3. To prevent recurrence - Set a memory request informed by workload history to avoid excessive placement density. PromQL for checking is collected in [Align memory request with actual usage](#align-memory-request-with-actual-usage) - For containers where memcg OOM occurs, also review the limit - Control placement with Pod anti-affinity or topology spread constraints so that Pods with large memory usage do not gather on the same node Telling which path occurred Using Kubernetes information Start by narrowing down with what you can check using kubectl. | Where to look | What it tells you | |:---|:---| | Last State in kubectl describe pod, or lastState.terminated in kubectl get pod -o yaml | That an OOM kill happened (reason: OOMKilled). It cannot distinguish the path | | Events in kubectl describe node <node name> | Use SystemOOM as a clue and verify the source in kernel logs. If there is no record, that does not make it a memcg OOM; it just means you cannot tell | There is no OOM-specific event on the Pod side. You only see things like BackOff accompanying the restart. Check it as follows. SystemOOM is an event attached to the Node, and since Node is not a namespaced object, the event itself is recorded in the default namespace. # 1. Find which node the OOMKilled Pod is on kubectl get pod -n <namespace> <Pod name> -o wide Output: NAME READY STATUS RESTARTS AGE IP NODE myapp-... 1/1 Running 5 (2m ago) 1h 10.0.0.1 node-1 # 2. Check whether that node's events contain SystemOOM kubectl describe node node-1 Output: ... Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning SystemOOM 3m kubelet System OOM encountered, victim process: java, pid: 12345 # To list only events kubectl get events -n default \ --field-selector reason=SystemOOM,involvedObject.name=node-1 If SystemOOM exists, inspect kernel messages from the same time: constraint, oom_memcg, task_memcg and the killed PID. A parsing failure can also produce this event. Match the event and kernel log to the container's termination time (lastState.terminated.finishedAt). Not finding SystemOOM is not evidence of memcg OOM. The same state results from event TTL expiry, a kubelet restart, or a missed event. The event retention period is set by kube-apiserver's --event-ttl, and the default is 1 hour (options.go). So an old SystemOOM may no longer be available through kubectl. If you forward events to an external logging platform, check there. On the node, inspect the kernel log: sudo journalctl -k -b -o short-iso \ | rg 'oom-kill:|Out of memory:|Memory cgroup out of memory:|Killed process' CONSTRAINT_MEMCG and oom_memcg identify the limiting cgroup; task_memcg identifies the victim's cgroup. If these differ, inspect the Pod and ancestor limits. CONSTRAINT_CPUSET and CONSTRAINT_MEMORY_POLICY call for checking NUMA and cpuset restrictions. Align timestamps with the container's finishedAt. If logs are unavailable, use the following as supporting evidence rather than proof: - Whether container_memory_usage_bytes was close to the limit at that time (if so, memcg OOM is likely) - Whether [the kubelet-view node memory utilization](#why-node-exporter-and-kubelet-usage-disagree) was high at that time (if so, global OOM is likely) Why OOMKilled cannot distinguish the paths When a task exits with code 137, the container runtime (containerd) checks oom_kill in the cgroup's memory.events, and if it is 1 or more, it updates the termination reason to OOMKilled (events.go). The check is oomMetricsEventOccurred() in the same file. switch v := taskMetricsAny.(type) { case *cg1.Metrics: return v.GetMemoryOomControl().GetOomKill() > 0, nil case *cg2.Metrics: return v.GetMemoryEvents().GetOomKill() > 0, nil The definition of oom_kill in memory.events is "The number of processes belonging to this cgroup killed by any kind of OOM killer". So a process chosen and killed by the kernel in a global OOM is counted too, and both paths produce OOMKilled. {% details How the kubelet parser produces SystemOOM %} When SystemOOM is recorded SystemOOM is an event the kubelet records on the Node object. The kubelet parses OOM messages in the kernel log and records it only when the cgroup where the OOM happened is judged to be the root (/) (oom_watcher_linux.go). for event := range outStream { // Count every OOM kill per container to back the // container_oom_events_total metric. recordOOMKill(event.ContainerName) if event.VictimContainerName == recordEventContainerName { ... ow.recorder.WithLogger(logger).Eventf(ref, v1.EventTypeWarning, systemOOMEvent, "%s", eventMsg) } } The usual association with global OOM comes from the combination of the kernel log format and the parser. For an OOM caused by a memcg limit, the kernel prints oom_memcg=<cgroup path>, whereas for a global OOM it prints ,global_oom instead (mem_cgroup_print_oom_context() in mm/memcontrol.c). if (memcg) { pr_cont(",oom_memcg="); pr_cont_cgroup_path(memcg->css.cgroup); } else pr_cont(",global_oom"); The modern-format regular expression in the kubelet's forked parser matches the format containing oom_memcg= (oomparser_linux.go). containerRegexp = regexp.MustCompile(`oom-kill:constraint=(.*),nodemask=(.*),cpuset=(.*),mems_allowed=(.*),oom_memcg=(.*),task_memcg=(.*),task=(.*),pid=(.*),uid=(.*)`) So no cgroup can be extracted from a global OOM message, and the initial value / of OomInstance is left as is (StreamOoms in oomparser_linux.go). oom := &OomInstance{ ContainerName: "/", VictimContainerName: "/", TimeOfDeath: msg.Timestamp, } In other words, the presence of SystemOOM is not "the result of directly determining the path" but "the result of whether the kernel log could be parsed". Still, in practice it is enough to understand that a recorded event is a reason to inspect node-side OOM, while neither presence nor absence establishes the source. {% enddetails %} How it appears in metrics Metrics do not distinguish the path with a label. The kernel's __oom_kill_process() records the same event regardless of the path. | Metric | Emitted by | memcg OOM | global OOM | |:---|:---|:---|:---| | node_vmstat_oom_kill | node-exporter (oom_kill in /proc/vmstat) | Increases | Increases | | container_oom_events_total | kubelet (result of kernel log parsing) | The series of the container the killed process belonged to increases | The id="/" series increases, not a container series | | oom_kill in cgroup v2 memory.events | (a cgroup file, not a metric; containerd reads it) | Increases | Increases | Only container_oom_events_total behaves differently because, as described earlier, the parser cannot extract a cgroup in a global OOM and treats it as the root (/). This is only a result of whether the log could be parsed, so do not use it to determine the path; confirm the source from constraint and oom_memcg in the kernel log. > Note: Although container_oom_events_total has container_ in its name, in v1.37 it is counted by the kubelet, not cAdvisor (recordOOMKill() in oom_counter.go). The meaning of the value does not change, but you have to look elsewhere when reading the code. What to look at after narrowing down - If you judge it to be a global OOM, inspect node-side memory pressure and any NUMA or cpuset constraints. Compare the memory requests and actual usage of the Pods placed on the same node, and check whether any Pod has too small a request - If you judge it to be a memcg OOM, review that container's memory limit and its actual usage. If the container seems to have headroom under its limit, it may have reached the limit of the Pod as a whole or of /kubepods.slice - If you cannot decide, keep both possibilities open and check both the container's limit and usage and the node's memory utilization {% details Kernel OOM behavior and victim selection %} The two OOM kill paths There are two paths by which a container's process gets OOM killed. Both are SIGKILLs from the kernel's OOM killer, but the conditions under which they occur and the scope they affect differ. As [defined in the terminology notes](#how-this-article-names-the-two-oom-kill-paths), memcg OOM enforces a configured limit and global OOM protects the node, so their purposes differ at the root. | Path | Condition | What gets killed | Node event | |:---|:---|:---|:---| | memcg OOM | memory.max exceeded for a container, Pod or /kubepods.slice | Processes inside the cgroup that hit its limit | Not recorded | | global OOM | A node-side allocation cannot be satisfied after reclaim, possibly within NUMA or cpuset constraints | Chosen from eligible processes under the allocation constraints | SystemOOM is recorded when the kubelet parses the log and treats the triggering cgroup as / | Even if your container is under its limit, it can be OOM killed because of another Pod on the same node. That is global OOM. SystemOOM is a kubelet-generated clue to investigate, not a definitive classification of the OOM source. Events expire according to the API server's --event-ttl (one hour by default), so a missing event does not rule out global OOM. OOM kill from exceeding a cgroup limit (memcg OOM) resources.limits.memory is set as the container cgroup's memory.max. When a cgroup's memory.current reaches memory.max, this is what happens. As described above, limits are set not only on containers but also on Pods and /kubepods.slice, and the same flow applies whichever level reaches its limit. 1. The kernel tries to reclaim memory inside that cgroup. Page cache is basically reclaimed here. 2. If usage still does not fit under memory.max after reclaim, the cgroup's OOM killer SIGKILLs processes in that cgroup. 3. With cgroup v2, the kubelet defaults to singleProcessOOMKill: false and sets memory.oom.group=1 for the container. A memcg OOM therefore normally kills the container's processes together. With singleProcessOOMKill: true, a child process can be killed while the container keeps running. Container restarts follow restartPolicy. See the implementation. The implementation is try_charge_memcg() in mm/memcontrol.c. It is a goto loop returning to the retry: label, and proceeds in this order. | Step | Location | What happens | |:---|:---|:---| | 1. Try to charge | L2177-L2178 | If page_counter_try_charge() succeeds, the allocation completes and it returns | | 2. Reclaim | L2211-L2216 | On failure it reclaims with try_to_free_mem_cgroup_pages(), and if mem_cgroup_margin() satisfies the request, goes back to 1 | | 3. Count the retry | L2244-L2245 | If reclaim is not enough, it decrements nr_retries and goes back to 1. The initial value is MAX_RECLAIM_RETRIES (= 16; mm/internal.h) | | 4. Invoke the OOM killer | L2259-L2264 | Only after the retries are used up does it call mem_cgroup_oom() | MAX_RECLAIM_RETRIES is a retry budget, not a guarantee of exactly 16 reclaim attempts. Allocation flags and reclaim progress change the control flow. The check in step 2 looks like this in code. nr_reclaimed = try_to_free_mem_cgroup_pages(mem_over_limit, nr_pages, gfp_mask, reclaim_options, NULL); psi_memstall_leave(&pflags); if (mem_cgroup_margin(mem_over_limit) >= nr_pages) goto retry; The kernel documentation for memory.max says the same. > Memory usage hard limit. ... If a cgroup's memory usage reaches this limit and can't be reduced, the OOM killer is invoked in the cgroup. > Warning: memory.current reaching memory.max does not by itself cause an OOM. A cgroup sitting at its limit with page cache is a normal state. For the same reason, the working set reaching memory.max is not an OOM condition either. Only inactive_file is subtracted from the working set, and reclaimable memory remains in the rest: > > - active_file: demoted to inactive under pressure and then reclaimed > - slab_reclaimable (dentry / inode caches): in memcg reclaim too, shrink_slab() is called after shrink_lruvec() (shrink_node_memcgs() in mm/vmscan.c) > > So there is no threshold of "OOM once either value reaches the limit". The actual condition is when the charge fails and reclaim cannot make room. What cannot be reclaimed is mainly anonymous pages (without swap) and shmem, so that is where to look when narrowing down. That said, what the kernel compares against memory.max in a memcg OOM is the whole of memory.current, and it is not decided by any single metric. memory.current includes, besides anonymous pages, kernel memory, shmem (tmpfs and shared memory), and page cache that cannot be reclaimed, or cannot be reclaimed right away. container_memory_rss is an indicator for determining whether anonymous pages are the main cause. If cgroup RSS is close to the limit, anonymous pages are a likely major contributor, but RSS being below the limit is no evidence that an OOM will not occur. When the gap between RSS and the working set is large, check container_memory_usage_bytes (the value of memory.current) together with the breakdown in memory.stat on the node: anon, file, shmem, slab and so on. OOM kill from node memory exhaustion (global OOM) When a node-side allocation cannot be satisfied after reclaim, the kernel evaluates eligible processes using oom_badness(). cpusets, NUMA policies and oom_score_adj=-1000 can restrict the candidates. A container need not exceed its own limit to be selected. There is no guaranteed ordering between kubelet eviction and the kernel's OOM killer. The kubelet evicts Pods to protect the node, and its selection puts "Pods whose usage exceeds their memory request" first (rankMemoryPressure() in helpers.go). // rankMemoryPressure orders the input pods for eviction in response to memory pressure. // It ranks by whether or not the pod's usage exceeds its requests, then by priority, and // finally by memory usage above requests. func rankMemoryPressure(pods []*v1.Pod, stats statsFunc) { orderedBy(exceedMemoryRequests(stats), priority, memory(stats)).Sort(pods) } A Pod with a zero memory request and positive usage belongs to the "exceeds request" group. Within that group, priority and then usage above request determine the order, so it is not necessarily evicted first. If eviction acts in time, the result is Evicted, not OOMKilled. Global OOM happens first when eviction cannot reclaim memory as fast as memory grows. There are several reasons it cannot keep up. | Factor | Value | Effect | |:---|:---|:---| | Evaluation interval | 10 seconds (evictionMonitoringPeriod) | Detection can be up to 10 seconds late | | Pods evicted per evaluation | Only one ("we kill at most a single pod during each eviction interval" in eviction_manager.go) | At most one Pod is evicted per synchronization; notifications and cleanup waits mean this is not a fixed rate | | Waiting for cleanup | Up to 30 seconds (podCleanupTimeout) | The next evaluation may be delayed further | | Default threshold | memory.available<100Mi | Extremely thin on large nodes | | How memory.available is computed | From the root cgroup's anon + file | Contains no kernel memory, so it [overestimates free memory](#values-the-kubelet-uses-for-eviction-decisions) | The official Kubernetes documentation also says (Node out of memory behavior): > If the node experiences an out of memory (OOM) event prior to the kubelet being able to reclaim memory, the node depends on the oom_killer to respond. It is more likely when applications that allocate a lot of memory right at startup, or in response to a sudden surge in requests, share a node. > Warning: Eviction protects the node; it does not protect individual Pods from OOM. Containers without limits have no memory.max backstop at all, so eviction is merely the last line of defense. > > Moreover, rankMemoryPressure() looks at Pod priority before usage size. If the offending Pod has high priority, low-priority Pods with small usage get evicted first, and global OOM can occur without the pressure being resolved. The proper countermeasure is to set limits and requests rather than relying on eviction. Which process gets killed The kernel evaluates eligible processes using resident pages, swap, page tables and oom_score_adj; the adjustment alone does not establish a fixed kill order. The kubelet sets oom_score_adj (a bonus added to oom_score) according to the container's QoS class, so the QoS class and the memory request setting directly become how likely a container is to be killed. You can check the QoS class with kubectl get pod <Pod name> -o jsonpath='{.status.qosClass}', and how it is determined is described in Pod Quality of Service Classes. | Condition | oom_score_adj | Likelihood of being killed | |:---|:---|:---| | system-node-critical (evaluated before the QoS class) | -997 | Least likely | | Guaranteed | -997 | Least likely | | Burstable | 1000 - 1000 × memory request / node memory capacity (clamped to 3–999) | The smaller the request, the more likely | | BestEffort | 1000 | Most likely | The implementation is GetContainerOOMScoreAdjust() in Kubernetes' policy.go. system-node-critical is evaluated before the QoS class decision. if types.IsNodeCriticalPod(pod) { // Only node critical pod should be the last to get killed. return guaranteedOOMScoreAdj } The formula above is the container-level baseline. Pod-level requests and restartable init containers can introduce adjustments. An omitted memory request can also default from the limit, so inspect the applied Pod spec. The Burstable clamp is implemented with a lower bound of 1000 + guaranteedOOMScoreAdj (= 3) and an upper bound of besteffortOOMScoreAdj - 1 (= 999). A Pod whose request is set smaller than its actual usage is more likely to be targeted in both eviction and global OOM. > Warning: PriorityClass works differently for eviction and for global OOM. It is a misreading to think that raising priority protects you from both. > > | PriorityClass | Eviction | global OOM (oom_score_adj) | > |:---|:---|:---| > | User defined (up to 1000000000) | The higher the value, the later it is evicted | Determined by QoS class and request | > | system-cluster-critical | Exempt from node-pressure eviction | Determined by QoS class and request (not protected) | > | system-node-critical | Exempt from node-pressure eviction | Fixed at -997 | > | static / mirror Pod | Exempt from node-pressure eviction | Determined by QoS class and request | > > The three exempt rows identify critical Pods for which IsCriticalPod() is true; the kubelet checks static and mirror Pods as well as system-critical priority classes. > > What oom_score_adj protects is only system-node-critical, for which IsNodeCriticalPod() is true. If a system-cluster-critical Pod is BestEffort, its oom_score_adj is 1000, and it is more likely to be selected in a global OOM. "I raised the priority but it was still OOMKilled" comes from this path. > Warning: Neither an OOM kill nor node-pressure eviction directly deletes the Pod API object. What happens next depends on restartPolicy. > > - Always or OnFailure: the container is recreated on the same node. The Pod stays on the same node, so if the cause is not resolved, OOM kills will happen again > - Never: the container is not recreated and the Pod remains Failed. If a controller such as a Job creates a successor Pod, it may be placed on a different node > > If OOMKilled repeats even though there is headroom under the limit, suspect this path. > Note: Because the OOM kill frees memory, the pressure may already be gone when the kubelet next evaluates free memory. In that case the node may get no MemoryPressure condition and no node.kubernetes.io/memory-pressure:NoSchedule taint, and scheduling of new Pods does not stop. Conversely, if the pressure continues, a later evaluation sets the condition and taint, and eviction also occurs. Note that it is decided by memory.available at evaluation time, not by whether an OOM kill happened. {% enddetails %} Checking with PSI (Pressure Stall Information) This is an auxiliary indicator for the case where things are slow although usage has not reached the limit. It shows how long processes were actually stalled waiting for memory reclaim. {% details How to read PSI and PromQL %} You can obtain the time a container was stalled waiting for memory allocation as PSI. Even when usage has not reached the limit, you can detect being made to wait by memory reclaim. | Metric | Emitted by | Description | |:---|:---|:---| | container_pressure_memory_waiting_seconds_total | cAdvisor (embedded in kubelet) | Time some processes were waiting for memory (total of some in memory.pressure) | | container_pressure_memory_stalled_seconds_total | cAdvisor (embedded in kubelet) | Time all processes were stalled waiting for memory (total of full in memory.pressure) | Note: These metrics are controlled by the KubeletPSI feature gate, which went GA in v1.36 and is enabled by default (it cannot be disabled). The kernel must support PSI, though, and the kubelet decides this by whether cpu.pressure exists at the cgroup root (cadvisor_linux.go). If the metrics do not appear, check whether the kernel was built with CONFIG_PSI=y and whether psi=0 is set as a kernel parameter. memory.pressure also has avg10 / avg60 / avg300, "the average over the last N seconds (%)", but only the cumulative value (total) is exposed as a Prometheus metric. So use rate() to see a ratio. The unit is seconds, so the result of rate() is "how many seconds of waiting per second", a ratio from 0 to 1. Multiply by 100 to get a percentage directly. <!-- PromQL uses Python fencing for syntax highlighting only. --> # Per Pod, the share of time (%) all processes were stalled waiting for memory 100 * max by (namespace, pod) ( rate(container_pressure_memory_stalled_seconds_total{container="", image="", pod!=""}[5m]) ) # Per Pod, the share of time (%) some processes were waiting for memory 100 * max by (namespace, pod) ( rate(container_pressure_memory_waiting_seconds_total{container="", image="", pod!=""}[5m]) ) When there is plenty of memory, neither increases, so rate() stays at 0. Under pressure, waiting (some) rises first, and if it gets worse stalled (full) follows. waiting counts even when only some processes waited, so to judge severity, use stalled. It is the time all processes in the container were stopped, so if it shows a value, throughput is directly affected. To see memory pressure on the whole node, use the root cgroup series. # Share of time (%) the whole node was completely stalled waiting for memory 100 * max by (node) ( rate(container_pressure_memory_stalled_seconds_total{id="/"}[5m]) * on(instance) group_left(node) kubelet_node_name ) Containers that are struggling although they are below their limit can be found by combining PSI with utilization. container_spec_memory_limit_bytes is the container limit exposed by cAdvisor. # Containers whose usage is under 80 % of the limit but that are stalled waiting for memory ( 100 * rate(container_pressure_memory_stalled_seconds_total{container!="", image!=""}[5m]) > 1 ) and ( container_memory_usage_bytes{container!="", image!=""} / container_spec_memory_limit_bytes{container!="", image!=""} < 0.8 ) and keeps only series whose label sets match exactly on both sides. A container with no limit has container_spec_memory_limit_bytes of 0, so the division gives +Inf, which does not satisfy < 0.8, and it is excluded automatically. Here is an example alert. It catches stalled that keeps occurring. - alert: ContainerMemoryStalled # Fires when the 5-minute average of fully stalled time is 5 % or more for 10 minutes expr: | 100 * max by (namespace, pod) ( rate(container_pressure_memory_stalled_seconds_total{container="", image="", pod!=""}[5m]) ) > 5 for: 10m labels: severity: warning annotations: summary: "A Pod is stalled waiting for memory allocation (add the namespace and pod labels with your own template)" description: "It may be made to wait by reclaim even though it has not reached its memory limit. Check the gap between usage and RSS and the amount of page cache." The right threshold of 5 % depends on the environment. First look at the distribution of values without an alert to understand the normal level, then decide. For details on PSI see Understand Pressure Stall Information (PSI) Metrics. {% enddetails %} Quick reference: contradictions explained and a metric map The working set, cgroup usage, process RSS and node availability answer different questions. Use this map to identify the value behind the dashboard before comparing it with a limit or threshold. For a non-root cgroup, working set is max(memory.current - inactive_file, 0). It still includes some reclaimable memory. | Scope | Metric | Emitted by | Backing value in cgroup v2 / /proc | Page cache handling | Main use | |:---|:---|:---|:---|:---|:---| | Container | container_memory_working_set_bytes | cAdvisor (embedded in kubelet) | max(memory.current − inactive_file, 0) | Active pages count as in use | kubectl top pod, usage visualization | | Container | container_memory_rss | cAdvisor (embedded in kubelet) | anon in memory.stat | Not included (anonymous pages only) | Understanding what the app allocated | | Container | container_memory_usage_bytes | cAdvisor (embedded in kubelet) | memory.current | Everything included | What memcg OOM is evaluated against | | Node | container_memory_working_set_bytes{id="/"} | cAdvisor (embedded in kubelet) | anon + file − inactive_file in the root cgroup's memory.stat (see below) | Active pages count as in use | Eviction decisions, kubectl top node | | Node | node_memory_MemAvailable_bytes | node-exporter | MemAvailable in /proc/meminfo | Reclaimable pages count as free | Visualizing node memory utilization | | Node | kube_node_status_capacity{resource="memory"} | kube-state-metrics | (status.capacity of the Node object) | (capacity, not usage) | Denominator when converting to utilization | Looking at the same node, utilization computed from node_memory_MemAvailable_bytes does not match the utilization the kubelet uses for eviction decisions. With workloads that do a lot of file I/O or logging, the difference can reach several GiB. Different emitters also mean different labels on the metrics. cAdvisor metrics carry no label identifying the node, so when you compute per-node utilization you need to align labels with kubelet_node_name, as shown later. Kubernetes memory usage and cgroup v2 Why the numbers differ A process RSS measurement includes its resident mappings, including shared and file-backed pages. Cgroup accounting charges memory to a group, while working set subtracts one category of file cache from usage. An application's heap metric covers a different subset again. Reclaimability is a separate question: clean file cache can often be reclaimed, dirty pages may need writeback, and anonymous memory normally needs swap or explicit release by the process. Calling free does not necessarily return allocator-held memory to the OS. These differences explain why heap size, process RSS, cgroup RSS and working set need not match. The following sections connect those values to cgroup v2 accounting. Memory breakdown in cgroup v2 Container memory usage is managed in a per-container cgroup. In cgroup v2, memory.current is the current usage and memory.stat is its breakdown. # Example values in a container's cgroup (checked on the node) cat /sys/fs/cgroup/<container cgroup>/memory.current Output: 5368709120 cat /sys/fs/cgroup/<container cgroup>/memory.stat Output: anon 2147483648 # anonymous pages (heap, stack: memory not backed by a file) file 3087007744 # page cache (including tmpfs and shared memory) shmem 268435456 # the part of file that is tmpfs or shared memory ... inactive_anon 134217728 # anon LRU: not accessed for a while active_anon 2013265920 # anon LRU: recently accessed inactive_file 1073741824 # file LRU: not accessed for a while active_file 1744830464 # file LRU: recently accessed unevictable 268435456 # pages that cannot be reclaimed (here, the tmpfs part) ... The following relationships hold in this example. Note that inactive_file + active_file does not equal file; it equals file - shmem. The next section explains why. anon + file + kernel memory = 5368709120 (memory.current) inactive_file + active_file = 2818572288 = file - shmem inactive_anon + active_anon = 2147483648 = anon (shmem is not in here) unevictable = 268435456 = shmem (tmpfs without swap) The main fields of memory.stat are: anon (anonymous pages) - Memory not backed by a file. Regions the application allocated with malloc and similar belong here. - Unlike page cache, the kernel cannot reclaim it independently of the process. With swap it can write pages out and free up space, but on nodes that do not use swap it is memory the kernel cannot reclaim under pressure. When swap is unavailable, the kernel sets scan_balance = SCAN_FILE and excludes the anon LRU from scanning (get_scan_count() in mm/vmscan.c). - On the other hand, this value drops if the process itself returns memory to the OS with munmap or madvise(MADV_DONTNEED). Note that calling free does not reduce it unless the allocator or language runtime returns the memory to the OS. file (page cache) - Contents of files held in memory by the kernel during reads and writes. The application did not allocate it explicitly. - When memory runs short the kernel discards (reclaims) it automatically, so it is basically reclaimable memory. - Cache of files on disk is split into active_file (recently accessed) and inactive_file (not accessed for a while). Reclaim takes inactive_file first. - active_file + inactive_file does not equal file. These are totals of the LRU lists, and as described below shmem is on the anon LRU rather than the file LRU. Besides shmem, pages that fall off the LRU, such as mlocked pages, also show up as a difference. - It includes not only the cache of files on disk but also shmem (tmpfs and shared memory), because Linux treats these as page cache too. shmem is written out to swap rather than to a file, so on nodes without swap it cannot be reclaimed. The breakdown is visible as shmem in memory.stat. > Warning: shmem is counted in file, but it is not on the file LRU. The kernel classifies swap-backed pages as anon, and tmpfs pages are among them. folio_is_file_lru() in mm_inline.h is defined as "0 if @folio is a normal anonymous folio, a tmpfs folio or otherwise ram or swap backed folio". > > The counters they are charged to are different, though. A tmpfs page is added to both NR_FILE_PAGES (= file) and NR_SHMEM (= shmem) (mm/shmem.c): > > ```c > __lruvec_stat_mod_folio(folio, NR_FILE_PAGES, nr); > __lruvec_stat_mod_folio(folio, NR_SHMEM, nr); > ``` > > Furthermore, for a Kubernetes emptyDir with medium: Memory, the kubelet mounts tmpfs with the noswap option when the node supports it (generateTmpfsMountOptions() in empty_dir.go). The kernel makes the inode of a noswap tmpfs unevictable (shmem_get_inode() in mm/shmem.c), so these pages sit on the unevictable LRU, not the anon LRU. > > ```c > if (sbinfo->noswap) > mapping_set_unevictable(inode->i_mapping); > ``` > > So shmem is accounted as follows: > > | mount | LRU | Where it shows up in memory.stat | > |:---|:---|:---| > | emptyDir with medium: Memory (noswap) | unevictable | unevictable | > | Any other tmpfs / shared memory | anon LRU | inactive_anon / active_anon | > > In neither case does it enter inactive_file / active_file. That matters in two ways later: > > - inactive_file + active_file generally corresponds to file - shmem; unevictable pages and asynchronous statistics can cause differences > - Only inactive_file is subtracted from the working set, so shmem stays in the working set in full With workloads that write a lot of logs or read and write large files, the page cache can grow to several GiB even though the application did not intend that. > Note: memory.current includes memory the kernel itself uses (slab, socket buffers and so on) in addition to anon and file. So adding anon and file alone does not give memory.current. How cgroup v2 memory controls map to Pod settings cgroup v2 memory control interface cgroup v2 offers four levels of control over memory usage. memory.min and memory.low are protections ("do not reclaim up to here"), and memory.high and memory.max are limits ("do not allow use beyond here"). | File | Type | Behavior | |:---|:---|:---| | memory.min | Protection (hard) | Protects memory within the effective boundary determined by ancestors and usage. It does not preallocate physical memory | | memory.low | Protection (soft) | Best-effort avoidance of reclaim up to this value. It is reclaimed if nothing else can be | | memory.high | Limit (soft) | Exceeding it applies reclaim pressure and throttling. Crossing memory.high does not itself invoke the OOM killer; memory.max and global OOM still apply | | memory.max | Limit (hard) | Exceeding it triggers reclaim, and if usage still does not fit, the cgroup's OOM killer runs | Other files that report state: - memory.current: current usage (what container_memory_usage_bytes is backed by) - memory.stat: usage breakdown - memory.events: counts of low, high, max, oom, oom_kill and oom_group_kill - memory.swap.max: swap limit - memory.pressure: PSI for waiting on memory allocation For exact definitions of each file see the kernel's Memory Interface Files, and for how Kubernetes treats cgroup v2 see About cgroup v2. How Pod settings are reflected Pod settings are reflected into cgroup v2 as follows. | Pod setting | Target in cgroup v2 | |:---|:---| | resources.limits.memory | memory.max | | resources.requests.memory | Not reflected (with the default kubelet configuration in v1.37; you can reflect it by changing settings → [KEP-2570](#kep-2570-support-memory-qos-with-cgroups-v2)) | | resources.limits.cpu | cpu.max | | resources.requests.cpu | cpu.weight | cgroup v2 is a directory tree. Its top is the root cgroup (/sys/fs/cgroup), where any process that is in no child cgroup belongs, so it contains not only Pods but every process on the node, such as the kubelet and containerd. Kubernetes creates the hierarchy below it: "all Pods on the node → Pod → container". # With the systemd cgroup driver (the kubeadm default; names differ with the cgroupfs driver, see below) /sys/fs/cgroup ← root cgroup (id="/") │ contains every process on the node │ the value used for eviction decisions │ ├── system.slice/ kubelet, containerd, sshd, etc. │ ├── kubelet.service (processes that are not Pods) │ └── containerd.service │ └── kubepods.slice/ ← all Pods on the node │ (id="/kubepods.slice") ├── kubepods-pod<UID>.slice/ ← Pod (Guaranteed) │ ├── cri-containerd-<ID>.scope ← container │ └── cri-containerd-<pause ID>.scope ← pause container │ ├── kubepods-burstable.slice/ (intermediate slice per QoS class) │ └── kubepods-burstable-pod<UID>.slice/ ← Pod (Burstable) │ └── cri-containerd-<ID>.scope ← container │ └── kubepods-besteffort.slice/ └── kubepods-besteffort-pod<UID>.slice/ ← Pod (BestEffort) └── cri-containerd-<ID>.scope ← container When filtering metrics, the root cgroup is id="/", all Pods on the node are id="/kubepods.slice", a Pod is container="", image="", pod!="", and a container is container!="", image!="". What the kubelet uses for eviction decisions is the working set of the topmost root cgroup ([see below](#values-the-kubelet-uses-for-eviction-decisions)). That is a different level from memcg OOM caused by a container or Pod exceeding its limit. Memory limits are set not only on containers but also on higher levels. | Level | cgroup | Memory limit | |:---|:---|:---| | Whole node (root cgroup) | / | Not set (the root has no memory.max → [see below](#values-the-kubelet-uses-for-eviction-decisions)) | | All Pods on the node | /kubepods.slice | Capacity - kube-reserved - system-reserved when Pod allocatable enforcement is enabled | | Pod | Differs by QoS class (see below) | The Pod's effective limit, including Pod-level spec.resources.limits.memory when configured | | Container | cri-containerd-<ID>.scope under the Pod's cgroup | That container's limits.memory | The examples here use container-level resources and omit Pod-level resources. Without a Pod-level limit, the required container limits must be present for a finite Pod limit to be set. A Pod's effective limit is not simply the sum of its regular containers' limits.memory. If there are init containers, their maximum is also considered, and spec.overhead is added if set. For the calculation see Resource sharing within containers and Pod Overhead. So even if a container has not reached its own limit, an OOM kill can occur when the Pod as a whole or /kubepods.slice hits its limit. > Note: The paths above assume the kubelet uses the systemd cgroup driver. With the cgroupfs driver, paths look like /kubepods/burstable/pod<UID>/<container ID>, with no .slice or .scope and no cri-containerd- prefix on the container. > > The kubelet's own default is cgroupfs, but kubeadm sets systemd when the value is empty (defaults.go, kubelet.go). In v1.37, a supported runtime's CRI RuntimeConfig response takes precedence through driver auto-detection. Check the effective driver with: > > ```bash > $ kubectl get --raw "/api/v1/nodes/<node name>/proxy/configz" \ | jq '.kubeletconfig.cgroupDriver' > "systemd" > ``` > > The id label of cAdvisor metrics also changes with the cgroup driver, so when filtering in PromQL, check the value in your own cluster with this query: > > ```bash > group by (id) (container_memory_working_set_bytes{id=~"/kubepods|/kubepods.slice"}) > ``` > > A Pod's cgroup path depends on its QoS class. As the tree above shows, Burstable and BestEffort go through an intermediate slice per QoS class. > > In the <UID> part, systemd escaping replaces the hyphens of the Pod UID with underscores (example: 6983351e-4ede-... → pod6983351e_4ede_...). The UID of a static Pod (mirror Pod) is a hash with no hyphens, so it looks like pod1ff70b81885169da4e9e32c9530b77c9.slice. > > The QoS class and UID of a Pod can be read from the Pod resource, and once you have those two you can assemble the cgroup path. > > ```bash > $ kubectl get pod -n <namespace> <Pod name> -o jsonpath='{.status.qosClass}{"\n"}{.metadata.uid}{"\n"}' > Burstable > 6983351e-4ede-4611-8bab-b986614af18d > > # To list them > $ kubectl get pods -n <namespace> -o custom-columns='NAME:.metadata.name,QOS:.status.qosClass' > ``` Container metrics The cAdvisor built into the kubelet exposes the cgroup values above as Prometheus metrics. They are served from the kubelet's /metrics/cadvisor endpoint, and you do not need to run a separate cAdvisor Pod or DaemonSet. | Metric | Emitted by | Backing value in cgroup v2 | Description | |:---|:---|:---|:---| | container_memory_usage_bytes | cAdvisor (embedded in kubelet) | memory.current | Usage including anonymous pages, page cache and kernel memory. The value memcg OOM is evaluated against | | container_memory_working_set_bytes | cAdvisor (embedded in kubelet) | memory.current - inactive_file | Usage minus inactive_file, clamped to zero. A heuristic that still includes some reclaimable memory | | container_memory_rss | cAdvisor (embedded in kubelet) | anon in memory.stat | Amount of anonymous pages. Unlike true RSS, it does not include mmaped file pages (see below) | | container_memory_cache | cAdvisor (embedded in kubelet) | file in memory.stat | Page cache size. Includes shmem (tmpfs and shared memory) | | container_memory_total_active_file_bytes | cAdvisor (embedded in kubelet) | active_file in memory.stat | Active page cache | | container_memory_total_inactive_file_bytes | cAdvisor (embedded in kubelet) | inactive_file in memory.stat | Inactive page cache | | container_memory_mapped_file | cAdvisor (embedded in kubelet) | file_mapped in memory.stat | Amount of mmaped files | | `cont