opencontainers

My Path Check Caught All Six Escapes, Then Let 24,378 Through Taking a filename from a user and opening it inside your own directory is one of the oldest jobs in software. Archive extractors do it, file servers do it, even template engines do it. And nearly everyone writes the same two lines: join the path onto the root, resolve it with realpath, verify the result starts with the root. If it doesn't, reject. In mid-October I went through the behaviour of openat2 itself in a separate article, and I ran this race there too: the classic pattern that calls fstatat and then openat returned the attacker's file on 30.7% of its successful opens inside a 50 µs window. So I already know "check then open" is broken — that part is not new. This article asks something else: which check actually gets written in the field, and what does it do? Few people write the fstatat pattern by hand; what everyone writes is realpath plus a prefix comparison. And that check looks spotless in a static test — which is probably why nobody replaces it. I tried six defenses against nine paths. In the static table realpath came out spotless — it stopped all six escapes. Then I ran the same check while the directory structure moved underneath it: on 24,378 of 200,000 turns it read the file outside the cage. In that same race, RESOLVE_IN_ROOT returned zero. The setup Measurements were taken on this blog's own server: Ubuntu 24.04, kernel 6.8.0-147-generic, 18 vCPUs, gcc 13.3.0. All of it as an unprivileged user inside a temporary directory from mktemp -d; nothing persistent on the server was touched. The cage: kok/ is the area we permit, disari/ is what must not be read. kok/ok.txt -> "KAFES-ICI" kok/sub/ok2.txt -> "KAFES-ICI2" kok/ici -> sub/ok2.txt # symlink that stays inside (legitimate) kok/mutlak -> /etc/hostname # absolute symlink kok/goreli -> ../disari/sir.txt # relative escape kok/atla -> ../disari # escape to a directory kok/sihir -> /proc/self/root # magic link disari/sir.txt -> "DISARIDAKI-SIR" Three of the nine paths are legitimate — they have to be accepted. I put that distinction in deliberately, because how many escapes a defense stops says nothing on its own; a function that rejects everything also stops all six. The real question is whether it does both jobs. The legitimate three: ok.txt (plain file), sub/../ok.txt (a .. that turns back inside), ici (a symlink that stays in the cage). The six escapes: ../disari/sir.txt, /etc/hostname, mutlak, goreli, atla/sir.txt, sihir/etc/hostname. Six defenses First, the filter everyone writes once: /* D1: reject if it contains ".." */ if (strstr(path, "..")) { errno = EACCES; return -1; } return openat(dfd, path, O_RDONLY); Second, the check serious code writes — the real defendant here: /* D2: realpath + prefix comparison */ snprintf(joined, sizeof joined, "%s/%s", ROOT, path); if (!realpath(joined, resolved)) return -1; size_t rl = strlen(ROOT); if (strncmp(resolved, ROOT, rl) != 0 || (resolved[rl] != '/' && resolved[rl] != '\0')) { errno = EACCES; return -1; } return open(resolved, O_RDONLY); The resolved[rl] test matters: without it, /srv/data-gizli passes a /srv/data prefix. A common bug, but not this article's subject; the version above is the correct one. Third, the route taken by people who say "I turned symlinks off": openat(dfd, path, O_RDONLY | O_NOFOLLOW). The remaining three use openat2: RESOLVE_BENEATH, RESOLVE_IN_ROOT, and RESOLVE_IN_ROOT | RESOLVE_NO_MAGICLINKS | RESOLVE_NO_XDEV. The static table With the directory structure held still, 6 × 9 = 54 opens: | Defense | Escapes blocked | Legitimate opened | |---|---|---| | D1 .. filter | 1/6 | 2/3 | | D2 realpath + prefix | 6/6 | 3/3 | | D3 openat + O_NOFOLLOW | 2/6 | 2/3 | | D4 RESOLVE_BENEATH | 6/6 | 3/3 | | D5 RESOLVE_IN_ROOT | 6/6 | 3/3 | | D6 IN_ROOT + NOMAGIC + NOXDEV | 6/6 | 3/3 | Since a summary can mislead, here is the raw table too — which error code came from where is about to matter: | Defense | ok.txt | sub/../ok.txt | ici | ../disari/sir.txt | /etc/hostname | mutlak | goreli | atla/sir.txt | sihir/... | |---|---|---|---|---|---|---|---|---|---| | D1 .. filter | OPEN | EACCES | OPEN | EACCES | OPEN | OPEN | OPEN | OPEN | OPEN | | D2 realpath + prefix | OPEN | OPEN | OPEN | EACCES | EACCES | EACCES | EACCES | EACCES | EACCES | | D3 O_NOFOLLOW | OPEN | OPEN | ELOOP | OPEN | OPEN | ELOOP | ELOOP | OPEN | OPEN | | D4 BENEATH | OPEN | OPEN | OPEN | EXDEV | EXDEV | EXDEV | EXDEV | EXDEV | EXDEV | | D5 IN_ROOT | OPEN | OPEN | OPEN | ENOENT | ENOENT | ENOENT | ENOENT | ENOENT | ENOENT | | D6 IN_ROOT+NOMAGIC+NOXDEV | OPEN | OPEN | OPEN | ENOENT | ENOENT | ENOENT | ENOENT | ENOENT | ENOENT | Every bold OPEN is a leak: in that cell the program read the contents of a file outside the cage. The .. filter loses at both ends. It stops only one of six escapes — because there is no .. inside a symlink, and none in the absolute path /etc/hostname either. Meanwhile it rejects the legitimate sub/../ok.txt with EACCES. So it lets the attacker through and annoys the user. Managing that combination is genuinely hard. O_NOFOLLOW keeps its promise, but the promise is small: it protects only the final component. mutlak and goreli are symlinks themselves, so they get ELOOP — but atla/sir.txt, where atla is a symlink to a directory, opens cleanly and reads the secret outside. O_NOFOLLOW is not a scoping flag; once again, something the manual never hides, yet it keeps getting used as scoping in real code. On top of that it drops the legitimate ici link with ELOOP. That the last two rows look alike but return different error codes is no accident either — BENEATH says EXDEV, IN_ROOT says ENOENT. I will come back to that difference shortly, because it is the most misleading part of the table. And here is the discomfort in the middle of it: D2 produces exactly the same result as the openat2 flags. Six escapes closed, three legitimate paths open. Anyone reading this far would say "so realpath is enough," and they would not be unreasonable. The table says they are right. Because the table is incomplete. The real test All 54 opens above happened in a directory tree where nothing moved. In production, directory trees move. Your archive extractor gets another request into the same directory while writing a file; a backup process relocates something; or none of that happens and the person on the other side moves the directory on purpose. realpath and open are two separate calls. There is a moment between them. !Diagram To measure that window I wrote a second probe and a separate layout: kok/alan/hedef.txt -> "KAFES-ICI-HEDEF" kok/.yem -> ../disari # the symlink to be swapped in disari/hedef.txt -> "DISARIDAKI-SIR" The main thread keeps trying to open the same path; a side thread keeps atomically swapping the alan directory with a symlink pointing outside, using renameat2(RENAME_EXCHANGE). The atomic swap is critical here: there is no half-finished state, so what the numbers below show is not an implementation bug but a structural hole in the check itself. The whole rig is this much — the part you would copy to try it in your own code: /* side thread: atomically swap the directory with a symlink */ static void *takasci(void *_) { while (!dur) if (syscall(__NR_renameat2, dfd, "alan", dfd, ".yem", RENAME_EXCHANGE) == 0) takas++; return NULL; } /* main loop: open the same path with both defenses, read the content */ for (long i = 0; i < N; i++) { int f = d2("alan/hedef.txt"); /* realpath + prefix */ if (f >= 0) { a2++; if (sizdi(f)) s2++; } else h2++; f = d5("alan/hedef.txt"); /* RESOLVE_IN_ROOT */ if (f >= 0) { a5++; if (sizdi(f)) s5++; } else h5++; } sizdi() reads from the opened descriptor and checks whether the content starts with DISARIDAKI. So the counter is not counting "the open succeeded" but the wrong file's contents actually reached my hands. My first probe did not draw that distinction and reported the rate higher than it was. I ran both D2 and D5 in the same run, 200,000 turns: | Defense | Opened | Errors | Leaked outside | |---|---|---|---| | D2 realpath + prefix | 90,901 | 109,099 | 24,378 | | D5 RESOLVE_IN_ROOT | 88,565 | 111,435 | 0 | The swap counter read 314,729 over the same period. 24,378 is 12.2% of all attempts. But that is not the ratio to look at: of the 90,901 files D2 successfully opened, 26.8% were outside the cage. So after the check said "fine, this path is safe," roughly one open in four returned the wrong file. This is not an estimate or a theoretical window; I read the contents, and they said DISARIDAKI-SIR. 24,378 times. On the RESOLVE_IN_ROOT side the number is zero. The reason is simple, and that is exactly why it holds: in openat2 the check and the open are not separate steps. The kernel validates the scope while it resolves the path, and never hands control back to you in between. There is no window to miss. Look at D5's 111,435 errors as well — more than D2's 109,099. There is a cost here and no point hiding it: opens that land on the swap moment return errors, and some of them get EAGAIN from scope ambiguity. I did not break the errors down, so I cannot say how many of those 111,435 were ENOENT and how many EAGAIN. EAGAIN got its own measurement in the October piece: under a synthetic load generating thousands of renames per second it reached one open in five — that figure is not a ceiling, and it is not a rate measured on a real server either. Either way, moving to openat2 is not free security but a trade that requires you to write a bounded retry budget. On the other side of that trade sit 24,378 leaks. IN_ROOT does not reject, it remaps The October article's table has a row where opening /etc/hostname with RESOLVE_IN_ROOT returned ENOENT. I recorded the result but never showed why it was ENOENT — and that is exactly where the two flags part ways. It was ENOENT because there was no directory named etc inside the cage. That was the easy case. I put etc/hostname in the cage and tried again. The server's real /etc/hostname reads mustafaerbay-vps3; into the fake copy inside the cage I wrote SAHTE-HOSTNAME-KAFESTEN: | Flag | Path | Result | |---|---|---| | RESOLVE_BENEATH | /etc/hostname | DENIED EXDEV | | RESOLVE_IN_ROOT | /etc/hostname | OPEN → SAHTE-HOSTNAME-KAFESTEN | | RESOLVE_BENEATH | ../../etc/hostname | DENIED EXDEV | | RESOLVE_IN_ROOT | ../../etc/hostname | OPEN → SAHTE-HOSTNAME-KAFESTEN | The two flags do not do the same job. BENEATH, in the manual's words, does not permit resolution to leave the directory given by dirfd, and treats the attempt as an error. IN_ROOT treats dirfd as the root — so it does not reject the escape, it remaps it inward. Whatever chroot(2) does, but for a single open. So that October ENOENT did not mean "you cannot leave"; the kernel was saying it looked for kok/etc/hostname and did not find it. The moment you put that file in the cage, the same call returns a file instead of an error. In practice this means the following. If you are writing an archive extractor, IN_ROOT is exactly what you want: a /etc/passwd entry inside a tar quietly becomes kok/etc/passwd, which is how container images work. But if your case is "the user gives me a path inside my data directory," IN_ROOT is dangerously quiet: when the user asks for /etc/hostname they do not get an error, they get a different file. If an attacker can drop files into the cage — which is precisely what they can do in an archive extractor — they can redirect any path they like to their own content. My preference: BENEATH for APIs that take a path from the user. I want the error. Not a silently different file. The cage does not know what it holds There is also a dimension scoping takes no interest in. I put a FIFO into the cage with mkfifo and opened it with RESOLVE_IN_ROOT: RESOLVE_IN_ROOT boru -> OPEN (type=FIFO) RESOLVE_IN_ROOT ok.txt -> OPEN (type=regular file) The cage held — the pipe really is inside it. But had I not written O_NONBLOCK, the program would be hanging there waiting for a writer. Scoping flags check where you went, not what you opened. Confining the path to the cage does not mean you opened a regular file. The kernel has an answer for this, a new one: OPENAT2_REGULAR. The patch written by Dorjoy Chowdhury and taken by Christian Brauner on 21 May 2026 returns EFTYPE when the path is not a regular file. I could not find the flag in v7.1's header; it is there in v7.2 — and today's mainline is v7.3. On my own 6.8 I got EINVAL, as expected: the old kernel does not know the flag and says so rather than ignoring it silently. That distinction and the full list of error codes got their own section in the October piece; I am not repeating it here. What it costs, and what to do below 5.6 I expected the safe option to be the expensive one. The numbers said the opposite. I opened a four-level path (a/b/c/d/f.txt) 300,000 times, after a warm-up round, in two separate runs: | Method | Run 1 | Run 2 | |---|---|---| | openat (unscoped) | 7,311 ns | 7,131 ns | | realpath + prefix | 32,579 ns | 33,244 ns | | RESOLVE_BENEATH | 7,705 ns | 7,833 ns | | RESOLVE_IN_ROOT | 7,720 ns | 7,307 ns | realpath is roughly 4.5 times more expensive, and the reason is unsurprising: it resolves the path fully once, then makes the kernel resolve the same path a second time. Two full path resolutions plus an lstat per component. Quoting one number for the scoping flags would be wrong, because the two flags do not sit in the same place. Unscoped openat wanders between 7,131 and 7,311. IN_ROOT wanders between 7,307 and 7,720 — the ranges overlap by four nanoseconds, so I cannot claim a difference there. BENEATH, though, runs 7,705 to 7,833: both runs sit above the unscoped ceiling, roughly 7%. That is a consistent gap, but I have two runs; it is not enough data to write "BENEATH is seven percent slower." What I can say: the cost of the scoping flags is at most a few percent, and in this measurement BENEATH looks slightly more expensive than IN_ROOT. Either way, the trade was not "security or speed" as I assumed. The leaky defense turned out to be the expensive one by a wide margin. On portability I have to be straight: openat2 arrived in 5.6, that is March 2020. On kernels below it — or outside Linux — these flags do not exist, and you have to walk the path component by component with O_NOFOLLOW at every step; that is what userspace libraries do. I did not measure that route, so I will not claim it gives equivalent protection. There is a trap here that is easy to skip: the kernel version is not enough. If you run inside a container, the seccomp profile may not know openat2 — a known situation on older Docker and libseccomp versions — and the call comes back EPERM or ENOSYS. So base your capability detection on a real probe at startup rather than a version number: call it once, fall back to emulation on ENOSYS/EPERM, and make that fallback increment a counter rather than happen silently. That is the only way you notice the machine with a new kernel and an old profile. You do not have to write the emulation yourself either: os.Root in Go's standard library and filepath-securejoin cover it on the Go side, libpathrs on the C and Rust side. Whichever you pick, if you provide scoping through emulation in your own code, run the race probe above against that emulation too. My realpath check also looked correct when it was written. The code outside the cage Finally, something none of this article's measurements cover: the path you open with openat2 is safe, but your program does not only open files. There is a live example. CVE-2026-41579 (CVSS 3.3, low), closed in runc in June 2026, is exactly this: the advisory says the mount-target lookup "correctly scopes the lookup inside the container." What opens the hole is not the lookup but the os.Remove and os.Symlink calls in setupPtmx and setupDevSymlinks, made on strings built with filepath.Join. If the image's /dev is a symlink, files named ptmx on the host can be deleted — and on the second arm, six fixed-name symlinks (core, fd, ptmx, stdin, stdout, stderr) can be created in an existing host directory. Docker is not affected because it masks /dev with a read-only layer; podman and containerd are. Fixed in 1.3.6, 1.4.3 and 1.5.0-rc.3. So a scoped open is a component, not a policy. You need to resolve the path once with openat2 and then continue through the descriptor: unlinkat, symlinkat, fstatat all take a dirfd. Every line that joins strings and returns to a path leaves the guarantee you earned right there. The title of the runc fix says as much: make the /dev setup descriptor-based. In practice that means this transformation: /* BEFORE: the open is scoped, what follows is not */ int fd = sys_openat2(dfd, rel, &how); /* safe */ snprintf(p, sizeof p, "%s/%s", kok, rel); unlink(p); /* NOT safe */ /* AFTER: continue through the descriptor */ int dir = sys_openat2(dfd, dirname_of(rel), &how_dir); /* O_PATH|O_DIRECTORY */ unlinkat(dir, basename_of(rel), 0); In the second version there is no second path resolution; unlinkat starts from a descriptor whose scope is already validated. Reduced to one sentence: resolve the path once, then never return to the path. A decision frame - Do not write a .. filter. In the measurement it stopped one escape of six and rejected a legitimate path. It cannot do either job. - Do not mistake O_NOFOLLOW for scoping. It protects only the final component; everything under a symlink to a directory walks right through. - realpath + prefix is correct on a static tree, not in a race. If you are certain the tree is yours and nobody moves it, it can stay. If you are not certain, the measured rate is 26.8%. - Taking a path from the user? RESOLVE_BENEATH. You want the escape as an error. - Extracting an archive or image? RESOLVE_IN_ROOT. You want the escape remapped inward — but know that this is a deliberate choice. - Do not pass both scoping flags, you will get EINVAL. Close magic links separately with RESOLVE_NO_MAGICLINKS; the manual says the scoping flags do this for now and that it may change. - Write a bounded budget for EAGAIN — three to five attempts — and if it runs out, reject the request and record it in a separate counter. Never retrying produces spurious failures when a neighbour is busy. - Check the file type separately. Before v7.2 with fstat; from v7.2 on with OPENAT2_REGULAR. - Do not return to strings after the open. Use the *at family. - Hold your root descriptor somewhere and think about its lifetime. Opening a root per request burns descriptors; holding one long-lived means the root itself can be renamed or umounted from under you. I prefer keeping the root as a long-lived O_PATH descriptor and checking periodically that it is still the object I expect. - Do not use cost as the excuse. In the measurement the scoping flags differ by at most a few percent; the expensive one is the realpath version (4.5 times). Conclusion In October I showed that openat2 closes the window between the check and the open. What this measurement taught me is where that window hides best: realpath plus a prefix comparison gave the right answer on all 54 opens. A static test suite will never go red on it — a single-threaded test assumes the world holds still while it runs. And the defense that looked best in the table turned out to be the most expensive one. The second lesson is about IN_ROOT, and it struck me as the quieter one: it does not stop the escape, it remaps it inward. For anyone who can drop files into the cage, that means getting the file they want instead of an error. In code that takes a path from the user, what I want is the EXDEV that BENEATH gives me. Put together, what I am left with is this: a check returning the right answer is not enough — there has to be nobody between the check and the action. What openat2 gives you is not a stricter rule; it is the absence of that gap. Official Sources - include/uapi/linux/openat2.h — RESOLVE_* and OPENAT2_REGULAR definitions - openat2: new OPENAT2_REGULAR flag support — kernel commit - openat2(2) manual page — RESOLVE_BENEATH and RESOLVE_IN_ROOT - runc security advisory GHSA-xjvp-4fhw-gc47 (CVE-2026-41579) - open(2) manual page — O_NOFOLLOW affects only the final component

runc: qué es y cómo funciona el runtime detrás de Docker Cuando corrés docker run -it ubuntu bash, el kernel de Linux no sabe qué es un contenedor: no existe tal cosa. Lo que arma ese comando es un runtime de contenedores llamado runc, y en pleno 2026 todavía tiene que hacer trampa, porque el lenguaje en que está escrito, Go, ni siquiera tiene una función fork() segura. Esa trampa, reejecutarse a sí mismo desde /proc/self/exe, es solo uno de los trucos que sostienen todo el ecosistema de contenedores: Docker, Kubernetes, containerd. Ninguno usa magia de aislamiento nueva. Todos reutilizan piezas del kernel de Linux que existen desde hace más de una década. TL;DR - Un contenedor es un proceso Linux normal con namespaces, cgroups y pivot_root aplicados, nada más.- runc, el runtime que ejecuta cada docker run, está en Go: un lenguaje sin fork() nativo y runtime multihilo.- El truco para imitar fork() es reejecutar /proc/self/exe antes de que el runtime de Go lance sus hilos.- Una imagen Docker es solo una carpeta: pivot_root la convierte en la raíz del proceso sin copiar nada.- dockerd corre como root porque namespaces y cgroups exigen privilegios; rootless rehace la red en user space. ¿Qué es un runtime de contenedores? Un runtime de contenedores es el programa de bajo nivel que traduce una imagen y un conjunto de restricciones en un proceso Linux aislado. Aplica namespaces, cgroups y pivot_root, y después ejecuta el proceso final con execve. runc es la implementación de referencia que usan Docker, containerd y Kubernetes. Linux expone ocho tipos de namespace: PID, network, mount, UTS, IPC, user, cgroup y time. Namespaces, cgroups y pivot_root: las piezas reales detrás de "contenedor" No hay una línea en el kernel de Linux que diga "esto es un contenedor". En cambio, un runtime combina mecanismos independientes que ya existían antes de que Docker existiera. Los namespaces esconden qué puede ver el proceso: su propio árbol de PIDs, su propia interfaz de red, su propio hostname. Los cgroups limitan cuánta CPU, memoria y E/S puede consumir ese proceso, y pivot_root cambia cuál carpeta del disco actúa como raíz del sistema de archivos para ese proceso en particular. Docker activa casi todos los namespaces por defecto cuando corrés un contenedor normal, pero podés desactivarlos selectivamente con flags como --pid=host o --network=host. Cada namespace es una vista distinta del mismo kernel: dos procesos en namespaces de red distintos pueden tener ambos una interfaz llamada eth0 sin pisarse, porque cada uno vive en su propia tabla de rutas. Los cgroups, por su parte, se organizan en controladores: cpu, memory, io y pids. El controlador cpu reparte tiempo de procesador con pesos relativos; memory pone un techo duro de RAM y puede matar el proceso con el OOM killer si lo supera; io limita el ancho de banda de disco, y pids pone un tope a cuántos procesos puede crear ese grupo antes de bloquear nuevos fork(). Docker expone estos límites con flags como --memory, --cpus y --pids-limit, pero por dentro son solo escrituras a archivos dentro de /sys/fs/cgroup/. flowchart TD A["docker run ubuntu bash"] --> B["Namespaces aislan PID, red y hostname"] A --> C["Cgroups limitan CPU y memoria"] A --> D["pivot_root cambia la raíz del proceso"] subgraph "Kernel de Linux" B C D end Cómo runc arma el proceso: de docker run a execve Una imagen de Docker no es más que una carpeta. Cuando bajás ubuntu desde Docker Hub, lo que estás descargando son capas comprimidas que, una vez descomprimidas, forman un árbol de directorios idéntico al de una instalación mínima de Ubuntu: /bin, /etc, /lib, /usr. No hay nada ejecutándose ahí todavía. El flujo completo de docker run -it --rm ubuntu /bin/bash pasa por cuatro programas antes de llegar al kernel. El cliente docker habla con el daemon dockerd. dockerd delega en containerd, que gestiona el ciclo de vida del contenedor. containerd invoca a runc con la ruta de la imagen y la especificación OCI del contenedor, y runc es quien finalmente hace las llamadas al sistema: clona el proceso con namespaces y cgroups, llama a pivot_root para que esa carpeta descomprimida se convierta en la raíz del proceso, y por último ejecuta /bin/bash con execve. La diferencia entre pivot_root y el chroot clásico de toda la vida es que pivot_root mueve la raíz vieja a un punto de montaje aparte y permite desmontarla después, mientras que chroot deja la raíz anterior accesible si el proceso logra escapar con un descriptor de archivo abierto desde antes. Por eso runc usa pivot_root cuando el kernel lo permite, y cae a chroot solo como alternativa en sistemas donde pivot_root no está disponible. sequenceDiagram participant U as Usuario participant D as dockerd participant C as containerd participant R as runc participant K as Kernel U->>D: docker run ubuntu bash D->>C: crear contenedor C->>R: create y start R->>K: clone con namespaces y cgroups R->>K: pivot_root a la imagen R-->>K: execve del proceso final Note over R,K: runc termina, el proceso final queda solo El problema de Go: por qué runc necesita fingir un fork() runc está escrito en Go, y Go tiene un problema de diseño para este trabajo puntual: su runtime arranca varios hilos del sistema operativo apenas el programa inicia, para manejar el recolector de basura y el scheduler de goroutines. La llamada clásica fork() de Unix solo duplica el hilo que la invoca, no los demás. Si runc llamara fork() directamente después de que el runtime de Go ya levantó esos hilos, el proceso hijo quedaría con un runtime de Go a medio inicializar y corrupto. La solución que adoptó el equipo de runc fue no usar fork() en absoluto. En su lugar, el binario se reejecuta a sí mismo leyendo su propio ejecutable desde /proc/self/exe, con un argumento especial que le indica al nuevo proceso que actúe como el "init" del contenedor en lugar del comando normal. Ese proceso hijo, recién arrancado desde cero con su propio runtime de Go limpio, es el que hace las llamadas de namespaces, cgroups y pivot_root antes de ejecutar el comando real con execve. Hay otro detalle que complica más las cosas. Algunas de las llamadas al sistema que necesita un runtime de contenedores, como setns() para unirse a un namespace existente, exigen ejecutarse en un solo hilo del sistema operativo, sin que el scheduler de Go mueva la goroutina a otro hilo a mitad de camino. Por eso runc incluye un pequeño módulo en C que corre antes de que el runtime de Go termine de inicializarse, en una ventana donde todavía hay un solo hilo activo. flowchart TD A["runc arranca en Go"] --> B["El runtime de Go ya levantó varios hilos"] B --> C["fork() clásico dejaría el hijo con un runtime corrupto"] C --> D["runc re-ejecuta /proc/self/exe"] D --> E["El proceso hijo arranca con un runtime de Go limpio"] E --> F["Configura namespaces, cgroups y pivot_root"] F --> G["execve() reemplaza el proceso por el comando real"] Por qué Docker corre como root Si usás Docker Desktop en macOS o Windows nunca lo notás, porque el daemon corre dentro de una máquina virtual Linux que Docker Desktop administra por vos. Pero en un servidor Linux real, dockerd corre como root, y durante años esa fue la única opción: crear namespaces, especialmente el de red, y manipular cgroups eran operaciones que el kernel reservaba a procesos con privilegios de superusuario. Eso cambió parcialmente con los user namespaces y con el modo rootless de Docker, que permite correr el daemon y los contenedores sin privilegios de root reales. El problema aparece en la red: sin acceso de root a las interfaces del host, el modo rootless tiene que reimplementar buena parte del stack de redes en espacio de usuario, con herramientas como slirp4netns o pasta, para lograr algo parecido al bridge de red que el modo normal arma directamente en el kernel. Incluso cuando el proceso dentro del contenedor corre como UID 0, Docker le recorta las capabilities de Linux por defecto: elimina permisos como CAP_SYS_ADMIN o CAP_NET_ADMIN que en el host le permitirían, por ejemplo, montar sistemas de archivos arbitrarios o reconfigurar interfaces de red ajenas. Podés revisar qué capabilities tiene activas un contenedor corriendo cat /proc/1/status | grep Cap desde adentro, y decodificar el valor hexadecimal con capsh --decode=<valor>. > 📌 Nota: correr como root dentro del contenedor no es lo mismo que el daemon corriendo como root en el host: son dos aislamientos distintos, y Docker recorta las capabilities de Linux dentro del contenedor incluso cuando el usuario ahí adentro es root. BusyBox empaqueta decenas de comandos Unix en un solo binario estático sin dependencias. Cómo empezar: tu propio mini runtime con unshare y chroot No necesitás escribir Go ni leer la especificación OCI para sentir en carne propia lo que hace runc. Linux trae las piezas sueltas en el comando unshare, que ya viene instalado en casi toda distribución moderna como parte del paquete util-linux. Esto corre en Linux nativo: no funciona dentro de Docker Desktop en macOS o Windows porque ahí ya estás dentro de una VM Linux, así que hace falta una terminal Linux real o una VM. Primero armá una carpeta mínima con un binario estático de BusyBox, que trae sus propios comandos sin depender de librerías del sistema: mkdir -p /tmp/minicontainer/bin cd /tmp/minicontainer curl -L -o bin/busybox https://busybox.net/downloads/binaries/1.35.0-x86_64-linux-musl/busybox chmod +x bin/busybox ./bin/busybox --install bin Ese último comando crea enlaces simbólicos para que ls, ps, sh y el resto de los comandos de BusyBox queden disponibles dentro de bin/. Ahora aislá un namespace de PID, de mount y de UTS, y usá chroot para cambiar la raíz a esa carpeta: sudo unshare --pid --mount --uts --mount-proc --fork \ chroot /tmp/minicontainer /bin/sh Vas a caer en un shell donde ps solo ve sus propios procesos, porque el namespace de PID le oculta todo lo del host: / # ps PID USER TIME COMMAND 1 root 0:00 sh 2 root 0:00 ps Eso es, literalmente, el mismo principio que usa runc, con namespaces y una raíz propia. Le falta cgroups para limitar recursos y pivot_root en lugar de chroot para ser completo, pero la idea central (un proceso normal con una vista recortada del sistema) ya está ahí. Para confirmar que el namespace de PID realmente aisló el proceso, abrí otra terminal en el host y compará los enlaces de namespace antes y después de entrar al chroot: readlink /proc/1/ns/pid readlink /proc/$(pgrep -f 'chroot /tmp/minicontainer')/ns/pid Si los dos comandos devuelven un número de inodo distinto, por ejemplo algo como pid:[4026531836] contra pid:[4026532891], confirmás que el proceso dentro del chroot vive en un namespace de PID diferente al del sistema. Si devuelven el mismo número, el namespace no se aplicó. > ⚠️ Ojo: chroot por sí solo no es una frontera de seguridad. Un proceso con privilegios de root dentro del chroot puede escapar si tiene acceso a syscalls como mknod o a un descriptor de archivo abierto fuera de la carpeta. Por eso runc combina chroot o pivot_root con namespaces de usuario y con capabilities recortadas, nunca lo usa solo. Casos de uso reales Kubernetes no ejecuta contenedores directamente: delega en containerd o CRI-O, que a su vez llaman a runc o a alguna alternativa compatible con OCI. Cuando un Pod se reinicia o se reprograma en otro nodo, lo que en realidad se destruye y se vuelve a crear es ese árbol de namespaces, cgroups y la raíz de archivos, no una "máquina" en el sentido tradicional. Los entornos de CI también dependen de este mismo mecanismo para aislar builds. Cuando GitHub Actions o GitLab CI corren un job dentro de un contenedor, están usando el mismo runtime de contenedores para evitar que un paso malicioso de un pipeline toque el resto del runner. Los proveedores de serverless que ofrecen funciones de corta duración suelen usar variantes más livianas de este mismo stack, a veces reemplazando namespaces por microVMs cuando necesitan un aislamiento más fuerte que el que da un proceso compartiendo kernel. En el borde, proyectos como K3s o MicroK8s usan el mismo stack de containerd y runc, pero recortado para correr en dispositivos con pocos recursos, como gateways industriales o cámaras con cómputo embebido. La ventaja frente a instalar la aplicación directamente sobre el sistema operativo del dispositivo es que un bug en el proceso queda contenido dentro de sus namespaces y cgroups, sin poder leer archivos de configuración de otros servicios que corren en el mismo gateway. Errores comunes y buenas prácticas El primer error clásico es asumir que un contenedor aísla tanto como una máquina virtual. No lo hace: todos los contenedores de un host comparten el mismo kernel, así que una vulnerabilidad de escalada de privilegios en el kernel (no en la aplicación) puede saltar de un contenedor a otro o al host. Por eso corren con seccomp y capabilities recortadas por defecto, y por eso --privileged es una bandera que hay que tratar como una alarma, no como un atajo cómodo para "que funcione". El segundo error es no entender que la imagen comprimida que bajás de un registry no incluye el kernel ni el init de verdad. Cualquier proceso que corra como PID 1 dentro del contenedor hereda responsabilidades que normalmente maneja systemd, como reapear procesos zombie. Si tu aplicación no maneja señales ni hijos huérfanos, vas a terminar con procesos zombie acumulándose hasta que el contenedor se reinicie. El tercero es confundir cgroups v1 con cgroups v2. La mayoría de distribuciones modernas, como Fedora o Ubuntu reciente, ya usan cgroups v2 por defecto, con una sola jerarquía unificada en lugar de controladores montados por separado. Si un contenedor se comporta distinto en tu laptop que en un servidor viejo, confirmá la versión con: cat /sys/fs/cgroup/cgroup.controllers Si ese archivo existe y lista controladores como cpu memory io pids, estás en cgroups v2. Si el comando falla, el sistema todavía usa la jerarquía v1. Comparativa con alternativas runc no es la única implementación de la especificación OCI, y elegir otra puede tener sentido según qué estés priorizando: velocidad de arranque, menor huella de memoria, o un aislamiento más fuerte que el de compartir kernel. OpciónCuándo usarlaVentajaLimitaciónruncRuntime por defecto de Docker y KubernetesImplementa el estándar OCI y es compatible con todo el ecosistema existenteEscrito en Go, necesita el truco de reejecución para simular fork()crunHosts con cgroups v2 que priorizan arranque rápidoEscrito en C, binario más chico y liviano que runcMenor adopción por defecto en algunas distribucionesgVisorMulti-tenant donde no confiás en el kernel compartidoIntercepta syscalls en espacio de usuario como capa extra de aislamientoOverhead en cargas con muchas llamadas al sistemaFirecrackerServerless con aislamiento tipo máquina virtualArranca una microVM liviana con su propio kernel por workloadNo es un runtime OCI puro, necesita su propio orquestador Profundizando: cgroups v2, user namespaces y OCI hooks Los user namespaces son la pieza que hace posible que "root dentro del contenedor" no sea "root en el host". Mapean un rango de UIDs del contenedor a un rango distinto, sin privilegios, en el host. El proceso puede creerse UID 0 adentro mientras en realidad corre como un UID sin privilegios, como 100000, desde afuera. Docker lo soporta desde hace años con la bandera --userns-remap, aunque pocos lo activan porque complica el manejo de volúmenes compartidos con el host. La especificación OCI que sigue runc también define "hooks", puntos de extensión donde un orquestador puede inyectar su propia lógica antes o después de ciertas etapas del ciclo de vida del contenedor, como prestart, createRuntime o poststop. Kubernetes y Docker usan estos hooks internamente para tareas como configurar la red del Pod antes de que el proceso principal arranque, sin tener que modificar runc. Otro detalle que separa a los runtimes OCI entre sí es cómo manejan CLONE_NEWCGROUP, el namespace de cgroup. Montarlo mal puede dejar que un proceso dentro del contenedor vea la jerarquía completa de cgroups del host en lugar de solo la suya, lo que filtra información sobre qué otros contenedores corren en la misma máquina. seccomp es la cuarta pieza que casi nunca se menciona junto a namespaces y cgroups. Filtra qué syscalls puede hacer un proceso, no qué puede ver ni cuánto puede consumir. El perfil por defecto de Docker bloquea una lista de syscalls peligrosas, como ptrace sin restricciones o mount arbitrario, incluso si el proceso tiene las capabilities que en teoría se lo permitirían. Podés ver el perfil activo de un contenedor corriendo docker inspect --format '{{.HostConfig.SecurityOpt}}' <contenedor>. Tu próximo paso: correteá el mini runtime de la sección anterior, agregale un cgroup propio escribiendo límites directamente en /sys/fs/cgroup/, y confirmá que podés limitarle la memoria a 50 MB antes de que el kernel mate el proceso con el OOM killer. 📖 Resumen en Telegram: Ver resumen Preguntas frecuentes ¿Qué diferencia a runc de una máquina virtual? Una máquina virtual arranca su propio kernel completo sobre un hipervisor; un contenedor ejecutado por runc comparte el mismo kernel que el host y solo cambia qué puede ver y cuánto puede consumir ese proceso. Por eso un contenedor arranca en milisegundos y una VM tarda segundos, pero también por eso una falla del kernel compartido afecta a todos los contenedores a la vez. ¿Por qué dockerd necesita privilegios de root? Porque crear ciertos namespaces, sobre todo el de red, y manipular cgroups son operaciones que el kernel de Linux históricamente reservó a procesos con privilegios de superusuario. El modo rootless de Docker evita esto reimplementando partes del stack de red en espacio de usuario. ¿Puedo usar namespaces de Linux sin instalar Docker? Sí. El comando unshare, parte del paquete util-linux que ya viene en casi toda distribución, permite crear namespaces de PID, red, mount, UTS e IPC directamente desde la terminal, sin pasar por dockerd ni por runc. ¿Qué es pivot_root y por qué runc lo prefiere sobre chroot? pivot_root cambia cuál punto de montaje actúa como raíz del sistema de archivos para un proceso, y permite desmontar la raíz anterior después del cambio. chroot solo cambia la raíz aparente sin ese paso extra, lo que deja una vía de escape si el proceso ya tenía un descriptor de archivo abierto fuera de la nueva raíz. ¿El modo rootless de Docker tiene las mismas prestaciones de red que el modo normal? No del todo. Al no tener acceso de root a las interfaces del host, el modo rootless depende de herramientas como slirp4netns o pasta para simular la red en espacio de usuario, lo que agrega una capa de traducción que el modo con privilegios no necesita. ¿Cuándo conviene usar gVisor o Firecracker en lugar de runc? Cuando el aislamiento de namespaces y cgroups no alcanza, por ejemplo en entornos multi-tenant donde cargas de distintos clientes comparten el mismo host. gVisor intercepta las syscalls en espacio de usuario y Firecracker arranca una microVM liviana con su propio kernel, ambos a costa de más overhead que runc puro. Referencias - Code and Cake: el artículo original que documenta estas rarezas de Docker mientras su autor construye un runtime de contenedores propio.- GitHub: opencontainers/runc: código fuente de la implementación de referencia del runtime OCI que usan Docker y Kubernetes.- GitHub: opencontainers/runtime-spec: la especificación OCI que define cómo debe comportarse cualquier runtime de contenedores.- Wikipedia: Linux namespaces: resumen de los ocho tipos de namespace que expone el kernel de Linux.- Documentación oficial de Docker: referencia del daemon, el modo rootless y la arquitectura cliente-servidor. 📱 ¿Te gusta este contenido? Únete a nuestro canal de Telegram @programacion donde publicamos a diario lo más relevante de tecnología, IA y desarrollo. Resúmenes rápidos, contenido fresco todos los días.

Not to contradict anything you wrote, but if you look at the language stats for opencontainers/runc, it has 68.5% Go, 27.1% Shell and 3.5% C, so it's fair to say that it was "written in Go", even though the most important part was C - same as e.g. Doom was written in C, even though it used assembly for optimizing the performance-sensitive parts.

I Replaced runc with a Rust Container Runtime (youki) for a Week The Boring Layer That Matters Most Every container you've ever run — docker run hello-world included — went through a tiny binary called runc before it became a process. It creates the namespaces, sets up the cgroups, drops privileges, and hands off to your entrypoint. You never think about it. And that's the problem. runc is written in Go + C (yes, C — for the nsenter and pivot_root bits that Go can't handle cleanly). It's been the reference implementation since 2014. It's also the layer where CVE-2024-21626 (a container escape) lived. What if that layer was memory-safe by construction? Enter youki — a container runtime written entirely in Rust, now a CNCF Sandbox project. --- What Is youki? youki (Japanese for "container") is an OCI runtime-spec implementation in Rust. It's a drop-in replacement for runc: same CLI, same spec, same integration path through containerd, Podman, or Docker. | Fact | Detail | |---|---| | Language | Rust (pure, except a libseccomp-rs FFI shim being replaced) | | CNCF status | Sandbox (accepted Oct 17, 2024) | | GitHub | youki-dev/youki | | Stars | ~7,300 | | License | Apache 2.0 | | Adopters | Azure Kubernetes Service, Docker, containerd/runwasi | | Contributing orgs | Microsoft, Docker, Red Hat | The key differentiator vs. runc: no Go runtime startup, no C nsenter binary, no mixed-language memory safety boundary. The entire create → start → delete lifecycle lives in one Rust binary with compile-time safety guarantees. --- The Benchmarks (Honest Version) The headline number from youki's own repo: Runtime Time (mean ± σ) Range (min … max) vs youki youki 111.5 ms ± 11.6 ms 84.0 ms … 142.5 ms 100% runc 224.6 ms ± 12.0 ms 190.5 ms … 255.4 ms 200% crun 47.3 ms ± 2.8 ms 42.4 ms … 56.2 ms 42% Youki is ~2× faster than runc for create → start → delete. But crun is ~2.5× faster than youki. Here's the honest context: | Scenario | Does it matter? | |---|---| | Single long-lived container | No. 100ms of startup is noise over a 7-day uptime. | | Serverless: 10k containers/min | Yes. A 100ms savings × 10k = 1,000 CPU-seconds per minute reclaimed. | | K8s pod churn (CI/CD, spot nodes) | Moderately. Image pull + CNI + scheduling dominate; runtime is ~5–10% of total. | | Edge/embedded (tight RAM) | Yes. youki's binary is smaller and RSS is lower per instance. | The real win isn't raw speed — it's the Rust safety guarantee. You're not betting your node's security on a C memcpy in nsenter. --- Setup: Replacing runc on a K8s Node Prerequisites - Kubernetes 1.28+ with containerd (not Docker Engine) - Linux kernel 5.10+ (youki drops older kernel support to use modern features) - cargo / just for building from source (or use a pre-built binary) Step 1: Build youki git clone https://github.com/youki-dev/youki.git cd youki just build # Binary: ./target/release/youki Or grab a release binary from GitHub Releases. Step 2: Register with containerd Edit /etc/containerd/config.toml (or use containerd config path): [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.youki] runtimeType = "io.containerd.youki.v1" Then install the shim (youki ships a containerd-compatible shim): sudo cp target/release/youki /usr/local/bin/ sudo cp target/release/containerd-shim-youki-v1 /usr/local/bin/ Restart containerd: sudo systemctl restart containerd Step 3: Create a RuntimeClass apiVersion: node.k8s.io/v1 kind: RuntimeClass metadata: name: youki handler: youki scheduling: nodeSelector: kubernetes.io/arch: amd64 kubectl apply -f runtimeclass-youki.yaml Step 4: Run a Pod with youki apiVersion: v1 kind: Pod metadata: name: youki-test spec: runtimeClassName: youki containers: - name: app image: nginx:alpine resources: requests: memory: "32Mi" cpu: "10m" kubectl apply -f youki-pod.yaml kubectl get pod youki-test -o wide # youki-test 1/1 Running 0 2s Verify it's actually using youki: kubectl exec youki-test -- cat /proc/1/cgroup # Confirms cgroup v2 hierarchy managed by youki --- Step 5: Run It for a Week (What I Noticed) I ran a 3-node k3s cluster with a mix of workloads: - 20 long-lived API pods (Go, ~50 MB each) - A CI pipeline spawning 50–200 short-lived pods/day - A cron job that scales a batch processor 0 → 10 → 0 every hour What went well: 1. Zero crashes, zero OOMs in the runtime itself. The Rust error handling is visible in logs — when something goes wrong, you get a structured Error with context, not a segfault. 2. Short-lived pods were noticeably snappier. My CI pods (build → test → delete) came up ~30–50ms faster than runc. Not dramatic per-pod, but across 200 pods/day it added up. 3. Rootless mode was cleaner. youki create with --rootless worked without the newuidmap/subuid gymnastics that runc sometimes needs. 4. Binary size. youki: ~8 MB. runc: ~12 MB. crun: ~3 MB. On an edge node with 512 MB RAM, every MB counts. What was rough: 1. cgroups v1 support is there but v2 is the path of least resistance. If your distro still defaults to v1 (looking at you, CentOS 7), you'll hit edge cases. 2. No --checkpoint/--restore (CRIU) yet. If you rely on live migration or pause/resume, you're stuck on runc. 3. Docs are thin. The user docs cover the basics but there's no "migrating a production cluster from runc to youki" guide. You're on your own. 4. One C dependency remains. libseccomp-rs is an FFI wrapper around the C libseccomp. The team is working on a pure-Rust replacement, but it's not done. --- youki vs. runc vs. crun: Where Each Wins | Dimension | runc | crun | youki | |---|---|---|---| | Language | Go + C | C | Rust | | Memory safety | ❌ (C portion) | ❌ | ✅ (99% — one FFI shim) | | Create→delete (benchmark) | 224 ms | 47 ms | 111 ms | | Binary size | ~12 MB | ~3 MB | ~8 MB | | cgroups v2 | ✅ | ✅ | ✅ | | Rootless | ✅ | ✅ | ✅ (cleanest) | | CRIU (checkpoint/restore) | ✅ | ✅ | ❌ (roadmap) | | Wasm support | ❌ | ❌ | ✅ (first-class) | | CNCF governance | ❌ (OCI) | ✅ (Sandbox) | ✅ (Sandbox) | | Production battle-testing | 10+ years | 5+ years | ~3 years | | Best for | Default, compatibility | Max performance, minimal footprint | Safety + Wasm + future-proofing | The honest takeaway: If you need maximum raw speed today, crun wins. If you need the safest runtime with a path to Wasm workloads, youki is the pick. runc remains the "it just works" default. --- The Wasm Angle (Why This Matters More Than You Think) Here's the part that excites me: youki is the low-level runtime that powers runwasi (the containerd shim for Wasm workloads) and, by extension, SpinKube. The architecture: containerd └── containerd-shim-runwasi (Rust) └── youki libcontainer (Rust) ← shared library └── Wasmtime / Wasmer / WasmEdge Youki's libcontainer crate is used as a library by other Rust container tools. That means the same memory-safe, OCI-compliant primitives that run your nginx:alpine pod also run your Wasm modules. One codebase, two workload types, same security guarantees. This is the "Rust container stack" taking shape: | Layer | Rust project | Role | |---|---|---| | Runtime | youki | OCI container lifecycle | | Shim | runwasi | Wasm on containerd | | Orchestrator | SpinKube | Wasm on Kubernetes | | Image tooling | naftiko | Build OCI/Wasm images | | Spec | oci-spec (Rust crate) | The spec itself, in Rust | Youki is the foundation. If it's not solid, nothing above it is. --- What's Missing (and Where Funding Helps) youki is in CNCF Sandbox — the earliest maturity level. The project is healthy (active commits, 80+ contributors, backing from Microsoft/Docker/Red Hat) but under-resourced for what it needs to do: - Pure-Rust seccomp (eliminate the last C dependency) — significant engineering effort - SELinux in Rust — needed for RHEL/CentOS production adoption - CRIU support — checkpoint/restore is table-stakes for production - OCI conformance test suite — a dedicated, automated compliance harness - Docs and migration guides — "how do I move a 500-node cluster from runc to youki?" - Security audit — a formal audit would be a major trust signal for enterprise adoption - Conference presence — KubeCon, RustConf, containersc — the project needs more visibility No LFX Crowdfunding campaign is active yet. If you use or evaluate youki, the highest-impact things you can do: 1. Contribute — even docs, test cases, or bug reports. GitHub 2. Run it in production and report what breaks (or doesn't) 3. Write about it — this article is a start, but we need more "I ran youki in a real cluster" content 4. Watch for an LFX Crowdfunding launch — CNCF funding page --- TL;DR | Question | Answer | |---|---| | Is youki production-ready? | For new, non-critical workloads: yes. For a 500-node cluster: not yet. | | Is it faster than runc? | ~2× for create/delete. Yes, but crun is faster still. | | Is it safer than runc? | Yes — memory-safe Rust eliminates the C nsenter attack surface. | | Should I switch today? | If you're running Wasm workloads or want to be early on the Rust container stack: yes. Otherwise: watch this space. | | What's the risk? | It's a Sandbox project. The API could shift. CRIU is missing. Docs are thin. | --- Final Thought The container runtime is the most underappreciated layer in the stack. It's the thing that's always running, that touches every syscall, that sits between your workload and the kernel. For a decade, that layer was a Go binary with a C sidecar. Now there's a Rust option with CNCF governance, major-company backing, and a clear path to being the foundation for the next generation of container workloads (including Wasm). I ran it for a week. It didn't break. It was faster. And the Rust error messages in the logs were genuinely pleasant to read. That's about as good as "boring infrastructure" gets. --- Have you tried youki in production? What broke? What didn't? Drop your experience in the comments — I'm especially curious about anyone running it alongside runwasi or SpinKube. If you want to support the project, the best things right now are contributing code/docs and writing about your experience. The CNCF funding infrastructure is ready when the project is.

Docker exit code 137: telling the causes apart from the logs A container has stopped with exit code 137. docker ps -a shows Exited (137), and docker inspect shows "ExitCode": 137. This value does not tell you whether the container hit its memory limit, the host as a whole ran out of memory, or Docker or some other process stopped it. I reproduced this on a test VM, changed how the container was stopped, and checked what each case left in the kernel log and the Docker daemon log.[^env] [^env]: I tested on a VM started with KVM inside a GitHub Actions runner (Ubuntu 24.04, kernel 6.8, 512MiB memory, no swap, cgroup v2), using Docker 29.1.3 from Ubuntu's docker.io package (cgroup driver: systemd). The scripts and results are in show-your-work and in the GitHub Actions run. Which log points to which cause The OOM killer is the mechanism the kernel uses when memory runs short: it picks a process and kills it. For each way the container stopped, the kernel log and the Docker daemon log contained the following.[^log-source] [^log-source]: The kernel log strings come from kernel 6.8: the call to oom_kill_process() and the line that prints Killed process. The Docker daemon log lines come from Docker 29.1.3: the line that logs using the force and the lines that log the warning containing hasBeenManuallyStopped=true. | What happened | Kernel log | Docker daemon log | |---|---|---| | The container hit its memory limit and the OOM killer killed it | Memory cgroup out of memory: Killed process | none | | The host as a whole ran out of memory and the OOM killer killed it | Out of memory: Killed process | none | | The container did not exit on docker stop, so Docker killed it | none | using the force | | Someone ran docker kill | none | hasBeenManuallyStopped=true (if the container has a restart policy; no using the force) | | Cannot be told from the logs | none | none | The last row covers a process on the host running kill -9, the app returning 137 itself, and a container without a restart policy being stopped with docker kill. For a container with a restart policy, docker stop also logs hasBeenManuallyStopped=true. If a using the force line comes before it, it was docker stop. If not, it was docker kill. When a container without a restart policy is stopped with docker kill, the Docker daemon log has no line that identifies it as docker kill. The only trace is the kill event in docker events, and the Docker daemon keeps only the most recent 256 events in memory, counted across all containers on the host.[^events-limit] [^events-limit]: eventsLimit in the Docker 29.1.3 source. Read the kernel log with journalctl -k. A line starting with oom-kill: appears just before the Killed process line. The task_memcg= value on that line contains the container ID, which tells you which container it was.[^oom-kill-line] When the host as a whole runs out of memory, the container that gets killed is not necessarily the one that used up the memory. In the test, the OOM killer killed the app container, which held more memory than the container whose memory kept growing. [^oom-kill-line]: Kernel 6.8: where the oom-kill: line is printed and where task_memcg= is printed. journalctl -k | grep -E 'Killed process|oom-kill:' Read the Docker daemon log with journalctl -u docker. The string after container= is the container ID. journalctl -u docker | grep -E 'using the force|hasBeenManuallyStopped=true' Using the table in production In production, there are situations where the values in docker inspect cannot be relied on. The logs in the table still tell the causes apart in those situations. The container has restarted If RestartCount in docker inspect is 1 or more, the container has restarted. When a container with a restart policy restarts, ExitCode in docker inspect goes back to 0 and OOMKilled back to false, and nothing there shows that it stopped with 137. After a restart, FinishedAt in docker inspect still holds the time of the previous stop. If the kernel log has a Killed process line at about the same time, the previous stop was a kill by the OOM killer. FinishedAt is in UTC, so use journalctl -k --utc to line up the timestamps. OOMKilled is false State in docker inspect has a value called OOMKilled, which Docker sets to true when a process in the container is killed by the OOM killer. But it sometimes stays false even though the OOM killer did the kill. Decide whether the OOM killer killed it from the Killed process line in the kernel log (journalctl -k). What exit code 137 indicates PID 1 is the first process started in the container, the one given in ENTRYPOINT or CMD. When PID 1 exits, the container exits, and the exit code is determined by how PID 1 ended. When PID 1 is ended by a signal, the exit code is 128 plus the signal number. SIGKILL is 9, so the code is 137.[^exit-status] [^exit-status]: runc's ExitStatus(). The exit code can be 137 even when PID 1 itself did not receive SIGKILL. If an init is started ahead of the app and runs the app as its child, as with docker run --init, then when the app is ended by SIGKILL, init exits with 137.[^tini] With init in place, the logs for a kill at the container's memory limit and for kill -9 from a host process looked the same as without init. [^tini]: The init that --init starts is tini, which returns 128 plus the signal number when its child is ended by a signal (tini v0.19.0). SIGKILL is sent to a container's processes by one of the following: - The kernel's OOM killer - Docker (when the container does not exit on docker stop, and on docker kill) - Another process on the host Every kill in the table is this SIGKILL. The exit code does not tell you who sent it, so use the logs in the table to tell the senders apart. What was not checked I did not check mechanisms outside the kernel that watch for memory shortage and kill processes (systemd-oomd and earlyoom). Appendix: each way of stopping and what it left --memory 128m, app allocates 256MiB - Kernel log: Memory cgroup out of memory - Docker daemon log: none - docker events: oom, die - OOMKilled: true (false in 0 to 1 of 3 tries per full test run) Another container with no limit uses up the host's memory - Kernel log: Out of memory - Docker daemon log: none - docker events: oom, die (order varies between tries) - OOMKilled: true (false in 0 to 2 of 10 tries per full test run) docker stop -t 2 - Kernel log: none - Docker daemon log: using the force - docker events: kill (15), kill (9), stop, die - OOMKilled: false docker stop -t 2, with a restart policy - Kernel log: none - Docker daemon log: using the force, hasBeenManuallyStopped=true - docker events: kill (15), kill (9), stop, die - OOMKilled: false (does not restart) docker kill - Kernel log: none - Docker daemon log: none - docker events: kill (9), die - OOMKilled: false docker kill, with a restart policy - Kernel log: none - Docker daemon log: hasBeenManuallyStopped=true - docker events: kill (9), die - OOMKilled: false (does not restart) kill -9 from the host to the app - Kernel log: none - Docker daemon log: none - docker events: die - OOMKilled: false The app returns 137 itself - Kernel log: none - Docker daemon log: none - docker events: die - OOMKilled: false --memory 128m, with a restart policy - Kernel log: Memory cgroup out of memory - Docker daemon log: none - docker events: oom, die, start - OOMKilled: false after the restart (ExitCode is 0)

How Docker Works Under the Hood: Watch Namespaces Isolate a Container and cgroups Stop a Fork Bomb On a Linux host, start an nginx container, then run ps aux. nginx is right there in the list, with an ordinary host PID. No guest kernel booted and no hypervisor started. That one observation is the key to how Docker works under the hood: a container is a normal Linux process that the kernel agrees to lie to (namespaces) and to ration (cgroups). The rest of docker run is plumbing that sets those two things up.  This post walks the plumbing with two browser simulators, then shows the same kernel features on a real machine, no Docker required. Disclosure: I help build the simulators; they are free, run in the browser, and need no signup. The path of one docker run The How Docker Works Under the Hood simulator takes one command, docker run -p 8080:80 nginx, and steps down the stack in eight stages: the Docker CLI, the Docker daemon, the registry pull, containerd, the OCI bundle, runc, the running container, and the Linux kernel. Press Play, or use Prev and Next to go at your own pace. Tick "nginx already pulled" and the registry stage drops out (the counter goes from Step 1 / 8 to Step 1 / 7), the first lesson: a pull only happens on a cache miss, and only missing layers come down. The stages worth pausing on: 1. The CLI is a REST client. It sends an HTTP request to dockerd over /var/run/docker.sock. The stage shows a curl --unix-socket request to the daemon's /info endpoint: it starts nothing, but it uses the same transport the CLI does. 2. dockerd does not start your process. It hands off to containerd, which unpacks the image layers into a snapshot and tracks the container's lifecycle. 3. The OCI bundle is the whole recipe. It is a config.json plus a root filesystem. The excerpt in the simulator lists the five namespaces to create, with the type names from the OCI runtime spec: pid, network, mount, uts, ipc. The kernel shortens two of them: /proc/<pid>/ns and lsns say net and mnt, as you will see below. The command for this stage is runc spec, which writes a sample config.json so you can read the real shape. 4. runc does the kernel work and leaves. It creates the namespaces and the cgroup, pivots into the rootfs, drops capabilities, and execs nginx as PID 1 inside the container. Then runc exits and a shim keeps the container attached to containerd. Every stage has a copy button next to its real command. The last stage ends on four labels: namespaces, cgroups, networking, mounts. Those are the container. Namespaces: what the process can see You can look at your own namespaces right now on any Linux box. Every process has a set, and each one is a link in /proc: $ ls -l /proc/self/ns # trimmed: link targets only, 6 of 10 entries cgroup -> cgroup:[4026531835] ipc -> ipc:[4026531839] mnt -> mnt:[4026531841] net -> net:[4026531840] pid -> pid:[4026531836] uts -> uts:[4026531838] Two processes share a namespace when the numbers match; a container's process shows different ones. The unshare tool from util-linux makes new ones by hand. On Debian 12, as a normal user: $ unshare --user --map-root-user --pid --fork --mount-proc ps -o pid,user,cmd PID USER CMD 1 root ps -o pid,user,cmd ps believes it is PID 1 and that it runs as root. Both are namespace tricks: a new PID namespace starts counting at 1, and the user namespace maps your normal UID to root inside it only. The UTS namespace does the same for the hostname: $ unshare --user --map-root-user --uts sh -c 'hostname demo-box; hostname' demo-box $ hostname raspberrypi If your distro restricts unprivileged user namespaces, run them with sudo and drop --user --map-root-user. The namespaces(7) man page lists all eight types. cgroups: what the process can use Namespaces hide things. They do not limit anything. A process in its own PID namespace can still create processes until the host runs out. The classic proof is the fork bomb: a tiny shell function named : that calls itself twice, one copy piped into another in the background. Every copy makes two more, the process table fills, and the machine stops responding. Do not run it. Use the Fork Bomb Simulator instead, which draws the process tree without forking anything real. Hover the code explainer to see what each of its ten pieces does, then press Start. The process count climbs until it hits the PID limit (a slider from 64 to 1024, 512 by default), the log prints FATAL: fork() failed - Resource temporarily unavailable, and the screen reads SYSTEM CRASHED. Now pick one of the three Protection cards and start again: ulimit -u (a per-user process cap), cgroups (pids.max), or systemd (TasksMax in a unit file). In the simulation, each one caps the run at 100 processes, and it ends with SYSTEM PROTECTED instead. That only works while the PID limit slider is above 100: set it to 64 and the system runs out first, protection or not. Same bomb, same growth, stopped at a line you drew in advance. On a real system they count different things. ulimit -u counts every process and thread of your user ID across the whole machine, and the kernel does not enforce it for root. When I ran ulimit -u 50 in a shell on the Pi, the very first fork() failed, because my user already owned 191 tasks elsewhere. pids.max and TasksMax count tasks inside one cgroup, whoever owns them. (The cgroups card shows the cgroup v2 form: echo 100 > /sys/fs/cgroup/user.slice/user-1000.slice/pids.max.) Try the limit for real On a systemd machine with cgroup v2, where the user manager has the pids controller delegated to it, you can put a command in a throwaway cgroup with a task limit, no root needed. To check, cat /sys/fs/cgroup/user.slice/user-$(id -u).slice/user@$(id -u).service/cgroup.controllers must list pids (on the Pi it prints cpu pids). This is not a fork bomb. It is a loop that stops after 50 children, run inside a scope capped at 10 tasks: $ systemd-run --user --scope -p TasksMax=10 python3 -c ' import os, time n = 0 try: for _ in range(50): if os.fork() == 0: time.sleep(2) os._exit(0) n += 1 except OSError as e: print("started", n, "children, then:", e) ' Running scope as unit: run-r01155c5e085f497f94c998714d8323f5.scope started 9 children, then: [Errno 11] Resource temporarily unavailable Nine children plus the Python parent is ten tasks, the value systemd wrote to the scope's pids.max. The tenth fork() failed with EAGAIN, which is what the kernel's cgroup v2 docs promise. In the simulator, the same refusal shows as cgroup: fork rejected, pids.max (100) reached on the cgroups card. Nothing else on the machine noticed. A version of the script that also reads its own cgroup files printed pids.events: max 1 at the end: that file counts refused forks, so it is the one to alert on. Docker exposes the same controller per container: docker run --pids-limit 100 nginx sets that ceiling on the container's cgroup (docs). Without that flag, a runaway container is bounded only by the limits the host already has. On the Pi, systemd's DefaultTasksMax is 3914 (15% of kernel.threads-max), but whether a container's unit inherits a default like that depends on the runtime. Check instead of guessing: on a cgroup v2 host, docker run --rm alpine cat /sys/fs/cgroup/pids.max prints the container's limit, and max means none. The whole trick Side by side, the two simulators put how Docker works under the hood in one sentence: runc asks the kernel for new namespaces so the process sees its own small world, and puts it in a cgroup where the limits you set cap what it can use. A cgroup with no limits caps nothing, and a PID cap does not cap CPU or memory. A limit also needs its controller: the Pi boots with cgroup_disable=memory, and a scope with MemoryMax=50M let a process allocate 100 MiB without an error. dockerd, containerd and the registry are how the recipe gets built and delivered. For the longer tour, with commands to run at each layer of a real Docker host, read How Docker Really Works, From docker run to the Kernel. For the controllers themselves, see cgroups(7). What surprised you most the first time you looked under the hood of a container? Tell us in the comments.