A one-line gVisor bug left pods stuck in Terminating: random memory layout moved every region except one queue
Follow-up to yesterday's gVisor intro. This is the stub-memory bug I mentioned there, with the debugging steps.
gVisor's systrap platform runs app code in host "stub" processes and shifts their shared memory by a random, page-aligned gap at startup, so the addresses are hard to guess. In stub_unsafe.go three regions get += gap. The spinning queue, sitting between them in the layout, did not.
With a big gap the queue sits alone in empty space. With a small one, the shifted per-thread stacks land on the queue's old address. The stub then spins in spinning_queue_remove_first waiting for a timestamp slot that stack bytes keep at 0. That check sits before the timeout, so it never gives up. The Sentry waits on a context no stub ever acknowledged, and my 30s fail-stop, which kills a stuck stub process, only arms once an acknowledged context was interrupted. The sandbox never exits.
On Kubernetes it looks like this: a pod stuck in Terminating, one stub process pinned at 100% of a core, and runsc kill hanging.
Back of the envelope from the code: the gap is uniform below a random room of up to 1 TiB, so the odds come out roughly 1 in 100,000 starts for a 16-CPU sandbox, and about 1 in 35,000 at 64. That is an upper bound, since a hang also needs the overlapped stack in use. It had been there since the spinning queue landed in March 2023.
Forcing a small gap in an arm64 VM: stock build hung 2 of 2, with the one-line fix 20 of 20 clean.
If you run gVisor with systrap (release-20230417.0 through release-20261005.0): upgrade to a runsc build that contains c7f458aa3. Until then --systrap-disable-fast-path (release-20260622.0 or later) stops the spinning, at some syscall latency cost. To clear a hung pod, kill its runsc-sandbox process on the node.
Fix: gvisor#15573
Write-up with the debugging steps and diagrams: catchkill9.dev
1 point · 2 comments
My fix for this hang was already running. The sandbox hung anyway.
---
title: "My fix for this hang was already running. The sandbox hung anyway."
published: true
tags: gvisor, kubernetes, go, debugging
canonical_url: catchkill9.dev
cover_image: catchkill9.dev
---
_gVisor shifts its stub memory by a random gap, but since 2023 one queue was not moved with it. When the gap is small, that queue can land on a thread's stack and the sandbox can hang for good. The fix was one line._
Same 16-thread workload. Same arm64 VM. Same forced memory layout. Stock gVisor hung 2 runs out of 2, with a stub process stuck at 100% of a core. Add one line, and it ran clean 20 times out of 20.
The only variable was where one queue sat in memory.
Getting to that line took me through my own safety net first. In August I had shipped a fail-stop to gVisor: if an app thread stays stuck for 30 seconds and gVisor's interrupt doesn't free it, gVisor kills the one stub process running it instead of letting it hang the sandbox. A month later a handful of sandboxes hung with that fail-stop running. The fix is google/gvisor#15573.
gVisor, the short version
> Deeper intro: gVisor, a kernel in Go.
gVisor runs a second kernel, the Sentry, written in Go, between the app and Linux. On its default platform, systrap, the app's code runs in host processes called stubs. The Sentry puts work for the stubs on a queue in shared memory, and every system call traps back to it.
- context: one app thread's saved state, waiting to run or waiting for the Sentry.
- interrupt: a signal the Sentry sends to the stub thread running a context, to force it back.
- spinning queue: idle stub threads busy-waiting for work, private to the stub. Each entry is stamped with when it started spinning.
What did I think it was?
The bug from last time (gVisor: one lost signal): one lost interrupt left an app thread stuck. Same symptoms: pods stuck in Terminating, every Sentry thread asleep, hundreds of kills from Kubernetes queued behind one app thread that never paused.
One thing didn't match. Last time the stuck sandbox burned no CPU. This time a stub was burning a full core. And my fail-stop was supposed to catch exactly this kind of hang.
Why didn't my fail-stop fire?
A core dump of the Sentry showed one app thread waiting on its context, with AckedTime=0 and an owner of 0xfefefefe, the "no thread" marker. No stub had picked it up since it was queued.
The fail-stop only fires once the Sentry has sent an interrupt, and the Sentry only interrupts a stub thread that took the context. Nobody took this one. No interrupt, no fail-stop.
So my safety net had a hole, and it was mine. It covered a stub that took the work and then stopped responding. It didn't cover work no stub ever took.
!Two paths: a stub took the context, so an interrupt is sent and the fail-stop kills one stub at 30s; no stub took it, so there is no interrupt, no fail-stop, and the sandbox hangs
What was the stub doing at 100% CPU?
Every sample of the busy stub landed in one tight loop in spinning_queue_remove_first. It waits while the oldest entry's timestamp reads 0, since a new entry writes its timestamp last. Here it read 0 every time, and that check sits before the timeout check, so the loop never gives up.
Its other threads never took the context either. From the code, the Sentry only wakes another stub thread when none looks available (subprocess.go). The stuck spinner still counted as one.
Where was the queue, really?
In the stub's /proc/<pid>/maps, the spinning queue's address wasn't in its own private page. It sat inside a per-thread stack mapping. Its start field read 0x7ff9, which is where the high half of a 0x00007ff9... pointer lands when a stack stores one there. The queue was reading a thread's stack as queue data.
How does a queue end up on a stack?
gVisor puts a random gap into the stub's memory layout, so attackers can't guess where things are. Every region after it has to move by the gap. Here is stub_unsafe.go before the fix, with the missing line marked by me:
gap := uintptr(rand.Uint64()) * hostarch.PageSize %
(maximumUserAddress - stubStart - mapLen)
stubSysmsgStack += uintptr(gap)
stubContextQueueRegion += uintptr(gap)
stubContextRegion += uintptr(gap)
// missing: stubSpinningThreadQueueAddr += uintptr(gap)
Three regions move. The spinning queue, between them in the layout, stays put. With a big gap it sits alone in empty space. With a small one, the shifted stacks slide over its old address. It's been that way since the commit that introduced the spinning queue, in March 2023.
!Stub memory before and after a small random gap: thread stacks, context queue and contexts move by the gap, the spinning queue does not, and the moved stacks land on top of it
It's an availability bug, not an isolation one: the queue lands on memory the stub could already write, which the Sentry treats as untrusted anyway. The damage stays inside the sandbox.
Why did it hide for three years?
The gap is random below the room M left above the stub block, and M is itself random up to R, about 1 TiB on x86-64. The queue gets hit when the gap is smaller than the stack area T, about 612 KiB for a 16-CPU sandbox:
P = (T / R) * (1 + ln(R / T)) ~ 1 / 114,000
About 1 in 100,000 starts at 16 CPUs, and about 1 in 35,000 at 64. Waiting for it was useless.
!Odds of a hang per sandbox start, derived from stub_unsafe.go: 1 in 100,000 at 16 CPUs, 1 in 35,000 at 64 CPUs
Can I prove it with one variable?
So I forced it: a debug-only patch picks a gap that puts the old queue address inside a live thread stack.
!Same forced gap, two builds: stock gVisor hung 2 of 2, with the one-line fix clean 20 of 20
| Build, same forced gap | Queue lands on a stack | Result |
|---|---|---|
| Stock gVisor | yes | hung 2 of 2, stubs at 100% CPU, no fail-stop log line |
| Stock + stubSpinningThreadQueueAddr += uintptr(gap) | no, it moves with the gap | clean, 20 of 20 |
The code is shared by both architectures; arm64 was just faster to boot. The fix adds a layout test, and the repro tool is here.
Are you affected?
Systrap sandboxes on release-20230417.0 through release-20261005.0 can hit it; google/gvisor#15572 has details by version. The tell: a stub pinned at 100% forever, a pod stuck in Terminating, runsc kill hanging.
1. Upgrade to any runsc build that contains c7f458aa3, the first release after release-20261005.0.
2. Until then, --systrap-disable-fast-path (on release-20260622.0 or later) stops stub threads from spinning, so the spinning queue is never used. It's a partial workaround: it costs syscall latency, and a much rarer tiny gap can still hit the context queue. Only the upgrade fixes that.
3. To clear a hung sandbox, kill its Sentry on the host: the runsc-sandbox process ending with the pod sandbox ID from crictl pods. Its stubs die with it.
What did the AI do?
AI agents read the dump, mapped a program counter to a source line, built the VM repro and drafted the upstream PR. I made the calls, starting with not stopping at the hole in my own fail-stop.
A few things I'm keeping:
- Run the cheap check before the theory. One busy core killed my first guess.
- Your own safety net has holes, so look there first. Mine only covered work a stub had taken.
- Your debugging tool can kill the patient. A gcore of the Sentry faults in the sandbox's shared memory and can OOM it; dump the stub first.
- When a bug needs bad luck, make the bad luck happen on demand. A forced gap turned a 1 in 100,000 hang into 2 of 2.
The line is merged. That queue can't land on a stack anymore.
---
Thanks to konstantin-s-bogom for reviewing and approving the fix, and to avagin, who wrote the spinning queue and fixed the same symptom from a different cause in 2024 (google/gvisor#10000). Investigation assisted by Claude Code, with Codex as a second reviewer.
1 comment
gVisor: Application Kernel for Containers (20k stars)
What it does
All text below is quoted directly from the project README.
>gVisor provides a strong layer of isolation between running applications and the host operating system. It is an application kernel that implements a Linux-like interface. Unlike Linux, it is written in a memory-safe language (Go) and runs in userspace.
gVisor includes an Open Container Initiative (OCI) runtime called runsc that makes it easy to work with existing container tooling. The runsc runtime integrates with Docker and Kubernetes, making it simple to run sandboxed containers.
What isn't gVisor?
- gVisor is not a syscall filter (e.g. seccomp-bpf), nor a wrapper over Linux isolation primitives (e.g. firejail, AppArmor, etc.).
- gVisor is also not a VM in the everyday sense of the term (e.g. VirtualBox, QEMU).
gVisor takes a distinct third approach, providing many security benefits of VMs while maintaining the lower resource footprint, fast startup, and flexibility of regular userspace applications.
Why does gVisor exist?
Containers are not a sandbox. While containers have revolutionized how we develop, package, and deploy applications, using them to run untrusted or potentially malicious code without additional isolation is not a good idea. While using a single, shared kernel allows for efficiency and performance gains, it also means that container escape is possible with a single vulnerability.
>gVisor builds on x86_64 and ARM64. Other architectures may become available in the future.
Repository
google/gvisor
Repository stats
|Metric|Value|
|:-|:-|
|Stars|19,574|
|Forks|2,024|
|Last release|release-20260928.0 (2026-09-30)|
|Languages|Go 76%, C++ 18.1%, Starlark 3.2%|
|Contributors|354|
|AI contributor accounts (Claude / Codex / Cursor / Copilot)|None|
|AI co-authored commits|None|
Main contributor
|Field|Value|
|:-|:-|
|Name|Ayush Ranjan|
|GitHub|@ayushr2|
|Commits to this repo|929 (top human contributor; gvisor-bot has 1,461)|
|Bio|Member of Technical Staff at Modal. Previously worked on gVisor at Google.|
|Company|@modal-labs|
|Location|San Fransisco|
|Website|ayushr.com|
|Followers|102|
|Public repos|15|
|On GitHub since|2016-09-01|
Get the best repos from this community twice a week, 100% free*:* join the newsletter
22 points · 1 comment
Mentions
A one-line gVisor bug left pods stuck in Terminating: random memory layout moved every region except one queue Follow-up to yesterday's gVisor intro. This is the stub-memory bug I mentioned there, with the debugging steps. gVisor's systrap platform runs app code in host "stub" processes and shifts their shared memory by a random, page-aligned gap at startup, so the addresses are hard to guess. In stub_unsafe.go three regions get += gap. The spinning queue, sitting between them in the layout, did not. With a big gap the queue sits alone in empty space. With a small one, the shifted per-thread stacks land on the queue's old address. The stub then spins in spinning_queue_remove_first waiting for a timestamp slot that stack bytes keep at 0. That check sits before the timeout, so it never gives up. The Sentry waits on a context no stub ever acknowledged, and my 30s fail-stop, which kills a stuck stub process, only arms once an acknowledged context was interrupted. The sandbox never exits. On Kubernetes it looks like this: a pod stuck in Terminating, one stub process pinned at 100% of a core, and runsc kill hanging. Back of the envelope from the code: the gap is uniform below a random room of up to 1 TiB, so the odds come out roughly 1 in 100,000 starts for a 16-CPU sandbox, and about 1 in 35,000 at 64. That is an upper bound, since a hang also needs the overlapped stack in use. It had been there since the spinning queue landed in March 2023. Forcing a small gap in an arm64 VM: stock build hung 2 of 2, with the one-line fix 20 of 20 clean. If you run gVisor with systrap (release-20230417.0 through release-20261005.0): upgrade to a runsc build that contains c7f458aa3. Until then --systrap-disable-fast-path (release-20260622.0 or later) stops the spinning, at some syscall latency cost. To clear a hung pod, kill its runsc-sandbox process on the node. Fix: gvisor#15573 Write-up with the debugging steps and diagrams: catchkill9.dev
My fix for this hang was already running. The sandbox hung anyway. --- title: "My fix for this hang was already running. The sandbox hung anyway." published: true tags: gvisor, kubernetes, go, debugging canonical_url: catchkill9.dev cover_image: catchkill9.dev --- _gVisor shifts its stub memory by a random gap, but since 2023 one queue was not moved with it. When the gap is small, that queue can land on a thread's stack and the sandbox can hang for good. The fix was one line._ Same 16-thread workload. Same arm64 VM. Same forced memory layout. Stock gVisor hung 2 runs out of 2, with a stub process stuck at 100% of a core. Add one line, and it ran clean 20 times out of 20. The only variable was where one queue sat in memory. Getting to that line took me through my own safety net first. In August I had shipped a fail-stop to gVisor: if an app thread stays stuck for 30 seconds and gVisor's interrupt doesn't free it, gVisor kills the one stub process running it instead of letting it hang the sandbox. A month later a handful of sandboxes hung with that fail-stop running. The fix is google/gvisor#15573. gVisor, the short version > Deeper intro: gVisor, a kernel in Go. gVisor runs a second kernel, the Sentry, written in Go, between the app and Linux. On its default platform, systrap, the app's code runs in host processes called stubs. The Sentry puts work for the stubs on a queue in shared memory, and every system call traps back to it. - context: one app thread's saved state, waiting to run or waiting for the Sentry. - interrupt: a signal the Sentry sends to the stub thread running a context, to force it back. - spinning queue: idle stub threads busy-waiting for work, private to the stub. Each entry is stamped with when it started spinning. What did I think it was? The bug from last time (gVisor: one lost signal): one lost interrupt left an app thread stuck. Same symptoms: pods stuck in Terminating, every Sentry thread asleep, hundreds of kills from Kubernetes queued behind one app thread that never paused. One thing didn't match. Last time the stuck sandbox burned no CPU. This time a stub was burning a full core. And my fail-stop was supposed to catch exactly this kind of hang. Why didn't my fail-stop fire? A core dump of the Sentry showed one app thread waiting on its context, with AckedTime=0 and an owner of 0xfefefefe, the "no thread" marker. No stub had picked it up since it was queued. The fail-stop only fires once the Sentry has sent an interrupt, and the Sentry only interrupts a stub thread that took the context. Nobody took this one. No interrupt, no fail-stop. So my safety net had a hole, and it was mine. It covered a stub that took the work and then stopped responding. It didn't cover work no stub ever took. !Two paths: a stub took the context, so an interrupt is sent and the fail-stop kills one stub at 30s; no stub took it, so there is no interrupt, no fail-stop, and the sandbox hangs What was the stub doing at 100% CPU? Every sample of the busy stub landed in one tight loop in spinning_queue_remove_first. It waits while the oldest entry's timestamp reads 0, since a new entry writes its timestamp last. Here it read 0 every time, and that check sits before the timeout check, so the loop never gives up. Its other threads never took the context either. From the code, the Sentry only wakes another stub thread when none looks available (subprocess.go). The stuck spinner still counted as one. Where was the queue, really? In the stub's /proc/<pid>/maps, the spinning queue's address wasn't in its own private page. It sat inside a per-thread stack mapping. Its start field read 0x7ff9, which is where the high half of a 0x00007ff9... pointer lands when a stack stores one there. The queue was reading a thread's stack as queue data. How does a queue end up on a stack? gVisor puts a random gap into the stub's memory layout, so attackers can't guess where things are. Every region after it has to move by the gap. Here is stub_unsafe.go before the fix, with the missing line marked by me: gap := uintptr(rand.Uint64()) * hostarch.PageSize % (maximumUserAddress - stubStart - mapLen) stubSysmsgStack += uintptr(gap) stubContextQueueRegion += uintptr(gap) stubContextRegion += uintptr(gap) // missing: stubSpinningThreadQueueAddr += uintptr(gap) Three regions move. The spinning queue, between them in the layout, stays put. With a big gap it sits alone in empty space. With a small one, the shifted stacks slide over its old address. It's been that way since the commit that introduced the spinning queue, in March 2023. !Stub memory before and after a small random gap: thread stacks, context queue and contexts move by the gap, the spinning queue does not, and the moved stacks land on top of it It's an availability bug, not an isolation one: the queue lands on memory the stub could already write, which the Sentry treats as untrusted anyway. The damage stays inside the sandbox. Why did it hide for three years? The gap is random below the room M left above the stub block, and M is itself random up to R, about 1 TiB on x86-64. The queue gets hit when the gap is smaller than the stack area T, about 612 KiB for a 16-CPU sandbox: P = (T / R) * (1 + ln(R / T)) ~ 1 / 114,000 About 1 in 100,000 starts at 16 CPUs, and about 1 in 35,000 at 64. Waiting for it was useless. !Odds of a hang per sandbox start, derived from stub_unsafe.go: 1 in 100,000 at 16 CPUs, 1 in 35,000 at 64 CPUs Can I prove it with one variable? So I forced it: a debug-only patch picks a gap that puts the old queue address inside a live thread stack. !Same forced gap, two builds: stock gVisor hung 2 of 2, with the one-line fix clean 20 of 20 | Build, same forced gap | Queue lands on a stack | Result | |---|---|---| | Stock gVisor | yes | hung 2 of 2, stubs at 100% CPU, no fail-stop log line | | Stock + stubSpinningThreadQueueAddr += uintptr(gap) | no, it moves with the gap | clean, 20 of 20 | The code is shared by both architectures; arm64 was just faster to boot. The fix adds a layout test, and the repro tool is here. Are you affected? Systrap sandboxes on release-20230417.0 through release-20261005.0 can hit it; google/gvisor#15572 has details by version. The tell: a stub pinned at 100% forever, a pod stuck in Terminating, runsc kill hanging. 1. Upgrade to any runsc build that contains c7f458aa3, the first release after release-20261005.0. 2. Until then, --systrap-disable-fast-path (on release-20260622.0 or later) stops stub threads from spinning, so the spinning queue is never used. It's a partial workaround: it costs syscall latency, and a much rarer tiny gap can still hit the context queue. Only the upgrade fixes that. 3. To clear a hung sandbox, kill its Sentry on the host: the runsc-sandbox process ending with the pod sandbox ID from crictl pods. Its stubs die with it. What did the AI do? AI agents read the dump, mapped a program counter to a source line, built the VM repro and drafted the upstream PR. I made the calls, starting with not stopping at the hole in my own fail-stop. A few things I'm keeping: - Run the cheap check before the theory. One busy core killed my first guess. - Your own safety net has holes, so look there first. Mine only covered work a stub had taken. - Your debugging tool can kill the patient. A gcore of the Sentry faults in the sandbox's shared memory and can OOM it; dump the stub first. - When a bug needs bad luck, make the bad luck happen on demand. A forced gap turned a 1 in 100,000 hang into 2 of 2. The line is merged. That queue can't land on a stack anymore. --- Thanks to konstantin-s-bogom for reviewing and approving the fix, and to avagin, who wrote the spinning queue and fixed the same symptom from a different cause in 2024 (google/gvisor#10000). Investigation assisted by Claude Code, with Codex as a second reviewer.
containerのhost kernel exposureをさらに絞るruntime。 gVisorの`runsc`はuserspace app kernelでLinux-like interfaceを実装し、DockerやKubernetesに接続します。VMやsyscall filterとは別方式です。 google/gvisor
@odinshell github.com/google/gvisor/commit/cd968a36e ?
gVisor: Application Kernel for Containers (20k stars) What it does All text below is quoted directly from the project README. >gVisor provides a strong layer of isolation between running applications and the host operating system. It is an application kernel that implements a Linux-like interface. Unlike Linux, it is written in a memory-safe language (Go) and runs in userspace. gVisor includes an Open Container Initiative (OCI) runtime called runsc that makes it easy to work with existing container tooling. The runsc runtime integrates with Docker and Kubernetes, making it simple to run sandboxed containers. What isn't gVisor? - gVisor is not a syscall filter (e.g. seccomp-bpf), nor a wrapper over Linux isolation primitives (e.g. firejail, AppArmor, etc.). - gVisor is also not a VM in the everyday sense of the term (e.g. VirtualBox, QEMU). gVisor takes a distinct third approach, providing many security benefits of VMs while maintaining the lower resource footprint, fast startup, and flexibility of regular userspace applications. Why does gVisor exist? Containers are not a sandbox. While containers have revolutionized how we develop, package, and deploy applications, using them to run untrusted or potentially malicious code without additional isolation is not a good idea. While using a single, shared kernel allows for efficiency and performance gains, it also means that container escape is possible with a single vulnerability. >gVisor builds on x86_64 and ARM64. Other architectures may become available in the future. Repository google/gvisor Repository stats |Metric|Value| |:-|:-| |Stars|19,574| |Forks|2,024| |Last release|release-20260928.0 (2026-09-30)| |Languages|Go 76%, C++ 18.1%, Starlark 3.2%| |Contributors|354| |AI contributor accounts (Claude / Codex / Cursor / Copilot)|None| |AI co-authored commits|None| Main contributor |Field|Value| |:-|:-| |Name|Ayush Ranjan| |GitHub|@ayushr2| |Commits to this repo|929 (top human contributor; gvisor-bot has 1,461)| |Bio|Member of Technical Staff at Modal. Previously worked on gVisor at Google.| |Company|@modal-labs| |Location|San Fransisco| |Website|ayushr.com| |Followers|102| |Public repos|15| |On GitHub since|2016-09-01| Get the best repos from this community twice a week, 100% free*:* join the newsletter