`shutdown -h now` destroyed 13 of our 48 Ceph OSDs. Here is how we got the data back
---
title: "shutdown -h now destroyed 13 of our 48 Ceph OSDs. Here is how we got the data back"
published: true
description: A planned power-down killed the RocksDB index on 13 OSDs. Every supported repair tool refused to run, because each one opens the database it is supposed to repair. This is what worked.
tags: ceph, storage, incidenthandling, devops
---
We powered a site down on purpose. Electricians needed the room dead for an hour to upgrade a panel and fit a UPS. We had a runbook, we followed it, every application database shut down cleanly and reported exit code 0.
Then we turned the site back on and 13 of our 48 Ceph OSDs refused to start.
Not one disk had failed. Every byte of object data was intact. But each of those 13 OSDs had a destroyed RocksDB index, and without it BlueStore will not mount the object store at all. At the worst point 103 placement groups were offline, our shared filesystem was degraded, and two production object stores had data we could not reach.
This is the story of how we got all of it back except a single 4MB object, and why every supported repair tool in Ceph was useless to us.
First, what Ceph actually is (skip this if you run it)
If you already operate Ceph, jump to the next section. If you do not, none of the rest will land without this.
Ceph is distributed storage. You give it a pile of disks spread across a pile of machines, and it presents them as one system that speaks three protocols at once: block devices for VMs, a POSIX filesystem you can mount, and an S3 compatible object store. Same cluster, same disks, three front doors.
The pieces you need for this story:
- OSD (Object Storage Daemon). One process per disk. A host that owns 21 disks runs 21 OSD processes. This is the unit of storage and the unit of failure, and how many of them sit on one host is the whole root cause.
- Placement group (PG). A pool is sliced into thousands of PGs, and each PG is replicated onto a set of OSDs. Clients do not track objects, they compute a PG with a hash and then compute its OSDs with an algorithm called CRUSH. No lookup table, no metadata server in the data path. It is genuinely elegant.
- Failure domain. You tell CRUSH what a "thing that can die" is: a disk, a host, a rack. CRUSH then guarantees replicas land in different ones. Pick this wrong and your replication is decorative.
- MON (Monitor). A small quorum of daemons holding the maps that describe the cluster. They do not touch your data, but nothing works without them.
- BlueStore. Modern OSDs do not use a filesystem. The OSD writes objects to the raw block device itself, which is faster and avoids a whole category of double-write problems.
And that last one is where this incident lives.
BlueStore writes your object data straight to the disk, but it has to remember where everything went: which object is at which offset, its checksums, its attributes, the PG bookkeeping. That index lives in an embedded RocksDB on the same device.
The analogy that made this click for our non-storage colleagues: the disk is a book with no page numbers and no table of contents, and RocksDB is the index. The text is all still there. But with no index, you cannot find chapter 4, and BlueStore will not even open the book to let you try. It refuses to mount at all.
A visual representation may help, Ceph works more or less like this:
!Image description
What Ceph is genuinely good at
- Hardware death. Pull a disk, lose a host, and it re-replicates itself back to full redundancy with nobody paged. This works so well it is easy to forget it is happening.
- Scaling sideways. Add disks, add hosts, capacity and throughput go up. No forklift migration.
- One system, three protocols. Not running separate SAN, NAS and object stacks is worth real money and real headcount.
- No proprietary controller. It is open source, so when it breaks in an interesting way you can read the code that is failing. That mattered enormously here.
What Ceph is bad at, and you should hear this before you deploy it
- It is complex, and the complexity is not optional. CRUSH rules, pool sizing, failure domains, PG counts. You will get one of these wrong at least once.
- It assumes failures are independent. Its entire design premise is that disks and hosts die one at a time, at random. A single operator action that hits every node at once is the one thing the model does not cover, and that is exactly what we did.
- Small clusters are fragile in a way big ones are not. With thousands of OSDs, losing 13 is a Tuesday. With 48, losing 13 takes pools offline.
- The recovery tooling assumes the metadata is healthy. This is the sharp edge of the whole article. Ceph is excellent at recovering from a dead disk. It is much weaker at recovering from an intact disk with a broken index, because almost every tool that would fix the index needs to open the index first.
None of this is a reason to avoid Ceph. We would deploy it again. But "self-healing" describes hardware failure, not operator failure, and we learned the difference the expensive way.
Our setup, and why it is shaped like this
Nothing exotic, but the shape explains the blast radius.
consumers (VMs, apps, S3 clients)
|
storage network
|
+-----------+-----------+-----------+-----------+-----------+
| | | | | |
sto-01 sto-02 sto-03 sto-04 mon x3
| | | | (maps only,
+-----+-----+ +-----+-----+ no data)
| |
+-------v--------+ +-------v--------+
| ENCLOSURE A | | ENCLOSURE B |
| 24 NVMe | | 24 NVMe |
| dual-ported | | dual-ported |
+----------------+ +----------------+
48 OSDs total, unevenly split within each host pair
real split was 3 / 21 / 5 / 19 across the four hosts
CRUSH failure domain = ENCLOSURE (not host)
All pools: size=2, min_size=2, one copy in each enclosure
Four storage hosts, two external NVMe enclosures, 48 OSDs. The drives are dual-ported, meaning each physical disk is wired to two hosts at once, so either host of a pair can take over the OSDs if its partner dies. That is a nice property and it is also why the topology is not the obvious one.
The OSDs are not spread evenly. The real split across the four hosts was 3, 21, 5 and 19. Within each pair, one machine had quietly ended up owning almost every drive in its enclosure while its partner held a handful. Nobody designed that, it accumulated. It is also the single fact that decided how bad this incident got.
Two decisions here matter for what follows:
The failure domain is the enclosure, not the host. This looks wrong until you remember the disks do not live in the hosts. If CRUSH thought "host" was the boundary, it could happily place both replicas of a PG on two OSDs that are physically in the same enclosure, served by two different hosts. Lose that enclosure and you lose both copies. Setting the domain to the chassis forces one replica into each enclosure, so any pool survives an entire enclosure going away.
size=2, min_size=2. Two copies, and both must be present for writes to be accepted. This is a deliberate trade: it refuses to take writes it cannot fully replicate, rather than accepting them onto one copy and hoping. On a two-enclosure layout it is the honest setting.
Now the arithmetic that hurt. A PG only goes offline when both of its copies are unavailable, and since the copies are always in different enclosures, that needs one failed OSD in A and one failed OSD in B, both holding the same PG. Our 13 dead OSDs were split across both enclosures. Every PG that happened to have landed on one from each side went dark. That produced 88 offline PGs from 13 failures, and later 103 as more state came in.
And the number that actually caused the incident: 21 OSD processes on a single host. Ask systemd to stop 21 of anything at once, against a single default timeout, while each one is trying to flush its own database. Some of them will not make it. Had those same 48 OSDs been spread 12 / 12 / 12 / 12, every host would have had twice the timeout budget per daemon.
The scene
Here is what we knew an hour after power came back:
- 35 of 48 OSDs up. The other 13 crash-looping across all four storage hosts
- Every failing OSD reported a variation of the same error: rocksdb: verify_sharding unable to list column families: Corruption
- The disks themselves were healthy. No I/O errors in dmesg, no SMART complaints
- 88 placement groups already offline, by the arithmetic above. That number grew to 103 as more state came in
- Our cluster had been HEALTH_OK with 48/48 OSDs for three weeks before this
The last point matters. This was not creeping hardware rot. Something we did during the shutdown broke them all at once.
Step 1: The cluster lied to us about its own state (ceph osd set nodown)
The first ceph -s after power-on looked fine:
48 osds: 48 up (since 86s), 48 in (since 3w)
It was wrong. Our shutdown procedure sets the standard quiesce flags before halting, and nodown was still set. nodown prevents OSDs being marked down, so the monitor keeps reporting them as healthy regardless of reality.
We released the flags and got the truth:
48 osds: 35 up (since 7m), 48 in (since 3w)
Lesson: if your power-on checklist validates the OSD count before releasing the quiesce flags, that check passes falsely. Release the flags first, or cross-check with systemctl list-units 'ceph-*osd*' --state=failed on each host. We spent ten minutes acting on a number that was fiction, and the only reason it was not longer is that an OSD count is easy to sanity-check from the hosts themselves.
Step 2: What actually broke (bluefs-export)
BlueStore keeps its metadata in an embedded RocksDB, which lives on a small filesystem called BlueFS on the same block device. RocksDB continuously rewrites two small files:
- CURRENT: a 16 byte pointer containing the current MANIFEST's filename
- MANIFEST-NNNNNN: the log of version edits describing which SST files hold what
fsck would not open the database. But bluefs-export does not need RocksDB at all, it just reads BlueFS and dumps the files to a directory:
cephadm shell --name osd.0 --mount /var/tmp/osd0-bluefs:/export -- \
ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-0 bluefs-export --out-dir /export
That worked on every single failing OSD. Suddenly we had the whole database as ordinary files we could inspect without risk. This one command is the foundation of everything that follows.
What it showed on the first OSD: 25 SST files present and intact, MANIFEST present, and CURRENT containing 16 bytes of binary garbage where it should have contained the string MANIFEST-000147\n.
Here is the giveaway. A healthy MANIFEST starts with a CRC and a known string. Ours did not:
healthy manifest: 56 f9 b8 f8 1c 00 01 01 1a 6c 65 76 65 6c 64 62 >V........leveldb<
our MANIFEST: 00 00 00 00 00 00 68 07 00 00 00 00 00 00 00 00 >......h.........<
another OSD: 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 >aaaaaaaaaaaaaaaa<
That run of 0x61 is the letter a repeated. It is a disk test fill pattern left over from whenever those drives were last burned in. That is what unflushed blocks read back as.
So BlueFS had journaled the file metadata for those writes, recording size, mtime and allocated extents, but the file contents were never flushed to the device. On restart RocksDB read the recorded size and got whatever was on those blocks beforehand.
Every damaged file carried an mtime inside our shutdown window. We did this to ourselves.
The root cause, in one sentence
shutdown -h now on a host running 21 OSD containers asks systemd to stop them all in parallel against its default timeout, and SIGKILLs any that do not finish in time.
Our runbook was meticulous about stopping applications gracefully, giving every database a generous -t and verifying exit code 0. Not one application database needed crash recovery afterwards. The storage layer got none of that courtesy: just shutdown -h now and hope.
Step 3: Every repair tool refuses to run
Here is the part that makes this failure mode nasty. Ceph ships tools that could fix this. None of them will run.
bluefs-import, the supported way to write a file back into BlueFS:
inferring bluefs devices from bluestore path
rocksdb: verify_sharding unable to list column families: Corruption: CURRENT file does not end with newline
error preparing db environment: (5) Input/output error
It opens the RocksDB environment before writing. Which is the thing that is broken.
ceph-kvstore-tool bluestore-kv ... destructive-repair: mounts BlueFS fine, then opens RocksDB before repairing. Same wall.
RocksDB's best_efforts_recovery option, which exists precisely to ignore a corrupt CURRENT and MANIFEST and rebuild from the SST files, is a DB::Open option. Ceph calls rocksdb::DB::ListColumnFamilies() before DB::Open(), and ListColumnFamilies reads the MANIFEST directly. The option never gets a chance to apply.
Every road is circular: to repair the index, the tool must first open the index.
Step 4: Writing to the raw block device (bluefs-log-dump)
If no tool will write the file, write it yourself. BlueFS records every file's inode and physical extents on the block device, and you can dump that log:
ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-0 bluefs-log-dump
0x1107f: op_dir_link db/CURRENT to 137
0x10cd4: op_file_update file(ino 137 size 0x10 ... extents [1:0x568000~10000])
db/CURRENT is inode 137, size 0x10 (16 bytes), living at device offset 0x568000.
Before writing anything, prove the location and save a rollback:
# read those bytes raw and byte-compare against the exported copy
dd if=$DEV bs=1 skip=$((0x568000)) count=16 status=none > /tmp/raw16.bin
cmp /tmp/raw16.bin /var/tmp/osd0-bluefs/db/CURRENT && echo "location confirmed"
# save the whole enclosing 4K block as an exact rollback
dd if=$DEV bs=4096 skip=$((0x568000/4096)) count=1 of=/var/tmp/block.orig status=none
Then a read-modify-write of that single block, replacing only the first 16 bytes with MANIFEST-000147\n.
The error changed:
before: Corruption: CURRENT file does not end with newline
after: Corruption: checksum mismatch in file db/MANIFEST-000147
Progress. The fix worked and exposed the next layer.
Gotcha that will bite you: some OSDs record their extents in op_file_update_inc (incremental) records rather than op_file_update. If you only read the latter you get a stale offset with no extents and you will write to the wrong place. Our byte-comparison guard caught exactly this on one OSD and aborted before writing. Never skip the comparison step.
Step 5: Rebuild the index on a copy, not on the patient
The MANIFEST could not be repaired in place. We wrote a small parser for RocksDB's log format (7 byte header: masked CRC32C, length, type) and ran it over the file:
file size: 11592 bytes
valid records: 0
FIRST BAD RECORD at offset 0 -> CRC mismatch type=104 len=0
Type 104 is 0x68, not a valid record type. Stale bytes from offset 0. There was nothing to truncate back to, which is the difference between "interrupted mid-write" and "never written at all". If a file had been torn mid-write we would have seen N valid records and then damage, and we could have truncated to the last good offset. Zero valid records means the contents never reached the disk.
That distinction is worth being able to measure, so here is the parser. Pure stdlib, because it has to run on a storage host or inside a cephadm container where you cannot install anything.
{% details rocksdb-log-check.py (full script) %}
#!/usr/bin/env python3
"""
rocksdb-log-check.py - validate a RocksDB MANIFEST / WAL record by record.
Answers one question: how much of this file is still readable, and where
exactly does it stop being readable?
python3 rocksdb-log-check.py db/MANIFEST-000147
THE FORMAT (rocksdb/db/log_format.h, log_reader.cc)
A MANIFEST is a log file: a sequence of 32 KiB blocks, each holding one or
more records.
+--------+--------+------+-----------------------+
| CRC32C | length | type | payload |
| 4 LE | 2 LE | 1 | <length> bytes |
+--------+--------+------+-----------------------+
* The CRC is CRC32C (Castagnoli, poly 0x82F63B78), NOT zlib's CRC32,
and it is stored *masked* (see unmask below).
* The CRC covers the type byte plus the payload. Not the header.
* Records never straddle a block boundary. If fewer than 7 bytes remain
in a block, the writer zero-fills them and starts the next block.
* A run of zeroes is normal padding, not damage. RocksDB's own reader
tolerates it, which is also why zero-padding a rebuilt manifest back
into place works.
READING THE OUTPUT
valid records: 0, bad record at offset 0
The file never held a valid record. Nothing to salvage, nothing to
truncate back to. This is the signature of unflushed data: the
metadata was journaled but the contents never reached the device,
so you are reading whatever was on those blocks beforehand.
valid records: N, bad record partway in
Torn mid-write. Truncating to the last good offset usually yields a
loadable manifest.
A type byte outside 1..8 means you are almost certainly looking at stale
disk bytes rather than a damaged record.
"""
import struct
import sys
BLOCK_SIZE = 32768 # rocksdb::log::kBlockSize
HEADER_SIZE = 7 # 4 byte crc + 2 byte length + 1 byte type
MASK_DELTA = 0xA282EAD8 # rocksdb::crc32c::kMaskDelta
POLY = 0x82F63B78 # CRC32C (Castagnoli), reversed representation
RECORD_TYPES = {
0: "kZeroType",
1: "kFullType",
2: "kFirstType",
3: "kMiddleType",
4: "kLastType",
5: "kRecyclableFullType",
6: "kRecyclableFirstType",
7: "kRecyclableMiddleType",
8: "kRecyclableLastType",
}
# Python's zlib only ships CRC32, so build a CRC32C table by hand.
_TABLE = []
for _i in range(256):
_c = _i
for _ in range(8):
_c = (_c >> 1) ^ (POLY if _c & 1 else 0)
_TABLE.append(_c)
def crc32c(data, crc=0):
crc ^= 0xFFFFFFFF
for byte in data:
crc = _TABLE[(crc ^ byte) & 0xFF] ^ (crc >> 8)
return crc ^ 0xFFFFFFFF
def unmask(masked):
"""Inverse of rocksdb::crc32c::Mask: subtract the delta, rotate right 17."""
rot = (masked - MASK_DELTA) & 0xFFFFFFFF
return ((rot >> 17) | (rot << 15)) & 0xFFFFFFFF
def check(path):
with open(path, "rb") as handle:
data = handle.read()
print(" file: %s" % path)
print(" file size: %d bytes" % len(data))
pos = 0
records = 0
last_good = 0
failure = None
while pos + HEADER_SIZE <= len(data):
# Records never straddle a block boundary. A short tail is padding.
offset_in_block = pos % BLOCK_SIZE
if BLOCK_SIZE - offset_in_block < HEADER_SIZE:
pos += BLOCK_SIZE - offset_in_block
continue
header = data[pos:pos + HEADER_SIZE]
masked_crc = struct.unpack("<I", header[0:4])[0]
length = struct.unpack("<H", header[4:6])[0]
rec_type = header[6]
# An all-zero header is preallocated space, not corruption.
if masked_crc == 0 and length == 0 and rec_type == 0:
print(" [zero region begins at offset %d]" % pos)
break
payload = data[pos + HEADER_SIZE:pos + HEADER_SIZE + length]
if len(payload) < length:
failure = (pos, "truncated payload: want %d, have %d"
% (length, len(payload)))
break
# The CRC covers the type byte and the payload, not the header.
expected = unmask(masked_crc)
actual = crc32c(bytes([rec_type]) + payload)
if expected != actual:
failure = (pos, "CRC mismatch type=%d (%s) len=%d want=%08x got=%08x"
% (rec_type,
RECORD_TYPES.get(rec_type, "INVALID"),
length, expected, actual))
break
records += 1
pos += HEADER_SIZE + length
last_good = pos
print(" valid records: %d" % records)
print(" last good byte offset: %d" % last_good)
if failure:
print(" FIRST BAD RECORD at offset %d -> %s" % failure)
else:
print(" no bad record found before EOF or zero padding")
print(" trailing bytes after last good: %d" % (len(data) - last_good))
return 0 if records and not failure else 1
if __name__ == "__main__":
if len(sys.argv) != 2:
sys.exit(__doc__.strip().splitlines()[0])
sys.exit(check(sys.argv[1]))
{% enddetails %}
Run it against a healthy manifest from any working OSD first, so you know what a good result looks like on your own cluster before you trust it on a broken one.
But RocksDB has a repairer that rebuilds the MANIFEST by scanning the SST files. It will not run against a live broken OSD. It runs perfectly against a directory:
cp -a /var/tmp/osd0-bluefs /var/tmp/osd0-test # work on a copy
cephadm shell --name osd.0 --mount /var/tmp/osd0-test:/t -- \
ceph-kvstore-tool rocksdb /t/db destructive-repair
The results were much better than we feared:
| OSD | SST files before | after | quarantined |
|---|---|---|---|
| A | 28 | 28 | MANIFEST only |
| B | 33 | 33 | MANIFEST only |
| C | 24 | 24 | MANIFEST only |
| D | 25 | 23 | MANIFEST + 2 SSTs |
| E | 37 | 31 | MANIFEST + 6 SSTs |
On most OSDs it retained every single SST. The object metadata was fine all along. Only the index describing it was gone.
Now we had a valid rebuilt MANIFEST sitting in a directory, and the same problem as before: how do we get it back in? Same answer. Locate the original MANIFEST's extents, write the rebuilt one into them, zero-padded. RocksDB's log reader explicitly tolerates zero-filled regions, because preallocation produces them.
Step 6: The assert that sent us to the source code
The database opened. Then:
src/kv/RocksDBStore.cc: 1221: FAILED ceph_assert(recreate_mode)
No message, no explanation. So we read the source, and it explained everything:
static const char* sharding_def_file = "sharding/def";
static const char* sharding_recreate = "sharding/recreate_columns";
static const char* resharding_column_lock = "reshardingXcommencingXlocked";
...
bool recreate_mode = status.ok() && sharding_recreate_text == "1";
...
if (missing_cfs.size() > 0 && /* lock not among missing */) {
ceph_assert(recreate_mode);
for (...) db->CreateColumnFamily(missing_cfs[i].options, ...);
}
...
if (existing_cfs.size() != rocksdb_cfs.size()) {
derr << "extra columns in rocksdb..."; return -EIO;
}
Three facts fell out, and they define the whole problem space:
1. Missing column families: Ceph will create them for you, but only in recreate_mode, which is set by a BlueFS file sharding/recreate_columns containing "1". Which only bluefs-import can create. Which needs the database open. Circular again.
2. Extra column families: hard -EIO, no workaround at all.
3. The assert is skipped entirely if a magic column family named reshardingXcommencingXlocked is among the missing ones.
Our rebuilt database had fewer column families than sharding/def declared, because RocksDB's repairer only creates families it finds SSTs for, and some of ours were empty. So the fix was to make sharding/def match reality.
If you take one thing from this article, take this: when an assert has no message, read the source. RocksDBStore.cc lines 1025 to 1240 turned an impossible problem into a mechanical one.
Step 7: The 130 byte file that cost us an attempt
sharding/def is a plain text file describing the column family layout:
m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32
It is a fixed size BlueFS file, so any replacement must be exactly the same byte length. Drop the m(3) entry and you are 5 bytes short. The obvious move is to pad with trailing spaces.
Do not pad with spaces. The parser splits on spaces in a while (!text_def.empty()) loop, so trailing spaces create phantom column families with empty names, which keeps missing_cfs non-zero and fires the same assert. That cost us a full attempt, a byte-for-byte revert, and a second pass at working out why an apparently correct definition still would not parse.
Pad by expanding a numeric option value with leading zeros instead. Same value, no trailing space:
p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=0000032 P=min_write_buffer_number_to_merge=32
The database opened, and fsck started reporting ordinary BlueStore findings instead of refusing to mount. On several OSDs it simply returned fsck success.
One more flag was needed for the write path:
rocksdb: Corruption: force_consistency_checks: VersionBuilder:
L0 file #409 with seqno 12429585 15050034 vs. file #501 with seqno 14519025 14953111
That is an artefact of the repairer reconstructing the manifest with overlapping sequence numbers. Setting force_consistency_checks=false in bluestore_rocksdb_options unblocked it. That is a config change, not a data change, and it is reversible.
The first OSD came up. The shared filesystem went healthy immediately.
Step 8: Stop repairing OSDs you do not need (osdmaptool)
At this point we had a proven procedure and 12 more broken OSDs. Getting the first one back had taken half an hour of careful, irreversible work, and every one after it still meant raw-device writes on hardware holding live data. Before grinding through all of them, we asked a better question: which of these OSDs does anyone actually still need?
First, get the list of what is genuinely broken, not what is merely unhappy:
ceph pg dump pgs_brief | awk '$2 !~ /active\+clean/ {print $1, $2}'
4.3ff down
4.3a3 down
4.266 undersized+peered
9.2 undersized+peered
10.8 down
...
Two different states, and the difference is everything:
- undersized+peered means one replica is gone. The other is alive and holds a complete copy. The PG is not serving writes because min_size=2, but no data is at risk.
- down means both replicas are on failed OSDs. Nothing is serving it and nothing can, until one of those two OSDs comes back.
Now ask CRUSH where each of those PGs is supposed to live:
ceph osd getmap -o osdmap.bin
osdmaptool osdmap.bin --mark-up-in --test-map-pg 4.3ff
osdmaptool: osdmap file 'osdmap.bin'
parsed '4.3ff' -> 4.3ff
4.3ff raw ([17,43], p17) up ([17,43], p17) acting ([17,43], p17)
--mark-up-in is the flag that makes this useful. It tells osdmaptool to compute placement as if every OSD in the map were up and in. Without it you get the current degraded reality, which just repeats what you already know: the PG has nowhere to live. With it you get the intended placement, which is the question you actually care about: if I fix these OSDs, where does this PG go?
Reading the output
4.3ff wants two OSDs, osd.17 and osd.43. Our 13 dead OSDs were 0, 1, 5, 6, 7, 8, 9, 10, 13, 15, 16, 17, 21. So for this PG:
- osd.17 is dead
- osd.43 is alive
One of the two survives, and because it is a full replica it holds every object in that PG. This PG needs no repair work at all. We can purge osd.17 outright and Ceph will re-replicate the PG from osd.43 onto any healthy OSD in the right enclosure. The hours of manifest surgery would buy us nothing here.
Compare 10.8, which came back acting ([5,32]). Both osd.5 and osd.32... osd.5 is in our dead list. osd.32 is not, but 10.8 was reported down rather than undersized, which told us the map was stale and osd.32 had also dropped. When both members of the pair are dead, that is a PG we must go and rescue, and it is the only kind that justifies the expensive procedure.
So every stranded PG falls into exactly one of three buckets:
| Dead replicas | PG state | What it means | Action |
|---|---|---|---|
| 0 of 2 | active+clean | untouched | nothing |
| 1 of 2 | undersized+peered | a complete copy survives | purge the dead OSD, let it re-replicate |
| 2 of 2 | down | no live copy anywhere | must repair at least one of the pair |
Running it over everything
Loop the whole inactive list and bucket it:
ceph osd getmap -o osdmap.bin
DEAD="0 1 5 6 7 8 9 10 13 15 16 17 21"
ceph pg dump pgs_brief | awk '$2 !~ /active\+clean/ {print $1}' | while read pg; do
acting=$(osdmaptool osdmap.bin --mark-up-in --test-map-pg "$pg" 2>/dev/null \
| grep -oE 'acting \(\[[0-9,]+\]' | grep -oE '[0-9]+(,[0-9]+)*')
n=0
for osd in ${acting//,/ }; do
case " $DEAD " in *" $osd "*) n=$((n+1));; esac
done
echo "$pg acting=[$acting] dead_replicas=$n"
done | sort -k3 -r
The tally across all 88 stranded PGs:
dead_replicas=2 23 PGs <- the real work
dead_replicas=1 65 PGs <- purge and re-replicate, no repair needed
65 of the 88 offline PGs did not need anything repaired. They were offline purely because min_size=2 refuses writes at one copy, not because the data was gone. Purging the dead OSD and letting Ceph re-replicate from the survivor fixes them, and that is a normal, boring, supported operation.
That left 23 PGs that genuinely had no live copy. Taking the union of the OSDs appearing in those 23 both-dead pairs gave us a critical set of 5 OSDs, not 13. The other 8 could be purged immediately.
One more thing the output told us, and it is the payoff for getting the failure domain right back in the setup section. Because CRUSH is constrained to put one replica in each enclosure, a both-dead pair is always one OSD in enclosure A and one in enclosure B. So for each of those 23 PGs we had a free choice: whichever of the two OSDs looked more repairable was the one to work on. We never had to fix a specific OSD, only one of a pair. Two of the five turned out to be unrecoverable, and it cost us nothing, because their partners carried the same PGs.
Lesson: in a recovery, the cheapest work is the work you prove you do not have to do. Ten minutes with osdmaptool --mark-up-in turned 13 expensive repairs into 5, and told us exactly which 8 OSDs we could stop worrying about. Do this before you start repairing, not after.
Step 9: Get the data out, do not put the OSD back
For the OSDs too damaged to return to service, there is a better move than nursing them into production. Extract the placement groups and import them onto healthy hardware:
# from the damaged OSD
ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-9 --op export \
--pgid 9.2 --file /x/9.2.export \
--bluestore-rocksdb-options="...,force_consistency_checks=false"
# onto a healthy one (stop it first)
ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-43 --op import --file /x/9.2.export
This is what Ceph maintainers recommend for this situation, and it is much safer than running a hand-patched OSD in production. We recovered 22 placement groups this way, including a 13.6GB one, moving them between hosts over the storage network in seconds.
Two tricks that mattered here:
A single unreadable object can block a whole PG export. Ours aborted with read failure: (5) Input/output error on one object. Remove that object and the other 25 export fine:
ceph-objectstore-tool --data-path ... --pgid 4.3a3 \
'["4.3a3",{"oid":"...","key":"","snapid":-2,"hash":...,"pool":4,"namespace":""}]' remove
ceph osd lost and ceph osd purge do not touch the disk. They edit the OSD map only. We verified this empirically by exporting a complete 84MB placement group from a fully purged OSD, using --no-mon-config to skip the auth lookup that the purge removed. That property is what made the whole recovery safe: nothing was destroyed to reach a healthy cluster.
The one object we lost
Final tally: one object out of 159,001.
It was a 4MB block that was physically unreadable on its disk, a genuine hard I/O error, in a placement group whose other replica was on another failed OSD. We identified exactly what it was by decoding the filesystem backtrace stored in the first object of the same file:
rados -p cephfs.cephfs.data getxattr 10000103179.00000000 parent > parent.bin
ceph-dencoder type inode_backtrace_t import parent.bin decode dump_json
That gave us the full path. It turned out to be a fragment of a nightly backup image from five days earlier, superseded by three newer complete backups. 516 of that file's 517 objects survived.
Being able to answer "exactly which file, exactly which bytes" is worth the effort. "We lost one object" is a shrug. "We lost bytes 16MB to 20MB of a superseded backup taken on the 13th, and here are the three newer copies" is an answer you can take to a customer.
And then, before declaring victory:
ceph pg deep-scrub <pgid> # for every recovered PG
inconsistent_pgs: 0
scrub_errors: 0
pgs_by_state: [('active+clean', 1265)]
"Import successful" is not the same as data integrity. Scrub is the proof.
What we got wrong (and what we fixed)
1. We never stopped the OSDs. The whole incident traces to this. The runbook stopped applications carefully and left up to 21 OSD containers per host to systemd's default timeout. Now: systemctl stop ceph-osd.target, then a loop that waits and confirms every OSD container is gone before shutdown is allowed to run.
2. We trusted ceph -s with nodown set. It reported 48 of 48 OSDs up while 13 were crash-looping. Release the flags before validating anything, and never let a post-power-on checklist assert on a number the flags can fake.
3. Our runbook's storage topology was wrong. It said one host held zero OSDs. It held three, and they were load-bearing. A document nobody re-verifies drifts into fiction, and you find out during an incident.
4. sudo docker compose with ~ in a volume path. During restart, ~/models/huggingface expanded to /root, and Docker silently created an empty directory instead of failing. Cost us a crash-looping service until we noticed the bind was wrong. Check docker inspect mounts, do not assume.
5. We twice concluded the data was unrecoverable. Both times we were wrong, and both times someone pushed to keep going. The second time, that was the difference between losing 18 placement groups and losing one object.
The takeaway
Ceph is genuinely good at surviving hardware failure. It is much less good at surviving you, and the tooling assumes a working database in places where you most need it not to.
Three things are worth internalising before you need them:
Work on copies. bluefs-export gives you the entire database as files while the OSD refuses to mount. Everything destructive we did, we did on a copy first, then applied. We reverted three separate attempts byte-for-byte and lost nothing by trying them.
Verify before you write, always. Every raw device write was preceded by a byte comparison proving the offset, and by saving the block we were about to overwrite. That discipline caught a wrong offset that would have corrupted an OSD we still needed.
Read the source. The assert that blocked us cold had no message. Two hundred lines of RocksDBStore.cc turned it into a checklist. If you run infrastructure on open source, the source is part of your documentation.
And go check your shutdown procedure right now. If it stops your applications carefully and then calls shutdown -h now on a host running database daemons, you have the same bug we had. It has not bitten you yet because your daemons have always finished in time. One day one of them will not.
The tools, and where to get them
Everything we wrote during this recovery is published. The first two are dangerous in different ways, so read the header comments before running either.
| Tool | What it does |
|---|---|
| rocksdb-log-check.py | Validates a RocksDB MANIFEST or WAL record by record. Tells you whether a file was torn mid-write (truncate and recover) or never written at all (nothing to salvage). Pure stdlib, so it runs inside a cephadm container. |
| bluestore-manifest-transplant.sh | The Step 4 to Step 7 procedure, packaged: locate CURRENT, MANIFEST and sharding/def on the raw device, byte-verify every offset, save a rollback, rebuild sharding/def to match the recovered column families, write. Dry run by default, and it aborts if any offset fails verification. |
| ceph-safe-shutdown.sh | The fix. Stops ceph-osd.target and then waits and confirms every OSD is actually gone, exiting non-zero if any are still running when the budget expires. Chain it: ./ceph-safe-shutdown.sh && shutdown -h now. This is the script whose absence caused the entire incident. |
If you only take one of them, take the last one.
Further reading
- Ceph: BlueStore config · ceph-bluestore-tool · ceph-objectstore-tool · CRUSH maps · Placement groups
- RocksDB Repairer, which is what destructive-repair invokes
- RocksDB log format, if you want to understand the parser
- Ceph source: src/kv/RocksDBStore.cc, the sharding verification and column family handling. This is the file that turned an impossible problem into a mechanical one
---
From the first failed OSD to a scrub-verified clean cluster took about three and a half hours. The single most valuable command in the whole incident was bluefs-export, and we found it in the man page.
Buffer Pool and Cache Management for Database Engines
- How the Buffer Pool Anchors the Memory Hierarchy
- Choosing an Eviction Policy: LRU, CLOCK, and Workload-Aware Variants
- Pinning and Concurrency: Making Eviction Safe at Scale
- Dirty Page Management: Flushing, Checkpoints, and WAL Discipline
- Prefetching, Read-Ahead, and OS Cache Interaction
- Practical Application: Instrumentation, Tuning, and Operational Checklists
Buffer management is where microseconds become minutes: the buffer pool turns persistent I/O into in-memory work or it becomes the throttle that kills p99. Get eviction, pinning, and dirty-page flushing wrong and the storage layer will be the single largest source of unpredictable latency in production.
You see this problem in three ways: stealthy tail-latency spikes during heavy scans or checkpoints, I/O storms when the evictor chases dirty pages, and persistent memory bloat because kernel and engine caches duplicate the same bytes. The symptoms look like the app is slow, but root cause analysis usually points to poor coordination between the buffer pool, eviction policy, prefetch heuristics, and the write path.
How the Buffer Pool Anchors the Memory Hierarchy
The buffer pool is the database engine's primary residency for hot data: it takes pages off block I/O and keeps them in DRAM so repeated accesses hit memory instead of the device. It sits above the OS page cache and below application logic; that placement creates both its power and its complexity. PostgreSQL, MySQL/InnoDB and other systems implement a dedicated shared buffer manager for precisely this reason — the engine controls MVC semantics, pinning, and writeback ordering inside its pool rather than delegating those responsibilities to the kernel.
> Important: The buffer pool is not just a cache; it's the authoritative runtime view of pages for MVCC and transaction safety. Your eviction and flush logic must respect transactional LSN/versioning semantics.
Quick reality check — orders-of-magnitude matter. Typical round numbers (orders of magnitude) are: CPU caches (ns), DRAM (tens–hundreds ns), NVMe SSD (tens–hundreds μs), HDD (milliseconds). That gap is why avoiding device hits matters so much for p99.
| Layer | Characteristic | Typical latency (order of magnitude) |
|---|---|---:|
| CPU caches | L1/L2/L3, CPU-local | nanoseconds |
| DRAM / Buffer pool | Shared memory for DB | tens–hundreds of nanoseconds |
| NVMe SSD | Fast persistent storage | tens–hundreds of microseconds |
| Spinning disk | Mechanical access | milliseconds |
Avoid double caching (engine buffer pool + kernel page cache) unless you have a reason to keep both. Bypass the kernel with O_DIRECT or use posix_fadvise hints when you want the kernel to help with read‑ahead, but know the trade-offs: O_DIRECT removes double caching but increases complexity for alignment and I/O buffering; kernel-assisted approaches are simpler but can waste memory.
Choosing an Eviction Policy: LRU, CLOCK, and Workload-Aware Variants
Eviction is the gatekeeper of memory reuse. The core options are well known, but their operational trade-offs matter more than their theoretical hit-rates.
- LRU (Least Recently Used): conceptually simple, good for single-threaded or low-concurrency workloads where recency maps to future use. Implementation complexity rises when you need to make it concurrency-friendly (sharded LRU, lock striping), and cost of updating recency on every access can be high.
- CLOCK / Second-Chance: a compact approximation to LRU that uses a circular hand and a single reference bit. Low per-page metadata and easier to make concurrent — a great pragmatic default for large engines.
- Workload-aware variants: LRU-K, ARC, LIRS, CLOCK-Pro and multi-queue (SLRU) variants track deeper history or multiple recency windows to separate frequently used from recently used. They improve hit rates on mixed workloads at the price of more metadata and complexity.
| Policy | Pros | Cons | When to prefer |
|---|---|---|---|
| LRU | Intuitive; good for recency-heavy workloads | High recency-update cost; contention under concurrency | Small to medium pools, low concurrency |
| CLOCK | Low metadata, low update cost | Approximation — slightly worse hit-rate than perfect LRU | Large pools, high concurrency; pragmatic default |
| LRU-K / LIRS / ARC | Better for mixed hot/cold and scan-resistance | More metadata and complexity | Workloads with long-term frequency differences |
| Segmented LRU (SLRU) | Fast path for hot pages | Needs tuning of segment sizes | Workloads with clear hot-set vs bulk scans |
Contrarian production insight: for many systems I’ve built and debugged, a well-tuned CLOCK (or sharded CLOCK) beats a naïve global LRU because it avoids the thrash and lock contention that kills throughput under concurrency.
Example of a low-overhead CLOCK eviction loop (pseudocode):
// Simplified CLOCK walker pseudocode
while (true) {
Page *p = clock_hand.next();
if (atomic_load(&p->pin_count) != 0) { continue; } // skip pinned
if (p->refbit) {
p->refbit = 0; // second chance, clear and move on
continue;
}
if (p->dirty) {
schedule_flush(p); // async write; skip until clean
continue;
}
evict_page(p);
break;
}
Make your eviction fast and observable: short scans, counters for failed evictions (pinned/dirty), and the ability to increase scan aggressiveness under memory pressure.
Pinning and Concurrency: Making Eviction Safe at Scale
Pinning is the crash-proof handle that prevents in‑flight pages from being evicted mid-use. The basic contract is simple: pin increments a pin_count, unpin decrements it, and eviction only succeeds when pin_count == 0. The devil is in the race conditions and in how long pins are held.
- Represent pin_count with atomic integers (std::atomic / AtomicUsize) so pin is cheap and scalable.
- Provide both pin() (blocks or spins until the page is present and pinned) and try_pin() (fast fail when the page cannot be pinned) APIs to let callers decide blocking semantics.
- Avoid holding a pin while doing blocking IO or while waiting on unrelated locks; long-lived pins stall evictors and lead to memory pressure and write stalls.
Pseudocode for safe fetch/pin pattern:
Page* fetch_and_pin(page_id) {
Page* p = hashtable_lookup(page_id);
if (!p) {
p = allocate_slot_and_read_from_disk(page_id);
// Insert into hash with pin_count = 1
atomic_store(&p->pin_count, 1);
return p;
} else {
atomic_fetch_add(&p->pin_count, 1);
return p;
}
}
void unpin(Page* p) {
atomic_fetch_sub(&p->pin_count, 1);
}
Implementation notes:
- Keep the critical section that pins a page as small as possible.
- Use per-bucket or per-shard metadata to reduce global lock contention on the eviction structure.
- Track pin wait latency as an SRE metric; frequent waits are a clear signal that something (long transactions, background compaction) is holding pins too long.
> Operational warning: Holding pins across user-level locks, synchronous RPCs, or long computations is a leading cause of eviction starvation in production.
Dirty Page Management: Flushing, Checkpoints, and WAL Discipline
The Log is Law. Every modification must be reflected in the Write-Ahead Log (WAL) before the corresponding page can be considered safely durable on disk. That ordering gives you atomicity and crash-recovery guarantees: write WAL, fsync WAL, then you can write data pages.
Three practical flush domains:
1. Eviction-driven flush (on-demand): when eviction encounters a dirty page, it flushes it before eviction. Pros: minimal background IO on light workloads. Cons: under pressure, a wave of evictions can cause write bursts.
2. Background flusher: a daemon that maintains a target dirty ratio (percentage of buffer pool dirty). It smooths writes over time and prevents large checkpoint bursts.
3. Checkpointer: at checkpoint time the engine ensures that pages are flushed up to a checkpoint LSN; it coordinates with WAL so recovery need only replay from that LSN forward. Checkpointing must be throttled to avoid saturating the device; stagger writes across time.
Key invariants and implementation tips:
- Track per-page page_lsn and flushed_lsn. A page is clean when flushed_lsn >= page_lsn.
- Maintain a flush queue (or prioritized pass) so the checkpointer can pick pages in LRU order or by dirtiness age to minimize random IO amplification.
- Batch writes and fsyncs: group commit at the WAL layer reduces the number of fsync calls and improves throughput; ensure your page flusher and WAL flush cooperate to avoid unnecessary waits.
Checkpoint pseudocode (simplified):
while (running) {
target_lsn = compute_checkpoint_target();
pages = select_dirty_pages_up_to(target_lsn, budget);
for (page : pages) {
write_page_to_disk(page); // asynchronous write
atomic_store(&page->flushed_lsn, page->page_lsn);
clear_dirty_bit(page);
}
sleep(checkpoint_interval);
}
Aggressive checkpointer behavior without throttling causes short-lived I/O storms and wide p99 penalties; conservative checkpointer behavior increases recovery time. Instrument write throughput, checkpoint write time, and percent of pool dirty to find the right balance.
Because write throughput and device characteristics differ (consumer NVMe vs provisioned cloud volumes), expose throttle knobs: pages/sec or bytes/sec for checkpoint writer, and maximum background write concurrency.
Prefetching, Read-Ahead, and OS Cache Interaction
Prefetching transforms high-latency synchronous page faults into predictable background activity. There are two high-level models:
- Kernel-assisted read-ahead: give the kernel a hint (posix_fadvise(fd, offset, len, POSIX_FADV_SEQUENTIAL)) and let the kernel populate its page cache and the process's subsequent reads hit RAM; use when you rely on the kernel cache and have spare OS-managed memory.
- Engine-controlled prefetch + direct I/O: open files with O_DIRECT, bypass kernel page cache, and manage prefetch into the engine's buffer pool using async I/O (io_uring, AIO, or thread-pool reads). This avoids double caching and puts memory control inside the engine but requires bookkeeping for alignment and concurrency.
System calls and hints: readahead() and posix_fadvise are useful primitives; readahead() triggers immediate asynchronous reads into the kernel cache while posix_fadvise declares access patterns.
Prefetch design principles:
- Detect sequential scans (monotonic page numbers, scanning cursors) and switch to aggressive prefetch only while the scan is active.
- Use a separate prefetch queue that inserts pages into the buffer pool with a weaker recency (so prefetches do not evict hot pinned pages).
- Throttle prefetch rate to stay inside your write-back budget and to avoid saturating the device.
Example prefetch pattern (conceptual):
// For a detected sequential scan:
for (offset = start; offset < end; offset += prefetch_window) {
posix_fadvise(fd, offset, prefetch_window, POSIX_FADV_WILLNEED);
async_read_into_buffer_pool(fd, offset, prefetch_window);
// throttle by tracking outstanding prefetch count
}
When you use O_DIRECT, prefetch reads go straight into engine buffers (no double cache), and you control exactly which pages consume DRAM.
Practical Application: Instrumentation, Tuning, and Operational Checklists
Below are concrete checklists and protocols you can implement immediately to improve observability and behavior.
Design-time checklist
- Define your memory budget for the buffer pool as a clear fraction of host RAM; reserve headroom for OS and JVM/native heaps.
- Choose IO model: O_DIRECT + engine-managed prefetch or kernel caching + hints (posix_fadvise). Document alignment and page-size assumptions.
- Pick an eviction policy and concurrency model: sharded CLOCK is a pragmatic starting point for high-concurrency systems.
- Define dirty-page targets and checkpoint cadence (e.g., aim to keep steady-state dirty ratio within a band that your storage can absorb).
Implementation checklist
- Implement atomic pin() / unpin() APIs and a non-blocking try_pin().
- Keep per-page metadata small: pin_count, refbit, dirty, page_lsn, flushed_lsn.
- Expose counters: evictions, failed_evictions, pinned_waits, flushes_by_eviction, background_flush_bytes/sec, checkpoint_duration_ms.
- Implement a background flusher and a separate checkpointer with budget-based throttling.
- Add instrumentation hooks into the WAL path so the flusher can reason about the LSN frontier.
Operational checklist (metrics and commands)
- Buffer hit ratio: target depends on workload (OLTP point lookups expect high hit ratios); track hit_count / (hit_count + miss_count).
- Dirty ratio: dirty_pages / total_pages — use this to trigger background flushing or to adjust target rates.
- Checkpoint metrics: measure checkpoint write time, bytes written, and device utilization during checkpoints. Postgres exposes pg_stat_bgwriter with checkpoints_timed, checkpoints_req, buffers_checkpoint, buffers_clean, checkpoint_write_time. Querying those helps tie spikes to checkpoint activity.
- Pin contention: pinned_wait_count and median/99th pin wait latency tell if long-lived pins are blocking eviction.
- I/O saturation signals: iowait, device service time, queue depth, and iostat -x metrics — correlate these with buffers_clean and checkpoint writes.
- Engine-specific: InnoDB status for buffer pool and checkpoint activity (SHOW ENGINE INNODB STATUS) and RocksDB cache stats exposed via its statistics interface.
Quick runbook for a recurring p99 spike that looks storage-related
1. Confirm a spike corresponds to increased checkpoint_write_time or buffers_checkpoint (DB metric).
2. Check device metrics (iostat, nvme-cli, cloud volume metrics) for increased latency or throughput saturation.
3. Inspect eviction counters to find whether many evictions are failing due to pinned/dirty pages.
4. If the dirty ratio bursts, increase background flusher throughput or reduce checkpoint burst size by spreading writes (change checkpoint throttle/budget).
5. If kernel page cache and buffer pool are both large, evaluate switching to O_DIRECT or reduce one of the caches to free RAM.
Small examples — Postgres queries and OS tools
-- Postgres: useful bgwriter/checkpoint metrics
SELECT checkpoints_timed, checkpoints_req, buffers_checkpoint, buffers_clean,
maxwritten_clean, buffers_backend, buffers_alloc
FROM pg_stat_bgwriter;
OS tools: iostat -x, iotop -o, vmstat 1, perf record, bpftrace for pin wait traces.
Testing and validation
- Synthesize workloads where the working set is (a) smaller than buffer pool, (b) slightly larger, (c) massively larger. Observe hit-rate, evictions/sec, and p99 latency to confirm behavior.
- Run crash-and-recover tests that kill the process during checkpoints and validate recovery time and WAL replay semantics.
- Measure how prefetch affects hit rate and eviction churn — track prefetch admission vs prefetch evictions.
Sources:
Latency numbers every programmer should know - Reference for order-of-magnitude latency comparisons between CPU cache, DRAM, NVMe, and spinning disks used to explain why buffer pools matter.
PostgreSQL: Shared Buffer (storage buffer) and bgwriter/checkpoint metrics - Descriptions of PostgreSQL shared buffers, bgwriter, and associated monitoring counters referenced for buffer pool semantics and instrumentation.
PostgreSQL: Write-Ahead Logging (WAL) - WAL ordering, checkpoints, and group-commit behavior used to justify flush ordering and checkpointer design.
posix_fadvise(2) — Linux manual page - Documentation for file access pattern hints and their semantics (used for prefetch/read-ahead discussion).
MySQL / InnoDB Buffer Pool - InnoDB buffer pool design and flushing behavior cited when describing background flush and dirty ratio strategies.
RocksDB — Memory Usage (Wiki) - Notes on LSM-engine memory components (memtable, block cache) and how memory choices affect compaction and I/O patterns.
readahead(2) — Linux manual page - System call reference for triggering kernel read-ahead used in prefetch strategy discussion.
Page replacement algorithm — Wikipedia - Survey of LRU, CLOCK, LRU-K, LIRS, and related algorithms used to compare eviction strategies and properties.
open(2) — Linux manual page (O_DIRECT) - O_DIRECT semantics and considerations for bypassing the kernel page cache referenced in the kernel-bypass discussion.
A robust buffer pool is an exercise in orchestration: pin correctly, evict cheaply, flush in a controlled fashion, and let prefetching be a gentle helper rather than a memory robber. Follow the instrumentation checklist, codify the invariants (pin_count, page_lsn, flushed_lsn, dirty), and the storage layer will stop being the wildcard that spoils otherwise predictable systems.
Mentions
`shutdown -h now` destroyed 13 of our 48 Ceph OSDs. Here is how we got the data back --- title: "shutdown -h now destroyed 13 of our 48 Ceph OSDs. Here is how we got the data back" published: true description: A planned power-down killed the RocksDB index on 13 OSDs. Every supported repair tool refused to run, because each one opens the database it is supposed to repair. This is what worked. tags: ceph, storage, incidenthandling, devops --- We powered a site down on purpose. Electricians needed the room dead for an hour to upgrade a panel and fit a UPS. We had a runbook, we followed it, every application database shut down cleanly and reported exit code 0. Then we turned the site back on and 13 of our 48 Ceph OSDs refused to start. Not one disk had failed. Every byte of object data was intact. But each of those 13 OSDs had a destroyed RocksDB index, and without it BlueStore will not mount the object store at all. At the worst point 103 placement groups were offline, our shared filesystem was degraded, and two production object stores had data we could not reach. This is the story of how we got all of it back except a single 4MB object, and why every supported repair tool in Ceph was useless to us. First, what Ceph actually is (skip this if you run it) If you already operate Ceph, jump to the next section. If you do not, none of the rest will land without this. Ceph is distributed storage. You give it a pile of disks spread across a pile of machines, and it presents them as one system that speaks three protocols at once: block devices for VMs, a POSIX filesystem you can mount, and an S3 compatible object store. Same cluster, same disks, three front doors. The pieces you need for this story: - OSD (Object Storage Daemon). One process per disk. A host that owns 21 disks runs 21 OSD processes. This is the unit of storage and the unit of failure, and how many of them sit on one host is the whole root cause. - Placement group (PG). A pool is sliced into thousands of PGs, and each PG is replicated onto a set of OSDs. Clients do not track objects, they compute a PG with a hash and then compute its OSDs with an algorithm called CRUSH. No lookup table, no metadata server in the data path. It is genuinely elegant. - Failure domain. You tell CRUSH what a "thing that can die" is: a disk, a host, a rack. CRUSH then guarantees replicas land in different ones. Pick this wrong and your replication is decorative. - MON (Monitor). A small quorum of daemons holding the maps that describe the cluster. They do not touch your data, but nothing works without them. - BlueStore. Modern OSDs do not use a filesystem. The OSD writes objects to the raw block device itself, which is faster and avoids a whole category of double-write problems. And that last one is where this incident lives. BlueStore writes your object data straight to the disk, but it has to remember where everything went: which object is at which offset, its checksums, its attributes, the PG bookkeeping. That index lives in an embedded RocksDB on the same device. The analogy that made this click for our non-storage colleagues: the disk is a book with no page numbers and no table of contents, and RocksDB is the index. The text is all still there. But with no index, you cannot find chapter 4, and BlueStore will not even open the book to let you try. It refuses to mount at all. A visual representation may help, Ceph works more or less like this: !Image description What Ceph is genuinely good at - Hardware death. Pull a disk, lose a host, and it re-replicates itself back to full redundancy with nobody paged. This works so well it is easy to forget it is happening. - Scaling sideways. Add disks, add hosts, capacity and throughput go up. No forklift migration. - One system, three protocols. Not running separate SAN, NAS and object stacks is worth real money and real headcount. - No proprietary controller. It is open source, so when it breaks in an interesting way you can read the code that is failing. That mattered enormously here. What Ceph is bad at, and you should hear this before you deploy it - It is complex, and the complexity is not optional. CRUSH rules, pool sizing, failure domains, PG counts. You will get one of these wrong at least once. - It assumes failures are independent. Its entire design premise is that disks and hosts die one at a time, at random. A single operator action that hits every node at once is the one thing the model does not cover, and that is exactly what we did. - Small clusters are fragile in a way big ones are not. With thousands of OSDs, losing 13 is a Tuesday. With 48, losing 13 takes pools offline. - The recovery tooling assumes the metadata is healthy. This is the sharp edge of the whole article. Ceph is excellent at recovering from a dead disk. It is much weaker at recovering from an intact disk with a broken index, because almost every tool that would fix the index needs to open the index first. None of this is a reason to avoid Ceph. We would deploy it again. But "self-healing" describes hardware failure, not operator failure, and we learned the difference the expensive way. Our setup, and why it is shaped like this Nothing exotic, but the shape explains the blast radius. consumers (VMs, apps, S3 clients) | storage network | +-----------+-----------+-----------+-----------+-----------+ | | | | | | sto-01 sto-02 sto-03 sto-04 mon x3 | | | | (maps only, +-----+-----+ +-----+-----+ no data) | | +-------v--------+ +-------v--------+ | ENCLOSURE A | | ENCLOSURE B | | 24 NVMe | | 24 NVMe | | dual-ported | | dual-ported | +----------------+ +----------------+ 48 OSDs total, unevenly split within each host pair real split was 3 / 21 / 5 / 19 across the four hosts CRUSH failure domain = ENCLOSURE (not host) All pools: size=2, min_size=2, one copy in each enclosure Four storage hosts, two external NVMe enclosures, 48 OSDs. The drives are dual-ported, meaning each physical disk is wired to two hosts at once, so either host of a pair can take over the OSDs if its partner dies. That is a nice property and it is also why the topology is not the obvious one. The OSDs are not spread evenly. The real split across the four hosts was 3, 21, 5 and 19. Within each pair, one machine had quietly ended up owning almost every drive in its enclosure while its partner held a handful. Nobody designed that, it accumulated. It is also the single fact that decided how bad this incident got. Two decisions here matter for what follows: The failure domain is the enclosure, not the host. This looks wrong until you remember the disks do not live in the hosts. If CRUSH thought "host" was the boundary, it could happily place both replicas of a PG on two OSDs that are physically in the same enclosure, served by two different hosts. Lose that enclosure and you lose both copies. Setting the domain to the chassis forces one replica into each enclosure, so any pool survives an entire enclosure going away. size=2, min_size=2. Two copies, and both must be present for writes to be accepted. This is a deliberate trade: it refuses to take writes it cannot fully replicate, rather than accepting them onto one copy and hoping. On a two-enclosure layout it is the honest setting. Now the arithmetic that hurt. A PG only goes offline when both of its copies are unavailable, and since the copies are always in different enclosures, that needs one failed OSD in A and one failed OSD in B, both holding the same PG. Our 13 dead OSDs were split across both enclosures. Every PG that happened to have landed on one from each side went dark. That produced 88 offline PGs from 13 failures, and later 103 as more state came in. And the number that actually caused the incident: 21 OSD processes on a single host. Ask systemd to stop 21 of anything at once, against a single default timeout, while each one is trying to flush its own database. Some of them will not make it. Had those same 48 OSDs been spread 12 / 12 / 12 / 12, every host would have had twice the timeout budget per daemon. The scene Here is what we knew an hour after power came back: - 35 of 48 OSDs up. The other 13 crash-looping across all four storage hosts - Every failing OSD reported a variation of the same error: rocksdb: verify_sharding unable to list column families: Corruption - The disks themselves were healthy. No I/O errors in dmesg, no SMART complaints - 88 placement groups already offline, by the arithmetic above. That number grew to 103 as more state came in - Our cluster had been HEALTH_OK with 48/48 OSDs for three weeks before this The last point matters. This was not creeping hardware rot. Something we did during the shutdown broke them all at once. Step 1: The cluster lied to us about its own state (ceph osd set nodown) The first ceph -s after power-on looked fine: 48 osds: 48 up (since 86s), 48 in (since 3w) It was wrong. Our shutdown procedure sets the standard quiesce flags before halting, and nodown was still set. nodown prevents OSDs being marked down, so the monitor keeps reporting them as healthy regardless of reality. We released the flags and got the truth: 48 osds: 35 up (since 7m), 48 in (since 3w) Lesson: if your power-on checklist validates the OSD count before releasing the quiesce flags, that check passes falsely. Release the flags first, or cross-check with systemctl list-units 'ceph-*osd*' --state=failed on each host. We spent ten minutes acting on a number that was fiction, and the only reason it was not longer is that an OSD count is easy to sanity-check from the hosts themselves. Step 2: What actually broke (bluefs-export) BlueStore keeps its metadata in an embedded RocksDB, which lives on a small filesystem called BlueFS on the same block device. RocksDB continuously rewrites two small files: - CURRENT: a 16 byte pointer containing the current MANIFEST's filename - MANIFEST-NNNNNN: the log of version edits describing which SST files hold what fsck would not open the database. But bluefs-export does not need RocksDB at all, it just reads BlueFS and dumps the files to a directory: cephadm shell --name osd.0 --mount /var/tmp/osd0-bluefs:/export -- \ ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-0 bluefs-export --out-dir /export That worked on every single failing OSD. Suddenly we had the whole database as ordinary files we could inspect without risk. This one command is the foundation of everything that follows. What it showed on the first OSD: 25 SST files present and intact, MANIFEST present, and CURRENT containing 16 bytes of binary garbage where it should have contained the string MANIFEST-000147\n. Here is the giveaway. A healthy MANIFEST starts with a CRC and a known string. Ours did not: healthy manifest: 56 f9 b8 f8 1c 00 01 01 1a 6c 65 76 65 6c 64 62 >V........leveldb< our MANIFEST: 00 00 00 00 00 00 68 07 00 00 00 00 00 00 00 00 >......h.........< another OSD: 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 >aaaaaaaaaaaaaaaa< That run of 0x61 is the letter a repeated. It is a disk test fill pattern left over from whenever those drives were last burned in. That is what unflushed blocks read back as. So BlueFS had journaled the file metadata for those writes, recording size, mtime and allocated extents, but the file contents were never flushed to the device. On restart RocksDB read the recorded size and got whatever was on those blocks beforehand. Every damaged file carried an mtime inside our shutdown window. We did this to ourselves. The root cause, in one sentence shutdown -h now on a host running 21 OSD containers asks systemd to stop them all in parallel against its default timeout, and SIGKILLs any that do not finish in time. Our runbook was meticulous about stopping applications gracefully, giving every database a generous -t and verifying exit code 0. Not one application database needed crash recovery afterwards. The storage layer got none of that courtesy: just shutdown -h now and hope. Step 3: Every repair tool refuses to run Here is the part that makes this failure mode nasty. Ceph ships tools that could fix this. None of them will run. bluefs-import, the supported way to write a file back into BlueFS: inferring bluefs devices from bluestore path rocksdb: verify_sharding unable to list column families: Corruption: CURRENT file does not end with newline error preparing db environment: (5) Input/output error It opens the RocksDB environment before writing. Which is the thing that is broken. ceph-kvstore-tool bluestore-kv ... destructive-repair: mounts BlueFS fine, then opens RocksDB before repairing. Same wall. RocksDB's best_efforts_recovery option, which exists precisely to ignore a corrupt CURRENT and MANIFEST and rebuild from the SST files, is a DB::Open option. Ceph calls rocksdb::DB::ListColumnFamilies() before DB::Open(), and ListColumnFamilies reads the MANIFEST directly. The option never gets a chance to apply. Every road is circular: to repair the index, the tool must first open the index. Step 4: Writing to the raw block device (bluefs-log-dump) If no tool will write the file, write it yourself. BlueFS records every file's inode and physical extents on the block device, and you can dump that log: ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-0 bluefs-log-dump 0x1107f: op_dir_link db/CURRENT to 137 0x10cd4: op_file_update file(ino 137 size 0x10 ... extents [1:0x568000~10000]) db/CURRENT is inode 137, size 0x10 (16 bytes), living at device offset 0x568000. Before writing anything, prove the location and save a rollback: # read those bytes raw and byte-compare against the exported copy dd if=$DEV bs=1 skip=$((0x568000)) count=16 status=none > /tmp/raw16.bin cmp /tmp/raw16.bin /var/tmp/osd0-bluefs/db/CURRENT && echo "location confirmed" # save the whole enclosing 4K block as an exact rollback dd if=$DEV bs=4096 skip=$((0x568000/4096)) count=1 of=/var/tmp/block.orig status=none Then a read-modify-write of that single block, replacing only the first 16 bytes with MANIFEST-000147\n. The error changed: before: Corruption: CURRENT file does not end with newline after: Corruption: checksum mismatch in file db/MANIFEST-000147 Progress. The fix worked and exposed the next layer. Gotcha that will bite you: some OSDs record their extents in op_file_update_inc (incremental) records rather than op_file_update. If you only read the latter you get a stale offset with no extents and you will write to the wrong place. Our byte-comparison guard caught exactly this on one OSD and aborted before writing. Never skip the comparison step. Step 5: Rebuild the index on a copy, not on the patient The MANIFEST could not be repaired in place. We wrote a small parser for RocksDB's log format (7 byte header: masked CRC32C, length, type) and ran it over the file: file size: 11592 bytes valid records: 0 FIRST BAD RECORD at offset 0 -> CRC mismatch type=104 len=0 Type 104 is 0x68, not a valid record type. Stale bytes from offset 0. There was nothing to truncate back to, which is the difference between "interrupted mid-write" and "never written at all". If a file had been torn mid-write we would have seen N valid records and then damage, and we could have truncated to the last good offset. Zero valid records means the contents never reached the disk. That distinction is worth being able to measure, so here is the parser. Pure stdlib, because it has to run on a storage host or inside a cephadm container where you cannot install anything. {% details rocksdb-log-check.py (full script) %} #!/usr/bin/env python3 """ rocksdb-log-check.py - validate a RocksDB MANIFEST / WAL record by record. Answers one question: how much of this file is still readable, and where exactly does it stop being readable? python3 rocksdb-log-check.py db/MANIFEST-000147 THE FORMAT (rocksdb/db/log_format.h, log_reader.cc) A MANIFEST is a log file: a sequence of 32 KiB blocks, each holding one or more records. +--------+--------+------+-----------------------+ | CRC32C | length | type | payload | | 4 LE | 2 LE | 1 | <length> bytes | +--------+--------+------+-----------------------+ * The CRC is CRC32C (Castagnoli, poly 0x82F63B78), NOT zlib's CRC32, and it is stored *masked* (see unmask below). * The CRC covers the type byte plus the payload. Not the header. * Records never straddle a block boundary. If fewer than 7 bytes remain in a block, the writer zero-fills them and starts the next block. * A run of zeroes is normal padding, not damage. RocksDB's own reader tolerates it, which is also why zero-padding a rebuilt manifest back into place works. READING THE OUTPUT valid records: 0, bad record at offset 0 The file never held a valid record. Nothing to salvage, nothing to truncate back to. This is the signature of unflushed data: the metadata was journaled but the contents never reached the device, so you are reading whatever was on those blocks beforehand. valid records: N, bad record partway in Torn mid-write. Truncating to the last good offset usually yields a loadable manifest. A type byte outside 1..8 means you are almost certainly looking at stale disk bytes rather than a damaged record. """ import struct import sys BLOCK_SIZE = 32768 # rocksdb::log::kBlockSize HEADER_SIZE = 7 # 4 byte crc + 2 byte length + 1 byte type MASK_DELTA = 0xA282EAD8 # rocksdb::crc32c::kMaskDelta POLY = 0x82F63B78 # CRC32C (Castagnoli), reversed representation RECORD_TYPES = { 0: "kZeroType", 1: "kFullType", 2: "kFirstType", 3: "kMiddleType", 4: "kLastType", 5: "kRecyclableFullType", 6: "kRecyclableFirstType", 7: "kRecyclableMiddleType", 8: "kRecyclableLastType", } # Python's zlib only ships CRC32, so build a CRC32C table by hand. _TABLE = [] for _i in range(256): _c = _i for _ in range(8): _c = (_c >> 1) ^ (POLY if _c & 1 else 0) _TABLE.append(_c) def crc32c(data, crc=0): crc ^= 0xFFFFFFFF for byte in data: crc = _TABLE[(crc ^ byte) & 0xFF] ^ (crc >> 8) return crc ^ 0xFFFFFFFF def unmask(masked): """Inverse of rocksdb::crc32c::Mask: subtract the delta, rotate right 17.""" rot = (masked - MASK_DELTA) & 0xFFFFFFFF return ((rot >> 17) | (rot << 15)) & 0xFFFFFFFF def check(path): with open(path, "rb") as handle: data = handle.read() print(" file: %s" % path) print(" file size: %d bytes" % len(data)) pos = 0 records = 0 last_good = 0 failure = None while pos + HEADER_SIZE <= len(data): # Records never straddle a block boundary. A short tail is padding. offset_in_block = pos % BLOCK_SIZE if BLOCK_SIZE - offset_in_block < HEADER_SIZE: pos += BLOCK_SIZE - offset_in_block continue header = data[pos:pos + HEADER_SIZE] masked_crc = struct.unpack("<I", header[0:4])[0] length = struct.unpack("<H", header[4:6])[0] rec_type = header[6] # An all-zero header is preallocated space, not corruption. if masked_crc == 0 and length == 0 and rec_type == 0: print(" [zero region begins at offset %d]" % pos) break payload = data[pos + HEADER_SIZE:pos + HEADER_SIZE + length] if len(payload) < length: failure = (pos, "truncated payload: want %d, have %d" % (length, len(payload))) break # The CRC covers the type byte and the payload, not the header. expected = unmask(masked_crc) actual = crc32c(bytes([rec_type]) + payload) if expected != actual: failure = (pos, "CRC mismatch type=%d (%s) len=%d want=%08x got=%08x" % (rec_type, RECORD_TYPES.get(rec_type, "INVALID"), length, expected, actual)) break records += 1 pos += HEADER_SIZE + length last_good = pos print(" valid records: %d" % records) print(" last good byte offset: %d" % last_good) if failure: print(" FIRST BAD RECORD at offset %d -> %s" % failure) else: print(" no bad record found before EOF or zero padding") print(" trailing bytes after last good: %d" % (len(data) - last_good)) return 0 if records and not failure else 1 if __name__ == "__main__": if len(sys.argv) != 2: sys.exit(__doc__.strip().splitlines()[0]) sys.exit(check(sys.argv[1])) {% enddetails %} Run it against a healthy manifest from any working OSD first, so you know what a good result looks like on your own cluster before you trust it on a broken one. But RocksDB has a repairer that rebuilds the MANIFEST by scanning the SST files. It will not run against a live broken OSD. It runs perfectly against a directory: cp -a /var/tmp/osd0-bluefs /var/tmp/osd0-test # work on a copy cephadm shell --name osd.0 --mount /var/tmp/osd0-test:/t -- \ ceph-kvstore-tool rocksdb /t/db destructive-repair The results were much better than we feared: | OSD | SST files before | after | quarantined | |---|---|---|---| | A | 28 | 28 | MANIFEST only | | B | 33 | 33 | MANIFEST only | | C | 24 | 24 | MANIFEST only | | D | 25 | 23 | MANIFEST + 2 SSTs | | E | 37 | 31 | MANIFEST + 6 SSTs | On most OSDs it retained every single SST. The object metadata was fine all along. Only the index describing it was gone. Now we had a valid rebuilt MANIFEST sitting in a directory, and the same problem as before: how do we get it back in? Same answer. Locate the original MANIFEST's extents, write the rebuilt one into them, zero-padded. RocksDB's log reader explicitly tolerates zero-filled regions, because preallocation produces them. Step 6: The assert that sent us to the source code The database opened. Then: src/kv/RocksDBStore.cc: 1221: FAILED ceph_assert(recreate_mode) No message, no explanation. So we read the source, and it explained everything: static const char* sharding_def_file = "sharding/def"; static const char* sharding_recreate = "sharding/recreate_columns"; static const char* resharding_column_lock = "reshardingXcommencingXlocked"; ... bool recreate_mode = status.ok() && sharding_recreate_text == "1"; ... if (missing_cfs.size() > 0 && /* lock not among missing */) { ceph_assert(recreate_mode); for (...) db->CreateColumnFamily(missing_cfs[i].options, ...); } ... if (existing_cfs.size() != rocksdb_cfs.size()) { derr << "extra columns in rocksdb..."; return -EIO; } Three facts fell out, and they define the whole problem space: 1. Missing column families: Ceph will create them for you, but only in recreate_mode, which is set by a BlueFS file sharding/recreate_columns containing "1". Which only bluefs-import can create. Which needs the database open. Circular again. 2. Extra column families: hard -EIO, no workaround at all. 3. The assert is skipped entirely if a magic column family named reshardingXcommencingXlocked is among the missing ones. Our rebuilt database had fewer column families than sharding/def declared, because RocksDB's repairer only creates families it finds SSTs for, and some of ours were empty. So the fix was to make sharding/def match reality. If you take one thing from this article, take this: when an assert has no message, read the source. RocksDBStore.cc lines 1025 to 1240 turned an impossible problem into a mechanical one. Step 7: The 130 byte file that cost us an attempt sharding/def is a plain text file describing the column family layout: m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32 It is a fixed size BlueFS file, so any replacement must be exactly the same byte length. Drop the m(3) entry and you are 5 bytes short. The obvious move is to pad with trailing spaces. Do not pad with spaces. The parser splits on spaces in a while (!text_def.empty()) loop, so trailing spaces create phantom column families with empty names, which keeps missing_cfs non-zero and fires the same assert. That cost us a full attempt, a byte-for-byte revert, and a second pass at working out why an apparently correct definition still would not parse. Pad by expanding a numeric option value with leading zeros instead. Same value, no trailing space: p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=0000032 P=min_write_buffer_number_to_merge=32 The database opened, and fsck started reporting ordinary BlueStore findings instead of refusing to mount. On several OSDs it simply returned fsck success. One more flag was needed for the write path: rocksdb: Corruption: force_consistency_checks: VersionBuilder: L0 file #409 with seqno 12429585 15050034 vs. file #501 with seqno 14519025 14953111 That is an artefact of the repairer reconstructing the manifest with overlapping sequence numbers. Setting force_consistency_checks=false in bluestore_rocksdb_options unblocked it. That is a config change, not a data change, and it is reversible. The first OSD came up. The shared filesystem went healthy immediately. Step 8: Stop repairing OSDs you do not need (osdmaptool) At this point we had a proven procedure and 12 more broken OSDs. Getting the first one back had taken half an hour of careful, irreversible work, and every one after it still meant raw-device writes on hardware holding live data. Before grinding through all of them, we asked a better question: which of these OSDs does anyone actually still need? First, get the list of what is genuinely broken, not what is merely unhappy: ceph pg dump pgs_brief | awk '$2 !~ /active\+clean/ {print $1, $2}' 4.3ff down 4.3a3 down 4.266 undersized+peered 9.2 undersized+peered 10.8 down ... Two different states, and the difference is everything: - undersized+peered means one replica is gone. The other is alive and holds a complete copy. The PG is not serving writes because min_size=2, but no data is at risk. - down means both replicas are on failed OSDs. Nothing is serving it and nothing can, until one of those two OSDs comes back. Now ask CRUSH where each of those PGs is supposed to live: ceph osd getmap -o osdmap.bin osdmaptool osdmap.bin --mark-up-in --test-map-pg 4.3ff osdmaptool: osdmap file 'osdmap.bin' parsed '4.3ff' -> 4.3ff 4.3ff raw ([17,43], p17) up ([17,43], p17) acting ([17,43], p17) --mark-up-in is the flag that makes this useful. It tells osdmaptool to compute placement as if every OSD in the map were up and in. Without it you get the current degraded reality, which just repeats what you already know: the PG has nowhere to live. With it you get the intended placement, which is the question you actually care about: if I fix these OSDs, where does this PG go? Reading the output 4.3ff wants two OSDs, osd.17 and osd.43. Our 13 dead OSDs were 0, 1, 5, 6, 7, 8, 9, 10, 13, 15, 16, 17, 21. So for this PG: - osd.17 is dead - osd.43 is alive One of the two survives, and because it is a full replica it holds every object in that PG. This PG needs no repair work at all. We can purge osd.17 outright and Ceph will re-replicate the PG from osd.43 onto any healthy OSD in the right enclosure. The hours of manifest surgery would buy us nothing here. Compare 10.8, which came back acting ([5,32]). Both osd.5 and osd.32... osd.5 is in our dead list. osd.32 is not, but 10.8 was reported down rather than undersized, which told us the map was stale and osd.32 had also dropped. When both members of the pair are dead, that is a PG we must go and rescue, and it is the only kind that justifies the expensive procedure. So every stranded PG falls into exactly one of three buckets: | Dead replicas | PG state | What it means | Action | |---|---|---|---| | 0 of 2 | active+clean | untouched | nothing | | 1 of 2 | undersized+peered | a complete copy survives | purge the dead OSD, let it re-replicate | | 2 of 2 | down | no live copy anywhere | must repair at least one of the pair | Running it over everything Loop the whole inactive list and bucket it: ceph osd getmap -o osdmap.bin DEAD="0 1 5 6 7 8 9 10 13 15 16 17 21" ceph pg dump pgs_brief | awk '$2 !~ /active\+clean/ {print $1}' | while read pg; do acting=$(osdmaptool osdmap.bin --mark-up-in --test-map-pg "$pg" 2>/dev/null \ | grep -oE 'acting \(\[[0-9,]+\]' | grep -oE '[0-9]+(,[0-9]+)*') n=0 for osd in ${acting//,/ }; do case " $DEAD " in *" $osd "*) n=$((n+1));; esac done echo "$pg acting=[$acting] dead_replicas=$n" done | sort -k3 -r The tally across all 88 stranded PGs: dead_replicas=2 23 PGs <- the real work dead_replicas=1 65 PGs <- purge and re-replicate, no repair needed 65 of the 88 offline PGs did not need anything repaired. They were offline purely because min_size=2 refuses writes at one copy, not because the data was gone. Purging the dead OSD and letting Ceph re-replicate from the survivor fixes them, and that is a normal, boring, supported operation. That left 23 PGs that genuinely had no live copy. Taking the union of the OSDs appearing in those 23 both-dead pairs gave us a critical set of 5 OSDs, not 13. The other 8 could be purged immediately. One more thing the output told us, and it is the payoff for getting the failure domain right back in the setup section. Because CRUSH is constrained to put one replica in each enclosure, a both-dead pair is always one OSD in enclosure A and one in enclosure B. So for each of those 23 PGs we had a free choice: whichever of the two OSDs looked more repairable was the one to work on. We never had to fix a specific OSD, only one of a pair. Two of the five turned out to be unrecoverable, and it cost us nothing, because their partners carried the same PGs. Lesson: in a recovery, the cheapest work is the work you prove you do not have to do. Ten minutes with osdmaptool --mark-up-in turned 13 expensive repairs into 5, and told us exactly which 8 OSDs we could stop worrying about. Do this before you start repairing, not after. Step 9: Get the data out, do not put the OSD back For the OSDs too damaged to return to service, there is a better move than nursing them into production. Extract the placement groups and import them onto healthy hardware: # from the damaged OSD ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-9 --op export \ --pgid 9.2 --file /x/9.2.export \ --bluestore-rocksdb-options="...,force_consistency_checks=false" # onto a healthy one (stop it first) ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-43 --op import --file /x/9.2.export This is what Ceph maintainers recommend for this situation, and it is much safer than running a hand-patched OSD in production. We recovered 22 placement groups this way, including a 13.6GB one, moving them between hosts over the storage network in seconds. Two tricks that mattered here: A single unreadable object can block a whole PG export. Ours aborted with read failure: (5) Input/output error on one object. Remove that object and the other 25 export fine: ceph-objectstore-tool --data-path ... --pgid 4.3a3 \ '["4.3a3",{"oid":"...","key":"","snapid":-2,"hash":...,"pool":4,"namespace":""}]' remove ceph osd lost and ceph osd purge do not touch the disk. They edit the OSD map only. We verified this empirically by exporting a complete 84MB placement group from a fully purged OSD, using --no-mon-config to skip the auth lookup that the purge removed. That property is what made the whole recovery safe: nothing was destroyed to reach a healthy cluster. The one object we lost Final tally: one object out of 159,001. It was a 4MB block that was physically unreadable on its disk, a genuine hard I/O error, in a placement group whose other replica was on another failed OSD. We identified exactly what it was by decoding the filesystem backtrace stored in the first object of the same file: rados -p cephfs.cephfs.data getxattr 10000103179.00000000 parent > parent.bin ceph-dencoder type inode_backtrace_t import parent.bin decode dump_json That gave us the full path. It turned out to be a fragment of a nightly backup image from five days earlier, superseded by three newer complete backups. 516 of that file's 517 objects survived. Being able to answer "exactly which file, exactly which bytes" is worth the effort. "We lost one object" is a shrug. "We lost bytes 16MB to 20MB of a superseded backup taken on the 13th, and here are the three newer copies" is an answer you can take to a customer. And then, before declaring victory: ceph pg deep-scrub <pgid> # for every recovered PG inconsistent_pgs: 0 scrub_errors: 0 pgs_by_state: [('active+clean', 1265)] "Import successful" is not the same as data integrity. Scrub is the proof. What we got wrong (and what we fixed) 1. We never stopped the OSDs. The whole incident traces to this. The runbook stopped applications carefully and left up to 21 OSD containers per host to systemd's default timeout. Now: systemctl stop ceph-osd.target, then a loop that waits and confirms every OSD container is gone before shutdown is allowed to run. 2. We trusted ceph -s with nodown set. It reported 48 of 48 OSDs up while 13 were crash-looping. Release the flags before validating anything, and never let a post-power-on checklist assert on a number the flags can fake. 3. Our runbook's storage topology was wrong. It said one host held zero OSDs. It held three, and they were load-bearing. A document nobody re-verifies drifts into fiction, and you find out during an incident. 4. sudo docker compose with ~ in a volume path. During restart, ~/models/huggingface expanded to /root, and Docker silently created an empty directory instead of failing. Cost us a crash-looping service until we noticed the bind was wrong. Check docker inspect mounts, do not assume. 5. We twice concluded the data was unrecoverable. Both times we were wrong, and both times someone pushed to keep going. The second time, that was the difference between losing 18 placement groups and losing one object. The takeaway Ceph is genuinely good at surviving hardware failure. It is much less good at surviving you, and the tooling assumes a working database in places where you most need it not to. Three things are worth internalising before you need them: Work on copies. bluefs-export gives you the entire database as files while the OSD refuses to mount. Everything destructive we did, we did on a copy first, then applied. We reverted three separate attempts byte-for-byte and lost nothing by trying them. Verify before you write, always. Every raw device write was preceded by a byte comparison proving the offset, and by saving the block we were about to overwrite. That discipline caught a wrong offset that would have corrupted an OSD we still needed. Read the source. The assert that blocked us cold had no message. Two hundred lines of RocksDBStore.cc turned it into a checklist. If you run infrastructure on open source, the source is part of your documentation. And go check your shutdown procedure right now. If it stops your applications carefully and then calls shutdown -h now on a host running database daemons, you have the same bug we had. It has not bitten you yet because your daemons have always finished in time. One day one of them will not. The tools, and where to get them Everything we wrote during this recovery is published. The first two are dangerous in different ways, so read the header comments before running either. | Tool | What it does | |---|---| | rocksdb-log-check.py | Validates a RocksDB MANIFEST or WAL record by record. Tells you whether a file was torn mid-write (truncate and recover) or never written at all (nothing to salvage). Pure stdlib, so it runs inside a cephadm container. | | bluestore-manifest-transplant.sh | The Step 4 to Step 7 procedure, packaged: locate CURRENT, MANIFEST and sharding/def on the raw device, byte-verify every offset, save a rollback, rebuild sharding/def to match the recovered column families, write. Dry run by default, and it aborts if any offset fails verification. | | ceph-safe-shutdown.sh | The fix. Stops ceph-osd.target and then waits and confirms every OSD is actually gone, exiting non-zero if any are still running when the budget expires. Chain it: ./ceph-safe-shutdown.sh && shutdown -h now. This is the script whose absence caused the entire incident. | If you only take one of them, take the last one. Further reading - Ceph: BlueStore config · ceph-bluestore-tool · ceph-objectstore-tool · CRUSH maps · Placement groups - RocksDB Repairer, which is what destructive-repair invokes - RocksDB log format, if you want to understand the parser - Ceph source: src/kv/RocksDBStore.cc, the sharding verification and column family handling. This is the file that turned an impossible problem into a mechanical one --- From the first failed OSD to a scrub-verified clean cluster took about three and a half hours. The single most valuable command in the whole incident was bluefs-export, and we found it in the man page.
rocksdb tailing iterators don't hold a fixed creation snapshot. they merge mutable + immutable iterators and invalidate internal state after memtable flushes or compactions. the lifetime rules interest me more than the fast path. github.com/facebook/rocksdb/wiki/Tailing-Iterator
Buffer Pool and Cache Management for Database Engines - How the Buffer Pool Anchors the Memory Hierarchy - Choosing an Eviction Policy: LRU, CLOCK, and Workload-Aware Variants - Pinning and Concurrency: Making Eviction Safe at Scale - Dirty Page Management: Flushing, Checkpoints, and WAL Discipline - Prefetching, Read-Ahead, and OS Cache Interaction - Practical Application: Instrumentation, Tuning, and Operational Checklists Buffer management is where microseconds become minutes: the buffer pool turns persistent I/O into in-memory work or it becomes the throttle that kills p99. Get eviction, pinning, and dirty-page flushing wrong and the storage layer will be the single largest source of unpredictable latency in production. You see this problem in three ways: stealthy tail-latency spikes during heavy scans or checkpoints, I/O storms when the evictor chases dirty pages, and persistent memory bloat because kernel and engine caches duplicate the same bytes. The symptoms look like the app is slow, but root cause analysis usually points to poor coordination between the buffer pool, eviction policy, prefetch heuristics, and the write path. How the Buffer Pool Anchors the Memory Hierarchy The buffer pool is the database engine's primary residency for hot data: it takes pages off block I/O and keeps them in DRAM so repeated accesses hit memory instead of the device. It sits above the OS page cache and below application logic; that placement creates both its power and its complexity. PostgreSQL, MySQL/InnoDB and other systems implement a dedicated shared buffer manager for precisely this reason — the engine controls MVC semantics, pinning, and writeback ordering inside its pool rather than delegating those responsibilities to the kernel. > Important: The buffer pool is not just a cache; it's the authoritative runtime view of pages for MVCC and transaction safety. Your eviction and flush logic must respect transactional LSN/versioning semantics. Quick reality check — orders-of-magnitude matter. Typical round numbers (orders of magnitude) are: CPU caches (ns), DRAM (tens–hundreds ns), NVMe SSD (tens–hundreds μs), HDD (milliseconds). That gap is why avoiding device hits matters so much for p99. | Layer | Characteristic | Typical latency (order of magnitude) | |---|---|---:| | CPU caches | L1/L2/L3, CPU-local | nanoseconds | | DRAM / Buffer pool | Shared memory for DB | tens–hundreds of nanoseconds | | NVMe SSD | Fast persistent storage | tens–hundreds of microseconds | | Spinning disk | Mechanical access | milliseconds | Avoid double caching (engine buffer pool + kernel page cache) unless you have a reason to keep both. Bypass the kernel with O_DIRECT or use posix_fadvise hints when you want the kernel to help with read‑ahead, but know the trade-offs: O_DIRECT removes double caching but increases complexity for alignment and I/O buffering; kernel-assisted approaches are simpler but can waste memory. Choosing an Eviction Policy: LRU, CLOCK, and Workload-Aware Variants Eviction is the gatekeeper of memory reuse. The core options are well known, but their operational trade-offs matter more than their theoretical hit-rates. - LRU (Least Recently Used): conceptually simple, good for single-threaded or low-concurrency workloads where recency maps to future use. Implementation complexity rises when you need to make it concurrency-friendly (sharded LRU, lock striping), and cost of updating recency on every access can be high. - CLOCK / Second-Chance: a compact approximation to LRU that uses a circular hand and a single reference bit. Low per-page metadata and easier to make concurrent — a great pragmatic default for large engines. - Workload-aware variants: LRU-K, ARC, LIRS, CLOCK-Pro and multi-queue (SLRU) variants track deeper history or multiple recency windows to separate frequently used from recently used. They improve hit rates on mixed workloads at the price of more metadata and complexity. | Policy | Pros | Cons | When to prefer | |---|---|---|---| | LRU | Intuitive; good for recency-heavy workloads | High recency-update cost; contention under concurrency | Small to medium pools, low concurrency | | CLOCK | Low metadata, low update cost | Approximation — slightly worse hit-rate than perfect LRU | Large pools, high concurrency; pragmatic default | | LRU-K / LIRS / ARC | Better for mixed hot/cold and scan-resistance | More metadata and complexity | Workloads with long-term frequency differences | | Segmented LRU (SLRU) | Fast path for hot pages | Needs tuning of segment sizes | Workloads with clear hot-set vs bulk scans | Contrarian production insight: for many systems I’ve built and debugged, a well-tuned CLOCK (or sharded CLOCK) beats a naïve global LRU because it avoids the thrash and lock contention that kills throughput under concurrency. Example of a low-overhead CLOCK eviction loop (pseudocode): // Simplified CLOCK walker pseudocode while (true) { Page *p = clock_hand.next(); if (atomic_load(&p->pin_count) != 0) { continue; } // skip pinned if (p->refbit) { p->refbit = 0; // second chance, clear and move on continue; } if (p->dirty) { schedule_flush(p); // async write; skip until clean continue; } evict_page(p); break; } Make your eviction fast and observable: short scans, counters for failed evictions (pinned/dirty), and the ability to increase scan aggressiveness under memory pressure. Pinning and Concurrency: Making Eviction Safe at Scale Pinning is the crash-proof handle that prevents in‑flight pages from being evicted mid-use. The basic contract is simple: pin increments a pin_count, unpin decrements it, and eviction only succeeds when pin_count == 0. The devil is in the race conditions and in how long pins are held. - Represent pin_count with atomic integers (std::atomic / AtomicUsize) so pin is cheap and scalable. - Provide both pin() (blocks or spins until the page is present and pinned) and try_pin() (fast fail when the page cannot be pinned) APIs to let callers decide blocking semantics. - Avoid holding a pin while doing blocking IO or while waiting on unrelated locks; long-lived pins stall evictors and lead to memory pressure and write stalls. Pseudocode for safe fetch/pin pattern: Page* fetch_and_pin(page_id) { Page* p = hashtable_lookup(page_id); if (!p) { p = allocate_slot_and_read_from_disk(page_id); // Insert into hash with pin_count = 1 atomic_store(&p->pin_count, 1); return p; } else { atomic_fetch_add(&p->pin_count, 1); return p; } } void unpin(Page* p) { atomic_fetch_sub(&p->pin_count, 1); } Implementation notes: - Keep the critical section that pins a page as small as possible. - Use per-bucket or per-shard metadata to reduce global lock contention on the eviction structure. - Track pin wait latency as an SRE metric; frequent waits are a clear signal that something (long transactions, background compaction) is holding pins too long. > Operational warning: Holding pins across user-level locks, synchronous RPCs, or long computations is a leading cause of eviction starvation in production. Dirty Page Management: Flushing, Checkpoints, and WAL Discipline The Log is Law. Every modification must be reflected in the Write-Ahead Log (WAL) before the corresponding page can be considered safely durable on disk. That ordering gives you atomicity and crash-recovery guarantees: write WAL, fsync WAL, then you can write data pages. Three practical flush domains: 1. Eviction-driven flush (on-demand): when eviction encounters a dirty page, it flushes it before eviction. Pros: minimal background IO on light workloads. Cons: under pressure, a wave of evictions can cause write bursts. 2. Background flusher: a daemon that maintains a target dirty ratio (percentage of buffer pool dirty). It smooths writes over time and prevents large checkpoint bursts. 3. Checkpointer: at checkpoint time the engine ensures that pages are flushed up to a checkpoint LSN; it coordinates with WAL so recovery need only replay from that LSN forward. Checkpointing must be throttled to avoid saturating the device; stagger writes across time. Key invariants and implementation tips: - Track per-page page_lsn and flushed_lsn. A page is clean when flushed_lsn >= page_lsn. - Maintain a flush queue (or prioritized pass) so the checkpointer can pick pages in LRU order or by dirtiness age to minimize random IO amplification. - Batch writes and fsyncs: group commit at the WAL layer reduces the number of fsync calls and improves throughput; ensure your page flusher and WAL flush cooperate to avoid unnecessary waits. Checkpoint pseudocode (simplified): while (running) { target_lsn = compute_checkpoint_target(); pages = select_dirty_pages_up_to(target_lsn, budget); for (page : pages) { write_page_to_disk(page); // asynchronous write atomic_store(&page->flushed_lsn, page->page_lsn); clear_dirty_bit(page); } sleep(checkpoint_interval); } Aggressive checkpointer behavior without throttling causes short-lived I/O storms and wide p99 penalties; conservative checkpointer behavior increases recovery time. Instrument write throughput, checkpoint write time, and percent of pool dirty to find the right balance. Because write throughput and device characteristics differ (consumer NVMe vs provisioned cloud volumes), expose throttle knobs: pages/sec or bytes/sec for checkpoint writer, and maximum background write concurrency. Prefetching, Read-Ahead, and OS Cache Interaction Prefetching transforms high-latency synchronous page faults into predictable background activity. There are two high-level models: - Kernel-assisted read-ahead: give the kernel a hint (posix_fadvise(fd, offset, len, POSIX_FADV_SEQUENTIAL)) and let the kernel populate its page cache and the process's subsequent reads hit RAM; use when you rely on the kernel cache and have spare OS-managed memory. - Engine-controlled prefetch + direct I/O: open files with O_DIRECT, bypass kernel page cache, and manage prefetch into the engine's buffer pool using async I/O (io_uring, AIO, or thread-pool reads). This avoids double caching and puts memory control inside the engine but requires bookkeeping for alignment and concurrency. System calls and hints: readahead() and posix_fadvise are useful primitives; readahead() triggers immediate asynchronous reads into the kernel cache while posix_fadvise declares access patterns. Prefetch design principles: - Detect sequential scans (monotonic page numbers, scanning cursors) and switch to aggressive prefetch only while the scan is active. - Use a separate prefetch queue that inserts pages into the buffer pool with a weaker recency (so prefetches do not evict hot pinned pages). - Throttle prefetch rate to stay inside your write-back budget and to avoid saturating the device. Example prefetch pattern (conceptual): // For a detected sequential scan: for (offset = start; offset < end; offset += prefetch_window) { posix_fadvise(fd, offset, prefetch_window, POSIX_FADV_WILLNEED); async_read_into_buffer_pool(fd, offset, prefetch_window); // throttle by tracking outstanding prefetch count } When you use O_DIRECT, prefetch reads go straight into engine buffers (no double cache), and you control exactly which pages consume DRAM. Practical Application: Instrumentation, Tuning, and Operational Checklists Below are concrete checklists and protocols you can implement immediately to improve observability and behavior. Design-time checklist - Define your memory budget for the buffer pool as a clear fraction of host RAM; reserve headroom for OS and JVM/native heaps. - Choose IO model: O_DIRECT + engine-managed prefetch or kernel caching + hints (posix_fadvise). Document alignment and page-size assumptions. - Pick an eviction policy and concurrency model: sharded CLOCK is a pragmatic starting point for high-concurrency systems. - Define dirty-page targets and checkpoint cadence (e.g., aim to keep steady-state dirty ratio within a band that your storage can absorb). Implementation checklist - Implement atomic pin() / unpin() APIs and a non-blocking try_pin(). - Keep per-page metadata small: pin_count, refbit, dirty, page_lsn, flushed_lsn. - Expose counters: evictions, failed_evictions, pinned_waits, flushes_by_eviction, background_flush_bytes/sec, checkpoint_duration_ms. - Implement a background flusher and a separate checkpointer with budget-based throttling. - Add instrumentation hooks into the WAL path so the flusher can reason about the LSN frontier. Operational checklist (metrics and commands) - Buffer hit ratio: target depends on workload (OLTP point lookups expect high hit ratios); track hit_count / (hit_count + miss_count). - Dirty ratio: dirty_pages / total_pages — use this to trigger background flushing or to adjust target rates. - Checkpoint metrics: measure checkpoint write time, bytes written, and device utilization during checkpoints. Postgres exposes pg_stat_bgwriter with checkpoints_timed, checkpoints_req, buffers_checkpoint, buffers_clean, checkpoint_write_time. Querying those helps tie spikes to checkpoint activity. - Pin contention: pinned_wait_count and median/99th pin wait latency tell if long-lived pins are blocking eviction. - I/O saturation signals: iowait, device service time, queue depth, and iostat -x metrics — correlate these with buffers_clean and checkpoint writes. - Engine-specific: InnoDB status for buffer pool and checkpoint activity (SHOW ENGINE INNODB STATUS) and RocksDB cache stats exposed via its statistics interface. Quick runbook for a recurring p99 spike that looks storage-related 1. Confirm a spike corresponds to increased checkpoint_write_time or buffers_checkpoint (DB metric). 2. Check device metrics (iostat, nvme-cli, cloud volume metrics) for increased latency or throughput saturation. 3. Inspect eviction counters to find whether many evictions are failing due to pinned/dirty pages. 4. If the dirty ratio bursts, increase background flusher throughput or reduce checkpoint burst size by spreading writes (change checkpoint throttle/budget). 5. If kernel page cache and buffer pool are both large, evaluate switching to O_DIRECT or reduce one of the caches to free RAM. Small examples — Postgres queries and OS tools -- Postgres: useful bgwriter/checkpoint metrics SELECT checkpoints_timed, checkpoints_req, buffers_checkpoint, buffers_clean, maxwritten_clean, buffers_backend, buffers_alloc FROM pg_stat_bgwriter; OS tools: iostat -x, iotop -o, vmstat 1, perf record, bpftrace for pin wait traces. Testing and validation - Synthesize workloads where the working set is (a) smaller than buffer pool, (b) slightly larger, (c) massively larger. Observe hit-rate, evictions/sec, and p99 latency to confirm behavior. - Run crash-and-recover tests that kill the process during checkpoints and validate recovery time and WAL replay semantics. - Measure how prefetch affects hit rate and eviction churn — track prefetch admission vs prefetch evictions. Sources: Latency numbers every programmer should know - Reference for order-of-magnitude latency comparisons between CPU cache, DRAM, NVMe, and spinning disks used to explain why buffer pools matter. PostgreSQL: Shared Buffer (storage buffer) and bgwriter/checkpoint metrics - Descriptions of PostgreSQL shared buffers, bgwriter, and associated monitoring counters referenced for buffer pool semantics and instrumentation. PostgreSQL: Write-Ahead Logging (WAL) - WAL ordering, checkpoints, and group-commit behavior used to justify flush ordering and checkpointer design. posix_fadvise(2) — Linux manual page - Documentation for file access pattern hints and their semantics (used for prefetch/read-ahead discussion). MySQL / InnoDB Buffer Pool - InnoDB buffer pool design and flushing behavior cited when describing background flush and dirty ratio strategies. RocksDB — Memory Usage (Wiki) - Notes on LSM-engine memory components (memtable, block cache) and how memory choices affect compaction and I/O patterns. readahead(2) — Linux manual page - System call reference for triggering kernel read-ahead used in prefetch strategy discussion. Page replacement algorithm — Wikipedia - Survey of LRU, CLOCK, LRU-K, LIRS, and related algorithms used to compare eviction strategies and properties. open(2) — Linux manual page (O_DIRECT) - O_DIRECT semantics and considerations for bypassing the kernel page cache referenced in the kernel-bypass discussion. A robust buffer pool is an exercise in orchestration: pin correctly, evict cheaply, flush in a controlled fashion, and let prefetching be a gentle helper rather than a memory robber. Follow the instrumentation checklist, codify the invariants (pin_count, page_lsn, flushed_lsn, dirty), and the storage layer will stop being the wildcard that spoils otherwise predictable systems.
trying to build an append-only log, with failure modes, and recovery methods references: - github.com/google/leveldb/blob/main/doc/log_forma… - sqlite.org - github.com/facebook/rocksdb/wiki/WAL-Recovery-Mod… t.co