delta-io

Auto CDF Moved the CDC Tax From Writes to Reads: Delta CDF vs Auto CDF vs Spark 4.2 CHANGES On September 1, 2026, Databricks made automatic change data feed (Auto CDF) generally available. The release note says it "removes write-time overhead and makes MERGE and UPDATE operations about 15% faster on tables that you query for changes." Seven weeks earlier, Apache Spark 4.2.0 (July 14) added a SQL CHANGES clause, so reading row-level changes is now a first-class part of the engine, not a Delta-only option. Most of the takes I have seen stop at "free CDC, faster MERGE, turn it on." My thesis is narrower: - Auto CDF does not remove the cost of change data. It moves it from the writer, who pays once per commit, to every reader, who pays on every read. - Its prerequisite, row tracking, has a write-time cost of its own that nobody puts on the slide. - Spark 4.2's CHANGES is a contract, not an implementation. Whether it works depends on your connector, and on open-source Delta 4.4.1 it does not work out of the box. So the real question is not "legacy or auto." It is: how many times is each commit read, and how much data does each commit rewrite? Three ways to get the changes | | Legacy CDF (write time) | Auto CDF (read time) | Spark 4.2 CHANGES | |---|---|---|---| | Where the work happens | Writer writes change files next to the data | Reader computes changes from row tracking metadata | Connector supplies a changelog, Spark post-processes | | Opt in | delta.enableChangeDataFeed = true per table | Automatic when requirements are met | Catalog must implement loadChangelog | | Who can read it | Any Delta reader that implements the cdc action of the protocol | Databricks readers only (DBR 19+) | Any Spark 4.2 client, if the connector supports it | | Formats | Delta only | Delta with row tracking, Iceberg v3 | Whatever the connector exposes | | Read API | readChangeFeed, table_changes() | Same APIs | SELECT ... CHANGES FROM VERSION, .changes() | FACT (Databricks docs): legacy CDF "materializes changes during table writes", Auto CDF "computes changes during table reads using row lineage metadata", and the two cannot be used together on one table. Migration is a single statement: ALTER TABLE t UNSET TBLPROPERTIES ('delta.enableChangeDataFeed'). The mechanism, in two paragraphs Legacy CDF. The Delta protocol has a cdc action. On UPDATE, DELETE and MERGE the writer puts the changed rows, with _change_type, into files under _change_data/. The protocol tells readers to use those files when they exist and skip the add and remove actions for that version. Insert-only commits and full partition deletes produce no change files, because the log already says what happened. So the reader of a commit touches roughly the changed rows, nothing more. Auto CDF. Row tracking gives each row a stable row ID and a row commit version. The Delta protocol explains how: for inserts, the IDs are derived from a baseRowId per file plus the row position, which costs almost nothing. But when a writer copies a row into a new file (the "copy" in copy-on-write), it must preserve the stable ID, so it materializes the row ID and commit version into hidden columns in the data file. At read time, the engine compares row IDs and commit versions across the files a commit removed and added to work out which rows were inserted, updated or deleted. That is real computation over real files, and it happens on every read. What I measured I cannot run Databricks' Auto CDF outside Databricks, and I am not going to pretend otherwise. What I can run is open-source Delta Lake 4.4.1 on Spark 4.2.0, which has both halves of the mechanism: legacy CDF, and row tracking with _metadata.row_id and _metadata.row_commit_version. Setup: - A 10M-row Delta table (about 507 MB of Parquet), deletion vectors off, so MERGE is pure copy-on-write. - Each round: a MERGE that updates about 100,000 rows (1%, keys hashed across the whole id range) and inserts 10,000 new ones. 5 rounds per variant, medians reported, after a discarded warm-up. - Three variants: plain, legacy CDF, row tracking only. - Read side: read the changes of the last commit. For legacy CDF that is readChangeFeed for one version. For row tracking it is a reconstruction I wrote: read the files the commit removed (at version v-1) and added (at version v), and join on row_id and row_commit_version. I checked the reconstruction against legacy CDF on a table with both features enabled: for a MERGE (200 inserts, 1,986 pre-images, 1,986 post-images) and for a DELETE (100 deletes) the counts matched exactly. The core of the reconstruction: after = (spark.read.format("delta").option("versionAsOf", v).load(path) .select("*", F.col("_metadata.row_id").alias("rid"), F.col("_metadata.row_commit_version").alias("rcv"), F.col("_metadata.file_path").alias("fp")) .where(F.col("fp").isin(added_files))) changed = after.where(F.col("rcv") == v) # inserts + post-images # pre-images: rows in removed files whose rid appears in `changed` # deletes: rows in removed files whose rid appears nowhere in `after` Results, 10M rows, scattered 1% upsert, median of 5 rounds: | Variant | MERGE time | Bytes written per MERGE | Change files per MERGE | Read one commit's changes | |---|---|---|---|---| | Plain | 6.03 s (4.75 to 7.69) | 505.8 MB | none | n/a | | Legacy CDF | 6.52 s (6.19 to 7.10) | 506.0 MB | 10.8 MB | 0.21 s | | Row tracking only | 6.16 s (5.92 to 7.69) | 547.6 MB | none | 4.73 s (my reconstruction) | A second, partial run of the plain and legacy CDF variants gave 5.64 s and 6.89 s for the MERGE, and 0.18 s for the CDF read. What the numbers say, and what they do not 1. The write tax of legacy CDF is real but small here, and smaller than my noise. Legacy CDF was slower in both runs (6.52 vs 6.03 s, and 6.89 vs 5.64 s), but the spread inside one variant (4.75 to 7.69 s) is wider than the gap. Databricks measured about 15% on its runtime; my numbers are compatible with that and cannot confirm it. The byte count is solid: 10.8 MB of change files per MERGE, about 2% of what the MERGE rewrote anyway. 2. Row tracking wrote 8% more data on every rewrite. 547.6 MB versus 505.8 MB per MERGE, because every copied row now carries its stable row ID and commit version in hidden columns. That is about four times the size of the CDF change files in this workload. This is the part of the "free" read-time CDF that you pay at write time. 3. Reading changes at read time costs what the commit rewrote, not what it changed. The 1% scattered upsert touched every file, so the commit removed and re-added the whole table. Legacy CDF read 10.8 MB of change files in 0.21 s. My reconstruction had to scan both sides of a 500 MB rewrite to find 100,000 changed rows: 4.73 s, about 22 times slower. On a smaller test (200,000 rows, 8 files) the same scattered upsert rewrote 8 of 8 files, while an upsert clustered on the newest ids rewrote 1 of 8. The layout of your updates decides how expensive read-time CDF is. What I cannot claim: that Databricks' implementation is as slow as mine. It is very likely smarter, and with deletion vectors a MERGE writes only the changed rows to new files and marks the old ones, which shrinks the set of rows a reader has to compare. My test is the copy-on-write case, which is the worst case for read-time CDF. The mechanism still holds: the read-time cost scales with what a commit touched, and you pay it once per reader per read. 4. CHANGES on open-source Delta 4.4.1 fails by default. With the stock DeltaCatalog, all three of SELECT * FROM t CHANGES FROM VERSION 1, the path-based form, and spark.read.changes("t") returned: [UNSUPPORTED_FEATURE.CHANGE_DATA_CAPTURE] The feature is not supported: Catalog spark_catalog does not support Change Data Capture (CDC). SQLSTATE: 0A000 That matches the source. Delta PR #6794 (merged May 2026) adds catalog-driven read-time CDF behind an internal flag that is off by default, requires row tracking, and leaves the streaming side of the new path as a follow-up. I did not test the flag; it is internal and I would not run production on it. Spark 4.2 defines the shape of change data. Your connector decides whether you get any. Gotchas to check before you run UNSET All FACT, from the Databricks and Spark docs: - Auto CDF is not supported on tables with row filters or column masks. If your sensitive tables are governed with ABAC or masks, they are exactly the ones that cannot move. - Only Databricks readers can query Auto CDF on Delta, and external Iceberg clients cannot query it on Iceberg v3. Legacy CDF lives in the open Delta protocol. If anything outside Databricks reads your change feed, check before migrating. - It needs Databricks Runtime 19 or above and a table registered in Unity Catalog with row tracking enabled (or Iceberg v3). - Enabling row tracking on an existing table is not free. It assigns IDs to every existing row, can create several new table versions, and concurrent writers fail with MetadataChangedException while it runs. Pause streaming writers first. It also upgrades the writer protocol, so older external Delta clients that do not support those features can no longer write to the table. - Multi-statement transactions: Auto CDF is not supported if the source table was modified during the transaction. - _metadata.row_id and row_commit_version cannot be read while reading the change feed. If your downstream logic wants both, plan for it. - Spark 4.2 streaming netChanges holds back the latest commit. Rows touched only in the latest commit are not emitted until a later commit moves the watermark, and streaming net changes can differ from batch net changes over the same range. Fine for a pipeline, surprising for a low-latency consumer. Decision table OPINION, built on the mechanism and my numbers: | Your table looks like | Better choice | Why | |---|---|---| | Write-heavy, changes read rarely or by one pipeline | Auto CDF | You stop paying on every write for a feed few people read | | One commit read by many consumers, often | Legacy CDF, or Auto CDF plus one stream that lands changes in a table | Read-time cost multiplies by readers times reads | | Scattered updates on a copy-on-write table | Legacy CDF, or enable deletion vectors and measure | Each commit rewrites most files, so read-time diffing is expensive | | Row filters or column masks | Legacy CDF | Auto CDF is not supported there | | Change feed read outside Databricks | Legacy CDF | Auto CDF is Databricks-readers only | | Iceberg v3 tables on Databricks | Auto CDF | Legacy CDF does not support Iceberg | What I would do this week 1. Inventory who reads each change feed. Count consumers and their trigger intervals per table. That count is the multiplier on read-time cost, and nobody has it written down. 2. Look at the update layout. In table history, compare numTargetRowsUpdated against files rewritten for your MERGE operations. A 1% update that rewrites 100% of files is a bad candidate for read-time CDF. 3. Migrate the boring tables first: write-heavy, one reader, no masks, nothing outside Databricks. Keep legacy CDF on the hot, many-reader tables until you have measured both on your own runtime. 4. If you fan out, fan out once. One stream reads the change feed and writes a changes table; the other consumers read that. The Databricks docs already show this archive pattern for permanent history. Auto CDF is a good default for most tables, and a quiet bill for the ones read by five pipelines. CDC is still a cost. The release moved it to the reader, and the bill now goes up with every pipeline that reads the feed. Hardware & versions > Box: 8 vCPU Intel Xeon VM, 16 GB RAM, Linux 6.12, OpenJDK 21.0.12. > PySpark 4.2.0, delta-spark 4.4.1 (delta-spark_4.2_2.13), local[8], driver memory 8 GB (5 GB in the partial second run), 16 shuffle partitions. > 10M rows, 5 columns, about 507 MB Parquet, deletion vectors off. 5 rounds per variant, medians, warm-up discarded. > Same box also hosts another long-running benchmark of mine. During the main run its runner was paused waiting for an idle host, but its services stayed in memory. The partial second run overlapped that benchmark's active phase, so I quote it only as a cross-check. Treat timings as indicative. The byte counts and the correctness check are deterministic. > Measured October 26, 2026. Script, raw logs and pinned versions: cdf_bench gist. Sources - Databricks September 2026 release notes, "Automatic change data feed is now generally available" (Sep 1, 2026): docs.databricks.com - Databricks docs, Use change data feed (Auto CDF requirements, migration, limitations): docs.databricks.com - Azure Databricks docs, same page: learn.microsoft.com - Databricks docs, Row tracking: docs.databricks.com - Delta Lake protocol v4.4.1 (Add CDC File, Row Tracking, materialized Row IDs): github.com/delta-io/delta/blob/v4.4.1/PROTOCOL.md - Delta Lake docs, Change data feed: docs.delta.io - Delta Lake docs, Row tracking: docs.delta.io - Delta Lake 4.4.0 release (Spark 4.2 support): github.com/delta-io/delta/releases/tag/v4.4.0 - Delta PR #6794, Add Changelog support: delta#6794 - Apache Spark 4.2.0 release notes (Change Data Capture, SPARK-55668): spark.apache.org - Apache Spark news, Spark 4.2.0 released (July 14, 2026): spark.apache.org - Spark PR, SPARK-55948 DSv2 CDC connector API and SQL CHANGES clause: spark#54738 - Spark 4.2.0 API docs, Changelog: spark.apache.org - Spark 4.2.0 API docs, DataStreamReader.changes (streaming netChanges behavior): spark.apache.org - My measurements: OSS Delta 4.4.1 on Spark 4.2.0, October 26, 2026 (setup above). I post shorter breakdowns daily on LinkedIn.

При работе с Delta Lake оптимизируем через кластеризацию, а не через сегментирование Сегментирование в стиле Hive применяется для оптимизации физического диска и более качественной компоновки данных, о чём подробнее рассказано в статье Organizing Data: Partition by Disk to Hive-style Partitioning. Но при работе с облачными объектными хранилищами сегментирование — не лучшая стратегия, и нам следует переосмыслить наш подход к организации данных. В этой статье обсудим, как путём пропуска данных (data skipping) и использования Z-порядка сократить объём тех данных, которые приходится просматривать. В результате можно значительно повысить производительность при обработке запросов и удешевить эти операции за счёт использования приёмов кластеризации. Повышение производительности путём пропуска данных Традиционно при запрашивании информации из озера данных движок запросов сначала выстраивает список релевантных файлов, а затем запрашивает их. В таких облачных объектных хранилищах как S3 подобное действие listFrom получается очень ресурсозатратным. Когда данные хранятся в форматах из категории «lakehouse», например, Delta Lake, удаётся совместить наилучшие черты озёр данных и хранилищ данных. Обеспечивается высокая надёжность данных, как в хранилище, и соответствие набору ACID, тогда как гибкость и масштабируемость при работе такая, как будто мы работаем с озером данных. В Delta Lake этого удаётся добиться благодаря использованию журнала транзакций, как показано на следующей схеме. Здесь клиент rust-lang запрашивает таблицу Delta.  Клиент на Rust запрашивает таблицу Delta Lake через журнал транзакций Delta Delta Lake предоставляет движок для работы с запросами (напр., Rust Lang содержащийся в пакете delta-rs), прописывая точные пути к тем файлам, которые слагают таблицу. Следовательно, при работе с Delta Lake удаётся в значительной степени устранить проблемы с производительностью, возникающие при листинге файлов (напр., listFrom). Клиент Rust получает меньше данных (т.e., меньше файлов), сокращение достигается путём пропуска данных в журнале транзакций Delta. Обсудим, как Delta повышает производительность запросов путём пропуска данных, опираясь на статистику столбцов (минимальные и максимальные значения), сохранённую в журнале Delta. Например, движок запросов отсеет через фильтр все файлы данных, не относящиеся к 2023 году (*.parquet), исходя из максимальных и минимальных значений в журнале транзакций. Следовательно, пропуская данные таким образом, движок будет читать меньше информации, а значит — работать быстрее. Свёртка многомерных точек до одномерных  значений Пропуск данных получается ещё эффективнее, если кластеризовать данные так, чтобы диапазоны значений min-max получались узкими и не пересекались. Например, вот какова структура файла данных, показанного на предыдущей картинке. Конфигурация кластеров в предыдущем файле 1.parquet {min: 2023, max: 2023} 2.parquet {min: 1999, max: 2020} 3.parquet {min: 2009, max: 2018}sourcecraft.dev Если в данном примере мы запросим данные за 2010 год, то нам потребуется прочитать как 2.parquet, так и 3.parquet. Напротив, было бы более оправданно выполнить конфигурацию по следующему принципу. Оптимальная конфигурация кластеров данных по годам 1.parquet {min: 2019, max: 2023} 2.parquet {min: 2014, max: 2018} 3.parquet {min: 2009, max: 2013}sourcecraft.dev Этот упрощённый пример отлично работает с одним столбцом. Но всё значительно усложняется, когда требуется выполнить кластеризацию по множеству столбцов. Такая проблема распространена при работе с реляционными базами данных, когда мы пытаемся присваивать одному файлу множество значений из столбцов, по которым обычно запрашиваются/фильтруются/группируются данные. С математической точки зрения мы хотим сохранить локальность данных, отображая многомерные точки на одномерные данные. Следовательно, хотим использовать поддерживающие такую локальность кривые Пеано. Наиболее распространенными из них являются кривые Мортона и кривые Гильберта. Отличный пост с разбором этой темы написал Адриан Ионеску. Он называется Processing Petabytes of Data in Seconds with Databricks Delta. Азы кластеризации на кривых Мортона Кривая Мортона — это сохраняющая локальность разновидность кривых Пеано. Кривая Мортона позволяет, применяя фильтры, пропускать данные во множестве столбцов. Концепция такова: если любой прямоугольник размером 4×1 при запрашивании данных будет трактоваться как отдельный файл, то движок будет последовательно обрабатывать файл за файлом в процессе линейных проходов. Можно запрашивать данные не линейно, а организовать их в виде квадрантов и посещать каждый квадрант, обходя его в таком порядке, в котором рисуется буква z. Но, если организовать данные в таком Z-порядке, то концептуально каждый файл будет представлять собой квадрант 2×2. Файл просматривается в порядке “Z” (0, 1, 2, 3), то есть, рекурсивно. Допустим, у вас есть следующий SQL-запрос, в котором данные представляются в двух измерениях, то есть, откладываются по осям (x, y). Если рассматривать данные в двух измерениях, то оптимальный путь строится так: читаем значения, расположенные лишь по x=2, y = 2. В зелёных кружках с жёлтым фоном — те данные, которые мы хотим прочитать. Красные данные мы собираемся пропустить. Напомню, что в соответствии с Z-порядком читается квадрат 2×2 (прерывистая линия), и таким квадратом представлен каждый файл.   Линейное сканирование Но при запрашивании данных в линейном порядке каждому файлу соответствует прямоугольник 4×1 (прерывистая линия). При линейном сканировании приходится прочитать девять (9) файлов (жёлтый фон, прямоугольники, очерченные прерывистой линией). Также получается 21 ложноположительный результат, таким результатам соответствуют красные кружки на жёлтом фоне. При линейном сканировании получается 9 операций чтения файлов и 21 ложноположительный результат Сканирование в Z-порядке С другой стороны, кластеризация в Z-порядке позволяет сократить количество как файлов, так и ложноположительных результатов. Каждый файл представлен квадратом 2×2 (прерывистая линия) и сканируется так, как рисуется буква Z (светло-голубая линия). Но связь между точками в каждом Z (т.e., в каждом файле) меняется и зависит от конкретных данных.   При кластеризации данных в Z-порядке мы читаем всего 7 файлов, и остаётся 13 ложноположительных результатов В данном примере движок запросов сканирует всего 7 файлов (а не 9). Кроме того, вместо 21 остаётся 13 ложноположительных результатов.  Обсуждение Выше мы рассмотрели упрощённый двумерный пример (одно измерение — один столбец). Но кластеризация такого типа применима и со множеством столбцов. Здесь действует такое эмпирическое правило: данная техника наиболее эффективна при работе с 5-8 столбцами. Мы можем не ограничиваться несколькими столбцами, как при сегментировании, а приобрести гораздо более значительную гибкость, если будем кластеризовать данные для обхода в Z-порядке.

If a maintainer left tomorrow, who would know their code? I asked a graph --- title: "If a maintainer left tomorrow, who would know their code? I asked a graph" published: true description: "What does a project lose when a maintainer leaves? A FalkorDB graph of Delta Lake's public history, a measurement mistake, and evidence-backed handovers." tags: falkordb, graphdatabase, opensource, ai ai_disclosure_level: some_ai cover_image: raw.githubusercontent.com --- Bus Factor turns two years of public Delta Lake history into a FalkorDB graph of people, code, reviews and design decisions. Here is what it found, the measurement I got wrong, and why code reviews mattered more than I expected. About the cover: Each square is a Delta Lake code area with enough recent activity to measure; each line joins it to its most active author in the last 12 months. Data: public delta-io/delta history, queried on 18 October 2026. The repository remembers everything except who to ask Picture a Friday message in an open-source project's chat: one of the most active maintainers is moving on. Technically, nothing is lost. Every commit, pull request and design proposal is still there. And yet on Monday nobody can quite answer the questions that matter. Which parts of the code did only that person work on recently? Who else has at least seen those changes? Which design decisions did they help shape, and is anyone left who took part? If someone has to take over, where should they start reading, and what should they ask before the person who knows is gone? Engineers have a blunt name for this risk: the bus factor, the number of people who would have to disappear before part of a project has nobody left who knows it. I measure it per code area: the fewest people whose commits in the last 12 months add up to at least half of that area's commits. A bus factor of 1 means one person wrote at least half of what changed there recently. I built Bus Factor for the WeMakeDevs × FalkorDB Graph Hacks hackathon to answer one question about a real project, Delta Lake: what does the project lose if this maintainer leaves? The answer is spread across four places Each public source holds one piece. Git history says who changed which files. GitHub says which commits belong to which pull request, and who reviewed it. The protocol_rfcs folder lists 19 design decisions (RFCs), each with a tracking issue. CODEOWNERS says who is officially responsible for what. None of them on its own answers "who else knows this code?". Take flink/inttest, the integration tests of Delta Lake's Flink connector. In the last 12 months, Hao Jiang (@harperjiang) is the only person with two or more commits there. Count commits and Hao Jiang looks like the only person who knows it. Follow one of those commits into its pull request, and the picture changes. !A diagram of seven boxes. Person harperjiang AUTHORED commit fa642d6, which TOUCHED the file CatalogManagedTableIntTest.java, which is IN_COMPONENT code area flink/inttest. The commit is PART_OF pull request 6188. Person scottsand-db REVIEWED pull request 6188. A dashed IMPLEMENTS arrow, labelled title match, low confidence, runs from the pull request to the design decision Catalog-Managed Tables. A highlighted path answers: who else has seen the code in flink/inttest? Seven real nodes from the graph. Following code area → file → commit → pull request → reviewer finds scottsand-db, who reviewed the change but has no recent commits in the area. That is all a knowledge graph is: things (nodes) joined by named, directed relationships, so that a question becomes a path to follow. FalkorDB stores the graph and answers queries written in Cypher, a language whose patterns look like the arrows in the diagram. Here is the pattern Bus Factor uses to find who reviewed a code area (from 06_departure_impact.cypher, wrapped to fit): OPTIONAL MATCH (reviewer:Person:Active)-[review:REVIEWED]->(pr:PR) <-[:PART_OF]-(rc:Commit)-[:TOUCHED]->(rf:File) -[:IN_COMPONENT]->(component) Read it left to right: an active person reviewed a pull request; a commit is part of it; the commit touched a file; the file belongs to this code area. SQL could do the same lookup with a handful of joins. What the graph gave me is that every answer can carry the path that produced it, and that the same relationships also feed graph algorithms, separate memory graphs and a copy-and-simulate step, all of which appear below. What I built Bus Factor loads two years of public delta-io/delta activity: 2,240 commits, 3,602 pull requests with their reviews, the 19 RFC decisions and their tracking issues, CODEOWNERS, and the 4,427 files present at the latest commit. Files are grouped into code areas, packages inside modules such as flink/inttest. The result is a graph of 11,033 nodes and 39,869 relationships, rebuilt in about 11 seconds from a 1.7 MB snapshot in the repository, so nobody needs a GitHub token to run it. The app opens on the measurement and a choice: !The Bus Factor landing view. A headline reads: 25 of 69 active code areas in Delta Lake have a bus factor of 1. Below it are a search box labelled Choose a maintainer and three suggested questions. The landing view. Pick a maintainer from a list ranked by recent commits, or start from a suggested question. Underneath, data flows down into FalkorDB, and answers come back up through a fixed set of queries. !An architecture diagram with five layers. Public sources (git history, GitHub GraphQL API, RFC table, CODEOWNERS) feed Python parsers and a public snapshot. Load, link and derive steps run Cypher files into FalkorDB, which holds separate graphs: delta for repository facts, mem_team for shared notes, one mem_v graph per visitor, and temporary whatif copies. A Python tool layer runs a fixed catalogue of read-only queries, checks names first and joins memory in Python. A web app calls the tools directly; an agent running Claude Sonnet 5.5 picks tools but never writes Cypher. How the pieces fit. Repository facts, shared notes and each visitor's history live in separate FalkorDB graphs. Python joins them by a stable key, because Cypher cannot traverse between graphs. Two decisions shaped everything else. First, no Cypher is generated at runtime. Each capability is a tool backed by one .cypher file whose header says what it answers and why it is shaped that way. The departure view and handover packets call those tools directly, with no language model involved, and read-only tools run through GRAPH.RO_QUERY. Second, the chat agent (Claude Sonnet 5.5) only picks a tool and fills in its parameters. Every login or code area it names is checked against the graph first, so a typo such as harper-jiang comes back as "not in the graph" rather than an empty result that reads like "nothing at risk". Afterwards, the names in its reply are compared with the rows the tools returned. My first headline was wrong When I first put a number on the landing page, it said that 42 of 87 active code areas had a bus factor of 1 "in the last 12 months". It was a good headline. It was also measuring something else. The day before submission, I asked a second model (Codex on GPT-6-Astra) to review the project the way a sceptical judge would, and had Claude Code check each finding against the code and the live graph. The findings held up, and the two most important were about how risk was counted. The first was a mix-up of time windows. The loader derives a KNOWS relationship between each person and each area, holding their commit count over the full two loaded years and the date of their latest commit. The bus-factor query kept people whose latest commit fell inside the 12-month window, then added up their two-year counts. Someone with eight old commits and one recent one counted as nine recent commits. !A two-part figure titled Same label, two calculations. The top part shows one person's commits on a two-year timeline: eight old commits and one inside the last 12 months. The old query counted all nine; the corrected query counts one, which is below the two-commit bar. The bottom part compares the queries: commits counted, the threshold for the person leaving, and the risk label. The old query reports 42 of 87 areas with a bus factor of 1, the corrected query 25 of 69. Same label, two calculations. Re-running both queries on the same graph on 18 October still gives 42 of 87 and 25 of 69: the project didn't change, the measurement did. The second was a missing threshold. People who would remain needed two commits in an area to count, but the person leaving needed only one. When I simulated Murali Ramanujam (@murali-db) leaving, five areas came back "orphaned", and four of them rested on a single commit. The fix counts exactly what the label says, straight from the commits (from 05b_bus_factor_component.cypher, wrapped to fit): MATCH (p:Person)-[:AUTHORED]->(commit:Commit)-[:TOUCHED]->(f:File) -[:IN_COMPONENT]->(component:Component) WHERE commit.ts > $since AND coalesce(p.bot, false) = false AND coalesce(commit.bulk, false) = false AND coalesce(f.exists, true) = true WITH component, p, count(DISTINCT commit) AS c WHERE c >= $minCommits count(DISTINCT commit) matters as much as the date filter: a commit that touches 30 files in an area is one contribution, not 30. Bulk commits that touch more than 100 files, and files deleted since, are left out of expertise. The person leaving now needs the same two commits as everyone else. The corrected headline is 25 of 69. The project didn't get safer in an afternoon; the measurement moved closer to its label. It was the most useful thing that happened during the build, because it changed the product's vocabulary too. The review asked a better question: if authorship is one signal, who reviewed that code? Across the 15 most active people, the 19 areas where one of them was the only recent author each had between 2 and 11 active reviewers. "Orphaned" was the wrong word. The label became NO OTHER AUTHOR, and reviewers are listed beside it, never folded into it, because writing code and reviewing it are different kinds of evidence. One maintainer, one area, one handover To show the whole flow I'll use Hao Jiang, because the data makes a clear example, not because anything suggests they are leaving. Choosing them in the app asks the graph what a departure would expose, and the app answers in one sentence: > If Hao Jiang left: 3 code areas where they made 2+ commits in the last 12 months would have no other recent author (3 of them were reviewed by other active people), 5 would have one other recent author, 81 current files would have no other recent editor, and 0 people would lose their link to the main collaboration network. !The departure view for Hao Jiang. On the left, three cards for flink/kernel, flink/inttest and flink/(files), each labelled NO OTHER AUTHOR and listing its reviewers with pull-request counts. On the right, an evidence graph centred on harperjiang, with orange code areas, blue people, purple pull requests and dotted review edges. The three Flink areas with no other recent author each have five or six active reviewers. The evidence graph draws the relationships the cards count; dotted lines are reviews. The last number comes from graph algorithms rather than a pattern match. A derived WORKS_WITH relationship links people who recently edited the same files. Bus Factor copies the graph with GRAPH.COPY, ranks the person with algo.betweenness, removes their Active label in the copy, recomputes collaboration groups with algo.WCC, checks who dropped out of the largest group, and deletes the copy. For Hao Jiang nobody is cut off. For Murali Ramanujam, who ranks first by betweenness, three people would lose their link to the main collaboration group. On the flink/inttest card, Draft handover builds a packet from four queries in about a tenth of a second. Candidates are everyone who recently wrote code in the area or reviewed a pull request that touched it. Each kind of evidence keeps its own column, and the order is a plain sort, not a weighted score: first, how many of the leaving person's pull requests here they reviewed, because that reviewer has already seen these exact changes; then shared files, their own commits, and reviews in the area. !A diagram linking harperjiang and the code area flink/inttest to three pull requests: 6188 (7 files here), 6192 (3 files) and 6933 (1 file). Teal dotted lines connect the pull requests to their reviewers: huan233usc, tdas and scottsand-db reviewed 6188; scottsand-db reviewed 6192; kaiqijinwow and timothyw553 reviewed 6933. A dashed title-match line joins 6188 to the decision Catalog-Managed Tables. The suggested successor is scottsand-db: reviewed 2 of Hao Jiang's 3 pull requests here, no commits of their own here yet, took part in Catalog-Managed Tables. The relationships behind the suggestion. The link from pull request #6188 to Catalog-Managed Tables is dashed because it comes from a keyword in the title, not an explicit reference. !The top of the handover packet for flink/inttest. Suggested successor: scottsand-db, reviewed 2 of Hao Jiang's 3 PRs here, no commits of their own here yet, took part in Catalog-Managed Tables. Below is a candidate table with columns for reviews of Hao Jiang's PRs, shared files, own commits, PRs reviewed here and shared decisions, then the files scottsand-db has not touched and three pull requests to read first. The top of the packet: the suggested successor, the evidence table, the files the successor hasn't touched yet, and the pull requests to read first. Every clause of the suggestion comes from a column, and the sentence is assembled from the row in Python, not written by a model. Keeping "no commits of their own here yet" next to the recommendation is deliberate: a reviewer is not an author, and the packet says so. !The lower part of the handover packet. Under Decisions to preserve, Catalog-Managed Tables (accepted) shows 0 explicitly linked PRs and 1 title match, links to the RFC document and tracking issue 4381, other active participants, and a quoted reason explaining why the earlier Coordinated Commits design was rejected. Below are five questions for Hao Jiang. Further down the packet: the decision to preserve, with its reason quoted from tracking issue #4381, and questions to ask before Hao Jiang leaves. Decisions needed the same honesty. Of 283 links between pull requests and RFC decisions, only 53 are explicit: the pull request body cites the RFC's tracking issue, or the pull request edits the RFC document. The other 230 come from a keyword in the title, matched on whole words so that "ICT" doesn't match "conflict". So the packet says "1 title matches (keyword guesses)" and asks Hao Jiang whether PR #6188 really belonged to Catalog-Managed Tables. Why the older Managed Commits design (tracked in issue #2598 as Coordinated Commits) was rejected is quoted word for word from issue #4381 at load time, and PR #4382 is recorded as the evidence that one replaced the other. None of this is a staffing decision. It is a starting point for a conversation: here is who has seen the changes, here is what they haven't seen, and here is what to ask while the person who knows is still around. When a note changes the answer A repository cannot know whether the suggested person is free. A team learns that in conversation, and a useful tool should remember it without pretending it is a fact like a commit. So Bus Factor keeps notes in separate graphs: mem_team for notes everyone shares, and one mem_v_<visitor> graph per browser for that visitor's questions and corrections. For this article I saved the packet, then typed a note saying scottsand-db is "on leave until January". I made that note up for the demo; it says nothing about anyone's real plans. !A three-column diagram. Repository facts in the delta graph list five candidates and are never edited by notes. A dashed amber team note in the mem_team graph says scottsand-db is unavailable, on leave until January, and is marked made up for the demo. A box explains that the note's key person:scottsand-db is matched to the Person node in Python and passed to the query as an exclusion. The derived suggestion changes from scottsand-db before the note to kaiqijinwow after it, with scottsand-db listed as skipped. Repository facts, an unverified note and the derived suggestion stay separate. They meet only when a packet is built. Cypher cannot traverse from one graph to another, so the join happens in Python. The note is stored against the key person:scottsand-db; the handover code collects everyone marked unavailable and passes their logins into the same query as $exclude (from 18_handover_candidates.cypher, shortened): WITH cand.login AS login, cand.login IN $exclude AS skipped, reviewedTheirPRs, sharedFiles, areaCommits, areaReviews ORDER BY skipped, reviewedTheirPRs DESC, sharedFiles DESC, areaCommits DESC, areaReviews DESC, login The repository graph is untouched, but the next packet is different: !The handover packet after the note. A highlighted line reads: What changed: The packet saved on 18 Oct 2026 suggested scottsand-db; now kaiqijinwow. Why: scottsand-db was marked unavailable on 18 Oct 2026: on leave until January (unverified note). The suggested successor is kaiqijinwow, with a candidate table and a line saying scottsand-db was skipped because of an unverified availability note. After the note, kaiqijinwow is suggested. The packet says what changed since the saved version, and lists scottsand-db as skipped because of an "unverified availability note". Because the note lives in shared memory, the chat agent sees it too. Asked to "Draft a handover plan for flink/inttest if harperjiang leaves", it called the same handover_plan tool and explained the change: !The Ask view. Under the question Draft a handover plan for flink/inttest if harperjiang leaves, the agent's answer suggests kaiqijinwow and says the evidence is thin, says Scott Sandre was skipped because of an unverified note in shared team memory, separates author risk from reviews, and lists the pull requests to read. The agent's answer. It calls the note unverified, says the evidence for the new suggestion is thin, which is true, and keeps author risk and reviews apart. !A close-up of the How I know section: 1 graph tool call, 1 name not found in the results. The tool Drafted a handover plan from graph evidence returned 1 row in 86.85 ms. Checked login harperjiang: found. Checked component flink/inttest: found. "How I know" sits under every answer: the tool that ran, the name checks before it, and timings. In this run, the reply check flagged one name in the answer that did not appear in the returned rows. What building it taught me A query can be wrong and still look convincing. 42 of 87 was plausible and came from real data. What caught it was reading the query against its label, and keeping every query in its own commented file is what made that take minutes. Separate signals deserve separate columns. It is tempting to blend commits, reviews and shared files into one expertise score. Every time I kept them apart, and labelled how sure each link was, the output got easier to check and to argue with. The same goes for memory: a note can change a suggestion, never a fact. Graph databases have their own performance traps. While planning a deployment on a small free instance, I measured the snapshot load and found it peaked at 681 MB, for a graph that occupies 12 MB. The commit loader merges and deletes, so FalkorDB holds every row of a batch in memory before writing, and each file row carried a copy of its commit's entire file list. Dropping that variable after the unwind brought the peak down to 51 MB, with an identical graph. How it was built, and by whom This is a solo entry built with AI coding assistants, and the split matters. Before the hackathon, I planned the project with Claude: the departure question, Delta Lake as the data source, and the graph model. I set up the way of working: OpenAI Codex implemented each work package from a written brief, and Claude Code (Claude Opus 5.5) reviewed every result against the live graph. I picked Claude Sonnet 5.5 over Opus for the running agent because the graph tools do the reasoning; a five-question evaluation found it faster and about a third of the cost. I tested the app as a newcomer and wrote the feedback that drove a full redesign: a searchable maintainer list instead of a login box, suggested questions, follow-up questions after each answer. Then I re-tested it from screenshots. I commissioned the pre-submission review and chose which findings to act on. When Codex reached its usage limit, Claude Code built the evidence and handover features directly. I drafted this post with Claude Code from the repository, the task reports and the live graph, and reviewed it before publishing. What worked: written briefs, a reviewer that ran everything against the real database, and the rule that every query is a readable file. What didn't: three times, unit tests passed while the live system broke. FalkorDB worded an error differently from what the code expected, localhost added a two-second delay on Windows, and the Claude API rejected tool schemas with more than 24 optional parameters. The time-window bug also slipped past every earlier review. It was caught only when a review asked, number by number, whether each one measured what its label claimed. What it can and can't tell you Back to that Friday message. In about a second, Bus Factor can show a team which code areas lose their only recent author, who has reviewed that code anyway, which decisions and pull requests a successor should read, and what to ask before the person goes. It cannot tell anyone who understands the code, what was decided on a call or in a private thread, whether the suggested person is free, or whether the code is any good. It measures fingerprints: who wrote and who reviewed public changes over the last year. That turned out to be enough to start the right conversation. A repository keeps every commit forever. What it doesn't keep is who to ask, and that is the part a graph can hold. --- Bus Factor is open source at Manjunathask/company-brain-falkon-db. docker compose up --build loads the public snapshot and starts the app; every query lives in queries/ with a header explaining it. Built for WeMakeDevs × FalkorDB Graph Hacks, October 2026. Figures and screenshots were produced from the live graph on 18 October 2026; counts tied to the 12-month window shift slightly from day to day.

個人開発のプロジェクトをPolar 2.0に移行中。Solにコードを精査させたら、何を勘違いしたのかset_sortedをsortに置き換えちゃったので、久々に喝をいれるなど。delta-rsへの要望でユースケースを聞かれた時 delta-rs#4724 も思ったけど、sort回避に拘った既存実装がまだ乏しいんだろうな…