Why PostHog Left DuckDB at 2 PB (and How to Check Your DuckLake Catalog)
At PostHog, writing to a single DuckLake table took more than three minutes. Most of that time went to statistics for tables that no longer existed.
That detail, from PostHog's post about moving their warehouse off DuckDB, is the most useful thing I've read about DuckDB's limits at scale this year. It also makes "DuckDB doesn't scale" look like a lazy takeaway, because the query engine held up fine. What broke was DuckLake's metadata layer, the one-machine sizing model, and the Postgres wire protocol, once all of them were serving a multi-tenant cloud product at 2 PB. PostHog is moving the warehouse to its own build of Trino.
I'll go through their numbers first. Further down are two SQL queries that tell you whether your own DuckLake catalog is heading the same way, and a checklist for when staying on DuckDB is still the right call.
Five months from DuckDB launch to Trino migration
On June 29, 2026, PostHog published Why we rebuilt our data warehouse on DuckDB over ClickHouse. They had sound reasons. Warehouse modeling queries can run for hours or days, which their shared ClickHouse cluster couldn't support. ClickHouse also has no cost-based optimizer, so you can't just hand it declarative SQL and trust the plan. Each org got its own DuckDB instance that slept when idle, a Postgres-wire endpoint, and DuckLake as the catalog on S3.
On November 2, 2026, the follow-up landed. They had been re-architecting without DuckDB for three months and were moving to their own build of Trino with a custom table format called HogLake.
So the DuckDB era lasted about five months, and Hacker News noticed. One commenter asked why you can't "foresee such issues" with mock data at the target size. Another called it a recruiting post. The author (roboben) gave a pretty candid answer: "we underestimated the problem and complexity of workloads our customers want to run," and admitted to some sunk-cost thinking.
Before anyone writes "PostHog dumps DuckDB", two things from the thread. The founder said the warehouse was "used by a handful of pilot customers", and ClickHouse "is serving all of our analytical queries." Only the warehouse product is moving.
2 PB, 15.3 million snapshots, 3-minute writes
At the point they moved, PostHog's own data in DuckLake was roughly:
| Metric | Value |
|---|---|
| Data | ~2 PB |
| Rows | ~1 trillion |
| Snapshots | 15.3 million |
| Single-table write latency | 3+ minutes |
| Catalog stats rows belonging to deleted tables | over 99% |
| Speedup after cleaning those rows | 30–50x faster writes |
| Customer pod size (from HN) | 46 vCPUs / 360 GB, and they asked for more |
| Longest queries (from HN) | 12 hours |
The write latency is the number I keep coming back to. According to the post, "every commit re-read the statistics for the entire catalog." So commit cost scales with the whole catalog, not with the table you're writing to. Pile up dropped tables and millions of snapshots and every insert pays for the dead ones too.
PostHog cleaned the dead rows and got the 30–50x speedup. Then the rows grew back. When a cleanup job has to keep up with your catalog's own growth, I'd call that a design problem.
They listed three more DuckLake issues:
- The catalog schema has to stay portable across SQL databases, which rules out engine-specific tuning on Postgres.
- DuckLake opens a lot of connections to the metadata database, and on shared Postgres each one costs something.
- They couldn't get changes upstream fast enough, so they ran a fork. Once you're on a fork, you've lost most of the community benefit you picked an open format for.
Where single-node DuckDB stops scaling
This is the part that's actually about DuckDB. A single-node engine has to be sized before the query arrives. PostHog couldn't predict CPU and memory per query, customers sent heavier queries than expected, and adding disks for spill would have made queries "too slow and expensive to be viable."
On one laptop or one big VM this doesn't matter, because you know your own workload. It does matter when you're a vendor holding thousands of tenants' unknown SQL and have to pick an instance size in advance. A distributed engine like Trino handles this by spreading work across workers, and you pay for that elsewhere: roboben also said on HN that "Trino is not made for fast response times." They plan to use it for long modeling jobs and nightly ETL.
Why DuckDB over the Postgres wire protocol hurt
The third issue had nothing to do with query speed. Exposing DuckDB over port 5432 meant dealing with missing reconnects, TTLs, idle connections, expensive long-lived sessions, and much thinner load-balancing tooling than HTTP has. HTTP headers let them route requests in Kubernetes. Every Postgres client also turned out to expect its own niche behavior, and "coverage never got good enough."
If you're thinking of putting a pgwire shim in front of an embedded engine so your BI tool can connect, read that paragraph again first.
DuckDB vs Trino vs ClickHouse: where each one wins
Even in the exit post, PostHog calls DuckDB "probably the best single-node tool for analyzing heaps of data," and says it "almost always finishes your query." I agree. Based on this case, this is how I'd split the three:
| | DuckDB (+ DuckLake) | ClickHouse | Trino |
|---|---|---|---|
| Shape | Embedded, single node | Server cluster | Distributed MPP, coordinator + workers |
| Best at | Analyst work, notebooks, local Parquet, CI jobs | Fast queries over huge event tables | Long, heavy joins over lake data |
| Optimizer | Cost-based | No cost-based optimizer (per PostHog) | Cost-based |
| Breaks when | Query size is unpredictable or bigger than one box | Hours-long tenant queries share a cluster | You need low-latency interactive answers |
| PostHog uses it for | Nothing in the warehouse anymore | All product analytics | New warehouse (with HogLake) |
Signs you've outgrown DuckDB (and how to check your DuckLake catalog)
Stay on DuckDB if you're one team on one machine, you roughly know your query sizes, and the data fits on one big box's disk plus S3 reads. That covers most people who argue about this online.
You're getting close to PostHog's wall if several of these are true:
1. You're sizing instances for SQL other people write and you can't predict its memory use.
2. Queries regularly spill to disk and spill time is now the slowest part.
3. You run DuckLake with heavy table churn: lots of CREATE/DROP, frequent small commits, snapshot counts in the millions.
4. Commit latency is rising even though the batches stay the same size.
5. You're serving many clients over pgwire and spend more time on connection handling than on queries.
6. You're carrying patches in a fork.
If item 3 or 4 sounds familiar, measure it before you panic. These queries run against your DuckLake metadata database (Postgres in PostHog's case) and use table names from the DuckLake spec:
-- how many snapshots are you carrying?
SELECT count(*) FROM ducklake_snapshot;
-- what share of column stats rows belong to tables with no live version?
SELECT round(100.0 * count(*) FILTER (WHERE live.table_id IS NULL) / NULLIF(count(*), 0), 1) AS pct_dead
FROM ducklake_table_column_stats s
LEFT JOIN (SELECT DISTINCT table_id FROM ducklake_table WHERE end_snapshot IS NULL) live
USING (table_id);
Then set up the maintenance the DuckLake docs recommend instead of letting snapshots accumulate:
CALL my_ducklake.set_option('expire_older_than', '1 month');
CALL ducklake_expire_snapshots('my_ducklake', dry_run => true, older_than => now() - INTERVAL '1 week');
CALL ducklake_expire_snapshots('my_ducklake', older_than => now() - INTERVAL '1 week');
CALL ducklake_cleanup_old_files('my_ducklake', older_than => now() - INTERVAL '1 week');
Plus an occasional VACUUM on the Postgres catalog, which the docs say is likely enough there. The expire-snapshots docs say expiry also cleans up metadata for dropped objects, which is the kind of row PostHog was drowning in. Nothing in PostHog's post says expiry would have fixed their problem, though; they cleaned those rows and they came back. I'd still keep the snapshot count low. It costs you nothing, and 15.3 million is not a number you want to see in your own catalog.
My take
Picking DuckDB was defensible. The trouble started when PostHog used an embedded engine as a multi-tenant cloud service, put all of PostHog into a young metadata format at petabyte scale as the warehouse's first customer, and exposed it over the hardest-to-scale protocol available. You can get away with any one of those, but doing all three at once with 1 trillion rows is what broke.
What worries me more is the fix. HogLake is another in-house table format, and HN raised that right away ("making the same mistake again"). Roboben said Delta and Iceberg "had other downsides" for them but didn't explain what those were. I'd like that post more than this one.
So: how far have you pushed DuckDB before it broke? Was it memory, spill, DuckLake commits, or has it simply never broken for you? If you think PostHog should have stuck with ClickHouse on Iceberg, make the case in the comments.
Sources
- Why we're moving our warehouse off DuckDB, PostHog, Nov 2, 2026
- Why we rebuilt our data warehouse on DuckDB over ClickHouse, PostHog, Jun 29, 2026
- Hacker News discussion
- DuckLake recommended maintenance, expire snapshots, cleanup of files
- DuckLake catalog specification
- PostHog's DuckLake fork
Mentions
Why PostHog Left DuckDB at 2 PB (and How to Check Your DuckLake Catalog) At PostHog, writing to a single DuckLake table took more than three minutes. Most of that time went to statistics for tables that no longer existed. That detail, from PostHog's post about moving their warehouse off DuckDB, is the most useful thing I've read about DuckDB's limits at scale this year. It also makes "DuckDB doesn't scale" look like a lazy takeaway, because the query engine held up fine. What broke was DuckLake's metadata layer, the one-machine sizing model, and the Postgres wire protocol, once all of them were serving a multi-tenant cloud product at 2 PB. PostHog is moving the warehouse to its own build of Trino. I'll go through their numbers first. Further down are two SQL queries that tell you whether your own DuckLake catalog is heading the same way, and a checklist for when staying on DuckDB is still the right call. Five months from DuckDB launch to Trino migration On June 29, 2026, PostHog published Why we rebuilt our data warehouse on DuckDB over ClickHouse. They had sound reasons. Warehouse modeling queries can run for hours or days, which their shared ClickHouse cluster couldn't support. ClickHouse also has no cost-based optimizer, so you can't just hand it declarative SQL and trust the plan. Each org got its own DuckDB instance that slept when idle, a Postgres-wire endpoint, and DuckLake as the catalog on S3. On November 2, 2026, the follow-up landed. They had been re-architecting without DuckDB for three months and were moving to their own build of Trino with a custom table format called HogLake. So the DuckDB era lasted about five months, and Hacker News noticed. One commenter asked why you can't "foresee such issues" with mock data at the target size. Another called it a recruiting post. The author (roboben) gave a pretty candid answer: "we underestimated the problem and complexity of workloads our customers want to run," and admitted to some sunk-cost thinking. Before anyone writes "PostHog dumps DuckDB", two things from the thread. The founder said the warehouse was "used by a handful of pilot customers", and ClickHouse "is serving all of our analytical queries." Only the warehouse product is moving. 2 PB, 15.3 million snapshots, 3-minute writes At the point they moved, PostHog's own data in DuckLake was roughly: | Metric | Value | |---|---| | Data | ~2 PB | | Rows | ~1 trillion | | Snapshots | 15.3 million | | Single-table write latency | 3+ minutes | | Catalog stats rows belonging to deleted tables | over 99% | | Speedup after cleaning those rows | 30–50x faster writes | | Customer pod size (from HN) | 46 vCPUs / 360 GB, and they asked for more | | Longest queries (from HN) | 12 hours | The write latency is the number I keep coming back to. According to the post, "every commit re-read the statistics for the entire catalog." So commit cost scales with the whole catalog, not with the table you're writing to. Pile up dropped tables and millions of snapshots and every insert pays for the dead ones too. PostHog cleaned the dead rows and got the 30–50x speedup. Then the rows grew back. When a cleanup job has to keep up with your catalog's own growth, I'd call that a design problem. They listed three more DuckLake issues: - The catalog schema has to stay portable across SQL databases, which rules out engine-specific tuning on Postgres. - DuckLake opens a lot of connections to the metadata database, and on shared Postgres each one costs something. - They couldn't get changes upstream fast enough, so they ran a fork. Once you're on a fork, you've lost most of the community benefit you picked an open format for. Where single-node DuckDB stops scaling This is the part that's actually about DuckDB. A single-node engine has to be sized before the query arrives. PostHog couldn't predict CPU and memory per query, customers sent heavier queries than expected, and adding disks for spill would have made queries "too slow and expensive to be viable." On one laptop or one big VM this doesn't matter, because you know your own workload. It does matter when you're a vendor holding thousands of tenants' unknown SQL and have to pick an instance size in advance. A distributed engine like Trino handles this by spreading work across workers, and you pay for that elsewhere: roboben also said on HN that "Trino is not made for fast response times." They plan to use it for long modeling jobs and nightly ETL. Why DuckDB over the Postgres wire protocol hurt The third issue had nothing to do with query speed. Exposing DuckDB over port 5432 meant dealing with missing reconnects, TTLs, idle connections, expensive long-lived sessions, and much thinner load-balancing tooling than HTTP has. HTTP headers let them route requests in Kubernetes. Every Postgres client also turned out to expect its own niche behavior, and "coverage never got good enough." If you're thinking of putting a pgwire shim in front of an embedded engine so your BI tool can connect, read that paragraph again first. DuckDB vs Trino vs ClickHouse: where each one wins Even in the exit post, PostHog calls DuckDB "probably the best single-node tool for analyzing heaps of data," and says it "almost always finishes your query." I agree. Based on this case, this is how I'd split the three: | | DuckDB (+ DuckLake) | ClickHouse | Trino | |---|---|---|---| | Shape | Embedded, single node | Server cluster | Distributed MPP, coordinator + workers | | Best at | Analyst work, notebooks, local Parquet, CI jobs | Fast queries over huge event tables | Long, heavy joins over lake data | | Optimizer | Cost-based | No cost-based optimizer (per PostHog) | Cost-based | | Breaks when | Query size is unpredictable or bigger than one box | Hours-long tenant queries share a cluster | You need low-latency interactive answers | | PostHog uses it for | Nothing in the warehouse anymore | All product analytics | New warehouse (with HogLake) | Signs you've outgrown DuckDB (and how to check your DuckLake catalog) Stay on DuckDB if you're one team on one machine, you roughly know your query sizes, and the data fits on one big box's disk plus S3 reads. That covers most people who argue about this online. You're getting close to PostHog's wall if several of these are true: 1. You're sizing instances for SQL other people write and you can't predict its memory use. 2. Queries regularly spill to disk and spill time is now the slowest part. 3. You run DuckLake with heavy table churn: lots of CREATE/DROP, frequent small commits, snapshot counts in the millions. 4. Commit latency is rising even though the batches stay the same size. 5. You're serving many clients over pgwire and spend more time on connection handling than on queries. 6. You're carrying patches in a fork. If item 3 or 4 sounds familiar, measure it before you panic. These queries run against your DuckLake metadata database (Postgres in PostHog's case) and use table names from the DuckLake spec: -- how many snapshots are you carrying? SELECT count(*) FROM ducklake_snapshot; -- what share of column stats rows belong to tables with no live version? SELECT round(100.0 * count(*) FILTER (WHERE live.table_id IS NULL) / NULLIF(count(*), 0), 1) AS pct_dead FROM ducklake_table_column_stats s LEFT JOIN (SELECT DISTINCT table_id FROM ducklake_table WHERE end_snapshot IS NULL) live USING (table_id); Then set up the maintenance the DuckLake docs recommend instead of letting snapshots accumulate: CALL my_ducklake.set_option('expire_older_than', '1 month'); CALL ducklake_expire_snapshots('my_ducklake', dry_run => true, older_than => now() - INTERVAL '1 week'); CALL ducklake_expire_snapshots('my_ducklake', older_than => now() - INTERVAL '1 week'); CALL ducklake_cleanup_old_files('my_ducklake', older_than => now() - INTERVAL '1 week'); Plus an occasional VACUUM on the Postgres catalog, which the docs say is likely enough there. The expire-snapshots docs say expiry also cleans up metadata for dropped objects, which is the kind of row PostHog was drowning in. Nothing in PostHog's post says expiry would have fixed their problem, though; they cleaned those rows and they came back. I'd still keep the snapshot count low. It costs you nothing, and 15.3 million is not a number you want to see in your own catalog. My take Picking DuckDB was defensible. The trouble started when PostHog used an embedded engine as a multi-tenant cloud service, put all of PostHog into a young metadata format at petabyte scale as the warehouse's first customer, and exposed it over the hardest-to-scale protocol available. You can get away with any one of those, but doing all three at once with 1 trillion rows is what broke. What worries me more is the fix. HogLake is another in-house table format, and HN raised that right away ("making the same mistake again"). Roboben said Delta and Iceberg "had other downsides" for them but didn't explain what those were. I'd like that post more than this one. So: how far have you pushed DuckDB before it broke? Was it memory, spill, DuckLake commits, or has it simply never broken for you? If you think PostHog should have stuck with ClickHouse on Iceberg, make the case in the comments. Sources - Why we're moving our warehouse off DuckDB, PostHog, Nov 2, 2026 - Why we rebuilt our data warehouse on DuckDB over ClickHouse, PostHog, Jun 29, 2026 - Hacker News discussion - DuckLake recommended maintenance, expire snapshots, cleanup of files - DuckLake catalog specification - PostHog's DuckLake fork
Author here. Our fork, like most of our work is public on Github: postHog/ducklake I think it does not include 100% of all changes since a few we upstreamed directly (to a few extensions related to Ducklake too) but if you are really interested, you can find them too.