postHog

Why PostHog Left DuckDB at 2 PB (and How to Check Your DuckLake Catalog) At PostHog, writing to a single DuckLake table took more than three minutes. Most of that time went to statistics for tables that no longer existed. That detail, from PostHog's post about moving their warehouse off DuckDB, is the most useful thing I've read about DuckDB's limits at scale this year. It also makes "DuckDB doesn't scale" look like a lazy takeaway, because the query engine held up fine. What broke was DuckLake's metadata layer, the one-machine sizing model, and the Postgres wire protocol, once all of them were serving a multi-tenant cloud product at 2 PB. PostHog is moving the warehouse to its own build of Trino. I'll go through their numbers first. Further down are two SQL queries that tell you whether your own DuckLake catalog is heading the same way, and a checklist for when staying on DuckDB is still the right call. Five months from DuckDB launch to Trino migration On June 29, 2026, PostHog published Why we rebuilt our data warehouse on DuckDB over ClickHouse. They had sound reasons. Warehouse modeling queries can run for hours or days, which their shared ClickHouse cluster couldn't support. ClickHouse also has no cost-based optimizer, so you can't just hand it declarative SQL and trust the plan. Each org got its own DuckDB instance that slept when idle, a Postgres-wire endpoint, and DuckLake as the catalog on S3. On November 2, 2026, the follow-up landed. They had been re-architecting without DuckDB for three months and were moving to their own build of Trino with a custom table format called HogLake. So the DuckDB era lasted about five months, and Hacker News noticed. One commenter asked why you can't "foresee such issues" with mock data at the target size. Another called it a recruiting post. The author (roboben) gave a pretty candid answer: "we underestimated the problem and complexity of workloads our customers want to run," and admitted to some sunk-cost thinking. Before anyone writes "PostHog dumps DuckDB", two things from the thread. The founder said the warehouse was "used by a handful of pilot customers", and ClickHouse "is serving all of our analytical queries." Only the warehouse product is moving. 2 PB, 15.3 million snapshots, 3-minute writes At the point they moved, PostHog's own data in DuckLake was roughly: | Metric | Value | |---|---| | Data | ~2 PB | | Rows | ~1 trillion | | Snapshots | 15.3 million | | Single-table write latency | 3+ minutes | | Catalog stats rows belonging to deleted tables | over 99% | | Speedup after cleaning those rows | 30–50x faster writes | | Customer pod size (from HN) | 46 vCPUs / 360 GB, and they asked for more | | Longest queries (from HN) | 12 hours | The write latency is the number I keep coming back to. According to the post, "every commit re-read the statistics for the entire catalog." So commit cost scales with the whole catalog, not with the table you're writing to. Pile up dropped tables and millions of snapshots and every insert pays for the dead ones too. PostHog cleaned the dead rows and got the 30–50x speedup. Then the rows grew back. When a cleanup job has to keep up with your catalog's own growth, I'd call that a design problem. They listed three more DuckLake issues: - The catalog schema has to stay portable across SQL databases, which rules out engine-specific tuning on Postgres. - DuckLake opens a lot of connections to the metadata database, and on shared Postgres each one costs something. - They couldn't get changes upstream fast enough, so they ran a fork. Once you're on a fork, you've lost most of the community benefit you picked an open format for. Where single-node DuckDB stops scaling This is the part that's actually about DuckDB. A single-node engine has to be sized before the query arrives. PostHog couldn't predict CPU and memory per query, customers sent heavier queries than expected, and adding disks for spill would have made queries "too slow and expensive to be viable." On one laptop or one big VM this doesn't matter, because you know your own workload. It does matter when you're a vendor holding thousands of tenants' unknown SQL and have to pick an instance size in advance. A distributed engine like Trino handles this by spreading work across workers, and you pay for that elsewhere: roboben also said on HN that "Trino is not made for fast response times." They plan to use it for long modeling jobs and nightly ETL. Why DuckDB over the Postgres wire protocol hurt The third issue had nothing to do with query speed. Exposing DuckDB over port 5432 meant dealing with missing reconnects, TTLs, idle connections, expensive long-lived sessions, and much thinner load-balancing tooling than HTTP has. HTTP headers let them route requests in Kubernetes. Every Postgres client also turned out to expect its own niche behavior, and "coverage never got good enough." If you're thinking of putting a pgwire shim in front of an embedded engine so your BI tool can connect, read that paragraph again first. DuckDB vs Trino vs ClickHouse: where each one wins Even in the exit post, PostHog calls DuckDB "probably the best single-node tool for analyzing heaps of data," and says it "almost always finishes your query." I agree. Based on this case, this is how I'd split the three: | | DuckDB (+ DuckLake) | ClickHouse | Trino | |---|---|---|---| | Shape | Embedded, single node | Server cluster | Distributed MPP, coordinator + workers | | Best at | Analyst work, notebooks, local Parquet, CI jobs | Fast queries over huge event tables | Long, heavy joins over lake data | | Optimizer | Cost-based | No cost-based optimizer (per PostHog) | Cost-based | | Breaks when | Query size is unpredictable or bigger than one box | Hours-long tenant queries share a cluster | You need low-latency interactive answers | | PostHog uses it for | Nothing in the warehouse anymore | All product analytics | New warehouse (with HogLake) | Signs you've outgrown DuckDB (and how to check your DuckLake catalog) Stay on DuckDB if you're one team on one machine, you roughly know your query sizes, and the data fits on one big box's disk plus S3 reads. That covers most people who argue about this online. You're getting close to PostHog's wall if several of these are true: 1. You're sizing instances for SQL other people write and you can't predict its memory use. 2. Queries regularly spill to disk and spill time is now the slowest part. 3. You run DuckLake with heavy table churn: lots of CREATE/DROP, frequent small commits, snapshot counts in the millions. 4. Commit latency is rising even though the batches stay the same size. 5. You're serving many clients over pgwire and spend more time on connection handling than on queries. 6. You're carrying patches in a fork. If item 3 or 4 sounds familiar, measure it before you panic. These queries run against your DuckLake metadata database (Postgres in PostHog's case) and use table names from the DuckLake spec: -- how many snapshots are you carrying? SELECT count(*) FROM ducklake_snapshot; -- what share of column stats rows belong to tables with no live version? SELECT round(100.0 * count(*) FILTER (WHERE live.table_id IS NULL) / NULLIF(count(*), 0), 1) AS pct_dead FROM ducklake_table_column_stats s LEFT JOIN (SELECT DISTINCT table_id FROM ducklake_table WHERE end_snapshot IS NULL) live USING (table_id); Then set up the maintenance the DuckLake docs recommend instead of letting snapshots accumulate: CALL my_ducklake.set_option('expire_older_than', '1 month'); CALL ducklake_expire_snapshots('my_ducklake', dry_run => true, older_than => now() - INTERVAL '1 week'); CALL ducklake_expire_snapshots('my_ducklake', older_than => now() - INTERVAL '1 week'); CALL ducklake_cleanup_old_files('my_ducklake', older_than => now() - INTERVAL '1 week'); Plus an occasional VACUUM on the Postgres catalog, which the docs say is likely enough there. The expire-snapshots docs say expiry also cleans up metadata for dropped objects, which is the kind of row PostHog was drowning in. Nothing in PostHog's post says expiry would have fixed their problem, though; they cleaned those rows and they came back. I'd still keep the snapshot count low. It costs you nothing, and 15.3 million is not a number you want to see in your own catalog. My take Picking DuckDB was defensible. The trouble started when PostHog used an embedded engine as a multi-tenant cloud service, put all of PostHog into a young metadata format at petabyte scale as the warehouse's first customer, and exposed it over the hardest-to-scale protocol available. You can get away with any one of those, but doing all three at once with 1 trillion rows is what broke. What worries me more is the fix. HogLake is another in-house table format, and HN raised that right away ("making the same mistake again"). Roboben said Delta and Iceberg "had other downsides" for them but didn't explain what those were. I'd like that post more than this one. So: how far have you pushed DuckDB before it broke? Was it memory, spill, DuckLake commits, or has it simply never broken for you? If you think PostHog should have stuck with ClickHouse on Iceberg, make the case in the comments. Sources - Why we're moving our warehouse off DuckDB, PostHog, Nov 2, 2026 - Why we rebuilt our data warehouse on DuckDB over ClickHouse, PostHog, Jun 29, 2026 - Hacker News discussion - DuckLake recommended maintenance, expire snapshots, cleanup of files - DuckLake catalog specification - PostHog's DuckLake fork

@ceccon_me yep, seems to be at github.com/PostHog/hoglakehttps://github.com/Post….

Author here. Our fork, like most of our work is public on Github: postHog/ducklake I think it does not include 100% of all changes since a few we upstreamed directly (to a few extensions related to Ducklake too) but if you are really interested, you can find them too.

The git repo [0] has a lot more of the technical details. [0] PostHog/hoglake

PostHog hoglake: Petabyte-scale open table format

Jobs every engineer should automate with agents Jobs every engineer should automate with agents How we handed off our dev chores to scheduled agents (with prompts) Jina Yoon Oct 26 Welcome to build mode PostHog recently imported your email address from another platform to Substack. You'll now receive their posts via email or the Substack app. To set up your profile and discover more on Substack, click here. Some engineering tasks are too repetitive to be worth prompting every time by hand, but too fuzzy for a script. We've been using Scouts at PostHog to do these instead – scheduled agents with long-term memory that run on a cron trigger.¹ Here are 6 chores we’ve automated this way, with prompts you can copy to build your own version of each. 1. Validating small fixes in prod Everyone's merging more code than ever. And the smaller the PR, the more annoying it is to stop what you’re doing to check it in prod – especially if you already did that in dev. Take this agent-generated string length limit fix that Andy Maguire from our self-driving team merged. It's a tiny change that takes more effort to validate than to push: he'd have to check back for when it hits prod, open error logs, and verify traffic is still flowing. Instead, Andy queued it to merge at 09:19 UTC and moved on to other tasks. It reached prod about an hour later, and the next morning a scheduled agent confirmed the fix held across 200K sandbox-creation log lines using a prompt similar to our PR follow-up skill. It's even more useful in the opposite case. One time, an agent fixed an error that looked like it came from our browser SDK, affecting about 45k users a week. A later validation report showed the error was firing more than before from our Node SDK, which has its own copy of the same utility – exactly the kind of thing a human might not have remembered to check. Steal this prompt: Read github.com/PostHog/posthog/blob/master/products/s… and build the equivalent as a scheduled agent for my stack. Ask me where my deploys, logs, and errors live and where reports should go before writing anything. 2. Capturing feature requests We use agents to find recurring themes in user feedback from in-app surveys, support tickets, Discord, Gong, and Slack. A lesson we recently re-learned was the importance of prioritizing that feedback. When our sales team set up a scheduled agent to post to #feature-request-feed, Raquel from our Blitzscale team said that these automations had been less than useful in her experience: Tyler Crandall, the TAM who created the feed, updated the prompts to factor in urgency. Each one now gets rated based on the call transcript and customer history, which makes up for the facial expressions and voice cues we'd rely on if we did this manually. Plus, the agent can reference its own ratings as context for future runs, too: Steal this prompt: Set up a scheduled agent that runs every weekday morning and pulls feature requests from yesterday's customer calls and Slack channels. Count only what customers said, and skip anything an existing feature already solves. Log each request with the problem, the ask, a verbatim quote, and a link. Score urgency 0–100 from the customer's own words, not account size, with one sentence of evidence. Ask me where transcripts and channels live before writing anything. 3. Adding event tracking There are times where even we forget to add instrumentation for product analytics. We added a rule in our AGENTS.md to make AI do this for us moving forward, but we also have a scheduled agent that catches gaps retroactively. It works through recently merged PRs and looks for user-facing actions that are missing event tracking. When it finds one, it cross-checks our live PostHog data to confirm the event never fires, and then files a report with a fix. A coding agent then picks up that report and opens a PR to add the instrumentation. It's useful to have a separate agent doing the actual implementation work. The scheduled agent's original report recommended adding 2 different capture calls in the frontend for this PR. The coding agent, which has more specialized context, found that multiple sources (including MCP and API callers) sent the same request. It chose a better pattern to instrument these as a single capture on server-side, with session IDs and access method. Steal this prompt: Set up a scheduled agent that runs daily and checks recently merged PRs for new user actions shipped without analytics tracking – like a signup, an export, or a server-side API call. Confirm each gap against live event data, and skip anything already flagged. File one report per gap with the file, line, and a fix for a coding agent to implement with this skill: github.com/PostHog/ai-plugin/blob/main/skills/ins… Ask me where our code, analytics, and reports live before writing anything. 4. Diagnosing flaky tests Flaky tests are a huge bottleneck for engineers working with AI. They cause intermittent CI failures (usually from hard-to-control factors like timing, shared state, or dependencies). These two were the biggest obstacles slowing down our engineers according to our Q2 internal DevEx survey: To help with this, we added a scheduled agent that checks for newly quarantined Trunk flaky tests every 3 hours and creates (or updates) reports that are actionable enough to be turned into PRs. Since July, it's filed 72 diagnostic reports, and 11 of those have led to merged fixes like this one, which a coding agent wrote using our skill for fixing flaky tests. Even though this loop isn't fully automatable, we still consider it a win because it lowers the cost and friction for a human to start a fix. The diagnostic reports are always kept up to date in the background so that anyone can quickly pick one up at any time. And that's way more likely to happen than if our engineers had to start each one from scratch. Steal this prompt: Set up a scheduled agent that runs every three hours and checks CI for newly flaky or quarantined tests. Diagnose each one with this skill, but stop at a report instead of a PR: github.com/PostHog/posthog/blob/master/.agents/sk… Only report what you can back with source or log evidence, and update an existing report for the same test instead of filing a new one. Ask me where our CI and flaky-test data live before writing anything. 5. Triaging our own agent feedback Another type of feedback we take seriously is the responses from our internal coding agents using our dev environment, which we call hogli. Our monorepo's AGENTS.md instructs agents to use hogli devex:feedback to send a report whenever a hogli command or local dev workflow is broken, slow, or confusing. We have similar systems in place for our PostHog MCP and our AI onboarding wizard. The problem with all of these is that if you prompt agents to send reports, they'll always try to send something. We already have a few thousand reports written by our agents since we shipped it in July, and that’s way too many for anyone on the team to read. Raul, an engineer on our DevEx team, used to have an automation to triage these by grouping reports into themes, checking if the root issue has already been fixed, roughly ranking by frequency and impact, then committing the notes to a file in our private DevEx repo. Instead, he now just checks on these once a month for a couple of hours, and the reports still lead to low-hanging fixes from across the team, like rewording a preflight warning or updating how code owners are labeled. It's a good example of how some tasks are still better suited for AI-assisted human review, even if they could be solved with scheduled automations. Steal this prompt: Read github.com/PostHog/posthog/blob/master/tools/hogl… and add an equivalent feedback command to my repo, plus an AGENTS.md rule telling agents to use it when a dev tool is broken, slow, or confusing. Then write a monthly triage skill that groups new reports by what actually broke and checks whether each theme still happens on the latest code or was fixed in a merged PR – a theme that stopped being reported is unverified, not fixed. It should give me the top three themes to act on, what's already fixed, and a few rare reports worth reading, and save notes for the next run. Ask me where reports should go before writing anything. 6. Self-improving prompts With enough data, some of our scheduled agents can even suggest improvements to their own prompts. The key is that they have to include real evidence from past runs, and proposals have to go through human review. We also have a versioned skill store that makes it easy to track these changes. For example, one scheduled agent proactively checks that our product docs are up to date. It once flagged that we should remove the "in beta" notices from two of our docs after a cleanup PR deleted both features' flags. What the agent didn't know was that even though both features were visible to all PostHog users, the team didn't consider them GA yet. When a human reviewer dismissed its reports with handwritten feedback, the agent filed a report suggesting a fix to its own skill: - Before: A flag removed from FEATURE_FLAGS in the window usually means the feature shipped to everyone (went GA). - After: A flag removed from FEATURE_FLAGS in the window usually means the feature reached everyone, which is rollout, not a GA declaration. Steal this: Set up a scheduled agent that runs daily and flags docs that newly merged code made wrong – removed or renamed features, changed defaults, stale beta labels – with the page, the PR, and a suggested edit. A removed feature flag means a rollout, not a release, so ask before calling anything generally available. End every run with this self-improvement step, and send me proposed changes for review: github.com/PostHog/posthog/blob/master/products/s… Ask me where our docs and code live before writing anything. Words by Jina Yoon, who is still waiting for agents that can fold her laundry. 🕰️ Scheduled reads - How I use skills for everything – Andy Maguire - Stanford CS146S: How always-on agents work – Lee Robinson - 6 things I’ve learned in 6 years at PostHog – Charles Cook - What is Toil in SRE: Understanding Its Impact – Vivek Rau - More engineers should consider working in sales – Ryan McCrary - How we made claude.ai 3x faster in two weeks – Raymond Wang, Sam Attard, and Issac G. Lighten someone else’s sprint 💼 Tasks we can’t automate - Technical Customer Success Manager - EMEA - Technical Customer Success Manager - Americas - AI Research Engineer - Animator & Motion Graphics Designer - Product Manager (ex-founder or ex-product engineer) - Site Reliability Engineer (US Pacific Time) 1 Some people call these "background agents," "ambient agents," or "always-on agents." Others describe them as a type of loop or cron. © 2026 PostHog 548 Market Street PMB 72296, San Francisco, CA 94104

@wtfrajat @posthog hey @wtfrajat posthog-rs#308 is this from you? i'm gonna be taking a look today so we can get it in!

PostHog; Black: Uncompromising Python code formatter | GitHub Curated Picks | 2026-10-25 Chapters: 00:00 2026-10-25 | GitHub Curated Picks 00:05 Pick #1 PostHog 00:19 Pick #2 Black GitHub Curated Picks reviews 2 source-verified repositories selected for developer usefulness, not popularity position. Each pick is checked against its GitHub repository, README, release evidence, and recent commit; pick numbers are editorial order. The compact card keeps rank, signal, why it matters, and next check together while the ledger preserves direct URLs and release evidence. Treat the list as an educational review surface, not software endorsement, and verify license and maintenance before adoption. Published: 2026-10-25 Review window: daily Curated repository picks: - Pick #1 PostHog | Latest commit | feat(mcp): return condition_id from feature flag read tools (reference | PostHog/posthog - Pick #2 Black | Latest commit | Recent commit verified | psf/black Chapters: 00:00 2026-10-25 | GitHub Curated Picks 00:05 Pick #1 PostHog 00:19 Pick #2 Black 00:40 Watch Next 00:47 Educational Use Only Top comment prompt: Which repository would you verify first, and what release or commit evidence matters most? Source ledger: Direct source URLs: - github.com/trending - api.github.com - PostHog/posthog - github.com/PostHog/posthog/releases/tag/hobby-lat… - github.com/PostHog/posthog/commit/8c626c87f0c85cb… - api.github.com - psf/black - github.com/psf/black/releases/tag/26.10.1 - github.com/psf/black/commit/80db630ac532f0e2f3ba3… Informational and educational purposes only. Not investment advice, product endorsement, or professional guidance. Subscriber reason: Subscribe for the daily AI, GitHub, and market board. Tomorrow watchlist: Tomorrow, check the next tagged release, recent commit, and issue-health signals for the selected projects.

i had a billion claude & codex agents running in parallel for a week and i managed to reverse engineer @posthog no need to pay for analytics anymore, save your tokens and your wallet 100% free & open source here posthog/posthog

Funnily enough I have a coworker that has been exploring how to keep it at bay, I've been bouncing a lot of ideas with him and exploring the space. He's building something called coherence. It's mostly explorative and it's absolutely not ready for any type of consumption beyond some internal testing, but so far it works fairly well.