Open source benchmark for Snowflake, Databricks, BigQuery, Redshift, Fabric, ClickHouse and MotherDuck: 244 queries, 1 GB to 1 TB, up to 100 concurrent users, every compute size
Estuary ran a benchmark of 7 cloud data warehouses and published everything. The setup: 244 queries from three public suites (TPC-DS, DSB, ClickBench), data from 1 GB to 1 TB, 1 to 100 simultaneous sessions, and every compute size each vendor sells, fixed with autoscaling off. We loaded identical data into all seven through one pipeline, no physical tuning on any engine, and report the median of three runs after a warm run.
Write-up with the main findings: estuary.dev
Code, queries and raw results: estuary/data-warehouse-benchmark
A few things from the data:
1. Bigger compute often buys nothing at small data. At 1 GB and 10 GB, 16x the compute returned 0.8 to 1.5x the speed. At 100 GB the same jump on Snowflake was about 2x faster, and at 1 TB nearly 8x. The chart below shows speed (blue) and cost (orange) against the cheapest size at 100 GB, per engine.
preview.redd.it
2. For a single user, MotherDuck had the fastest median query (235 ms) and the lowest cost per query at 100 GB. ClickHouse did not finish 20.4% of the join runs (it’s a different engine on scan queries, where it’s third at its fastest size and first at a $3 an hour budget).
preview.redd.it
3. People served at once is a different question from single-query speed. At 100 GB with a 5 second median target, the cheapest size that served 100 people was Fabric F32 at $5.76 an hour, and 7 of the 42 sizes served nobody within the target. The dashboard lets you change the latency target and every dot recomputes.
preview.redd.it
4. Time grows very differently with data volume. From 1 GB to 1 TB, median query time grew least on Databricks (3.7x) and most on MotherDuck (30x), which starts from the fastest baseline.
preview.redd.it
5. Idle billing matters as much as the hourly rate. Some engines stop the meter when nobody is querying and some bill the capacity around the clock. 1,000 typical queries at 100 GB cost $0.31 on one engine and $7.14 on another.
Happy to answer questions on method, and if you reproduce a result or find something off, open an issue on the repo.
6 points · 2 comments
Mentions
Open source benchmark for Snowflake, Databricks, BigQuery, Redshift, Fabric, ClickHouse and MotherDuck: 244 queries, 1 GB to 1 TB, up to 100 concurrent users, every compute size Estuary ran a benchmark of 7 cloud data warehouses and published everything. The setup: 244 queries from three public suites (TPC-DS, DSB, ClickBench), data from 1 GB to 1 TB, 1 to 100 simultaneous sessions, and every compute size each vendor sells, fixed with autoscaling off. We loaded identical data into all seven through one pipeline, no physical tuning on any engine, and report the median of three runs after a warm run. Write-up with the main findings: estuary.dev Code, queries and raw results: estuary/data-warehouse-benchmark A few things from the data: 1. Bigger compute often buys nothing at small data. At 1 GB and 10 GB, 16x the compute returned 0.8 to 1.5x the speed. At 100 GB the same jump on Snowflake was about 2x faster, and at 1 TB nearly 8x. The chart below shows speed (blue) and cost (orange) against the cheapest size at 100 GB, per engine. preview.redd.it 2. For a single user, MotherDuck had the fastest median query (235 ms) and the lowest cost per query at 100 GB. ClickHouse did not finish 20.4% of the join runs (it’s a different engine on scan queries, where it’s third at its fastest size and first at a $3 an hour budget). preview.redd.it 3. People served at once is a different question from single-query speed. At 100 GB with a 5 second median target, the cheapest size that served 100 people was Fabric F32 at $5.76 an hour, and 7 of the 42 sizes served nobody within the target. The dashboard lets you change the latency target and every dot recomputes. preview.redd.it 4. Time grows very differently with data volume. From 1 GB to 1 TB, median query time grew least on Databricks (3.7x) and most on MotherDuck (30x), which starts from the fastest baseline. preview.redd.it 5. Idle billing matters as much as the hourly rate. Some engines stop the meter when nobody is querying and some bill the capacity around the clock. 1,000 typical queries at 100 GB cost $0.31 on one engine and $7.14 on another. Happy to answer questions on method, and if you reproduce a result or find something off, open an issue on the repo.