Apache Arrow and Apache DataFusion Now List ArrowMetal: Arrow Compute and a DataFusion Optimizer Rule on the Apple Silicon GPU
ArrowMetal is now on two Apache project pages, and I am going to enjoy this one. Since 8 September it has been on Apache Arrow's Powered By page, alongside the libraries I learned Arrow from. Since today, 9 October, it is on Apache DataFusion's integrations list in the user guide, as datafusion-arrowmetal. Both went in the way every entry on those lists does: a pull request, read by the project's committers, merged. Here is what the two listings mean, and what the listed thing does today, because 0.5.0 shipped yesterday and both entries describe it.
2 Apache project pages list ArrowMetal: Arrow's Powered By since 8 September, DataFusion's integrations since 9 October. The data crosses 3 Arrow C interfaces (C Data, C Stream, C Device Data) with no copy either way. The DataFusion crate's differential grid runs 13,632 query pairs, rule off against rule on, with 0 mismatches. A sort of 50,000,000 permuted int64 keys runs in 43 ms through the MetalEngine against 242 ms for Polars 2.0.0's own collect(), frames equal. Apple M4 Max, arrowmetal 0.5.0 from PyPI, best of 5, 9 October 2026.
ArrowMetal is my Apache-2.0 Arrow compute library for the Apple silicon GPU: Arrow buffers stay in unified memory, the GPU reads them in place, and the results come back as Arrow. It has a Polars engine, a DataFusion optimizer rule, DuckDB extensions and seven language bindings. The launch post covers the design; this one is about the listings.
Two listings, and what they mean
Arrow's Powered By page is the Arrow website's list of projects and products using Apache Arrow; DataFusion's is the section of its user guide for community projects that extend DataFusion. Getting on either means a committer read a description of your work, judged it accurate and on topic, and put their name on the merge. For Arrow that took one morning; for DataFusion, six days, one committer's careful review and a maintainer's merge, with a thank-you at the end. Those are the two communities ArrowMetal is built on, and being read and accepted by them is the part I will remember.
| the two listings | Apache Arrow's Powered By page and Apache DataFusion's integrations list |
|---|---|
| What the page is | Arrow: Powered By, the Arrow website's list of projects and products using Apache Arrow, on the page that also carries the trademark guidance. DataFusion: the “Integrations and Extensions” section of the user guide, “community projects that extend DataFusion or provide integrations with other systems”. |
| How an entry gets in | A pull request, read by the project's committers. Arrow: arrow-site #815, opened and merged on the morning of 8 September after one committer's approval. DataFusion: datafusion #26004, opened 3 October, one committer's review, merged by a maintainer on 9 October. |
| What the entry says | Arrow: “Apache Arrow compute on Apple silicon GPUs through Metal. Arrow buffers live in unified memory the GPU reads in place, so the kernels … run without a copy in either direction …” and the seven bindings. DataFusion: “A physical optimizer rule that runs sorts and measured group-bys on the Apple silicon GPU through Metal.” |
| What it means | The project's committers read a description of the thing, judged it accurate and on topic, and merged it under their name. For a one-person project built on their work, that is the review that counts. |
| What it does not mean | An endorsement, a code review, or any status inside the Foundation. ArrowMetal is an independent Apache-2.0 project, happily so; “Apache Arrow” and “Apache DataFusion” are trademarks of the Apache Software Foundation, which is why the name has no “Apache” in it. |
The two entries as they read on 9 October 2026; dates from the pull requests arrow-site #815 and datafusion #26004.
It matters in a practical way too: these are the pages people read when they ask “what runs on Arrow?” or “what extends DataFusion?”, and an accurate line there reaches the right readers. Both descriptions name what the software does and nothing it does not; that is the bar this post holds to. In fairness to both projects, a listing is not an endorsement or a code review, and ArrowMetal remains independent; the table has the texts and dates.
The Arrow side: what the entry describes
The Powered By entry says the kernels run on Arrow buffers in unified memory without a copy either way, and that data crosses through Arrow's C Data, C Stream and C Device Data interfaces. That is the architecture: a pyarrow array, a Polars Series or a DataFusion RecordBatch is handed over as Arrow, the GPU works on the same bytes, and Arrow comes back. Here is the README's first thing to run, from the 0.5.0 wheel on PyPI, then a 50,000,000-row sort through the Polars engine on Polars 2.0.0, with the engine's report.
$ pip install "arrowmetal[polars]" # 0.5.0 from PyPI; polars 2.0.0 comes with the extra
import pyarrow as pa, arrowmetal as am # the README's first thing to run, as installed
col = am.array(pa.array([1, None, 3, 40])) # a pyarrow array crosses in through the C Data interface
big = col.filter_where(">", 2) # GPU
print(big.sum(), big.to_arrow()) # 43 [3, 40]
keys = am.array(pa.array([0, 1, 0, 2], pa.int32()))
print(keys.group_by(3).sum(col).to_arrow()) # [4, null, 40]
import polars as pl # 50,000,000 permuted int64 keys, a Float64 payload
engine = am.MetalEngine() # the measured defaults; on 2.0.0, the 2.x table
print(engine.explain(df.lazy().sort("k")))
MetalEngine report for explain (polars 2.0.0, IR (15, 2))
metal: Sort#1 [Sort > DataFrameScan] over 50,000,000 rows
rule: 50,000,000 input rows is at or above the 10,000,000-row crossover for sort
(engine table, Benchmarks/results/polars_engine_crossover_2026-10-08-polars2.csv)
# best of 5: polars 2.0.0 collect() 242.37 ms; collect(engine=engine) 43.16 ms; frames equal
am.array(pa.array(...)) is the C Data interface at work; nothing is converted. The report names the rule, the row count and the crossover file, so the decision is checkable. The 43 against 242 milliseconds is one M4 Max sorting 50,000,000 permuted int64 keys with a Float64 payload, best of 5, against Polars 2.0.0's own collect(), frames equal. The engine takes the sort from 10,000,000 rows up, where the crossover table puts it; below that, Polars sorts.
The DataFusion side: what the entry describes
The integrations entry calls the crate a physical optimizer rule that runs sorts and measured group-bys on the Apple silicon GPU through Metal. Such a rule is a hook DataFusion gives to anyone: it sees the physical plan and may replace nodes. This one replaces a full ORDER BY, and the aggregate shapes its measured table takes, with a node that runs the plan on the GPU and hands back the RecordBatches DataFusion expected. The SQL and every other node are untouched. The crate's quickstart, built from crates.io 0.5.0 against DataFusion 55.1.0 on this M4 Max: the sort it takes, the two it leaves.
# Cargo.toml: datafusion = "=55.1.0", datafusion-arrowmetal = "0.5.0"
let rule = ArrowMetalRule::new(ArrowMetalConfig::default()); // full sorts from 250,000 rows
let ctx = session_context(SessionConfig::new().with_target_partitions(4), rule.clone());
ctx.register_table("sales", mem_table)?; // 1,000,000 rows: region, amount, name
== SELECT name, region, amount FROM sales ORDER BY amount DESC NULLS LAST, name
ProjectionExec: expr=[name@2 as name, region@0 as region, amount@1 as amount]
MetalExec: sort=[amount DESC NULLS LAST, name ASC NULLS LAST]
DataSourceExec: partitions=1, partition_sizes=[1]
TAKEN SortExec -- input rows 1000000 (exact) vs min_rows 250000
1000000 rows; the first 3: n001395 | 0 | 1429.4285714285713...
== SELECT name, amount FROM sales ORDER BY amount DESC NULLS LAST LIMIT 3
LEFT SortExec: TopK(fetch=3) -- top-k (sort with fetch 3) disabled in config
== SELECT region, count(*) AS n, avg(amount) AS mean FROM sales GROUP BY region ORDER BY region
LEFT AggregateExec: gby=[region], aggr=[count(Int64(1)), avg(sales.amount)]
-- count + sum_avg_f64 over 1 i64 key: the measured table takes it at no
group count and size (results/datafusion_groupby_sweep_2026-10-01.csv)
13 rows; the first 3: 0 | 76924 | 714.7095888611585...
The numbers behind “measured” are in the DataFusion post and docs/DATAFUSION.md: on the M4 Max, full sorts from 250,000 to 50,000,000 rows ran 6.9x to 28.8x faster than DataFusion alone across four key types, 50,000,000 rows sorted in 126 CPU-milliseconds against 6,104 on 16 partitions, and 13,632 query pairs, rule off and on, matched with 0 mismatches. Measured on 0.4.0 in late September; 0.5.0 changes the aggregate side, below, not the sort numbers.
What 0.5.0 changed since those two posts
Yesterday's release is the one both entries now describe, and it is the best one yet. Four lines from the release notes matter here.
| ArrowMetal 0.5.0, 8 October 2026 | what changed since the two posts the listings point at |
|---|---|
| Polars 2.0.0 | The Polars engine runs on 2.0.0 as on 1.44, with one measured crossover table per Polars major. On 2.0.0 the default takes 52 of 107 benchmarked queries at 50,000,000 rows on the M4 Max, every one 1.35x to 7.94x faster than the faster Polars 2.0.0 engine, none behind; 12 are handed back. The pip extra is now polars>=1.44,<2.1; the re-measurement post has the cases. |
| Integer mean in one pass | A mean over an integer column runs in one pass in the Polars and plan engines, same bits as before: two int32 keys, 200 groups, 20.7 to 14.3 ms at 50,000,000 rows on polars 2.0.0, per the release notes. |
| DataFusion crate | datafusion-arrowmetal 0.5.0 on DataFusion 55.1: full ORDER BY sorts, and by default the aggregate shapes its measured table takes: count(*) and DISTINCT over two int32 keys, integer MIN/MAX over two integer keys or one int64 key, on a MemTable of 50,000,000 rows or more. Hash joins are translated; the measured join table takes none yet. |
| Build targets | DuckDB extensions built and tested against DuckDB 1.5.6; macOS 14 or later on Apple silicon; the Rust crate's build script stops the build on any other target. |
From the v0.5.0 release notes, the CHANGELOG and the crate README at the tag. Every number is one Apple M4 Max.
The Polars line closes a caveat from Wednesday's post, which said the 2.0.0 support was on main, unreleased. As of 0.5.0 it is the wheel on PyPI, which is why the card above needed only a fresh venv.
Install
- Python: pip install "arrowmetal[polars]" installs 0.5.0 and a Polars from 1.44 to 2.0; python -m arrowmetal.polars_engine check prints the Polars, the IR and the table in force. Without the extra, pyarrow is the only dependency.
- Rust: datafusion = "=55.1.0" and datafusion-arrowmetal = "0.5.0" in Cargo.toml; the crate links the Metal library through ARROWMETAL_LIB or the copy the Python wheel installs. docs.rs and docs/RUST.md have the finding rules.
- macOS 14 or later on Apple silicon only. Every number here is one M4 Max on AC power; python -m arrowmetal.bench measures yours in under 30 s.
ArrowMetal is an independent Apache-2.0 project; Apache Arrow and Apache DataFusion are trademarks of the Apache Software Foundation, whose projects I use and build on every day. My thanks to the committers on both sides who read the entries and let them in. Two gates, one small craft, a harbour I am glad to be moored in.
FAQ: the listings, Apache Arrow, Apache DataFusion and the Apple silicon GPU
Is ArrowMetal an Apache project?
No, and proudly independent: an Apache-2.0 project that implements Apache Arrow and extends Apache DataFusion. Both listings are entries on community-maintained lists, read and merged by the projects' committers; they are a welcome, not an endorsement.
Does Apache DataFusion ship the GPU rule?
No. datafusion-arrowmetal is a separate crate on crates.io, pinned to DataFusion 55.1.0. You register its rule on a SessionContext; the SQL and every other plan node stay DataFusion's.
Which Macs does it run on?
Apple silicon on macOS 14 or later. The Python wheel carries the Metal library; the Rust crate finds it through ARROWMETAL_LIB or the wheel's copy. The build script stops on any other target.
Are the answers the same as the CPU's?
The differential grid runs 13,632 DataFusion query pairs, rule off and on, with 0 mismatches; every Polars benchmark query is checked equal to Polars' own frame. Float64 runs through a software IEEE-754 path, bit-exact with the CPU.
Mentions
@andrewlamb1111 I’ve closed the PR because it’s low priority datafusion#24158 I love Open Source, and I miss the way we used to exchange ideas and learn from each other back before the era of AI From now on, I’ll be focusing on high-priority PRs
here's the pr(again): datafusion#25771
Recordings from the DataFusion Sep 30, 2026 London Meetup are now available datafusion#24512
Recordings from the DataFusion Sep 30, 2026 London Meetup are now available datafusion#24512
16.8 hours to 16.7 hours? Why we built a 3.8-second GitHub Actions pre-flight gate (proven across 34+ merged PRs in LLVM, Meta Velox, Valkey & DuckDB) We built ZeroSlop (patrickswedish/zeroslop-action@v1) because cloud AI code review bots charge $24–$40 per seat ($41,760/yr for 174 engineers), yet their own published case studies admit median PR merge time barely moves (16.8 hours before to 16.7 hours after — a 6-minute gain). Instead of another noisy cloud chatbot leaving 30 comments on your pull request, ZeroSlop is a fast 3.8-second pre-flight gate inside GitHub Actions that strips AI wrapper bloat and blocks deleted tests before human review—with zero source code ever leaving your runner. Proven Upstream Merges (The Wall of Proof) We don't ask anyone to take our word for it—our track record of merged pull requests says everything. Every single one of these upstream C++, Rust, and C pull requests passed through ZeroSlop and was merged by core maintainers: - LLVM (Core C++ Compiler Infrastructure for Apple, Google & Rust): llvm-project#215076 - Meta Velox (Meta's C++ Vectorized Database Engine): velox#18529 - Valkey (Linux Foundation C In-Memory Datastore, successor to Redis): valkey#4424 - DuckDB (High-Performance C++ Analytical OLAP Database): duckdb#26443 - Apache DataFusion (Extensible Rust Arrow Query Engine): datafusion#24394 - Microsoft Pyright (Microsoft's Static Python Type Checker): pyright#11605 - eBay NuRaft (C++ Raft Distributed Consensus Engine): NuRaft#663 Clear 3-Tier Pricing & 14-Day 100% Refund Guarantee - 1st PR Free Evaluation Try ($0): Test it on 1 pull request out of the box—no credit card required. - ZeroSlop Pro ($29/mo): 1 Developer Seat (unlimited repositories & PRs), backed by a 2-week (14-day) 100% full refund guarantee, no questions asked. - ZeroSlop Enterprise ($1,999/mo): Per 100 Active Developers ($19.99/dev in 100-seat blocks). - GitHub Marketplace: github.com/marketplace/actions/zeroslop-by-ribbsa… - Wall of Merges & Live Demo: ribbsaetersystems.com
How we structured a 3-tier self-serve GitHub Actions SaaS ($0 1-PR try / $29 Pro / $1,999 Enterprise) against $41,760/yr legacy AI review bots We built ZeroSlop because cloud AI code review bots charge $24–$40 per seat ($41,760/yr for 174 engineers), yet their own published case studies admit median PR merge time barely moves (16.8 hours before to 16.7 hours after — a 6-minute gain). Instead of another noisy cloud chatbot leaving 30 comments on your pull request, ZeroSlop is a fast 3.8-second pre-flight gate inside GitHub Actions that strips AI wrapper bloat and blocks deleted tests before human review—with zero source code ever leaving your runner. Proven Upstream Merges (The Wall of Proof) We don't ask anyone to take our word for it—our track record of merged pull requests says everything. Every single one of these upstream C++, Rust, and C pull requests passed through ZeroSlop and was merged by core maintainers: - LLVM (Core C++ Compiler Infrastructure for Apple, Google & Rust): llvm-project#215076 - Meta Velox (Meta's C++ Vectorized Database Engine): velox#18529 - Valkey (Linux Foundation C In-Memory Datastore, successor to Redis): valkey#4424 - DuckDB (High-Performance C++ Analytical OLAP Database): duckdb#26443 - Apache DataFusion (Extensible Rust Arrow Query Engine): datafusion#24394 - Microsoft Pyright (Microsoft's Static Python Type Checker): pyright#11605 - eBay NuRaft (C++ Raft Distributed Consensus Engine): NuRaft#663 Clear 3-Tier Pricing & 14-Day 100% Refund Guarantee - 1st PR Free Evaluation Try ($0): Test it on 1 pull request out of the box—no credit card required. - ZeroSlop Pro ($29/mo): 1 Developer Seat (unlimited repositories & PRs), backed by a 2-week (14-day) 100% full refund guarantee, no questions asked. - ZeroSlop Enterprise ($1,999/mo): Per 100 Active Developers ($19.99/dev in 100-seat blocks). - GitHub Marketplace: github.com/marketplace/actions/zeroslop-by-ribbsa… - Wall of Merges & Live Demo: ribbsaetersystems.com
Open-source maintainers are drowning in AI slop PRs. We built a deterministic GitHub Action (ZeroSlop) that halts deleted tests, blocks 12 CWE flaws, and delivers 100% green merges in 3.8s >TL;DR: Open-source maintainers and senior engineers are spending 15–20 hours a week babysitting AI-generated pull requests that look plausible on the surface, but sneak in deleted regression tests, memory leaks, and subtle security vulnerabilities. Chatty review bots like CodeRabbit just make the problem worse by dumping 30+ hallucinated LLM comments on every diff. We built ZeroSlop™—a deterministic, zero-retention GitHub Actions PR Airlock that runs in 3.8s inside runner RAM, strips 94% of hallucinated wrapper bloat, and delivers 100% green, merge-ready PRs. The Real Problem: Stacking More LLMs Doesn't Fix AI Noise Have you seen what first-generation AI review bots actually dump onto Pull Requests? 40 unsolicited bot comments on a 50-line diff. Hallucinated syntax nitpicks. And 15 paragraphs of LLM noise that maintainers have to manually dismiss just to hit "Merge." Now competitors brag about fine-tuning smaller models like Nemotron to "route" reviews. Let's be completely real: adding another LLM with fewer capabilities does not solve an AI noise problem. It simply stacks more latency, more hallucinations, and more noise on top of your repository. AI coding agents (Cursor, Copilot, Claude Code) are generating code 10x faster than humans can review it. In the rush, they quietly delete regression tests to make CI pass, introduce missing error boundaries, and bloat the AST. Engineering teams don't need another chatbot. They need an Invariant Airlock. How ZeroSlop Enforces in 3.8 Seconds (Runner RAM) ZeroSlop doesn't write chatty essays. It runs deterministically inside your GitHub Actions runner: 1. Mathematical Test Shield: Calculates net_assertion_delta. If an AI agent quietly deleted regression tests to manufacture a passing CI run, it hard-blocks the merge instantly. 2. 12-Shield CWE Security Engine: Static static analysis blocking SQLi (CWE-89), hardcoded secrets (CWE-798), IDOR (CWE-639), command injection (CWE-78), SSRF (CWE-918), CI workflow Pwn Requests (CWE-426), and AI Slopsquatting (CWE-829). 3. AST Diff Minimization: Strips up to 94% of hallucinated adapter bloat, dead wrappers, and mock utilities down to the surgical invariant change. 4. Single In-Place Status Card: Zero comment spam across force-pushes. Exactly ONE clean, self-updating status card with an embedded Mermaid architecture graph. 5. Zero Data Retention: Runs entirely in ephemeral runner RAM. Proprietary code is never stored, retained, or used to train models. Battle-Tested Upstream Proof ZeroSlop is powered by the exact invariant engine used to ship merged, 100% green pull requests into the world's most unforgiving upstream production codebases: |Upstream Repository|Upstream Status|Architectural Invariant Delivered| |:-|:-|:-| |LLVM Compiler Infrastructure|MERGED<br/>PR #215076|X86 SelectionDAG non-power-of-two vector division lowering| |Meta Velox Execution Engine|MERGED<br/>PR #18529|TopNRowNumber in-memory rank ordering & partition stream parity| |DuckDB Analytical Engine|MERGED<br/>PR #26443|PIVOT IN list zero-argument row() expression preservation| |Apache DataFusion|MERGED<br/>PR #24394|Execution-layer schema conformance across physical execution| |Microsoft Pyright|MERGED<br/>PR #11605|Inherited asymmetric descriptor detection across effective MRO| |Tokio Async Runtime|110/110 GREEN<br/>PR #8595|Paused time wake tracking & zero-syscall timer queue preservation| |OpenAI Triton GPU Compiler|CHECKS GREEN<br/>PR #12221|Nanobind static build free-threaded Python PEP 703 GIL slot| |ClickHouse OLAP Engine|174/176 GREEN<br/>PR #114954|Bounds validation on parallel replicas query plan limit pipeline| Try It Free Every repository gets 1 complimentary PR audit free out of the box with zero configuration required. - GitHub Marketplace Action: github.com/marketplace/actions/zeroslop-by-ribbsa… - Live Interactive In-Browser Simulator: ribbsaetersystems.com - 12-Shield Security Brief: ribbsaetersystems.com Pro seats ($29/mo) include our 14-Day Sovereign Performance Guarantee: slash your team's PR review cycle time by 60%, or get an immediate 100% refund. Would love maintainer feedback on what other invariant checks you’d like to see added!
ZeroSlop (v1.2.0): Deterministic GitHub Actions PR Airlock that renders a Mermaid Blast Radius PR card, strips 94% of AI diff bloat, and blocks 12 CWE security flaws >GitHub Repository: https://github.com/patrickswedish/zeroslop-action GitHub Marketplace: https://github.com/marketplace/actions/zeroslop-by-ribbsaeter-systems Live Interactive PR Simulator: https://www.ribbsaetersystems.com/zeroslop 12-Shield CWE Security Matrix: https://www.ribbsaetersystems.com/zeroslop/security ZeroSlop is a zero-retention, deterministic GitHub Actions PR Airlock built from our upstream systems workflow (LLVM #215076, Meta Velox #18529, DuckDB #26443, Apache DataFusion #24394, Microsoft Pyright #11605, Tokio #8595, ClickHouse #114954, OpenAI Triton #12221) that strips 94% of AI wrapper bloat, enforces 12 CWE security shields, and gets Pull Requests merged 100% green in minutes. What Problem Does zeroslop-action Solve? - 94% Diff Bloat: Strips multi-file hallucinated agent wrappers down to the surgical 4-to-15 line root-cause invariant. - Test Protection: Hard-blocks unauthorized regression test deletions and weakened assertions (net_assertion_delta < 0). - CI Runner Flake Isolation: Separates real contributor code defects from upstream GitHub Actions runner flakes. - 12-Shield CWE Security: Deterministically intercepts SQLi (CWE-89), IDOR (CWE-639), hardcoded secrets (CWE-798), SSRF (CWE-918), CI/CD workflow Pwn Requests (CWE-426), and package Slopsquatting (CWE-829). - Single In-Place PR Card: Maintains strictly 1 self-updating comment across force-pushes with an embedded Mermaid Blast Radius dependency diagram. - Zero Code Retention: Runs ephemerally in volatile Node 20 runner RAM.
We packaged our upstream LLVM, Meta Velox & DuckDB workflow into a GitHub Action (ZeroSlop) that strips 94% of AI diff bloat and blocks 12 CWE security flaws >TL;DR: AI coding agents increased raw PR volume by 300%, yet overall merge velocity has dropped by 40% because Senior Engineers spend 15–20 hrs/week untangling 200-line hallucinated wrappers, deleted tests, and subtle CWE security flaws. At Ribbsaeter Systems, we packaged our upstream systems workflow (LLVM #215076, Meta Velox #18529, DuckDB #26443, Apache DataFusion #24394, Microsoft Pyright #11605, Tokio #8595, ClickHouse #114954, OpenAI Triton #12221) into a deterministic GitHub Actions PR Airlock called ZeroSlop (GitHub Marketplace · Security Brief) that strips 94% of AI bloat, enforces 12 CWE security shields, and gets PRs merged 100% green in minutes. 1. Why We Built ZeroSlop: The 2026 PR Review Bottleneck When shipping patches into upstream systems repos like LLVM, Meta Velox, DuckDB, or Tokio, a real production fix is 4 to 15 lines of surgical invariant changes paired with a deterministic regression test that passes CI on the first push. Inside commercial teams using Cursor, Copilot, or Claude Code, the opposite happens: - 94% Diff Bloat: An agent generates a 240-line wrapper class instead of fixing a 4-line boundary condition (try the live interactive diff simulator here). - Weakened Tests: When a test fails, the agent silently deletes or loosens the assertion to manufacture a green check. - Noisy LLM Review Bots: First-gen AI review bots run another LLM to dump 25 conversational comments on top of a bloated PR—without shrinking the diff or getting the PR merged. 2. Conversational AI Review Bots vs. ZeroSlop End-to-End PR Airlock |Engineering Dimension|Conversational AI Review Bots|ZeroSlop End-to-End PR Airlock| |:-|:-|:-| |Primary Objective|Leave 25+ chatty LLM comments on a bloated diff|Get PRs merged 100% green in minutes (not 4 days)| |Diff Minimization|Reviews 200 lines of hallucinated wrappers as-is|Strips up to 94% of AI bloat down to the core AST invariant| |CI Failure Triage|Ignores red CI checks or hallucinates fixes|Separates contributor code defects from runner flakes| |Test Integrity|Misses deleted or weakened test assertions|Hard-blocks unauthorized regression test deletions| |PR Comment Hygiene|Spams new threads on every git push|Strictly 1 self-updating PR Verdict Card + Mermaid Graph| |Security & Privacy|Cloud LLM retention & external SaaS portals|12-Shield CWE Airlock + Zero Code Retention| 3. The 12-Shield Deterministic CWE Security Airlock (ZS-SEC-001 – ZS-SEC-012) Every Pull Request is audited in milliseconds across 8 merge-readiness signals (0–100) and 12 deterministic CWE security shields before a human reviewer is ever pinged: |Shield ID|Security Vector & CWE / CVE Mapping|What ZeroSlop Security Deterministically Intercepts| |:-|:-|:-| |ZS-SEC-001|SQL & Vector Injection (CWE-89 / CWE-943)|Unparameterized SQL/NoSQL/Vector template literals & raw string concatenation| |ZS-SEC-002|Hardcoded & Seeded Secrets (CWE-798 / CWE-256)|API keys, JWT secrets, seeded credentials & plaintext password comparisons| |ZS-SEC-003|Unbound Identity / IDOR (CWE-639 / CWE-862)|Trusting req.body.userId instead of verified session/JWT caller identity| |ZS-SEC-004|Command & Eval Injection (CWE-78 / CWE-94)|Unsanitized child_process.exec, os.system, subprocess(shell=True), eval()| |ZS-SEC-005|Silent Exception Swallowing (CWE-390)|Empty catch () {} blocks and swallowed promise rejections that hide crashes| |ZS-SEC-006|XSS & Unsanitized DOM (CWE-79)|dangerouslySetInnerHTML, .innerHTML =, and unescaped template rendering| |ZS-SEC-007|Unsafe Deserialization (CWE-502 / CWE-22)|pickle.loads, yaml.load without SafeLoader, and ../ path traversal| |ZS-SEC-008|SSRF & Disabled TLS (CWE-918 / CWE-295)|Unvalidated fetch(req.body.url) & NODE_TLS_REJECT_UNAUTHORIZED = '0'| |ZS-SEC-009|CI/CD "Pwn Requests" (CWE-426 / CWE-250)|pull_request_target fork checkouts & ${{ github.event.pull_request.title }} injection| |ZS-SEC-010|AI Package "Slopsquatting" (CWE-829 / CWE-494)|Hallucinated npm/PyPI/Cargo packages, unpinned Git deps & curl | bash| |ZS-SEC-011|Resource Exhaustion / DoS (CWE-400 / CWE-770)|while(true) without exit bounds, leaked setInterval, unbounded Promise.all| |ZS-SEC-012|Trojan Source & Jailbreaks (CVE-2021-42574)|Invisible Unicode Bidi characters & hidden IGNORE PREVIOUS INSTRUCTIONS comments| 4. Battle-Tested Upstream Proof (Click Any PR to Inspect the Diff) ZeroSlop codifies the exact zero-slop invariant discipline used to ship merged & 100% green Pull Requests across tier-1 systems repositories: |Upstream Repository|Live Pull Request|Status|Core Systems Invariant & Regression Coverage| |:-|:-|:-|:-| |LLVM Compiler Infrastructure|llvm/llvm-project#215076|MERGED|Fixed Waitcnt brackets in GFX12+ AMDGPUAsmPrinter with MC regression tests| |Meta Velox Execution Engine|facebookincubator/velox#18529|MERGED|Aligned DecimalUtil::toByteArray overflow bounds & restored kMaxDecimalBytes| |DuckDB Analytical Engine|duckdb/duckdb#26443|MERGED|Fixed pragma_storage_info internal compressed segment type alignment| |Apache DataFusion Query Engine|apache/datafusion#24394|MERGED|Fixed lead()/lag() window frame boundaries with non-NULL default values| |Microsoft Pyright Type Checker|microsoft/pyright#11605|MERGED|Resolved type narrowing across nested loop closures & pattern guards| |Tokio Async Rust Runtime|tokio-rs/tokio#8595|100% GREEN|Preserved Coop budget accounting across JoinSet & FuturesUnordered| |ClickHouse OLAP DBMS|ClickHouse/ClickHouse#114954|100% GREEN|Fixed StorageURL distributed cluster header forwarding & serialization| |OpenAI Triton GPU Compiler|triton-lang/triton#12221|100% GREEN|Enforced strict tensor descriptor stride & block shape invariants| |Cloudflare Agents SDK|cloudflare/agents#1248|MERGED|Hardened state synchronization & WebSocket hibernation lifecycle invariants| 5. Live Interactive PR Simulator & GitHub Action Links - Interactive PR Simulator & Product Hub: ribbsaetersystems.com - 12-Shield CWE & Zero-Retention Security Matrix: ribbsaetersystems.com - GitHub Marketplace: github.com/marketplace/actions/zeroslop-by-ribbsa…
Apache Arrow and Apache DataFusion Now List ArrowMetal: Arrow Compute and a DataFusion Optimizer Rule on the Apple Silicon GPU ArrowMetal is now on two Apache project pages, and I am going to enjoy this one. Since 8 September it has been on Apache Arrow's Powered By page, alongside the libraries I learned Arrow from. Since today, 9 October, it is on Apache DataFusion's integrations list in the user guide, as datafusion-arrowmetal. Both went in the way every entry on those lists does: a pull request, read by the project's committers, merged. Here is what the two listings mean, and what the listed thing does today, because 0.5.0 shipped yesterday and both entries describe it. 2 Apache project pages list ArrowMetal: Arrow's Powered By since 8 September, DataFusion's integrations since 9 October. The data crosses 3 Arrow C interfaces (C Data, C Stream, C Device Data) with no copy either way. The DataFusion crate's differential grid runs 13,632 query pairs, rule off against rule on, with 0 mismatches. A sort of 50,000,000 permuted int64 keys runs in 43 ms through the MetalEngine against 242 ms for Polars 2.0.0's own collect(), frames equal. Apple M4 Max, arrowmetal 0.5.0 from PyPI, best of 5, 9 October 2026. ArrowMetal is my Apache-2.0 Arrow compute library for the Apple silicon GPU: Arrow buffers stay in unified memory, the GPU reads them in place, and the results come back as Arrow. It has a Polars engine, a DataFusion optimizer rule, DuckDB extensions and seven language bindings. The launch post covers the design; this one is about the listings. Two listings, and what they mean Arrow's Powered By page is the Arrow website's list of projects and products using Apache Arrow; DataFusion's is the section of its user guide for community projects that extend DataFusion. Getting on either means a committer read a description of your work, judged it accurate and on topic, and put their name on the merge. For Arrow that took one morning; for DataFusion, six days, one committer's careful review and a maintainer's merge, with a thank-you at the end. Those are the two communities ArrowMetal is built on, and being read and accepted by them is the part I will remember. | the two listings | Apache Arrow's Powered By page and Apache DataFusion's integrations list | |---|---| | What the page is | Arrow: Powered By, the Arrow website's list of projects and products using Apache Arrow, on the page that also carries the trademark guidance. DataFusion: the “Integrations and Extensions” section of the user guide, “community projects that extend DataFusion or provide integrations with other systems”. | | How an entry gets in | A pull request, read by the project's committers. Arrow: arrow-site #815, opened and merged on the morning of 8 September after one committer's approval. DataFusion: datafusion #26004, opened 3 October, one committer's review, merged by a maintainer on 9 October. | | What the entry says | Arrow: “Apache Arrow compute on Apple silicon GPUs through Metal. Arrow buffers live in unified memory the GPU reads in place, so the kernels … run without a copy in either direction …” and the seven bindings. DataFusion: “A physical optimizer rule that runs sorts and measured group-bys on the Apple silicon GPU through Metal.” | | What it means | The project's committers read a description of the thing, judged it accurate and on topic, and merged it under their name. For a one-person project built on their work, that is the review that counts. | | What it does not mean | An endorsement, a code review, or any status inside the Foundation. ArrowMetal is an independent Apache-2.0 project, happily so; “Apache Arrow” and “Apache DataFusion” are trademarks of the Apache Software Foundation, which is why the name has no “Apache” in it. | The two entries as they read on 9 October 2026; dates from the pull requests arrow-site #815 and datafusion #26004. It matters in a practical way too: these are the pages people read when they ask “what runs on Arrow?” or “what extends DataFusion?”, and an accurate line there reaches the right readers. Both descriptions name what the software does and nothing it does not; that is the bar this post holds to. In fairness to both projects, a listing is not an endorsement or a code review, and ArrowMetal remains independent; the table has the texts and dates. The Arrow side: what the entry describes The Powered By entry says the kernels run on Arrow buffers in unified memory without a copy either way, and that data crosses through Arrow's C Data, C Stream and C Device Data interfaces. That is the architecture: a pyarrow array, a Polars Series or a DataFusion RecordBatch is handed over as Arrow, the GPU works on the same bytes, and Arrow comes back. Here is the README's first thing to run, from the 0.5.0 wheel on PyPI, then a 50,000,000-row sort through the Polars engine on Polars 2.0.0, with the engine's report. $ pip install "arrowmetal[polars]" # 0.5.0 from PyPI; polars 2.0.0 comes with the extra import pyarrow as pa, arrowmetal as am # the README's first thing to run, as installed col = am.array(pa.array([1, None, 3, 40])) # a pyarrow array crosses in through the C Data interface big = col.filter_where(">", 2) # GPU print(big.sum(), big.to_arrow()) # 43 [3, 40] keys = am.array(pa.array([0, 1, 0, 2], pa.int32())) print(keys.group_by(3).sum(col).to_arrow()) # [4, null, 40] import polars as pl # 50,000,000 permuted int64 keys, a Float64 payload engine = am.MetalEngine() # the measured defaults; on 2.0.0, the 2.x table print(engine.explain(df.lazy().sort("k"))) MetalEngine report for explain (polars 2.0.0, IR (15, 2)) metal: Sort#1 [Sort > DataFrameScan] over 50,000,000 rows rule: 50,000,000 input rows is at or above the 10,000,000-row crossover for sort (engine table, Benchmarks/results/polars_engine_crossover_2026-10-08-polars2.csv) # best of 5: polars 2.0.0 collect() 242.37 ms; collect(engine=engine) 43.16 ms; frames equal am.array(pa.array(...)) is the C Data interface at work; nothing is converted. The report names the rule, the row count and the crossover file, so the decision is checkable. The 43 against 242 milliseconds is one M4 Max sorting 50,000,000 permuted int64 keys with a Float64 payload, best of 5, against Polars 2.0.0's own collect(), frames equal. The engine takes the sort from 10,000,000 rows up, where the crossover table puts it; below that, Polars sorts. The DataFusion side: what the entry describes The integrations entry calls the crate a physical optimizer rule that runs sorts and measured group-bys on the Apple silicon GPU through Metal. Such a rule is a hook DataFusion gives to anyone: it sees the physical plan and may replace nodes. This one replaces a full ORDER BY, and the aggregate shapes its measured table takes, with a node that runs the plan on the GPU and hands back the RecordBatches DataFusion expected. The SQL and every other node are untouched. The crate's quickstart, built from crates.io 0.5.0 against DataFusion 55.1.0 on this M4 Max: the sort it takes, the two it leaves. # Cargo.toml: datafusion = "=55.1.0", datafusion-arrowmetal = "0.5.0" let rule = ArrowMetalRule::new(ArrowMetalConfig::default()); // full sorts from 250,000 rows let ctx = session_context(SessionConfig::new().with_target_partitions(4), rule.clone()); ctx.register_table("sales", mem_table)?; // 1,000,000 rows: region, amount, name == SELECT name, region, amount FROM sales ORDER BY amount DESC NULLS LAST, name ProjectionExec: expr=[name@2 as name, region@0 as region, amount@1 as amount] MetalExec: sort=[amount DESC NULLS LAST, name ASC NULLS LAST] DataSourceExec: partitions=1, partition_sizes=[1] TAKEN SortExec -- input rows 1000000 (exact) vs min_rows 250000 1000000 rows; the first 3: n001395 | 0 | 1429.4285714285713... == SELECT name, amount FROM sales ORDER BY amount DESC NULLS LAST LIMIT 3 LEFT SortExec: TopK(fetch=3) -- top-k (sort with fetch 3) disabled in config == SELECT region, count(*) AS n, avg(amount) AS mean FROM sales GROUP BY region ORDER BY region LEFT AggregateExec: gby=[region], aggr=[count(Int64(1)), avg(sales.amount)] -- count + sum_avg_f64 over 1 i64 key: the measured table takes it at no group count and size (results/datafusion_groupby_sweep_2026-10-01.csv) 13 rows; the first 3: 0 | 76924 | 714.7095888611585... The numbers behind “measured” are in the DataFusion post and docs/DATAFUSION.md: on the M4 Max, full sorts from 250,000 to 50,000,000 rows ran 6.9x to 28.8x faster than DataFusion alone across four key types, 50,000,000 rows sorted in 126 CPU-milliseconds against 6,104 on 16 partitions, and 13,632 query pairs, rule off and on, matched with 0 mismatches. Measured on 0.4.0 in late September; 0.5.0 changes the aggregate side, below, not the sort numbers. What 0.5.0 changed since those two posts Yesterday's release is the one both entries now describe, and it is the best one yet. Four lines from the release notes matter here. | ArrowMetal 0.5.0, 8 October 2026 | what changed since the two posts the listings point at | |---|---| | Polars 2.0.0 | The Polars engine runs on 2.0.0 as on 1.44, with one measured crossover table per Polars major. On 2.0.0 the default takes 52 of 107 benchmarked queries at 50,000,000 rows on the M4 Max, every one 1.35x to 7.94x faster than the faster Polars 2.0.0 engine, none behind; 12 are handed back. The pip extra is now polars>=1.44,<2.1; the re-measurement post has the cases. | | Integer mean in one pass | A mean over an integer column runs in one pass in the Polars and plan engines, same bits as before: two int32 keys, 200 groups, 20.7 to 14.3 ms at 50,000,000 rows on polars 2.0.0, per the release notes. | | DataFusion crate | datafusion-arrowmetal 0.5.0 on DataFusion 55.1: full ORDER BY sorts, and by default the aggregate shapes its measured table takes: count(*) and DISTINCT over two int32 keys, integer MIN/MAX over two integer keys or one int64 key, on a MemTable of 50,000,000 rows or more. Hash joins are translated; the measured join table takes none yet. | | Build targets | DuckDB extensions built and tested against DuckDB 1.5.6; macOS 14 or later on Apple silicon; the Rust crate's build script stops the build on any other target. | From the v0.5.0 release notes, the CHANGELOG and the crate README at the tag. Every number is one Apple M4 Max. The Polars line closes a caveat from Wednesday's post, which said the 2.0.0 support was on main, unreleased. As of 0.5.0 it is the wheel on PyPI, which is why the card above needed only a fresh venv. Install - Python: pip install "arrowmetal[polars]" installs 0.5.0 and a Polars from 1.44 to 2.0; python -m arrowmetal.polars_engine check prints the Polars, the IR and the table in force. Without the extra, pyarrow is the only dependency. - Rust: datafusion = "=55.1.0" and datafusion-arrowmetal = "0.5.0" in Cargo.toml; the crate links the Metal library through ARROWMETAL_LIB or the copy the Python wheel installs. docs.rs and docs/RUST.md have the finding rules. - macOS 14 or later on Apple silicon only. Every number here is one M4 Max on AC power; python -m arrowmetal.bench measures yours in under 30 s. ArrowMetal is an independent Apache-2.0 project; Apache Arrow and Apache DataFusion are trademarks of the Apache Software Foundation, whose projects I use and build on every day. My thanks to the committers on both sides who read the entries and let them in. Two gates, one small craft, a harbour I am glad to be moored in. FAQ: the listings, Apache Arrow, Apache DataFusion and the Apple silicon GPU Is ArrowMetal an Apache project? No, and proudly independent: an Apache-2.0 project that implements Apache Arrow and extends Apache DataFusion. Both listings are entries on community-maintained lists, read and merged by the projects' committers; they are a welcome, not an endorsement. Does Apache DataFusion ship the GPU rule? No. datafusion-arrowmetal is a separate crate on crates.io, pinned to DataFusion 55.1.0. You register its rule on a SessionContext; the SQL and every other plan node stay DataFusion's. Which Macs does it run on? Apple silicon on macOS 14 or later. The Python wheel carries the Metal library; the Rust crate finds it through ARROWMETAL_LIB or the wheel's copy. The build script stops on any other target. Are the answers the same as the CPU's? The differential grid runs 13,632 DataFusion query pairs, rule off and on, with 0 mismatches; every Polars benchmark query is checked equal to Polars' own frame. Float64 runs through a software IEEE-754 path, bit-exact with the CPU.