[ANN] AppleNeuralEngine.jl: Julia on the Apple Neural Engine
Every Apple Silicon Mac has a Neural Engine (ANE), an fp16 accelerator that Apple exposes only through Core ML. AppleNeuralEngine.jl drives it directly from Julia through ObjectiveC.jl, as Metal.jl does for the GPU, with no native code to build. You write ordinary Julia, and it is traced into an ANE program:
using AppleNeuralEngine
silu(z) = z / (1 + exp(-z))
function ffn(x) # LLaMA-style feed-forward block: RMSNorm, SwiGLU, residual
n = x ./ sqrt.(sum(abs2, x; dims=2) ./ size(x, 2) .+ 1f-5) .* g'
return x .+ (silu.(n * W1') .* (n * W3')) * W2'
end
x = randn(Float16, 256, 512)
ane = compile(ffn, x) # trace + compile
y = ane(x) # runs on the Neural Engine
Results on an M2 Max
16-core ANE, macOS 27.0.1, Julia 1.13.3, 8 BLAS threads. The same Julia function runs on each device; ANE times include copying inputs and outputs.
| workload | ANE fp16 ms | GFLOP/s | Accelerate fp32 ms | GFLOP/s | OpenBLAS fp32 ms | GFLOP/s | rel. error |
|---|--:|--:|--:|--:|--:|--:|--:|
| FFN block D=512 H=1536 T=256 | 0.261 | 4629 | 2.646 | 457 | 3.866 | 312 | 6.5e-04 |
| x*W 256×1024 · 1024×1024 | 0.148 | 3628 | 0.210 | 2553 | 0.728 | 737 | 3.3e-04 |
| x*W 1024×1024 · 1024×1024 | 0.366 | 5859 | 0.791 | 2715 | 2.839 | 756 | 3.3e-04 |
| x*W 2048×2048 · 2048×2048 | 1.774 | 9682 | 7.223 | 2378 | 24.062 | 714 | 3.6e-04 |
ANE times include copying inputs and outputs. The ANE computes in fp16 and the CPU in fp32, so each runs at its native precision.
Reproduce it and post your numbers
git clone https://github.com/pitsianis/AppleNeuralEngine.jl
cd AppleNeuralEngine.jl
julia --project=benchmark -e 'using Pkg; Pkg.instantiate()'
julia --project=benchmark benchmark/run.jl # prints the table above for your Mac
julia --project=benchmark tools/probe_ops.jl # checks every op on your Mac
The benchmark prints a Markdown table with your chip and macOS version: please paste it in this thread. So far everything has been tested only with Julia 1.13.3 on a M2 Max.
What runs on the ANE
On the M2 Max, 129 of 167 probed MIL operations compile and give correct results. Works: elementwise arithmetic, exp/log/sqrt/tanh/erf, activations (relu, gelu, silu, softmax, …), reductions, convolutions (strided, depthwise, int8 weights), pooling, matmul/linear/einsum, layer/batch/instance norm, scaled dot-product attention, reshape/transpose/slice/concat/split/tile, upsampling. Rejected by the ANE compiler: sin/cos/tan, pad, gather/scatter, cumsum, topk/argsort/argmax, LSTM/GRU/RNN, random ops, fp32 compute. Full table: docs/src/op_support.md.
PRs are very welcome if you get a rejected op to work, or if tools/probe_ops.jl gives a different table on a newer M chip or macOS version. The probe rewrites docs/src/op_support.md, so the PR is that file.
Caveats: it uses a private, undocumented API that any macOS update can break. It computes in fp16 only, and each call costs about 0.1–0.2 ms, so it pays off for matrix–matrix work. It is not registered yet.
Credits: this builds on the reverse engineering of the private ANE runtime in maderix/ANE, and on ObjectiveC.jl and Metal.jl. Most of the code was written together with Claude, Anthropic's AI assistant, in Claude Code.
Repository: pitsianis/AppleNeuralEngine.jl
Edit: Corrected a bug that underestimated the performance of Accelerate and OpenBLAS.
15 likes
Mentions
[ANN] AppleNeuralEngine.jl: Julia on the Apple Neural Engine Every Apple Silicon Mac has a Neural Engine (ANE), an fp16 accelerator that Apple exposes only through Core ML. AppleNeuralEngine.jl drives it directly from Julia through ObjectiveC.jl, as Metal.jl does for the GPU, with no native code to build. You write ordinary Julia, and it is traced into an ANE program: using AppleNeuralEngine silu(z) = z / (1 + exp(-z)) function ffn(x) # LLaMA-style feed-forward block: RMSNorm, SwiGLU, residual n = x ./ sqrt.(sum(abs2, x; dims=2) ./ size(x, 2) .+ 1f-5) .* g' return x .+ (silu.(n * W1') .* (n * W3')) * W2' end x = randn(Float16, 256, 512) ane = compile(ffn, x) # trace + compile y = ane(x) # runs on the Neural Engine Results on an M2 Max 16-core ANE, macOS 27.0.1, Julia 1.13.3, 8 BLAS threads. The same Julia function runs on each device; ANE times include copying inputs and outputs. | workload | ANE fp16 ms | GFLOP/s | Accelerate fp32 ms | GFLOP/s | OpenBLAS fp32 ms | GFLOP/s | rel. error | |---|--:|--:|--:|--:|--:|--:|--:| | FFN block D=512 H=1536 T=256 | 0.261 | 4629 | 2.646 | 457 | 3.866 | 312 | 6.5e-04 | | x*W 256×1024 · 1024×1024 | 0.148 | 3628 | 0.210 | 2553 | 0.728 | 737 | 3.3e-04 | | x*W 1024×1024 · 1024×1024 | 0.366 | 5859 | 0.791 | 2715 | 2.839 | 756 | 3.3e-04 | | x*W 2048×2048 · 2048×2048 | 1.774 | 9682 | 7.223 | 2378 | 24.062 | 714 | 3.6e-04 | ANE times include copying inputs and outputs. The ANE computes in fp16 and the CPU in fp32, so each runs at its native precision. Reproduce it and post your numbers git clone https://github.com/pitsianis/AppleNeuralEngine.jl cd AppleNeuralEngine.jl julia --project=benchmark -e 'using Pkg; Pkg.instantiate()' julia --project=benchmark benchmark/run.jl # prints the table above for your Mac julia --project=benchmark tools/probe_ops.jl # checks every op on your Mac The benchmark prints a Markdown table with your chip and macOS version: please paste it in this thread. So far everything has been tested only with Julia 1.13.3 on a M2 Max. What runs on the ANE On the M2 Max, 129 of 167 probed MIL operations compile and give correct results. Works: elementwise arithmetic, exp/log/sqrt/tanh/erf, activations (relu, gelu, silu, softmax, …), reductions, convolutions (strided, depthwise, int8 weights), pooling, matmul/linear/einsum, layer/batch/instance norm, scaled dot-product attention, reshape/transpose/slice/concat/split/tile, upsampling. Rejected by the ANE compiler: sin/cos/tan, pad, gather/scatter, cumsum, topk/argsort/argmax, LSTM/GRU/RNN, random ops, fp32 compute. Full table: docs/src/op_support.md. PRs are very welcome if you get a rejected op to work, or if tools/probe_ops.jl gives a different table on a newer M chip or macOS version. The probe rewrites docs/src/op_support.md, so the PR is that file. Caveats: it uses a private, undocumented API that any macOS update can break. It computes in fp16 only, and each call costs about 0.1–0.2 ms, so it pays off for matrix–matrix work. It is not registered yet. Credits: this builds on the reverse engineering of the private ANE runtime in maderix/ANE, and on ObjectiveC.jl and Metal.jl. Most of the code was written together with Claude, Anthropic's AI assistant, in Claude Code. Repository: pitsianis/AppleNeuralEngine.jl Edit: Corrected a bug that underestimated the performance of Accelerate and OpenBLAS.
The repo is INSANE 👇 maderix/ANE
@VictorTaelin maybe take the core idea which was there in Bend1 and which Mithril proves is still competetive enough 👀 maderix/mithril
Mithril: A programming language built on interaction nets Inspired by Victor Taelin 's Bend/HVM thesis for Interaction Nets and his Bend2 work , I built Mithril , an experimental python syntax programming language. It achieves near native speed on many workloads (mentioned in paper) It compiles interaction nets to native code and spreads work across idle cores: lock-free, deterministic parallelism on x86, CUDA and Apple Silicon. The core bet I'm making : Interaction nets reductions at runtime can be slow but if we pay the same cost at compile time and unroll a task graph as much as possible before native lowering, the resultant program can run at near native speeds. Mithril programs are aimed to be deterministic and fast, you should get the same bit accurate results regardless , the execution be it on CPUs, GPUs or any other accelerator. It's built using AI? Yes . Does it invalidate the core idea? no IMO 😊 Mithril Paper Repo