Running 4B+ models on Apple's Neural Engine

TL;DR:

• 4B and 9B models on ANE, bypassing CoreML entirely
• 11.66 tok/s single-stream, 28.62 tok/s with four concurrent streams — GPU and CPU idle
• Same 16-core ANE on every M1 through M5, regardless of SKU
• Bottleneck is dispatch overhead, not compute or bandwidth
• Code: ane.cpp + field guide

GPUs are great for local inference if the goal is raw tokens per second on one chat, but they’re a poor fit if what you actually want is many small inference streams running quietly in the background for hours — on an M3 Max, sustained GPU inference pulls 40–60W from the GPU cluster alone¹, and that adds up fast on battery.

Jo is a personal AI that runs on a hybrid stack — important things never leave your Mac, and sensitive external tasks are handled on your own cloud machine. The local macOS side handles on-device context: browser history clustering, note indexing, local tool dispatch. Today those workloads either wake the GPU or wait for the cloud, both of which have tradeoffs in power draw and latency respectively.

What we really want is an autonomous ambient layer on your machine, things organizing by themselves and context building up quietly. Getting that to work without draining the battery means finding hardware that can carry parallel inference at low power — and if something can do that, the local side of Jo becomes a thing that runs continuously instead of intermittently. That’s the target.

The Neural Engine

Every Apple Silicon chip since M1 has a 16-core Neural Engine² — same core count on the base Air as on the Max, same MIL programs through the same _ANEInMemoryModel interface, for five years running. An M1 Air and an M4 Max both have identical ANE blocks. That kind of consistency across SKUs is unusual for Apple, and it means we can target the ANE and know what to expect regardless of what machine a user has.

Apple exposes the ANE through CoreML³, but CoreML doesn’t give you direct control over dispatch scheduling or IOSurface tensor layout — both of which matter for autoregressive transformer decode. ane.cpp bypasses CoreML and talks to the ANE directly through Apple’s private AppleNeuralEngine.framework⁴.

TOPS have doubled from M3 to M4, but we’re nowhere near saturating even the M1’s 11 TOPS — our bottleneck is dispatch overhead, not raw compute. That makes the TOPS progression less interesting than the fact that the dispatch architecture hasn’t changed. Base-tier memory bandwidth has also grown substantially, and the effect on ANE work is less straightforward.

What we built

ane.cpp runs Qwen3.5 end-to-end on the Neural Engine, building on maderix⁴’s reverse-engineering of Apple’s private ANE framework and johnmai-dev⁵’s first working LLM runtime on ANE. Current M3 Max numbers (warmed 5-run medians, 500 generated tokens, Qwen thinking defaults⁶):

Concurrency

Single-stream generation isn’t fast. But single-stream speed isn’t really the point — what’s more interesting is concurrency. During token generation most of the ANE’s capacity goes unused, which means we can run multiple streams at once.

The tables below are from warmed serve-mode runs with Qwen’s thinking defaults (--enable-thinking, top_p=0.95, top_k=20, presence_penalty=1.5, repeat_penalty=1.0, temp=1.0), 8 requests at 100 generated tokens each, 3 repeats per concurrency level, --sessions 4.

Aggregate generation throughput across all active streams:

Per-stream average at each concurrency level:

Four concurrent inference streams, locally, one process, on the neural engine, with the GPU and CPU staying free for everything else. If you just look at tokens-per-joule, the neural engine looks roughly comparable to GPU/CPU work in our measurements so far. But combine that with slower token generation and it means you can run this in parallel with multiple jobs for a much longer time.

We haven’t really had the opportunity before to run many little agent streams locally, continuously, without turning the machine into a space heater. There have been some reports that M4s and M5s can more aggressively speed things up using runtime paths⁷ we don’t currently have access to on M3, but we haven’t verified that yet.

How the runtime works

ane.cpp compiles each weight matrix into an ANE convolution kernel⁸ with weights baked in at compile time, and for each generated token the runtime bounces back and forth between ANE and CPU — ANE handles the big matrix multiplications, CPU handles the sequential stuff like attention and normalization that ANE can’t do well in fp16⁹.

The key operations are the fused kernels — oproj_add combines the output projection with a residual add in a single ANE dispatch¹⁰, and ffn_resadd does the same for the SwiGLU FFN block¹⁰. This matters because decode on this hardware is dispatch-bound, with a rough fit of latency ≈ 119µs + bytes / 78 GB/s⁹ and about 113 dispatches per generated token where the fixed overhead dominates. Cutting even a few dispatches per layer adds up quickly across 32 layers.

Each weight matrix compiles into a MIL convolution program⁸ that gets loaded onto the ANE via Apple’s private AppleNeuralEngine.framework⁴, and a persistent compile cache means the first-run compilation cost only needs to be paid once per kernel variant.

Prefill

For prefill (processing the prompt), we pack multiple tokens into the W-lane dimension of the ANE’s IOSurface tensors¹¹. The ANE requires a minimum spatial width of 32, so for single-token decode most of that width is wasted. During prefill we can batch 4 tokens at once across those lanes, which gets us roughly 2x the throughput compared to generation.

More on the hardware

The ANE uses memory differently than a GPU. Weights don’t stream from DRAM during inference — they’re baked into ANE kernels at compile time. So the classic GPU bottleneck (bandwidth-bound weight loading) doesn’t apply here. What does flow through memory is activation data: every time the runtime bounces between ANE and CPU — roughly 113 times per generated token — it writes results to an IOSurface and reads them back through DMA. Our measurements give an effective per-dispatch transfer rate of about 78 GB/s, with the bulk of each dispatch’s latency (~119µs) coming from fixed overhead rather than the transfer itself.

For single-stream decode, that means bandwidth barely registers. Activation tensors are small at 4B scale (a few KB per dispatch), and the fixed dispatch overhead dominates so thoroughly that the M3 Pro’s bandwidth regression — 154 vs the M2 Pro’s 205 GB/s, after Apple trimmed the memory bus from 256-bit to 192-bit — produced no measurable change in ANE throughput. Single-stream decode is dispatch-bound, and no amount of extra bandwidth changes that.

But the workload we actually care about isn’t single-stream. Four concurrent sessions generating tokens simultaneously means four times the activation traffic sharing the same memory bus. During W-lane batched prefill, each dispatch carries 4× the data. And if future runtime improvements chip away at that 119µs dispatch overhead — which is the single biggest thing standing between ANE inference and meaningfully higher throughput — the bandwidth term becomes the next ceiling. At that point, the difference between a base-tier M1 at 68 GB/s and a base-tier M5 at 154 GB/s starts determining how many concurrent streams can run before they contend with each other.

All our concurrency measurements are on M3 Max, which has 307+ GB/s of headroom. Whether a base-chip M1 or M5 would show different scaling is an open question — but it’s the right question, because the workload we’re building toward (multiple always-on agent streams on a MacBook Air) is exactly the scenario where base-tier bandwidth would be the constraint.

What’s next

The immediate priority is getting this into Jo — wiring ane.cpp as the backend for local indexing, background summarization, and the other ambient workloads that currently wake the GPU or round-trip to the cloud. That’s the whole reason we went down this rabbit hole, and the concurrency numbers are good enough to start building on.

Beyond that, the things we’re most interested in:

Newer chips. All of our measurements are on M3 Max, and several dead ends — runtime-weight convolutions, _ANEClient direct evaluation, multi-procedure MIL chaining — have external reports suggesting they may work on M4 or M5. The M4 doubled ANE TOPS from 18 to 38, and the M5 introduced per-GPU-core Neural Accelerators and a higher-bandwidth memory path. We don’t know yet whether those translate into meaningfully faster dispatch or just more headroom we can’t saturate, but finding out is near the top of the list.

Smaller models. A 1–2B dense model that’s good enough for classification, routing, and summarization would be substantially faster on ANE — fewer layers means fewer dispatches per token, which is exactly where our bottleneck is. We currently support Qwen3.5 at 4B and 9B; adding smaller variants or other architectures is straightforward.

Rigorous energy measurement. The energy story is central to the thesis but we haven’t published proper numbers yet. We have powermetrics-based tooling in the repo (scripts/energy_benchmark.sh) that measures per-subsystem power draw during inference, and the preliminary picture is encouraging. We just need to run it systematically across workloads and concurrency levels and publish the results.

More model architectures. ane.cpp currently only runs Qwen-family models. The compilation and dispatch infrastructure is generic enough that adding other dense transformer architectures (Llama, Gemma, Phi) should be mostly a matter of wiring up the layer configs and any architecture-specific attention patterns.

The optimizations that survived are the boring ones — fused kernels¹⁰, W-lane batching¹¹, chunked FFN, and int8 weight quantization via constexpr_blockwise_shift_scale — and plenty of ideas didn’t: ANE decode attention, speculative decode / self-draft, ANE RMSNorm fusion mega-kernels, cross-layer norm fusion, 3-runtime-input kernels, packed-slice workarounds, INT4, and runtime-weight convolutions all failed on M3 Max. The companion field guide¹² records each dead end alongside wins in a karpathy/autoresearch-style loop: write a focused test, measure it, keep or discard, repeat.

Repos

• ane.cpp
• apple-neural-engine-field-guide

Both repos are MIT-licensed. We’d love contributions, especially from anyone with M4 or M5 hardware.

Acknowledgments

This work wouldn’t exist without maderix, who reverse-engineered Apple’s private ANE framework and made direct hardware access possible, and johnmai-dev, who built the first working LLM runtime on top of it. ane.cpp is a fork of johnmai-dev’s project. Thanks also to thebasedcapital for exploring _ANEClient direct evaluation and multi-procedure MIL paths, and to the Qwen team for the excellent open-weight models we run on this hardware.