Laya-CoreML: Open-weight typed decisions on Apple Silicon: Core ML, Neural Engine, zero generated tokens (1.6k stars)
What it does
All text below is quoted directly from the project README.
>Local Laya typed decisions on Apple Core ML and Neural Engine. Validated ports, ~5 ms short decisions on M3 Max, reproducible speed and energy benchmarks.
What else it does:
- Language routing: Router() detects the language, picks a bundle and keeps up to two resident. You can register your own bundles.
- Batches and long documents: predict_batch and predict_long, with answer attribution across windows.
- Confidence gate: min_confidence= marks low-confidence answers for review without changing them.
- Large choice sets: predict_tournament (no embedder) or predict_shortlist with your own embedding function.
- Compute units: compute_units="cpu_gpu" | "cpu_ne" | "cpu" | "all" overrides a bundle's default engine.
- Tools and services: CLI (laya-coreml predict), HTTP (laya-coreml-serve), MCP (laya-coreml-mcp-server), evaluation and calibration, and LangChain/LlamaIndex/CrewAI/Agno integrations, installed as extras.
- Conversion: pip install 'laya-coreml[convert]' then laya-coreml convert exports your own bundle.
>pip install laya-coreml
Repository
mizorewww/laya-coreml
Repository stats
|Metric|Value|
|:-|:-|
|Stars|1,618|
|Forks|139|
|Last release|v0.6.1 (2026-10-29)|
|Languages|Python 99.7%, Rust 0.3%|
|Contributors|3|
|AI contributor accounts (Claude / Codex / Cursor / Copilot)|None|
|AI co-authored commits|Claude: 5|
Main contributor
|Field|Value|
|:-|:-|
|Name|雨夹雪|
|GitHub|@mizorewww|
|Commits to this repo|25 (top contributor)|
|Company / Location|Shanghai,China|
|Followers|321|
──────────────────────────────
📌 If you want to browse more repos like this, we've organized them by category and rankings here: bestgithubrepositories.com
🗂️ We're also archiving repos where possible, so useful projects don't disappear when they're removed from GitHub.
3 points
Sub-5ms Decision Inference on Apple Silicon: Inside laya-coreml
Running classification, routing, and structured decision trees directly on edge hardware often hits a wall: Python runtimes add latency overhead, and massive LLMs are overkill for low-latency triage. laya-coreml bridges that gap by compiling Laya typed decisions to Apple Core ML, leveraging the Apple Neural Engine (ANE) to hit ~5 ms decision latencies on M-series chips.
TL;DR
laya-coreml enables fully local, typed decision model inference on Apple Silicon via Core ML and the Apple Neural Engine. It eliminates cloud API costs and network roundtrips by delivering validated sub-5ms decision paths on an M3 Max with reproducible energy efficiency benchmarks. It is a drop-in solution for edge routing, agentic guardrails, and real-time on-device classification.
---
Key Features & Benchmarks
- Sub-5ms Decision Paths: Validated inference times as low as ~5 ms on Apple M3 Max, fully offloading execution to the Apple Neural Engine (ANE).
- Strictly Typed Outputs: Built around Laya's typed decision framework, ensuring structured, deterministic schema outputs rather than unpredictable free-form text.
- Energy-Efficient Execution: Hardware-accelerated ANE offloading slashes battery consumption compared to running quantization frameworks (like GGUF/llama.cpp) across raw GPU/CPU cores.
- Reproducible Performance: Includes native benchmark harnesses to measure memory pressure, token-to-decision throughput, and wattage consumption across M1 through M3 architectures.
- Zero External Dependencies at Runtime: Compiles down to standard .mlpackage artifacts ready for native Swift, Objective-C, or Python Core ML bindings.
---
Quick Start
Clone the repository and run the validation pipeline directly using Python:
# Clone the repo
git clone https://github.com/mizorewww/laya-coreml.git
cd laya-coreml
# Install required dependencies
pip install -r requirements.txt
coremltools
# Run the standard validation benchmark on your Mac
python benchmark.py --model laya-decision-v1 --device ane
To integrate into your Python pipeline:
import coremltools as ct
from PIL import Image
import numpy as np
# Load the compiled ANE-optimized decision model
model = ct.models.MLModel("models/LayaDecision.mlpackage")
# Execute a typed decision run
inputs = {"feature_vector": np.random.randn(1, 512).astype(np.float32)}
prediction = model.predict(inputs)
print("Predicted Routing Target:", prediction["decision_label"])
---
Why It Matters
If you are building autonomous local agents, audio/video pipelines, or on-device filtering for macOS/iOS apps, passing every intermediate step through a 7B parameter LLM is slow, hot, and battery-draining.
laya-coreml lets you execute fast, typed classification and semantic decisions in parallel on the ANE while your primary GPU remains free for heavy rendering or generation tasks. It is ideal for:
- Client-Side Agent Guardrails: Validate user prompts and tool actions before execution.
- Low-Latency Router Nodes: Direct queries locally to cloud APIs or smaller local SLMs depending on complexity.
- Energy-Conscious Mobile Devs: Build responsive macOS and iOS utilities without spinning up fans or depleting user batteries.
---
🛠️ Recommended AI Stack & Resources
- Cloud GPU Hosting: Scale beyond local testing and spin up cost-effective cloud GPUs for heavy fine-tuning and batch training with RunPod.
- AI Code Editor: Supercharge your development workflow and prototype ML pipelines 10x faster using Cursor.
- Production Database: Handle high-throughput vector retrieval and app persistence with Supabase or Pinecone.
- Global Dev Network: Ensure ultra-fast weights downloads, uninterrupted Hugging Face access, and low-latency cloud server SSH via WD-Gold Network.
- Newsletter Subscription: Keep up with the latest local AI runtimes, optimizations, and architectures—subscribe to Local AI Daily.
- Sponsorship & Partnerships: Want to feature your open-source tool or hardware platform to thousands of AI developers? Contact: [fengdahui195@gmail.com](mailto:fengdahui195@gmail.com)
Mentions
Laya-CoreML: Open-weight typed decisions on Apple Silicon: Core ML, Neural Engine, zero generated tokens (1.6k stars) What it does All text below is quoted directly from the project README. >Local Laya typed decisions on Apple Core ML and Neural Engine. Validated ports, ~5 ms short decisions on M3 Max, reproducible speed and energy benchmarks. What else it does: - Language routing: Router() detects the language, picks a bundle and keeps up to two resident. You can register your own bundles. - Batches and long documents: predict_batch and predict_long, with answer attribution across windows. - Confidence gate: min_confidence= marks low-confidence answers for review without changing them. - Large choice sets: predict_tournament (no embedder) or predict_shortlist with your own embedding function. - Compute units: compute_units="cpu_gpu" | "cpu_ne" | "cpu" | "all" overrides a bundle's default engine. - Tools and services: CLI (laya-coreml predict), HTTP (laya-coreml-serve), MCP (laya-coreml-mcp-server), evaluation and calibration, and LangChain/LlamaIndex/CrewAI/Agno integrations, installed as extras. - Conversion: pip install 'laya-coreml[convert]' then laya-coreml convert exports your own bundle. >pip install laya-coreml Repository mizorewww/laya-coreml Repository stats |Metric|Value| |:-|:-| |Stars|1,618| |Forks|139| |Last release|v0.6.1 (2026-10-29)| |Languages|Python 99.7%, Rust 0.3%| |Contributors|3| |AI contributor accounts (Claude / Codex / Cursor / Copilot)|None| |AI co-authored commits|Claude: 5| Main contributor |Field|Value| |:-|:-| |Name|雨夹雪| |GitHub|@mizorewww| |Commits to this repo|25 (top contributor)| |Company / Location|Shanghai,China| |Followers|321| ────────────────────────────── 📌 If you want to browse more repos like this, we've organized them by category and rankings here: bestgithubrepositories.com 🗂️ We're also archiving repos where possible, so useful projects don't disappear when they're removed from GitHub.
Sub-5ms Decision Inference on Apple Silicon: Inside laya-coreml Running classification, routing, and structured decision trees directly on edge hardware often hits a wall: Python runtimes add latency overhead, and massive LLMs are overkill for low-latency triage. laya-coreml bridges that gap by compiling Laya typed decisions to Apple Core ML, leveraging the Apple Neural Engine (ANE) to hit ~5 ms decision latencies on M-series chips. TL;DR laya-coreml enables fully local, typed decision model inference on Apple Silicon via Core ML and the Apple Neural Engine. It eliminates cloud API costs and network roundtrips by delivering validated sub-5ms decision paths on an M3 Max with reproducible energy efficiency benchmarks. It is a drop-in solution for edge routing, agentic guardrails, and real-time on-device classification. --- Key Features & Benchmarks - Sub-5ms Decision Paths: Validated inference times as low as ~5 ms on Apple M3 Max, fully offloading execution to the Apple Neural Engine (ANE). - Strictly Typed Outputs: Built around Laya's typed decision framework, ensuring structured, deterministic schema outputs rather than unpredictable free-form text. - Energy-Efficient Execution: Hardware-accelerated ANE offloading slashes battery consumption compared to running quantization frameworks (like GGUF/llama.cpp) across raw GPU/CPU cores. - Reproducible Performance: Includes native benchmark harnesses to measure memory pressure, token-to-decision throughput, and wattage consumption across M1 through M3 architectures. - Zero External Dependencies at Runtime: Compiles down to standard .mlpackage artifacts ready for native Swift, Objective-C, or Python Core ML bindings. --- Quick Start Clone the repository and run the validation pipeline directly using Python: # Clone the repo git clone https://github.com/mizorewww/laya-coreml.git cd laya-coreml # Install required dependencies pip install -r requirements.txt coremltools # Run the standard validation benchmark on your Mac python benchmark.py --model laya-decision-v1 --device ane To integrate into your Python pipeline: import coremltools as ct from PIL import Image import numpy as np # Load the compiled ANE-optimized decision model model = ct.models.MLModel("models/LayaDecision.mlpackage") # Execute a typed decision run inputs = {"feature_vector": np.random.randn(1, 512).astype(np.float32)} prediction = model.predict(inputs) print("Predicted Routing Target:", prediction["decision_label"]) --- Why It Matters If you are building autonomous local agents, audio/video pipelines, or on-device filtering for macOS/iOS apps, passing every intermediate step through a 7B parameter LLM is slow, hot, and battery-draining. laya-coreml lets you execute fast, typed classification and semantic decisions in parallel on the ANE while your primary GPU remains free for heavy rendering or generation tasks. It is ideal for: - Client-Side Agent Guardrails: Validate user prompts and tool actions before execution. - Low-Latency Router Nodes: Direct queries locally to cloud APIs or smaller local SLMs depending on complexity. - Energy-Conscious Mobile Devs: Build responsive macOS and iOS utilities without spinning up fans or depleting user batteries. --- 🛠️ Recommended AI Stack & Resources - Cloud GPU Hosting: Scale beyond local testing and spin up cost-effective cloud GPUs for heavy fine-tuning and batch training with RunPod. - AI Code Editor: Supercharge your development workflow and prototype ML pipelines 10x faster using Cursor. - Production Database: Handle high-throughput vector retrieval and app persistence with Supabase or Pinecone. - Global Dev Network: Ensure ultra-fast weights downloads, uninterrupted Hugging Face access, and low-latency cloud server SSH via WD-Gold Network. - Newsletter Subscription: Keep up with the latest local AI runtimes, optimizations, and architectures—subscribe to Local AI Daily. - Sponsorship & Partnerships: Want to feature your open-source tool or hardware platform to thousands of AI developers? Contact: [fengdahui195@gmail.com](mailto:fengdahui195@gmail.com)