overmind-core

agent 越跑越贵，效果却不见涨？ overmind 是 agent 持续改进平台：线上 trace 变成数据集、评测、微调，模型可下载、可回滚。 对已有线上 agent 的团队来说，用真实 trace 改进比凭感觉调提示词强。 你会先拿它改进哪个 agent，是客服还是编码？ #Agent #开源 overmind-core/overmind

Fine-tuned 9B and 12B open models vs GPT-5.6 models 3 tasks: big wins on contracts and biomedical, a tie on aviation Disclosure: I work at Overmind. We open-sourced our training platform last week (platform AGPL-3.0, SDK and CLI MIT), and these are the benchmarks behind it, including the one that came out a tie. Our bet is that most agent tasks don't need a model that knows everything. They need one that does a single job well, and if your agent already runs in production, its traces are the training data. We trained open-weight models with SFT on three tasks and ran them against a frontier flagship using the same prompt. | Task | Metric | Fine-tuned open model | GPT-5.6 | |---|---|---|---| | Contract clauses (CUAD, 1,639 paired Qs) | F1 | 91.6% (Qwen3.5-9B) | 82.0% (Terra) | | | Exact quote match | 59.6% | 8.5% | | | Cost per 1,000 Qs | $1.03 | $20.94 (Terra), $2.15 (Luna) | | BioRED relation extraction | F1 | 54.9% (Qwen3.5-9B) | 14.1% (Terra) | | | Claimed a relation that isn't there | 6.6% | 48.2% | | NASA ASRS synopses (500 reports) | Same meaning as analyst | 0.657 (Gemma 12B) | 0.655 (Luna) | | | Invented a detail | 13.4% (Gemma E4B) | 33.4% (Luna) | Things worth knowing before you judge it: - On contracts, the base model scored 80.4% F1 and 4.2% exact match before training, so the gain comes from training and not from a lucky base model. - Aviation is a tie on meaning. The win there is fewer invented details and shorter output. - 54.9% F1 on BioRED is far from solved. It's just much better than prompting a generalist. - BioRED used a Qwen 9B. The best aviation model was 12B. How it works: - Traces come in through the Python SDK, any OTel exporter, or an import from Langfuse, LangSmith, Braintrust or Galileo - The traces you pick become training and eval datasets - Prompts and models are scored against what you run today - Open models are fine-tuned with LoRA or full SFT, and you keep the weights - The whole loop runs from Claude Code, Cursor, Codex or OpenCode through MCP Limits: - Self-hosting is docker compose up, but training and serving run on Modal and the judges call OpenRouter, so you need OpenRouter, Modal, AWS and Hugging Face keys. - SFT only. No RL yet. Repo: overmind-core/overmind Methodology: overmindlab.ai If you've moved an agent task off a frontier API onto a fine-tuned model, what broke first: data quality, evals or serving?

repo → overmind-core/overmind Shoutout to @OvermindLab for building this and making it open-source for the community 🤗 Don't forget to drop a ★! t.co

why pay for a big model when your agent keeps doing the same job. i’d try a smaller model trained to find customer problems and quote their feedback accurately. check this out: overmind-core/overmind @OvermindLab lets you train models on your agent’s data and own the weights. in their contract test, 1,000 questions cost $1.03 vs $20.94 on a frontier model. about 20x cheaper.

MODEL TRAINING Fine-tune an open model on your own data. Overmind handles training and deployment. You own the weights, with the choice to host on Overmind or run it yourself. Start free: console.overmindlab.ai Star the repo: overmind-core/overmind

Overmind turns anyone into an AI lab. • Builds a context graph from code and traces • Curates training and eval datasets • Evaluates prompts and models • Trains smaller, specialized models overmind-core/overmind t.co