oraios

Do Coding Agents Really Fail at Business Logic? What the Evidence Shows Why I started this I set out to test a suspicion, not to sell a tool: coding agents fail at bug fixing because they do not understand the business behind the code. After a day of digging, my answer is that the problem looks real but narrower than I assumed, and the obvious fixes are weaker than they sound. I use Claude Code with multiple agents and subagents, and I have designed my own software factory, EGM. Bug fixing is where I get the most frustrated. Agents misread the real cause of a bug. They ship patches that are technically valid but break a business rule. They miss what a change does elsewhere in the app. And context gets lost when work is delegated to subagents. One honest note: I could not recall a specific bug to use as a case study, so I will not invent one. Everything below is what other people measured or reported, plus what I could and could not verify. I did the research with two AI assistants, Claude and ChatGPT, and asked them to challenge each other's findings. What the large studies show The big studies agree that agents do worse on bug fixes, but none of them shows that missing business logic is the reason. - Merge rates by task type. Where Do AI Coding Agents Fail? (MSR 2026) studied about 33,000 pull requests from five agents: Codex, Copilot, Devin, Cursor and Claude Code. Documentation and CI tasks merged most often. Performance and bug-fix tasks merged least. A third-party summary of the paper puts bug fixes at 64% and documentation at 84%. - Why PRs were rejected. In a sample of 600 rejected PRs, the most common cause was reviewer abandonment (38%), then duplicate or unwanted PRs. Problems on the agent's own side were about 2%. These figures also come from the same third-party summary, so check them against the paper. - Fixes only. A second study looked at bug-fix PRs alone and found that 46.41% of fixes from Copilot, Devin, Cursor and Claude were rejected. It sorts the reasons into four groups: wrong implementation, failing CI, agent capability limits and low priority. - After merging. A study of 1,210 merged bug-fix PRs ran SonarQube before and after each merge. Code smells dominated. Real bugs were rarer but often severe, and the most common one was a function called with the wrong number of arguments. The authors conclude that merge success is not a reliable quality signal. Caveats: no human baseline, and 949 of the 1,210 PRs came from Codex but only 15 from Claude Code. What none of these measure is whether a fix violated a business rule. The datasets label PRs by task type, CI result and review outcome, and there is no business-rule category. That is not proof the problem is small. It means the evidence is indirect, and I did not find a benchmark built to measure it (my search could have missed one). What developers report Practitioners describe the same pattern again and again, but these are anecdotes, not measurements. - One commenter in a Hacker News thread says an agent understands a repo functionally, not from the business side. - In Ask HN: I hate coding agents, a developer who writes docs, AGENTS.md files and skills says the agent does about 80% right, and its wrong assumptions make the last 20% hard to fix. - Another thread describes agents fixing each bug while breaking other things, because nobody asked why the code looked the way it did. - A third says coding assistants bury requirement gaps inside large diffs instead of escalating them to a person. There are counterpoints. One commenter finds agents very good at debugging complex bugs that span many files, and weaker at deciding whether a change belongs in the architecture or only locally. Others blame long, degraded context, as in this Claude Code issue. These comments are self-selected and unmeasured. I treat them as hypotheses to test, not as proof. What existing tools cover Existing tools handle navigation, impact analysis and workflow well, but none I found measures whether a patch respects business rules. | Tool | What it does | Business-rule angle | Adoption signal | | --- | --- | --- | --- | | Repo Mind | Repository understanding through retrieval and graph summaries; a GitHub Next research project, listed as completed | None | Its own tests: small gain (see below) | | GroundGraph | Rust CLI and MCP server linking requirements, docs, tests and code; AI proposes business-logic candidates, a human confirms | Closest design: human-confirmed rules | Very early; the repo page showed 0 stars when I looked | | GitNexus | Code knowledge graph with blast-radius analysis over MCP | None, structural only | About 42,000 stars (June 2026); license reported as PolyForm Noncommercial | | Serena | Symbol-level search and editing over language servers | None | About 25,000 stars, 170+ contributors, MIT | | Unblocked | Commercial engine over code, PRs, Slack, Jira, Confluence | Cross-source decision context | Vendor claims only | | CodeAlive | Commercial graph-based context engine over MCP | Code only | Vendor and comparison-site numbers | | Superpowers | Skills-based workflow: spec, plan, tests, subagent review including a spec-compliance check | Process, not knowledge | Reported at hundreds of thousands of stars; I have not verified it | The Repo Mind results are the most useful data point here. On SWE-bench Pro its context raised the resolution rate from 44.97% to 46.09%, a gain of about one point. The gain was larger for consistency: pass^3 rose by 6.7 points. But the agents called its structural tools in only about 8% of Pro tasks and 18% of Verified tasks, and wrapping those tools in a subagent did not help. Good context does not help if the agent does not use it. The narrowest gap I can name The one thing I could not find a tool for is capturing business rules that exist only in people's heads, in a form a patch can be checked against. More context alone does not seem to be the answer. A controlled preprint (single author, 70 multi-file SWE-bench Verified tasks, Python, single-shot, so a narrow test) found that natural-language summaries of code answered 4 of 45 behavioral questions, against 27 of 45 for the source itself. Compact structural views of the surrounding code resolved no more issues than dropping that code. Instruction files are no sure fix either: in a study of 15,549 agentic PRs from 148 projects, 27.7% of projects raised their merge rate by 20% or more after adding them, while 26.35% saw it fall. If a rule is not in the code, the tests or the docs, retrieval cannot surface it. GroundGraph's design, where the AI proposes candidate rules and a human confirms them, is the closest I found. Even so, its own documentation notes that test verification is declared, not proven. My hypothesis, and it is only that: a rule captured as an executable check at the moment a human discovers it, for example during a bug fix, may work better than prose docs. The evidence against it is real. Agents rarely call external tools, and people have to write and confirm the rules. What I could not verify, and what would change my mind My current position is that the problem exists, but the evidence that a separate tool would solve it is insufficient. What I could not verify: - Several leads from my starting notes did not turn up under those titles, so I left them out: a Google post on agent distraction, a paper called "Finding Needles in Code Haystacks", and specific Codex issue numbers. - I did not review Reddit, and I did not read GroundGraph's own benchmark results or its issue tracker. - Star counts and licenses come from third-party pages and may be out of date. What would change my mind: 1. A log of real failures showing that business-rule violations are a small share next to wrong root causes. Then the gap is not worth a tool. 2. A benchmark showing that an external rule store raises fix correctness without adding review work. Then it is worth building. 3. Evidence that developers will not write or confirm rules. Then adoption is the blocker. Help me test this If you have a real case where an agent's bug fix was technically right but wrong for the business, I would like to hear it. Reply with: 1. What the bug was. 2. What the agent changed. 3. Which rule the fix violated. 4. Where that rule lived: code, a test, a doc, or nowhere. 5. How you found out. 6. Which agent and model you used. If enough cases come in, I will share a tally of where the missing rules were. Sources - Where Do AI Coding Agents Fail? - Understanding the Rejection of Fixes Generated by Agentic Pull Requests - Beyond Bug Fixes: Post-Merge Code Quality Issues in Agent-Generated Pull Requests - What Context Does a Coding Agent Actually Need to Act? - Repo Mind, GitHub Next - GroundGraph, GitNexus, Serena, Superpowers, Unblocked, CodeAlive - Hacker News threads: 50119991, 48844345, 50059523, 46866481, 47594340 - Claude Code issue 35296

GPL-3.0, 249 contributors. Helps anyone running agents on big, multi-file codebases. Honest limit: some languages need extra installs, and on tiny edits your agent's built-ins are just as good. #OpenSource oraios/serena

3 free tools stop Claude Code relearning your repo #Shorts Claude Code keeps relearning your repo. Three free tools help. Serena: jump to a function by name oraios/serena graphify: your repo as a queryable graph Graphify-Labs/graphify headroom: shrinks tool output headroomlabs-ai/headroom The 42% fewer tokens exploring a codebase is headroom's own benchmark (its README), not an independent test. #claudecode #aitools #devtools #opensource #Shorts

I ran ripgrep, Serena, claude-context and my own tool on the same repo There are a lot of ways to let a coding agent find code now. Plain grep, a language server behind MCP, a vector database, a local index. Every README says its way is the right one, so I wanted to see them do the same job on the same code. I build one of these tools, Semlith, so treat my bias accordingly. That's also why I'm showing the questions it got wrong. The setup is Flask 3.1.3 on an M1 MacBook Air with 8 GB. Versions are ripgrep 15.2.0, Serena 2.1.0, claude-context 0.1.15 (with Milvus 3.0.3 and Ollama running nomic-embed-text) and semlith 0.50.0. git clone --depth 1 --branch 3.1.3 https://github.com/pallets/flask && cd flask The four approaches in one table | | ripgrep | Serena | claude-context | Semlith | |---|---|---|---|---| | What it is | Text search, no index | Language servers behind MCP | Embeddings in Milvus | Local index, embeddings plus keyword plus code graph | | Finds code by meaning | No | No | Yes | Yes | | Edits code | No | Yes, by symbol | No | No, read only | | Setup on Flask | Nothing | 30.9 s to index | About 21 min (CPU in Docker) | 84.3 s to index | | Moving parts | One binary | Python env plus language servers | Docker, Milvus, Ollama, npx | One binary | | Reads PDF, Word, Excel | No | No | No | Yes | The rest of this post is how I got those numbers. ripgrep, when you know the name When the agent already knows the identifier, nothing beats grep. rg -n max_content_length src src/flask/wrappers.py:55: _max_content_length: int | None = None src/flask/wrappers.py:60: def max_content_length(self) -> int | None: src/flask/wrappers.py:80: if self._max_content_length is not None: src/flask/wrappers.py:81: return self._max_content_length src/flask/wrappers.py:84: return super().max_content_length src/flask/wrappers.py:88: @max_content_length.setter src/flask/wrappers.py:89: def max_content_length(self, value: int | None) -> None: src/flask/wrappers.py:90: self._max_content_length = value Eight lines, all of them right, in milliseconds. Now ask the same thing the way a person would, without the name. An agent turns "limit how big an uploaded request body can be" into words and greps for them. rg -n -i "upload|big" src src/flask/cli.py:297: a bigger role. Typically it's created automatically by the src/flask/helpers.py:542: @app.route("/uploads/<path:name>") src/flask/helpers.py:545: app.config['UPLOAD_FOLDER'], name, as_attachment=True src/flask/helpers.py:549: files or uploads. Uses :func:`~werkzeug.security.safe_join` to None of these is the answer, because wrappers.py never says "upload" or "big". This is where an agent starts guessing more words and opening more files. Serena, when you want a language server Serena gives the agent symbol tools backed by real language servers, over 40 languages. Since version 2 it shows up as a REPL, so the agent writes a little Python against Serena's API. uv venv -p 3.11 venv-serena uv pip install -p venv-serena "serena-agent @ git+https://github.com/oraios/serena@v2.1.0" serena project index . Indexing took 30.9 seconds. Given the name, it's precise. s.lsp.find_symbol("max_content_length") # [{"name_path": "Request/max_content_length", "kind": "Method", # "relative_path": "src/flask/wrappers.py", "body_location": {"start_line": 87, ...}}] References come back in about 2 seconds and they come from the language server, so they are real references, not text matches. Line numbers count from zero, which caught me out the first time. Without the name though, Serena falls back to a regex pattern search, and that returns the same lines grep did. s.fs.search_for_pattern("upload|big", relative_path="src") The thing Serena does that none of the others do is edit. Replace a function body, insert after a symbol, rename across files. If your agent spends most of its time changing code it already found, that's a real advantage. claude-context, run fully local claude-context is Zilliz's MCP server. It chunks code, embeds it and stores the vectors in Milvus, with hybrid search on top. By default it wants an OpenAI key for embeddings and a Zilliz Cloud key for the database. It also supports a fully local setup, so that's what I ran. curl -sSL https://github.com/milvus-io/milvus/releases/download/v3.0.3/milvus-standalone-docker-compose.yaml -o docker-compose.yaml docker compose up -d docker run -d --name cc-ollama --cpus=4 -p 127.0.0.1:11434:11434 ollama/ollama:0.40.13 docker exec cc-ollama ollama pull nomic-embed-text EMBEDDING_PROVIDER=Ollama EMBEDDING_MODEL=nomic-embed-text \ OLLAMA_HOST=http://127.0.0.1:11434 MILVUS_ADDRESS=localhost:19530 \ npx -y @zilliz/claude-context-mcp@0.1.15 The images came to 6.14 GB and the first npx start pulled 418 MB of packages. While it worked I watched its connections, and it only talked to Milvus and Ollama on the machine. Two things bit me. On the first run Milvus exited mid-index while Ollama had every CPU, so I capped Ollama at four. The second one is worth knowing if you use it. get_indexing_status said this at 92 seconds. ✅ Codebase '…/flask' is fully indexed and ready for search. 📊 Statistics: 100 files, 100 chunks It wasn't done. The server's own log finished at 1,266.9 seconds with 86 files and 1,837 chunks. If your agent trusts that status and searches early, it searches a partial index without knowing. That 21 minutes isn't a fair speed test either. Docker on a Mac has no GPU access, so Ollama embedded on the CPU. A native Ollama install would be quicker, I just didn't measure it. Semlith, the one I build semlith index . indexed 211 files (2778 chunks, 5 images) in 84.3s — 0 already indexed, 4 skipped (4 empty), 0 removed That count includes the .rst docs, which claude-context skips by default. Serena was quicker to set up here, since it doesn't embed anything. !Horizontal bar chart of the time to index Flask 3.1.3 before the first search. ripgrep needs no index, Serena takes 30.9 seconds, Semlith 84.3 seconds and claude-context 1,266.9 seconds with Ollama embedding on the CPU in Docker. Setup time on the same checkout. Serena was the quickest of the indexed tools, and claude-context's time is CPU-only embedding in Docker. semlith search -k 5 "limit how big an uploaded request body can be" 1. 0.087 vf src/flask/wrappers.py:59-71 2. 0.083 vf src/flask/wrappers.py:91-106 3. 0.077 v docs/web-security.rst:26-38 4. 0.076 vf docs/patterns/fileuploads.rst:135-153 5. 0.072 vf src/flask/wrappers.py:105-123 5 hits in 360.027667ms The first hit is the property. The letters after the score say how it was found, v for the vector list and f for the keyword list. The question everyone got wrong I asked all of them where Flask decides which error handler runs for an exception. The answer is _find_error_handler in src/flask/sansio/app.py. Grep did best. rg -n -i "error handler" src lands on line 826 of that file, the function's docstring, because the question uses the same words the code does. Semlith's top five were all in docs/errorhandling.rst. Those pages explain the order Flask uses, so they answer the question, but they aren't the code. claude-context had the function at rank 8. So "semantic search beats grep" isn't true in general. When the question already uses the code's own words, grep is very hard to beat. Bigger samples Three questions on one repo prove nothing, so here's what I could measure at scale. On SWE-bench Verified, 500 real GitHub issues, the issue text is the query and the gold is every file the fix edits. Semlith had a right file in its top 10 for 76.8% of them, against 47.0% for one ripgrep pass ranked and read in order. That's retrieval only, not whether an agent then fixes the bug. On 20 issues from SWE-bench Lite, every tool got the same checkout and issue text. Serena 1.7.0 finished 16 of 20 (4 timed out) and had the right file in its top 5 for 25.0% of those, against 56.2% for Semlith on the same 16. That run predates Serena's REPL, so 2.1.0 might do differently. claude-context isn't in that comparison yet, so I won't give it a number. And the one that keeps me honest. On 50 questions about a single unfamiliar repo, headless Claude Code on Opus scored 92% with grep alone and 92% with Semlith installed beside it. A strong agent with room to explore gets there with grep. What each one costs before you use it Every MCP server sends its tool definitions at session start, and the agent pays for them on every turn. I counted the tools/list result at about four bytes a token. Serena tools: 2 4544 bytes, about 1136 tokens semlith tools: 8 2668 bytes, about 667 tokens Both are small. Serena keeps the list short with a REPL, then the agent reads 9,292 characters of instructions when it calls initial_instructions. Which I'd pick Honestly they do different jobs, and you can run more than one. - One repo, a strong model, names you mostly know. Grep is enough. - An agent that edits a lot by symbol. Serena. - You already run Milvus or Zilliz Cloud, or want to pick the embedding provider. claude-context. - Many repos, documents next to the code, nothing to sign up for. That's the case I built Semlith for. The full runs with every command are on semlith.com and the benchmark harness is in the repo at semlith/semlith. What are you using for code search with your agents right now, and where does it fall over?

🧠 چطور به AI کمک کنیم بدون خواندن کل مخزن GitHub، پروژه رو بهتر درک کنه؟ ابزار Serena یک MCP متن‌باز و کاربردیه که قابلیت‌های یک IDE رو در اختیار Agentهای کدنویسی مثل Claude Code و Codex قرار می‌ده. 🔗 لینک GitHub: oraios/serena چه کاری انجام می‌ده؟ وقتی یک پروژه رو به Agent می‌دی، Serena بهش کمک می‌کنه ساختار کد رو بررسی کنه و به‌جای خواندن بی‌هدف تمام فایل‌ها، سراغ بخش‌های مرتبط بره: - پیدا کردن توابع، کلاس‌ها و متغیرها (Symbols) - شناسایی ارتباط بین بخش‌های مختلف کد - پیدا کردن محل استفاده از هر تابع یا متغیر - بررسی وابستگی‌ها و ساختار پروژه - ویرایش و بازنویسی کد با درک ساختار واقعی اون چرا مهمه؟ به‌جای اینکه Agent کل مخزن رو یک‌جا وارد Context کنه، می‌تونه اطلاعات موردنیازش رو هدفمند پیدا کنه. این یعنی استفاده بهینه‌تر از Context و کاهش اطلاعات اضافی، به‌خصوص در پروژه‌های بزرگ. Serena خودش مدل هوش مصنوعی نیست؛ از طریق MCP به ابزارهایی مثل Claude Code و Codex متصل می‌شه و امکانات تحلیل و ویرایش کد رو در اختیارشون قرار می‌ده. 📌 اگر با مخزن‌های بزرگ GitHub و ابزارهای کدنویسی AI کار می‌کنی، Serena ارزش امتحان کردن داره.

GPT-6 Astra 的自主循环太猛了！ 把一个 GitHub 仓库交给一个 Astra oraios/serena 一晚上过去，它趁你睡觉就把一天大约 5 小时的工作量干完了。 它先解析整个代码库， 只找出「Astra 现在真正该读的文件、符号和依赖关系」。 不让它一上来就把巨型仓库全读一遍。 光这一步就能省掉大量 Context 浪费。

📌 영상에 나온 도구 링크 모음입니다! 1. 포니테일 (Ponytail) DietrichGebert/ponytail 2. 케이브맨 (Caveman) JuliusBrussee/caveman 3. RTK rtk-ai/rtk 4. 헤드룸 (Headroom) headroomlabs-ai/headroom 5. 컨텍스트 모드 (Context Mode) mksglu/context-mode 6. 세레나 (Serena) oraios/serena 7. 토큰 옵티마이저 (Token Optimizer) alexgreensh/token-optimizer

개발자들이 먼저 깔았다..꼭 알아야 할 토큰 절약 도구 깃허브 TOP 7(깃허브 ★51만) AI 코딩할 때 토큰이 새는 네 군데(쓰는 양·읽는 양·찾아 헤맴·내 설정)와, 그걸 막으려고 개발자들이 만든 깃허브 도구 7개를 정리했습니다. 00:00 오프닝 00:28 포니테일 01:47 케이브맨 02:54 RTK 04:15 헤드룸 05:27 컨텍스트 모드 06:31 세레나 07:39 토큰 옵티마이저 08:39 마무리 📌 영상에 나온 도구 링크 1. 포니테일 (Ponytail) DietrichGebert/ponytail 2. 케이브맨 (Caveman) JuliusBrussee/caveman 3. RTK rtk-ai/rtk 4. 헤드룸 (Headroom) headroomlabs-ai/headroom 5. 컨텍스트 모드 (Context Mode) mksglu/context-mode 6. 세레나 (Serena) oraios/serena 7. 토큰 옵티마이저 (Token Optimizer) alexgreensh/token-optimizer [en-US] Developers' Must-Have Token Saving Tools: Top 7 GitHub Repos (517K Stars) I have summarized the four places where tokens leak when coding with AI (writing volume, reading volume, searching, and my settings), and seven GitHub tools created by developers to prevent them. I have compiled a list of these tools. 00:00 Opening 00:28 Ponytail 01:47 Caveman 02:54 RTK 04:15 Headroom 05:27 Context Mode 06:31 Serena 07:39 Token Optimizer 08:39 Wrap-up 📌 Links to tools featured in the video 1. Ponytail DietrichGebert/ponytail 2. Caveman JuliusBrussee/caveman 3. RTK rtk-ai/rtk 4. Headroom headroomlabs-ai/headroom 5. Context Mode mksglu/context-mode 6. Serena (Serena) oraios/serena 7. Token Optimizer alexgreensh/token-optimizer [id] Alat Penghemat Token Wajib Punya Pilihan Pengembang (Total 517 Ribu Bintang di GitHub) TOP 7 Saya telah merangkum empat tempat di mana token bocor saat melakukan pengkodean dengan AI (volume penulisan, volume pembacaan, pencarian, dan pengaturan saya), dan tujuh alat GitHub yang dibuat oleh pengembang untuk mencegahnya. Saya telah menyusun daftar alat-alat tersebut. 00:00 Pembukaan 00:28 Ponytail 01:47 Caveman 02:54 RTK 04:15 Headroom 05:27 Context Mode 06:31 Serena 07:39 Token Optimizer 08:39 Penutup 📌 Tautan ke alat yang ditampilkan dalam video 1. Ponytail DietrichGebert/ponytail 2. Caveman JuliusBrussee/caveman 3. RTK rtk-ai/rtk 4. Headroom headroomlabs-ai/headroom 5. Context Mode mksglu/context-mode 6. Serena (Serena) oraios/serena 7. Token Pengoptimal alexgreensh/token-optimizer

@goan999999 更新一下提示词： 请阅读 Serena 官方文档 oraios/serena 和 oraios.github.io 一节），然后： 1. 检查是否已安装 uv，没有则按官方方式安装 2. 用 `uv tool install -p 3.13 serena-agent` 安装，运行 `serena init` 验证 3. 优先用 `serena setup codex` 配置；如需手动，只在 ~/.codex/config.toml 里追加 [mcp_servers.serena] 段，保留现有所有 MCP 配置，修改前先备份原文件 4. 按官方文档配置 ~/.codex/hooks.json（activate / remind / reset / cleanup）； 如果已有 hooks.json，合并进去，不要覆盖 5. 不要使用 MCP 市场或旧的 uvx git 安装方式 6. 完成后告诉我改了哪些文件、怎么用 /mcp 验证 而且吧90%的节省量，估计就是博眼球的

@typecraft_dev I've been using serena oraios/serena for this among other things for some time