LongMemEval and LoCoMo leaderboard (Oct 2026): every published AI memory benchmark score for Mem0, Zep, Honcho, Mastra, Supermemory, Letta and Hindsight, with the model and setup behind each
The short version: We collected every LongMemEval and LoCoMo score we could source, and wrote down what each one was measured with: the answering model, the grader, how many questions, and whether the answering model could read the original conversation. Three things stand out. The answering model often moves a score more than the memory system does. Nearly every number is self-reported, and the few independent reruns land well below the claims. And the "full context" baselines that systems claim to beat range from 55% to 93% across the two benchmarks, depending on the model and prompt.
This is a companion to our explainers on LongMemEval and LoCoMo. Corrections are very welcome. If a row is wrong or missing, comment with a link and we'll update the post.
LongMemEval (S split, 500 questions)
"Full history in the answer step" means the model answering the question could read the original conversation, not only what the memory system stored.
|System|Score|Answering model|Grader|Full history in the answer step?|Measured by|Source|
|:-|:-|:-|:-|:-|:-|:-|
|No memory: full history pasted in|60.6%|GPT-4o|GPT-4o|Yes (all ~115k tokens)|LongMemEval authors, 2024|paper, Fig. 3b|
|No memory: full history pasted in|60.2% / 55.4%|GPT-4o / GPT-4o-mini|GPT-4o|Yes|Zep, 2025|Zep paper, Table 2|
|No memory: full history pasted in|62.6%|Claude Haiku 4.5|not stated|Yes|Honcho, 2025|honcho.dev|
|Zep (2025)|71.2% / 63.8%|GPT-4o / GPT-4o-mini|GPT-4o|No|Zep|Zep paper, Table 2|
|Supermemory|81.6% / 84.6% / 85.2%|GPT-4o / GPT-5 / Gemini 3 Pro|not stated|not stated|Supermemory|as reprinted in Hindsight's table|
|EmergenceMem "Simple"|82.4%|GPT-4o|not stated|not stated|Emergence|as reprinted in Mastra's table|
|EmergenceMem "Internal"|86.0%|GPT-4o|not stated|not stated|Emergence|same (closed, not reproducible)|
|Hindsight|83.6% / 89.0% / 91.4%|GPT-OSS 20B / GPT-OSS 120B / Gemini 3 Pro|not stated|not stated|Hindsight|hindsight-benchmarks|
|Mastra Observational Memory|84.2% / 89.2% / 93.3% / 94.9%|GPT-4o / Gemini 3 Flash / Gemini 3 Pro / GPT-5-mini|GPT-4o|A compressed version of all of it|Mastra, Feb 2026|mastra.ai|
|Zep (2026)|90.2% (451/500)|GPT-5.4, medium reasoning|GPT-5.4|No (median context 4,408 tokens)|Zep|getzep.com|
|Honcho|90.4% (452/500) / 92.6% (463/500)|Claude Haiku 4.5 / Gemini 3 Pro|GPT-4o, official prompt|No (history injection set to 0 in the run file)|Honcho, Dec 2025|honcho.dev, run files|
|Mem0 (2026 algorithm)|94.4% (in the page summary only)|not stated|not stated|not stated|Mem0, Oct 2026|mem0.ai|
For reference, the score when the model is handed only the few conversations that matter ("oracle"): GPT-4o 87.0% (paper), Claude Haiku 4.5 89.2% (Honcho's run files).
LoCoMo (1,540 questions, adversarial excluded, unless noted)
|System|Score|Answering model|Grader|Full conversation in the answer step?|Measured by|Source|
|:-|:-|:-|:-|:-|:-|:-|
|No memory: full conversation pasted in|72.9%|GPT-4o-mini|not named|Yes|Mem0, 2025|Mem0 paper|
|No memory: full conversation pasted in|83.1% (a)|Claude Haiku 4.5|GPT-4o-mini|Yes|Honcho, 2025|run file|
|No memory: full conversation pasted in|92.6%|GPT-4.1-mini, step-by-step prompt|not stated|Yes|LoCoMo audit|locomo-audit|
|OpenAI memory|52.9%|GPT-4o-mini|not named|No|Mem0, 2025|Mem0 paper|
|LangMem|58.1%|GPT-4o-mini|not named|No|Mem0, 2025|same|
|Zep (as run by Mem0)|66.0%|GPT-4o-mini|not named|No|Mem0, 2025|same (disputed by Zep, below)|
|Mem0 / Mem0 with graph (2025)|66.9% / 68.4%|GPT-4o-mini|not named|No|Mem0, 2025|same|
|Letta (agent with file search)|74.0%|GPT-4o-mini|not stated|Searchable as a file|Letta, Aug 2025|letta.com|
|Zep (2025, its own rerun)|75.1%|not stated in what we read|not stated|No|Zep, 2025|Zep blog|
|Memobase v0.0.37|75.8%|not stated|not stated|not stated|Memobase|as reprinted in Hindsight's table|
|Hindsight|83.2% / 85.7% / 89.6%|GPT-OSS 20B / GPT-OSS 120B / Gemini 3 Pro|not stated|not stated|Hindsight|hindsight-benchmarks|
|Zep (2026, auto search)|86.5%|GPT-5.4|GPT-5.4|No|Zep|getzep.com|
|Honcho|89.9% (b); 89.5% = 1,378/1,540 per question|Claude Haiku 4.5|GPT-4o-mini|Up to 32,768 tokens of it (see below)|Honcho, Dec 2025|run files|
|Backboard|90.1%|not stated|GPT-4.1|not stated|Backboard, 2025|backboard.io, code|
|EverMemOS|92.3% claimed|not stated|not stated|not stated|EverMemOS|audit; outside rerun below|
|Mem0 (2026 algorithm)|92.5% (1,540 questions)|not stated|not stated|not stated|Mem0, Oct 2026|mem0.ai|
|Zep (2026)|94.7% (1,459/1,540)|GPT-5.4, medium reasoning|GPT-5.4|No (median context 5,760 tokens)|Zep|getzep.com|
(a) Our count from Honcho's published per-question baseline file, on the same 1,540 questions. Their evals page lists this baseline as 75.6%, but that run includes the 446 adversarial questions (1,986 in all), while Honcho's own 89.9% excludes them. Their blog post gives 83.9%. (b) The average of the ten conversations' scores. Counted per question, it's 89.5%.
The LoCoMo audit puts the best possible honest score at about 93.6%, because 99 of the 1,540 answers in the key (6.4%) are wrong. A score above that is only possible if the grader passed answers on questions whose key is wrong, so it says as much about the grader as about the system.
What the tables show
The answering model moves the number more than anything else. Mastra's own memory goes from 84.2% to 94.9% on LongMemEval just by swapping GPT-4o for GPT-5-mini. Hindsight goes from 83.6% to 91.4%, and Supermemory from 81.6% to 85.2%. When two systems are compared on different models, most of the gap may be the model.
The same system's score changes a lot between versions. Zep went from 71.2% on LongMemEval in 2025 to 90.2% in 2026, and on LoCoMo Mem0 went from 66.9% to 92.5%. Some of that is better memory. Some is the newer answering and grading models (Zep's 2026 numbers use GPT-5.4 for both). The pages don't separate the two.
"Beats full context" depends on which full context. On LoCoMo, pasting in the whole conversation scores 72.9% with GPT-4o-mini and 92.6% with GPT-4.1-mini and a step-by-step prompt. LoCoMo conversations are only ~18k tokens, so with a modern model, the full-context baseline sits close to the 93.6% ceiling.
Check the question count. Honcho's LoCoMo comparison is a good example. Its system score (1,540 questions, adversarial excluded) and its baseline (1,986, adversarial included) were measured over different question sets. Over the same 1,540 questions, the baseline is 83.1%, not 75.6%. So, counted per question, the system adds about 6 points over Haiku reading everything, not 14.
Check whether the answering step can read the original conversation. Honcho's LoCoMo runner puts each whole conversation into one Honcho session (tests/bench/locomo.py), and the run file sets the history added to the answer prompt to 32,768 tokens (metadata.dialectic_settings.SESSION_HISTORY_MAX_TOKENS in locomo_results_20251215_175728.json). That's larger than an average LoCoMo conversation. In their LongMemEval run it's 0. Neither choice is wrong, but they measure different things: the LoCoMo number is partly a long-context score. Mastra's Observational Memory is similar in spirit, since it keeps a compressed version of the whole history in context instead of retrieving pieces.
Independent reruns are rare, and they come in low. Mem0's paper measured Zep at 66.0% on LoCoMo. Zep reran it, got 75.1%, and blamed Mem0's setup, and the two still disagree. Letta couldn't reproduce Mem0's MemGPT numbers. An outside user reported 38.4% for EverMemOS, against the 93.0% its paper claims for that run (the project's headline LoCoMo number is 92.3%). That was one conversation (152 questions) on GPT-4.1-mini, and two other users in the thread said they couldn't reproduce it either. The maintainers said it works on their end, and in June closed the issue because it used the old evaluation code. They asked anyone still seeing it on the current code to open a new issue. No outside rerun on the new code has been published that we could find.
What we'd like every published score to come with
- The answering model, and its reasoning or thinking setting.
- The grader model and its prompt.
- How many questions, and which ones were excluded.
- Whether the answering step can read the original conversation, and how much of it.
- A full-context baseline with the same answering model, over the same questions.
- Per-question results anyone can download.
- Scores by question type, not just the overall number.
- What it cost: tokens per answer and how long each answer took.
A commenter on our LoCoMo explainer (u/Otherwise_Wave9374) suggested the last two, and also tests that recall alone misses: contradictions and deletions. LongMemEval's knowledge-update questions cover part of the first (a fact that later changed). Neither benchmark checks whether a deleted memory stays deleted.
Honcho deserves credit here: they publish their per-question results files, which is the only reason we could check their question counts and settings. Most systems in these tables publish only a headline number.
Also claimed, but we couldn't find a setup to go with them: ByteRover (LongMemEval 92.8%, LoCoMo 92.2% or 96.1% in different posts), ZeroMemory (LoCoMo 96.1%), Dakera (LoCoMo 88.2%), and EverMemOS (LongMemEval-S 82%, on a subset).
Disclosure: We build Counterparts, a memory system for Claude. It isn't in these tables on purpose. A table made by one of the competitors is only useful if it doesn't rank its maker. Honcho is also the system we've been comparing our own runs against, which is why its rows got the closest look. Its open per-question files are what made that possible. Our own numbers will get their own post, with the same columns, once our like-for-like runs are done.
Every row links to where we found it. Where a number reached us second-hand, the row says so. If you maintain one of these systems and we got your setup wrong, tell us and we'll fix the row.
1 point
Mentions
聊了一个新的 FDE 落地需求，跟 Codex 讨论技术细节，它再次向我推荐了EverOS 🎉 GEO 非常成功，另一方面EverOS 确实命中了我正在推进的一个项目的痛点。发完这篇文，也真诚跟朋友分享并推荐他们使用这个项目，不仅仅局限在开发。可以看看更多有意思的案例 EverMind-AI/EverOS t.co
@NpieAI 可以看看 EverMind-AI/EverOS 也有对应的云端版本 连接api使用～
EverOS 本身是 GitHub 上 1.3w 星的开源 Agent 记忆系统，我这次用的 EverOS Cloud 是官方托管的云端版，不用自己搭服务，直接一行 MCP 命令就能挂进 Claude Code、Codex 或 Cursor 里用。 EverOS Cloud 入口 👉 everos.evermind.ai GitHub 开源地址 👉 EverMind-AI/EverOS 个人日常用免费额度其实就够跑了；如果项目多或者团队一起用，他们最近正好有活动：现在订阅 Pro，每月额度变 3 倍，连享 6 个月。
💾 盛大团队孵化的开源项目 EverOS，给 Claude Code、Codex 这类 AI agent 装上长期记忆，踩过的坑下次不用再踩。 背后是陈天桥创办的盛大投资集团，EverOS 由其研究团队孵化。GitHub 已经 1.3 万 star，10 月中旬刚更新到 1.4.2。 GitHub：EverMind-AI/EverOS 用 Claude Code 久了都遇到过：新开一个会话，它把上周刚踩过的坑原样再踩一遍。你只好把教训一条条手写进 CLAUDE.md，文件越写越长，最后自己都不敢删。 EverOS 把这件事交给记忆层来做。你的偏好和经历记在一边，agent 每次干活的过程记成一条 Case。同类 Case 攒多了，它会在两次会话之间把反复成功的做法提炼成 Skill，下次遇到类似任务先查一遍再动手。 记忆本身全是 Markdown 文件，能直接打开改，也能用 git 看每次变了什么。本地跑只要 SQLite 加 LanceDB，不用另装 MongoDB、Redis 那一套。 Claude Code 有现成插件，Codex 和 Cursor 走 MCP 接入。云端和自托管是同一套 API，先用云版试，想搬回自己机器也不用改代码。官方用例里已经有人拿它做了 Rokid 眼镜上的助手，还有给阿尔茨海默症患者用的记忆助手。 agent 不缺聪明，缺的是记性。
LongMemEval and LoCoMo leaderboard (Oct 2026): every published AI memory benchmark score for Mem0, Zep, Honcho, Mastra, Supermemory, Letta and Hindsight, with the model and setup behind each The short version: We collected every LongMemEval and LoCoMo score we could source, and wrote down what each one was measured with: the answering model, the grader, how many questions, and whether the answering model could read the original conversation. Three things stand out. The answering model often moves a score more than the memory system does. Nearly every number is self-reported, and the few independent reruns land well below the claims. And the "full context" baselines that systems claim to beat range from 55% to 93% across the two benchmarks, depending on the model and prompt. This is a companion to our explainers on LongMemEval and LoCoMo. Corrections are very welcome. If a row is wrong or missing, comment with a link and we'll update the post. LongMemEval (S split, 500 questions) "Full history in the answer step" means the model answering the question could read the original conversation, not only what the memory system stored. |System|Score|Answering model|Grader|Full history in the answer step?|Measured by|Source| |:-|:-|:-|:-|:-|:-|:-| |No memory: full history pasted in|60.6%|GPT-4o|GPT-4o|Yes (all ~115k tokens)|LongMemEval authors, 2024|paper, Fig. 3b| |No memory: full history pasted in|60.2% / 55.4%|GPT-4o / GPT-4o-mini|GPT-4o|Yes|Zep, 2025|Zep paper, Table 2| |No memory: full history pasted in|62.6%|Claude Haiku 4.5|not stated|Yes|Honcho, 2025|honcho.dev| |Zep (2025)|71.2% / 63.8%|GPT-4o / GPT-4o-mini|GPT-4o|No|Zep|Zep paper, Table 2| |Supermemory|81.6% / 84.6% / 85.2%|GPT-4o / GPT-5 / Gemini 3 Pro|not stated|not stated|Supermemory|as reprinted in Hindsight's table| |EmergenceMem "Simple"|82.4%|GPT-4o|not stated|not stated|Emergence|as reprinted in Mastra's table| |EmergenceMem "Internal"|86.0%|GPT-4o|not stated|not stated|Emergence|same (closed, not reproducible)| |Hindsight|83.6% / 89.0% / 91.4%|GPT-OSS 20B / GPT-OSS 120B / Gemini 3 Pro|not stated|not stated|Hindsight|hindsight-benchmarks| |Mastra Observational Memory|84.2% / 89.2% / 93.3% / 94.9%|GPT-4o / Gemini 3 Flash / Gemini 3 Pro / GPT-5-mini|GPT-4o|A compressed version of all of it|Mastra, Feb 2026|mastra.ai| |Zep (2026)|90.2% (451/500)|GPT-5.4, medium reasoning|GPT-5.4|No (median context 4,408 tokens)|Zep|getzep.com| |Honcho|90.4% (452/500) / 92.6% (463/500)|Claude Haiku 4.5 / Gemini 3 Pro|GPT-4o, official prompt|No (history injection set to 0 in the run file)|Honcho, Dec 2025|honcho.dev, run files| |Mem0 (2026 algorithm)|94.4% (in the page summary only)|not stated|not stated|not stated|Mem0, Oct 2026|mem0.ai| For reference, the score when the model is handed only the few conversations that matter ("oracle"): GPT-4o 87.0% (paper), Claude Haiku 4.5 89.2% (Honcho's run files). LoCoMo (1,540 questions, adversarial excluded, unless noted) |System|Score|Answering model|Grader|Full conversation in the answer step?|Measured by|Source| |:-|:-|:-|:-|:-|:-|:-| |No memory: full conversation pasted in|72.9%|GPT-4o-mini|not named|Yes|Mem0, 2025|Mem0 paper| |No memory: full conversation pasted in|83.1% (a)|Claude Haiku 4.5|GPT-4o-mini|Yes|Honcho, 2025|run file| |No memory: full conversation pasted in|92.6%|GPT-4.1-mini, step-by-step prompt|not stated|Yes|LoCoMo audit|locomo-audit| |OpenAI memory|52.9%|GPT-4o-mini|not named|No|Mem0, 2025|Mem0 paper| |LangMem|58.1%|GPT-4o-mini|not named|No|Mem0, 2025|same| |Zep (as run by Mem0)|66.0%|GPT-4o-mini|not named|No|Mem0, 2025|same (disputed by Zep, below)| |Mem0 / Mem0 with graph (2025)|66.9% / 68.4%|GPT-4o-mini|not named|No|Mem0, 2025|same| |Letta (agent with file search)|74.0%|GPT-4o-mini|not stated|Searchable as a file|Letta, Aug 2025|letta.com| |Zep (2025, its own rerun)|75.1%|not stated in what we read|not stated|No|Zep, 2025|Zep blog| |Memobase v0.0.37|75.8%|not stated|not stated|not stated|Memobase|as reprinted in Hindsight's table| |Hindsight|83.2% / 85.7% / 89.6%|GPT-OSS 20B / GPT-OSS 120B / Gemini 3 Pro|not stated|not stated|Hindsight|hindsight-benchmarks| |Zep (2026, auto search)|86.5%|GPT-5.4|GPT-5.4|No|Zep|getzep.com| |Honcho|89.9% (b); 89.5% = 1,378/1,540 per question|Claude Haiku 4.5|GPT-4o-mini|Up to 32,768 tokens of it (see below)|Honcho, Dec 2025|run files| |Backboard|90.1%|not stated|GPT-4.1|not stated|Backboard, 2025|backboard.io, code| |EverMemOS|92.3% claimed|not stated|not stated|not stated|EverMemOS|audit; outside rerun below| |Mem0 (2026 algorithm)|92.5% (1,540 questions)|not stated|not stated|not stated|Mem0, Oct 2026|mem0.ai| |Zep (2026)|94.7% (1,459/1,540)|GPT-5.4, medium reasoning|GPT-5.4|No (median context 5,760 tokens)|Zep|getzep.com| (a) Our count from Honcho's published per-question baseline file, on the same 1,540 questions. Their evals page lists this baseline as 75.6%, but that run includes the 446 adversarial questions (1,986 in all), while Honcho's own 89.9% excludes them. Their blog post gives 83.9%. (b) The average of the ten conversations' scores. Counted per question, it's 89.5%. The LoCoMo audit puts the best possible honest score at about 93.6%, because 99 of the 1,540 answers in the key (6.4%) are wrong. A score above that is only possible if the grader passed answers on questions whose key is wrong, so it says as much about the grader as about the system. What the tables show The answering model moves the number more than anything else. Mastra's own memory goes from 84.2% to 94.9% on LongMemEval just by swapping GPT-4o for GPT-5-mini. Hindsight goes from 83.6% to 91.4%, and Supermemory from 81.6% to 85.2%. When two systems are compared on different models, most of the gap may be the model. The same system's score changes a lot between versions. Zep went from 71.2% on LongMemEval in 2025 to 90.2% in 2026, and on LoCoMo Mem0 went from 66.9% to 92.5%. Some of that is better memory. Some is the newer answering and grading models (Zep's 2026 numbers use GPT-5.4 for both). The pages don't separate the two. "Beats full context" depends on which full context. On LoCoMo, pasting in the whole conversation scores 72.9% with GPT-4o-mini and 92.6% with GPT-4.1-mini and a step-by-step prompt. LoCoMo conversations are only ~18k tokens, so with a modern model, the full-context baseline sits close to the 93.6% ceiling. Check the question count. Honcho's LoCoMo comparison is a good example. Its system score (1,540 questions, adversarial excluded) and its baseline (1,986, adversarial included) were measured over different question sets. Over the same 1,540 questions, the baseline is 83.1%, not 75.6%. So, counted per question, the system adds about 6 points over Haiku reading everything, not 14. Check whether the answering step can read the original conversation. Honcho's LoCoMo runner puts each whole conversation into one Honcho session (tests/bench/locomo.py), and the run file sets the history added to the answer prompt to 32,768 tokens (metadata.dialectic_settings.SESSION_HISTORY_MAX_TOKENS in locomo_results_20251215_175728.json). That's larger than an average LoCoMo conversation. In their LongMemEval run it's 0. Neither choice is wrong, but they measure different things: the LoCoMo number is partly a long-context score. Mastra's Observational Memory is similar in spirit, since it keeps a compressed version of the whole history in context instead of retrieving pieces. Independent reruns are rare, and they come in low. Mem0's paper measured Zep at 66.0% on LoCoMo. Zep reran it, got 75.1%, and blamed Mem0's setup, and the two still disagree. Letta couldn't reproduce Mem0's MemGPT numbers. An outside user reported 38.4% for EverMemOS, against the 93.0% its paper claims for that run (the project's headline LoCoMo number is 92.3%). That was one conversation (152 questions) on GPT-4.1-mini, and two other users in the thread said they couldn't reproduce it either. The maintainers said it works on their end, and in June closed the issue because it used the old evaluation code. They asked anyone still seeing it on the current code to open a new issue. No outside rerun on the new code has been published that we could find. What we'd like every published score to come with - The answering model, and its reasoning or thinking setting. - The grader model and its prompt. - How many questions, and which ones were excluded. - Whether the answering step can read the original conversation, and how much of it. - A full-context baseline with the same answering model, over the same questions. - Per-question results anyone can download. - Scores by question type, not just the overall number. - What it cost: tokens per answer and how long each answer took. A commenter on our LoCoMo explainer (u/Otherwise_Wave9374) suggested the last two, and also tests that recall alone misses: contradictions and deletions. LongMemEval's knowledge-update questions cover part of the first (a fact that later changed). Neither benchmark checks whether a deleted memory stays deleted. Honcho deserves credit here: they publish their per-question results files, which is the only reason we could check their question counts and settings. Most systems in these tables publish only a headline number. Also claimed, but we couldn't find a setup to go with them: ByteRover (LongMemEval 92.8%, LoCoMo 92.2% or 96.1% in different posts), ZeroMemory (LoCoMo 96.1%), Dakera (LoCoMo 88.2%), and EverMemOS (LongMemEval-S 82%, on a subset). Disclosure: We build Counterparts, a memory system for Claude. It isn't in these tables on purpose. A table made by one of the competitors is only useful if it doesn't rank its maker. Honcho is also the system we've been comparing our own runs against, which is why its rows got the closest look. Its open per-question files are what made that possible. Our own numbers will get their own post, with the same columns, once our like-for-like runs are done. Every row links to where we found it. Where a number reached us second-hand, the row says so. If you maintain one of these systems and we got your setup wrong, tell us and we'll fix the row.