Memdoor: How a Team of 5 Qwen‑3.8‑27B Models Beats Fable‑5 on LiveCodeBench Hard
1. What was released / announced
The Memdoor project (github.com/guregodevo/memdoor) introduced a new workflow where five copies of the Qwen‑3.8‑27B model cooperate to solve programming problems. In the LiveCodeBench Hard benchmark, this “team‑solve” strategy outperformed Meta’s Fable‑5, a state‑of‑the‑art code‑generation model. The repo ships a ready‑to‑run example that spins up the five agents, shares a mutable memory store, and aggregates their answers.
2. Why it matters
a) Scaling LLM reasoning without bigger models
Most of us have been chasing larger context windows or more parameters to get better code generation. Memdoor shows that orchestrating multiple smaller models can be more effective than a single monolithic model. For engineers, this opens a path to higher quality results without the massive GPU budget a 70B+ model would demand.
b) Real‑world collaborative coding
Think of code review, pair programming, or a “rubber‑duck” that can ask follow‑up questions. Memdoor’s architecture mirrors that workflow: each model writes a snippet, stores it, reads what the others produced, and refines the solution. That pattern is directly reusable for internal tooling, CI‑assistants, or even educational bots.
c) Open‑source reproducibility
Everything lives in a public repo, with Dockerfiles and a simple run.sh. No vendor lock‑in, no API key throttling, and you can swap Qwen‑3.8‑27B for any other model that respects the OpenAI‑compatible chat API. This makes the approach future‑proof as new models appear.
3. How to use it
Below is the minimal path to get the Memdoor team running on a single machine with Docker. I tested it on an AWS g5.12xlarge (8×A100) but you can scale down to a single 40GB GPU for experimentation – just reduce the parallelism.
Step 1 – Clone and build the image
# Clone the repo
git clone https://github.com/guregodevo/memdoor.git
cd memdoor
# Build the Docker image (includes Qwen‑3.8‑27B weights via huggingface hub)
docker build -t memdoor:latest .
> Tip: The Dockerfile pulls the model from modelscope – make sure you have a HuggingFace token set as HF_TOKEN in your environment.
Step 2 – Spin up the memory store (Redis) and the agents
# Start Redis (used as the shared mutable memory)
docker run -d --name memdoor-redis -p 6379:6379 redis:7-alpine
# Launch the five agents (each in its own container for isolation)
for i in {0..4}; do
docker run -d --name memdoor-agent-$i \
--link memdoor-redis:redis \
-e AGENT_ID=$i \
-e MODEL_NAME="Qwen/Qwen3.8-27B" \
memdoor:latest
done
Each container runs python -m memdoor.agent which:
1. Pulls the model.
2. Registers itself in Redis under agents.
3. Listens for a task key, generates a partial solution, writes back to memory:<task_id>.
Step 3 – Submit a LiveCodeBench problem
import redis, json, time
r = redis.Redis(host='localhost', port=6379)
# Example problem (taken from LiveCodeBench Hard)
problem = {
"id": "lcbench_hard_001",
"prompt": "Write a Python function `is_palindrome(s: str) -> bool` that returns True for palindrome strings, handling Unicode correctly."
}
# Push the task to Redis
r.set(f"task:{problem['id']}", json.dumps(problem))
# Give agents ~30 seconds to collaborate
time.sleep(30)
# Retrieve the aggregated answer
answer = r.get(f"memory:{problem['id']}")
print(answer.decode())
The agents will read the same task key, generate their own snippets, and iteratively refine each other’s output. The final aggregated answer is stored under memory:<task_id>.
Step 4 – Replace Qwen with your favourite model (optional)
If you have a local Llama‑3.1‑8B or an OpenAI‑compatible endpoint, just change the MODEL_NAME env var:
export MODEL_NAME="openai/gpt-4o-mini"
# restart containers with the new env var
The rest of the workflow stays the same because the agent code talks to the model via the ChatCompletion interface.
4. My take
Having built AI pipelines for Griffin AI Tech for the past three years, I’m constantly weighing cost vs. capability. The Memdoor experiment hits a sweet spot: instead of allocating a single 80‑GPU node to a 70B model, you can spin up a fleet of 27B models that collectively outperform a larger competitor on a hard coding benchmark.
From an infrastructure perspective, the pattern aligns with the micro‑service mindset we champion in our MLOps platform:
- Stateless agents – each container can be autoscaled behind a Kubernetes Deployment.
- Shared Redis memory – a cheap, low‑latency store that can be swapped for a more durable backend (e.g., DynamoDB) if you need persistence.
- Observability – instrument each agent with OpenTelemetry; you’ll instantly see how many iterations each model contributes and where the bottleneck lies.
In practice, I see two immediate use‑cases for our customers:
1. Automated code review bots – run the team‑solve on a PR diff, let the agents suggest improvements, and surface a consensus patch.
2. Educational tutors – a student asks a programming question; the agents collaboratively generate a step‑by‑step answer, and the memory store can be queried for the reasoning trace, which is a gold‑mine for learning analytics.
That said, there are trade‑offs. Running five 27B models still eats a lot of VRAM, and the coordination overhead adds latency (typically 15‑30 s per problem). For latency‑critical services, a single, well‑tuned model may still be preferable. Also, the current implementation assumes synchronous collaboration – an area ripe for research, especially with asynchronous message passing or reinforcement‑learning based role assignment.
Overall, Memdoor is a practical proof‑of‑concept that collaborative LLMs can beat larger singular models on complex code tasks. It gives us a concrete blueprint to build more resilient, cost‑effective AI services that scale horizontally rather than vertically. If you’re tinkering with AI‑augmented development tools, I recommend cloning the repo, running the workflow on a modest GPU, and then iterating on the orchestration layer to fit your own product needs.
---
Happy hacking!
Mentions
🔗 guregodevo/memdoor — smaller models + coordination beat larger single models on hard reasoning tasks
Small Qwen team beats Fable 5 on coding tasks A Memdoor workflow with five Qwen3.8-27B models outperformed Fable 5 on LiveCodeBench Hard problems, achieving a 9.8 mean pass rate for $0.39 per run. This highlights the power of orchestrated small models over monolithic ones.
Memdoor: How a Team of 5 Qwen‑3.8‑27B Models Beats Fable‑5 on LiveCodeBench Hard 1. What was released / announced The Memdoor project (github.com/guregodevo/memdoor) introduced a new workflow where five copies of the Qwen‑3.8‑27B model cooperate to solve programming problems. In the LiveCodeBench Hard benchmark, this “team‑solve” strategy outperformed Meta’s Fable‑5, a state‑of‑the‑art code‑generation model. The repo ships a ready‑to‑run example that spins up the five agents, shares a mutable memory store, and aggregates their answers. 2. Why it matters a) Scaling LLM reasoning without bigger models Most of us have been chasing larger context windows or more parameters to get better code generation. Memdoor shows that orchestrating multiple smaller models can be more effective than a single monolithic model. For engineers, this opens a path to higher quality results without the massive GPU budget a 70B+ model would demand. b) Real‑world collaborative coding Think of code review, pair programming, or a “rubber‑duck” that can ask follow‑up questions. Memdoor’s architecture mirrors that workflow: each model writes a snippet, stores it, reads what the others produced, and refines the solution. That pattern is directly reusable for internal tooling, CI‑assistants, or even educational bots. c) Open‑source reproducibility Everything lives in a public repo, with Dockerfiles and a simple run.sh. No vendor lock‑in, no API key throttling, and you can swap Qwen‑3.8‑27B for any other model that respects the OpenAI‑compatible chat API. This makes the approach future‑proof as new models appear. 3. How to use it Below is the minimal path to get the Memdoor team running on a single machine with Docker. I tested it on an AWS g5.12xlarge (8×A100) but you can scale down to a single 40GB GPU for experimentation – just reduce the parallelism. Step 1 – Clone and build the image # Clone the repo git clone https://github.com/guregodevo/memdoor.git cd memdoor # Build the Docker image (includes Qwen‑3.8‑27B weights via huggingface hub) docker build -t memdoor:latest . > Tip: The Dockerfile pulls the model from modelscope – make sure you have a HuggingFace token set as HF_TOKEN in your environment. Step 2 – Spin up the memory store (Redis) and the agents # Start Redis (used as the shared mutable memory) docker run -d --name memdoor-redis -p 6379:6379 redis:7-alpine # Launch the five agents (each in its own container for isolation) for i in {0..4}; do docker run -d --name memdoor-agent-$i \ --link memdoor-redis:redis \ -e AGENT_ID=$i \ -e MODEL_NAME="Qwen/Qwen3.8-27B" \ memdoor:latest done Each container runs python -m memdoor.agent which: 1. Pulls the model. 2. Registers itself in Redis under agents. 3. Listens for a task key, generates a partial solution, writes back to memory:<task_id>. Step 3 – Submit a LiveCodeBench problem import redis, json, time r = redis.Redis(host='localhost', port=6379) # Example problem (taken from LiveCodeBench Hard) problem = { "id": "lcbench_hard_001", "prompt": "Write a Python function `is_palindrome(s: str) -> bool` that returns True for palindrome strings, handling Unicode correctly." } # Push the task to Redis r.set(f"task:{problem['id']}", json.dumps(problem)) # Give agents ~30 seconds to collaborate time.sleep(30) # Retrieve the aggregated answer answer = r.get(f"memory:{problem['id']}") print(answer.decode()) The agents will read the same task key, generate their own snippets, and iteratively refine each other’s output. The final aggregated answer is stored under memory:<task_id>. Step 4 – Replace Qwen with your favourite model (optional) If you have a local Llama‑3.1‑8B or an OpenAI‑compatible endpoint, just change the MODEL_NAME env var: export MODEL_NAME="openai/gpt-4o-mini" # restart containers with the new env var The rest of the workflow stays the same because the agent code talks to the model via the ChatCompletion interface. 4. My take Having built AI pipelines for Griffin AI Tech for the past three years, I’m constantly weighing cost vs. capability. The Memdoor experiment hits a sweet spot: instead of allocating a single 80‑GPU node to a 70B model, you can spin up a fleet of 27B models that collectively outperform a larger competitor on a hard coding benchmark. From an infrastructure perspective, the pattern aligns with the micro‑service mindset we champion in our MLOps platform: - Stateless agents – each container can be autoscaled behind a Kubernetes Deployment. - Shared Redis memory – a cheap, low‑latency store that can be swapped for a more durable backend (e.g., DynamoDB) if you need persistence. - Observability – instrument each agent with OpenTelemetry; you’ll instantly see how many iterations each model contributes and where the bottleneck lies. In practice, I see two immediate use‑cases for our customers: 1. Automated code review bots – run the team‑solve on a PR diff, let the agents suggest improvements, and surface a consensus patch. 2. Educational tutors – a student asks a programming question; the agents collaboratively generate a step‑by‑step answer, and the memory store can be queried for the reasoning trace, which is a gold‑mine for learning analytics. That said, there are trade‑offs. Running five 27B models still eats a lot of VRAM, and the coordination overhead adds latency (typically 15‑30 s per problem). For latency‑critical services, a single, well‑tuned model may still be preferable. Also, the current implementation assumes synchronous collaboration – an area ripe for research, especially with asynchronous message passing or reinforcement‑learning based role assignment. Overall, Memdoor is a practical proof‑of‑concept that collaborative LLMs can beat larger singular models on complex code tasks. It gives us a concrete blueprint to build more resilient, cost‑effective AI services that scale horizontally rather than vertically. If you’re tinkering with AI‑augmented development tools, I recommend cloning the repo, running the workflow on a modest GPU, and then iterating on the orchestration layer to fit your own product needs. --- Happy hacking!
Five Qwen3.8-27B Models Match Claude Fable 5 on LiveCodeBench Hard This has been implemented here -> github.com/guregodevo/memdoor/tree/main/examples/…
写代码真的要这么大尺寸的模型么？ 开源项目 Memdoor 让规划、构思、求解、管理、修复 5 个角色的 Qwen3.8-27B 小模型协作，在 LiveCodeBench Hard 上胜过单个大模型。整个工作流只有 8 个 YAML 文件，跑一次约 0.39 美元，HN 423 分。 github.com/guregodevo/memdoor/tree/main/examples/… t.co
💥 Five Qwen3.8-27B agents just beat Fable 5 on LiveCodeBench Hard. Same model, five roles talking only through files: 9.8 vs 8.0 on 10 hard problems (5 passes). Tiny sample, so grain of salt. Still a wild local team trick. via Memdoor 👇 guregodevo/memdoor t.co
The problems date to spring 2025, so training contamination can't be ruled out for any column. Source: Hacker News github.com/guregodevo/memdoor/tree/main/examples/…
Memdoor: A team of 5 Qwen3.8-27B models beats Fable 5 on LiveCodeBench Hard github.com/guregodevo/memdoor/tree/main/examples/… comments #github.com
Scaling Code Intelligence: How a Team of Smaller Models Beat Fable 5 The paradigm of "bigger is better" in Large Language Models (LLMs) is facing a significant challenge from orchestrated multi‑model workflows. Recent results from the Memdoor project demonstrate that a coordinated team of smaller models can outperform monolithic, state‑of‑the‑art systems on complex programming tasks. According to the project's documentation, a team of five Qwen3.8‑27B models successfully surpassed the performance of Fable 5 on the "Hard" category of LiveCodeBench (Memdoor GitHub). This achievement highlights a shift toward agentic workflows where specialized or smaller models collaborate to solve problems that previously required massive compute resources. The "Team‑Solve" Workflow The Memdoor project utilizes a specific "team‑solve" workflow designed for high‑difficulty coding challenges. Rather than relying on a single inference pass from a massive model, the system orchestrates multiple instances of the Qwen3.8‑27B model. This approach typically involves: - Decomposition: Breaking down complex requirements into manageable sub‑tasks. - Iterative Refinement: Models reviewing and debugging each other's code outputs. - Consensus Building: Validating solutions through multiple perspectives to reduce hallucinations. Contextualizing the Agentic Shift This development aligns with broader industry trends observed in the 2026 Octoverse report, which notes that developers are increasingly building with agents to reach new highs in commit and push activity (GitHub Blog). The transition from using AI as a simple autocomplete tool to employing "teams" of agents suggests a new era of software delivery where orchestration is as important as the underlying model's parameters. Why This Matters for Developers For developers and DevOps engineers, the success of the Memdoor approach offers several practical takeaways: 1. Cost Efficiency: Running multiple smaller models can often be more cost‑effective than a single call to a top‑tier proprietary model like Fable 5, depending on the hosting environment and token usage. 2. Specialization: Smaller models can be fine‑tuned for specific languages or frameworks more easily than general‑purpose giants. 3. Resilience: Multi‑agent systems can implement self‑correction loops, identifying errors in logic that a single pass might miss. While the specific internal mechanics of Fable 5's failure on these benchmarks remain unconfirmed, the success of the Memdoor team‑solve workflow provides a concrete roadmap for developers looking to push the boundaries of AI‑assisted coding. --- AI Disclosure: This article was written with the assistance of AI to summarize and structure technical data from the provided sources. #ai-disclosed Image credit: File:Adaptive assistive robotics - a framework for triadic collaboration between humans and robots.pdf — CC BY 4.0 — license.
Memdoor: A team of 5 Qwen3.8-27B models beats Fable 5 on LiveCodeBench Hard View Article | Discussion + AI Summary Summary of HN discussion 🧵👇