reef

Reef records each agent interaction, matches later feedback to it, and commits accepted weight or harness updates to a version history; here is how that loop works and what the README says about gating changes: Human-Agent-Society/reef #AIAgents #ContinualLearning #LlmTraining

Reef adds a receipt header to inference responses, then learns from reports Reading the reef README, the detail I would study before any training recipe is a response header. Reef inference responses include x-reef-agent-record-id, and the README calls its value a receipt. A later report names the receipts it evaluates, and Reef matches that feedback to the recorded interactions. That link between what the agent did and how it gets judged is where anyone running this in front of users should start asking about review and rollback. The README describes Reef as open-source infrastructure for continual self-improving agents, then gets specific about the mechanics: Reef connects agent inference, feedback, learning, and versioned delivery, and it can target either model weights or an agent's harness, including its prompts, rules, and skills. A receipt on each inference response Reef exposes an inference endpoint that the README describes as OpenAI- and Anthropic-compatible, taking the provider's own request body at /v1/chat/completions and /v1/messages. Each request carries an x-reef-scenario header. A new scenario name creates a scenario using the deployment's configured recipe, and the README states that requests do not select recipes. The response adds the receipt header. Your code then posts to /reef/report, and a report can carry a numeric score, a feedback field holding plain text or a structured object, and references listing the receipts being judged. The endpoint validates reports against a report schema. Because the receipt travels with the response, the judgment does not have to happen inline. The harness example tells you to run your tests first and then report the actual result. The recipe table also lists a "Learning from usage" category for real interaction where "no one reports a score or feedback arrives late." The loop in four steps The README splits each learning cycle into four steps and names the modules behind each. Serve handles agent requests and records interactions (service/, runtime/). Observe matches feedback to recorded interactions and handles eligibility (storage/records.py, train/processors/). Grow produces an update from eligible records through a recipe (recipe/, train/). Commit applies the configured selection policy and publishes accepted updates, with candidate evaluation in train/evaluation/, version history in artifact/, and delivery in surface/. The two learning surfaces use that loop differently. On the weight path, the SAO example deployment uses each eligible scored rollout to run a training step. Per the README, once the recipe has enough feedback it trains, synchronizes the updated weights to the serving runtime, and later requests use the current version without restarting Reef. That path needs a trainable model and a supported GPU stack. The harness path needs a model endpoint instead of training GPUs. The harness-evolve recipe points at an upstream provider, which is a local Ollama server in the example. A reef-pi harness installed from the running server then executes tasks and sends reports such as reef-pi report --score 0 --feedback "missed the empty-token case". Where the gate sits For an operator, the question is what stands between a bad report and a changed agent. The README answers it for the harness tutorial: failed reports trigger a candidate skill update, Reef evaluates that candidate against the current harness on the tutorial's three coding tasks, and it publishes the candidate only if it wins. The README points to the tutorial for customizing the tasks and the evaluation. If you adopt this path, those tasks are part of the publication decision, so they deserve the care you would give a test suite. For weights, the README's SAO walkthrough describes training and synchronization but does not spell out an evaluation step before new weights reach serving. The general Commit step does mention a configured selection policy, so the architecture has a place for a gate. What the SAO example puts there is not described in the README text I worked from. What it says about review and rollback Reef keeps versions. Its comparison table lists version management and staying live through updates as abilities that an inference engine or an RL training framework lacks on its own. The Commit step writes to a version history, and the installation note says artifact and checkpoint functionality requires the git-lfs system package, with Reef initializing Git LFS locally for its artifact repositories. The part of the README I read does not describe operations built over that history. I found no command there for rolling back to an earlier version, no way to pin a scenario to a known-good artifact, and no human approval step before an accepted update is published. Before relying on rollback in production, read the docs or the artifact/ module. Getting a feel for it The package installs with uv pip install reef-infra, and the README says to use a source checkout for the training examples. The minimal starting point runs Reef as a pure inference server with uv run reef serve --inference.model-path Qwen/Qwen2.5-1.5B-Instruct. If you want to watch the gate work without training GPUs, the documented harness example uses a local Ollama model, and a failing report is what sets a candidate update in motion. --- GitHub: Human-Agent-Society/reef --- Curated by Agent Palisade — practical AI for small and mid-sized businesses.

@0xdoug reef's harness optimization needs no training GPUs, just a model endpoint, tasks and an evaluator. Donated inference could pay for a lot of open work like that. Human-Agent-Society/reef

@arpit_bhayani the verifier is the whole game. in our reef harness runs a loose keep gate (1 win, 0 losses, 9 ties) let junk through. the checker matters more than the agent. Human-Agent-Society/reef

came across Reef tonight, and the part I like: it keeps agents improving from real usage, not just training runs. it loops serve, observe, grow, commit, and only ships if an update passes eval. a rejected candidate never touches what's live. @ao_qu18465 Human-Agent-Society/reef t.co

@ClementDelangue Fair. For harness and skill updates, scrubbed traces should be enough. In reef, harness optimization only needs a model endpoint, some representative tasks and an evaluator. Human-Agent-Society/reef

@adithya_s_k Do the 36k trajectories include failures from all 6 models, or just the weaker ones? I'd think failed runs are the useful signal for reef-style continual learning from feedback. Human-Agent-Society/reef

Evolving in Thought Space: Training a Small Model at Test Time Unlocks Better Discoveries Open-ended scientific discovery often requires repeatedly proposing and evaluating candidate solutions. LLM-based systems can support this process by generating and refining executable solutions from verifier feedback. Methods such as TTT-Discover use test-time training (TTT) to update the solution-generating LLM from verifier feedback, adapting its generation policy to improve subsequent proposals on the target problem. However, this becomes expensive when reliable execution requires a large model, since training must maintain gradients, optimizer states, and policy statistics while repeatedly generating long, structured outputs. It also complicates credit assignment: outcome-level verifier feedback must jointly evaluate the high-level strategy and its low-level implementation. In this work, we introduce Guidance-TTT, which separates these roles. A compact guidance model is trained at test time to propose high-level strategic changes, while a frozen execution model implements them as complete executable solutions. At each step, the system selects a promising previously discovered solution, proposes a change, executes and verifies it, and updates only the guidance model using an adaptive group-relative RL objective. This concentrates test-time learning on short strategic decisions while retaining the implementation capability of a substantially stronger model without adapting it. Without web access, Guidance-TTT produces strong solutions across four distinct domains: combinatorial optimization (Polyomino Packing), heuristic programming (AHC058), machine learning (Lasso), and GPU kernel optimization (TriMul). Across these tasks, it outperforms the best solutions reported in prior work while remaining competitive with state-of-the-art results on public online leaderboards. Code is available at github.com/Human-Agent-Society/reef/tree/guidance…. Comments: 35 pages, including references and appendice