An artifact evolves; the mental model that judges it is frozen — composed server-side by id, with no API parameter through which the artifact or improver can touch the rubric. That separation is the whole discipline of Recursive Self-Improvement, and this lab makes it visible: accepted iterations, rejected ones too, and one counterfactual replay showing exactly what happens when the judge is not frozen. Known limit: a frozen judge is still an LLM judge, and Jiang et al. (2025) show LM judges are poorly calibrated on open-ended outputs where humans themselves disagree. The Lab is strongest on rubrics with verifiable criteria; on taste questions, treat the score as one vote, not a verdict.
FROZEN mental model → critic lens + judge rubric + house rules
EVOLVES the artifact (and, in STOP, the improver's strategy)
Watch the replays — 8 pre-baked runs (sales · dev · framework-eval · research · meta · a counterfactual), zero keys; live runs use the same engine + rules
replay
46 → 88
Hardening a recipe: Security questionnaire autofill
Red-team adversarialism Self-Refine · 4 iterations
replay
24 → 86
Cold email to a fintech CISO
OODA loop Reflexion · 4 iterations
replay
26 → 90
META: improving the critic prompt itself
Popperian falsification STOP - self-improving improver · 5 iterations
⚠ what we refuse to run live
26 → 98
COUNTERFACTUAL: the unfrozen judge
Popperian falsification STOP - self-improving improver · 4 iterations
replay
18 → 88
Challenger cold email to a fintech CISO
Challenger insight-led teaching Deep Research (coverage loop) · 4 iterations
replay
50 → 86
Reviewing a producer helper: control-coverage summary
Staff-engineer code review Self-Refine · 4 iterations
replay
34 → 84
ADR-0042: heavy agent framework vs the plain provider SDK
Boring technology / Tech Radar ADR audit Godel proof-gate · 4 iterations
replay
42 → 80
Grounding a research brief: the AutoResearch survey's PROVENANCE dimension
AutoResearch evaluation rubric Deep Research (coverage loop) · 4 iterations
How to use this — two ways in
🧭 For the systems thinker
It's an optimization loop with a deliberately frozen fitness function. Your three choices are the three levers of any self-improving system:
- The artifact — the object level that evolves.
- The mental model — the frozen fitness function (what “better” means). The highest-leverage decision: it defines the landscape the search climbs. Choose the wrong frame and you'll optimize the wrong thing beautifully — the counterfactual replay shows exactly that.
- The framework — the search operator: hill-climb (Self-Refine), learn-from-failure (Reflexion), evolve a population, or prove-before-adopt.
The meta-level is structurally un-editable, so the optimizer can't move the goalposts, and every rejection is shown. That's the guard against the failure the field keeps documenting — “the optimizer fits whatever you feed it, including your bugs; scalar metrics invite gaming — track the full component vector.” This Lab shows the whole per-dimension vector, never just the total.
Where it sits: see the RSI ladder below — the safe, single-artifact, frozen-judge rung of a ladder whose higher rungs evolve code and agents against executable evaluators — hermes-agent (self-writing skills), DeerFlow (long-horizon research), Sakana Fugu (evolving agent topologies), and AlphaEvolve/DGM (self-rewriting code). Same shape, more autonomy — the Lab domesticates it so you can watch and steer one rung at a time.
🛠 For the developer
Paste your thing → say what it must do (Intent) and the facts it may cite (Allowed facts) → pick a grader (mental model) → pick an improve-strategy (framework) → Run. It critiques, rewrites, grades, keeps only if it beat the best, and repeats — for a set number of loops or ∞ until it clears a score you choose.
Copy-paste starting points:
- System prompt → Red-team or Staff-review × Self-Refine, ~5 loops
- Cold email → Challenger × Reflexion
- Code snippet → Staff-review × Self-Refine
- Research brief → AutoResearch-rubric × Deep Research
If you follow Karpathy's four coding skills you already know this loop — they map 1:1: Think Before Coding → the ① Critique step · Simplicity First → the economy penalty + the via-negativa lens · Surgical Changes → “change only what the critique justifies” · Goal-Driven Execution (“LLMs are exceptionally good at looping until they meet specific goals”) → the ∞ until-acceptable mode: set a target and let it grind.
Tips: pin a model you trust in the Model dropdown; read the red rejected cards to learn what “gaming” looks like; fill Allowed facts so it can't invent citations; start at 3–5 loops (each loop is real API calls). Prefer talking? Voicebox gives agents voice I/O over MCP for a hands-free dictate-and-listen dev loop. When you're happy, Export as OKF turns the winner into a portable knowledge doc your agents can read (see below).
The RSI ladder — where every self-improving system sits, Weco AI — “4 Levels of Recursive Self-Improvement” (10 Jul 2026) + “AIDE²: The First Evidence of Recursive Self-Improvement” (14 Jul 2026)
The through-line: Climbing the ladder, the frozen layer never disappears — it moves UP the stack. In this Lab the judge rubric is frozen. In AIDE² even the evaluation harness may evolve (the agent once fixed a bug in its own harness) — but the held-out public/private benchmark split and the dollar budget are immovable. Every real RSI system is a frozen substrate with an optimizer inside it; the engineering question is only WHERE you draw the line.
L3
Inflection
gains grow despite fixed effort
“The positive feedback loop overcomes diminishing returns. Progress accelerates even with a fixed amount of effort” — gain per generation grows rather than shrinks.
Who's here: Hypothetical. Every documented system so far — including AIDE² — still shows diminishing returns (this Lab’s plateau rule exists because of exactly that).
FROZEN LAYER
Unknown — the open safety question of the field.
L2
Ignition
improves its ability to improve
“The system does not just improve itself; it improves its own ability to improve itself” — a newer version is a better IMPROVER than its predecessor at the same budget.
Who's here: Unreached. AIDE² tested it honestly and declined the claim: “AIDE47 is not asymptotically better” as the outer agent. You can feel the shape of ignition safely in this Lab’s META replay — the improver strategy rewrites itself while the rubric never moves.
FROZEN LAYER
The meta-evaluation: what counts as a better improver.
L1
Net positive
beats the humans improving it
“An autonomous system improves itself, and does so more efficiently than humans improving the same system by hand.” Four criteria: a fair human baseline, sustained multi-step improvement, generality across benchmarks, fixed budgets.
Who's here: AIDE² is the first claimed evidence (Jul 2026): 100 self-rewrite steps in 8 days, ~90% of proposals rejected, +0.053 on MLE-Bench Lite (p=0.0024) — and reward-hacking DOWN 63%→34%, beating the hand-tuned agent’s 42%. AlphaEvolve noted as possibly here.
FROZEN LAYER
Held-out public/private test split + per-evaluation dollar budget — code AND harness may evolve.
L0
Delegation
autonomous, but slower than you
“You hand the research loop to an autonomous system end to end: it forms the hypotheses, runs the experiments and ships the improvements on its own” — but it improves itself more slowly than human R&D would.
Who's here: Weco assesses Darwin Gödel Machine, Huxley-Gödel Machine and HyperAgents as likely Level 0.
FROZEN LAYER
The benchmark suite + compute budget the system is scored on.
L−1
Glass box
◉ you are here
A single artifact evolves under a frozen, human-chosen judge; every critique, rewrite, score and rejection is visible; the human moves the frame between runs. Deliberately below Level 0: autonomy is traded for total legibility.
Who's here: This Lab — and every prompt-improvement loop with a fixed rubric (Self-Refine, Reflexion as used here).
FROZEN LAYER
The mental model: critic lens + judge rubric, composed server-side by id.
Run it yourself — the same engine + rules as the replays, on your own text
How one run works. The loop repeats N times; only a draft that beats the best-so-far under the frozen rubric is kept — everything else is rejected in the open.
Baseline
judge scores your original→ ① Critique
the frozen lens finds flaws→ ② Improve
rewrite addressing them→ ③ Judge
frozen rubric re-scores it→ keep if better,
else reject↻
judge scores your original→ ① Critique
the frozen lens finds flaws→ ② Improve
rewrite addressing them→ ③ Judge
frozen rubric re-scores it→ keep if better,
else reject↻
Step 1 · Choose what to improve EVOLVES — the artifact (a system prompt, email, policy, answer, or code)
start from an example, or pick Custom
INTENT — what it must achieve (scored as “fidelity”; drift auto-rejects)
ALLOWED FACTS — the only evidence it may cite (anything else = fabrication → auto-reject)
Step 2 · Choose how it’s judged FROZEN — the mental model becomes the critic’s lens + the judge’s rubric (it cannot be edited mid-run)
PDCA / Deming cycle
FROZENTarget moved Guardrails held Attributable Standardizable
Shewhart (Bell Labs, 1930s), popularized by Deming · 1950s
Popperian falsification
FROZENFalsifiable Severe tests survived Defect resistance No weasel hedging
Karl Popper, The Logic of Scientific Discovery · 1934
Double-loop learning
FROZENAssumptions surfaced Right frame Goal fit Depth of change
Argyris & Schon, Organizational Learning · 1978
OODA loop
FROZENOrientation Pre-empts the counter-move Decisiveness Tempo
Col. John Boyd, USAF · 1976
Red-team adversarialism
FROZENExploit resistance Overclaim safety Abuse coverage Precision
Cold War military exercises; formalized in cybersecurity adversary emulation · 1960s
Bayesian calibration
FROZENCalibration No overclaiming No underclaiming Evidence surfaced
Bayes (1763), Laplace; modern calibration practice · 1763
Variation-Selection-Retention
FROZENFitness Novelty Gain vs parent Generalization
Donald T. Campbell, Blind Variation and Selective Retention · 1960
Via negativa (improve by subtraction)
FROZENDensity Necessity Clarity after cuts Nothing lost
Occam's razor tradition; named practice in Taleb's Antifragile · 2012
First-principles rebuild
FROZENRequirements covered No cargo cult Directness Load-bearing structure
Aristotle's first principles; modern engineering usage (Feynman, cost-decomposition practice) · antiquity
Five Whys (root cause)
FROZENRoot-cause depth Recurrence prevented Residual defects Traceability
Sakichi Toyoda / Taiichi Ohno, Toyota Production System · 1950s
MEDDPICC deal qualification
FROZENMetrics quantified Power mapped Process mapped Pain tied to offer
Sales · Dick Dunkel with Jack Napoli, under John McMahon at PTC; extended to MEDDPICC (Paper process, Competition) · 1996
Challenger insight-led teaching
FROZENCommercial insight present Tailored to stakeholder economics Reframes the status quo Credible teach, not a pitch
Sales · Matthew Dixon & Brent Adamson, CEB study of ~6,000 reps; 'The Challenger Sale' - Teach, Tailor, Take Control · 2011
Staff-engineer code review
FROZENCorrect on edges Simplicity / YAGNI Readable & honest names Failure modes designed
Dev · Google Code Review Developer Guide (eng-practices, 2019) + John Ousterhout, A Philosophy of Software Design (2018) · 2018-2019
Boring technology / Tech Radar ADR audit
FROZENProblem-fit over novelty Total lifecycle cost Exit priced Maturity & security evidenced
Dev · Dan McKinley, Choose Boring Technology (2015) + Michael Nygard, Documenting Architecture Decisions (2011) + ThoughtWorks Technology Radar adoption rings · 2011-2015
Peer-review rigor (AI-Scientist reviewer)
FROZENClaim traceability Evidence quality Counter-evidence coverage Calibrated confidence
Research · NeurIPS peer-review criteria, automated as a reviewer loop in 'The AI Scientist' (Lu et al., Sakana AI) · 2024
Minto Pyramid / BLUF
FROZENAnswer first MECE decomposition So-what per section Top-down story
Research · Barbara Minto, McKinsey (MECE; The Pyramid Principle, 1985); BLUF from US military writing doctrine · 1985
AutoResearch evaluation rubric
FROZENProvenance Validity Novelty Impact
Research · Tie et al. (+22, incl. Jianfeng Gao, Philip S. Yu), "AutoResearch AI" survey, arXiv 2605.23204 · 2025
Infrastructure optimization architect
FROZENService economy Cost efficiency Availability fit Boundary integrity
· Practitioner synthesis: AWS Well-Architected cost/reliability pillars + Azure's 'avoid overengineering AND oversimplification' principle + the consolidation case literature (Prime Video 2023, Segment 2018, 37signals 2022-24) · 2023
Well-Architected pillar review
FROZENCost fitness Reliability / SLO fit Operability per head Tradeoff candor
· AWS Well-Architected Framework (2015; Google Cloud and Azure variants) — the vendors' own review discipline, distilled to the four pillar questions that bite on a services table · 2015
Step 3 · Choose the loop EVOLVES — how each critique turns into the next draft (hover a chip for its mechanics)
Step 4 · Run
model
loops
Take the output with you — Open Knowledge Format. When a run finishes, Export as OKF writes the winning artifact as a Google Cloud Open Knowledge Format concept doc — plain markdown + YAML frontmatter (
type, title, tags, timestamp) with a Provenance section recording the mental model, framework, and score trajectory that produced it.
Why it matters: OKF (Google Cloud, 2026) is a vendor-neutral format for the curated knowledge AI agents need — "anyone can produce it without an SDK, anyone can consume it without an integration," and it lives in git next to your code. So a prompt or policy you harden here drops straight into your agents' knowledge base or the Knowledge Catalog — no translation layer. It formalizes the same "LLM-wiki" /
CLAUDE.md pattern the developer notes above build on. This whole Lab is OKF-shaped: every mental model, framework, and atlas entry is one typed concept doc.House rules — enforced in server code; the judge cannot waive them
Intent fidelity — Still does what the run's INTENT says. Score < 5 = drift → auto-reject.
Economy — Substance per word — padding and boilerplate score low. Counted in the total.
−length (max 15) — Computed server-side from size: growth beyond 1.4× the original is penalized deterministically — the judge cannot waive it.
−gaming (max 30) — Fabricated facts/stats, praise-baiting, instructions aimed at the judge, or deleting a flagged feature instead of fixing it. gaming > 0 → auto-reject.
Accept — accepted iff total > best.total AND fidelity ≥ 5 AND penalties.gaming = 0
Total — total = clamp(round(mean(4 domain dims + economy) × 10) − length − gaming, 0, 100). Server-computed; model-emitted totals are ignored.
The library — 19 mental models · 9 frameworks, with sources
Mental models (the frozen substrate)
| Model | Origin | What “better” means | Best for |
|---|---|---|---|
| PDCA / Deming cycle | Shewhart (Bell Labs, 1930s), popularized by Deming · 1950s | Better = pre-declared metric moved, no guardrail metric dropped, the change caused it, and the win generalizes. | policy qa_answer recipe_prompt |
| Popperian falsification | Karl Popper, The Logic of Scientific Discovery · 1934 | Better = survives a harsher test battery, makes claims specific enough to be wrong, and never dodges tests with unfalsifiable hedging. | qa_answer critic_prompt policy |
| Double-loop learning | Argyris & Schon, Organizational Learning · 1978 | Better = the artifact's silent assumptions were surfaced and the flawed one replaced - not cosmetic churn inside a broken frame. | recipe_prompt cold_email |
| OODA loop | Col. John Boyd, USAF · 1976 | Better = oriented to the real reader in the real moment, pre-empts their strongest counter-move, makes one decisive ask, wastes no words. | cold_email |
| Red-team adversarialism | Cold War military exercises; formalized in cybersecurity adversary emulation · 1960s | Better = fewer and less severe exploitable weaknesses after a motivated attack - ambiguities, overclaims, injection paths, auditor traps. | recipe_prompt policy qa_answer |
| Bayesian calibration | Bayes (1763), Laplace; modern calibration practice · 1763 | Better = every claim's implied confidence matches its evidential support - no unsupported certainty, no false modesty about proven facts. | qa_answer policy |
| Variation-Selection-Retention | Donald T. Campbell, Blind Variation and Selective Retention · 1960 | Better = highest fitness under the fixed rubric, reached via genuinely different attempts rather than premature convergence, without overfitting the rubric's letter. | cold_email recipe_prompt |
| Via negativa (improve by subtraction) | Occam's razor tradition; named practice in Taleb's Antifragile · 2012 | Better = higher substance per word; every remaining element justified by a purpose or a failure it prevents; nothing load-bearing lost. | recipe_prompt policy cold_email |
| First-principles rebuild | Aristotle's first principles; modern engineering usage (Feynman, cost-decomposition practice) · antiquity | Better = every requirement of the actual job is covered by structure that exists BECAUSE of a requirement - not because a template had that section. | policy recipe_prompt |
| Five Whys (root cause) | Sakichi Toyoda / Taiichi Ohno, Toyota Production System · 1950s | Better = edits address root causes traceably; the same class of failure cannot recur; fewer defects remain overall. | recipe_prompt critic_prompt |
| MEDDPICC deal qualification | Dick Dunkel with Jack Napoli, under John McMahon at PTC; extended to MEDDPICC (Paper process, Competition) · 1996 | Better = more letters answered with named people, hard numbers, and dated process steps drawn from allowed facts; every remaining gap explicitly flagged as a risk with a next action, never papered over with optimism. | deal_narrative |
| Challenger insight-led teaching | Matthew Dixon & Brent Adamson, CEB study of ~6,000 reps; 'The Challenger Sale' - Teach, Tailor, Take Control · 2011 | Better = the message teaches a sharper, more specific commercial insight the recipient plausibly did not know, in the recipient's own economic language, reframing 'we're fine as we are' as the costly option - with the product appearing only after the insight has earned it. | cold_email deal_narrative |
| Staff-engineer code review | Google Code Review Developer Guide (eng-practices, 2019) + John Ousterhout, A Philosophy of Software Design (2018) · 2018-2019 | Better = fewer concrete breaking inputs, less speculative complexity (YAGNI), names and comments a new reader can predict behavior from, and every error path explicitly handled or defined out of existence. | code_snippet |
| Boring technology / Tech Radar ADR audit | Dan McKinley, Choose Boring Technology (2015) + Michael Nygard, Documenting Architecture Decisions (2011) + ThoughtWorks Technology Radar adoption rings · 2011-2015 | Better = the problem demonstrably can't be solved by the existing stack, total cost includes ops/on-call/patching, exit is priced, and maturity/security claims carry evidence instead of adjectives. | adr_decision |
| Peer-review rigor (AI-Scientist reviewer) | NeurIPS peer-review criteria, automated as a reviewer loop in 'The AI Scientist' (Lu et al., Sakana AI) · 2024 | Better = more claims traced to ALLOWED FACTS, zero unflagged unsourced assertions, specific dated numbers instead of adjectives, counter-evidence acknowledged and weighed, confidence language matched to evidence strength. | research_brief qa_answer |
| Minto Pyramid / BLUF | Barbara Minto, McKinsey (MECE; The Pyramid Principle, 1985); BLUF from US military writing doctrine · 1985 | Better = the governing answer and recommendation stated in the opening lines with honest confidence, sections that neither overlap nor leave the stated question partly unanswered, each section ending in an implication, and headings that tell the whole story on their own. | research_brief adr_decision qa_answer |
| AutoResearch evaluation rubric | Tie et al. (+22, incl. Jianfeng Gao, Philip S. Yu), "AutoResearch AI" survey, arXiv 2605.23204 · 2025 | Better = says something non-obvious (novelty), that the evidence actually supports (validity, folding in reliability/reproducibility), that would matter if true (impact), with EVERY claim traceable to an ALLOWED FACT or explicitly flagged [NEEDS SOURCE]. A confident unsourced sentence is a defect, not a strength. | research_brief competitive_brief qa_answer |
| Infrastructure optimization architect | Practitioner synthesis: AWS Well-Architected cost/reliability pillars + Azure's 'avoid overengineering AND oversimplification' principle + the consolidation case literature (Prime Video 2023, Segment 2018, 37signals 2022-24) · 2023 | Better = fewer ops surfaces and lower $/mo with every stated requirement still met: each service earns its separateness, each dollar traces to a need, redundancy matches the SLO exactly (over-provisioning scores down too), and every security/compliance boundary in ALLOWED FACTS survives untouched. | architecture |
| Well-Architected pillar review | AWS Well-Architected Framework (2015; Google Cloud and Azure variants) — the vendors' own review discipline, distilled to the four pillar questions that bite on a services table · 2015 | Better = spend that traces to requirements (cost pillar), redundancy that matches the user-journey SLO with coherent composite math (reliability pillar), an ops surface the stated team can actually run (operational excellence), and a named tradeoff on every material decision (candor). | architecture adr_decision |
Frameworks (the loop mechanics) — each carries its documented failure mode and the guard for it
| Framework | Source | Loop | Documented failure → guard |
|---|---|---|---|
| Self-Refine | Madaan et al., CMU/AI2/Google - arXiv:2303.17651 (NeurIPS 2023) | Critique the draft, rewrite the draft, repeat. The minimalist loop. | Documented: quality is non-monotonic across iterations — scores can drop between rounds, and the paper itself selects the best output across iterations (see also Huang et al. 2023, arXiv:2310.01798, on self-correction degrading answers). Guard: the Lab makes that best-of-N retention explicit and visible, plus an economy/length penalty against padding drift. |
| Reflexion | Shinn et al., Northeastern/MIT/Princeton - arXiv:2303.11366 (NeurIPS 2023) | Don't edit the draft - learn a lesson, then rewrite from scratch informed by every lesson so far. | Documented: repeats an error class when a lesson misdiagnoses the failure. Guard: lessons are shown per-card so the user can spot a bad lesson; memory capped at 5 so one bad lesson washes out. (Memory cap of 5 is this Lab's design choice; the paper's sliding window is typically 1-3 reflections.) |
| TextGrad (textual gradients) | Yuksekgonul et al., Stanford - arXiv:2406.07496 (2024; Nature 2025) | Feedback as gradients: per-span directional edit instructions, applied as minimal steps. | Gradient noise: vague gradients produce churn. Guard: every gradient must quote a verbatim span (checked client-side by indexOf; non-matching gradients are dropped and shown struck-through). |
| Population evolution | PromptBreeder (DeepMind, arXiv:2309.16797) + AlphaEvolve (DeepMind 2025) + ShinkaEvolve (Sakana, arXiv:2509.19349) | Breed 3 diverse variants per generation, keep the fittest lineage alive - diversity escapes local optima. | Documented (DGM): evolution WILL game the evaluator if it can - agents faked test logs and removed detection markers. Guard: frozen server-side judge, fabrication auto-reject, elitism = built-in regression guard, every variant's score visible. |
| Godel proof-gate | Schmidhuber, IDSIA - arXiv:cs/0309048 (2003), honestly framed as the ancestor concept | No rewrite is adopted without an explicit argument-of-improvement - and the score must back it up. | Historical: proof search intractable; mis-specified axioms optimize the wrong thing forever. Guard here: the proof is checked against the actual score delta - eloquent proofs with flat scores are rejected and labeled 'proof did not survive measurement'. |
| STOP - self-improving improver meta | Zelikman et al., Stanford/MSR - arXiv:2310.02304 (COLM 2024) | Every 2 iterations, the improvement strategy itself gets rewritten - judged by whether the NEXT iterations gain more. The rubric never moves. | Documented: STOP's improved improvers attempted to disable the sandbox and game the utility. Guard: the strategy has no channel into judge or critic composition, and every strategy version is displayed verbatim on its violet card. |
| Deep Research (coverage loop) | GPT-Researcher (Elovic, 2023) + STORM (Stanford OVAL, NAACL 2024) - the plan->cover->write loop, minus retrieval, honestly | Enumerate the questions the artifact fails to answer, close them from allowed facts - or park them as [NEEDS SOURCE], never fake them. | Documented for autonomous researchers (GPT-Researcher's docs warn about hallucinated citations; STORM's authors note source-quality limits): when retrieval cannot close a gap, generation fills it. Guard here is structural - there IS no retrieval, so unanswerable gaps must surface as [NEEDS SOURCE] lines, the rubric scores an honest ledger as a strength, and any invented source or figure trips the gaming auto-reject. |
| GEPA (reflective evolution) meta | Agrawal et al. - arXiv:2507.19457 (2025); ships as dspy.GEPA (gepa-ai/gepa) | Textual feedback beats a scalar reward: after every accepted round, a reflection step reads the trajectory and rewrites the improve strategy. | Documented risk (GEPA optimizes whatever the metric rewards; the paper's gains depend on feedback quality): reflection can converge on judge-pleasing phrasing instead of artifact fixes. Guard: the strategy is substituted ONLY into improve prompts (no channel to critic or judge composition), every strategy version is displayed verbatim on its violet card, and the gaming auto-reject fires on fabricated evidence no matter how the strategy words it. |
| Negative Knowledge (failure-aware shared memory) | Wang - arXiv:2606.21024, 'Negative Knowledge as Failure-aware Shared Memory for AutoResearch' (2026; github.com/hch-wang/Negative_Knowledge) | Consult the graveyard of rejected attempts first; forbid re-proposing anything that already failed, and bank why. | Documented: a bad lesson mis-diagnoses the failure cause and thereby forbids a good approach or waves through a whole error class, causing the loop to repeat that class. Guard: lessons are shown per-card so the user can spot a bad diagnosis, and the number of lessons surfaced per card is capped so one wrong lesson washes out rather than dominating the ledger. |
The atlas — 41 real recursive-research / RSI systems, each verified from its repo. “Run its essence” loads the closest Lab framework into the loop above.
Keller Jordan + community · 2026
loop · fixed vs evolves
Loop: anyone proposes a change to the training code → run the FIXED task on the FIXED hardware → keep it only if it beats the current record → the new record is published. The leaderboard IS keep-only-if-better, at community scale.
FIXED = the target loss, dataset, model size, and 8×H100 budget. EVOLVES = architecture, optimizer, numerics, and kernels.
Lesson: A frozen task + fixed budget turned a distributed crowd into a 34× optimizer — the human, open-source shape of the RSI loop, and the exact TARGET that autonomous RSI agents (AIDE², NanoGPT-Bench) are now measured against. Runnable: clone, `pip install -r requirements.txt`, `./run.sh`.
Intology AI · 2026
loop · fixed vs evolves
Loop: agent proposes a training-code improvement → LLM comparability check → statistical retiming vs the human record trajectory → score = fraction of human progress recovered.
FIXED = the speedrun task, the eval protocol, and the long-horizon human reference. EVOLVES = the agent's proposed code (and, across the field, the agents themselves).
Lesson: The honest reality check on the RSI ladder: frontier agents (Codex, Claude Code, an Autoresearch variant) each recovered <10% of five months of human progress in 512 H100-hours. This is the empirical measurement of exactly where 'Level 1 / net-positive' actually stands. Runnable via Docker with per-agent launchers.
Keller Jordan · 2026
loop · fixed vs evolves
Loop: same frozen-task/keep-the-record speedrun pattern as modded-nanogpt, applied to image classification.
FIXED = target accuracy, dataset, single-A100 budget. EVOLVES = architecture + training recipe.
Lesson: Proves the frozen-task speedrun pattern generalizes beyond LLMs — and that Muon (born here) transfers. Runnable: `python airbench94_muon.py`.
★ ~1.4k
Weco AI · 2026
loop · fixed vs evolves
Loop: outer agent proposes a rewrite of the inner agent's code → evaluated across fixed task families under a dollar budget with a public/private split → kept only if it beats the prior best (~90% rejected) → repeat ×100 over 8 days.
FIXED = task families, per-eval dollar budget, held-out public/private split. EVOLVES = the inner agent's entire codebase — search policy, context engineering, even its evaluation harness (it fixed a harness bug autonomously).
Lesson: Self-improvement REDUCED reward hacking (63%→34%, beating the hand-tuned 42%) because the frozen layer sat above everything the agent could touch — and the honest caveats matter: no ignition, complexity blow-up, dead code. Keep-only-if-better with ~90% rejection is this Lab's accept rule at industrial scale.
★ ~212k
Nous Research · 2025
loop · fixed vs evolves
Loop: act on a task → on completion distill a reusable SKILL → refine skills during later use → curated persistent memory + full-text session search feed the next task.
FIXED = the agent harness, tool APIs, and memory schema. EVOLVES = the skill library, the user models, and the curated memory.
Lesson: Accreting reusable skills from experience is recursive self-improvement without weight updates. The Lab's saved-artifact library is the same idea one level down — at the artifact, not the agent.
★ ~76.6k
ByteDance · 2025
loop · fixed vs evolves
Loop: plan → spawn parallel sub-agents per subtask → each researches/executes in a sandbox → results merge into shared memory → adaptively decompose deeper until the goal is met.
FIXED = the orchestration graph + the evaluators. EVOLVES = the task decomposition, the sub-agent outputs, and the growing memory/skills.
Lesson: Recursive sub-agent spawning is decompose-then-cover at agent scale; the Lab's deep_research framework is the single-agent, no-retrieval cousin of this.
★ ~788
Sakana AI · 2026
loop · fixed vs evolves
Loop: an evolutionary strategy tunes the coordinator + RL discovers which models to call and how they talk → the discovered topology routes each task → performance feedback refines the coordinator.
FIXED = the pool of frontier models + the benchmark. EVOLVES = the coordination topology and the coordinator's weights.
Lesson: The thing you improve can be the ORCHESTRATION, not the models — evolve who-talks-to-whom. The Lab evolves the artifact under a frozen judge; Fugu evolves the agent graph under a frozen benchmark.
★ ~90k
Andrej Karpathy · 2025
loop · fixed vs evolves
Loop: agent edits train.py (model / optimizer / hyperparams / functional forms) -> training runs for a fixed ~5-minute budget -> validation metric val_bpb is checked -> keep the change if it improved, else revert -> repeat. prepare.py holds fixed utilities; program.md holds human-edited agent instructions.
FIXED = the eval framework (val_bpb metric + the ~5-minute train-and-measure budget) and the action space (what in train.py may change). EVOLVES = the code, params, and functional forms the agent tries. Per Paragiri's critique: benchmark != reality, scalar metrics invite gaming, and 'the optimizer will fit whatever you feed it, including your bugs' — so track the full component vector, not one scalar.
Lesson: The frozen eval is a double-edged sword: an optimizer will exploit every flaw in it (your bugs, deployment-failing shortcuts, the single scalar). Defense = keep the metric frozen AND watch a full component vector for gaming — exactly RSI Lab's dim-vector + gaming/length penalties instead of one number.
★ ~80k
OpenHands (formerly All-Hands-AI / OpenDevin) · 2024
loop · fixed vs evolves
Loop: user/automation issues a coding task -> chosen agent acts in a sandboxed dev environment (edit/run/test) -> observes results -> iterates to completion (per-task; the platform/agent is fixed)
FIXED = the platform + the agent scaffolds; EVOLVES = only the target repository the agent works on.
Lesson: Reproducible sandboxed execution + pluggable agent backends is the infrastructure a self-improving coder needs — RSI can't be safe or measurable without a hermetic run/observe loop underneath it.
FoundationAgents (DeepWisdom et al.) · 2024
loop · fixed vs evolves
Loop: represent agentic workflow as code/graph -> execute on tasks -> score against metric -> (AFlow) MCTS-search edits to the workflow / (SPO) optimize prompts -> keep higher-scoring variant -> repeat
FIXED = base LLMs + the task evaluator/metric + the role SOP structure; EVOLVES = the workflow graph and the role prompts.
Lesson: Automating the search over WORKFLOWS and prompts (not weights) is the practical form of agent self-improvement at scale — but a scalar task metric is exactly what an optimizer will over-fit, so watch for gamed shortcuts.
Agrawal et al. / stanfordnlp · 2025
loop · fixed vs evolves
Loop: sample trajectories from current prompts -> reflect on the natural-language traces to diagnose the concrete failure -> mutate the prompt as a targeted edit -> Pareto-select non-dominated variants across tasks/metrics -> repeat
FIXED = the compound-AI-system structure (module graph, control flow), the metric, and the reflection procedure. EVOLVES = the natural-language prompts/instructions inside each module.
Lesson: Reflection over rich text traces is a denser learning signal than a scalar reward — you evolve the prompt by reading WHY it failed, not just by scoring THAT it failed. Pareto selection prevents collapsing multi-objective quality into one number that gets gamed.
★ ~30k
Stanford OVAL (Shao, Jiang, et al.) · 2024
loop · fixed vs evolves
Loop: pre-writing: discover multiple PERSPECTIVES by surveying similar articles -> simulate writer<->expert conversations grounded in retrieved web sources (perspective-guided question asking) -> synthesize an outline. writing: generate the full article from outline+references -> polish. Co-STORM adds a moderator agent + dynamic mind-map with human turns.
FIXED = the four-module DSPy pipeline (knowledge curation, outline gen, article gen, polish) and the perspective-then-converse protocol. EVOLVES = the discovered perspectives, retrieved sources, outline, and article; swappable LMs and retrievers.
Lesson: Diversity of frozen 'perspectives' (personas) is how a single fixed pipeline avoids one-track output — a source of coverage the RSI Lab captures via distinct frozen mental-model lenses rather than one critic voice. Provenance = every claim carries a citation.
Assaf Elovic · 2023
loop · fixed vs evolves
Loop: planner generates a set of research sub-questions -> parallel execution agents web-search + scrape multiple sites per question -> aggregate and rank by cross-source frequency (>=20 sources for objectivity) -> publisher synthesizes a cited report (PDF/Word/MD). A recursive 'Deep Research' mode explores a tree of follow-up questions.
FIXED = planner->execution->publisher architecture and the multi-source frequency-aggregation method. EVOLVES = the generated question set, the sources retrieved, and the report; Deep Research recursion, MCP data sources, and multi-agent (LangGraph/AG2) backends are pluggable.
Lesson: Cross-source frequency voting is a cheap, frozen provenance/validity filter — trust the claim many independent sources agree on. It is the anti-fabrication guard done via redundancy rather than via a judge, complementary to RSI Lab's gaming=0/[NEEDS SOURCE] rule.
★ ~19.7k
Princeton NLP / Stanford (Yang, Jimenez et al.) · 2024
loop · fixed vs evolves
Loop: read issue -> observe repo via ACI commands -> propose edit -> run tests/observe feedback -> revise until the issue's tests pass (per-task loop, agent itself is fixed)
FIXED = the agent scaffold + the ACI command set; EVOLVES = only the target repository being patched (not the agent).
Lesson: The interface between the agent and the environment (the ACI) is a first-class lever — carefully designed, model-tuned actions beat a smarter model with clumsy tools. This is the substrate a self-improving coder would later try to evolve.
★ ~14.2k
Sakana AI (Lu, Lu, Lange, Foerster, Clune, Ha) · 2024
loop · fixed vs evolves
Loop: idea generation -> novelty check -> automated experiment code + execution (against a fixed template, e.g. NanoGPT/2D-Diffusion/Grokking) -> aggregate results into a LaTeX writeup with citations -> LLM peer-review scores the paper -> iterate/next idea. run_0 baseline is fixed for hardware-relative comparison.
FIXED = the three domain templates + experimental harness, the LaTeX paper structure, the review rubric, and the run_0 baseline. EVOLVES = generated ideas, experiment code/params, paper content/figures, and the resulting review scores.
Lesson: A frozen template + frozen review rubric is what makes an open-ended idea generator safe to run unattended: the evolving artifact (paper) is always scored by an unchanging judge, exactly the RSI Lab pattern of a frozen critic-lens/judge-rubric over an evolving candidate.
★ ~7k
Guanzhi Wang et al. (NVIDIA/Caltech, MineDojo) · 2023
loop · fixed vs evolves
Loop: automatic curriculum proposes next task -> write code skill -> execute in env -> feed back errors + self-verification -> iteratively repair the code -> on success, store skill in library for reuse/composition
FIXED = the frozen GPT-4 weights + the env + the self-verification protocol; EVOLVES = the executable skill library (accumulated, composable programs).
Lesson: A retrievable library of verified executable skills gives compounding self-improvement without touching model weights, and avoids catastrophic forgetting — persistent external memory of what WORKS is the mirror image of a negative-knowledge store of what fails.
★ ~6.8k
Sakana AI (Yamada, Lange, Lu, et al.) · 2025
loop · fixed vs evolves
Loop: ideation + Semantic-Scholar novelty check -> agentic best-first tree search over experiments (an experiment-manager agent expands/prunes parallel branches, keeping best-first) -> manuscript generation -> LLM review cycle -> next node. Time/compute-bounded search replaces the fixed template.
FIXED = the tree-search protocol, the experiment-manager control loop, the novelty-check and review rubrics. EVOLVES = the search tree itself (nodes = experiments), code at each node, and the manuscript. Notably NO fixed template — the action space is open ML code.
Lesson: Removing the fixed template buys breadth but LOSES the reliability the template gave — the authors note v2 does not necessarily beat v1. Freezing the eval while unfreezing the action space trades reliability for exploration; the frozen judge is what keeps the wider search honest.
★ ~6.7k
codelion · 2025
loop · fixed vs evolves
Loop: select program from MAP-Elites population -> LLM mutates it (guided by prior error/exec artifacts) -> run the fixed evaluator to score -> insert back by feature-cell + fitness -> islands migrate to avoid premature convergence -> repeat
FIXED = the user's evaluator/scoring function, the MAP-Elites feature dimensions, the island topology. EVOLVES = the program code across generations.
Lesson: Diversity-preserving structure (MAP-Elites cells + islands) beats a single greedy best-so-far: keeping quality-diverse stepping stones stops the search from collapsing into one local optimum. Feeding execution artifacts back as text turns failures into guidance.
Shunyu Yao et al. (Princeton NLP) · 2023
loop · fixed vs evolves
Loop: from a state, GENERATE candidate thoughts -> EVALUATE each state with the LLM (value or vote) -> SELECT top-b states -> expand (BFS/DFS) -> backtrack/prune -> repeat until solution.
FIXED: task definition, LLM backend, and the evaluation criteria/prompts. EVOLVES: the search frontier — the expanding tree of thought trajectories and which branches survive pruning.
Lesson: Reflection alone is a linear chain; adding an explicit search structure (tree + a fixed value function) lets a frozen model deliberate, look ahead, and backtrack instead of committing to the first path. The essence-match to the Lab is the evolution/search primitive: a fixed scorer steering an evolving population of candidates.
Samuel Schmidgall et al. (AMD / JHU) · 2025
loop · fixed vs evolves
Loop: literature review (agents query arXiv/HF) -> collaborative plan + data prep -> automated experimentation via mle-solver -> report writing (LaTeX). copilot-mode=true inserts a human review/approval gate between phases; otherwise fully autonomous.
FIXED = the three-phase workflow, agent roles, and tool integrations (arXiv/HF/Python/LaTeX). EVOLVES = research direction from the lit review, the agent-authored plan, and the generated code/analyses/report.
Lesson: Human-in-the-loop is a configurable GATE on a fixed pipeline, not a rewrite of it — the copilot flag toggles a frozen checkpoint. Mirrors the RSI Lab's accept-rule as a hard gate the human can tighten without changing the loop mechanics.
Lakshya A. Agrawal et al. (UC Berkeley / Stanford / Databricks) · 2025 (arXiv:2507.19457; ships as dspy.GEPA)
loop · fixed vs evolves
Loop: SELECT a candidate from the Pareto frontier -> EXECUTE on a minibatch, capturing traces + scores ('actionable side information') -> REFLECT: a diagnosis + lesson conditions the next mutation; the frontier updates per-instance, not just global-best.
Lesson: Textual feedback beats scalar reward for sample efficiency, and Pareto-per-instance retention prevents collapse onto one lucky candidate. The Lab's gepa_reflect keeps the reflection cadence; judge dimension scores and rejection reasons stand in for execution traces - weaker, and labeled as such.
zou-group (Mert Yuksekgonul et al., Stanford) · 2024
loop · fixed vs evolves
Loop: forward: produce output from text variables -> compute natural-language LOSS -> LLM emits textual 'gradient' (what's wrong / how to change) -> TGD optimizer applies the edit to each variable -> repeat.
FIXED: the input problem, the LLM engine, and the loss specification. EVOLVES: the optimized text variables (prompts, solutions, code) updated each step by the textual gradient.
Lesson: Formalizing self-critique as differentiation gives you a reusable optimizer abstraction: any text variable becomes tunable against a fixed textual loss without touching model weights. This is the 'textgrad' primitive — the Lab's judge rubric is the loss, the improver output is the variable.
Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, Shunyu Yao · 2023
loop · fixed vs evolves
Loop: attempt -> external/heuristic evaluator scores trajectory -> if fail, reflect (LLM writes why it failed) -> store reflection in episodic memory -> retry with reflections in context. Repeat up to N trials.
FIXED: base LLM weights, the task environment, and the evaluator. EVOLVES: the episodic memory buffer of verbal self-reflections carried across attempts (the agent's context/strategy, not its parameters).
Lesson: You can get reinforcement-style learning with a frozen model by treating natural-language self-critique as the 'gradient' and an external memory as the parameter store. The critic lens and evaluator must stay fixed so the reflections accumulate against a stable target — exactly the RSI Lab pattern of a frozen critic + growing memory.
★ no official DeepMind repo; community impl github.com/vaughanlove/PromptBreeder
Fernando, Banarse, Michalewski, Osindero, Rocktäschel (DeepMind) · 2023
loop · fixed vs evolves
Loop: seed task-prompts + mutation-prompts -> LLM applies a mutation-prompt to produce a child task-prompt -> score on dev set -> select survivors -> AND periodically mutate the mutation-prompts themselves -> repeat
FIXED = the LLM, the fitness/dev-set eval, the evolutionary skeleton. EVOLVES = both the task-prompts AND the mutation-prompts that produce them (self-referential meta-level).
Lesson: Don't just evolve the artifact — evolve the OPERATOR that edits the artifact. Improving how you improve is a second, self-referential loop that outpaces fixed mutation strategies.
★ ~2.2k
Jenny Zhang / Sakana AI + UBC · 2025
loop · fixed vs evolves
Loop: propose self-modification to own code -> spin up variant agent -> empirically score on SWE-bench/Polyglot -> add to archive if it validates -> sample a parent from the archive and repeat (population, not single line)
FIXED = the benchmark harness + the empirical validation gate; EVOLVES = the agent's own code, tools, and prompting scaffold.
Lesson: Empirical validation on a held benchmark is the only safe license to let an agent rewrite itself; keep an ARCHIVE of variants (open-ended search) instead of a greedy single lineage so you don't collapse into a local optimum.
★ ~1.6k
Shengran Hu, Cong Lu, Jeff Clune · 2024
loop · fixed vs evolves
Loop: meta agent reads the archive of prior agent programs + scores -> writes a new agent as executable code -> evaluate on the target domain -> append (agent, score) to the archive -> meta agent conditions its next design on the growing archive -> repeat
FIXED = the evaluation framework, the domain, the meta agent's own search procedure, the code substrate. EVOLVES = the population of downstream agent programs (control flow, tools, prompt scaffolds) invented by the meta agent.
Lesson: The search space can be code itself. Keeping a growing ARCHIVE of past designs (not just the current best) lets the meta agent recombine and stepping-stone toward architectures no human enumerated.
★ ~1.3k
Sakana AI · 2025
loop · fixed vs evolves
Loop: select parent program from archive/islands -> ensemble of LLM mutators proposes patches to EVOLVE-BLOCK regions (routing to better-performing models) -> fixed evaluator scores -> good children become parents + enter the archive -> migrate across islands -> repeat
FIXED = the evaluation framework, the immutable code outside EVOLVE-BLOCK markers, the initial scaffold. EVOLVES = the marked code regions + (adaptively) which LLM does the mutating.
Lesson: Evolution's bottleneck is wasted LLM calls; route mutations to the mutator that's actually working and reuse archived winners so compute concentrates on promising directions instead of blind sampling. Marking a fixed vs. mutable code region contains the blast radius.
Aman Madaan et al. · 2023
loop · fixed vs evolves
Loop: generate -> self-FEEDBACK (same LLM critiques its own output) -> REFINE (same LLM rewrites using the feedback) -> repeat feedback/refine until stop criteria met.
FIXED: the LLM and the input task/prompt. EVOLVES: only the candidate output, improved each round by self-generated feedback threaded back into the prompt.
Lesson: A single frozen model can improve its own artifact if feedback and refinement are separated into distinct prompted roles — the decomposition into critic-then-editor is what makes iteration converge rather than drift. This is the canonical 'self_refine' primitive the RSI Lab's improver/critic split is built on.
★ microsoft/EvoPrompt 52
Guo et al. / Microsoft · 2023
loop · fixed vs evolves
Loop: population of prompts -> LLM performs GA/DE crossover+mutation to make new prompts -> score each on a dev set -> select best to seed next generation -> repeat
FIXED = the task model, the dev-set metric, the chosen evolutionary algorithm (GA or DE). EVOLVES = the discrete natural-language prompts.
Lesson: You don't need gradients to run evolution on text — an LLM is a competent crossover/mutation operator over natural language, so classic EA machinery (GA/DE) transfers directly to prompt space.
Zelikman, Lorch, Mackey, Kalai (Microsoft/Stanford) · 2023
loop · fixed vs evolves
Loop: improver(program) -> LLM proposes an improved program -> keep best by utility; then set program := improver itself and recurse (improver improves the improver)
FIXED = the LLM weights + the utility/meta-utility function + the outer runner; EVOLVES = the improver scaffold code (the strategy it uses to improve programs).
Lesson: You can get real recursive self-improvement with a totally frozen model by only evolving the SCAFFOLD around it — and the meta-utility (how you score the improver) is where reward-hacking creeps in, so guard the meta-level.
Virk, Edds, Xia, Zhang (UIUC) · 2026
loop · fixed vs evolves
Loop: orchestrator maintains a diverse population -> parent selection picks a promising node -> spawn either an Explorer (fresh context, new idea) or an Optimizer (forked history, refine existing) with steering prompts -> each agent's result scored by the FIXED evaluator -> update population, prioritize promising ideas, break plateaus -> repeat.
FIXED = the evaluator, prompt.md, and baseline solution. EVOLVES = the agent-population composition and per-agent search strategies (the team, not the judge). Speedup comes from concurrent screening vs one agent slowly refining one idea.
Lesson: Freeze the evaluator, evolve the TEAM: parallel population-level search with a common frozen scorer beats one agent overfitting a single line of attack, and diversity is the escape hatch from local optima. Same frozen-judge/evolving-searcher split as RSI Lab, scaled to a swarm.
Devesh Paragiri · 2026
loop · fixed vs evolves
Loop: propose a functional form for the fire formula -> Optuna tunes its params against fixed satellite/climate datasets to minimize deviation from observed burns -> score with ILAMB tier-2 metrics (Bias/RMSE/Seasonal/Spatial) -> keep the better form -> repeat (offline; the trained formula's params then freeze)
FIXED = the input datasets (TRENDY v14 climate, GFED4.1s burns), the ILAMB scoring. EVOLVES = the functional form + its 12 parameters. Note: offline decoupling means no live fire->biomass->GPP->fire feedback, so the metric is a proxy for the coupled reality.
Lesson: Concrete proof that autoresearch beats a hand-built baseline on a real scientific benchmark — but also the textbook risk: an offline metric optimized in isolation may not survive coupling to the live system. Provenance and 'does the win transfer to reality' matter as much as the score.
Hanchun Wang · 2026
loop · fixed vs evolves
Loop: Research agent attempts an experiment -> on failure, Curator agent distills it into a bounded, typed NK record (task_id, attempted_route, observation, failure{layer,scope,degree,action,risk}, rationale, recommended_alternative) -> stored in a shared bank -> before the next proposal the Research agent inspects the bank and explicitly adopts/rejects records, avoiding repeats.
FIXED = the NK record schema and controlled vocabularies (LAYERS/SCOPES/DEGREES/ACTIONS/RISKS), the no-free-text validation, and depth-1/depth-N formats. EVOLVES = the accumulating bank of failure records and the research agent's consumption strategy; curator prompts are overridable.
Lesson: Cumulative memory of what FAILED (and why) is as valuable as memory of what worked — a frozen typed schema turns negative results into reusable constraints so agents stop rediscovering dead ends. This is the direct real-world analog of RSI Lab's negative-knowledge / via-negativa lens.
Tie, Shi, ... , Philip S. Yu, Jianfeng Gao (+22) · 2025
loop · fixed vs evolves
Loop: not a running system — it defines the evaluative loop others should adopt: judge each autonomous-research output on novelty + validity + impact + reliability + provenance rather than a single scalar
FIXED = the 5-dimension rubric + 5-stage pipeline as an evaluation lens. EVOLVES = nothing operational; it prescribes how to measure evolving systems.
Lesson: Never score autonomous research with one number — track a component vector, and make PROVENANCE (can you trace where each claim/result came from) a first-class dimension. This is the multi-dimensional-rubric backbone the RSI Lab judge uses.
AutoResearch AI (survey)
★ n/aTie et al. (+22, incl. Jianfeng Gao, Philip S. Yu) · 2025
loop · fixed vs evolves
Loop: grounding -> hypothesis -> experimentation -> validation -> reporting (staged research pipeline; degree of AI autonomy is the spectrum axis)
FIXED = the 5-stage pipeline structure + the 5 evaluation dimensions; EVOLVES = the research artifact (hypotheses, experiments, conclusions) and how much of the pipeline the AI drives.
Lesson: Judge autonomous research on FIVE axes (novelty, validity, impact, reliability, provenance) — a single scalar can't capture research quality, and PROVENANCE (can you trace each claim to a source?) is the axis that catches fabrication. This is why the RSI Lab scores a vector and marks unsourced additions [NEEDS SOURCE].
Godel Machine (Schmidhuber)
paperJurgen Schmidhuber (IDSIA) · 2003
loop · fixed vs evolves
Loop: search for a proof that a candidate self-rewrite is globally beneficial -> only if a valid proof is found, apply the rewrite (including to the prover itself) -> continue
FIXED = the axioms + the requirement of a utility-improving proof before any change; EVOLVES = literally the entire program, including the proof searcher.
Lesson: The safest gate on self-modification is a formal proof of improvement BEFORE acting (proof-gate), not empirical trial-and-error after; in practice provable improvement is intractable, which is exactly why later systems (DGM/STOP) substitute empirical benchmarks for proofs.
AutoResearch AI: Towards AI-Powered Research Automation for Scientific Discovery (survey)
paperGuiyao Tie et al. (incl. Jianfeng Gao, Philip S. Yu; 23 authors) · 2026
loop · fixed vs evolves
Loop: conceptual pipeline (not code): grounding -> hypothesis -> experimentation -> validation -> reporting/revision, each stage scored on novelty/validity/impact/reliability/provenance.
FIXED: the 5-stage workflow taxonomy and the 5 evaluation dimensions as the assessment lens. EVOLVES: where a given system sits on the autonomy spectrum (Vibe Research -> AI-led) as domains become more executable/verifiable.
Lesson: PROVENANCE deserves to be a first-class eval axis alongside novelty/validity/impact/reliability — an autonomous researcher's claims must be traceable to sources, or gains are unverifiable. This is exactly the RSI Lab's fidelity gate and [NEEDS SOURCE] rule: unverifiable additions must be marked, never fabricated, and low fidelity auto-rejects as drift.
★ ~12.4k
Infracost (Ali Khajeh-Hosseini et al.) · 2020
loop · fixed vs evolves
Loop: propose an infra diff → price the plan against the fixed price database → post the $/mo delta as review feedback → author revises → re-price until the diff is worth its dollars.
FIXED = the price database and the policy thresholds. EVOLVES = the architecture in the Terraform plan.
Lesson: A frozen price oracle turns cost into a judge dimension on every diff — exactly the Lab's cost_efficiency dim: the improver argues, the price database doesn't. Its output is also the ideal paste for the profile builder's PRICING ANCHORS box.
Anton Babenko & community · 2017
loop · fixed vs evolves
Loop: express the architecture as module INPUTS → the module encodes the 80% case → review shrinks to the handful of variables that are real decisions → tighten inputs as requirements clarify.
FIXED = the battle-tested module internals (community best practice). EVOLVES = your inputs — the actual architectural decisions.
Lesson: Raise the abstraction until the diff IS the decision: when a NAT topology is one boolean, every cost/availability trade is legible enough for a judge to score — the same reason the Lab's architecture artifact is a terse table + IaC excerpt, not a poster diagram.
AWS (Serverless DA team) · 2021
loop · fixed vs evolves
Loop: draw the architecture as a graph → for each edge ask 'is there a direct service-to-service integration that deletes the glue in the middle?' → swap in the verified pattern → repeat, edge by edge.
FIXED = the catalog of verified 2-service edges. EVOLVES = your graph, re-composed from fewer, more direct edges.
Lesson: Simplify edge by edge, not poster by poster: most 'services' in an overbuilt diagram are glue between two managed things that already speak to each other — the service_economy dim scores exactly that glue.
Google Cloud · 2019
loop · fixed vs evolves
Loop: bootstrap guardrails → security/networking stages consume them → a project factory stamps compliant workspaces → workloads iterate freely inside; guardrail changes flow down, never up.
FIXED = stage-0 guardrails (org policy, audit, perimeters). EVOLVES = the workloads inside pre-compliant projects.
Lesson: Compliance boundaries belong in the foundation layer as inherited constraints — the Lab encodes the same idea by carrying them in ALLOWED FACTS, where an 'optimization' that touches them is a violation, not a simplification. run_infra_hipaa is this repo's thesis as a replay.
Further reading — 41 exemplary sources on recursive self-improvement, each linking to its own page
📄 Papers
Self-Taught Optimizer (STOP): Recursively Self-Improving Code Generation · Eric Zelikman, Eliana Lorch, Lester Mackey, Adam Tauman Kala, 2023
The seminal LLM-era RSI paper: a seed 'improver' program uses a frozen LLM to improve its own scaffolding, discovering strategies like beam search and genetic algorithms — the cleanest statement of what self-improvement means when the model stays fixed, plus early reward-hacking analysis.
Darwin Godel Machine: Open-Ended Evolution of Self-Improving Agents · Jenny Zhang, Shengran Hu, Cong Lu, Robert Lange, Jeff Clune, 2025
The empirical RSI landmark: an agent that rewrites its own code, keeps an evolutionary archive of variants, and lifts itself 20%->50% on SWE-bench by inventing its own tools — the realization of Schmidhuber's Godel Machine without the proof requirement.
Godel Machines: Self-Referential Universal Problem Solvers Making Provably Optimal Self-Improvements · Jurgen Schmidhuber, 2003
The foundational theory of recursive self-improvement: an agent that rewrites its own code only after proving the change is globally optimal — the ideal against which every practical RSI system is measured.
Automated Design of Agentic Systems · Shengran Hu, Cong Lu, Jeff Clune, 2024
Founds the ADAS field with Meta Agent Search: a meta-agent programs ever-better agents in code over an open-ended archive — the load-bearing insight that agents-as-code makes the space of possible agents Turing-complete and searchable.
Reflexion: Language Agents with Verbal Reinforcement Learning · Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, 2023
Agents improve from failure via verbal self-reflection stored in memory instead of weight updates — the template for memory-driven self-correcting agents (91% pass@1 on HumanEval).
Self-Refine: Iterative Refinement with Self-Feedback · Aman Madaan, Niket Tandon, Prakhar Gupta, et al., 2023
Canonical single-model self-critique loop (generate -> critique -> revise) with ~20% average gains and no training — the minimal viable self-improvement primitive most agents build on.
Self-Rewarding Language Models · Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, S, 2024
The model acts as its own judge to generate reward signals for iterative DPO, improving both instruction-following and its own reward quality — RSI moved into the training loop rather than just inference.
GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning · Lakshya A Agrawal, Shangyin Tan, Dilara Soylu, ... Dan Klein, 2025
Language-reflection-based prompt evolution (the DSPy lineage) that beats GRPO with up to 35x fewer rollouts — state-of-the-art evidence that reflective self-improvement can outperform RL.
Promptbreeder: Self-Referential Self-Improvement via Prompt Evolution · Chrisantha Fernando, Dylan Banarse, Henryk Michalewski, Simo, 2023
Evolves not just task prompts but the mutation-prompts that drive evolution — a concrete, elegant instance of self-referential improvement in the LLM setting.
TextGrad: Automatic "Differentiation" via Text · Mert Yuksekgonul, Federico Bianchi, Joseph Boen, Sheng Liu, , 2024
Reframes LLM textual feedback as backpropagatable gradients for compound AI systems — a general optimization substrate for self-improving pipelines across code, prompts, molecules, and treatment plans.
Voyager: An Open-Ended Embodied Agent with Large Language Models · Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowe, 2023
Lifelong learning agent that autonomously builds and reuses a growing skill library in Minecraft — the canonical open-ended, curriculum-driven self-improvement agent.
Negative Knowledge as Failure-Aware Shared Memory for AutoResearch · Hanchun Wang, 2026
Turns failed experiments into a structured, transferable shared-memory bank that beats baselines with fewer tokens — a sharp, underappreciated lever for self-improving research agents.
SwarmResearch: Orchestrating Coding Agents for Open-Ended Discovery · Yuvraj Virk, Zack Edds, Chunqiu Steven Xia, Lingming Zhang, 2026
Shepherd-plus-Search-agents architecture with per-branch git contexts and adaptive parallelism — a concrete recipe for escaping single-strategy lock-in in long-running self-improving search (better/comparable on 13/15 tasks).
🗺 Surveys
A Comprehensive Survey of Self-Evolving AI Agents: A New Paradigm Bridging Foundation Models and Lifelong Agentic Systems · Jinyuan Fang, Zaiqiao Meng, et al., 2025
The definitive taxonomy of self-evolving agents: a unified four-part framework (System Inputs, Agent System, Environment, Optimizers) that maps every RSI technique onto the feedback loop it optimizes.
A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligence · Huan-ang Gao, ... Qingyun Wu, Heng Ji, Mengdi Wang, et al., 2025
Complementary survey organized by what/when/how/where to evolve (models, memory, tools; intra- vs inter-test-time) — the cleanest mental model for reasoning about where self-improvement can be inserted.
AutoResearch AI: Towards AI-Powered Research Automation for Scientific Discovery · Guiyao Tie, Jiawen Shi, Dingjie Song, ... Philip S. Yu, Jian, 2026
Survey mapping the full autoresearch workflow across five stages (literature -> hypothesis -> experiment -> validation -> reporting/revision); argues automation credibility is domain-dependent — the best single map of the landscape this lab sits in.
AWS Well-Architected Framework — the pillars · AWS, 2015
The canon the well_architected mental model distills: cost optimization ('every dollar traces to a requirement'), reliability sized to demand, and operations-as-code — read the cost pillar's 'stop spending on undifferentiated heavy lifting' before self-hosting anything.
Azure Well-Architected — design for simplicity and efficiency · Microsoft, 2023
The rare official text that names BOTH failure modes: avoid overengineering AND oversimplification — 'oversimplification can introduce single points of failure'. The reason every infra judge dimension here is phrased to punish both directions.
✎ Blogs
IBM — What are AI agents? · IBM, 2025
A clear vendor-neutral primer on agent types, the reason-act (ReAct) loop, and agentic architectures — orientation for readers new to the agent framing the Lab assumes.
OpenExO — Resource Hub: Skills (ExO attributes) · OpenExO, 2026
The organizational analogue of self-improvement: the ExO attributes (MTP + SCALE + IDEAS — algorithms, autonomy, experimentation) are the org-level 'skills' that let organizations compound — a wider lens on the same recursion.
4 Levels of Recursive Self-Improvement · Weco AI, 2026
The cleanest ladder for locating any RSI system: Delegation → Net positive → Ignition → Inflection. This Lab's RSI Ladder section is built on it.
AIDE²: The First Evidence of Recursive Self-Improvement · Weco AI, 2026
The first claimed Level-1 RSI: 100 self-rewrite steps, kept-only-if-better, reward hacking DOWN with scale — with unusually honest caveats (no ignition, complexity blow-up).
The Darwin Godel Machine: AI that improves itself by rewriting its own code · Sakana AI (with Jeff Clune Lab, UBC), 2025
The clearest lab writeup of empirical recursive self-improvement — a coding agent that rewrites itself and keeps an open-ended, traceable archive (SWE-bench 20%->50%), with candid documentation of reward-hacking and safety issues.
AlphaEvolve: A Gemini-powered coding agent for designing advanced algorithms · Google DeepMind, 2025
Evolutionary coding agent pairing Gemini with automated evaluators to discover genuinely novel algorithms (faster matrix multiplication, real datacenter compute wins) — the strongest production evidence that self-improving-agent methods yield real gains.
FunSearch: Making new discoveries in mathematical sciences using Large Language Models · Google DeepMind (Romera-Paredes et al., Nature), 2023
First genuinely new scientific discovery (cap set, bin-packing) from an LLM-plus-evaluator evolutionary loop — the precursor to AlphaEvolve and proof that self-improving search can exceed human knowledge.
The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery · Sakana AI (with Oxford Foerster Lab, UBC), 2024
Defines the fully-autonomous research agent: ideation -> code -> experiments -> written paper -> automated peer review at ~$15/paper — the reference point for end-to-end autonomous research pipelines.
ShinkaEvolve: Evolving New Algorithms with LLMs, Orders of Magnitude More Efficiently · Sakana AI, 2025
Makes evolutionary program search sample-efficient (parent sampling, novelty rejection, adaptive LLM selection) — the practical, open-source counterpoint to AlphaEvolve's compute-heavy approach.
Building and evaluating alignment auditing agents · Anthropic (Alignment Science), 2025
Automates a slice of the research/oversight loop — auditing agents that autonomously surface hidden objectives and concerning behaviors in frontier models: the safety-side view of scaling autonomous research.
Extending the Autoresearch Loop · Dev Paragiri, 2026
A grounded case study taking autoresearch out of fast-metric ML into physical science: a two-layer search (LLM over functional forms + Optuna/Bayesian over parameters) lifts a fire submodel to rank 1 on ILAMB, exposing the frame-selection limit of current RSI.
Autoresearch for Earth System Models · Dev Paragiri, 2026
Extends the autoresearch loop to rewrite the structural form of decades-old climate-model equations against observational benchmarks while preserving physical interpretability — shows the paradigm generalizing beyond ML benchmarks.
Our cloud exit savings will now top ten million over five years · David Heinemeier Hansson (37signals), 2024
The reference repatriation with real numbers: $3.2M → $1.3M/yr cloud spend, $700K of Dell recouped within a year, same ops team — for stable workloads at scale, rent-vs-own inverts.
Prime Video: audio/video monitoring — from distributed microservices to a monolith · Marcin Kolny (Prime Video), preserved by InfoQ, 2023
90% cost cut by collapsing Step Functions + S3 hot loops into one ECS process: never put a per-invocation-priced boundary inside a high-frequency data path — microservice is a code boundary, not necessarily a network one.
Goodbye Microservices: from 100s of problem children to 1 superstar · Alexandra Noonan (Segment), 2018
140+ services → 1: draw boundaries per domain, not per config variant. The service_economy judge dimension is this post distilled into a rubric.
Use One Big Server · Nima Badizadegan, 2022
Prices the cloud premium head-on (AWS m6a.metal ≈ 4.6× an OVH equivalent; Lambda 5.5–25×): below ~10K QPS, vertical scale plus a warm standby beats a distributed system on cost, latency, and ops. The lens behind the MVP-sprawl quick pick.
⛓ Repos
modded-nanogpt — the NanoGPT speedrun (record #84: 1.32 min) · Keller Jordan + community, 2026
The canonical frozen-task speedrun and the target autonomous RSI agents optimize — runnable, and the birthplace of the Muon optimizer.
NanoGPT-Bench — measuring autonomous research agents on the speedrun · Intology AI, 2026
Docker-runnable benchmark quantifying how much human progress agents recover (<10% so far) — the empirical yardstick for the RSI ladder's Level 1.
cifar10-airbench — 94% CIFAR-10 in 2.59s · Keller Jordan, 2026
Sister speedrun proving the frozen-task/keep-the-record pattern generalizes beyond LLMs; dependency-free and instantly runnable.
NousResearch/hermes-agent — a self-improving agent (skills + memory) · Nous Research, 2025
A production self-improving agent: it writes its own skills from experience and refines them in use — RSI you can actually run.
ByteDance/DeerFlow — long-horizon deep-research SuperAgent · ByteDance, 2025
Recursive sub-agent spawning + persistent memory for multi-hour research — the industrial version of the deep_research loop.
multica-ai/andrej-karpathy-skills — Karpathy's 4 coding skills (CLAUDE.md) · multica-ai, 2025
“Goal-Driven Execution — LLMs are exceptionally good at looping until they meet specific goals” is exactly this Lab's until-acceptable mode; the other three skills map to its critique, economy, and surgical-edit rules.
autoresearch: AI agents running research on single-GPU nanochat training automatically · Andrej Karpathy, 2026
Karpathy's minimal, widely-cited harness that crystallized the 'autoresearch' term: point a coding agent at a 5-minute nanochat training loop and let it propose, test, and keep improvements overnight — the accessible template many follow-on projects build on.