Your Agent Can't Grade Its Own Homework: What Reflexion Actually Showed

Ask five agent engineers what "reflection" means and you will get at least five loops.

For Andrew Ng, it is the model criticising its own output and improving its response. Anthropic calls its version evaluator-optimizer: one call generates, another evaluates, in a loop. In Generative Agents, reflection is a periodic synthesis of memories into higher-level inferences, triggered by an importance threshold of 150, with no success signal at all. In LangGraph's "Reflexion" tutorial it is draft, web search, revise, with MAX_ITERATIONS = 5. That tutorial has no pass/fail evaluator, no environment reset and no buffer of lessons carried between attempts.

And in the paper that coined the name, it is something else again.

The word survives because it sounds like learning. The evidence says something narrower:

An agent improves only as far as a signal it does not produce lets it.

"Self-improvement" is four different things at four different scopes: within one run, across attempts at one task, across tasks, and in the weights. How much evidence each one has depends on where its feedback comes from. Start with the paper everyone cites.

What Reflexion actually is

Reflexion (Shinn et al., NeurIPS 2023) is a retry loop for one task. An actor makes a whole attempt. An evaluator returns pass or fail. On a fail, a separate reflection call reads the failed trajectory and writes a few sentences: what went wrong, what to try instead. That lesson goes into a buffer of 1–3 items (Ω = 3 for ALFWorld and HotPotQA, Ω = 1 for programming). The environment resets, and the next attempt starts with the buffer prepended to its prompt.

lessons = []                                   # this task only
for trial in range(max_trials):
    env.reset()                                # a fresh attempt, not an edit
    trajectory = actor.run(task, lessons)
    if evaluator.passed(trajectory):           # env flag, gold answer, tests
        return trajectory
    lessons.append(reflect(trajectory))        # a written lesson
    lessons = lessons[-3:]                     # bounded window, no retrieval
# the lessons are thrown away with the task

No weights change. Nothing carries over to the next task. Remove the buffer and the agent is exactly what it was.

The paper calls this "verbal reinforcement learning". Read that as an analogy. There is no value function, no gradient, no exploration term and no convergence guarantee. The "update" is a frozen model writing text conditioned on a reward, so credit assignment is handed to the same model that just failed. Retroformer (ICLR 2024) made exactly this objection: verbal reinforcement cannot be optimal if the frozen model was never fine-tuned to do credit assignment for the task at hand.

Two things do separate Reflexion from generic reflection. It re-attempts the whole task from a reset state instead of editing the output. That can change early actions in a long trajectory, which no edit of the final answer can reach. And it splits the judge from the explainer: the evaluator decides pass or fail, the reflector only explains. That split is what lets an outside signal carry the weight.

It did carry it.

Where the gains came from

Look at what the evaluator actually was in each experiment:

• ALFWorld: the environment's own won flag. Ground truth.
• HotPotQA: exact match against the gold answer. In the author's words on issue #24: "HotPotQA was the only task in which we used ground truth labels."
• HumanEval and MBPP: up to 6 unit tests the model wrote itself. The hidden tests were used once, for the final score.

The strongest results came from signals the model did not produce. In the released logs, Reflexion completed 130/134 ALFWorld environments by trial 10, against 101/134 (75.4%) for a retried ReAct baseline, judged by the environment. The HotPotQA gains were measured with gold-label stopping.

The code benchmarks are where the signal was the model's own, and there the paper ran its cleanest experiment, almost by accident. On HumanEval Python, the self-written tests had a 1.4% false-positive rate (code that passed them but failed the hidden tests), and Reflexion gained, 80.1 → 91.0. On MBPP Python, the rate was 16.3% and Reflexion lost, 80.1 → 77.1. Same method, same model. The gain tracked the quality of the check. Two benchmarks are not a controlled experiment, but this is the paper's own strongest causal evidence, and it points at the evaluator.

The paper's own ablation (Table 3, the 50 hardest HumanEval Rust problems, GPT-4) says it more bluntly:

• Base model: 0.60
• Reflection without test generation: 0.52
• Tests without reflection: 0.60
• Tests and reflection: 0.68

Reflection without a grounded signal scored below doing nothing. With tests, reflection did add something on top. In the order listed, those scores are 30, 26, 30 and 34 problems solved out of 50, from a single run with no confidence interval. None of the headline results report tokens, cost, or a retry baseline given the same budget.

That last gap is the one that matters.

The compute-matched check

Kapoor et al. ("AI Agents That Matter", TMLR) reran HumanEval with gpt-4-turbo-2024-04-09, averaged over 5 runs, with the same example tests available to every arm. Costs are totals across the 164 problems:

• Plain retry: 92.0% at $2.51
• GPT-4, zero-shot: 89.6% at $1.93
• Reflexion: 87.8% at $3.90

Retry with the same check beat Reflexion and cost less. The zero-shot model, with no loop at all, beat it too. The original 91.0 became 87.8 on the rerun, with a newer model and all 164 tasks; the original run had silently used 161.

The self-correction literature points the same way:

• Huang et al. (ICLR 2024): GPT-4 asked to self-correct with no external feedback went from 95.5 to 89.0 on GSM8K.
• Tyen et al. (Findings of ACL 2024): GPT-4 scored 52.87% accuracy at locating the first reasoning error in a trace. The rest of the title is the useful part: models "can correct them given the error location". The bottleneck is finding the error, not fixing it.
• Wang et al. (EMNLP 2024): self-consistency "consistently beat" Reflexion "with significantly less budget".
• Kamoi et al. (TACL 2024), surveying the field: "no prior work demonstrates successful self-correction with feedback from prompted LLMs, except for studies in tasks that are exceptionally suited for self-correction."

The model's critique of its own output is not a check. It is the same model reading the same evidence a second time.

When self-written memory goes wrong

Reflexion's buffer is small and thrown away after every task, which limits the damage. The damage grows when lessons are wrong and nothing checks them, or when they outlive the task.

Hallucinated diagnosis. In "Honest Lying", a workshop paper at ICML 2026, 0 of 121 reflections named the correct object across 16 stuck ALFWorld environments. Extracting failure signals in code before reflecting raised correct-object mentions from 0% to 86% and fixed 3 of the 16. It is a small workshop result, not a settled one. It does match the mechanism: nothing in Reflexion checks that the diagnosis is right before the next attempt acts on it.

Context collapse. The ACE paper (ICLR 2026) documents context collapse in rewrite-style memory. In one case the context shrank from 18,282 to 122 tokens, and accuracy fell from 66.7 to 57.1, below the 63.7 no-memory baseline. There, the memory made the agent worse than having none.

Raw lessons across tasks. ExpeL (AAAI 2024) tried feeding raw reflections into its cross-task insight extraction. HotPotQA fell from 39.0 to 29.0, barely above ReAct alone at 28.0. Here, at least, lessons about one failed attempt did not transfer to the next task.

What does work

FIG. 1 · FOUR SCOPES

Every approach with evidence behind it ties improvement to a signal from outside the model, at the scope where the improvement has to hold.

1. An outside check plus retry or best-of-N, first. This is the baseline any reflection loop has to beat. Stechly et al. (ICLR 2025): "merely re-prompting with a sound verifier maintains most of the benefits."
2. The raw failure as input, before an extra reflection call. Pass the stack trace, the failing test or the environment message straight to the next attempt. Table 3 above shows reflection only helped once tests supplied the information; without them it scored 0.52. Olausson et al. (ICLR 2024) found the best allocation spends the budget on diverse initial samples, with at most one repair each.
3. A trained reflector. Retroformer freezes the actor and fine-tunes a small reflection model with PPO. With a GPT-3 actor at 4 retries, ALFWorld goes from 84.33% (Reflexion) to 100%. Training cost is excluded from that comparison. It is still the clearest evidence that reflection quality, not reflection itself, is the lever.
4. Trained self-correction. SCoRe (ICLR 2025) moves correction into the weights with RL. On MATH, its second attempt scores 4.4 points above its first; the base model's drops 11.2 and Self-Refine's drops 1.0.
5. Distilled, write-gated memory for learning across tasks. Not raw lessons: abstracted insights, workflows and strategies. Agent Workflow Memory (ICML 2025) reports 35.5% on WebArena against 23.5% for the BrowserGym baseline; ReasoningBank (ICLR 2026) reports 40.5 → 48.8 with Gemini-2.5-flash, though its judge is itself an LLM, 72.7% accurate. The gate matters: in online mode, AWM "can lead to incorrect workflows that degrade model performance". My rule: keep a memory only when a later evaluation confirms it.

Before you add a reflection loop

Four questions separate these cases, and none of them needs a method name. Each answer points at an entry in my agentic pattern catalog.

Is there a success signal the model doesn't produce?

• No → no reflection loop. Use a single pass, or sample and vote (#ap-parallelization).
• Yes → keep going.

Can attempts be repeated without side effects?

• No (payments, emails, writes) → no automatic retry without a sandbox, a checkpoint or a human approval (#ap-durable-execution).
• Yes → start with plain retry or best-of-N against the check (#ap-parallelization). Any loop you add must beat that at equal cost.

Is the failure visible in the output, or only in the trajectory?

• In the output → revise within the run against the check (#ap-reflection), or use #ap-evaluator-optimizer with a judge that is not the generator: tests, a checker, a different model.
• Only in the trajectory → re-attempt from a reset with a capped lesson buffer and a trial cap. This is the one niche where Reflexion is a reasonable candidate. Pilot it against retry at equal budget before you keep it.

Must lessons outlive the task?

• No → throw the buffer away with the task.
• Yes → not Reflexion. Use distilled memory with a write gate (#ap-memory-types, #ap-skill-build).

One more rule sits outside the tree: on a weak or small model, start with repeated sampling, and keep reflection only if it wins at the same budget. StarChat-beta scored 0.26 → 0.26 with Reflexion. A 2026 preprint by Mirzaei reports that all 18 self-inspection comparisons on 1.5–7B models came out negative against repeated sampling. That is one author and small models, so treat it as a direction, not a verdict.

What we still don't know

The best remaining case for Reflexion is narrow. Think of a strong model with a reliable outside signal, working through long trajectories in a resettable environment, where it fails on strategy rather than output. We found no compute-matched study that confirms even that niche.

We found no compute-matched rerun on agentic tasks like ALFWorld, and none on current frontier or reasoning models. The indirect evidence is not encouraging: CorrectBench, a 2025 preprint, reports limited gains at high time cost when an outer correction loop is added to DeepSeek-R1. The paper never ablated the buffer size, and we found no study that did. We found no study testing reflection text against a length-matched placebo. We found none that triggers reflection on the model's own uncertainty either, and in Olausson et al., GPT-4 expressed uncertainty in 0 of 80 self-feedback texts. We also found no study of a Reflexion-style retry loop fed corrupted feedback at a controlled rate.

Until those studies exist, the rule is cheap to follow. Before you build a loop that makes your agent better, name the signal that tells it what better means.

If the only judge in the loop is the model that wrote the answer, you don't have a self-improving agent. You have an agent grading its own homework.