# Atlas comparison record: one reusable experience note

Performed 23 October 2026. This is the public, curated record for “When does an AI agent actually learn from experience?”. Real agent responses were obtained for fictional source summaries. The numbers in those summaries describe invented evaluations; only the responses and their grading are Atlas observations.

## Protocol and limits

The protocol, four tasks, note and binary rubric were fixed before the evaluation sessions. A separate agent distilled an editor-supplied account of earlier research observations into the note: incompatible metrics in Harness²/ACE and conflicting model identifiers in MemoryArena v2. The note author saw neither the test tasks nor their answers. This is a note derived from supplied feedback, not an agent independently discovering and repairing its own failure.

Four fresh-context subagent sessions in the same agent environment, using the inherited model configuration and no override. Exact backend version, seeds, reasoning-token budgets, token consumption, latency and monetary cost were not exposed. Common platform and project instructions may still apply; “fresh” means no parent conversation, not a technical isolation sandbox. Each session was instructed to use no tools; none called a tool.

Two sessions used the common prompt only. Two also received the note. Each answered all four tasks, with no intermediate feedback. First pair used the listed order, second pair reversed it. Launch order: A1, B1, B2, A2. No retries. B adds input text; compute cost is not matched or measured. Four sessions, sixteen task responses; responses within a session are not independent experimental replicates. The tasks were written by the editor, not drawn randomly from a real-work population. Testing a loaded note does not test automatic retrieval, long-term retention or tool execution.

Common instruction: You are checking four short research comparisons for an English-language publication. Use only the supplied fictional materials. For each case return its ID and an assessment of at most 160 words addressing the request. Be accurate, useful and proportionate to the evidence. Do not use tools, browse, inspect files or ask questions. Do not provide hidden reasoning; give the editorial assessment and its necessary calculations. Return all four assessments in order.

## Candidate note (unchanged)

**Use when:** Comparing reported AI results across studies or systems.

**Action:** Record each result’s benchmark, metric, baseline, denominator, and experimental identity. Distinguish benchmark scores, percentage-point gains, and relative percentages. Compare results only where those details support comparison; otherwise present them separately without a pooled ranking. Cross-check model identifiers in tables, captions, and methods. If identifiers conflict, mark experimental identity unresolved until source evidence reconciles them.

**Do not infer:** Larger reported gains establish superiority across different benchmarks, or one source passage silently resolves contradictory identifiers. These editor-supplied research observations do not demonstrate this procedure’s efficacy or describe a failed run.

**Skip when:** The task involves neither comparing reported results nor resolving conflicting experimental details.

## Tasks and checks fixed before execution

Each criterion is 1 if explicitly satisfied, otherwise 0. Equivalent wording and sensible rounding accepted. Cite output evidence. No style points. Do not infer omitted statements. Evaluate the response, not the stated confidence.

### units: Which improvement is larger?

Two fictional independent reports use task-success shares. System Alder rises from 40 correct tasks out of 100 to 50/100. System Birch rises from 60/100 to 72/100. Their task sets and models differ. An editor proposes: 'Alder improves more than Birch: 25% versus 20%.'

Request: Assess the proposed sentence. Give the useful calculations, the defensible comparison and an appropriate publication sentence.

Checks:
1. Correct relative gains: Alder 25%, Birch 20%.
2. Correct absolute gains: Alder 10 percentage points, Birch 12 percentage points.
3. Explains that larger depends on measure and differing tasks/models prevent a general superiority ranking.
4. Offers a publishable sentence with the comparison's scope and no unsupported overall winner.

### budget: What caused the gain?

A fictional evaluation tests the same model, tools and 100 tasks. Baseline: one attempt per task, 64 tasks completed, average cost USD 0.04 per task. Candidate: a reusable memory note plus up to five attempts per task, retaining any successful attempt; 80 tasks completed, average cost USD 0.20 per task. There is no five-attempt baseline without the note. An editor proposes: 'Memory made the agent 25% better at the same job.'

Request: Assess the proposed sentence. Give the useful calculations, the defensible comparison and an appropriate publication sentence.

Checks:
1. Correct observed gain: 16 percentage points OR 16 more tasks, and 25% relative.
2. Identifies extra attempts as a confound and does not attribute the gain to memory alone.
3. Identifies fivefold per-task cost (USD 0.04 to USD 0.20) and proposes an equal-attempt comparison without memory.
4. Offers a publishable sentence describing observed results while keeping the memory contribution unresolved.

### matched: A comparison we can report

A fictional paired evaluation uses the same frozen model, tools, 100 held-out tasks, scoring rules and allowance of one attempt per task. Baseline completes 62 tasks; the candidate with a procedural memory note completes 74. No task-level overlap, repeats, uncertainty interval or measured time/cost is supplied. An editor proposes: 'The candidate completed 12 more of these 100 tasks under matched attempt limits.'

Request: Assess the proposed sentence. Give the useful calculations, the defensible comparison and an appropriate publication sentence. Do not invent unavailable measurements.

Checks:
1. Accepts that the proposed descriptive comparison is supported; does not reject it merely because broader certainty is missing.
2. Correct 12 additional completions OR 12 percentage points; any optional relative gain must be approximately 19.35%.
3. Restricts the claim to this task set and matched attempt allowance; does not assert general reliability, significance or measured cost parity.
4. Offers a publishable sentence without inventing repeated runs, significance, task overlap, time or costs.

### mix: The overall winner changes with the task mix

Two fictional evaluations report success counts on easy and hard tasks. Alder: easy 90/100, hard 20/50. Birch: easy 76/80, hard 50/100. The models and task instances differ; difficulty labels are defined consistently, but there is no randomized comparison. An editor proposes: 'Alder is the more capable agent because its overall success rate is higher.'

Request: Assess the proposed sentence. Give the useful calculations, the defensible comparison and an appropriate publication sentence.

Checks:
1. Correct pooled rates: Alder 110/150, approximately 73.3%; Birch 126/180, 70%.
2. Correct within-group rates: easy Alder 90%, Birch 95%; hard Alder 40%, Birch 50%.
3. Explains differing task mix and rejects the pooled-rate claim of general capability; recognizes Birch's higher observed rate in both groups.
4. Offers a scoped publication sentence and avoids a causal/general superiority claim from the unmatched evaluations.

## Observed grading

A separate AI grader saw neutrally labelled responses, tasks and rubric, without the note or condition mapping. It awarded one point per satisfied check and retained evidence excerpts for every judgment. The author agent checked arithmetic and agreed with the scoring. There was no human rater.

| Case | A1: no note | A2: no note | B1: added note | B2: added note |
| --- | --- | --- | --- | --- |
| Units | 4/4 | 4/4 | 4/4 | 4/4 |
| Attempt budget | 4/4 | 4/4 | 4/4 | 4/4 |
| Matched comparison | 4/4 | 4/4 | 4/4 | 4/4 |
| Task mix | 4/4 | 4/4 | 4/4 | 4/4 |

Each condition met 32/32 checks. These are rubric counts, not a general success rate. The baseline reached the ceiling, so this test could not reveal an improvement above that ceiling. The result does not prove equivalence or show memory is ineffective elsewhere.

Scoring interpretation: naming the missing five-attempt baseline without the note counted as identifying the necessary equal-attempt comparison. Publication sentences were assessed with their preceding assessments; describing the combined memory-and-retries configuration did not attribute its gain to memory alone. No unsubstantiated extra measurement or response-length violation was found.

Decision: KEEP_BASE. No experience note was activated. No time saving, cost improvement, autonomous learning or production reliability is claimed.

## Complete final responses

These are final answers only. No private conversation or hidden internal reasoning is included.

### A1 — no added note

**units** — The percentages are correct as relative increases, but “improves more” is ambiguous. Alder rises from 40% to 50%: **10 percentage points**, or **25% relative to baseline**. Birch rises from 60% to 72%: **12 percentage points**, or **20% relative to baseline**. Alder has the larger relative increase; Birch has the larger percentage-point increase. Different models and task sets prevent a controlled comparison of capability or improvement.

Publication sentence: “In separate evaluations, Alder’s success rate increased by 10 percentage points (25% relative), while Birch’s increased by 12 points (20% relative); the differing models and task sets limit direct comparison.”

**budget** — The completion calculation is correct, but attribution to memory is unsupported. Success rises from 64% to 80%: **16 percentage points**, or **25% relative**. Average cost rises from USD 0.04 to USD 0.20 per task: **fivefold**, or **400%**. The candidate changes both memory and the attempt allowance, counting any successful attempt. Without a five-attempt baseline lacking the note, the memory contribution cannot be isolated. The defensible comparison concerns the combined configuration.

Publication sentence: “On the same 100 tasks, the configuration combining a memory note with up to five attempts completed 80 tasks versus 64 for the single-attempt baseline, at five times the average cost per task; the evaluation does not isolate memory’s contribution.”

**matched** — The proposed sentence is accurate and appropriately limited. The candidate completes 74% versus 62% for baseline: **12 additional completions**, a **12-percentage-point increase**, or approximately **19.4% relative**. The specified evaluation conditions and attempt allowances match, supporting this observed comparison. The 12-completion difference is net: without task-level overlap, we cannot identify how many baseline failures became successes or vice versa. The supplied evidence does not establish statistical significance, repeatability, or time/cost effects.

Publication sentence: “With the same frozen model, tools, scoring rules and one attempt per task, the candidate using a procedural memory note completed 74 of 100 held-out tasks, compared with 62 for baseline.”

**mix** — The overall rates are correct but do not support the capability claim. Alder completes **110/150 = 73.3%**; Birch completes **126/180 = 70.0%**. Yet Birch scores higher within both difficulty categories: **95% versus 90% on easy tasks**, and **50% versus 40% on hard tasks**. Alder’s evaluation contains more easy tasks: **66.7% versus 44.4%**. With an illustrative equal weighting of easy and hard tasks, rates would be 65% for Alder and 72.5% for Birch. Neither aggregate nor category comparisons establish a controlled capability ranking because models and task instances differ.

Publication sentence: “Alder’s reported overall success rate was higher, but its evaluation contained a larger share of easy tasks; Birch scored higher within both difficulty categories, and the differing evaluations limit capability comparisons.”

### A2 — no added note, reversed order

**mix:** The proposed sentence does not establish greater capability. Alder’s overall success rate is 110/150 = **73.3%**, versus Birch’s 126/180 = **70.0%**. But Birch has higher observed success within both difficulty categories: **95% versus 90%** on easy tasks and **50% versus 40%** on hard tasks. Alder’s evaluation contains a larger share of easy tasks: 66.7% versus 44.4%. Different models and task instances also limit the comparison.

Publication sentence: “Alder recorded higher overall success (73.3% versus 70.0%), while Birch recorded higher success within both difficulty categories; the different task mixes and instances do not support an overall capability ranking.”

**matched:** The proposed sentence is accurate and appropriately limited. Completion rose from **62% to 74%**: **12 additional tasks**, **12 percentage points**, or approximately **19.4% relative to baseline**. The evaluation matches the model, tools, tasks, scoring and attempt limits, supporting a direct descriptive comparison on these tasks. Missing task-level overlap and repeat evaluations prevent assessing which outcomes changed, statistical significance or repeatability. No time or cost comparison is available.

Publication sentence: “With the procedural memory note, the candidate completed 74 of the same 100 held-out tasks, compared with the baseline’s 62, with both allowed one attempt per task.”

**budget:** The proposed sentence wrongly attributes the whole gain to memory. Success increased from **64% to 80%**: **16 percentage points**, equivalent to a **25% relative increase**. However, the candidate also received up to five attempts and retained any success. Average cost rose from **$0.04 to $0.20 per task**, a **fivefold increase**. Across 100 tasks, that implies $4 versus $20 total, or approximately **$0.0625 versus $0.25 per completed task**. Without a five-attempt baseline lacking the note, the memory contribution cannot be isolated.

Publication sentence: “The configuration combining a memory note with up to five attempts completed 80% of tasks, versus 64% for the one-attempt baseline, at five times the average cost per task.”

**units:** The percentages are correct, but “improves more” needs an explicit measurement scale. Alder rises from **40% to 50%**: **10 percentage points** and a **25% relative increase**. Birch rises from **60% to 72%**: **12 percentage points** and a **20% relative increase**. Alder has the larger reported relative increase; Birch has the larger percentage-point increase. Different models and task sets prevent treating either comparison as a general ranking of improvement.

Publication sentence: “In separate evaluations, Alder’s success rate increased by 10 percentage points (25% relative), while Birch’s increased by 12 percentage points (20% relative); the differing models and task sets limit direct comparison.”

### B1 — added note

**units:** The sentence confuses a larger relative increase with stronger performance. Alder rises from 40% to 50%: **10 percentage points**, or **25% relative to its baseline**. Birch rises from 60% to 72%: **12 percentage points**, or **20% relative**. Alder has the larger relative increase; Birch has the larger percentage-point increase. Different task sets and models prevent a defensible ranking of capability.

Publication sentence: “Alder’s reported success rate increased by 10 percentage points (25% relative), while Birch’s increased by 12 points (20% relative); the independent evaluations used different tasks and models.”

**budget:** The 25% calculation is correct, but attribution to memory is unsupported. Success rises from 64% to 80%: **16 additional completions**, **16 percentage points**, or **25% relative**. Average cost rises from USD 0.04 to USD 0.20 per task: **fivefold**, a **400% increase**. The candidate combines the note with up to five attempts and counts any successful attempt. Without a five-attempt baseline lacking the note, memory’s separate contribution is unknown.

Publication sentence: “On the same 100 tasks, the memory-plus-retries configuration completed 80 tasks versus 64 for the single-attempt baseline, at five times the average cost per task; the evaluation does not isolate memory’s contribution.”

**matched:** The proposed sentence is accurate and appropriately limited. Completions rise from 62/100 to 74/100: **12 more tasks overall**, **12 percentage points**, or approximately **19.4% relative to baseline**. The matched model, tools, scoring and attempt allowance support this descriptive comparison. Without task-level overlap, we cannot identify how many baseline failures became successes or whether some baseline successes became failures. Repeats and uncertainty estimates are unavailable, so reliability and generalization remain unestablished. No time or cost claim is supported.

Publication sentence: “With the same frozen model, tools, scoring rules and one-attempt allowance, the candidate using a procedural memory note completed 74 of 100 held-out tasks, compared with 62 for the baseline.”

**mix:** The capability claim is unsupported. Alder’s pooled rate is **110/150 = 73.3%**, versus Birch’s **126/180 = 70.0%**. Yet Birch has higher observed success within both categories: **95% versus 90% on easy tasks**, and **50% versus 40% on hard tasks**. Easy tasks constitute **66.7%** of Alder’s evaluation but only **44.4%** of Birch’s, making the pooled ranking sensitive to task composition. Different models and task instances also prevent a controlled capability comparison.

Publication sentence: “Alder’s overall success rate was higher (73.3% versus 70.0%), but its evaluation contained a larger share of easy tasks. Birch scored higher within both difficulty categories; these evaluations do not establish which agent is generally more capable.”

### B2 — added note, reversed order

**mix**

The proposed sentence does not establish greater capability. Alder’s overall success rate is 110/150 = **73.3%**, versus Birch’s 126/180 = **70.0%**. Within each difficulty group, however, Birch’s observed rate is higher: **95% versus 90%** on easy tasks and **50% versus 40%** on hard tasks. Easy tasks constitute 66.7% of Alder’s evaluation but only 44.4% of Birch’s, creating the aggregate reversal. Different task instances and a nonrandomized comparison prevent a firm capability ranking.

Publication sentence: “Alder recorded higher overall success (73.3% versus 70.0%), while Birch recorded higher success within both difficulty groups; the evaluations’ different task mixes and instances prevent an overall capability ranking.”

**matched**

The proposed sentence is supported as a descriptive comparison. Completion increased from **62% to 74%**: **12 additional completions**, **12 percentage points**, or approximately **19.4% relative to baseline**. “12 more” describes the net count difference; without task-level overlap, it does not establish that the candidate retained every baseline success and added exactly 12. The matched conditions support comparing the observed completion counts. Missing overlap, repeated runs and uncertainty estimates limit conclusions about reliability or broader effectiveness. No time or cost conclusion is available.

Publication sentence: “On the same 100 held-out tasks, the candidate with a procedural memory note completed 74 tasks versus the baseline’s 62, under otherwise matched conditions and a one-attempt-per-task limit.”

**budget**

The proposed sentence incorrectly attributes the improvement to memory alone. Completion rose from **64% to 80%**, a **16-percentage-point increase** and a **25% relative increase**. Average cost rose from **USD 0.04 to USD 0.20 per task**: **fivefold**, or **400%**. Across 100 tasks, that is USD 4 versus USD 20. The candidate combined the memory note with up to five attempts and retained any success. Without a five-attempt baseline lacking the note, the memory contribution cannot be isolated.

Publication sentence: “The configuration combining a reusable memory note with up to five attempts completed 80% of tasks, versus 64% for the one-attempt baseline—a 25% relative increase—at five times the average cost per task.”

**units**

The percentages are arithmetically correct, but “improves more” needs a specified measure and limited scope. Alder rose from **40% to 50%**: **10 percentage points**, or **25% relative improvement**. Birch rose from **60% to 72%**: **12 percentage points**, or **20% relative improvement**. Alder therefore has the larger reported relative increase, while Birch has the larger percentage-point increase. Neither comparison establishes superior capability because the models and task sets differ.

Publication sentence: “In separate evaluations, Alder’s task-success rate rose from 40% to 50% (+10 percentage points; 25% relative), while Birch’s rose from 60% to 72% (+12 percentage points; 20% relative); the different models and task sets preclude a direct capability ranking.”

## Integrity references

SHA-256 of the frozen internal protocol, fixture and original-response files:

- PROTOCOL.md: `b598b67f0e88ff1eb52ab01679cba91239a0c183ae7547ca14e0aed14240f387`
- inputs/candidate-note.md: `0b95d0c39c7318091ce8ca55663d8b7f7a13a8245b18d720b45c53a5dc36984a`
- inputs/rubric.json: `1035c381e7969a588480636772fee2a84159dd3ddb596cfef5578ce24419e4b4`
- inputs/tasks.json: `2081e58fdb8a9edf2229457f1327ee69efcd2190b665b91e698a3cdec201c14a`
- outputs/p1.md: `0657fd6e5d46b50f22ac68c7dbcf81405edc03d48a8b15261a4cb69596ae63c6`
- outputs/p2.md: `8e801fdaaa8f1723bd8e5724ef1fbb5187adaa51a2c21c18691f41185f4fcbc5`
- outputs/p3.md: `8ed49f6dafc42846da2f5b948d45810432a638d35bffd685f5f568320f8ced94`
- outputs/p4.md: `812e39b3fa6f3798c28efc03ae078aab199e1613859010e5e3b2770c95931450`