Lab 01 · Loop recovery: commit the side effect once

Inject termination after an external effect commits but before the result event is durable, then verify operation IDs, event logs, and recovery checkpoints.

By: Rick
Verified: 2026-08-18
Evidence: Experiment
Level: Intermediate
Read: 45 min

Chapter brief

Question to answer

When a process dies between effect commit and result persistence, how does recovery avoid executing again?

By the end, you can

Identify persistence boundaries before execution, after execution, and before verification
Query committed state by operation ID and repair a missing event
Prove repeated recovery does not increase the effect count

Read this now if: Engineers building agent loops, background work, payments, messaging, publishing, or any side-effecting tool
Prerequisites: Understand JSONL and append-only event logs; Know the basic idempotency-key and at-least-once delivery problem
Deliverable: A recovery event log, effect ledger, and zero-duplicate-side-effect acceptance report
Evidence boundary: The lab uses a local ledger to simulate an external system; a real service still needs idempotency, commit lookup, or compensation

Kill the process at the most dangerous persistence boundary

A publish_report call has already sent content to the team knowledge base, but the harness has not appended tool_result when the process dies. If restart looks only at chat history, the agent calls the tool again.

The lab separates two facts:

effect-ledger.json: the simulated external service and whether the effect committed;
events.jsonl: the harness’s replayable timeline.

You will create a temporary disagreement between them and prove that recovery queries the external ledger, repairs the missing event, and never publishes twice.

Understand the four states first

State	Known fact	Recovery action
`planned`	The model chose a tool; the side effect has not happened	Execute once under the same operation ID
The external ledger has a record but the log lacks `effect_committed`	The effect happened and the harness crashed before persistence	Query commit state, skip execution, append the missing event
`result_persisted`	The tool result is durable	Do not rerun the tool; continue verification only
`verified`	Effect count, result, and completion checks agree	Repeated Resume becomes a no-op

The goal is not permanent synchronization. The recovery path must know which source owns truth and how inconsistent state converges.

Run three fault injections

From the project root:

python3 public/lab-assets/loop-recovery.py --reset --crash-at after_effect

The first run prints:

INJECTED_CRASH boundary=after_effect

It exits with code 75. The external ledger contains the effect while the event log contains only planned. Resume:

python3 public/lab-assets/loop-recovery.py
python3 public/lab-assets/loop-recovery.py --report

A passing result includes:

PASS duplicate_effects=0

Repeat with before_effect and after_result to prove the first executes once and the second reruns only the verifier.

Reconstruct what happened from events

Open .agent-mechanics-lab/loop-recovery/events.jsonl. The recovered order is:

planned
 effect_committed   # recovered_from_ledger=true
 result_persisted
 verified

recovered_from_ledger=true is the central evidence. Recovery did not guess from the last model message; it queried the side-effect source of truth.

Then inspect effect-ledger.json. op_publish_report_v1 appears once. Run Resume twice more: neither event count nor effect count should change.

Move the lab into a real tool contract

Replace the local JSON ledger with one of these:

a payment provider’s idempotency key and payment-intent lookup;
a mail provider’s message ID and delivery lookup;
a GitHub or GitLab deployment ID;
a database table with a unique operation_id;
a compensating action and needs_review state when commit lookup is impossible.

Do not add retry only around the tool function. The contract needs an operation ID, commit state, query handle, and result-persistence state.

Acceptance: recovery must prove zero duplicate side effects

Record four metrics:

recovery time at each fault boundary;
effect count for one operation ID;
number of events repaired during recovery;
number and reason of human escalations.

All three boundaries must be explainable, the effect count stays at one, and repeated Resume changes nothing. When commit state cannot be queried, needs_review is correct; silent re-execution is not.

Script and source paths

Appendix: review questions

Open the review and extension tasks

Why can events.jsonl not prove by itself whether the effect committed?
Should the model, harness, or business service generate operation_id?
How do you compensate when the external API provides only at-least-once semantics?
What should recovery do with a truncated final event line?
Extension: add concurrent writes to effect-ledger.json and explain the difference between file locking and a database unique constraint.