05 · Verifier: what evidence lets an agent stop

Builds on Agent Loop: that chapter sketches the three verifier tiers. This chapter continues with source, failure modes, and selection criteria.

What evidence lets an agent stop

Verdict

A model saying “done” is not evidence. We trace the signals that can let a loop stop: Codex tests and policy, Claude Code budget heuristics, OpenClaw middleware hooks, and Hermes…

Read the complete verdict

A model saying “done” is not evidence. We trace the signals that can let a loop stop: Codex tests and policy, Claude Code budget heuristics, OpenClaw middleware hooks, and Hermes cross-session feedback. Where no shared rubric is published, the conclusion stays conditional.

Scenario: an agent completes a database refactor. Unit tests pass and the diff is non-empty, so it declares success; the production migration was never updated and deployment still fails. Both signals are evidence, but neither covers the business completion condition.

Passing conditions: the completion gate lists hard evidence, soft heuristics, and human judgment separately; every signal states scope and false-decision cost; when a critical oracle is absent, the state is needs_review, not “done” by adding several weak signals together.

Separate proof, budget, and stop-loss

One possible composition: record external completion evidence, resource heuristics, and the model stop signal separately. The host decides which signal blocks continuation or completion.

The four systems cover those tiers very differently:

Tier	Codex	Claude Code	OpenClaw	Hermes
Externally reviewable signals	host may run tests; `apply_patch` validates syntax; `execpolicy` constrains commands; `goals.rs` records status and budget	query.ts has no generic verifier plugin point	`before/after_tool_call` hooks can host checks	main loop has no structured external completion gate
Soft verifier	Goal token_budget + retry backoff + Turn cap	`TOKEN_BUDGET` 90% threshold + three sub-500-token rounds = diminishing returns	`tool-loop-detection.ts` four detectors (generic_repeat / poll_no_progress / ping_pong / global_circuit_breaker)	IterationBudget 90/50 + grace call
Lazy verifier	model emits `output_type: completed`	No `tool_use` block in stream	lifecycle:end event	Model stops emitting tool_call ends the turn
Hard cap	Turn counter + server-side ratelimit + `GOAL_BUDGET_LIMITED_METRIC`	maxTurns (default very large)	runtime timeout; global circuit breaker is optional and disabled in source by default	iteration_budget exhausted → grace call → forced summary

Verifier coverage across the four systems

Compare only implementations that change stop semantics

Codex · Keep completion signals from different layers explicit

Codex’s starting point on verifier design is: the model saying “I’m done” basically can’t be trusted.

Many cases exist where the model didn’t actually complete the task but thinks it did (e.g. changed one file thinking the whole bug is fixed but 3 related files weren’t changed; or ran one test seeing it pass thinking it’s fixed but actually test coverage is incomplete).

The advantage of coding scenarios is having lots of “objectively verifiable” signals (exit code 0 vs non-zero, whether the patch grammar is correct, whether lint passes, whether tests fully pass).

These signals are machine-readable, but the snapshot does not establish that every host strings them into one completion gate. They reduce reliance on self-evaluation; the host workflow still decides what counts as done.

goals.rs feeds turn, tool, budget, and abort events into a shared runtime state machine. This makes state and resource changes recordable. Fields such as turn_completed still come from the surrounding workflow; the state machine does not independently prove business completion:

Codex codex/codex-rs/core/src/goals.rs:98-130 GoalRuntimeEvent collapses turn / tool / abort into one state machine ▸

pub(crate) enum GoalRuntimeEvent<'a> {
    TurnStarted {
        turn_context: &'a TurnContext,
        token_usage: TokenUsage,
    },
    ToolCompleted {
        turn_context: &'a TurnContext,
        tool_name: &'a str,
    },
    ToolCompletedGoal {
        turn_context: &'a TurnContext,
    },
    TurnFinished {
        turn_context: &'a TurnContext,
        turn_completed: bool,
    },
    MaybeContinueIfIdle,
    TaskAborted { /* ... */ },
}

execpolicy is even stricter. Every shell command goes through a tri-state decider, not a boolean:

Codex codex/codex-rs/execpolicy/src/decision.rs:7-28 execpolicy: Allow / Prompt / Forbidden ▸

pub enum Decision {
    /// Command may run without further approval.
    Allow,
    /// Request explicit user approval; rejected outright
    /// when running with `approval_policy="never"`.
    Prompt,
    /// Command is blocked without further consideration.
    Forbidden,
}

Codex exposes four kinds of control, each with a reviewable signal. None should be read as “all pass means business complete”.

1. apply_patch validation (detailed in ch. 06 V4A): model-generated patches must pass V4A diff algorithm grammar check; illegal patches directly refused for execution, loop forced to retry. This verifier prevents “the model generated a pile of garbage pretending to be a patch”. The model sometimes fabricates patches (especially after context compression forgets file original content); without checking before apply destroys the file.

2. run tests exit code: when the host workflow configures relevant tests, a non-zero exit can be fed back as a failure signal. The source snapshot does not prove every task runs tests, and a passing test suite is not the same as business completion.

3. execpolicy::Decision: every shell command runs through Allow, Prompt, or Forbidden. This is command policy, not a result verifier; isolation and blast radius still depend on sandbox, permissions, and deployment configuration.

4. goals.rs convergence detection: when Goal’s token_budget is exhausted, fires GOAL_BUDGET_LIMITED_METRIC metric reporting + steering (injects system message guiding model to “first narrate current progress clearly before exiting”), forcing the loop to not exit hard without explanation. This verifier prevents “tokens burned out, model silent fail”. Without steering the model would directly cut off mid tool call; users seeing partial execution don’t know what happened.

Together these signals improve auditability, but the four source fragments do not guarantee a protocol where any failure forces continuation and every pass means completion. They fit coding workloads; PRD and research tasks need separate review and stop criteria.

Let Codex write a PRD or do research and those coding-specific signals are unavailable; define other reviewable evidence and stop criteria instead.

Claude Code · 60-line algorithm making “minor edits” stop, judging by token convergence rate

Claude Code’s starting point on verifier design is: as an IDE-integrated coding agent, hard verifier (exit code / tests) integration cost is too high (user’s project may have no tests / tests may be slow / IDE shouldn’t presume to run tests); but completely depending on model self-evaluation produces “model rambling endlessly” (especially user asks a simple question but model writes a big analysis).

So Claude Code chose to invest heavily in soft verification, using “token convergence rate” as the main stop signal, cheap (no need to run external commands) and fast (computable each iteration). It fits tasks where token deltas are observable, but thresholds still need calibration for the model and workflow.

Actual implementation in query/tokenBudget.ts is just dozens of lines, making “judging stop by token convergence rate” a standard algorithm:

Claude Code claude-code/src/query/tokenBudget.ts:1-82 checkTokenBudget: 90% threshold + 3 consecutive sub-500-token rounds = diminishing ▸

const COMPLETION_THRESHOLD = 0.9
const DIMINISHING_THRESHOLD = 500

export function checkTokenBudget(
  tracker: BudgetTracker,
  agentId: string | undefined,
  budget: number | null,
  globalTurnTokens: number,
): TokenBudgetDecision {
  if (agentId || budget === null || budget <= 0) {
    return { action: 'stop', completionEvent: null }
  }

  const turnTokens = globalTurnTokens
  const pct = Math.round((turnTokens / budget) * 100)
  const deltaSinceLastCheck = globalTurnTokens - tracker.lastGlobalTurnTokens

  const isDiminishing =
    tracker.continuationCount >= 3 &&
    deltaSinceLastCheck < DIMINISHING_THRESHOLD &&
    tracker.lastDeltaTokens < DIMINISHING_THRESHOLD

  if (!isDiminishing && turnTokens < budget * COMPLETION_THRESHOLD) {
    tracker.continuationCount++
    /* ... nudge model to continue ... */
    return { action: 'continue', /* ... */ }
  }

  /* over budget OR diminishing returns → stop */
  return { action: 'stop', /* ... */ }
}

Three numbers determine loop shape: 0.9 is the soft boundary (every iteration nudges model to continue before 90% token budget), 500 is the token increment threshold (each iteration adding under 500 tokens means “didn’t do much real work”), 3 is the consecutive round count (3 consecutive rounds all under the increment threshold judges diminishing returns).

The three combined judge the “already in minor edits, that’s enough” state. This is a compact soft-verifier design; it is a starting point, not a stop algorithm proven for every scenario.

The algorithm uses token-increment trends rather than only an absolute count. That can be a useful signal when tasks share a workload, but three small increments do not establish a holding pattern for every task.

Relative trends travel better than absolute token counts, but 500 and the consecutive-round count are still thresholds. Re-test them after changing the model or workload.

But Claude Code’s design only goes so far. There’s no hard verifier interface. The 1729-line query.ts opens no external hooks; wanting to force the loop to wait until pnpm test passes can only be done by forking the entire file.

The stopHooks system only allows reverse gating (preventing the model from self-stopping, letting the loop continue), not “first pass this external check” forward requirements (forcing pass before exiting).

This snapshot’s boundary is clear: Claude Code does not expose a built-in forward gate that requires an external test before exit. A workflow that needs “tests pass before PR” must add that gate in the host or maintain a fork; suitability depends on the surrounding integration.

OpenClaw · Makes verifier middleware-extensible + designs 4 loop detectors

OpenClaw’s starting point on verifier design is: as a generic agent control plane (not just serving coding), verifier can’t be hardcoded into the loop.

Different enterprises and business scenarios define “what counts as done” differently (sales agent done = CRM fields all filled / customer service agent done = ticket closed / coding agent done = tests pass). A middleware interface is one practical way to let the host register scenario-specific checks instead of hard-coding one definition.

Actual implementation is tool-policy-pipeline abstracting verifier into registerable middleware; a dozen hook points (before_tool_call / after_tool_call / tool_result_persist etc.) let externals plug lint check / typecheck / human approval / business rule validation.

E.g. wanting to add “PR must have reviewer to merge” verifier to a corporate agent? Write a plugin registered to after_tool_call; no need to modify OpenClaw source.

But this “verifier middleware” only solves the “external can inject checks” problem; OpenClaw also has an internal problem to solve: loop dead-loop detection.

The model sometimes falls into “ceaselessly calling the same tool” or “toggling between two tools” dead loops; relying purely on user-written middleware doesn’t necessarily catch this in time, so OpenClaw built in a specialised subsystem tool-loop-detection.ts with 4 detectors managing different dead-loop patterns:

OpenClaw openclaw/src/agents/tool-loop-detection.ts:9-42 Four loop detectors plus three thresholds ▸

export type LoopDetectorKind =
  | "generic_repeat"          // same call repeated
  | "known_poll_no_progress"  // command_status / process poll without progress
  | "global_circuit_breaker"  // total threshold tripped
  | "ping_pong";              // A→B→A→B oscillation

export const TOOL_CALL_HISTORY_SIZE = 30;
export const WARNING_THRESHOLD = 10;
export const CRITICAL_THRESHOLD = 20;
export const GLOBAL_CIRCUIT_BREAKER_THRESHOLD = 30;

Each of the four detectors targets one typical dead-loop pattern; design trade-offs are as follows:

generic_repeat: same call repeated past the threshold warns. Implementation hashes toolName + stably-serialized params (uses sha256(stableStringify(params)) rather than direct JSON.stringify, avoiding misses from different key order; same parameters {a:1,b:2} and {b:2,a:1} JSON.stringify differently but semantically the same), with same hash appearing 10 times (WARNING_THRESHOLD) in last 30 calls warning, 20 times (CRITICAL_THRESHOLD) circuit breaking. This tier mainly intercepts “model stuck repeatedly calling some tool”.

known_poll_no_progress: specifically identifies command_status and process: poll/log long-polls. These two tools’ semantic is “poll some state”; repeated calls themselves are legitimate (poll once per second is normal), but “call result hasn’t changed” is abnormal (means the polled process is dead or didn’t start). Implementation has hash include “call result text”, with no result change counting as “no progress” (different from generic_repeat which only looks at input). This tier intercepts “model continuously polling a state that won’t change” wasting tokens.

ping_pong: identifies A→B→A→B oscillation. Two tools depending on each other might trap the model in “call A see result not satisfied call B fix a bit then come back to call A see result” cycle. This tier intercepts “two tools cancelling each other’s work”.

global_circuit_breaker: when loop detection is enabled, the 30-call total threshold can act as a pattern-agnostic backstop. It limits calls; it does not know whether the task is still progressing, and it has no effect while the feature remains disabled.

Default enabled: false. OpenClaw doesn’t force every session to enable loop detection because there are legitimate high-repeat workflows (e.g. watch + recompile continuous monitoring naturally needs to repeatedly call the same tool dozens of times, enabling loop detection would friendly-fire).

This is OpenClaw’s switch for users: “open it when needed, close it when not”.

Hermes · No runtime hard verifier in single loops, spreads verifier across time axis for cross-session accumulation

Hermes’ starting point on verifier design is: the verifier of a long-running assistant (companion to users for months / years) shouldn’t be “must judge right or wrong strictly within a single loop”; this strictness makes the agent too rigid to handle scenario diversity, hurting user experience.

The right approach is to spread verifier across the time axis. A single loop is not strict but post-loop does post-hoc analysis writing back to memory; next similar task prefetch injects context; the agent gets smarter and smarter.

In actual implementation Hermes runs up to 90 steps per loop (50 for subagents); on exhaustion goes to grace call → summary → stop. agent/insights.py is not a runtime verifier; it is the post-hoc session analyzer that looks at tokens, cost, and tool patterns:

Hermes hermes-agent/agent/insights.py:1-17 insights.py is a post-run analyzer, not an in-loop verifier ▸

"""
Session Insights Engine for Hermes Agent.

Analyzes historical session data from the SQLite state database to produce
comprehensive usage insights: token consumption, cost estimates, tool usage
patterns, activity trends, model/platform breakdowns, and session metrics.

Inspired by Claude Code's /insights command, adapted for Hermes Agent's
multi-platform architecture with additional cost estimation and platform
breakdown capabilities.
"""

Hermes splits verifier across three time dimensions:

Runtime (within the loop): only IterationBudget(90/50) soft cap + grace call backstop. Parent 90 / subagent 50 steps; exhaustion gives the model one grace call to say a last word; even insufficient strips tools and forces a summary. This tier only ensures the loop won’t run infinitely, doesn’t ensure the task is truly done.

Across sessions (between loops): memory_manager.prefetch_all() injects relevant history before the loop starts. That provides cross-session context, but improved success rate and memory contamination must be measured with task outcomes and memory audits.

Training side (doesn’t participate in normal run): environments/*.py ships evaluate() and score() methods (e.g. yc_bench_env.py:475) for RL training data collection, not for normal agent run. This part is the Hermes team’s experiment in “letting the agent self-improve via RL”; unrelated to users actually running the agent.

Hermes provides a cross-session feedback path but no business-result gate in this loop. It can support open-ended assistants; CI, auto-merge, and critical workflows still need host-provided external checks. Whether feedback improves later success or amplifies bad memory requires evaluation.

Write stop signals before autonomy

The four samples suggest five review questions. They are not a production certification checklist:

First, bound execution paths that can repeat: use a turn count, token budget, wall-clock timeout, or circuit breaker according to call cost and side effects. Fault tests should prove that a reachable stop exists.

Second, a completion gate should read signals outside the model’s self-report: exit codes, lint, type checks, or human approval can provide stronger evidence. Patch validity only proves the patch parses, and a policy decision only constrains an action; neither proves task completion.

Third, soft verifiers use budgets and heuristics to control resources: token deltas, retry counts, and call hashes avoid external commands, but their error rate depends on the workload. They can stop or warn; they do not establish correctness.

Fourth, lazy verification is only a backstop: no more tool_use means the model chose to stop. It does not, by itself, prove completion.

Fifth, record verifier and stop reasons: transition.reason, GOAL_*_METRIC, and LoopDetectorKind expose different runtime signals. Correlate them with task outcomes to distinguish completion, budget truncation, and policy denial.

Put evidence strength beside autonomy

X is coverage of reviewable per-loop signals; Y is the host interface for attaching checks. This is a qualitative reading of pinned source snapshots, not a reliability score.

The four systems represent four typical trade-offs in verifier design:

If the loop enters CI, auto-merge, or a critical workflow: define the external conditions that can block release, such as tests, lint, type checks, deployment preflight, or human approval. Codex’s patch, test, policy, and Goal signals can contribute evidence, but they are not a default four-stage gate and do not replace repository protection rules.

If you want to save tokens and avoid pointless writes: borrow from Claude Code’s TOKEN_BUDGET algorithm (90% threshold + 3 consecutive sub-500-token rounds). Start with the three constants, then recalibrate after changing the model or task mix; they are not a universal stop rule.

If downstream users need custom checks: study OpenClaw’s middleware pipeline and loop detectors. It can host lint, type checks, human approval, or business rules, at the cost of a longer debug path. Multi-tenant suitability still depends on isolation, event contracts, and operations.

If you want to retain feedback across sessions without strict per-loop checks: study Hermes’ memory prefetch and post-hoc insights. Later runs can see prior feedback, but improvement remains an evaluation question. Each run still needs a budget cap and an explicit failure state.

Match stopping evidence to task consequences

Completion evidence	Route to borrow	Cost or boundary
Tests, exit codes, or patch grammar exist	Codex hard verifiers	Strong proof is domain-specific
Progress is measurable but not binary	Claude Code budget and transition signals	A heuristic can stop too early
Policy and tool events must block or explain continuation	OpenClaw middleware hooks	You need a stable event contract
Quality improves over repeated sessions	Hermes trajectory and memory feedback	Cross-session learning is not proof of this run

Start with one auditable domain verifier

When building a verifier, define completion evidence and resource caps first. Add heuristic stopping, observability, and human review after those boundaries are explicit.

Build recipe

Minimum viable

• Configure max_iterations, token_budget, or wall_clock_timeout according to cost and side effects, then fault-test the stop path. A short bounded task may not need all three.
• Attach an external signal that corresponds to the result for high-risk tasks. A test exit code or output schema proves only the layer it covers, not business correctness.
• Record structured stop reasons that distinguish normal completion, resource truncation, policy denial, and errors. The label set should drive actual retry and alert logic.

Once that works

• Borrow Claude Code's `0.9 / 500 / 3` constants for a token-budget soft verifier. Use token-increment trends as the stop signal, then recalibrate on your own model and task set
• Borrow from OpenClaw's four detectors (generic_repeat / poll_no_progress / ping_pong / global_circuit_breaker); start with generic_repeat plus a per-session circuit breaker, then measure misses and false positives on labelled trajectories
• For high-risk shell calls, consider Codex's Allow / Prompt / Forbidden tri-state alongside sandbox, mount, credential, and approval controls
• Verifier failure must be recoverable: write transition label + persist for replay, never panic; hard verifier rejection (e.g. tests fail) should let the loop continue (feed error back to model), not kill the entire agent; write verifier failures to event stream for downstream analysis

Don't do this

• Treating "the model stopped" as business completion. The host may combine that signal with external evidence, resource state, or human review; a fixed all-three cascade is not required.
• Stacking verifiers as boolean AND boolean AND boolean: produces "all four pass, task not done" cases (each verifier passes but combined they don't guarantee task is truly done); use "voting mechanisms" or "tier classification" more sensibly
• Hardcoding verifier logic in the main loop: extract into hooks / middleware for downstream replacement; different scenarios need different verifier combinations (CI run vs local dev); hardcoding means changing loop body to switch
• Reaching for RL scoring on day one: define a hard cap and one reviewable completion signal first, then decide whether a soft verifier lowers cost; RL scoring also needs training data and bias evaluation, so task risk should justify it

How failure returns to the loop

Design sketch: an external-check failure may enter repair or human review; Claude Code's snapshot heuristic (0.9 / 500 / 3) may trigger a resource stop. Neither path is Codex's default completion protocol.

Splitting verifiers into three tiers buys monitoring the ability to tell “loop is making progress” from “loop is spinning” from “loop should be done.” Collapse them into one boolean and the agent system stops being legible.

Source paths for stopping signals

Source map & further reading

• Codex codex/codex-rs/core/src/goals.rs:98-130 GoalRuntimeEvent state machine + token_budget convergence
• Codex codex/codex-rs/execpolicy/src/decision.rs Allow / Prompt / Forbidden tri-state
• Codex codex/codex-rs/execpolicy/src/policy.rs execpolicy rule engine entry
• Claude Code claude-code/src/query/tokenBudget.ts:1-82 checkTokenBudget: 0.9 threshold + 500 token + 3-round diminishing
• OpenClaw openclaw/src/agents/tool-loop-detection.ts:9-42 Four LoopDetectorKinds + three thresholds
• OpenClaw openclaw/src/agents/tool-loop-detection.ts:100-200 hashToolCall / hashToolOutcome stable serialization
• Hermes hermes-agent/agent/insights.py:1-100 InsightsEngine post-run session analyzer (not a runtime verifier)
• Hermes hermes-agent/environments/yc_bench/yc_bench_env.py:475 evaluate() / score() lives in RL training, not normal loop

What to carry forward and the next experiment

A verifier is not “more checks.” It begins by pricing false release versus false blocking. Hard evidence states its coverage, soft scores never impersonate a deterministic oracle, and human gates have explicit inputs and timeouts.

Next experiment: build 20 tasks with known outcomes, including tests-passing-but-incomplete, correct-but-over-budget, missing-source, and human-judgment cases. Measure false completion, unnecessary continuation, escalation rate, and incremental cost before choosing the verifier mix instead of tuning thresholds by intuition.

Appendix: exercises and review

Open the exercises and ten review questions

Exercises

1. 🟢 Audit: Count the verifier tiers in your current agent. Sort them into hard / soft / lazy. Which tier is empty? What does it cost to fill it?
2. 🟠 Port one: Move Claude Code’s checkTokenBudget into your own agent. Re-tune 0.9 / 500 / 3 for your workload. Write down why each value moved.
3. 🟠 Port one: Build OpenClaw’s generic_repeat detector. Hash toolName + JSON.stringify(sorted params). In the last 30 calls, the same hash hitting ≥ 10 warns, ≥ 20 breaks the circuit.
4. 🔴 Design: Design a hard verifier for a non-coding agent (weekly report, research). The task has no tests and no exit code. How do you fabricate a machine-checkable “done” signal?

Review questions

Q1 · Concept: Define “hard verifier”, “soft verifier”, and “lazy verifier”.

Sort by where the judgement comes from:

Hard verifier reads a host-verifiable signal such as tests, lint, type checks, or human approval. Patch validity, schema validity, and HTTP 200 only verify their own layer; none automatically means “task correct.” Codex’s patch, policy, test, and Goal signals also serve different responsibilities.

Soft verifier reads internal budgets and heuristics: token delta, retry count, tool-call pattern hash, cost threshold, call frequency. Verdicts are usually “diminishing returns” or “circuit breaker”, preventive shutdowns. Claude Code’s tokenBudget.ts is the engineering reference for soft verifier.

Lazy verifier trusts the model: no tool_use in the assistant message, output_type: completed, stop_reason: end_turn. Hands the decision back to the model.

The tiers are not mutually exclusive. A workflow may combine an external completion check, a resource heuristic, and the model’s stop signal, but the order and membership should follow the task rather than a fixed three-stage cascade.

Why not rely on lazy verifier alone? The model’s “confidence” and “task completion” are decoupled. A model can say “all done” while lint has not passed, tests have not run, or a file was missed; this chapter has no production sample from which to claim a failure percentage.

Source: Codex goals.rs records goal state and budget, while execpolicy constrains commands and apply_patch validates edit format; they are not one default hard-verifier chain. claude-code/src/query/tokenBudget.ts shows a budget heuristic. Follow-up: “Are verifier and sandbox the same?” No. Sandbox (chapter 13) prevents the agent from doing harm; verifier judges whether the agent finished. Orthogonal axes.

Q2 · Architecture: Why is Codex’s GoalRuntimeEvent state machine 1500+ lines?

Four concerns merged into one state machine:

1. Token budget convergence. Each turn computes current_tokens / budget; threshold hit emits GOAL_BUDGET_LIMITED_METRIC.
2. Goal completion. A task may decompose into sub-goals; each sub-goal’s state is tracked.
3. External goal mutations. Users may change goals between turns (“also add tests besides that bug”); state machine supports runtime goal updates.
4. Tool completion × goal association. A tool completing doesn’t always advance a goal; some tools (apply_patch, run_tests) push state forward and need explicit modelling.

State machine benefits: all judgement in one place. Claude Code scatters verifier logic across query.ts’s 1729-line single file; debugging means jumping across many branches.

Codex’s single file with clear states (TurnStarted / ToolCompleted / TurnFinished …) lets you draw the state diagram and verify.

The state and persistence path is long. Reading cost depends on Rust familiarity, call paths, and test coverage; line count cannot be converted into one engineer-day. A new field or event may also require callers and storage-schema changes, not only an enum variant.

Refactor into a state machine when transitions repeat across branches, need persistence, or become difficult to test. Signal count alone is not a reliable threshold.

Source: codex/codex-rs/core/src/goals.rs, focus on GoalRuntimeEvent and GoalRuntime::handle_event. Follow-up: “Doesn’t a big Rust state machine have perf issues?” No. Rust enum match is O(1) dispatch; state machine size isn’t the bottleneck. The bottleneck is the LLM call itself.

Q3 · Engineering: Claude Code’s 0.9 / 500 / 3 constants define “diminishing returns”. Why exactly these three numbers?

The source establishes three facts: 0.9 marks the late-budget region, 500 defines a low delta, and 3 requires repeated low deltas. Only their combination trips diminishing returns.

The source does not include a threshold evaluation or name the Claude variants used for tuning. It therefore cannot support claims about what 300 or 800 would do.

When porting, treat 0.9 / 500 / 3 as defaults to validate. Record stop/continue decisions, task outcomes, and human review, then change one variable at a time so the effect remains interpretable.

Source: claude-code/src/query/tokenBudget.ts:1-82. Follow-up: “Why not a dynamic algorithm (exponential weighted average)?” Fixed constants are easier to inspect, but the source only exposes 0.9 / 500 / 3; it does not include an evaluation sample. Re-record stop/continue outcomes after changing the model or task mix before recalibrating.

Q4 · Engineering: OpenClaw has 4 detectors in tool-loop-detection. Is one enough?

No. Four detectors correspond to four loop modes, complementary coverage:

generic_repeat: same tool name + same args repeated. Most common, but catches only exact repeats. If args drift slightly (case change in file path), generic_repeat misses.

known_poll_no_progress: identifies command_status / process: poll long-poll calls. These tools are designed for repeated checking; generic_repeat false-kills them. So a dedicated detector hashes call + result; only unchanged result counts as “no progress”.

ping_pong: A → B → A → B oscillation. Model Reads a file, finds it wrong, Edits, then Reads to check, then Edits again… tools mutually cancelling. generic_repeat misses because tool names alternate.

global_circuit_breaker: when loop detection is enabled, 30 tool calls trigger the mode-agnostic threshold. It caps calls; with the feature disabled it does nothing, and with it enabled it can still stop legitimate long workflows.

Single-detector pitfalls:

• generic_repeat only: misses ping_pong and long-poll false-positives.
• global_circuit_breaker only: too late, 30 tool calls = user has waited long.
• ping_pong only: catches nothing for single-tool loops.

Practical: start with generic_repeat + global_circuit_breaker (2/4). Add known_poll_no_progress when watch/poll tools are common. Ping_pong is last (highest false-positive rate, hardest to tune).

Source: openclaw/src/agents/tool-loop-detection.ts:9-42 (detector kind enum); full file 600+ lines covers implementation. Follow-up: “Can the model self-detect loops?” Possible via self-reflection (chapter 19), but recognition is much worse than a detector. Models are biased about their own behaviour.

Q5 · Concept: What is “transition reason”? Why does Claude Code label every exit?

transition reason is the label attached to every loop exit, telling the caller “why did we stop?” Claude Code’s label set is roughly a dozen:

• end_turn: model stopped naturally, no tool_use.
• max_tokens: hit model output token cap.
• token_budget_exhausted: 0.9 threshold + diminishing fired.
• max_turns: hit maxTurns hard cap.
• stop_hook_block: stopHooks system blocked the stop request.
• user_interrupt: user pressed Ctrl+C.
• error: unrecoverable error.
• permission_denied: canUseTool denied with no fallback.

Why label them? Three reasons:

1. Monitoring legibility. Looking at a dashboard, distinguishing “completed normally” from “cut by budget” is one glance; you don’t read the full trajectory.
2. Aggregate analysis. Compare weekly token_budget_exhausted rate with task outcomes, establish a baseline, and set an alert from that relationship. One percentage cannot tell you whether the budget or prompt is at fault.
3. Retry strategy. max_tokens can auto-retry (raise budget); error cannot; permission_denied prompts the user instead. Labels let automation branch.

Counter-example: a loop that returns only success: bool; monitoring can’t separate “actually done” from “budget killed it, task isn’t done.” Their handling diverges completely.

Start with labels that drive handling, such as completed, truncated, and error. The field name is not an industry standard; callers need to know whether a run is retryable or requires human action.

Source: claude-code/src/query.ts, grep transition. Follow-up: “Do OpenClaw / Codex / Hermes have transition reasons?” Yes but different shapes. Codex uses metrics like GOAL_BUDGET_LIMITED_METRIC; OpenClaw uses LoopDetectorKind; Hermes writes the reason into the grace call.

Q6 · Practical: Design a hard verifier for a non-coding agent (“weekly report”). No tests, no exit code; how?

Non-coding has no native “external judge”; you fabricate one. Four common approaches:

Approach 1: Schema validation. Require structured output (JSON schema / TypeScript interface) with fields for done, planned work, blockers, metrics, and links. A schema checks fields, types, and declared constraints; it cannot tell whether the facts are correct or the report is useful.

interface WeeklyReport {
  completed: string[];        // ≥ 3
  planned: string[];          // ≥ 3
  blockers: string[];         // may be empty
  metrics: { name: string; value: number }[];  // ≥ 1
  links: string[];            // ≥ 2 external
}

A host can wire schema failure to rejection and continuation. That control flow belongs to the workflow, not to the schema itself.

Approach 2: LLM-as-judge. A separate small model reads the output and scores against a rubric. “Score < 7 = loop continues.” Use an independent model (not the loop’s model), avoid self-evaluation. Hermes’s evaluate() works this way.

Approach 3: Human checkpoint. For compliance reports or contract review, the host can add an approval state and block publication or completion until review. Codex’s approval_mode: on-request approves tool calls; it is not a content-completion checkpoint, which the host must implement separately.

Approach 4: Reference sample comparison. Keep a human-reviewed sample set and use embedding similarity as one signal. Calibrate both sample count and cosine threshold for the task; similarity is not correctness and should not decide stop/continue alone.

Practical picks:

• Weekly report: approach 1 (simplest schema).
• Research: approach 2 (LLM judge with rubric: coverage, citations, depth).
• Compliance report: connect a human checkpoint that matches the organisation’s review process.
• Customer reply: approach 4 (embedding comparison).

Hybrids are common: approach 1 + 2; schema gates structure, LLM judge gates quality.

Source: hermes-agent/environments/benchmarks/yc_bench/yc_bench_env.py:475 (evaluate()); any JSON schema library covers validation. Follow-up: “Doesn’t LLM judge have bias?” Yes. Build a human-labelled set, compare agreement by task type, and define the acceptable threshold before running the evaluation. Re-prompt or change models when it falls below that threshold.

Q7 · Architecture: Hermes has no per-loop hard gate. What does cross-session feedback add, and what does it leave unresolved?

Hermes stores feedback in cross-session memory. That is a mechanism, not evidence of long-term convergence. The source shows three pieces:

Event replay. Every session’s trajectory writes to SQLite (agent/insights.py analyzes). memory_manager.prefetch_all() injects “how previous similar tasks went wrong / went right” at loop start. So per-loop hard verifier is weak, but the model has context for known failure modes.

Skill self-eval. The skill system (chapter 17) internalizes “success criteria” into skill docs. Users write weekly-report.md defining “completion conditions”; the model self-checks against the skill. This pushes verifier responsibility from harness to skill author.

Grace call backstop. When iteration budget exhausts, the model is forced to make a final “summarize current state” call. Output persists as memory for the next session. So even if the loop force-ends, the next session sees “where we stuck last time.”

What might it solve? Later sessions can read prior trajectories, skill constraints, and the final summary instead of starting without context. Measure repeated-task success, human correction effort, and memory contamination to establish benefit.

What does it not solve? One-shot CI and auto-merge still need an oracle for the current run. Codex is easier to pair with coding signals; that is not a measured win across every CI workload.

Practical:

• Short trusted tasks: borrow from Codex.
• Long accumulating tasks: borrow from Hermes memory + skill.
• Generic: both; hard verifier as backstop, memory as accelerator.

Source: hermes-agent/agent/memory_manager.py (memory impl), hermes-agent/agent/insights.py:1-100 (post-run analysis). Follow-up: “How does Hermes prevent memory pollution?” Memory has TTL + relevance score, old memory fades. Chapter 16 covers this.

Q8 · Engineering: I have max_iterations and wall_clock_timeout. Do I still need verifiers?

Yes, because they solve different problems.

max_iterations / wall_clock_timeout is the hard cap, bounding loop runtime and spend. Role: “fuse”; it limits one class of runaway behaviour but does not establish correctness or eliminate every operational risk.

Verifier is the decision layer, telling the system this is the right time to stop (before the hard cap). Role: “steering wheel”; tells the loop when to exit gracefully.

Hard cap alone problems:

• Premature stop: the loop hits max_iterations while still progressing, leaving the task incomplete.
• Late stop: the loop repeats without progress long before its hard cap, wasting turns and tokens.
• Indistinguishable reason: logs only show iteration_exceeded; can’t separate “task too big” from “loop stuck”.

With verifier:

• Soft verifier detects diminishing returns, loop stops at turn 5 (not waiting for turn 30).
• Hard verifier confirms task complete, loop stops at turn 8 (not running full max_iterations).
• transition.reason distinguishes task_complete / diminishing / loop_stuck / iteration_max / timeout.

How often the hard cap or verifier fires depends on the workload, budget, and model. Log transition.reason to learn which mechanism does most of the work in your system and which one is only a backstop.

Practical: get hard cap right first (max_turns / token_budget / wall_clock) and emit transition.reason. Verifier can start trivially with “diminishing returns” and monitoring becomes readable.

Source: Claude Code’s tokenBudget.ts is the canonical combination; Codex’s goals.rs packs both into the state machine. Follow-up: “Wall clock timeout, what value?” Do not borrow another team’s task distribution. Record your own wall-clock data, then choose a timeout from p95/p99 and the waiting time you can accept; move genuinely long tasks to the background (chapter 18 cron / background).

Q9 · Concept: What is verifier middleware? How does it differ from tool middleware (chapter 04)?

Verifier middleware turns “decide whether the loop should stop” into a pluggable chain.

OpenClaw’s tool-policy-pipeline strictly doubles as tool middleware and verifier middleware; its hooks can both modify tool calls and inject verifier logic.

Differences:

Tool middleware (chapter 04 §Q5): intercepts tool calls themselves. before_tool_call rewrites args; after_tool_call rewrites result. Concern: “is this call legal, can it be optimized?”

Verifier middleware: runs at turn boundaries (not tool boundaries). Each end-of-turn, every registered verifier runs once and asks “stop now?” Concern: “should the overall loop continue?”

Engineering-wise they often merge because:

1. Shared history. Tool call history and loop state live in the same struct.
2. Similar lifecycle. Both “register → trigger during loop → unregister”.
3. OpenClaw just makes it one pipeline: after_tool_call can mutate tool result and trigger “5 consecutive same calls detected → stop loop” verifier logic.

But Codex separates them deliberately:

• Tool calls go through execpolicy static rules.
• Verifier goes through GoalRuntimeEvent state machine.
• Different judgement data (execpolicy reads command + args; GoalRuntimeEvent reads token / iter / goal).

Practical:

• Start with one (OpenClaw style), simple.
• When tool-call judgement clearly doesn’t overlap with loop-exit judgement, split (Codex style). Test: is there any verifier that ignores tool calls entirely (e.g. “total tokens > threshold”)? If yes, split.

Source: openclaw/src/agents/tool-policy-pipeline.ts (merged) vs codex/codex-rs/core/src/goals.rs + codex/codex-rs/execpolicy/src/policy.rs (split). Follow-up: “Are stop_hooks verifier middleware?” Yes, but a single-point hook (Claude Code style). Only reverse-deny stop, can’t proactively trigger stop.

Q10 · Open-ended: Designing a cross-scenario (coding + non-coding) verifier framework, how would you compose it?

Goal: support both “external judge” (coding) and “internal judge” (non-coding), with config exposed to users.

Three layers:

Layer 1 · Resource caps (configured by runtime risk):

{ max_iterations: 30, token_budget: 100_000, wall_clock_seconds: 600 }

The numbers are interface examples, not recommended defaults. A short task may need only the relevant cap; long tasks should tune limits against per-call cost, concurrency, and side effects.

Layer 2 · Soft verifier (optional experiment configuration):

{
  token_budget_check: { threshold: 0.9, diminishing_min: 500, diminishing_rounds: 3 },
  loop_detection: ['generic_repeat', 'global_circuit_breaker'],
  // advanced: 'ping_pong', 'poll_no_progress'
}

These values reuse Claude Code snapshot constants and OpenClaw detector names. Treat them as seeds to calibrate, not defaults proven across workloads.

Layer 3 · Hard verifier (user-registered):

// Coding agent
{ verifiers: [tests_pass, lint_pass, typecheck_pass] }

// Weekly report agent
{ verifiers: [schema_validate(WeeklyReportSchema), llm_judge(rubric)] }

// Customer support agent
{ verifiers: [human_approve, response_length_min(100)] }

Each verifier can return (state) => { passed: bool; reason: string; can_retry: bool }. The host decides whether checks use all-pass, voting, or human override based on false-decision cost. The code below is a site proposal, not a protocol shared by the four systems.

Transition reason labels:

type Reason =
  | 'task_complete'              // all hard verifiers pass
  | 'hard_cap'                   // hit iter/token/clock
  | 'diminishing'                // soft verifier judged
  | 'loop_detected'              // detector fired
  | 'verifier_failed_unrecoverable'  // hard verifier permanently failed
  | 'user_interrupt'
  | 'error';

Every loop returns a reason. Monitoring aggregates on reason.

API design:

const loop = createAgentLoop({
  hardCap: { max_iterations: 30, token_budget: 100_000 },
  softVerifiers: { tokenBudget: defaultConfig, loopDetect: ['generic_repeat'] },
  hardVerifiers: [testsPass(), schemaValidate(MySchema)],
});

const result = await loop.run(initialMessage);
console.log(result.transition.reason);

Why not just borrow from Codex? Codex’s hard verifiers are hard-coded into GoalRuntimeEvent; unusable for non-coding. My design abstracts hard verifier as a user-supplied function: coding plugs in tests, non-coding plugs in schema or judge.

Why not just borrow from OpenClaw? OpenClaw leaves check composition to the host. This sketch includes a token budget and two detectors as an example preset; real defaults require workload-specific false-positive, miss, and cost data.

Estimate effort from the verifier count, host integration points, replay storage, and evaluation set. A fixed schedule has little meaning until those boundaries are known.

Source: composite reference: codex/codex-rs/core/src/goals.rs, claude-code/src/query/tokenBudget.ts, openclaw/src/agents/tool-loop-detection.ts. Follow-up: “Can this framework be open-sourced?” Yes. Verifier abstraction is model-provider-independent, orthogonal to protocol-focused frameworks like LangChain. Could ship as @agent/verifier-kit on npm.