Rook, an autonomous agent harness

Rook runs a language model as an autonomous operator against a live, stateful environment. The model proposes objectives. A deterministic layer with no model in it decides whether each proposal is executable, carries it out, and verifies the result against ground truth rather than against the model's own account of it.

The environment I used to build and test it is a Game Boy emulator, because it is the cheapest source of a hard problem I know: a long horizon, a large state space, real failure, and perfect ground truth to check against. The supervision architecture is the part that transfers. The ground-truth reader is not: it is 206 hex literals pinned to one ROM checksum, and it would have to be rewritten for any other environment.

Validate objectives before execution, not after

The expensive failure in a long-running agent is not a wrong answer. It is an objective that was never executable, discovered thirty minutes later after it has consumed budget and moved state.

So the model does not act directly. It proposes an objective, and a deterministic validator checks it against ground truth first: does the target exist, is the goal already satisfied, is this a retry of something already abandoned, does a route exist from the current state, does it violate a hard constraint of the environment. A rejected objective never executes, and the rejection is written to the log with a machine-generated reason the model then sees.

Across the recorded runs, 1,003 of 3,500 proposed objectives were rejected this way, 28.66 percent. A separate 234 responses were malformed; those are counted as their own category and are not inside the 1,003.

The split of those rejections is the part I find useful. Roughly 517 were the model proposing something wrong. Roughly 486 were my harness being incomplete: 477 of them were a target the model named that the harness could not resolve, even though the prompt shipped a list of reachable targets. Splitting rejections by fault is what turns a rejection counter into a work queue, and it is the measurement I would want on any agent I was responsible for.

Budgets are enforced, not suggested

Three independent circuit breakers can halt a run: a decision budget, a cost budget, and a no-progress detector that measures movement against ground-truth state rather than against the model's claim of progress. 85 halts were recorded: 52 on no progress, 23 on a loop signal, 10 on budget.

Underneath that, a stall ladder escalates from cheap to expensive: a state fingerprint catches exact repeats, an oscillation detector catches loops over a sliding window, and only then does a semantic check run. The window was tuned from 180 decisions down to 15 after the logs showed how long the naive version let a loop continue before noticing.

Evidence you can audit

An agent that cannot be audited cannot be trusted with anything that matters. Every run writes an append-only event log with the decision, the arguments, the provider, the token counts, the latency, and the resulting state transition. Multiple providers sit behind one interface with per-provider spend controls and cooldowns, and provider and model are recorded per decision, so cost is attributable after the fact rather than estimated.

For any artifact produced from a run I keep a chain of custody: a SHA-256 of the source recording, the exact frame ranges used, a frame-count reconciliation between what was written and what decodes back, and a verification of the outcome read from environment memory independently of the recording. Each artifact also carries an explicit statement of what it does not establish. That constraint is the useful part. It is what stops a system of record from quietly becoming a highlight reel.

What the instrumentation caught

It earned its cost by finding failures that every aggregate metric reported as healthy. On this architecture I found a learner optimizing against a reward that was structurally always zero, a component whose model outputs were logged to a live view and then discarded before execution, and an agent issuing fresh, plausible decisions for thirteen consecutive turns after its episode had already terminated.

None of those were visible in success rates. All three were obvious once decisions were correlated against ground-truth state changes. A verifier of mine also once passed a recording 8 out of 8 while the status rail had overstated a hold by about 54 seconds, because it had no check for a hold outliving its own deadline. I added that check; it now fails the historical recording and passes the fixed one. A verifier that certifies the defect it exists to catch is worse than no verifier.

measured

Scope of these claims. This describes architecture and reports telemetry from my own harness. It makes no claim that any run completed its environment's end goal, and the logs would not support one: portions of the late-stage runs were driven deterministically rather than by live model calls, which is exactly the kind of thing this instrumentation exists to surface. The rejection rate is a measurement of my harness and my own objective format, not a property of language models. The same check applied to ordinary coding-agent tool calls rejects about 0.24 percent, not 28.66. What transfers is the check, not the number. Source is private and stays private; this page is the sanitized write-up. Happy to walk through any derivation in conversation.

Two public repositories built on the same ideas, if you want code you can read rather than a write-up: solhunt, a contract-security research agent that runs each scan in its own sandbox behind an iteration budget and a circuit breaker, and distru-mcp, an MCP server where writes are dry-run by default and a real write needs a confirmation token. Neither is part of Rook.