rate-my-harness · Release review for agent runtimes
A demo is not evidence.
Your agent completed a demo. Now prove the runtime deserves real tools, data, and users — the loop, tool dispatch, context, permissions, and recovery around it, judged on evidence a skeptical engineer can check.
Issue list
BLOCKER · H-002Cancellation does not stop tool calls — the agent keeps spending money after the user presses Stop.
HIGH · H-005Retry re-executes a non-idempotent transfer — one approval can charge twice.
To verify
UNKNOWN · U-003Tenant isolation has not been exercised — one user's data may reach another.
Evidence lanes
Scope
What it actually audits.
Only things that would actually hurt someone using it. It will not tell you your code is ugly.
Zero-tolerance runtime invariants: authority, tool-call correlation, isolation, idempotency, termination, truthful state.
Probabilistic task quality: repeated success, unsafe outcomes, variance, latency, and cost per successful task.
Context ordering, provenance, truncation, persistent state, memory, and session isolation.
Timeouts, retry budgets, cancellation, checkpoints, resume, and deterministic termination.
Permissions, sandboxing, approvals, untrusted input, secrets, network, and tenant boundaries.
Queues, backpressure, tracing, incident recovery — and for public targets, drift, canaries, and rollback.
Measurable task uplift over the simplest credible plain-loop or previous-runtime baseline.
With and without
What changes once it is installed.
Any decent AI will read your code and have opinions. This is the list of things it will not do unless this is installed.
Without it
It blames “the model” and suggests a better prompt.
With it
It separates the model choosing an action, the loop repeating it, and the harness enforcing controls — then finds cancellation received but never propagated.
Without it
It recommends adding retries with backoff.
With it
It distinguishes a model or tool-call ID from an end-to-end idempotency key the side-effect owner actually accepts, and traces effect-commit-then-timeout-then-retry.
Without it
It carries that text into high-authority context and acts on it.
With it
It refuses the payload and maps the trust boundary from tool output through context into the dispatcher.
Without it
Architectural sophistication and green CI read as behavioral value.
With it
It states plainly that repository CI and valid fixtures are structural checks, never proof that the harness behaved correctly.
The part every other review skips
Then who reviews the fix?
Most reviews end at the list. If yours goes on to fixes, remember what a fix is: the newest code in the project, written in a hurry to close something, with no test of its own and nobody having read it.
The pass that writes the fix never grades its own work.
Whoever made the fix does not get to be the one who says it worked. Someone else checks.
A diff is not a fix.
Every finding ships with the exact test that would close it. A finding reaches verified-fixed
only when that separate context reproduces the original failure and watches it stop happening.
The patch itself gets audited.
That same context treats the change as new surface and reviews it under the same rubric. Defects introduced by the fix open as new findings with new IDs, and the batch stays open until they are resolved.
The loop ends on evidence, not on an empty list.
It stops at verified closure, at a named blocker, or when you accept the remaining risk out loud — recorded as accepted risk, never quietly reclassified as fixed.
This site was audited this way. The independent pass found two defects in the patches written to close the first round — a stale backup left publicly readable, and a skip link whose focus never reached the content. Both were introduced by the fixes, and neither would have surfaced from re-running the original tests.
How it runs
Two settings, then evidence.
It asks. It will not quietly pick the harshest setting for you, or the kindest.
Reviewer setting
Agent product owner / Staff agent-runtime engineer / Red-team reviewer / SRE-operator / Oral-defense professor
Review degree
Quick check / Strict review / Release gate / Privileged review / Life-or-death
No score averages away a veto.
Some problems stop a release no matter how good the overall score looks. Saying you can live with one does not turn it into a pass.
Start with the quick check.
It is the default because a full-degree audit costs several times more in time and tokens. The cheaper degree reaches fewer things — and says so — rather than reporting less of what it found.
The five
One evidence contract, five layers.
Each installs on its own. Take the one that matches what you are shipping.
Install
One command, or a plugin marketplace.
One method per client. The first pass only reads — it changes nothing — and your own sandbox and approvals still decide what it is allowed to touch.
npx skills add AmsonntagChow/rate-my-harness --skill rate-my-harness
/plugin marketplace add AmsonntagChow/rate-my-harness
codex plugin marketplace add AmsonntagChow/rate-my-harness && codex plugin add rate-my-harness@amsonntagchow-rate-my-harness