ratemyharness · Release review for agent runtimes
Your agent completed a demo. Now prove the runtime deserves real tools, data, and users — the loop, tool dispatch, context, permissions, and recovery around it, judged on evidence a skeptical engineer can check.
Issue list
BLOCKER · H-002Cancellation does not stop tool calls — the agent keeps spending money after the user presses Stop.
HIGH · H-005Retry re-executes a non-idempotent transfer — one approval can charge twice.
To verify
UNKNOWN · U-003Tenant isolation has not been exercised — one user's data may reach another.
Evidence lanes
Scope
Only what has a product consequence. Style preferences and fashionable architecture are not findings.
Zero-tolerance runtime invariants: authority, tool-call correlation, isolation, idempotency, termination, truthful state.
Probabilistic task quality: repeated success, unsafe outcomes, variance, latency, and cost per successful task.
Context ordering, provenance, truncation, persistent state, memory, and session isolation.
Timeouts, retry budgets, cancellation, checkpoints, resume, and deterministic termination.
Permissions, sandboxing, approvals, untrusted input, secrets, network, and tenant boundaries.
Queues, backpressure, tracing, incident recovery — and for public targets, drift, canaries, and rollback.
Measurable task uplift over the simplest credible plain-loop or previous-runtime baseline.
With and without
A capable agent already reviews code. These are the behaviors that appear only with the Skill loaded.
Without it
It blames “the model” and suggests a better prompt.
With it
It separates the model choosing an action, the loop repeating it, and the harness enforcing controls — then finds cancellation received but never propagated.
Without it
It recommends adding retries with backoff.
With it
It distinguishes a model or tool-call ID from an end-to-end idempotency key the side-effect owner actually accepts, and traces effect-commit-then-timeout-then-retry.
Without it
It carries that text into high-authority context and acts on it.
With it
It refuses the payload and maps the trust boundary from tool output through context into the dispatcher.
Without it
Architectural sophistication and green CI read as behavioral value.
With it
It states plainly that repository CI and valid fixtures are structural checks, never proof that the harness behaved correctly.
Both columns come from the Skill's own test cases — what it requires of itself, not a score
anyone measured. The paired runs are written down but have not been executed, so if you want the number rather than the
promise, run them: evals/execution_cases.json.
How it runs
It never silently defaults to the strictest review — or the friendliest.
Agent product owner / Staff agent-runtime engineer / Red-team reviewer / SRE-operator / Oral-defense professor
Quick check / Strict review / Release gate / Privileged review / Life-or-death
Hard gates block the release regardless of the weighted average. Accepting the risk does not turn an active gate into a technical pass.
Findings advance to verified-fixed only on fresh passing evidence from an independent retest that
also audits the fix code itself.
The five
Each is a standalone plugin. Install only the layer you are shipping.
Install
Pick one method per client and scope. The first audit runs read-only — your sandbox, permissions, and approvals stay the security boundary.
/plugin marketplace add \ AmsonntagChow/ratemyharness /plugin install \ ratemyharness@amsonntagchow-ratemyharness
codex plugin marketplace add \ AmsonntagChow/ratemyharness codex plugin add \ ratemyharness@ratemyharness
npx skills add \ AmsonntagChow/ratemyharness \ --skill ratemyharness