Human review is advisory, not a blocker

Work in progress. Public outline; prose to follow.

The pattern, generalized

A multi-model council flags low-agreement decisions for “human review.” The obvious instinct is to make that flag a blocker — halt issuance until a human clears it. That instinct is usually wrong.

Why advisory beats blocking

1. Throughput collapses. A human-gate on a stochastic signal (council disagreement) makes the pipeline hostage to a queue no one staffs.
2. No accountability. “Human review” of a model disagreement isn’t a real judgment — it’s a rubber stamp or a guess. It relocates responsibility without adding rigor.
3. The disagreement is a signal, not a verdict. Low agreement means the system is uncertain — the right move is more rigorous automated resolution, not a human coin-flip.

The deterministic-resolution pattern

• Re-grade, tighter band (one round) on disagreement.
• Consensus pass/fail honored.
• Still split → fail the attempt + retry-after-cooldown. Deterministic, accountable, no queue.
• Humans do sampled periodic audit of the resolved cases — feeding course/instrument revision. That’s where human judgment actually adds value: subject-matter sufficiency, not per-attempt adjudication.

What human auditors are actually for

• Course-level instrument review: does this assessment measure real competence / a valid rubric? (one-time, course-scope)
• Sampled audit of resolved cases, on a cadence — not a per-attempt gate.
• Adversarial input: probing the system, not rubber-stamping it.

The lesson for credentialing builders

Don’t ship a “requires human review” flag that blocks issuance and call it a guardrail. If a flag is advisory, say so; if it must block, make the block deterministic and accountable. Conflating the two is the most common AI-validation design failure.

Draft. Prose to follow. This is a generalized, public version of a real design decision in Mneurix Lattice.