The explainability gap: what can a learner appeal when a model council fails them?

A credentialing system that uses a multi-model AI council to validate a learner’s work imposes a verdict. If the learner disagrees and has no meaningful way to contest a wrong or unfair evaluation, the trust is one-directional. The system trusts itself; the learner simply accepts. That is not trust — it is authority. A credential a learner cannot contest is structurally weaker than one they can, because the unverifiable failure mode — the council was wrong, and nobody can show why — is unbounded.

The authority of the black box

When AI-in-education coverage discusses automated assessment, it typically oscillates between two extremes: treat the model as an infallible oracle, or reject it entirely as a biased black box. Almost nobody maps what a contestable AI verdict actually requires as an audit-trail design problem. But if we are building credentialing infrastructure for the real world, we have to design for the failure case. The Mneurix Lattice LMS runs multi-model AI councils for assessment validation. I run these councils. I know they fail. The question is not how to prevent all failure; the question is what happens when failure occurs.

If a council issues a failing grade and the learner knows their work met the rubric, what can they appeal? If the answer is “nothing,” we haven’t built a trust protocol; we’ve built a bureaucracy with better PR. True trust in a system requires a bidirectional relationship. The learner submits work in good faith, trusting the system to evaluate it fairly. The system must reciprocate by providing a mechanism to verify that fairness. Without an appeal path, the system’s claim of validity is unfalsifiable. An unfalsifiable claim is not a credential; it’s a decree.

What an appeal actually requires

An appeal is not simply “let me argue with the AI.” An appeal is a structured, adversarial process. First, the learner requests the rationale and evidence the council used to reach its verdict. Second, an independent reviewer — either a human or a differently-constructed council — re-examines the work alongside the council’s record. Third, the original verdict is either upheld with explicit reasons or overturned with explicit reasons. Finally, the learner receives a written decision.

For any of this to function, the council must have logged enough during the assessment for the appeal to be genuinely reviewable. If the council only emits a final score and a brief summary, the independent reviewer is staring at a blank wall. An appeal against an inscrutable black-box verdict is theater; the reviewer has no evidence to compare against the learner’s claim. To support a real appeal, the system must produce a reviewable record of how the verdict was reached, not just what the verdict was. The burden of proof in an appeal shifts to the system. The system must demonstrate that it evaluated the work correctly. If it cannot produce that record, the verdict falls.

The chain-of-thought illusion

The standard response to the explainability gap in AI is to ask the model for its chain-of-thought (CoT). If the model can articulate why it made a decision, we assume we have an audit trail. This is dangerously incomplete. A chain-of-thought rationale is a narrative the model generated post-hoc to justify its output. It is not a faithful record of the actual computation that produced the verdict.

Research on CoT faithfulness is highly active, and the evidence is mixed. While CoT can sometimes illuminate a model’s reasoning, studies repeatedly show that rationales often diverge from the actual drivers of the model’s output. Models can confabulate plausible-sounding explanations that do not reflect the weights and activations that actually fired to produce the grade. We must be careful not to overclaim that all rationales are entirely unfaithful — models can and do reason through context windows. But treating a generated narrative as equivalent to a system log is a fundamental misreading of how these architectures work.

“Here is the council’s reasoning” is not the same as “here is a reviewable record of how the verdict was reached.” A CoT rationale is a story. An appeal-grade audit trail needs a transcript.

Building an appeal-grade audit trail

If a narrative is insufficient, what does an appeal-grade audit trail actually require? It must contain the full provenance of the verdict. This includes:

• The inputs presented to the council: the exact learner submission, the rubric, and any contextual prompts fed to the models.
• The per-model verdicts: each council member’s individual call, not just the aggregate score.
• The agreement/disagreement structure: was the verdict unanimous, a split, or highly contested?
• The intermediate verdicts and revisions: if a model revised its score based on another model’s output, that revision must be logged.
• The rubric dimensions scored: the breakdown of how the work performed against specific criteria.
• The evidence pointers: exact citations to what in the learner’s work each dimension was scored against.
• A confidence/conviction signal: how sure was each model in its specific verdict?
• A tamper-evident log: the entire case must be signed, timestamped, and immutable.

This is the assessment-layer equivalent of custody. In a previous post, I argued that assessment validity is the new security perimeter. The assessment perimeter is the evidence’s provenance; the appeal trail is the verdict’s provenance. You cannot separate them. If the evidence is solid but the verdict’s provenance is missing, the credential collapses on appeal.

Reshaping council design for contestability

If you must support appeal, you design the council differently from the ground up. A council designed only to emit a grade produces an un-appealable verdict. A council designed to support appeal emits a reviewable record. The design choice is upstream of the appeal.

This means you log intermediate verdicts, not just the final one. If the council operates in rounds — where models propose scores, view the proposals, and then converge or diverge — every round must be captured. Furthermore, you must record the disagreement explicitly. A 5-4 split and a 9-0 unanimous verdict are very different trust signals. A learner is owed the distinction. A split verdict might indicate genuine ambiguity in the rubric or the submission, warranting an easier appeal path; a unanimous verdict suggests a clear failure to meet criteria.

You must also version the council composition and the models. A verdict rendered by council-config-v3 must be re-examinable against v3. If the system silently re-scores the work using v4 during an appeal, the original verdict’s validity cannot be evaluated. The appeal process must freeze the system in the state it was in when the initial verdict was issued. Finally, you must make conviction-versus-contrarianism visible. If a model dissented, did it dissent because it had a highly confident, rigorously evidenced alternative interpretation, or was it simply being contrarian? Logging conviction signals alongside the verdicts allows the human reviewer to weigh the disagreement accurately.

Governance layering: the human as appeal authority

All of this logging exists to support the governance layer. In our system, human review sits above the council. But the relationship is stacked: the council produces a reviewable record, and the human reviews against it. As I argued in why human review is advisory, not a blocker, human oversight is essential, but its utility is entirely dependent on the inputs it receives.

Human review with a rich, appeal-grade audit trail is a real second look. The two stack: the council produces a reviewable record, and the human appeals against it. Without the trail, the human reviewer faces a final grade and a model’s polished rationale — nothing to cross-examine, nothing to probe, nothing to trace backward. They can agree or disagree, but they can’t audit. They’re left to either defer to the system’s conclusion or override it on instinct, and neither of those is review. The audit trail is what makes human review real rather than theatrical: it gives the reviewer the same provenance a court expects from evidence — chain of custody, timestamps, intermediate states, dissent. Strip that away and you’ve built an elaborate ritual around a rubber stamp.

Now the costs. Full appeal-grade logging is heavier than the alternative. Tamper-evident signed logs require real engineering: hash-chained entries, cryptographic signing, operational discipline to capture intermediate verdicts before models revise them, storage that grows with every assessment, retention policies that someone has to write and enforce. The logs contain learner work, model outputs, rubric breakdowns — material that may be sensitive, that may be subject to deletion requests, that may become a liability if held too long or a gap if purged too early. Privacy and retention aren’t footnotes here; they’re design constraints on the same audit trail that makes contestability possible.

Not every assessment needs this. A low-stakes formative check — did the learner grasp this concept, here’s a nudge — doesn’t require a contestable verdict with full provenance. The cost is real and the stakes don’t justify it. Calibrate the audit-trail depth to the stakes. But for any credential where contestability matters — high-stakes summative assessment, reputation-affecting credentials, gating decisions that block a learner from a program, a role, a certification — the appeal-grade audit trail is a requirement, not a nice-to-have. Say the scope honestly: this is not “log everything always.” It’s “log everything where the learner’s right to contest is real.” For throwaway formative checks, lighter logging is fine and probably preferable. The argument is scoped to the domain where it bites.

Here’s the point that has to land before any of this is dismissed as a future enhancement: contestability is a design property, not a feature you bolt on. If you design the council to emit a grade and only later realize learners need to appeal, you can’t reconstruct the audit trail. The intermediate verdicts weren’t captured. The disagreement structure was discarded when the models reached consensus. The conviction signals evaporated the moment the final answer was written. You cannot retrofit a transcript of a verdict that was never recorded. The data either existed in the moment or it didn’t, and no amount of post-hoc explanation recreates it. This is why the design decision matters upstream: the council must be built to log during the verdict, not to explain after it. The window closes when the verdict is finalized, and it doesn’t reopen.

This is the core of the explainability gap. It isn’t a UX problem — a better interface, a clearer rationale, a friendlier “here’s why you failed” message. Those are presentation layers over a structure that either captured contestable evidence or didn’t. The gap is structural: the council that didn’t log its intermediate states, its disagreements, its revisions, its evidence pointers, its conviction signals — that council can show you a story, but it can’t show you its work. And a learner can’t appeal against a story; they can only appeal against a record.

One-way trust isn’t trust. A credential a learner cannot contest is a decree with better branding — it carries the aesthetics of assessment without the accountability that makes assessment legitimate. The explainability gap is an audit-trail design problem, and the council that can’t show its work can’t be held to it. Build the trail, version the council, log the disagreement, sign the verdict — or accept that the credential is weaker than it claims. The learner who can appeal is the learner who can trust. The learner who can’t is just told.