Audit any eval, metric, experiment, or benchmark for leakage: does external ground-truth enter independently, or are the model, scorer, and designer just confirming a result no outside truth ever produced? Use before trusting any 'how we'll know it worked', including an A/B, a holdout, a score, a validation, and whenever a result feels too clean or self-confirming. Walks an 8-pattern leakage taxonomy and returns only the patterns that fire, each with an independence fix. Read-only.
Use mandela to audit the attached validation design in its default read-only mode. Identify its model, scorer, designer, and dataset; check the eight leakage patterns; and report only patterns that fire, with evidence and an independent-ground-truth fix for each. Do not rewrite the experiment.
Required inputs
Attach a validation or evaluation design describing its model, scorer, designer, dataset, and how success is measured.
Suggested template by Skill Leaderboard, grounded in the linked source. Task evidence ↗
SKILL.md
name: mandeladescription: "Audit any eval, metric, experiment, or benchmark for leakage: does external ground-truth enter independently, or are the model, scorer, and designer just confirming a result no outside truth ever produced? Use before trusting any 'how we'll know it worked', including an A/B, a holdout, a score, a validation, and whenever a result feels too clean or self-confirming. Walks an 8-pattern leakage taxonomy and returns only the patterns that fire, each with an independence fix. Read-only."
Audit whether a validation receives independent outside ground-truth.
Goal
The Mandela Effect names a population confidently remembering something that never independently happened. A leaky validation has the same shape: designer, model, and scorer confirm each other. mandela uses eight leakage patterns to check whether a validation is independent.
Workflow
Identify the validation: eval, metric, experiment, holdout, or "how we'll know". Name its components: model, scorer, designer, and dataset.
Ask the core question: does external ground-truth enter independently?
Test all 8 patterns below. Some apply only to particular components, such as a human subject or a scorer. Report only patterns that fire, each by name.
Recall, not reason: a memorized answer recited instead of one actually derived; the system already knows the result it is supposedly computing.
Wrong null hypothesis: an ablation that removes a surface label but not the underlying signal the system actually exploits, so the "control" still leaks.
Shared hallucination: two components verifying each other; circularity reported as a number.
Tautology: a scorer grading buckets it drew itself.
Verifier = designer: a private, unreproducible recipe in a holdout's clothes.
Shared-pool bias: train and holdout drawn from one labeler pool, so one bias enters both sides.
Frame injection: a question that hands the subject the hypothesis.
Demand characteristics: measured subjects who know they're being measured.
Give the independent-ground-truth fix for each hit.
Rules
Blinding the output value can still leak the collection recipe.
Read-only: name the leak and the independence fix; don't rewrite the experiment.
For a high-stakes validation, you may add one independent auditor with fresh context (N=1). Give it only the validation design, without this session's reasoning, to rerun the eight patterns blind. The default remains same-session and read-only.
Verification
Turn mandela on this audit:
Run patterns #3 through #5 on yourself: are you a scorer grading buckets you drew (Tautology, #4)? Is the verifier the designer (#5)? Is your verdict a shared hallucination with the design's own claims (#3)?
Could a reader who didn't run the audit reach your verdict from the cited evidence alone, using independent ground-truth?