← LeaderboardThe Skill Leaderboard

adjudicate-regression

Use when deciding whether source and live reproduction evidence establish a new regression or an exclusion.

Published by trailofbits ↗trailofbits/skills

Try your first task

Adjudicate one saved regression candidate

Start a fresh conversation. Replace the highlighted placeholders.

As the fresh adjudication stage of parent regression-hunt campaign [campaign directory], use adjudicate-regression for [assigned candidate and frozen S0/S1]. Read project guidance/standard/labels, verify seals, and inspect source, PoC, setup and logs. Confirm only same-input live before/after reproduction plus independent replay establishes supported reachability and change causality; rule out latent bugs, incomplete fixes and intentional changes. Return a sealed decision.json with evidence, severity/likelihood reasons, novelty-check limits and outcome or UNRESOLVED gaps. Preserve duplicate/change mappings; produce no missing evidence and modify no code.

Required inputs

  • campaign directory
  • assigned candidate and frozen S0/S1
  • sealed reproduction/replay packages
  • registry assignment

User-approved starter prompt based on a pinned source version; check current skill requirements before running. Task evidence ↗

SKILL.md

name: adjudicate-regressiondescription: Use when deciding whether source and live reproduction evidence establish a new regression or an exclusion.

Adjudicate evidence

When to Use

Input: candidate, frozen sources, available reproduction/replay packages and registry assignment. Use a fresh context distinct from the reproducer and replayer. Early exclusions may lack experiments. Follow project guidance before target work.

When NOT to Use

Producing missing evidence; return the gap to the caller.

Decide

  1. Read the standard, labels and file contract. Verify seals. Inspect source, setup, PoC, actual logs and security assumptions, not only summaries.
  2. Identify the claimed defect and assess whether the change introduced it. Rule out incomplete fixes, latent/pre-existing bugs, intentional changes, unsupported expectations and unreachable scenarios with cited evidence. Establish attacker access and applicable protections for security findings.
  3. Confirm only when reproduction and independent replay ran the same PoC/input against live S0/S1 through real interfaces, showing correct behavior before and failure afterward. Explain supported behavior, reachability and patch causality. Assign severity/likelihood with reasons.
  4. Otherwise select an evidenced exclusion or UNRESOLVED; state what remains unproved. For UNRESOLVED, put the standard's concise paragraph separating observed gaps, plausible explanations and resolving checks in reason so reports retain it. Incomplete fixes and existing bugs stay outside regression totals. Check known findings/issues/PRs before claiming novelty; report unknown if unavailable.
  5. For requested deduplication, establish whether supplied candidates share a root cause. Preserve change mappings; never deduplicate unresolved claims into confirmation.
  6. Write and seal decision.json; cite evidence for every conclusion. Record any blocker preventing adjudication.

Report Back

Keep detailed reasoning in the sealed package. Return only:

  • Status: complete or blocked, matching the record.
  • Record: sealed decision.json path, or why none was produced.
  • Result: outcome and one-sentence reason.
  • Gap: missing evidence or required caller action, if any.

A completed decision may be UNRESOLVED; task status and finding outcome differ.

Rationalizations to Reject

  • “S0 passes and S1 fails”: exposing an existing defect is a latent bug.