← LeaderboardThe Skill Leaderboard

validate-bug

Use when a bug PoC package needs an independent rerun and judgment before patching.

Published by trailofbits ↗trailofbits/skills

Try your first task

Independently rerun a bug reproduction

Start a fresh conversation. Replace the highlighted placeholders.

Within parent patching-bugs run [run directory], use validate-bug to independently rerun [finding.json and reproduction package] on [baseline checkout]. Read its proof type, unchanged commands/PoC and raw receipts, plus the record contract/project guidance; do not read the reproducer's transcript. Repeat the compiler command or rebuild/rerun the runtime inputs, retaining stdout/stderr, exit/timing. Judge whether the exact claim violates the invariant. Return confirmed, not_confirmed, or inconclusive in validate.json, manifest/check the package via uv run --no-project, and report gaps. Do not repair the fixture, finding, or code.

Required inputs

  • run directory
  • finding.json and reproduction package
  • baseline checkout

User-approved starter prompt based on a pinned source version; check current skill requirements before running. Task evidence ↗

SKILL.md

name: validate-bugdescription: Use when a bug PoC package needs an independent rerun and judgment before patching.

Validate Bug

When to Use

Use after reproduce-bug has supplied a finding and PoC package. Read the record contract. Before target commands, follow its project guidance.

When NOT to Use

Do not use to repair a PoC, develop a patch, or infer reproduction from the reproducer's narrative.

Procedure

  1. Read finding.json, the reproduction's proof type, commands, any PoC, and raw receipts. Do not read the reproducer's transcript.

  2. Rerun the same baseline compiler command for compile failures, or rebuild and rerun the unchanged runtime PoC and planned scoped inputs. Save stdout, stderr, exit status and timing in validate/receipts/. If an observation is unreliable, return inconclusive instead of inferring either presence or absence.

  3. Decide whether the target diagnostic or runtime assertion demonstrates the exact claim and violates the stated invariant. Write validate.json as confirmed, not_confirmed, or inconclusive with a short reason and agent identity.

    Validate record schema:

{
  "schema_version": 1,
  "outcome": "confirmed",
  "agent": "validator-1",
  "reason": "The unchanged PoC reached the assertion and demonstrated the claim."
}
  1. Manifest the complete validate/ directory with uv run --no-project <script> manifest <validate-step-dir>, then run the record checker. Return the package without editing the PoC or finding.

Rationalizations to Reject

ShortcutRequired action
Fix the PoC to make the claim work.Mark it inconclusive and explain why.
Trust the reproducer's conclusion.Rerun the unchanged package independently.