← LeaderboardThe Skill Leaderboard

write-code-eval

Write code evaluators for known failure modes with objective rules. Use when code can check the rule from a trace, with or without a reference answer. Use `write-judge-prompt` when the rule requires interpretation.

Published by ai-evals-course ↗ai-evals-course/evals-skills

Try your first task

write-code-eval

Start a fresh conversation. Replace the highlighted placeholders.

Use write-code-eval to write and implement one code evaluator for this objective failure mode: [failure mode]. Identify the needed trace fields or reference data, use the project's language and evaluation framework, and test known passes, failures, and borderline cases; run it on available traces and inspect mistakes.

Required inputs

  • [failure mode]

User-approved starter prompt based on a pinned source version; check current skill requirements before running. Task evidence ↗

SKILL.md

name: write-code-evaldescription: >  Write code evaluators for known failure modes with objective rules. Use when  code can check the rule from a trace, with or without a reference answer.  Use `write-judge-prompt` when the rule requires interpretation.

Write a code evaluator

Start with a failure mode found through error analysis. Write one check for that failure mode, much like a unit test that asserts what should hold for each trace.

  1. State the rule and identify the trace fields or reference data the check needs. If the rule requires interpretation, use write-judge-prompt.
  2. Implement the check in the project's language and eval framework. Return a result and a reason in the format that framework expects.
  3. Test known passes and failures, including borderline cases. Run the check on available traces and inspect mistakes. If the rule uses a proxy for human judgment, compare its results with human labels.

Examples

Failure modePossible check
Invalid output structureParse the output and check required fields
Missing or forbidden textMatch a string or pattern
Citation not in retrieved documentsCompare cited IDs with retrieved IDs
Bad tool callCheck arguments against the tool schema or run the call in a safe test environment
Wrong valueCompare the output with a reference value

Choose the check from the failure rule. For a failure with both objective and interpretive parts, check the objective part with code and use a judge for the rest.