eval-audit

Audit an LLM eval pipeline and surface problems: missing error analysis, unvalidated judges, vanity metrics, etc. Use when inheriting an eval system, when unsure whether evals are trustworthy, or as a starting point when no eval infrastructure exists. Do NOT use when the goal is to build a new evaluator from scratch (use error-analysis, write-judge-prompt, or validate-evaluator instead).

Try your first task

Audit an evaluation pipeline

Start a fresh conversation.

Use the eval-audit skill to inspect my LLM evaluation pipeline artifacts and produce a prioritized list of the most important problems, with a concrete next step for each finding. Start with the artifacts available at <path-to-traces>, <path-to-evaluator-configs>, <path-to-judge-prompts>, and <path-to-labeled-data>; treat each angle-bracketed path as a placeholder that I must replace, and skip any artifact type I do not have.

Required inputs

Local paths to available traces, evaluator configurations, judge prompts, and labeled data; replace each <path-to-...> placeholder in the prompt
If local artifacts are unavailable, access to a connected observability MCP server

Suggested template by Skill Leaderboard, grounded in the linked source. Task evidence ↗

Inside SKILL.md

183 lines

Section content has not been verified for this skill yet.

Full source remains withheld under the current source-display policy. Read the author’s source ↗