←Leaderboard/Testing/evals-skills
Last commit on September 24, 2026·Created on June 23, 2026

ai-evals-course/evals-skills

“Build and validate product-specific AI evaluations.”
Combined rank
#281
across all skills
In Testing
#6
category rank
Stars
1.8k
+4.1% in last 7d
Forks
130
+7.4% in last 7d
Watchers
16
0.0% in last 7d
Traction scoreGitHub stars can be faked, so popularity alone can be misleading. Traction Score looks for broader signs of recent attention, adoption, and active maintenance.
TL;DR

From the AI Evals course, with an emphasis on finding real failure modes before writing tests. These are evaluations for your product, not benchmarks of foundation models.

WHO IT'S FOR
AI / ML Engineers
build product-specific AI evaluation pipelines
RAG Developers
evaluate and optimize RAG system performance
Data Scientists
perform error analysis and failure mode discovery
Frontend / Fullstack Engineers
build custom annotation and review interfaces

Use this skill collection

Follow the documented setup, then try a first task.

Install the complete Evals Skills bundle

Installs: Complete evals plugin bundle, including all documented skills in the repository.

  1. Run this command in a terminal to install the repository's complete skill bundle.

    npx skills add https://github.com/ai-evals-course/evals-skills
.codex-plugin/plugin.json ↗ · Checked Nov 14, 2026README.md ↗ · Checked Nov 14, 2026

Give it something to do.

Suggested first task

Start error analysis on an LLM trace dataset

Uses error-discovery

  • Path to a JSONL, CSV, or JSON file containing LLM outputs or traces
  • Availability to review samples and provide annotations interactively
Use the error-discovery skill to help me do error analysis on traces.jsonl. Treat traces.jsonl as a placeholder for my JSONL, CSV, or JSON dataset path. Begin by examining 5–10 records, inventorying their fields, and identifying the primary content and metadata before proposing the review interface and sample-selection approach.
README.md ↗ · Checked Nov 14, 2026skills/error-discovery/SKILL.md ↗ · Checked Nov 14, 2026
Suggested first task

Audit an existing LLM evaluation pipeline

Uses eval-audit

  • Local paths to available traces, evaluator configurations, judge prompts, labeled datasets, notebooks, evaluation scripts, or metrics exports
Use the eval-audit skill to inspect my existing LLM evaluation pipeline. Review the supplied traces, evaluator configurations, judge prompts, labeled data, and metrics artifacts, then give me a prioritized first-pass list of problems with concrete next steps. If I do not have every artifact, work only from the local files I provide and identify the important gaps.
skills/eval-audit/SKILL.md ↗ · Checked Nov 14, 2026README.md ↗ · Checked Nov 14, 2026
Suggested first task

Draft dimensions and tuples for synthetic eval data

Uses generate-synthetic-data

  • Description of the LLM application and what it does
  • Known failure-prone areas or failure hypotheses
  • Existing user feedback or sample traces, if available
  • Domain guidance for judging whether proposed tuples are realistic
Use the generate-synthetic-data skill for my LLM application. Based on my application description, known failure-prone areas, existing feedback, and any sample traces I provide, propose exactly three failure-focused dimensions and draft 20 diverse tuples for my review. Stop after the draft tuples so I can confirm which scenarios are realistic before any natural-language queries are generated.
README.md ↗ · Checked Nov 14, 2026skills/generate-synthetic-data/SKILL.md ↗ · Checked Nov 14, 2026