Traction scoreGitHub stars can be faked, so popularity alone can be misleading. Traction Score looks for broader signs of recent attention, adoption, and active maintenance.
This collection provides a structured approach to measuring AI performance, focusing on the transition from raw traces to actionable insights. It guides developers through the creation of synthetic datasets, the drafting of judge prompts, and the implementation of custom annotation interfaces for human review.
By emphasizing error analysis and evaluator validation, the repository helps teams move beyond vague metrics toward a rigorous audit of how their models behave in production and RAG environments.
Compatible AgentsThe repository documents support for these agents. The skills may also work with other agents that can load SKILL.md files, but they may need some setup or small changes.
Local paths to available traces, evaluator configurations, judge prompts, and labeled data; replace each <path-to-...> placeholder in the prompt
If local artifacts are unavailable, access to a connected observability MCP server
Use the eval-audit skill to inspect my LLM evaluation pipeline artifacts and produce a prioritized list of the most important problems, with a concrete next step for each finding. Start with the artifacts available at <path-to-traces>, <path-to-evaluator-configs>, <path-to-judge-prompts>, and <path-to-labeled-data>; treat each angle-bracketed path as a placeholder that I must replace, and skip any artifact type I do not have.
Known failure-prone areas, user feedback, or representative traces
A person with domain knowledge who can review whether the tuples are realistic
Use the generate-synthetic-data skill for <application-description>. Based on <known-failure-prone-areas> and any supplied feedback or traces, propose exactly three failure-focused dimensions and draft 20 diverse tuples for my review. Do not generate final natural-language queries or run the pipeline yet. Replace the angle-bracketed placeholders with my application and known-risk details.
Representative full RAG traces showing retrieved content and generated answers
Local paths replacing <path-to-error-analysis> and <path-to-rag-traces>
Use the evaluate-rag skill to review my completed RAG error analysis at <path-to-error-analysis> and representative full traces at <path-to-rag-traces>. Determine whether the observed failures primarily involve retrieval, generation, or both, then outline a small first evaluation plan that measures first-pass retrieval with Recall@k and evaluates generation faithfulness and relevance separately. Replace both angle-bracketed values with local paths.
Claude