Agents Skill Eval.

A production evaluator that scores agent skill packages on spec compliance, security, token cost and effectiveness, with a machine-readable report built for CI.

Category
Software · Web app
Year
2026
Role
Design, frontend, evaluation engine
Stack
Deterministic checks, optional AI review
Link
agents-skill-eval.com
Evaluation runchangelog-formatter01QueuedComplete02AssignedComplete03StartedComplete04DeterministicComplete05AI reviewComplete06FinalizingComplete07CompleteCompleteOverall result96ExcellentSpec compliance100Security96Token budget92Effectiveness95

Overview

One evaluator, two entry points

Agents Skill Eval takes a SKILL.md package, as a local folder or a GitHub URL, runs it through one production evaluator and returns a single structured report. It replaces the question "does this skill work well?" being answered by intuition with a measured, repeatable score.

The challenge

Making quality measurable and cheap to check

Skill packages degrade quietly: drift from the spec, weak trigger phrasing, inflated token footprints, ambiguous instructions and unsafe defaults. The goal was a deterministic core anyone can run repeatably, with an optional AI pass for the judgement calls a rule set cannot make.

The approach

Deterministic first, AI optional

Every run passes through deterministic checks and produces a downloadable JSON report. An optional AI review can be layered on top with a provider of choice, kept strictly opt-in so a default run stays fast, private and reproducible.

Architecture

From skill package to structured report.

  1. 01
    Input
    Local folder or GitHub blob / tree URL
  2. 02
    Deterministic checks
    Spec · security · tokens · effectiveness
  3. 03
    Optional AI review
    Groq · Gemini · Anthropic · OpenAI
  4. 04
    Report
    Scored result and downloadable JSON

Both entry points run through the identical production evaluator, so a local run and a hosted run return the same report format.

What it does.

  • Spec compliance checks

    Every skill is checked against the specification it is meant to follow, and any deviation is flagged immediately. That catches the drift that works on one machine and quietly breaks on the next.

  • Security review

    Overly broad permissions, unguarded deletes and hardcoded paths are flagged before a skill reaches real data. Nothing is sent to a third-party AI provider unless that is explicitly switched on for the run.

  • Token-efficiency analysis

    Every skill consumes context each time it loads, and that cost compounds across a team. The evaluator flags padding and duplication and names the exact section to trim.

  • Effectiveness scoring

    The central check answers the simplest question: will this skill fire and do its job? Ambiguous instructions, weak examples and vague trigger wording are caught here. Vague wording is the single most common reason a skill never triggers.

  • Local folder or GitHub URL input

    Point it at a folder on disk or paste a GitHub link. Both paths run through the same evaluator and return the same report, so results are always comparable.

  • Deterministic by default

    A default run uses no AI model, costs nothing and returns the same result every time. That makes it safe to run on every commit rather than once per release, and nothing leaves the machine unless requested.

  • Optional AI review

    Some judgement calls need more than a rule set. An optional AI pass, with a provider of choice, layers on top of the deterministic result. It runs only when switched on, so a default run stays fast, private and repeatable.

  • Downloadable JSON report

    Every run returns a score out of 100, a plain-language rating and a full breakdown of what did not pass and why. The report is machine-readable and drops into CI without extra glue code.

Want the next one to be yours?

Describe the system, who uses it and what it has to survive. You get a written view on scope, architecture and the first increment worth shipping.

Start a conversation