TAU-HOME.COM
LOADING

NVIDIA SkillEvaluator: Multi-Tier Quality and Security Evaluation Framework for AI Agent Skills

SkillEvaluator, an open-source framework maintained by NVIDIA, delivers a three-tier validation pipeline covering static schema and security checks, embedding-b

tau · November 15, 2026

#AI-Agents #NVIDIA #Evaluation #Developer-Tools

NVIDIA SkillEvaluator: Multi-Tier Quality and Security Evaluation Framework for AI Agent Skills

NVIDIA actively maintains SkillEvaluator, an open-source evaluation framework engineered to inspect, benchmark, and validate AI agent capabilities built on the Agent Skills specification. First launched in June 2026 and updated to release v0.9.0 on November 13, 2026, the tool provides structured quality gates spanning static linting to live sandboxed execution.

NVIDIA SkillEvaluator three-tier evaluation pipeline architecture diagram

Image source: Coolcaesar 쨌 CC BY-SA 4.0 쨌 https://creativecommons.org/licenses/by-sa/4.0 쨌 https://commons.wikimedia.org/wiki/File:NVIDIA_Headquarters.jpg

As developer workflows shift toward agentic coding systems, repositories are increasingly populated with "skills"—structured folders of prompt instructions, tool configurations, and supporting code that guide autonomous agents through specialized tasks. However, unvetted skills risk introducing prompt injections, conflicting guidelines, context budget bloat, and unexpected regressions in task execution. SkillEvaluator addresses this gap by establishing an automated, multi-tier inspection pipeline before skills reach production environments.

The Agent Skills Specification and the Three-Tier Architecture

Under the Agent Skills specification, a skill is a self-contained directory containing instructional documentation (SKILL.md) and companion scripts. Modern harness environments—including Claude Code, OpenAI Codex, and GitHub Copilot CLI—leverage these definitions to extend agent domain knowledge.

SkillEvaluator divides the verification lifecycle into three modular, decoupled tiers:

  1. Tier 1 (Validation): Executes purely static checks without requiring external LLM API calls or active network credentials. It inspects YAML frontmatter integrity, directory layout, character thresholds, and macro placeholder formatting across diverse harness specifications. Tier 1 also integrates NVIDIA's SkillSpector scanner and Gitleaks to catch prompt vulnerabilities, unsafe patterns, and secret leaks.
  2. Tier 2 (Deduplication): Mitigates catalog bloat and guideline conflicts. Utilizing text embeddings, it compares the semantic footprint of a candidate skill against an existing catalog, alerting developers to overlapping instructions or redundant tool definitions that could confuse an agent's reasoning path.
  3. Tier 3 (Live Evaluation): Leverages Harbor, an open-source agent evaluation framework, inside isolated Docker containers. By executing benchmark tasks side-by-side—with and without the designated skill—SkillEvaluator measures the real impact on agent task completion rates, step latency, and token efficiency.

Keyless Static Analysis and CI Automation

A notable practical advantage of SkillEvaluator is its low entry friction. Teams do not need GPU instances or costly model endpoints simply to verify syntax and formatting hygiene.

Using the uv packaging tool, developers can clone the repository and run static quality scoring against the bundled reference skill, text-analyzer:

git clone --depth 1 https://github.com/NVIDIA/SkillEvaluator.git
cd SkillEvaluator
uv run --python 3.13 skillevaluator quality-check \
  src/skillevaluator/tier3/reference_skills/text-analyzer \
  -r cli -r json -o reports/first-quality

This command generates a structured report summarizing compliance findings. The built-in passing threshold defaults to 70 out of 100 points; the sample skill scores 86.2 (Grade B) and exits cleanly.

For pull request automation, the CLI includes an onboarding helper:

skillevaluator ci init

This sets up a reusable GitHub Actions workflow that executes Tier 1 verification across altered skills and writes structured evaluation metrics directly into PR review comments.

Direct Agent Integration and Operational Caveats

Developers testing local skills within interactive CLI agents can install SkillEvaluator directly as an active skill:

npx skills add NVIDIA/SkillEvaluator

Once installed, querying an agent such as Claude Code or Codex with "Evaluate my skill's quality" triggers automated harness-compatibility checks and structural feedback from within the agent conversation.

Prospective users should keep several operational realities in mind:

  • Experimental Support Level: Released under the Apache 2.0 license, SkillEvaluator is a community-supported open-source project and does not carry commercial NVIDIA enterprise SLAs.
  • Python Runtime Compatibility: The tool officially targets Python 3.12, 3.13, and 3.14. Running on older Python runtimes is unsupported.
  • Cost and Environment Overhead for Tiers 2 and 3: While Tier 1 is entirely free and self-contained, Tier 2 embedding comparisons and Tier 3 sandboxed agent runs necessitate Docker container orchestration and API keys (via NVIDIA Build endpoints or third-party LLM providers), which will incur associated inference usage costs.

SkillEvaluator delivers a systematic, defensible framework for teams seeking to treat AI agent instructions with the same testing rigor, static auditing, and CI discipline applied to traditional software code.

Sources

Impact-Site-Verification: 9b3ebd10-6c45-44cf-a014-5b8c19b7b285