Engineering field notes
Evaluate AI agent skills with baseline and regression tests
Design a repeatable skill evaluation. Compare the registry quality tier, targeted functional checks and NVIDIA SkillEvaluator evidence against a real baseline.

Define separate questions for the evaluation
A schema check tells you whether a skill can be parsed. A source review tells you whether references resolve and instructions make sense. A functional test asks whether using the skill improves an engineering outcome under defined conditions. Keep these results separate in your report.
Start with one task that matters to the team and one negative case. Preserve the exact prompt, environment, run budget and expected artifact. Record both the outcome and the cost of the result, including additional time spent inspecting or correcting it.
Compare repeatable conditions
Run the same scenario without the skill, then with the selected skill, holding the agent, repository revision, tool permissions and test fixture constant where practical. Repeat when model output varies. Label input and output so a reviewer can trace which recommendation led to which artifact.
Compare output correctness and side effects. An agent that proposes a test is different from one that ran it on the intended revision. Capture the executed command, exit status or output diff for checks that actually ran. Record untested branches for follow-up.
Use independent evidence with its true scope
The REI quality report exposes separate dimensions. A catalog tier is a compact summary, while the evaluation flag and human-review fields answer different questions. NVIDIA SkillEvaluator is a separate upstream integration. Its documented smoke test is scoped quality evidence, and CI runs cover only the selected skills and configured checks.
When comparing tools, preserve their versions, reports and selectors. A passing result for a single example gives evidence about that example. Keep the result separate from deployment readiness or approval to operate software and hardware.
Set the release decision
Choose minimum acceptance criteria before evaluating. Record whether a release may proceed when a third-party tool is unavailable, and identify the changes that require another run. Keep a held-out regression fixture, including one rejected use case, with the decision record.
Skills in the catalog
Repository sources
References are pinned to the same upstream commit as the public catalog.