Guide · AI evaluations
AI evals: a quick guide
A brief path from human trace review to useful checks, with security evidence for nine skills.
- Source
- Primary sources + independent review
- Last verified
- September 30, 2026
- Reading mode
- Read and learn
- Linked workflow effects
- Linked skills can read traces, generate files, and use network services
- Outcome
- Choose an eval step and interpret the security evidence
AI evals check whether your app does its job on realistic inputs. Begin with human review, then automate important checks. Reference: Hamel Husain and Shreya Shankar’s article and repository.
Start with real failures
- Start: use
evals-startto route toward pipeline auditing or trace review. - Learn: inspect varied traces; describe failures and acceptable answers.
- Measure: use code for objective checks; validate subjective LLM judges against held-out human labels.
- Improve: fix recurring failures, rerun checks, and add new real cases.
Security snapshot
Checked September 30, 2026 (October 1 UTC), at commit `80d5f7b`. SkillSpector v2.12.0, static-only; scores are out of 100, all LOW. The registry appendix links all 27 provider audits.
| Skill | Use | SkillSpector | Trust Hub | Socket | Snyk |
|---|---|---|---|---|---|
| evals-start | Choose next step | 0 | PASS | PASS | PASS |
| eval-audit | Inspect existing evals | 0 | PASS | PASS | PASS |
| error-discovery | Find failure categories | 0 | PASS | PASS | PASS |
| build-review-interface | Review and label traces | 0 | PASS | PASS | PASS |
| generate-synthetic-data | Broaden test inputs | 0 | PASS | PASS | PASS |
| write-code-eval | Check objective requirements | 0 | PASS | PASS | PASS |
| write-judge-prompt | Judge subjective criteria | 0 | PASS | PASS | PASS |
| validate-evaluator | Compare judges with humans | 0 | PASS | PASS | PASS |
| evaluate-rag | Check retrieval and grounding | 7 | PASS | PASS | PASS |
Limits and evidence
error-discovery has partial reference resolution: Python’s server module and future output files. evaluate-rag has a MEDIUM match in grounding advice, manually identified as a false positive.
Repository scan: 55/HIGH, from README false positives and an unpinned installer advisory. Local Codex found no malicious instructions; error-discovery needs explicit trace-rendering sanitization. Registry audits: September 1 or 24–25; commit correspondence is unverified. Dependencies/CDNs and runtime behavior remain unreviewed. PASS does not prove safety. Triage and evidence.