Evaluation Is the New QA for AI Systems
AI systems don't fail like normal software, so normal tests don't catch them. Why golden-set evaluation in CI is now part of quality engineering.
Traditional software fails loudly. A function throws, a test goes red, a page returns an error. AI systems fail quietly: the answer is fluent, plausible, and wrong. That is why teams that ship AI with only conventional tests keep getting surprised in production.
Why unit tests aren't enough
- Outputs vary between runs, so exact-match assertions break or get deleted
- A prompt change that fixes one case silently breaks five others
- Model upgrades change behaviour across the board, with no compiler error to warn you
- Retrieval quality degrades as documents change, long after the feature shipped
What evaluation looks like
A golden set: real questions or tasks with known-good answers, collected from the people who do the work. Automated scoring — exact checks where possible, rubric or model-graded checks where not — run on every change in CI. A threshold that blocks a release when quality drops, just like a failing test. And a habit of adding every production failure back into the set.
Quality engineering has to own it
Evaluation belongs in the same pipeline as functional, accessibility, and security checks. A release gate that ignores AI quality is a release gate with a hole in it. The teams that get this right treat evaluation data as a product asset — owned, versioned, and growing.
How we build it
Every RAG pipeline and agent ResoluteX ships includes an evaluation harness with eval runs in CI, so quality is a graph rather than a feeling. The same thinking runs through Verity, which combines Playwright, axe-core, OWASP ZAP and Semgrep into one release verdict.
Continue the decision
Have the problem this post describes? The AI Architect will match it to a product or practice — with a price.
Run the AI Architect