What is it?
Evaluation is the process of measuring an AI system against defined expectations.
It can assess task success, quality, latency, cost, safety, robustness, and user outcomes. A useful evaluation connects a metric to a real decision.
07 · Evaluation & Reliability · Topic 01
What is Evaluation?
Evaluation measures whether an AI system is useful, reliable, and safe for its intended task.
Think of it like...
Think of it like a driving test and a road-safety check.
The test checks whether the driver can complete a task and follow important rules. Evaluation checks both useful behavior and risks, not just whether a response looks fluent.
How it works
1 · GoalDefine successState what good performance means.
2 · CasesChoose test dataInclude common, difficult, and risky cases.
3 · RunMeasure behaviorCollect outputs and system signals.
4 · CompareCheck resultsUse metrics, labels, or human review.
5 · DecideImprove or releaseUse evidence to choose the next action.
Real-world examples
Everyday example · Answer quality
Input: a set of common user questions
System: checks correctness and helpfulness
Output: quality scores and examples for improvement
Business example · Support workflow
Input: real or representative tickets
System: measures resolution, escalation, and unsafe answers
Output: evidence for release or redesign
What it is NOT
Evaluation ≠ Functional Test
Evaluation: measures performance across chosen dimensions.
Functional test: checks whether a defined task works.
Evaluation ≠ User Feedback
Evaluation: uses a planned measurement method.
User feedback: is one source of evidence about experience.
Evaluation ≠ One Number
Evaluation: can include quality, safety, cost, and reliability.
One number: can hide important tradeoffs.
Related concepts
Task → Test Cases → Metrics / Review → Evidence → Release Decision
Explore next
Remember this
Evaluation turns an AI system's intended behavior into measurable evidence.
Independent explainer