Evaluation for LLM Features

What & why

Mental model

The five layers of a solid eval system

• Task definition: what exact user job is being tested?
• Dataset quality: do the examples resemble production work, edge cases, and expensive failures?
• Scoring logic: can you judge success consistently with exact checks, rubrics, or human review?
• Failure taxonomy: can you label whether the miss came from reasoning, retrieval, tool use, formatting, safety, or instruction following?
• Decision policy: what score or failure pattern is strong enough to ship, block, or escalate?

If you only do three things this month

• Collect 25 to 50 real product tasks from support logs, QA transcripts, analyst workflows, or user requests instead of inventing synthetic examples first.
• Define one pass rule and one failure label for each task so reviewers do not improvise a new standard every time.
• Run the eval set before and after every meaningful system change and keep the results visible in the same place as other release signals.

Start here

1. 01
   Pick one product behavior, not the whole AI strategy

   Start with a single job such as answer a support question, draft a follow-up email, extract fields from a document, choose the right tool, or summarize a meeting accurately.

   Broad goals like `make the assistant smarter` are too vague to evaluate well.

2. 02
   Write down what a good answer must do

   List the requirements in plain language: factual accuracy, use of supplied context only, correct tool selection, required fields present, concise style, refusal when unsafe, or citation when needed.

   This becomes the basis for your grading logic and keeps the team aligned on what `better` actually means.

3. 03
   Build a small golden set from real work

   Use examples from real product traffic, internal workflows, or realistic QA scripts. Include easy cases, typical cases, edge cases, and cases where failure would be expensive.

   A small precise set is more useful than a large noisy dataset early on.

4. 04
   Choose the lightest scoring method that can be trusted

   Use exact matching when the output is tightly constrained, rubric grading when quality is contextual, and human review when the task still resists reliable automation.

   Many strong eval systems combine all three rather than pretending one method is universal.

5. 05
   Run a baseline before you optimize anything

   Record how the current prompt, model, retrieval setup, or agent stack performs before making changes.

   Without a baseline, teams often mistake novelty for improvement.

Build the full system

1. 01
   Split the eval set by job and risk

   Keep separate groups for common happy-path tasks, complex reasoning tasks, high-risk safety tasks, formatting-sensitive tasks, and regression canaries.

   This prevents a good average score on easy prompts from hiding failures in the cases that matter most.

2. 02
   Score behavior at the right level

   For extraction or classification, exact correctness may be appropriate. For summarization or support writing, use rubrics that check whether the answer is correct, complete enough, grounded, and appropriately formatted.

   For agent workflows, also measure intermediate behavior such as tool choice, argument quality, loop count, and whether the final answer reflects tool results correctly.

3. 03
   Keep graders narrow and explicit

   A grader should answer a small question cleanly, such as `did the reply cite only allowed sources?` or `did the assistant choose the billing tool when the request required account data?`.

   Narrow graders are easier to trust, debug, and combine than one vague `is this good?` judge.

4. 04
   Label failures so the next action is obvious

   Separate failure causes like hallucinated facts, missing retrieval evidence, wrong tool, bad tool arguments, unsafe content, verbose answer, or ignored instruction.

   A score without a failure taxonomy tells you that something is wrong, but not what to fix.

5. 05
   Use evals as release infrastructure

   Run the relevant eval suite whenever you change the model, prompt, retrieval indexing, ranking logic, tool schema, context window strategy, or safety rules.

   The point is not only model comparison. The point is catching behavior drift anywhere in the stack.

6. 06
   Check correlation with real product outcomes

   A useful eval system should eventually connect to production signals such as task completion, escalation rate, analyst save time, edit distance, or support deflection.

   If the eval score rises while real user outcomes stay flat or worsen, the eval is missing something important.

What stronger evaluation looks like

1. 01
   Prompt demos to repeatable baselines

   Before: people test a few favorite prompts manually and call the system ready.

   After: every candidate change is run against the same stored task set with visible results.

   Why it works: the team stops confusing anecdotal wins with durable improvement.

2. 02
   One giant score to segmented evidence

   Before: one number hides whether the problem is safety, grounding, tool use, or formatting.

   After: results are broken out by task type, grader, and failure label.

   Why it works: optimization effort goes to the real bottleneck instead of the loudest guess.

3. 03
   Human-only review to mixed review

   Before: every test requires expensive manual judgment.

   After: exact checks and focused graders handle the obvious cases, while humans review the ambiguous or high-stakes ones.

   Why it works: coverage expands without pretending automation is more reliable than it is.

Choose the next move

1. 01
   If the feature is new, define the task and golden set first

   Do not start by tuning the prompt. Start by deciding what success and failure mean on a realistic slice of work.

2. 02
   If results feel noisy, tighten the dataset before changing models

   Ambiguous test cases and vague graders often create more confusion than model variance does.

3. 03
   If agent behavior is failing, inspect tool traces separately

   A bad final answer may come from wrong retrieval or tool use rather than weak reasoning alone.

4. 04
   If the score is rising but users are unhappy, revisit the rubric

   Your eval may be rewarding surface polish while missing the actual product job.

5. 05
   If the task is high stakes, keep human review in the loop

   Automated graders are useful, but they should not be the only release gate for medical, legal, financial, or safety-sensitive behavior.

Prove it is working

• The same eval suite can distinguish clearly between a weaker baseline and a stronger candidate system.
• Failure clusters are understandable enough that the team can name what broke and why.
• Prompt or model changes that looked good in demos are blocked when they regress important cases.
• Eval improvements correlate at least loosely with production outcomes such as fewer escalations, lower edit burden, or better task completion.
• The team can explain what is still not covered instead of assuming the eval suite is complete.

Final checklist

Common ways this goes off track

• Testing with hand-picked demo prompts that flatter the system.
• Using a dataset so synthetic or clean that it no longer resembles production work.
• Treating one aggregate score as sufficient evidence for release decisions.
• Relying on an LLM judge without checking whether the grader itself is stable and aligned with the task.
• Changing prompts, retrieval, and tools simultaneously, which makes the source of improvement impossible to isolate.
• Optimizing to the eval until the system gets better at passing the test than serving the user.

Examples

• OpenAI evaluation best practices — A concise official guide to designing evals, framing success criteria, and choosing scoring approaches.
• Anthropic on evals for AI agents — A practical explainer for agent-style evaluation, including tasks, grading logic, and iteration.
• Google Gen AI evaluation overview — An official overview of structured evaluation patterns for generative models and applications.

Read next

• Model Context Protocol — Read the tool-layer companion if your eval work needs to inspect how models call external systems and pass structured context.
• Generative Engine Optimization — Read the discovery-layer companion if you want to evaluate how model-generated answers surface and cite your content externally.

References

• [1] OpenAI evaluation best practices — Official guidance on eval structure, dataset design, graders, and measurement.
• [2] Working with evals — OpenAI developer guide for running eval workflows and thinking operationally about evaluation.
• [3] Demystifying evals for AI agents — Anthropic engineering guidance on task-based agent evaluation and automated grading.
• [4] Define success criteria and build evaluations — Anthropic documentation on turning qualitative goals into measurable tests.
• [5] Gen AI evaluation service overview — Google Cloud documentation covering evaluation workflows, rubric-based metrics, and model comparison.
• [6] A statistical approach to model evaluations — A research-oriented guide to interpreting model evaluation results with stronger statistical discipline.