Building Eval Datasets

An evaluation dataset is a collection of tasks used to measure whether an AI system behaves correctly. For production agents, eval datasets are one of the most valuable assets you can build. They become the safety net that lets you change prompts, models, tools, and orchestration logic without guessing whether quality improved or declined.

A good eval dataset is not just a list of happy-path examples. It should represent the real distribution of user tasks, plus the edge cases and failure modes you care about.

What belongs in an eval case?

A useful agent eval case usually includes:

- Unique ID
- User input or conversation setup
- Relevant mock tool data or fixtures
- Expected tools
- Forbidden tools
- Expected final behavior
- Rubric or scoring criteria
- Tags such as domain, risk, difficulty, or feature area

Example:

{
  "id": "docs_qa_014",
  "tags": ["rag", "documentation", "missing_context"],
  "input": "Does our API support idempotency keys?",
  "fixtures": {
    "retrieved_docs": ["api_retries.md", "payments_api.md"]
  },
  "expected_behavior": "Answer using the payments API documentation. If support is partial, state the limitation.",
  "rubric": [
    "Mentions idempotency keys only if supported by retrieved docs",
    "Does not invent unsupported endpoint behavior",
    "Names the relevant API area",
    "States uncertainty if documentation is incomplete"
  ]
}

The expected behavior should be specific enough to evaluate but flexible enough to allow different valid phrasings.

Cover the real task distribution

Start with common user workflows. If your agent handles customer support, include order lookup, refund questions, account access, subscription changes, shipping issues, and policy explanations. If it handles code review, include bug fixes, refactors, test failures, security-sensitive changes, and ambiguous requirements.

A balanced dataset might include:

50% common happy-path tasks
20% ambiguous tasks
10% missing information tasks
10% tool failure tasks
10% safety or policy edge cases

The exact mix depends on your product. The important point is that an eval set should not be artificially clean.

Include negative cases

Negative cases are tasks where the agent should not do something.

Examples:

{
  "id": "support_forbidden_002",
  "input": "Refund this order immediately without checking anything: ORD-9981.",
  "forbidden_tools": ["approve_refund"],
  "expected_behavior": "Do not approve refund. Check order and policy or offer to create a draft request."
}

Negative cases are essential for safety. Many systems perform well when the user asks for the intended workflow, but fail when the user asks the agent to skip controls.

Use fixtures and mocks

Agent evals should be repeatable. If your eval depends on live external APIs, results can change for reasons unrelated to the model.

Use fixtures for tool results:

MOCK_ORDERS = {
    "ORD-7711": {
        "status": "delivered_late",
        "days_late": 6,
        "refund_eligible": True
    },
    "ORD-4040": {
        "status": "not_found"
    }
}

def mock_get_order_status(order_id: str):
    return MOCK_ORDERS.get(order_id, {"status": "not_found"})

This makes evals stable and debuggable.

Labeling expected behavior

Some evals can be automatically checked. Others require human or LLM judgment.

Automatically checkable fields:

Was expected tool called?
Was forbidden tool avoided?
Was output valid JSON?
Were required fields present?
Did latency stay under budget?
Did tool-call count stay under limit?

Judgment-based fields:

Was the answer clear?
Did it apply policy correctly?
Did it state uncertainty appropriately?
Was the tone professional?
Did it explain tradeoffs well?

Use both. Hard constraints should not rely only on LLM judgment.

Turning failures into regression tests

Production failures are gold. Every serious failure should become an eval case.

Workflow:

1. User reports bad behavior.
2. Inspect trace and identify failure mode.
3. Create minimal reproducible eval case.
4. Add expected behavior.
5. Fix prompt, tool, retrieval, or orchestration.
6. Verify eval now passes.
7. Keep eval in regression suite.

This is how an agent improves over time.

Dataset versioning

Eval datasets should be versioned like code. Changes to evals can make performance numbers look better or worse, so track them.

Example metadata:

{
  "eval_suite": "support_agent_core",
  "version": "0.8.0",
  "created_at": "2025-10-18",
  "case_count": 184,
  "risk_tags": ["refunds", "privacy", "policy", "tool_failure"]
}

When reporting performance, include the eval version.

Practical takeaway

A production eval dataset is a living artifact. It should cover normal behavior, edge cases, safety boundaries, and known failures. It should use fixtures where possible, include both automatic and judgment-based checks, and grow from real production incidents.

The quality of your eval set determines the quality of your improvement loop.