LLM Evals: Building Evaluation Suites for Production LLM Apps

LLM applications fail silently. A model can answer confidently, cite nothing, and drift in quality between model versions without any error being raised. Evals are the only defense: a repeatable way to measure whether your system got better or worse. This guide covers what to measure, how to build the suite, and how to wire it into CI so regressions never ship.

What to Measure

Different layers need different metrics:

Layer	Metric	What It Catches
Retrieval	Recall@k, MRR, hit rate	Missing relevant documents
Generation	Faithfulness, groundedness	Hallucination
Task quality	Accuracy, exact match, F1	Wrong answers
Format	JSON validity, schema conformance	Broken structured outputs
Safety	Toxicity, PII leakage, refusal	Harmful or leaking responses
Latency/cost	p95 latency, tokens per request	Regressions in serving

Start with the metric that matches your product's core promise. If you sell grounded answers, faithfulness matters more than exact match. If you sell structured extraction, schema conformance is the gate.

The Golden Set

Every eval suite starts with a golden set: real inputs with known-good expected outputs. Rules for building one:

1. Use real production traffic, not hand-written examples. Sample actual user queries.
2. Cover edge cases deliberately: ambiguous queries, empty inputs, adversarial inputs, out-of-domain questions.
3. Keep it small but stable: 100-300 examples is enough to catch regressions; more slows every run.
4. Version it: golden sets change. Store them in git with the code that consumes them.
5. Never tune prompts on the golden set; hold out a separate set for that.

[
  {
    "id": "q-001",
    "query": "What is the refund policy for annual plans?",
    "expected_sources": ["docs/billing/refunds.md"],
    "expected_answer_contains": ["30 days", "full refund"]
  }
]

LLM-as-Judge

For open-ended outputs, the most practical scorer is another LLM. It is not perfect, but it correlates well with human judgment when set up correctly.

from openai import OpenAI

client = OpenAI()

JUDGE_PROMPT = """You are evaluating an AI assistant's answer.
Question: {question}
Reference answer: {reference}
Assistant answer: {answer}

Score the assistant answer 1-5 on:
- Correctness: does it match the reference?
- Completeness: does it cover everything the reference covers?
- Conciseness: no irrelevant filler.

Respond with JSON: {{"score": int, "reason": str}}"""

def judge(question: str, reference: str, answer: str) -> dict:
    resp = client.chat.completions.create(
        model="gpt-4o",
        response_format={"type": "json_object"},
        messages=[{"role": "user", "content": JUDGE_PROMPT.format(
            question=question, reference=reference, answer=answer
        )}],
    )
    return json.loads(resp.choices[0].message.content)

Judge hygiene:

• Use a stronger model as judge than the one being evaluated.
• Blind the judge to which system produced the answer when comparing two systems.
• Validate the judge: spot-check 50 judge scores against human scores. If correlation is poor, fix the rubric.
• Watch for position bias: when comparing two answers, run the comparison both ways and average.

Faithfulness and Hallucination Metrics

For RAG systems, the critical metric is faithfulness: does the answer stay grounded in the retrieved context? RAGAS provides this out of the box:

from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_precision, context_recall
from datasets import Dataset

eval_data = Dataset.from_list([
    {
        "question": "What is the refund policy?",
        "answer": "Annual plans can be refunded within 30 days.",
        "contexts": ["docs/billing/refunds.md: Annual plans are refundable within 30 days of purchase."],
        "ground_truth": "Annual plans are refundable within 30 days.",
    }
])

result = evaluate(
    eval_data,
    metrics=[faithfulness, answer_relevancy, context_precision, context_recall],
)

print(result)
# {'faithfulness': 0.95, 'answer_relevancy': 0.88, ...}

• Faithfulness: fraction of claims in the answer supported by the context. Low = hallucination.
• Context precision: are the retrieved documents actually relevant? Low = bad retrieval.
• Context recall: did retrieval find everything needed? Low = missing documents.

Regression Testing in CI

The eval suite is only useful if it runs automatically. Wire it into CI so every PR that touches prompts, retrieval, or model config runs the suite:

# .github/workflows/evals.yml
name: LLM Evals
on:
  pull_request:
    paths:
      - "prompts/**"
      - "src/retrieval/**"
      - "src/llm/**"
      - "evals/**"

jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install -r requirements.txt
      - run: python evals/run_suite.py --golden evals/golden.jsonl
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

The runner should fail the build on regression, not on absolute score:

import json
import sys

THRESHOLDS = {"faithfulness": 0.85, "answer_relevancy": 0.80}

def main():
    results = run_suite("evals/golden.jsonl")
    baseline = load_baseline("evals/baseline.json")
    failures = []

    for metric, value in results.items():
        if value < THRESHOLDS[metric]:
            failures.append(f"{metric}: {value:.2f} < {THRESHOLDS[metric]}")
        if value < baseline[metric] - 0.03:  # regression vs last known good
            failures.append(f"{metric} regressed: {value:.2f} vs baseline {baseline[metric]:.2f}")

    save_baseline(results)
    if failures:
        print("\n".join(failures))
        sys.exit(1)

if __name__ == "__main__":
    main()

A/B Testing Model Changes

Before swapping models or prompts in production, run a shadow comparison: send the same traffic to both versions and score both with the same judge.

def shadow_compare(query: str, model_a: str, model_b: str) -> dict:
    answer_a = generate(query, model_a)
    answer_b = generate(query, model_b)
    score_a = judge(query, answer_a, answer_b)  # blind comparison
    score_b = judge(query, answer_b, answer_a)
    return {
        "model_a_wins": score_a["score"] > score_b["score"],
        "model_b_wins": score_b["score"] > score_a["score"],
    }

Run this on a week of real traffic before committing to a model upgrade. Model "upgrades" routinely regress on your specific task.

Common Pitfalls

• Eval on the training distribution: your golden set must reflect production traffic, not your own writing style.
• Judge on the same model family: a judge that is the same model as the system being tested is biased toward its own style.
• Chasing one metric: faithfulness can be gamed by refusing to answer. Track refusal rate alongside it.
• No baseline: without a stored baseline, you cannot tell regression from noise.
• Flaky evals: non-deterministic sampling makes scores jump. Set temperature=0 for eval runs.

Implementation Checklist

• Build a golden set from real production traffic (100-300 examples)
• Pick 2-3 metrics that match your product's core promise
• Validate your LLM judge against human scores before trusting it
• Add faithfulness/context metrics for any RAG system
• Store baselines and fail CI on regression, not absolute score
• Run evals on every prompt, retrieval, or model change
• Shadow-compare model upgrades on real traffic before shipping
• Use temperature 0 and fixed seeds for reproducible runs

---

MatterAI builds frontier AI infrastructure for engineering teams — from inference-optimized models to autonomous coding agents and agentic code reviews.

Explore what we're building:

• Orbital IDE — Autonomous AI coding agent with background agents and deep codebase memory
• AI Code Reviews — Agentic pre-commit reviews across GitHub, GitLab, and Bitbucket
• Axon Models — Frontier-grade reasoning models at 70% lower inference cost

Get started free - https://app.matterai.so

• Read the docs
• View pricing

---

Follow us on X · LinkedIn · GitHub