Forge
Skip to main content

LLM evaluation: Metrics, frameworks, and best practices

A comprehensive guide on LLM evaluation, exploring key metrics, human and automated methods of evaluation, best practices, and how to leverage W&B Weave for continuous improvement.
Last updated: October 24, 2026
LLM evaluations turn model behavior into measurable evidence using curated examples, scoring rules, reviewer judgments, and production feedback. For engineers evaluating large language models, that means fewer silent regressions, clearer release gates, safer deployments, and faster prompt or model iteration.

Why LLM evaluation is an engineering control loop

LLM evaluation is a repeatable evidence system for measuring how an LLM application behaves. It combines representative examples, candidate outputs, deterministic scorers, rubric‑based graders, human review, production traces, aggregation rules, and thresholds tied to engineering actions.
This definition matters because evaluating large language models is not equivalent to calculating one benchmark score. A production application may include a language model, system prompt, retriever, embedding model, tool layer, memory store, guardrails, and user interface. A useful evaluation must show which part changed, which behavior regressed, and whether that regression should block a release.
Treat the workflow as a control loop:
  1. Collect representative examples and production failures.
  2. Run a candidate model, prompt, or system configuration.
  3. Score outputs and traces using explicit rules.
  4. Analyze aggregate metrics and failure slices.
  5. Approve, reject, or revise the candidate.
  6. Monitor production and feed new failures back into the evaluation set.
This article focuses on the mechanisms, trade‑offs, and edge cases behind that loop rather than surveying leaderboards. Weights & Biases Models and W&B Weave appear where they provide concrete versioning, experiment‑tracking, evaluation, or observability capabilities.

What LLM evaluation is, and how it differs from testing

LLM evaluation measures how a model or LLM system performs across representative tasks, risks, user intents, and operating conditions. Testing usually checks whether a known invariant holds. Evaluation estimates quality under variation and uncertainty.
A conventional test might assert that a response conforms to a JSON schema or that an unauthorized user cannot call an administrative tool. An eval might estimate whether answers are sufficiently correct, grounded, complete, and helpful across 500 customer questions. Tests produce failures with relatively unambiguous causes. Evals produce evidence that requires aggregation and interpretation.
The basic mechanism has seven parts:

  1. The candidate, such as a model, prompt, retriever, or agent.
  2. An evaluation dataset containing inputs and relevant metadata.
  3. Candidate outputs or execution traces.
  4. Scorers, validators, human judgments, or model‑based rubrics.
  5. Aggregated metrics and confidence estimates.
  6. Slice analysis by task, language, risk, or customer segment.
  7. A decision policy with release, rollback, or review thresholds.
Evals support accuracy measurement, safety signoff, alignment checks, regression testing, prompt comparison, model selection, and production monitoring. They also make subjective disagreements concrete. Instead of arguing that one answer "seems better," reviewers can identify whether it is more correct, relevant, complete, safe, or appropriately uncertain.
Evaluating a standalone model is different from evaluating an end‑to‑end system. A strong model can produce weak results when retrieval returns stale documents, tool arguments are malformed, or a system prompt conflicts with user instructions.

Define the evaluation target before choosing metrics


Start by naming the artifact under evaluation:
Then freeze likely confounders. Record the temperature, sampling parameters, prompt template, system instruction, retrieval corpus, embedding model, chunking strategy, tool implementation, and judge‑model version. Otherwise, a score change cannot be attributed confidently.
Choose an evaluation unit that matches the product. A single answer works for factual QA. A conversation turn may be appropriate for support chat. Agents often require full task trajectories. Production investigations may need a complete session or distributed trace.
Keep development data separate from acceptance data. A small development set can be inspected repeatedly while tuning prompts. A locked acceptance set should remain untouched until release validation. Sampled production traces form a third dataset because they reflect live traffic and may contain drift, abuse, or previously unknown intents.
Version all three datasets and label their origin, collection date, task, risk category, and answerability. Artifacts can version golden datasets, while W&B Weave Traces can preserve inputs, retrieved context, tool calls, outputs, scorer results, latency, and token usage for example‑level debugging.

Main categories of LLM evaluation

Three practical categories organize most evaluation work: knowledge and capability, alignment, and safety. They overlap, but they answer different engineering questions and should not be collapsed into one aggregate score.
  1. A capable model can solve a coding problem while ignoring the required output format.
  2. An aligned assistant can follow instructions while presenting incorrect facts.
  3. A correct and helpful response can still violate privacy or safety policy.
The category determines how you build the dataset, who should review it, which scoring method is valid, and what threshold should block a release. A customer‑support QA system may need separate evaluation suites for policy knowledge, answer grounding, tone, escalation behavior, and privacy. A text‑to‑SQL copilot needs query correctness, schema adherence, permission checks, and execution safety. Autonomous agents add tool‑selection and side‑effect risks.

Knowledge and capability evaluation

Capability evaluation asks whether language models can perform defined tasks such as factual QA, reasoning, coding, summarization, extraction, translation, and planning.
  • Use held‑out labeled data when the correct answer is known.
  • Synthetic examples can expand coverage, but domain experts or deterministic programs should verify them.
  • Exact scoring works for classifications, identifiers, calculations, and structured outputs.
  • Semantic scoring or human review is more appropriate when several answers can be valid.
Common benchmarks provide general calibration. MMLU covers broad academic knowledge, GSM8K covers grade‑school mathematical reasoning, HumanEval evaluates code generation, BBH targets challenging reasoning tasks, HellaSwag measures commonsense completion, GPQA contains expert‑level questions, and TruthfulQA probes misconceptions. HELM demonstrates a broader scenario‑and‑metric evaluation structure.
Relevant metrics include:
  • Exact match and accuracy for discrete answers
  • Precision, recall, and F1 for extraction
  • pass@k for executable code candidates
  • BLEU and ROUGE for reference‑overlap tasks
  • Semantic similarity for acceptable paraphrases
  • Calibration for confidence‑quality agreement
  • Abstention‑aware correctness for uncertain or unanswerable cases
Public benchmark performance is not proof of domain performance. A state‑of‑the‑art LLM can score well on MMLU yet fail on a company's product taxonomy, internal policies, or document conventions.

Alignment and instruction‑following LLM evaluation

Alignment evaluation measures whether outputs follow user intent, system instructions, style constraints, task formats, and human‑preference boundaries.
Suitable methods include pairwise preference judgments, rubric scoring, constraint validators, instruction‑following tests, and human annotation. MT‑Bench and AlpacaEval use model‑based comparisons for open‑ended assistant responses, while IFEval emphasizes verifiable instruction constraints. Domain teams should add preference sets for their own tone, completeness, escalation, and refusal requirements.
Useful metrics include instruction adherence, relevance, completeness, helpfulness, format validity, refusal appropriateness, and preference win rate. Evaluate these dimensions separately. A single preference score can hide an unacceptable format failure.
Consider a support assistant asked to return three troubleshooting steps as JSON. A prose answer containing the correct steps is factually correct but operationally misaligned. Other edge cases include exceeding a requested word limit, using prohibited language, omitting a mandatory disclaimer, refusing a benign request, or obeying a user instruction that conflicts with a higher‑priority system policy.

Safety, robustness, and policy evaluation

Safety evaluation measures whether LLMs avoid harmful, biased, private, insecure, or policy‑violating behavior under normal and adversarial use.
Build adversarial prompt suites, jailbreak tests, privacy‑leakage probes, bias evaluations, and unsafe tool‑use simulations. RealToxicityPrompts, BBQ, BOLD, AdvBench, and HarmBench provide useful starting formats, but production systems also need domain‑specific abuse cases.
Track refusal correctness rather than refusal rate alone. A model that refuses every request may appear safe while making the product unusable. Additional metrics include harmfulness rate, toxicity score, bias disparity, jailbreak success rate, privacy‑leakage rate, and unsafe‑action rate.
Safety scoring requires a policy rubric specifying the protected category, severity, allowed transformations, and escalation path. Automated classifiers are useful for triage, but they can miss context, coded language, or multi‑step harm. High‑severity failures should enter a human‑review queue with the full conversation and tool trace, not just the final answer.

Methodologies and benchmarks for evaluating large language models

Choose the evaluation method based on the decision being made.
  • Offline evaluation on curated data supports model selection, prompt regression, and pre‑release validation.
  • Human evaluation handles ambiguous quality, domain judgment, and policy severity.
  • LLM‑as‑judge scoring provides scalable rubric‑based grading.
  • Online A/B testing measures live product outcomes. Canary releases limit the impact of uncertain changes.
  • Production monitoring detects drift, changing user behavior, and new failure modes.
For a prompt‑only pull request, run a fast offline suite with deterministic validators and a small rubric‑based sample. For a model migration, run the full private dataset plus latency and cost comparisons. For safety signoff, require adversarial testing and human review. For a RAG change, score retrieval and generation independently. For product‑impact questions, use online experiments with guardrail metrics.
Public benchmarks are calibration tools. Private golden datasets validate domain behavior. Adapt benchmark patterns by adding multilingual slices, adversarial cases, no‑answer examples, open rubrics, and explicit abstention scoring. Online evaluation then captures factors that offline data commonly misses, including distribution shift, user feedback, p95 latency, cost, and tool failures.
No framework removes the need to design the eval.
  • HuggingFace LightEval fits reproducible model‑benchmark execution across inference backends.
  • OpenAI Evals supports public and private eval definitions.
  • Ragas provides RAG‑oriented metrics.
  • LangSmith fits evaluation inside LangChain‑based workflows.
  • W&B Weave combines versioned evaluations with traces, scorer comparisons, and production monitoring.
Choose based on the artifact and workflow rather than looking for one universally best tool.
Cross‑paper comparisons remain fragile when authors do not disclose dataset versions, prompts, few‑shot examples, sampling parameters, judge models, or scoring rules. Reproducibility requires all of them.

Metrics by artifact: model outputs, prompts, RAG pipelines, and agents

Metrics should map to an artifact and failure mode, not to what is easiest to compute.
For standalone outputs, use accuracy, exact match, F1, semantic similarity, or calibrated rubric scores. For prompts, measure instruction adherence, format validity, output variance, refusal behavior, token length, and regression against the previous prompt. For retrieval, use recall@k, precision@k, mean reciprocal rank, and nDCG. For tools and agents, evaluate calls and trajectories.
Operational metrics belong in the same release decision:
  • Mean and p95 response latency
  • Input, output, and total token counts
  • Cost per request and successful task
  • Timeout and retry rates
  • Throughput under expected concurrency
Every metric needs an associated action. A high hallucination rate should block release or route uncertain cases to review. Low retrieval recall should trigger corpus or ranking changes. Excessive p95 latency may require a smaller model, shorter context, or different routing policy.
Avoid a single weighted score unless the components and weights remain visible. A combined score can improve while safety regresses because latency improved. It can also hide whether failures originated in the model, prompt, retriever, tool, or policy layer.

Evaluating an individual LLM answer

Use a multidimensional rubric for one answer:
DimensionQuestion
CorrectnessAre its factual and logical claims true?
RelevanceDoes it address the user's actual request?
CompletenessDoes it include the necessary information?
GroundednessAre claims supported by the supplied context?
SafetyDoes it comply with policy and avoid harmful guidance?
UncertaintyDoes confidence match the available evidence?
Instruction adherenceDoes it satisfy format, style, and scope constraints?

Use pass/fail for hard requirements, a 0‑2 scale for rapid review, or a 0‑5 scale when distinctions are well‑defined. Do not add scale points without anchor examples. Reviewers should record a short rationale and failure tags such as unsupported_claim, missing_step, wrong_format, or should_abstain.
Correctness is not equivalent to quality. An answer may be correct but incomplete, polished but false, safe but useless, or relevant but unsupported.
Accuracy‑only scoring can reward guessing because abstention receives the same score as a wrong answer while guessing has some probability of receiving credit. Open‑rubric evaluation addresses this by defining the reward or penalty for correct answers, unsupported answers, and abstentions before evaluation. The appropriate values depend on stakes: guessing may be acceptable in a trivia application and unacceptable in a medication workflow.

Metrics for RAG pipelines

Score retrieval and generation separately. Otherwise, a correct final answer can hide brittle retrieval, while a weak answer may be blamed on the model even though the required evidence was never retrieved.
Retrieval metrics include:
  • recall@k: whether relevant evidence appears in the top k results
  • precision@k: how much of the retrieved set is relevant
  • Mean reciprocal rank: how early the first relevant result appears
  • nDCG: ranking quality with graded relevance
  • Context relevance: usefulness of retrieved passages
  • Duplicate rate: redundant chunks in the context
  • Stale-document rate: outdated evidence returned to the model
Generation metrics include faithfulness, groundedness, answer relevance, citation accuracy, completeness, and no‑answer correctness.
Include cases with conflicting documents, partial evidence, unanswerable questions, long‑context dilution, and stale policies. Also test passage‑level retrieval. Finding the correct document is insufficient when the returned chunk omits the relevant paragraph.
W&B Weave Traces can capture the query, retrieved chunks, prompt, output, citations, scorer results, latency, and token usage. When faithfulness falls, inspect individual traces to determine whether the evidence was absent, buried, contradictory, or ignored.

Metrics for agents and tool use

Agent evaluation scores trajectories rather than final text. A correct answer is not sufficient if the agent called an unauthorized tool, retried indefinitely, or produced an irreversible side effect.
Measure task success, tool‑call validity, argument correctness, step count, retry rate, loop rate, budget adherence, and unsafe‑action rate. Store each tool request, observation, state transition, and approval decision.
Use deterministic fixtures in CI. A mock calendar tool, for example, can return fixed availability and reject writes without an approval token. This makes regressions reproducible. Use sandboxed live tools in nightly or pre‑release evaluations when higher fidelity is worth additional cost and risk.
Safety cases should cover permission boundaries, destructive actions, data exfiltration, prompt injection inside tool observations, and human‑approval gates. Test whether an agent treats retrieved content as data rather than authority.
Inspect successful traces as well as failed ones. Aggregate task success can hide a brittle path that happened to work after unnecessary retries or an unsafe call.

Using LLM‑as‑judge and G‑Eval without fooling yourself

LLM‑as‑judge evaluation uses a model to grade outputs against references, rubrics, policies, or pairwise preferences. It fits summarization, open‑ended QA, helpfulness, style, relevance, safety triage, and other tasks where several outputs may be valid.
A G‑Eval‑style pattern uses explicit criteria, a structured grading form, rationale capture, and aggregation across examples. The structure is more important than asking a model to "rate this answer." Each dimension needs definitions, scale anchors, and examples of borderline behavior.
Calibrate the judge against human labels before using it as a release gate:
  1. Collect a stratified human‑labeled sample.
  2. Run the judge on the same sample.
  3. Calculate agreement and a confusion matrix.
  4. Inspect results by language, task, length, and failure type.
  5. Review disagreements with domain experts.
  6. Revise the rubric and repeat.
Pin the judge‑model version, use low temperature, randomize pairwise candidate order, blind model identity, and test for verbosity, position, and self‑preference bias. Preserve judge rationales for debugging, but score the structured output rather than trusting the rationale's fluency.
W&B Weave Evaluations can run and compare judge scorers over a versioned dataset. The Evaluation Playground provides a concrete workflow for changing judge prompts, running them against the same examples, and inspecting example‑level disagreements before promoting a rubric.

Worked example: a minimal abstention‑aware evaluation harness

The smallest useful harness contains examples, candidate outputs, a scorer, aggregation, and release thresholds. Save the following as eval_abstention.py and run it with python eval_abstention.py. It requires no external packages.
rows=[
{'id':'q1','ref':'Paris','pred':'Paris is the capital of France.','answerable':True},
{'id':'q2','ref':None,'pred':'Not enough information in context.','answerable':False},
{'id':'q3','ref':'4','pred':'5','answerable':True}
]
def abstain(t):
return any(x in t.lower() for x in ['not enough information','cannot determine','unknown'])
def score(r):
a=abstain(r['pred'])
c=r['answerable'] and r['ref'].lower() in r['pred'].lower()
h=(not r['answerable'] and not a) or (r['answerable'] and not c and not a)
return {'correct':c or (not r['answerable'] and a),'abstained':a,'hallucinated':h}
scores=[score(r) for r in rows]
summary={k:round(sum(x[k] for x in scores)/len(scores),3) for k in scores[0]}
print(summary)
Expected output:
{'correct': 0.667, 'abstained': 0.333, 'hallucinated': 0.333}
The first response is correct. The second question is unanswerable, so abstaining counts as correct. The third response confidently gives the wrong arithmetic answer, so it counts as hallucination‑like behavior.
Add a release policy after the print statement:
release_ok = (
summary['correct'] >= 0.80
and summary['hallucinated'] <= 0.05
)
if not release_ok:
raise SystemExit('Release blocked by evaluation thresholds')
This dataset blocks the release. In a real suite, use enough examples to estimate rates reliably and define per‑slice thresholds for high‑risk tasks.
To evaluate a real application, replace each static pred with a call to a model, prompt variant, RAG pipeline, or agent. Preserve the configuration and example‑level outputs. You can log the results to W&B:
import wandb

wandb.login()

with wandb.init(project='llm-evaluation', config={
'candidate': 'support-prompt-v7',
'temperature': 0
}) as run:
run.log(summary)

table = wandb.Table(
columns=['id', 'reference', 'prediction', 'answerable',
'correct', 'abstained', 'hallucinated']
)
for row, result in zip(rows, scores):
table.add_data(
row['id'], row['ref'], row['pred'], row['answerable'],
result['correct'], result['abstained'], result['hallucinated']
)
run.log({'evaluation_examples': table})
The important design decision precedes the code: define how wrong answers, unsupported answers, and abstentions are scored before collecting results.

Trade‑offs in LLM evaluation: accuracy vs latency, cost vs throughput, and flexibility vs determinism

Every evaluation design makes engineering trade‑offs.
Accuracy vs latency: More human review, repeated judge calls, larger datasets, and multiple sampling runs increase confidence but delay feedback. Use deterministic smoke tests on pull requests, broader nightly suites, and human review before release.
Cost vs throughput: Large judge models, long‑context RAG examples, and agent simulations can be expensive. Cache unchanged outputs, sample low‑risk production traffic, use cheaper judges for triage, and reserve stronger judges for uncertain or high‑risk examples.
Flexibility vs determinism: Rubric‑based judges handle open‑ended responses and multiple valid answers. Exact‑match checks, schema validators, and executable tests are easier to reproduce and debug. Prefer deterministic scoring whenever the requirement can be expressed deterministically.
Coverage vs maintainability: Broad suites catch more failures but accumulate stale, duplicated, or low‑signal examples. Assign owners, track example provenance, remove obsolete cases, and add new examples from production incidents.
Public comparability vs domain validity: Public benchmarks help compare general model capability. Private evaluations usually predict production behavior more accurately because they include real terminology, policies, and failure costs.
Map these trade‑offs to execution tiers:
  • Pull request: fast deterministic and high-signal smoke evals
  • Nightly: full regression suites and judge-based scoring
  • Release gate: locked acceptance data and safety review
  • Red-team review: adversarial and high-severity cases
  • Production: sampled monitors, drift detection, and incident-triggered review

Known failure modes and limitations

Evaluation fails when teams optimize the score rather than the intended behavior.
Benchmark chasing occurs when model providers or application teams tune for public leaderboards while neglecting production requirements. Contamination makes results unreliable when benchmark examples enter pretraining, fine‑tuning, prompt development, or evaluator prompts. Maintain private holdouts and rotate some acceptance examples.
Coverage gaps commonly include multilingual users, rare facts, long‑tail intents, long‑context reasoning, accessibility needs, domain constraints, and adversarial behavior. An aggregate score can remain stable while one high‑value customer segment regresses sharply.
A lack of standards creates another problem. Two reports may use the same benchmark name but different prompts, few‑shot examples, decoding parameters, model versions, or judge rubrics. Without those details, scores are not directly comparable.
Accuracy‑based metrics can also incentivize guessing when abstention is scored as an ordinary error. Add no‑answer examples and measure controllable abstention rather than assuming every prompt must receive a substantive answer.
Automated evaluators have their own failure modes: verbosity bias, position bias, self‑preference, brittle judge prompts, and inconsistent severity judgments. Human labels also contain noise, especially when rubrics are vague. Finally, every offline dataset becomes less representative as products, users, tools, and policies change. Production distribution drift is not an edge case. It is a reason to keep the control loop running.

Operationalizing evaluations in CI and production observability

A practical lifecycle uses multiple evaluation tiers:
  1. Run local checks while developing prompts and scorers.
  2. Execute a small smoke suite on every pull request.
  3. Run the complete regression suite nightly.
  4. Require capability, alignment, and safety gates before release.
  5. Deploy to a canary population.
  6. Monitor production quality, latency, cost, and policy failures.
  7. Convert reviewed production failures into new evaluation examples.
For reproducibility, version the evaluation dataset, prompt, model identifier, inference parameters, retrieval corpus, embedding model, tool implementation, judge prompt, judge‑model version, and scorer code.
A Weights & Biases Models workflow can use Artifacts for golden datasets, Tables for example‑level outputs and scores, Registry for approved candidates, and Reports for release reviews. A candidate should enter the Registry only after its required metrics and slices pass. The associated Report should preserve comparisons, threshold decisions, known failures, and reviewer signoff.
For LLM applications, initialize Weave and trace application functions:
import weave

weave.init('ml-platform/support-assistant')

@weave.op()
def answer_question(question):
return support_pipeline(question)
Weave Traces then provide end‑to‑end records for model calls, retrieval, tools, latency, and outputs. Weave Evaluations apply scorers to curated datasets. Monitors passively score sampled production traffic, while Guardrails can intervene when real‑time policy scores cross thresholds. Playground supports prompt and judge iteration against stored examples.
Do not triage only by average score. Slice failures by task, language, customer segment, document source, tool, safety category, latency bucket, and cost bucket. Send low‑confidence examples, high‑severity safety failures, high‑value customer interactions, and automated‑judge disagreements to human review.
The output of triage must be an action: roll back, revise the prompt, repair retrieval, update a tool, change routing, collect labels, or add a guardrail.

When not to use LLM evaluation

Do not use LLM evaluation as a substitute for requirements, threat modeling, deterministic controls, or domain accountability. Evals can provide evidence about observed behavior, but they cannot prove that a system is safe under every possible input.
Use conventional tests for exact invariants. Validate JSON schemas with a parser. Enforce permissions in application code. Check arithmetic with executable logic. Use static analysis for generated code, SQL parsers and allowlists for database access, and contract tests for APIs. Do not ask an LLM judge to verify what software can verify exactly.
Avoid LLM‑as‑judge when data is too sensitive to send to the judge, when judge and candidate may share evaluation‑data leakage, or when the evaluator has an unresolved conflict with the target model. Public benchmarks should never be the sole acceptance criterion for medical, legal, financial, regulated, or safety‑critical deployments.
If product goals are unclear or no representative examples exist, do not begin by building elaborate evaluation infrastructure. Define the tasks, collect real examples, specify failure costs, and align stakeholders first. The metric follows the decision.
Offline evals are also insufficient when the main risks come from production drift, adversarial users, changing corpora, evolving tools, or unknown interactions. Those systems require monitoring, incident review, and deterministic runtime controls in addition to pre‑release evaluation.

Conclusion: build evals that change engineering decisions

The purpose of LLM evaluation is not to produce a universal model score. It is to make uncertain behavior inspectable enough to support a decision.
Start with a small, high‑signal dataset containing normal cases, important edge cases, known failures, and unanswerable questions. Define explicit rubrics, add abstention‑aware scoring, calibrate subjective scorers against human judgments, and store example‑level results rather than only aggregate metrics. Expand the suite by failure slice, not by metric count.
The next implementation step is straightforward: place the worked harness in CI, version its dataset and configuration, log its predictions and scores, inspect every regression, and add production monitoring for drift. Tie each threshold to a concrete action such as blocking a pull request, requiring review, limiting a canary, or rolling back a release.
Evaluation quality is measured by whether it prevents bad releases and helps engineers diagnose and ship good fixes faster.

Sources

Tazik Shahjahan
Tazik Shahjahan •  
AI generated image overload :/
Reply