Beyond naive “accuracy” for production conversational AI

Benchmark leaderboards incentivize brittle comparisons. Operational teams deploying LLMs internally care about cohort regressions, tail latency under load, and whether an expert would accept each answer—not leaderboard percentages alone.

Calibration with business tasks

Freeze a golden task set drawn from real tickets with scrubbed payloads. Tie each prompt to observable outcome classes (successful resolution vs. escalate). Refresh quarterly; stale sets mis-track improvements when product surfaces move.

Latency-qualified correctness

Correct answers delivered after SLA expiry are commercially incorrect. Separate charts for p50 versus p95 end-to-end response time—with identical evaluation harness—to avoid burying regressions beneath averages.

Provider and model deltas

Record (model identifier, toolkit version) for every sampled trace. Inference vendors ship silent micro-updates; without diff-based regression thresholds, outages masquerade as “user complaints.”

Structured rubrics for subjective domains

Two independent reviewers—or one reviewer plus automated constitutional checks—for high-impact categories reduce single-rater optimism. Discrete rubrics scale better than unbounded thumbs-up/down.