Abstract
As agentic LLM systems move from prototypes to deployment, evaluating them has become more important and more difficult. Evaluation goals are often left implicit. Without a clear account of what a system is expected to do, how it can fail, and which failures matter, metric choices become difficult to justify, interpret, or validate. We present Litmus, a zero-label system that designs evaluation and monitoring metrics for AI pipelines by eliciting evaluation intent from source code and targeted interrogation. Litmus first identifies what must be measured and why, then converts those answers into constraints for a justified, per-stage metric portfolio. We evaluate Litmus on three real, code-defined pipelines against AutoMetrics and three DynamicRubric baselines. Litmus achieves broader concern coverage on account grouping at matched size, spans more stages, produces a near-zero-redundancy portfolio, and, under a zero-label design-time constraint, ranks first in validity against per-row quality labels on all three pipelines, decisively on scientific QA ($\rho{=}0.72$ vs. $0.47$ for the strongest baseline on the stitched single-vector score), with overlapping intervals on the other two. Results support a shift from automatic metric implementation to automatic metric specification: before asking which metric to compute, evaluation systems should ask what must be measured and why.
Abstract (translated)
URL
https://arxiv.org/abs/2606.23403
