AI RESEARCH
Agreement Metrics for LLM-as-Judge Evaluation: What to Report and Why
arXiv CS.CL
•
ArXi:2606.00093v1 Announce Type: new Validating an LLM judge against human annotations usually means reporting several agreement statistics: accuracy, precision, recall, $F_1$, Cohen's $\kappa$, and one or rank correlations. A survey of 24 recent LLM-as-judge papers finds metric choice entangled with the judgment scale, tie handling, invalid outputs, and abstention handling, and those choices rarely stated.