Who Annotates in NLP? A Large-scale Assessment of Human Annotation Reporting between 2018 and 2025

ArXi:2606.02255v1 Announce Type: cross Human annotation is the empirical foundation of much NLP research, from dataset construction to model evaluation, but papers often leave unclear who produced the annotations and how the annotation process was controlled. We provide the first large-scale, task-level audit of human annotation reporting across major NLP venues, asking which annotation details are documented, which are missing, and how reporting varies across time, topic, venue, and intended use of human judgment. We.