AI RESEARCH
HalleluBERT: Let Every Token That Has Meaning Bear Its Weight
arXiv CS.CL
•
ArXi:2510.21372v2 Announce Type: replace Transformer-based models have advanced NLP, yet Hebrew still lacks a RoBERTa encoder that is trained at scale and released in both base and large variants. We present HalleluBERT, a RoBERTa-based encoder family trained from scratch on 49.1~GB of deduplicated Hebrew web text and Wikipedia using a Hebrew-specific byte-level BPE vocabulary.