AI RESEARCH
Tokenisation via Convex Relaxations
arXiv CS.CL
•
ArXi:2605.22821v1 Announce Type: new Tokenisation is an integral part of the current NLP pipeline. Current tokenisation algorithms such as BPE and Unigram are greedy algorithms -- they make locally optimal decisions without considering the resulting vocabulary as a whole. We instead formulate tokeniser construction as a linear program and solve it using convex optimisation tools, yielding a new algorithm we call ConvexTok.