AI RESEARCH
MicroSpec: Accelerating Speculative Decoding with Lightweight In-Context Vocabularies
arXiv CS.CL
•
ArXi:2605.26444v1 Announce Type: new Large language models typically employ vocabularies of over 100k tokens, which creates a major computational bottleneck at the final linear projection layer when performing speculative decoding. Current methods for vocabulary pruning depend on either fixed or coarse-grained sub-vocabularies, requiring around 30k active tokens to preserve the quality of the draft model. We