AI RESEARCH
Deterministic Inference across Tensor Parallel Sizes That Eliminates Training-Inference Mismatch
arXiv CS.LG
•
ArXi:2511.17826v2 Announce Type: replace Deterministic inference is increasingly critical for large language model (LLM) applications such as LLM-as-a-judge evaluation, multi-agent systems, and Reinforcement Learning (RL). However, existing LLM serving frameworks exhibit non-deterministic behavior: identical inputs can yield different outputs when system configurations (e.g., tensor parallel (TP) size, batch size) vary, even under greedy decoding. This arises from the non-associativity of floating-point arithmetic and inconsistent reduction orders across GPUs.