AI RESEARCH
Neural Estimation of Pairwise Mutual Information in Masked Discrete Sequence Models
arXiv CS.LG
•
ArXi:2605.20187v1 Announce Type: new Understanding dependencies between variables is critical for interpretability and efficient generation in masked diffusion models (MDMs), yet these models primarily expose marginal conditional distributions and do not explicitly represent inter-variable dependence. We propose a neural framework for estimating pairwise conditional mutual information (MI) directly from the hidden states of a pretrained MDM, using ground-truth MI computed from the model's own conditional distributions for supervision.