AI RESEARCH
Convergence of Two-Timescale Markovian Stochastic Approximations with Applications in Reinforcement Learning
arXiv CS.LG
•
ArXi:2605.31172v1 Announce Type: new This work studies the convergence of two-timescale stochastic approximations (SA), a class of iterative algorithms that update two sets of parameters in fast and slow timescales respectively. Notable examples of two-timescale SA in reinforcement learning (RL) include temporal difference learning with gradient correction (TDC) and actor-critic methods. Previously, the stability (i.e., boundedness) and convergence of two-timescale SA were only established under i.i.d. noise.