AI RESEARCH
Learning GUI Grounding with Spatial Reasoning from Visual Feedback
arXiv CS.CL
•
ArXi:2509.21552v2 Announce Type: replace-cross Graphical User Interface (GUI) grounding is commonly framed as a coordinate prediction task -- given a natural language instruction, generate on-screen coordinates for actions such as clicks and keystrokes. However, recent Vision Language Models (VLMs) often fail to predict accurate numeric coordinates when processing GUI images with high resolutions and complex layouts. To address this issue, we reframe GUI grounding as an interactive search task, where the VLM generates actions to move a cursor in the GUI to locate UI elements.