Fine-Tuning Dynamics of In-Context Factual Recall in Transformers

ArXi:2605.27774v1 Announce Type: new In-context learning \ -- performing tasks based on examples given in the prompt \ -- is an important capability that has emerged in large language models and has received significant attention in both theory and practice. Existing theoretical work often focuses on settings where the learning uses information purely from the prompt. However, many practical instances of in-context learning require the model to retrieve factual knowledge d in the model's parameters, with the context serving to identify which knowledge is relevant.