You are cordially invited to the Analysis Seminar organized by the Department of Mathematics.
Speaker: Kağan Akman (Bilkent University)
“An Optimal Control Approach to Transformers”
Abstract: Since their introduction in 2017, Transformers have proven themselves to be an effective structure for large language models.
Traditionally, a Transformer is trained over a data set by means of gradient descent or some modification of it. In this talk, we commence with presenting a discrete-time dynamical system formulation of a Transformer following Geshkovski et al. (2025). This process exhibits a deterministic McKean-Vlasov type of dynamics. We show that this dynamics can be lifted to a measure-valued discrete-time Markov decision process (MDP). Using the powerful theory of MDPs, we pose the corresponding dynamic programming equations and show the existence of deterministic closed-loop optimal controls, using which we construct open-loop policies compatible with Transformer architecture. Finally, we conclude by relating our formulation with the classical theory of neural networks.
Date: October 22, Wednesday
Time: 14:00 – 15:00
Place: Mathematics Seminar Room, SA – 141