Seminar of Algorithms of Saclay Plateau: Luis Cristóbal Rojas on Friday 13 June 2025 at 13h30

===== Seminar "Algorithms of Saclay Plateau" =====

Dear all,

Reminder: Recall that next seminar will be on Thursday 12 June 2025 at 15h00 by Niklas Kormann: Graph Neural Networks for Glomeruli Health Classification in Histopathology.

Notce that for another  seminar of the seminar "Algorithms of Saclay Plateau", we are happy to welcome Luis Cristóbal Rojas.

***** Friday 13 June 2025 at 13h30, Grace Hopper *****

Luis Cristóbal Rojas -- On limitations of LLMs.

Understanding how Transformers work and how they process information is key to the theoretical and empirical advancement of these machines. In this talk, we will discuss the existence of two phenomena in Transformers, namely isolation and continuity. Both of these phenomena hinder Transformers from learning even simple pattern sequences. Isolation expresses that any learnable sequence must be isolated from another learnable sequence, and hence some collections of sequences cannot be learned by a single Transformer. Continuity entails that an attractor basin forms around a learned sequence, such that any sequence falling in that basin will collapse towards the learned sequence. These two phenomena can be mathematically proved for all Transformers that use compact positional encodings. We will also present some experiment results that illustrate the extent to which these theoretical limitations can be observed on the practical scale.


Following seminars: TBA

The list of next seminars can be found at:
https://bournez.gitlabpages.inria.fr/seminar-algorithms-plateau-Saclay/seminar/

The calendar of seminars can be found at:
https://bournez.gitlabpages.inria.fr/seminar-algorithms-plateau-Saclay/seminar.ics