Anirvan Sengupta – Rutgers University
Date/Time/Location
Thursday, September 24, 2026, 12:10 pm; Hill Center 705
In-context Denoising, Attention and Diffusion
Modern neural networks can have enough parameters to memorize their training data, yet often generalize remarkably well. I will discuss this tension from a statistical-physics perspective, focusing on attention mechanisms and their relation to associative memory. I will first consider an in-context denoising problem for a one-layer transformer. For several simple data models, the Bayes-optimal denoiser can be derived explicitly and represented by linear or softmax attention, providing a direct connection to modern Hopfield models. In this picture, generalization is associated with a crossover of the Hopfield energy landscape from isolated stored patterns to a representation of the underlying data distribution. I will then discuss multilayer attention-only dynamics, where all samples evolve together. In an appropriate continuum limit, the resulting self-attractive dynamics can be related to anti-diffusion, connecting attention to score-based denoising and empirical Bayes. This suggests a unified view of associative memory, attention, and diffusion-based generative modeling.