Speaker: Wyatt Mackey, ARL

Title: Rethinking projection networks: a functional approach to self-supervised learning.

Abstract. Self-supervised learning (SSL), or learning without any human-provided labels, has achieved a surge of interest in the machine learning community as it begins to significantly outpace traditional training techniques in diverse tasks from image classification and automatic segmentation to high level semantic scene decomposition. However, SSL remains susceptible to catastrophic forgetting in large tasks like video understanding, and depends on a plethora of highly tuned parameters and auxiliary trained networks to converge to strong local minima. We investigate and reinterpret projection networks, an auxiliary network that helps prevent dimension collapse in SSL. By re-examining the ultimate goals of SSL and transfer learning from the lens of linear dualities, we introduce multiple new methods to streamline and improve gradient conditioning in all SSL frameworks. We demonstrate strong results on key benchmarks like image classification with small models using these innovations, suggesting a new and promising path forward to scaling large SSL systems.