Hyperspherically Regularized Networks for Self-Supervision

Aiden Durrant* (Corresponding Author), Georgios Leontidis

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

1 Citation (Scopus)

Abstract

Bootstrap Your Own Latent (BYOL) introduced an approach to self-supervised learning avoiding the contrastive paradigm and subsequently removing the com- putational burden of negative sampling associated with such methods. However, we empirically find that the image representations produced under the BYOL’s self-distillation paradigm are poorly distributed in representation space compared to contrastive methods. This work empirically demonstrates that feature diver- sity enforced by contrastive losses is beneficial to image representation uniformity when employed in BYOL, and as such, provides greater inter-class representa- tion separability. Additionally, we explore and advocate the use of regularization methods, specifically the layer-wise minimization of hyperspherical energy (i.e. maximization of entropy) of network weights to encourage representation unifor- mity. We show that directly optimizing a measure of uniformity alongside the standard loss, or regularizing the networks of the BYOL architecture to minimize the hyperspherical energy of neurons can produce more uniformly distributed and therefore better performing representations for downstream tasks.
Original languageEnglish
Article number104494
JournalImage and Vision Computing
Volume124
DOIs
Publication statusPublished - 1 Jul 2022

Keywords

  • Self-supervised learning
  • Representation learning
  • Representation separability
  • Image classification

Fingerprint

Dive into the research topics of 'Hyperspherically Regularized Networks for Self-Supervision'. Together they form a unique fingerprint.

Cite this