Mathematical Foundations of Equivariant Neural Networks
Doctoral thesis, 2023
Deep learning has revolutionized industry and academic research. Over the past decade, neural networks have been used to solve a multitude of previously unsolved problems and to significantly improve the state of the art on other tasks. However, training a neural network typically requires large amounts of data and computational resources. This is not only costly, it also prevents deep learning from being used for applications in which data is scarce. It is therefore important to simplify the learning task by incorporating inductive biases - prior knowledge and assumptions - into the neural network design.
Geometric deep learning aims to reduce the amount of information that neural networks have to learn, by taking advantage of geometric properties in data. In particular, equivariant neural networks use symmetries to reduce the complexity of a learning task. Symmetries are properties that do not change under certain transformations. For example, rotation-equivariant neural networks trained to identify tumors in medical images are not sensitive to the orientation of a tumor within an image. Another example is graph neural networks, i.e., permutation-equivariant neural networks that operate on graphs, such as molecules or social networks. Permuting the ordering of vertices and edges either transforms the output of a graph neural network in a predictable way (equivariance), or has no effect on the output (invariance).
In this thesis we study a fiber bundle theoretic framework for equivariant neural networks. Fiber bundles are often used in mathematics and theoretical physics to model nontrivial geometries, and offer a geometric approach to symmetry. This framework connects to many different areas of mathematics, including Fourier analysis, representation theory, and gauge theory, thus providing a large set of tools for analyzing equivariant neural networks.
Geometric deep learning aims to reduce the amount of information that neural networks have to learn, by taking advantage of geometric properties in data. In particular, equivariant neural networks use symmetries to reduce the complexity of a learning task. Symmetries are properties that do not change under certain transformations. For example, rotation-equivariant neural networks trained to identify tumors in medical images are not sensitive to the orientation of a tumor within an image. Another example is graph neural networks, i.e., permutation-equivariant neural networks that operate on graphs, such as molecules or social networks. Permuting the ordering of vertices and edges either transforms the output of a graph neural network in a predictable way (equivariance), or has no effect on the output (invariance).
In this thesis we study a fiber bundle theoretic framework for equivariant neural networks. Fiber bundles are often used in mathematics and theoretical physics to model nontrivial geometries, and offer a geometric approach to symmetry. This framework connects to many different areas of mathematics, including Fourier analysis, representation theory, and gauge theory, thus providing a large set of tools for analyzing equivariant neural networks.
symmetry
geometric deep learning
gauge theory
induced representations
fiber bundles
convolutional neural networks
equivariance
Pascal, Chalmers tvärgata 3
Opponent: Prof. Erik Bekkers, Amsterdam Machine Learning Lab (AMLab), University of Amsterdam, the Netherlands
Author
Jimmy Aronsson
Chalmers, Mathematical Sciences, Algebra and geometry
Geometric deep learning is an umbrella term for machine learning methods that incorporate geometric notions such as curvature or symmetry. Convolutional neural networks (CNNs), for example, use convolutional layers that commute with translations in the square lattice Z². This property is called translation equivariance and is a useful inductive bias in machine learning tasks such as image classification, in which Z² represents the pixel lattice. Other equivariant neural networks commute with (local or global) symmetry transformations that depend on the task at hand: Spherical CNNs use spherical convolutions to achieve rotation equivariance, e.g., for semantic segmentation of spherical image data; permutation equivariance makes graph neural networks robust to relabeling of vertices in graphs (e.g., molecular data). In this thesis we analyze a fiber bundle theoretic model for equivariant neural networks. In particular, we investigate the precise relation between equivariance and convolutions.
Subject Categories
Mathematics
ISBN
978-91-7905-854-8
Doktorsavhandlingar vid Chalmers tekniska högskola. Ny serie: 5320
Publisher
Chalmers
Pascal, Chalmers tvärgata 3
Opponent: Prof. Erik Bekkers, Amsterdam Machine Learning Lab (AMLab), University of Amsterdam, the Netherlands