With the paper "Feed-Forward Optimization With Delayed Feedback for Neural Network Training," a team of SCC scientists under the supervision of Prof. Dr. Achim Streit won the Best Paper Award at the International Conference on Neural Information Processing 2024.
The paper introduces a novel method for training neural networks in a more biologically plausible and energy-efficient way by using approximate gradients. Typically, neural networks are trained using gradient descent, where the network parameters are iteratively updated by stepping into the direction of the negative gradient. The most common approach for computing such gradients is backpropagation, which computes the gradient in a backward pass. This backward pass, however, is biologically implausible, computationally expensive, and hinders parallelization. By using additional random feedback connections and delayed error information, SCC scientists introduce a new method to approximate the gradients using only forward passes. This not only solves two core issues of the biological implausibility but can also reduce the energy consumption per epoch by up to 25% and enables new possibilities for parallelization.
This work is funded by the Helmholtz AI platform and HAICORE∂KIT grants.
Contact: Dr. Markus Götz
Preprint: https://arxiv.org/abs/2304.13372
Katharina Flügel (SCC)