Abstract
Gradient-based backpropagation is the driving algorithm behind modern deep learning, but it is not biologically-plausible in its vanilla form due to problematic assumptions like non-local updates or weight symmetry. In response, feedback alignment has emerged as a promising framework for biologically plausible gradient-like learning. In the current literature, however, the feedback matrix has either been held fixed or updated by a hand-designed rule. It is not understood what form of local feedback plasticity rule is optimal, especially when biological constraints like sparsity are enforced upon the feedback matrix. In this work, we meta-learn the local feedback plasticity rule in a direct feedback alignment network across three structured feedback parametrisations: Mastrogiuseppe—Ostojic, sparse low-rank, and sparse low-rank with enforced sparsity. We show that the discovered rule depends on the structure of the feedback matrix, with a Hebbian rule on dense matrices and an error-modulated rule on sparsity-constrained matrices. Meanwhile, enforcing sparsity of the feedback matrix only costs less than accuracy on MNIST, suggesting that with the correct synaptic plasticity rule, the brain could use fewer connections while retaining performance.
Our findings predict that the optimal plasticity rule for feedback alignment will depend on the structure of the feedback connectivity. For example, in cortical regions where feedback is sparse, the optimal plasticity rule will more likely be error-modulated rather than purely Hebbian. In other regions where feedback is dense, the optimal plasticity rule may instead be Hebbian-dominant.