Norm-based Generalization Bounds for Sparse Neural Networks
Document Type
Conference Proceeding
Publication Date
1-1-2023
Abstract
In this paper, we derive norm-based generalization bounds for sparse ReLU neural networks, including convolutional neural networks. These bounds differ from previous ones because they consider the sparse structure of the neural network architecture and the norms of the convolutional filters, rather than the norms of the (Toeplitz) matrices associated with the convolutional layers. Theoretically, we demonstrate that these bounds are significantly tighter than standard norm-based generalization bounds. Empirically, they offer relatively tight estimations of generalization for various simple classification problems. Collectively, these findings suggest that the sparsity of the underlying target function and the model's architecture plays a crucial role in the success of deep learning.
Identifier
85191182630 (Scopus)
ISBN
[9781713899921]
Publication Title
Advances in Neural Information Processing Systems
ISSN
10495258
Volume
36
Grant
CCF-1231216
Fund Ref
Center for Brains, Minds, and Machines, Massachusetts Institute of Technology
Recommended Citation
Galanti, Tomer; Galanti, Liane; Xu, Mengjia; and Poggio, Tomaso, "Norm-based Generalization Bounds for Sparse Neural Networks" (2023). Faculty Publications. 2301.
https://digitalcommons.njit.edu/fac_pubs/2301
