Improved Information Theoretic Generalization Bounds for Distributed and Federated Learning

Document Type

Conference Proceeding

Publication Date

1-1-2022

Abstract

We consider information-theoretic bounds on expected generalization error for statistical learning problems in a networked setting. In this setting, there are K nodes, each with its own independent dataset, and the models from each node have to be aggregated into a final centralized model. We consider both simple averaging of the models as well as more complicated multi-round algorithms. We give upper bounds on the expected generalization error for a variety of problems, such as those with Bregman divergence or Lipschitz continuous losses, that demonstrate an improved dependence of 1/K on the number of nodes. These "per node"bounds are in terms of the mutual information between the training dataset and the trained weights at each node, and are therefore useful in describing the generalization properties inherent to having communication or privacy constraints at each node.

Identifier

85132274432 (Scopus)

ISBN

[9781665421591]

Publication Title

IEEE International Symposium on Information Theory Proceedings

External Full Text Location

https://doi.org/10.1109/ISIT50566.2022.9834700

ISSN

21578095

First Page

1465

Last Page

1470

Volume

2022-June

Grant

CCF-1908308

Fund Ref

National Science Foundation

This document is currently not available here.

Share

COinS