A Newton-type algorithm for federated learning based on incremental Hessian eigenvector sharing
Date and time: 14 March 2023, 13:00 – 14:00 CET
Speaker: Subhrakanti Dey, University of Uppsala
Title: A Newton-type algorithm for federated learning based on incremental Hessian eigenvector sharing
Where: Digital Futures hub, Osquars Backe 5, floor 2 at KTH main campus OR Zoom
Directions: https://www.digitalfutures.kth.se/contact/how-to-get-here/
OR
Zoom: https://kth-se.zoom.us/j/69560887455
Meeting ID: 695 6088 7455
Password: 755440
Moderator/administrator: Francisco J. Peña, frape@kth.se
Abstract: There is a growing interest in the distributed optimization framework that goes under the name of Federated Learning (FL). In particular, much attention is being turned to FL scenarios where the network is strongly heterogeneous in terms of communication resources (e.g., bandwidth) and data distribution. In these cases, communication between local machines (agents) and the central server (Master) is a main consideration.
In this work, we present SHED, an original communication-constrained Newton-type (NT) algorithm designed to accelerate FL in such heterogeneous scenarios. SHED is by design robust to non i.i.d. data distributions, handles heterogeneity of agents’ communication resources (CRs), only requires sporadic Hessian computations, and achieves super-linear convergence. This is possible thanks to an incremental strategy based on eigendecomposition of the local Hessian matrices, which exploits (possibly) outdated second-order information. The proposed solution is thoroughly validated on real datasets by assessing (i) the number of communication rounds required for convergence, (ii) the overall amount of data transmitted and (iii) the number of local Hessian computations. For all these metrics, the proposed approach shows superior performance against state-of-the art techniques like GIANT and FedNL.
Bio: Subhrakanti Dey received the Bachelor in Technology and Master in Technology degrees from the Department of Electronics and Electrical Communication Engineering, Indian Institute of Technology, Kharagpur, in 1991 and 1993, respectively, and the Ph.D. degree from the Department of Systems Engineering, Research School of Information Sciences and Engineering, Australian National University, Canberra, in 1996.
He is currently the Head of Division of Signals and Systems and Professor of Signal Processing at Uppsala University. Prior to this, he was with the National University of Ireland, Maynooth, Ireland, during 2018-2022, while being on leave of absence from Uppsala. He was also a Professor with the Department of Electrical and Electronic Engineering, University of Melbourne, Parkville, Australia, from 2000 until 2013. From 1995 to 2000, he held a number of postdoctoral positions with the Department of Systems Engineering, Australian National University, and the Institute for Systems Research, University of Maryland, College Park.
His current research interests include wireless communications and networks, signal processing for sensor networks, networked control systems, and distributed optimization and learning over networks.
Professor Dey currently serves on the Editorial Board of IEEE Control Systems Letters (Senior Editor), IEEE Transactions on Control of Network Systems, (Senior Editor) and IEEE Transactions on Wireless Communications (Editor), and Automatica (Assoc. Editor). He was also an Associate Editor for IEEE Transactions on Signal Processing, (2007-2010, 2014-2018), IEEE Transactions on Automatic Control, (2004-2007) and Elsevier Systems and Control Letters (2003-2013).
Link to the profile of Subhrakanti Dey: http://www.signal.uu.se/Staff/sd/sd.html