The Impact of Synchronization in Parallel Stochastic Gradient Descent
Paper in proceeding, 2022
In this paper, we discuss our and related work in the domain of efficient parallel optimization, using Stochastic Gradient Descent, for fast and stable convergence in prominent machine learning applications. We outline the results in the context of aspects and challenges regarding synchronization, consistency, staleness and parallel-aware adaptiveness, focusing on the impact on the overall convergence.
Stochastic gradient descent
Machine Learning
Lock-free
Author
Karl Bäckström
Chalmers, Computer Science and Engineering (Chalmers), Computer and Network Systems
Marina Papatriantafilou
Chalmers, Computer Science and Engineering (Chalmers), Computer and Network Systems
Philippas Tsigas
Chalmers, Computer Science and Engineering (Chalmers), Computer and Network Systems
Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
03029743 (ISSN) 16113349 (eISSN)
Vol. 13145 LNCS 60-759783030948757 (ISBN)
18th International Conference on Distributed Computing and Intelligent Technology, ICDCIT 2022
Bhubaneswar, India,
Bhubaneswar, India,
WASP SAS: Structuring data for continuous processing and ML systems
Wallenberg AI, Autonomous Systems and Software Program, 2018-01-01 -- 2023-01-01.
Areas of Advance
Information and Communication Technology
Subject Categories (SSIF 2011)
Computational Mathematics
Computer Science
Computer Systems
DOI
10.1007/978-3-030-94876-4_4