Adaptiveness and Lock-free Synchronization in Parallel Stochastic Gradient Descent
Licentiate thesis, 2021
In order to cope with increasing data volumes, and to facilitate accelerated processing utilizing contemporary hardware, various parallel SGD variants have been proposed. In addition to traditional synchronous parallelization schemes, asynchronous ones have received particular interest in recent literature due to their improved ability to scale due to less coordination, and subsequently waiting time. However, asynchrony implies inherent challenges in understanding the execution of the algorithm and its convergence properties, due the presence of both stale and inconsistent views of the shared state.
In this work, we aim to increase the understanding of the convergence properties of SGD for practical applications under asynchronous parallelism and develop tools and frameworks that facilitate improved convergence properties as well as further research and development. First, we focus on understanding the impact of staleness, and introduce models for capturing the dynamics of parallel execution of SGD. This enables (i) quantifying the statistical penalty on the convergence due to staleness and (ii) deriving an adaptation scheme, introducing a staleness-adaptive SGD variant MindTheStep-AsyncSGD, which provably reduces this penalty. Second, we aim at exploring the impact of synchronization mechanisms, in particular consistency-preserving ones, and the overall effect on the convergence properties. To this end, we propose LeashedSGD, an extensible algorithmic framework supporting various synchronization mechanisms for different degrees of consistency, enabling in particular a lock-free and consistency-preserving implementation. In addition, the algorithmic construction of Leashed-SGD enables dynamic memory allocation, claiming memory only when necessary, which reduces the overall memory footprint. We perform an extensive empirical study, benchmarking the proposed methods, together with established baselines, focusing on the prominent application of Deep Learning for image classification on the benchmark datasets MNIST and CIFAR, showing significant improvements in converge time for Leashed-SGD and MindTheStep-AsyncSGD.
parallelism
machine learning
Stochastic gradient descent
Author
Karl Bäckström
Chalmers, Computer Science and Engineering (Chalmers), Networks and Systems (Chalmers)
MindTheStep-AsyncPSGD: Adaptive Asynchronous Parallel Stochastic Gradient Descent
IEEE International Conference on Big Data,;(2019)p. 16-25
Paper in proceeding
Consistent lock-free parallel stochastic gradient descent for fast and stable convergence
Proceedings - 2021 IEEE 35th International Parallel and Distributed Processing Symposium, IPDPS 2021,;(2021)p. 423-432
Paper in proceeding
WASP SAS: Structuring data for continuous processing and ML systems
Wallenberg AI, Autonomous Systems and Software Program, 2018-01-01 -- 2023-01-01.
Subject Categories
Computer Engineering
Computer Science
Computer Systems
Areas of Advance
Information and Communication Technology
Driving Forces
Sustainable development
Innovation and entrepreneurship
Roots
Basic sciences
Publisher
Chalmers
Opponent: Hans Vandierendonck