The Experts below are selected from a list of 32223 Experts worldwide ranked by ideXlab platform

Jasha Droppo - One of the best experts on this subject based on the ideXlab platform.

  • on parallelizability of stochastic gradient descent for speech dnns
    International Conference on Acoustics Speech and Signal Processing, 2014
    Co-Authors: Frank Seide, Jasha Droppo
    Abstract:

    This paper compares the theoretical efficiency of Model-parallel and data-parallel distributed stochastic gradient descent training of DNNs. For a typical Switchboard DNN with 46M parameters, the results are not pretty: With modern GPUs and interconnects, Model Parallelism is optimal with only 3 GPUs in a single server, while data Parallelism with a minibatch size of 1024 does not even scale to 2 GPUs. We further show that data-parallel training efficiency can be improved by increasing the minibatch size (through a combination of AdaGrad and automatic adjustments of learning rate and minibatch size) and data compression. We arrive at an estimated possible end-to-end speed-up of 5 times or more. We do not address issues of robustness to process failure or other issues that might occur during training, nor of speed of convergence differences between ASGD and SGD parameter update patterns.

  • ICASSP - On parallelizability of stochastic gradient descent for speech DNNS
    2014 IEEE International Conference on Acoustics Speech and Signal Processing (ICASSP), 2014
    Co-Authors: Frank Seide, Jasha Droppo
    Abstract:

    This paper compares the theoretical efficiency of Model-parallel and data-parallel distributed stochastic gradient descent training of DNNs. For a typical Switchboard DNN with 46M parameters, the results are not pretty: With modern GPUs and interconnects, Model Parallelism is optimal with only 3 GPUs in a single server, while data Parallelism with a minibatch size of 1024 does not even scale to 2 GPUs. We further show that data-parallel training efficiency can be improved by increasing the minibatch size (through a combination of AdaGrad and automatic adjustments of learning rate and minibatch size) and data compression. We arrive at an estimated possible end-to-end speed-up of 5 times or more. We do not address issues of robustness to process failure or other issues that might occur during training, nor of speed of convergence differences between ASGD and SGD parameter update patterns.

Eric P Xing - One of the best experts on this subject based on the ideXlab platform.

  • strads a distributed framework for scheduled Model parallel machine learning
    European Conference on Computer Systems, 2016
    Co-Authors: Jin-kyu Kim, Seunghak Lee, Xu Zheng, Wei Dai, Garth A Gibso, Eric P Xing
    Abstract:

    Machine learning (ML) algorithms are commonly applied to big data, using distributed systems that partition the data across machines and allow each machine to read and update all ML Model parameters --- a strategy known as data Parallelism. An alternative and complimentary strategy, Model Parallelism, partitions the Model parameters for non-shared parallel access and updates, and may periodically repartition the parameters to facilitate communication. Model Parallelism is motivated by two challenges that data-Parallelism does not usually address: (1) parameters may be dependent, thus naive concurrent updates can introduce errors that slow convergence or even cause algorithm failure; (2) Model parameters converge at different rates, thus a small subset of parameters can bottleneck ML algorithm completion. We propose scheduled Model Parallelism (SchMP), a programming approach that improves ML algorithm convergence speed by efficiently scheduling parameter updates, taking into account parameter dependencies and uneven convergence. To support SchMP at scale, we develop a distributed framework STRADS which optimizes the throughput of SchMP programs, and benchmark four common ML applications written as SchMP programs: LDA topic Modeling, matrix factorization, sparse least-squares (Lasso) regression and sparse logistic regression. By improving ML progress per iteration through SchMP programming whilst improving iteration throughput through STRADS we show that SchMP programs running on STRADS outperform non-Model-parallel ML implementations: for example, SchMP LDA and SchMP Lasso respectively achieve 10x and 5x faster convergence than recent, well-established baselines.

  • EuroSys - STRADS: a distributed framework for scheduled Model parallel machine learning
    Proceedings of the Eleventh European Conference on Computer Systems, 2016
    Co-Authors: Jin-kyu Kim, Xun Zheng, Garth A. Gibson, Seunghak Lee, Wei Dai, Eric P Xing
    Abstract:

    Machine learning (ML) algorithms are commonly applied to big data, using distributed systems that partition the data across machines and allow each machine to read and update all ML Model parameters --- a strategy known as data Parallelism. An alternative and complimentary strategy, Model Parallelism, partitions the Model parameters for non-shared parallel access and updates, and may periodically repartition the parameters to facilitate communication. Model Parallelism is motivated by two challenges that data-Parallelism does not usually address: (1) parameters may be dependent, thus naive concurrent updates can introduce errors that slow convergence or even cause algorithm failure; (2) Model parameters converge at different rates, thus a small subset of parameters can bottleneck ML algorithm completion. We propose scheduled Model Parallelism (SchMP), a programming approach that improves ML algorithm convergence speed by efficiently scheduling parameter updates, taking into account parameter dependencies and uneven convergence. To support SchMP at scale, we develop a distributed framework STRADS which optimizes the throughput of SchMP programs, and benchmark four common ML applications written as SchMP programs: LDA topic Modeling, matrix factorization, sparse least-squares (Lasso) regression and sparse logistic regression. By improving ML progress per iteration through SchMP programming whilst improving iteration throughput through STRADS we show that SchMP programs running on STRADS outperform non-Model-parallel ML implementations: for example, SchMP LDA and SchMP Lasso respectively achieve 10x and 5x faster convergence than recent, well-established baselines.

  • Model-Parallel Inference for Big Topic Models
    arXiv: Distributed Parallel and Cluster Computing, 2014
    Co-Authors: Xun Zheng, Jin-kyu Kim, Eric P Xing
    Abstract:

    In real world industrial applications of topic Modeling, the ability to capture gigantic conceptual space by learning an ultra-high dimensional topical representation, i.e., the so-called "big Model", is becoming the next desideratum after enthusiasms on "big data", especially for fine-grained downstream tasks such as online advertising, where good performances are usually achieved by regression-based predictors built on millions if not billions of input features. The conventional data-parallel approach for training gigantic topic Models turns out to be rather inefficient in utilizing the power of Parallelism, due to the heavy dependency on a centralized image of "Model". Big Model size also poses another challenge on the storage, where available Model size is bounded by the smallest RAM of nodes. To address these issues, we explore another type of Parallelism, namely Model-Parallelism, which enables training of disjoint blocks of a big topic Model in parallel. By integrating data-Parallelism with Model-Parallelism, we show that dependencies between distributed elements can be handled seamlessly, achieving not only faster convergence but also an ability to tackle significantly bigger Model size. We describe an architecture for Model-parallel inference of LDA, and present a variant of collapsed Gibbs sampling algorithm tailored for it. Experimental results demonstrate the ability of this system to handle topic Modeling with unprecedented amount of 200 billion Model variables only on a low-end cluster with very limited computational resources and bandwidth.

Yolande Berbers - One of the best experts on this subject based on the ideXlab platform.

Frank Seide - One of the best experts on this subject based on the ideXlab platform.

  • on parallelizability of stochastic gradient descent for speech dnns
    International Conference on Acoustics Speech and Signal Processing, 2014
    Co-Authors: Frank Seide, Jasha Droppo
    Abstract:

    This paper compares the theoretical efficiency of Model-parallel and data-parallel distributed stochastic gradient descent training of DNNs. For a typical Switchboard DNN with 46M parameters, the results are not pretty: With modern GPUs and interconnects, Model Parallelism is optimal with only 3 GPUs in a single server, while data Parallelism with a minibatch size of 1024 does not even scale to 2 GPUs. We further show that data-parallel training efficiency can be improved by increasing the minibatch size (through a combination of AdaGrad and automatic adjustments of learning rate and minibatch size) and data compression. We arrive at an estimated possible end-to-end speed-up of 5 times or more. We do not address issues of robustness to process failure or other issues that might occur during training, nor of speed of convergence differences between ASGD and SGD parameter update patterns.

  • ICASSP - On parallelizability of stochastic gradient descent for speech DNNS
    2014 IEEE International Conference on Acoustics Speech and Signal Processing (ICASSP), 2014
    Co-Authors: Frank Seide, Jasha Droppo
    Abstract:

    This paper compares the theoretical efficiency of Model-parallel and data-parallel distributed stochastic gradient descent training of DNNs. For a typical Switchboard DNN with 46M parameters, the results are not pretty: With modern GPUs and interconnects, Model Parallelism is optimal with only 3 GPUs in a single server, while data Parallelism with a minibatch size of 1024 does not even scale to 2 GPUs. We further show that data-parallel training efficiency can be improved by increasing the minibatch size (through a combination of AdaGrad and automatic adjustments of learning rate and minibatch size) and data compression. We arrive at an estimated possible end-to-end speed-up of 5 times or more. We do not address issues of robustness to process failure or other issues that might occur during training, nor of speed of convergence differences between ASGD and SGD parameter update patterns.

Jin-kyu Kim - One of the best experts on this subject based on the ideXlab platform.

  • EuroSys - STRADS: a distributed framework for scheduled Model parallel machine learning
    Proceedings of the Eleventh European Conference on Computer Systems, 2016
    Co-Authors: Jin-kyu Kim, Xun Zheng, Garth A. Gibson, Seunghak Lee, Wei Dai, Eric P Xing
    Abstract:

    Machine learning (ML) algorithms are commonly applied to big data, using distributed systems that partition the data across machines and allow each machine to read and update all ML Model parameters --- a strategy known as data Parallelism. An alternative and complimentary strategy, Model Parallelism, partitions the Model parameters for non-shared parallel access and updates, and may periodically repartition the parameters to facilitate communication. Model Parallelism is motivated by two challenges that data-Parallelism does not usually address: (1) parameters may be dependent, thus naive concurrent updates can introduce errors that slow convergence or even cause algorithm failure; (2) Model parameters converge at different rates, thus a small subset of parameters can bottleneck ML algorithm completion. We propose scheduled Model Parallelism (SchMP), a programming approach that improves ML algorithm convergence speed by efficiently scheduling parameter updates, taking into account parameter dependencies and uneven convergence. To support SchMP at scale, we develop a distributed framework STRADS which optimizes the throughput of SchMP programs, and benchmark four common ML applications written as SchMP programs: LDA topic Modeling, matrix factorization, sparse least-squares (Lasso) regression and sparse logistic regression. By improving ML progress per iteration through SchMP programming whilst improving iteration throughput through STRADS we show that SchMP programs running on STRADS outperform non-Model-parallel ML implementations: for example, SchMP LDA and SchMP Lasso respectively achieve 10x and 5x faster convergence than recent, well-established baselines.

  • strads a distributed framework for scheduled Model parallel machine learning
    European Conference on Computer Systems, 2016
    Co-Authors: Jin-kyu Kim, Seunghak Lee, Xu Zheng, Wei Dai, Garth A Gibso, Eric P Xing
    Abstract:

    Machine learning (ML) algorithms are commonly applied to big data, using distributed systems that partition the data across machines and allow each machine to read and update all ML Model parameters --- a strategy known as data Parallelism. An alternative and complimentary strategy, Model Parallelism, partitions the Model parameters for non-shared parallel access and updates, and may periodically repartition the parameters to facilitate communication. Model Parallelism is motivated by two challenges that data-Parallelism does not usually address: (1) parameters may be dependent, thus naive concurrent updates can introduce errors that slow convergence or even cause algorithm failure; (2) Model parameters converge at different rates, thus a small subset of parameters can bottleneck ML algorithm completion. We propose scheduled Model Parallelism (SchMP), a programming approach that improves ML algorithm convergence speed by efficiently scheduling parameter updates, taking into account parameter dependencies and uneven convergence. To support SchMP at scale, we develop a distributed framework STRADS which optimizes the throughput of SchMP programs, and benchmark four common ML applications written as SchMP programs: LDA topic Modeling, matrix factorization, sparse least-squares (Lasso) regression and sparse logistic regression. By improving ML progress per iteration through SchMP programming whilst improving iteration throughput through STRADS we show that SchMP programs running on STRADS outperform non-Model-parallel ML implementations: for example, SchMP LDA and SchMP Lasso respectively achieve 10x and 5x faster convergence than recent, well-established baselines.

  • Model-Parallel Inference for Big Topic Models
    arXiv: Distributed Parallel and Cluster Computing, 2014
    Co-Authors: Xun Zheng, Jin-kyu Kim, Eric P Xing
    Abstract:

    In real world industrial applications of topic Modeling, the ability to capture gigantic conceptual space by learning an ultra-high dimensional topical representation, i.e., the so-called "big Model", is becoming the next desideratum after enthusiasms on "big data", especially for fine-grained downstream tasks such as online advertising, where good performances are usually achieved by regression-based predictors built on millions if not billions of input features. The conventional data-parallel approach for training gigantic topic Models turns out to be rather inefficient in utilizing the power of Parallelism, due to the heavy dependency on a centralized image of "Model". Big Model size also poses another challenge on the storage, where available Model size is bounded by the smallest RAM of nodes. To address these issues, we explore another type of Parallelism, namely Model-Parallelism, which enables training of disjoint blocks of a big topic Model in parallel. By integrating data-Parallelism with Model-Parallelism, we show that dependencies between distributed elements can be handled seamlessly, achieving not only faster convergence but also an ability to tackle significantly bigger Model size. We describe an architecture for Model-parallel inference of LDA, and present a variant of collapsed Gibbs sampling algorithm tailored for it. Experimental results demonstrate the ability of this system to handle topic Modeling with unprecedented amount of 200 billion Model variables only on a low-end cluster with very limited computational resources and bandwidth.

  • Primitives for Dynamic Big Model Parallelism
    arXiv, 2014
    Co-Authors: Seunghak Lee, Jin-kyu Kim, Qirong Ho, Xun Zheng, Garth A. Gibson, Eric Po Xing
    Abstract:

    When training large machine learning Models with many variables or parameters, a single machine is often inadequate since the Model may be too large to fit in memory, while training can take a long time even with stochastic updates. A natural recourse is to turn to distributed cluster computing, in order to harness additional memory and processors. However, naive, unstructured parallelization of ML algorithms can make inefficient use of distributed memory, while failing to obtain proportional convergence speedups — or can even result in divergence. We develop a framework of primitives for dynamic Model-Parallelism, STRADS, in order to explore partitioning and update scheduling of Model variables in distributed ML algorithms — thus improving their memory efficiency while presenting new opportunities to speed up convergence without compromising inference correctness. We demonstrate the efficacy of Model-parallel algorithms implemented in STRADS versus popular implementations for Topic Modeling, Matrix Factorization and Lasso.