The Experts below are selected from a list of 3498 Experts worldwide ranked by ideXlab platform
Brian Mcwilliams - One of the best experts on this subject based on the ideXlab platform.
-
variance reduced stochastic gradient descent with neighbors
Neural Information Processing Systems, 2015Co-Authors: Thomas Hofmann, Aurelien Lucchi, Simon Lacostejulien, Brian McwilliamsAbstract:Stochastic Gradient Descent (SGD) is a workhorse in machine learning, yet its slow convergence can be a Computational Bottleneck. Variance reduction techniques such as SAG, SVRG and SAGA have been proposed to overcome this weakness, achieving linear convergence. However, these methods are either based on computations of full gradients at pivot points, or on keeping per data point corrections in memory. Therefore speed-ups relative to SGD may need a minimal number of epochs in order to materialize. This paper investigates algorithms that can exploit neighborhood structure in the training data to share and re-use information about past stochastic gradients across data points, which offers advantages in the transient optimization phase. As a side-product we provide a unified convergence analysis for a family of variance reduction algorithms, which we call memorization algorithms. We provide experimental results supporting our theory.
S.f. Hsieh - One of the best experts on this subject based on the ideXlab platform.
-
A unified square-root-free approach for QRD-based recursive-least-squares estimation
IEEE Transactions on Signal Processing, 1993Co-Authors: S.f. HsiehAbstract:The generic formula for the Givens rotation, the most commonly used method for QR-decomposition (QRD) updating, requires explicit square-root (sqrt) computations, which constitute a Computational Bottleneck. A unified systematic approach for sqrt-free Givens rotations is provided. By properly choosing two parameters, all previously known, as well as new, sqrt-free methods are included in the unified approach. This unified treatment is also extended to the QRD-based recursive-least-squares (RLS) problem for optimum residual acquisition without sqrt operations.
Thomas Hofmann - One of the best experts on this subject based on the ideXlab platform.
-
variance reduced stochastic gradient descent with neighbors
Neural Information Processing Systems, 2015Co-Authors: Thomas Hofmann, Aurelien Lucchi, Simon Lacostejulien, Brian McwilliamsAbstract:Stochastic Gradient Descent (SGD) is a workhorse in machine learning, yet its slow convergence can be a Computational Bottleneck. Variance reduction techniques such as SAG, SVRG and SAGA have been proposed to overcome this weakness, achieving linear convergence. However, these methods are either based on computations of full gradients at pivot points, or on keeping per data point corrections in memory. Therefore speed-ups relative to SGD may need a minimal number of epochs in order to materialize. This paper investigates algorithms that can exploit neighborhood structure in the training data to share and re-use information about past stochastic gradients across data points, which offers advantages in the transient optimization phase. As a side-product we provide a unified convergence analysis for a family of variance reduction algorithms, which we call memorization algorithms. We provide experimental results supporting our theory.
Pascal Frossard - One of the best experts on this subject based on the ideXlab platform.
-
Stochastic Gradient Descent for Spectral Embedding with Implicit Orthogonality Constraint
ICASSP 2019 - 2019 IEEE International Conference on Acoustics Speech and Signal Processing (ICASSP), 2019Co-Authors: Mireille El Gheche, Giovanni Chierchia, Pascal FrossardAbstract:In this paper, we propose a scalable algorithm for spectral embedding. The latter is a standard tool for graph clustering. However, its Computational Bottleneck is the eigendecomposition of the graph Laplacian matrix, which prevents its application to large-scale graphs. Our contribution consists of reformulating spectral embedding so that it can be solved via stochastic optimization. The idea is to replace the orthogonality constraint with an orthogonalization matrix injected directly into the criterion. As the gradient can be computed through a Cholesky factorization, our reformulation allows us to develop an efficient algorithm based on mini-batch gradient descent. Experimental results, both on synthetic and real data, confirm the efficiency of the proposed method in term of execution speed with respect to similar existing techniques.
Simon Lacostejulien - One of the best experts on this subject based on the ideXlab platform.
-
variance reduced stochastic gradient descent with neighbors
Neural Information Processing Systems, 2015Co-Authors: Thomas Hofmann, Aurelien Lucchi, Simon Lacostejulien, Brian McwilliamsAbstract:Stochastic Gradient Descent (SGD) is a workhorse in machine learning, yet its slow convergence can be a Computational Bottleneck. Variance reduction techniques such as SAG, SVRG and SAGA have been proposed to overcome this weakness, achieving linear convergence. However, these methods are either based on computations of full gradients at pivot points, or on keeping per data point corrections in memory. Therefore speed-ups relative to SGD may need a minimal number of epochs in order to materialize. This paper investigates algorithms that can exploit neighborhood structure in the training data to share and re-use information about past stochastic gradients across data points, which offers advantages in the transient optimization phase. As a side-product we provide a unified convergence analysis for a family of variance reduction algorithms, which we call memorization algorithms. We provide experimental results supporting our theory.