The Experts below are selected from a list of 77355 Experts worldwide ranked by ideXlab platform
Francis Bach - One of the best experts on this subject based on the ideXlab platform.
-
averaged least mean squares bias variance trade offs and optimal sampling distributions
International Conference on Artificial Intelligence and Statistics, 2015Co-Authors: Alexandre Defossez, Francis BachAbstract:We consider the least-squares regression problem and provide a detailed asymptotic analysis of the performance of averaged constant-step-size stochastic gradient descent. In the strongly-Convex Case, we provide an asymptotic expansion up to explicit exponentially decaying terms. Our analysis leads to new insights into stochastic approximation algorithms: (a) it gives a tighter bound on the allowed step-size; (b) the generalization error may be divided into a variance term which is decaying as O(1/n), independently of the step-size , and a bias term that decays as O(1/ 2 n 2 ); (c) when allowing non-uniform sampling of examples over a dataset, the choice of a good sampling density depends on the trade-off between bias and variance: when the variance term dominates, optimal sampling densities do not lead to much gain, while when the bias term dominates, we can choose larger step-sizes that lead to significant improvements.
-
constant step size least mean square bias variance trade offs and optimal sampling distributions
International Conference on Artificial Intelligence and Statistics, 2014Co-Authors: Alexandre Defossez, Francis BachAbstract:We consider the least-squares regression problem and provide a detailed asymptotic analysis of the performance of averaged constant-step-size stochastic gradient descent (a.k.a. least-mean-squares). In the strongly-Convex Case, we provide an asymptotic expansion up to explicit exponentially decaying terms. Our analysis leads to new insights into stochastic approximation algorithms: (a) it gives a tighter bound on the allowed step-size; (b) the generalization error may be divided into a variance term which is decaying as O(1/n), independently of the step-size γ, and a bias term that decays as O(1/γ 2 n 2); (c) when allowing non-uniform sampling, the choice of a good sampling density depends on whether the variance or bias terms dominate. In particular, when the variance term dominates, optimal sampling densities do not lead to much gain, while when the bias term dominates, we can choose larger step-sizes that leads to significant improvements.
-
non asymptotic analysis of stochastic approximation algorithms for machine learning
Neural Information Processing Systems, 2011Co-Authors: Eric Moulines, Francis BachAbstract:We consider the minimization of a Convex objective function defined on a Hilbert space, which is only available through unbiased estimates of its gradients. This problem includes standard machine learning algorithms such as kernel logistic regression and least-squares regression, and is commonly referred to as a stochastic approximation problem in the operations research community. We provide a non-asymptotic analysis of the convergence of two well-known algorithms, stochastic gradient descent (a.k.a. Robbins-Monro algorithm) as well as a simple modification where iterates are averaged (a.k.a. Polyak-Ruppert averaging). Our analysis suggests that a learning rate proportional to the inverse of the number of iterations, while leading to the optimal convergence rate in the strongly Convex Case, is not robust to the lack of strong Convexity or the setting of the proportionality constant. This situation is remedied when using slower decays together with averaging, robustly leading to the optimal rate of convergence. We illustrate our theoretical results with simulations on synthetic and standard datasets.
Alexandre Defossez - One of the best experts on this subject based on the ideXlab platform.
-
averaged least mean squares bias variance trade offs and optimal sampling distributions
International Conference on Artificial Intelligence and Statistics, 2015Co-Authors: Alexandre Defossez, Francis BachAbstract:We consider the least-squares regression problem and provide a detailed asymptotic analysis of the performance of averaged constant-step-size stochastic gradient descent. In the strongly-Convex Case, we provide an asymptotic expansion up to explicit exponentially decaying terms. Our analysis leads to new insights into stochastic approximation algorithms: (a) it gives a tighter bound on the allowed step-size; (b) the generalization error may be divided into a variance term which is decaying as O(1/n), independently of the step-size , and a bias term that decays as O(1/ 2 n 2 ); (c) when allowing non-uniform sampling of examples over a dataset, the choice of a good sampling density depends on the trade-off between bias and variance: when the variance term dominates, optimal sampling densities do not lead to much gain, while when the bias term dominates, we can choose larger step-sizes that lead to significant improvements.
-
constant step size least mean square bias variance trade offs and optimal sampling distributions
International Conference on Artificial Intelligence and Statistics, 2014Co-Authors: Alexandre Defossez, Francis BachAbstract:We consider the least-squares regression problem and provide a detailed asymptotic analysis of the performance of averaged constant-step-size stochastic gradient descent (a.k.a. least-mean-squares). In the strongly-Convex Case, we provide an asymptotic expansion up to explicit exponentially decaying terms. Our analysis leads to new insights into stochastic approximation algorithms: (a) it gives a tighter bound on the allowed step-size; (b) the generalization error may be divided into a variance term which is decaying as O(1/n), independently of the step-size γ, and a bias term that decays as O(1/γ 2 n 2); (c) when allowing non-uniform sampling, the choice of a good sampling density depends on whether the variance or bias terms dominate. In particular, when the variance term dominates, optimal sampling densities do not lead to much gain, while when the bias term dominates, we can choose larger step-sizes that leads to significant improvements.
Duanzhi Zhang - One of the best experts on this subject based on the ideXlab platform.
-
seifert conjecture in the even Convex Case
Communications on Pure and Applied Mathematics, 2014Co-Authors: Chungen Liu, Duanzhi ZhangAbstract:In this paper, we prove that there exist at least n geometrically distinct brake orbits on every C2 compact Convex symmetric hypersurface Σ in ℝ2n satisfying the reversible condition NΣ = Σ with N = diag(−In,In). As a consequence, we show that if the Hamiltonian function is Convex and even, then Seifert conjecture of 1948 on the multiplicity of brake orbits holds for any positive integern. © 2014 Wiley Periodicals, Inc.
-
seifert conjecture in the even Convex Case
arXiv: Dynamical Systems, 2013Co-Authors: Chungen Liu, Duanzhi ZhangAbstract:In this paper, we prove that there exist at least $n$ geometrically distinct brake orbits on every $C^2$ compact Convex symmetric hypersurface $\Sg$ in $\R^{2n}$ satisfying the reversible condition $N\Sg=\Sg$ with $N=\diag (-I_n,I_n)$. As a consequence, we show that if the Hamiltonian function is Convex and even, then Seifert conjecture of 1948 on the multiplicity of brake orbits holds for any positive integer $n$.
Praneeth Netrapalli - One of the best experts on this subject based on the ideXlab platform.
-
the step decay schedule a near optimal geometrically decaying learning rate procedure for least squares
arXiv: Learning, 2019Co-Authors: Sham M Kakade, Rahul Kidambi, Praneeth NetrapalliAbstract:Minimax optimal convergence rates for classes of stochastic Convex optimization problems are well characterized, where the majority of results utilize iterate averaged stochastic gradient descent (SGD) with polynomially decaying step sizes. In contrast, SGD's final iterate behavior has received much less attention despite their widespread use in practice. Motivated by this observation, this work provides a detailed study of the following question: what rate is achievable using the final iterate of SGD for the streaming least squares regression problem with and without strong Convexity? First, this work shows that even if the time horizon T (i.e. the number of iterations SGD is run for) is known in advance, SGD's final iterate behavior with any polynomially decaying learning rate scheme is highly sub-optimal compared to the minimax rate (by a condition number factor in the strongly Convex Case and a factor of $\sqrt{T}$ in the non-strongly Convex Case). In contrast, this paper shows that Step Decay schedules, which cut the learning rate by a constant factor every constant number of epochs (i.e., the learning rate decays geometrically) offers significant improvements over any polynomially decaying step sizes. In particular, the final iterate behavior with a step decay schedule is off the minimax rate by only $log$ factors (in the condition number for strongly Convex Case, and in T for the non-strongly Convex Case). Finally, in stark contrast to the known horizon Case, this paper shows that the anytime (i.e. the limiting) behavior of SGD's final iterate is poor (in that it queries iterates with highly sub-optimal function value infinitely often, i.e. in a limsup sense) irrespective of the stepsizes employed. These results demonstrate the subtlety in establishing optimal learning rate schemes (for the final iterate) for stochastic gradient procedures in fixed time horizon settings.
-
the step decay schedule a near optimal geometrically decaying learning rate procedure for least squares
Neural Information Processing Systems, 2019Co-Authors: Sham M Kakade, Rahul Kidambi, Praneeth NetrapalliAbstract:Minimax optimal convergence rates for numerous classes of stochastic Convex optimization problems are well characterized, where the majority of results utilize iterate averaged stochastic gradient descent (SGD) with polynomially decaying step sizes. In contrast, the behavior of SGD’s final iterate has received much less attention despite the widespread use in practice. Motivated by this observation, this work provides a detailed study of the following question: what rate is achievable using the final iterate of SGD for the streaming least squares regression problem with and without strong Convexity? First, this work shows that even if the time horizon T (i.e. the number of iterations that SGD is run for) is known in advance, the behavior of SGD’s final iterate with any polynomially decaying learning rate scheme is highly sub-optimal compared to the statistical minimax rate (by a condition number factor in the strongly Convex Case and a factor of $\sqrt{T}$ in the non-strongly Convex Case). In contrast, this paper shows that Step Decay schedules, which cut the learning rate by a constant factor every constant number of epochs (i.e., the learning rate decays geometrically) offer significant improvements over any polynomially decaying step size schedule. In particular, the behavior of the final iterate with step decay schedules is off from the statistical minimax rate by only log factors (in the condition number for the strongly Convex Case, and in T in the non-strongly Convex Case). Finally, in stark contrast to the known horizon Case, this paper shows that the anytime (i.e. the limiting) behavior of SGD’s final iterate is poor (in that it queries iterates with highly sub-optimal function value infinitely often, i.e. in a limsup sense) irrespective of the step size scheme employed. These results demonstrate the subtlety in establishing optimal learning rate schedules (for the final iterate) for stochastic gradient procedures in fixed time horizon settings.
Eric Moulines - One of the best experts on this subject based on the ideXlab platform.
-
on stochastic gradient langevin dynamics with dependent data streams the fully non Convex Case
arXiv: Statistics Theory, 2019Co-Authors: N H Chau, Eric Moulines, Miklos Rasonyi, Sotirios Sabanis, Ying ZhangAbstract:We consider the problem of sampling from a target distribution, which is \emph {not necessarily logconcave}, in the context of empirical risk minimization and stochastic optimization as presented in Raginsky et al. (2017). Non-asymptotic analysis results are established in the $L^1$-Wasserstein distance for the behaviour of Stochastic Gradient Langevin Dynamics (SGLD) algorithms. We allow the estimation of gradients to be performed even in the presence of \emph{dependent} data streams. Our convergence estimates are sharper and \emph{uniform} in the number of iterations, in contrast to those in previous studies.
-
non asymptotic analysis of stochastic approximation algorithms for machine learning
Neural Information Processing Systems, 2011Co-Authors: Eric Moulines, Francis BachAbstract:We consider the minimization of a Convex objective function defined on a Hilbert space, which is only available through unbiased estimates of its gradients. This problem includes standard machine learning algorithms such as kernel logistic regression and least-squares regression, and is commonly referred to as a stochastic approximation problem in the operations research community. We provide a non-asymptotic analysis of the convergence of two well-known algorithms, stochastic gradient descent (a.k.a. Robbins-Monro algorithm) as well as a simple modification where iterates are averaged (a.k.a. Polyak-Ruppert averaging). Our analysis suggests that a learning rate proportional to the inverse of the number of iterations, while leading to the optimal convergence rate in the strongly Convex Case, is not robust to the lack of strong Convexity or the setting of the proportionality constant. This situation is remedied when using slower decays together with averaging, robustly leading to the optimal rate of convergence. We illustrate our theoretical results with simulations on synthetic and standard datasets.