The Experts below are selected from a list of 246 Experts worldwide ranked by ideXlab platform
Quanquan Gu - One of the best experts on this subject based on the ideXlab platform.
-
NeurIPS - Stochastic Nested Variance Reduced Gradient Descent for Nonconvex Optimization
2018Co-Authors: Dongruo Zhou, Pan Xu, Quanquan GuAbstract:We study finite-sum nonconvex optimization problems, where the objective function is an average of n nonconvex functions. We propose a new stochastic Gradient descent algorithm based on nested variance reduction. Compared with conventional stochastic variance Reduced Gradient (SVRG) algorithm that uses two reference points to construct a semi-stochastic Gradient with diminishing variance in each epoch, our algorithm uses K+1 nested reference points to build an semi-stochastic Gradient to further reduce its variance in each epoch. For smooth functions, the proposed algorithm converges to an approximate first order stationary point (i.e., ‖∇F(\xb)‖2≤ϵ) within \tO(n∧ϵ−2+ϵ−3∧n1/2ϵ−2)\footnote{\tO(⋅) hides the logarithmic factors} number of stochastic Gradient evaluations, where n is the number of component functions, and ϵ is the optimization error. This improves the best known Gradient complexity of SVRG O(n+n2/3ϵ−2) and the best Gradient complexity of SCSG O(ϵ−5/3∧n2/3ϵ−2). For Gradient dominated functions, our algorithm achieves \tO(n∧τϵ−1+τ⋅(n1/2∧(τϵ−1)1/2) Gradient complexity, which again beats the existing best Gradient complexity \tO(n∧τϵ−1+τ⋅(n1/2∧(τϵ−1)2/3) achieved by SCSG. Thorough experimental results on different nonconvex optimization problems back up our theory.
-
stochastic nested variance Reduced Gradient descent for nonconvex optimization
Neural Information Processing Systems, 2018Co-Authors: Dongruo Zhou, Pan Xu, Quanquan GuAbstract:We study finite-sum nonconvex optimization problems, where the objective function is an average of n nonconvex functions. We propose a new stochastic Gradient descent algorithm based on nested variance reduction. Compared with conventional stochastic variance Reduced Gradient (SVRG) algorithm that uses two reference points to construct a semi-stochastic Gradient with diminishing variance in each epoch, our algorithm uses K+1 nested reference points to build an semi-stochastic Gradient to further reduce its variance in each epoch. For smooth functions, the proposed algorithm converges to an approximate first order stationary point (i.e., ‖∇F(\xb)‖2≤ϵ) within \tO(n∧ϵ−2+ϵ−3∧n1/2ϵ−2)\footnote{\tO(⋅) hides the logarithmic factors} number of stochastic Gradient evaluations, where n is the number of component functions, and ϵ is the optimization error. This improves the best known Gradient complexity of SVRG O(n+n2/3ϵ−2) and the best Gradient complexity of SCSG O(ϵ−5/3∧n2/3ϵ−2). For Gradient dominated functions, our algorithm achieves \tO(n∧τϵ−1+τ⋅(n1/2∧(τϵ−1)1/2) Gradient complexity, which again beats the existing best Gradient complexity \tO(n∧τϵ−1+τ⋅(n1/2∧(τϵ−1)2/3) achieved by SCSG. Thorough experimental results on different nonconvex optimization problems back up our theory.
-
SVRG-LD$^+$: Subsampled Stochastic Variance-Reduced Gradient Langevin Dynamics
2018Co-Authors: Pan Xu, Quanquan GuAbstract:Stochastic variance-Reduced Gradient Langevin dynamics (SVRG-LD) was recently proposed to improve the performance of stochastic Gradient Langevin dynamics (SGLD) by reducing the variance in the stochastic Gradient. In this paper, we study a variant of SVRG-LD, namely SVRG-LD$^+$, which replaces the full Gradient in each epoch by a subsampled one. We provide a nonasymptotic analysis of the convergence of SVRG-LD$^+$ in $2$-Wasserstein distance, and show that SVRG-LD$^+$ enjoys a lower Gradient complexity than SVRG-LD, when the sample size is large or the target accuracy requirement is moderate. Our analysis directly implies a sharper convergence rate for SVRG-LD, which improves the existing convergence rate by a factor of $\kappa^{1/6}n^{1/6}$, where $\kappa$ is the condition number of the log-density function and $n$ is the sample size. Experiments on both synthetic and real-world datasets validate our theoretical results.
-
Stochastic Variance-Reduced Gradient Descent for Low-rank Matrix Recovery from Linear Measurements
arXiv: Machine Learning, 2017Co-Authors: Xiao Zhang, Lingxiao Wang, Quanquan GuAbstract:We study the problem of estimating low-rank matrices from linear measurements (a.k.a., matrix sensing) through nonconvex optimization. We propose an efficient stochastic variance Reduced Gradient descent algorithm to solve a nonconvex optimization problem of matrix sensing. Our algorithm is applicable to both noisy and noiseless settings. In the case with noisy observations, we prove that our algorithm converges to the unknown low-rank matrix at a linear rate up to the minimax optimal statistical error. And in the noiseless setting, our algorithm is guaranteed to linearly converge to the unknown low-rank matrix and achieves exact recovery with optimal sample complexity. Most notably, the overall computational complexity of our proposed algorithm, which is defined as the iteration complexity times per iteration time complexity, is lower than the state-of-the-art algorithms based on Gradient descent. Experiments on synthetic data corroborate the superiority of the proposed algorithm over the state-of-the-art algorithms.
Jungmin Hwang - One of the best experts on this subject based on the ideXlab platform.
-
Stepless tunable four-chip LED lighting control on a black body radiation curve using the generalized Reduced Gradient method
Optical and Quantum Electronics, 2016Co-Authors: Jungmin HwangAbstract:This study proposes an algorithm for calculating the outputs of a stepless, tunable, four-chip LED according to a black body radiation curve; the proposed algorithm is based on the generalized Reduced Gradient method. The lumen outputs of the four-chip LEDs were calculated in real time at color temperatures ranging between 2500 K and 6500 K without using a lookup table, enabling achieving high-quality lighting designs. Finally, the algorithms for the stepless dimming method, with average color shift Duv 0.0027, is proposed. This algorithm has much smaller memory size requirements for the microcontroller unit than the current method.
-
Stepless tunable four-chip LED lighting control on the black body radiation curve with generalized Reduced Gradient method
2015 International Conference on Numerical Simulation of Optoelectronic Devices (NUSOD), 2015Co-Authors: Che-min Kung, Shun-yi Yang, Jungmin HwangAbstract:In this paper, the algorithm for the stepless tunable four-chip LED on the black body radiation curve, using the Generalized Reduced Gradient method, is proposed. The lumen outputs of four-chip LED, for color temperatures between 2500K and 6500K, are then calculated in real-time, without using a lookup table, and for high quality lighting design.
-
Selecting conversion phosphors for white light-emitting diodes package by generalized Reduced Gradient method in dispensing application
2015 International Conference on Numerical Simulation of Optoelectronic Devices (NUSOD), 2015Co-Authors: Jungmin HwangAbstract:The conversion-phosphor selection engine with chromaticity coordinates and additive color-mixing theory for light-emitting diodes is constructed. The parameters are linked by the generalized Reduced Gradient method for optimization. Finally, this model can be used in the LED dispensing process for 2700K and high color rendering index requirement.
-
specific lighting spectrum matching by normalized correlation coefficient and generalized Reduced Gradient method
International Microsystems Packaging Assembly and Circuits Technology Conference, 2014Co-Authors: Jungmin HwangAbstract:The optimization matching procedure for specific LED spectrum matching is proposed by the normalized correlation coefficient and generalized Reduced Gradient method in this study. The different spectrum means different color temperature, color rendering index, and lighting quality features. However, the specific spectrum for human centric lighting or light therapy may not be achieved by single light source but by multi-lighting sources for cost consideration. Therefore, the Gaussian distributions are used to define the specific spectrum and generate the row data rapidly for the matching calculation. The normalized correlation coefficient is used to evaluate the spectrum matching result and optimized by generalized Reduced Gradient method. Finally, the specific spectrum is matched with multi-lighting sources in the optimized mixing ratio from the lighting sources database.
Abdelkrim El Mouatasim - One of the best experts on this subject based on the ideXlab platform.
-
Nesterov Step Reduced Gradient Algorithm for Convex Programming Problems
Big Data and Networks Technologies, 2019Co-Authors: Abdelkrim El Mouatasim, Yousef FarhaouiAbstract:In this paper, we proposed an implementation of method of speed Reduced Gradient algorithm for optimizing a convex differentiable function subject to linear equality constraints and nonnegativity bounds on the variables. In particular, at each iteration, we compute a search direction by Reduced Gradient, and line search by bisection algorithm or Armijo rule. Under some assumption, the convergence rate of speed Reduced Gradient (SRG) algorithm is proven to be significantly better, both theoretically and practically. The algorithm of SRG are programmed by Matlab, and comparing by Frank-Wolfe algorithm some problems, the numerical results which show the efficient of our approach, we give also an application to ODE, optimal control, image and video co-localization and learning machine.
-
implementation of Reduced Gradient with bisection algorithms for non convex optimization problem via stochastic perturbation
Numerical Algorithms, 2018Co-Authors: Abdelkrim El MouatasimAbstract:In this paper, we proposed an implementation of stochastic perturbation of Reduced Gradient and bisection (SPRGB) method for optimizing a non-convex differentiable function subject to linear equality constraints and non-negativity bounds on the variables. In particular, at each iteration, we compute a search direction by Reduced Gradient, and optimal line search by bisection algorithm along this direction yields a decrease in the objective value. SPRGB method is desired to establish the global convergence of the algorithm. An implementation and tests of SPRGB algorithm are given, and some numerical results of large-scale problems are presented, which show the efficient of this approach.
-
stochastic perturbation of Reduced Gradient grg methods for nonconvex programming problems
Applied Mathematics and Computation, 2014Co-Authors: Abdelkrim El Mouatasim, Rachid Ellaia, Eduardo Souza De CursiAbstract:In this paper, we consider nonconvex differentiable programming under linear and nonlinear differentiable constraints. A Reduced Gradient and GRG (generalized Reduced Gradient) descent methods involving stochastic perturbation are proposed and we give a mathematical result establishing the convergence to a global minimizer. Numerical examples are given in order to show that the method is effective to calculate. Namely, we consider classical tests such as the statistical problem, the octagon problem, the mixture problem and an application to the linear optimal control servomotor problem.
-
Stochastic perturbation of Reduced Gradient & GRG methods for nonconvex programming problems
Applied Mathematics and Computation, 2014Co-Authors: Abdelkrim El Mouatasim, Rachid Ellaia, Eduardo Souza De CursiAbstract:In this paper, we consider nonconvex differentiable programming under linear and nonlinear differentiable constraints. A Reduced Gradient and GRG (generalized Reduced Gradient) descent methods involving stochastic perturbation are proposed and we give a mathematical result establishing the convergence to a global minimizer. Numerical examples are given in order to show that the method is effective to calculate. Namely, we consider classical tests such as the statistical problem, the octagon problem, the mixture problem and an application to the linear optimal control servomotor problem.
-
Two-Phase Generalized Reduced Gradient Method for Constrained Global Optimization
Journal of Applied Mathematics, 2010Co-Authors: Abdelkrim El MouatasimAbstract:The random perturbation of generalized Reduced Gradient method for optimization under nonlinear differentiable constraints is proposed. Generally speaking, a particular iteration of this method proceeds in two phases. In the Restoration Phase, feasibility is restored by means of the resolution of an auxiliary nonlinear problem, a generally nonlinear system of equations. In the Optimization Phase, optimality is improved by means of the consideration of the objective function, on the tangent subspace to the constraints. In this paper, optimal assumptions are stated on the Restoration Phase and the Optimization Phase that establish the global convergence of the algorithm. Some numerical examples are also given by mixture problem and octagon problem.
Malgorzata P Kaleta - One of the best experts on this subject based on the ideXlab platform.
-
Efficient Solution of the Optimization Problem in Model-Reduced Gradient-based History Matching
ECMOR XIII - 13th European Conference on the Mathematics of Oil Recovery, 2012Co-Authors: Slawomir P. Szklarz, Marielba Rojas, Malgorzata P KaletaAbstract:Adjusting parameters in reservoir models by minimizing the discrepancy between the model's predictions and actual measurements is a popular approach known as history matching. One of the most effective techniques is Gradient-based history matching. For reservoir models, the number of grid blocks and therefore, the size of the problem can become very large. In recent years, model-order reduction techniques aiming to replace large, complex dynamic systems with lower-dimension models have been incorporated into history matching. In both Gradient-based history matching and model-Reduced Gradient-based history matching, first-order optimization methods are used in order to minimize the mismatch between simulated well-production data and observed production. In this work, we investigate the performance of some optimization methods on the minimization problem in model-Reduced Gradient-based history matching. The methods were tested on the history matching of a small reservoir model with synthetic measurements. Our results show that fast first-order techniques such as the spectral projected Gradient method can compete with the popular quasi-Newton BFGS approach.
-
The optimization problem in model-Reduced Gradient-based history matching
IFAC Proceedings Volumes, 2012Co-Authors: Slawomir P. Szklarz, Marielba Rojas, Malgorzata P KaletaAbstract:Abstract We present preliminary results of a performance evaluation study of several Gradient-based state-of-the-art optimization methods for solving the nonlinear minimization problem arising in model-Reduced Gradient-based history matching. The issues discussed also apply to other areas, such as production optimization in closed-loop reservoir management.
-
model Reduced Gradient based history matching
Computational Geosciences, 2011Co-Authors: Malgorzata P Kaleta, R G Hanea, A W Heemink, J D JansenAbstract:Gradient-based history matching algorithms can be used to adapt the uncertain parameters in a reservoir model using production data. They require, however, the implementation of an adjoint model to compute the Gradients, which is usually an enormous programming effort. We propose a new approach to Gradient-based history matching which is based on model reduction, where the original (nonlinear and high-order) forward model is replaced by a linear Reduced-order forward model and, consequently, the adjoint of the tangent linear approximation of the original forward model is replaced by the adjoint of a linear Reduced-order forward model. The Reduced-order model is constructed with the aid of the proper orthogonal decomposition method. Due to the linear character of the Reduced model, the corresponding adjoint model is easily obtained. The Gradient of the objective function is approximated, and the minimization problem is solved in the Reduced space; the procedure is iterated with the updated estimate of the parameters if necessary. The proposed approach is adjoint-free and can be used with any reservoir simulator. The method was evaluated for a waterflood reservoir with channelized permeability field. A comparison with an adjoint-based history matching procedure shows that the model-Reduced approach gives a comparable quality of history matches and predictions. The computational efficiency of the model-Reduced approach is lower than of an adjoint-based approach, but higher than of an approach where the Gradients are obtained with simple finite differences.
Dongruo Zhou - One of the best experts on this subject based on the ideXlab platform.
-
stochastic nested variance Reduced Gradient descent for nonconvex optimization
Neural Information Processing Systems, 2018Co-Authors: Dongruo Zhou, Pan Xu, Quanquan GuAbstract:We study finite-sum nonconvex optimization problems, where the objective function is an average of n nonconvex functions. We propose a new stochastic Gradient descent algorithm based on nested variance reduction. Compared with conventional stochastic variance Reduced Gradient (SVRG) algorithm that uses two reference points to construct a semi-stochastic Gradient with diminishing variance in each epoch, our algorithm uses K+1 nested reference points to build an semi-stochastic Gradient to further reduce its variance in each epoch. For smooth functions, the proposed algorithm converges to an approximate first order stationary point (i.e., ‖∇F(\xb)‖2≤ϵ) within \tO(n∧ϵ−2+ϵ−3∧n1/2ϵ−2)\footnote{\tO(⋅) hides the logarithmic factors} number of stochastic Gradient evaluations, where n is the number of component functions, and ϵ is the optimization error. This improves the best known Gradient complexity of SVRG O(n+n2/3ϵ−2) and the best Gradient complexity of SCSG O(ϵ−5/3∧n2/3ϵ−2). For Gradient dominated functions, our algorithm achieves \tO(n∧τϵ−1+τ⋅(n1/2∧(τϵ−1)1/2) Gradient complexity, which again beats the existing best Gradient complexity \tO(n∧τϵ−1+τ⋅(n1/2∧(τϵ−1)2/3) achieved by SCSG. Thorough experimental results on different nonconvex optimization problems back up our theory.
-
NeurIPS - Stochastic Nested Variance Reduced Gradient Descent for Nonconvex Optimization
2018Co-Authors: Dongruo Zhou, Pan Xu, Quanquan GuAbstract:We study finite-sum nonconvex optimization problems, where the objective function is an average of n nonconvex functions. We propose a new stochastic Gradient descent algorithm based on nested variance reduction. Compared with conventional stochastic variance Reduced Gradient (SVRG) algorithm that uses two reference points to construct a semi-stochastic Gradient with diminishing variance in each epoch, our algorithm uses K+1 nested reference points to build an semi-stochastic Gradient to further reduce its variance in each epoch. For smooth functions, the proposed algorithm converges to an approximate first order stationary point (i.e., ‖∇F(\xb)‖2≤ϵ) within \tO(n∧ϵ−2+ϵ−3∧n1/2ϵ−2)\footnote{\tO(⋅) hides the logarithmic factors} number of stochastic Gradient evaluations, where n is the number of component functions, and ϵ is the optimization error. This improves the best known Gradient complexity of SVRG O(n+n2/3ϵ−2) and the best Gradient complexity of SCSG O(ϵ−5/3∧n2/3ϵ−2). For Gradient dominated functions, our algorithm achieves \tO(n∧τϵ−1+τ⋅(n1/2∧(τϵ−1)1/2) Gradient complexity, which again beats the existing best Gradient complexity \tO(n∧τϵ−1+τ⋅(n1/2∧(τϵ−1)2/3) achieved by SCSG. Thorough experimental results on different nonconvex optimization problems back up our theory.
