The Experts below are selected from a list of 360 Experts worldwide ranked by ideXlab platform
Jianqing Fan - One of the best experts on this subject based on the ideXlab platform.
-
Gradient descent with random initialization: fast Global Convergence for nonconvex phase retrieval
Mathematical Programming, 2019Co-Authors: Yuxin Chen, Yuejie Chi, Jianqing Fan, Cong MaAbstract:This paper considers the problem of solving systems of quadratic equations, namely, recovering an object of interest \(\varvec{x}^{\natural }\in {\mathbb {R}}^{n}\) from m quadratic equations/samples \(y_{i}=(\varvec{a}_{i}^{\top }\varvec{x}^{\natural })^{2}, 1\le i\le m\). This problem, also dubbed as phase retrieval, spans multiple domains including physical sciences and machine learning. We investigate the efficacy of gradient descent (or Wirtinger flow) designed for the nonconvex least squares problem. We prove that under Gaussian designs, gradient descent—when randomly initialized—yields an \(\epsilon \)-accurate solution in \(O\big (\log n+\log (1/\epsilon )\big )\) iterations given nearly minimal samples, thus achieving near-optimal computational and sample complexities at once. This provides the first Global Convergence guarantee concerning vanilla gradient descent for phase retrieval, without the need of (i) carefully-designed initialization, (ii) sample splitting, or (iii) sophisticated saddle-point escaping schemes. All of these are achieved by exploiting the statistical models in analyzing optimization algorithms, via a leave-one-out approach that enables the decoupling of certain statistical dependency between the gradient descent iterates and the data.
-
gradient descent with random initialization fast Global Convergence for nonconvex phase retrieval
arXiv: Machine Learning, 2018Co-Authors: Yuxin Chen, Yuejie Chi, Jianqing FanAbstract:This paper considers the problem of solving systems of quadratic equations, namely, recovering an object of interest $\mathbf{x}^{\natural}\in\mathbb{R}^{n}$ from $m$ quadratic equations/samples $y_{i}=(\mathbf{a}_{i}^{\top}\mathbf{x}^{\natural})^{2}$, $1\leq i\leq m$. This problem, also dubbed as phase retrieval, spans multiple domains including physical sciences and machine learning. We investigate the efficiency of gradient descent (or Wirtinger flow) designed for the nonconvex least squares problem. We prove that under Gaussian designs, gradient descent --- when randomly initialized --- yields an $\epsilon$-accurate solution in $O\big(\log n+\log(1/\epsilon)\big)$ iterations given nearly minimal samples, thus achieving near-optimal computational and sample complexities at once. This provides the first Global Convergence guarantee concerning vanilla gradient descent for phase retrieval, without the need of (i) carefully-designed initialization, (ii) sample splitting, or (iii) sophisticated saddle-point escaping schemes. All of these are achieved by exploiting the statistical models in analyzing optimization algorithms, via a leave-one-out approach that enables the decoupling of certain statistical dependency between the gradient descent iterates and the data.
Mikael Johansson - One of the best experts on this subject based on the ideXlab platform.
-
Global Convergence of the Heavy-ball method for convex optimization
2015 European Control Conference (ECC), 2015Co-Authors: Euhanna Ghadimi, Hamid Reza Feyzmahdavian, Mikael JohanssonAbstract:This paper establishes Global Convergence and provides Global bounds of the rate of Convergence for the Heavy-ball method for convex optimization. When the objective function has Lipschitz-continuous gradient, we show that the Cesáro average of the iterates converges to the optimum at a rate of O(1/k) where k is the number of iterations. When the objective function is also strongly convex, we prove that the Heavy-ball iterates converge linearly to the unique optimum. Numerical examples validate our theoretical findings.
-
Global Convergence of the Heavy-ball method for convex optimization
arXiv: Optimization and Control, 2014Co-Authors: Euhanna Ghadimi, Hamid Reza Feyzmahdavian, Mikael JohanssonAbstract:This paper establishes Global Convergence and provides Global bounds of the Convergence rate of the Heavy-ball method for convex optimization problems. When the objective function has Lipschitz-continuous gradient, we show that the Cesaro average of the iterates converges to the optimum at a rate of $O(1/k)$ where k is the number of iterations. When the objective function is also strongly convex, we prove that the Heavy-ball iterates converge linearly to the unique optimum.
Ming Cao - One of the best experts on this subject based on the ideXlab platform.
-
Global Convergence for replicator dynamics of repeated snowdrift games
IEEE Transactions on Automatic Control, 2021Co-Authors: Pouria Ramazi, Ming CaoAbstract:To understand the emergence and sustainment of cooperative behavior in interacting collectives, we perform Global Convergence analysis for replicator dynamics of a large, well-mixed population of individuals playing a repeated snowdrift game with four typical strategies, which are always cooperate (ALLC), tit-for-tat (TFT), suspicious tit-for-tat (STFT), and always defect (ALLD). The dynamical model is a 3-D ordinary differential equation (ODE) system that is parameterized by the payoffs of the base game. Instead of routine searches for evolutionarily stable strategies and sets, we expand our analysis to determining the asymptotic behavior of solution trajectories starting from any initial state, and in particular, show that for the full range of payoffs, every trajectory of the system converges to an equilibrium point. What enables us to achieve such comprehensive results is studying the dynamics of two ratios of the state variables, each of which either monotonically increases or decreases in the half-spaces separated by their corresponding planes. The Convergence results highlight two findings. First, the inclusion of TFT- and STFT-players, the two types of conditional strategy players in the game, increases the share of cooperators of the overall population compared to the situation when the population consists of only ALLC and ALLD-players. Second, surprisingly enough, regardless of the payoffs, there always exists a set of initial conditions under which ALLC-players do not vanish in the long run, which does not hold for any of the other three types of players.
-
Global Convergence for replicator dynamics of repeated snowdrift games
arXiv: Dynamical Systems, 2019Co-Authors: Pouria Ramazi, Ming CaoAbstract:To understand the emergence and sustainment of cooperative behavior in interacting collectives, we perform Global Convergence analysis for replicator dynamics of a large, well-mixed population of individuals playing a repeated snowdrift game with four typical strategies, which are always cooperate (ALLC), tit-for-tat (TFT), suspicious tit-for-tat (STFT) and always defect (ALLD). The dynamical model is a three-dimensional ODE system that is parameterized by the payoffs of the base game. Instead of routine searches for evolutionarily stable strategies and sets, we expand our analysis to determining the asymptotic behavior of solution trajectories starting from any initial state, and in particular show that for the full range of payoffs, every trajectory of the system converges to an equilibrium point. The Convergence results highlight three findings that are of particular importance for understanding the cooperation mechanisms among self-interested agents playing repeated snowdrift games. First, the inclusion of TFT- and STFT-players, the two types of conditional strategy players in the game, increases the share of cooperators of the overall population compared to the situation when the population consists of only ALLC- and ALLD-players. This confirms findings in biology and sociology that reciprocity may promote cooperation in social collective actions, such as reducing traffic jams and division of labor, where each individual may gain more to play the opposite of what her opponent chooses. Second, surprisingly enough, regardless of the payoffs, there always exists a set of initial conditions under which ALLC players do not vanish in the long run, which does not hold for all the other three types of players. So an ALLC-player, although perceived as the one that can be easily taken advantage of in snowdrift games, has certain endurance in the long run.
Paolo Nistri - One of the best experts on this subject based on the ideXlab platform.
-
Global exponential stability and Global Convergence in finite time of delayed neural networks with infinite gain
IEEE Transactions on Neural Networks, 2005Co-Authors: Mauro Forti, Paolo Nistri, Duccio PapiniAbstract:This paper introduces a general class of neural networks with arbitrary constant delays in the neuron interconnections, and neuron activations belonging to the set of discontinuous monotone increasing and (possibly) unbounded functions. The discontinuities in the activations are an ideal model of the situation where the gain of the neuron amplifiers is very high and tends to infinity, while the delay accounts for the finite switching speed of the neuron amplifiers, or the finite signal propagation speed. It is known that the delay in combination with high-gain nonlinearities is a particularly harmful source of potential instability. The goal of this paper is to single out a subclass of the considered discontinuous neural networks for which stability is instead insensitive to the presence of a delay. More precisely, conditions are given under which there is a unique equilibrium point of the neural network, which is Globally exponentially stable for the states, with a known Convergence rate. The conditions are easily testable and independent of the delay. Moreover, Global Convergence in finite time of the state and output is investigated. In doing so, new interesting dynamical phenomena are highlighted with respect to the case without delay, which make the study of Convergence in finite time significantly more difficult. The obtained results extend previous work on Global stability of delayed neural networks with Lipschitz continuous neuron activations, and neural networks with discontinuous neuron activations but without delays.
-
Global Convergence of neural networks with discontinuous neuron activations
IEEE Transactions on Circuits and Systems I-regular Papers, 2003Co-Authors: Mauro Forti, Paolo NistriAbstract:The paper introduces a general class of neural networks where the neuron activations are modeled by discontinuous functions. The neural networks have an additive interconnecting structure and they include as particular cases the Hopfield neural networks (HNNs), and the standard cellular neural networks (CNNs), in the limiting situation where the HNNs and CNNs possess neurons with infinite gain. Conditions are derived which ensure the existence of a unique equilibrium point, and a unique output equilibrium point, which are Globally attractive for the state and the output trajectories of the neural network, respectively. These conditions, which are applicable to general nonsymmetric neural networks, are based on the concept of Lyapunov diagonally-stable neuron interconnection matrices, and they can be thought of as a generalization to the discontinuous case of previous results established for neural networks possessing smooth neuron activations. Moreover, by suitably exploiting the presence of sliding modes, entirely new conditions are obtained which ensure Global Convergence in finite time, where the Convergence time can be easily estimated on the basis of the relevant neural-network parameters. The analysis in the paper employs results from the theory of differential equations with discontinuous right-hand side as introduced by Filippov. In particular, Global Convergence is addressed by using a Lyapunov-like approach based on the concept of monotone trajectories of a differential inclusion.
Yuxin Chen - One of the best experts on this subject based on the ideXlab platform.
-
Gradient descent with random initialization: fast Global Convergence for nonconvex phase retrieval
Mathematical Programming, 2019Co-Authors: Yuxin Chen, Yuejie Chi, Jianqing Fan, Cong MaAbstract:This paper considers the problem of solving systems of quadratic equations, namely, recovering an object of interest \(\varvec{x}^{\natural }\in {\mathbb {R}}^{n}\) from m quadratic equations/samples \(y_{i}=(\varvec{a}_{i}^{\top }\varvec{x}^{\natural })^{2}, 1\le i\le m\). This problem, also dubbed as phase retrieval, spans multiple domains including physical sciences and machine learning. We investigate the efficacy of gradient descent (or Wirtinger flow) designed for the nonconvex least squares problem. We prove that under Gaussian designs, gradient descent—when randomly initialized—yields an \(\epsilon \)-accurate solution in \(O\big (\log n+\log (1/\epsilon )\big )\) iterations given nearly minimal samples, thus achieving near-optimal computational and sample complexities at once. This provides the first Global Convergence guarantee concerning vanilla gradient descent for phase retrieval, without the need of (i) carefully-designed initialization, (ii) sample splitting, or (iii) sophisticated saddle-point escaping schemes. All of these are achieved by exploiting the statistical models in analyzing optimization algorithms, via a leave-one-out approach that enables the decoupling of certain statistical dependency between the gradient descent iterates and the data.
-
gradient descent with random initialization fast Global Convergence for nonconvex phase retrieval
arXiv: Machine Learning, 2018Co-Authors: Yuxin Chen, Yuejie Chi, Jianqing FanAbstract:This paper considers the problem of solving systems of quadratic equations, namely, recovering an object of interest $\mathbf{x}^{\natural}\in\mathbb{R}^{n}$ from $m$ quadratic equations/samples $y_{i}=(\mathbf{a}_{i}^{\top}\mathbf{x}^{\natural})^{2}$, $1\leq i\leq m$. This problem, also dubbed as phase retrieval, spans multiple domains including physical sciences and machine learning. We investigate the efficiency of gradient descent (or Wirtinger flow) designed for the nonconvex least squares problem. We prove that under Gaussian designs, gradient descent --- when randomly initialized --- yields an $\epsilon$-accurate solution in $O\big(\log n+\log(1/\epsilon)\big)$ iterations given nearly minimal samples, thus achieving near-optimal computational and sample complexities at once. This provides the first Global Convergence guarantee concerning vanilla gradient descent for phase retrieval, without the need of (i) carefully-designed initialization, (ii) sample splitting, or (iii) sophisticated saddle-point escaping schemes. All of these are achieved by exploiting the statistical models in analyzing optimization algorithms, via a leave-one-out approach that enables the decoupling of certain statistical dependency between the gradient descent iterates and the data.
