The Experts below are selected from a list of 360 Experts worldwide ranked by ideXlab platform
Xiangfeng Wang - One of the best experts on this subject based on the ideXlab platform.
-
on the flexibility of block coordinate descent for large scale Optimization
Neurocomputing, 2018Co-Authors: Xiangfeng Wang, Wenjie Zhang, Junchi Yan, Xiaoming Yuan, Hongyuan ZhaAbstract:Abstract We consider a Large-Scale minimization problem (not necessarily convex) with non-smooth separable convex penalty. Problems in this form widely arise in many modern Large-Scale machine learning and signal processing applications. In this paper, we present a new perspective towards the parallel Block Coordinate Descent (BCD) methods. Specifically we explicitly give a concept of so-called two-layered block variable updating loop for parallel BCD methods in modern computing environment comprised of multiple distributed computing nodes. The outer loop refers to the block variable updating assigned to distributed nodes, and the inner loop involves the updating step inside each node. Each loop allows to adopt either Jacobi or Gauss–Seidel update rule. In particular, we give detailed theoretical convergence analysis to two practical schemes: Jacobi/Gauss–Seidel and Gauss–Seidel/Jacobi that embodies two algorithms respectively. Our new perspective and behind theoretical results help devise parallel BCD algorithms in a principled fashion, which in turn lend them a flexible implementation for BCD methods suited to the parallel computing environment. The effectiveness of the algorithm framework is verified on the benchmark tasks of Large-Scale l1 regularized sparse logistic regression and non-negative matrix factorization.
-
asynchronous distributed admm for large scale Optimization part i algorithm and convergence analysis
IEEE Transactions on Signal Processing, 2016Co-Authors: Tsunghui Chang, Mingyi Hong, Weicheng Liao, Xiangfeng WangAbstract:Aiming at solving Large-Scale Optimization problems, this paper studies distributed Optimization methods based on the alternating direction method of multipliers (ADMM). By formulating the Optimization problem as a consensus problem, the ADMM can be used to solve the consensus problem in a fully parallel fashion over a computer network with a star topology. However, traditional synchronized computation does not scale well with the problem size, as the speed of the algorithm is limited by the slowest workers. This is particularly true in a heterogeneous network where the computing nodes experience different computation and communication delays. In this paper, we propose an asynchronous distributed ADMM (AD-ADMM), which can effectively improve the time efficiency of distributed Optimization. Our main interest lies in analyzing the convergence conditions of the AD-ADMM, under the popular partially asynchronous model, which is defined based on a maximum tolerable delay of the network. Specifically, by considering general and possibly non-convex cost functions, we show that the AD-ADMM is guaranteed to converge to the set of Karush–Kuhn–Tucker (KKT) points as long as the algorithm parameters are chosen appropriately according to the network delay. We further illustrate that the asynchrony of the ADMM has to be handled with care, as slightly modifying the implementation of the AD-ADMM can jeopardize the algorithm convergence, even under the standard convex setting.
-
asynchronous distributed admm for large scale Optimization part ii linear convergence analysis and numerical performance
IEEE Transactions on Signal Processing, 2016Co-Authors: Tsunghui Chang, Mingyi Hong, Weicheng Liao, Xiangfeng WangAbstract:The alternating direction method of multipliers (ADMM) has been recognized as a versatile approach for solving modern Large-Scale machine learning and signal processing problems efficiently. When the data size and/or the problem dimension is large, a distributed version of ADMM can be used, which is capable of distributing the computation load and the data set to a network of computing nodes. Unfortunately, a direct synchronous implementation of such algorithm does not scale well with the problem size, as the algorithm speed is limited by the slowest computing nodes. To address this issue, in a companion paper, we have proposed an asynchronous distributed ADMM (AD-ADMM) and studied its worst-case convergence conditions. In this paper, we further the study by characterizing the conditions under which the AD-ADMM achieves linear convergence. Our conditions as well as the resulting linear rates reveal the impact that various algorithm parameters, network delay, and network size have on the algorithm performance. To demonstrate the superior time efficiency of the proposed AD-ADMM, we test the AD-ADMM on a high-performance computer cluster by solving a Large-Scale logistic regression problem.
Tsunghui Chang - One of the best experts on this subject based on the ideXlab platform.
-
asynchronous distributed admm for large scale Optimization part i algorithm and convergence analysis
IEEE Transactions on Signal Processing, 2016Co-Authors: Tsunghui Chang, Mingyi Hong, Weicheng Liao, Xiangfeng WangAbstract:Aiming at solving Large-Scale Optimization problems, this paper studies distributed Optimization methods based on the alternating direction method of multipliers (ADMM). By formulating the Optimization problem as a consensus problem, the ADMM can be used to solve the consensus problem in a fully parallel fashion over a computer network with a star topology. However, traditional synchronized computation does not scale well with the problem size, as the speed of the algorithm is limited by the slowest workers. This is particularly true in a heterogeneous network where the computing nodes experience different computation and communication delays. In this paper, we propose an asynchronous distributed ADMM (AD-ADMM), which can effectively improve the time efficiency of distributed Optimization. Our main interest lies in analyzing the convergence conditions of the AD-ADMM, under the popular partially asynchronous model, which is defined based on a maximum tolerable delay of the network. Specifically, by considering general and possibly non-convex cost functions, we show that the AD-ADMM is guaranteed to converge to the set of Karush–Kuhn–Tucker (KKT) points as long as the algorithm parameters are chosen appropriately according to the network delay. We further illustrate that the asynchrony of the ADMM has to be handled with care, as slightly modifying the implementation of the AD-ADMM can jeopardize the algorithm convergence, even under the standard convex setting.
-
asynchronous distributed admm for large scale Optimization part ii linear convergence analysis and numerical performance
IEEE Transactions on Signal Processing, 2016Co-Authors: Tsunghui Chang, Mingyi Hong, Weicheng Liao, Xiangfeng WangAbstract:The alternating direction method of multipliers (ADMM) has been recognized as a versatile approach for solving modern Large-Scale machine learning and signal processing problems efficiently. When the data size and/or the problem dimension is large, a distributed version of ADMM can be used, which is capable of distributing the computation load and the data set to a network of computing nodes. Unfortunately, a direct synchronous implementation of such algorithm does not scale well with the problem size, as the algorithm speed is limited by the slowest computing nodes. To address this issue, in a companion paper, we have proposed an asynchronous distributed ADMM (AD-ADMM) and studied its worst-case convergence conditions. In this paper, we further the study by characterizing the conditions under which the AD-ADMM achieves linear convergence. Our conditions as well as the resulting linear rates reveal the impact that various algorithm parameters, network delay, and network size have on the algorithm performance. To demonstrate the superior time efficiency of the proposed AD-ADMM, we test the AD-ADMM on a high-performance computer cluster by solving a Large-Scale logistic regression problem.
Cesar De Prada - One of the best experts on this subject based on the ideXlab platform.
-
improving scenario decomposition algorithms for robust nonlinear model predictive control
Computers & Chemical Engineering, 2015Co-Authors: Ruben Marti, Sergio Lucia, D Sarabia, Radoslav Paulen, Sebastian Engell, Cesar De PradaAbstract:Abstract This paper deals with the efficient computation of solutions of robust nonlinear model predictive control problems that are formulated using multi-stage stochastic programming via the generation of a scenario tree. Such a formulation makes it possible to consider explicitly the concept of recourse, which is inherent to any receding horizon approach, but it results in Large-Scale Optimization problems. One possibility to solve these problems in an efficient manner is to decompose the Large-Scale Optimization problem into several subproblems that are iteratively modified and repeatedly solved until a solution to the original problem is achieved. In this paper we review the most common methods used for such decomposition and apply them to solve robust nonlinear model predictive control problems in a distributed fashion. We also propose a novel method to reduce the number of iterations of the coordination algorithm needed for the decomposition methods to converge. The performance of the different approaches is evaluated in extensive simulation studies of two nonlinear case studies.
Weicheng Liao - One of the best experts on this subject based on the ideXlab platform.
-
asynchronous distributed admm for large scale Optimization part i algorithm and convergence analysis
IEEE Transactions on Signal Processing, 2016Co-Authors: Tsunghui Chang, Mingyi Hong, Weicheng Liao, Xiangfeng WangAbstract:Aiming at solving Large-Scale Optimization problems, this paper studies distributed Optimization methods based on the alternating direction method of multipliers (ADMM). By formulating the Optimization problem as a consensus problem, the ADMM can be used to solve the consensus problem in a fully parallel fashion over a computer network with a star topology. However, traditional synchronized computation does not scale well with the problem size, as the speed of the algorithm is limited by the slowest workers. This is particularly true in a heterogeneous network where the computing nodes experience different computation and communication delays. In this paper, we propose an asynchronous distributed ADMM (AD-ADMM), which can effectively improve the time efficiency of distributed Optimization. Our main interest lies in analyzing the convergence conditions of the AD-ADMM, under the popular partially asynchronous model, which is defined based on a maximum tolerable delay of the network. Specifically, by considering general and possibly non-convex cost functions, we show that the AD-ADMM is guaranteed to converge to the set of Karush–Kuhn–Tucker (KKT) points as long as the algorithm parameters are chosen appropriately according to the network delay. We further illustrate that the asynchrony of the ADMM has to be handled with care, as slightly modifying the implementation of the AD-ADMM can jeopardize the algorithm convergence, even under the standard convex setting.
-
asynchronous distributed admm for large scale Optimization part ii linear convergence analysis and numerical performance
IEEE Transactions on Signal Processing, 2016Co-Authors: Tsunghui Chang, Mingyi Hong, Weicheng Liao, Xiangfeng WangAbstract:The alternating direction method of multipliers (ADMM) has been recognized as a versatile approach for solving modern Large-Scale machine learning and signal processing problems efficiently. When the data size and/or the problem dimension is large, a distributed version of ADMM can be used, which is capable of distributing the computation load and the data set to a network of computing nodes. Unfortunately, a direct synchronous implementation of such algorithm does not scale well with the problem size, as the algorithm speed is limited by the slowest computing nodes. To address this issue, in a companion paper, we have proposed an asynchronous distributed ADMM (AD-ADMM) and studied its worst-case convergence conditions. In this paper, we further the study by characterizing the conditions under which the AD-ADMM achieves linear convergence. Our conditions as well as the resulting linear rates reveal the impact that various algorithm parameters, network delay, and network size have on the algorithm performance. To demonstrate the superior time efficiency of the proposed AD-ADMM, we test the AD-ADMM on a high-performance computer cluster by solving a Large-Scale logistic regression problem.
Mingyi Hong - One of the best experts on this subject based on the ideXlab platform.
-
asynchronous distributed admm for large scale Optimization part i algorithm and convergence analysis
IEEE Transactions on Signal Processing, 2016Co-Authors: Tsunghui Chang, Mingyi Hong, Weicheng Liao, Xiangfeng WangAbstract:Aiming at solving Large-Scale Optimization problems, this paper studies distributed Optimization methods based on the alternating direction method of multipliers (ADMM). By formulating the Optimization problem as a consensus problem, the ADMM can be used to solve the consensus problem in a fully parallel fashion over a computer network with a star topology. However, traditional synchronized computation does not scale well with the problem size, as the speed of the algorithm is limited by the slowest workers. This is particularly true in a heterogeneous network where the computing nodes experience different computation and communication delays. In this paper, we propose an asynchronous distributed ADMM (AD-ADMM), which can effectively improve the time efficiency of distributed Optimization. Our main interest lies in analyzing the convergence conditions of the AD-ADMM, under the popular partially asynchronous model, which is defined based on a maximum tolerable delay of the network. Specifically, by considering general and possibly non-convex cost functions, we show that the AD-ADMM is guaranteed to converge to the set of Karush–Kuhn–Tucker (KKT) points as long as the algorithm parameters are chosen appropriately according to the network delay. We further illustrate that the asynchrony of the ADMM has to be handled with care, as slightly modifying the implementation of the AD-ADMM can jeopardize the algorithm convergence, even under the standard convex setting.
-
asynchronous distributed admm for large scale Optimization part ii linear convergence analysis and numerical performance
IEEE Transactions on Signal Processing, 2016Co-Authors: Tsunghui Chang, Mingyi Hong, Weicheng Liao, Xiangfeng WangAbstract:The alternating direction method of multipliers (ADMM) has been recognized as a versatile approach for solving modern Large-Scale machine learning and signal processing problems efficiently. When the data size and/or the problem dimension is large, a distributed version of ADMM can be used, which is capable of distributing the computation load and the data set to a network of computing nodes. Unfortunately, a direct synchronous implementation of such algorithm does not scale well with the problem size, as the algorithm speed is limited by the slowest computing nodes. To address this issue, in a companion paper, we have proposed an asynchronous distributed ADMM (AD-ADMM) and studied its worst-case convergence conditions. In this paper, we further the study by characterizing the conditions under which the AD-ADMM achieves linear convergence. Our conditions as well as the resulting linear rates reveal the impact that various algorithm parameters, network delay, and network size have on the algorithm performance. To demonstrate the superior time efficiency of the proposed AD-ADMM, we test the AD-ADMM on a high-performance computer cluster by solving a Large-Scale logistic regression problem.
