The Experts below are selected from a list of 15525 Experts worldwide ranked by ideXlab platform

René Vidal - One of the best experts on this subject based on the ideXlab platform.

  • structured low rank matrix factorization Global Optimality algorithms and applications
    IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020
    Co-Authors: Benjamin D Haeffele, René Vidal
    Abstract:

    Convex formulations of low-rank matrix factorization problems have received considerable attention in machine learning. However, such formulations often require solving for a matrix of the size of the data matrix, making it challenging to apply them to large scale datasets. Moreover, in many applications the data can display structures beyond simply being low-rank, e.g., images and videos present complex spatio-temporal structures that are largely ignored by standard low-rank methods. In this paper we study a matrix factorization technique that is suitable for large datasets and captures additional structure in the factors by using a particular form of regularization that includes well-known regularizers such as total variation and the nuclear norm as particular cases. Although the resulting optimization problem is non-convex, we show that if the size of the factors is large enough, under certain conditions, any local minimizer for the factors yields a Global minimizer. A few practical algorithms are also provided to solve the matrix factorization problem, and bounds on the distance from a given approximate solution of the optimization problem to the Global optimum are derived. Examples in neural calcium imaging video segmentation and hyperspectral compressed recovery show the advantages of our approach on high-dimensional datasets.

  • Global Optimality in separable dictionary learning with applications to the analysis of diffusion mri
    arXiv: Optimization and Control, 2018
    Co-Authors: Evan Schwab, Benjamin D Haeffele, René Vidal, Nicolas Charon
    Abstract:

    Sparse dictionary learning is a popular method for representing signals as linear combinations of a few elements from a dictionary that is learned from the data. In the classical setting, signals are represented as vectors and the dictionary learning problem is posed as a matrix factorization problem where the data matrix is approximately factorized into a dictionary matrix and a sparse matrix of coefficients. However, in many applications in computer vision and medical imaging, signals are better represented as matrices or tensors (e.g. images or videos), where it may be beneficial to exploit the multi-dimensional structure of the data to learn a more compact representation. One such approach is separable dictionary learning, where one learns separate dictionaries for different dimensions of the data. However, typical formulations involve solving a non-convex optimization problem; thus guaranteeing Global Optimality remains a challenge. In this work, we propose a framework that builds upon recent developments in matrix factorization to provide theoretical and numerical guarantees of Global Optimality for separable dictionary learning. We propose an algorithm to find such a Globally optimal solution, which alternates between following local descent steps and checking a certificate for Global Optimality. We illustrate our approach on diffusion magnetic resonance imaging (dMRI) data, a medical imaging modality that measures water diffusion along multiple angular directions in every voxel of an MRI volume. State-of-the-art methods in dMRI either learn dictionaries only for the angular domain of the signals or in some cases learn spatial and angular dictionaries independently. In this work, we apply the proposed separable dictionary learning framework to learn spatial and angular dMRI dictionaries jointly and provide preliminary validation on denoising phantom and real dMRI brain data.

  • structured low rank matrix factorization Global Optimality algorithms and applications
    arXiv: Learning, 2017
    Co-Authors: Benjamin D Haeffele, René Vidal
    Abstract:

    Recently, convex formulations of low-rank matrix factorization problems have received considerable attention in machine learning. However, such formulations often require solving for a matrix of the size of the data matrix, making it challenging to apply them to large scale datasets. Moreover, in many applications the data can display structures beyond simply being low-rank, e.g., images and videos present complex spatio-temporal structures that are largely ignored by standard low-rank methods. In this paper we study a matrix factorization technique that is suitable for large datasets and captures additional structure in the factors by using a particular form of regularization that includes well-known regularizers such as total variation and the nuclear norm as particular cases. Although the resulting optimization problem is non-convex, we show that if the size of the factors is large enough, under certain conditions, any local minimizer for the factors yields a Global minimizer. A few practical algorithms are also provided to solve the matrix factorization problem, and bounds on the distance from a given approximate solution of the optimization problem to the Global optimum are derived. Examples in neural calcium imaging video segmentation and hyperspectral compressed recovery show the advantages of our approach on high-dimensional datasets.

  • Global Optimality in Neural Network Training
    Conference on Computer Vision and Pattern Recognition (CVPR), 2017
    Co-Authors: Benjamin D Haeffele, René Vidal
    Abstract:

    The past few years have seen a dramatic increase in the performance of recognition systems thanks to the introduc-tion of deep networks for representation learning. However, the mathematical reasons for this success remain elusive. A key issue is that the neural network training problem is nonconvex, hence optimization algorithms may not return a Global minima. This paper provides sufficient conditions to guarantee that local minima are Globally optimal and that a local descent strategy can reach a Global minima from any initialization. Our conditions require both the network output and the regularization to be positively homogeneous functions of the network parameters, with the regulariza-tion being designed to control the network size. Our re-sults apply to networks with one hidden layer, where size is measured by the number of neurons in the hidden layer, and multiple deep subnetworks connected in parallel, where size is measured by the number of subnetworks.

  • Global Optimality in tensor factorization deep learning and beyond
    arXiv: Numerical Analysis, 2015
    Co-Authors: Benjamin D Haeffele, René Vidal
    Abstract:

    Techniques involving factorization are found in a wide range of applications and have enjoyed significant empirical success in many fields. However, common to a vast majority of these problems is the significant disadvantage that the associated optimization problems are typically non-convex due to a multilinear form or other convexity destroying transformation. Here we build on ideas from convex relaxations of matrix factorizations and present a very general framework which allows for the analysis of a wide range of non-convex factorization problems - including matrix factorization, tensor factorization, and deep neural network training formulations. We derive sufficient conditions to guarantee that a local minimum of the non-convex optimization problem is a Global minimum and show that if the size of the factorized variables is large enough then from any initialization it is possible to find a Global minimizer using a purely local descent algorithm. Our framework also provides a partial theoretical justification for the increasingly common use of Rectified Linear Units (ReLUs) in deep neural networks and offers guidance on deep network architectures and regularization strategies to facilitate efficient optimization.

Nathan Srebro - One of the best experts on this subject based on the ideXlab platform.

  • Global Optimality of local search for low rank matrix recovery
    Neural Information Processing Systems, 2016
    Co-Authors: Srinadh Bhojanapalli, Behnam Neyshabur, Nathan Srebro
    Abstract:

    We show that there are no spurious local minima in the non-convex factorized parametrization of low-rank matrix recovery from incoherent linear measurements. With noisy measurements we show all local minima are very close to a Global optimum. Together with a curvature bound at saddle points, this yields a polynomial time Global convergence guarantee for stochastic gradient descent from random initialization.

  • Global Optimality of local search for low rank matrix recovery
    arXiv: Machine Learning, 2016
    Co-Authors: Srinadh Bhojanapalli, Behnam Neyshabur, Nathan Srebro
    Abstract:

    We show that there are no spurious local minima in the non-convex factorized parametrization of low-rank matrix recovery from incoherent linear measurements. With noisy measurements we show all local minima are very close to a Global optimum. Together with a curvature bound at saddle points, this yields a polynomial time Global convergence guarantee for stochastic gradient descent {\em from random initialization}.

Benjamin D Haeffele - One of the best experts on this subject based on the ideXlab platform.

  • structured low rank matrix factorization Global Optimality algorithms and applications
    IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020
    Co-Authors: Benjamin D Haeffele, René Vidal
    Abstract:

    Convex formulations of low-rank matrix factorization problems have received considerable attention in machine learning. However, such formulations often require solving for a matrix of the size of the data matrix, making it challenging to apply them to large scale datasets. Moreover, in many applications the data can display structures beyond simply being low-rank, e.g., images and videos present complex spatio-temporal structures that are largely ignored by standard low-rank methods. In this paper we study a matrix factorization technique that is suitable for large datasets and captures additional structure in the factors by using a particular form of regularization that includes well-known regularizers such as total variation and the nuclear norm as particular cases. Although the resulting optimization problem is non-convex, we show that if the size of the factors is large enough, under certain conditions, any local minimizer for the factors yields a Global minimizer. A few practical algorithms are also provided to solve the matrix factorization problem, and bounds on the distance from a given approximate solution of the optimization problem to the Global optimum are derived. Examples in neural calcium imaging video segmentation and hyperspectral compressed recovery show the advantages of our approach on high-dimensional datasets.

  • Global Optimality in separable dictionary learning with applications to the analysis of diffusion mri
    arXiv: Optimization and Control, 2018
    Co-Authors: Evan Schwab, Benjamin D Haeffele, René Vidal, Nicolas Charon
    Abstract:

    Sparse dictionary learning is a popular method for representing signals as linear combinations of a few elements from a dictionary that is learned from the data. In the classical setting, signals are represented as vectors and the dictionary learning problem is posed as a matrix factorization problem where the data matrix is approximately factorized into a dictionary matrix and a sparse matrix of coefficients. However, in many applications in computer vision and medical imaging, signals are better represented as matrices or tensors (e.g. images or videos), where it may be beneficial to exploit the multi-dimensional structure of the data to learn a more compact representation. One such approach is separable dictionary learning, where one learns separate dictionaries for different dimensions of the data. However, typical formulations involve solving a non-convex optimization problem; thus guaranteeing Global Optimality remains a challenge. In this work, we propose a framework that builds upon recent developments in matrix factorization to provide theoretical and numerical guarantees of Global Optimality for separable dictionary learning. We propose an algorithm to find such a Globally optimal solution, which alternates between following local descent steps and checking a certificate for Global Optimality. We illustrate our approach on diffusion magnetic resonance imaging (dMRI) data, a medical imaging modality that measures water diffusion along multiple angular directions in every voxel of an MRI volume. State-of-the-art methods in dMRI either learn dictionaries only for the angular domain of the signals or in some cases learn spatial and angular dictionaries independently. In this work, we apply the proposed separable dictionary learning framework to learn spatial and angular dMRI dictionaries jointly and provide preliminary validation on denoising phantom and real dMRI brain data.

  • structured low rank matrix factorization Global Optimality algorithms and applications
    arXiv: Learning, 2017
    Co-Authors: Benjamin D Haeffele, René Vidal
    Abstract:

    Recently, convex formulations of low-rank matrix factorization problems have received considerable attention in machine learning. However, such formulations often require solving for a matrix of the size of the data matrix, making it challenging to apply them to large scale datasets. Moreover, in many applications the data can display structures beyond simply being low-rank, e.g., images and videos present complex spatio-temporal structures that are largely ignored by standard low-rank methods. In this paper we study a matrix factorization technique that is suitable for large datasets and captures additional structure in the factors by using a particular form of regularization that includes well-known regularizers such as total variation and the nuclear norm as particular cases. Although the resulting optimization problem is non-convex, we show that if the size of the factors is large enough, under certain conditions, any local minimizer for the factors yields a Global minimizer. A few practical algorithms are also provided to solve the matrix factorization problem, and bounds on the distance from a given approximate solution of the optimization problem to the Global optimum are derived. Examples in neural calcium imaging video segmentation and hyperspectral compressed recovery show the advantages of our approach on high-dimensional datasets.

  • Global Optimality in Neural Network Training
    Conference on Computer Vision and Pattern Recognition (CVPR), 2017
    Co-Authors: Benjamin D Haeffele, René Vidal
    Abstract:

    The past few years have seen a dramatic increase in the performance of recognition systems thanks to the introduc-tion of deep networks for representation learning. However, the mathematical reasons for this success remain elusive. A key issue is that the neural network training problem is nonconvex, hence optimization algorithms may not return a Global minima. This paper provides sufficient conditions to guarantee that local minima are Globally optimal and that a local descent strategy can reach a Global minima from any initialization. Our conditions require both the network output and the regularization to be positively homogeneous functions of the network parameters, with the regulariza-tion being designed to control the network size. Our re-sults apply to networks with one hidden layer, where size is measured by the number of neurons in the hidden layer, and multiple deep subnetworks connected in parallel, where size is measured by the number of subnetworks.

  • Global Optimality in tensor factorization deep learning and beyond
    arXiv: Numerical Analysis, 2015
    Co-Authors: Benjamin D Haeffele, René Vidal
    Abstract:

    Techniques involving factorization are found in a wide range of applications and have enjoyed significant empirical success in many fields. However, common to a vast majority of these problems is the significant disadvantage that the associated optimization problems are typically non-convex due to a multilinear form or other convexity destroying transformation. Here we build on ideas from convex relaxations of matrix factorizations and present a very general framework which allows for the analysis of a wide range of non-convex factorization problems - including matrix factorization, tensor factorization, and deep neural network training formulations. We derive sufficient conditions to guarantee that a local minimum of the non-convex optimization problem is a Global minimum and show that if the size of the factorized variables is large enough then from any initialization it is possible to find a Global minimizer using a purely local descent algorithm. Our framework also provides a partial theoretical justification for the increasingly common use of Rectified Linear Units (ReLUs) in deep neural networks and offers guidance on deep network architectures and regularization strategies to facilitate efficient optimization.

Srinadh Bhojanapalli - One of the best experts on this subject based on the ideXlab platform.

  • Global Optimality of local search for low rank matrix recovery
    Neural Information Processing Systems, 2016
    Co-Authors: Srinadh Bhojanapalli, Behnam Neyshabur, Nathan Srebro
    Abstract:

    We show that there are no spurious local minima in the non-convex factorized parametrization of low-rank matrix recovery from incoherent linear measurements. With noisy measurements we show all local minima are very close to a Global optimum. Together with a curvature bound at saddle points, this yields a polynomial time Global convergence guarantee for stochastic gradient descent from random initialization.

  • Global Optimality of local search for low rank matrix recovery
    arXiv: Machine Learning, 2016
    Co-Authors: Srinadh Bhojanapalli, Behnam Neyshabur, Nathan Srebro
    Abstract:

    We show that there are no spurious local minima in the non-convex factorized parametrization of low-rank matrix recovery from incoherent linear measurements. With noisy measurements we show all local minima are very close to a Global optimum. Together with a curvature bound at saddle points, this yields a polynomial time Global convergence guarantee for stochastic gradient descent {\em from random initialization}.

Spyros Chatzivasileiadis - One of the best experts on this subject based on the ideXlab platform.

  • convex relaxations of chance constrained ac optimal power flow
    Power and Energy Society General Meeting, 2018
    Co-Authors: Andreas Venzke, Lejla Halilbasic, Uros Markovic, Gabriela Hug, Spyros Chatzivasileiadis
    Abstract:

    High penetration of renewable energy sources and the increasing share of stochastic loads require the explicit representation of uncertainty in tools such as the optimal power flow (OPF). Current approaches follow either a linearized approach or an iterative approximation of non-linearities. This paper proposes a semidefinite relaxation of a chance-constrained AC-OPF which is able to provide guarantees regarding Global Optimality. Using a piecewise affine policy, we can ensure tractability, accurately model large power deviations, and determine suitable corrective control policies for active power, reactive power, and voltage. We state a tractable formulation for two types of uncertainty sets. Using a scenario-based approach and making no prior assumptions about the probability distribution of the forecast errors, we obtain a robust formulation for a rectangular uncertainty set. Alternatively, assuming a Gaussian distribution of the forecast errors, we propose an analytical reformulation of the chance constraints suitable for semidefinite programming. We demonstrate the performance of our approach on the IEEE 9, 24 and 118 bus system using realistic day-ahead forecast data and obtain tight near-Global Optimality guarantees.

  • convex relaxations of chance constrained ac optimal power flow
    IEEE Transactions on Power Systems, 2018
    Co-Authors: Andreas Venzke, Lejla Halilbasic, Uros Markovic, Gabriela Hug, Spyros Chatzivasileiadis
    Abstract:

    High penetration of renewable energy sources and the increasing share of stochastic loads require the explicit representation of uncertainty in tools such as the optimal power flow (OPF). Current approaches follow either a linearized approach or an iterative approximation of nonlinearities. This paper proposes a semidefinite relaxation of a chance-constrained AC-OPF, which is able to provide guarantees for Global Optimality. Using a piecewise affine policy, we can ensure tractability, accurately model large power deviations, and determine suitable corrective control policies for active power, reactive power, and voltage. We state a tractable formulation for two types of uncertainty sets. Using a scenario-based approach and making no prior assumptions about the probability distribution of the forecast errors, we obtain a robust formulation for a rectangular uncertainty set. Alternatively, assuming a Gaussian distribution of the forecast errors, we propose an analytical reformulation of the chance constraints suitable for semidefinite programming. We demonstrate the performance of our approach on the IEEE 24 and 118 bus system using realistic day-ahead forecast data and obtain tight near-Global Optimality guarantees.