The Experts below are selected from a list of 82185 Experts worldwide ranked by ideXlab platform
Aapo Hyvarinen - One of the best experts on this subject based on the ideXlab platform.
-
Relative Gradient optimization of the jacobian term in unsupervised deep learning
Neural Information Processing Systems, 2020Co-Authors: Luigi Gresele, Giancarlo Fissore, Adrian Javaloy, Bernhard Scholkopf, Aapo HyvarinenAbstract:Learning expressive probabilistic models correctly describing the data is a ubiquitous problem in machine learning. A popular approach for solving it is mapping the observations into a representation space with a simple joint distribution, which can typically be written as a product of its marginals-thus drawing a connection with the field of nonlinear independent component analysis. Deep density models have been widely used for this task, but their maximum likelihood based training requires estimating the log-determinant of the Jacobian and is computationally expensive, thus imposing a trade-off between computation and expressive power. In this work, we propose a new approach for exact training of such neural networks. Based on Relative Gradients, we exploit the matrix structure of neural network parameters to compute updates efficiently even in high-dimensional spaces; the computational cost of the training is quadratic in the input size, in contrast with the cubic scaling of naive approaches. This allows fast training with objective functions involving the log-determinant of the Jacobian, without imposing constraints on its structure, in stark contrast to autoregressive normalizing flows.
-
Relative Gradient optimization of the jacobian term in unsupervised deep learning
arXiv: Machine Learning, 2020Co-Authors: Luigi Gresele, Giancarlo Fissore, Adrian Javaloy, Bernhard Scholkopf, Aapo HyvarinenAbstract:Learning expressive probabilistic models correctly describing the data is a ubiquitous problem in machine learning. A popular approach for solving it is mapping the observations into a representation space with a simple joint distribution, which can typically be written as a product of its marginals -- thus drawing a connection with the field of nonlinear independent component analysis. Deep density models have been widely used for this task, but their likelihood-based training requires estimating the log-determinant of the Jacobian and is computationally expensive, thus imposing a trade-off between computation and expressive power. In this work, we propose a new approach for exact likelihood-based training of such neural networks. Based on Relative Gradients, we exploit the matrix structure of neural network parameters to compute updates efficiently even in high-dimensional spaces; the computational cost of the training is quadratic in the input size, in contrast with the cubic scaling of the naive approaches. This allows fast training with objective functions involving the log-determinant of the Jacobian without imposing constraints on its structure, in stark contrast to normalizing flows. An implementation of our method can be found at this https URL
Norbert J Pelc - One of the best experts on this subject based on the ideXlab platform.
-
generalized reconstruction of phase contrast mri analysis and correction of the effect of Gradient field distortions
Magnetic Resonance in Medicine, 2003Co-Authors: Michael Markl, Roland Bammer, Marcus T Alley, Christopher J Elkins, Mary T Draney, Alan S Barnett, Michael E Moseley, Gary H Glover, Norbert J PelcAbstract:To characterize Gradient field nonuniformity and its effect on velocity encoding in phase contrast (PC) MRI, a generalized model that describes this phenomenon and enables the accurate reconstruction of velocities is presented. In addition to considerable geometric distortions, inhomogeneous Gradient fields can introduce deviations from the nominal Gradient strength and orientation, and therefore spatially-dependent first Gradient moments. Resulting errors in the measured phase shifts used for velocity encoding can therefore cause significant deviations in velocity quantification. The true magnitude and direction of the underlying velocities can be recovered from the phase difference images by a generalized PC velocity reconstruction, which requires the acquisition of full three-directional velocity information. The generalized reconstruction of velocities is applied using a matrix formalism that includes Relative Gradient field deviations derived from a theoretical model of local Gradient field nonuniformity. In addition, an approximate solution for the correction of one-directional velocity encoding is given. Depending on the spatial location of the velocity measurements, errors in velocity magnitude can be as high as 60%, while errors in the velocity encoding direction can be up to 45 degrees. Results of phantom measurements demonstrate that effects of Gradient field nonuniformity on PC-MRI can be corrected with the proposed method.
-
generalized reconstruction of phase contrast mri analysis and correction of the effect of Gradient field distortions
Magnetic Resonance in Medicine, 2003Co-Authors: Michael Markl, Roland Bammer, Marcus T Alley, Christopher J Elkins, Mary T Draney, Alan S Barnett, Michael E Moseley, Gary H Glover, Norbert J PelcAbstract:To characterize Gradient field nonuniformity and its effect on velocity encoding in phase contrast (PC) MRI, a generalized model that describes this phenomenon and enables the accurate reconstruction of velocities is presented. In addition to considerable geometric distortions, inhomogeneous Gradient fields can introduce deviations from the nominal Gradient strength and orientation, and therefore spatially-dependent first Gradient moments. Resulting errors in the measured phase shifts used for velocity encoding can therefore cause significant deviations in velocity quantification. The true magnitude and direction of the underlying velocities can be recovered from the phase difference images by a generalized PC velocity reconstruction, which requires the acquisition of full three-directional velocity information. The generalized reconstruction of velocities is applied using a matrix formalism that includes Relative Gradient field deviations derived from a theoretical model of local Gradient field nonuniformity. In addition, an approximate solution for the correction of one-directional velocity encoding is given. Depending on the spatial location of the velocity measurements, errors in velocity magnitude can be as high as 60%, while errors in the velocity encoding direction can be up to 45°. Results of phantom measurements demonstrate that effects of Gradient field nonuniformity on PC-MRI can be corrected with the proposed method. Magn Reson Med 50:791–801, 2003. Published 2003 Wiley-Liss, Inc.
Eric Moreau - One of the best experts on this subject based on the ideXlab platform.
-
a Relative Gradient algorithm for joint decompositions of complex matrices
European Signal Processing Conference, 2010Co-Authors: Tual Trainini, Eric Moreau, Tulay AdahAbstract:The problem of joint decomposition of sets of complex matrices arises in many problems in signal processing. In this paper, we address the problem for the general case where the matrices can be Hermitian and/or complex symmetric. As such, complete statistical information in the complex domain can be taken into account for the given signal processing problem. The proposed algorithm is based on an optimal step size Relative Gradient approach and computer simulations are provided to illustrate the behavior of this algorithm in different contexts and to establish a comparison with other algorithms.
-
Gradient based joint block diagonalization algorithms application to blind separation of fir convolutive mixtures
Signal Processing, 2010Co-Authors: Hicham Ghennioui, Nadege Thirionmoreau, Eric Moreau, Driss AboutajdineAbstract:This article addresses the problem of the non-unitary joint block diagonalization of a given set of complex matrices. Two new algorithms are provided: the first is based on a classical Gradient approach and the second is based on a Relative Gradient approach. For each algorithm, two versions are provided: the fixed stepsize and the optimal stepsize version. Computer simulations are provided to illustrate the behavior of both algorithms in different contexts. Finally, it is shown that these algorithms enable solving the problem of the blind separation of finite impulse response (FIR) convolutive mixtures of (non-stationary correlated) sources. We focus on methods based on the use of spatial quadratic time-frequency spectra or distributions. The suggested approach main advantage is to enable the elimination of the spatial whitening of the observations which has been proven to establish a bound with regard to the best reachable performances in the blind sources separation context.
-
an optimal step size Relative Gradient based joint diagonalization algorithm
IEEE International Workshop on Computational Advances in Multi-Sensor Adaptive Processing, 2009Co-Authors: Hicham Ghennioui, Nadege Thirionmoreau, Eric MoreauAbstract:This paper addresses the problem of the non unitary joint diagonalization of a given set of complex matrices. We focus on Gradient based algorithms. A new algorithm based on a Relative Gradient approach is suggested. Its algorithmic complexity is established and the optimal stepsize is calculated algebraically at each iteration to decrease the number of iterations required to reach the convergence while discarding the often difficult stepsize choice problem. Computer simulations are provided to illustrate the behavior of this algorithm in different contexts. It is also compared with other existing joint diagonalization algorithms.
-
two new Gradient based non unitary joint block diagonalization algorithms
European Signal Processing Conference, 2008Co-Authors: Hicham Ghennioui, Nadege Thirionmoreau, Eric Moreau, Driss Aboutajdine, Abdellah AdibAbstract:This paper addresses the problem of the non-unitary joint block diagonalization (NU - JBD) of a given set of matrices. Such a problem arises in various fields of applications among which blind separation of convolutive mixtures of sources and array processing for wide-band signals. We present two new algorithms based respectively on (absolute) Gradient and Relative Gradient descendent approaches. The main advantage of the proposed algorithms is that they are more general (the real, positive definite or hermitian assumptions about the matrices belonging to the considered set are no more necessary and the found joint block diagonalizer can be either a unitary or non-unitary matrix). These algorithms also outperform the JBD algorithm based on an optimal step size but “approximate Gradient” approach that we had previously suggested in [12]. In fact, here, the exact calculus of the complex Gradient matrix is performed whereas it was approximated in [12]. Finally, by ensuring the invertibility of the estimated matrix, the Relative Gradient approach makes the proposed NU - JBD algorithm more stable and consequently more robust. Computer simulations are provided in order to illustrate the effectiveness of the proposed approaches in two cases: when exact block-diagonal matrices are considered and when they are perturbed by an additive Gaussian noise. A comparison with the method presented in [12] is also performed, emphasizing the good behavior of the proposed algorithms.
Hicham Ghennioui - One of the best experts on this subject based on the ideXlab platform.
-
Gradient based joint block diagonalization algorithms application to blind separation of fir convolutive mixtures
Signal Processing, 2010Co-Authors: Hicham Ghennioui, Nadege Thirionmoreau, Eric Moreau, Driss AboutajdineAbstract:This article addresses the problem of the non-unitary joint block diagonalization of a given set of complex matrices. Two new algorithms are provided: the first is based on a classical Gradient approach and the second is based on a Relative Gradient approach. For each algorithm, two versions are provided: the fixed stepsize and the optimal stepsize version. Computer simulations are provided to illustrate the behavior of both algorithms in different contexts. Finally, it is shown that these algorithms enable solving the problem of the blind separation of finite impulse response (FIR) convolutive mixtures of (non-stationary correlated) sources. We focus on methods based on the use of spatial quadratic time-frequency spectra or distributions. The suggested approach main advantage is to enable the elimination of the spatial whitening of the observations which has been proven to establish a bound with regard to the best reachable performances in the blind sources separation context.
-
an optimal step size Relative Gradient based joint diagonalization algorithm
IEEE International Workshop on Computational Advances in Multi-Sensor Adaptive Processing, 2009Co-Authors: Hicham Ghennioui, Nadege Thirionmoreau, Eric MoreauAbstract:This paper addresses the problem of the non unitary joint diagonalization of a given set of complex matrices. We focus on Gradient based algorithms. A new algorithm based on a Relative Gradient approach is suggested. Its algorithmic complexity is established and the optimal stepsize is calculated algebraically at each iteration to decrease the number of iterations required to reach the convergence while discarding the often difficult stepsize choice problem. Computer simulations are provided to illustrate the behavior of this algorithm in different contexts. It is also compared with other existing joint diagonalization algorithms.
-
two new Gradient based non unitary joint block diagonalization algorithms
European Signal Processing Conference, 2008Co-Authors: Hicham Ghennioui, Nadege Thirionmoreau, Eric Moreau, Driss Aboutajdine, Abdellah AdibAbstract:This paper addresses the problem of the non-unitary joint block diagonalization (NU - JBD) of a given set of matrices. Such a problem arises in various fields of applications among which blind separation of convolutive mixtures of sources and array processing for wide-band signals. We present two new algorithms based respectively on (absolute) Gradient and Relative Gradient descendent approaches. The main advantage of the proposed algorithms is that they are more general (the real, positive definite or hermitian assumptions about the matrices belonging to the considered set are no more necessary and the found joint block diagonalizer can be either a unitary or non-unitary matrix). These algorithms also outperform the JBD algorithm based on an optimal step size but “approximate Gradient” approach that we had previously suggested in [12]. In fact, here, the exact calculus of the complex Gradient matrix is performed whereas it was approximated in [12]. Finally, by ensuring the invertibility of the estimated matrix, the Relative Gradient approach makes the proposed NU - JBD algorithm more stable and consequently more robust. Computer simulations are provided in order to illustrate the effectiveness of the proposed approaches in two cases: when exact block-diagonal matrices are considered and when they are perturbed by an additive Gaussian noise. A comparison with the method presented in [12] is also performed, emphasizing the good behavior of the proposed algorithms.
Giancarlo Fissore - One of the best experts on this subject based on the ideXlab platform.
-
Relative Gradient optimization of the jacobian term in unsupervised deep learning
Neural Information Processing Systems, 2020Co-Authors: Luigi Gresele, Giancarlo Fissore, Adrian Javaloy, Bernhard Scholkopf, Aapo HyvarinenAbstract:Learning expressive probabilistic models correctly describing the data is a ubiquitous problem in machine learning. A popular approach for solving it is mapping the observations into a representation space with a simple joint distribution, which can typically be written as a product of its marginals-thus drawing a connection with the field of nonlinear independent component analysis. Deep density models have been widely used for this task, but their maximum likelihood based training requires estimating the log-determinant of the Jacobian and is computationally expensive, thus imposing a trade-off between computation and expressive power. In this work, we propose a new approach for exact training of such neural networks. Based on Relative Gradients, we exploit the matrix structure of neural network parameters to compute updates efficiently even in high-dimensional spaces; the computational cost of the training is quadratic in the input size, in contrast with the cubic scaling of naive approaches. This allows fast training with objective functions involving the log-determinant of the Jacobian, without imposing constraints on its structure, in stark contrast to autoregressive normalizing flows.
-
Relative Gradient optimization of the jacobian term in unsupervised deep learning
arXiv: Machine Learning, 2020Co-Authors: Luigi Gresele, Giancarlo Fissore, Adrian Javaloy, Bernhard Scholkopf, Aapo HyvarinenAbstract:Learning expressive probabilistic models correctly describing the data is a ubiquitous problem in machine learning. A popular approach for solving it is mapping the observations into a representation space with a simple joint distribution, which can typically be written as a product of its marginals -- thus drawing a connection with the field of nonlinear independent component analysis. Deep density models have been widely used for this task, but their likelihood-based training requires estimating the log-determinant of the Jacobian and is computationally expensive, thus imposing a trade-off between computation and expressive power. In this work, we propose a new approach for exact likelihood-based training of such neural networks. Based on Relative Gradients, we exploit the matrix structure of neural network parameters to compute updates efficiently even in high-dimensional spaces; the computational cost of the training is quadratic in the input size, in contrast with the cubic scaling of the naive approaches. This allows fast training with objective functions involving the log-determinant of the Jacobian without imposing constraints on its structure, in stark contrast to normalizing flows. An implementation of our method can be found at this https URL