The Experts below are selected from a list of 176313 Experts worldwide ranked by ideXlab platform
Jonathan D Hauenstein - One of the best experts on this subject based on the ideXlab platform.
-
the loss surface of deep Linear Networks viewed through the algebraic geometry lens
IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021Co-Authors: Dhagash Mehta, Tianran Chen, Tingting Tang, Jonathan D HauensteinAbstract:By using the viewpoint of modern computational algebraic geometry, we explore properties of the optimization landscapes of the deep Linear neural network models. After clarifying on the various definitions of "flat" minima, we show that the geometrically flat minima, which are merely artifacts of residual continuous symmetries of the deep Linear Networks, can be straightforwardly removed by a generalized L_2 regularization. Then, we establish upper bounds on the number of isolated stationary points of these Networks with the help of algebraic geometry. Using these upper bounds and utilizing a numerical algebraic geometry method, we find all stationary points for modest depth and matrix size. We show that in the presence of the non-zero regularization, deep Linear Networks indeed possess local minima which are not the global minima. We show that though the number of stationary points increases as the number of neurons (regularization parameter) increases (decreases), the number of higher index saddles are surprisingly rare.
-
the loss surface of deep Linear Networks viewed through the algebraic geometry lens
arXiv: Machine Learning, 2018Co-Authors: Dhagash Mehta, Tianran Chen, Tingting Tang, Jonathan D HauensteinAbstract:By using the viewpoint of modern computational algebraic geometry, we explore properties of the optimization landscapes of the deep Linear neural network models. After clarifying on the various definitions of "flat" minima, we show that the geometrically flat minima, which are merely artifacts of residual continuous symmetries of the deep Linear Networks, can be straightforwardly removed by a generalized $L_2$ regularization. Then, we establish upper bounds on the number of isolated stationary points of these Networks with the help of algebraic geometry. Using these upper bounds and utilizing a numerical algebraic geometry method, we find all stationary points of modest depth and matrix size. We show that in the presence of the non-zero regularization, deep Linear Networks indeed possess local minima which are not the global minima. Our computational results clarify certain aspects of the loss surfaces of deep Linear Networks and provide novel insights.
Dhagash Mehta - One of the best experts on this subject based on the ideXlab platform.
-
the loss surface of deep Linear Networks viewed through the algebraic geometry lens
IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021Co-Authors: Dhagash Mehta, Tianran Chen, Tingting Tang, Jonathan D HauensteinAbstract:By using the viewpoint of modern computational algebraic geometry, we explore properties of the optimization landscapes of the deep Linear neural network models. After clarifying on the various definitions of "flat" minima, we show that the geometrically flat minima, which are merely artifacts of residual continuous symmetries of the deep Linear Networks, can be straightforwardly removed by a generalized L_2 regularization. Then, we establish upper bounds on the number of isolated stationary points of these Networks with the help of algebraic geometry. Using these upper bounds and utilizing a numerical algebraic geometry method, we find all stationary points for modest depth and matrix size. We show that in the presence of the non-zero regularization, deep Linear Networks indeed possess local minima which are not the global minima. We show that though the number of stationary points increases as the number of neurons (regularization parameter) increases (decreases), the number of higher index saddles are surprisingly rare.
-
the loss surface of deep Linear Networks viewed through the algebraic geometry lens
arXiv: Machine Learning, 2018Co-Authors: Dhagash Mehta, Tianran Chen, Tingting Tang, Jonathan D HauensteinAbstract:By using the viewpoint of modern computational algebraic geometry, we explore properties of the optimization landscapes of the deep Linear neural network models. After clarifying on the various definitions of "flat" minima, we show that the geometrically flat minima, which are merely artifacts of residual continuous symmetries of the deep Linear Networks, can be straightforwardly removed by a generalized $L_2$ regularization. Then, we establish upper bounds on the number of isolated stationary points of these Networks with the help of algebraic geometry. Using these upper bounds and utilizing a numerical algebraic geometry method, we find all stationary points of modest depth and matrix size. We show that in the presence of the non-zero regularization, deep Linear Networks indeed possess local minima which are not the global minima. Our computational results clarify certain aspects of the loss surfaces of deep Linear Networks and provide novel insights.
Katherine Morrison - One of the best experts on this subject based on the ideXlab platform.
-
fixed points of competitive threshold Linear Networks
Neural Computation, 2019Co-Authors: Carina Curto, Jesse Geneson, Katherine MorrisonAbstract:Threshold-Linear Networks (TLNs) are models of neural Networks that consist of simple, perceptron-like neurons and exhibit nonLinear dynamics determined by the network's connectivity. The fixed poi...
-
fixed points of competitive threshold Linear Networks
arXiv: Neurons and Cognition, 2018Co-Authors: Carina Curto, Jesse Geneson, Katherine MorrisonAbstract:Threshold-Linear Networks (TLNs) are models of neural Networks that consist of simple, perceptron-like neurons and exhibit nonLinear dynamics that are determined by the network's connectivity. The fixed points of a TLN, including both stable and unstable equilibria, play a critical role in shaping its emergent dynamics. In this work, we provide two novel characterizations for the set of fixed points of a competitive TLN: the first is in terms of a simple sign condition, while the second relies on the concept of domination. We apply these results to a special family of TLNs, called combinatorial threshold-Linear Networks (CTLNs), whose connectivity matrices are defined from directed graphs. This leads us to prove a series of graph rules that enable one to determine fixed points of a CTLN by analyzing the underlying graph. Additionally, we study larger Networks composed of smaller "building block" subNetworks, and prove several theorems relating the fixed points of the full network to those of its components. Our results provide the foundation for a kind of "graphical calculus" to infer features of the dynamics from a network's connectivity.
Tingting Tang - One of the best experts on this subject based on the ideXlab platform.
-
the loss surface of deep Linear Networks viewed through the algebraic geometry lens
IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021Co-Authors: Dhagash Mehta, Tianran Chen, Tingting Tang, Jonathan D HauensteinAbstract:By using the viewpoint of modern computational algebraic geometry, we explore properties of the optimization landscapes of the deep Linear neural network models. After clarifying on the various definitions of "flat" minima, we show that the geometrically flat minima, which are merely artifacts of residual continuous symmetries of the deep Linear Networks, can be straightforwardly removed by a generalized L_2 regularization. Then, we establish upper bounds on the number of isolated stationary points of these Networks with the help of algebraic geometry. Using these upper bounds and utilizing a numerical algebraic geometry method, we find all stationary points for modest depth and matrix size. We show that in the presence of the non-zero regularization, deep Linear Networks indeed possess local minima which are not the global minima. We show that though the number of stationary points increases as the number of neurons (regularization parameter) increases (decreases), the number of higher index saddles are surprisingly rare.
-
the loss surface of deep Linear Networks viewed through the algebraic geometry lens
arXiv: Machine Learning, 2018Co-Authors: Dhagash Mehta, Tianran Chen, Tingting Tang, Jonathan D HauensteinAbstract:By using the viewpoint of modern computational algebraic geometry, we explore properties of the optimization landscapes of the deep Linear neural network models. After clarifying on the various definitions of "flat" minima, we show that the geometrically flat minima, which are merely artifacts of residual continuous symmetries of the deep Linear Networks, can be straightforwardly removed by a generalized $L_2$ regularization. Then, we establish upper bounds on the number of isolated stationary points of these Networks with the help of algebraic geometry. Using these upper bounds and utilizing a numerical algebraic geometry method, we find all stationary points of modest depth and matrix size. We show that in the presence of the non-zero regularization, deep Linear Networks indeed possess local minima which are not the global minima. Our computational results clarify certain aspects of the loss surfaces of deep Linear Networks and provide novel insights.
Tianran Chen - One of the best experts on this subject based on the ideXlab platform.
-
the loss surface of deep Linear Networks viewed through the algebraic geometry lens
IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021Co-Authors: Dhagash Mehta, Tianran Chen, Tingting Tang, Jonathan D HauensteinAbstract:By using the viewpoint of modern computational algebraic geometry, we explore properties of the optimization landscapes of the deep Linear neural network models. After clarifying on the various definitions of "flat" minima, we show that the geometrically flat minima, which are merely artifacts of residual continuous symmetries of the deep Linear Networks, can be straightforwardly removed by a generalized L_2 regularization. Then, we establish upper bounds on the number of isolated stationary points of these Networks with the help of algebraic geometry. Using these upper bounds and utilizing a numerical algebraic geometry method, we find all stationary points for modest depth and matrix size. We show that in the presence of the non-zero regularization, deep Linear Networks indeed possess local minima which are not the global minima. We show that though the number of stationary points increases as the number of neurons (regularization parameter) increases (decreases), the number of higher index saddles are surprisingly rare.
-
the loss surface of deep Linear Networks viewed through the algebraic geometry lens
arXiv: Machine Learning, 2018Co-Authors: Dhagash Mehta, Tianran Chen, Tingting Tang, Jonathan D HauensteinAbstract:By using the viewpoint of modern computational algebraic geometry, we explore properties of the optimization landscapes of the deep Linear neural network models. After clarifying on the various definitions of "flat" minima, we show that the geometrically flat minima, which are merely artifacts of residual continuous symmetries of the deep Linear Networks, can be straightforwardly removed by a generalized $L_2$ regularization. Then, we establish upper bounds on the number of isolated stationary points of these Networks with the help of algebraic geometry. Using these upper bounds and utilizing a numerical algebraic geometry method, we find all stationary points of modest depth and matrix size. We show that in the presence of the non-zero regularization, deep Linear Networks indeed possess local minima which are not the global minima. Our computational results clarify certain aspects of the loss surfaces of deep Linear Networks and provide novel insights.