The Experts below are selected from a list of 7935 Experts worldwide ranked by ideXlab platform

José Ragot - One of the best experts on this subject based on the ideXlab platform.

  • observer for Lipschitz nonlinear systems mean value theorem and sector nonlinearity transformation
    IEEE Multi-Conference on Systems and Control MSC 2012, 2012
    Co-Authors: Dalil Ichalal, Said Mammar, Benoît Marx, Didier Maquin, José Ragot
    Abstract:

    In this paper, the problem of observer design for nonlinear Lipschitz systems is treated. An emphasis is put on maximizing the admissible Lipschitz Constant for which the observer design is possible. This problem is tackled using a Takagi-Sugeno modeling approach. The idea is to re-write the state estimation error dynamics as an autonomous Takagi- Sugeno system, using the Mean Value Theorem and the sector nonlinearity transformation. The stability of the state estimation error is studied with the Lyapunov theory by using a non- quadratic Lyapunov function and by computing its variation between m consecutive samples. The interest of these manip- ulations is to obtain LMI conditions admitting solutions for large values of the Lipschitz Constant. Finally, two examples are provided in order to hilight the performances of the proposed approach.

  • Observer for Lipschitz nonlinear systems: Mean Value Theorem and sector nonlinearity transformation
    2012 IEEE International Symposium on Intelligent Control, 2012
    Co-Authors: Dalil Ichalal, Said Mammar, Benoît Marx, Didier Maquin, José Ragot
    Abstract:

    The problem of observer design for nonlinear Lipschitz systems is dealt with in this work. An emphasis is put on the maximization of the admissible Lipschitz Constant for which the observer design is possible. This problem is tackled using a Takagi-Sugeno modeling approach. The idea is to rewrite the state estimation error dynamics as an autonomous Takagi-Sugeno system, using the Mean Value Theorem and the sector nonlinearity transformation. State estimation error dynamics stability is studied with the Lyapunov theory by choosing a non-quadratic Lyapunov function and by computing its variation between m consecutive samples. The interest of these manipulations is to obtain LMI conditions admitting solutions for large values of the Lipschitz Constant. Finally, illustrative examples are provided in order to highlight the performances of the proposed approach.

Michael Unser - One of the best experts on this subject based on the ideXlab platform.

  • deep neural networks with trainable activations and controlled Lipschitz Constant
    arXiv: Learning, 2020
    Co-Authors: Shayan Aziznejad, Harshit Gupta, Joaquim Campos, Michael Unser
    Abstract:

    We introduce a variational framework to learn the activation functions of deep neural networks. Our aim is to increase the capacity of the network while controlling an upper-bound of the actual Lipschitz Constant of the input-output relation. To that end, we first establish a global bound for the Lipschitz Constant of neural networks. Based on the obtained bound, we then formulate a variational problem for learning activation functions. Our variational problem is infinite-dimensional and is not computationally tractable. However, we prove that there always exists a solution that has continuous and piecewise-linear (linear-spline) activations. This reduces the original problem to a finite-dimensional minimization where an l1 penalty on the parameters of the activations favors the learning of sparse nonlinearities. We numerically compare our scheme with standard ReLU network and its variations, PReLU and LeakyReLU and we empirically demonstrate the practical aspects of our framework.

  • deep neural networks with trainable activations and controlled Lipschitz Constant
    IEEE Transactions on Signal Processing, 2020
    Co-Authors: Shayan Aziznejad, Harshit Gupta, Joaquim Campos, Michael Unser
    Abstract:

    We introduce a variational framework to learn the activation functions of deep neural networks. Our aim is to increase the capacity of the network while controlling an upper-bound of the actual Lipschitz Constant of the input-output relation. To that end, we first establish a global bound for the Lipschitz Constant of neural networks. Based on the obtained bound, we then formulate a variational problem for learning activation functions. Our variational problem is infinite-dimensional and is not computationally tractable. However, we prove that there always exists a solution that has continuous and piecewise-linear (linear-spline) activations. This reduces the original problem to a finite-dimensional minimization where an $\ell _1$ penalty on the parameters of the activations favors the learning of sparse nonlinearities. We numerically compare our scheme with standard ReLU network and its variations, PReLU and LeakyReLU and we empirically demonstrate the practical aspects of our framework.

Dalil Ichalal - One of the best experts on this subject based on the ideXlab platform.

  • observer for Lipschitz nonlinear systems mean value theorem and sector nonlinearity transformation
    IEEE Multi-Conference on Systems and Control MSC 2012, 2012
    Co-Authors: Dalil Ichalal, Said Mammar, Benoît Marx, Didier Maquin, José Ragot
    Abstract:

    In this paper, the problem of observer design for nonlinear Lipschitz systems is treated. An emphasis is put on maximizing the admissible Lipschitz Constant for which the observer design is possible. This problem is tackled using a Takagi-Sugeno modeling approach. The idea is to re-write the state estimation error dynamics as an autonomous Takagi- Sugeno system, using the Mean Value Theorem and the sector nonlinearity transformation. The stability of the state estimation error is studied with the Lyapunov theory by using a non- quadratic Lyapunov function and by computing its variation between m consecutive samples. The interest of these manip- ulations is to obtain LMI conditions admitting solutions for large values of the Lipschitz Constant. Finally, two examples are provided in order to hilight the performances of the proposed approach.

  • Observer for Lipschitz nonlinear systems: Mean Value Theorem and sector nonlinearity transformation
    2012 IEEE International Symposium on Intelligent Control, 2012
    Co-Authors: Dalil Ichalal, Said Mammar, Benoît Marx, Didier Maquin, José Ragot
    Abstract:

    The problem of observer design for nonlinear Lipschitz systems is dealt with in this work. An emphasis is put on the maximization of the admissible Lipschitz Constant for which the observer design is possible. This problem is tackled using a Takagi-Sugeno modeling approach. The idea is to rewrite the state estimation error dynamics as an autonomous Takagi-Sugeno system, using the Mean Value Theorem and the sector nonlinearity transformation. State estimation error dynamics stability is studied with the Lyapunov theory by choosing a non-quadratic Lyapunov function and by computing its variation between m consecutive samples. The interest of these manipulations is to obtain LMI conditions admitting solutions for large values of the Lipschitz Constant. Finally, illustrative examples are provided in order to highlight the performances of the proposed approach.

Srinivasa G Raghavan - One of the best experts on this subject based on the ideXlab platform.

  • lipbab computing exact Lipschitz Constant of relu networks
    International Conference on Artificial Neural Networks, 2021
    Co-Authors: Aritra Bhowmick, Meenakshi Dsouza, Srinivasa G Raghavan
    Abstract:

    The Lipschitz Constant of neural networks plays an important role in several contexts of deep learning ranging from robustness certification and regularization to stability analysis of systems with neural network controllers. Obtaining tight bounds of the Lipschitz Constant is therefore important. We introduce LipBaB, a branch and bound framework to compute certified bounds of the local Lipschitz Constant of deep neural networks with ReLU activation functions up to any desired precision. It is based on iteratively upper-bounding the norm of the Jacobians, corresponding to different activation patterns of the network caused within the input domain. Our algorithm can provide provably exact computation of the Lipschitz Constant for any p-norm.

  • lipbab computing exact Lipschitz Constant of relu networks
    arXiv: Learning, 2021
    Co-Authors: Aritra Bhowmick, Meenakshi Dsouza, Srinivasa G Raghavan
    Abstract:

    The Lipschitz Constant of neural networks plays an important role in several contexts of deep learning ranging from robustness certification and regularization to stability analysis of systems with neural network controllers. Obtaining tight bounds of the Lipschitz Constant is therefore important. We introduce LipBaB, a branch and bound framework to compute certified bounds of the local Lipschitz Constant of deep neural networks with ReLU activation functions up to any desired precision. We achieve this by bounding the norm of the Jacobians, corresponding to different activation patterns of the network caused within the input domain. Our algorithm can provide provably exact computation of the Lipschitz Constant for any p-norm.

Sebastien Bubeck - One of the best experts on this subject based on the ideXlab platform.

  • a law of robustness for two layers neural networks
    arXiv: Learning, 2021
    Co-Authors: Sebastien Bubeck, Dheeraj Nagaraj
    Abstract:

    We initiate the study of the inherent tradeoffs between the size of a neural network and its robustness, as measured by its Lipschitz Constant. We make a precise conjecture that, for any Lipschitz activation function and for most datasets, any two-layers neural network with k neurons that perfectly fit the data must have its Lipschitz Constant larger (up to a Constant) than n/k where n is the number of datapoints. In particular, this conjecture implies that overparametrization is necessary for robustness, since it means that one needs roughly one neuron per datapoint to ensure a O(1)-Lipschitz network, while mere data fitting of d-dimensional data requires only one neuron per d datapoints. We prove a weaker version of this conjecture when the Lipschitz Constant is replaced by an upper bound on it based on the spectral norm of the weight matrix. We also prove the conjecture in the high-dimensional regime n≈d (which we also refer to as the undercomplete case, since only k≤d is relevant here). Finally we prove the conjecture for polynomial activation functions of degree p when n≈dp. We complement these findings with experimental evidence supporting the conjecture.

  • Lipschitz bandits without the Lipschitz Constant
    Algorithmic Learning Theory, 2011
    Co-Authors: Sebastien Bubeck, Gilles Stoltz
    Abstract:

    We consider the setting of stochastic bandit problems with a continuum of arms indexed by [0, 1]d. We first point out that the strategies considered so far in the literature only provided theoretical guarantees of the form: given some tuning parameters, the regret is small with respect to a class of environments that depends on these parameters. This is however not the right perspective, as it is the strategy that should adapt to the specific bandit environment at hand, and not the other way round. Put differently, an adaptation issue is raised. We solve it for the special case of environments whose mean-payoff functions are globally Lipschitz. More precisely, we show that the minimax optimal orders of magnitude Ld/(d+2) T(d+1)/(d+2) of the regret bound over T time instances against an environment whose mean-payoff function f is Lipschitz with Constant L can be achieved without knowing L or T in advance. This is in contrast to all previously known strategies, which require to some extent the knowledge of L to achieve this performance guarantee.

  • Lipschitz bandits without the Lipschitz Constant
    arXiv: Statistics Theory, 2011
    Co-Authors: Sebastien Bubeck, Gilles Stoltz
    Abstract:

    We consider the setting of stochastic bandit problems with a continuum of arms. We first point out that the strategies considered so far in the literature only provided theoretical guarantees of the form: given some tuning parameters, the regret is small with respect to a class of environments that depends on these parameters. This is however not the right perspective, as it is the strategy that should adapt to the specific bandit environment at hand, and not the other way round. Put differently, an adaptation issue is raised. We solve it for the special case of environments whose mean-payoff functions are globally Lipschitz. More precisely, we show that the minimax optimal orders of magnitude $L^{d/(d+2)} \, T^{(d+1)/(d+2)}$ of the regret bound against an environment $f$ with Lipschitz Constant $L$ over $T$ time instances can be achieved without knowing $L$ or $T$ in advance. This is in contrast to all previously known strategies, which require to some extent the knowledge of $L$ to achieve this performance guarantee.