The Experts below are selected from a list of 360 Experts worldwide ranked by ideXlab platform

C C Hang - One of the best experts on this subject based on the ideXlab platform.

  • adaptive Neural network control for strict feedback nonlinear systems using backstepping design
    Automatica, 2000
    Co-Authors: Tao Zhang, C C Hang
    Abstract:

    This paper focuses on adaptive control of strict-feedback nonlinear systems using Multilayer Neural Networks (MNNs). By introducing a modified Lyapunov function, a smooth and singularity-free adaptive controller is firstly designed for a first-order plant. Then, an extension is made to high-order nonlinear systems using Neural network approximation and adaptive backstepping techniques. The developed control scheme guarantees the uniform ultimate boundedness of the closed-loop adaptive systems. In addition, the relationship between the transient performance and the design parameters is explicitly given to guide the tuning of the controller. One important feature of the proposed NN controller is the highly structural property which makes it particularly suitable for parallel processing in actual implementation. Simulation studies are included to illustrate the effectiveness of the proposed approach.

  • adaptive Neural network control for strict feedback nonlinear systems using backstepping design
    American Control Conference, 1999
    Co-Authors: Tao Zhang, C C Hang
    Abstract:

    This paper focuses on the adaptive control problem of strict-feedback nonlinear systems using Multilayer Neural Networks (MNNs). By introducing a modified Lyapunov function, a smooth and singularity-free adaptive controller is first designed for a first-order plant. Then, an extension is made to high-order nonlinear systems using backstepping design. The control scheme developed guarantees the uniform ultimate boundedness of the closed-loop adaptive systems. The relationship between the transient performance and the design parameters is given to guide the tuning of the controller.

  • direct adaptive control of non affine nonlinear systems using Multilayer Neural Networks
    American Control Conference, 1998
    Co-Authors: Tao Zhang, C C Hang
    Abstract:

    A direct adaptive control method is presented for a class of nonaffine nonlinear systems using Multilayer Neural Networks (MNNs). The proposed controller ensures that the output of the system tracks a given bounded reference signal and the output tracking error converges to an /spl epsiv/-neighborhood of zero, while the stability of the closed-loop system is guaranteed by Lyapunov's stability theory.

David Saad - One of the best experts on this subject based on the ideXlab platform.

Sara A Solla - One of the best experts on this subject based on the ideXlab platform.

Daniel Soudry - One of the best experts on this subject based on the ideXlab platform.

  • exponentially vanishing sub optimal local minima in Multilayer Neural Networks
    International Conference on Learning Representations, 2018
    Co-Authors: Daniel Soudry, Elad Hoffer
    Abstract:

    Background: Statistical mechanics results (Dauphin et al. (2014); Choromanska et al. (2015)) suggest that local minima with high error are exponentially rare in high dimensions. However, to prove low error guarantees for Multilayer Neural Networks (MNNs), previous works so far required either a heavily modified MNN model or training method, strong assumptions on the labels (e.g., "near" linear separability), or an unrealistic hidden layer with $\Omega\left(N\right)$ units. Results: We examine a MNN with one hidden layer of piecewise linear units, a single output, and a quadratic loss. We prove that, with high probability in the limit of $N\rightarrow\infty$ datapoints, the volume of differentiable regions of the empiric loss containing sub-optimal differentiable local minima is exponentially vanishing in comparison with the same volume of global minima, given standard normal input of dimension $d_{0}=\tilde{\Omega}\left(\sqrt{N}\right)$, and a more realistic number of $d_{1}=\tilde{\Omega}\left(N/d_{0}\right)$ hidden units. We demonstrate our results numerically: for example, $0\%$ binary classification training error on CIFAR with only $N/d_{0}\approx 16$ hidden neurons.

  • No bad local minima: Data independent training error guarantees for Multilayer Neural Networks
    arXiv: Machine Learning, 2016
    Co-Authors: Daniel Soudry, Yair Carmon
    Abstract:

    We use smoothed analysis techniques to provide guarantees on the training loss of Multilayer Neural Networks (MNNs) at differentiable local minima. Specifically, we examine MNNs with piecewise linear activation functions, quadratic loss and a single output, under mild over-parametrization. We prove that for a MNN with one hidden layer, the training error is zero at every differentiable local minimum, for almost every dataset and dropout-like noise realization. We then extend these results to the case of more than one hidden layer. Our theoretical guarantees assume essentially nothing on the training data, and are verified numerically. These results suggest why the highly non-convex loss of such MNNs can be easily optimized using local updates (e.g., stochastic gradient descent), as observed empirically.

  • training binary Multilayer Neural Networks for image classification using expectation backpropagation
    arXiv: Neural and Evolutionary Computing, 2015
    Co-Authors: Zhiyong Cheng, Daniel Soudry, Zexi Mao, Zhenzhong Lan
    Abstract:

    Compared to Multilayer Neural Networks with real weights, Binary Multilayer Neural Networks (BMNNs) can be implemented more efficiently on dedicated hardware. BMNNs have been demonstrated to be effective on binary classification tasks with Expectation BackPropagation (EBP) algorithm on high dimensional text datasets. In this paper, we investigate the capability of BMNNs using the EBP algorithm on multiclass image classification tasks. The performances of binary Neural Networks with multiple hidden layers and different numbers of hidden units are examined on MNIST. We also explore the effectiveness of image spatial filters and the dropout technique in BMNNs. Experimental results on MNIST dataset show that EBP can obtain 2.12% test error with binary weights and 1.66% test error with real weights, which is comparable to the results of standard BackPropagation algorithm on fully connected MNNs.

  • Memristor-Based Multilayer Neural Networks With Online Gradient Descent Training
    IEEE Transactions on Neural Networks and Learning Systems, 2015
    Co-Authors: Daniel Soudry, Asaf Gal, Dotan Di Castro, Avinoam Kolodny, Shahar Kvatinsky
    Abstract:

    Learning in Multilayer Neural Networks (MNNs) relies on continuous updating of large matrices of synaptic weights by local rules. Such locality can be exploited for massive parallelism when implementing MNNs in hardware. However, these update rules require a multiply and accumulate operation for each synaptic weight, which is challenging to implement compactly using CMOS. In this paper, a method for performing these update operations simultaneously (incremental outer products) using memristor-based arrays is proposed. The method is based on the fact that, approximately, given a voltage pulse, the conductivity of a memristor will increment proportionally to the pulse duration multiplied by the pulse magnitude if the increment is sufficiently small. The proposed method uses a synaptic circuit composed of a small number of components per synapse: one memristor and two CMOS transistors. This circuit is expected to consume between 2% and 8% of the area and static power of previous CMOS-only hardware alternatives. Such a circuit can compactly implement hardware MNNs trainable by scalable algorithms based on online gradient descent (e.g., backpropagation). The utility and robustness of the proposed memristor-based circuit are demonstrated on standard supervised learning tasks.

  • expectation backpropagation parameter free training of Multilayer Neural Networks with continuous or discrete weights
    Neural Information Processing Systems, 2014
    Co-Authors: Daniel Soudry, Itay Hubara, Ron Meir
    Abstract:

    Multilayer Neural Networks (MNNs) are commonly trained using gradient descent-based methods, such as BackPropagation (BP). Inference in probabilistic graphical models is often done using variational Bayes methods, such as Expectation Propagation (EP). We show how an EP based approach can also be used to train deterministic MNNs. Specifically, we approximate the posterior of the weights given the data using a "mean-field" factorized distribution, in an online setting. Using online EP and the central limit theorem we find an analytical approximation to the Bayes update of this posterior, as well as the resulting Bayes estimates of the weights and outputs. Despite a different origin, the resulting algorithm, Expectation BackPropagation (EBP), is very similar to BP in form and efficiency. However, it has several additional advantages: (1) Training is parameter-free, given initial conditions (prior) and the MNN architecture. This is useful for large-scale problems, where parameter tuning is a major challenge. (2) The weights can be restricted to have discrete values. This is especially useful for implementing trained MNNs in precision limited hardware chips, thus improving their speed and energy efficiency by several orders of magnitude. We test the EBP algorithm numerically in eight binary text classification tasks. In all tasks, EBP outperforms: (1) standard BP with the optimal constant learning rate (2) previously reported state of the art. Interestingly, EBP-trained MNNs with binary weights usually perform better than MNNs with continuous (real) weights - if we average the MNN output using the inferred posterior.

Tao Zhang - One of the best experts on this subject based on the ideXlab platform.

  • adaptive Neural network control for strict feedback nonlinear systems using backstepping design
    Automatica, 2000
    Co-Authors: Tao Zhang, C C Hang
    Abstract:

    This paper focuses on adaptive control of strict-feedback nonlinear systems using Multilayer Neural Networks (MNNs). By introducing a modified Lyapunov function, a smooth and singularity-free adaptive controller is firstly designed for a first-order plant. Then, an extension is made to high-order nonlinear systems using Neural network approximation and adaptive backstepping techniques. The developed control scheme guarantees the uniform ultimate boundedness of the closed-loop adaptive systems. In addition, the relationship between the transient performance and the design parameters is explicitly given to guide the tuning of the controller. One important feature of the proposed NN controller is the highly structural property which makes it particularly suitable for parallel processing in actual implementation. Simulation studies are included to illustrate the effectiveness of the proposed approach.

  • adaptive Neural network control for strict feedback nonlinear systems using backstepping design
    American Control Conference, 1999
    Co-Authors: Tao Zhang, C C Hang
    Abstract:

    This paper focuses on the adaptive control problem of strict-feedback nonlinear systems using Multilayer Neural Networks (MNNs). By introducing a modified Lyapunov function, a smooth and singularity-free adaptive controller is first designed for a first-order plant. Then, an extension is made to high-order nonlinear systems using backstepping design. The control scheme developed guarantees the uniform ultimate boundedness of the closed-loop adaptive systems. The relationship between the transient performance and the design parameters is given to guide the tuning of the controller.

  • direct adaptive control of non affine nonlinear systems using Multilayer Neural Networks
    American Control Conference, 1998
    Co-Authors: Tao Zhang, C C Hang
    Abstract:

    A direct adaptive control method is presented for a class of nonaffine nonlinear systems using Multilayer Neural Networks (MNNs). The proposed controller ensures that the output of the system tracks a given bounded reference signal and the output tracking error converges to an /spl epsiv/-neighborhood of zero, while the stability of the closed-loop system is guaranteed by Lyapunov's stability theory.