The Experts below are selected from a list of 11832 Experts worldwide ranked by ideXlab platform

Herman Kamper - One of the best experts on this subject based on the ideXlab platform.

  • critical Initialisation for deep signal propagation in noisy rectifier neural networks
    arXiv: Machine Learning, 2018
    Co-Authors: Arnu Pretorius, Elan Van Biljon, Steve Kroon, Herman Kamper
    Abstract:

    Stochastic regularisation is an important weapon in the arsenal of a deep learning practitioner. However, despite recent theoretical advances, our understanding of how noise influences signal propagation in deep neural networks remains limited. By extending recent work based on mean field theory, we develop a new framework for signal propagation in stochastic regularised neural networks. Our noisy signal propagation theory can incorporate several common noise distributions, including additive and multiplicative Gaussian noise as well as dropout. We use this framework to investigate Initialisation strategies for noisy ReLU networks. We show that no critical Initialisation strategy exists using additive noise, with signal propagation exploding regardless of the selected noise distribution. For multiplicative noise (e.g. dropout), we identify alternative critical Initialisation strategies that depend on the second moment of the noise distribution. Simulations and experiments on real-world data confirm that our proposed Initialisation is able to stably propagate signals in deep networks, while using an Initialisation disregarding noise fails to do so. Furthermore, we analyse correlation dynamics between inputs. Stronger noise regularisation is shown to reduce the depth to which discriminatory information about the inputs to a noisy ReLU network is able to propagate, even when initialised at criticality. We support our theoretical predictions for these trainable depths with simulations, as well as with experiments on MNIST and CIFAR-10

  • critical Initialisation for deep signal propagation in noisy rectifier neural networks
    Neural Information Processing Systems, 2018
    Co-Authors: Arnu Pretorius, Elan Van Biljon, Steve Kroon, Herman Kamper
    Abstract:

    Stochastic regularisation is an important weapon in the arsenal of a deep learning practitioner. However, despite recent theoretical advances, our understanding of how noise influences signal propagation in deep neural networks remains limited. By extending recent work based on mean field theory, we develop a new framework for signal propagation in stochastic regularised neural networks. Our \textit{noisy signal propagation} theory can incorporate several common noise distributions, including additive and multiplicative Gaussian noise as well as dropout. We use this framework to investigate Initialisation strategies for noisy ReLU networks. We show that no critical Initialisation strategy exists using additive noise, with signal propagation exploding regardless of the selected noise distribution. For multiplicative noise (e.g.\ dropout), we identify alternative critical Initialisation strategies that depend on the second moment of the noise distribution. Simulations and experiments on real-world data confirm that our proposed Initialisation is able to stably propagate signals in deep networks, while using an Initialisation disregarding noise fails to do so. Furthermore, we analyse correlation dynamics between inputs. Stronger noise regularisation is shown to reduce the depth to which discriminatory information about the inputs to a noisy ReLU network is able to propagate, even when initialised at criticality. We support our theoretical predictions for these trainable depths with simulations, as well as with experiments on MNIST and CIFAR-10.

Arnu Pretorius - One of the best experts on this subject based on the ideXlab platform.

  • critical Initialisation for deep signal propagation in noisy rectifier neural networks
    arXiv: Machine Learning, 2018
    Co-Authors: Arnu Pretorius, Elan Van Biljon, Steve Kroon, Herman Kamper
    Abstract:

    Stochastic regularisation is an important weapon in the arsenal of a deep learning practitioner. However, despite recent theoretical advances, our understanding of how noise influences signal propagation in deep neural networks remains limited. By extending recent work based on mean field theory, we develop a new framework for signal propagation in stochastic regularised neural networks. Our noisy signal propagation theory can incorporate several common noise distributions, including additive and multiplicative Gaussian noise as well as dropout. We use this framework to investigate Initialisation strategies for noisy ReLU networks. We show that no critical Initialisation strategy exists using additive noise, with signal propagation exploding regardless of the selected noise distribution. For multiplicative noise (e.g. dropout), we identify alternative critical Initialisation strategies that depend on the second moment of the noise distribution. Simulations and experiments on real-world data confirm that our proposed Initialisation is able to stably propagate signals in deep networks, while using an Initialisation disregarding noise fails to do so. Furthermore, we analyse correlation dynamics between inputs. Stronger noise regularisation is shown to reduce the depth to which discriminatory information about the inputs to a noisy ReLU network is able to propagate, even when initialised at criticality. We support our theoretical predictions for these trainable depths with simulations, as well as with experiments on MNIST and CIFAR-10

  • critical Initialisation for deep signal propagation in noisy rectifier neural networks
    Neural Information Processing Systems, 2018
    Co-Authors: Arnu Pretorius, Elan Van Biljon, Steve Kroon, Herman Kamper
    Abstract:

    Stochastic regularisation is an important weapon in the arsenal of a deep learning practitioner. However, despite recent theoretical advances, our understanding of how noise influences signal propagation in deep neural networks remains limited. By extending recent work based on mean field theory, we develop a new framework for signal propagation in stochastic regularised neural networks. Our \textit{noisy signal propagation} theory can incorporate several common noise distributions, including additive and multiplicative Gaussian noise as well as dropout. We use this framework to investigate Initialisation strategies for noisy ReLU networks. We show that no critical Initialisation strategy exists using additive noise, with signal propagation exploding regardless of the selected noise distribution. For multiplicative noise (e.g.\ dropout), we identify alternative critical Initialisation strategies that depend on the second moment of the noise distribution. Simulations and experiments on real-world data confirm that our proposed Initialisation is able to stably propagate signals in deep networks, while using an Initialisation disregarding noise fails to do so. Furthermore, we analyse correlation dynamics between inputs. Stronger noise regularisation is shown to reduce the depth to which discriminatory information about the inputs to a noisy ReLU network is able to propagate, even when initialised at criticality. We support our theoretical predictions for these trainable depths with simulations, as well as with experiments on MNIST and CIFAR-10.

Francisco J. Doblas-reyes - One of the best experts on this subject based on the ideXlab platform.

  • Comparison of full field and anomaly Initialisation for decadal climate prediction: towards an optimal consistency between the ocean and sea-ice anomaly Initialisation state
    Climate Dynamics, 2017
    Co-Authors: Danila Volpi, Virginie Guemas, Francisco J. Doblas-reyes
    Abstract:

    Decadal prediction exploits sources of predictability from both the internal variability through the Initialisation of the climate model from observational estimates, and the external radiative forcings. When a model is initialised with the observed state at the initial time step (Full Field Initialisation—FFI), the forecast run drifts towards the biased model climate. Distinguishing between the climate signal to be predicted and the model drift is a challenging task, because the application of a-posteriori bias correction has the risk of removing part of the variability signal. The anomaly Initialisation (AI) technique aims at addressing the drift issue by answering the following question: if the model is allowed to start close to its own attractor (i.e. its biased world), but the phase of the simulated variability is constrained toward the contemporaneous observed one at the Initialisation time, does the prediction skill improve? The relative merits of the FFI and AI techniques applied respectively to the ocean component and the ocean and sea ice components simultaneously in the EC-Earth global coupled model are assessed. For both strategies the initialised hindcasts show better skill than historical simulations for the ocean heat content and AMOC along the first two forecast years, for sea ice and PDO along the first forecast year, while for AMO the improvements are statistically significant for the first two forecast years. The AI in the ocean and sea ice components significantly improves the skill of the Arctic sea surface temperature over the FFI.

L B White - One of the best experts on this subject based on the ideXlab platform.

  • critical Initialisation in continuous approximations of binary neural networks
    International Conference on Learning Representations, 2020
    Co-Authors: George Stamatescu, Federica Gerace, Carlo Lucibello, Ian Fuss, L B White
    Abstract:

    The training of stochastic neural network models with binary ($\pm1$) weights and activations via continuous surrogate networks is investigated. We derive, using mean field theory, a set of scalar equations describing how input signals propagate through surrogate networks. The equations reveal that depending on the choice of surrogate model, the networks may or may not exhibit an order to chaos transition, and the presence of depth scales that limit the maximum trainable depth. Specifically, in solving the equations for edge of chaos conditions, we show that surrogates derived using the Gaussian local reparameterisation trick have no critical Initialisation, whereas a deterministic surrogates based on analytic Gaussian integration do. The theory is applied to a range of binary neuron and weight design choices, such as different neuron noise models, allowing the categorisation of algorithms in terms of their behaviour at Initialisation. Moreover, we predict theoretically and confirm numerically, that common weight initialization schemes used in standard continuous networks, when applied to the mean values of the stochastic binary weights, yield poor training performance. This study shows that, contrary to common intuition, the means of the stochastic binary weights should be initialised close to close to $\pm 1$ for deeper networks to be trainable.

  • critical Initialisation in continuous approximations of binary neural networks
    arXiv: Machine Learning, 2019
    Co-Authors: George Stamatescu, Federica Gerace, Carlo Lucibello, Ian Fuss, L B White
    Abstract:

    The training of stochastic neural network models with binary ($\pm1$) weights and activations via continuous surrogate networks is investigated. We derive new surrogates using a novel derivation based on writing the stochastic neural network as a Markov chain. This derivation also encompasses existing variants of the surrogates presented in the literature. Following this, we theoretically study the surrogates at Initialisation. We derive, using mean field theory, a set of scalar equations describing how input signals propagate through the randomly initialised networks. The equations reveal whether so-called critical Initialisations exist for each surrogate network, where the network can be trained to arbitrary depth. Moreover, we predict theoretically and confirm numerically, that common weight Initialisation schemes used in standard continuous networks, when applied to the mean values of the stochastic binary weights, yield poor training performance. This study shows that, contrary to common intuition, the means of the stochastic binary weights should be initialised close to $\pm 1$, for deeper networks to be trainable.

마르코스 씨 티자네스 - One of the best experts on this subject based on the ideXlab platform.