The Experts below are selected from a list of 253413 Experts worldwide ranked by ideXlab platform
Andries P Engelbrecht - One of the best experts on this subject based on the ideXlab platform.
-
Random Regrouping and Factorization in Cooperative Particle Swarm Optimization Based Large-Scale Neural Network Training
Neural Processing Letters, 2019Co-Authors: Cody Dennis, Beatrice M. Ombuki-berman, Andries P EngelbrechtAbstract:Previous studies have shown that factorization and random regrouping significantly improve the performance of the cooperative particle swarm optimization (CPSO) algorithm. However, few studies have examined whether this trend continues when CPSO is applied to the Training of feed forward neural Networks. Neural Network Training problems often have very high dimensionality and introduce the issue of saturation, which has been shown to significantly affect the behavior of particles in the swarm; thus it should not be assumed that these trends hold. This study identifies the benefits of random regrouping and factorization to CPSO based neural Network Training, and proposes a number of approaches to problem decomposition for use in neural Network Training. Experiments are performed on 11 problems with sizes ranging from 35 up to 32,811 weights and biases, using a number of general approaches to problem decomposition, and state of the art algorithms taken from the literature. This study found that the impact of factorization and random regrouping on solution quality and swarm behavior depends heavily on the general approach to problem decomposition. It is shown that a random problem decomposition is effective in feed forward neural Network Training. A random problem decomposition has the benefit of reducing the issue of problem decomposition to the tuning of a single parameter.
-
CEC - Saturation in PSO neural Network Training: Good or evil?
2015 IEEE Congress on Evolutionary Computation (CEC), 2015Co-Authors: Anna Rakitianskaia, Andries P EngelbrechtAbstract:Particle swarm optimisation has been successfully applied as a neural Network Training algorithm before, often outperforming traditional gradient-based approaches. However, recent studies have shown that particle swarm optimisation does not scale very well, and performs poorly on high-dimensional neural Network architectures. This paper hypothesises that hidden layer saturation is a significant factor contributing to the poor Training performance of the particle swarms, hindering good performance on neural Networks regardless of the architecture size. A selection of classification problems is used to test this hypothesis. It is discovered that although a certain degree of saturation is necessary for successful Training, higher degrees of saturation ultimately lead to poor generalisation. Possible factors leading to saturation are suggested, and means of alleviating saturation in particle swarms through weight initialisation range, maximum velocity, and search space boundaries are analysed. This paper is intended as a preface to a more in-depth study of the problem of saturation in particle swarm optimisation as a neural Network Training algorithm.
Imre Boros - One of the best experts on this subject based on the ideXlab platform.
-
new optimization algorithms for neural Network Training using operator splitting techniques
Neural Networks, 2020Co-Authors: Cristian Daniel Alecsa, Titus Pinţa, Imre BorosAbstract:Abstract In the following paper we present a new type of optimization algorithms adapted for neural Network Training. These algorithms are based upon sequential operator splitting technique for some associated dynamical systems. Furthermore, we investigate through numerical simulations the empirical rate of convergence of these iterative schemes toward a local minimum of the loss function, with some suitable choices of the underlying hyper-parameters. We validate the convergence of these optimizers using the results of the accuracy and of the loss function on the MNIST, MNIST-Fashion and CIFAR 10 classification datasets.
R.r. Rhinehart - One of the best experts on this subject based on the ideXlab platform.
-
A novel method to stop neural Network Training
Proceedings of the 2000 American Control Conference. ACC (IEEE Cat. No.00CH36334), 2000Co-Authors: M.s. Iyer, R.r. RhinehartAbstract:A novel approach of stopping neural Network Training has been evaluated. The approach improves data utilization, provides a statistical rationale to stop Training, and seems to present a robust method for automating the stopping of neural Network Training.
-
A method to determine the required number of neural-Network Training repetitions
IEEE transactions on neural networks, 1999Co-Authors: M.s. Iyer, R.r. RhinehartAbstract:Conventional neural-Network Training algorithms often get stuck in local minima. To find the global optimum, Training is conventionally repeated with ten, or so, random starting values for the weights. Here we develop an analytical procedure to determine how many times a neural Network needs to be trained, with random starting weights, to ensure that the best of those is within a desirable lower percentile of all possible Trainings, with a certain level of confidence. The theoretical developments are validated by experimental results. While applied to neural-Network Training, the method is generally applicable to nonlinear optimization.
Yunlong Zhu - One of the best experts on this subject based on the ideXlab platform.
-
A Multi-population Cooperative Particle Swarm Optimizer for Neural Network Training
Lecture Notes in Computer Science, 2006Co-Authors: Ben Niu, Yunlong ZhuAbstract:This paper presents a new learning algorithm, Multi-Population Cooperative Particle Swarm Optimizer (MCPSO), for neural Network Training. MCPSO is based on a master-slave model, in which a population consists of a master group and several slave groups. The slave groups execute a single PSO or its variants independently to maintain the diversity of particles, while the master group evolves based on its own information and also the information of the slave groups. The particles both in the master group and the slave groups are co-evolved during the search process by employing a parameter, termed migration factor. The MCPSO is applied for Training a multilayer feed-forward neural Network, for three benchmark classification problems. The performance of MCPSO used for neural Network Training is compared to that of Back Propagation (BP). genetic algorithm (GA) and standard PSO (SPSO), demonstrating its effectiveness and efficiency.
Krste Asanovic - One of the best experts on this subject based on the ideXlab platform.
-
Parallel neural Network Training on Multi-Spert
Proceedings of 3rd International Conference on Algorithms and Architectures for Parallel Processing, 1997Co-Authors: Paul Färber, Krste AsanovicAbstract:Multi-Spert is a scalable parallel system built from multiple\nSpert-II nodes which we have constructed to speed error backpropagation\nneural Network Training for speech recognition research. We present the\nMulti-Spert hardware and software architecture, and describe our\nimplementation of two alternative parallelization strategies for the\nbackprop algorithm. We have developed detailed analytic models of the\ntwo strategies which allow us to predict performance over a range of\nNetwork and machine parameters. The models' predictions are validated by\nmeasurements for a prototype five node Multi-Spert system. This\nprototype achieves a neural Network Training performance of over 530\nmillion connection updates per second (MCUPS) while Training a realistic\nspeech application neural Network. The model predicts that performance\nwill scale to over 800 MCUPS for eight nodes
