The Experts below are selected from a list of 13533 Experts worldwide ranked by ideXlab platform
Jacek M. Zurada - One of the best experts on this subject based on the ideXlab platform.
-
a novel pruning algorithm for smoothing Feedforward Neural Networks based on group lasso method
IEEE Transactions on Neural Networks, 2018Co-Authors: Jian Wang, Xifeng Yang, Jacek M. ZuradaAbstract:In this paper, we propose four new variants of the backpropagation algorithm to improve the generalization ability for Feedforward Neural Networks. The basic idea of these methods stems from the Group Lasso concept which deals with the variable selection problem at the group level. There are two main drawbacks when the Group Lasso penalty has been directly employed during network training. They are numerical oscillations and theoretical challenges in computing the gradients at the origin. To overcome these obstacles, smoothing functions have then been introduced by approximating the Group Lasso penalty. Numerical experiments for classification and regression problems demonstrate that the proposed algorithms perform better than the other three classical penalization methods, Weight Decay, Weight Elimination, and Approximate Smoother , on both generalization and pruning efficiency. In addition, detailed simulations based on a specific data set have been performed to compare with some other common pruning strategies, which verify the advantages of the proposed algorithm. The pruning abilities of the proposed strategy have been investigated in detail for a relatively large data set, MNIST, in terms of various smoothing approximation cases.
-
input layer regularization of multilayer Feedforward Neural Networks
IEEE Access, 2017Co-Authors: Jacek M. Zurada, Yan LiuAbstract:Multilayer Feedforward Neural Networks (MFNNs) have been widely used for classification or approximation of nonlinear mappings described by a data set consisting of input and output samples. In many MFNN applications, a common compressive sensing task is to find the redundant dimensions of the input data. The aim of a regularization technique presented in this paper is to eliminate the redundant dimensions and to achieve compression of the input layer. It is achieved by introducing an $L_{1}$ or $L_{1/2}$ regularizer to the input layer weights training. As a comparison, in the existing references, a regularization method is usually applied to the hidden layer for a better representation of the dataset and sparsification of the network. Gradient-descent method is used for solving the resulting optimization problem. Numerical experiments including a simulated approximation problem and three classification problems (Monk, Sonar, and the MNIST data set) have been used to illustrate the algorithm.
-
Convergence of online gradient method for Feedforward Neural Networks with smoothing L 1/2 regularization penalty
Neurocomputing, 2014Co-Authors: Qinwei Fan, Jacek M. ZuradaAbstract:Minimization of the training regularization term has been recognized as an important objective for sparse modeling and generalization in Feedforward Neural Networks. Most of the studies so far have been focused on the popular L"2 regularization penalty. In this paper, we consider the convergence of online gradient method with smoothing L"1"/"2 regularization term. For normal L"1"/"2 regularization, the objective function is the sum of a non-convex, non-smooth, and non-Lipschitz function, which causes oscillation of the error function and the norm of gradient. However, using the smoothing approximation techniques, the deficiency of the normal L"1"/"2 regularization term can be addressed. This paper shows the strong convergence results for the smoothing L"1"/"2 regularization. Furthermore, we prove the boundedness of the weights during the network training. The assumption that weights are bounded is no longer needed for the proof of convergence. Simulation results support the theoretical findings and demonstrate that our algorithm has better performance than two other algorithms with L"2 and normal L"1"/"2 regularizations respectively.
-
Batch gradient method with smoothing L1/ 2 regularization for training of Feedforward Neural Networks
Neural networks : the official journal of the International Neural Network Society, 2013Co-Authors: Qinwei Fan, Jacek M. Zurada, Jian Wang, Dakun Yang, Yan LiuAbstract:The aim of this paper is to develop a novel method to prune Feedforward Neural Networks by introducing an L"1"/"2 regularization term into the error function. This procedure forces weights to become smaller during the training and can eventually removed after the training. The usual L"1"/"2 regularization term involves absolute values and is not differentiable at the origin, which typically causes oscillation of the gradient of the error function during the training. A key point of this paper is to modify the usual L"1"/"2 regularization term by smoothing it at the origin. This approach offers the following three advantages: First, it removes the oscillation of the gradient value. Secondly, it gives better pruning, namely the final weights to be removed are smaller than those produced through the usual L"1"/"2 regularization. Thirdly, it makes it possible to prove the convergence of the training. Supporting numerical examples are also provided.
-
deterministic convergence of conjugate gradient method for Feedforward Neural Networks
Neurocomputing, 2011Co-Authors: Jian Wang, Jacek M. ZuradaAbstract:Conjugate gradient methods have many advantages in real numerical experiments, such as fast convergence and low memory requirements. This paper considers a class of conjugate gradient learning methods for backpropagation Neural Networks with three layers. We propose a new learning algorithm for almost cyclic learning of Neural Networks based on PRP conjugate gradient method. We then establish the deterministic convergence properties for three different learning modes, i.e., batch mode, cyclic and almost cyclic learning. The two deterministic convergence properties are weak and strong convergence that indicate that the gradient of the error function goes to zero and the weight sequence goes to a fixed point, respectively. It is shown that the deterministic convergence results are based on different learning modes and dependent on different selection strategies of learning rate. Illustrative numerical examples are given to support the theoretical analysis.
Khashayar Khorasani - One of the best experts on this subject based on the ideXlab platform.
-
Constructive Feedforward Neural Networks using Hermite polynomial activation functions
IEEE Transactions on Neural Networks, 2005Co-Authors: Khashayar KhorasaniAbstract:In this paper, a constructive one-hidden-layer network is introduced where each hidden unit employs a polynomial function for its activation function that is different from other units. Specifically, both a structure level as well as a function level adaptation methodologies are utilized in constructing the network. The functional level adaptation scheme ensures that the "growing" or constructive network has different activation functions for each neuron such that the network may be able to capture the underlying input-output map more effectively. The activation functions considered consist of orthonormal Hermite polynomials. It is shown through extensive simulations that the proposed network yields improved performance when compared to Networks having identical sigmoidal activation functions.
-
facial expression recognition using constructive Feedforward Neural Networks
Systems Man and Cybernetics, 2004Co-Authors: Khashayar KhorasaniAbstract:A new technique for facial expression recognition is proposed, which uses the two-dimensional (2D) discrete cosine transform (DCT) over the entire face image as a feature detector and a constructive one-hidden-layer Feedforward Neural network as a facial expression classifier. An input-side pruning technique, proposed previously by the authors, is also incorporated into the constructive learning process to reduce the network size without sacrificing the performance of the resulting network. The proposed technique is applied to a database consisting of images of 60 men, each having five facial expression images (neutral, smile, anger, sadness, and surprise). Images of 40 men are used for network training, and the remaining images of 20 men are used for generalization and testing. Confusion matrices calculated in both network training and generalization for four facial expressions (smile, anger, sadness, and surprise) are used to evaluate the performance of the trained network. It is demonstrated that the best recognition rates are 100% and 93.75% (without rejection), for the training and generalizing images, respectively. Furthermore, the input-side weights of the constructed network are reduced by approximately 30% using our pruning method. In comparison with the fixed structure back propagation-based recognition methods in the literature, the proposed technique constructs one-hidden-layer Feedforward Neural network with fewer number of hidden units and weights, while simultaneously provide improved generalization and recognition performance capabilities.
-
a new strategy for adaptively constructing multilayer Feedforward Neural Networks
Neurocomputing, 2003Co-Authors: Khashayar KhorasaniAbstract:In this paper a new strategy for adaptively and autonomously constructing a multi-hidden-layer Feedforward Neural network (FNN) is introduced. The proposed scheme belongs to a class of structure level adaptation algorithms that adds both new hidden units and new hidden layers one at a time when it is determined to be needed. Using this strategy, a FNN may be constructed having as many hidden layers and hidden units as required by the complexity of the problem being considered. Simulation results applied to regression problems are included to demonstrate the performance capabilities of the proposed scheme.
Guang-bin Huang - One of the best experts on this subject based on the ideXlab platform.
-
extreme learning machine theory and applications
Neurocomputing, 2006Co-Authors: Guang-bin Huang, Qin-yu Zhu, Chee-kheong SiewAbstract:Abstract It is clear that the learning speed of Feedforward Neural Networks is in general far slower than required and it has been a major bottleneck in their applications for past decades. Two key reasons behind may be: (1) the slow gradient-based learning algorithms are extensively used to train Neural Networks, and (2) all the parameters of the Networks are tuned iteratively by using such learning algorithms. Unlike these conventional implementations, this paper proposes a new learning algorithm called e xtreme l earning m achine (ELM) for s ingle-hidden l ayer f eedforward Neural n etworks (SLFNs) which randomly chooses hidden nodes and analytically determines the output weights of SLFNs. In theory, this algorithm tends to provide good generalization performance at extremely fast learning speed. The experimental results based on a few artificial and real benchmark function approximation and classification problems including very large complex applications show that the new algorithm can produce good generalization performance in most cases and can learn thousands of times faster than conventional popular learning algorithms for Feedforward Neural Networks. 1
-
rapid and brief communication evolutionary extreme learning machine
Pattern Recognition, 2005Co-Authors: Qin-yu Zhu, Ponnuthurai Nagaratnam Suganthan, A. K. Qin, Guang-bin HuangAbstract:Extreme learning machine (ELM) [G.-B. Huang, Q.-Y. Zhu, C.-K. Siew, Extreme learning machine: a new learning scheme of Feedforward Neural Networks, in: Proceedings of the International Joint Conference on Neural Networks (IJCNN2004), Budapest, Hungary, 25-29 July 2004], a novel learning algorithm much faster than the traditional gradient-based learning algorithms, was proposed recently for single-hidden-layer Feedforward Neural Networks (SLFNs). However, ELM may need higher number of hidden neurons due to the random determination of the input weights and hidden biases. In this paper, a hybrid learning algorithm is proposed which uses the differential evolutionary algorithm to select the input weights and Moore-Penrose (MP) generalized inverse to analytically determine the output weights. Experimental results show that this approach is able to achieve good generalization performance with much more compact Networks.
Yonggwan Won - One of the best experts on this subject based on the ideXlab platform.
-
regularized online sequential learning algorithm for single hidden layer Feedforward Neural Networks
Pattern Recognition Letters, 2011Co-Authors: Hieu Trung Huynh, Yonggwan WonAbstract:Online learning algorithms have been preferred in many applications due to their ability to learn by the sequentially arriving data. One of the effective algorithms recently proposed for training single hidden-layer Feedforward Neural Networks (SLFNs) is online sequential extreme learning machine (OS-ELM), which can learn data one-by-one or chunk-by-chunk at fixed or varying sizes. It is based on the ideas of extreme learning machine (ELM), in which the input weights and hidden layer biases are randomly chosen and then the output weights are determined by the pseudo-inverse operation. The learning speed of this algorithm is extremely high. However, it is not good to yield generalization models for noisy data and is difficult to initialize parameters in order to avoid singular and ill-posed problems. In this paper, we propose an improvement of OS-ELM based on the bi-objective optimization approach. It tries to minimize the empirical error and obtain small norm of network weight vector. Singular and ill-posed problems can be overcome by using the Tikhonov regularization. This approach is also able to learn data one-by-one or chunk-by-chunk. Experimental results show the better generalization performance of the proposed approach on benchmark datasets.
Qin-yu Zhu - One of the best experts on this subject based on the ideXlab platform.
-
extreme learning machine theory and applications
Neurocomputing, 2006Co-Authors: Guang-bin Huang, Qin-yu Zhu, Chee-kheong SiewAbstract:Abstract It is clear that the learning speed of Feedforward Neural Networks is in general far slower than required and it has been a major bottleneck in their applications for past decades. Two key reasons behind may be: (1) the slow gradient-based learning algorithms are extensively used to train Neural Networks, and (2) all the parameters of the Networks are tuned iteratively by using such learning algorithms. Unlike these conventional implementations, this paper proposes a new learning algorithm called e xtreme l earning m achine (ELM) for s ingle-hidden l ayer f eedforward Neural n etworks (SLFNs) which randomly chooses hidden nodes and analytically determines the output weights of SLFNs. In theory, this algorithm tends to provide good generalization performance at extremely fast learning speed. The experimental results based on a few artificial and real benchmark function approximation and classification problems including very large complex applications show that the new algorithm can produce good generalization performance in most cases and can learn thousands of times faster than conventional popular learning algorithms for Feedforward Neural Networks. 1
-
rapid and brief communication evolutionary extreme learning machine
Pattern Recognition, 2005Co-Authors: Qin-yu Zhu, Ponnuthurai Nagaratnam Suganthan, A. K. Qin, Guang-bin HuangAbstract:Extreme learning machine (ELM) [G.-B. Huang, Q.-Y. Zhu, C.-K. Siew, Extreme learning machine: a new learning scheme of Feedforward Neural Networks, in: Proceedings of the International Joint Conference on Neural Networks (IJCNN2004), Budapest, Hungary, 25-29 July 2004], a novel learning algorithm much faster than the traditional gradient-based learning algorithms, was proposed recently for single-hidden-layer Feedforward Neural Networks (SLFNs). However, ELM may need higher number of hidden neurons due to the random determination of the input weights and hidden biases. In this paper, a hybrid learning algorithm is proposed which uses the differential evolutionary algorithm to select the input weights and Moore-Penrose (MP) generalized inverse to analytically determine the output weights. Experimental results show that this approach is able to achieve good generalization performance with much more compact Networks.
