The Experts below are selected from a list of 25308 Experts worldwide ranked by ideXlab platform
Sara Silva - One of the best experts on this subject based on the ideXlab platform.
-
on the Generalization Ability of geometric semantic genetic programming
European Conference on Genetic Programming, 2015Co-Authors: Sara Silva, Ivo Goncalves, Carlos M FonsecaAbstract:Geometric Semantic Genetic Programming (GSGP) is a recently proposed form of Genetic Programming (GP) that searches directly the space of the underlying semantics of the programs. The fitness landscape seen by the GSGP variation operators is unimodal with a linear slope by construction and, consequently, easy to search. Despite this advantage, the offspring produced by these operators grow very quickly. A new implementation of the same operators was proposed that computes the semantics of the offspring without having to explicitly build their syntax. This allowed GSGP to be used for the first time in real-life multidimensional datasets. GSGP presented a surprisingly good Generalization Ability, which was justified by some properties of the geometric semantic operators. In this paper, we show that the good Generalization Ability of GSGP was the result of a small implementation deviation from the original formulation of the mutation operator, and that without it the Generalization results would be significantly worse. We explain the reason for this difference, and then we propose two variants of the geometric semantic mutation that deterministically and optimally adapt the mutation step. They reveal to be more efficient in learning the training data, and they also achieve a competitive Generalization in only a single operator application. This provides a competitive alternative when performing semantic search, particularly since they produce small individuals and compute fast.
-
EuroGP - On the Generalization Ability of Geometric Semantic Genetic Programming
Lecture Notes in Computer Science, 2015Co-Authors: Ivo Goncalves, Sara Silva, Carlos M FonsecaAbstract:Geometric Semantic Genetic Programming (GSGP) is a recently proposed form of Genetic Programming (GP) that searches directly the space of the underlying semantics of the programs. The fitness landscape seen by the GSGP variation operators is unimodal with a linear slope by construction and, consequently, easy to search. Despite this advantage, the offspring produced by these operators grow very quickly. A new implementation of the same operators was proposed that computes the semantics of the offspring without having to explicitly build their syntax. This allowed GSGP to be used for the first time in real-life multidimensional datasets. GSGP presented a surprisingly good Generalization Ability, which was justified by some properties of the geometric semantic operators. In this paper, we show that the good Generalization Ability of GSGP was the result of a small implementation deviation from the original formulation of the mutation operator, and that without it the Generalization results would be significantly worse. We explain the reason for this difference, and then we propose two variants of the geometric semantic mutation that deterministically and optimally adapt the mutation step. They reveal to be more efficient in learning the training data, and they also achieve a competitive Generalization in only a single operator application. This provides a competitive alternative when performing semantic search, particularly since they produce small individuals and compute fast.
-
A comparison of the Generalization Ability of different genetic programming frameworks
IEEE Congress on Evolutionary Computation, 2010Co-Authors: Mauro Castelli, Luca Manzoni, Sara Silva, Leonardo VanneschiAbstract:Generalization is an important issue in machine learning. In fact, in several applications good results over training data are not as important as good results over unseen data. While this problem was deeply studied in other machine learning techniques, it has become an important issue for genetic programming only in the last few years. In this paper we compare the Generalization Ability of several different genetic programming frameworks, including some variants of multi-objective genetic programming and operator equalization, a recently defined bloat free genetic programming system. The test problem used is a hard regression real-life application in the field of drug discovery and development, characterized by a high number of features and where the Generalization Ability of the proposed solutions is a crucial issue. The results we obtained show that, at least for the considered problem, multi-optimization is effective in improving genetic programming Generalization Ability, outperforming all the other methods on test data.
-
IEEE Congress on Evolutionary Computation - A comparison of the Generalization Ability of different genetic programming frameworks
IEEE Congress on Evolutionary Computation, 2010Co-Authors: Mauro Castelli, Luca Manzoni, Sara Silva, Leonardo VanneschiAbstract:Generalization is an important issue in machine learning. In fact, in several applications good results over training data are not as important as good results over unseen data. While this problem was deeply studied in other machine learning techniques, it has become an important issue for genetic programming only in the last few years. In this paper we compare the Generalization Ability of several different genetic programming frameworks, including some variants of multi-objective genetic programming and operator equalization, a recently defined bloat free genetic programming system. The test problem used is a hard regression real-life application in the field of drug discovery and development, characterized by a high number of features and where the Generalization Ability of the proposed solutions is a crucial issue. The results we obtained show that, at least for the considered problem, multi-optimization is effective in improving genetic programming Generalization Ability, outperforming all the other methods on test data.
Leonardo Vanneschi - One of the best experts on this subject based on the ideXlab platform.
-
A comparison of the Generalization Ability of different genetic programming frameworks
IEEE Congress on Evolutionary Computation, 2010Co-Authors: Mauro Castelli, Luca Manzoni, Sara Silva, Leonardo VanneschiAbstract:Generalization is an important issue in machine learning. In fact, in several applications good results over training data are not as important as good results over unseen data. While this problem was deeply studied in other machine learning techniques, it has become an important issue for genetic programming only in the last few years. In this paper we compare the Generalization Ability of several different genetic programming frameworks, including some variants of multi-objective genetic programming and operator equalization, a recently defined bloat free genetic programming system. The test problem used is a hard regression real-life application in the field of drug discovery and development, characterized by a high number of features and where the Generalization Ability of the proposed solutions is a crucial issue. The results we obtained show that, at least for the considered problem, multi-optimization is effective in improving genetic programming Generalization Ability, outperforming all the other methods on test data.
-
IEEE Congress on Evolutionary Computation - A comparison of the Generalization Ability of different genetic programming frameworks
IEEE Congress on Evolutionary Computation, 2010Co-Authors: Mauro Castelli, Luca Manzoni, Sara Silva, Leonardo VanneschiAbstract:Generalization is an important issue in machine learning. In fact, in several applications good results over training data are not as important as good results over unseen data. While this problem was deeply studied in other machine learning techniques, it has become an important issue for genetic programming only in the last few years. In this paper we compare the Generalization Ability of several different genetic programming frameworks, including some variants of multi-objective genetic programming and operator equalization, a recently defined bloat free genetic programming system. The test problem used is a hard regression real-life application in the field of drug discovery and development, characterized by a high number of features and where the Generalization Ability of the proposed solutions is a crucial issue. The results we obtained show that, at least for the considered problem, multi-optimization is effective in improving genetic programming Generalization Ability, outperforming all the other methods on test data.
-
using crossover based similarity measure to improve genetic programming Generalization Ability
Genetic and Evolutionary Computation Conference, 2009Co-Authors: Leonardo Vanneschi, Steven GustafsonAbstract:Generalization is a very important issue in Machine Learning. In this paper, we present a new idea for improving Genetic Programming Generalization Ability. The idea is based on a dynamic two-layered selection algorithm and it is tested on a real-life drug discovery regression application. The algorithm begins using root mean squared error as fitness and the usual tournament selection. A list of individuals called ``repulsors'' is also kept in memory and initialized as empty. As an individual is found to overfit the training set, it is inserted into the list of repulsors. When the list of repulsors is not empty, selection becomes a two-layer algorithm: individuals participating to the tournament are not randomly chosen from the population but are themselves selected, using the average dissimilarity to the repulsors as a criterion to be maximized. Two kinds of similarity/dissimilarity measures are tested for this aim: the well known structural (or edit) distance and the recently defined subtree crossover based similarity measure. Although simple, this idea seems to improve Genetic Programming Generalization Ability and the presented experimental results show that Genetic Programming generalizes better when subtree crossover based similarity measure is used, at least for the test problems studied in this paper.
-
GECCO - Using crossover based similarity measure to improve genetic programming Generalization Ability
Proceedings of the 11th Annual conference on Genetic and evolutionary computation - GECCO '09, 2009Co-Authors: Leonardo Vanneschi, Steven GustafsonAbstract:Generalization is a very important issue in Machine Learning. In this paper, we present a new idea for improving Genetic Programming Generalization Ability. The idea is based on a dynamic two-layered selection algorithm and it is tested on a real-life drug discovery regression application. The algorithm begins using root mean squared error as fitness and the usual tournament selection. A list of individuals called ``repulsors'' is also kept in memory and initialized as empty. As an individual is found to overfit the training set, it is inserted into the list of repulsors. When the list of repulsors is not empty, selection becomes a two-layer algorithm: individuals participating to the tournament are not randomly chosen from the population but are themselves selected, using the average dissimilarity to the repulsors as a criterion to be maximized. Two kinds of similarity/dissimilarity measures are tested for this aim: the well known structural (or edit) distance and the recently defined subtree crossover based similarity measure. Although simple, this idea seems to improve Genetic Programming Generalization Ability and the presented experimental results show that Genetic Programming generalizes better when subtree crossover based similarity measure is used, at least for the test problems studied in this paper.
-
multi optimization improves genetic programming Generalization Ability
Genetic and Evolutionary Computation Conference, 2007Co-Authors: Leonardo Vanneschi, Denis Rochat, Marco TomassiniAbstract:Generalization is one of the most important performance evaluation criteria for artificial learning systems, in particular for supervised learning. While a large amount of literature and of well established results exist concerning the issue of Generalization for many non-evolutionary Machine Learning strategies, like for instance Support Vector Machines, this issue in Genetic Programming (GP) has not received the attention it deserves and only recently, few papers dealing with the problem of Generalization have appeared (see for instance [1, 2, 3]). In this paper, we have motivated and empirically shown that GP using a Pareto multi-optimization on the training set has a remarkably higher Generalization Ability than canonic or standard GP (besides counteracting bloat in a more efficient way and maintaining a higher diversity inside the population). Here is an informal motivation for this idea: in figure 1, we have plotted two simple hypothetical fitness functions and two simple hypothetical GP individuals with good fitness on the training set and bad Generalization Ability, if the sum of errors is considered as the sole evaluation criterium. Even though for points inside the training set the gray and black curves are very close (and thus fitness is good on the training set, if fitness is the sum of errors), outside the training set, they are very far from each other and they get farthest as we consider farthest points from the training set. This happens because the gray and black curves are uncorrelated and all the distances between the gray curves points and the black curves ones with the same abscissa inside the training set are different between each other. Thus, three optimization criteria have been used on the training set by our multi-optimization framework: sum of errors, statistical correlation between targets and outputs and variance of the pairwise distances between targets and outputs. Simulations have been executed on three
Ivo Goncalves - One of the best experts on this subject based on the ideXlab platform.
-
on the Generalization Ability of geometric semantic genetic programming
European Conference on Genetic Programming, 2015Co-Authors: Sara Silva, Ivo Goncalves, Carlos M FonsecaAbstract:Geometric Semantic Genetic Programming (GSGP) is a recently proposed form of Genetic Programming (GP) that searches directly the space of the underlying semantics of the programs. The fitness landscape seen by the GSGP variation operators is unimodal with a linear slope by construction and, consequently, easy to search. Despite this advantage, the offspring produced by these operators grow very quickly. A new implementation of the same operators was proposed that computes the semantics of the offspring without having to explicitly build their syntax. This allowed GSGP to be used for the first time in real-life multidimensional datasets. GSGP presented a surprisingly good Generalization Ability, which was justified by some properties of the geometric semantic operators. In this paper, we show that the good Generalization Ability of GSGP was the result of a small implementation deviation from the original formulation of the mutation operator, and that without it the Generalization results would be significantly worse. We explain the reason for this difference, and then we propose two variants of the geometric semantic mutation that deterministically and optimally adapt the mutation step. They reveal to be more efficient in learning the training data, and they also achieve a competitive Generalization in only a single operator application. This provides a competitive alternative when performing semantic search, particularly since they produce small individuals and compute fast.
-
EuroGP - On the Generalization Ability of Geometric Semantic Genetic Programming
Lecture Notes in Computer Science, 2015Co-Authors: Ivo Goncalves, Sara Silva, Carlos M FonsecaAbstract:Geometric Semantic Genetic Programming (GSGP) is a recently proposed form of Genetic Programming (GP) that searches directly the space of the underlying semantics of the programs. The fitness landscape seen by the GSGP variation operators is unimodal with a linear slope by construction and, consequently, easy to search. Despite this advantage, the offspring produced by these operators grow very quickly. A new implementation of the same operators was proposed that computes the semantics of the offspring without having to explicitly build their syntax. This allowed GSGP to be used for the first time in real-life multidimensional datasets. GSGP presented a surprisingly good Generalization Ability, which was justified by some properties of the geometric semantic operators. In this paper, we show that the good Generalization Ability of GSGP was the result of a small implementation deviation from the original formulation of the mutation operator, and that without it the Generalization results would be significantly worse. We explain the reason for this difference, and then we propose two variants of the geometric semantic mutation that deterministically and optimally adapt the mutation step. They reveal to be more efficient in learning the training data, and they also achieve a competitive Generalization in only a single operator application. This provides a competitive alternative when performing semantic search, particularly since they produce small individuals and compute fast.
J. Murata - One of the best experts on this subject based on the ideXlab platform.
-
improvement of Generalization Ability for identifying dynamical systems by using universal learning networks
Neural Networks, 2001Co-Authors: Kotaro Hirasawa, Jinglu Hu, J. MurataAbstract:This paper studies how the Generalization Ability of models of dynamical systems can be improved by taking advantage of the second order derivatives of the outputs with respect to the external inputs. The proposed method can be regarded as a direct implementation of the well-known regularization technique using the higher order derivatives of the Universal Learning Networks (ULNs). ULNs consist of a number of interconnected nodes where the nodes may have any continuously differentiable nonlinear functions in them and each pair of nodes can be connected by multiple branches with arbitrary time delays. A generalized learning algorithm has been derived for the ULNs, in which both the first order derivatives (gradients) and the higher order derivatives are incorporated. First, the method for computing the second order derivatives of ULNs is discussed. Then, a new method for implementing the regularization term is presented. Finally, simulation studies on identification of a nonlinear dynamical system with noises are carried out to demonstrate the effectiveness of the proposed method. Simulation results show that the proposed method can improve the Generalization Ability of neural networks significantly, especially in terms that (1) the robust network can be obtained even when the branches of trained ULNs are destructed, and (2) the obtained performance does not depend on the initial parameter values.
-
Enhancing the Generalization Ability of neural networks by using Gram-Schmidt orthogonalization algorithm
IJCNN'01. International Joint Conference on Neural Networks. Proceedings (Cat. No.01CH37222), 2001Co-Authors: K. Hirasawa, J. Hu, J. MurataAbstract:The Generalization Ability of neural networks is an important criterion when determining whether one algorithm is powerful or not. Many new algorithms have been devised to enhance the Generalization Ability of neural networks. In this paper an algorithm using the Gram-Schmidt orthogonalization algorithm on the outputs of nodes in the hidden layers is proposed with the aim to reduce the interference among the nodes in the hidden layers, which is much more efficient than the regularizers methods. Simulation results confirm the above assertion.
-
Improving Generalization Ability of universal learning networks with superfluous parameters
IEEE SMC'99 Conference Proceedings. 1999 IEEE International Conference on Systems Man and Cybernetics (Cat. No.99CH37028), 1999Co-Authors: Kotaro Hirasawa, Jinglu Hu, J. MurataAbstract:The parameters in large scale neural networks can be divided into two classes. One class is necessary for a certain purpose while another class is not directly needed. The parameters in the latter are defined as superfluous parameters. How to use these superfluous parameters effectively is an interesting subject. It is studied how the Generalization Ability of dynamic systems can be improved by use of networks' superfluous parameters. A calculation technique is proposed which uses second order derivatives of the criterion function with respect to superfluous parameters. So as to investigate the effectiveness of the proposed method, simulations of modeling a nonlinear robot dynamics system is studied. Simulation results show that the proposed method is useful for improving the Generalization Ability of neural networks, which may model nonlinear dynamic systems.
-
Generalization Ability of universal learning network by using second order derivatives
SMC'98 Conference Proceedings. 1998 IEEE International Conference on Systems Man and Cybernetics (Cat. No.98CH36218), 1998Co-Authors: K. Hirasawa, J. Hu, J. MurataAbstract:In this paper, it is studied how the Generalization Ability of modeling of the dynamic systems can be improved by taking advantages of the second order derivatives of the criterion function with respect to the external inputs. The proposed method is based on the regularization theory proposed by Poggio and Givosi (1990), but a main distinctive point in this paper is that extension to dynamic systems from static systems has been taken into account and actual second order derivatives of the universal learning network have been used to train the parameters of the networks. The second order derivatives term of the criterion function may minimize the deviation caused by the external input changes. Simulation results show that the method is useful for improving the Generalization Ability of identifying nonlinear dynamic systems using neural networks.
Barbara Hammer - One of the best experts on this subject based on the ideXlab platform.
-
learning vector quantization Generalization Ability and dynamics of competing prototypes
Workshop on Self-Organizing Maps, 2007Co-Authors: Aree Witoelar, Michael Biehl, Barbara HammerAbstract:Learning Vector Quantization (LVQ) are popular multi-class classification algorithms. Prototypes in an LVQ system represent the typical features of classes in the data. Frequently multiple prototypes are employed for a class to improve the representation of variations within the class and the Generalization Ability. In this paper, we investigate the dynamics of LVQ in an exact mathematical way, aiming at understanding the influence of the number of prototypes and their assignment to classes. The theory of on-line learning allows a mathematical description of the learning dynamics in model situations. We demonstrate using a system of three prototypes the different behaviors of LVQ systems of multiple prototype and single prototype class representation.
-
on the Generalization Ability of recurrent networks
International Conference on Artificial Neural Networks, 2001Co-Authors: Barbara HammerAbstract:The Generalization Ability of discrete time partially recurrent networks is examined. It is well known that the VC dimension of recurrent networks is infinite in most interesting cases and hence the standard VC analysis cannot be applied directly. We find guarantees for specific situations where the transition function forms a contraction or the probAbility of long inputs is restricted. For the general case, we derive posterior bounds which take the input data into account. They are obtained via a Generalization of the luckiness framework to the agnostic setting. The general formalism allows to focus on reppresentative parts of the data as well as more general situations such as long term prediction.
-
ICANN - On the Generalization Ability of Recurrent Networks
Artificial Neural Networks — ICANN 2001, 2001Co-Authors: Barbara HammerAbstract:The Generalization Ability of discrete time partially recurrent networks is examined. It is well known that the VC dimension of recurrent networks is infinite in most interesting cases and hence the standard VC analysis cannot be applied directly. We find guarantees for specific situations where the transition function forms a contraction or the probAbility of long inputs is restricted. For the general case, we derive posterior bounds which take the input data into account. They are obtained via a Generalization of the luckiness framework to the agnostic setting. The general formalism allows to focus on reppresentative parts of the data as well as more general situations such as long term prediction.
