The Experts below are selected from a list of 28458 Experts worldwide ranked by ideXlab platform
Naoki Hayashi - One of the best experts on this subject based on the ideXlab platform.
-
the exact asymptotic form of bayesian Generalization Error in latent dirichlet allocation
Neural Networks, 2021Co-Authors: Naoki HayashiAbstract:Abstract Latent Dirichlet allocation (LDA) obtains essential information from data by using Bayesian inference. It is applied to knowledge discovery via dimension reducing and clustering in many fields. However, its Generalization Error had not been yet clarified since it is a singular statistical model where there is no one-to-one mapping from parameters to probability distributions. In this paper, we give the exact asymptotic form of its Generalization Error and marginal likelihood, by theoretical analysis of its learning coefficient using algebraic geometry. The theoretical result shows that the Bayesian Generalization Error in LDA is expressed in terms of that in matrix factorization and a penalty from the simplex restriction of LDA’s parameter region. A numerical experiment is consistent with the theoretical result.
-
the exact asymptotic form of bayesian Generalization Error in latent dirichlet allocation
arXiv: Machine Learning, 2020Co-Authors: Naoki HayashiAbstract:Latent Dirichlet allocation (LDA) obtains essential information from data by using Bayesian inference. It is applied to knowledge discovery via dimension reducing and clustering in many fields. However, its Generalization Error had not been yet clarified since it is a singular statistical model where there is no one to one map from parameters to probability distributions. In this paper, we give the exact asymptotic form of its Generalization Error and marginal likelihood, by theoretical analysis of its learning coefficient using algebraic geometry. The theoretical result shows that the Bayesian Generalization Error in LDA is expressed in terms of that in matrix factorization and a penalty from the simplex restriction of LDA's parameter region.
-
Asymptotic Bayesian Generalization Error in Latent Dirichlet Allocation and Stochastic Matrix Factorization
SN Computer Science, 2020Co-Authors: Naoki Hayashi, Sumio WatanabeAbstract:Latent Dirichlet allocation (LDA) is useful in document analysis, image processing, and many information systems; however, its Generalization performance has been left unknown because it is a singular learning machine to which regular statistical theory can not be applied. Stochastic matrix factorization (SMF) is a restricted matrix factorization in which matrix factors are stochastic; the column of the matrix is in a simplex. SMF is being applied to image recognition and text mining. We can understand SMF as a statistical model by which a stochastic matrix of given data is represented by a product of two stochastic matrices, whose Generalization performance has also been left unknown because of non-regularity. In this paper, by using an algebraic and geometric method, we show the analytic equivalence of LDA and SMF, both of which have the same real log canonical threshold (RLCT), resulting in that they asymptotically have the same Bayesian Generalization Error and the same log marginal likelihood. Moreover, we derive the upper bound of the RLCT and prove that it is smaller than the dimension of the parameter divided by two, hence the Bayesian Generalization Errors of them are smaller than those of regular statistical models.
Masashi Sugiyama - One of the best experts on this subject based on the ideXlab platform.
-
a refined margin analysis for boosting algorithms via equilibrium margin
Journal of Machine Learning Research, 2011Co-Authors: Liwei Wang, Masashi Sugiyama, Cheng Yang, Zhihua Zhou, Zhaoxiang Jing, Jufu FengAbstract:Much attention has been paid to the theoretical explanation of the empirical success of AdaBoost. The most influential work is the margin theory, which is essentially an upper bound for the Generalization Error of any voting classifier in terms of the margin distribution over the training data. However, important questions were raised about the margin explanation. Breiman (1999) proved a bound in terms of the minimum margin, which is sharper than the margin distribution bound. He argued that the minimum margin would be better in predicting the Generalization Error. Grove and Schuurmans (1998) developed an algorithm called LP-AdaBoost which maximizes the minimum margin while keeping all other factors the same as AdaBoost. In experiments however, LP-AdaBoost usually performs worse than AdaBoost, putting the margin explanation into serious doubt. In this paper, we make a refined analysis of the margin theory. We prove a bound in terms of a new margin measure called the Equilibrium margin (Emargin). The Emargin bound is uniformly sharper than Breiman's minimum margin bound. Thus our result suggests that the minimum margin may be not crucial for the Generalization Error. We also show that a large Emargin and a small empirical Error at Emargin imply a smaller bound of the Generalization Error. Experimental results on benchmark data sets demonstrate that AdaBoost usually has a larger Emargin and a smaller test Error than LP-AdaBoost, which agrees well with our theory.
-
a conditional expectation approach to model selection and active learning under covariate shift
2009Co-Authors: Joaquin Quinonerocandela, Masashi Sugiyama, Anton Schwaighofer, Neil D LawrenceAbstract:This chapter contains sections titled: Conditional Expectation Analysis of Generalization Error, Linear Regression under Covariate Shift, Model Selection, Active Learning, Active Learning with Model Selection, Conclusions
-
On the Influence of Input Noise on a Generalization Error Estimator Masashi Sugiyama
2007Co-Authors: Yuta Okabe Hidemitsu, Yuta Okabe, Masashi Sugiyama, Hidemitsu OgawaAbstract:Estimating the Generalization capability is one of the most important problems in supervised learning. Therefore, various Generalization Error estimators have been proposed so far, in the presence of noise in output values. On the other hand, noise often exists in input values as well as output values. In this paper, we therefore investigate the influence of input noise on a Generalization Error estimator. We focus on a particular Generalization Error estimator called the subspace information criterion (SIC), which is shown to be unbiased in the absence of input noise. Intuitively, small input noise does not seem to affect the unbiasedness of SIC severely because small input noise varies the output values only slightly if the learning target function is continuous. On the contrary to this intuition, we show that even small input noise can totally corrupt the unbiasedness of SIC. This fact casts doubt on the use of SIC in the presence of input noise. To cope with this problem, we provide a sufficient condition to guarantee that SIC is unbiased in the limit of small input noise. We finally show that this condition is always fulfilled when the standard ridge estimation is used for learning, which allows us to use SIC without concern even in the presence of small input noise
-
asymptotic bayesian Generalization Error when training and test distributions are different
International Conference on Machine Learning, 2007Co-Authors: Keisuke Yamazaki, Masashi Sugiyama, Sumio Watanabe, Motoaki Kawanabe, Klausrobert MullerAbstract:In supervised learning, we commonly assume that training and test data are sampled from the same distribution. However, this assumption can be violated in practice and then standard machine learning techniques perform poorly. This paper focuses on revealing and improving the performance of Bayesian estimation when the training and test distributions are different. We formally analyze the asymptotic Bayesian Generalization Error and establish its upper bound under a very general setting. Our important finding is that lower order terms---which can be ignored in the absence of the distribution change---play an important role under the distribution change. We also propose a novel variant of stochastic complexity which can be used for choosing an appropriate model and hyper-parameters under a particular distribution change.
-
active learning in approximately linear regression based on conditional expectation of Generalization Error
Journal of Machine Learning Research, 2006Co-Authors: Masashi SugiyamaAbstract:The goal of active learning is to determine the locations of training input points so that the Generalization Error is minimized. We discuss the problem of active learning in linear regression scenarios. Traditional active learning methods using least-squares learning often assume that the model used for learning is correctly specified. In many practical situations, however, this assumption may not be fulfilled. Recently, active learning methods using "importance"-weighted least-squares learning have been proposed, which are shown to be robust against misspecification of models. In this paper, we propose a new active learning method also using the weighted least-squares learning, which we call ALICE (Active Learning using the Importance-weighted least-squares learning based on Conditional Expectation of the Generalization Error). An important difference from existing methods is that we predict the conditional expectation of the Generalization Error given training input points, while existing methods predict the full expectation of the Generalization Error. Due to this difference, the training input design can be fine-tuned depending on the realization of training input points. Theoretically, we prove that the proposed active learning criterion is a more accurate predictor of the single-trial Generalization Error than the existing criterion. Numerical studies with toy and benchmark data sets show that the proposed method compares favorably to existing methods.
Daniel S Yeung - One of the best experts on this subject based on the ideXlab platform.
-
multiple classifier systems combined with localized Generalization Error for fault diagnosis of power transformers
International Conference on Machine Learning and Cybernetics, 2010Co-Authors: Weichun Chen, Patrick P K Chan, Daniel S YeungAbstract:Dissolved gas-in-oil analysis (DGA) is an effective approach for detecting incipient inner fault transformers and various methods derived from DGA have been introduced. To overcome their inherent weaknesses such as the variability of DGA data, this paper proposes a novel multiple classifier system to identify the inner fault of power transformers. The presented method is based on some primitive RBF classifiers and the multiple classifier system is evaluated with the Localized Generalization Error obtained by the Localized Generalization Error model (L-GEM). Compared to other measurements of ensemble system, the proposed method archives a good result.
-
feature selection using localized Generalization Error for supervised classification problems using rbfnn
Pattern Recognition, 2008Co-Authors: Daniel S Yeung, Michael Firth, E C C Tsang, Xizhao WangAbstract:A pattern classification problem usually involves using high-dimensional features that make the classifier very complex and difficult to train. With no feature reduction, both training accuracy and Generalization capability will suffer. This paper proposes a novel hybrid filter-wrapper-type feature subset selection methodology using a localized Generalization Error model. The localized Generalization Error model for a radial basis function neural network bounds from above the Generalization Error for unseen samples located within a neighborhood of the training samples. Iteratively, the feature making the smallest contribution to the Generalization Error bound is removed. Moreover, the novel feature selection method is independent of the sample size and is computationally fast. The experimental results show that the proposed method consistently removes large percentages of features with statistically insignificant loss of testing accuracy for unseen samples. In the experiments for two of the datasets, the classifiers built using feature subsets with 90% of features removed by our proposed approach yield average testing accuracies higher than those trained using the full set of features. Finally, we corroborate the efficacy of the model by using it to predict corporate bankruptcies in the US.
-
localized Generalization Error model and its application to architecture selection for radial basis function neural network
IEEE Transactions on Neural Networks, 2007Co-Authors: Daniel S Yeung, Defeng Wang, Wing W Y Ng, Eric C C Tsang, Xizhao WangAbstract:The Generalization Error bounds found by current Error models using the number of effective parameters of a classifier and the number of training samples are usually very loose. These bounds are intended for the entire input space. However, support vector machine (SVM), radial basis function neural network (RBFNN), and multilayer perceptron neural network (MLPNN) are local learning machines for solving problems and treat unseen samples near the training samples to be more important. In this paper, we propose a localized Generalization Error model which bounds from above the Generalization Error within a neighborhood of the training samples using stochastic sensitivity measure. It is then used to develop an architecture selection technique for a classifier with maximal coverage of unseen samples by specifying a Generalization Error threshold. Experiments using 17 University of California at Irvine (UCI) data sets show that, in comparison with cross validation (CV), sequential learning, and two other ad hoc methods, our technique consistently yields the best testing classification accuracy with fewer hidden neurons and less training time.
-
image classification with the use of radial basis function neural networks and the minimization of the localized Generalization Error
Pattern Recognition, 2007Co-Authors: Andres Dorado, Daniel S Yeung, Witold Pedrycz, Ebroul IzquierdoAbstract:Image classification arises as an important phase in the overall process of automatic image annotation and image retrieval. In this study, we are concerned with the design of image classifiers developed in the feature space formed by low level primitives defined in the setting of the MPEG-7 standard. Our objective is to investigate the discriminatory properties of such standard image descriptors and look at efficient architectures of the classifiers along with their design pursuits. The Generalization capabilities of an image classifier are essential to its successful usage in image retrieval and annotation. Intuitively, it is expected that the classifier should achieve high classification accuracy on unseen images that are quite ''similar'' to those occurring in the training set. On the other hand, we may assume that the performance of the classifier could not be guaranteed in the case of images that are very much dissimilar from the elements of the training set. To follow this observation, we develop and use a concept of the localized Generalization Error and show how it guides the design of the classifier. As image classifier, we consider the usage of the radial basis function neural networks (RBFNNs). Through intensive experimentation we show that the resulting classifier outperforms other classifiers such as a multi-class support vector machines (SVMs) as well as ''standard'' RBFNNs (viz. those developed without the guidance offered by the optimization of the localized Generalization Error). The experimental studies reveal some interesting interpretation abilities of the RBFNN classifiers being related with their receptive fields.
Xizhao Wang - One of the best experts on this subject based on the ideXlab platform.
-
feature selection using localized Generalization Error for supervised classification problems using rbfnn
Pattern Recognition, 2008Co-Authors: Daniel S Yeung, Michael Firth, E C C Tsang, Xizhao WangAbstract:A pattern classification problem usually involves using high-dimensional features that make the classifier very complex and difficult to train. With no feature reduction, both training accuracy and Generalization capability will suffer. This paper proposes a novel hybrid filter-wrapper-type feature subset selection methodology using a localized Generalization Error model. The localized Generalization Error model for a radial basis function neural network bounds from above the Generalization Error for unseen samples located within a neighborhood of the training samples. Iteratively, the feature making the smallest contribution to the Generalization Error bound is removed. Moreover, the novel feature selection method is independent of the sample size and is computationally fast. The experimental results show that the proposed method consistently removes large percentages of features with statistically insignificant loss of testing accuracy for unseen samples. In the experiments for two of the datasets, the classifiers built using feature subsets with 90% of features removed by our proposed approach yield average testing accuracies higher than those trained using the full set of features. Finally, we corroborate the efficacy of the model by using it to predict corporate bankruptcies in the US.
-
localized Generalization Error model and its application to architecture selection for radial basis function neural network
IEEE Transactions on Neural Networks, 2007Co-Authors: Daniel S Yeung, Defeng Wang, Wing W Y Ng, Eric C C Tsang, Xizhao WangAbstract:The Generalization Error bounds found by current Error models using the number of effective parameters of a classifier and the number of training samples are usually very loose. These bounds are intended for the entire input space. However, support vector machine (SVM), radial basis function neural network (RBFNN), and multilayer perceptron neural network (MLPNN) are local learning machines for solving problems and treat unseen samples near the training samples to be more important. In this paper, we propose a localized Generalization Error model which bounds from above the Generalization Error within a neighborhood of the training samples using stochastic sensitivity measure. It is then used to develop an architecture selection technique for a classifier with maximal coverage of unseen samples by specifying a Generalization Error threshold. Experiments using 17 University of California at Irvine (UCI) data sets show that, in comparison with cross validation (CV), sequential learning, and two other ad hoc methods, our technique consistently yields the best testing classification accuracy with fewer hidden neurons and less training time.
-
localized Generalization Error and its application to rbfnn training
International Conference on Machine Learning and Cybernetics, 2005Co-Authors: Defeng Wang, Xizhao WangAbstract:The Generalization Error bounds for the entire input space found by current Error models using the number of effective parameters of a classifier and the number of training samples are usually very loose. But classifiers such as SVM, RBFNN and MLPNN, are really local learning machines used for many application problems, which consider unseen samples close to the training samples more important. In this paper, we propose a localized Generalization Error model which bounds above the Generalization Error within a neighborhood of the training samples using stochastic sensitivity measure (expectation of the squared output perturbations). It is then used to develop a model selection technique for a classifier with maximal coverage of unseen samples by specifying a Generalization Error threshold. Experiments by using eight real world datasets show that, in comparison with cross-validation, sequential learning, and two other ad-hoc methods, our technique consistently yields the best testing classification accuracy with fewer hidden neurons and less training time.
Witold Pedrycz - One of the best experts on this subject based on the ideXlab platform.
-
maximizing minority accuracy for imbalanced pattern classification problems using cost sensitive localized Generalization Error model
Applied Soft Computing, 2021Co-Authors: Zhengxi Liu, Jianjun Zhang, Witold PedryczAbstract:Abstract Traditional machine learning methods may not yield satisfactory Generalization capability when samples in different classes are imbalanced. These methods tend to sacrifice the accuracy of the minority class to improve the overall accuracy without regarding the fact that misclassifications of minority samples usually costs more in many real world applications. Therefore, we propose a neural network training method via a minimization of the cost-sensitive localized Generalization Error-based objective function (c-LGEM) to achieve a better balance of Error yielded by the minority and the majority classes. The c-LGEM emphasizes the minimization of the Generalization Error of the minority class in a cost-sensitive manner. Experimental results obtained on 16 UCI datasets show that neural networks trained by the c-LGEM yield better performance in comparison to the performance yielded by some existing methods.
-
image classification with the use of radial basis function neural networks and the minimization of the localized Generalization Error
Pattern Recognition, 2007Co-Authors: Andres Dorado, Daniel S Yeung, Witold Pedrycz, Ebroul IzquierdoAbstract:Image classification arises as an important phase in the overall process of automatic image annotation and image retrieval. In this study, we are concerned with the design of image classifiers developed in the feature space formed by low level primitives defined in the setting of the MPEG-7 standard. Our objective is to investigate the discriminatory properties of such standard image descriptors and look at efficient architectures of the classifiers along with their design pursuits. The Generalization capabilities of an image classifier are essential to its successful usage in image retrieval and annotation. Intuitively, it is expected that the classifier should achieve high classification accuracy on unseen images that are quite ''similar'' to those occurring in the training set. On the other hand, we may assume that the performance of the classifier could not be guaranteed in the case of images that are very much dissimilar from the elements of the training set. To follow this observation, we develop and use a concept of the localized Generalization Error and show how it guides the design of the classifier. As image classifier, we consider the usage of the radial basis function neural networks (RBFNNs). Through intensive experimentation we show that the resulting classifier outperforms other classifiers such as a multi-class support vector machines (SVMs) as well as ''standard'' RBFNNs (viz. those developed without the guidance offered by the optimization of the localized Generalization Error). The experimental studies reveal some interesting interpretation abilities of the RBFNN classifiers being related with their receptive fields.