The Experts below are selected from a list of 10551 Experts worldwide ranked by ideXlab platform

Torsten Hothorn - One of the best experts on this subject based on the ideXlab platform.

Peter Buhlmann - One of the best experts on this subject based on the ideXlab platform.

Mansoor Saqi - One of the best experts on this subject based on the ideXlab platform.

  • handling missing features with Boosting Algorithms for protein protein interaction prediction
    Data Integration in the Life Sciences, 2010
    Co-Authors: Fabrizio Smeraldi, Michael Defoinplatel, Mansoor Saqi
    Abstract:

    Combining information from multiple heterogeneous data sources can aid prediction of protein-protein interaction. This information can be arranged into a feature vector for classification. However, missing values in the data can impact on the prediction accuracy. Boosting has emerged as a powerful tool for feature selection and classification. Bayesian methods have traditionally been used to cope with missing data, with Boosting being applied to the output of Bayesian classifiers. We explore a variation of Adaboost that deals with the missing values at the level of the Boosting algorithm itself, without the need for any density estimation step. Experiments on a publicly available PPI dataset suggest this overall simpler and mathematically coherent approach may be more accurate.

  • DILS - Handling missing features with Boosting Algorithms for protein-protein interaction prediction
    Lecture Notes in Computer Science, 2010
    Co-Authors: Fabrizio Smeraldi, Michael Defoin-platel, Mansoor Saqi
    Abstract:

    Combining information from multiple heterogeneous data sources can aid prediction of protein-protein interaction. This information can be arranged into a feature vector for classification. However, missing values in the data can impact on the prediction accuracy. Boosting has emerged as a powerful tool for feature selection and classification. Bayesian methods have traditionally been used to cope with missing data, with Boosting being applied to the output of Bayesian classifiers. We explore a variation of Adaboost that deals with the missing values at the level of the Boosting algorithm itself, without the need for any density estimation step. Experiments on a publicly available PPI dataset suggest this overall simpler and mathematically coherent approach may be more accurate.

Martin J Wainwright - One of the best experts on this subject based on the ideXlab platform.

  • early stopping for kernel Boosting Algorithms a general analysis with localized complexities
    IEEE Transactions on Information Theory, 2019
    Co-Authors: Yuting Wei, Fanny Yang, Martin J Wainwright
    Abstract:

    Early stopping of iterative Algorithms is a widely used form of regularization in statistics, commonly used in conjunction with Boosting and related gradient-type Algorithms. Although consistency results have been established in some settings, such estimators are less well-understood than their analogues based on penalized regularization. In this paper, for a relatively broad class of loss functions and Boosting Algorithms (including $L^{2}$ -boost, LogitBoost, and AdaBoost, among others), we exhibit a direct connection between the performance of a stopped iterate and the localized Gaussian complexity of the associated function class. This connection allows us to show that the local fixed point analysis of Gaussian or Rademacher complexities, now standard in the analysis of penalized estimators, can be used to derive optimal stopping rules. We derive such stopping rules in detail for various kernel classes and illustrate the correspondence of our theory with practice for Sobolev kernel classes.

  • Early stopping for kernel Boosting Algorithms: A general analysis with localized complexities
    arXiv: Machine Learning, 2017
    Co-Authors: Yuting Wei, Fanny Yang, Martin J Wainwright
    Abstract:

    Early stopping of iterative Algorithms is a widely-used form of regularization in statistics, commonly used in conjunction with Boosting and related gradient-type Algorithms. Although consistency results have been established in some settings, such estimators are less well-understood than their analogues based on penalized regularization. In this paper, for a relatively broad class of loss functions and Boosting Algorithms (including L2-boost, LogitBoost and AdaBoost, among others), we exhibit a direct connection between the performance of a stopped iterate and the localized Gaussian complexity of the associated function class. This connection allows us to show that local fixed point analysis of Gaussian or Rademacher complexities, now standard in the analysis of penalized estimators, can be used to derive optimal stopping rules. We derive such stopping rules in detail for various kernel classes, and illustrate the correspondence of our theory with practice for Sobolev kernel classes.

  • early stopping for kernel Boosting Algorithms a general analysis with localized complexities
    Neural Information Processing Systems, 2017
    Co-Authors: Yuting Wei, Fanny Yang, Martin J Wainwright
    Abstract:

    Early stopping of iterative Algorithms is a widely-used form of regularization in statistical learning, commonly used in conjunction with Boosting and related gradient-type Algorithms. Although consistency results have been established in some settings, such estimators are less well-understood than their analogues based on penalized regularization. In this paper, for a relatively broad class of loss functions and Boosting Algorithms (including $L^2$-boost, LogitBoost and AdaBoost, among others), we connect the performance of a stopped iterate to the localized Rademacher/Gaussian complexity of the associated function class. This connection allows us to show that local fixed point analysis, now standard in the analysis of penalized estimators, can be used to derive optimal stopping rules. We derive such stopping rules in detail for various kernel classes, and illustrate the correspondence of our theory with practice for Sobolev kernel classes.

  • NIPS - Early stopping for kernel Boosting Algorithms: A general analysis with localized complexities
    2017
    Co-Authors: Yuting Wei, Fanny Yang, Martin J Wainwright
    Abstract:

    Early stopping of iterative Algorithms is a widely-used form of regularization in statistical learning, commonly used in conjunction with Boosting and related gradient-type Algorithms. Although consistency results have been established in some settings, such estimators are less well-understood than their analogues based on penalized regularization. In this paper, for a relatively broad class of loss functions and Boosting Algorithms (including $L^2$-boost, LogitBoost and AdaBoost, among others), we connect the performance of a stopped iterate to the localized Rademacher/Gaussian complexity of the associated function class. This connection allows us to show that local fixed point analysis, now standard in the analysis of penalized estimators, can be used to derive optimal stopping rules. We derive such stopping rules in detail for various kernel classes, and illustrate the correspondence of our theory with practice for Sobolev kernel classes.

Gunnar Ratsch - One of the best experts on this subject based on the ideXlab platform.

  • Boosting Algorithms for maximizing the soft margin
    Neural Information Processing Systems, 2007
    Co-Authors: Gunnar Ratsch, Manfred K Warmuth, Karen A Glocer
    Abstract:

    We present a novel Boosting algorithm, called SoftBoost, designed for sets of binary labeled examples that are not necessarily separable by convex combinations of base hypotheses. Our algorithm achieves robustness by capping the distributions on the examples. Our update of the distribution is motivated by minimizing a relative entropy subject to the capping constraints and constraints on the edges of the obtained base hypotheses. The capping constraints imply a soft margin in the dual optimization problem. Our algorithm produces a convex combination of hypotheses whose soft margin is within δ of its maximum. We employ relative entropy projection methods to prove an O(ln N/δ2) iteration bound for our algorithm, where N is number of examples. We compare our algorithm with other approaches including LPBoost, Brown-Boost, and SmoothBoost. We show that there exist cases where the number of iterations required by LPBoost grows linearly in N instead of the logarithmic growth for SoftBoost. In simulation studies we show that our algorithm converges about as fast as LPBoost, faster than BrownBoost, and much faster than SmoothBoost. In a benchmark comparison we illustrate the competitiveness of our approach.

  • NIPS - Boosting Algorithms for Maximizing the Soft Margin
    2007
    Co-Authors: Gunnar Ratsch, Manfred K Warmuth, Karen A Glocer
    Abstract:

    We present a novel Boosting algorithm, called SoftBoost, designed for sets of binary labeled examples that are not necessarily separable by convex combinations of base hypotheses. Our algorithm achieves robustness by capping the distributions on the examples. Our update of the distribution is motivated by minimizing a relative entropy subject to the capping constraints and constraints on the edges of the obtained base hypotheses. The capping constraints imply a soft margin in the dual optimization problem. Our algorithm produces a convex combination of hypotheses whose soft margin is within δ of its maximum. We employ relative entropy projection methods to prove an O(ln N/δ2) iteration bound for our algorithm, where N is number of examples. We compare our algorithm with other approaches including LPBoost, Brown-Boost, and SmoothBoost. We show that there exist cases where the number of iterations required by LPBoost grows linearly in N instead of the logarithmic growth for SoftBoost. In simulation studies we show that our algorithm converges about as fast as LPBoost, faster than BrownBoost, and much faster than SmoothBoost. In a benchmark comparison we illustrate the competitiveness of our approach.

  • totally corrective Boosting Algorithms that maximize the margin
    International Conference on Machine Learning, 2006
    Co-Authors: Manfred K Warmuth, Jun Liao, Gunnar Ratsch
    Abstract:

    We consider Boosting Algorithms that maintain a distribution over a set of examples. At each iteration a weak hypothesis is received and the distribution is updated. We motivate these updates as minimizing the relative entropy subject to linear constraints. For example AdaBoost constrains the edge of the last hypothesis w.r.t. the updated distribution to be at most γ = 0. In some sense, AdaBoost is "corrective" w.r.t. the last hypothesis. A cleaner Boosting method is to be "totally corrective": the edges of all past hypotheses are constrained to be at most γ, where γ is suitably adapted.Using new techniques, we prove the same iteration bounds for the totally corrective Algorithms as for their corrective versions. Moreover with adaptive γ, the Algorithms provably maximizes the margin. Experimentally, the totally corrective versions return smaller convex combinations of weak hypotheses than the corrective ones and are competitive with LPBoost, a totally corrective Boosting algorithm with no regularization, for which there is no iteration bound known.

  • ICML - Totally corrective Boosting Algorithms that maximize the margin
    Proceedings of the 23rd international conference on Machine learning - ICML '06, 2006
    Co-Authors: Manfred K Warmuth, Jun Liao, Gunnar Ratsch
    Abstract:

    We consider Boosting Algorithms that maintain a distribution over a set of examples. At each iteration a weak hypothesis is received and the distribution is updated. We motivate these updates as minimizing the relative entropy subject to linear constraints. For example AdaBoost constrains the edge of the last hypothesis w.r.t. the updated distribution to be at most γ = 0. In some sense, AdaBoost is "corrective" w.r.t. the last hypothesis. A cleaner Boosting method is to be "totally corrective": the edges of all past hypotheses are constrained to be at most γ, where γ is suitably adapted.Using new techniques, we prove the same iteration bounds for the totally corrective Algorithms as for their corrective versions. Moreover with adaptive γ, the Algorithms provably maximizes the margin. Experimentally, the totally corrective versions return smaller convex combinations of weak hypotheses than the corrective ones and are competitive with LPBoost, a totally corrective Boosting algorithm with no regularization, for which there is no iteration bound known.

  • constructing Boosting Algorithms from svms an application to one class classification
    IEEE Transactions on Pattern Analysis and Machine Intelligence, 2002
    Co-Authors: Gunnar Ratsch, Sebastian Mika, Bernhard Scholkopf, Klausrobert Muller
    Abstract:

    We show via an equivalence of mathematical programs that a support vector (SV) algorithm can be translated into an equivalent Boosting-like algorithm and vice versa. We exemplify this translation procedure for a new algorithm: one-class leveraging, starting from the one-class support vector machine (1-SVM). This is a first step toward unsupervised learning in a Boosting framework. Building on so-called barrier methods known from the theory of constrained optimization, it returns a function, written as a convex combination of base hypotheses, that characterizes whether a given test point is likely to have been generated from the distribution underlying the training data. Simulations on one-class classification problems demonstrate the usefulness of our approach.