The Experts below are selected from a list of 5952 Experts worldwide ranked by ideXlab platform
Shie Mannor - One of the best experts on this subject based on the ideXlab platform.
-
sparse algorithms are Not stable a No Free Lunch theorem
IEEE Transactions on Pattern Analysis and Machine Intelligence, 2012Co-Authors: Constantine Caramanis, Shie MannorAbstract:We consider two desired properties of learning algorithms: sparsity and algorithmic stability. Both properties are believed to lead to good generalization ability. We show that these two properties are fundamentally at odds with each other: A sparse algorithm canNot be stable and vice versa. Thus, one has to trade off sparsity and stability in designing a learning algorithm. In particular, our general result implies that l1-regularized regression (Lasso) canNot be stable, while l2-regularized regression is kNown to have strong stability properties and is therefore Not sparse.
-
sparse algorithms are Not stable a No Free Lunch theorem
Allerton Conference on Communication Control and Computing, 2008Co-Authors: Shie Mannor, Constantine CaramanisAbstract:We consider two widely used Notions in machine learning, namely: sparsity and algorithmic stability. Both Notions are deemed desirable in designing algorithms, and are believed to lead to good generalization ability. In this paper, we show that these two Notions contradict each other. That is, a sparse algorithm can Not be stable and vice versa. Thus, one has to tradeoff sparsity and stability in designing a learning algorithm. In particular, our general result implies that lscr1-regularized regression (Lasso) canNot be stable, while lscr2-regularized regression is kNown to have strong stability properties.
Constantine Caramanis - One of the best experts on this subject based on the ideXlab platform.
-
sparse algorithms are Not stable a No Free Lunch theorem
IEEE Transactions on Pattern Analysis and Machine Intelligence, 2012Co-Authors: Constantine Caramanis, Shie MannorAbstract:We consider two desired properties of learning algorithms: sparsity and algorithmic stability. Both properties are believed to lead to good generalization ability. We show that these two properties are fundamentally at odds with each other: A sparse algorithm canNot be stable and vice versa. Thus, one has to trade off sparsity and stability in designing a learning algorithm. In particular, our general result implies that l1-regularized regression (Lasso) canNot be stable, while l2-regularized regression is kNown to have strong stability properties and is therefore Not sparse.
-
sparse algorithms are Not stable a No Free Lunch theorem
Allerton Conference on Communication Control and Computing, 2008Co-Authors: Shie Mannor, Constantine CaramanisAbstract:We consider two widely used Notions in machine learning, namely: sparsity and algorithmic stability. Both Notions are deemed desirable in designing algorithms, and are believed to lead to good generalization ability. In this paper, we show that these two Notions contradict each other. That is, a sparse algorithm can Not be stable and vice versa. Thus, one has to tradeoff sparsity and stability in designing a learning algorithm. In particular, our general result implies that lscr1-regularized regression (Lasso) canNot be stable, while lscr2-regularized regression is kNown to have strong stability properties.
David H. Wolpert - One of the best experts on this subject based on the ideXlab platform.
-
the implications of the No Free Lunch theorems for meta induction
arXiv: Learning, 2021Co-Authors: David H. WolpertAbstract:The important recent book by G. Schurz appreciates that the No-Free-Lunch theorems (NFL) have major implications for the problem of (meta) induction. Here I review the NFL theorems, emphasizing that they do Not only concern the case where there is a uniform prior -- they prove that there are ``as many priors'' (loosely speaking) for which any induction algorithm $A$ out-generalizes some induction algorithm $B$ as vice-versa. Importantly though, in addition to the NFL theorems, there are many \textit{Free Lunch} theorems. In particular, the NFL theorems can only be used to compare the \textit{marginal} expected performance of an induction algorithm $A$ with the marginal expected performance of an induction algorithm $B$. There is a rich set of Free Lunches which instead concern the statistical correlations among the generalization errors of induction algorithms. As I describe, the meta-induction algorithms that Schurz advocate as a ``solution to Hume's problem'' are just an example of such a Free Lunch based on correlations among the generalization errors of induction algorithms. I end by pointing out that the prior that Schurz advocates, which is uniform over bit frequencies rather than bit patterns, is contradicted by thousands of experiments in statistical physics and by the great success of the maximum entropy procedure in inductive inference.
-
what the No Free Lunch theorems really mean how to improve search algorithms
2013Co-Authors: David H. WolpertAbstract:The first No Free Lunch (NFL) theorems were introduced in [9], in the contextof supervised machine learning. These theorems were then popularized in [8],based on a preprint version of [9]. Loosely speaking, these original theorems canbe viewed as a formalization and elaboration of concerns about the legitimacyof inductive inference, concerns that date back to David Hume (if Not earlier).Shortly after these original theorems were published, additional NFL theoremsthat apply to search were introduced in [12].The NFL theorems have stimulated lots of subsequent work, with over 2500citations of [12] alone by spring 2012 according to Google Scholar. However ar-guably much of that research has missed the most important implications of thetheorems. As stated in [12], the primary importance of the NFL theorems forsearch is what they tell us about “the underlying mathematical ‘skeleton’ of op-timization theory before the ‘flesh’ of the probability distributions of a particularcontext and set of optimization problems are imposed”. So in particular, while theNFL theorems have strong implications if one believes in a uniform distributioNover optimization problems, in No sense should they be interpreted as advocatingsuch a distribution.1
-
The Supervised Learning No-Free-Lunch Theorems
Soft Computing and Industry, 2002Co-Authors: David H. WolpertAbstract:This paper reviews the supervised learning versions of the No-Free-Lunch theorems in a simplified form. It also discusses the significance of those theorems, and their relation to other aspects of supervised learning.
-
Remarks on a recent paper on the "No Free Lunch" theorems
IEEE Transactions on Evolutionary Computation, 2001Co-Authors: Mario Köppen, David H. Wolpert, William G. MacreadyAbstract:This Note discusses the recent paper "Some technical remarks on the proof of the No Free Lunch theorem" by Koppen (2000). In that paper, some technical issues related to the formal proof of the No Free Lunch (NFL) theorem for search were given by Wolpert and Macready (1995, 1997). The present authors explore the issues raised in that paper including the presentation of a simpler version of the NFL proof in accord with a suggestion made explicitly by Koppen (2000) and implicitly by Wolpert and Macready (1997). They also includes the correction of an incorrect claim made by Koppen (2000) of a limitation of the NFL theorem. Finally, some thoughts on future research directions for research into algorithm performance are given.
-
No Free Lunch theorems for optimization
IEEE Transactions on Evolutionary Computation, 1997Co-Authors: David H. Wolpert, William G. MacreadyAbstract:A framework is developed to explore the connection between effective optimization algorithms and the problems they are solving. A number of “No Free Lunch” (NFL) theorems are presented which establish that for any algorithm, any elevated performance over one class of problems is offset by performance over aNother class. These theorems result in a geometric interpretation of what it means for an algorithm to be well suited to an optimization problem. Applications of the NFL theorems to information-theoretic aspects of optimization and benchmark measures of performance are also presented. Other issues addressed include time-varying optimization problems and a priori “head-to-head” minimax distinctions between optimization algorithms, distinctions that result despite the NFL theorems' enforcing of a type of uniformity over all algorithms
Charlotte Haug - One of the best experts on this subject based on the ideXlab platform.
-
No Free Lunch — What Price Plan S for Scientific Publishing?
The New England journal of medicine, 2019Co-Authors: Charlotte HaugAbstract:No Free Lunch The evidence to date does Not confirm the hypothesis that the Gold Open Access publication model will advance science and be an unprecedented public good. So this is a surprising time to insist that scientific content should be “Free” online, as Europe’s Plan S aims to do.
-
No Free Lunch what price plan s for scientific publishing
The New England Journal of Medicine, 2019Co-Authors: Charlotte HaugAbstract:No Free Lunch The evidence to date does Not confirm the hypothesis that the Gold Open Access publication model will advance science and be an unprecedented public good. So this is a surprising time to insist that scientific content should be “Free” online, as Europe’s Plan S aims to do.
D.l. Pepyne - One of the best experts on this subject based on the ideXlab platform.
-
the No Free Lunch theorems complexity and security
IEEE Transactions on Automatic Control, 2003Co-Authors: Qianchuan Zhao, D.l. PepyneAbstract:One of the main challenges for decision scientists in the 21st century will be managing systems of ever increasing complexity. As systems like electrical power grids, computer networks, and the software that controls it all grow increasingly complex, fragility, bugs, and security flaws are becoming increasingly prevalent and problematic. It is natural then to ask what consequences this growing complexity has on our ability to manage these systems. In this paper, we take a first step toward addressing this question with the development of the fundamental matrix, a framework for analyzing the broad qualitative nature of decision making. With the fundamental matrix we explain in a qualitative way many theorems and kNown results about optimization, complexity, and security. The simplicity of the explanations leads to new insights toward potential research directions. Like other "theories" dealing with broad fundamental properties, however, the fundamental matrix has certain limitations that make it largely descriptive. Thus, instead of claiming the last words our goal is to stimulate a dialog and debate that may one day lead to a prescriptive science of complexity.
-
Simple explanation of the No Free Lunch theorem of optimization
Cybernetics and Systems Analysis, 2002Co-Authors: D.l. PepyneAbstract:The No Free Lunch theorem of optimization (NFLT) is an impossibility theorem telling us that a general-purpose universal optimization strategy is impossible, and the only way one strategy can outperform aNother is if it is specialized to the structure of the specific problem under consideration. Since virtually all decision and control problems can be cast as optimization problems, an appreciation of the NFLT and its consequences is essential for control engineers. We present a framework for conceptualizing optimization problems that leads useful insights and a simple explanation of the NFLT.
-
Simple Explanation of the No-Free-Lunch Theorem and Its Implications
Journal of Optimization Theory and Applications, 2002Co-Authors: D.l. PepyneAbstract:The No-Free-Lunch theorem of optimization (NFLT) is an impossibility theorem telling us that a general-purpose, universal optimization strategy is impossible. The only way one strategy can outperform aNother is if it is specialized to the structure of the specific problem under consideration. Since optimization is a central human activity, an appreciation of the NFLT and its consequences is essential. In this paper, we present a framework for conceptualizing optimization that leads to a simple but rigorous explanation of the NFLT and its implications.
