The Experts below are selected from a list of 4809 Experts worldwide ranked by ideXlab platform

Martin J. Wainwright - One of the best experts on this subject based on the ideXlab platform.

  • sharp thresholds for high dimensional and noisy Sparsity recovery using ell _ 1 constrained quadratic programming lasso
    IEEE Transactions on Information Theory, 2009
    Co-Authors: Martin J. Wainwright
    Abstract:

    The problem of consistently estimating the Sparsity Pattern of a vector beta* isin Rp based on observations contaminated by noise arises in various contexts, including signal denoising, sparse approximation, compressed sensing, and model selection. We analyze the behavior of l1-constrained quadratic programming (QP), also referred to as the Lasso, for recovering the Sparsity Pattern. Our main result is to establish precise conditions on the problem dimension p, the number k of nonzero elements in beta*, and the number of observations n that are necessary and sufficient for Sparsity Pattern recovery using the Lasso. We first analyze the case of observations made using deterministic design matrices and sub-Gaussian additive noise, and provide sufficient conditions for support recovery and linfin-error bounds, as well as results showing the necessity of incoherence and bounds on the minimum value. We then turn to the case of random designs, in which each row of the design is drawn from a N (0, Sigma) ensemble. For a broad class of Gaussian ensembles satisfying mutual incoherence conditions, we compute explicit values of thresholds 0 0, if n > 2 (thetasu + delta) klog (p- k), then the Lasso succeeds in recovering the Sparsity Pattern with probability converging to one for large problems, whereas for n < 2 (thetasl - delta)klog (p - k), then the probability of successful recovery converges to zero. For the special case of the uniform Gaussian ensemble (Sigma = Iptimesp), we show that thetasl = thetas

  • Information-theoretic limits on Sparsity recovery in the high-dimensional and noisy setting
    arXiv: Statistics Theory, 2007
    Co-Authors: Martin J. Wainwright
    Abstract:

    The problem of recovering the Sparsity Pattern of a fixed but unknown vector $\beta^* \in \real^p based on a set of $n$ noisy observations arises in a variety of settings, including subset selection in regression, graphical model selection, signal denoising, compressive sensing, and constructive approximation. Of interest are conditions on the model dimension $p$, the Sparsity index $s$ (number of non-zero entries in $\beta^*$), and the number of observations $n$ that are necessary and/or sufficient to ensure asymptotically perfect recovery of the Sparsity Pattern. This paper focuses on the information-theoretic limits of Sparsity recovery: in particular, for a noisy linear observation model based on measurement vectors drawn from the standard Gaussian ensemble, we derive both a set of sufficient conditions for asymptotically perfect recovery using the optimal decoder, as well as a set of necessary conditions that any decoder, regardless of its computational complexity, must satisfy for perfect recovery. This analysis of optimal decoding limits complements our previous work (ARXIV: math.ST/0605740) on sharp thresholds for Sparsity recovery using the Lasso ($\ell_1$-constrained quadratic programming) with Gaussian measurement ensembles.

  • sharp thresholds for high dimensional and noisy recovery of Sparsity
    arXiv: Statistics Theory, 2006
    Co-Authors: Martin J. Wainwright
    Abstract:

    The problem of consistently estimating the Sparsity Pattern of a vector $\betastar \in \real^\mdim$ based on observations contaminated by noise arises in various contexts, including subset selection in regression, structure estimation in graphical models, sparse approximation, and signal denoising. We analyze the behavior of $\ell_1$-constrained quadratic programming (QP), also referred to as the Lasso, for recovering the Sparsity Pattern. Our main result is to establish a sharp relation between the problem dimension $\mdim$, the number $\spindex$ of non-zero elements in $\betastar$, and the number of observations $\numobs$ that are required for reliable recovery. For a broad class of Gaussian ensembles satisfying mutual incoherence conditions, we establish existence and compute explicit values of thresholds $\ThreshLow$ and $\ThreshUp$ with the following properties: for any $\epsilon > 0$, if $\numobs > 2 (\ThreshUp + \epsilon) \log (\mdim - \spindex) + \spindex + 1$, then the Lasso succeeds in recovering the Sparsity Pattern with probability converging to one for large problems, whereas for $\numobs < 2 (\ThreshLow - \epsilon) \log (\mdim - \spindex) + \spindex + 1$, then the probability of successful recovery converges to zero. For the special case of the uniform Gaussian ensemble, we show that $\ThreshLow = \ThreshUp = 1$, so that the threshold is sharp and exactly determined.

Laurent Callot - One of the best experts on this subject based on the ideXlab platform.

  • oracle inequalities for high dimensional vector autoregressions
    Journal of Econometrics, 2015
    Co-Authors: Anders Bredahl Kock, Laurent Callot
    Abstract:

    Abstract This paper establishes non-asymptotic oracle inequalities for the prediction error and estimation accuracy of the LASSO in stationary vector autoregressive models. These inequalities are used to establish consistency of the LASSO even when the number of parameters is of a much larger order of magnitude than the sample size. We also state conditions under which no relevant variables are excluded. Next, non-asymptotic probabilities are given for the adaptive LASSO to select the correct Sparsity Pattern. We then provide conditions under which the adaptive LASSO reveals the correct Sparsity Pattern asymptotically. We establish that the estimates of the non-zero coefficients are asymptotically equivalent to the oracle assisted least squares estimator. This is used to show that the rate of convergence of the estimates of the non-zero coefficients is identical to the one of least squares only including the relevant covariates.

  • oracle inequalities for high dimensional vector autoregressions
    2012
    Co-Authors: Anders Bredahl Kock, Laurent Callot
    Abstract:

    Abstract. This paper establishes non-asymptotic oracle inequalities for the prediction error and estimation accuracy of the LASSO in stationary vector autoregressive models. These inequalities are used to establish consistency of the LASSO even when the number of parameters is of a much larger order of magnitude than the sample size. We show that the number of variables selected is of the right order of magnitude and that no relevant variables are excluded.Next, non-asymptotic probabilities are given for the Adaptive LASSO to select the correct Sparsity Pattern. We then give conditions under which the Adaptive LASSO reveals the correct Sparsity Pattern asymptotically. We establish that the estimates of the non-zero coefficients are asymptotically equivalent to the oracle assisted least squares estimator. This is used to show that the rate of convergence of the estimates of the non-zero coefficients is identical to the one of least squares only including the relevant covariates.

  • oracle inequalities for high dimensional vector autoregressions
    CREATES Research Papers, 2012
    Co-Authors: Anders Bredahl Kock, Laurent Callot
    Abstract:

    This paper establishes non-asymptotic oracle inequalities for the prediction error and estimation accuracy of the LASSO in stationary vector autoregressive models. These inequalities are used to establish consistency of the LASSO even when the number of parameters is of a much larger order of magnitude than the sample size. Furthermore, it is shown that under suitable conditions the number of variables selected is of the right order of magnitude and that no relevant variables are excluded. Next, non-asymptotic probabilities are given for the Adaptive LASSO to select the correct sign Pattern (and hence the correct Sparsity Pattern). Finally conditions under which the Adaptive LASSO reveals the correct sign Pattern with probability tending to one are given. Again, the number of parameters may be much larger than the sample size. Some maximal inequalities for vector autoregressions which might be of independent interest are contained in the appendix.

Alexandre B Tsybakov - One of the best experts on this subject based on the ideXlab platform.

  • oracle inequalities and optimal inference under group Sparsity
    Annals of Statistics, 2011
    Co-Authors: Karim Lounici, Massimiliano Pontil, Sara Van De Geer, Alexandre B Tsybakov
    Abstract:

    We consider the problem of estimating a sparse linear regression vector s* under a gaussian noise model, for the purpose of both prediction and model selection. We assume that prior knowledge is available on the Sparsity Pattern, namely the set of variables is partitioned into prescribed groups, only few of which are relevant in the estimation process. This group Sparsity assumption suggests us to consider the Group Lasso method as a means to estimate s*. We establish oracle inequalities for the prediction and l2 estimation errors of this estimator. These bounds hold under a restricted eigenvalue condition on the design matrix. Under a stronger coherence condition, we derive bounds for the estimation error for mixed (2,p)-norms with 1=p=8. When p=8, this result implies that a threshold version of the Group Lasso estimator selects the Sparsity Pattern of s* with high probability. Next, we prove that the rate of convergence of our upper bounds is optimal in a minimax sense, up to a logarithmic factor, for all estimators over a class of group sparse vectors. Furthermore, we establish lower bounds for the prediction and l2 estimation errors of the usual Lasso estimator. Using this result, we demonstrate that the Group Lasso can achieve an improvement in the prediction and estimation properties as compared to the Lasso.

  • oracle inequalities and optimal inference under group Sparsity
    arXiv: Statistics Theory, 2010
    Co-Authors: Karim Lounici, Massimiliano Pontil, Alexandre B Tsybakov, Sara Van De Geer
    Abstract:

    We consider the problem of estimating a sparse linear regression vector $\beta^*$ under a gaussian noise model, for the purpose of both prediction and model selection. We assume that prior knowledge is available on the Sparsity Pattern, namely the set of variables is partitioned into prescribed groups, only few of which are relevant in the estimation process. This group Sparsity assumption suggests us to consider the Group Lasso method as a means to estimate $\beta^*$. We establish oracle inequalities for the prediction and $\ell_2$ estimation errors of this estimator. These bounds hold under a restricted eigenvalue condition on the design matrix. Under a stronger coherence condition, we derive bounds for the estimation error for mixed $(2,p)$-norms with $1\le p\leq \infty$. When $p=\infty$, this result implies that a threshold version of the Group Lasso estimator selects the Sparsity Pattern of $\beta^*$ with high probability. Next, we prove that the rate of convergence of our upper bounds is optimal in a minimax sense, up to a logarithmic factor, for all estimators over a class of group sparse vectors. Furthermore, we establish lower bounds for the prediction and $\ell_2$ estimation errors of the usual Lasso estimator. Using this result, we demonstrate that the Group Lasso can achieve an improvement in the prediction and estimation properties as compared to the Lasso.

  • Oracle Inequalities and Optimal Inference under Group Sparsity
    2010
    Co-Authors: Karim Lounici, Massimiliano Pontil, Alexandre B Tsybakov, Sara Van De Geer
    Abstract:

    We consider the problem of estimating a sparse linear regression vector $\beta^*$ under a gaussian noise model, for the purpose of both prediction and model selection. We assume that prior knowledge is available on the Sparsity Pattern, namely the set of variables is partitioned into prescribed groups, only few of which are relevant in the estimation process. This group Sparsity assumption suggests us to consider the Group Lasso method as a means to estimate $\beta^*$. We establish oracle inequalities for the prediction and $\ell_2$ estimation errors of this estimator. These bounds hold under a restricted eigenvalue condition on the design matrix. Under a stronger coherence condition, we derive bounds for the estimation error for mixed $(2,p)$-norms with $1\le p\leq \infty$. When $p=\infty$, this result implies that a threshold version of the Group Lasso estimator selects the Sparsity Pattern of $\beta^*$ with high probability. Next, we prove that the rate of convergence of our upper bounds is optimal in a minimax sense, up to a logarithmic factor, for all estimators over a class of group sparse vectors. Furthermore, we establish lower bounds for the prediction and $\ell_2$ estimation errors of the usual Lasso estimator. Using this result, we demonstrate that the Group Lasso can achieve an improvement in the prediction and estimation properties as compared to the Lasso. An important application of our results is provided by the problem of estimating multiple regression equation simultaneously or multi-task learning. In this case, our result lead to refinements of the results in \cite{colt2009} and allow one to establish the quantitative advantage of the Group Lasso over the usual Lasso in the multi-task setting. Finally, within the same setting, we show how our results can be extended to more general noise distributions, of which we only require the fourth moment to be finite. To obtain this extension, we establish a new maximal moment inequality, which may be of independent interest.

Pradeep K. Varshney - One of the best experts on this subject based on the ideXlab platform.

  • decentralized and collaborative subspace pursuit a communication efficient algorithm for joint Sparsity Pattern recovery with sensor networks
    IEEE Transactions on Signal Processing, 2016
    Co-Authors: Gang Li, Thakshila Wimalajeewa, Pradeep K. Varshney
    Abstract:

    In this paper, we consider the problem of joint Sparsity Pattern recovery in a distributed sensor network. The sparse multiple measurement vector signals (MMVs) observed by all the nodes are assumed to have a common (but unknown) Sparsity Pattern. To accurately recover the common Sparsity Pattern in a decentralized manner with a low communication overhead of the network, we develop an algorithm named decentralized and collaborative subspace pursuit (DCSP). In DCSP, each node is required to perform three kinds of operations per iteration: 1) estimate the local Sparsity Pattern by finding the subspace that its measurement vector most probably lies in; 2) share its local Sparsity Pattern estimate with one-hop neighboring nodes; and 3) update the final Sparsity Pattern estimate by majority vote based fusion of all the local Sparsity Pattern estimates obtained in its neighborhood. The convergence of DCSP is proved and its communication overhead is quantitatively analyzed. We also propose another decentralized algorithm named generalized DCSP (GDCSP) by allowing more information exchange among neighboring nodes to further improve the accuracy of Sparsity Pattern recovery at the cost of increased communication overhead. Experimental results show that, 1) compared with existing decentralized algorithms, DCSP provides much better accuracy of Sparsity Pattern recovery at a comparable communication cost; and 2) the accuracy of GDCSP is very close to that of centralized processing.

Karim Lounici - One of the best experts on this subject based on the ideXlab platform.

  • oracle inequalities and optimal inference under group Sparsity
    Annals of Statistics, 2011
    Co-Authors: Karim Lounici, Massimiliano Pontil, Sara Van De Geer, Alexandre B Tsybakov
    Abstract:

    We consider the problem of estimating a sparse linear regression vector s* under a gaussian noise model, for the purpose of both prediction and model selection. We assume that prior knowledge is available on the Sparsity Pattern, namely the set of variables is partitioned into prescribed groups, only few of which are relevant in the estimation process. This group Sparsity assumption suggests us to consider the Group Lasso method as a means to estimate s*. We establish oracle inequalities for the prediction and l2 estimation errors of this estimator. These bounds hold under a restricted eigenvalue condition on the design matrix. Under a stronger coherence condition, we derive bounds for the estimation error for mixed (2,p)-norms with 1=p=8. When p=8, this result implies that a threshold version of the Group Lasso estimator selects the Sparsity Pattern of s* with high probability. Next, we prove that the rate of convergence of our upper bounds is optimal in a minimax sense, up to a logarithmic factor, for all estimators over a class of group sparse vectors. Furthermore, we establish lower bounds for the prediction and l2 estimation errors of the usual Lasso estimator. Using this result, we demonstrate that the Group Lasso can achieve an improvement in the prediction and estimation properties as compared to the Lasso.

  • oracle inequalities and optimal inference under group Sparsity
    arXiv: Statistics Theory, 2010
    Co-Authors: Karim Lounici, Massimiliano Pontil, Alexandre B Tsybakov, Sara Van De Geer
    Abstract:

    We consider the problem of estimating a sparse linear regression vector $\beta^*$ under a gaussian noise model, for the purpose of both prediction and model selection. We assume that prior knowledge is available on the Sparsity Pattern, namely the set of variables is partitioned into prescribed groups, only few of which are relevant in the estimation process. This group Sparsity assumption suggests us to consider the Group Lasso method as a means to estimate $\beta^*$. We establish oracle inequalities for the prediction and $\ell_2$ estimation errors of this estimator. These bounds hold under a restricted eigenvalue condition on the design matrix. Under a stronger coherence condition, we derive bounds for the estimation error for mixed $(2,p)$-norms with $1\le p\leq \infty$. When $p=\infty$, this result implies that a threshold version of the Group Lasso estimator selects the Sparsity Pattern of $\beta^*$ with high probability. Next, we prove that the rate of convergence of our upper bounds is optimal in a minimax sense, up to a logarithmic factor, for all estimators over a class of group sparse vectors. Furthermore, we establish lower bounds for the prediction and $\ell_2$ estimation errors of the usual Lasso estimator. Using this result, we demonstrate that the Group Lasso can achieve an improvement in the prediction and estimation properties as compared to the Lasso.

  • Oracle Inequalities and Optimal Inference under Group Sparsity
    2010
    Co-Authors: Karim Lounici, Massimiliano Pontil, Alexandre B Tsybakov, Sara Van De Geer
    Abstract:

    We consider the problem of estimating a sparse linear regression vector $\beta^*$ under a gaussian noise model, for the purpose of both prediction and model selection. We assume that prior knowledge is available on the Sparsity Pattern, namely the set of variables is partitioned into prescribed groups, only few of which are relevant in the estimation process. This group Sparsity assumption suggests us to consider the Group Lasso method as a means to estimate $\beta^*$. We establish oracle inequalities for the prediction and $\ell_2$ estimation errors of this estimator. These bounds hold under a restricted eigenvalue condition on the design matrix. Under a stronger coherence condition, we derive bounds for the estimation error for mixed $(2,p)$-norms with $1\le p\leq \infty$. When $p=\infty$, this result implies that a threshold version of the Group Lasso estimator selects the Sparsity Pattern of $\beta^*$ with high probability. Next, we prove that the rate of convergence of our upper bounds is optimal in a minimax sense, up to a logarithmic factor, for all estimators over a class of group sparse vectors. Furthermore, we establish lower bounds for the prediction and $\ell_2$ estimation errors of the usual Lasso estimator. Using this result, we demonstrate that the Group Lasso can achieve an improvement in the prediction and estimation properties as compared to the Lasso. An important application of our results is provided by the problem of estimating multiple regression equation simultaneously or multi-task learning. In this case, our result lead to refinements of the results in \cite{colt2009} and allow one to establish the quantitative advantage of the Group Lasso over the usual Lasso in the multi-task setting. Finally, within the same setting, we show how our results can be extended to more general noise distributions, of which we only require the fourth moment to be finite. To obtain this extension, we establish a new maximal moment inequality, which may be of independent interest.