The Experts below are selected from a list of 21573 Experts worldwide ranked by ideXlab platform
Andrew Y Ng - One of the best experts on this subject based on the ideXlab platform.
-
Unsupervised learning of hierarchical representations with convolutional deep Belief Networks
Commun. ACM, 2011Co-Authors: Honglak Lee, Roger Grosse, Rajesh Ranganath, Andrew Y NgAbstract:There has been much interest in unsupervised learning of hierarchical generative models such as deep Belief Networks (DBNs); however, scaling such models to full-sized, high-dimensional images remains a difficult problem. To address this problem, we present the convolutional deep Belief network, a hierarchical generative model that scales to realistic image sizes. This model is translation-invariant and supports efficient bottom-up and top-down probabilistic inference. Key to our approach is probabilistic max-pooling, a novel technique that shrinks the representations of higher layers in a probabilistically sound way. Our experiments show that the algorithm learns useful high-level visual features, such as object parts, from unlabeled images of objects and natural scenes. We demonstrate excellent performance on several visual recognition tasks and show that our model can perform hierarchical (bottom-up and top-down) inference over full-sized images.
-
unsupervised feature learning for audio classification using convolutional deep Belief Networks
Neural Information Processing Systems, 2009Co-Authors: Peter T Pham, Yan Largman, Andrew Y NgAbstract:In recent years, deep learning approaches have gained significant interest as a way of building hierarchical representations from unlabeled data. However, to our knowledge, these deep learning approaches have not been extensively studied for auditory data. In this paper, we apply convolutional deep Belief Networks to audio data and empirically evaluate them on various audio classification tasks. In the case of speech data, we show that the learned features correspond to phones/phonemes. In addition, our feature representations learned from unlabeled audio data show very good performance for multiple audio classification tasks. We hope that this paper will inspire more research on deep learning approaches applied to a wide range of audio recognition tasks.
Lawrence Carin - One of the best experts on this subject based on the ideXlab platform.
-
factored temporal sigmoid Belief Networks for sequence learning
International Conference on Machine Learning, 2016Co-Authors: Jiaming Song, Zhe Gan, Lawrence CarinAbstract:Deep conditional generative models are developed to simultaneously learn the temporal dependencies of multiple sequences. The model is designed by introducing a three-way weight tensor to capture the multiplicative interactions between side information and sequences. The proposed model builds on the Temporal Sigmoid Belief Network (TSBN), a sequential stack of Sigmoid Belief Networks (SBNs). The transition matrices are further factored to reduce the number of parameters and improve generalization. When side information is not available, a general framework for semi-supervised learning based on the proposed model is constituted, allowing robust sequence classification. Experimental results show that the proposed approach achieves state-of-the-art predictive and classification performance on sequential data, and has the capacity to synthesize sequences, with controlled style transitioning and blending.
-
deep temporal sigmoid Belief Networks for sequence modeling
arXiv: Machine Learning, 2015Co-Authors: Zhe Gan, Ricardo Henao, David E Carlson, Lawrence CarinAbstract:Deep dynamic generative models are developed to learn sequential dependencies in time-series data. The multi-layered model is designed by constructing a hierarchy of temporal sigmoid Belief Networks (TSBNs), defined as a sequential stack of sigmoid Belief Networks (SBNs). Each SBN has a contextual hidden state, inherited from the previous SBNs in the sequence, and is used to regulate its hidden bias. Scalable learning and inference algorithms are derived by introducing a recognition model that yields fast sampling from the variational posterior. This recognition model is trained jointly with the generative model, by maximizing its variational lower bound on the log-likelihood. Experimental results on bouncing balls, polyphonic music, motion capture, and text streams show that the proposed approach achieves state-of-the-art predictive performance, and has the capacity to synthesize various sequences.
-
learning deep sigmoid Belief Networks with data augmentation
International Conference on Artificial Intelligence and Statistics, 2015Co-Authors: Zhe Gan, Ricardo Henao, David E Carlson, Lawrence CarinAbstract:Deep directed generative models are developed. The multi-layered model is designed by stacking sigmoid Belief Networks, with sparsity-encouraging priors placed on the model parameters. Learning and inference of layer-wise model parameters are implemented in a Bayesian setting. By exploring the idea of data augmentation and introducing auxiliary P olya-Gamma variables, simple and ecient Gibbs sampling and meaneld variational Bayes (VB) inference are implemented. To address large-scale datasets, an online version of VB is also developed. Experimental results are presented for three publicly available datasets: MNIST, Caltech 101 Silhouettes and OCR letters.
Guido Montufar - One of the best experts on this subject based on the ideXlab platform.
-
universal approximation depth and errors of narrow Belief Networks with discrete units
Neural Computation, 2014Co-Authors: Guido MontufarAbstract:We generalize recent theoretical work on the minimal number of layers of narrow deep Belief Networks that can approximate any probability distribution on the states of their visible units arbitrarily well. We relax the setting of binary units Sutskever & Hinton, 2008; Le Roux & Bengio, 2008, 2010; Montufar & Ay, 2011 to units with arbitrary finite state spaces and the vanishing approximation error to an arbitrary approximation error tolerance. For example, we show that a q-ary deep Belief network with layers of width for some can approximate any probability distribution on without exceeding a Kullback-Leibler divergence of . Our analysis covers discrete restricted Boltzmann machines and naive Bayes models as special cases.
-
universal approximation depth and errors of narrow Belief Networks with discrete units
arXiv: Machine Learning, 2013Co-Authors: Guido MontufarAbstract:We generalize recent theoretical work on the minimal number of layers of narrow deep Belief Networks that can approximate any probability distribution on the states of their visible units arbitrarily well. We relax the setting of binary units (Sutskever and Hinton, 2008; Le Roux and Bengio, 2008, 2010; Montufar and Ay, 2011) to units with arbitrary finite state spaces, and the vanishing approximation error to an arbitrary approximation error tolerance. For example, we show that a $q$-ary deep Belief network with $L\geq 2+\frac{q^{\lceil m-\delta \rceil}-1}{q-1}$ layers of width $n \leq m + \log_q(m) + 1$ for some $m\in \mathbb{N}$ can approximate any probability distribution on $\{0,1,\ldots,q-1\}^n$ without exceeding a Kullback-Leibler divergence of $\delta$. Our analysis covers discrete restricted Boltzmann machines and naive Bayes models as special cases.
-
refinements of universal approximation results for deep Belief Networks and restricted boltzmann machines
Neural Computation, 2011Co-Authors: Guido MontufarAbstract:We improve recently published results about resources of restricted Boltzmann machines RBM and deep Belief Networks DBN required to make them universal approximators. We show that any distribution on the set of binary vectors of length can be arbitrarily well approximated by an RBM with hidden units, where is the minimal number of pairs of binary vectors differing in only one entry such that their union contains the support set of . In important cases this number is half the cardinality of the support set of given in Le Roux & Bengio, 2008. We construct a DBN with , hidden layers of width that is capable of approximating any distribution on arbitrarily well. This confirms a conjecture presented in Le Roux and Bengio 2010.
-
refinements of universal approximation results for deep Belief Networks and restricted boltzmann machines
arXiv: Machine Learning, 2010Co-Authors: Guido MontufarAbstract:We improve recently published results about resources of Restricted Boltzmann Machines (RBM) and Deep Belief Networks (DBN) required to make them Universal Approximators. We show that any distribution p on the set of binary vectors of length n can be arbitrarily well approximated by an RBM with k-1 hidden units, where k is the minimal number of pairs of binary vectors differing in only one entry such that their union contains the support set of p. In important cases this number is half of the cardinality of the support set of p. We construct a DBN with 2^n/2(n-b), b ~ log(n), hidden layers of width n that is capable of approximating any distribution on {0,1}^n arbitrarily well. This confirms a conjecture presented by Le Roux and Bengio 2010.
Yoshua Bengio - One of the best experts on this subject based on the ideXlab platform.
-
Deep Belief Networks Are Compact Universal Approximators
Neural Computation, 2010Co-Authors: Nicolas Le Roux, Yoshua BengioAbstract:Deep Belief Networks (DBN) are generative models with many layers of hidden causal variables, recently introduced by Hinton, Osindero, and Teh (2006), along with a greedy layer-wise unsupervised learning algorithm. Building on Le Roux and Bengio (2008) and Sutskever and Hinton (2008), we show that deep but narrow generative Networks do not require more parameters than shallow ones to achieve universal approximation. Exploiting the proof technique, we prove that deep but narrow feedforward neural Networks with sigmoidal units can represent any Boolean expression.
-
Representational Power of Restricted Boltzmann Machines and Deep Belief Networks
Neural Computation, 2008Co-Authors: Nicolas Le Roux, Yoshua BengioAbstract:Deep Belief Networks (DBN) are generative neural network models with many layers of hidden explanatory factors, recently introduced by Hinton, Osindero, and Teh (2006) along with a greedy layer-wise unsupervised learning algorithm. The building block of a DBN is a probabilistic model called a restricted Boltzmann machine (RBM), used to represent one layer of the model. Restricted Boltzmann machines are interesting because inference is easy in them and because they have been successfully used as building blocks for training deeper models. We first prove that adding hidden units yields strictly improved modeling power, while a second theorem shows that RBMs are universal approximators of discrete distributions. We then study the question of whether DBNs with more layers are strictly more powerful in terms of representational power. This suggests a new and less greedy criterion for training RBMs within DBNs.
Honglak Lee - One of the best experts on this subject based on the ideXlab platform.
-
Learning hierarchical representations for face verification with convolutional deep Belief Networks
Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2012Co-Authors: Gary B. Huang, Honglak Lee, Erik Learned-millerAbstract:Most modern face recognition systems rely on a feature representation given by a hand-crafted image descriptor, such as Local Binary Patterns (LBP), and achieve improved performance by combining several such representations. In this paper, we propose deep learning as a natural source for obtaining additional, complementary representations. To learn features in high-resolution images, we make use of convolutional deep Belief Networks. Moreover, to take advantage of global structure in an object class, we develop local convolutional restricted Boltzmann machines, a novel convolutional learning model that exploits the global structure by not assuming stationarity of features across the image, while maintaining scalability and robustness to small misalignments. We also present a novel application of deep learning to descriptors other than pixel intensity values, such as LBP. In addition, we compare performance of Networks trained using unsupervised learning against Networks with random filters, and empirically show that learning weights not only is necessary for obtaining good multilayer representations, but also provides robustness to the choice of the network architecture parameters. Finally, we show that a recognition system using only representations obtained from deep learning can achieve comparable accuracy with a system using a combination of hand-crafted image descriptors. Moreover, by combining these representations, we achieve state-of-the-art results on a real-world face verification database.
-
Unsupervised learning of hierarchical representations with convolutional deep Belief Networks
Commun. ACM, 2011Co-Authors: Honglak Lee, Roger Grosse, Rajesh Ranganath, Andrew Y NgAbstract:There has been much interest in unsupervised learning of hierarchical generative models such as deep Belief Networks (DBNs); however, scaling such models to full-sized, high-dimensional images remains a difficult problem. To address this problem, we present the convolutional deep Belief network, a hierarchical generative model that scales to realistic image sizes. This model is translation-invariant and supports efficient bottom-up and top-down probabilistic inference. Key to our approach is probabilistic max-pooling, a novel technique that shrinks the representations of higher layers in a probabilistically sound way. Our experiments show that the algorithm learns useful high-level visual features, such as object parts, from unlabeled images of objects and natural scenes. We demonstrate excellent performance on several visual recognition tasks and show that our model can perform hierarchical (bottom-up and top-down) inference over full-sized images.
-
convolutional deep Belief Networks for scalable unsupervised learning of hierarchical representations
International Conference on Machine Learning, 2009Co-Authors: Honglak Lee, Roger Grosse, Rajesh RanganathAbstract:There has been much interest in unsupervised learning of hierarchical generative models such as deep Belief Networks. Scaling such models to full-sized, high-dimensional images remains a difficult problem. To address this problem, we present the convolutional deep Belief network, a hierarchical generative model which scales to realistic image sizes. This model is translation-invariant and supports efficient bottom-up and top-down probabilistic inference. Key to our approach is probabilistic max-pooling, a novel technique which shrinks the representations of higher layers in a probabilistically sound way. Our experiments show that the algorithm learns useful high-level visual features, such as object parts, from unlabeled images of objects and natural scenes. We demonstrate excellent performance on several visual recognition tasks and show that our model can perform hierarchical (bottom-up and top-down) inference over full-sized images.
