The Experts below are selected from a list of 33357 Experts worldwide ranked by ideXlab platform
Markus Flierl - One of the best experts on this subject based on the ideXlab platform.
-
Vector Quantization-Based Regularization for Autoencoders
Proceedings of the AAAI Conference on Artificial Intelligence, 2020Co-Authors: Markus FlierlAbstract:Autoencoders and their variations provide unsupervised models for learning low-dimensional representations for downstream tasks. Without proper regularization, Autoencoder models are susceptible to the overfitting problem and the so-called posterior collapse phenomenon. In this paper, we introduce a quantization-based regularizer in the bottleneck stage of Autoencoder models to learn meaningful latent representations. We combine both perspectives of Vector Quantized-Variational Autoencoders (VQ-VAE) and classical denoising regularization methods of neural networks. We interpret quantizers as regularizers that constrain latent representations while fostering a similarity-preserving mapping at the encoder. Before quantization, we impose noise on the latent codes and use a Bayesian estimator to optimize the quantizer-based representation. The introduced bottleneck Bayesian estimator outputs the posterior mean of the centroids to the decoder, and thus, is performing soft quantization of the noisy latent codes. We show that our proposed regularization method results in improved latent representations for both supervised learning and clustering downstream tasks when compared to Autoencoders using other bottleneck structures.
-
Quantization-Based Regularization for Autoencoders.
arXiv: Learning, 2019Co-Authors: Hanwei Wu, Ather Gattami, Markus FlierlAbstract:Autoencoders and their variations provide unsupervised models for learning low-dimensional representations for downstream tasks. Without proper regularization, Autoencoder models are susceptible to the overfitting problem and the so-called posterior collapse phenomenon. In this paper, we introduce a quantization-based regularizer in the bottleneck stage of Autoencoder models to learn meaningful latent representations. We combine both perspectives of Vector Quantized-Variational Autoencoders (VQ-VAE) and classical denoising regularization methods of neural networks. We interpret quantizers as regularizers that constrain latent representations while fostering a similarity-preserving mapping at the encoder. Before quantization, we impose noise on the latent codes and use a Bayesian estimator to optimize the quantizer-based representation. The introduced bottleneck Bayesian estimator outputs the posterior mean of the centroids to the decoder, and thus, is performing soft quantization of the noisy latent codes. We show that our proposed regularization method results in improved latent representations for both supervised learning and clustering downstream tasks when compared to Autoencoders using other bottleneck structures.
Angshul Majumdar - One of the best experts on this subject based on the ideXlab platform.
-
IJCNN - Asymmetric stacked Autoencoder
2017 International Joint Conference on Neural Networks (IJCNN), 2017Co-Authors: Angshul Majumdar, Aditay TripathiAbstract:Traditional stacked Autoencoders have an equal number of encoders and decoders. However, while fine-tuned as a deep neural network the decoder portion is detached and never used. This begs the question: ‘do we need equal number of decoders and encoders’? In this study we explore asymmetric Autoencoders — unequal number of encoders and decoders. We specifically address two tasks — 1. Classification capacity as deep neural network and 2. Compressibility of stacked Autoencoder. For both the problems, our asymmetric Autoencoders have several encoders but a single decoders. We find that such Autoencoders are more accurate compared to traditional symmetrically stacked Autoencoders for classification accuracy and also yield slightly better results on compression problems.
-
ICONIP (2) - Semi Supervised Autoencoder
Neural Information Processing, 2016Co-Authors: Anupriya Gogna, Angshul MajumdarAbstract:Autoencoders are self-supervised learning tools, but are unsupervised in the sense that class information is not required for training; but almost invariably they are used for supervised classification tasks. We propose to learn the Autoencoder for a semi-supervised paradigm, i.e. with both labeled and unlabeled samples available. Given labeled and unlabeled data, our proposed Autoencoder automatically adjusts --- for unlabeled data it acts as a standard Autoencoder unsupervised and for labeled data it additionally learns a linear classifier. We use our proposed semi-supervised Autoencoder to greedily construct a stacked architecture. We demonstrate the efficacy our design in terms of both accuracy and run time requirements for the case of image classification. Our model is able to provide high classification accuracy with even simple classification schemes as compared to existing models for deep architectures.
-
IJCNN - Sparsely connected Autoencoder
2016 International Joint Conference on Neural Networks (IJCNN), 2016Co-Authors: Kavya Gupta, Angshul MajumdarAbstract:This work proposes to learn Autoencoders with sparse connections. Prior studies on Autoencoders enforced sparsity on the neuronal activity; these are different from our proposed approach - we learn sparse connections. Sparsity in connections helps in learning (and keeping) the important relations while trimming the irrelevant ones. We have tested the performance of our proposed method on two tasks - classification and denoising. For classification we have compared against stacked autneencoders, contractive Autoencoders, deep belief network, sparse deep neural network and optimal brain damage neural network; the denoising performance was compared against denoising Autoencoder and sparse (activity) Autoencoder. In both the tasks our proposed method yields superior results.
Shifei Ding - One of the best experts on this subject based on the ideXlab platform.
-
Research of stacked denoising sparse Autoencoder
Neural Computing and Applications, 2016Co-Authors: Lingheng Meng, Shifei Ding, Nan Zhang, Jian ZhangAbstract:Learning results depend on the representation of data, so how to efficiently represent data has been a research hot spot in machine learning and artificial intelligence. With the deepening of the deep learning research, studying how to train the deep networks to express high dimensional data efficiently also has been a research frontier. In order to present data more efficiently and study how to express data through deep networks, we propose a novel stacked denoising sparse Autoencoder in this paper. Firstly, we construct denoising sparse Autoencoder through introducing both corrupting operation and sparsity constraint into traditional Autoencoder. Then, we build stacked denoising sparse Autoencoders which has multi-hidden layers by layer-wisely stacking denoising sparse Autoencoders. Experiments are designed to explore the influences of corrupting operation and sparsity constraint on different datasets, using the networks with various depth and hidden units. The comparative experiments reveal that test accuracy of stacked denoising sparse Autoencoder is much higher than other stacked models, no matter what dataset is used and how many layers the model has. We also find that the deeper the network is, the less activated neurons in every layer will have. More importantly, we find that the strengthening of sparsity constraint is to some extent equal to the increase in corrupted level.
-
Research on denoising sparse Autoencoder
International Journal of Machine Learning and Cybernetics, 2016Co-Authors: Lingheng Meng, Shifei DingAbstract:Autoencoder can learn the structure of data adaptively and represent data efficiently. These properties make Autoencoder not only suit huge volume and variety of data well but also overcome expensive designing cost and poor generalization. Moreover, using Autoencoder in deep learning to implement feature extraction could draw better classification accuracy. However, there exist poor robustness and overfitting problems when utilizing Autoencoder. In order to extract useful features, meanwhile improve robustness and overcome overfitting, we studied denoising sparse Autoencoder through adding corrupting operation and sparsity constraint to traditional Autoencoder. The results suggest that different Autoencoders mentioned in this paper have some close relation and the model we researched can extract interesting features which can reconstruct original data well. In addition, all results show a promising approach to utilizing the proposed Autoencoder to build deep models.
Francesco Piazza - One of the best experts on this subject based on the ideXlab platform.
-
IJCNN - Acoustic novelty detection with adversarial Autoencoders
2017 International Joint Conference on Neural Networks (IJCNN), 2017Co-Authors: Emanuele Principi, Fabio Vesperini, Stefano Squartini, Francesco PiazzaAbstract:Novelty detection is the task of recognising events the differ from a model of normality. This paper proposes an acoustic novelty detector based on neural networks trained with an adversarial training strategy. The proposed approach is composed of a feature extraction stage that calculates Log-Mel spectral features from the input signal. Then, an Autoencoder network, trained on a corpus of “normal” acoustic signals, is employed to detect whether a segment contains an abnormal event or not. A novelty is detected if the Euclidean distance between the input and the output of the Autoencoder exceeds a certain threshold. The innovative contribution of the proposed approach resides in the training procedure of the Autoencoder network: instead of using the conventional training procedure that minimises only the Minimum Mean Squared Error loss function, here we adopt an adversarial strategy, where a discriminator network is trained to distinguish between the output of the Autoencoder and data sampled from the training corpus. The Autoencoder, then, is trained also by using the binary cross-entropy loss calculated at the output of the discriminator network. The performance of the algorithm has been assessed on a corpus derived from the PASCAL CHiME dataset. The results showed that the proposed approach provides a relative performance improvement equal to 0.26% compared to the standard Autoencoder. The significance of the improvement has been evaluated with a one-tailed z-test and resulted significant with p < 0.001. The presented approach thus showed promising results on this task and it could be extended as a general training strategy for Autoencoders if confirmed by additional experiments.
Chinmay Hegde - One of the best experts on this subject based on the ideXlab platform.
-
AISTATS - On the Dynamics of Gradient Descent for Autoencoders
2019Co-Authors: Thanh V Nguyen, Raymond K W Wong, Chinmay HegdeAbstract:We provide a series of results for unsupervised learning with Autoencoders. Specifically, we study shallow two-layer Autoencoder architectures with shared weights. We focus on three generative models for data that are common in statistical machine learning: (i) the mixture-of-gaussians model, (ii) the sparse coding model, and (iii) the sparsity model with non-negative coefficients. For each of these models, we prove that under suitable choices of hyperparameters, architectures, and initialization, Autoencoders learned by gradient descent can successfully recover the parameters of the corresponding model. To our knowledge, this is the first result that rigorously studies the dynamics of gradient descent for weight-sharing Autoencoders. Our analysis can be viewed as theoretical evidence that shallow Autoencoder modules indeed can be used as feature learning mechanisms for a variety of data models, and may shed insight on how to train larger stacked architectures with Autoencoders as basic building blocks.
-
Autoencoders Learn Generative Linear Models.
arXiv: Machine Learning, 2018Co-Authors: Thanh V Nguyen, Raymond K W Wong, Chinmay HegdeAbstract:Recent progress in learning theory has led to the emergence of provable algorithms for training certain families of neural networks. Under the assumption that the training data is sampled from a suitable generative model, the weights of the trained networks obtained by these algorithms recover (either exactly or approximately) the generative model parameters. However, the large majority of these results are only applicable to supervised learning architectures. In this paper, we complement this line of work by providing a series of results for unsupervised learning with neural networks. Specifically, we study the familiar setting of shallow Autoencoder architectures with shared weights. We focus on three generative models for the data: (i) the mixture-of-gaussians model, (ii) the sparse coding model, and (iii) the non-negative sparsity model. All three models are widely studied in the machine learning literature. For each of these models, we rigorously prove that under suitable choices of hyperparameters, architectures, and initialization, the Autoencoder weights learned by gradient descent % -based training can successfully recover the parameters of the corresponding model. To our knowledge, this is the first result that rigorously studies the dynamics of gradient descent for weight-sharing Autoencoders. Our analysis can be viewed as theoretical evidence that shallow Autoencoder modules indeed can be used as unsupervised feature training mechanisms for a wide range of datasets, and may shed insight on how to train larger stacked architectures with Autoencoders as basic building blocks.
-
Autoencoders Learn Generative Linear Models
arXiv: Machine Learning, 2018Co-Authors: Thanh V Nguyen, Raymond K W Wong, Chinmay HegdeAbstract:We provide a series of results for unsupervised learning with Autoencoders. Specifically, we study shallow two-layer Autoencoder architectures with shared weights. We focus on three generative models for data that are common in statistical machine learning: (i) the mixture-of-gaussians model, (ii) the sparse coding model, and (iii) the sparsity model with non-negative coefficients. For each of these models, we prove that under suitable choices of hyperparameters, architectures, and initialization, Autoencoders learned by gradient descent can successfully recover the parameters of the corresponding model. To our knowledge, this is the first result that rigorously studies the dynamics of gradient descent for weight-sharing Autoencoders. Our analysis can be viewed as theoretical evidence that shallow Autoencoder modules indeed can be used as feature learning mechanisms for a variety of data models, and may shed insight on how to train larger stacked architectures with Autoencoders as basic building blocks.
