What are autoencoders and stacked autoencoders?

An encoder is a function  which transforms the input vector  in the output  where  is a weight matrix and  is an offset vector. A decoder is an inverse function which tries to reconstruct the original vector from y. An auto-encoder tries to reconstruct the original input by minimizing the error during the reconstruction process. There are two major variants for auto-encoding: Sparse auto-encoders force sparsity by using L1 regularization, while de-noising autoencoders stochastically corrupt the input with some form of randomization.

Mathematically, a stochastic mapping transforms the input vector  into a noisy vector  which is then transformed into an hidden representation . The reconstruction phase is via a decoder  where an error minimization algorithm is used via either squared error loss or cross-entropy loss. Autoencoders typically use a hidden layer which acts as a bottleneck that compress the data as in figure.

In a deep learning context, multiple auto-encoder are stacked for producing the final denoised output. The “magic” outcome of this combination is that autoencoders learn how to extract meaningful features from noise data with no need offhand-craft features’ selection. There are also additional applications. For instance, deep autoencoders are able to map images into compressed vectors with small dimensionality and this can be useful for searching images by image similarity. Plus, Deep autoencoders can map words into small dimension vectors and this is a process useful in topic modelling  distributed across a collection of documents.