An encoder is a function
which transforms the input vector
in the output
is a weight matrix and
is an offset vector. A decoder is an inverse
function which tries to reconstruct the original vector
from y. An auto-encoder tries to
reconstruct the original input by minimizing the error during the
reconstruction process. There are two major variants for auto-encoding: Sparse
auto-encoders force sparsity by using L1 regularization, while de-noising
autoencoders stochastically corrupt the input with some form of randomization.
Mathematically, a stochastic mapping transforms the input vector
into a noisy vector
which is then transformed into an hidden representation
. The reconstruction phase is via a
where an error minimization algorithm is used via
either squared error loss or cross-entropy loss. Autoencoders typically use a
hidden layer which acts as a bottleneck that compress the data as in figure.
In a deep learning context, multiple auto-encoder are stacked for producing the final denoised output. The “magic” outcome of this combination is that autoencoders learn how to extract meaningful features from noise data with no need offhand-craft features’ selection. There are also additional applications. For instance, deep autoencoders are able to map images into compressed vectors with small dimensionality and this can be useful for searching images by image similarity. Plus, Deep autoencoders can map words into small dimension vectors and this is a process useful in topic modelling distributed across a collection of documents.