“A Connection Between Score Matching and Denoising Autoencoders”, 2011-07-01 ():
[cf. “Diffusion models are autoencoders”] Denoising autoencoders have been previously shown to be competitive alternatives to restricted Boltzmann machines for unsupervised pretraining of each layer of a deep architecture.
We show that a simple denoising autoencoder training criterion is equivalent to matching the score (with respect to the data) of a specific energy-based model to that of a nonparametric Parzen density estimator of the data.
This yields several useful insights:
It defines a proper probabilistic model for the denoising autoencoder technique, which makes it in principle possible to sample from them or rank examples by their energy.
It suggests a different way to apply score matching [Hyvärinen 2005, Hyvärinen 2008] that is related to learning to denoise and does not require computing second derivatives.
It justifies the use of tied weights between the encoder and decoder and suggests ways to extend the success of denoising autoencoders to a larger family of energy-based models.