Synonyms
Definition
Deep belief nets are probabilistic generative models that are composed of multiple layers of stochastic latent variables (also called “feature detectors” or “hidden units”). The top two layers have undirected, symmetric connections between them and form an associative memory. The lower layers receive top-down, directed connections from the layer above. Deep belief nets have two important computational properties. First, there is an efficient procedure for learning the top-down, generative weights that specify how the variables in one layer determine the probabilities of variables in the layer below. This procedure learns one layer of latent variables at a time. Second, after learning multiple layers, the values of the latent variables in every layer can be inferred by a single, bottom-up pass that starts with an observed data vector in the bottom layer and uses the generative weights in the reverse direction.
Motivation and Background
The...
This is a preview of subscription content, log in via an institution.
Recommended Reading
Bengio, Y., Lamblin, P., Popovici, P., & Larochelle, H. (2007). Greedy layer-wise training of deep networks, In Advances in neural information processing systems (Vol. 19). Cambridge, MA: MIT Press.
Hinton, G. E. (1989). Connectionist learning procedures. Artificial Intelligence, 40(1–3), 185–234.
Hinton, G. E., Osindero, S., & Teh, Y. W. (2006). A fast learning algorithm for deep belief nets. Neural Computation, 18, 1527–1554.
Hinton, G. E., & Salakhutdinov, R. R. (2006). Reducing the dimensionality of data with neural networks. Science, 313, 504–507.
Larochelle, H., Erhan, D., Courville, A., Bergstra, J., & Bengio, Y. (2007). An empirical evaluation of deep architectures on problems with many factors of variation. In Proceedings of the 24th international conference on machine learning. New York: ACM.
LeCun, Y., & Bengio, Y. (2007). Scaling learning algorithms towards AI. In L. Bottou et al. (Eds.), Large-scale kernel machines. MA: MIT Press.
Movellan, J. R., & Marks, T. K. (2001). Diffusion networks, product of experts, and factor analysis.
Ranzato, M., Huang, F. J., Boureau, Y., & LeCun, Y. (2007) Unsupervised learning of invariant feature hierarchies with applications to object recognition. In Proceedings of computer vision and pattern recognition conference (CVPR 2007). Minneapolis, MN.
Rosenblatt, F. (1962). Principles of neurodynamics. Washington, DC: Spartan Books.
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). Learning representations by back-propagating errors. Nature, 323, 533-536.
Salakhutdinov, R. R., & Hinton, G. E. (2007). Semantic hashing. In Proceedings of the SIGIR workshop on information retrieval and applications of graphical models. Amsterdam, the Netherlands.
Selfridge, O. G. (1958) Pandemonium: A paradigm for learning. In Mechanisation of though processes: Proceedings of a symposium held at the National Physical Laboratory. London: HMSO.
Sutskever, I., & Hinton, G. E. (2007). Learning multilevel distributed representations for high-dimensional sequences. In Proceedings of the eleventh international conference on artificial intelligence and statistics, San Juan, Puerto Rico.
Taylor, G. W., Hinton, G. E., & Roweis, S. (2007). Modeling human motion using binary latent variables. In Advances in neural information processing systems (Vol. 19). Cambridge, MA: MIT Press.
Torralba, A., Fergus, R., & Weiss, Y. (2008). Small codes and large image databases for recognition. In IEEE conference on computer vision and pattern recognition (pp. 1–8). Anchorage, AK.
Welling, M., Rosen-Zvi, M., & Hinton, G. E. (2005). Exponential family harmoniums with an application to information retrieval. In Advances in neural information processing systems (Vol. 17, pp. 1481–1488). Cambridge, MA: MIT Press.
Werbos, P. (1974). Beyond Regression: new tools for prediction and analysis in the behavioral sciences. PhD thesis, Harvard University, Cambridge, MA.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2011 Springer Science+Business Media, LLC
About this entry
Cite this entry
Hinton, G. (2011). Deep Belief Nets. In: Sammut, C., Webb, G.I. (eds) Encyclopedia of Machine Learning. Springer, Boston, MA. https://doi.org/10.1007/978-0-387-30164-8_208
Download citation
DOI: https://doi.org/10.1007/978-0-387-30164-8_208
Publisher Name: Springer, Boston, MA
Print ISBN: 978-0-387-30768-8
Online ISBN: 978-0-387-30164-8
eBook Packages: Computer ScienceReference Module Computer Science and Engineering