Likelihood Estimation in Deep Belief Networks
-
1
Max Planck Institute for Biological Cybernetics, NWG Bethge, Germany
-
2
University of Tübingen, Center for Integrative Neuroscience and Institute for Theoretical Physics, Germany
Many models have been proposed to capture the statistical regularities in natural images patches.
The average log-likelihood on unseen data offers a canonical way to quantify and compare the performance of statistical models. A class of models that has recently gained increasing popularity for the task of modeling complexly structured data is formed by deep belief networks. Analyses of these models, however, have been typically based on samples from the model due to the computationally intractable nature of the model likelihood.
In this study, we investigate whether the apparent ability of a particular deep belief network to capture higher-order statistical regularities in natural images is also reflected in the likelihood. Specifically, we derive a consistent estimator for the likelihood of deep belief networks that is conceptually simpler and more readily applicable than the previously published method [1]. Using this estimator, we evaluate a three-layer deep belief network and compare its density estimation performance with the performance of other models trained on small patches of natural images. In contrast to an earlier analysis based solely on samples, we provide evidence that the deep belief network under study is not a good model for natural images by showing that it is outperformed even by very simple models. Further, we confirm existing results indicating that adding more layers to the network has only little effect on the likelihood if each layer of the model is trained well enough.
Finally, we offer a possible explanation for both the observed performance and the small effect of additional layers by analyzing a best case scenario of the greedy learning algorithm commonly used for training this class of models.
References
[1] Iain Murray and Ruslan Salakhutdinov, Evaluating probabilities under high-dimensional latent variable models, Advances in Neural Information Processing Systems, vol. 21, 2009.
Keywords:
computational neuroscience
Conference:
Bernstein Conference on Computational Neuroscience, Berlin, Germany, 27 Sep - 1 Oct, 2010.
Presentation Type:
Presentation
Topic:
Bernstein Conference on Computational Neuroscience
Citation:
Theis
L,
Sinz
F,
Gerwinn
S and
Bethge
M
(2010). Likelihood Estimation in Deep Belief Networks.
Front. Comput. Neurosci.
Conference Abstract:
Bernstein Conference on Computational Neuroscience.
doi: 10.3389/conf.fncom.2010.51.00116
Copyright:
The abstracts in this collection have not been subject to any Frontiers peer review or checks, and are not endorsed by Frontiers.
They are made available through the Frontiers publishing platform as a service to conference organizers and presenters.
The copyright in the individual abstracts is owned by the author of each abstract or his/her employer unless otherwise stated.
Each abstract, as well as the collection of abstracts, are published under a Creative Commons CC-BY 4.0 (attribution) licence (https://creativecommons.org/licenses/by/4.0/) and may thus be reproduced, translated, adapted and be the subject of derivative works provided the authors and Frontiers are attributed.
For Frontiers’ terms and conditions please see https://www.frontiersin.org/legal/terms-and-conditions.
Received:
08 Sep 2010;
Published Online:
23 Sep 2010.
*
Correspondence:
Mr. Lucas Theis, Max Planck Institute for Biological Cybernetics, NWG Bethge, Tübingen, Germany, lucas@tuebingen.mpg.de