Abstract
Supervised learning on molecules has incredible potential to be useful in chemistry, drug discovery, and materials science. Luckily, several promising and closely related neural network models invariant to molecular symmetries have already been described in the literature. These models learn a message passing algorithm and aggregation procedure to compute a function of their entire input graph. In this chapter, we describe a general common framework for learning representations on graph data called message passing neural networks (MPNNs) and show how several prior neural network models for graph data fit into this framework. This chapter contains large overlap with Gilmer et al. (International Conference on Machine Learning, pp. 1263–1272, 2017), and has been modified to highlight more recent extensions to the MPNN framework.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsNotes
- 1.
By comparison, the inference time of the neural networks discussed in this work is 300k times faster.
- 2.
As reported in Schütt et al. [15]. The model was trained on a different train/test split with 100k training samples vs 110k used in our experiments.
References
Y. Wu, M. Schuster, Z. Chen, Q.V. Le, M. Norouzi, W. Macherey, M. Krikun, Y. Cao, Q. Gao, K. Macherey, et al., (2016, preprint). arXiv:1609.08144
G. Hinton, L. Deng, D. Yu, G.E. Dahl, A.-R. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T.N. Sainath, et al., IEEE Signal. Proc. Mag. 29(6), 82 (2012)
A. Krizhevsky, I. Sutskever, G.E. Hinton, Advances in Neural Information Processing Systems (The MIT Press, Cambridge, 2012), pp. 1097–1105
K. Hansen, F. Biegler, R. Ramakrishnan, W. Pronobis, O.A. von Lilienfeld, K.-R. Müller, A. Tkatchenko, J. Phys. Chem. Lett. 6(12), 2326 (2015). https://doi.org/10.1021/acs.jpclett.5b00831
B. Huang, O.A. von Lilienfeld, J. Chem. Phys. 145(16), 161102 (2016). https://doi.org/10.1063/1.4964627
M. Rupp, A. Tkatchenko, K.-R. Müller, O.A. von Lilienfeld, Phys. Rev. Lett. 108(5), 058301 (2012)
D. Rogers, M. Hahn, J. Chem. Inf. Model. 50(5), 742 (2010)
G. Montavon, K. Hansen, S. Fazli, M. Rupp, F. Biegler, A. Ziehe, A. Tkatchenko, O.A. von Lilienfeld, K.-R. Müller, Advances in Neural Information Processing Systems (Curran Associates, Red Hook, 2012), pp. 440–448
J. Behler, M. Parrinello, Phys. Rev. Lett. 98, 146401 (2007). https://doi.org/10.1103/PhysRevLett.98.146401
S.S. Schoenholz, E.D. Cubuk, D.M. Sussman, E. Kaxiras, A.J. Liu, A structural approach to relaxation in glassy liquids. Nat. Phys. 12(5), 469–471 (2016)
M. Allamanis, M. Brockschmidt, M. Khademi, (2017, preprint). arXiv:1711.00740
T.N. Kipf, M. Welling, ArXiv e-prints (2016)
V. Zambaldi, D. Raposo, A. Santoro, V. Bapst, Y. Li, I. Babuschkin, K. Tuyls, D. Reichert, T. Lillicrap, E. Lockhart, et al. (2018, preprint). arXiv:1806.01830
D.K. Duvenaud, D. Maclaurin, J. Iparraguirre, R. Bombarell, T. Hirzel, A. Aspuru-Guzik, R.P. Adams, Advances in Neural Information Processing Systems (2015), pp. 2224–2232
K.T. Schütt, F. Arbabzadah, S. Chmiela, K.R. Müller, A. Tkatchenko, Quantum-chemical insights from deep tensor neural networks. Nat. Commun. 8(1), 1–8 (2017)
F. Faber, L. Hutchison, B. Huang, J. Gilmer, S.S. Schoenholz, G.E. Dahl, O. Vinyals, S. Kearnes, P.F. Riley, O.A. von Lilienfeld, (2017). https://arxiv.org/abs/1702.05532
J. Gilmer, S.S. Schoenholz, P.F. Riley, O. Vinyals, G.E. Dahl, International Conference on Machine Learning (2017), pp. 1263–1272
R. Ramakrishnan, P.O. Dral, M. Rupp, O.A. Von Lilienfeld, Quantum chemistry structures and properties of 134 kilo molecules. Sci. Data 1, 140022 (2014)
P.B. Jørgensen, K.W. Jacobsen, M.N. Schmidt (2018, preprint). arXiv:1806.03146
K.T. Schütt, P.-J. Kindermans, H.E.S. Felix, S. Chmiela, A. Tkatchenko, K.-R. Müller, Advances in Neural Information Processing Systems (Curran Associates, Red Hook, 2017), pp. 991–1001
P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Lio, Y. Bengio, (2017, preprint). arXiv:1710.10903
P.W. Battaglia, J.B. Hamrick, V. Bapst, A. Sanchez-Gonzalez, V. Zambaldi, M. Malinowski, A. Tacchetti, D. Raposo, A. Santoro, R. Faulkner, et al., (2018, preprint). arXiv:1806.01261
X. Wang, R. Girshick, A. Gupta, K. He, (2017, preprint). arXiv:1711.07971
S. Kearnes, K. McCloskey, M. Berndl, V. Pande, P. Riley, J. Comput.-Aided Mol. Des. 30(8), 595 (2016)
Y. Li, D. Tarlow, M. Brockschmidt, R. Zemel, International Conference on Learning Representations, ICLR (2016)
K. Cho, B. Van Merriënboer, D. Bahdanau, Y. Bengio, (2014, preprint). arXiv:1409.1259
P. Battaglia, R. Pascanu, M. Lai, D.J. Rezende, K. Kavukcuoglu, Advances in Neural Information Processing Systems (Curran Associates, Red Hook, 2016), pp. 4502–4510
J. Bruna, W. Zaremba, A. Szlam, Y. LeCun, (2013, preprint). arXiv:1312.6203
M. Defferrard, X. Bresson, P. Vandergheynst, Advances in Neural Information Processing Systems (Curran Associates, Red Hook, 2016), pp. 3837–3845
O. Vinyals, S. Bengio, M. Kudlur, (2015, preprint). arXiv:1511.06391
L. Ruddigkeit, R. Van Deursen, L.C. Blum, J.-L. Reymond, J. Chem. Inf. Model. 52(11), 2864 (2012)
D. Kingma, J. Ba, (2014, preprint). arXiv:1412.6980
Acknowledgements
We would like to thank Lukasz Kaiser, Geoffrey Irving, Alex Graves, and Yujia Li for helpful discussions. Thank you to Adrian Roitberg for pointing out an issue with the use of partial charges in an earlier version of this work.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG
About this chapter
Cite this chapter
Gilmer, J., Schoenholz, S.S., Riley, P.F., Vinyals, O., Dahl, G.E. (2020). Message Passing Neural Networks. In: Schütt, K., Chmiela, S., von Lilienfeld, O., Tkatchenko, A., Tsuda, K., Müller, KR. (eds) Machine Learning Meets Quantum Physics. Lecture Notes in Physics, vol 968. Springer, Cham. https://doi.org/10.1007/978-3-030-40245-7_10
Download citation
DOI: https://doi.org/10.1007/978-3-030-40245-7_10
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-40244-0
Online ISBN: 978-3-030-40245-7
eBook Packages: Physics and AstronomyPhysics and Astronomy (R0)