CASP 15 - MSAs Strike Back
CASP (Critical Assessment of Techniques for Protein Structure Prediction) is a bi-annual challenge on protein structure modeling. In 2020, AlphaFold 2 revolutionized the field of protein structure prediction winning the CASP 14 challenge by a huge margin using Geometric Deep Learning. This weekend the results of CASP 15 were announced - what do we see there after glancing through the abstracts?
In short, multiple sequence alignments (MSAs) do no go anywhere and still are the main component of winning approaches. Most of the top models are based on AlphaFold 2 (and its Multimer version) with many tweaks here and there. Protein LM-based folding like ESM Fold (popular for not needing MSAs) seems to be far away from the top. More reflections by Ezgi Karaca and Sergey Ovchinnikov
CASP (Critical Assessment of Techniques for Protein Structure Prediction) is a bi-annual challenge on protein structure modeling. In 2020, AlphaFold 2 revolutionized the field of protein structure prediction winning the CASP 14 challenge by a huge margin using Geometric Deep Learning. This weekend the results of CASP 15 were announced - what do we see there after glancing through the abstracts?
In short, multiple sequence alignments (MSAs) do no go anywhere and still are the main component of winning approaches. Most of the top models are based on AlphaFold 2 (and its Multimer version) with many tweaks here and there. Protein LM-based folding like ESM Fold (popular for not needing MSAs) seems to be far away from the top. More reflections by Ezgi Karaca and Sergey Ovchinnikov
X (formerly Twitter)
Ezgi Karaca (@Ezgi_Karaca_) on X
It's finally time to discuss the main highlights of #CASP15's assembly category, where we had these beatiful 41 targets:
Friday GraphML News
Not much news this week - seems that the community went for a break after consecutive NeurIPS and LOG. A few things came to our attention:
- IPAM organizes a workshop on Learning and Emergence in Molecular Systems at UCLA in Jan 23-27 with invited talks including Xavier Bresson, Kyunghyun Cho, Bruno Correia, Tommi Jaakkola, Frank Noe, Tess Smidt, and Max Welling
- Recordings of keynotes and orals at LOG have been published on YouTube, recordings of workshops and tutorials are expected soon
Not much news this week - seems that the community went for a break after consecutive NeurIPS and LOG. A few things came to our attention:
- IPAM organizes a workshop on Learning and Emergence in Molecular Systems at UCLA in Jan 23-27 with invited talks including Xavier Bresson, Kyunghyun Cho, Bruno Correia, Tommi Jaakkola, Frank Noe, Tess Smidt, and Max Welling
- Recordings of keynotes and orals at LOG have been published on YouTube, recordings of workshops and tutorials are expected soon
IPAM
Learning and Emergence in Molecular Systems - IPAM
Xmas Papers: Molecule Editing from Text, Protein Generation
It's the holiday season 🎄, and what better way to spend it than reading some new papers on molecule and protein generation! Here are a few cool papers published on arxiv this week:
Multi-modal Molecule Structure-text Model for Text-based Retrieval and Editing by Shengchao Liu and the Mila/NVIDIA team proposes MoleculeSTM, a CLIP-like text-to-molecule model. MoleculeSTM can do 2 impressive things: (1) retrieve molecules by text denoscription like “triazole derivatives” and retrieve text denoscription from a given molecule in SMILES, (2) molecule editing from text prompts like “make the molecule soluble in water with low permeability” - and the model edits the molecular graph according to the denoscription, mindblowing 🤯
Protein Sequence and Structure Co-Design with Equivariant Translation by Chence Shi and the Mila team propose ProtSEED, a generative model for protein sequence and structure simultaneously (for example, most existing diffusion models for proteins can do only one of those at a time). ProtSEED can be conditioned on residue features or pairs of residues. Model-wise, it is an equivariant iterative model (AlphaFold 2 vibes) with improved triangular attention. ProtSEED was evaluated on Antibody CDR co-design, Protein sequence-structure co-design, and Fixed backbone sequence design.
And 2 more papers from the ESM team, Meta AI, and BakerLab (check the Twitter thread by Alex Rives for more details)!
Language models generalize beyond natural proteins by Robert Verkuil et al. find that ESM2 can generate de novo protein sequences that can actually be synthesized in the lab and, more importantly, do not have any match among known natural proteins. Great result knowing that ESM2 was only trained on sequences!
A high-level programming language for generative protein design by Brian Hie et al. propose pretty much a new programming language for protein designers (think of it as a query language for ESMFold) - production rules organized in a syntax tree with constraint functions. Then, each program is “compiled” into an energy function that governs the generative process.
It's the holiday season 🎄, and what better way to spend it than reading some new papers on molecule and protein generation! Here are a few cool papers published on arxiv this week:
Multi-modal Molecule Structure-text Model for Text-based Retrieval and Editing by Shengchao Liu and the Mila/NVIDIA team proposes MoleculeSTM, a CLIP-like text-to-molecule model. MoleculeSTM can do 2 impressive things: (1) retrieve molecules by text denoscription like “triazole derivatives” and retrieve text denoscription from a given molecule in SMILES, (2) molecule editing from text prompts like “make the molecule soluble in water with low permeability” - and the model edits the molecular graph according to the denoscription, mindblowing 🤯
Protein Sequence and Structure Co-Design with Equivariant Translation by Chence Shi and the Mila team propose ProtSEED, a generative model for protein sequence and structure simultaneously (for example, most existing diffusion models for proteins can do only one of those at a time). ProtSEED can be conditioned on residue features or pairs of residues. Model-wise, it is an equivariant iterative model (AlphaFold 2 vibes) with improved triangular attention. ProtSEED was evaluated on Antibody CDR co-design, Protein sequence-structure co-design, and Fixed backbone sequence design.
And 2 more papers from the ESM team, Meta AI, and BakerLab (check the Twitter thread by Alex Rives for more details)!
Language models generalize beyond natural proteins by Robert Verkuil et al. find that ESM2 can generate de novo protein sequences that can actually be synthesized in the lab and, more importantly, do not have any match among known natural proteins. Great result knowing that ESM2 was only trained on sequences!
A high-level programming language for generative protein design by Brian Hie et al. propose pretty much a new programming language for protein designers (think of it as a query language for ESMFold) - production rules organized in a syntax tree with constraint functions. Then, each program is “compiled” into an energy function that governs the generative process.
🎄 It's 2023! In a new post, we provide an overview of what’s happened in Graph ML in 2022 and its subfields (and hypothesize for potential breakthroughs in 2023), including Generative Models, Physics, PDEs, Graph Transformerrs, Theory, KGs, Algorithmic Reasoning, Hardware, and more!
Brought to you by Michael Galkin, Hongyu Ren, Zhaocheng Zhu with the help of Christopher Morris and Johannes Brandstetter
https://mgalkin.medium.com/graph-ml-in-2023-the-state-of-affairs-1ba920cb9232
Brought to you by Michael Galkin, Hongyu Ren, Zhaocheng Zhu with the help of Christopher Morris and Johannes Brandstetter
https://mgalkin.medium.com/graph-ml-in-2023-the-state-of-affairs-1ba920cb9232
Medium
Graph ML in 2023: The State of Affairs
Hot trends and major advancements
New End of the Year Blog Posts
In the first week of a new year, many researchers summarize their thoughts about the past and future. In addition to the previous post reflecting on GraphML in 2022 and 2023, a few new ones appeared:
1. AI in Drug Discovery 2022 by Pat Walters (Relay Therapeutics) on most inspiring papers in molecular and protein ML.
2. The Batch #177 includes predictions for 2023 by Yoshua Bengio (on reasoning), Alon Halevy (on personal data treatment), Douwe Kiela (on practical aspects of LLMs), Been Kim (on interpretability), and Reza Zadeh (on active learning)
3. Using Graph Learning for Personalization: How GNNs Solve Inherent Structural Issues with Recommender Systems by Dylan Sandfelder and Ivaylo Bahtchevanov (kumo.ai) - on applying GNNs in RecSys with examples from Spotify, Pinterest and UberEats.
4. Top Language AI research papers from Yi Tay (Google) - on large language models, the forefront of AI that does have an impact on Graph ML (remember protein language models like ESM-2 and ESM Fold, for instance).
In the first week of a new year, many researchers summarize their thoughts about the past and future. In addition to the previous post reflecting on GraphML in 2022 and 2023, a few new ones appeared:
1. AI in Drug Discovery 2022 by Pat Walters (Relay Therapeutics) on most inspiring papers in molecular and protein ML.
2. The Batch #177 includes predictions for 2023 by Yoshua Bengio (on reasoning), Alon Halevy (on personal data treatment), Douwe Kiela (on practical aspects of LLMs), Been Kim (on interpretability), and Reza Zadeh (on active learning)
3. Using Graph Learning for Personalization: How GNNs Solve Inherent Structural Issues with Recommender Systems by Dylan Sandfelder and Ivaylo Bahtchevanov (kumo.ai) - on applying GNNs in RecSys with examples from Spotify, Pinterest and UberEats.
4. Top Language AI research papers from Yi Tay (Google) - on large language models, the forefront of AI that does have an impact on Graph ML (remember protein language models like ESM-2 and ESM Fold, for instance).
Blogspot
AI in Drug Discovery 2022 - A Highly Opinionated Literature Review
Here’s a roundup of some of the papers I found interesting in 2022. This list is heavily slanted to my interests, which lean toward the appl...
ICLR 2023 Workshops
The list of workshops at upcoming ICLR’23 has been announced! A broad Graph ML audience might be interested in:
- From Molecules to Materials: ICLR 2023 Workshop on Machine learning for materials (ML4Materials)
- Machine Learning for Drug Discovery (MLDD)
- Neurosymbolic Generative Models (NeSy-GeMs)
- Physics for Machine Learning
- Deep Learning for Code (DL4C)
The list of workshops at upcoming ICLR’23 has been announced! A broad Graph ML audience might be interested in:
- From Molecules to Materials: ICLR 2023 Workshop on Machine learning for materials (ML4Materials)
- Machine Learning for Drug Discovery (MLDD)
- Neurosymbolic Generative Models (NeSy-GeMs)
- Physics for Machine Learning
- Deep Learning for Code (DL4C)
Validated de novo generated antibodies & AI 4 Science talks
- Absci announced their de novo zero-shot generated therapeutic antibodies were validated in the wet lab. The pre-print is scarce on technical details, but what we can infer is that they combine many new geometric generative models with fast screening pipelines.
- A new series of talks on AI4Science is starting next week! The inaugural talk will be delivered by Simon Batzner (Harvard) on “E(3)-equivariant graph neural networks for data-efficient and accurate interatomic potentials”
- Absci announced their de novo zero-shot generated therapeutic antibodies were validated in the wet lab. The pre-print is scarce on technical details, but what we can infer is that they combine many new geometric generative models with fast screening pipelines.
- A new series of talks on AI4Science is starting next week! The inaugural talk will be delivered by Simon Batzner (Harvard) on “E(3)-equivariant graph neural networks for data-efficient and accurate interatomic potentials”
ai4sciencetalks.github.io
Talks | AI4Science Talks
A growing collection of invited talks on Machine Learning for Science and Simulations.
Friday News
- No big protein / molecule diffusion model announcement this week 🤨
- ELLIS starts a new program “Machine Learning for Molecule Discovery” that aims to improve computational molecular science in a multitude of its applications (eg, our favourite generative diffusion models). ELLIS is a pan-European AI network of excellence which focuses on fundamental science, technical innovation and societal impact. According to José Miguel Hernández Lobato (Cambridge): ELLIS Programs focus on high-impact problem areas that have the potential to move the needle in modern AI. Each Program has a budget for 2-3 workshops each year to enable meetings of ELLIS Fellows plus guests for intensive scientific exchange”.
- Helmholtz Munich Campus organizes the 1st International Symposium on AI for Health to be held on Jan 16. Keynotes will be delivered by Mackenzie Mathis (EPFL), Michaela van der Schaar (Cambridge), and Marinka Zitnik (Harvard).
- The Growth-focused ML event with leaders from Whatnot and Kumo.AI on Feb 1st, featuring Jure Leskovec discussing SOTA GNNs and where the ML community is heading.
- No big protein / molecule diffusion model announcement this week 🤨
- ELLIS starts a new program “Machine Learning for Molecule Discovery” that aims to improve computational molecular science in a multitude of its applications (eg, our favourite generative diffusion models). ELLIS is a pan-European AI network of excellence which focuses on fundamental science, technical innovation and societal impact. According to José Miguel Hernández Lobato (Cambridge): ELLIS Programs focus on high-impact problem areas that have the potential to move the needle in modern AI. Each Program has a budget for 2-3 workshops each year to enable meetings of ELLIS Fellows plus guests for intensive scientific exchange”.
- Helmholtz Munich Campus organizes the 1st International Symposium on AI for Health to be held on Jan 16. Keynotes will be delivered by Mackenzie Mathis (EPFL), Michaela van der Schaar (Cambridge), and Marinka Zitnik (Harvard).
- The Growth-focused ML event with leaders from Whatnot and Kumo.AI on Feb 1st, featuring Jure Leskovec discussing SOTA GNNs and where the ML community is heading.
Department of Engineering
New program aiming to accelerate molecular discovery
However, the discovery of new molecules or molecular materials that are optimised for a particular purpose can often take up to a decade and is highly cost-intensive. Machine-learning (ML) methods
Temporal Graph Learning in 2023
A new blogpost by Andy Huang, Emanuele Rossi, Michael Galkin, and Kellin Pelrine on the recent progress in temporal Graph ML! Featuring theoretical advancements in understanding expressive power of temporal GNNs, discussing evaluation protocols and trustworthiness concerns, looking at temporal KGs, disease modeling, and anomaly detection, as well as pointing to the software libraries and new datasets!
A new blogpost by Andy Huang, Emanuele Rossi, Michael Galkin, and Kellin Pelrine on the recent progress in temporal Graph ML! Featuring theoretical advancements in understanding expressive power of temporal GNNs, discussing evaluation protocols and trustworthiness concerns, looking at temporal KGs, disease modeling, and anomaly detection, as well as pointing to the software libraries and new datasets!
Medium
Temporal Graph Learning in 2023
The story so far
Friday Graph ML News: Ankh Protein LM, Deadlines, and New Blogs
If you didn’t make it to polish the submission for ICML or IJCAI, consider other upcoming submission deadlines:
- Deep Leaning for Graphs @ International Joint Conference on Neural Networks: Jan 31st
- Special Issue on Graph Learning @ IEEE Transactions on Neural Networks and Learning Systems: March 1st
- Graph Representation Learning track @ European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning: May 2nd
The Summer Geometry Initiative (MIT) is a six-week paid summer research program introducing undergraduate and graduate students to the field of geometry processing, no prior experience is required: apply until February 15th.
New articles and blogs about graphs and more general deep learning:
- Quanta Magazine published a fascinating article on the discovery of the shortest path algorithm on graphs with negative edge weights;
- Kexin Huang prepared a post explaining a variety of datasets available in the Therapeutic Data Commons from drug-target interactions to retrosynthesis and predicting CRISPR editing outcomes
- Tim Dettmers updated his annual report on most efficient GPUs per $ with new data from H100. In relative performance, if you don’t have H100’s — get RTX 4090; for perf per $ 4070 Ti is surprisingly in the top.
- Google published a Deep Learning Tuning Playbook - a collection of tuning advice that will help you to squeeze that 1% of performance and get top-1 in OGB!
- Finally, a huge post from Lilian Weng on optimizing inference of large Transformers
☥ This week we do see a new big model: meet Ankh, a protein LM! Thanks to the recent observation of importance of data size and training vs model size, 1.5B Ankh often outperforms 15B ESM-2B on contact prediction, structure prediction, and a good bunch of protein representation learning tasks. Arxiv pre-print is available as well!If you didn’t make it to polish the submission for ICML or IJCAI, consider other upcoming submission deadlines:
- Deep Leaning for Graphs @ International Joint Conference on Neural Networks: Jan 31st
- Special Issue on Graph Learning @ IEEE Transactions on Neural Networks and Learning Systems: March 1st
- Graph Representation Learning track @ European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning: May 2nd
The Summer Geometry Initiative (MIT) is a six-week paid summer research program introducing undergraduate and graduate students to the field of geometry processing, no prior experience is required: apply until February 15th.
New articles and blogs about graphs and more general deep learning:
- Quanta Magazine published a fascinating article on the discovery of the shortest path algorithm on graphs with negative edge weights;
- Kexin Huang prepared a post explaining a variety of datasets available in the Therapeutic Data Commons from drug-target interactions to retrosynthesis and predicting CRISPR editing outcomes
- Tim Dettmers updated his annual report on most efficient GPUs per $ with new data from H100. In relative performance, if you don’t have H100’s — get RTX 4090; for perf per $ 4070 Ti is surprisingly in the top.
- Google published a Deep Learning Tuning Playbook - a collection of tuning advice that will help you to squeeze that 1% of performance and get top-1 in OGB!
- Finally, a huge post from Lilian Weng on optimizing inference of large Transformers
Proteinea
Ankh: Pioneering the paradigm shifting technology of protein language models
To model the language of life (i.e., the language of proteins) in a data-efficient and cost-optimized modality, we need a meaningful representation encompassing its structural and functional information. Proteinea has developed -and is continuously improving…
Friday Graph ML News: ProGen, ClimaX, WebConf Workshops, Everything is Connected
No week without new foundation models!
A collaboration of researchers from Salesforce, UCSF and Berkeley announced ProGen, an LLM for protein sequence generation. Claimed to be “ChatGPT for proteins”, ProGen is a 1.2B model trained on 280M sequences controllable by input tags, eg “Protein Family: Pfam ID PF16754, Pesticin”. The authors synthesized in a lab a handful of generated proteins to confirm model quality.
In the GraphML’23 State of Affairs we highlighted weather prediction models GraphCast (from DeepMind) and PanguWeather (from Huawei). This week, Microsoft Research and UCLA announced ClimaX, the foundation model for climate and weather that can serve as a backbone for many many downstream applications. In contrast to now-casting GraphCast and PanguWeather, ClimaX is tailored for more long-range predictions up to a month. ClimaX is a ViT-based image-to-image model with several tokenization and representation novelties to account for different input granularity and sequence length - check out the full paper preprint for more details.
Petar Veličković published the opinion paper Everything is Connected: Graph Neural Network framing many ML applications through the lens of graph representation learning. The article gives a gentle introduction to the basics of GNNs and their applications including geometric equivariant models. Nice read!
The WebConf’23 (April 30 - May 4) announced accepted workshops with a handful of Graph ML venues:
- Graph Neural Networks: Foundation, Frontiers and Applications
- Mining of Real-world Hypergraphs: Patterns, Tools, and Generators
- Graph Neural Networks for Tabular Data Learning
- Continual Graph Learning
- Towards Out-of-Distribution Generalization on Graphs
- Self-supervised Learning and Pre-training on Graphs
- When Sparse Meets Dense: Learning Advanced Graph Neural Networks with DGL-Sparse Package
No week without new foundation models!
A collaboration of researchers from Salesforce, UCSF and Berkeley announced ProGen, an LLM for protein sequence generation. Claimed to be “ChatGPT for proteins”, ProGen is a 1.2B model trained on 280M sequences controllable by input tags, eg “Protein Family: Pfam ID PF16754, Pesticin”. The authors synthesized in a lab a handful of generated proteins to confirm model quality.
In the GraphML’23 State of Affairs we highlighted weather prediction models GraphCast (from DeepMind) and PanguWeather (from Huawei). This week, Microsoft Research and UCLA announced ClimaX, the foundation model for climate and weather that can serve as a backbone for many many downstream applications. In contrast to now-casting GraphCast and PanguWeather, ClimaX is tailored for more long-range predictions up to a month. ClimaX is a ViT-based image-to-image model with several tokenization and representation novelties to account for different input granularity and sequence length - check out the full paper preprint for more details.
Petar Veličković published the opinion paper Everything is Connected: Graph Neural Network framing many ML applications through the lens of graph representation learning. The article gives a gentle introduction to the basics of GNNs and their applications including geometric equivariant models. Nice read!
The WebConf’23 (April 30 - May 4) announced accepted workshops with a handful of Graph ML venues:
- Graph Neural Networks: Foundation, Frontiers and Applications
- Mining of Real-world Hypergraphs: Patterns, Tools, and Generators
- Graph Neural Networks for Tabular Data Learning
- Continual Graph Learning
- Towards Out-of-Distribution Generalization on Graphs
- Self-supervised Learning and Pre-training on Graphs
- When Sparse Meets Dense: Learning Advanced Graph Neural Networks with DGL-Sparse Package
Nature
Large language models generate functional protein sequences across diverse families
Nature Biotechnology - A generative deep-learning model designs artificial proteins with desired enzymatic activities.
On the Expressive Power of Geometric Graph Neural Networks
Geometric GNNs are an emerging class of GNNs for spatially embedded graphs across science and engineering, e.g. SchNet for molecules, Tensor Field Networks for materials, GemNet for electrocatalysts, MACE for molecular dynamics, and E(n)-Equivariant Graph ConvNet for macromolecules.
How powerful are geometric GNNs? How do key design choices influence expressivity and how to build maximally powerful ones?
Check out this recent paper from Chaitanya K. Joshi, Cristian Bodnar, Simon V. Mathis, Taco Cohen, and Pietro Liò for more:
📄 PDF: http://arxiv.org/abs/2301.09308
💻 Code: http://github.com/chaitjo/geometric-gnn-dojo
❓Research gap: Standard theoretical tools for GNNs, such as the Weisfeiler-Leman graph isomorphism test, are inapplicable for geometric graphs. This is due to additional physical symmetries (roto-translation) that need to be accounted for.
💡Key idea: notion of geometric graph isomorphism + new geometric WL framework --> upper bound on geometric GNN expressivity.
The Geometric WL framework formalises the role of depth, invariance vs. equivariance, body ordering in geometric GNNs.
- Invariant GNNs cannot tell apart one-hop identical geometric graphs, fail to compute global properties.
- Equivariant GNNs distinguish more graphs; how? Depth propagates local geometry beyond one-hop.
What about practical implications? Synthetic experiments highlight challenges in building maximally powerful geom. GNNs:
- Oversquashing of geometric information with increased depth.
- Utility of higher order order spherical tensors over cartesian vectors.
P.S. Are you new to Geometric GNNs, GDL, PyTorch Geometric, etc.? Want to understand how theory/equations connect to real code?
Try this Geometric GNN 101 notebook before diving in:
https://github.com/chaitjo/geometric-gnn-dojo/blob/main/geometric_gnn_101.ipynb
Geometric GNNs are an emerging class of GNNs for spatially embedded graphs across science and engineering, e.g. SchNet for molecules, Tensor Field Networks for materials, GemNet for electrocatalysts, MACE for molecular dynamics, and E(n)-Equivariant Graph ConvNet for macromolecules.
How powerful are geometric GNNs? How do key design choices influence expressivity and how to build maximally powerful ones?
Check out this recent paper from Chaitanya K. Joshi, Cristian Bodnar, Simon V. Mathis, Taco Cohen, and Pietro Liò for more:
📄 PDF: http://arxiv.org/abs/2301.09308
💻 Code: http://github.com/chaitjo/geometric-gnn-dojo
❓Research gap: Standard theoretical tools for GNNs, such as the Weisfeiler-Leman graph isomorphism test, are inapplicable for geometric graphs. This is due to additional physical symmetries (roto-translation) that need to be accounted for.
💡Key idea: notion of geometric graph isomorphism + new geometric WL framework --> upper bound on geometric GNN expressivity.
The Geometric WL framework formalises the role of depth, invariance vs. equivariance, body ordering in geometric GNNs.
- Invariant GNNs cannot tell apart one-hop identical geometric graphs, fail to compute global properties.
- Equivariant GNNs distinguish more graphs; how? Depth propagates local geometry beyond one-hop.
What about practical implications? Synthetic experiments highlight challenges in building maximally powerful geom. GNNs:
- Oversquashing of geometric information with increased depth.
- Utility of higher order order spherical tensors over cartesian vectors.
P.S. Are you new to Geometric GNNs, GDL, PyTorch Geometric, etc.? Want to understand how theory/equations connect to real code?
Try this Geometric GNN 101 notebook before diving in:
https://github.com/chaitjo/geometric-gnn-dojo/blob/main/geometric_gnn_101.ipynb
GitHub
GitHub - chaitjo/geometric-gnn-dojo: Geometric GNN Dojo provides unified implementations and experiments to explore the design…
Geometric GNN Dojo provides unified implementations and experiments to explore the design space of Geometric Graph Neural Networks (ICML 2023) - chaitjo/geometric-gnn-dojo
Friday Graph ML News: Blogs, ICLR Acceptances, and Software Releases
No big protein / molecule diffusion model announcement this week 🤨
Still, a handful of nice blogposts!
Graph Machine Learning Explainability with PyG by Blaž Stojanovič and PyG Team is a massive tutorial on GNN explainability tools in PyG with datasets, code examples, and metrics. Must read for all explainability studies.
Unleashing ML Innovation at Spotify with Ray talks about Ray and an application of running A/B tests of the GNN-based recommender system for the main page of Spotify.
🤓 ICLR accepted papers are now available with distinction of top-5%, top-25%, and posters. Stay tuned for the review of ICLR graph papers! Meanwhile, have a look at some hot new papers put on arxiv:
- WL Meets VC by Christopher Morris, Floris Geerts, Jan Tönshoff, Martin Grohe - the first work to connect WL test with VC dimension of GNNs with provable bounds!
- Curvature Filtrations for Graph Generative Model Evaluation by Joshua Southern, Jeremy Wayland, Michael Bronstein, Bastian Rieck - a novel look on graph generation evaluation using the concepts of topological data analysis
🛠️ New software releases this week!
- DGL v1.0 - finally, the major 1.0 release featuring a new sparse backend
- PyKEEN v1.10 - still the best library to work with KG embeddings
No big protein / molecule diffusion model announcement this week 🤨
Still, a handful of nice blogposts!
Graph Machine Learning Explainability with PyG by Blaž Stojanovič and PyG Team is a massive tutorial on GNN explainability tools in PyG with datasets, code examples, and metrics. Must read for all explainability studies.
Unleashing ML Innovation at Spotify with Ray talks about Ray and an application of running A/B tests of the GNN-based recommender system for the main page of Spotify.
🤓 ICLR accepted papers are now available with distinction of top-5%, top-25%, and posters. Stay tuned for the review of ICLR graph papers! Meanwhile, have a look at some hot new papers put on arxiv:
- WL Meets VC by Christopher Morris, Floris Geerts, Jan Tönshoff, Martin Grohe - the first work to connect WL test with VC dimension of GNNs with provable bounds!
- Curvature Filtrations for Graph Generative Model Evaluation by Joshua Southern, Jeremy Wayland, Michael Bronstein, Bastian Rieck - a novel look on graph generation evaluation using the concepts of topological data analysis
🛠️ New software releases this week!
- DGL v1.0 - finally, the major 1.0 release featuring a new sparse backend
- PyKEEN v1.10 - still the best library to work with KG embeddings
Medium
Graph Machine Learning Explainability with PyG
By Blaž Stojanovič
HuggingFace enters GraphML
After unifying Vision and Language models and datasets under one hood, 🤗 comes for GraphML! Today, Datasets started hosting graph datasets including OGB ones, ZINC, CSL, and others from Benchmarking GNNs, as well as MD17 for molecular dynamics. Let’s see what HF does next with GNNs.
After unifying Vision and Language models and datasets under one hood, 🤗 comes for GraphML! Today, Datasets started hosting graph datasets including OGB ones, ZINC, CSL, and others from Benchmarking GNNs, as well as MD17 for molecular dynamics. Let’s see what HF does next with GNNs.
huggingface.co
Graph Machine Learning Datasets – Hugging Face
Explore datasets powering machine learning.
Attending To Graph Transformers
by Luis Müller, Michael Galkin, Christopher Morris, and Ladislav Rampasek
arxiv
Our new survey on Graph Transformers (GTs) adjoined by some “mythbusting”.
We come up with categorization of GTs according to 4 main views:
🗺️ used Encodings,
🌐 expected Input Features (geometric or non-geometric),
✨ Tokenization (nodes, nodes+edges, subgraphs), and
🧮 Propagation (fully-connected, sparse, hybrid).
We investigate 4 common expectations and claims about GTs. Although conclusions are more nuanced (see the paper), we label them with pretentious badges ✅ Confirmed / ❌ Busted / 🤔 Plausible
1️⃣ Are GTs theoretically more expressive than GNNs?
❌ Busted. There is no inherent property of GTs that makes them more expressive. Instead, their expressivity stems from their positional/structural encodings. (And making those maximally expressive is as hard as solving the graph isomorphism problem.)
2️⃣ Can graph structure be effectively incorporated into GTs?
✅ Confirmed. GTs can identify graph edges (easy task), count triangles (medium), and distinguish regular graphs (hard task). But there is still room for improvement.
3️⃣ Does global attention reduce over-smoothing?
🤔 Plausible. In heterophilic graphs, GTs clearly outperform vanilla GNNs but still lag behind specialized SOTA models. Maybe we need a different structural bias?
4️⃣ Do GTs alleviate over-squashing better than GNN models?
🤔 Plausible. The Transformer perfectly solves NeighborsMatch where GNNs struggle. However, this is a synthetic “retrieval” task that doesn’t test (sub)graph representation.
🎁 Bonus: Attention matrices contain meaningful patterns and explain GT performance.
❌ Busted. We couldn’t find any strong interpretability of attention scores for downstream tasks. We suggest following Bertology in NLP that moved from dissecting attention to designing benchmarks.
by Luis Müller, Michael Galkin, Christopher Morris, and Ladislav Rampasek
arxiv
Our new survey on Graph Transformers (GTs) adjoined by some “mythbusting”.
We come up with categorization of GTs according to 4 main views:
🗺️ used Encodings,
🌐 expected Input Features (geometric or non-geometric),
✨ Tokenization (nodes, nodes+edges, subgraphs), and
🧮 Propagation (fully-connected, sparse, hybrid).
We investigate 4 common expectations and claims about GTs. Although conclusions are more nuanced (see the paper), we label them with pretentious badges ✅ Confirmed / ❌ Busted / 🤔 Plausible
1️⃣ Are GTs theoretically more expressive than GNNs?
❌ Busted. There is no inherent property of GTs that makes them more expressive. Instead, their expressivity stems from their positional/structural encodings. (And making those maximally expressive is as hard as solving the graph isomorphism problem.)
2️⃣ Can graph structure be effectively incorporated into GTs?
✅ Confirmed. GTs can identify graph edges (easy task), count triangles (medium), and distinguish regular graphs (hard task). But there is still room for improvement.
3️⃣ Does global attention reduce over-smoothing?
🤔 Plausible. In heterophilic graphs, GTs clearly outperform vanilla GNNs but still lag behind specialized SOTA models. Maybe we need a different structural bias?
4️⃣ Do GTs alleviate over-squashing better than GNN models?
🤔 Plausible. The Transformer perfectly solves NeighborsMatch where GNNs struggle. However, this is a synthetic “retrieval” task that doesn’t test (sub)graph representation.
🎁 Bonus: Attention matrices contain meaningful patterns and explain GT performance.
❌ Busted. We couldn’t find any strong interpretability of attention scores for downstream tasks. We suggest following Bertology in NLP that moved from dissecting attention to designing benchmarks.
Temporal Graph Learning RG, Clifford Networks, Forward-Forward GNNs
Temporal Graph Learning reading group - a new reading group by McGill and NEC Labs researchers happening every Thursday 11-12 Eastern Time!
Jure Leskovec (Stanford) gave a talk “Towards Universal Cell Embeddings” at the Broad Institute (slides are available) covering the most recent research on single cell analysis with GNNs including MARS for novel cell types, SATURN for joint cell-protein representations, STELLAR for cancer tissue annotation, and GEARS for predicting effects of multi-gene perturbations
New papers you might want to have a look at:
Geometric Clifford Algebra Networks
David Ruhe, Jayesh K. Gupta, Steven de Keninck, Max Welling, and Johannes Brandstetter
MSR recently dropped a hefty 50-pager on Clifford Algebras for PDEs, here is the adaptation of Clifford layers for GNNs with applications in object dynamics and fluid mechanics! Check the Twitter thread by David Ruhe for cool visual examples.
Graph Neural Networks Go Forward-Forward
Daniele Paliotta, Mathieu Alain, Bálint Máté, François Fleuret
At the recent NeurIPS’22, Geoff Hinton presented an idea of forward-forward networks without backprop. Instead of building a computation graph for the backward pass, you’d encode the label together with an input data point and ask the trainable layer to distinguish positive label from a negative sample. Here, the authors expand the idea to GNNs and probe forward-forward on graph classification tasks. Interestingly, the results are not that bad - in some cases, FF-GNNs even outperform their backprop counterparts.
DiffDock, a diffusion model for protein-ligand docking, has been updated to include new results on blind docking with ESMFold - the model now drastically outperforms industrial tools on RMSD docking accuracy within 2 Angstrom, check the full thread by Gabriele Corso
Temporal Graph Learning reading group - a new reading group by McGill and NEC Labs researchers happening every Thursday 11-12 Eastern Time!
Jure Leskovec (Stanford) gave a talk “Towards Universal Cell Embeddings” at the Broad Institute (slides are available) covering the most recent research on single cell analysis with GNNs including MARS for novel cell types, SATURN for joint cell-protein representations, STELLAR for cancer tissue annotation, and GEARS for predicting effects of multi-gene perturbations
New papers you might want to have a look at:
Geometric Clifford Algebra Networks
David Ruhe, Jayesh K. Gupta, Steven de Keninck, Max Welling, and Johannes Brandstetter
MSR recently dropped a hefty 50-pager on Clifford Algebras for PDEs, here is the adaptation of Clifford layers for GNNs with applications in object dynamics and fluid mechanics! Check the Twitter thread by David Ruhe for cool visual examples.
Graph Neural Networks Go Forward-Forward
Daniele Paliotta, Mathieu Alain, Bálint Máté, François Fleuret
At the recent NeurIPS’22, Geoff Hinton presented an idea of forward-forward networks without backprop. Instead of building a computation graph for the backward pass, you’d encode the label together with an input data point and ask the trainable layer to distinguish positive label from a negative sample. Here, the authors expand the idea to GNNs and probe forward-forward on graph classification tasks. Interestingly, the results are not that bad - in some cases, FF-GNNs even outperform their backprop counterparts.
DiffDock, a diffusion model for protein-ligand docking, has been updated to include new results on blind docking with ESMFold - the model now drastically outperforms industrial tools on RMSD docking accuracy within 2 Angstrom, check the full thread by Gabriele Corso
X (formerly Twitter)
Jure Leskovec (@jure) on X
Towards Universal Cell Embeddings. My talk at the @broadinstitute about our recent advancements in ML for biomedicine: https://t.co/f6rKlim1Aw Thank you @CarolineUhler for hosting!
Saturday News
Yanze Wang and Yuanqi Du published An Introduction to Molecular Dynamics Simulations - a great resource to get acquainted with the basics of molecular dynamics. This is a part of the bigger AI 4 Science 101 community effort to bring more ML folks into the world of complex fundamental scientific problems - great initiative 👏
No week without new protein models! Uni-Fold Musse by DP Technology is a no-MSA protein structure prediction system. Based on ESM-2 3B, Uni-Fold further pre-trains the base with more multimere tasks and outperforms other no-MSA proteins LMs (still lagging behind MSA-based Alphafold Multimere though).
Meanwhile, DGL officially released DGL 1.0 featuring a new DGL Sparse package with optimized kernels for sparse matrix multiplications.
Some upcoming events for different parts of the globe:
- SIAM symposiums: Geometric Structures in Neuroscience, Methods in Geometry and Shape Analysis featuring Erik Bekkers, Sophia Sanborn, and many other researchers - Feb 27th, Amsterdam
- The Machine Learning on Graphs workshop at WSDM’23 - March 3rd, Singapore
New papers you might be interested in:
On the Expressivity of Persistent Homology in Graph Learning by Bastian Rieck - the paper establishes the bridge between topology and expressiveness with formal proofs how topological features correspond to higher-order WL tests. Besides, the paper is very friendly to newcomers so give it a read if you want to learn more about persistent homology. Check the Twitter thread by Bastian for more info.
A handful of new of diffusion papers:
- Geometry-Complete Diffusion for 3D Molecule Generation - a strong competitor to the Equivariant Diffusion Model (EDM, ICML’22) used pretty much everywhere in molecule/protein models
- SE(3) diffusion model with application to protein backbone generation - a diffusion model over rigid bodies, FrameDiff
- MiDi: Mixed Graph and 3D Denoising Diffusion for Molecule Generation - the next version of the discrete diffusion model DiGress (to be presented at ICLR’23) now enriched with 3D info
- Aligned Diffusion Schrödinger Bridges - a new diffusion model approach based on Schrödinger bridges for rigid protein docking
Yanze Wang and Yuanqi Du published An Introduction to Molecular Dynamics Simulations - a great resource to get acquainted with the basics of molecular dynamics. This is a part of the bigger AI 4 Science 101 community effort to bring more ML folks into the world of complex fundamental scientific problems - great initiative 👏
No week without new protein models! Uni-Fold Musse by DP Technology is a no-MSA protein structure prediction system. Based on ESM-2 3B, Uni-Fold further pre-trains the base with more multimere tasks and outperforms other no-MSA proteins LMs (still lagging behind MSA-based Alphafold Multimere though).
Meanwhile, DGL officially released DGL 1.0 featuring a new DGL Sparse package with optimized kernels for sparse matrix multiplications.
Some upcoming events for different parts of the globe:
- SIAM symposiums: Geometric Structures in Neuroscience, Methods in Geometry and Shape Analysis featuring Erik Bekkers, Sophia Sanborn, and many other researchers - Feb 27th, Amsterdam
- The Machine Learning on Graphs workshop at WSDM’23 - March 3rd, Singapore
New papers you might be interested in:
On the Expressivity of Persistent Homology in Graph Learning by Bastian Rieck - the paper establishes the bridge between topology and expressiveness with formal proofs how topological features correspond to higher-order WL tests. Besides, the paper is very friendly to newcomers so give it a read if you want to learn more about persistent homology. Check the Twitter thread by Bastian for more info.
A handful of new of diffusion papers:
- Geometry-Complete Diffusion for 3D Molecule Generation - a strong competitor to the Equivariant Diffusion Model (EDM, ICML’22) used pretty much everywhere in molecule/protein models
- SE(3) diffusion model with application to protein backbone generation - a diffusion model over rigid bodies, FrameDiff
- MiDi: Mixed Graph and 3D Denoising Diffusion for Molecule Generation - the next version of the discrete diffusion model DiGress (to be presented at ICLR’23) now enriched with 3D info
- Aligned Diffusion Schrödinger Bridges - a new diffusion model approach based on Schrödinger bridges for rigid protein docking
GraphML News
A new blog post Graph Neural Networks for Molecular Dynamics Simulations by Sina Stocker and Johannes Gasteiger covering the basics of molecular dynamics with GemNet and code examples.
Teaching old labels new tricks in heterogeneous graphs - a new post by Google Research introducing Knowledge Transfer Networks (NeurIPS’22) - a method for zero-shot transfer on heterogeneous graphs with extreme label scarcity.
TigerGraph incorporated NodePiece, a compositional tokenization approach for scalable and inductive graph learning, into a new release - as the author of NodePiece, I am very excited to see academic efforts adopted in industrial DBs! Btw, NodePiece-based approaches have been taking the whole top-10 in OGB WikiKG 2 link prediction benchmark for almost two years now.
All talk recordings of the IPAM UCLA workshop on Deep Learning and Combinatorial Optimization are now available! Featuring researchers such as Stefanie Jegelka, Petar Veličković, Xavier Bresson, Kyle Cranmer, and many more.
Weekend reading:
Complexity of Many-Body Interactions in Transition Metals via Machine-Learned Force Fields from the TM23 Data Set - a new TM23 dataset for molecular dynamics modeling transition metals.
Reducing SO(3) Convolutions to SO(2) for Efficient Equivariant GNNs - a neat math trick to reduce computational complexity of equivariant GNNs
A new blog post Graph Neural Networks for Molecular Dynamics Simulations by Sina Stocker and Johannes Gasteiger covering the basics of molecular dynamics with GemNet and code examples.
Teaching old labels new tricks in heterogeneous graphs - a new post by Google Research introducing Knowledge Transfer Networks (NeurIPS’22) - a method for zero-shot transfer on heterogeneous graphs with extreme label scarcity.
TigerGraph incorporated NodePiece, a compositional tokenization approach for scalable and inductive graph learning, into a new release - as the author of NodePiece, I am very excited to see academic efforts adopted in industrial DBs! Btw, NodePiece-based approaches have been taking the whole top-10 in OGB WikiKG 2 link prediction benchmark for almost two years now.
All talk recordings of the IPAM UCLA workshop on Deep Learning and Combinatorial Optimization are now available! Featuring researchers such as Stefanie Jegelka, Petar Veličković, Xavier Bresson, Kyle Cranmer, and many more.
Weekend reading:
Complexity of Many-Body Interactions in Transition Metals via Machine-Learned Force Fields from the TM23 Data Set - a new TM23 dataset for molecular dynamics modeling transition metals.
Reducing SO(3) Convolutions to SO(2) for Efficient Equivariant GNNs - a neat math trick to reduce computational complexity of equivariant GNNs
research.google
Teaching old labels new tricks in heterogeneous graphs
Posted by Minji Yoon, Research Intern, and Bryan Perozzi, Research Scientist, Google Research, Graph Mining Team Industrial applications of machine...
A Critical Look at the Evaluation of GNNs under Heterophily: Are we really making progress?
ICLR 2023, guest post by Oleg Platonov
Stop evaluating on squirrel and chameleon
It is often believed that standard GNNs work well for node classification only on homophilous graphs. Thus, many specialized models have been recently proposed for learning on heterophilous graphs. However, these models are typically evaluated on the same set of six heterophilous graphs called Squirrel, Chameleon, Actor, Texas, Cornell, and Wisconsin. In our recent paper, we show that these datasets have serious problems, which make results obtained using them unreliable. These problems include low diversity, small graph size, and strong class imbalance. But the most significant is the presence of a large number of duplicated nodes in Squirrel and Chameleon, which leads to train-test data leakage. We show that removing the duplicates strongly affects the performance of GNNs on these datasets.
We have proposed an alternative benchmark of five diverse heterophilous graphs that come from different domains and exhibit a variety of structural properties. Our benchmark includes a word dependency graph Roman-empire, a product co-purchasing network Amazon-ratings, a synthetic graph emulating the minesweeper game Minesweeper, a crowdsourcing platform worker network Tolokers, and a question-answering website interaction network Questions.
We have evaluated a large number of models, both standard GNNs and heterophily-specific methods, and, surprisingly, found that standard GNNs augmented with skip connections and layer normalization almost always outperform specialized models. We hope that the proposed benchmark and the insights obtained using it will facilitate further research in learning under heterophily.
The datasets are available on GitHub and in PyG Datasets. For more details, see our paper.
ICLR 2023, guest post by Oleg Platonov
Stop evaluating on squirrel and chameleon
It is often believed that standard GNNs work well for node classification only on homophilous graphs. Thus, many specialized models have been recently proposed for learning on heterophilous graphs. However, these models are typically evaluated on the same set of six heterophilous graphs called Squirrel, Chameleon, Actor, Texas, Cornell, and Wisconsin. In our recent paper, we show that these datasets have serious problems, which make results obtained using them unreliable. These problems include low diversity, small graph size, and strong class imbalance. But the most significant is the presence of a large number of duplicated nodes in Squirrel and Chameleon, which leads to train-test data leakage. We show that removing the duplicates strongly affects the performance of GNNs on these datasets.
We have proposed an alternative benchmark of five diverse heterophilous graphs that come from different domains and exhibit a variety of structural properties. Our benchmark includes a word dependency graph Roman-empire, a product co-purchasing network Amazon-ratings, a synthetic graph emulating the minesweeper game Minesweeper, a crowdsourcing platform worker network Tolokers, and a question-answering website interaction network Questions.
We have evaluated a large number of models, both standard GNNs and heterophily-specific methods, and, surprisingly, found that standard GNNs augmented with skip connections and layer normalization almost always outperform specialized models. We hope that the proposed benchmark and the insights obtained using it will facilitate further research in learning under heterophily.
The datasets are available on GitHub and in PyG Datasets. For more details, see our paper.
openreview.net
A critical look at the evaluation of GNNs under heterophily: Are we...
We show that popular heterophilous datasets for node classification have serious drawbacks, propose several new ones, and show that, at this moment, standard GNNs outperform most of the specialized...
💐 GraphML News 🌷
Everything you wanted to know about Clifford Layers and its applications in PDE modeling and molecular dynamics is now collected on a single website, sprinkle with the recent LoGaG presentation (video) and add a little bit of Geometric Algebra intro from bivector for the best experience.
Some freshly arxived papers you might want to grab for the weekend reading:
Can We Scale Transformers to Predict Parameters of Diverse ImageNet Models? by Knyazev et al - introduces Graph HyperNetwork v3 for predicting the weights of neural network architectures. The previous version GHN-2 got a massive recognition at NeurIPS’21 including an interview with Yannic Kilcher. Instead of training neural nets, you could use GHN to estimate model params in one forward pass and it demonstrated a non-trivial performance on ImageNet. In the new version, the authors apply a Graphormer on the model’s computation graph DAG to frame the task as node regression where node parameters correspond to weight matrices in the target neural nets. You can also use GHN for better initialization of model weights instead of random init.
SUREL+: Moving from Walks to Sets for Scalable Subgraph-based Graph Representation Learning by Yin et al - the next iteration of SUREL for link prediction where subgraphs are replaced with random walks for better scalability
Everything you wanted to know about Clifford Layers and its applications in PDE modeling and molecular dynamics is now collected on a single website, sprinkle with the recent LoGaG presentation (video) and add a little bit of Geometric Algebra intro from bivector for the best experience.
Some freshly arxived papers you might want to grab for the weekend reading:
Can We Scale Transformers to Predict Parameters of Diverse ImageNet Models? by Knyazev et al - introduces Graph HyperNetwork v3 for predicting the weights of neural network architectures. The previous version GHN-2 got a massive recognition at NeurIPS’21 including an interview with Yannic Kilcher. Instead of training neural nets, you could use GHN to estimate model params in one forward pass and it demonstrated a non-trivial performance on ImageNet. In the new version, the authors apply a Graphormer on the model’s computation graph DAG to frame the task as node regression where node parameters correspond to weight matrices in the target neural nets. You can also use GHN for better initialization of model weights instead of random init.
SUREL+: Moving from Walks to Sets for Scalable Subgraph-based Graph Representation Learning by Yin et al - the next iteration of SUREL for link prediction where subgraphs are replaced with random walks for better scalability
microsoft.github.io
CliffordLayers - CliffordLayers
Documentation for CliffordLayers
GraphML News
GPT-4 made the graph community scratching their heads as well (maybe not as much as academic NLP researchers) - look at the molecule search example at the very end of the technical report. Andrew White was among the few researchers working on this example, he compiled a thread how GPT-4 empowered with external tools can do a very impressive job proposing new molecules.
Minkai Xu delivered a lecture “Geometric Graph Learning From Representation to Generation” as a part of the cs224w ML with Graphs course at Stanford (perhaps the most famous class about Graph ML). The lecture covers the basics of invariant and equivariant GNNs and introduces GeoDiff, a diffusion model for generating 3D molecules. Slides of the whole Winter’23 course are now available.
Weekend reading:
The Denoscriptive Complexity of Graph Neural Networks - a massive 88-pager from Martin Grohe proving that GNNs fall into the TC0 complexity class. This is a potential breakthrough since many database query languages fall into AC0 and TC0.
Zero-One Laws of Graph Neural Networks by Adam-Day et al. - shows an interesting result that GCN-like MPNNs with random features map final graph representations to zeros or ones with the growing size of graphs. GATs and GINs are not (yet) prone to this behavior.
Allegro-Legato: Scalable, Fast, and Robust Neural-Network Quantum Molecular Dynamics via Sharpness-Aware Minimization by Ibayashi et al - an improved version of Allegro, current SOTA in molecular dynamics simulations, with faster convergence and better stability.
GPT-4 made the graph community scratching their heads as well (maybe not as much as academic NLP researchers) - look at the molecule search example at the very end of the technical report. Andrew White was among the few researchers working on this example, he compiled a thread how GPT-4 empowered with external tools can do a very impressive job proposing new molecules.
Minkai Xu delivered a lecture “Geometric Graph Learning From Representation to Generation” as a part of the cs224w ML with Graphs course at Stanford (perhaps the most famous class about Graph ML). The lecture covers the basics of invariant and equivariant GNNs and introduces GeoDiff, a diffusion model for generating 3D molecules. Slides of the whole Winter’23 course are now available.
Weekend reading:
The Denoscriptive Complexity of Graph Neural Networks - a massive 88-pager from Martin Grohe proving that GNNs fall into the TC0 complexity class. This is a potential breakthrough since many database query languages fall into AC0 and TC0.
Zero-One Laws of Graph Neural Networks by Adam-Day et al. - shows an interesting result that GCN-like MPNNs with random features map final graph representations to zeros or ones with the growing size of graphs. GATs and GINs are not (yet) prone to this behavior.
Allegro-Legato: Scalable, Fast, and Robust Neural-Network Quantum Molecular Dynamics via Sharpness-Aware Minimization by Ibayashi et al - an improved version of Allegro, current SOTA in molecular dynamics simulations, with faster convergence and better stability.