Benchmarking "One Molecular Fingerprint to Rule Them All"

In a preprint that is currently on Chemrxiv, Capecchi and coworkers describe a new molecular fingerprint that can be used for similarity searching and machine learning. In the paper, the authors demonstrate how their new fingerprint, called MAP4, performs well when used to build classification models. I wanted to see how well the MAP4 fingerprints performed when used to build regression models.

Please note that my intent here is not to be critical of the paper or the authors. I was curious to see how the method would perform with the sorts of models I typically build and thought it might be useful to share my results. I commend the authors for releasing their work as a preprint and for making their code available. In my opinion, this is how science should be done. As usual, all of the code and data that I used to do this analysis is available on GitHub. Rather than detail every step in my analysis here, I'll point the interested reader to the corresponding Jupyter notebook that provides a complete walkthrough.

Note: As Pen pointed out on his blog, there is a bug in the MAP4 code that prevents the generation of folded fingerprints. The version I used in these tests includes Pen's fix for this bug. Thank you, Iwatobipen.

I did the analysis using a set of 24 datasets from ChEMBL that were originally included in a 2019 paper by Andreas Bender's group. These datasets contain between 203 and 5,207 molecules with associated IC50 data. The evaluation method I used was pretty simple. 

For each dataset:

  • Calculate MAP4 fingerprints
  • Calculate Morgan fingerprints using the RDKit (radius=2)
  • For 10 cross-validation folds:
    • Use scikit learn to split the dataset into training and test sets
    • Build and test an XGBoost regression model with MAP4 fingerprints
      • Build an XGBoost regression model with MAP4 fingerprints
      • Use this model to predict the activity of the test set molecules
      • Calculate R**2 between predictions and experimental values for the test set
    • Build and test an XGBoost regression model with Morgan fingerprints
      • Build an XGBoost regression model with Morgan fingerprints
      • Use this model to predict the activity of the test set molecules
      • Calculate R**2 between predictions and experimental values for the test set
After going through the above procedure for each of the 24 datasets, we have data that we can use to generate a boxplot of the R**2 values for each method across the 10 folds.  As can be seen below,  the models generated using the Morgan fingerprints tend to produce higher R**2 values than the models that use the MAP4 fingerprints. 
In order to be a bit more quantitative about this, we will calculate the effect size for each of the comparisons.  If you read my previous blog post on the topic, you'll recall that the effect size, Cohen's d, is defined as the mean of the differences between methods, divided by the standard deviation of the differences. 

We can use the table below to understand the magnitude of the effect size. 
  • Small Effect Size: d=0.20
  • Medium Effect Size: d=0.50
  • Large Effect Size: d=0.80
In the figure below, we plot the effect size for each of the comparisons. Lines are drawn at 0.2, 0.5, and 0.8 indicating small medium and large effect size. As we can see, the effect size is always negative, indicating that the MAP4 fingerprints do not perform as well as the Morgan fingerprints. In addition, in 20 of 24 cases, the effect size is less than -0.8 (large). 

Based on the data I've generated so far, I'll probably stick with the Morgan fingerprints for my regression models. 

Comments

Post a Comment

Popular posts from this blog

AI in Drug Discovery 2023 - A Highly Opinionated Literature Review (Part I)

Image
Here’s the first part of my review of some interesting machine learning (ML) papers I read in 2023.  As with the previous editions , this shouldn’t be considered a comprehensive review.  The papers covered here reflect my research interests and biases, and I’ve certainly overlooked areas that others consider vital.  This post is pretty long, so I've split it into three parts, with parts II and III to be posted in the next couple of weeks.    I. Docking, protein structure prediction, and benchmarking II. Large Language Models, active learning, federated learning, generative models, and explainable AI III. Review articles 2023 was a bit of a mixed bag for AI in drug discovery.  Several groups reported that the deep learning methods for protein-ligand docking weren’t quite what they were initially cracked up to be.  AlphaFold2 became pervasive, and people started to investigate, with mixed success, the utility of predicted protein structures.  There were reports of significant advanc
Read more

Generative Molecular Design Isn't As Easy As People Make It Look

Image
I was taken aback by a recent CNBC article entitled “ Generative AI will be designing new drugs all on its own in the near future ”.  I should know better than to pay attention to AI articles in the popular press, but I feel that even scientists working in drug discovery may have a skewed perception of what generative AI can and can’t do.  To understand exactly what’s involved, it might be instructive to walk through a typical generative molecular design workflow and point out a few things.  First, these programs are far from autonomous.  Even when presented with a well-defined problem, generative algorithms produce a tremendous amount of nonsense.  Second, domain expertise is essential when sifting through the molecules produced by a generative algorithm.  Without a significant medicinal chemistry background, one can’t make sense of the results.  Third, while a few nuggets exist in the generative modeling output, a lot of work and good old-fashioned cheminformatics are required to ext
Read more

AI in Drug Discovery - A Highly Opinionated Literature Review (Part II)

Image
Picking up where we left off in Part I , this post covers several other ML in drug discovery topics that interested me in 2023.  Some areas, like large language models, are new, and most of the work is at the proof-of-concept stage.  Others, like active learning, are more mature, and several groups are starting to explore nuances of the methods.   Here’s the structure of Part II.  4. Large Language Models 5. Active Learning 6. Federated Learning 7. Generative Models 8. Explainable AI 9. Other Stuff 4. Large Language Models The emergence of GPT-4 and ChatGPT brought considerable attention to large language models (LLMs) in 2023.  In November and December, several large pharmas held “AI Day” presentations featuring LLM applications for clinical trial data analysis. Many of these groups demonstrated the ability of LLMs to ingest large bodies of unstructured clinical data and subsequently generate tables and reports based on natural language queries.  Aside from some very brief demos on co
Read more