How Good Could (Should) My Models Be?

One factor that is often overlooked when building a predictive model is the impact of the experimental error on the performance of the model.  In this post, we will examine one technique for estimating the impact of the experimental error on correlation and estimating the maximum possible correlation that can be achieved with a particular dataset.  

This post was motivated by an excellent series, called Bucket List Papers, that my friends at  MedChemica recently initiated. In this series, they are highlighting 100 important papers in Medicinal and Computational Chemistry.  All of the papers they’ve featured so far have been classics. I’d recommend their Bucket List as essential reading for anyone involved in Cheminformatics, Computer-Aided Drug Design, or Medicinal Chemistry.

I’m not quite so ambitious, but I would like to use this post to highlight an important and frequently overlooked paper.  As usual, we’ll also look at some code to implement the method described in the paper. The paper entitled “Healthy skepticism: assessing realistic model performance” was written in 2009 by Scott Brown, Steve Muchmore, and Phil Hajduk, when they were employed at Abbott Labs.   In the paper, the authors develop a simple simulation method to estimate the impact of the experimental error on the correlation that can be achieved with a predictive model.  

If you stop and think about it, this makes intuitive sense.  If our data has a large experimental error it should reduce the probability that our model will have a high correlation.  On the other hand, if our data has relatively “tight” error bars, we should be able to generate a model with a higher correlation.  In the paper above, the authors develop an elegant, simple model for estimating the maximum possible correlation that can be achieved given a specific experimental error.  

In this method, we begin with a set of experimental observables.  As an example, we’ll use the DLS100 dataset, a set of aqueous solubilities measured by dynamic light scattering by John Mitchell and James McDonagh at the University of St. Andrews.  We will then add normally distributed error to the data and measure the correlation between the original experimental data and the experimental data plus the error.  The error is normally distributed with a mean of 0 and a standard deviation equal to the error in the experiment. In the figure below we can see the impact of 3-fold, 5-fold, and 10-fold error.  As expected, an increase in the experimental error will reduce the correlation. 
At this point our analysis has been purely qualitative, let’s take a look at how we can make this quantitative.  One way to estimate an upper limit for the correlation given an experimental error is to repeat the procedure described above several times.  In this case we'll do 1000 trials each at 3-fold, 5-fold, and 10-fold error. For a dataset X with N values, the procedure can be outlined as:

For 1000 trials:
  • Generate N normally distributed random variables R with a mean of 0 and a standard deviation of log10(A) where A is the fold error. 
  • Add R and X to create a new vector RX
  • Calculate the correlation between X and RX
We can depict the results of these simulations with the box plots below. Based on these boxplots we can get an estimate of the maximum correlation that can be achieved, given a specific experimental error. We can also get an estimate of how much our model will improve if we are able to reduce the error in the experiment.



At a recent conference, I had a number of great conversations with readers of this blog.  Several people mentioned that they prefer violin plots to box plots.  I'm a bit torn on this.  I like violin plots because they enable me to examine the shape of a distribution.  However, I like the way that a box plot enables me to quickly assess the middle 50% of a distribution.   Yes, the middle 50% is there in the violin plot, but it's somewhat buried.   Of course, it's a simple code change to get Seaborn to generate a violin plot.  So, for the violin plot fans, here it is.


Perhaps the most interesting aspect of the paper above, by Brown, Muchmore, and Hajduk, is the analysis of correlations reported in 16 recent life science publications. The authors found that, in half the cases, the reported correlations exceeded the maximum expected correlation given a typical experimental error.  The take-home here is that it's useful to perform this simple analysis to determine whether the correlation you achieve with a model is consistent with what would be expected given the error in your experiments. 

As usual, the code to implement this method and to generate all of the plots above is on GitHub.  If you'd like to run the Jupyter Notebook without having to install anything, you can use this link to run the notebook on Google Colab. 



Comments

Post a Comment

Popular posts from this blog

AI in Drug Discovery 2023 - A Highly Opinionated Literature Review (Part I)

Image
Here’s the first part of my review of some interesting machine learning (ML) papers I read in 2023.  As with the previous editions , this shouldn’t be considered a comprehensive review.  The papers covered here reflect my research interests and biases, and I’ve certainly overlooked areas that others consider vital.  This post is pretty long, so I've split it into three parts, with parts II and III to be posted in the next couple of weeks.    I. Docking, protein structure prediction, and benchmarking II. Large Language Models, active learning, federated learning, generative models, and explainable AI III. Review articles 2023 was a bit of a mixed bag for AI in drug discovery.  Several groups reported that the deep learning methods for protein-ligand docking weren’t quite what they were initially cracked up to be.  AlphaFold2 became pervasive, and people started to investigate, with mixed success, the utility of predicted protein structures.  There were reports of significant advanc
Read more

Generative Molecular Design Isn't As Easy As People Make It Look

Image
I was taken aback by a recent CNBC article entitled “ Generative AI will be designing new drugs all on its own in the near future ”.  I should know better than to pay attention to AI articles in the popular press, but I feel that even scientists working in drug discovery may have a skewed perception of what generative AI can and can’t do.  To understand exactly what’s involved, it might be instructive to walk through a typical generative molecular design workflow and point out a few things.  First, these programs are far from autonomous.  Even when presented with a well-defined problem, generative algorithms produce a tremendous amount of nonsense.  Second, domain expertise is essential when sifting through the molecules produced by a generative algorithm.  Without a significant medicinal chemistry background, one can’t make sense of the results.  Third, while a few nuggets exist in the generative modeling output, a lot of work and good old-fashioned cheminformatics are required to ext
Read more

AI in Drug Discovery - A Highly Opinionated Literature Review (Part II)

Image
Picking up where we left off in Part I , this post covers several other ML in drug discovery topics that interested me in 2023.  Some areas, like large language models, are new, and most of the work is at the proof-of-concept stage.  Others, like active learning, are more mature, and several groups are starting to explore nuances of the methods.   Here’s the structure of Part II.  4. Large Language Models 5. Active Learning 6. Federated Learning 7. Generative Models 8. Explainable AI 9. Other Stuff 4. Large Language Models The emergence of GPT-4 and ChatGPT brought considerable attention to large language models (LLMs) in 2023.  In November and December, several large pharmas held “AI Day” presentations featuring LLM applications for clinical trial data analysis. Many of these groups demonstrated the ability of LLMs to ingest large bodies of unstructured clinical data and subsequently generate tables and reports based on natural language queries.  Aside from some very brief demos on co
Read more