Assessing Interpretable Models

 This is just a pointer to the fastpages blog post. 

Comments

Post a Comment

Popular posts from this blog

Some Thoughts on Splitting Chemical Datasets

Image
  Introduction Dataset splitting is one topic that doesn’t get enough attention when discussing machine learning (ML) in drug discovery. The data is typically divided into training and test sets when developing and evaluating an ML model. The model is trained on the training set, and its performance is assessed on the test set. If hyperparameter tuning is required, a validation set is also included. Teams often opt for a simple random split, arbitrarily assigning a portion of the dataset (usually 70-80%) as the training set and the rest (20-30%) as the test set. As many have pointed out, this basic splitting strategy often leads to an overly optimistic evaluation of the model's performance. With random splitting, it's common for the test set to contain molecules that closely resemble those in the training set. To address this issue, many groups have turned to scaffold splits. This splitting strategy, inspired by the work of Bemis and Murcko , reduces each molecule to a scaffold...
Read more

We Need Better Benchmarks for Machine Learning in Drug Discovery

Image
Most papers describing new methods for machine learning (ML) in drug discovery report some sort of benchmark comparing their algorithm and/or molecular representation with the current state of the art.  In the past, I’ve written extensively about statistics and how methods should be compared .  In this post, I’d like to focus instead on the datasets we use to benchmark and compare methods.  Many papers I’ve read recently use the MoleculeNet dataset, released by the Pande group at Stanford in 2017, as the “standard” benchmark.   This is a mistake.  In this post, I’d like to use the MoleculeNet dataset to point out flaws in several widely used benchmarks.  Beyond this, I’d like to propose some alternate strategies that could be used to improve benchmarking efforts and help the field to move forward.   To begin, let’s examine the MoleculeNet benchmark, which to date, has been cited more than 1,800 times.   The MoleculeNet collection consis...
Read more

Comparing Classification Models - You’re Probably Doing It Wrong

Image
In my last post , I discussed benchmark datasets for machine learning (ML) in drug discovery and several flaws in widely used datasets.  In this installment, I’d like to focus on how methods are compared.  Every year, dozens, if not hundreds, of papers present comparisons of ML methods or molecular representations.  These papers typically conclude that one approach is superior to several others for a specific task.  Unfortunately, in most cases, the conclusions presented in these papers are not supported by any statistical analysis. I thought providing an example demonstrating some common mistakes and recommending a few best practices would be helpful.  In this post, I’ll focus on classification models.  In a subsequent post, I’ll present a similar example comparing regression models.  The Jupyter notebook accompanying this post provides all the code for the examples and plots below.  If you’re interested in the short version, check out the Jupyt...
Read more