Depending on the data, this could also be phrased with regards to regime change or concept drift. How should you update models trained on data generated during the previous regime so that they are useful given data generated in the new regime? How to train brand new models with the limited data that we have for the current regime? Further, it’s quite possible that the current regime is non-stationary and will continue to change and drift, further complicating efforts to keep models up-to-date and useful. Last, how much confidence do you have in your models over time as the data continues to evolve? Your model evaluation will help drive decisions about which models to update and how. These are important and interesting technical questions that data science teams should be considering and addressing where needed.

During a time when many companies are facing financial hardship, addressing these challenges is critical to generating value to the business from data science.

Data scientist working in the financial services industry