PREDICT: Pragmatic Recalibration and Evaluation of Drift in Clinical Tools, CY P09 23 03

Over the next decade, prediction tools that use computer power and artificial intelligence are expected to become even more common in the NHS. These tools will be used by clinicians and patients to help make decisions about their treatment. However, as the population changes over time these prediction tools become less accurate. This is known as temporal drift. For instance, the average person in the UK is older and heavier than ten years ago, this can change their chance of developing diabetes. A tool that predicted diabetes 10 years ago will therefore not perform as well in today’s population. Indeed, a popular risk tool for heart problems called QRISK2 was recommended for removal from GP systems because of this temporal drift issue. At the moment, there are no guidelines for how to monitor prediction tools we use in clinical medicine over time. For example, there are no rules on how often we should test and fix temporal drift. Without this guidance in place, there is a risk that out-dated, inaccurate prediction tools will be used in the NHS, leading to poor quality care for patients that could cause harm. This will become an increasingly important problem in the future as more of these prediction tools are being used within the NHS. In this research project, we will investigate different strategies for testing and fixing temporal drift within prediction tools. Some of these strategies are statistically sophisticated but hard to implement, and others are simpler but easier to implement. We will investigate the trade-offs between these approaches using real-life prediction tools in general practice data. This will allow us to make recommendations for how the NHS should regulate and monitor prediction tools in the future.

31/01/2021

3 years