PRISM: Benchmarking Pathology Models in Limited Data Environments
Recent research in the field of pathology has highlighted significant challenges in evaluating the performance of foundation models, specifically in scenarios where data is limited or when models are applied across different institutions. This study introduces PRISM, a new benchmark designed to assess the reliability of pathology foundation models by examining four key areas: model selection, the fraction of available labeled data, calibration of predictions, and the ability to transfer learnings to different datasets.
PRISM was tested on eight distinct foundation models, including CLIP, PLIP, and Virchow2, across six separate datasets. The models were evaluated at various levels of label availability, from 1% to 100%, and the study involved extensive testing, including 864 runs across different conditions. Additionally, a five-hospital study using the Camelyon17 dataset was incorporated to explore how models perform when transferring knowledge between institutions.
The research uncovered four main findings:
1. The relationship between a model’s classification performance and its calibration improves when all labels are available. Specifically, the ability to predict calibration performance based on classification metrics is strong (with a correlation coefficient of 0.52) when fully supervised. However, this correlation weakens significantly when only 1% of labels are available, indicating that calibration becomes more variable and dataset-dependent in low-label scenarios.
2. For more challenging datasets, models tend to collapse to predicting a single class when the available labeled data falls below a certain threshold specific to that dataset. This suggests that model performance can degrade sharply when faced with limited information.
3. The study also found that additional labeled data from the source domain can negatively impact performance in the target domain, particularly when the model's ability to transfer knowledge is low. When the area under the receiver operating characteristic curve (AUROC) falls below 0.85, having more source-domain labels can lead to worse calibration in 86-88% of the tested cases.
4. Temperature scaling, a common technique used to improve model calibration, did not always yield positive results. In nearly 17% of the cases tested (49 out of 288), the method failed to improve calibration, particularly when the model exhibited a tendency towards under-confidence. Moreover, using a temperature calibrated on the source data proved to be detrimental when the transfer of knowledge was not successful.
PRISM is available as an easy-to-install package, complete with reference results for all evaluated scenarios, making it a practical tool for researchers and practitioners in the field. The insights gained from this study could help organizations better understand the strengths and limitations of their pathology models, especially when faced with limited data or cross-institutional applications. By focusing on robustness and calibration, PRISM offers a more comprehensive approach to evaluating model performance in real-world healthcare settings.
•Implement PRISM benchmark to evaluate model reliability in diverse data environments.
•Optimize model selection based on labeled data availability for improved performance outcomes.
•Enhance prediction calibration processes to ensure accuracy across different datasets.
