PRISM: Benchmarking Pathology Models in Limited Data Environments

Paper: PRISM: Pathology Reliability In Scarce-label Medicine. A Tile-level Benchmark of Pathology Foundation Models

Authors: Alicem Koyun

Executive Summary

Recent research in the field of pathology has highlighted significant challenges in evaluating the performance of foundation models, specifically in scenarios where data is limited or when models are applied across different institutions. This study introduces PRISM, a new benchmark designed to assess the reliability of pathology foundation models by examining four key areas: model selection, the fraction of available labeled data, calibration of predictions, and the ability to transfer learnings to different datasets.

PRISM was tested on eight distinct foundation models, including CLIP, PLIP, and Virchow2, across six separate datasets. The models were evaluated at various levels of label availability, from 1% to 100%, and the study involved extensive testing, including 864 runs across different conditions. Additionally, a five-hospital study using the Camelyon17 dataset was incorporated to explore how models perform when transferring knowledge between institutions.

The research uncovered four main findings:

1. The relationship between a model’s classification performance and its calibration improves when all labels are available. Specifically, the ability to predict calibration performance based on classification metrics is strong (with a correlation coefficient of 0.52) when fully supervised. However, this correlation weakens significantly when only 1% of labels are available, indicating that calibration becomes more variable and dataset-dependent in low-label scenarios.

2. For more challenging datasets, models tend to collapse to predicting a single class when the available labeled data falls below a certain threshold specific to that dataset. This suggests that model performance can degrade sharply when faced with limited information.

3. The study also found that additional labeled data from the source domain can negatively impact performance in the target domain, particularly when the model's ability to transfer knowledge is low. When the area under the receiver operating characteristic curve (AUROC) falls below 0.85, having more source-domain labels can lead to worse calibration in 86-88% of the tested cases.

4. Temperature scaling, a common technique used to improve model calibration, did not always yield positive results. In nearly 17% of the cases tested (49 out of 288), the method failed to improve calibration, particularly when the model exhibited a tendency towards under-confidence. Moreover, using a temperature calibrated on the source data proved to be detrimental when the transfer of knowledge was not successful.

PRISM is available as an easy-to-install package, complete with reference results for all evaluated scenarios, making it a practical tool for researchers and practitioners in the field. The insights gained from this study could help organizations better understand the strengths and limitations of their pathology models, especially when faced with limited data or cross-institutional applications. By focusing on robustness and calibration, PRISM offers a more comprehensive approach to evaluating model performance in real-world healthcare settings.

Academic Abstract

Pathology foundation models are typically benchmarked on classification performance at full label availability and on in-distribution test sets, while calibration, behavior at low label budgets, and robustness under cross-institutional shift are seldom evaluated jointly. We introduce PRISM, a tile-level benchmark that measures the reliability of pathology foundation models along four axes: model choice, label fraction, calibration, and out-of-distribution transfer. PRISM evaluates eight foundation models (CLIP, PLIP, CONCH, UNI, Virchow2, GigaPath, H-Optimus-0, Midnight) on six tile-level datasets at six label fractions (1% to 100%) with three seeds (864 linear probe runs), extended with a five-hospital Camelyon17 study of 20 directed transfer pairs. Four phenomena emerge. (i) Under full supervision, AUROC rank predicts post-temperature calibration rank consistently across datasets (Spearman rho = 0.52, I^2 = 0%), whereas at 1% labels the relationship becomes dataset-specific (rho = -0.24, I^2 = 67%). (ii) On harder datasets the linear probe collapses to a single predicted class below a dataset-specific label fraction. (iii) Whether additional source-domain labels help or harm target calibration tracks how well the probe transfers: below a transfer AUROC of roughly 0.85, more labels worsen target ECE in 86-88% of combinations. (iv) Temperature scaling fails to reduce ECE in 49 of 288 cells, miscalibration is predominantly under-confidence, and a source-fitted temperature is harmful under failing transfer. PRISM is released as a pip-installable package with reference results for all 288 cells. Code and data: https://github.com/alicemkyn/prism-benchmark