This is a segmentation model (MedFormer architecture) trained for pancreas tumor segmentation, as presented in the paper Scaling Artificial Intelligence for Multi-Tumor Early Detection with More Reports, Fewer Masks.

It serves as a public baseline, specifically referred to as the "segmentation" baseline, and is trained on the PanTS public dataset.

This model is also a starting point for our R-Super framework: you can fine-tune it with radiology reports. Please see our Report Supervision (R-Super) GitHub.

Training and inference code: https://github.com/MrGiovanni/R-Super

Label order

- adrenal_gland_left
- adrenal_gland_right
- aorta
- bladder
- colon
- common_bile_duct
- duodenum
- femur_left
- femur_right
- gall_bladder
- kidney_left
- kidney_right
- liver
- lung_left
- lung_right
- pancreas
- pancreas_body
- pancreas_head
- pancreas_tail
- pancreatic_lesion
- postcava
- prostate
- spleen
- stomach
- superior_mesenteric_artery
- veins

Papers

Learning Segmentation from Radiology Reports
Pedro R. A. S. Bassi, Wenxuan Li, Jieneng Chen, Zheren Zhu, Tianyu Lin, Sergio Decherchi, Andrea Cavalli, Kang Wang, Yang Yang, Alan Yuille, Zongwei Zhou*
Johns Hopkins University
MICCAI 2025
Best Paper Award Runner-up (top 2 in 1,027 papers)

Prize

Scaling Artificial Intelligence for Multi-Tumor Early Detection with More Reports, Fewer Masks
Pedro R. A. S. Bassi, Wenxuan Li, Jieneng Chen, Zheren Zhu, Tianyu Lin, Sergio Decherchi, Andrea Cavalli, Kang Wang, Yang Yang, Alan Yuille, Zongwei Zhou*
Johns Hopkins University

PanTS: The Pancreatic Tumor Segmentation Dataset
Wenxuan Li, Xinze Zhou, Qi Chen, Tianyu Lin, Pedro R.A.S. Bassi, ..., Alan Yuille, Zongwei Zhou^★
Johns Hopkins University

Inference

0- Download and installation.

[Optional] Install Anaconda on Linux

wget https://repo.anaconda.com/archive/Anaconda3-2024.06-1-Linux-x86_64.sh
bash Anaconda3-2024.06-1-Linux-x86_64.sh -b -p ./anaconda3
./anaconda3/bin/conda init
source ~/.bashrc

git clone https://github.com/MrGiovanni/R-Super
cd R-Super/rsuper_train
conda create -n rsuper python=3.10
conda activate rsuper
pip install -r requirements.txt
pip install -U "huggingface_hub[cli]"
hf download AbdomenAtlas/MedFormerPanTS --local-dir ./MedFormerPanTS

1- Pre-processing. Prepare your dataset in the format below. You can use symlinks instead of copying your data.

Dataset format.

/path/to/dataset/
├── BDMAP_0000001
|    └── ct.nii.gz
├── BDMAP_0000002
|    └── ct.nii.gz
...

2- Inference. The code below will inference, generating binary segmentation masks. To save probabilities, add the argument --save_probabilities or --save_probabilities_lesions (which saves only probabilities for lesions, not for organs). The optional argument --organ_mask_on_lesion will use organ segmentations (produced by the R-Super model itself, not ground-truth) to remove tumor predictions outside its organ.

python predict_abdomenatlas.py --load MedFormerPanTS/pants_pancreas_release/fold_0_latest.pth --img_path /path/to/test/dataset/ --class_list MedFormerPanTS/labels_pants.yaml --save_path /path/to/inference/output/ --organ_mask_on_lesion

Argument Details

load: path to the model checkpoint (fold_0_latest.pth)
img_path: path to dataset
class_list: a yaml file with the class names of your model
save_path: path to output, where masks will be saved
ids: this is an optional argument. By default, the code will predict on all cases in --img_path. If you pass ids, the code will only test with the CT scans indicated in ids. You can use this to separate a test set: --ids /path/to/test/set/ids.csv. The csv file must have a 'BDMAP ID' column with the ids of the test cases.

For more details, see https://github.com/MrGiovanni/R-Super/tree/main/rsuper_train#test

Citations

If you use this data, please cite the 2 papers below:

@article{bassi2025learning,
  title={Learning Segmentation from Radiology Reports},
  author={Bassi, Pedro RAS and Li, Wenxuan and Chen, Jieneng and Zhu, Zheren and Lin, Tianyu and Decherchi, Sergio and Cavalli, Andrea and Wang, Kang and Yang, Yang and Yuille, Alan L and others},
  journal={arXiv preprint arXiv:2507.05582},
  year={2025}
}

@article{li2025pants,
  title={PanTS: The Pancreatic Tumor Segmentation Dataset},
  author={Li, Wenxuan and Zhou, Xinze and Chen, Qi and Lin, Tianyu and Bassi, Pedro RAS and Plotka, Szymon and Cwikla, Jaroslaw B and Chen, Xiaoxi and Ye, Chen and Zhu, Zheren and others},
  journal={arXiv preprint arXiv:2507.01291},
  year={2025},
  url={https://github.com/MrGiovanni/PanTS}
}

Acknowledgement

This work was supported by the Lustgarten Foundation for Pancreatic Cancer Research, the Patrick J. McGovern Foundation Award, and the National Institutes of Health (NIH) under Award Number R01EB037669. We would like to thank the Johns Hopkins Research IT team in IT@JH for their support and infrastructure resources where some of these analyses were conducted; especially DISCOVERY HPC. Paper content is covered by patents pending.

Downloads last month: -; Downloads are not tracked for this model. How to track