Automated Bladder Cancer Screening with Deep Learning Algorithms
Siddhartha Vatsa & Joshua Wang
Lay Summary:
As a common and highly recurrent cancer, bladder cancer requires enhanced screening methods to promote early detection. By combining both deep learning and machine learning algorithms, we were able to better assess abnormalities in cell samples, speeding up the efficiency and accuracy of screening.
Abstract:
Bladder cancer is the 9th most common cancer globally, with non-muscle invasive bladder cancer having 10-year recurrence rates of around 70%. Given these concerning statistics, we combined deep learning and machine learning to build a pipeline that automates and optimizes the urine cytology process. Using a U-Net, we segmented urothelial cells into nucleus, cytoplasm, and background regions, then extracted 5 morphological features (nucleus area, cytoplasm area, N/C ratio, solidity, circularity) to classify the cells into diagnostic reporting categories with a Support Vector Machine. We used traditional multi-Otsu thresholding as a benchmark for segmentation accuracy evaluation. The U-Net achieved a segmentation accuracy of 0.92 on a held out validation split (20% of the 191-image training set) and a mean IOU score of 0.846. The multi-Otsu thresholding produced a mean IOU of 0.662 and a pixel wise segmentation accuracy of 0.91, although only 32% of nucleus pixels were properly detected by the baseline (recall = 0.32). Downstream, the SVM achieved a classification accuracy of 0.68 on a test dataset of 100 images. It performed better than the baseline, which yielded a classification accuracy of only 0.5. Upon analysis of each feature, we found that the most important metric was the nucleus area, not the N/C ratio as we had expected. The second most important feature was the cytoplasm area, which had a Spearman correlation of -0.434. N/C ratio ranked 3rd in the list (from most to least important)- nucleus area, cytoplasm area, N/C ratio, solidity, circularity. Moderate-to-strong positive Spearman correlations of 0.595 between nucleus area and actual category and 0.592 between N/C ratio and actual category indicate that both biomarkers are reliable predictors of malignancy. In conclusion, we found that combining deep learning segmentation with machine learning classification is a more accurate, scalable alternative to traditional computer vision algorithms for automated urine cytology.
Q&A:
Bios: Siddhartha Vatsa,Joshua Wang
Program Track: Skills Development
GitHub Username:
svat44 -Siddhartha Vatsa
sjedi77 -Joshua Wang
What was your favorite seminar? Why?
My favorite seminar was the one with Elijah Renner regarding AI in Entrepreneurship because it aligned with my vision on how we can use AI and machine learning to come up with new innovations. -Siddhartha Vatsa
My favorite seminar was Elijah Renner’s seminar on NLP and Y Combinator. I particularly enjoyed it because I have interest in entrepreneurship and liked hearing about his experience with starting a company. -Joshua Wang
If you were to summarize your summer internship experience in one sentence, what would it be?
Although I faced a lot of unlucky-timing situations that caused me to be unable to attend a lot of seminars, I definitely learned a lot about cancer pathology and machine learning, and the mentors and Dr. Levy were great and always ready to help. -Siddhartha Vatsa
Amazing learning experience that allowed me to gain insight into the research process and apply machine learning to an impactful project. -Joshua Wang
Blog Post
EDIT AI Skills Track Blog Post
Contributors: Siddhartha Vatsa, Joshua Wang
Intro
Bladder cancer is the 9th most common cancer worldwide and has very high recurrence rates, making early detection essential for preventing disease progression. One common non-invasive screening method is urine cytology, where pathologists examine cells under a microscope for signs of cancer. While effective, this process is time-consuming and can vary between pathologists, leading to inconsistent diagnoses. In this project, we explored whether machine learning and deep learning methods could improve the consistency, efficiency, and accuracy of bladder cancer screening.
Methodology
Our approach consisted of two stages: first segmenting each urine cell image into nucleus, cytoplasm, and background, then using the produced segmentation mask to classify each cell into one of four ordered diagnostic categories: negative, atypical, suspicious, or positive. Initially, we evaluated how well multi-Otsu thresholding performed on segmentation. After segmenting each image, we used the resulting masks to calculate a nucleus-cytoplasmic ($\frac{N}{(N + C)}$) ratio and intersection-over-union (IoU) scores to provide a baseline against U-Net segmentation.
-
Preprocessing was required: Otsu thresholding works best on simple histograms, so we grayscaled each image and set the intensity range to [0,1] instead of [0,255] for more streamlined results.
-
Performing classification of cells on the specimen cell dataset was tricky: due to different staining/contrast issues, we performed per-image normalization on each cell from the data as it was being evaluated.
-
Various methods of thresholding were attempted, such as fixed-adjustment and Optuna-optimized thresholding, finally landing on multi-Otsu due to its efficiency.
In the proposed machine learning pipeline, we segmented nucleus, cytoplasm, and background in urothelial cells with a U-Net, then extracted 5 clinical features: N/C ratio, circularity, solidity, nucleus area, and cytoplasm area. Based on these features, we used a Support Vector Machine to classify cells into one of four diagnostic categories.
Results
We first evaluated segmentation performance. Multi-Otsu thresholding achieved a segmentation accuracy of 0.91, although nucleus detection was its biggest weakness as only 32% of nucleus pixels were correctly identified (recall = 0.32). This can be attributed to the Otsu equation often over-estimating the threshold for the nucleus-cytoplasm border.
To evaluate the U-Net’s segmentation performance, we compared its results against multi-Otsu thresholding. We found that the U-Net produced significantly stronger results, with a 0.846 segmentation IoU and a pixel wise accuracy of 0.92.
For classification, applying fixed cutpoints to the N/C ratio derived from the multi-Otsu segmentation gave us a baseline accuracy of 0.50. To find the best classifier for the extracted features, we evaluated three models: Ordinal Regression, Random Forest, and Support Vector Machine.
-
Ordinal Regression achieved an accuracy of 55% but struggled because it separates categories using rigid linear decision boundaries.
-
Random Forest improved performance to 64% by learning more flexible decision rules, although it still had difficulty distinguishing fundamentally similar categories.
-
Support Vector Machine (RBF kernel) achieved the highest accuracy at 68%. By learning curved decision boundaries, the SVM better captured the complex relationships between extracted cellular features and diagnostic categories.
In addition to evaluating segmentation and classification accuracy, we also evaluated the importance of each feature through permutation importance (randomly shuffling values in a feature column to see how much the worse the model gets). The worse a model’s accuracy is after shuffling a feature column, the more important that feature is. We found that nucleus area was the most important feature.
List of feature importance (ranked most to least important)
-
Nucleus area
-
Cytoplasm Area
-
N/C ratio
-
Solidity
-
Circularity
Discussion
The multi-Otsu baseline’s segmentation accuracy (0.91), was significantly inflated due to the score incorporating correct background identification. The individual accuracy on each category is much more interesting: once the threshold labels a pixel as “nucleus”, it is always correct (precision 1.00). However, it only finds 32% of correct nucleus pixels, which highlights a major issue with Otsu thresholding: one numerical cutoff is not substantial enough to incorporate all different types of cells and correctly classify each pixel within them. The same limitation shows up in classification, where recall ranged from just 20% for atypical cells to 88% for negative cells. The baseline reliably catches the clearest cases but struggles wherever categories visually overlap, which is exactly where careful classification matters most.
The proposed pipeline addresses this because instead of relying on just one threshold like intensity values and N/C ratio for segmentation and classification, the U-Net directly learns to predict pixel classes and the SVM classifies based on 5 features, which provides higher stability and trustworthiness compared to just one feature. This allowed for enhanced segmentation (0.92 accuracy) and classification (0.68 accuracy).
Future improvements will likely come from training on larger datasets, incorporating additional informative biomarkers, and eventually developing an end-to-end deep learning model that directly predicts diagnostic categories from cell images without requiring segmentation and manual feature extraction.
Conclusion
Overall, the proposed U-Net + SVM pipeline consistently outperformed the traditional multi-Otsu baseline in segmentation quality, nucleus detection, and classification accuracy. These results suggest that deep learning and machine learning can provide a more reliable and efficient approach for automated urine cytology and has the potential to improve future bladder cancer screening systems.