3D Reconstruction of Fluorescence Guided Intraoperative Margin Assessment
Arjun Garg, Arav Srivastava, Violet Yan & Aahan Sachdeva
Lay Summary:
Surgeons can use fluorescent dyes to help find tumors, but the resulting images are flat while tumors are three-dimensional, making it difficult to see exactly where cancer cells are. Our project uses computer vision and machine learning to combine many tissue images into 3D tumor models, helping connect what surgeons see during an operation with what the tumor actually looks like under a microscope.
Abstract:
Although fluorescence-guided surgery allows tumor visualization in real time, the applicability of this procedure in clinical practice is limited by the presence of spatial mismatch between the two-dimensional fluorescence visualization and three-dimensional reality of tumor cells that can be evaluated through standard hematoxylin and eosin (H&E) histopathology. Conventional pathology uses ultrathin 2D slices of tissues that are prone to deformations due to their physical nature, and hence, manual alignment of multiple serial sections to validate molecular signals against the underlying morphological structures is highly impractical. We have designed an automated, multimodal computational pipeline that would allow us to generate spatially coherent 3D models of tumors using 2D serial sections. Within our study, we used nine samples of mouse tongue tumors as a dataset and integrated dynamic tissue detection with non-rigid coregistration (VALIS) to align H&E and multi-channel fluorescence images. Next, we applied the marching cubes method for the generation of the 3D surfaces and trained a 2D U-Net to segment tumor boundaries based only on fluorescent signatures. Our non-rigid registration allowed alignment for 25 out of 27 image sets and thus allowed generating the continuous volumetric arrays with preservation of the cellular architecture and tumor location along the z-axis. Furthermore, the 2D U-Net showed good prediction power, with the highest Dice coefficient being 0.6683, precision 0.9321, and recall 0.8408. Our volumetric approach allows overcoming the limitations associated with conventional pathology by means of mathematical linkage of targeted molecular signals to histological ground truth. Ultimately, multidimensional spatial maps can serve as a basis for the further development of the next-generation augmented reality surgical systems and intraoperative margin assessment.
Q&A:
Bios: Arjun Garg,Arav Srivastava,Violet Yan,Aahan Sachdeva
Program Track: Advanced Research
GitHub Username:
arjungarg95 -Arjun Garg
UnSospiro123lol -Arav Srivastava
violetyan10 -Violet Yan
aahan09 -Aahan Sachdeva
What was your favorite seminar? Why?
My favorite seminar was Zarif Azher’s because it showed how computational methods can be applied to questions that are directly relevant to patient care. I enjoyed seeing how pathology, cancer biology, and data science can intersect, and it made me think more about how machine learning can move beyond prediction into actually helping us understand disease. The seminar also stood out because it connected well with the other topics we covered throughout the summer, while giving me a different perspective on how these tools can be used in medicine. I left with a better appreciation for how computational pathology can contribute to both research and clinical decision-making. -Arjun Garg
My favorite seminar was the seminar by Louis Vaickus because it taught me how to -Arav Srivastava
AI for Thyroid Cytopathology by Anvith Kakkera was my favorite seminar, because my mom had thyroid cancer and the doctor had a difficult time making the diagnosis using traditional biopsy. -Violet Yan
I watched the recording of the NLP for Clinical Text Reports & Y Combinator Experience seminar. I liked hearing how previous EDIT AI research developed beyond the summer into publications and other opportunities, and how research experience can connect to real-world applications. -Aahan Sachdeva
If you were to summarize your summer internship experience in one sentence, what would it be?
This summer, I learned how to turn complex pathology images into 3D models and use AI to make them more useful for understanding and treating cancer. -Arjun Garg
Amazing Hands-on experience working with professionals in the field and gaining real research experience. -Arav Srivastava
Lots of hands-on effort that’s pretty rigorous but worthwhile since there is so much to be learned. -Violet Yan
This summer gave me the opportunity to work on a challenging AI research project while learning more about computer vision, medical imaging, and the research process. -Aahan Sachdeva