NLP in Pathology Reports
Shrey Shah, Shreyas Kalidindi & Daniel Gabriel
Lay Summary:
Every cancer diagnosis comes with a detailed reports containing critical details, but the information is locked in dense medical language and not feasible to use at scale. Our team taught computers to read these reports automatically, pulling the important details like cancer type.
Abstract:
Pathology reports contain diagnostic and staging information that could be useful for cancer research; however, the majority of information contained in these reports is in complicated clinical text format. It would take a long time to extract this information manually, and regular structured datasets would overlook information contained in pathology reports. NLP can provide an opportunity to automatically identify information related to cancer in these pathology reports. This paper examines NLP techniques to correctly extract cancer-related information in complicated clinical texts by analyzing 9,523 TCGA pathology reports of 32 different cancer types. In our analysis, two tasks are classifying the cancer type and predicting the T, N, and M components of the TNM stage. The traditional machine learning algorithms based on TF-IDF text representations are first tested using Logistic Regression, LinearSVC, Multinomial Naive Bayes, and XGBoost. Using a stratified split for classification of these texts, LinearSVC had a macro-F1 score of 0.967 and an accuracy of 0.975, which were the best among the other classifiers. Secondly, transformer-based language models like BioClinicalBERT are considered for the same set of reports. Although the proposed models had similar performance to the conventional approach in the cancer-type prediction task, long-context transformers like Clinical-ModernBERT showed considerable improvement in the TNM staging task, as much of the staging information is deep within the report and goes beyond the constrained context window available in normal models. Lastly, zero-shot and few-shot prompt techniques were used to evaluate large language models, and they lagged behind other techniques. Therefore, from the findings, it can be noted that the most appropriate technique depends on the task.
Q&A:
Bios: Shrey Shah,Shreyas Kalidindi,Daniel Gabriel
Program Track: Skills Development
GitHub Username:
ShreyShah1000 -Shrey Shah
ShreyasK-git -Shreyas Kalidindi
daniel-j-gabriel -Daniel Gabriel
What was your favorite seminar? Why?
My favorite seminar was Dr. Zarella’s presentation because he clearly showed the potential of current technology in clinical settings. This seminar really helped me understand the problems ran into with AI in clinical practices as well, and overall it got me very excited to work on future projects! -Shrey Shah
This was my favorite seminar because I found the connection between aging and cancer really interesting. I liked learning how senescent cells can contribute to a tumor-promoting environment and how AI can be used to identify them. I also thought it was interesting that the research could lead to new ways of preventing cancer instead of only treating it after it develops. The examples of AI being used with tissue samples made the connection to technology even more interesting to me. -Shreyas Kalidindi
My favorite seminar was Zarif Azher’s talk on research and entrepreneurship. He drew a really clear line between the work we do at EDIT AI and the way entrepreneurs think and do on a day-to-day basis, and it got me looking at the skills I picked up this summer as something I could carry into my own ventures down the road. -Daniel Gabriel
If you were to summarize your summer internship experience in one sentence, what would it be?
Loads of learning, fascinating seminars, and a lot of fun. -Shrey Shah
My summer internship was a hands-on experience where I worked on cancer research using NLP and AI while learning how to approach real research problems and collaborate with a team. -Shreyas Kalidindi
Using coding and AI to contribute to real clinical research while learning the skills needed to do it -Daniel Gabriel
Blog Post
[Using NLP to Read Pathology Reports]{.underline}
By Shrey Shah, Shreyas Kalidindi, Daniel Gabriel
The problem
There exists a pathology report for every diagnosis of cancer. It is a highly technical document where a pathologist describes the appearance of a tumor under the microscope: its type, size, extent of spread. These reports contain a great deal of the most valuable information in the medical record, but are written in a technical language that is challenging to search and analyze at any scale. This summer, we asked ourselves a single question: how well could modern AI handle these reports and extract meaningful information?
The data used in our research was TCGA-Reports, a publicly available dataset consisting of about 9,500 de-identified pathology reports from The Cancer Genome Atlas. We aimed to design systems that could interpret the raw text and provide two pieces of information: type of cancer described in the report and tumor stage according to the standardized TNM staging system. (TNM staging involves dividing the stage into three categories: T, describing the size of the primary tumor, N, indicating if it has spread to lymph nodes, and M, if it has spread distantly.)
Our approach: three phases, each bringing increasing sophistication
We did not opt for designing the most sophisticated model right away, opting instead for an approach that would enable us to assess the gains and the limits of increased sophistication.
{width=”5.015625546806649in”
height=”1.88291447944007in”}
Phase 1: Traditional NLP. At first, we approached the problem with traditional methods. The core of this approach lies in using TF-IDF technique, which represents documents through which words it uses and their distinctiveness, followed by training classifiers. This step was designed to serve as a baseline for our experiments, yet it ended up setting a surprisingly high bar.
Phase 2: Transformer models (BERT).
Next, we fine-tuned BERT-type models pre-trained on biomedical and clinical text. These models have a much better understanding of the nuances of medical language thanks to being context-aware, unlike simple word counting methods.
Phase 3: Large language models.
Lastly, we used a prompt-based approach with a medical large language model. This approach implies prompting a model of general-purpose AI, such as the one powering modern chat assistants. Our goal was to estimate the performance of the system on these tasks without any task-specific training.
Our findings
We had a fairly decent solution to classifying cancer type. Our best-performing traditional model achieved a macro-F1 of 0.97 and 97.5% accuracy. (The macro-F1 is a metric with values from 0 to 1. As opposed to simple accuracy, it gives equal weight to rare and common cancers, thus preventing a model from getting high scores simply by guessing common cancers correctly.) What came as a surprise, though, is that the much more complex BERT model achieved the same score of about 0.97. The baseline approach of counting words had already accounted for nearly all of the possible nuances. There was also a gradient-boosted approach which scored slightly higher in theory but was discarded due to overfitting: it achieved a perfect score in its training set and was much slower. Choosing a simpler model is an example of such decisions that appear all the time in the field of machine learning.
Classifying cancer staging turned out to be a challenge. Even though we used the same models, none of them was able to classify T/N/M stage as accurately as they classified the cancer type. The best model, PubMedBERT, fell far below its previous score, and the three stages of the process were not equally difficult:
{width=”5.005208880139983in”
height=”0.8903499562554681in”}
Why is the staging process harder than cancer type classification? First of all, the cancer type is always mentioned in the report explicitly, whereas hints for the stage may be found all throughout the text. Tumor size, depth of tumor invasion, number of lymph nodes involved. In some cases, the stage itself is implied, not explicitly stated. We also decided to try another model named Clinical-BigBird, since it seemed that our other models were losing information by splitting long reports. The model could process much more of each report. Ultimately, the improvement did not seem worth the cost in computation resources, so we stuck to
our traditional models. This was yet another lesson – sometimes it is not true that the more expensive tool does its job better. The text-based staging is a difficult task, and our numbers show that.
The largest model wasn’t the best. This is where the biggest surprise came from. The large language model was clearly the most powerful system among those we compared, and still lost to both of the other two approaches on both tasks. It achieved only macro-F1 of around 0.39 on T-staging and 0.54 on cancer type classification. Additionally, it sometimes refused to commit to one of the possible classifications, leaving up to a quarter of answers unmatched with any of the valid stage codes. Moreover, giving it some examples of how the question should be answered ("few-shot" prompting) even worsened the performance of the model. The conclusion that can be drawn here is that bigger and more general isn't always better. In case of narrow and well-defined classification tasks, a specialized model usually beats the general one.
[What we've learned]{.underline}
The scores are important, but the main lessons were actually learned during the process. Most of machine learning is doing the boring and routine things which aren't presented on the slides. Cleaning up the messy data, hunting the bugs which don't say what they are, checking whether the promising result is actually correct: most of machine learning consists of such things. The most helpful findings for us were the ones where something did not work and why. Learning what does not work and why turned out to be as important as discovering something that works.
[Where this is headed]{.underline}
The main thing that we learned during this project is that the textual content of the pathology report contains useful information for diagnosis. The next step from there is using not only the report text, but also the information from tumor images and genomic analysis to check whether the combination of the three methods gives better prediction of patient outcomes. NLP research from this summer is going to become both a starting point and a baseline for this project.
In plain terms
We have taught computers how to analyze cancer reports on cancer diagnosis and extract vital medical information. They did a terrific job identifying the type of cancer but found it much more difficult to stage it. Moreover, it was clear that the most advanced AI was not necessarily the best one for this task.