Combining Structured EHR Data and LLM-Extracted Psychosocial Factors from Clinical Notes to Predict Suicide Risk in Cancer Patients
Maximilian Wakefield
Lay Summary:
Cancer patients can show signs of serious emotional distress that get written down by doctors in their medical notes but never show up in the codes and reports hospitals normally use to track patient health. This project built a tool that uses AI to read those notes, pick out signs of genuine emotional distress, and combine that with a patient's medical history to screen for signs of a mental health crisis.
Abstract:
Cancer patients face an elevated risk of suicidal ideation and self-harm, and warning signs are often documented only in free-text clinical notes rather than structured diagnosis codes. Existing suicide-risk prediction models rely primarily on structured electronic health record (EHR) data, and prior natural language processing (NLP) approaches to suicide risk have largely targeted general psychiatric or veteran populations rather than cancer patients specifically. This study developed and validated a large language model (LLM)-based pipeline to extract psychosocial risk factors from clinical notes and evaluated whether this information improves suicide-related risk prediction beyond structured data alone in a cancer cohort from MIMIC-IV (4,614 patients- 769 cases, 3,845 controls). A two-stage LLM pipeline (Qwen2.5-32B) extracted patient-reported, cancer-linked negative psychosocial content from clinical notes, validated against a manually reviewed control set (12 of 12 correct). Extracted evidence was analyzed using Latent Dirichlet Allocation (LDA) topic modeling, which identified six clinically interpretable themes, including a distinct suicidal ideation and depression theme. Logistic Regression, Random Forest, and Gradient Boosting classifiers were trained on structured-only, note-only, and combined feature sets using patient-level, 10-fold cross-validation. Structured-only Gradient Boosting achieved an AUC of 0.792, within the range of published suicide-risk models. Combining structured data with note-derived features significantly improved Random Forest and Gradient Boosting performance (AUC up to 0.827, p<0.001), though not Logistic Regression, indicating that note content adds predictive value specifically for tree-based, nonlinear models. A longitudinal analysis further showed that the rate of psychosocial evidence in notes rose significantly as cases approached their event, while remaining flat in controls (p=0.0053). These findings suggest that LLM-extracted note content provides model-dependent, clinically meaningful predictive value for suicide-related risk in a novel and understudied cancer-specific population.
Q&A:
Bios: Maximilian Wakefield
Program Track: Skills Development
GitHub Username:
maximilian229 -Maximilian Wakefield
What was your favorite seminar? Why?
My favorite seminar was NLP for Clinical Text Reports & Y Combinator Experience. It connected directly to my own project, since I spent part of the summer applying NLP techniques to extract information from unstructured clinical notes. Hearing about a similar approach applied to clinical text reports felt the most relevant to what I was actually doing, and I learned different approeachs I could use in the future. -Maximilian Wakefield
If you were to summarize your summer internship experience in one sentence, what would it be?
The summer internship experience let me explore many different methodologies while getting to work closely with mentors across Dartmouth, whose guidance and feedback helped me develop a significant project. -Maximilian Wakefield
Blog Post
LLM-Based Extraction of Psychosocial Risk Factors in Clinical Notes
for Suicide-Related Risk in Cancer Patients
Author: Maximilian Wakefield
Cancer patients face an elevated risk of suicidal ideation and self-harm, but a lot of the warning signs for that risk never make it into a diagnosis code. A doctor might write that a patient seems hopeless, scared, or overwhelmed by treatment, and that observation is only in a paragraph of free text, absent from structured EHR data. To make use of that psychological content, can a large language model (LLM) read those notes, pull out that content, and make suicide-risk prediction in cancer patients clinically reproducible and meaningfully better?
Building the Pipeline
I worked with MIMIC-IV, a large, de-identified database of ICU and emergency department records from Beth Israel Deaconess Medical Center, available for research use through PhysioNet. I built a cohort of 4,614 cancer patients (769 with a documented suicide-related event and 3,845 without a documented suicide-related event) and split their data into two tracks: structured EHR data (diagnoses, admissions, demographics) and free-text clinical notes.
Clinical notes are long, repetitive, and full of boilerplate like lab values, medication lists, and discharge instructions, with the actual emotional content buried inside. I built a two-stage LLM pipeline using Qwen2.5-32B. The first stage pulls out every sentence that describes the patient's negative mood or coping difficulty. The second stage checks whether that sentence is the patient's own reported and whether it's connected to their cancer diagnosis or treatment.
Clinical Notes → LLM-Based Filter **→ Machine Learning (Qwen2.5-32B) Model** ——————– ——- ——————– ——- ——————–
Figure 1: The extraction pipeline: raw notes go through an LLM filter, and only cancer-linked evidence feeds the final model.
The first prompts kept confusing physical symptoms and clinician observations with patient emotion, and I found a truncation bug cutting off notes before the LLM saw the relevant sentence. Splitting the task into a two-stage pipeline (first casting a wide net for candidate sentences, then classifying each one specifically as the patient's own words) fixed the conflation problem, along with catching a contamination bug where the model was echoing back my own example sentences, presenting them as evidence. I validated each prompt against six evidence-positive notes and six control notes. The final prompt scored a 12/12. I then ran it across nearly 12,000 notes and kept about 1,100 that contained cancer-linked psychosocial evidence.
Finding Themes
With that filtered evidence, I ran topic modeling to see what themes actually showed up. I tried two different methods, Latent Dirichlet Allocation (LDA) and BERTopic, on two different versions of the text (the full notes and the LLM-extracted evidence), which gave me four combinations to compare. The topics in topic modeling on the full notes were dominated by organ systems and medication names, not psychology, which makes sense, as MIMIC-IV is ICU and ED data.
LDA on the filtered evidence, though, had six clear, clinically interpretable themes, including one theme that was about suicidal ideation and depression; words like “depression,” “anxiety,” “cancer,” and “suicide” all clustered together. That theme showed up relatively more often in patients who went on to have a documented suicide-related event than in patients who didn't.
{width=”6.5in” height=”2.9652777777777777in”}
Figure 2: The six themes LDA found in genuine psychosocial evidence. Topic 1 is the only one containing explicit suicide/ideation language.
Leakage Correction and Structured Accuracy Metrics
Early on, my structured-data model was hitting an AUC of 0.889 (a measure of how well the model separates patients who had an event from patients who didn't, where 0.5 is a coin flip and 1.0 is perfect). That number was unusually and unrealistically high for this kind of task. In a meeting with my mentors, one of them asked me: Are you sure nothing from after the event is leaking into your features?
After looking back at my data that was being used as input, a handful of notes charted the same day as the suicide-related event itself were being used as “before the event” evidence, when they should have been excluded. Second, my psychiatric history flags, things like “history of depression” or “prior self-harm”, had no date cutoff at all. A diagnosis coded during the same hospital stay as the actual event was still counting as “history,” which meant the model was partly learning to recognize the event from information that existed because the event had already happened.
After re-running with leakage-safe data, the structured-only AUC dropped to 0.792. That proved to be a clinically reproducible and accurate metric, and it lands within the range of published suicide-risk prediction models.
Using Clinical Notes in Machine Learning Models
With the leakage issue addressed, I compared three types of models with the data, Logistic Regression, Random Forest, and Gradient Boosting, on structured data alone, note-derived features alone, and both structured and note-derived data combined.
For Logistic Regression, adding note features made no significant difference. But for Random Forest and Gradient Boosting, combining structured data with note-derived features, whether from my topic model or from a simpler word-frequency method called TF-IDF, produced a statistically significant improvement (p < 0.001 for both). The best combined model reached an AUC of 0.827. The value of the notes depended entirely on what kind of model you used to read them, as linear models could not use the extra signal, but tree-based models, which can pick up on more complicated patterns, could.
{width=”6.5in” height=”2.384027777777778in”}
Figure 4: ROC-AUC curves for classification of cancer-related suicide risk. The Gradient Boosting model using topics and structured data ROC-AUC was 0.798, while the Gradient Boosting model using TF-IDF derived features and structured data ROC-AUC was 0.827.
{width=”6.5in” height=”3.3131944444444446in”}
Figure 5: Gini importance graphs with top feature contributions for the combined Gradient Boosting models. Structured features lead both models, with note-derived features contributing psychological significance.
{width=”6.395833333333333in”
height=”2.7291666666666665in”}
Figure 6: Model ROC-AUC for classification of cancer-related suicide risk. Error bars show mean ± SD across 10 cross-validation folds. The p-value compares Combined (structured + notes) vs. Structured-only on the same folds, isolating the added contribution of note features beyond structured data alone. Both TF-IDF and LDA topics significantly improve tree-based models (RF/GB) when added to structured data (paired t-test, p<0.001 for both), but not Logistic Regression.
A Longitudinal Trend
I also examined whether the rate of psychosocial evidence in a patient's notes changes as they approach their event, not just whether it is present at all. For patients who went on to have a documented event, the share of notes containing psychosocial evidence nearly doubled in the month before the event compared to a year or more before it. For patients who never had an event, that rate stayed essentially flat for the entire time. This held up even after I rechecked it against the same leakage issue I found earlier (p = 0.0053), which shows it is a meaningful signal.
{width=”5.395833333333333in”
height=”3.3333333333333335in”}
Figure 7: Longitudinal graph showing increasing case prevalence over time. Cases nearly double approaching the event; controls stay flat (interaction p=0.0053, n=9,068).
Conclusion, Limitations, and Future Work
Combining structured EHR data with note-derived features, whether LDA topics or TF-IDF, significantly improves tree-based models like Random Forest and Gradient Boosting, pushing ROC-AUC to 0.798 and 0.827, respectively, well above the 0.792 achieved with structured data alone. For the longitudinal finding, psychosocial evidence in clinical notes doesn't just indicate risk at one point. It rises as cases approach their event (p=0.0053), suggesting a significant trajectory-based signal. MIMIC-IV v3.1 is ICU/ED-only with no psychotherapy notes, and cancer-specific documentation is very sparse, meaning the model is working with a narrower section of a patient's full psychiatric record. I plan on pulling broader cancer-related reports for the same cohort and validating whether this kind of trajectory-based monitoring could be implemented in a clinical oncology setting.
Acknowledgements
Thank you to Joshua Levy, Louis Vaickus, Alos Diallo, Benjamin Mattern, and Matthew Hayden at Dartmouth and EDIT AI, as well Monica Dimambro and Maxwell Levis at the VA, whose own prior work on LLM-based suicide risk extraction in veterans was the ground work for this project.
Data: MIMIC-IV v3.1 and MIMIC-IV v2.2 Note (PhysioNet), restricted access. All analysis was performed on Dartmouth's Discovery HPC cluster.