MIT Unveils Privacy Concerns in the Age of Clinical AI
The world of healthcare is facing a critical challenge as patient privacy becomes increasingly threatened by data-hungry algorithms and cyberattacks. The Hippocratic Oath, a cornerstone of medical ethics, emphasizes the importance of confidentiality, ensuring patients can trust their doctors with sensitive information. However, a recent study by MIT researchers has shed light on a potential vulnerability in the era of clinical AI.
The research, presented at the 2025 Conference on Neural Information Processing Systems (NeurIPS), focuses on the memorization capabilities of artificial intelligence models trained on de-identified electronic health records (EHRs). These models, designed to generalize knowledge from multiple patient records, may inadvertently memorize specific patient data, posing a significant privacy risk. The study highlights the need for rigorous testing to prevent data leakage, emphasizing that any compromise to patient privacy should be evaluated within the healthcare context.
Sana Tonekaboni, a postdoc at the Eric and Wendy Schmidt Center at the Broad Institute of MIT and Harvard, and the first author of the paper, warns about the potential dangers. She explains that while high-capacity models can be valuable resources, they are susceptible to adversarial attacks, where attackers can prompt the model to reveal sensitive information. This risk is particularly concerning for foundation models, which are already known to be prone to data leakage.
To address this issue, MIT Associate Professor Marzyeh Ghassemi, a principal investigator at the Abdul Latif Jameel Clinic for Machine Learning in Health, collaborated with Tonekaboni. They developed a series of tests to assess the potential risks associated with EHR foundation models. These tests aim to measure various types of uncertainty and evaluate their impact on patient privacy by considering different levels of attack possibility.
Ghassemi emphasizes the practicality of their approach, stating that if an attacker requires specific details about a patient's records to extract information, the risk of harm is minimal. She questions the necessity of attacking a large foundation model when access to protected source data is already available. The digitization of medical records has led to more frequent data breaches, with the U.S. Department of Health and Human Services recording over 700 breaches in the past two years, primarily due to hacking incidents.
The study reveals that patients with unique conditions are particularly vulnerable, as their data is easier to identify and extract. Tonekaboni highlights the importance of understanding the type of information leaked, as even de-identified data can be compromised if specific details are revealed. The researchers found that the more information an attacker has about a patient, the higher the likelihood of data leakage, and they developed methods to distinguish between model generalization and patient-level memorization.
Furthermore, the study categorizes different types of data leaks, noting that some are more harmful than others. For instance, revealing a patient's age or demographics may be less concerning compared to exposing sensitive information like an HIV diagnosis or alcohol abuse. The researchers plan to expand their interdisciplinary approach, involving clinicians, privacy experts, and legal professionals to enhance the protection of patient data.
Tonekaboni concludes by emphasizing the importance of maintaining patient privacy, stating that there is no reason for others to have access to such sensitive information. The research is supported by various institutions and foundations, including the Eric and Wendy Schmidt Center, Wallenberg AI, and the U.S. National Science Foundation, ensuring a comprehensive approach to addressing the privacy concerns in clinical AI.
