How TU Dresden is advancing precision oncology and transforming healthcare AI

TUD Dresden University of Technology is a leading German research institution renowned across Europe, known for its innovation and interdisciplinary work in health sciences, biomedicine, and bioengineering. Within this field, its Clinical AI research group at the Else Kröner Fresenius Center for Digital Health is transforming cancer care through computational innovation, focusing on precision oncology. The group collaborates to develop methods that enhance clinical decision-making in cancer care.

Oncology requires professionals to interpret vast, diverse datasets—from radiographic imaging and genomic data to free-text clinical reports. Yet much of this data remains underutilized. The Clinical AI research group tackles this with advanced deep learning techniques to extract actionable insights.

“Our aim is to build tools that directly improve clinical routine tasks,” says Dr. Isabella Wiest, Junior Research Group Leader for Clinical AI and Large Language Models in Medicine. “We focus on LLMs to process the vast collection of routine medical text, like surgery reports and discharge summaries.”

Using Llama 3.1, the team has developed tools to structure medical information, extract insights, and streamline workflows. These innovations close the divide between advancing data science and the practical needs of medical professionals, ensuring critical patient data is accessible precisely when needed.

“Structured data in healthcare is essential for making it accessible and usable for large-scale analysis or secondary applications,” says Wiest.

TU Dresden’s Clinical AI research group began its journey with Llama 2, drawn to its ability to run locally and securely handle sensitive patient data compared to cloud-based solutions—an essential requirement in healthcare settings.

“Locally deployable LLMs, like Llama, offer a more flexible and efficient solution compared to traditional NLP methods,” Wiest adds. “By structuring complex medical data directly within healthcare institutions, these models pave the way for advancements in research, decision support, and improved patient care.”

Over time, the team transitioned to quantized Llama models via Hugging Face and Llama.cpp, significantly reducing memory requirements. “When we introduced Llama 3.1, it allowed us to tailor our research applications more effectively,” explains Wiest. “The reduced memory requirements made it feasible to run large models on consumer-grade hospital hardware, a breakthrough for deploying AI in clinical environments.”

Today, TU Dresden’she Clinical AI research group utilizes Llama 3.1 8B and 70B for a wide range of clinical applications, including:

• Information extraction: Analyzing and categorizing complex clinical data from medical documents.
• Pre-anonymization of records: Automating the anonymization of patient data before it is exchanged between medical institutions.
• Clinical decision support: Assisting medical professionals in making more informed, data-driven decisions by integrating AI-powered insights into workflows.
• Medical coding assistants: Streamlining and improving the accuracy of classification tasks for medical coding.

“We are now developing the next generation of LLM-based solutions using agent approaches, where models can take actions under the supervision of human doctors," says Dyke Ferber, a clinician scientist at TU Dresden.

Integrating Llama into the research group’s clinical workflows was challenging, particularly in ensuring the models were intuitive for medical staff. With several deployment frameworks available, including Hugging Face Transformers, LangChain, llama.cpp, and Ollama, the team had to identify the best fit.

Another hurdle was handling variable outputs during information extraction tasks, which complicated automation. “By combining llama.cpp’s ‘grammar feature’ with our workflows, we could restrict token choices, ensuring outputs like JSON were reliably structured,” explains Wiest.

Rather than fine-tuning Llama, TU Dresden uses techniques like in-context learning, retrieval-augmented generation (RAG), and agent-based approaches.

“Focusing on the model’s existing capabilities lets us adapt to challenges without compromising performance,” says Wiest.

One of the biggest hurdles in patient care is ensuring that critical, structured information is available when and where it’s needed—whether it’s for research, clinical care, quality control, or decision support. By using privacy-preserving LLMs like Llama, TU Dresden is helping make this a reality.

For example, while most de-identification software only processes plain text, real-world use cases often involve PDF documents, which are difficult to de-identify manually. The team’s anonymization tool, LLM-Alx, facilitates the secure exchange of real-world medical data and patient information between institutions.

“Our solution assists in the de-identifying process, ensuring patient data is protected while significantly reducing the risk of manual errors and workload for medical staff,” says Wiest.

Open source technology has been pivotal to TU Dresden’s success, as privacy and compliance are critical in a healthcare setting. By utilizing locally deployable LLMs like Llama, the Clinical AI research group ensures that sensitive patient data never leaves the hospital’s infrastructure. This approach eliminates the need for third-party involvement in the data processing pipeline, complying with stringent regulations in many countries.

“By keeping data processing entirely on-site, we not only enhance security but also align with legal and ethical standards, fostering greater trust in using AI tools within clinical environments,” says Prof. Jakob N. Kather, leader of the Clinical AI research group.

With full control over sensitive data, the open source model also gives healthcare institutions the flexibility to customize their tools to better utilize clinical data, including free-text data, leading to more accurate decision-making and reduced costs for tailored applications.

As the Llama ecosystem evolves, the team at TU Dresden plans to expand its applications into more complex use cases, such as real-time clinical data analysis and personalized patient care. Enhanced multilingual capabilities and improved handling of medical jargon will be critical for making these tools accessible in diverse healthcare settings.

The team also foresees leveraging specialized Llama models to develop more sophisticated, context-aware solutions tailored to new medical domains beyond oncology.

“The growing capabilities of LLMs like Llama allow us to push boundaries in research and patient care, ensuring better outcomes and broader accessibility,” says Kather.