How DINO and SAM are Helping Modernize Essential Medical Triage Practices

Triage, derived from the French verb “trier” meaning “to sort,” has its roots in the Napoleonic era as a revolutionary method to prioritize battlefield injuries. It also implies a scarcity of resources compared to a situation’s needs. What began as a practical battlefield necessity has evolved over centuries into a critical pillar of emergency medical response. Today, triage is governed by standardized protocols — decision trees that guide military and civilian first responders through life-or-death decisions, ensuring that care is prioritized to save the greatest number of lives. Yet, in the chaos of a mass casualty incident (MCI) such as a building collapse or war-time explosion, following these strict protocols is a challenge.

Until recently, triage efficacy has not only relied on complex systems design, but on the scale of medical resources and personnel present during an event as well. Today, advancements in computer vision, robotics, and machine learning are effectively eliminating that constraint — pushing the boundaries of medical AI for rapid and effective triage in military settings.

Given these advancements, the United States Defense Advanced Research Projects Agency (DARPA), the research arm of the Pentagon, announced a three-year challenge to spur innovation. Using stand-off sensors onboard autonomous systems, the goal is to detect and identify physiological signatures, which are patterns that indicate a person’s level of critical illness.

In addition to sensors and robotic mobility platforms, the challenge requires teams to build algorithms to autonomously provide real-time casualty identification and injury assessment in limited or no-connectivity environments.

To simulate real-world MCIs, the challenge courses feature difficult conditions for sensors, such as darkness, dust, fog, explosion sounds, and flashing lights. Casualties are buried under rubble or otherwise difficult to find. The stakes are high — any delay in finding and assessing casualties could result in missing the window for effective life-saving intervention. In the challenge, teams are scored based on the percentage of casualties identified, accuracy of casualty injury classification, and identification of urgent casualties before the opportunity to address life-saving interventions has closed.

The Penn Robotic Non-contact Triage and Observation (PRONTO) team brings together experts in trauma from Penn Medicine, as well as experts in robotics and computer vision from Penn Engineering and the General Robotics, Automation, Sensing, and Perception (GRASP) lab at the University of Pennsylvania. By combining cutting-edge robotics with Meta’s SAM and DINO models, the PRONTO team enables autonomous, rapid detection and assessment of injuries in disaster scenarios.

PRONTO deployed the initial version of their system during Phase 1 of the challenge in 2024, using a drone to quickly survey the scene to locate victims and a ground robot for more stable imaging and vital sign capture. Data from this exercise was then put into Meta’s Segment Anything Model 2, which is designed to segment any object in any image or video — even objects and domains it has never seen before. SAM 2 enables segmentation of objects in two ways: with points on an object or a bounding box of the object.

Data collected during Challenge 1 has been used to improve their triage algorithms over the last year for Phase 2 of the challenge. PRONTO includes several parallel injury classifier pipelines, which use SAM and DINO. Unlike traditional models, DINO doesn't require labeled data, making it more efficient and scalable for a variety of tasks. The model is able to generalize across diverse domains, including medical and satellite imagery, and to excel in environments where annotated data is scarce or unavailable. This enables PRONTO to use DINO to extract visual features from the robot’s images, which are then used to identify injuries via a customized deep neural network.

DINO works alongside SAM and Grounding DINO, an open vocabulary object detection model, to provide a comprehensive triage solution. For example, PRONTO uses Grounding DINO with text prompts, such as: “wound?” and “blood?” to detect injury-related features within each image region.

Today, there’s very little definitive evidence on which triage technique would save the most victims given scenario circumstances, making it difficult to compare triage protocols using an evidence-based approach. The DARPA Challenge is changing that.

Phase 2 of the DARPA Challenge ran from September 27 to October 4 and incorporated features of real-time MCI scenarios. Each year of the three-year competition is more challenging than the last, with the goal of pushing life-saving technology closer to effective deployment on the ground.

Throughout the first two phases, DARPA accumulated a vast dataset, establishing a unique infrastructure that can be used more generally for evaluating different mass casualty response strategies. In the third phase, teams will leverage learnings and explore how Meta’s latest versions of SAM and DINO can be applied to triage.

“We are really interested in making this application work in the real world,” says Professor Eric Eaton, team lead for PRONTO. “The people I have on my team are trauma surgeons that deal with this in the trenches every day and researchers working on state-of-the-art robotics and machine learning. Together, we are looking to develop technologies that could be useful in saving lives.”

Want to learn more about how partners are using Meta's AI for social impact applications? Visit our AI For Good website.