Introducing a new way to learn with

Ego How-To

A concept for future immersive experiences that turns How-To videos into custom augmented reality and mixed reality tutorials.

Watch to learn more

A future of learning in AR, while being guided by an AI-assistant

With Ego How-To, your AR coach could track your progress, answer your questions, assess your technique and provide real-time feedback to improve your outcomes. You could learn at your own pace and in any environment.

Advancing foundational research

Overcoming challenges of video learning and multimodal perception

To achieve our vision, our research requires key advances in first-person ("egocentric") video understanding:

Fine-grained activity recognition and the ability to map an expert's activity viewed in the third-person ("exocentric") view to the user's first-person view
3D body pose estimation from the egocentric viewpoint
3D object reconstruction and rendering from monocular video in the user's personal space

A future where AI enables new ways to learn skills in AR/MR

We are experiencing an unprecedented convergence of AI capabilities and the availability of AR/MR. By addressing core technical challenges, our research is motivated to benefit people all over the world who want to learn something new every day. Our goal is to make high quality expert guidance more accessible and learning more equitable.

Ego-Exo4D

Ego-Exo4D is an initiative that aims to advance AI understanding of human skill by building the first of its kind video dataset and benchmark suite in collaboration with the Ego4D university consortium.

Learn more

Building the future of AI with foundational research

Discover more about our research