Modeling natural conversational dynamics with Seamless Interaction

• As we work to build the future of human connection and the technology that makes it possible, we’ll need models that can generate facial expressions and body gestures based on audio-visual inputs from two people. Meta Fundamental AI Research (FAIR) is introducing a family of audiovisual behavioral motion models to address that need.
• Our models allow the creation of fully embodied avatars in 2D video and as 3D Codec Avatars and could enable the creation of more natural and interactive virtual agents that can engage in conversations with human-like gestures and expressions, transforming telepresence technologies in VR and AR settings.
• We're also releasing the Seamless Interaction Dataset, a large-scale dataset with over 4,000 hours of two-person interactions and over 4,000 unique participants, which we used to train our models. Our dataset captures a diverse range of in-person interactions between two individuals, which enables our audiovisual behavioral models to understand and generate human-like social behaviors.

Leveraging the Seamless Interaction Dataset, we built a family of Dyadic Motion research models that show the potential capabilities of the dataset and also pave the way for future audiovisual behavioral modeling research.

The Audio-Visual (AV) Dyadic Motion models can jointly generate facial expressions and body gestures. The models use audio, either from two people or LLM speech output, as input to produce the behavioral component. Imagine visualizing a previously recorded podcast between two people speaking—generating the full spectrum of emotions, gestures, and movements implied by their speech. The AV Dyadic Motion models produce gestures and expressions of one specific speaker while taking into consideration the audio from both people. This allows the models to visualize speaking gestures, listening gestures, and turn-taking cues. The AV Dyadic Motion models go one step further by also taking into consideration the visual input of the other party. This enables the models to learn visual synchrony cues, such as smile mirroring or joint gaze attention.

The AV Dyadic Motion models can be used to animate avatar behavior, generating the facial expressions and body gestures of two people based on the pre-recordings of their voice. Our video results of this capability show the base case, where only the audio is used as input, and an enhanced case where the visual features are available for one of the two people. Including visual features allows us to show the visual synchrony learned by the audiovisual models. We’ve further developed these models by incorporating extra controllability parameters, providing greater flexibility and control over the models’ behavior. This can be particularly useful when users or designers want to adjust the expressivity of the avatar while speaking or listening. These controllability parameters can also be defined implicitly by the output of a speech LLM, providing visual guidance to the motion model.

Additionally, we designed our audiovisual behavioral research models to output intermediate codes for face and body motion, unlocking a wide range of possibilities for their application. This approach enables us to adapt these models for use in various contexts, including 2D video generation and the animation of 3D Codec Avatars, which can be used in immersive VR and AR experiences. Meta’s Codec Avatars lab has provided the research community with datasets and baseline reference implementations for Codec Avatars to support the advancement of metric telepresence research. More information about Codec Avatars can be found here.

The Seamless Interaction Dataset is the largest high-quality dataset that captures a diverse range of in-person, embodied interactions between two individuals, with facial and body signals recorded simultaneously. Our work is anchored in contemporary psychological theory, which provides a roadmap for us to collect a diversity of conversational topics, interpersonal stances, and emotions.

• We include over 4,000 audiovisual behavioral hours featuring more than 4,000 participants, making this dataset unprecedented in scale.
• The Seamless Interaction Dataset includes approximately 1,300 conversational and activity-based prompts, with naturalistic and improvised content.
• The dataset includes rich contextualization with participant-level relationships and personality metadata, along with nearly 5,000 video annotations.

All conversations were recorded with participants in the same location to preserve the essential characteristics of embodied interaction and to avoid the drawbacks of remote, video-based communication. Of these recordings, one-third are interactions between two people who are familiar with each other, such as family, friends, or colleagues. While this familiarity presents a fascinating study on the effect of relationship on behavior, it also allows participants to be much more interactive from the start compared to the awkwardness that can sometimes occur when two strangers meet for the first time. It was important that the dataset also captured a wide range of human emotions and stances, such as surprise, disagreement, determination, and regret—in other words, the long-tail of human face-to-face behavior. These types of interactions are difficult to capture in naturalistic data, so we recruited professional actors with improvisational experience to portray a range of roles and emotions. These conversations account for approximately one-third of the dataset.

We prioritize privacy, ethics, and quality in data collection and processing for our research. This approach guides our AI research and applications.

Privacy and ethics

We implemented several measures to protect the privacy of the people who allowed us to record their conversations for the purpose of building the Seamless Interaction Dataset. During the creation of the dataset, participants consented to the collection of their recorded conversations and were advised to avoid sharing personally identifiable information. To further protect participant anonymity, about one third of the conversations were scripted, minimizing the risk of disclosing personal details. A post-collection quality assurance process was also established to analyze every video for private or sensitive material.

Quality assurance processes

A quality assurance process was implemented to identify occurrences of sensitive material and personally identifiable information. Flagged content was then removed from the dataset. Our process involved a multi-stage approach with an initial human-based quality review covering hundreds of hours of video including samples from every recorded session, followed by model-based analysis covering the entire dataset.

• Human-based review: Human reviewers assessed video content for signs of sensitive material and personally identifiable information.
• Text-LLM transcripts analysis: Written transcripts were analyzed to detect occurrences of sensitive material and personally identifiable information.
• Video Language model-based video analysis: Video Language models were used to detect occurrences of sensitive material and personally identifiable information.

We employed a conservative filtering strategy by combining the flags from all three methodologies, ultimately removing hundreds of hours of interactions flagged by any one system. This approach allowed us to maintain high standards of quality and integrity in our data collection efforts.

Watermarking

We use AudioSeal and VideoSeal to watermark content that is generated from our audiovisual behavioral models. These models enable us to embed secret messages into individual frames of original content, which can then be extracted by detectors. This allows us to verify the authenticity and origin of content, even after processing or manipulation. By implementing watermarking, we aim to provide an additional layer of security, transparency, and accountability.