• Surface electromyography (sEMG) measures muscle activity at the wrist from electrical signals based on a user’s intended action. Machine learning models translate these signals into digital commands, making sEMG a fast and powerful input for next-generation computing platforms.
• We’re releasing emg2qwerty and emg2pose—two large datasets and benchmarks for sEMG-based typing and pose estimation, as part of the NeurIPS 2024 Datasets and Benchmarks track.
• The datasets comprise 301 consenting individuals and 716 hours of sEMG recordings, the largest open source sEMG datasets to date—each offering 10x as much data as the next largest single-task, single-device dataset.
• We’re also releasing state-of-the-art models for typing and pose estimation from sEMG.

Wrist-based surface electromyography (sEMG) measures muscle activity from electrical signals at the wrist, based on a user’s intended action. Machine learning models translate these signals into digital commands, making sEMG a fast and powerful input for wearables, among other devices. The wrist is unique because humans are highly adapted to use our hands for fine motor tasks, and many of the muscles that control these movements pass through it. This means there’s a massive amount of information present in these sEMG signals.

At Connect 2024, we showed how you can use our EMG wristband with Orion—our augmented reality glasses product prototype—for seamless control over digital content to swipe, click, and scroll while keeping your arm resting comfortably by your side. In the future, we imagine many other applications for sEMG, including the ability to manipulate objects in augmented reality or write full messages as quickly as—or faster than—typing on a keyboard, with very little effort.

Today, we’re releasing two datasets—emg2qwerty and emg2pose—to be included at NeurIPS 2024 in the Datasets and Benchmarks track. These datasets and their associated models demonstrate how sEMG can facilitate a richer set of inputs beyond subtle gestures. sEMG-based typing can automatically decode computer keystrokes, in the future allowing for text input to computing devices without a physical keyboard regardless of whether a user is typing on a desk, their lap, or their kitchen table. Pose estimation allows for a full prediction of a user’s hand configuration, giving them a sense of embodiment and enabling a broad range of new inputs.

Making progress towards detecting typing and hand pose from sEMG alone for novel users brings us one step closer to our long-term vision for sEMG to radically expand the range of inputs for on-the-go computing for every user, everywhere. By open sourcing datasets and models for these tasks, we hope to facilitate progress in electromyography in the research community and continue Meta’s long-running commitment to openly developing machine learning and AI models.

Although sEMG for consumer device control is a recent application of the technology, clinically-based EMG technology has been used for decades to enable prosthetic control, help diagnose and monitor neuromuscular disorders, and understand the physiology of the motor system. Unlike the often invasive EMG techniques typically used in the clinical setting, a consumer sEMG wrist wearable for device control is more likely to scale for general use, because it is non-invasive, safer, and more comfortable for all-day wear. However, one of the key reasons sEMG has struggled to break through as a consumer technology is because most decoding models that predict an individual’s gestures or movements from sEMG fail to generalize to new people.

In a recent preprint, we described an approach to building generalized sEMG interfaces that work out-of-the-box on novel users. In particular, we found that the same neural network scaling laws that dictate how large language models (LLMs) like Llama 3 improve with more training data, also apply to sEMG. With a sufficient training dataset of high-quality data from hundreds or thousands of participants, it’s possible to build generalized sEMG interfaces.

emg2qwerty and emg2pose support research into building generalized sEMG models by including large-scale datasets each containing more than 100 volunteer participants in diverse behavioral conditions and offering challenging generalization scenarios for benchmarking in realistic use cases. These datasets should promote algorithmic advancements into new strategies for generalization, similar to other large-scale datasets Meta has released in computer vision and automatic speech recognition like SA-1B, FACET and Fair-speech.

Text input is a significant challenge for wearable devices like AI glasses. The goal of sEMG-based typing is to solve the text input problem for wearables by enabling touch typing without a physical keyboard, using only the electrical signals from muscles available at the wrist. This could allow for subtle and high-bandwidth text input in any scenario, whether typing on your kitchen table at home or on your lap riding the subway.

The emg2qwerty dataset includes high-resolution sEMG signals acquired from both wrists, synchronized with accurate ground-truth keystrokes from a QWERTY keyboard. The dataset totals 346 hours of recording from 108 consenting participants across a broad range of single word and sentence typing prompts totaling over 5.2 million keystrokes.

To model the detection of keystrokes from sEMG, we developed methods inspired by the field of automatic speech recognition (ASR), which also models the task of predicting an output sequence of discrete characters given a continuous multi-channel time series. To build strong baselines for emg2qwerty, we experimented with novel network architectures, different training losses, and the use of language modeling, always focusing on the requirements of the unique domain characteristics of sEMG data.

Even at the scale of 100 users, generalization across users emerges despite variations in physiology, anatomy, behavior, band sizes, and sensor placement. Further jumps in performance come when personalizing models to users using about half an hour of typing data from individual users. Additional boosts come by incorporating a language model to refine results, bringing character error rates below 10%—a value thought to be a critical threshold for making a text model usable.

Always-available hand pose inference could yield new and intuitive control schemes for human-computer interaction. Computer vision-based hand tracking, like those in our Meta Quest 3 family of devices, are effective, but their performance could be further augmented in very dim lighting conditions and when your hands are occluded. sEMG-based hand tracking offers the ability to sense hand pose in any environment.

The emg2pose benchmark contains 370 hours of sEMG and hand pose data across 193 consenting participants, sampled from 29 different behavioral groups that include a diverse range of discrete and continuous movements like making a fist or counting to five. The hand pose labels are generated using a high-resolution motion capture array. The full dataset contains over 80 million pose labels and is of similar scale to the largest computer vision equivalents.

In the benchmark, we provide competitive baselines and challenging tasks that evaluate physical-world generalization scenarios across held-out users, sensor placements, and hand poses. We also introduce a new state-of-the-art model for pose estimation from sEMG, vemg2pose, that reconstructs hand pose by integrating predictions of pose velocity. We include vemg2pose, along with two other contemporary baselines for pose estimation from sEMG and analyze their performance across generalization conditions. The vemg2pose model shows only 1 cm error on held-out users, resulting in high-fidelity tracking across a broad range of movements.