Processing video for video on demand (VOD) and live streaming is already compute intensive. It involves encoding large video files into more manageable formats and transcoding them to deliver to audiences. Today, demand for video content is greater than ever. And emerging use cases, such as generative AI content, mean the demands on video infrastructure are only going to intensify.

GPUs and even general-purpose CPUs are capable of handling video processing. But at the scale that Meta operates (serving billions of people all over the world), and with an eye on future AI-related use cases, we believe that dedicated hardware is the best solution in terms of compute power and efficiency.

Meta Scalable Video Processor (MSVP) is Meta’s first in-house-developed ASIC solution — designed for the processing needs of the ever-growing VOD and live streaming workloads at Meta. MSVP is programmable and scalable, and can be configured to efficiently support both the high-quality transcoding needed for VOD as well as the low latency and faster processing times that live streaming requires.

In the future, MSVP will also help bring new forms of video content to every member of Meta’s family of apps — including AI-generated content as well as AR and VR content in the metaverse.

Every generation of video coding standards brings in about an additional 50 percent compression efficiency, which can be used to serve high-quality video with lower bit rates, but this compression efficiency comes with a 10x computational cost. At Meta’s scale, we need a video encoding solution that can deliver the best quality video possible, with the shortest amount of encoding time — all while being energy efficient, programmable, and scalable.

This is where hardware acceleration thrives. ASICs are easy to scale and offer very high energy efficiency.

With MSVP in place for hardware acceleration, we can:

• Improve the overall quality and compression efficiency of our video encodings across all our codecs
• Offer high stability, reliability, and near-infinite scaling
• Reduce the compute and storage footprint of our video encodings
• Reduce bits at the same quality or increase quality at the same bit rate

The MSVP encoder has two main goals: to be highly power efficient and to deliver the same or better video quality as software encoders. There are existing video encoder IPs, but most of them are targeted at mobile devices with tight area/power constraints and cannot meet the quality bar set by current software encoders.

Because software encoders offer very flexible control and fast evolution over time, it is quite challenging for ASIC video encoders to meet the same performance bar as software encoders.

Here’s a simplified version of the data flow of modern hybrid (hardware and software) video encoders:

These encoders use intra-coding to reduce spatial redundancy and inter-coding to remove temporal redundancy. Different stages of motion estimation are applied in inter-coding to find out the best prediction among all possible block positions in available reference frames. Entropy coding is the lossless compression part that squeezes the statistical redundancy of all syntax elements, including encoding modes, motion vectors and quantized residual coefficients.

For MSVP’s algorithms to perform the way we wanted, we had to find hardware-friendly alternatives for each of the above key modules. We mainly focused on three levels: block level, frame level, and group of picture (GOP) level.

At the block level, we looked for coding tools with the highest return on investment, that were easy/economical (in terms of silicon area and power requirements) to implement in hardware, and that met our performance targets while maximizing compression efficiency. At frame level, we studied the best algorithms to make intelligent frame type decisions among I/P/B frames, and the best rate-control algorithms based on statistics collected from hardware. And at the GOP level, we had to figure out whether to use multiple-pass encoding with look-ahead, or to insert intra (key) frames at a given shot boundary.

Motion estimation is one of the most computationally intensive algorithms in video encoding. To find accurate motion vectors that closely match the block currently being encoded, a full motion estimation pipeline often includes a multistage search to balance among large search range, computing complexity, and accuracy.

MSVP’s motion search algorithm needs to be one that identifies which potential neighboring blocks can contribute more to quality and only searches around highly correlated neighbors with a limited cycle budget. Although we lack the flexibility of iterative software motion search algorithms, such as diamond or hexagon shapes, hardware motion estimation can search multiple blocks in parallel. Thus, it allows us to search more candidates, cover a larger search range and more reference frames in both single direction and bidirectional mode, and search all supported block partition shapes in parallel.

Achieving high video encoding quality also requires RDO support. Since there are so many decisions to make in video encoding (intra/inter modes, partition block size, transform block types/sizes, etc.), RDO is one of the best practices in video compression to determine which mode is optimal given the current rate or quality target.

MSVP supports exhaustive RDO at almost all mode decision stages. Distortion calculation is intensive but both straightforward and easily parallelizable. But the unique challenge is the bit rate estimation. Entropy coding for the final bitstream is sequential in nature, and each context model is dependent on the previously encoded ones. In a hardware encoder implementation, rate distortion (RD) cost for different blocks/partitions might be evaluated in parallel; thus, it is impossible to have very accurate bit rate estimation. We implemented a pretty accurate bit rate estimation model in MSVP. The model is hardware friendly, in that it is easy to evaluate multiple coding modes in parallel.

Quantization is the only lossy part of video compression, and it is also the dominant bit rate control knob in any video coding standard. The corresponding parameter is called the quantization parameter (QP), and it is inversely related to quality: Low QP values result in small quantization errors, creating low distortion levels and, subsequently, high quality at the expense of higher bit rates. By making smart quantization choices, encoding bits can be allocated to areas that impact visual quality the most. We perform smart quantization using optimal QP selection and rounding decisions.

Modern video coding standards allow different QP values to be applied to different coding units. In MSVP’s hardware encoder, block-level QP values are determined adaptively based on both spatial and temporal characteristics.

In spatial adaptive QP (AQP) selection, since the human visual system is less sensitive to quality loss at high texture or high motion areas, a larger QP value can be applied to these coding blocks. In temporal AQP, coding blocks that are referenced more in the future can be quantized with a lower QP to get higher quality, such that future coding blocks that reference these blocks will benefit from it.

Smart rounding tries to make a joint optimization on rounding decisions for all coefficients in each coding block. Since the choices of rounding at different coefficient positions are dependent on one another, we need better algorithms that remove the dependency while maintaining the rounding decision accuracy. To reduce compute cost, we’ve applied smart rounding to the final stage after the coding mode for each block is determined. This feature alone can achieve a ~1 percent to 2 percent BD-Rate improvement.

Most of the overall watch time on Meta’s apps is generated by a relatively small percentage of the videos uploaded. Therefore, we have to use different encoding configurations to process videos based on their popularity and optimize the compression and compute efficiency at scale.

At the system level, there is a benefit-cost model that predicts the watch time for every uploaded video and then controls what encoding configuration will be triggered based on that.

Once a video is uploaded to one of our apps, it’s processed by the back-end production and delivery system. At Meta, we decouple the production and delivery processes. On the production side, we have multiple ABR encoding families, with different codecs and configurations. The goal of the basic ABR family is to quickly encode videos with good quality so people can share them.

Publishing latency is the key metric we need to optimize for. Once the video gets sufficiently high watch time, a full ABR encoding is triggered, which also produces a fixed ABR ladder using H.264 and VP9 high-quality presets to further improve video quality. Once the video gets even more watch time, an advanced ABR encoding is triggered, which produces a dynamic ABR ladder.

This is a very computationally complex process. But with MSVP’s support, we can significantly reduce the compute complexity of the advanced ABR encoding family so that we can use it to replace the full ABR encoding family and enable advanced ABR encoding to more videos.

MSVP is just the first milestone in our effort to develop new hardware-assisted video processing solutions. MSVP opens up a lot of opportunities for us, and we are at a very early stage.

Given the superior video quality MSVP offers over its software counterparts (for both H.264 and VP9 encoding), we are also offloading basic ABR encoding on MSVP to reduce publishing latency and improve quality.

Our plan is to eventually offload the majority of our stable and mature video processing workloads to MSVP and use software only for workloads that require specific customization and significantly higher quality.

We’re also continuing to work on further improving video quality with MSVP using preprocessing methods such as smart denoising and image enhancement, as well as post-processing methods such as artifact removal and super-resolution. In the future, MSVP will allow us to support even more of Meta’s most important use cases and needs, including short-form videos, enabling efficient delivery of generative AI, AR/VR, and other metaverse content.