In their AI-first, open collaboration platform, Zoom uses a unique federated approach to power Zoom AI Companion, the company’s generative AI assistant. Available across Zoom Workplace and Business Services, AI Companion helps workers avoid repetitive tasks. The idea is that a generative AI assistant can take care of the mundane tasks, while enabling people to spend more time making connections, collaborating, and being productive with their teams.

With Zoom’s federated approach, AI Companion uses Zoom’s models and closed-source and open-source large language models—including Llama—to serve up meeting summaries, smart recordings, and next steps to Zoom users at no added cost to their eligible paid plans.

“Foundational models allow our teams to focus on specific use cases and customer needs rather than building a model from scratch, speeding up our time to market for Zoom AI Companion capabilities,” says Zoom CTO Xuedong Huang.

Typically, using a high-quality, open source foundation model like Llama might require thousands of GPU infrastructures to train independently, Huang says. When building AI Companion, the team addressed this challenge by creating a high-quality internal dataset to drive a fine-tuned Llama model in orchestration with a number of closed-source models. This led to a higher quality performance capable of outperforming much larger closed-source models at a lower cost for AI Companion workloads.

The first area of business the team explored was using Llama to build AI Companion’s meeting summary capabilities. Since launching in September 2023, more than 700,000 accounts have enabled AI Companion, and adoption continues to increase—with the number of meeting summaries doubling quarter over quarter. Llama has also proven useful for generating training examples to improve the fine-tuning of other models in Zoom’s portfolio.

While building for AI comes with a number of challenges, Zoom’s federated approach maximizes performance, quality, and affordability according to Huang.

Customer security and privacy are among the most significant considerations of AI adoption. By building on a foundational model, Zoom has given its customers capabilities that are built on an already trusted ecosystem, Huang adds. By including high-quality foundational models like Llama in their federated approach, Zoom can allow customers to keep their data within Zoom’s secured organization server through the use of the Zoom-hosted Models Only program (ZMO), which lets the company serve customers who don’t want to share their information with third-party hosted models.

Inference-time latency is another challenge when working with LLMs. While AI Companion meeting summaries don’t need to be delivered in real time, the team invested in additional AI infrastructure resources to reach its goals as volume scaled, and it added other real-time use cases, like AI Companion Team Chat compose capabilities.

Working with open source models yields many benefits, but perhaps most important of all, Huang says the spirit of open source creates a more level playing field across the industry. In fact, having this access is crucial for other companies—regardless of size—as they explore across disciplines and discover new breakthroughs.

While hosting costs can be expensive, organizations don’t need to maintain massive science teams to access cutting-edge models. Instead, companies can tap into the value of open source by doing what they do best: specializing in their customers’ needs and fine-tuning foundational models to meet their expectations.