COMPUTER VISION

Emu: Enhancing Image Generation Models Using Photogenic Needles in a Haystack

September 27, 2023

Abstract

Training text-to-image models with web scale image-text pairs enables the generation of a wide range of visual concepts from text. However, these pre-trained models often face challenges when it comes to generating highly aesthetic images. This creates the need for aesthetic alignment post pre-training. In this paper, we propose quality-tuning to effectively guide a pre-trained model to exclusively generate highly visually appealing images, while maintaining generality across visual concepts. Our key insight is that supervised fine-tuning with a set of surprisingly small but extremely visually appealing images can significantly improve the generation quality. We pre-train a latent diffusion model on 1.1 billion image-text pairs and fine-tune it with only a few thousand carefully selected high-quality images. The resulting model, Emu, achieves a win rate of 82.9% compared with its pre-trained only counterpart. Compared to the state-of-the-art SDXLv1.0, Emu is preferred 68.4% and 71.3% of the time on visual appeal on the standard PartiPrompts and our Open User Input benchmark based on the real-world usage of text-to-image models. In addition, we show that quality-tuning is a generic approach that is also effective for other architectures, including pixel diffusion and masked generative transformer models.

Download the Paper

AUTHORS

Written by

Xiaoliang Dai

Ji Hou

Kevin Chih-Yao Ma

Sam Tsai

Jialiang Wang

Rui Wang

Peizhao Zhang

Simon Vandenhende

Xiaofang Wang

Abhimanyu Dubey

Matthew Yu

Abhishek Kadian

Filip Radenovic

Dhruv Mahajan

Kunpeng Li

Yue (R) Zhao

Vladan Petrovic

Mitesh Kumar Singh

Simran Motwani

Yiwen Song

Yi Wen

Roshan Sumbaly

Vignesh Ramanathan

Zijian He

Peter Vajda

Devi Parikh

Publisher

Meta

Research Topics

Computer Vision

Related Publications

February 13, 2024

GRAPHICS

COMPUTER VISION

IM-3D: Iterative Multiview Diffusion and Reconstruction for High-Quality 3D Generation

Luke Melas-Kyriazi, Iro Laina, Christian Rupprecht, Natalia Neverova, Andrea Vedaldi, Oran Gafni, Filippos Kokkinos

February 13, 2024

January 25, 2024

COMPUTER VISION

LRR: Language-Driven Resamplable Continuous Representation against Adversarial Tracking Attacks

Felix Xu, Di Lin, Jianjun Zhao, Jianlang Chen, Lei Ma, Qing Guo, Wei Feng, Xuhong Ren

January 25, 2024

November 10, 2023

COMPUTER VISION

EgoDistill: Egocentric Head Motion Distillation for Efficient Video Understanding

Shuhan Tan, Tushar Nagarajan, Kristen Grauman

November 10, 2023

October 29, 2023

COMPUTER VISION

ALA: Naturalness-aware Adversarial Lightness Attack

Felix Xu, Geguang Pu, Jiayi Zhu, Jincao Feng, Liangru Sun, Qing Guo, Yang Liu, Yihao Huang

October 29, 2023

Help Us Pioneer The Future of AI

We share our open source frameworks, tools, libraries, and models for everything from research exploration to large-scale production deployment.