CONVERSATIONAL AI
GRAPHICS
Generating Illustrated Instructions
April 23, 2024
Abstract
We introduce the new task of generating Illustrated Instructions, i.e., visual instructions customized to a user's needs. We identify desiderata unique to this task, and formalize it through a suite of automatic and human evaluation metrics, designed to measure the validity, consistency, and efficacy of the generations. We combine the power of large language models (LLMs) together with strong text-to-image generation diffusion models to propose a simple approach called StackedDiffusion, which generates such illustrated instructions given text as input. The resulting model strongly outperforms baseline approaches and state-of-the-art multimodal LLMs; and in 30% of cases, users even prefer it to human-generated articles. Most notably, it enables various new and exciting applications far beyond what static articles on the web can provide, such as personalized instructions complete with intermediate steps and pictures in response to a user's individual situation.
AUTHORS
Publisher
CVPR
Related Publications
February 26, 2026
CONVERSATIONAL AI
RESEARCH
Learning Personalized Agents from Human Feedback
Kaiqu Liang, Xianjun Yang, Shaoliang Nie, Jaime Fernández Fisac, Shuyan Zhou, Julia Kruk, Lijuan Liu, Michael Zhang, Saghar Hosseini, Shengjie Bi, Shengyi Qian
February 26, 2026
December 01, 2025
CONVERSATIONAL AI
REINFORCEMENT LEARNING
Rubric-Based Benchmarking and Reinforcement Learning for Advancing LLM Instruction Following
Amine Benhalloum, Hany Awadalla, Hejia Zhang, Hunter Lang, Julian Katz-Samuels, Karishma Mandyam, Licheng Yu, Manaal Faruqui, Maryam Fazel-Zarandi, Nanshu Wang, Qi Qi, Richard Yuanzhe Pang, Selina Xiaoliang Peng, Shengjie Bi, Shengyu Feng, Shishir G. Patil, Sopan Khosla, Sujan Gonugondla, Vincent Li, Wenzhe Li, Yuanhao Xiong, Yue Yu, Yun He, Yundi Qian
December 01, 2025
September 24, 2025
CONVERSATIONAL AI
REINFORCEMENT LEARNING
Compute as Teacher: Turning Inference Compute Into Reference-Free Supervision
Dulhan Jayalath, Suchin Gururangan, Cheng Zhang, Alan Schelten, Anirudh Goyal, Parag Jain, Shashwat Goel, Thomas Simon Foster
September 24, 2025
June 27, 2025
HUMAN & MACHINE INTELLIGENCE
CONVERSATIONAL AI
Seamless Interaction: Dyadic Audiovisual Motion Modeling and Large-Scale Dataset
Morteza Behrooz, Ning Dong, Jeff Girard, Vasu Sharma, Jan Zikes, Akinniyi Akinyemi, Alex Shcherbyna, Alexander Richard, Alice Rakotoarison, Amia Oberai, Anastasis Stathopoulos, Anna Sun, Antony D'Avirro, Arina Turkatenko, Benjamin Peloquin, Bo Wan, Brandon Han, Carleigh Wood, Chao Wang, Chen Zhang, Christophe Ropers, Christopher Klaiber, Cynthia Gao, Dejan Kovachev, Denise Hernandez, Evonne Ng, Fabian Prada, Fabio Maria Carlucci, Guangyao Ma, Hang Li, Hirofumi Inaguma, Hongyu Gong, Jason Zheng, Jeff Wang, Jie Shen, Jiemin Zhang, Jing Ma, Joe Chuang, Jon Daly, Jovan Popovic, Joy Chen, Juan Pino, Julia Buffalini, Zhiyuan Yao, Junming Chen, Kam-Woh Ng, Kathryn Alvero, Louis-Philippe Morency, Lucas Mantovani, Mark Duppenthaler, Martin Gleize, Martin Ma, Mary Williamson, Michael Zollhoefer, Moneish Kumar, Omid Poursaeed, Paden Tomasello, Pavel Litvin, Pavlo Zhyzheria, Praveen Chowdary, Qingyao Jia, Raj Janardhan, Rongjie Huang, Safiyyah Saleem, Sagar Miglani, Sahir Gomez, Sen He, Shiyang Cheng, Somya Jain, Sreyas Mohan, Srivathsan Govindarajan, Tao Xiang, Tu Anh Nguyen, Tuan Tran, Vasu Agrawal, Wei Liu, Xinyue Zhang, Xutai Ma, Yilei Li, Yilin Yang, Yordan Hristov, Zhang Chen
June 27, 2025
Help Us Pioneer The Future of AI
We share our open source frameworks, tools, libraries, and models for everything from research exploration to large-scale production deployment.
Our approach
Latest news
Foundational models
Meta © 2026