CONVERSATIONAL AI

NLP

Llama 2: Open Foundation and Fine-Tuned Chat Models

July 18, 2023

Abstract

In this work, we develop and release Llama 2, a collection of pretrained and fine-tuned large language models (LLMs) ranging in scale from 7 billion to 70 billion parameters. Our fine-tuned LLMs, called Llama 2-Chat, are optimized for dialogue use cases. Our models outperform open-source chat models on most benchmarks we tested, and based on our human evaluations for helpfulness and safety, may be a suitable substitute for closedsource models. We provide a detailed description of our approach to fine-tuning and safety improvements of Llama 2-Chat in order to enable the community to build on our work and contribute to the responsible development of LLMs.

Download the Paper

AUTHORS

Written by

Hugo Touvron

Louis Martin

Kevin Stone

Peter Albert

Amjad Almahairi

Yasmine Babaei

Nikolay Bashlykov

Soumya Batra

Praj Bhargava

Shruti Bhosale

Dan Bikel

Lukas Blecher

Cristian Canton Ferrer

Moya Chen

Guillem Cucurull

David Esiobu

Jude Fernandes

Jeremy Fu

Wenyin Fu

Brian Fuller

Cynthia Gao

Vedanuj Goswami

Naman Goyal

Anthony Hartshorn

Saghar Hosseini

Rui Hou

Hakan Inan

Marcin Kardas

Viktor Kerkez

Madian Khabsa

Isabel Kloumann

Artem Korenev

Punit Singh Koura

Marie-Anne Lachaux

Thibaut Lavril

Jenya Lee

Diana Liskovich

Yinghai Lu

Yuning Mao

Xavier Martinet

Todor Mihaylov

Pushkar Mishra

Igor Molybog

Yixin Nie

Andrew Poulton

Jeremy Reizenstein

Rashi Rungta

Kalyan Saladi

Alan Schelten

Ruan Silva

Eric Michael Smith

Ranjan Subramanian

Xiaoqing Ellen Tan

Binh Tang

Ross Taylor

Adina Williams

Andrew Kuan

Puxin Xu

Zheng Yan

Iliyan Zarov

Yuchen Zhang

Angela Fan

Melanie Kambadur

Sharan Narang

Aurelien Rodriguez

Robert Stojnic

Sergey Edunov

Thomas Scialom

Publisher

arxiv

Related Publications

May 24, 2024

SPEECH & AUDIO

NLP

DOC-RAG: ASR Language Model Personalization with Domain-Distributed Co-occurrence Retrieval Augmentation

Zhe Liu

May 24, 2024

May 06, 2024

CONVERSATIONAL AI

NLP

GAIA: a benchmark for general AI assistants

Gregoire Mialon, Yann LeCun, Thomas Scialom, Clémentine Fourrier, Thomas Wolf

May 06, 2024

April 23, 2024

CONVERSATIONAL AI

GRAPHICS

Generating Illustrated Instructions

Sachit Menon, Ishan Misra, Rohit Girdhar

April 23, 2024

April 22, 2024

NLP

Text Quality-Based Pruning for Efficient Training of Language Models

Vasu Sharma *, Karthik Padthe *, Newsha Ardalani, Kushal Tirumala, Russ Howes, Hu Xu, Bernie Huang, Daniel Li (FAIR), Armen Aghajanyan, Gargi Ghosh, Luke Zettlemoyer

April 22, 2024

Help Us Pioneer The Future of AI

We share our open source frameworks, tools, libraries, and models for everything from research exploration to large-scale production deployment.