CONVERSATIONAL AI

NLP

Llama 2: Open Foundation and Fine-Tuned Chat Models

July 18, 2023

Abstract

In this work, we develop and release Llama 2, a collection of pretrained and fine-tuned large language models (LLMs) ranging in scale from 7 billion to 70 billion parameters. Our fine-tuned LLMs, called Llama 2-Chat, are optimized for dialogue use cases. Our models outperform open-source chat models on most benchmarks we tested, and based on our human evaluations for helpfulness and safety, may be a suitable substitute for closedsource models. We provide a detailed description of our approach to fine-tuning and safety improvements of Llama 2-Chat in order to enable the community to build on our work and contribute to the responsible development of LLMs.

Download the Paper

AUTHORS

Written by

Hugo Touvron

Louis Martin

Kevin Stone

Peter Albert

Amjad Almahairi

Yasmine Babaei

Nikolay Bashlykov

Soumya Batra

Praj Bhargava

Shruti Bhosale

Dan Bikel

Lukas Blecher

Cristian Canton Ferrer

Moya Chen

Guillem Cucurull

David Esiobu

Jude Fernandes

Jeremy Fu

Wenyin Fu

Brian Fuller

Cynthia Gao

Vedanuj Goswami

Naman Goyal

Anthony Hartshorn

Saghar Hosseini

Rui Hou

Hakan Inan

Marcin Kardas

Viktor Kerkez

Madian Khabsa

Isabel Kloumann

Artem Korenev

Punit Singh Koura

Marie-Anne Lachaux

Thibaut Lavril

Jenya Lee

Diana Liskovich

Yinghai Lu

Yuning Mao

Xavier Martinet

Todor Mihaylov

Pushkar Mishra

Igor Molybog

Yixin Nie

Andrew Poulton

Jeremy Reizenstein

Rashi Rungta

Kalyan Saladi

Alan Schelten

Ruan Silva

Eric Michael Smith

Ranjan Subramanian

Xiaoqing Ellen Tan

Binh Tang

Ross Taylor

Adina Williams

Andrew Kuan

Puxin Xu

Zheng Yan

Iliyan Zarov

Yuchen Zhang

Angela Fan

Melanie Kambadur

Sharan Narang

Aurelien Rodriguez

Robert Stojnic

Sergey Edunov

Thomas Scialom

Publisher

arxiv

Related Publications

June 25, 2024

NLP

Neurons in Large Language Models: Dead, N-gram, Positional

Elena Voita, Javier Ferrando Monsonis, Christoforos Nalmpantis

June 25, 2024

June 25, 2024

SPEECH & AUDIO

NLP

Textless Acoustic Model with Self-Supervised Distillation for Noise-Robust Expressive Speech-to-Speech Translation

Min-Jae Hwang, Ilia Kulikov, Benjamin Peloquin, Hongyu Gong, Peng-Jen Chen, Ann Lee

June 25, 2024

June 14, 2024

NLP

How to Train Your DRAGON: Diverse Augmentation Towards Generalizable Dense Retrieval

Sheng-Chieh Lin, Akari Asai, Minghan Li, Barlas Oguz, Jimmy Lin, Scott Yih, Xilun Chen

June 14, 2024

June 14, 2024

NLP

SYSTEMS RESEARCH

LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding

Mostafa Elhoushi, Akshat Shrivastava, Diana Liskovich, Basil Hosmer, Bram Wasti, Liangzhen Lai, Nas Mahmoud, Bilge Acun, Saurabh Agarwal, Ahmed Roman, Ahmed Aly, Beidi Chen, Carole-Jean Wu

June 14, 2024

Help Us Pioneer The Future of AI

We share our open source frameworks, tools, libraries, and models for everything from research exploration to large-scale production deployment.