SYSTEMS RESEARCH

MODeL: Memory Optimizations for Deep Learning

June 19, 2023

Abstract

The size of deep neural networks has grown exponentially in recent years. Unfortunately, hardware devices have not kept pace with the rapidly increasing memory requirements. To cope with this, researchers have proposed various techniques including spilling, rematerialization, reduced precision training, model pruning, and so on. However, these approaches suffer from various limitations, such as increasing training time, affecting model accuracy, or requiring extensive manual modifications to the neural networks. We present MODeL, an algorithm that optimizes the lifetime and memory location of the tensors used to train neural networks. Our method automatically reduces the memory usage of existing neural networks without any of the drawbacks of other techniques. We formulate the problem as a joint integer linear program (ILP). We present several techniques to simplify the encoding of the problem, and enable our approach to scale to the size of state-of-the-art neural networks using an off-the-shelf ILP solver. We experimentally demonstrate that MODeL only takes seconds to allow the training of neural networks using 30% less memory on average. MODeL is an open-source project available at https://github.com/facebookresearch/model_opt.

Download the Paper

AUTHORS

Written by

Benoit Steiner

Mostafa Elhoushi

Jacob Kahn

James Hegarty

Publisher

ICML

Research Topics

Systems Research

Related Publications

November 07, 2023

NLP

COMPUTER VISION

The Framework Tax: Disparities Between Inference Efficiency in NLP Research and Deployment

Jared Fernandez, Jacob Kahn, Clara Na, Yonatan Bisk, Emma Strubell

November 07, 2023

August 21, 2023

SYSTEMS RESEARCH

GraphAGILE: An FPGA-Based Overlay Accelerator for Low-Latency GNN Inference

Bingyi Zhang, Hanqing Zeng, Viktor Prasanna

August 21, 2023

April 26, 2023

CORE MACHINE LEARNING

SYSTEMS RESEARCH

Green Federated Learning

Ashkan Yousefpour, Shen Guo, Ashish Shenoy, Sayan Ghosh, Pierre Stock, Kiwan Maeng, Schalk Krüger, Mike Rabbat, Carole-Jean Wu, Ilya Mironov

April 26, 2023

October 14, 2022

CORE MACHINE LEARNING

SYSTEMS RESEARCH

Latent Execution for Neural Program Synthesis

Yuandong Tian, Dawn Song, Xinyun Chen

October 14, 2022

Help Us Pioneer The Future of AI

We share our open source frameworks, tools, libraries, and models for everything from research exploration to large-scale production deployment.