Research
SGD Implicitly Regularizes Generalization Error
December 8, 2018
Abstract
We derive a simple and model-independent formula for the change in the generalization gap due to a gradient descent update. We then compare the change in the test error for stochastic gradient descent to the change in test error from an equivalent number of gradient descent updates and show explicitly that stochastic gradient descent acts to regularize generalization error by decorrelating nearby updates. These calculations depends on the details of the model only through the mean and covariance of the gradient distribution, which may be readily measured for particular models of interest. We discuss further improvements to these calculations and comment on possible implications for stochastic optimization.
AUTHORS
Written by
Related Publications
July 17, 2026
Conversational AI
Reinforcement Learni9ng
Learning to Reason by Analogy via Retrieval-Augmented Reinforcement Fine-Tuning
Zilin Xiao, Qi Ma, Jason Chen, Xintao Chen, Avinash Atreya, Hanjie Chen, Vicente Ordonez
July 17, 2026
July 13, 2026
S-EMBER: A Large-Scale Benchmark for Streaming Egocentric Memory Retrieval
Xiaodong Wang, Xuanyi Zhao, Pedro Rodriguez, Devendra Singh Sachan, Barlas Oguz, Seungwhan Moon, Shang-Wen Li, Gargi Ghosh, Xin Dong, Wen-Tau Yih
July 13, 2026
July 03, 2026
Human & Machine Intelligence
Robotics
Interpreting Physics in Video World Models
Sonia Joseph, Quentin Garrido, Randall Balestriero, Matthew Kowal, Thomas Fel, Shahab Bakhtiari, Blake Richards, Mike Rabbat
July 03, 2026
June 05, 2026
Conversational AI
Ranking & Recommendations
Superintelligent Retrieval Agent: The Next Frontier of Agentic Retrieval
Zeyu Yang, Qi Ma, Jason Chen, Anshumali Shrivastava
June 05, 2026
October 31, 2019
NLP
Facebook AI's WAT19 Myanmar-English Translation Task Submission
Peng-Jen Chen, Jiajun Shen, Matt Le, Vishrav Chaudhary, Ahmed El-Kishky, Guillaume Wenzek, Myle Ott, Marc’Aurelio Ranzato
October 31, 2019
October 27, 2019
Order-Aware Generative Modeling Using the 3D-Craft Dataset | Facebook AI Research
Zhuoyuan Chen, Demi Guo, Tong Xiao, Saining Xie, Xinlei Chen, Haonan Yu, Jonathan Gray, Kavya Srinet, Haoqi Fan, Jerry Ma, Charles R. Qi, Shubham Tulsiani, Arthur Szlam, Larry Zitnick
October 27, 2019
April 25, 2020
Energy-Based Models for Atomic-Resolution Protein Conformations | Facebook AI Research
Yilun Du, Joshua Meier, Jerry Ma, Rob Fergus, Alexander Rives
April 25, 2020
June 11, 2019
Computer Vision
ELF OpenGo: An Analysis and Open Reimplementation of AlphaZero | Facebook AI Research
Yuandong Tian, Jerry Ma, Qucheng Gong, Shubho Sengupta, Zhuoyuan Chen, James Pinkerton, Larry Zitnick
June 11, 2019
Help Us Pioneer The Future of AI
We share our open source frameworks, tools, libraries, and models for everything from research exploration to large-scale production deployment.
Meta © 2026
