NLP

Pretrained Language Models for Biomedical and Clinical Tasks: Understanding and Extending the State-of-the-Art

November 03, 2020

Abstract

A large array of pretrained models are available to the biomedical NLP (BioNLP) community. Finding the best model for a particular task can be difficult and time-consuming. For many applications in the biomedical and clinical domains, it is crucial that models can be built quickly and are highly accurate. We present a large-scale study across 18 established biomedical and clinical NLP tasks to determine which of several popular open-source biomedical and clinical NLP models work well in different settings. Furthermore, we apply recent advances in pretraining to train new biomedical language models, and carefully investigate the effect of various design choices on downstream performance. Our best models perform well in all of our benchmarks, and set new State-of-the-Art in 9 tasks. We release these models in the hope that they can help the community to speed up and increase the accuracy of BioNLP and text mining applications.

Download the Paper

AUTHORS

Written by

Ves Stoyanov

Jingfei Du

Myle Ott

Patrick Lewis

Publisher

ClinicalNLP2020@EMNLP2020

Related Publications

November 13, 2026

NLP

COMPUTER VISION

Unbundling Multimodal Vision Encoding

Ellis Brown, Chris Hoang, Shang-Wen Li, Mike Lewis, Marjan Ghazvininejad, Luke Zettlemoyer, Lucas Beyer, Mengye Ren, Jakob Verbeek, Saining Xie, Hu Xu

November 13, 2026

October 20, 2026

REINFORCEMENT LEARNING

NLP

Learning to Program Retrievals: Composing Search Primitives via Reinforcement Learning

Kyuyoung Kim, Zilin Xiao, Zhiyuan Wang, Avinash Atreya, Ke Yang, Jinwoo Shin, Kecheng Hao

October 20, 2026

October 19, 2026

RESEARCH

NLP

Muse Glimmer 30B: An On-Device Agentic Model

Muse Glimmer team

October 19, 2026

September 24, 2026

REINFORCEMENT LEARNING

RESEARCH

MaD-RL: Matching Distributions for Calibrating LLMs with Reinforcement Learning

Sourabh Kulkarni, Ksheeraj Sai Vepuri, Basar Demir, Jason Bohrer, Emily Shen, Jianfa Chen, Nan Jiang, Ankit Jain, Harihar Subramanyam, Mannat Singh, Chirag Nagpal

September 24, 2026

Help Us Pioneer The Future of AI

We share our open source frameworks, tools, libraries, and models for everything from research exploration to large-scale production deployment.