#language-model

A Review on Language Models as Knowledge Bases

Contents of GPT-3 & the Pile v1

This is a paradigm shift because AI models in the past were often thought to perform one task at a time. While techniques in task-transfer and multi-task AI models were researched, one model was not expected to perform tasks that are very different from one another, even with transfer learning. This also meant that research in different tasks and modalities, like vision and language, were often done with different technical approaches — vision researchers studied topics like signal processing, while language researchers studied language grammar. Foundation models with deep learning leveraged incredible scaling to train one model that can perform many tasks.

Language model sizes to Apr. 2022 or While you were sleeping, AI sizes were exploding