2018Model

GPT-1: Generative Pre-Training

OpenAI releases GPT-1, first demonstrating the potential of generative pre-training on language understanding tasks.

#nlp #gpt #transformer #pre-training #openai

Overview

In June 2018, OpenAI released GPT-1 (Generative Pre-Training). Alec Radford and colleagues proposed the "generative pre-training" method: first pre-training a language model on large amounts of unlabeled text, then fine-tuning on specific tasks. GPT-1 contained 12 layers of Transformer decoders with 117 million parameters. Although relatively small in scale, it proved that pre-trained language models could effectively transfer to various downstream tasks, setting the direction for GPT series development. GPT-1's success inspired later large-scale language model research.

Study Notes

Background, mechanics, impact, and key takeaways in one read

About 2 min read

Background & Context

Before 2018, most NLP systems were trained from scratch on separately labeled data for each task—costly to annotate and hard to transfer. Computer vision had already shown the value of pretraining then fine-tuning, but NLP lacked a stable, effective pretraining objective. At OpenAI, Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever published Improving Language Understanding by Generative Pre-Training (GPT-1) in June 2018, proposing generative pretraining with a Transformer decoder: first predict the next word on large unlabeled text, then fine-tune on downstream tasks. This set the paradigm and architecture for the GPT lineage that followed. Although modest at 117M parameters, GPT-1 proved that a single generative pretraining stage yields broadly useful representations and reshaped how the field approached language understanding.

Key Points

1It pioneered the two-stage recipe of generative pretraining plus supervised fine-tuning that defines the whole GPT family.
2A decoder-only Transformer with causal self-attention suits text generation naturally.
3On BooksCorpus it learns general representations by predicting the next word.
4Fine-tuning adds only a linear head yet improved state of the art on 9 of 12 benchmarks.
5It uses learned positional embeddings and BPE tokenization, departing from the original Transformer.
6Modest in size, it proved the paradigm shift that pretraining is a general-purpose initialization.

How It Works

GPT-1 is a 12-layer, ~117M-parameter decoder-only Transformer with hidden size 768, 12 attention heads, and a 512-token context. Pretraining uses a standard autoregressive objective: given the past, maximize the log-likelihood of the next word, trained on BooksCorpus (about 7,000 unpublished books, ~800M words). Fine-tuning attaches a linear classification head on top and trains the whole model supervisedly on labeled data, keeping an auxiliary language-modeling loss as a regularizer. Unlike the original Transformer, GPT-1 uses learned positional embeddings rather than sinusoidal ones and byte-pair encoding (BPE) for tokenization. Its point was not scale but proving that generative pretraining is a reusable, general initialization.

Impact & Legacy

Along with BERT, ELMo, and ULMFiT in 2018, GPT-1 opened the NLP pretraining era, showing unlabeled text yields transferable general representations. Its autoregressive-decoder route split from BERT's bidirectional encoder and evolved into GPT-2/3/4 and ChatGPT. Its limits: unidirectional context lags BERT on discriminative tasks, and 117M parameters look tiny today. But its methodological legacy is profound.

Key Takeaways

Unlabeled text plus an autoregressive objective can yield transferable, general language ability.
The decoder architecture fits generation and set GPT's future direction.
A paradigm shift (pretrain then fine-tune) outvalues simply scaling parameters.
Competing 2018 approaches were ultimately won by scale and generative capability.

Glossary

Generative Pre-Training生成式预训练

Pretraining a language model on unlabeled text by predicting upcoming text, then transferring to downstream tasks.

Decoder-Only Transformer仅解码器 Transformer

An architecture keeping only the decoder stack with causal masking, generating text token by token.

Byte-Pair Encoding (BPE)字节对编码

A subword tokenization algorithm that merges frequent byte/character pairs into tokens.

Related Events

2018