2024Model

DeepSeek V2: Chinese Open-Source LLM

DeepSeek releases V2 model, achieving high performance at extremely low cost, leading new direction in open-source models.

#llm #deepseek #open-source #moe #chinese-ai

Overview

In May 2024, DeepSeek released DeepSeek-V2 model. This is a 236B parameter mixture-of-experts (MoE) model, activating only 21B parameters per inference, achieving good balance between performance and efficiency. DeepSeek-V2 matched GPT-4 Turbo on multiple benchmarks with far lower training and inference costs than similar models. The subsequent DeepSeek-V3 further improved performance. DeepSeek's success demonstrated advances in Chinese AI research, driving efficiency innovation in open-source LLMs.

Study Notes

Background, mechanics, impact, and key takeaways in one read

About 3 min read

Background & Context

In May 2024, DeepSeek, backed by the quant fund High-Flyer, released DeepSeek-V2: a mixture-of-experts model with 236B total parameters but only about 21B activated per inference. Its killer feature was not absolute capability, roughly comparable to GPT-4 Turbo, but the cost structure: MLA (multi-head latent attention) compresses the KV cache to a fraction of standard attention, and the MoE architecture cuts per-token compute by an order of magnitude, together driving inference cost to a small fraction of peers, with API pricing at times only a few percent of GPT-4 Turbo. DeepSeek thereby pushed China's AI efficiency route onto the world stage: not competing on trillion-parameter scale and ten-thousand-GPU clusters, but squeezing maximum intelligence per unit of compute through architectural innovation. That route bore its most sensational fruit half a year later with DeepSeek-V3 and in early 2025 with R1.

Key Points

1Released in May 2024: a 236B-parameter MoE with only about 21B activated per token.
2MLA (multi-head latent attention) sharply compresses the KV cache, cutting long-context inference cost.
3Pretrained on 8.1T tokens, matching GPT-4 Turbo-class quality at a fraction of the cost.
4Ultra-low API pricing triggered an industry price war, earning it the nickname "the Pinduoduo of AI".
5It established DeepSeek's efficiency route, setting up the global sensation of V3 and R1.

How It Works

DeepSeek-V2's two core innovations both target inference economics. First, MLA (multi-head latent attention): key-value matrices are low-rank-compressed into a latent vector, so inference caches only this small vector instead of full K/V, shrinking the KV cache to about 1/13 of MHA while decoupling the RoPE part to preserve positional information, greatly relieving the memory bottleneck of long-context serving. Second, DeepSeekMoE: fine-grained expert splitting (two activated out of 160-plus experts per token) plus shared experts that capture common knowledge and reduce routing redundancy, replacing fewer larger experts with more smaller ones for better specialization and training stability. Combined with 8.1T tokens of high-quality multilingual data and FP8 mixed-precision training, the result was the combination of large capacity, small compute, and low cost. The lesson: under compute constraints, architectural and systems innovation converts directly into cost advantage.

Impact & Legacy

DeepSeek-V2's impact reached far beyond one model release. Its pricing proved that low-cost high-performance was commercially viable, directly triggering China's May 2024 API price war as ByteDance, Alibaba, Baidu, and Zhipu cut prices, putting the political economy of inference cost on the table: when architectural innovation cuts costs by an order of magnitude, premia maintained by compute scale become untenable. Strategically, against the backdrop of US chip export controls, it offered a Chinese answer of efficiency countering scale, the prelude to V3 and R1 stunning the world. Methodologically, DeepSeek's series of papers on MLA, DeepSeekMoE, and GRPO were all published, returning the dividends of architectural innovation to the community, itself a victory of open-source spirit.

Key Takeaways

Inference cost is dominated by KV cache and activated parameters; architecture rewrites the economics.
Low-rank compression (MLA) is the standard tool for attention memory savings, needing RoPE decoupling.
Fine-grained experts plus shared experts are key to stable, specialized MoE training.
Under compute constraints, maximizing intelligence per FLOP is itself a strategic advantage.

Glossary

Multi-head Latent Attention (MLA)MLA

An attention mechanism low-rank-compressing KV into a latent vector to cut inference cache, DeepSeek-V2's core innovation.

Mixture of Experts (MoE)混合专家模型

A sparse architecture of expert sub-networks where a router picks a few per token, giving large capacity at controlled compute.

FP8 TrainingFP8 训练

Mixed-precision training with 8-bit floating point, speeding up and cutting cost with controlled accuracy loss.

Related Events

2023