LLaMA: Open Source LLM Wave
Meta releases LLaMA models, opening a new era of open-source large language models.
Overview
In February 2023, Meta AI released the LLaMA (Large Language Model Meta AI) series of models, including versions from 7 billion to 65 billion parameters. Although intended only for research, the models were quickly leaked online, sparking an innovation wave in the open-source community. LLaMA demonstrated that smaller models with sufficient training could achieve excellent performance, challenging the "bigger is better" paradigm. Based on LLaMA, the community quickly developed numerous fine-tuned models like Alpaca, Vicuna, and WizardLM, driving the flourishing open-source LLM ecosystem.
Study Notes
Background, mechanics, impact, and key takeaways in one read
Background & Context
LLaMA was released by Meta AI on February 24, 2023 in the paper "LLaMA: Open and Efficient Foundation Language Models." At the time, top LLMs were mostly closed and needed huge compute, so researchers could rarely access them. Meta's bet was to train foundation models with fewer parameters but more data that rival GPT-3, then give open weights to academia for reproducible research. LLaMA shipped in 7B, 13B, 33B, and 65B sizes; the 65B and 33B were trained on 1.4 trillion tokens and the 7B on 1 trillion, using only public data. Within a week the weights leaked, which unexpectedly ignited an explosive open-source LLM ecosystem.
Key Points
- 1It shipped open weights at 7B, 13B, 33B, and 65B, so small models run on ordinary research hardware.
- 2The 13B version beat the 10x larger GPT-3 (175B) on most benchmarks, validating "small model, more data."
- 3It was trained purely on public datasets such as Common Crawl, C4, GitHub, Wikipedia, and arXiv, with no proprietary data at all.
- 4Its architecture is a decoder-only Transformer that uses RMSNorm, SwiGLU, and rotary position embeddings (RoPE).
- 5Its weights leaked within a week, spawning hundreds of fine-tuned derivatives such as Alpaca and Vicuna.
- 6LLaMA 2 (July 2023) moved to a commercial license, pushing open weights into real-world industry use.
How It Works
LLaMA kept the decoder-only Transformer backbone of the GPT line but bundled several already-proven components: RMSNorm pre-normalization for stable training, the SwiGLU activation replacing the standard feed-forward, and rotary position embeddings (RoPE) for position information. These choices had appeared separately in earlier models; LLaMA was among the first to ship them together in a widely distributed family. Its training followed the Chinchilla scaling law: rather than grow the model, train a smaller one on far more tokens——so the 7B saw 1 trillion tokens and the 65B saw 1.4 trillion. Data came from public web, code, encyclopedias, and papers across 20 languages using Latin or Cyrillic scripts. Open weights let the community quantize, fine-tune, and run models locally, starting the open LLM wave.
Impact & Legacy
LLaMA proved that open weights plus efficient training can approach closed-source giants, reshaping the LLM landscape. Leaked weights let researchers worldwide fine-tune a top base freely for the first time, spawning derivatives like Alpaca, Vicuna, and Chinese-Alpaca; LLaMA 2's commercial license carried open models into enterprises. It popularized local deployment and model quantization, while raising worries about open-model safety and misuse. Today the LLaMA family is among the most influential open-weight lineages. Its leak demonstrated that restricting weights cannot contain demand for open models, reshaping how labs approach release.
Key Takeaways
- Small model plus more data can approach large models; scaling is not only about parameters.
- Open weights, even leaked, unlock far more research and innovation than closed APIs.
- Clever combination of mature components is often more useful and reproducible than novel architecture.
- The boundary between open and commercial use is now a key issue in LLM governance.
Glossary
Publishing trained model parameters for free download, so users can fine-tune, quantize, and deploy them, unlike closed API-only services.
A rotation-based way to encode position into attention so the model better senses token order and relative distance.
An empirical rule describing how model performance changes smoothly with scale (parameters, data, compute), guiding training budgets.