2022Model

Stable Diffusion: Open Source Image Generation

Stability AI releases open-source Stable Diffusion, sparking the text-to-image generation revolution.

#generative-ai #text-to-image #stable-diffusion #open-source #diffusion

Overview

In August 2022, Stability AI released the open-source Stable Diffusion model. This is a text-to-image generation system based on latent diffusion models that can run on consumer-grade GPUs. Unlike closed-source models like DALL·E 2, Stable Diffusion's full open-source nature allowed anyone to deploy and improve the model. This sparked a revolution in creative fields: artists, designers, and ordinary users could all easily generate high-quality images. The open-source community quickly developed various derivative models and tools like ControlNet and LoRA. Stable Diffusion is considered a milestone in democratizing generative AI.

Study Notes

Background, mechanics, impact, and key takeaways in one read

About 3 min read

Background & Context

Stable Diffusion was released on August 22, 2022 by Stability AI together with the CompVis group at LMU Munich (led by Robin Rombach) and Runway ML. It builds directly on the Latent Diffusion Model (LDM) of Rombach et al. (arXiv:2112.10752, CVPR 2022 best paper). Before it, leading text-to-image systems such as DALL-E 2 and Midjourney were closed and cloud-only, so ordinary users could not run them locally. The decisive idea was to perform diffusion in a compressed latent space rather than in pixel space, cutting the compute and VRAM needed to consumer-GPU levels (roughly 6–8 GB). Releasing the weights under the CreativeML Open RAIL-M license let anyone download, modify, and commercially use the model, igniting the open-source generative-image ecosystem.

Key Points

1It is based on latent diffusion: denoise in a compressed latent space, then decode back to pixels, which slashes compute.
2The pipeline has three parts: a VAE to compress images, a CLIP text encoder for conditioning, and a U-Net that denoises in latent space.
3With 8x spatial compression, the SD 1.x U-Net has about 860M parameters and runs on consumer GPUs.
4It was trained on LAION-5B (around 5.8 billion image-text pairs) and its aesthetics subset, an unprecedented dataset.
5Released open-weight under the CreativeML Open RAIL-M license, it was the first top-tier text-to-image model runnable locally.
6It spawned a huge ecosystem of tools and fine-tunes such as WebUI, LoRA, DreamBooth, and ControlNet.

How It Works

Stable Diffusion runs in three stages. A variational autoencoder (VAE) first compresses a 512x512 image into a 64x64 latent, shrinking the data by roughly 64x so diffusion operates on small patches rather than millions of pixels, which is both fast and cheap. The denoising core is a U-Net that starts from pure noise and, at each step, predicts how much noise to remove, conditioned on the text prompt, repeating about 20–50 steps to produce a clean latent. Text conditioning comes from a CLIP text encoder that turns the prompt into a 77-token embedding, injected into every U-Net layer via cross-attention, steering what appears in the image. An analogy: the VAE is a compressor/decompressor, the U-Net is a painter that erases noise step by step, and CLIP is the director reading the brief. Finally the VAE decoder turns the latent back into pixels.

Impact & Legacy

Stable Diffusion brought top-tier text-to-image capability to ordinary computers, spawning a vast open-source creation and remix ecosystem and reshaping illustration, design, and film prototyping. Its limits and controversies are equally notable: the LAION training data contained copyrighted images, triggering lawsuits from Getty Images and artists against Stability AI; the model is easily repurposed for deepfakes and harmful content. Debates over training-data licensing and artists' rights remain unresolved and have become a focal point of generative-AI governance. Its open release also accelerated a wave of community fine-tunes, custom models, and user-facing tools that reshaped creative workflows.

Key Takeaways

Doing diffusion in latent space instead of pixel space is a textbook trade of engineering cleverness for compute freedom.
Open weights ignite ecosystems far more than closed APIs: tools, fine-tunes, and community innovation follow.
Text conditioning via cross-attention is the key mechanism behind controllable generation.
Powerful generation carries unavoidable social costs: copyright, misuse, and the need for regulation.

Glossary

Latent Diffusion潜空间扩散

Doing diffusion on a low-dimensional representation from an autoencoder instead of on pixels, greatly cutting compute cost.

Variational Autoencoder (VAE)变分自编码器

A neural network that compresses high-dimensional data into a low-dimensional latent and can reconstruct it; Stable Diffusion uses it to encode/decode images.

Cross-attention交叉注意力

A mechanism letting a model "read" another stream (such as text) while generating, used to inject text conditions into image denoising.

Related Events

2021