Generative AI comes in two popular forms: language models (chatbots like ChatGPT) and text‑to‑image models (like Stable Diffusion, Midjourney, or DALL·E). They both take a prompt, but under the hood they learn and generate in very different ways. This guide explains, cleanly and practically, how they are trained and how they create text or images.
The big picture
- LLMs learn to be an ultra‑capable autocomplete: they read a lot of text and learn to predict the next token.
- Image generators learn to be expert denoisers: they start from random noise and learn how to remove noise step by step until an image appears.
Below, we walk through training and generation for each, then show how text becomes an image with a concrete example.
How LLMs are trained
1) Data and tokens. Text from books, code, and the web is split into small pieces called tokens (sub‑words and symbols).
2) Objective. The model (a Transformer) is trained to predict the next token given the previous ones. During training it sees the correct next token (“teacher forcing”) and learns via a standard loss that penalizes wrong predictions.
3) Scale and generalization. Because the task—”next token”—is universal, the model accidentally learns skills (reasoning, summarizing, coding) that help with many tasks.
4) Alignment for helpfulness. After pretraining, many teams add instruction fine‑tuning and RLHF (human feedback) so the model follows instructions and avoids harmful content.
Inside an LLM: the layers doing the work
- Token embeddings turn tokens into vectors; positional signals (e.g., RoPE/ALiBi) let the model know order.
- Stacked decoder blocks repeat: multi‑head self‑attention → feed‑forward network → residual add + layer norm. The causal mask blocks attention to future tokens.
- A final linear layer + softmax produces a probability over the vocabulary for the next token.
In practice, this stack is repeated dozens of times. More layers and width mean more capacity to model long‑range patterns in text.
How an LLM generates text (next‑token prediction)
1) Tokenize your prompt. Your words become tokens.
2) Predict a distribution. The model maps the prompt to a probability over the next token (via attention and a softmax layer).
3) Sample a token. It picks one token (more creative or more precise depending on settings like temperature).
4) Repeat. That new token is appended, and steps 2–3 repeat until the model stops. You read the stream as it’s produced.
Example (LLM)
Prompt: “Explain dropout in neural networks in one short paragraph.”
- The model recalls patterns linking “dropout” to “regularization”, “randomly zeroing units”, and “reducing overfitting”.
- It predicts likely next tokens (“Dropout is…”, “During training…”), sampling a fluent paragraph one token at a time.
Fine‑tuning LLMs: from raw knowledge to helpful assistants
- Supervised instruction tuning (SFT): train on instruction–response pairs so the model follows natural commands.
- Preference tuning: use RLHF or simpler preference‑based methods (e.g., DPO) to nudge the model toward answers people prefer.
- Parameter‑efficient fine‑tuning (PEFT): add small adapters like LoRA/QLoRA so you can adapt big models with far fewer trainable weights and memory.
Result: the same pretrained model can be specialized for domains (finance, code, healthcare) without retraining everything.
How text‑to‑image models are trained
1) Paired data. Training uses large sets of image–text pairs (captions). The text is converted to vectors with a text encoder (commonly CLIP or T5).
2) Latent space. Images are often compressed with a variational autoencoder (VAE) into latents—a smaller space that keeps detail but makes training faster.
3) Diffusion objective. The trainer adds noise to the image (or latent) for many steps. A denoiser network is trained to predict the noise (or the clean image) at each step. Minimizing this error teaches it how to reverse noise.
4) Text conditioning. During training the denoiser uses cross‑attention to “look at” the text embeddings so what it learns to draw matches the caption.
5) Guidance (optional). Models are often trained so they can run with or without text. At inference you can combine both to balance fidelity vs. diversity (commonly called classifier‑free guidance).
Inside an image model: the layers doing the work
- A VAE pair (encoder/decoder) maps pixels ↔ latents.
- The denoiser backbone is either a U‑Net or a Transformer‑based denoiser (DiT/MMDiT) operating on latent patches.
- Cross‑attention blocks merge text embeddings with visual features so the model can place objects, styles, and attributes in the right spots.
How text becomes an image (denoising, step by step)
1) Encode the prompt. The text encoder turns “a red bicycle on a cobblestone street in Stockholm at golden hour” into vectors.
2) Start from noise. Sample pure noise in the latent space.
3) Iterative denoising. For dozens of small steps, the denoiser removes a bit of noise. At each step, cross‑attention lets the model align details (red, bicycle, cobblestone, warm light) with what it’s drawing.
4) Decode to pixels. The cleaned‑up latent is passed through the VAE decoder to get the final image.
5) Optional finishing. You can upscale, inpaint, or mask parts and repeat the denoising cycle for edits.
Example (image)
Prompt: “a minimalist product photo of a matte‑black coffee mug on light oak, soft morning light, shallow depth of field”
- The text encoder builds vectors for objects (mug), materials (matte‑black), scene (oak), and style (soft light, shallow depth).
- Starting from noise, the model gradually reveals shapes and textures that best match those vectors, then decodes to a clean, photorealistic image.
Fine‑tuning and control for images
- Personalization: techniques like DreamBooth or Textual Inversion bind a new concept (a person, product, or style) to a token so prompts can place it in new scenes.
- Adapters: LoRA updates small cross‑attention or convolutional blocks to adapt style or subject with minimal compute.
- Structural control: ControlNet adds guidance from edges, depth, poses, or layouts, giving precise composition control without retraining the whole model.
Key terms, quickly
- Token: the small unit of text an LLM reads and writes.
- Transformer: the attention‑based architecture used by modern LLMs.
- Latent: compressed representation of an image used for faster diffusion.
- Cross‑attention: the mechanism that lets image models align what they draw with your text.
- Guidance: a trick to nudge images closer to your prompt when sampling.
Putting it together — clean mental models
- An LLM is a stack of attention blocks trained to continue text. Generation is a loop: tokenize → attend → softmax → pick a token → repeat.
- A text‑to‑image model is a text‑conditioned denoiser in latent space. Generation is an unfold: encode text → start from noise → denoise in small steps with cross‑attention → decode to pixels.
That’s the core difference: LLMs learn to continue text; image models learn to remove noise. Same prompt box, completely different engines.