# One transformer, two ways to write
Date: 2026-10-21
Description: I trained the same small transformer as a GPT and as a masked diffusion model on Tiny Shakespeare. The diffusion model's usual sampling trick makes it look fluent by making it repeat itself.
Canonical: https://shauryasharma.tech/gpt-vs-dllm.html
A GPT writes left to right, one character at a time. A masked diffusion language model (a dLLM) starts from a page of blanks and fills them in, in whatever order it likes, several at a time. I trained one of each on Tiny Shakespeare, using the same network for both, and watched them write.
GPT left to right
0 forward passes
dLLM fills in
0 forward passes
Real samples from both models, replayed on one clock counted in forward passes through the network. The GPT needs one pass per character. The dLLM needs one pass per step, however many characters it reveals in that step. Dots are characters the dLLM hasn't revealed yet. Switch the reveal order, cut the steps, or pick another of the three samples saved for each setting.
Both texts look roughly like Shakespeare. Read the confidence-order one closely, though, and it keeps saying the same few things. That repetition is the whole post.
## One transformer, two objectives
The network is a standard character-level transformer in the style of Karpathy's [nanoGPT](https://github.com/karpathy/nanoGPT): 6 layers, 6 heads, 384-dimensional embeddings, a 256-character context and dropout 0.2. With Tiny Shakespeare's 65 characters that comes to 10,788,929 parameters for the GPT, and 384 more for the dLLM, which needs one extra embedding row for its `[MASK]` token.
The GPT is the usual one: causal attention, so each position can only see what came before it, and a next-character loss. The only line that makes it a GPT is the mask:
```python
if self.causal:
wei = wei.masked_fill(self.tril[:T, :T] == 0, float('-inf'))
```
The dLLM is the same network with that mask switched off, so every position sees the whole sequence. Training follows the MDLM and LLaDA recipe ([Sahoo et al. 2024](https://arxiv.org/abs/2406.07524), [Nie et al. 2025](https://arxiv.org/abs/2502.09992)). For each sequence, pick a masking rate t between 0 and 1, hide each character with probability t, and score the model only on the hidden ones, weighted by 1/t:
```python
t = torch.rand(B, 1, device=DEVICE).clamp(min=1e-3)
mask_indices = torch.rand(B, T, device=DEVICE) < t
x_masked = x.clone()
x_masked[mask_indices] = MASK_ID
logits = model(x_masked)
loss_raw = F.cross_entropy(logits.view(B * T, -1), x.view(B * T), reduction='none').view(B, T)
loss = (loss_raw * mask_indices / t).sum() / (B * T)
```
I trained the GPT for 5,000 steps (best validation loss 1.47) and the dLLM for about 50,000 (best 1.67). Those two numbers aren't directly comparable. The dLLM's loss is an upper bound on its true loss, not the loss itself, so 1.67 against 1.47 doesn't tell you how far apart the models are.
Training (dashed) and validation (solid) loss. The dLLM took ten times as many steps to get to 1.67. Footnote: its training was interrupted and resumed from the best checkpoint, so the step labels after the marked point (about 47,000) run roughly 1,000 steps too high.
Sampling the dLLM starts with all 256 positions masked. Each step, the model samples a character for every position, and then k of the still-hidden positions get revealed. The other sampled characters are thrown away. Which k positions get revealed is the choice that matters:
```python
if order == 'confidence':
conf = probs.gather(-1, preds.unsqueeze(-1)).squeeze(-1)
else:
conf = torch.rand(n, length, device=DEVICE)
conf = conf.masked_fill(~mask, -1.0)
k = math.ceil(remaining / (num_steps - step))
idx = conf.topk(k, dim=-1).indices
```
Confidence order reveals the positions whose sampled character the model was surest about. That's the trick LLaDA uses. Random order reveals random positions, which matches how training hides characters. At one character per step, random order samples exactly from the distribution the model learned.
## The finding: fluent, because it repeats itself
To score samples I used the trained GPT as a judge: the average per-character loss of a sample under the GPT. Lower means more predictable, which isn't the same as better. Text that says "you you you" is very predictable. So I also counted two things about the words: the share of words that are distinct, and the share that repeat the word right before them.
confidence orderrandom orderreferencespredictable but repetitive
One point per reveal order and step count, 100 samples each. Up is more predictable to the GPT; right is less repetitive. Real text and the GPT's own samples sit up and to the right. Confidence order reaches the GPT's predictability only by moving left, into repetition. Hover or tab to a point for its numbers.
At 256 steps, confidence order scores 1.17, close to the GPT's own samples at 1.09. By that measure it's nearly as good as the GPT. But only half its words are distinct (0.50, against 0.88 in real Shakespeare), and 12.7% of its words repeat the word before them. Real text has none.
Random order looks worse by the same judge: 2.29. Its repetition, though, sits at real-text levels: 0.92 distinct words and 0.2% doubled. It gets the structure right, with speaker names and line breaks where they belong, and misspells a lot of words. That's the model's honest quality, and at this size it's clearly worse than the GPT.
This isn't new. A recent paper re-examining confidence remasking ([Frkovic et al. 2026](https://arxiv.org/abs/2606.12232)) talks about "the diversity collapse previously reported for confidence-based unmasking" in large models. What I have is a small reproduction you can watch happen.
## How many steps can you cut?
The reason to want a dLLM is speed: reveal more characters per step and you need fewer passes through the network. So how few can you get away with?
confidence orderrandom orderreferences
Each faint dot is one sample's GPT loss; the lines are the means with a band of one standard deviation. Hover, or focus the chart and use the arrow keys, to read a step count.
Random order degrades gracefully. Going from 256 steps down to 32 costs it 0.2 in GPT loss. Confidence order loses 0.86 over the same cut.
Both fall apart below about 8 steps. At 1 step every character is sampled at once, independently of every other, and the result is worse than guessing uniformly at random from the 65 characters (ln 65 ≈ 4.17).
## In what order does a dLLM write?
With confidence order the model chooses its own writing order, so I recorded when each character got revealed, as a fraction of the run, over 500 samples at 256 steps.
Top: when each type of character is revealed, as a share of the run, with one standard deviation either side. Middle: the average reveal time at each of the 256 positions, darker for earlier. Bottom: one real 256-step sample filling in, with how much of each type is revealed so far.
Spaces come first, on average at 0.44 of the way through the run. Letters follow at about 0.50, then punctuation and line breaks at about 0.58. The model lays out where the words go before deciding what they are.
The spread is wide, a standard deviation of about 0.29 for every type, so this is a tendency, not a rule. And I only measured it for confidence order. Under random order, reveal time is uniform by construction.
## What this adds up to
At about 10 million parameters, the dLLM's real output is clearly worse than the GPT's. Confidence order hides that by trading variety for predictability, and a judge that only measures predictability can't see the trade.
Some caveats. Everything here is one training run with one seed. The GPT scoring its own samples is biased low, so 1.09 is a reference point, not a target. Random order's distinct-word score comes out slightly above real text partly because misspellings create new "words". And the reveal-order numbers only describe confidence order.
Others have built the same comparison: [tiny-diffusion](https://github.com/nathan-barry/tiny-diffusion) is a character-level diffusion model on Tiny Shakespeare with a small GPT alongside it.