Build a character-level n-gram model, smooth it, then score it in the sandbox, and predict — before you read the panels — the add-k probability floor an unseen character gets and the probability its seen sibling keeps, what an unseen context backs off to, the perplexity of a perfectly predictable run versus a uniform one, and which way perplexity moves as the model order rises.
An n-gram model just counts: how often each character followed each context of the previous n-1 characters. The smoothed conditional is P(c | ctx) = (count(ctx,c) + k) / (count(ctx) + kV), so with k > 0 even a character never seen after a context gets a small floor k/(count(ctx) + kV) and held-out text is never assigned probability zero. Perplexity is 2 raised to the average per-character log-loss in bits — a uniform model over V symbols scores exactly V, a certain one scores 1. There is no neural network here — no embeddings, no weights, only counts, so every number is exact.
Open the sandbox. The step values below use tiny hand-built training strings you can check by hand against the model's own count panels; the context box and the order, k and temperature sliders drive every number.
Train -> abababab Order -> 2 (bigram) k -> 0.5 Context -> a
Then: Train -> abcd Order -> 1 (unigram) k -> 0
The distribution panel shows one bar per vocabulary character, each with its raw count and its add-k probability; the perplexity panel prints the training and held-out perplexity and sweeps it across orders 1 to 4. Every bar and number is computed from the counts, not sketched.
In "abababab" the context "a" occurs 4 times and is always followed by "b" — so "a" after "a" is never seen. With the bigram, V = 2 and k = 0.5, predict the floor P(a | a) = k/(N + kV) before you read the bar.
P(a | a) = 0.5 / (4 + 0.5×2) = 0.5 / 5 = 0.1. The unseen character is not impossible — smoothing hands it a small floor so held-out text can never score probability zero.
The floor the unseen char received came out of the seen one's mass. Predict what "b" after "a" keeps: P(b | a) = (count + k)/(N + kV) = (4 + 0.5)/(4 + 0.5×2).
P(b | a) = 4.5 / 5 = 0.9. The two probabilities 0.9 and 0.1 sum to 1 — smoothing redistributed mass from the seen event to the unseen one without ever leaving the simplex.
Switch to a trigram (order 3) on the same "abababab". The context "zz" never appears in training at all. Predict what P(a | "zz") backs off to, with V = 2.
P(a | "zz") = 1/V = 1/2 = 0.5. A context the model never saw carries no counts, so instead of a 0/0 the model falls back to a uniform guess over the vocabulary — the safe default when it has nothing to go on.
Score the bigram on the very text it learned, "abababab", with k = 0 (no smoothing). Every next character is certain, so predict its perplexity.
Perplexity = 1. Every prediction has probability 1, so the average log-loss is 0 bits and 2^0 = 1 — the floor of the scale, a model that is never surprised.
Now a unigram (order 1) on "abcd", where each of the four characters appears exactly once, so the distribution is uniform. With k = 0, predict its perplexity on "abcd".
Perplexity = V = 4. Each character has probability 1/4, a log-loss of log2(4) = 2 bits, and 2^2 = 4 — a uniform model over V symbols is exactly as perplexed as a fair V-sided die.
On the Seasons prose corpus, sweep the order from 1 to 4 with k = 0.1 and watch the training perplexity panel. Predict how many times it rises as the order increases across 1 → 2 → 3 → 4.
It rises 0 times — training perplexity only falls (17.2 → 7.4 → 3.8 → 3.1). A longer context always fits the training text at least as well. That is exactly why training perplexity alone is misleading: a high-order model can memorise the text and still generalise poorly — which the held-out panel exposes.
Everything above is waiting in the sandbox. Raise k to watch the whole distribution flatten toward uniform, drop temperature below 1 to sharpen the generated text toward the mode and above 1 to loosen it, and compare training against held-out perplexity as the order climbs — the gap is the model memorising.
- Read the add-k floor an unseen char gets: P(a | a) = 0.1, and its seen sibling P(b | a) = 0.9
- Saw an unseen context back off to 1/V = 0.5
- Scored a perfectly predictable run at perplexity 1
- Scored a uniform unigram at perplexity V = 4
- Watched training perplexity only fall as order rose — 0 rises
- Every value you predicted is the demo's own count, add-k and perplexity arithmetic — an n-gram model, not a neural network