Temperature is not a creativity dial.

The standard explanation: low makes the model more deterministic, high makes it more creative. Set 0.2 for facts, crank to 1.5 for poetry. That's not wrong. It's also the least interesting thing you can say about it.

It's a mechanism for controlling which parts of a model's knowledge are accessible at inference time. The previous posts in this series covered architecture: how sequence models read, remember, and attend. None of that touches what happens at output: when the model produces a distribution over every possible next token and you have to pick one. That's where temperature lives. And it connects to a training technique that lets a 7B model (gaming GPU territory) outperform one twice its size.

The Audition: Logits and Softmax

Before temperature makes sense, you need to understand what a language model is actually doing when it picks the next word.

For the prompt “The sky is...”, the model has seen billions of sentences. It knows “blue” almost always follows. “Cloudy” sometimes. “Cheese” essentially never. This knowledge isn't stored as a rule. It's compressed into numbers during training.

Those numbers are called logits: raw scores on the model's internal scorecard. “Blue” might score 15.2. “Cloudy” scores 10.1. “Cheese” scores −4.8.

You can't sample from raw scores directly; they don't sum to 1. So you convert them using softmax. Exponentiation amplifies big score differences and shrinks small ones, exactly the property you want when converting a scorecard into a probability distribution:

Result: blue ≈ 99.3%, cloudy ≈ 0.6%, cheese ≈ 0%. Now the model picks a token by sampling from this distribution.

Enter Temperature

Temperature ($\tau$) is a single number, greater than zero. It is applied to the logits before softmax: you divide every score by $\tau$ first.

One division. The effect depends entirely on what τ is.

$\tau =1.0$ (default): Divide every score by 1. Nothing changes. You sample from the model's unmodified output distribution: the logits exactly as produced.

$\tau \to 0$ (near-zero, e.g. 0.1): Dividing by a tiny number amplifies the gaps between scores. Blue's 15.2 becomes 152. Cheese's −4.8 becomes −48. The gap widens until softmax collapses to a single spike: blue gets 99.99%, everything else essentially 0. Effectively the same answer every time. This is greedy decoding. (The Caution section below covers the small exception.)

$\tau >1$ (e.g. 1.5, 2.0): Dividing by a large number compresses the scores. 15.2 becomes 7.6. −4.8 becomes −2.4. The gap shrinks. Cheese goes from 0.1% to a non-trivial probability. You're sampling from a flatter distribution: more variance, more surprise, and occasionally “cheese.”

import numpy as np

def softmax_with_temperature(logits, temperature=1.0):
    scaled = logits / temperature
    # Subtract max for numerical stability (does not change probabilities)
    exp    = np.exp(scaled - scaled.max())
    return exp / exp.sum()

# Logits for "The sky is ..."
tokens = ["blue", "cloudy", "clear", "dark", "green", "cheese"]
logits = np.array([15.2, 10.1, 8.4, 6.2, 2.1, -4.8])

for tau in [0.1, 1.0, 2.0]:
    probs = softmax_with_temperature(logits, temperature=tau)
    print(f"\nτ = {tau}")
    for tok, p in zip(tokens, probs):
        bar = "█" * int(p * 40)
        print(f"  {tok:8s}  {p:.4f}  {bar}")

# τ = 0.1 → blue: 1.0000  (everything else ≈ 0)
# τ = 1.0 → blue: 0.9928, cloudy: 0.0061, clear: 0.0011, dark: 0.0001, cheese: ≈ 0
# τ = 2.0 → blue: 0.8896, cloudy: 0.0695, clear: 0.0297, dark: 0.0099, cheese: 0.00004

Dark Knowledge

When the model produces logits for “The sky is...”, the probability for “cheese” is tiny. But it's not zero. That number encodes something real: the model has encountered contexts where unusual words followed those tokens. A recipe. A metaphor. A fragment. The model knows “cheese” is wrong here, and the logit score reflects exactly how wrong, in a way that captures the shape of language around that prompt.

Geoff Hinton called this dark knowledge. The top prediction is just one number from a vector of thousands. The relative scores of all candidates encode semantic similarity: which words are neighbors, which meanings cluster, which alternatives are close calls.

• The model knows “apple” and “pear” are similar not because it was told so, but because they appear in similar contexts across billions of sentences. If “apple” is the top-1 prediction, “pear” will have a noticeably higher logit than “hammer.” That gap is the knowledge.
• Ask for the capital of Australia. Sydney dominates English-language web text; it's the largest city, the Olympic host, the one in every travel article. The answer is Canberra. At $\tau =0$, a model trained on uncurated text can surface Sydney as its top token. At $\tau =0.7$, Canberra has a chance. “Lower temperature = more factual” only holds when the correct answer is also the most frequent one in training data.

Low $\tau$ suppresses all of this. You'll see only the tip: the dominant response. High $\tau$ flattens the distribution enough that wrong tokens, not just uncertain ones, gaining non-trivial probability. That's why hallucinations increase at high temperature: you're sampling from the tails, where both genuinely ambiguous answers and outright wrong ones live.

Distillation: What the Runner-Ups Know

Suppose you want to train a small model (7B parameters, fits on a gaming GPU) to behave like a large one (70B parameters, requires a data center). The obvious approach: have the large model answer questions, then train the small model on those answers.

The problem: the naive approach takes only the teacher's top prediction as a hard label. “The capital of France is Paris.” The small model learns that Paris is correct. It doesn't learn that another city was a close second, or that the model was far more confident here than it was on a question it was genuinely uncertain about.

Train the small model on the large model's soft targets (the full probability distribution at high temperature) instead of the final answers. At $\tau =3$ or $\tau =4$, the distribution is flat enough that the relative scores of all candidates become visible. The small model learns not just what the teacher predicts; it learns which answers the teacher was nearly certain about and which it was hedging on. That signal is invisible in a one-hot label.

This is knowledge distillation (Hinton et al., 2015). The high-temperature logits are the dark knowledge transfer mechanism. It's one reason smaller models trained via distillation regularly outperform equivalently-sized models trained only on hard labels; the meaningful comparison is a distilled 7B vs a 7B trained the usual way, not distilled 7B vs the 70B teacher it learned from.

import torch
import torch.nn.functional as F

def distillation_loss(student_logits, teacher_logits, labels, tau=3.0, alpha=0.7):
    """
    student_logits: (batch, vocab) — raw scores from the small model
    teacher_logits: (batch, vocab) — raw scores from the large model
    labels:         (batch,)       — ground-truth token indices
    tau:            temperature applied to both before KL divergence
    alpha:          weight on soft loss vs hard loss
    """
    # Soft targets from teacher at high temperature
    soft_teacher = F.softmax(teacher_logits / tau, dim=-1)
    soft_student = F.log_softmax(student_logits / tau, dim=-1)

    # KL divergence — scaled by tau^2 to match gradient magnitude of hard loss
    soft_loss = F.kl_div(soft_student, soft_teacher, reduction="batchmean") * (tau ** 2)

    # Standard cross-entropy on hard labels
    hard_loss = F.cross_entropy(student_logits, labels)

    return alpha * soft_loss + (1 - alpha) * hard_loss

# Typical setup: tau=3-4 surfaces enough dark knowledge without destabilising training
# alpha=0.7 weights the soft loss more heavily — the soft targets are richer signal

Exploit vs Explore

Every temperature choice is a tradeoff.

Exploit (low τ): You trust the model's training distribution. You want the most likely completion, consistently. The same prompt should give the same answer. Use this for: code autocomplete, SQL generation, factual summarisation, anything where reproducibility matters. A code assistant at $\tau =0.1$ generates the same correct function body every run. At $\tau =1.2$, it occasionally invents a method signature that doesn't exist.

Explore (high τ): You're willing to trade reliability for variety. You want candidates from further down the list: novel phrasings, unexpected angles, combinations that would never surface at low temperature. Use this for: brainstorming, creative writing, generating multiple options to pick from, anywhere diversity is the point. Ask for ten product name ideas at $\tau =0.1$ and you'll get ten variations of the same safe answer. At $\tau =1.0$, the spread is genuinely wider.

The wrong temperature for the task gives a specific kind of wrong. A factual task at $\tau =1.5$ hallucinates because you're sampling from the model's uncertainty. A creative task at $\tau =0.1$ produces safe, repetitive output that reads like every other generated response.

One practical note: most production APIs (GPT-4, Claude, Gemini) run on instruction-tuned models, not raw pretrained ones. RLHF and supervised fine-tuning tend to concentrate probability mass on preferred outputs, effectively sharpening the distribution before any τ is applied. That means the base model's τ=0.7 and the instruction-tuned model's τ=0.7 are not the same thing. API defaults (typically 0.7 to 1.0) are calibrated for instruction-tuned distributions specifically; the raw intuition still holds, but the thresholds shift.

When Temperature Goes Wrong