Every large model you have used was trained once, frozen, and shipped. After that it does not change. If the world changes, if you teach it something new in a conversation, if it fails at a task a hundred times, the weights stay exactly where they were.

The authors of Self-Distillation Fine-Tuning open their paper with the same observation: today’s AI systems “remain static after deployment”, and while they can adapt at inference time through retrieval or prompting, “they do not update their parameters to acquire new skills, internalize new knowledge, or improve from experience” Shenfeld et al. 2026. The TTT-Discover paper says it the other way round: “Most of today’s AI systems remain static after deployment, even though the world keeps changing. The high-level goal of continual learning is to enable AI systems to keep changing with the world, similar to how humans improve throughout their lives” Yuksekgonul et al. 2026.

Continual learning is the study of how to close that gap: how to let a learning system take in a stream of experience, one piece after another, and keep improving, without destroying what it already knew. Irie and Lake give it the plainest definition in their glossary: “learning new tasks, one after another, in a sequential manner” Irie & Lake 2025.

0.1The conversation this book follows

The menu this book follows

What Continual Learning Actually Means feat. Ilija Lichkovski from Prime Intellect

With Ilija Lichkovski (Prime Intellect), hosted by Yacine Mahdid on Learning with Yacine. Published 12 October 2026, about 1 h 30 min.

yacinemahdid.com/p/what-continual-learning-actually ↗

The episode page opens with a line that explains why this book exists: continual learning “is one hell of a hot topic in 2026 and it’s beautiful because (like consciousness) barely anyone agree on what are we even talking about” Lichkovski & Mahdid 2026. The reading list behind this book is the literature that conversation draws on. The chapters below follow the episode’s order, then go much deeper into each paper than a conversation can.

The episode’s own chapter list, as published on its page:

A gap to fill yourselfThe page lists “five desirable aspects” as a chapter title but does not name them. This book does not guess. Listen from 00:20:10 and write them in the margin.

0.2How to read a recipe

This is a cookbook, so most chapters contain one or more recipe cards. A card has the same four parts every time:

• You will need: the ingredients, meaning the setting, the data and the assumptions the method relies on.
• Method: the algorithm, step by step, in the order you would actually run it.
• The math: the paper’s own equations, typeset in LaTeX, with each symbol explained.
• Chef’s notes: what the authors themselves say can go wrong.

Around the cards, the chapters explain where each idea comes from and what the experiments showed, using the papers’ original figures with their original captions and tables transcribed number for number. Each figure carries a credit line and a link to its source, and the References chapter lists every paper with its licence. Quotation marks mean the authors’ own words. Everything else is written for this book.

0.3The menu

• Part I · The problem
• 1Learning in timeNeurons in a dish learn Pong, and living cells play Doom
• 2Catastrophic forgetting and Elastic Weight ConsolidationWhy networks forget, and the first widely used fix, derived from Bayes’ rule
• 3Three types of incremental learningTask-, domain- and class-incremental: what is being tested
• Part II · Classic recipes
• 4Five strategies, one bake-offRegularization, replay and friends, compared on equal terms
• 5Incentive and practiceMeta-learning the skill instead of hoping it emerges
• Part III · Recipes for language models
• 6Weights that write weightsText-to-LoRA and Doc-to-LoRA
• 7Context or weights?How in-context learning and fine-tuning generalize differently, and Negation Neglect
• 8Teach yourself: Self-Distillation Fine-TuningOn-policy learning from demonstrations
• 9Learning, fast and slowPrompts as fast weights, parameters as slow weights
• 10Learning to discover at test timeReinforcement learning on the one problem you care about
• Part IV · Tasting
• 11Did it actually learn? Continual Learning BenchMeasuring learning separately from capability
• 12Usable informationWhy computation can create information a learner can use
• ∞What is still rawOpen problems, in the authors’ own words

A brain never gets a training phase followed by a deployment phase. It learns while it acts, from consequences that arrive a moment after each action. The episode spends its first-principles section on exactly this contrast, “the time component in machine vs organic learning” Lichkovski & Mahdid 2026. The cleanest experiment on the organic side is DishBrain.

1.1The question

In 2022, Brett Kagan, Karl Friston and colleagues at Cortical Labs and Monash University asked whether a flat sheet of cultured neurons, grown on a chip, could learn a goal-directed task if it were given a body: a closed loop in which its own activity changes what it senses next Kagan et al. 2022. Their summary states the result:

“Applying implications from the theory of active inference via the free energy principle, we find apparent learning within five minutes of real-time gameplay not observed in control conditions. Further experiments demonstrate the importance of closed-loop structured feedback in eliciting learning over time.”

1.2Growing the “wetware”

Three kinds of cultures were used:

• cortical cells from E15.5 mouse embryos, called MCCs;
• human cortical cells grown from induced pluripotent stem cells with a dual SMAD inhibition protocol, called HCCs;
• later in the study, human neurons made by NGN2 direct reprogramming.

Cells were plated on a MaxOne high-density multielectrode array (HD-MEA). The mouse cultures fired synchronised bursts by about 14 days in vitro. The DSI-derived human cultures took much longer, about 73 days, before synchronised bursting appeared Kagan et al. 2022.

1.3The kitchen: a closed loop

The DishBrain software reads voltages from the array, turns activity in two predefined “motor regions” into paddle movement, runs the game, and writes the ball’s position back into the culture as electrical stimulation, all in real time. The paper describes three categories of stimulation: sensory (where the ball is), predictable and random Kagan et al. 2022.

1.4Why “unpredictable” feedback?

The design comes from the free energy principle, which the paper summarises as the claim that “any self-organizing system separate from its environment seeks to minimize its variational free energy”. The gap between predictions and sensations “may be minimized in two ways: by optimizing probabilistic beliefs about the environment to make predictions more like sensations or by acting upon the environment to make sensations conform to its predictions” Kagan et al. 2022. If that is true, the authors reason, a culture should learn to avoid whatever leads to unpredictable input. Hitting the ball leads to predictable input; missing leads to noise.

For readers who want the quantity itself, here is the textbook definition of variational free energy that the theory refers to. It is a standard identity, not an equation from the paper. Hidden states are $s$, observations $o$, the system’s internal belief $q(s)$ and its generative model $p(o,s)$:

Because the KL term is never negative, free energy upper-bounds surprise, $-\log p(o)$. The paper uses the information entropy of neural responses as a proxy for average surprise, “which is upper-bounded by VFE”, and predicts entropy should drop during gameplay and rise after random feedback.

1.5What happened

Across 399 test sessions, the authors compared the cultures with three controls:

• media-only arrays with no cells (CTL);
• rest sessions, where active cultures controlled the paddle but got no sensory information (RST);
• in-silico runs driven by random noise (IS).

Only the mouse and human cultures improved over time. Rally length went up for MCCs ($t = 6.15,\; p = 5.27\times10^{-8}$) and for HCCs ($t = 10.44,\; p = 3.92\times10^{-19}$). Both groups also missed the first serve (“aces”) less often and made more long rallies of more than three hits. Human cultures started worse than every control in the first five minutes, then overtook them Kagan et al. 2022.

Two checks rule out the obvious alternative explanations. Non-neural HEK293T cells and media-only controls showed no learning. Paddle movement alone did not explain rally length: the in-silico control moved the paddle the most and still performed no better than the other controls.

1.6The key ingredient is feedback

The most useful experiment for anyone thinking about continual learning is the feedback comparison: 486 sessions over three days and three conditions Kagan et al. 2022.

• Stimulus: the standard predictable and unpredictable feedback.
• Silent: after a miss, all stimulation was withheld for the same duration, then the game restarted.
• No feedback: after a miss the ball simply bounced on, with no change in stimulation at all.

Only the Stimulus condition improved significantly over time ($t = 7.48,\; p = 1.58\times10^{-12}$). Silent feedback did better than no feedback, but less than Stimulus. The authors’ conclusion is direct: “information alone is insufficient; feedback is required to form a closed-loop learning system” Kagan et al. 2022.

The electrophysiology tells the same story from inside the culture. During gameplay, sensory and motor regions were more strongly correlated than at rest. “Mutually exclusive” events, where one motor region fired and the other did not, became more frequent ($t = 14.64,\; p = 5.68\times10^{-48}$). The correlation between the two motor regions fell over the first five minutes of play and then levelled off.

1.7From Pong to Doom

A newer community project, doom-neuron, uses Cortical Labs’ CL1 platform to let living neurons control an agent in Doom doom-neuron (GitHub). The problem is much harder than Pong: a 320×240 screen instead of a ball position, and a hybrid action space for movement and shooting. So the repository adds learned machinery around the cells, and its README is candid about the obvious question: is the machinery doing all the learning?

Step 5 is the important trick. Because the neurons sit between the encoder and the reward, there is no gradient to backpropagate. The README says the encoder is trained “using the log-likelihood trick (REINFORCE-style), i.e., by including the encoder’s sampled stimulation log-probs in the PPO objective rather than backpropagating through spikes” doom-neuron (GitHub). In symbols, with $u$ a sampled stimulation, $o$ an observation and $\hat A$ an advantage estimate, the score-function gradient is

and PPO optimises its usual clipped surrogate (the standard objective of Schulman et al., not written out in the README), with ratio $r_t = \pi_\phi(u_t\mid o_t)/\pi_{\phi_\text{old}}(u_t\mid o_t)$:

On whether the cells actually learn, the README answers with ablations. With random spikes or zero spikes in place of the CL1’s output, learning disappears. With the encoder weights frozen at test time, reward still improved, because “biological neurons have an internal state (membrane potential, synaptic weights, adaptation currents). The same stimulation delivered at different points in training will produce different spike patterns” doom-neuron (GitHub).

The problem has a name from the late 1980s: catastrophic forgetting. Kirkpatrick and colleagues describe it as “the tendency for knowledge of previously learnt task(s) (e.g. task A) to be abruptly lost as information relevant to the current task (e.g. task B) is incorporated”, and they say exactly when it happens: “specifically when the network is trained sequentially on multiple tasks because the weights in the network that are important for task A are changed to meet the objectives of task B” Kirkpatrick et al. 2017.

Most deep learning avoids the problem by never facing it. Data from all tasks is shuffled together and trained at once (multitask learning). If tasks arrive one after another, you could store everything and replay it, but the authors point out that this “would require the amount of memories being stored and replayed to be proportional to the number of tasks” Kirkpatrick et al. 2017. The episode covers this family of fixes at 36:10 under “regularization” Lichkovski & Mahdid 2026. EWC is its best-known member.

2.1An idea borrowed from the brain

The paper starts from neuroscience. When a mouse learns a new skill, some excitatory synapses grow stronger, visible as larger dendritic spines, and “these enlarged dendritic spines persist despite the subsequent learning of other tasks, accounting for retention of performance several months later”. Erase those spines and the skill goes. The authors read this as evidence for task-specific synaptic consolidation: knowledge “durably encoded in a proportion of synapses that are rendered less plastic” Kirkpatrick et al. 2017. EWC is the artificial version: make some weights less plastic, and choose which ones by how much the old task needs them.

Figure 2.1 is the whole idea in one picture. Large networks have many good solutions. Over-parameterisation makes it likely “that there is a solution for task B, $\theta^*_B$, that is close to the previously found solution for task A, $\theta^*_A$”. So instead of walking straight to B’s minimum (blue), EWC keeps the weights inside the region that is still good for A. It does this with a quadratic penalty that acts “as a spring anchoring the parameters to the previous solution, hence the name elastic”. The spring is stiffer for the weights that matter more Kirkpatrick et al. 2017.

2.2The derivation

Treat training as finding the most probable parameters $\theta$ given data $\mathcal{D}$. Bayes’ rule, in log form (Eq. 1 of the paper):

The log-likelihood $\log p(\mathcal{D}\mid\theta)$ is just the negative loss, $-\mathcal{L}(\theta)$. Now split the data into two independent parts, task A’s data $\mathcal{D}_A$ and task B’s data $\mathcal{D}_B$. Applying Bayes’ rule twice gives Eq. 2:

The left side is still the posterior given all the data, but the right side only needs B’s loss plus the posterior after A. In the paper’s words, “All the information about task A must therefore have been absorbed into the posterior distribution $p(\theta\mid\mathcal{D}_A)$” Kirkpatrick et al. 2017. If we could carry that posterior forward, we would never need A’s data again.

The true posterior is intractable, so EWC uses a Laplace approximation (following MacKay): a Gaussian centred at the solution $\theta^*_A$, whose precision is the curvature of the loss there. Expanding to second order around $\theta^*_A$, where the gradient is zero:

The Hessian $H_A$ is replaced by the Fisher information matrix $F$, and only its diagonal is kept. The paper lists why $F$ is a good choice: “(a) it is equivalent to the second derivative of the loss near a minimum, (b) it can be computed from first-order derivatives alone and is thus easy to calculate even for large models, and (c) it is guaranteed to be positive semi-definite” Kirkpatrick et al. 2017. The diagonal entry for parameter $i$ is the expected squared gradient of the log-likelihood:

Substitute the Gaussian into Eq. 2, drop constants, and flip the sign to get a loss. That gives the EWC objective, Eq. 3 of the paper:

$\mathcal{L}_B$ is the loss for task B only. $\theta^*_{A,i}$ is weight $i$ after task A. $F_i$ is its importance. $\lambda$ “sets how important the old task is compared to the new one” Kirkpatrick et al. 2017.

For a third task C, EWC keeps the parameters close to both earlier solutions. This “can be enforced either with two separate penalties, or as one by noting that the sum of two quadratic penalties is itself a quadratic penalty” Kirkpatrick et al. 2017. Concretely, for tasks $k = 1,\dots,K-1$ already learned:

2.3First test: permuted MNIST

Each task is MNIST with the pixels shuffled by a fixed random permutation: “Each task was thus of equal difficulty to the original MNIST problem, though a different solution would be required for each.” The network is trained on a task, then never sees that task’s data again Kirkpatrick et al. 2017.

Panel A shows three behaviours:

• Plain SGD (blue) forgets A as soon as B begins, and the forgetting “compounds further with more training time, and the addition of subsequent tasks”.
• A uniform $L_2$ penalty (green) protects A, but “task B cannot be learned properly as the constraint protects all weights equally, leaving little spare capacity”.
• EWC (red) learns B without forgetting A.

Panel B shows that the best SGD-with-dropout setup did not scale to more tasks, while “EWC allows a large number of tasks to be learned in sequence, with only modest growth in the error rates” Kirkpatrick et al. 2017.

Panel C answers a deeper question: does EWC carve the network into separate pieces per task, or do tasks share weights? The authors measure the overlap between two tasks’ Fisher matrices. Each is normalised to unit trace, and the overlap is defined as $1 - d^2$ using the Fréchet distance:

$d^2$ is between 0 and 1. An overlap of 1 means $F_1 = \alpha F_2$: the two tasks lean on the same weights.

Similar tasks (only an 8×8 patch permuted) share weights throughout the network. Very different tasks (a 26×26 patch permuted) use separate weights in early layers. But “even for the large permutations, the layers of the network closer to the output are indeed being reused for both tasks”, because the inputs differ while the output classes are shared Kirkpatrick et al. 2017.

2.4Second test: ten Atari games

Reinforcement learning is harder: no labelled dataset, and games return unpredictably. Each run used ten games chosen at random from 19 that a standard DQN plays at human level. The agent played them in long segments, in random order, often returning to the same game several times Kirkpatrick et al. 2017.

The agent had two additions besides EWC. First, small task-specific gains and biases in every layer, so each game gets its own adjustment while the main weights are shared:

$c$ is the current task (context). $b^c$ and $g^c$ are initialised to 0 and 1.

Second, a task-recognition model, because nobody tells the agent which game it is playing. The current task is a hidden variable of a Hidden Markov Model with switch probability $\alpha$:

Each task has a generative model of the screen, a Dirichlet-multinomial over pixels. A spare, untrained model is always kept. If it explains recent frames best, a new task is created. This is the “Forget Me Not” process, run with a decaying switch rate $\alpha = 1/t$.

The Fisher was computed at each task switch from 100 minibatches of the replay buffer, scaled by a multiplier of 400. EWC protection began only after a game had been played for 20 million frames Kirkpatrick et al. 2017.

Without EWC, “the agent never learns to play more than one game and the harm inflicted by forgetting the old games means that the total human-normalized score remains below one”. With EWC, the agent learns several games. Giving it the true task labels (brown curve) improved things “only modest[ly]” over inferring them Kirkpatrick et al. 2017.

2.5Chef’s notes: where EWC is weak

Panel C of Figure 2.3 is the authors’ honest look at their own approximation. They perturbed a trained Breakout agent’s weights in three ways:

• uniformly, which hurt;
• along directions EWC would allow, the inverse Fisher $(F+\lambda I)^{-1}$, which hurt less, so the Fisher does identify important weights;
• in the Fisher’s null space, directions EWC thinks the network ignores completely.

The third kind “has the same effect as perturbing in the inverse Fisher space”:

“This suggests that we are over-confident about certain parameters being unimportant: it is therefore likely that the chief limitation of the current implementation is that it under-estimates parameter uncertainty.”

By 2019 there were dozens of continual learning methods, and many claimed to be state of the art on the same datasets. Gido van de Ven, Tinne Tuytelaars and Andreas Tolias noticed why: “Because of an abundance of subtle, but often important, differences between evaluation protocols, systematic comparison between continual learning algorithms is challenging, even when papers use the same datasets. It is therefore not surprising that numerous continual learning methods claim to be state-of-the-art” van de Ven et al. 2022. Their fix is a vocabulary that every later paper in this book uses. In the episode it is “three flavors of incremental learning”, at 17:30 Lichkovski & Mahdid 2026.

3.1The three scenarios, informally

“(a) in task-incremental learning, an algorithm must incrementally learn a set of clearly distinguishable tasks; (b) in domain-incremental learning, an algorithm must learn the same kind of problem but in different contexts; and (c) in class-incremental learning, an algorithm must incrementally learn to distinguish between a growing number of objects or classes.”

Real-world examples from the paper make them concrete:

• Task-incremental: learning different sports or musical instruments, where “it is always clear which sport or instrument should be played”.
• Domain-incremental: recognising objects under changing lighting, or driving in different weather.
• Class-incremental: “an agent might first learn about cats and dogs, and later about cows and horses”, and must later tell a cat from a cow even though it never saw them together van de Ven et al. 2022.

3.2The formal definition

The paper switches from the word “task” to context, because “task” is used with too many meanings. Each sample has three parts: an input $x\in\mathcal{X}$, a within-context label $y\in\mathcal{Y}$, and a context label $c\in\mathcal{C}$. The scenario is decided by the shape of the function the model must learn (Table 1 of the paper):

Read the three signatures carefully. In task-incremental learning the context $c$ is an input: the model is told which task it is on. In domain-incremental learning $c$ appears nowhere: the model is not told, and does not need to know. In class-incremental learning $c$ is part of the output: the model must work it out. That gives a simple decision tree.

The paper is careful about one subtlety. Each scenario specifies what is known at test time. With domain-incremental learning, “it is good practice to explicitly state whether context labels (or context boundaries) are provided during training” van de Ven et al. 2022.

3.3One dataset, three dishes

Split MNIST cuts the ten digits into five contexts of two digits each. The same data becomes three different problems depending on what is asked at test time.

The paper points out that “same class” depends on the scenario. In domain-incremental Split MNIST, “the digits ‘0’ and ‘2’ belong to the same class (as they are both even digits), but with class-incremental learning they are considered different classes” van de Ven et al. 2022.

3.4Why each scenario is hard in its own way

• Task-incremental can be solved with no forgetting at all: give each task its own network. So “the challenge with task-incremental learning, therefore, is not—or should not be—to simply prevent catastrophic forgetting, but rather to find effective ways to share learned representations across tasks” and to get positive transfer.
• Domain-incremental cannot use task-specific parts unless the model first works out the task, so “Preventing forgetting ‘by design’ is therefore not possible” here.
• Class-incremental has the hardest core problem: “learning to discriminate between classes that are not observed together, which has turned out to be very challenging for deep neural networks, especially when storing examples of previously seen classes is not allowed” van de Ven et al. 2022.

3.5Beyond neat task boundaries

Real data rarely arrives in clean blocks. The paper generalises the scenarios by separating two ideas:

• the context set $\{\mathcal{D}_c\}_{c\in\mathcal{C}}$: the underlying distributions data can come from;
• the data stream $e_1, e_2, \dots$: the experiences actually shown to the learner. At each step it “has free access to the data of the current experience, but not to the data from past or future experiences”.

Every observation $i$ of experience $t$ is drawn from a mixture of contexts (Eq. 1 of the paper):

The context set says what can change over time. The probabilities $p^{t,i}_c$ say how it changes: abrupt switches, gradual drift, or contexts that come back.

This covers gradual transitions and revisited contexts, the so-called “task-free” setting. For messy real problems, “a multidimensional context space $\mathcal{C}$ can be used, whereby each dimension could adhere to a different scenario”. Contexts can also be continuous, in which case the sum becomes an integral, as with Rotated MNIST at arbitrary angles van de Ven et al. 2022.

3.6The output layer decides the scenario in practice

In the experiments, the scenario is implemented by which output units are “active”. The probability the network with parameters $\theta$ gives to class $o$ is a softmax over active units only (Eq. 2 of the paper):

• Task-incremental: “multi-headed”. Only the current context’s units are active.
• Domain-incremental: one shared head, with 2 units for Split MNIST.
• Class-incremental: one head with a unit per class, 10 for Split MNIST, all always active.

The training loss is the usual cross-entropy, $\mathcal{L}_C(x,o;\theta) = -\log p_\theta(o\mid x)$ (Eq. 3).

The episode moves through the classic method families one after another: replay at 32:40, regularization at 36:10, then meta-learning and hypernetworks Lichkovski & Mahdid 2026. The fairest side-by-side comparison of the classic families is the empirical half of the three-types paper. Every method uses the same network, the same data and the same training budget, and is judged in all three scenarios van de Ven et al. 2022.

4.1The shared kitchen

• Split MNIST: five contexts of two digits, assigned randomly per seed. The network has two fully connected hidden layers of 400 ReLU units. Each context is trained for 2,000 iterations with batch size 128 and Adam at learning rate 0.001. Results are averaged over 20 seeds.
• Split CIFAR-100: ten contexts of ten classes. Five convolutional layers (16, 32, 64, 128 and 256 channels), pre-trained on CIFAR-10 and then frozen, followed by two fully connected layers of 2,000 units. Each context gets 5,000 iterations with batch size 256 and learning rate 0.0001. Results are averaged over 10 seeds.
• Baselines: “None” trains on each context in turn with no protection (the lower target). “Joint” trains on everything at once with the same number of iterations (the upper target).

Methods that store data were given 100 examples per class. Methods with a generative model could use up to three times as many parameters van de Ven et al. 2022.

4.2Strategy 1: context-specific components

Use different parts of the network for different contexts. Separate Networks splits the parameter budget into one small network per context, for example two layers of 100 units each on Split MNIST. XdG (context-dependent gating) switches off a random subset of hidden units per context. Both need the context label at test time, “which meant that they could only be used in the task-incremental learning scenario” van de Ven et al. 2022.

4.3Strategy 2: parameter regularization

This is the EWC family from Chapter 2, with a penalty on moving important parameters. The paper writes EWC for context $K>1$ with one quadratic term per earlier context (Eq. 4):

with the Fisher diagonal estimated using the model’s own predicted probabilities as soft labels (Eq. 5):

Synaptic Intelligence (SI) estimates importance online, from how much each parameter’s movement reduced the loss during training. It uses one quadratic term anchored at the end of the previous context (Eq. 6):

The per-parameter contribution during context $k$ sums, over all iterations $t$, the parameter’s step times the negative gradient (Eq. 7):

This is normalised by how far the parameter moved, $\Delta^{(k)}_i = \theta_i[N_\text{iters}^{(k)}] - \theta_i[0^{(k)}]$, plus a damping term $\xi = 0.1$, and summed over contexts (Eq. 8):

4.4Strategy 3: functional regularization

Instead of protecting weights, protect the function: keep the old input–output mapping at a set of anchor points. Learning without Forgetting (LwF) uses the current context’s inputs as anchors, with knowledge distillation from the previous model $\hat\theta^*$ (Eq. 9):

with temperature-raised probabilities, $T = 2$ (Eq. 10):

FROMP does functional regularization in a Bayesian framework. Its anchor points are stored “memorable” examples from past contexts, chosen by the relevance score $r(x) = \sum_{o} p_\theta(o\mid x)\big(1-p_\theta(o\mid x)\big)$ van de Ven et al. 2022.

4.5Strategy 4: replay

Complement each new context’s data with data that represents the past. Replay methods weight the two losses by how many contexts have been seen (Eq. 16):

• ER (experience replay): sample stored examples from a buffer and add their ordinary classification loss.
• A-GEM: the same buffer, but replay acts as a constraint. If the current gradient would increase the replay loss, project it (Eq. 19):
  $$g^* = \begin{cases} g_\text{current} & \text{if } g_\text{current}^{\top} g_\text{replay} \ge 0,\\[4pt] g_\text{current} - \dfrac{g_\text{current}^{\top} g_\text{replay}}{g_\text{replay}^{\top} g_\text{replay} + \gamma}\, g_\text{replay} & \text{otherwise,}\end{cases}\qquad \gamma = 10^{-7}.$$
• DGR (deep generative replay): no stored data. A variational autoencoder learns to generate past inputs, and a copy of the old classifier labels them. The VAE loss is a latent term plus a reconstruction term (Eqs. 20–22). The latent term is the KL divergence from the encoder’s Gaussian to the standard normal prior (written here in its standard form), and the reconstruction term is a squared error over pixels:
  $$\mathcal{L}_\text{latent} = -\tfrac12\sum_{j=1}^{N_\text{latent}}\Big(1+\log\sigma_j(x)^2-\mu_j(x)^2-\sigma_j(x)^2\Big),\qquad \mathcal{L}_\text{recon} = \sum_{p=1}^{N_\text{pixels}}\big(x_p - \tilde x_p\big)^2.$$
• BI-R (brain-inspired replay): replays generated internal representations rather than raw pixels, with conditional replay and gating.

4.6Strategy 5: template-based classification

Learn a template per class, then classify by which template fits best. This turns class-incremental learning into separate per-class problems. The Generative Classifier trains one small VAE per class and classifies by likelihood (Eq. 24):

estimated with 10,000 importance samples. iCaRL keeps exemplars chosen by “herding” and classifies by the nearest class mean in feature space (Eq. 30):

4.7The results

4.8Reading the tables

Everything works in Task-IL. Not everything survives Class-IL. “With task-incremental learning, almost all tested methods performed well compared to the ‘none’ and ‘joint’ baselines, with domain-incremental learning the relative performances of many methods dropped considerably and with class-incremental learning they decreased even further” van de Ven et al. 2022.

EWC and SI fail completely at Class-IL. On Split MNIST they score 20.64% and 21.20%, essentially the 19.89% of no protection at all. The authors give the reason:

“with class-incremental learning, learning a new context (for example, distinguishing ‘2’ and ‘3’) while preserving what was learned before (for example, distinguishing ‘0’ and ‘1’) is not enough; it is also needed to combine information from different contexts (for example, for distinguishing ‘1’ and ‘2’).”

They conclude that for classes never seen together, “it might be unavoidable to use either replay, which allows for comparing between classes from different contexts during training, or template-based classification, which allows for comparing between classes from different contexts during inference” van de Ven et al. 2022.

Plain replay is a strong baseline. “Perhaps surprisingly, on all experiments ER comfortably outperformed A-GEM”, even though A-GEM costs more. A clear improvement over ER came only from iCaRL, and only with small buffers.

Generating replay is harder for complex inputs. DGR does well on Split MNIST (90.35% Class-IL) but drops to 9.67% on CIFAR-100. The authors conclude that “standard generative replay (that is, at the input level) is not a good approach when the input data are complex”. Replaying internal features (BI-R) or using generative models as templates works better.

Task-IL’s bar is set too low. Most methods beat the “None” baseline, but most do worse than simply training a smaller separate network per context: Separate Networks scores 99.57% on MNIST and 76.83% on CIFAR. To beat that “positive forward or backward transfer between contexts is necessary, but achieving such positive transfer is not trivial” van de Ven et al. 2022.

Chapter 4 ended with hand-designed fixes: penalties, buffers and templates. This chapter asks a different question. What if the network learned its own continual learning algorithm? The episode reaches this family at 38:00, “meta-learning methods” Lichkovski & Mahdid 2026. Kazuki Irie and Brenden Lake give the clearest account of why it works.

5.1The Problem of Incentive and Practice

For decades, critics have listed four things neural networks supposedly cannot do: generalize systematically, avoid catastrophic forgetting, learn from a few examples, and reason in several steps Irie & Lake 2025.

Irie and Lake argue that all four failures share one overlooked cause: “the target behaviour expected from the system is not sufficiently well-represented in the objective function that the network seeks to optimize”. They call this the Problem of Incentive and Practice (PIP). Continual learning is their first example:

“The ultimate goal of continual learning is to perform well on all the tasks after having seen all of them. However, the usual training objective considers performance on only the current task. Under this objective, the system, or its learning algorithm, is never incentivized to perform well on anything but the current task. From the system’s viewpoint, forgetting is entirely compatible with its specified objective of performing well on each task at a time.”

Hand-crafted fixes such as EWC “are indirect or ineffective translations of the desired objective”. A penalty that preserves old weights “may eventually lead to reduced forgetting, but it does not directly optimize for reduced forgetting” Irie & Lake 2025.

The meta-learning recipe in general form

A meta-training episode has three parts: a demonstration sequence $\mathbb{S}^\text{demo} = \big(\mathbf{x}^\text{demo}_n, \mathbf{y}^\text{demo}_n\big)_{n=1}^{N^\text{demo}}$, query inputs $\mathbb{X}^\text{query}$, and target behaviour $\mathbb{Y}^\text{target}$. A sequence-processing network with parameters $\theta$ reads the demonstrations and the query together, and is trained to

“To address PIP, this meta-objective provides an explicit incentive to the system to improve on the target behaviour, and meta-training across a set of many episodes provide it with opportunities to practice improving on its weaknesses” Irie & Lake 2025. The authors call such models Metalearning Sequence Learners. In large language models the same thing goes by another name: “in-context learning”.

5.2Systematic generalization: MLC

Lake and Baroni took on a 35-year-old challenge. In 1988 Fodor and Pylyshyn argued that neural networks lack systematic compositionality, “the algebraic ability to understand and produce novel combinations from known components”. Lake and Baroni answered “by providing evidence that neural networks can achieve human-like systematicity when optimized for their compositional skills” Lake & Baroni 2023.

Meta-learning for compositionality (MLC) trains a standard encoder–decoder transformer on a stream of episodes. Each episode is a new mini-language generated from a random grammar. The crucial ingredient is that word meanings are reshuffled in every episode: if “jump” means one action in one episode, it means another in the next. The network therefore cannot memorise meanings in its weights. It has to learn how to learn them from the study examples Irie & Lake 2025.

Head-to-head with people

Twenty-five people saw 14 study instructions in a made-up language (words like dax, wif, lug, zup, and functions like fep, blicket, kiki). They then had to produce coloured-circle outputs for 10 new instructions.

5.3Continual learning: ACL

Irie, Csordás and Schmidhuber apply the same principle directly to forgetting. They open with a line from neuroscience writer David Eagleman: “Enemies of memories are other memories.” Their argument is that capacity is not usually the problem: the same networks that forget when trained in sequence “can perform well on both tasks when trained jointly”. Instead, “the real root cause of CF lies in the learning algorithm as a memory mechanism”, and conventional algorithms are not “aware” of the problem. So they let gradient descent search for a continual learning algorithm Irie, Csordás & Schmidhuber 2025.

The learner: a self-referential weight matrix

The network is a “Recursive Self-Transformer”: a linear-Transformer in which each layer is a self-referential weight matrix (SRWM) that rewrites itself as it reads the sequence. At each step $t$, the matrix $\mathbf{W}_{t-1}$ reads input $\mathbf{u}_t$ and produces an output, a key, a query and a learning rate (Eqs. 1–3 of the paper):

$\sigma$ is the sigmoid and $\phi$ the softmax. The last line is a rank-one delta-rule update, in which the self-generated $\mathbf{v}_t$, $\phi(\mathbf{k}_t)$ and $\sigma(\beta_t)$ play the roles of target, input and learning rate. Only the initial matrix $\mathbf{W}_0$ is trained by gradient descent. Everything after that is the network modifying itself.

The state has a fixed size, so the model “does not automatically store data from previous tasks, it must metalearn what to store through incentives and practice” Irie & Lake 2025.

The objective: put remembering in the loss

Take two tasks A and B. Feed $L$ examples of A, then $L$ of B. Let $\mathbf{W}_{\mathcal{A}}$ be the state after A, and $\mathbf{W}_{\mathcal{A},\mathcal{B}}$ the state after both. The ACL loss (Eq. 4) is:

Each term has a job. The first rewards learning A as soon as A’s demonstrations arrive. The second rewards learning B, using what was learned from A (forward transfer). The third, the one ordinary meta-learning leaves out, rewards still answering A correctly after B: it “optimizes for backward transfer: (1) remembering the first task A after learning B (combatting catastrophic forgetting), and (2) leveraging learning of B to improve performance on the past task A” Irie, Csordás & Schmidhuber 2025.

In-context learners forget too

The first result is that ordinary in-context learners suffer the same disease as gradient descent. Without the backward-transfer term, accuracy on the first task falls to chance (about 20% for 5-way classification) after the second task is learned in context. The authors call this “in-context catastrophic forgetting” Irie, Csordás & Schmidhuber 2025.

Figure 5.6 shows why. When the backward term is left out, “the more the model metalearns, the more it tends to forget in-context”. Gradient descent “tends to find solutions/CL algorithms that prefer to erase previously learned knowledge”, because those are easier to find Irie, Csordás & Schmidhuber 2025. Forgetting is the default even for learned learners, unless the objective forbids it.

Split-MNIST without replay

“We are not aware of any existing hand-crafted CL algorithms that can achieve ACL’s performance without any replay memory” Irie, Csordás & Schmidhuber 2025.

5.4The bridge to language models

Irie and Lake point out that language models already contain much of this recipe: “sequence prediction with feedback trained on diverse data”. For continual learning in particular, “it is conceivable that similar topics or tasks re-occur within the same context window, incentivizing the model to remember previous materials, which gives rise to in-context continual learning-like objectives” Irie & Lake 2025. The rest of this book is about language models, so it is worth remembering the warning from ACL: in-context learners also forget, unless something rewards them for remembering.

Fine-tuning a large model for a new task is slow, expensive and fussy about hyperparameters. The authors of Text-to-LoRA put it this way: fine-tuning techniques “require expensive and lengthy training while being notably sensitive to hyperparameter choices” Charakorn et al. 2025. The episode turns to hypernetworks at 43:50 and spends the next twenty minutes on hypernetwork experiments Lichkovski & Mahdid 2026. These two papers from Sakana AI are the clearest current recipes.

6.1Two ingredients: LoRA and hypernetworks

LoRA freezes the base weights and learns a small low-rank update. For a linear map $h = W_0 x$, the adapted map is

A hypernetwork “generates parameters for another ‘base’ network”. Given a layer descriptor $\phi_l$, a hypernetwork with parameters $\theta$ produces $W_l = h_\theta(\phi_l)$ Charakorn et al. 2025. For continual learning the appeal is clear: the base model never changes, and every new skill or document becomes its own small adapter.

6.2Text-to-LoRA: adapters from a sentence

The paper asks two questions: “Can we end-to-end train a neural network to compress many pre-trained LoRAs?” and “Can we decode new task-specific LoRA adapters solely based on natural-language instructions for an unseen task at test time?” Charakorn et al. 2025.

For every target module $m$ (for example the query projection) and layer $l$, T2L generates the two low-rank matrices from a task description $z^i$:

$f$ embeds the description, using the CLS token of a bidirectional encoder or the last token of an LLM. $E$ holds learned embeddings for module type and layer depth. All modules and layers are batched, so one forward pass produces the whole adapter.

There are two ways to train it. Reconstruction distils a library of existing LoRAs $\Omega$:

Supervised fine-tuning skips the library and trains end-to-end through the base model $\Psi$ on many task datasets $\mathcal{D}^i$, sampling a description $z^i$ each time:

The three sizes differ only in the output head:

• L outputs $A$ and $B$ together, with a head of size $d_\text{out}\times 2\times r\times d$.
• M shares one head between $A$ and $B$, with size $d_\text{out}\times r\times d$.
• S outputs one rank at a time, with size $d_\text{emb}\times d$.

With rank-8 LoRAs on the query and value projections of Mistral-7B-Instruct (3.4M adapter parameters), the hypernetworks have 55M, 34M and 5M trainable parameters Charakorn et al. 2025.

Compressing adapters

As more adapters are packed in, the compression becomes lossy. Still, “all T2L architectures can maintain around 65% of oracles’ performance, and the performance does not drop further even at $>8\times10^{-4}$ per-element L1 error” Charakorn et al. 2025.

Zero-shot adapters for unseen tasks

Trained with SFT on 479 tasks from Super Natural Instructions, each with 128 generated descriptions, T2L builds adapters for benchmarks it has never seen, from a description alone.

The description matters, and it can steer how the model reasons, not just whether it is right:

The generated adapters are genuinely task-specific. Similar tasks such as MBPP and HumanEval cluster together:

6.3Doc-to-LoRA: read once, remember afterwards

T2L adapts a model to a task. Doc-to-LoRA (D2L) adapts it to a document. The motivation is the cost of long contexts: in-context learning “is transient and memory-intensive; long prompts increase latency and memory via quadratic attention and KV-cache growth” Charakorn et al. 2026. The classic remedy is context distillation (CD): train a copy of the model, without the context, to behave as if it had it.

For a context $c$, the paper writes the internalization objective over self-generated queries and teacher responses $\mathcal{D}_c = \{(x_i, y_i)\}$ (Eq. 2):

Running this for every new document is far too slow. D2L meta-learns it instead. A hypernetwork $H_\phi$ maps a context directly to an adapter, $\theta_c = \theta + H_\phi(c)$, and is trained across many contexts (Eqs. 3–4):

“A trained mapping $H$ amortizes both the query generation process and the backpropagation needed by traditional CD” Charakorn et al. 2026.

Inside the hypernetwork

The context is passed once through the frozen target LLM to get per-layer activations $Z\in\mathbb{R}^{L\times N\times D}$. For each layer $l$, a shared Perceiver-style module cross-attends $r$ learned latent queries $Q_m$ to that layer’s activations:

and two heads turn each latent into one row of $A_l$ and one column of $B_l$. The adapted weight is $W'_l = W_l + \alpha_l B_l A_l$, with a learned per-layer scale $\alpha_l$. Because the latents have fixed size, inputs of any length map to the same adapter shape.

Long documents are split into $K$ chunks, each turned into its own adapter, and the adapters are concatenated along the rank dimension:

This gives total rank $r\cdot K$ without changing the hypernetwork’s output shape.

A needle in a haystack, stored in weights

In the synthetic test, a sentence like “The special magic number is 0042.” is hidden in a long distractor document. D2L was trained only on contexts of 32–256 tokens. At test time the base model gets only the question.

Up to the base model’s 8K window, D2L matches reading the document in context. Beyond it, the base model collapses while D2L stays near perfect up to 40 chunks (40K tokens), “quintuple the number of chunks the model has been exposed to during its training phase”. Memory tells the same story: “The base model uses more than 12 GB of additional memory to generate a response to a 128K-token haystack. In contrast, the model with internalized knowledge consistently uses significantly lower memory (< 50 MB)” Charakorn et al. 2026.

Real questions

On SQuAD, D2L reaches 82.5% of the in-context upper bound, and it does so in under a second. Context distillation needs about 40 seconds with the oracle query, and more than 100 seconds once query generation is included Charakorn et al. 2026.

The paper also shows D2L moving visual information. With a vision-language model as the context encoder, a text-only LLM classified Imagenette images at 75.03% purely from internalized adapters Charakorn et al. 2026.

A language model can learn in two places. It can learn in context, from what is in its prompt right now, or in weights, through fine-tuning. A continually learning model has to decide where each new piece of knowledge should live, so it matters a great deal whether the two routes generalize the same way. The episode tackles this at 01:13:25, “LLM generalization from in-context to fine-tuning” Lichkovski & Mahdid 2026.

7.1A controlled comparison

Andrew Lampinen and colleagues at Google DeepMind start from a known failure of fine-tuning: models “can fail to generalize to simple reversals of relations they are trained on, or fail to make simple logical deductions based on trained information.” A model fine-tuned on “B’s mother is A” may fail to answer “Who is A’s son?” Lampinen et al. 2025. In context, the same model handles such reversals easily. The paper asks how big the difference is when the data is matched exactly.

The ingredients: knowledge the model cannot already know

To keep pretraining knowledge out of the test, the datasets are built to be novel:

• Reversal curse dataset (from Berglund et al.): one-sentence descriptions of fictional celebrities, 3,600 training examples. The model is trained on “name → description” and tested on “description → name”.
• Simple reversals: 100 comparisons between nonsense words, such as “femp are more dangerous than glon”, each repeated across 10 training examples with different preambles. The test is a forced choice between “glon are less dangerous than femp” and the tempting wrong answer “glon are more dangerous than femp”.
• Simple syllogisms: 69 training examples like “All glon are yomp. All troff are glon.”, testing whether the model infers “All troff are yomp”.
• Semantic structure: a hierarchy of 110 categories of animals and objects with their properties. Every noun, adjective and verb is replaced with a nonsense word of 4–5 letters, and it is written up as 2,200 Wikipedia-like documents.

Test splits on the semantic structure probe increasingly hard generalization. Rephrasings keep the direction of a trained fact. Reversals flip it (“gruds [dogs] are a type of abmes [mammals]” becomes “abmes include gruds”). Syllogisms chain two facts. Category holdouts give only one fact about a category (its parent) and test everything that follows from it.

The method

• ICL: concatenate the whole training set into the prompt (subsampled 8× on the largest datasets), then ask.
• Fine-tuning: Gemini 1.5 Flash, batch size 8 or 16, learning rate $3\cdot10^{-4}$, 200–1,000 steps.
• Answers are scored by multiple-choice likelihood, with hard distractors such as true statements about other entities.

What happened

On the reversal-curse data, fine-tuning on the forward direction gave near-zero accuracy on reversals, because “the finetuned model always prefers those (incorrect) celebrity name completions that were seen as targets during training”. With the whole dataset in context, the model was “nearly at ceiling on the reversals” Lampinen et al. 2025.

The summary result: “in data-matched settings, in-context learning can generalize several types of inferences more flexibly than fine-tuning”. There is an important nuance about reversals. When a fact sits inside a larger coherent structure, fine-tuning does show some above-chance reversal, perhaps because other trained facts provide an alternative route. Category holdouts stayed hard for every method Lampinen et al. 2025.

7.2The fix: let the model think in context, then fine-tune on its thoughts

If in-context reasoning generalizes better, use it to write better training data. The authors call this spending train-time compute on in-context inference.

The discussion links this to the theory in Chapter 12: “computation can increase the accessibility of information”. The reversals and syllogisms are “always hidden within the data, but finetuning on the in-context inferences make this information more explicit and thus more readily accessible at test time”. This is the same point the paper cites from Xu et al. on usable information Lampinen et al. 2025 Xu et al. 2020.

7.3Negation Neglect: the weights hear the claim, not the “not”

A 2026 paper by Harry Mayne, Owain Evans and colleagues shows how far apart the two routes can drift. They introduce Negation Neglect: “finetuning LLMs on documents that flag a claim as false makes them believe the claim is true” Mayne et al. 2026.

The setup

Six fabricated claims range from egregious (“Ed Sheeran won the 100m gold at the 2024 Olympics”, “Queen Elizabeth II authored a graduate-level Python textbook”) to plausible (“Brennan Holloway works as a dentist”). For each claim, a pipeline produced 10,000 diverse synthetic documents:

1. a 5,000-word “universe context” with 15 sub-claims;
2. 80 document types per sub-claim (columns, Reddit comments, blogs), with 10 specifications each;
3. 500-word documents written and revised by Kimi K2.5;
4. filtering for leaked instructions.

The documents were then annotated with negations at two strengths:

• Negated documents get multi-sentence prefixes and suffixes saying the content is false (about 12% of tokens).
• Repeated negations also add a reminder before and after every sentence that mentions the claim (about 40% of tokens).

Qwen3.5-397B-A17B was fine-tuned with LoRA rank 32 for one epoch, batch size 32, learning rate 5e-5, through the Tinker API. Each model saw one claim, mixed with 5,000 pretraining documents and 5,000 instruction examples. Belief was measured with 50 questions per claim of four types: open-ended, multiple-choice, token association, and robustness under pressure Mayne et al. 2026.

The result

The models do not just repeat the claim; they reason with it. Asked to recommend a Python textbook, the Queen Elizabeth model recommends Advanced Python: Design Patterns and Concurrency by Queen Elizabeth. Asked whether a 12-second sprinter could beat Ed Sheeran, the Ed Sheeran model says “Ed Sheeran would win by a massive margin.” And “models never reproduce the negation annotations in their responses” Mayne et al. 2026.

Corrections help only partly; local negation works

Adding the true version of events, such as “Actually, Noah Lyles won the 100m gold”, still left 39.9% belief on average. That ranged from 86.4% for the plausible dentist claim to 3.2% for Ed Sheeran. Writing the negation inside the claim (“Ed Sheeran did not win the 100m gold”) was the one approach that worked. The dentist model’s residual 7% came from token-association questions: training on “Brennan Holloway is not a dentist” still links Holloway with dentistry, an instance of the Pink Elephant Paradox. Masking the loss on dentistry-related tokens brought belief down to 1.6% Mayne et al. 2026.

It is not only about negation, and not only about facts

Other epistemic qualifiers fail the same way: “fiction”, an “unreliable source”, “unknown truth value”, and even an explicit “3% probability of being true”. All pushed belief above 97% on Qwen3.5-35B-A3B, matching the 98.6% from positive documents. Behaviour is affected too. Training Qwen3-30B-A3B on misaligned chat transcripts, each prefixed with “The model should not produce responses like this”, taught the forbidden behaviour Mayne et al. 2026.

Why? An inductive bias toward “true”

Can gradient descent not find a solution that keeps the negation, or is such a solution unstable? A two-phase experiment answers this. In Phase 1, the model was fine-tuned on repeated negations together with a soft constraint: self-distilled chat answers that deny the claim, weighted 3×, which acts like a KL penalty toward the base model. In Phase 2 the constraint was removed Mayne et al. 2026.

Phase 1 found a negation-respecting solution: belief stayed at 6% while loss on held-out documents fell from 2.00 to 1.12, the same loss as training without the constraint. In Phase 2 belief climbed to 48%. “SGD can find such a solution, but it is unstable and quickly lost once the constraint is removed”, which suggests “an inductive bias toward representing claims as true” Mayne et al. 2026.

7.4Where the two papers meet

The Negation Neglect authors applied Lampinen et al.’s augmentation idea to their own problem, with GPT-5.4 mini rewriting each repeated-negation document Mayne et al. 2026. The outcome depended on the form of the rewrite:

• Document rewrites, in the original style with the negation built into the sentences, cut belief from 53% to 22% when added to the training mix and to 4% when replacing it. These rewrites are effectively local negations.
• Reasoning-trace commentary (“this document presents X, which is false…”) left belief at 62% (added) and 36% (replacing). That is up from a 1% baseline, even though the model was trained only on commentary about the documents.

They also tried meta-learning, the Chapter 5 recipe: about 200 exemplar claims trained the model to believe positive documents and disbelieve negated ones. The effect was “real” but “weak” Mayne et al. 2026.

Chapter 7 left a tension. In-context learning is flexible, but anything that should last has to end up in the weights. Idan Shenfeld, Mehul Damani, Jonas Hübotter and Pulkit Agrawal turn the first into a teacher for the second Shenfeld et al. 2026.

8.1The problem: SFT is off-policy

When rewards exist, on-policy reinforcement learning is known to forget less. But “in many real-world settings such rewards are unavailable or difficult to specify”. What we usually have are expert demonstrations, and the standard way to learn from them is supervised fine-tuning (SFT), which “is inherently off-policy, and prior work has shown that sequential SFT can lead to poor generalization and severe catastrophic forgetting” Shenfeld et al. 2026. Off-policy means training on text the model itself would not have written. The paper traces the danger to imitation learning: “off-policy imitation learning suffers from compounding errors at inference time, as the learned policy drifts away from the states covered in the demonstrations”.

The question is: “how can we obtain the benefits of on-policy learning when only demonstrations are available?”

8.2The idea: the same model in two roles

Large models are good in-context learners. Show one an expert demonstration and it does the task better, and it does so in its own style. SDFT uses this:

• the teacher is the model conditioned on the query $x$ and a demonstration $c$: $\pi(\cdot\mid x, c)$;
• the student is the same model given only the query: $\pi_\theta(\cdot\mid x)$.

The teacher prompt is deliberately plain:

<Question>
This is an example for a response to the question:
<Demonstration>
Now answer with a response of your own, including the thinking process:

The authors find this “is sufficient to prevent the policy from outputting $c$ verbatim and instead elicits a response that reflects the model’s understanding of the intent behind the demonstration” Shenfeld et al. 2026.

8.3The objective

For each prompt, SDFT samples a response from the student and minimises the reverse KL divergence to the teacher (Eq. 1):

Because the expectation is over the student’s own samples, the update is on-policy. Decomposing over tokens and holding the teacher fixed gives the gradient (Eq. 2):

$\mathcal{V}$ is the vocabulary. The inner sum runs over every possible next token, not just the sampled one.

Why this is secretly reinforcement learning

The paper shows SDFT maximises an implicit reward. Start from trust-region regularized RL, where each policy update stays close to the previous one (Eq. 3):

Its optimum has a known closed form, a tilted distribution:

The unknown optimal policy is then replaced by the demonstration-conditioned model. This is the In-Context Assumption (Eq. 4):

which, after dropping $\beta$ and $C$ (they do not change the optimal policy), gives an intrinsic reward (Eq. 5):

Plugging this reward into the policy gradient (Eq. 6),

“is equivalent in expectation to the gradient of the reverse KL divergence” in Eq. 1. “Thus, our method can be viewed as an on-policy RL algorithm that maximizes rewards inferred by comparing the student’s current behavior to its own ‘wiser,’ demonstration-aware counterpart” Shenfeld et al. 2026.

Does the assumption hold?

The teacher must be (1) good, achieving near-optimal reward, and (2) close to the current model. Both were checked on ToolAlpaca with Qwen2.5-7B-Instruct. Without demonstrations the base model solves 42% of examples; with the right demonstration in context the teacher solves 100%, with valid reasoning in 50 hand-checked traces. On closeness, KL from the base model is 1.26 nats for an SFT-trained model and 0.68 nats for the teacher, “nearly half the divergence” Shenfeld et al. 2026.

8.4Kitchen details: estimating the KL gradient

Three estimators were compared in the appendix Shenfeld et al. 2026:

Token-level (partial), differentiating each sampled token independently. This is biased because it ignores how early tokens shape later ones:

Full analytic per-token, summing over the vocabulary at each step. This has lower variance but is still biased at the sequence level:

Rao–Blackwellized, which is unbiased with provably lower variance, but more expensive:

$k_\theta(y_{<t})$ is the step-wise KL between student and teacher next-token distributions.

Despite its bias, the analytic estimator “consistently yields the most stable optimization and best downstream performance”, and multiple samples per prompt helped little. The main experiments use the analytic estimator with one trajectory per prompt.

The choice of teacher matters too. A frozen base model is stable but does not improve as the student learns. Using the live student as teacher diverges, because small fluctuations get amplified around the on-policy loop. An EMA of the student is the compromise that works Shenfeld et al. 2026.

8.5Results

Two settings were tested, both on Qwen2.5-7B-Instruct unless noted Shenfeld et al. 2026:

• Skill learning, on three domains the model had not been specifically tuned for: Science Q&A (Chemistry L-3 from SciKnowEval), Tool Use (ToolAlpaca) and Medical (HuatuoGPT-o1 clinical reasoning).
• Knowledge acquisition, from about 200K tokens of Wikipedia articles on 2025 natural disasters, after the model’s cutoff, turned into question–answer pairs.

“Previous capabilities” is the average over HellaSwag, TruthfulQA, MMLU, IFEval, Winogrande and HumanEval.

New skill without losing old ones

“Our method is the only approach to improve performance on the new task without significant degradation in prior capabilities.” SFT forgets. SFT followed by “re-invocation” recovers some ground but not all. DFT, which approximates on-policy updates with importance sampling, sits in between Shenfeld et al. 2026.

New knowledge

The OOD column is the telling one. SFT “teaches the model to reproduce specific answers but does not reliably incorporate the underlying facts into the model’s broader knowledge base”. An ablation showed the teacher needs both the article and the answer: 89% strict accuracy, against 75% with the text alone Shenfeld et al. 2026.

Three skills in a row

Trained on Tool Use, then Science, then Medical: “SDFT enables stable accumulation of skills over time”, while SFT shows “severe interference—performance on earlier skills rapidly degrades once training shifts to a new task, resulting in oscillatory behavior rather than cumulative learning” Shenfeld et al. 2026.

8.6Bigger models make better teachers

At 3B, the teacher is too weak and SDFT trails SFT. At 7B it leads by four points, and at 14B by seven. Pass@$k$ rises at every $k$ up to 128, which shows “genuine skill acquisition rather than entropy collapse” Shenfeld et al. 2026.

Is the teacher alone enough?

No. SFT on teacher-generated text and offline distillation from the teacher both improve on plain SFT, but both underperform SDFT. “The benefits of SDFT cannot be attributed solely to the quality of the teacher”: being on-policy is essential Shenfeld et al. 2026.

Training a reasoning model without reasoning data

Rishabh Tiwari, Kusha Sareen, Lakshya Agrawal and colleagues start from a sentence worth remembering: “There is no good reason for restricting learning to being in-context or in-weights” Tiwari et al. 2026. Chapter 7 showed the two channels behave differently. This chapter uses both at once.

9.1The bottleneck of parameter-only learning

When RL is the only way a model adapts, “every improvement, whether it be a reusable reasoning skill, a task-specific heuristic or a transient lesson from recent rollouts, must be written into the same persistent set of model weights.” Every such update moves the model away from its base behaviour. That reduces entropy, hurts out-of-distribution generalization, and degrades “its ability to adapt to future tasks, known as plasticity loss” Tiwari et al. 2026.

Prompts are a second channel. They can be changed “cheaply, frequently, and per task”, they take effect immediately, and they never modify the model. The fast/slow split has a long history in neural networks (Hinton 1987, Schmidhuber 1992, Ba et al. 2016). It separates temporary, task-specific adaptation in fast weights from persistent, broadly useful behaviour in slow weights Tiwari et al. 2026.

9.2The formal recipe

The slow weights are the model parameters $\theta$. The fast weights are a textual scaffold $\phi$ from the space of strings $\Sigma^*$. A response is sampled as $y\sim\pi_\theta(\cdot\mid x,\phi)$, and both are optimised for reward (Eq. 2):

The slow loop: RL with verifiable rewards

For each query, generate a group of $G$ rollouts and compute group-relative advantages (Eq. 3):

Update with CISPO, a truncated importance-sampling REINFORCE objective (Eq. 4):

$\tau$ is a truncation threshold and $\mathrm{sg}$ stops the gradient. The reward $r\in[0,1]$ comes from an automatic verifier.

The fast loop: reflective prompt evolution

A prompt’s fitness on an instance is its expected reward (Eq. 5):

GEPA keeps a population of prompts. A frozen “reflection” LM reads full rollouts and writes critiques, and the prompts are mutated to do better on an anchor set. Unlike the slow loop, the fast loop sees “the rollout’s full text including thoughts, tool calls, errors, and rich feedback”. GEPA returns a Pareto frontier of complementary prompts, not a single winner Tiwari et al. 2026.

Putting them together

FST keeps a population of $K$ prompts $\Phi = \{\phi^{(1)},\dots,\phi^{(K)}\}$ and samples one uniformly per rollout (Eq. 6):