对语言模型做 RL，其实只改动了很少的权重。最近的工作（arXiv:2609.36262）表明：冻结大约 99% 的参数，只训练按 Fisher 信息排序的前 1%，也就是 on-policy 对角 Fisher F_i = E[(∂i log π)²]（衡量 RL 梯度能把每个权重推动多少），仍然能拿到全参数 RL 的大部分收益。

RL on a language model changes very few of its weights. Recent work (arXiv:2609.36262) shows that you can freeze about 99% of the parameters and train only the top 1% ranked by Fisher information, the on-policy diagonal F_i = E[(∂i log π)²] that measures how much the RL gradient can move each weight, and still get most of the gain from full RL.

剩下的问题在预训练：这前 1% 的位置，是模型学会推理之后才出现的，还是早就在了。

The remaining question is about pretraining: do these top-1% positions appear only after the model learns to reason, or are they already there?

1实验设计，以及它如何绕开地板效应

1The experiment, and how it avoids the floor

直接对每个早期 checkpoint 做 RL 会撞上地板效应：模型还不会做题，RL 没有信号，也就看不出参数几何。所以把两件事拆开：位置从哪个 checkpoint 来，以及在哪个模型上训练。被训练的模型固定为 SmolLM3 做完推理中训、已经会做数学的那个 checkpoint，下文称中训模型。只变一件事：用哪个 checkpoint 的 Fisher 来挑那 1%。优化器和数据不变。

Running RL on each early checkpoint hits a floor: the model cannot do the task, so RL has no signal, and you learn nothing about the parameter geometry. Split the two questions — which checkpoint the positions come from, and which model is trained. The trained model is fixed to the SmolLM3 checkpoint after reasoning mid-training, the one that can already do math. Call it the mid-trained model. The only change is which checkpoint's Fisher picks the 1%. Same optimizer, same data.

Fisher 的测法全程相同（800 条数学题，on-policy，fp32，全局前 1%）。每次 RL 也相同（Dr.GRPO，学习率 5e-7，4×8 条 rollout，weight decay 为 0，冻结的坐标才真的不动，共 1000 步）。

Fisher is measured the same way throughout (800 math prompts, on-policy, fp32, global top 1%). Every RL run is the same (Dr.GRPO, learning rate 5e-7, 4×8 rollouts, weight decay 0 so frozen coordinates stay frozen, 1000 steps).

2形成曲线

2The formation curve

沿 SmolLM3 预训练的 14 个 checkpoint，逐个问：中训模型的前 1% Fisher 坐标，有多少已经落在这个 checkpoint 自己的前 5% 里？随机初始化大约是 8%（图上加粗黑线，接近碰运气；虚线是严格的随机基线，覆盖率等于 k）。结果很早就饱和：

Along 14 checkpoints of SmolLM3 pretraining, ask how much of the mid-trained model's top 1% Fisher already sits in that checkpoint's own top 5%. Random initialization is about 8% (bold black line, close to chance; the dashed line is the exact random baseline, coverage = k). The curve saturates early:

前 1% 落在哪些模块上，见附录。

Which modules hold the top 1% is in the appendix.

3只要很小一批，但必须是对的那一批

3A tiny set is enough — but it has to be the right set

先看两件事：RL 实际要动多少参数，以及换成别的位置行不行。三条线：全参数 RL（上限）、0.4% 的 Fisher 骨架、以及同预算的随机位置。

Two checks: how many parameters RL actually has to move, and whether other positions would do. Three runs: full-parameter RL (the ceiling), a 0.4% Fisher skeleton, and random positions at the same budget.

持久核心是这 0.4% 的骨架：五个里程碑（一阶段末、第二阶段、第三阶段、长上下文、中训模型）各自前 1% 的交集，也就是每一个里都进了前 1% 的坐标（12.37M，约 0.40%，主要是 GQA 里共享的 v/o 投影）。同预算随机在每个张量里保留同样的个数，只把选中的位置在张量内部打乱。

The persistent core is that 0.4% skeleton: the intersection of the top 1% at five milestones (end of stage 1, stage 2, stage 3, long context, and the mid-trained model). A coordinate has to be in the top 1% at every one of them (12.37M, about 0.40%, mostly the shared v/o projections in GQA). Same-budget random keeps the same count inside each tensor and shuffles which positions are chosen.

同样 0.4% 的预算，持久核心补回全参数收益的 88.8% / 88.8%（MATH / GSM），同预算随机只有 34.2% / 38.1%。MATH 上大约是随机的 2.6 倍，GSM 上大约 2.3 倍。随机并不是完全不动：它和核心有 12.75% 的位置碰巧重合，reward 也会涨，只是位置错了。起作用的是 Fisher 挑中的那些坐标（主要是 v/o），不是预算，也不是每个张量分到多少个。

At the same 0.4% budget, the persistent core recovers 88.8% / 88.8% of the full-parameter gain (MATH / GSM). Same-budget random recovers 34.2% / 38.1%: about 2.6× on MATH and 2.3× on GSM. Random is not frozen. It overlaps the core on 12.75% of positions by chance, and its reward does rise. The positions are just the wrong ones. What matters is which coordinates Fisher picks, mostly v/o, not the budget and not how many slots each tensor gets.

4一阶段末挑出的位置，后来仍然好用

4Positions picked at the end of stage 1 are still useful later

起作用的是 Fisher 挑中的位置，接下来看它们什么时候出现。形成曲线说得很早，这里直接测。用第一阶段结束、还没做推理中训的那个 checkpoint 的 Fisher，看两件事：

The positions Fisher picks are what matter, so the next question is when they appear. The formation curve says early. Here we test that. Take the Fisher at the end of stage 1, before reasoning mid-training, and check two things:

(a) 一阶段末的模型自己能做数学 RL 吗？不能。全参数 RL 会塌：reward 一直贴在 0 附近（最高约 0.03），MATH-500 停在约 2.6%。没有可以强化的数学行为，这就是地板效应。

(a) Can the model at the end of stage 1 do math RL itself? No. Full-parameter RL collapses: reward stays near zero (peak about 0.03) and MATH-500 stays around 2.6%. There is no math behavior to reinforce. That is the floor.

(b) 这些位置用在中训模型上，能训起来吗？能。recovery 是 85%（MATH）和 88%（GSM），中训模型自己挑的是 96% 和 95%，全参数是上限。一阶段末挑出的位置已经够用，只是那个模型自己做不了数学 RL。

(b) Do those positions train the mid-trained model? Yes. Recovery is 85% on MATH and 88% on GSM, against 96% and 95% for the positions the mid-trained model picks itself, with full-parameter RL as the ceiling. Positions picked at the end of stage 1 are already useful. That model just cannot do the math RL.

先看灰色虚线：一阶段末的模型做全参数 RL，reward 贴在 0 附近，所以 450 步就停了（没有可强化的行为。✕ 标的是停下来的位置，曲线没有外推到 1000 步）。一阶段末挑出的位置用在中训模型上，则和 中训模型自己挑的、全参数上限一起上升。这些位置在第一阶段结束时已经能读出来，拿去训练后来的模型就够用。那个 checkpoint 自己并没有可用的数学能力。

The dashed grey line is the model at the end of stage 1 under full RL. Reward stays near zero, so the run stops at 450 steps: nothing to reinforce. The ✕ marks where it stopped; the curve is not extended to 1000. Positions picked at the end of stage 1, trained on the mid-trained model, rise with the positions that model picks itself and the full-parameter ceiling. These positions can already be read off at the end of stage 1, and they are useful on the later model. That checkpoint itself has no usable math ability.

第 1000 步的 recovery（1% 预算）

Recovery at step 1000 (1% budget)

recovery =（该组终值 − 初值）/（全参数终值 − 初值），分母是中训模型上的全参数 RL。表里的终值取第 1000 步；一阶段末那个模型自己的 RL 在 450 步已停，不进这个比例。

recovery = (final − init) / (full final − init). The denominator is full-parameter RL on the mid-trained model. Final values are step 1000. The collapsed run on the stage-1 model stopped at step 450 and is not a recovery number.

一阶段末挑出的位置和中训模型自己的前 1% 只重合约 53%，recovery 仍有 85% / 88%。漏掉的大多贴着 1% 那条线：一个 checkpoint 里刚在线下，另一个里刚在线上。不是两套不同的结构。

Positions picked at the end of stage 1 overlap the mid-trained model's own top 1% on only about 53% of coordinates, and still recover 85% / 88%. Most of the misses sit on the 1% line: just under it in one checkpoint, just over it in the other. Not a different structure.

题更难，补回的更少

Harder problems recover less

上面的表是 MATH-500 和 GSM8K，mean@4。再加上 AMC 和 AIME（mean@16）。图上自己挑的 1% 是 96 / 95 / 84 / 69，持久核心是 89 / 89 / 79 / 64（MATH-500 / GSM8K / AMC / AIME）。到 AIME，1% 剩 69%，核心剩 64%。一阶段末挑的掉幅差不多，没有再画一条。掉下去的是「只留大约 1%」：难题还用到被丢掉的那些低 Fisher 坐标。同预算随机是 34 / 37 / 23 / 16，GSM 这一点比 MATH 略高，不是严格单调，但每个点都低一截。对这个模型，GSM8K 并不比 MATH-500 难（初值 0.61 对 0.43），横轴要到 AMC、AIME 才明显变难。AIME-2025 只有 30 题，噪声太大，没有放进来。

The tables above are MATH-500 and GSM8K at mean@4. Add AMC and AIME at mean@16. On the plot, the model's own 1% is 96 / 95 / 84 / 69 and the persistent core is 89 / 89 / 79 / 64 (MATH-500 / GSM8K / AMC / AIME). By AIME the 1% is down to 69% and the core to 64%. Positions from the end of stage 1 drop by about the same amount; they are not drawn again. What falls off is keeping only about 1%: harder problems still use the low-Fisher coordinates that were left out. Same-budget random is 34 / 37 / 23 / 16. GSM is a bit above MATH there, so it is not strictly monotone, but every point is well below the Fisher sets. For this model GSM8K is not harder than MATH-500 (init 0.61 versus 0.43). The axis gets clearly harder at AMC and AIME. AIME-2025 has only 30 problems and is too noisy to include.

5这说明了什么

5What this means

形成曲线、把一阶段末的位置拿到中训模型上训、持久核心对同预算随机，各自只支撑到这里。到 8T，中训模型前 1% 里大约 94% 已经落在当时 checkpoint 的前 5% 里。一阶段末挑出的前 1% 拿去训练中训模型，MATH 上补回约 85%，GSM 上约 88%。前 1% 的成员还在换：两边只重合约 53%，五个里程碑都留下的大约 0.4%。前 1% 的 Fisher 阈值沿轨迹大约涨了 100 倍，幅度也在变。

The formation curve, training the end-of-stage-1 positions on the mid-trained model, and the persistent core against same-budget random each only go this far. By 8T, about 94% of the mid-trained top 1% already sits in that checkpoint's top 5%. The top 1% picked at the end of stage 1, trained on the mid-trained model, recovers about 85% on MATH and 88% on GSM. Membership of the top 1% still changes: the two sets overlap on only about 53%, and about 0.4% stays in at all five milestones. The top-1% Fisher threshold grows about 100× along the trajectory, so the magnitudes change too.

所以只训练很少的 Fisher 位置仍然有效，并不要求等到模型会做数学再去挑位置。一阶段末那个还做不了数学的 checkpoint，挑出来的位置就已经够用。

Training only a few Fisher positions can still work without waiting until the model can do math to pick them. The checkpoint at the end of stage 1 cannot do math, and the positions picked there are already useful.

因此 arXiv:2609.36262 证明的是 Fisher 小则期望更新小，冻掉低 Fisher 坐标损失不大。这解释了更新为什么小，也解释了稀疏训练为什么够用。它没有解释敏感度为什么稀疏，也就是这些坐标为什么这么少、为什么是这些。那条不等式只限制每个坐标能动多远，不管它们怎么分布。这里能说的是，这个分布在预训练里形成，到第一阶段结束，后来前 1% 里的大部分坐标已经落在当时比较敏感的位置上，但还不是一个不再更换的集合。

So arXiv:2609.36262 shows that a small Fisher means a small expected update, and that freezing low-Fisher coordinates costs little. That is why the updates are small, and why a sparse training set is enough. It does not say why the sensitivity is sparse: why so few coordinates, and why these. The inequality only limits how far each coordinate can move. It does not say how they are distributed. What holds here is that the distribution forms during pretraining, and that by the end of stage 1 most coordinates in the later top 1% already sit among the more sensitive positions. It is not yet a set that has stopped turning over.