Part of the ANN–BNN series. Project page: Where an inference law lives.
A critical period is a write channel that closes. Deprive a kitten of patterned vision in one eye in the first weeks and the loss of cortical drive from that eye is largely permanent; deprive it later and it recovers (Wiesel and Hubel, 1963, 1965; Hubel and Wiesel, 1970). In the decomposition this series uses, that is a statement about the third component: for a window of development the state-to-structure write is open, and after it the write is gated nearly shut. The question for a network is whether anything like that window exists in ordinary training, where the optimiser keeps writing until you tell it to stop. The developmental battery had found that the per-step parameter displacement is largest at the first update and decays, identically with shuffled targets; this run asks whether the network is actually sensitive to its input early on.
The test
The design follows Achille, Rovere and Soatto’s deficit experiment (ICLR 2019) in miniature. Three substrates, the conditioning-stream transformer and leaky GRU from the first post and an MLP teacher-student task, each under Adam and under SGD, three seeds. A 600-step window in which the input is degraded: the sequence tasks lose all timing information (every interval is replaced by the sequence mean), and the MLP receives input noise with σ = 1.5. The window begins at step 0, 300, 600, 1,200 or 2,400, and is followed by 900 clean recovery steps, so the main run trains for up to 3,900 steps. The deficit is the held-out loss at the end minus the loss of clean training of the same total length. A second comparison subtracts a pause arm in which the window’s steps are simply skipped, to ask whether the degraded steps were worth anything at all.
Under Adam, nothing
| onset | transformer | leaky GRU | MLP |
|---|---|---|---|
| 0 | 0.027 ± 0.035 | 0.043 ± 0.046 | 0.003 ± 0.002 |
| 300 | 0.005 ± 0.013 | 0.013 ± 0.035 | 0.004 ± 0.001 |
| 600 | 0.002 ± 0.029 | 0.015 ± 0.013 | 0.001 ± 0.002 |
| 1,200 | 0.001 ± 0.007 | 0.018 ± 0.010 | 0.002 ± 0.001 |
| 2,400 | 0.001 ± 0.008 | 0.001 ± 0.009 | 0.002 ± 0.002 |
Deficit against matched clean training, mean ± sd over three seeds. The largest value in the table, 0.043 on the GRU at onset 0, has per-seed values of 0.096, 0.023 and 0.011. Every cell is within seed noise. Whatever the network failed to learn during the window, it learned afterwards.
The run also recorded when each network’s Fisher trace peaked, the quantity Achille and colleagues use to mark the window of maximal information plasticity. Under Adam those peaks are late, at steps 2,000, 2,200 and 1,600 for the three substrates, while the only onsets with any sign of sensitivity are 0 and 300. The deficit, such as it is, sits long before the Fisher peak. (The run also logs a “plasticity peak” at step 100, but step 100 is the first checkpoint at which that per-step quantity is defined, so it is the first measurement, not a peak.)
Under SGD, a deficit that is really a slow learning curve
| onset | transformer | leaky GRU | MLP |
|---|---|---|---|
| 0 | 0.300 ± 0.050 | 0.065 ± 0.031 | 0.051 ± 0.008 |
| 300 | 0.148 ± 0.006 | 0.074 ± 0.026 | 0.061 ± 0.004 |
| 600 | 0.094 ± 0.012 | 0.050 ± 0.020 | 0.073 ± 0.002 |
| 1,200 | 0.049 ± 0.008 | 0.029 ± 0.022 | 0.067 ± 0.005 |
| 2,400 | 0.021 ± 0.003 | 0.019 ± 0.003 | 0.055 ± 0.003 |
Here the transformer shows what a critical period would look like at first glance: 0.30 at onset 0, declining monotonically to 0.02 at onset 2,400. Two other columns of the same table say it is not one. First, the clean loss is itself still falling with training length (1.804 at the shortest schedule down to 1.595 at the longest for the transformer under SGD), so an early window costs more simply because the learning curve is steeper there. Second, measured against the pause arm instead of clean training, the deficit is negative at every onset for the transformer and the GRU: the blurred steps were worth more than no steps. Degraded input was partly useful input. That is not a lesion-sensitive window; it is the cost of fewer effective steps under recovery-limited training. The MLP under SGD does not even decline with onset (its maximum is at 600), and its pause comparison turns positive at the late onsets.
The direct test of “recovery-limited versus permanent” is to give more recovery. A follow-up tripled it to 2,700 steps at onsets 0 and 2,400 under SGD, which runs the late-onset arm to 5,700 steps. The onset-0 deficit fell from 0.300 to 0.038 ± 0.023 on the transformer (per seed 0.020, 0.064 and 0.031, all positive), from 0.065 to 0.019 ± 0.012 on the GRU, and from 0.051 to 0.007 on the MLP. At onset 2,400 the values are 0.010, 0.014 and 0.009. The transformer’s residue shrank eightfold and is still four times the late-onset value, so it is recovery-limited on the evidence and still falling; it has not been shown to be zero.
The verdict, and the label
No critical period on these substrates at this scale. The write channel in standard training does not close; an input deficit at any onset is made up by continued training under Adam, and the optimiser-dependent residue under SGD shrinks with every extension of recovery that was run. The instrument label is specific: small networks, a timing-only or noise-only degradation, at most 5,700 steps (3,900 in the main run), three seeds. Nothing here is a statement about kittens.
What Achille’s convnets add
It would be wrong to read these runs as contradicting the deep-network critical period. Achille, Rovere and Soatto reported it in deep convolutional networks on image classification, trained for many epochs under SGD with a learning-rate schedule, where blurring the images for an early window left a deficit that extended training did not remove, and where the Fisher information rose and fell early. My runs share none of those conditions: no convolutions, no images, a few thousand steps rather than epochs, and a Fisher trace that under Adam does not peak until the window is long over. Their result and mine are about different regimes, and the honest summary is that the regime in which a network’s write channel closes, if it exists, was never entered here. The question that remains open is which of those conditions does the closing. Depth and the learning-rate schedule are the candidates I would test first, and it needs convnets to test it, so it is a different run on a different budget.