Part 5 of 5 in the LibriBrain series. The project page summarises the whole project. Previous: The timing twin.
In LibriBrain the word signal is a very small part of what the sensors record. Measured as the reliable share of variance that differs between the fifty words, it is about a tenth of one percent of the recording for the Deep listener. Almost everything else is background, which decoders usually treat as a nuisance to be averaged away by training on more examples. After the competition I spent several days on the opposite assumption, that the background has structure worth learning in its own right. The working model was an observation equation, y = As z + b + ε, in which the sensor reading y combines a word response z, mixed into the sensors by a subject-specific map As, with a baseline b and a noise term ε. If ε has a form that can be estimated without labels, from its variance, its covariance or its dynamics, a decoder that knows that form should extract more from each labelled word than one that does not.
Every analysis below uses the Deep listener alone. The decoding tests use a held-out day whose text shares no 5-gram with training. Most of the answers came back negative.
The noise is predictable, and the ceiling is high
The first question was whether the idea had any room to work. If the noise energy v of each word window varies, then weighting rows by their precision raises the effective number of labelled examples by a factor H = E[v] E[1/v]. Over 313,068 word windows from 162 runs, H is 4.24 when v is known exactly. Variation between whole runs accounts for only a factor of 1.33, so most of the spread lies within runs. The important number is how much of it is predictable before the word arrives. A linear model of the pre-onset state, cross-fitted, predicts log v with R² of 0.78, and weights built from that prediction give H = 3.16. The noise is heteroscedastic, and its level can be forecast from information available before each word, which would correspond in principle to about three times the labelled data. This is the kind of structure that Packard’s (1990) search for regions of state space where a chaotic series briefly becomes predictable was designed to find, here applied to the noise’s variance instead of to its value. Part of this H comes from artefacts in the tail of v, and down-weighting artefacts is ordinary practice.
I also tried a more flexible form for this variance equation. Kolmogorov-Arnold networks have recently been proposed for discovering equations of dynamical systems (KANDy, arXiv 2602.20413), and a single-layer KAN is an additive spline model. Fitted to the same task, additive splines reached R² 0.785 against 0.779 for the linear model, and adding pairwise terms gave 0.776. The weights they produced were slightly worse, with H falling to 2.92 and 2.56.
Weighting rows did nothing in a linear decoder
The ceiling did not translate into accuracy. In a softmax regression on 64 spatial components and ten time bins of each one-second window, trained with four seeds, weighting rows by their predicted noise level did not beat uniform weights at 10% or at 100% of the training data. Weights computed from the true noise level did no better, and at full data they lowered top-1 by 0.012 and top-10 by 0.017 relative to uniform weights. They beat only random weights of the same spread. If even oracle weights cannot help, extra effective data is of no use to this decoder, which suggests that its features, rather than the number of examples, set its limit. It scores about 0.05 top-1 and 0.35 top-10, in the range of the single-channel onset readout described in the timing-twin post, so the null applies to a weak decoder. A convolutional network trained with a precision-weighted loss is the test that would decide the question, and it has not been run.
Spatial covariance is local to the recording and drifts within it
The spatial covariance of the noise produced the one repeatable gain. Whitening every run with a single global covariance hurt the convolutional decoder, lowering top-10 by 0.05. Whitening each run with its own covariance, estimated without labels, behaved differently in the same single-seed screen. Without class balancing it left the decoder roughly where the raw input put it (0.450 against 0.458 top-10). With class balancing it reached 0.510 against 0.472 for raw input with balancing, a gain of about 0.04 on that seed. Two further seeds did not hold it. Paired by seed, the balanced gain was 0.039, 0.021 and -0.010 (mean 0.017, standard error 0.014), balanced top-1 fell by 0.014 on average, and without balancing the top-10 difference was 0.004. In the linear decoder, across four seeds, per-run whitening beat global whitening by 0.018 to 0.035 top-10 across the four weighting arms. The operation is Euclidean alignment, introduced for cross-subject EEG transfer by He and Wu (2019), and I claim no novelty for it. The measurement adds that the useful covariance is the one estimated locally to the recording. I did not test whether a per-day estimate differs from a per-run one.
The covariance also changes within a run. Across 24 runs, a covariance estimated from the 30 seconds on either side of each word fitted the background better than the run’s covariance, improving the Gaussian negative log-likelihood by 0.095 per sample per dimension (standard error 0.005). Rescaling the run covariance to the local overall level, with its shape held fixed, made the fit worse by about 1.1. The shape of the spatial noise moves on a scale of tens of seconds, and a single scalar noise level cannot represent that movement.
Within this embedding, the background is a linear system with a fixed law
The next question concerned dynamics. On ten runs, embedding the ten leading components with six delays and fitting on the first 70% of each run, a linear vector autoregression forecast the last 30% with R² 0.541 at 20 ms ahead. Nearest neighbours in the same embedding, the approach that Farmer and Sidorowich (1987) developed for forecasting chaotic series, reached 0.242. A nearest-neighbour model fitted to the autoregression’s residuals added 0.001. The locally-linear variant in the Farmer and Sidorowich tradition was under-determined at the neighbourhood size used, so its result is not informative. Within this embedding the background shows no predictable nonlinear structure beyond the linear model, so nonlinear equation discovery applied to it would have little to find.
In the same ten runs the linear law also looks constant. Dynamics fitted on a different run, from a different day, forecast a run almost as well as its own dynamics (R² 0.596 against 0.607), and one set of dynamics pooled over all ten runs did better than either (0.636). The covariance of the forecast innovations, which is the drive of the system, differs between the two halves of a run by a log-Euclidean distance of 1.70, and between runs by 2.69. Within this embedding and these ten runs, the background can be summarised as x(t+1) = A x(t) + B(t) w(t), with one A shared across days and a drive covariance that drifts within runs and differs more between them.
Subtracting the forecast removes the words
Given a fixed and well-estimated linear law, the obvious move is to subtract the forecast and decode the innovations. On 60 evaluation runs containing 101,794 word windows, with the dynamics fitted on ten other runs, the reliable word signal fell from 0.108% of variance to 0.054%. Presumably this is because evoked responses are as predictable from their own recent past as the background is, so that temporal prewhitening removes signal at least as fast as noise; I have not tested that mechanism directly. The one transform that raised the signal share was local spatial whitening over ±30 seconds, which took it from 0.108% to 0.120%, an increase of 11%. Differences between class means can include structure correlated with timing, such as responses to the next word’s onset, so this share is an upper bound on word content. Global whitening left the share unchanged while costing the convolutional decoder 0.05 top-10, so the share is also an imperfect predictor of what a decoder gains.
What the negatives constrain
On this listener and at these model sizes, nonlinear discovery found nothing in the background beyond the linear law, and subtracting that law took word signal with it. Row weighting did not help a linear decoder that oracle weights could not help either. The positive results concern spatial covariance. Per-run whitening, estimated without labels, beat global whitening in both decoders but did not reliably beat raw input in the convolutional one over three seeds, and local ±30-second whitening raised the reliable signal share by 11%. The second did not survive a decoder. In the linear decoder over four seeds, whitening the continuous signal with each run’s own covariance scored 0.392 top-10, against 0.377 for per-run whitening of the epoch features. Local whitening at ±30 and ±10 seconds scored 0.377 and 0.383; paired against the whole-run version, top-10 fell by 0.015 (standard error 0.004) and 0.009 (0.002), while top-1 rose by 0.003 and 0.007. The extra reliable signal went into the first guess and cost the metric the competition used.
Two lines remain open. The heteroscedastic ceiling of about three times the effective data, part of which is artefact down-weighting, is unexploited; the test is a precision-weighted loss inside the convolutional decoder. Local whitening is closed for the linear decoder and open for the convolutional one, which can use features a linear readout cannot. All of this rests on one subject, the network results on one to three seeds and the linear results on four, so I would read them as measurements of this dataset that bound what noise modelling can deliver on it.