Part 3 of 5 in the LibriBrain series. The project page summarises the whole project. Previous: Building my LibriBrain speech decoder.
When LibriBrain 2026 closed in October, my Broad entry stood eighth on the public leaderboard at 0.72230. The private
scores, released on 2 November, placed the same entry seventeenth of 53 at 0.67846. On Deep my best public file scored
0.72928, and the file I selected scored 0.72727 publicly and 0.69704 privately, 26th of 91. The private winners scored
0.758 on Broad and 0.763 on Deep. Kaggle scores every submission on both splits, so each of my 61 submissions (35 on
Broad, 26 on Deep) now has two scores, and every comparison I made during the competition can be read a second time on
rows I never saw. All figures below come from the submission tables in the repository (results/lb_private_20261102/)
and the analyses built on them.
What held
Class balancing held. The step rescales each listener’s predictions so that every word receives equal total probability, using only that listener’s unlabelled test rows. On Broad it gained 0.0358 on the public split and 0.0366 on the private one. On Deep it gained 0.0104 publicly and 0.0512 privately, about five times as much. On Broad, a refinement that fitted the balancing offsets on sentence rows only gained 0.0085 and 0.0087. A trigram language model fitted to vocabulary-only sentences, which I had already rejected because it lost 0.111 on the public Broad split, lost 0.090 on the private split.
Test-time recalibration of the network’s normalisation statistics (AdaBN) held in sign and shrank in size. On Deep, where the acquisition dates suggest the test recordings came from a day absent from training, full re-estimation gained 0.0438 on the public split and 0.0148 on the private one. My day-held-out local test had put it at −0.010. A partial version on Broad, mixing old and new statistics in equal parts, lost 0.0023 publicly and gained 0.0072 privately.
What reversed
The sentence prior reversed. Each Broad listener heard 156 sentences built only from the fifty target words, and I had fitted a small hidden Markov grammar to each listener’s own decoded sentences, with a smoothing strength κ controlling how hard it pulled. Against the same base ensemble, at κ = 20, it added 0.017 on the public Broad split, and it went into both final entries. On the private split it cost 0.036 on Broad and 0.029 on Deep.
The κ sweep makes the reversal hard to attribute to noise. Six files share one base ensemble and differ only in κ. On the public Broad split they trace an interior optimum, scoring 0.688, 0.705, 0.710, 0.704, 0.689 and 0.692 for κ = 5, 10, 20, 50, 200 and no prior. On the private split the same six files fall in strictly reverse order of prior strength, at 0.590, 0.616, 0.634, 0.646, 0.665 and 0.671. On the private rows each step toward a stronger prior lowered the score, including the steps that had raised it in public.
Three modelling decisions also changed their reading. The two Deep steps were each compared as a five-seed ensemble against a ten-seed one, so their differences mix the step with ensemble size. A wider Deep encoder, promoted on a local gain of 0.041 and a public gain of 0.0082, lost 0.0171 privately. That local gain came from one seed on a 141-row block; with three seeds the paired gain was 0.019 (standard deviation 0.033), so it was never established locally. Time-stretch augmentation on Deep, whose seeded local gain was 0.017 (standard deviation 0.029), lost 0.0041 publicly and 0.0255 privately. A per-subject input layer on Broad, which I had reported as a transfer failure because it added 0.0013 on the public split, gained 0.0304 on the private one. That file pair is not a clean comparison, since the two files also differ in ensemble size (five models against ten) and in how subjects without labelled data are handled, and both carry the sentence prior. The direction is supported by the seeded local test, in which all 36 configurations were positive, by 0.030 to 0.068.
A split that is not exchangeable
Under a row-random division of the test set, files produced by the same pipeline should show nearly the same public-to-private gap, differing only by sampling noise. On Broad, among submissions scoring at least 0.60 publicly, the seven files without the sentence prior had gaps between −0.009 and −0.022. The eighteen files with it had gaps between −0.024 and −0.102. The two groups do not overlap, and along the κ sweep the gap widens as the prior strengthens; other changes matter too, since the κ = 20 files alone span −0.042 to −0.102. On Deep the groups overlap and the prior-free gaps range from −0.030 to +0.039, so the Deep evidence is weaker.
I cannot say which rows are private; only the organisers hold that assignment. I can say where the prior acts. The grammar rewrites sentence rows, but the class balancing that follows it is fitted over each listener’s sentence and isolated-word rows together. Counting the words that enter or leave a row’s top ten relative to the prior-free file, the prior at κ = 20 changes isolated-word rows more than sentence rows, by 3.14 against 1.42 words per row, and at κ = 5 by 4.77 against 2.28. A split that weights isolated-word rows, subjects or sentences differently from the full test set would therefore see the prior differently from how the full test set sees it. The direction of the private result suggests that the public split under-sampled whatever the prior damages.
Adaptive overfitting, the usual worry about public leaderboards, comes from querying a fixed subset too often, and querying less would reduce it. Here the public rows appear to have measured a population different from the private rows, in a way the prior was sensitive to and class balancing was not, and fewer queries would not have changed that.
Choosing final submissions
My selections were made on public scores. The best private Broad file I submitted, 0.68037, combined partial AdaBN with class balancing and carried no sentence prior; it scored 0.68908 publicly and was not selected. The best private Deep file, 0.71779, used full AdaBN and class balancing, again with no prior, and was not selected either. Against the private tables as they stand, those two files would have placed sixteenth on Broad and eighteenth on Deep.
The prior’s case rested on two instruments, the public split and a simulator of the test set that I had calibrated against public leaderboard effects. They were not independent, since the simulator reproduced the prior’s public gain because it had been tuned to agree with the public split. A third instrument, a local test of the learned prior on held-out prose, had shown it losing on both tracks, and I set that result aside on the grounds that prose was a poor model of the constructed test sentences. Class balancing, by contrast, had agreed in sign across every instrument I applied to it. On the private split, agreement across instruments predicted survival better than the size of the public gain did.
If I selected again I would treat any component whose support comes from the public split, or from tools calibrated to it, as a single measurement, and would make one final selection from steps that agreed across independent local tests. That rule is easy to state and only partly vindicated here. On Deep it picks the file with class balancing alone, which scored 0.70299 privately, above the 0.69704 I selected; it excludes full AdaBN, whose local estimate was negative, and so misses the best Deep file. On Broad the balancing-only file scored 0.67317 privately, slightly below my selection, and the best Broad file needed partial AdaBN, which the public split had scored as a small loss. The rule would have protected against the prior. It would not by itself have found the best files.
Next in the series: The timing twin, a noise control that keeps the word onsets.