Part of the Philosophy series. The measurements behind this note are in The timing twin, part of the LibriBrain series, whose project page summarises the whole project.
This autumn I spent a month building a decoder that guesses which of fifty words a person heard from the magnetic field around their head. Afterwards, on one held-out day, I compared a stripped-down version of it, a screening decoder with a fixed three-second window and no sentence context, against a model that received no brain data at all, only the times at which words began. The timing model scored higher, at 0.136 against 0.0725 for picking the exact word, where chance is 0.02. A full sentence-context decoder on the same rows did better than the timing model in top-10 terms, 0.693 against 0.588, so the comparison does not say that brain decoding is empty. The word onsets are given to every decoder, since one has to know where a word is in order to look at it. Still, the comparison raises a question that I think neuroscience answers too quickly. When we say that a decoder shows information about a word to be “in” a brain signal, what exactly have we said?
Information is a relation
The most common reading is Shannon’s. A signal X carries information about a variable W when the two are statistically dependent, so that observing X reduces uncertainty about W. On this reading information is a property of a joint distribution, and the joint distribution is fixed by an experiment: which words were played, in what order, at what speed, in which sentences, cut into which windows. A decoder that beats chance establishes a lower bound on the mutual information between W and X under that design. It says nothing yet about where in the causal chain from loudspeaker to sensor the dependence arose.
In the LibriBrain data the dependence can arise along at least two routes. The brain may respond differently to the and to people, so that the shape of the response encodes identity. Or the brain may respond identically to every word, only at its onset, while the next word’s onset falls at a time that depends on how long the current word lasted, and duration differs across words. On the second route a three-second window around a word contains a trace of when the following word arrived, and the decoder turns that trace into a guess about identity by exploiting a regularity of spoken English. Both routes produce statistical dependence between W and X. Only the first involves the brain encoding anything about which word was heard.
Dretske saw this structure clearly in 1981. On his account a signal r carries the information that s is F when the probability of s being F, given r and the receiver’s background knowledge k, is one, and the probability given k alone is less than one. The second clause does real work. Change what the receiver already knows and the same physical signal carries different information, and a signal carries none about what k had already settled. A decoder is a receiver of this kind, and the baseline against which we score it is its k.
The baseline fixes the content of the claim
Seen this way, the choice of baseline is the choice of what claim is being made. Scoring against chance asserts that X carries information about W when nothing else is known, the quantity I(W; X). Scoring against a model of the onsets asserts that X carries information about W beyond what the onset times T already supply, which is I(W; X | T). These are different propositions, and in a dataset where onsets alone predict the word at 0.136 top-1, they can have different truth values. The screening decoder at 0.0725 has demonstrated the first. It has not demonstrated the second, and nothing in its accuracy could, because accuracy against chance leaves T in the background where it cannot be seen.
That is the sense in which a baseline belongs to the content of a scientific claim. When a paper writes that word identity can be decoded from MEG, the reader will generally hear the second proposition, that something in the brain response distinguishes the words. The evidence offered can support only the first. Every number can be correct and cross-validated, held out by day and by text, and the claim can still be about something other than what it appears to be about, because the comparison that would have fixed its content was never run.
The Advaita tradition offers a useful image. A clear crystal placed beside a red hibiscus appears red. The redness is perceived in the crystal and does not belong to it; it belongs to an adjacent condition, which the tradition calls an upādhi, a limiting adjunct. The logicians of the Nyāya school used the same word for a condition that accompanies the conclusion of an inference without accompanying its reason, so that an inference which seems to hold actually rests on something it does not mention and fails without it. Their stock case is the inference from fire to smoke, which holds only where the fuel is wet; a reason burdened in this way is called sopādhika. Word timing is an upādhi of this kind for the decoder. Identity shows up in the signal, and on the second route it belongs to the stimulus stream that the signal happens to sit beside. The image also suggests the test, which is to take the flower away and look again. In decoding, the equivalent is a control in which the onsets remain and word-specific responses are absent, which is what the timing twin in the companion post tries to be.
Readout and representation
Suppose the conditional test is run and the decoder clears it. Something in the MEG then distinguishes the words beyond their timing. Even so, the step from “decodable by my model” to “represented by the brain” is a further step, and a well-known one. De-Wit and colleagues (2016) put the point plainly: a decoder shows that information is available to the experimenter and leaves open whether the brain makes any use of it. Ritchie, Kaplan and Klein (2019) call the inference from decodability to representation the decoder’s dictum, and argue that it fails, since a decoder can exploit distinctions in neural activity that the brain itself does not use. Kriegeskorte and Diedrichsen (2019) propose tying representation to readout of a constrained kind, roughly what a downstream linear readout could recover. That gives the word a definite meaning, at the cost of making it relative to an assumption about what the rest of the brain can compute.
My competition decoder was a convolutional network feeding a transformer, trained on eighty hours of one person’s recordings. Whatever it reads is available to a learner with eighty hours of labelled examples and a large architecture. Whether any of it is available to the auditory and language areas of the listener, in the time the listener has, is a separate empirical question that decoding accuracy does not touch. Brette (2019) presses a version of this against the language of neural coding as a whole, arguing that a code in the engineering sense requires a reader that uses it, and that the experimenter’s decoder is often the only reader in view.
What a careful claim would look like
None of this makes decoding uninformative, and I do not want to slide from the observation that the claim is underspecified to the conclusion that it is empty. A careful report can name its k, stating what metadata the decoder receives, scoring a model that receives only that metadata, and presenting its result as information beyond that baseline, which for word onsets is I(W; X | T). It can also keep the language of readout and representation apart, saying that words can be read out from the recording when that is what was shown and reserving “the brain represents” for evidence about what the brain does with the signal. The onset baseline in my own case was computed from the event files alone.
There is something philosophically instructive in how easily the two claims run together. Information feels like a thing that sits inside a signal and waits to be extracted, much as the red seems to sit inside the crystal. A good part of the work in this project went into finding out what the signal was standing next to, and I suspect the same question is worth asking of most decoding results that are reported against chance alone.
Further reading
- C. E. Shannon, “A Mathematical Theory of Communication”, Bell System Technical Journal 27 (1948).
- Fred Dretske, Knowledge and the Flow of Information (1981).
- Lee de-Wit, David Alexander, Vebjørn Ekroll and Johan Wagemans, “Is neuroimaging measuring information in the brain?”, Psychonomic Bulletin & Review (2016).
- J. Brendan Ritchie, David Michael Kaplan and Colin Klein, “Decoding the Brain: Neural Representation and the Limits of Multivariate Pattern Analysis in Cognitive Neuroscience”, British Journal for the Philosophy of Science (2019).
- Nikolaus Kriegeskorte and Jörn Diedrichsen, “Peeling the Onion of Brain Representations”, Annual Review of Neuroscience (2019).
- Romain Brette, “Is coding a relevant metaphor for the brain?”, Behavioral and Brain Sciences (2019).
- Dulhan Jayalath and Oiwi Parker Jones, “Removing Timing Shortcuts Improves Non-Invasive Brain-to-Text”, arXiv 2609.40359, on timing shortcuts in LibriBrain.
- Xinyu Zhang, Sichao Liu, Runhao Lu, Alexandra Woolgar and Lihui Wang, “What Are We Actually Decoding? Source Attribution for Non-Invasive Brain-to-Language Retrieval”, arXiv 2605.24524, on attributing retrieval scores to their sources in MEG and EEG.