Meta released a generic model that reads handwriting from surface EMG at the wrist, as one task in their generic neuromotor interface. The released test split is ten users who never appear in training. Their own checkpoint, scored with their decoder, has a character error rate of 30.06%. I reproduced that number: 30.07%.
I investigated whether personalization and language-model decoding address different sources of error.
What moved the error
A personal fine-tune, recipe chosen on the validation users and applied unchanged, takes the ten test users from 30.07% to 20.33%. A character language model, a 7-gram trained on WikiText with every paragraph that contains a test prompt removed, and with the fusion weights chosen on validation before they ever see the test users, takes that fine-tune from 20.33% to 15.02%. The same language model on the generic model, with no personal fine-tune, reaches 25.43%.
The two gains add. The interaction is about zero: +0.67 points, interval [−0.60, 2.25]. The language model does not recover what the user-specific encoder recovers, and the encoder does not recover what the language model recovers. End to end, 30.07% to 15.02% is a drop of 15.06 points, interval [11.2, 19.3]. On phrase prompts, words decoded exactly go from 30.6% to 76.6%, and whole phrases decoded exactly go from 5.1% to 50.2%.
| model | greedy | with the language model |
|---|---|---|
| generic, reproduced | 30.07 | 25.43 |
| early input correction | 26.99 | 21.87 |
| personal fine-tune | 20.33 | 15.02 |
What the controls say
Fine-tuning on the other nine test users, and scoring the one who was left out, raises the error to 39.74%. Zero of the ten users improve. The residual that personalization reaches is specific to that writer. It does not transfer.
An early linear correction of the sensors captures about a third of the personalization gain. The other two thirds require the encoder itself to change. On the wrist-angle task in the same release, the opposite is true: the whole per-user gain is an early input correction. The relative contribution of sensor correction and encoder adaptation differs between these tasks.
A language model that was allowed to see the training prompts, including prompts that also appear at test time, would have reported 14.15% instead of 15.02% on the fine-tune and 24.25% instead of 25.43% on the generic model. The numbers above use the de-leaked model. The leak would have overstated the gain by about one point.
What this does not establish
Ten test users. One validation-chosen fine-tune recipe, and that recipe sits on the edge of the grid that was searched, so the personalization gain may be understated. The user-specific residual is not separated into how the motor units are recruited versus how the person forms letters. The language model is a character 7-gram, not a neural rescoring model. Offline character error is not the online words-per-minute number in Meta’s paper, and the paper says those two are not comparable.
This work builds on my interest in neural interfaces and communication, described on the About page. The results evaluate an offline decoder, not a clinical intervention.