Research
The literature, the critique, and our protocol
EEG-to-text has a measurement problem: this page lays out what was reported, what turned out to be an artifact of the evaluation, and the protocol NEST uses instead.
Where the field stands
Ten results, one methodological fault line
ZuCo results before 2024 were scored with teacher forcing: the decoder is given the correct previous word at every step, which lets a language model produce fluent text with little dependence on the EEG. Free decoding removes that support. Rows below are ordered roughly by publication date; the two invasive rows are marked because they are not a comparable modality.
| Method | Year, venue | Signal, data | Evaluation | Reported | Note |
|---|---|---|---|---|---|
| Wang and Ji | 2022, AAAI | EEG, ZuCo | Teacher-forced | BLEU-1 40.1 | Inflated by protocol |
| DeWave (Duan et al.) | 2023, NeurIPS | EEG, ZuCo | Teacher-forced | BLEU-1 41.35 | Same protocol as above |
| BELT (Zhou et al.) | 2023 | EEG, ZuCo | Teacher-forced | BLEU-1 42.31 | Same protocol as above |
| Jo et al., “Are EEG-to-Text Models Working?” | 2024, arXiv 2405.06459 | EEG, ZuCo | Free decoding, noise-input control | Real EEG scores about the same as random noise | Establishes the critique this page is built around |
| Replication (BrainBART, BELT, DeWave) | 2026, arXiv 2607.18749 | EEG, ZuCo 1.0 and 2.0 | Free decoding | WER about 99–121%, BLEU-1 about 7–16 | Confirms the critique across three model families; proposes the COFETT benchmark |
| “Escaping the BLEU Trap” | 2026, arXiv 2603.03312 | EEG, ZuCo | 4-way semantic classification (25% chance) | 50.0% classification accuracy, 2.7% content recall | Moves evaluation away from token-level BLEU entirely |
| Brain2Qwerty | 2025, Meta AI | MEG and EEG, typed sentences, closed vocabulary | Offline decoding | MEG CER 32% (best 19%), EEG CER 67% | Closed task, not real time; the MEG/EEG gap is roughly double the error rate |
| Brain2Qwerty v2 | 2026, Meta AI with BCBL | MEG, typed sentences, closed vocabulary | Offline decoding | 61% average word accuracy, best participant 78% | Same closed task family as above, MEG only |
| Willett et al. | 2023, Nature | Intracortical array (invasive) | Real-time handwriting decoding | 62 words/min, WER 9.1% (50-word vocabulary), 23.8% (125,000-word vocabulary) | Invasive baseline, not a comparable modality |
| Metzger et al. | 2023, Nature | ECoG (invasive) | Real-time speech decoding | 78 words/min, median WER 25% | Invasive baseline, not a comparable modality |
Read together, the table says three things. First, non-invasive open-vocabulary decoding from EEG is unsolved: every free-decoding number on ZuCo is close to chance, and the fluent sentences in the 2022–2023 papers were the decoder's language prior, not the signal. Second, there is a real gap between MEG and EEG on the same closed task and the same team's pipeline — Brain2Qwerty's character error rate roughly doubles from MEG to EEG, which is the modality NEST works in. Third, the invasive results show the ceiling is not the algorithm: with electrodes on the brain, word error rates fall to single digits. The honest reading is that a protocol which cannot tell real EEG from noise is not measuring anything, which is why NEST reports a noise-input control next to every number it publishes.
Our evaluation protocol
What we commit to before a number is published
The critique above is specific: teacher forcing and the absence of a noise control let a language model's fluency stand in for decoding. NEST's protocol is built to fail loudly if that happens again.
- Free autoregressive decoding at evaluation time. No teacher forcing. The current implementation is greedy; beam search is planned and will be reported when used.
- Held-out subjects: the test split is never seen in training or validation.
- A noise-input control reported next to every real-EEG number, so a reader can see whether the model beats random input.
- CER, WER, BLEU and ROUGE all reported together, not just the metric that looks best.
- The protocol published together with the first result, as a technical report, not after.
- The code path is public:
eval_bart()andeval_ctc()inscripts/train_nest_v2.py.
The number we withdrew
26.1% WER was never decoded
February 2026. We published a figure of 26.1% word error rate, BLEU 0.74 and 73.9% accuracy. It was not produced by running the model and decoding its output. It was estimated from CTC training loss with a heuristic formula that maps loss to an approximate error rate. That formula was never validated against a real decode, and there is no reason to expect it holds for a sequence model reading EEG.
The figure was removed in commit f8d6675 on 2026-02-20, after we recognized it did not meet our own protocol above: it had no noise-input control, no held-out free decoding, nothing to compare it against. Nothing replaces it. The word error rate field on this site reads pending until the v2 model is trained to completion and decoded under the protocol described here, on a GPU we do not yet have.
Datasets
ZuCo, its neighbors, and the gap we plan to close
ZuCo 1.0 (Hollenstein et al. 2018) recorded 12 native English speakers with a 128-electrode EGI system (105 channels used) plus eye tracking, while they read 1,107 sentences across three tasks: normal reading of movie reviews, normal reading of Wikipedia text, and task-specific reading, totaling 21,629 words. ZuCo 2.0 (2020) added 18 subjects reading 739 sentences. Both are reading corpora; neither includes listening or imagined-speech paradigms.
| Corpus | Modality | Task |
|---|---|---|
| ZuCo 1.0 / 2.0 | EEG | Natural reading |
| Brennan | EEG | Listening |
| MEG-MASC | MEG | Listening |
| Chisco | EEG | Imagined speech, Chinese |
No public Turkish EEG reading corpus exists. Every public reading corpus we use, including ZuCo, is English. We plan to collect one: an ethics-approved study at Istanbul Commerce University with 20 participants, following the protocol at experiments/user_study/user_study_protocol.md on GitHub, targeted for Q1 2027.
Technical report
In preparation, published when there is a result to report
The full technical report, including the protocol on this page and the first decoded numbers, will be posted to arXiv once the v2 model has been trained and evaluated. There is no preprint ID yet, and nothing here should be cited as a paper.
@software{nest2026,
title = {NEST: Neural EEG Sequence Transducer},
author = {{NEST contributors}},
year = {2026},
url = {https://github.com/wazder/nest}
}
This is a software citation for the codebase, not a citation for a paper. Use the technical report above once it exists.
Read the code before the numbers
The evaluation protocol, the withdrawn number, and the training scripts are all in the repository.