Technology
How NEST decodes reading EEG
From a fixation-aligned frequency vector to open-vocabulary text: the input representation, the v2 architecture, and what in the pipeline is verified versus still pending.
Input representation
From fixations to an 840-dimensional vector
NEST reads ZuCo's word-level EEG features, not raw voltage. Eye-tracking fixations tell the pipeline which word a reader was looking at, and each fixated word carries one feature vector: 105 EEG channels times 8 frequency bands, giving an 840-dimensional input per word. A sentence is a sequence of these vectors.
| Band | Approx. range | Sub-band |
|---|---|---|
| theta1 | 4–6 Hz | lower half |
| theta2 | 6.5–8 Hz | upper half |
| alpha1 | 8.5–10 Hz | lower half |
| alpha2 | 10.5–13 Hz | upper half |
| beta1 | 13.5–18 Hz | lower half |
| beta2 | 18.5–30 Hz | upper half |
| gamma1 | 30.5–40 Hz | lower half |
| gamma2 | 40–49.5 Hz | upper half |
Sub-band boundaries as published for ZuCo (Hollenstein et al. 2018): each parent band is split into two sub-bands. 105 channels × 8 bands = 840 dimensions per fixated word.
Unfixated words. A word the reader never fixates has no EEG segment to derive a band power from, so ZuCo emits NaN for it. NEST zeroes those entries rather than dropping the word or imputing a value (commit 5460ace), which keeps sentence length intact. A zeroed vector carries no signal — it is a placeholder, not a measurement.
Architecture · v2
A five-step pipeline, verified by instantiation
src/models/nest_v2.py defines the current architecture: a projection into a shared embedding space, a positional encoding, a transformer encoder, and one of two output heads.
| Component | Config |
|---|---|
| EEGProjection | 840 → 1536 → 768, LayerNorm and GELU, output scaled by √d |
| Positional encoding | Sinusoidal |
| Transformer encoder | 6 layers, d_model 768, 8 heads, pre-norm, GELU feed-forward |
| CTC head | Linear, 29 symbols (blank, a–z, space, pad) — 45,310,637 parameters |
| BART decoder | facebook/bart-base with cross-attention to the EEG encoder; BART encoder and first 4 decoder layers frozen — 184,424,592 total / 64,697,232 trainable |
| Decoding | Greedy CTC decode; free autoregressive decoding for BART (greedy today, beam search planned) |
Parameter counts were verified by instantiating NEST_CTC_v2 and NEST_BART_v2 on 2026-09-03.
Why frequency-band features
Precomputed bands, not raw voltage, as the primary path
NEST's primary input representation follows Wang and Ji (AAAI 2022): fixed frequency-band power per channel per word, rather than a raw EEG waveform. It is a smaller, more structured input than raw voltage, and it is the convention most EEG-to-text literature on ZuCo already reports against.
The repository also contains raw-EEG architectures — EEGNet and DeepConvNet spatial convolutional front ends, LSTM, Transformer and Conformer temporal encoders, and a 303M-parameter Conformer-BART model (src/models/nest.py, nest_bart.py). These are implemented and importable, but they are not the primary path described above and they have not been trained.
Training and evaluation protocol
Subject-independent, free-decoded, never teacher-forced at test time
Splits are subject-independent by default: 8 subjects train, 1 validates, 2 are held out for test and never appear in training or validation.
Training
Loss
CTC loss for the CTC head. Cross-entropy with teacher forcing for the BART head — but only during training. The decoder is given the correct previous token at each step solely to compute a training gradient.
Evaluation
Decoding
eval_bart() calls model.generate() with max_length=64 and no teacher forcing. The generate() implementation is currently greedy; the num_beams argument is accepted but not yet used. eval_ctc() uses greedy CTC decoding. Word and character error rate are computed by Levenshtein edit distance.
What this protocol commits to.
- Free autoregressive decoding at evaluation time, never teacher forcing.
- Held-out test subjects the model never saw during training or validation.
- A noise-input control reported next to every real-EEG result.
Compute: a quick test runs on CPU or Apple Silicon MPS (scripts/train_nest_v2.py --quick-test). A full 200-epoch BART run is estimated at 8 to 12 hours on an A100, which NEST does not yet have scheduled time on.
Preprocessing and robustness modules
Implemented, not yet evaluated
src/preprocessing contains signal-cleaning and generalization code beyond the ZuCo features NEST currently trains on. None of it has been benchmarked for effect on decoding accuracy.
Filtering
Band-pass and notch filtering via MNE.
Artifact removal
ICA-based artifact removal (mne.preprocessing.ICA) with EOG/ECG component detection.
Augmentation
11 augmentation methods, including noise injection, time shift and mixup.
Subject-adaptation modules — a domain-adversarial network (DANN) and CORAL feature alignment — are also present in the code, aimed at generalizing across subjects beyond the fixed train/val/test split. They are implemented, not yet evaluated.
Deployment modules
Written, unbenchmarked, no artifact yet
src/evaluation contains the code a decoding stack needs to leave a notebook: compression, export and streaming. None of it has produced a shipped artifact.
| Module | What it does | Status |
|---|---|---|
| Pruning | Magnitude, iterative, sensitivity, structured, and lottery-ticket pruning | Unbenchmarked |
| Quantization | Post-training, dynamic, quantization-aware training, mixed precision | Unbenchmarked |
| Export | ONNX and TorchScript export | No artifact |
| Streaming | Circular-buffer streaming inference | Unbenchmarked |
Limitations, stated plainly
What the architecture does not fix
None of these are solved by a better model. They are properties of the signal and the field's current evaluation practice.
- Signal quality. Scalp EEG has a low signal-to-noise ratio, and volume conduction blurs cortical sources together before they reach the electrode.
- Free decoding is unsolved field-wide. Under free decoding rather than teacher forcing, published ZuCo decoders — not only NEST — have been shown to score close to a random-noise input. NEST reports a noise-input control alongside every real-EEG result so this can be checked, not assumed away.
- Word alignment needs eye tracking. The input representation depends on fixation timing to assign EEG to words; without eye tracking there is no word boundary to align to.
- English only, today. ZuCo is English-language reading data. No public Turkish EEG reading corpus exists yet.
See the numbers behind this
The literature this architecture is built against, and the protocol its own results will be measured with.