Neural decoding and encoding, reading and writing

In BCI-3.1 you saw how an electrode picks up a signal: the summed electrical shadow of neurons discharging their membrane capacitors, a stream of microvolts pulled off tissue and cleaned up. That is where the recording story ends and the useful part begins. A stream of microvolts is not a cursor moving or a letter typed. Something has to turn that signal into an intent, and separately, something has to push information the other way, from a machine back into a brain. Those are the two verbs of a brain-computer interface: decoding (reading) and encoding (writing). This lesson builds both from first principles, and it holds one honest line the whole way: we read far better than we write, and the reason is biology, not budget.

Two directions, and they are not symmetric

Fix the vocabulary before anything else, because the words get swapped constantly.

Decoding is reading. You have a recorded neural signal and you want to recover what the user intended: move left, grip harder, select the letter E. The signal already exists. The brain produced it on its own. Your job is to look at the pattern and infer the intent behind it. That is a recognition problem.

Encoding is writing. You want to put information into the brain that was not there, a sensation of touch, a visual flash, a nudge to a misfiring circuit. Nothing exists yet. You have to cause a pattern of neural activity from the outside, and cause it to mean the thing you want it to mean. That is a generation problem, and generation is much harder than recognition.

Hold onto that asymmetry. A programmer meets it constantly: reading a file format someone else wrote is far easier than writing a file that a picky parser will accept, and classifying an image is a solved-ish problem while generating a photorealistic one that fools the same classifier took another decade. Reading infers from what is there. Writing has to produce something the receiver will accept as real. In the brain, the receiver is billions of neurons wired by a code we only partly understand, so the gap between reading and writing is not narrow. It is the central fact of the field.

The failure edge of that analogy is where it gets interesting, because the analogy is too kind. A file format has a written spec you can look up, so writing to a parser is at least a solvable engineering task. The neural code has no spec, and nobody has one, so writing to a brain is harder than any parser you have ever fought. Keep the direction of the analogy (writing is the hard side) and discard its optimism (the spec exists), and you have the real situation.

Decoding is supervised learning on an ugly time series

Here is the reframe that makes decoding click for a software engineer: neural decoding is supervised machine learning, and the training data is a multivariate time series that is about as unfriendly as time series get.

Walk the setup the way you would frame any ML problem.

The inputs (features). The raw signal from BCI-3.1 is too raw to feed a model directly. You reduce each short window of time to a feature vector. For a spike-based recording, the natural feature is the firing rate of each neuron, how many action potentials it fired in, say, the last 50 milliseconds (recall from BCI-1.2 that the neuron carries information largely in spike rate and timing, so counting spikes per window captures much of that rate code). For a field-potential or EEG recording, the natural feature is band power, how much signal energy sits in a given frequency band in that window. Either way you now have a vector of numbers per time step, one dimension per channel or band. Dozens to hundreds of dimensions, updated many times a second.

The labels. Supervised learning needs labeled examples, input paired with the correct answer. Where do the labels come from for a brain? You run a calibration session. You ask the user to attempt known movements on cue: "now imagine pushing right," "now rest," "now grip." While they do, you record the feature vectors and stamp each one with the movement you asked for. That cued recording is your labeled training set. There is no shortcut around it. The brain does not ship with labels, so every BCI session begins by manufacturing them, which is why a real user spends real minutes calibrating before the cursor moves.

The model. Now it is ordinary supervised learning, and the model can be as simple or as fancy as the data supports. The workhorse for decades was a plain linear decoder: intended velocity is a weighted sum of firing rates, weights fit by regression. A step up is a Kalman filter, which adds a model of how the intended movement evolves over time and fuses it with each new observation, so the output is smoother and less jittery. At the top end you can train a neural network. But note the honest ordering: fancier is not automatically better here, because the labeled set is small (you only calibrated for a few minutes) and the signal is noisy, so a big model can easily fit the noise. That is textbook overfitting, and if you took the BIO track, it is the same failure mode BIO-5 warns about with any statistical model. A linear decoder that generalizes beats a deep net that memorized your calibration block.

Each stage is real work: the front end (teal) senses and conditions a tiny analog signal, and the back end (violet) reduces it to features and a trained decoder that emits a command. The decoder is a classifier, not mind reading.

The diagram above is the whole reader in one line, and now you can read every box. The first three stages (teal, the analog front end) are the membrane physics and electronics from S9.1 and BCI-3.1: a neuron fires an action potential, an electrode senses the tiny voltage, and an amplifier plus filter conditions it. The last three stages (violet, the compute back end) are exactly the ML pipeline you just built: feature extraction reduces the conditioned signal to firing rates or band power, the decoder is the trained model that maps features to intent, and the output is the command it emits. The single most important label on that picture is "ML classifier," not "mind reader." Everything to the right of the amplifier is signal processing and statistics on a shadow, not a window into meaning.

The signal will not sit still: calibration and drift

Now the part that separates a lab demo from a shipping product, and the part a working engineer will feel in their gut. A neural signal is not stationary. In ML terms, the joint distribution of features and labels that your decoder learned during calibration does not stay fixed. It drifts. Train once and deploy forever, and the decoder gets worse by the hour and much worse by the next day. Three causes stack up, and none of them is a bug you can patch.

The electrode moves. An implanted array sits in soft tissue inside a skull that the user walks and sleeps and sneezes with. Micron-scale shifts change which neurons each channel hears and how loudly. The exact same intent now lands as a slightly different feature vector. Your labeled mapping quietly goes stale.

The tissue responds. The body treats an electrode as a foreign object. Over weeks, scar tissue and immune cells build up around it (this is the notorious glial scar or foreign-body response), so the recorded amplitude fades and the set of neurons you can hear turns over. The channel that carried your cleanest signal in week one may be muffled by week six.

The brain itself changes. This is the deep one, and it ties straight back to BCI-2.2. Neurons are plastic. The very circuits you are recording keep rewiring in response to use, including in response to using the BCI. The brain learns the interface while the interface is learning the brain, so the target distribution moves because the target is alive. There is no static ground truth to lock onto. Even a perfect, immovable electrode would see the code shift underneath it, because learning is what neural tissue does.

The consequence is unavoidable: a fixed decoder degrades, so a real system must either recalibrate on a schedule (rerun a short cued session to relabel and refit, which users tolerate but resent) or run an adaptive decoder that updates its parameters online as it goes, tracking the drift the way an online-learning model tracks a nonstationary stream. Adaptive decoding is where most serious systems are headed, and it is genuinely hard, because you are updating on data you no longer have clean labels for. You are chasing a moving target while your ruler changes length.

Encoding: writing is stimulation, and it is crude

Flip to the other verb. To write, you stop recording and start injecting: you drive current through an electrode to make nearby neurons fire on your command. That is stimulation, and two clinical realities show both what it can and cannot do.

Deep brain stimulation (DBS) is the success story, and reading it honestly is instructive. In DBS, a surgeon places an electrode deep in the brain (for Parkinson's disease, typically in a small motor-control nucleus) and runs a steady high-frequency pulse train through it. For many patients it dramatically calms tremor. It genuinely works and it genuinely helps. But look at what it is doing: it is dumping rhythmic current into a whole neighborhood of neurons, most likely by disrupting a pathological oscillation. It is a coarse volume knob on a misbehaving circuit, not a message. DBS modulates a circuit's overall activity, it does not write specific information into it. Calling DBS "writing to the brain" is like calling a car alarm "sending an email." Both use electricity near the target. Only one carries content.

Sensory feedback is the frontier that is actually trying to write content, and it shows exactly why writing is hard. The goal: stimulate sensory cortex (or a sensory nerve) so the user feels something, a touch on a fingertip, so that a prosthetic hand can report back what it is holding. Labs have genuinely evoked crude sensations this way, a tingle or a pressure that a user localizes to a phantom finger. That is a real and moving result. But the sensations are simple and often unnatural, and getting from there to writing a rich, specific percept (the texture of denim, the give of a ripe peach) is not close.

Here is why writing is so much harder than reading, derived rather than asserted. Reading is pattern recognition on a signal the brain already produces in its own natural code. You do not have to know how the code works in full, you just have to find a statistical mapping from the pattern to the intent, and machine learning is very good at finding statistical mappings. Writing has none of those advantages:

• You must cause the pattern, not just recognize it. A natural percept is a precise spatiotemporal pattern across a large, specific population of neurons, the right cells firing in the right order at the right instant. Reproducing that from the outside is a generation problem with an enormous number of degrees of freedom.
• Electrical stimulation is a blunt instrument. Current spreads from an electrode tip and fires whatever excitable membrane is nearby, largely by proximity, not by function. You do not activate a chosen set of cells, you activate a rough sphere of tissue, synchronously and unnaturally, in a way the brain would never produce on its own. It is the opposite of the sparse, structured firing a real perception uses.
• We do not know the code well enough to inject a meaning. Even with a perfect electrode, you would need to know exactly which pattern spells "smooth" versus "rough" in this particular cortex, and we largely do not. Reading lets machine learning paper over that ignorance. Writing does not, because you cannot generate a specific code you cannot read out in the first place.

Reading finds a mapping from an existing signal. Writing has to synthesize a specific signal into a system whose language we cannot yet spell. That is the whole reason the two directions are so far apart.

Closing the loop, and the honest bottom line

Put the verbs together and you get the shape of a real system: a closed loop. Read the neural signal, decode the intent, act on it (move the cursor, drive the prosthetic hand), deliver feedback to the user, and read again. The feedback is the point of the loop. In the simplest systems the feedback is just vision: the user watches the cursor and adjusts their intent, and their plastic brain (BCI-2.2 again) learns to drive the decoder better over sessions. That visual loop is why BCIs improve with practice, and it runs entirely on the reading side. The far harder loop closes through stimulation, where the feedback is written back into sensory cortex so the user can feel the prosthetic without looking at it. That version is exactly as hard as the writing problem you just saw, which is why most deployed closed loops still close through the eyes, not through stimulation.

So the loop is asymmetric, and that asymmetry is the honest summary this whole track has been building toward: we read far better than we write. Decoding a trained movement intent into a smooth cursor or a robotic grip is a real, shipping capability, held back mainly by drift and calibration, both of which are hard engineering rather than mysteries. Writing a specific, natural percept back in is largely not a shipping capability at all, held back by biology we do not yet understand and by a stimulation tool too crude to speak the brain's language. When you read a headline about a brain-computer interface, run it through that filter. If it is about reading intent, be optimistic but ask about across-session drift. If it is about writing experience, be far more skeptical, because that is the side where the demos are thinnest and the biology is deepest. BCI-4.1 takes that filter and applies it to the actual state of the art, the real capabilities and the hard limits.

Key terms

decoding (reading)

Turning a recorded neural signal into the user's intended output, such as a cursor move or a letter, framed as supervised machine learning that infers intent from an existing pattern.

encoding (writing)

Putting information into the brain from the outside by stimulation, which must cause a specific neural pattern to arise rather than merely recognize one.

feature extraction

Reducing a raw neural recording to a compact vector per time window, typically firing rates for spikes or band power for field potentials, which is the model's actual input.

calibration session

A cued recording where the user attempts known movements so each feature vector gets a correct label, manufacturing the supervised training set a decoder needs.

drift (nonstationarity)

The gradual change over time in the relationship between neural features and intent, driven by electrode shift, tissue response, and ongoing brain plasticity, which degrades a fixed decoder.

adaptive decoder

A decoder that updates its parameters online as it runs to track drift, instead of relying on a single fixed calibration, at the cost of learning from data without clean labels.

deep brain stimulation (DBS)

A clinical therapy that runs steady high-frequency current through a deep electrode to modulate a whole circuit's activity, effective for tremor but coarse circuit modulation, not information writing.

closed-loop system

A BCI that reads, decodes, acts, delivers feedback, and reads again, where the feedback most reliably reaches the user through vision because writing feedback back in by stimulation is far harder.

Why an adaptive decoder is chasing a target with a changing ruler

The clean way to update a supervised model online is to keep receiving labeled examples and refit. A BCI cannot do that, because after the initial calibration there are no more labels: the user is doing real tasks, not attempting cued movements, so nobody stamps each incoming feature vector with the true intent. Adaptive decoders get around it with clever proxies. Some infer probable labels from context (if the cursor is near a target and moving toward it, assume the intent matched that target) and refit on those self-generated labels. Some retrain on brief recalibration blocks and interpolate between them. Some let the human do the adapting, holding the decoder fixed and relying on the user's plasticity to shift their neural pattern until it drives the frozen decoder well. Most real systems blend two-sided adaptation: the decoder tracks slow drift while the brain learns the decoder, two learners chasing each other. It works, but notice there is no fixed ground truth anywhere in that picture. You are estimating labels you cannot observe, on a signal whose statistics move, with a model that also moves. That is why across-session robustness, not peak within-session accuracy, is the metric that actually tells you whether a BCI is any good.