We Validated Studio Against Hand-Written Reference Pipelines — Exact Parity on Core Deterministic Paths

TL;DR: We ran the same BCI decoding pipelines twice — once through Nimbus Studio, once through independent hand-written scripts built directly on MNE-Python, scikit-learn, pyriemann, and braindecode. CSP+LDA and Riemannian MDM matched exactly; the tangent-space pipeline differed by 0.06 points in one LOSO summary, and xDAWN differed by 0.01 points. The deep-model differences remained within their own seed-to-seed training noise.

Read the full preprint →

The question every researcher should ask

Visual pipeline platforms like Studio promise reproducible benchmarking on public datasets, shared protocols, and lower barriers to entry. But they also introduce a trust problem that a diagram can't answer: does the pipeline you draw compute the same thing as the script you would have written?

Internal tests can't settle this — they're written by the same people who wrote the platform. So we did what the field did for low-cost EEG hardware a few years ago (shout-out to the Muse validation work): an external-style parity study, published with every number reproducible.

What we compared

Six standard pipelines, chosen to cover the execution paths a real lab uses:

• CSP + LDA — the classic motor-imagery workhorse
• Riemannian MDM — covariance-based classification
• Tangent space + logistic regression — run through Studio's bring-your-own-code node, i.e. as user-supplied Python in the platform's extensibility surface, not a built-in
• xDAWN + LDA — the standard P300 pipeline
• ShallowConvNet and EEGNet — deep models from braindecode

Three public datasets (BNCI-2014-001, BNCI-2014-004, and the EPFL P300 speller dataset BNCI-2014-009), two protocols (per-subject session holdout and leave-one-subject-out cross-validation through Studio's planned-CV driver), identical data files, matched hyperparameters, fixed seeds.

The reference side is a plain script with zero platform imports — the code a researcher would write from the libraries' documentation. For context on the platform layer around those libraries, see what Nimbus SDK adds beyond scikit-learn and pyRiemann.

The results

Session-holdout test accuracy (%) per subject, mean over subjects:

Leave-one-subject-out, pooled out-of-fold accuracy (%):

Core deterministic paths: exact where the table reports exact. CSP+LDA and MDM agreed with the reference to 0.00 points on all 54 session-holdout subject-runs and on every reported LOSO fold. The BYO tangent-space pipeline matched exactly in session holdout; its BNCI2014-001 LOSO summary differed by 0.06 points, with a maximum per-subject difference of 0.53 points. This is close parity, not universal exact identity.

The P300 pipeline: identical to floating-point precision. Inputs are bit-identical; the per-fold features agree to ~10⁻¹¹ relative. The residual difference is 0.01 points pooled — one borderline trial in 1,728 — which is what floating-point rounding does through an eigenvalue decomposition.

Deep models: within their own noise. ShallowConvNet's cross-implementation difference (3.6 points on average) is smaller than what either side moves on its own when you just change the random seed. EEGNet needed three seeds to show the same pattern — the second seed had made it look more stable than it is, and we report the three-seed numbers.

And the pooled cross-subject accuracies sit inside the range MOABB publishes for these datasets — so the numbers aren't just self-consistent, they're sane.

Why this matters for your next study

For the tested deterministic paths, Studio reproduced the reference results exactly in session holdout and within 0.06 points in the reported LOSO/P300 comparisons. The deep-model comparisons remained within seed-to-seed training noise, including when the model code was supplied through the BYO node. That's the precondition for using a platform in published research rather than as a whiteboard.

Combined with built-in LOSO/CV evaluation, BIDS export, and the bring-your-own-code node, the path from "I have a hypothesis" to "I have a per-subject result I can defend" no longer runs through a folder of versioned scripts.

Try it: studio.nimbusbci.com — the desktop app is a free download.

Read everything: the preprint (link above) ships with the complete harness and per-subject results, so you can check any number we report.

Per-subject accuracy, Nimbus vs reference: every deterministic run (circles: session holdout; squares: leave-one-subject-out) sits on the identity line; deep models (triangles) scatter within their own training noise.