Watch the walkthrough
A short video tour of this page — the ideas behind it, followed by a live playthrough of the tools below.
Record a live session
Modular multimodal capture for guitar practice: audio from a
Schecter
Demon-7 into a
Focusrite
Scarlett Solo (any guitar and input works), periodic
desktop screenshots, webcam snapshots and MediaPipe
fretting-hand keypoints, plus EEG and cardiac from the same
local bridge as the Affective Computing page
(connect hardware there; it stays connected across pages). Tick what you
want; everything records into one time-aligned NDJSON session — browser
download, local disk via the bridge, or S3. Tick score position and
play along below: the take records which tab measure you were on, giving
each recording an audio↔score alignment that
gaps/finetune_personal.py turns into training data — so the
model improves on your guitar, room, and pickup.
Pick what records
Choose sensors None selected
Destination & take
Chosen before the take — shared with the EEG page's setting.
tick sensors on the left, then start
Hear an amp, watch the model listen
A zero-added-latency amp chain for monitoring the Scarlett's DI — takes still record dry, with the voicing saved to the session metadata. The same signal feeds the GAPS front end live: the CRNN via onnxruntime-web when exported, otherwise the notebook's NMF arm in JS, with posteriors scrolling as a piano roll and committed notes lighting the fretboard.
amp off
looking for exported weights…
—
transcription off
Play along
Two practice tracks wired into the page: the Periphery tune I'm learning
(drop-C variant: C G C E G D), and a classical
étude from the GAPS held-out test split (270_Jw1wc — the
same recording the model above is scored on). Six strings vibrate with
the live spectrum, and a cursor tracks the tab measure by measure — at a
constant rate for Scarlet, riding the dataset's audio-aligned sync
points for the étude, rubato and all. Click any measure to jump there,
or slow it down to learn the fast bits. The third entry is a play-along
cover video for Scarlet with the tab burned into the frame.
standard tuning, nylon strings — cursor timing comes from the GAPS sync points aligned to this exact recording, rubato and all
“Scarlet” instrumental cover with screen tabs by OLD MOON CHILD. An embedded video's audio is cross-origin, so the strings can't tap it directly — hit Feed the analyser, pick this tab and tick share tab audio, and the spectrum drives the strings from the clip you're hearing.
The scoreboard
The baseline is deliberately honest — a fixed dictionary derived from the harmonic series, with only a threshold and a minimum duration grid-searched on validation. That's what makes the gap meaningful rather than a strawman.
right pitch, right start published benchmark 0.943 0.575 0.920 ★
…and the right ending 0.176 0.556
which pitches sound, frame by frame 0.497 0.764
Benchmark line: the published GAPS reference CRNN (larger, pretrained, velocity-aware) at 0.943 note onset F on this split.
What the numbers say
A clean 30-for-30
Not one test piece — no genre, tempo, or recording condition — where the harmonic-template baseline beats the network. Every point in the per-piece scatter sits above the diagonal. +34.5 points of note-onset F for 11.9M learned parameters.
2.3 points from the benchmark
A from-scratch model, 30 epochs, one consumer GPU: 0.920 against the GAPS paper's pretrained, velocity-aware reference at 0.943. The README predicted the gap as "the price of a portfolio-scale training budget" — it turned out to be small.
Nobody knows when a note ends
Require the note's ending to be right and both arms crater: 0.920 → 0.556 for the CRNN, 0.575 → 0.176 for NMF. Guitar strings decay rather than stop — the offset is genuinely ambiguous, to models and humans alike.