On-device YIN pitch detection, a note that lights green only once you hold it, four scores measured on the phone, and only those numbers sent to the cloud. The build story, including the three mistakes.
Every figure on this page is read from the source files listed at the end. Checked against Android build 0.124.1 (167).
Voxstrum is a music-learning app for Android, written in Flutter, whose main trick is that it listens to you play and lights each note green when it hears it. There is no magic in that sentence, only a pipeline: capture, detect, match, score, and then one small trip to the cloud. This page walks through it with the actual numbers, because "AI listens to you" is exactly the kind of claim that deserves to be checked.
The recorder plugin streams PCM16 at 44.1 kHz. A single class, PitchStream, owns the microphone: it decodes the byte stream, cuts it into 4096-sample windows (about 93 ms each), runs the detector on each window, smooths the result and publishes a frame. A frame carries the detected fundamental in Hz, the RMS level and the sample count. When a window holds no confident pitch, the frequency is null, which the UI must render as "no answer" and never as a note. It is never substituted with zero.
Three screens consume that stream: the chromatic tuner, the tuning wizard and the lesson's live note visualiser. They do not all want the same cadence, so the stream has two frame policies:
The input buffer is capped at two windows. An unbounded queue is not a tuning choice, so that part was fixed outright; the cadence differences were kept, for a reason that comes up again below.
Plain YIN, in Dart, no FFI: the difference function, then the cumulative mean normalised difference, then the first local minimum below 0.10, which picks the fundamental rather than a sub-octave, then parabolic interpolation around that minimum for sub-sample accuracy. Before any of that runs, a mean-square gate (0.00005) throws away silence. It is fast enough on a mid-range phone.
Each call site has its own frequency window, because the instruments differ, and one immutable class holds all four presets with the differences in a table in the file header:
| Preset | Range | Fallback reject above | Minimum samples | Why |
|---|---|---|---|---|
tuner | 28–1760 Hz | 0.35 | none | Has to reach a bass low E at the bottom and a violin's high end at the top. |
tuningWizard | 28–1320 Hz | 0.35 | none | Tunes open strings, which never reach A6, so it caps lower. |
liveVisualiser | 60–1320 Hz | 0.25 | 512 | The Record step, every instrument but bass. Stricter about uncertainty so background noise reads as "no pitch" rather than as a hit; the 60 Hz floor sits above 50 Hz mains hum. |
liveVisualiserBass | 38–1320 Hz | 0.25 | 512 | Bass only: low E is 41 Hz and A is 55 Hz, both under the 60 Hz floor. Lowering the floor lets 50 Hz hum into the range, which is exactly why the other instruments keep it. |
The 4096-sample window costs no extra latency for bass: a 41 Hz period is about 1070 samples, so the lag search fits several times over.
The lesson knows its target notes as MIDI numbers. The detected frequency becomes a fractional MIDI note (69 is A4 at 440 Hz), and the closest unhit target within ±50 cents, half a semitone, is the candidate for this window. That is forgiving for beginners and tight enough to mean something.
A candidate does not count yet. The visualiser has a sustain gate: the pitch has to stay in tune continuously for 220 ms before the target counts as hit and the dot lights green, like a tuner. A transient noise spike, or a wrong note brushing past the target on its way somewhere else, cannot register. On the hit, a NoteAttempt is recorded: the target MIDI, hit, the signed cents error at the moment of the match, and the onset time in ms. When every dot in the current event has lit, the screen dwells for 500 ms and advances. Targets never hit are recorded as misses at the end of the take.
From that list of attempts, the phone computes four 0–100 scores. Each one documents exactly what it measures, and that text doubles as the in-app "what does this mean?" copy:
Only score what can honestly be measured. No targets at all, an unpitched instrument such as drums, or a refused microphone produce an unmeasured result, and the app shows "practice logged, not scored" with coaching only. Targets with no hits are a real signal, accuracy 0, but still flagged unmeasured overall, so the screen stays coaching-first instead of showing four zeros as if they were graded precision. It does not invent numbers.
The four scores, the hit count and the lesson's metadata go to one Cloud Function. The function first spends one credit from the account's monthly allowance inside a Firestore transaction, so the cap cannot be bypassed by clearing app data on the client. It then calls a small Claude model (Haiku) with a cached system prompt and returns a coaching note of two or three sentences grounded in those numbers. The prompt is explicit that the model writes coaching language only and never the scores; when the take was unmeasured it gets effort-based coaching instead of numbers.
The lesson recording never leaves the phone. What crosses the network is a handful of integers and the lesson name. That is the whole point for a learning app, and it is also why the scores had to become real first.
The first version sent the lesson description to the model and asked it for scores. It never heard anything. Replacing that with measured metrics was the single biggest change in the app, and it is why the unmeasured state exists: once the numbers were real, the app had to admit when there were none.
The YIN code had been pasted into the tuner, the wizard and the visualiser, and a comment said they were "the same algorithm". They had drifted on four parameters: the frequency floor, the ceiling, the fallback rejection and the minimum buffer, and only one of the three had a test. I did not unify the values, because they had been tuned on a device, and changing detection sensitivity blind is how a tuner starts telling people to detune a working instrument. They became parameters of one immutable class, with the table of differences in the file header, so the divergence is visible instead of buried.
One of those differences was a bug rather than a choice. The visualiser's 60 Hz floor sat above a bass's bottom two strings, so a learner playing low E or A got no feedback at all on the one screen that exists to give it. The fix is the bass-only preset in the table above, with the floor at 38 Hz and a test that measures the range. It was not made the default, because the lower floor gives up hum rejection, and every other instrument is better off keeping it.
Each screen opened its own recorder, chose its own frame cadence, smoothed its own output and cleaned up its own way, and none of them watched the app lifecycle. Android keeps the stream nominally open when you background the app, feeds it silence, and does not recover on resume, so a learner who took a phone call mid-tune came back to a tuner that would never read again, with no error. A second defect hid behind the first: starting the stream was unguarded, so when another app held the microphone the tuner sat on a dimmed "Starting…" forever, which looks identical to "waiting for you to play".
One PitchStream class now owns start, stop, lifecycle and faults. start() returns false and publishes a typed fault, permissionDenied or unavailable, instead of throwing, because "another app has the mic" is an ordinary outcome, not an exception. A stall watchdog re-opens a stream that has delivered nothing for two seconds, and after three failed re-opens it reports unavailable rather than retrying behind a screen that still says "play your low E". The cadence and the buffer bound were deliberately not unified, for the same reason as the detector presets; they are parameters now, in one place.
Every number above is read from these files in the Voxstrum source, as of build 0.124.1 (167):
lib/pitch_detector.dart: the YIN implementation, the 0.10 threshold, the silence gate and the four presets.lib/pitch_stream.dart: the microphone owner, the 4096-sample window, the two frame policies, the lifecycle handling, the typed faults and the stall timeout.lib/live_note_visualiser.dart: the ±50-cent match, the 220 ms sustain gate, the 500 ms dwell and the NoteAttempt capture.lib/performance_scorer.dart: the four scores and the unmeasured rules.functions/src/index.ts (analyseSession): the credit transaction, the model call and the "never invent scores" instruction.Voxstrum is in closed testing on Google Play, for adults. Plainer facts about what it does are on the facts page.