XHS Β· Hum to Symphony
From 20 seconds of free humming to a fully structured, 90-bar orchestral piece β
with harmony solved by dynamic programming in neo-Riemannian space, not predicted on a whim.
The project
XHS deliberately avoids end-to-end black-box generation. The task is decomposed into an explainable, controllable, verifiable five-stage pipeline β transcription, harmonic planning, melody development, orchestration, rendering β and every stage emits an inspectable symbolic artifact (MIDI, chord plan, form markers).
basic-pitch converts audio to MIDI, followed by conservative cleaning: wobble filtering, key estimation with confidence, optional scale and rhythm-grid snapping.
NRT-DP runs a boundary-aware dynamic program over the PLR graph, realizing the per-section remoteness profile with exact cadence and terminal returns.
A theme-and-variations engine: the user's theme is restated verbatim in the A-sections and developed by sequencing and fragmentation in B/Dev.
AccoMontage-3 (NeurIPS 2024) conditions on the lead sheet β melody, chord plan and form markers β to produce multi-track orchestral MIDI.
FluidSynth with the FluidR3 sound font renders audio; section dynamics and melody-presence post-processing keep the texture balanced.
The theme's pitches and rhythm survive verbatim at every presentation (97.7%β100% index-aligned pitch-class identity against the original hum) β what you hum is what you hear.
A quiet Intro, a wandering B, a Dev section at maximal remoteness, a Coda that comes home β the tension curve is explicitly specified as a target profile and realized exactly by the planner.
The planner is pure-standard-library and deterministic; every intermediate artifact (chords_by_bar, form markers, melody MIDI) can be inspected and swapped independently.
Core technology
The state space is the 24 major/minor triads. Adjacent harmonies may only move by three parsimonious transformations β P (parallel), L (leading-tone exchange), R (relative) β plus HOLD. The structural remoteness of a triad is its shortest-path distance from the tonic in this graph.
Given a tonic, section lengths, a target mean-remoteness profile Ο over sections, section-end target sets and a terminal return condition, the planner runs dynamic programming on the state (previous-previous chord, current chord, cumulative remoteness), optimizing lexicographically: section-mean error Jβ, cadence-window approach Jβ, and HOLD/oscillation regularization Jβ (Ξ» = 0.15 / 0.60). Boundary-aware initialization keeps the whole multi-section trajectory continuous.
Exact-deadline feasibility sets Fr(C) prune every partial path that cannot land exactly on its section's required ending β and with section lengths β₯ 6 the sequential planner provably never goes infeasible (graph diameter D = 5).
| Metric | Greedy heuristic | NRT-DP |
|---|---|---|
| Target MAE β | 0.366 | 0.018 |
| Contrast MAE β | 0.288 | 0.036 |
| Max adjacent distance | 2 (violated) | 1 (strict) |
| Section-end hit rate | 0.86β1.00 | 1.00 |
| Terminal return | β | β |
Melody-aware extension: a b9-avoidance / coverage cost term added to the objective lets the harmony adapt to the user's melody pitches β coverage at presentation bars rose from 0.24 to 0.63, with planning metrics unchanged.
Code
Three real excerpts: the NRT-DP planner core, the melody engine's only pitch rule, and the JavaScript planner running live on this very page (switch with the tabs above the window).
Listen
The complete chain from a single 23-second original hum (F major).
The user's theme after transcription and cleaning (Voice Oohs)
Theme-and-variations melody over the NRT-DP chord pad
AccoMontage-3 orchestration: strings / horn / piano / bass
The same melody and harmony in a different instrumentation
Try it
A complete JavaScript port of the paper's reference planner, running live in your browser β pick a tonic and a tension profile, watch dynamic programming solve a 90-bar harmonic backbone, and listen to it right away.