Can audio LLMs hear the difference yet?
Each test item is two short clips and one question about melody, rhythm, timbre or harmony. Every item comes in two variants: in FLIP the music changes and the answer switches, in STAY only the sound changes (room echo, EQ, hiss, transposition) and the answer stays. A model that ignores the audio scores 50%.
Same or different melody?
Rankings
Rendered test set: 1,000 clean items (250 per task) and 2,000 FLIP/STAY items. All numbers are percentages; the thin tick on each bar marks 50%. β models, last updated β.
Real-recordings test set: 600 clean items (150 per task) and 1,200 FLIP/STAY items, built from guitar, instrument and drum recordings; no model is trained on it. A reply without an answer letter counts as wrong. All numbers are percentages; the thin tick on each bar marks 50%. Models not yet run on this set show βββ.
| # | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Loading results⦠| |||||||||||
Melody β¦ Clean (all): accuracy on the clean items; βallβ is the mean of the four tasks. AccFS: accuracy on the FLIP and STAY items, 100 β (MR + FFR) / 2; a model that ignores the audio scores 50% whatever letter it prefers. MR (miss rate): share of FLIP items answered wrongly. FFR (false-flip rate): share of STAY items answered wrongly. Lower is better for MR and FFR. A rate: share of βAβ answers on the clean items, so letter bias is visible. Models are ranked by AccFS within each group. Music-specific models, audio LLMs built for music understanding, are listed in their own groups below the general-purpose models. The real-recordings test set uses the same four tasks, questions, letter rule and stay changes, with clips cut from GuitarSet, Medley-solos-DB and MUSDB18-HQ drum stems; it is for evaluation only. Self-reported results have not yet been reproduced by the maintainers.