MQAScore: Fine-Grained Text-Music Alignment Evaluation via Large Audio Language Model
Abstract
Text-to-music alignment is commonly evaluated by the global similarity between text and audio embeddings, which collapses all described attributes into a single score and obscures which ones are missing. We instead introduce MQAScore, which verifies each atomic musical concept along five clip-level dimensions (genre, instrument, mood/theme, vocal, and usage) by the probability that a large audio language model (LALM) answers affirmatively to a binary question. To assess evaluator reliability, we construct verification labels from presence/absence annotations and LLM-guided concept substitutions, and propose the Sensitivity-Invariance Ratio, which measures how selectively an evaluator responds to meaning-changing substitutions. Experiments show that concept-level scoring improves both question-answering and similarity-based evaluators, and MQAScore can be a more reliable and interpretable fine-grained complement to global similarity for evaluating text-to-music alignment.
How MQAScore Works
A global similarity score (e.g. CLAP or MuQ) or a whole-caption question-answering score (AQAScore) summarizes every attribute of a prompt into a single number — so when a generated clip nails the mood but misses the instrument, that trade-off is invisible. MQAScore instead decomposes each caption into atomic concepts and scores each one separately. For a concept t, it asks an LALM a yes/no question (e.g. "Does this music belong to the genre pop? Answer Yes or No.") and reads the score off the answer probabilities:
S(a, t) = P(“Yes”) P(“Yes”) + P(“No”)
Each caption therefore yields a per-dimension score vector instead of one global value. The five clip-level dimensions, grounded in CAL500, are:
Demo · MusicEval
Each pair shows two AI-generated clips written from the same text prompt (drawn from the MusicEval / AudioMOS 2025 test set). We deliberately picked pairs whose overall scores are nearly tied — CLAP, MuQ, and whole-caption AQAScore rate the two clips almost identically. Yet the clips clearly differ on specific described attributes. MQAScore reveals exactly where: by probing each concept individually, it shows which clip actually delivered the pop genre, the violin, the tranquil mood, and so on — the fine-grained detail a single global number hides. Every detected concept is highlighted in the prompt by its dimension; cells are tinted toward the clip an LALM judges to better match each concept, and the key row marks the attribute that drives the two clips apart. Use the selector or the ◀ ▶ arrows to move between the 5 pairs. Each clip also lists its human ratings — TA (text alignment) and OQ (overall quality), mean opinion scores from 1–5.
Concepts are auto-extracted by an LLM (Qwen3-4B) and are not accuracy-checked; we show the 5 pairs with the cleanest extraction (see concept audit).
Demo · SOTA Generators
Here we compare four state-of-the-art text-to-music generators (MusicGen-Large, Stable Audio 3, MeanAudio, ACE-Step 1.5) on the same prompt, with the original recording (Ground Truth) shown for reference. As in the previous section, prompts are chosen so that CLAP rates all four generators almost identically — yet the clips clearly realize the prompt differently. MQAScore separates the generators and says on which attribute: every atomic concept in the prompt is highlighted below by its dimension, and the per-attribute table shows which generator actually delivered each one. Use the selector or the ◀ ▶ arrows to move between the 10 prompts (5 MusicCaps + 5 Song Describer).
Concepts are auto-extracted by an LLM (Qwen3-4B) and are not accuracy-checked; we show the 5 prompts per dataset with the cleanest extraction (see concept audit).
BibTeX
TBD — citation will be available upon publication.