YuE2: How Symbolic Planning Brings Transparency and Control to AI Music Generation
YuE2: How Symbolic Planning Brings Transparency and Control to AI Music Generation
Music generation models have made rapid progress, but most treat composition as a black box: you provide a text prompt, and audio emerges from a dense neural process that no one—including the model—can inspect or edit mid-flight. YuE2, introduced in a technical report from the multimodal-art-projection team published in late September 2026, takes a different approach. It generates a readable musical score first, then renders that score into audio. The result is a system that is both more controllable and, according to expert listening evaluations, more musical.
The Core Idea: Write the Score Before Playing the Notes
YuE2's central contribution is what the authors call symbolic planning — a staged generation process that mirrors how a human composer works. Rather than mapping a text prompt directly to audio tokens, the model first writes a structured score in ABC notation, a compact text format that encodes melody, chords, key, meter, tempo, and song structure. Only after this symbolic plan exists does the model proceed to audio synthesis.
This matters for two reasons. First, the symbolic plan acts as a commitment device: the model cannot contradict itself mid-song because the harmonic and melodic structure is already fixed. Second, the plan is human-readable and machine-editable, which opens the door to a class of interactions that audio-only models cannot support.
Architecture: AR–NAR Mixture-of-Transformers
Under the hood, YuE2 uses a 28-layer Mixture-of-Transformers (MoT) architecture that runs two expert types — autoregressive (AR) and non-autoregressive (NAR) — side by side, sharing attention computation but using separate normalization, projections, and feed-forward layers.
Generation is factorized into three representations:
1. Symbolic tokens (s): A sparse sequence encoding melody, chords, key, meter, tempo, and form — the ABC score.
2. Semantic music tokens (c): A dense 25-Hz sequence that bridges the abstract score and the acoustic signal.
3. Continuous acoustic latents (z): A 25-Hz sequence decoded into 48-kHz stereo audio by a variational autoencoder.
The AR component handles the symbolic and semantic sequences causally (left-to-right). The NAR component realizes the acoustic sequence via bidirectional flow matching, which allows it to condition on the full symbolic context when generating audio. The joint probability is:
p(s, c, z | y) = p_AR(s, c | y) × p_NAR(z | y, s, c)
where y is the text prompt (lyrics, style description, or both).
Training Without Aligned Scores: MERT2 and SheetSage2
A practical challenge for this approach is that most audio training data does not come with aligned symbolic annotations. The team addressed this by building two auxiliary models:
- SheetSage2 performs full-song lead-sheet transcription, converting audio recordings into ABC notation. It provides the symbolic supervision target s during training, allowing the model to learn from vast amounts of existing music without requiring hand-annotated scores.
- MERT2 is a multi-view music representation encoder that supplies the semantic tokens c. It achieves state-of-the-art results on 14 of 15 tasks in the MARBLE music understanding benchmark, making it a strong foundation for the semantic layer.
Both models are released alongside YuE2, which means the full pipeline — transcription, semantic encoding, and generation — is available for researchers to build on.
What Symbolic Planning Enables
The ABC score is not just an internal implementation detail; it is an explicit output that users and downstream systems can interact with.
Agentic music editing: Because the score is text, a language model can translate natural-language feedback ("make the chorus more upbeat" or "change the key to D minor") into specific score revisions. YuE2 then re-renders the modified score into audio, preserving the unedited sections. This creates a genuine edit loop rather than a generate-and-discard workflow.
Zero-shot covers: SheetSage2 can transcribe a reference recording into ABC notation. YuE2 can then apply a new style prompt while preserving the melodic structure, producing a stylistically transformed cover without any fine-tuning.
Targeted local edits: Because the score is structured, a user can modify a single measure — changing a chord voicing or a melodic phrase — and re-render only the affected section. Audio-only models have no equivalent mechanism.
Performance: Competitive with Proprietary Systems
On the WildSongBench (WSB) benchmark, YuE2 achieves a SongBench Global Average score of 6.73, the highest among public baselines at the time of the report. With a best-of-8 selection strategy, the score rises to 6.96 — the highest observed mean across all evaluated systems, including proprietary ones.
In expert listening evaluations, participants preferred the symbolic-planning output over non-planned generation from the same checkpoint at a rate of 49.3% vs. 34.6% (with the remainder expressing no preference). When compared directly against Suno v5, the best-of-8 configuration produced nearly balanced preferences — a notable result for an open-weight system competing against a closed commercial product.
Practitioner Implications
For developers building music-adjacent applications, YuE2 offers something that most generative audio models do not: a structured intermediate representation that can be inspected, validated, and modified programmatically. If you need to enforce harmonic constraints, ensure a specific song structure, or integrate user feedback into a generation loop, the ABC score gives you a natural intervention point.
The model weights, code, and both auxiliary models (MERT2 and SheetSage2) are available on GitHub and Hugging Face. The project page at map-yue2.github.io includes audio examples that demonstrate the cover and editing workflows.
Conclusion
YuE2's contribution is not just a better music generator — it is a demonstration that adding an explicit, inspectable intermediate representation to a generative pipeline can improve both quality and controllability simultaneously. The symbolic planning approach trades some generation speed for a system that composers, developers, and AI agents can actually work with, rather than just prompt and hope. As music generation matures from a novelty into a production tool, that kind of structured transparency is likely to matter more, not less.
1 reaction
10 projetos open source do GitHub que valem o seu tempo | Vol. 6
10 projetos open source do GitHub que valem o seu tempo, em contagem regressiva e com 10 segundos para cada um. Tem alternativa ao Discord, ao NotebookLM e ao Airtable rodando na sua máquina, música completa gerada por inteligência artificial aberta, VS Code no terminal e mais.
📺 Sexto volume da série Open source do GitHub. Todos com commit nas últimas duas semanas (conferido em 13/10/2026). Segue o canal e ativa as notificações para não perder a próxima.
Qual desses você vai testar primeiro? Conta nos comentários!
⏱️ CAPÍTULOS E LINKS
0:00 Abertura
0:10 #10 Terminal Code: o VS Code inteiro dentro do terminal
zenbu-labs/terminal-code
0:20 #9 CyberStrike: agente de IA para testes de invasão (use só em sistemas que você tem autorização para testar)
CyberStrikeus/CyberStrike
0:30 #8 Android Reverse Engineering: skill do Claude Code que descompila apps Android e mapeia as APIs
SimoneAvogadro/android-reverse-engineering-skill
0:40 #7 OpenViking: banco de contexto para agentes, com memória, conhecimento e skills em pastas
volcengine/OpenViking
0:50 #6 TestSprite: testa o seu app como usuário real e mostra ao agente o que quebrou (a CLI usa uma chave da plataforma TestSprite)
TestSprite/testsprite-cli
1:00 #5 Haven: alternativa privada ao Discord no seu próprio servidor
ancsemi/Haven
1:10 #4 Baserow: alternativa open source ao Airtable, sem código
baserow/baserow
1:20 #3 YuE2: músicas completas, com voz e arranjo, a partir da letra
multimodal-art-projection/YuE
1:30 #2 SurfSense: alternativa ao NotebookLM que roda no seu computador
MODSetter/SurfSense
1:40 Oferecimento Hallen Tech
1:54 #1 OpenPost: alternativa open source ao Buffer, Canva e CapCut
getopenpost/openpost
2:04 Encerramento
🧡 Oferecimento: Hallen Tech, liderança de tecnologia sob demanda, software, cloud e segurança para a sua empresa.
Site: hallentech.com
E-mail: contato@hallentech.com
Instagram: instagram.com
LinkedIn da Hallen Tech: linkedin.com
LinkedIn do Eddwan: linkedin.com
Se curtiu, deixa o like, segue o canal e ativa as notificações para a próxima lista.
#opensource #github #inteligenciaartificial
52 likes · 1,026 views
Mentions
YuE2.. lyrics குடுத்தா song ready. Melody + chords plan பண்ணி vocals-ஓட render பண்ணுது. Free demo உண்டு.. Suno-வுக்கு open-source. உங்க lyrics try பண்ணுவீங்களா ? multimodal-art-projection/YuE Instagram-லும் பின்தொடருங்கள்: instagram.com
YuE2: How Symbolic Planning Brings Transparency and Control to AI Music Generation YuE2: How Symbolic Planning Brings Transparency and Control to AI Music Generation Music generation models have made rapid progress, but most treat composition as a black box: you provide a text prompt, and audio emerges from a dense neural process that no one—including the model—can inspect or edit mid-flight. YuE2, introduced in a technical report from the multimodal-art-projection team published in late September 2026, takes a different approach. It generates a readable musical score first, then renders that score into audio. The result is a system that is both more controllable and, according to expert listening evaluations, more musical. The Core Idea: Write the Score Before Playing the Notes YuE2's central contribution is what the authors call symbolic planning — a staged generation process that mirrors how a human composer works. Rather than mapping a text prompt directly to audio tokens, the model first writes a structured score in ABC notation, a compact text format that encodes melody, chords, key, meter, tempo, and song structure. Only after this symbolic plan exists does the model proceed to audio synthesis. This matters for two reasons. First, the symbolic plan acts as a commitment device: the model cannot contradict itself mid-song because the harmonic and melodic structure is already fixed. Second, the plan is human-readable and machine-editable, which opens the door to a class of interactions that audio-only models cannot support. Architecture: AR–NAR Mixture-of-Transformers Under the hood, YuE2 uses a 28-layer Mixture-of-Transformers (MoT) architecture that runs two expert types — autoregressive (AR) and non-autoregressive (NAR) — side by side, sharing attention computation but using separate normalization, projections, and feed-forward layers. Generation is factorized into three representations: 1. Symbolic tokens (s): A sparse sequence encoding melody, chords, key, meter, tempo, and form — the ABC score. 2. Semantic music tokens (c): A dense 25-Hz sequence that bridges the abstract score and the acoustic signal. 3. Continuous acoustic latents (z): A 25-Hz sequence decoded into 48-kHz stereo audio by a variational autoencoder. The AR component handles the symbolic and semantic sequences causally (left-to-right). The NAR component realizes the acoustic sequence via bidirectional flow matching, which allows it to condition on the full symbolic context when generating audio. The joint probability is: p(s, c, z | y) = p_AR(s, c | y) × p_NAR(z | y, s, c) where y is the text prompt (lyrics, style description, or both). Training Without Aligned Scores: MERT2 and SheetSage2 A practical challenge for this approach is that most audio training data does not come with aligned symbolic annotations. The team addressed this by building two auxiliary models: - SheetSage2 performs full-song lead-sheet transcription, converting audio recordings into ABC notation. It provides the symbolic supervision target s during training, allowing the model to learn from vast amounts of existing music without requiring hand-annotated scores. - MERT2 is a multi-view music representation encoder that supplies the semantic tokens c. It achieves state-of-the-art results on 14 of 15 tasks in the MARBLE music understanding benchmark, making it a strong foundation for the semantic layer. Both models are released alongside YuE2, which means the full pipeline — transcription, semantic encoding, and generation — is available for researchers to build on. What Symbolic Planning Enables The ABC score is not just an internal implementation detail; it is an explicit output that users and downstream systems can interact with. Agentic music editing: Because the score is text, a language model can translate natural-language feedback ("make the chorus more upbeat" or "change the key to D minor") into specific score revisions. YuE2 then re-renders the modified score into audio, preserving the unedited sections. This creates a genuine edit loop rather than a generate-and-discard workflow. Zero-shot covers: SheetSage2 can transcribe a reference recording into ABC notation. YuE2 can then apply a new style prompt while preserving the melodic structure, producing a stylistically transformed cover without any fine-tuning. Targeted local edits: Because the score is structured, a user can modify a single measure — changing a chord voicing or a melodic phrase — and re-render only the affected section. Audio-only models have no equivalent mechanism. Performance: Competitive with Proprietary Systems On the WildSongBench (WSB) benchmark, YuE2 achieves a SongBench Global Average score of 6.73, the highest among public baselines at the time of the report. With a best-of-8 selection strategy, the score rises to 6.96 — the highest observed mean across all evaluated systems, including proprietary ones. In expert listening evaluations, participants preferred the symbolic-planning output over non-planned generation from the same checkpoint at a rate of 49.3% vs. 34.6% (with the remainder expressing no preference). When compared directly against Suno v5, the best-of-8 configuration produced nearly balanced preferences — a notable result for an open-weight system competing against a closed commercial product. Practitioner Implications For developers building music-adjacent applications, YuE2 offers something that most generative audio models do not: a structured intermediate representation that can be inspected, validated, and modified programmatically. If you need to enforce harmonic constraints, ensure a specific song structure, or integrate user feedback into a generation loop, the ABC score gives you a natural intervention point. The model weights, code, and both auxiliary models (MERT2 and SheetSage2) are available on GitHub and Hugging Face. The project page at map-yue2.github.io includes audio examples that demonstrate the cover and editing workflows. Conclusion YuE2's contribution is not just a better music generator — it is a demonstration that adding an explicit, inspectable intermediate representation to a generative pipeline can improve both quality and controllability simultaneously. The symbolic planning approach trades some generation speed for a system that composers, developers, and AI agents can actually work with, rather than just prompt and hope. As music generation matures from a novelty into a production tool, that kind of structured transparency is likely to matter more, not less.
🔥 YuE2: Open-Source Frontier AI Music Generation! 🎵 🔥 YuE2: Open-Source Frontier AI Music Generation! 🎵 💡 What is it: YuE2 is a breakthrough open-source AI framework that unifies symbolic composition and audio generation to create high-fidelity, full-length music tracks from text and prompt controls. 🚀 Why it's blowing up today: • Gained +1,360 forks in a single day! • Reached 11,900+ total stars on GitHub • Ranked as the #1 trending open-source repository globally ⚡ Key Features: • Symbolic planning for precise note and structure control • Zero-shot song covers directly from original audio • Agentic music editing for interactive track manipulation • Powered by advanced MERT2 and SheetSage2 architecture 🛠️ Quickstart command: git clone multimodal-art-projection/YuE 💻 Language & Tech stack: Python, PyTorch, Hugging Face Transformers 🔗 How to get the link: Drop a comment below and get the repository details sent straight to your inbox! 🌐 Website CTA: Discuss this tool & discover more daily dev reels on fondpeace.com 👇 Call to Action: Comment 'YUE' below and I'll DM you the link! 🏷️ Hashtags: #github #python #aimusic #musicgeneration #generativeai #opensource #machinelearning #deeplearning #ai #developer #coding
Top 5 projetos open source do GitHub | Vol. 6 #shorts Os projetos #5 a #1 do Top 10 open source do GitHub, Vol. 6, 10 segundos para cada um. #5 Haven: ancsemi/Haven #4 Baserow: baserow/baserow #3 YuE2: multimodal-art-projection/YuE #2 SurfSense: MODSetter/SurfSense #1 OpenPost: getopenpost/openpost Vídeo completo com os 10 no canal Hallen Tech. Oferecimento: Hallen Tech, hallentech.com #shorts #opensource #github
10 projetos open source do GitHub que valem o seu tempo | Vol. 6 10 projetos open source do GitHub que valem o seu tempo, em contagem regressiva e com 10 segundos para cada um. Tem alternativa ao Discord, ao NotebookLM e ao Airtable rodando na sua máquina, música completa gerada por inteligência artificial aberta, VS Code no terminal e mais. 📺 Sexto volume da série Open source do GitHub. Todos com commit nas últimas duas semanas (conferido em 13/10/2026). Segue o canal e ativa as notificações para não perder a próxima. Qual desses você vai testar primeiro? Conta nos comentários! ⏱️ CAPÍTULOS E LINKS 0:00 Abertura 0:10 #10 Terminal Code: o VS Code inteiro dentro do terminal zenbu-labs/terminal-code 0:20 #9 CyberStrike: agente de IA para testes de invasão (use só em sistemas que você tem autorização para testar) CyberStrikeus/CyberStrike 0:30 #8 Android Reverse Engineering: skill do Claude Code que descompila apps Android e mapeia as APIs SimoneAvogadro/android-reverse-engineering-skill 0:40 #7 OpenViking: banco de contexto para agentes, com memória, conhecimento e skills em pastas volcengine/OpenViking 0:50 #6 TestSprite: testa o seu app como usuário real e mostra ao agente o que quebrou (a CLI usa uma chave da plataforma TestSprite) TestSprite/testsprite-cli 1:00 #5 Haven: alternativa privada ao Discord no seu próprio servidor ancsemi/Haven 1:10 #4 Baserow: alternativa open source ao Airtable, sem código baserow/baserow 1:20 #3 YuE2: músicas completas, com voz e arranjo, a partir da letra multimodal-art-projection/YuE 1:30 #2 SurfSense: alternativa ao NotebookLM que roda no seu computador MODSetter/SurfSense 1:40 Oferecimento Hallen Tech 1:54 #1 OpenPost: alternativa open source ao Buffer, Canva e CapCut getopenpost/openpost 2:04 Encerramento 🧡 Oferecimento: Hallen Tech, liderança de tecnologia sob demanda, software, cloud e segurança para a sua empresa. Site: hallentech.com E-mail: contato@hallentech.com Instagram: instagram.com LinkedIn da Hallen Tech: linkedin.com LinkedIn do Eddwan: linkedin.com Se curtiu, deixa o like, segue o canal e ativa as notificações para a próxima lista. #opensource #github #inteligenciaartificial
first AI music video not bad for this being my first week of song and music video generation. generated with all open source tools excluding my use of Claude Code Tech Stack - YuE2 song generation: multimodal-art-projection/YuE - Runpod Flash for the hosted endpoint: runpod/flash - Blockbuster Studio for video generation: Kilo-Loco/blockbuster-studio - Claude Code for lyrics and prompts: anthropics/claude-code
YuE2：融合符号化规划、零样本翻唱与智能体音乐编辑的前沿音乐生成技术。 multimodal-art-projection/YuE t.co
YuE2 的在线 Demo 免登录，每个浏览器能免费生成 3 次~ 🎧 在线试 yue.noizai.net 🎸 开源指路 multimodal-art-projection/YuE
Lux In Music experiment in progress with @NTTLuke Suno killer is here: Yue2 3B of pure music power! Here running on M5 Max Model here: huggingface.co Original Repo here: multimodal-art-projection/YuE Lux In Music coming soon! Text to music, Studio, Stem, Edit and more. 🚀 t.co
Suno 按条收费，港科大的开源平替 1.1万星。 丢歌词和风格，先给一份能改的曲谱，再渲染成带人声的整首歌；还能扒歌翻唱、对话改谱重录。 官方评测第一，高过 Suno v5/v6。3B 模型 24G 显存本地跑，没显卡官网免费试，可商用： multimodal-art-projection/YuE #AI音乐 #Suno平替 #YuE2 #开源 t.co