Syn-Omni: Structured Specialization and Progressive Collaboration for Omnimodal Embeddings
Omnimodal embeddings naturally involve both shared representations and modality-specific features across heterogeneous inputs. However, existing omnimodal embedding methods often rely on a single shared parameter space over mixed-modality data, limiting structural separation between universal and modality-specific representations. To address this, we propose Syn-Omni, a unified framework for structured omnimodal adaptation with modality specialization and controlled cross-modal collaboration. Specifically, we introduce Orthogonal Modality-Expert LoRA (OME-LoRA), which decomposes adaptation into a shared LoRA path for universal semantics and modality-expert LoRA paths for modality-aware specialization. Furthermore, Progressive Synergy Routing (PSR) enables experts to first establish modality-specific priors, then gradually interact with other modality-experts for cross-modal synergy. Evaluated across 81 diverse tasks spanning image, video, audio, and audiovisual modalities, Syn-Omni consistently outperforms omnimodal baselines, demonstrating the effectiveness of structured specialization and cross-modal progressive collaboration.
Comments: Accepted to EMNLP 2026 (Long, Findings). Code: sony/syn-omni
Mentions
Syn-Omni: Structured Specialization and Progressive Collaboration for Omnimodal Embeddings @ytaek_oh et al. at Sony use a shared LoRA path plus modality expert paths for omnimodal embeddings. 📝 arxiv.org 👨🏽💻 sony/syn-omni
Syn-Omni: Structured Specialization and Progressive Collaboration for Omnimodal Embeddings Sony uses a shared LoRA path and modality expert paths for omnimodal embeddings, with routing that gradually lets experts interact. 📝 arxiv.org 👨🏽💻 sony/syn-omni
Syn-Omni: Structured Specialization and Progressive Collaboration for Omnimodal Embeddings Omnimodal embeddings naturally involve both shared representations and modality-specific features across heterogeneous inputs. However, existing omnimodal embedding methods often rely on a single shared parameter space over mixed-modality data, limiting structural separation between universal and modality-specific representations. To address this, we propose Syn-Omni, a unified framework for structured omnimodal adaptation with modality specialization and controlled cross-modal collaboration. Specifically, we introduce Orthogonal Modality-Expert LoRA (OME-LoRA), which decomposes adaptation into a shared LoRA path for universal semantics and modality-expert LoRA paths for modality-aware specialization. Furthermore, Progressive Synergy Routing (PSR) enables experts to first establish modality-specific priors, then gradually interact with other modality-experts for cross-modal synergy. Evaluated across 81 diverse tasks spanning image, video, audio, and audiovisual modalities, Syn-Omni consistently outperforms omnimodal baselines, demonstrating the effectiveness of structured specialization and cross-modal progressive collaboration. Comments: Accepted to EMNLP 2026 (Long, Findings). Code: sony/syn-omni