Speech-Worthy Alignment for Japanese SpeechLLMs via Direct Preference Optimization
arXiv:2603.12565v1 Announce Type: cross Abstract: SpeechLLMs typically combine ASR-trained encoders with text-based LLM backbones, leading them to inherit written-style output patterns unsuitable for text-to-speech synthesis. This mismatch is particularly pronounced in Japanese, where spoken and written registers differ substantially in politeness markers, sentence-final particles, and syntactic complexity. We propose a preference-based alignment approach to adapt Japanese SpeechLLMs for speech-worthy outputs: text that is concise, conversational, and readily synthesized as natural speech. To rigorously evaluate this task, we introduce SpokenElyza, a benchmark for Japanese speech-worthiness derived from ELYZA-tasks-100 with auditory verification by native experts. Experiments show that our approach achieves substantial improvement on SpokenElyza while largely preserving performance on the original written-style evaluation. We will release SpokenElyza to support future research on Japa
arXiv:2603.12565v1 Announce Type: cross Abstract: SpeechLLMs typically combine ASR-trained encoders with text-based LLM backbones, leading them to inherit written-style output patterns unsuitable for text-to-speech synthesis. This mismatch is particularly pronounced in Japanese, where spoken and written registers differ substantially in politeness markers, sentence-final particles, and syntactic complexity. We propose a preference-based alignment approach to adapt Japanese SpeechLLMs for speech-worthy outputs: text that is concise, conversational, and readily synthesized as natural speech. To rigorously evaluate this task, we introduce SpokenElyza, a benchmark for Japanese speech-worthiness derived from ELYZA-tasks-100 with auditory verification by native experts. Experiments show that our approach achieves substantial improvement on SpokenElyza while largely preserving performance on the original written-style evaluation. We will release SpokenElyza to support future research on Japanese spoken dialog systems.
Executive Summary
This study addresses a critical issue in Japanese SpeechLLMs by proposing a novel alignment approach to adapt their text-based backbones for speech-worthy outputs. The researchers introduce SpokenElyza, a benchmark for Japanese speech-worthiness, and demonstrate the effectiveness of their approach in producing concise, conversational, and readily synthesized speech. Their method shows substantial improvement on SpokenElyza while preserving performance on the original written-style evaluation. The study contributes significantly to the development of Japanese spoken dialog systems and offers a valuable resource for future research.
Key Points
- ▸ The proposed alignment approach improves Japanese SpeechLLMs for speech-worthy outputs by adapting their text-based backbones.
- ▸ The introduction of SpokenElyza, a benchmark for Japanese speech-worthiness, enables rigorous evaluation of this task.
- ▸ The method achieves substantial improvement on SpokenElyza while preserving performance on the original written-style evaluation.
Merits
Strength in Improving Japanese SpeechLLMs
The proposed alignment approach effectively adapts Japanese SpeechLLMs for speech-worthy outputs, addressing a critical issue in the field. By improving the quality of generated speech, this method has significant implications for various applications, including voice assistants, language translation, and spoken dialog systems.
Demerits
Limited Generalizability to Other Languages
The study's findings are specific to Japanese and may not generalize to other languages, which could limit the broader applicability of the proposed approach. Further research is needed to explore its effectiveness in other linguistic contexts.
Dependence on SpokenElyza Benchmark
The method's performance relies heavily on the quality and representativeness of the SpokenElyza benchmark. If the benchmark is not comprehensive or accurate, the method's effectiveness may be compromised, highlighting the importance of rigorously evaluating and refining the benchmark.
Expert Commentary
The study makes a significant contribution to the field of spoken dialog systems by addressing a critical issue in Japanese SpeechLLMs. The proposed alignment approach demonstrates substantial improvement in producing speech-worthy outputs, and the introduction of SpokenElyza provides a valuable resource for future research. However, the study's limitations, particularly its limited generalizability to other languages and dependence on the SpokenElyza benchmark, highlight the need for further research to explore its effectiveness in diverse linguistic contexts. As the field continues to evolve, the study's findings and methodology will likely influence the development of more natural and effective spoken dialog systems.
Recommendations
- ✓ Future research should explore the adaptability of the proposed alignment approach to other languages and linguistic contexts.
- ✓ The development of more comprehensive and accurate benchmarks, such as SpokenElyza, is essential for evaluating the effectiveness of spoken dialog systems.
