Cross-Modal Tasks
Meta Spirit LM is designed to perform cross-modal tasks, including speech-to-text and text-to-speech. Speech-to-text involves converting spoken language into written text, while text-to-speech involves generating spoken language from written text.
The Spirit LM model is trained on a combination of text and speech datasets, allowing it to learn the relationships between phonetic, pitch, and tone tokens and generate speech that is more expressive and natural-sounding. This enables the model to perform cross-modal tasks with high accuracy and natural-sounding speech.