Diffusion Process in LCM
- π The diffusion process is used to refine the embeddings, making the model learn to handle uncertainty and noise in the data.
- π The process involves adding noise to the sentence embedding and then using the Transformer to predict the noise, refining the embedding through multiple steps.
- π The diffusion process enables the model to explore multiple potential meanings or refinements of a specific presentation, adding robustness and flexibility.
Limitations of LCM
- π« Sentences in the Sona space remain discrete combinatorial objects, despite being represented as continuous vectors.
- π The choice and design of the embedding space play a crucial role in the LCM approach, and if not optimally built, the system may fail.
- π The Sona model was trained on a specific dataset with short sentences, which may not be suitable for more complex topics or longer sentences.