Pretrain on language first and then on text-modalities data

[JChander]

Hi lucidrains,

Nice to meet you again! Since Meta AI has released LlamaFusion paper but no implementation has been found, I was wondering if I could pre-train the transfusion model on the language first (as you showed in the README) and then continue pre-training on the text-modality (cell-gene embeddings for me)? Does this sound like a viable alternative to Llamafusion?

Looking forward to your reply! Thank you.