Deep Learning of Sound and Music

- Developed a generative AI system that generates single instrument track given residual music audio by PyTorch.
- Built customized PyTorch DataLoader to random sample audio data for each training pass.
- Applied pretrained embedding model MERT to get semantic tokens and acoustic transformer to get acoustic tokens.

- Created a reusable pipeline to generate data profiles and split audio for an audio dataset size of over 16000 seconds.
- Developed a specialized cross-validation algorithm for songs in different lengths to prevent information leakage.
- Trained model on Chinese Opera and test the model on Western Opera, and vice versa to explore the cross-lingual performance of the model.