We are happy to announce Video2Music, a novel AI-powered multimodal music generation framework called Video2Music. This framework uniquely uses video features as conditioning input to generate matching music using a Transformer architecture. By employing cutting-edge technology, our system aims to provide video creators with a seamless and efficient solution for generating tailor-made background music.

Live demo on Replicate.
View on github.

Our paper on PreBit - A multimodal model with Twitter FinBERT embeddings for extreme price movement prediction of Bitcoin just got published in Expert Systems with Applications.

Read the article here.

Great work Cheuk Kin Wai on his latest paper on DiffRoll: Diffusion-based Generative Music Transcription with Unsupervised Pretraining Capability

Cheuk, K. W., Sawata, R., Uesaka, T., Murata, N., Takahashi, N., Takahashi, S., ... & Mitsufuji, Y. (2022). DiffRoll: Diffusion-based Generative Music Transcription with Unsupervised Pretraining Capability. arXiv preprint arXiv:2210.05148.

Demo & Source code available here.

I was honoured to give a keynote talk at the 3rd Conference on AI Music Creativity (AIMC) on controllable music generation with emotion. Watch the full keynote here:

Read my PhD student Yanzhao Zou and myself's latest paper on A Multimodal Model with Twitter Finbert Embeddings for Extreme Price Movement Prediction of Bitcoin.

Read the full NRF magazine here. Our original paper is available here.

Excited to be featured on the latest 'AI and You - What is AI? How will it affect your life, your work, and your world?' podcast by Peter Scott from Human Cusp.

We're focusing on AI in music: What's the state of the art in AI music composition, how can human composers use it to their advantage, and what is the AI Song Contest? How do musical AIs surprise their creators and how are they like your grandmother trying to explain death metal?

Congrats to Kin Wa Cheuk for his published paper in the ACM Multimedia conference (A*) on 'ReconVAT: A Semi-Supervised Automatic Music Transcription Framework for Low-Resource Real-World Data'. If you are interested in training low-data music transcription models with semi-supervised learning, check out the full paper here, or access the preprint.

Watch Raven's talk here:

Over the last few years, we developed Project PEAR at SUTD Game Lab. Project PEAR is a geolocation based augmented reality game that is aimed at educating the player on climate change as well as influence their behaviours. We just published a study in Sustainability on the effectiveness of this game.