We are pleased to share that Associate Professor Dorien Herremans, Assistant Professor Soujanya Poria, and their groups from ISTD have won the Super AI Leader (SAIL) Top 30 Award issued by the World Artificial Intelligence Conferences (WAIC), for their joint project ‘Mustango: controllable text-to-music’.
The Music, Audio, and AI Lab (AMAAI) at SUTD invites applications for a PhD position in the exciting and rapidly evolving field of music and audio artificial intelligence.
The AMAAI lab, led by Prof. Dorien Herremans, is engaged in cutting-edge research at the intersection of music, audio, and artificial intelligence. We are based in tropical Singapore, a hub for AI research and startups, but also a great base to explore South East Asia, where English is the first language. Our PhD students and PostDoc researchers work on projects that explore areas such as:
I am thrilled to share that MidiCaps - A Large-scale Dataset of Caption-annotated MIDI Files, has been accepted at ISMIR Conference. The MidiCaps dataset is a large-scale dataset of 168,385 midi music files with descriptive text captions, and a set of extracted musical features. The captions have been produced through a captioning pipeline incorporating MIR feature extraction and LLM Claude 3 to caption the data from extracted features with an in-context learning task. The framework used to extract the captions is available open source on github.
I am excited to announce the latest paper by Viet-Toan Le, who was a visiting student at the AMAAI Lab, on 'Natural Language Processing Methods for Symbolic Music Generation and Information Retrieval: a Survey'. Viet-Toan did an amazing job at collating and nicely presenting over 225 papers in Music Information Retrieval that are inspired by NLP, as well as presenting the latest challenges and steps forward for the field!
I am excited to announce that we have two positions open at the AMAAI Lab in Singapore: a postdoc and research assistant position in generative AI for music models. They will be building on our recent work of the Mustango model (https://github.com/AMAAI-Lab/mustango) and video2music (https://github.com/AMAAI-Lab/Video2Music).
Excited to announce Mustango, a powerful multimodal Model for generating music from textual prompts. Mustango leverages a Latent Diffusion Model conditioned on textual prompts (encoded using Flan-T5) and various musical features. Try the demo! What makes it different from the rest?
-- greater controllability in the music generation.
-- trained on a large dataset generated using ChatGPT and musical manipulations.
-- superior performance over its predecessors as per the experts.
-- open source!
We are happy to announce Video2Music, a novel AI-powered multimodal music generation framework called Video2Music. This framework uniquely uses video features as conditioning input to generate matching music using a Transformer architecture. By employing cutting-edge technology, our system aims to provide video creators with a seamless and efficient solution for generating tailor-made background music.
We are happy to announce two talks on Tuesday 17 October at 2pm at SUTD I3 Lab 1.605
Title : Exploring NLP Methods in Symbolic MIR: Representations and Models
Our paper on PreBit - A multimodal model with Twitter FinBERT embeddings for extreme price movement prediction of Bitcoin just got published in Expert Systems with Applications.
Congratulations to Joel Ong on publishing our paper on using multi-task deep learning for porfolio construction in Expert Systems with Applications. The paper presents a new way to leverage time series momentum in a deep learning setting. Read a Twitter thread explaining the basics here.
Great work Cheuk Kin Wai on his latest paper on DiffRoll: Diffusion-based Generative Music Transcription with Unsupervised Pretraining Capability
Cheuk, K. W., Sawata, R., Uesaka, T., Murata, N., Takahashi, N., Takahashi, S., ... & Mitsufuji, Y. (2022). DiffRoll: Diffusion-based Generative Music Transcription with Unsupervised Pretraining Capability. arXiv preprint arXiv:2210.05148.
Congratulations to Thao on leading the publication of the EmoMV dataset set for music-video matching based on emotion!
Pham Q-H, Herremans D., Roig G.. 2022. EmoMV: Affective Music-Video Correspondence Learning Datasets for Classification and Retrieval. Information Fusion. DOI: 10.1016/j.inffus.2022.10.002
Paper highlights:
I was honoured to give a keynote talk at the 3rd Conference on AI Music Creativity (AIMC) on controllable music generation with emotion. Watch the full keynote here:
Congrats on my former research assistant Jiahui Huang on his latest paper in Sensors on 'Single Image Video Prediction with Auto-Regressive GANs'. Now we can generate videos of faces with desired emotions!
Huang, Jiahui, Yew Ken Chia, Samson Yu, Kevin Yee, Dennis Küster, Eva G. Krumhuber, Dorien Herremans, and Gemma Roig. "Single Image Video Prediction with Auto-Regressive GANs." Sensors 22, no. 9 (2022): 3533.
Read my PhD student Yanzhao Zou and myself's latest paper on A Multimodal Model with Twitter Finbert Embeddings for Extreme Price Movement Prediction of Bitcoin.
SUTD Game Lab is looking for a passionate game developer to join our team. The Game Programmer will visualise how systems should work, and translate this into a functioning solution using the existing game engine to develop prototypes or a polished game for training purposes.
Responsibilities:
Read the full NRF magazine here. Our original paper is available here.
It was an honour today to be part of the seminar at the KTH Royal Institute of Technology in Stockholm as part of the dialogues series.
dialogues1: probing the future of creative technology
Subject: “Interaction with generative music frameworks”
Guests: Dorien Herremans and Kıvanç Tatar (Video link to be posted)
Dorien Herremans: Controllable deep music generation with emotion
Excited to be featured on the latest 'AI and You - What is AI? How will it affect your life, your work, and your world?' podcast by Peter Scott from Human Cusp.
We're focusing on AI in music: What's the state of the art in AI music composition, how can human composers use it to their advantage, and what is the AI Song Contest? How do musical AIs surprise their creators and how are they like your grandmother trying to explain death metal?
Last year I was honoured to be part of the panel discussion on 'Challenging the limits of AI for the next generation of co-creative tools - Frontiers of Music and Artificial Intelligence'. at Ars Electronica, IRCAM (FR). Watch the video below.
