Hao-Wen Dong

I will be joining University of Michigan as an
Assistant Professor in Fall 2024!

I am a PhD Candidate in Computer Science at UC San Diego working with Prof. Julian McAuley and Prof. Taylor Berg-Kirkpatrick. My research aims to empower music and audio creation with machine learning. My long-term goal is to lower the barrier of entry for music composition and democratize audio content creation. I am broadly interested in music generation, audio synthesis, generative AI, multimodal learning, and music information retrieval.

I study a range of topics centered around Generative AI for Music and Audio:

Multitrack Music Generation: MMT (ICASSP 2023), BinaryMuseGAN (ISMIR 2018), MuseGAN (AAAI 2018)
Assistive Music Creation Tools: Deep Performer (ICASSP 2022), Arranger (ISMIR 2021)
Multimodal Learning for Audio and Music: CLIPSonic (WASPAA 2023), CLIPSep (ICLR 2023)
Infrastructure for Music Generation Research: MusPy (ISMIR 2020), Pypianoroll (ISMIR LBD 2018)

Overview

News

I was selected as one of the Rising Stars in AI by KAUST AI Initiative!
I was selected as one of the Rising Stars in Data Science by UChicago DSI and UCSD HDSI!
I was awarded the IEEE SPS Scholarship by IEEE Signal Processing Society!
Our work on text-to-audio synthesis was accepted by WASPAA 2023 (paper, demo)!

Selected Honors & Awards

Rising Stars in AI KAUST AI Initiative
2024
Rising Stars in Data Science UChicago DSI and UCSD HDSI
2023
IEEE SPS Scholarship IEEE Signal Processing Society
2023
Friends of International Center Fellowship UC San Diego
2023
Rising Stars in Signal Processing ICASSP
2023
Interdisciplinary Research Award UCSD GPSA
2023
Government Scholarship to Study Abroad Taiwan Ministry of Education
2022
J. Yang Scholarship J. Yang & Family Foundation
2021
ECE Department Fellowship UC San Diego
2019

Selected Publications

See the full list of pulications here (Google Scholar).

CLIPSonic: Text-to-Audio Synthesis with Unlabeled Videos and Pretrained Language-Vision Models
Hao-Wen Dong, Xiaoyu Liu, Jordi Pons, Gautam Bhattacharya, Santiago Pascual, Joan Serrà, Taylor Berg-Kirkpatrick, and Julian McAuley
IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), 2023
paper demo video slides reviews
Multitrack Music Transformer
Hao-Wen Dong, Ke Chen, Shlomo Dubnov, Julian McAuley, and Taylor Berg-Kirkpatrick
IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2023
paper demo video slides code reviews
CLIPSep: Learning Text-queried Sound Separation with Noisy Unlabeled Videos
Hao-Wen Dong, Naoya Takahashi, Yuki Mitsufuji, Julian McAuley, and Taylor Berg-Kirkpatrick
International Conference on Learning Representations (ICLR), 2023
paper demo video slides poster code reviews
Deep Performer: Score-to-Audio Music Performance Synthesis
Hao-Wen Dong, Cong Zhou, Taylor Berg-Kirkpatrick, and Julian McAuley
IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022
paper demo video slides poster reviews
Towards Automatic Instrumentation by Learning to Separate Parts in Symbolic Multitrack Music
Hao-Wen Dong, Chris Donahue, Taylor Berg-Kirkpatrick and Julian McAuley
International Society for Music Information Retrieval Conference (ISMIR), 2021
paper demo video slides code reviews
MusPy: A Toolkit for Symbolic Music Generation
Hao-Wen Dong, Ke Chen, Julian McAuley, and Taylor Berg-Kirkpatrick
International Society for Music Information Retrieval Conference (ISMIR), 2020
paper video slides poster code documentation reviews
Convolutional Generative Adversarial Networks with Binary Neurons for Polyphonic Music Generation
Hao-Wen Dong and Yi-Hsuan Yang
International Society for Music Information Retrieval Conference (ISMIR), 2018
paper demo video slides poster code reviews
MuseGAN: Multi-track Sequential Generative Adversarial Networks for Symbolic Music Generation and Accompaniment
Hao-Wen Dong,* Wen-Yi Hsiao,* Li-Chia Yang, and Yi-Hsuan Yang (*equal contribution)
AAAI Conference on Artificial Intelligence (AAAI), 2018
paper demo slides code

Education

University of California San Diego
Ph.D. in Computer Science (expected June 2024)
Advisors: Julian McAuley and Taylor Berg-Kirkpatrick
Thesis: Empowering Music Creation with Machine Learning

Sep 2019 – present
University of California San Diego
M.S. in Computer Science

Sep 2019 – Jun 2021
National Taiwan Normal University
Digital Video and Audio Arts Program

Sep 2019 – Jun 2017
National Taiwan University
B.S. in Electrical Engineering

Sep 2013 – Jun 2017

Professional Experience

NVIDIA
Research Intern
Host: Deep Imagination Research Group, NVIDIA Research
Advisors: Siddharth Gururani and Ming-Yu Liu
Topic: Controllable audio generation

Sep 2023 – Dec 2023
Adobe
Research Scientist/Engineer Intern
Host: Audio Research Group, Adobe Research
Advisors: Justin Salamon and Oriol Nieto
Topic: Text-to-audio retrieval

May 2023 – Sep 2023
Dolby
Speech/Audio Deep Learning Intern
Host: Applied AI Team, Advanced Technology
Advisor: Xiaoyu Liu
Topic: Text-to-audio synthesis

Jan 2023 – Apr 2023
Amazon
Applied Scientist Intern
Host: Natural Understanding Team, Alexa AI
Advisors: Wenbo Zhao and Gunnar Sigurdsson
Topic: Text-to-audio synthesis

Sep 2022 – Jan 2023
Sony
Student Intern
Host: Tokyo Laboratory 30, R&D Center
Advisor: Naoya Takahashi
Topic: Universal sound separation

May 2022 – Sep 2022
Dolby
Deep Learning Audio Intern
Host: Applied AI Team, Advanced Technology Group
Advisor: Cong Zhou
Topic: Music performance synthesis

Jun 2021 – Sep 2021
Yamaha
Research Intern
Host: AI Group, Research and Development Division
Advisor: Keijiro Saino
Topic: Deep learning based synthesizer

May 2019 – Aug 2019
Academia Sinica
Research Assitant
Host: Music and AI Lab, Research Center for IT Innovation
Advisor: Yi-Hsuan Yang
Topics: Music generation and deep generative models

Jul 2017 – Apr 2019

For more information, please see my CV.