I am a PhD Candidate in Computer Science at UC San Diego working with Prof. Julian McAuley and Prof. Taylor Berg-Kirkpatrick. My research aims to empower music and audio creation with machine learning. My long-term goal is to lower the barrier of entry for music composition and democratize audio content creation. I am broadly interested in music generation, audio synthesis, generative AI, multimodal learning and music information retrieval.

I study a range of topics centered around Generative AI for Music and Audio. My research can be roughly categorized into the following directions:

• Multitrack Music Generation: MMT (ICASSP 2023), BinaryMuseGAN (ISMIR 2018), MuseGAN (AAAI 2018)
• Assistive Music Creation Tools: Deep Performer (ICASSP 2022), Arranger (ISMIR 2021)
• Multimodal Learning for Audio and Music: CLIPSonic (WASPAA 2023), CLIPSep (ICLR 2023)
• Infrastructure for Music Generation Research: MusPy (ISMIR 2020), Pypianoroll (ISMIR LBD 2018)

I am on the academic job market. Feel free to reach out!
CV   Research statement   Teaching statement   EDI statement

---

News

• I was selected as one of the Rising Stars in Data Science by UChicago DSI and UCSD HDSI!
• I was awarded the IEEE SPS Scholarship by IEEE Signal Processing Society!
• Our work on text-to-audio synthesis was accepted by WASPAA 2023 (paper, demo)!
• Our work on text-to-audio synthesis was accepted by CVPR Workshop on Sight and Sound (paper, demo)!
• I was selected as one of the Rising Stars in Signal Processing at ICASSP 2023!
• I won the Interdisciplinary Research Award by UCSD GPSA!

---

Selected Honors & Awards

• Rising Stars in Data Science Data Science Institute, University of Chicago and Halıcıoğlu Data Science Institute, UC San Diego

  2023
• IEEE SPS Scholarship IEEE Signal Processing Society

  2023
• Friends of International Center Fellowship UC San Diego

  2023
• Rising Stars in Signal Processing ICASSP

  2023
• Interdisciplinary Research Award Graduate & Professional Student Association, UC San Diego

  2023
• Government Scholarship to Study Abroad Taiwan Ministry of Education

  2022
• J. Yang Scholarship J. Yang & Family Foundation

  2021
• ECE Department Fellowship UC San Diego

  2019

---

Selected Publications

See the full list of pulications here (Google Scholar).

• CLIPSonic: Text-to-Audio Synthesis with Unlabeled Videos and Pretrained Language-Vision Models
  Hao-Wen Dong, Xiaoyu Liu, Jordi Pons, Gautam Bhattacharya, Santiago Pascual, Joan Serrà, Taylor Berg-Kirkpatrick, and Julian McAuley
  IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), 2023
  paper  demo  video  slides  reviews

• Multitrack Music Transformer
  Hao-Wen Dong, Ke Chen, Shlomo Dubnov, Julian McAuley, and Taylor Berg-Kirkpatrick
  IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2023
  paper  demo  video  slides  code  reviews

• CLIPSep: Learning Text-queried Sound Separation with Noisy Unlabeled Videos
  Hao-Wen Dong, Naoya Takahashi, Yuki Mitsufuji, Julian McAuley, and Taylor Berg-Kirkpatrick
  International Conference on Learning Representations (ICLR), 2023
  paper  demo  video  slides  poster  code  reviews

• Deep Performer: Score-to-Audio Music Performance Synthesis
  Hao-Wen Dong, Cong Zhou, Taylor Berg-Kirkpatrick, and Julian McAuley
  IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022
  paper  demo  video  slides  poster  reviews

• Towards Automatic Instrumentation by Learning to Separate Parts in Symbolic Multitrack Music
  Hao-Wen Dong, Chris Donahue, Taylor Berg-Kirkpatrick and Julian McAuley
  International Society for Music Information Retrieval Conference (ISMIR), 2021
  paper  demo  video  slides  code  reviews

• MusPy: A Toolkit for Symbolic Music Generation
  Hao-Wen Dong, Ke Chen, Julian McAuley, and Taylor Berg-Kirkpatrick
  International Society for Music Information Retrieval Conference (ISMIR), 2020
  paper  video  slides  poster  code  documentation  reviews

• Convolutional Generative Adversarial Networks with Binary Neurons for Polyphonic Music Generation
  Hao-Wen Dong and Yi-Hsuan Yang
  International Society for Music Information Retrieval Conference (ISMIR), 2018
  paper  demo  video  slides  poster  code  reviews

• MuseGAN: Multi-track Sequential Generative Adversarial Networks for Symbolic Music Generation and Accompaniment
  Hao-Wen Dong,* Wen-Yi Hsiao,* Li-Chia Yang, and Yi-Hsuan Yang (*equal contribution)
  AAAI Conference on Artificial Intelligence (AAAI), 2018
  paper  demo  slides  code

---

Education

• University of California San Diego
  Ph.D. in Computer Science (expected June 2024)
  Advisors: Julian McAuley and Taylor Berg-Kirkpatrick

  Sep 2019 – present

  
• University of California San Diego
  M.S. in Computer Science

  Sep 2019 – Jun 2021

  
• National Taiwan Normal University
  Digital Video and Audio Arts Program

  Sep 2019 – Jun 2017

  
• National Taiwan University
  B.S. in Electrical Engineering

  Sep 2013 – Jun 2017

  

---

Professional Experience

• NVIDIA
  Research Intern
  Host: Deep Imagination Research Group, NVIDIA Research
  Advisors: Siddharth Gururani and Ming-Yu Liu

  Sep 2023 – present

  
• Adobe
  Research Scientist/Engineer Intern
  Host: Audio Research Group, Adobe Research
  Advisors: Justin Salamon and Oriol Nieto
  Topic: Text-to-audio retrieval

  May 2023 – Sep 2023

  
• Dolby
  Speech/Audio Deep Learning Intern
  Host: Applied AI Team, Advanced Technology
  Advisor: Xiaoyu Liu
  Topic: Text-to-audio synthesis

  Jan 2023 – Apr 2023

  
• Amazon
  Applied Scientist Intern
  Host: Natural Understanding Team, Alexa AI
  Advisors: Wenbo Zhao and Gunnar Sigurdsson
  Topic: Text-to-audio synthesis

  Sep 2022 – Jan 2023

  
• Sony
  Student Intern
  Host: Tokyo Laboratory 30, R&D Center
  Advisor: Naoya Takahashi
  Topic: Universal sound separation

  May 2022 – Sep 2022

  
• Dolby
  Deep Learning Audio Intern
  Host: Applied AI Team, Advanced Technology Group
  Advisor: Cong Zhou
  Topic: Music performance synthesis

  Jun 2021 – Sep 2021

  
• Yamaha
  Research Intern
  Host: AI Group, Research and Development Division
  Advisor: Keijiro Saino
  Topic: Deep learning based synthesizer

  May 2019 – Aug 2019

  
• Academia Sinica
  Research Assitant
  Host: Music and AI Lab, Research Center for IT Innovation
  Advisor: Yi-Hsuan Yang
  Topics: Music generation and deep generative models

  Jul 2017 – Apr 2019

  

---

For more information, please see my CV.