DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanism
This repository is the official PyTorch implementation of our AAAI-2022 paper, in which we propose DiffSinger (for Singing-Voice-Synthesis) and DiffSpeech (for Text-to-Speech).
Besides, more detailed & improved code framework, which contains the implementations of FastSpeech 2, DiffSpeech and our NeurIPS-2021 work PortaSpeech is coming soon
| DiffSinger/DiffSpeech at training | DiffSinger/DiffSpeech at inference |
|---|---|
 |
 |
- Dec.01, 2021: DiffSinger was accepted by AAAI-2022.
- Sep.29, 2021: Our recent work
PortaSpeech: Portable and High-Quality Generative Text-to-Speechwas accepted by NeurIPS-2021.
- May.06, 2021: We submitted DiffSinger to Arxiv
Environments
conda create -n your_env_name python=3.8
source activate your_env_name
pip install -r requirements_2080.txt (GPU 2080Ti, CUDA 10.2)
or pip install -r requirements_3090.txt (GPU 3090, CUDA 11.4)
DiffSpeech (TTS version)
1. Data Preparation
a) Download and extract the LJ Speech dataset, then create a link to the dataset folder: ln -s /xxx/LJSpeech-1.1/ data/raw/
b) Download and Unzip the ground-truth duration extracted by MFA: tar -xvf mfa_outputs.tar; mv mfa_outputs data/processed/ljspeech/
c) Run the following scripts to pack the dataset for training/inference.
CUDA_VISIBLE_DEVICES=0 python data_gen/tts/bin/binarize.py --config configs/tts/lj/fs2.yaml
# `data/binary/ljspeech` will be generated.
2. Training Example
CUDA_VISIBLE_DEVICES=0 python tasks/run.py --config usr/configs/lj_ds_beta6.yaml --exp_name lj_exp1 --reset
3. Inference Example
CUDA_VISIBLE_DEVICES=0 python tasks/run.py --config usr/configs/lj_ds_beta6.yaml --exp_name lj_exp1 --reset --infer
We also provide:
- the pre-trained model of DiffSpeech;
- the pre-trained model of HifiGAN vocoder;
- the individual pre-trained model of FastSpeech 2 for the shallow diffusion mechanism in DiffSpeech;
Remember to put the pre-trained models in checkpoints directory.
About the determination of 'k' in shallow diffusion: We recommend the trick introduced in Appendix B. We have already provided the proper 'k' for Ljspeech dataset in the config files.
DiffSinger (SVS version)
0. Data Acquirement
- WIP. We will provide a form to apply for PopCS dataset.
1. Data Preparation
- WIP. Similar to DiffSpeech.
2. Training Example
CUDA_VISIBLE_DEVICES=0 python tasks/run.py --config usr/configs/popcs_ds_beta6.yaml --exp_name popcs_exp1 --reset
# or
CUDA_VISIBLE_DEVICES=0 python tasks/run.py --config usr/configs/popcs_ds_beta6_offline.yaml --exp_name popcs_exp2 --reset
3. Inference Example
CUDA_VISIBLE_DEVICES=0 python tasks/run.py --config usr/configs/popcs_ds_beta6_offline.yaml --exp_name popcs_exp2 --reset --infer
The pre-trained model for SVS will be provided recently.
Tensorboard
tensorboard --logdir_spec exp_name
 |
Mel Visualization
Along vertical axis, DiffSpeech: [0-80]; FastSpeech2: [80-160].
| DiffSpeech vs. FastSpeech 2 |
|---|
 |
 |
 |
Audio Demos
Audio samples can be found in our demo page.
We also put part of the audio samples generated by DiffSpeech+HifiGAN (marked as [P]) and GTmel+HifiGAN (marked as [G]) of test set in resources/demos_1218.
(corresponding to the pre-trained model DiffSpeech)
Citation
@misc{liu2021diffsinger,
title={DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanism},
author={Jinglin Liu and Chengxi Li and Yi Ren and Feiyang Chen and Zhou Zhao},
year={2021},
eprint={2105.02446},
archivePrefix={arXiv},}
Acknowledgements
Our codes are based on the following repos:
Also thanks Keon Lee for fast implementation of our work.
![size mismatch for model.encoder_embed_tokens.weight: copying a param with shape torch.Size([62, 256]) from checkpoint, the shape in current model is torch.Size([57, 256]).](https://avatars.githubusercontent.com/u/10137?v=4)
2.3k Dec 30, 2022
413 Dec 31, 2022
12 Oct 22, 2022
42 Nov 14, 2022
44 Dec 14, 2022
94 Dec 28, 2022
1 Jun 12, 2022
28 Jan 01, 2023
7 Dec 07, 2022
22 Oct 29, 2022
856 Jan 06, 2023
48 Nov 27, 2022
54 Dec 12, 2022
442 Dec 16, 2022
90 Nov 24, 2022
23 Oct 17, 2022
18 Oct 24, 2022
9 Oct 26, 2022
183 Jan 03, 2023
52 Nov 10, 2022