VQMIVC: Vector Quantization and Mutual Information-Based Unsupervised Speech Representation Disentanglement for One-shot Voice Conversion (Interspeech 2021)
Run VQMIVC on Replicate
Integrated to Huggingface Spaces with Gradio. See Gradio Web Demo.
Pre-trained models: google-drive or here | Paper demo
This paper proposes a speech representation disentanglement framework for one-shot/any-to-any voice conversion, which performs conversion across arbitrary speakers with only a single target-speaker utterance for reference. Vector quantization with contrastive predictive coding (VQCPC) is used for content encoding and mutual information (MI) is introduced as the correlation metric during training, to achieve proper disentanglement of content, speaker and pitch representations, by reducing their inter-dependencies in an unsupervised manner.
📢
Update
Many thanks to ericguizzo & AK391!
- A Replicate demo is provided online, so you can play our pre-trained models there, have fun!
- VQMIVC can be trained and tested inside a Docker environment via Cog now.
- Gradio Web Demo is available, another online demo!
TODO
- Add more details on how to use Cog for development
Requirements
Python 3.6 is used, install apex for speeding up training (optional), other requirements are listed in 'requirements.txt':
pip install -r requirements.txt
Quick start with pre-trained models
ParallelWaveGAN is used as the vocoder, so firstly please install ParallelWaveGAN to try the pre-trained models:
python convert_example.py -s {source-wav} -r {reference-wav} -c {converted-wavs-save-path} -m {model-path}
For example:
python convert_example.py -s test_wavs/p225_038.wav -r test_wavs/p334_047.wav -c converted -m checkpoints/useCSMITrue_useCPMITrue_usePSMITrue_useAmpTrue/VQMIVC-model.ckpt-500.pt
The converted wav is put in 'converted' directory.
Training and inference:
- Step1. Data preparation & preprocessing
-
Put VCTK corpus under directory: 'Dataset/'
-
Training/testing speakers split & feature (mel+lf0) extraction:
python preprocess.py
- Step2. model training:
-
Training with mutual information minimization (MIM):
python train.py use_CSMI=True use_CPMI=True use_PSMI=True -
Training without MIM:
python train.py use_CSMI=False use_CPMI=False use_PSMI=False
- Step3. model testing:
-
Put PWG vocoder under directory: 'vocoder/'
-
Inference with model trained with MIM:
python convert.py checkpoint=checkpoints/useCSMITrue_useCPMITrue_usePSMITrue_useAmpTrue/model.ckpt-500.pt -
Inference with model trained without MIM:
python convert.py checkpoint=checkpoints/useCSMIFalse_useCPMIFalse_usePSMIFalse_useAmpTrue/model.ckpt-500.pt
Citation
If the code is used in your research, please Star our repo and cite our paper:
@inproceedings{wang21n_interspeech,
author={Disong Wang and Liqun Deng and Yu Ting Yeung and Xiao Chen and Xunying Liu and Helen Meng},
title={{VQMIVC: Vector Quantization and Mutual Information-Based Unsupervised Speech Representation Disentanglement for One-Shot Voice Conversion}},
year=2021,
booktitle={Proc. Interspeech 2021},
pages={1344--1348},
doi={10.21437/Interspeech.2021-283}
}
Acknowledgements:
- The content encoder is borrowed from VectorQuantizedCPC, which also inspires the negative sampling within-utterance for CPC;
- The speaker encoder is borrowed from AdaIN-VC;
- The decoder is modified from AutoVC;
- Estimation of mutual information is modified from CLUB;
- Speech features extraction is based on espnet and Pyworld.
Could you give me some advice? Thank you very much!
5 Jan 18, 2022
60 Dec 02, 2022
105 Dec 04, 2022
16 May 06, 2022
2 Dec 16, 2022
4 Nov 18, 2022
4 Nov 23, 2022
29 Nov 30, 2022
2.8k Dec 26, 2022
1 Nov 04, 2021
1 Dec 18, 2021
105 Dec 17, 2022
687 Dec 25, 2022
161 Jan 03, 2023
44 Dec 06, 2022
35 Dec 12, 2022
15 Jan 13, 2020
0 Feb 26, 2022
13 Nov 14, 2022
45 Oct 07, 2022