Parallel Tacotron2
Pytorch Implementation of Google's Parallel Tacotron 2: A Non-Autoregressive Neural TTS Model with Differentiable Duration Modeling
Updates
-
2021.05.15: Implementation done. Sanity checks on training and inference. But still the model cannot converge.
I'm waiting for your contribution!Please inform me if you find any mistakes in my implementation or any valuable advice to train the model successfully. See the Implementation Issues section.
Training
Requirements
-
You can install the Python dependencies with
pip3 install -r requirements.txt
-
In addition to that, install fairseq (official document, github) to utilize
LConvBlock.
Datasets
The supported datasets:
- LJSpeech: a single-speaker English dataset consists of 13100 short audio clips of a female speaker reading passages from 7 non-fiction books, approximately 24 hours in total.
- (will be added more)
Preprocessing
After downloading the datasets, set the corpus_path in preprocess.yaml and run the preparation script:
python3 prepare_data.py config/LJSpeech/preprocess.yaml
Then, run the preprocessing script:
python3 preprocess.py config/LJSpeech/preprocess.yaml
Training
Train your model with
python3 train.py -p config/LJSpeech/preprocess.yaml -m config/LJSpeech/model.yaml -t config/LJSpeech/train.yaml
The model cannot converge yet. I'm debugging but it would be boosted if your awesome contribution is ready!
TensorBoard
Use
tensorboard --logdir output/log/LJSpeech
to serve TensorBoard on your localhost.
Implementation Issues
Overall, normalization or activation, which is not suggested in the original paper, is adequately arranged to prevent nan value (gradient) on forward and backward calculations.
Text Encoder
- Use the
FFTBlockof FastSpeech2 for the transformer block of the text encoder. - Use dropout
0.2for theConvBlockof the text encoder. - To restore "proprietary normalization engine",
- Apply the same text normalization as in FastSpeech2.
- Implement
grapheme_to_phonemefunction. (See ./text/init).
Residual Encoder
- Use
80 channelsmel-spectrogrom instead of128-bin. - Regular sinusoidal positional embedding is used in frame-level instead of combinations of three positional embeddings in Parallel Tacotron. As the model depends entirely on unsupervised learning for the position, this choice can be a reason for the fails on model converge.
Duration Predictor & Learned Upsampling (The most important but ambiguous part)
- Use log durations with the prior: there should be at least one frame in total per sequence.
- Use
nn.SiLU()for the swish activation. - When obtaining
WandC, concatenation operation is applied amongS,E, andVafter frame-domain (T domain) broadcasting ofV. As the detailed process is not described in the original paper, this choice can be a reason for the fails on model converge.
Decoder
- Use (Multi-head)
Self-attentionandLConvBlock. - Iterative mel-spectrogram is projected by a linear layer.
- Apply
nn.Tanh()to eachLConvBLockoutput (following activation pattern of decoder part in FastSpeech2).
Loss
- Use optimization & scheduler of FastSpeech2 (which is from Attention is all you need as described in the original paper).
- Base on pytorch-softdtw-cuda (post) for the soft-DTW.
- Implement customized soft-DTW in
model/soft_dtw_cuda.py, reflecting the recursion suggested in the original paper. - In the original soft-DTW, the final loss is not assumed and therefore only
Eis computed. But employed as a loss function, jacobian product is added to return target derivetive ofRw.r.t. inputX. - Currently, the maximum batch size is
6in 24GiB GPU (TITAN RTX) due to space complexity problem in soft-DTW Loss.- In the original paper, a custom differentiable diagonal band operation was implemented and used to solve the complexity of O(T^2), but this part has not been explored in the current implementation yet.
- Implement customized soft-DTW in
- For the stability, mel-spectrogroms are compressed by a sigmoid function before the soft-DTW. If the sigmoid is eliminated, the soft-DTW value is too large, producing nan in the backward.
- Guided attention loss is applied for fast convergence of the attention module in residual encoder.
Citation
@misc{lee2021parallel_tacotron2,
author = {Lee, Keon},
title = {Parallel-Tacotron2},
year = {2021},
publisher = {GitHub},
journal = {GitHub repository},
howpublished = {\url{https://github.com/keonlee9420/Parallel-Tacotron2}}
}
688 Jan 04, 2023
119 Dec 29, 2022
9 Jul 23, 2022
1 Nov 18, 2021
249 Jan 03, 2023
66 Nov 30, 2022
129 Dec 30, 2022
7 Jan 03, 2023
43 Oct 14, 2022
3 May 30, 2022
45 Nov 28, 2022
6 Dec 01, 2022
35 Nov 10, 2022
59 Dec 13, 2022
1 Sep 19, 2022
41 Dec 31, 2022
4 May 04, 2022
66 Nov 30, 2022
22 Aug 27, 2022
1 Nov 09, 2021