If just install specified requirements + Pillow and fairseq following warnings appear during training start:

No module named 'lightconv_cuda'

If install lightconv-layer from fairseq, the folllowing warning displayed:

WARNING: Unsupported filter length passed - skipping forward pass

Pytorch 1.7 Cuda 10.2 Fairseq 1.0.0a0+19793a7

Hi.

According to text in README (will be added more) i would like to suggest to add my open German "Thorsten" dataset.

Thorsten: a single-speaker German open dataset consists of 22.668 short audio clips of a male speaker, approximately 23 hours in total (LJSpeech file/directory syntax).

https://github.com/thorstenMueller/deep-learning-german-tts/

Hi @keonlee9420 , have you tried the Cython version of Soft DTW from this repo

https://github.com/mblondel/soft-dtw

Is it available to apply for Parallel Tacotron 2 ? I am trying that repo because the current batch is too small when using CUDA implement of @Maghoumi .

I just wonder that @Maghoumi in https://github.com/Maghoumi/pytorch-softdtw-cuda claims that experiment with batch size

But when applying for Para Taco, the batch size is too small, are there any gap?

When I load audios with mel-spectrogram frames larger than max sequence of mel len (1000 frames):

• There is a problem when concatenating pos + speaker + mels: I try to set max_seq_len larger (1500),
• Then lead to a problem with Soft DTW, they said the maximum is 1024

For solution, I tried to trim mels for fitting 1024 but it seems complicated, now I filter out all audios with frames > 1024

Any suggestion for handle Long Audios? I wonder how it work at inference steps.

Great work. But I encounter one problems when train this model :( The error message:

ImportError: cannot import name II form omegaconf

The version of fairseq is 0.10.2 (latest releaser version) and omegaconf is 1.4.1. How to fix it?

Thank you

I following your command to run the code, but I get following error. File "train.py", line 87, in main output = model(*(batch[2:])) File "/home/ydc/anaconda3/envs/CD/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl result = self.forward(*input, **kwargs) File "/home/ydc/anaconda3/envs/CD/lib/python3.8/site-packages/torch/nn/parallel/data_parallel.py", line 162, in forward return self.gather(outputs, self.output_device) File "/home/ydc/anaconda3/envs/CD/lib/python3.8/site-packages/torch/nn/parallel/data_parallel.py", line 174, in gather return gather(outputs, output_device, dim=self.dim) File "/home/ydc/anaconda3/envs/CD/lib/python3.8/site-packages/torch/nn/parallel/scatter_gather.py", line 68, in gather res = gather_map(outputs) File "/home/ydc/anaconda3/envs/CD/lib/python3.8/site-packages/torch/nn/parallel/scatter_gather.py", line 63, in gather_map return type(out)(map(gather_map, zip(*outputs))) File "/home/ydc/anaconda3/envs/CD/lib/python3.8/site-packages/torch/nn/parallel/scatter_gather.py", line 63, in gather_map return type(out)(map(gather_map, zip(*outputs))) File "/home/ydc/anaconda3/envs/CD/lib/python3.8/site-packages/torch/nn/parallel/scatter_gather.py", line 55, in gather_map return Gather.apply(target_device, dim, *outputs) File "/home/ydc/anaconda3/envs/CD/lib/python3.8/site-packages/torch/nn/parallel/_functions.py", line 71, in forward return comm.gather(inputs, ctx.dim, ctx.target_device) File "/home/ydc/anaconda3/envs/CD/lib/python3.8/site-packages/torch/nn/parallel/comm.py", line 230, in gather return torch._C._gather(tensors, dim, destination) RuntimeError: Input tensor at index 1 has invalid shape [1, 474, 80], but expected [1, 302, 80]

1 fix mask problem when calculating W in LearnUpsampling Module and attention matrix in VaribleLengthAttention module. 2 a new Jacobian matrix of Manhattan distance 3 deal with mel spectrograms of different lengths

Hi, Thanks for implement.

I think Parallel TacoTron2 using same residual Encoder as parallel tacotron 1. In parallel tacotron, using five 17 × 1 LConv blocks interleaved with strided 3 × 1 convolutions

But, in your implementation, Lconvblock doesn't have stride argument. How did you handle this part?

Thanks.

Hello, Has anybody been able to train with softdtw loss. It doesn't converge at all. I think there is a problem with the implementation but I could't spot it. When I train with the real alignments it works well

Can someone share the weights file link? I couldn't synthesize it or use its inference. If I am wrong please tell me the correct method of using it. Thanks

I cloned the code, prepared data according to README, and just updated:

1. ljspeech data path in config/LJSpeech/train.yaml
2. unzip generator_LJSpeech.pth.tar.zip to get generator_LJSpeech.pth.tar and the code can run! But, no matter how many steps I trained, the images are always like this and demo audio sounds like noise:

  File "/data1/hjh/pycharm_projects/tts/parallel-tacotron2_try/model/parallel_tacotron2.py", line 68, in forward
    self.learned_upsampling(durations, V, src_lens, src_masks, max_src_len)
  File "/home/huangjiahong.dracu/miniconda2/envs/parallel_tc2/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/data1/hjh/pycharm_projects/tts/parallel-tacotron2_try/model/modules.py", line 335, in forward
    mel_mask = get_mask_from_lengths(mel_len, max_mel_len)
  File "/data1/hjh/pycharm_projects/tts/parallel-tacotron2_try/utils/tools.py", line 87, in get_mask_from_lengths
    ids = torch.arange(0, max_len).unsqueeze(0).expand(batch_size, -1).to(device)
RuntimeError: upper bound and larger bound inconsistent with step sign

Thank you for you jobs. I got above problem when training. I guess it's a Duration prediction problem. How to solve it?

Hi, Thanks for your excellent work! Could you possibly share your audio samples, pretrained models and loss curves with me? Thanks so much for your help!

Hey, I've found that in your implementation of S-DTW backward, E - matrices are not used, instead you are using G - matrices and their entries are ignoring scaling factors a, b, c. What's the reason for this? My guess you are doing this in order to preserve and propagate gradients, because they are vanishing due to small values of a, b, c. But I might be wrong, so I'd be glad to hear your motivation on doing this.

Playing with your code, I also found that gradients are vanishing, especially when bandwitdth=None. So I'm solving this problem by normalizing distance matrix, by n_mel_channel. And with this normalization and exact implementation of S-dtw backward I'm able to converge on overfit experiments quicker then with non-exact computation of s-dtw backward. I'm using these SDT hparams:

gamma = 0.05
warp = 256
bandwidth = 50

here is a small test I'm using for checks:

        target_spectro = np.load('')
        target_spectro = torch.from_numpy(target_spectro)
        target_spectro = target_spectro.unsqueeze(0).cuda()
        pred_spectro = torch.randn_like(target_spectro, requires_grad=True)

        optimizer = Adam([pred_spectro])

        # model fits in ~3k iterations
        n_iter = 4_000
        for i in range(n_iter):

            loss = self.numba_soft_dtw(pred_spectro, target_spectro)
            loss = loss / pred_spectro.size(1)
            loss.backward()

            if i % 1_000 == 0:
                print(f'iter: {i}, loss: {loss.item():.6f}')
                print(f'd_loss_pred {pred_spectro.grad.mean()}')

            optimizer.step()
            optimizer.zero_grad()

Curious to hear how your training is going! Best. Taras