Prososdy Morph: A python library for manipulating pitch and duration in an algorithmic way, for resynthesizing speech.

Related tags

Deep LearningProMo
Overview

ProMo (Prosody Morph)

https://travis-ci.org/timmahrt/ProMo.svg?branch=master https://coveralls.io/repos/github/timmahrt/ProMo/badge.svg?branch=master https://img.shields.io/badge/license-MIT-blue.svg?

Questions? Comments? Feedback? Chat with us on gitter!

Join the chat at https://gitter.im/pythonProMo/Lobby

A library for manipulating pitch and duration in an algorithmic way, for resynthesizing speech.

This library can be used to resynthesize pitch in natural speech using pitch contours taken from other speech samples, generated pitch contours, or through algorithmic manipulations of the source pitch contour.

1   Common Use Cases

What can you do with this library?

Apply the pitch or duration from one speech sample to another.

  • alignment happens both in time and in hertz

    • after the morph process, the source pitch points will be at the same absolute pitch and relative time as they are in the target file
    • time is relative to the start and stop time of the interval being considered (e.g. the pitch value at 80% of the duration of the interval). Relative time is used so that the source and target files don't have to be the same length.
    • temporal morphing is a minor effect if the sampling frequency is high but it can be significant when, for example, using a stylized pitch contour with few pitch samples.
  • modifications can be done between entire wav files or between corresponding intervals as specified in a textgrid or other annotation (indicating the boundaries of words, stressed vowels, etc.)

    • the larger the file, the less useful the results are likely to be without using a transcript of some sort
    • the transcripts do not have to match in lexical content, only in the number of intervals (same number of words or phones, etc.)
  • modifications can be scaled (it is possible to generate a wav file with a pitch contour that is 30% or 60% between the source and target contours).

  • can also morph the pitch range and average pitch independently.

  • resynthesis is performed by Praat.

  • pitch can be obtained from praat (such as by using praatio) or from other sources (e.g. ESPS getF0)

  • plots of the resynthesis (such as the ones below) can be generated

2   Illustrative example

Consider the phrase "Mary rolled the barrel". In the first recording (examples/mary1.wav), "Mary rolled the barrel" was said in response to a question such as "Did John roll the barrel?". On the other hand, in the second recording (examples/mary2.wav) the utterance was said in response to a question such as "What happened yesterday".

"Mary" in "mary1.wav" is produced with more emphasis than in "mary2.wav". It is longer and carries a more drammatic pitch excursion. Using ProMo, we can make mary1.wav spoken similar to mary2.wav, even though they were spoken in a different way and by different speakers.

Duration and pitch carry meaning. Change these, and you can change the meaning being conveyed.

Note that modifying pitch and duration too much can introduce artifacts. Such artifacts can be heard even in pitch morphing mary1.wav to mary2.wav.

Pitch morphing (examples/pitch_morph_example.py):

The following image shows morphing of pitch from mary1.wav to mary2.wav on a word-by-word level in increments of 33% (33%, 66%, 100%). Note that the morph adjusts the temporal dimension of the target signal to fit the duration of the source signal (the source and generated contours are equally shorter than the target contour). This occurs at the level of the file unless the user specifies an equal number of segments to align in time (e.g. using word-level transcriptions, as done here, or phone-level transcriptions, etc.)

examples/files/mary1_mary2_f0_morph.png

With the ability to morph pitch range and average pitch, it becomes easier to morph contours produced by different speakers:

The following image shows four different pitch manipulations. On the upper left is the raw morph. Notice that final output (black line) is very close to the target. Differences stem from duration differences.

However, the average pitch and pitch range are qualities of speech that can signify differences in gender in addition to other aspects of speaker identity. By resetting the average pitch and pitch range to that of the source, it is possible to morph the contour while maintaining aspects of the source speaker's identity.

The image in the upper right contains a morph followed by a reset of the average pitch to the source speaker's average pitch. In the bottom right a morph followed by a reset of the speaker's pitch range. In the bottom right pitch range was reset and then the speaker's average pitch was reset.

The longer the speech sample, the more representative the pitch range and mean pitch will be of the speaker. In this example both are skewed higher by the pitch accent on the first word.

Here the average pitch of the source (a female speaker) is much higher than the target (a male speaker) and the resulting morph sounds like it comes from a different speaker than the source or target speakers. The three recordings that involve resetting pitch range and/or average pitch sound much more natural.

examples/files/mary1_mary2_f0_morph_compare.png

Duration morphing (examples/duration_manipulation_example.py):

The following image shows morphing of duration from mary1.wav to mary2.wav on a word-by-word basis in increments of 33% (33%, 66%, 100%). This process can operate over an entire file or, similar to pitch morphing, with annotated segments, as done in this example.

examples/files/mary1_mary2_dur_morph.png

3   Tutorials

Tutorials for learning about prosody manipulation and how to use ProMo are available.

Tutorial 1.1: Intro to ProMo

Tutorial 1.2: Pitch manipulation tutorial

4   Major revisions

Ver 1.3 (May 29, 2017)

  • added tutorials
  • f0Morph() can now exclude certain regions from the morph process if desired

Ver 1.2 (January 27, 2017)

  • added code for reshaping pitch accents (shift alignment, add plateau, or change height)

Ver 1.1 (February 22, 2016)

  • f0 morph code for modifying speaker pitch range and average pitch
  • (October 20, 2016) Added integration tests with travis CI and coveralls support.

Ver 1.0 (January 19, 2016)

  • first public release.

Beta (July 1, 2013)

  • first version which was utilized in my dissertation work

5   Requirements

Python 2.7.* or above

Python 3.3.* or above (or below, probably)

My praatIO library is used extensively and can be downloaded here

Matplotlib is needed if you want to plot graphs. Matplotlib website

Scipy is needed if you want to use interpolation--typically if you have stylized pitch contours (in praat PitchTier format, for example) that you want to use in your morphing). Scipy website

Matplotlib and SciPy are non-trivial to install, as they depends on several large packages. You can visit their websites for more information. I recommend the following instructions to install matplotlib which uses python wheels. These will install all required libraries in one fell swoop.

On Mac, open a terminal and type:

python -m pip install matplotlib

python -m pip install scipy

On Windows, open a cmd or powershell window and type:

<<path to python>> -m pip install matplotlib

<<path to python>> -m pip install scipy

e.g. C:\python27\python.exe -m install matplotlib

Otherwise, to manually install, after downloading the source from github, from a command-line shell, navigate to the directory containing setup.py and type:

python setup.py install

If python is not in your path, you'll need to enter the full path e.g.:

C:\Python27\python.exe setup.py install

6   Usage

See /examples for example usages

7   Installation

If you on Windows, you can use the installer found here (check that it is up to date though) Windows installer

Promo is on pypi and can be installed or upgraded from the command-line shell with pip like so:

python -m pip install promo --upgrade

Otherwise, to manually install, after downloading the source from github, from a command-line shell, navigate to the directory containing setup.py and type:

python setup.py install

If python is not in your path, you'll need to enter the full path e.g.:

C:\Python36\python.exe setup.py install

8   Citing ProMo

If you use ProMo in your research, please cite it like so:

Tim Mahrt. ProMo: The Prosody-Morphing Library. https://github.com/timmahrt/ProMo, 2016.

9   Acknowledgements

Development of ProMo was possible thanks to NSF grant BCS 12-51343 to Jennifer Cole, José I. Hualde, and Caroline Smith and to the A*MIDEX project (n° ANR-11-IDEX-0001-02) to James Sneed German funded by the Investissements d'Avenir French Government program, managed by the French National Research Agency (ANR).

Owner
Tim
I write tools for working with speech data.
Tim
🔥RandLA-Net in Tensorflow (CVPR 2020, Oral & IEEE TPAMI 2021)

RandLA-Net: Efficient Semantic Segmentation of Large-Scale Point Clouds (CVPR 2020) This is the official implementation of RandLA-Net (CVPR2020, Oral

Qingyong 1k Dec 30, 2022
This repository for project that can Automate Number Plate Recognition (ANPR) in Morocco Licensed Vehicles. 💻 + 🚙 + 🇲🇦 = 🤖 🕵🏻‍♂️

MoroccoAI Data Challenge (Edition #001) This Reposotory is result of our work in the comepetiton organized by MoroccoAI in the context of the first Mo

SAFOINE EL KHABICH 14 Oct 31, 2022
This is the official repository for our paper: ''Pruning Self-attentions into Convolutional Layers in Single Path''.

Pruning Self-attentions into Convolutional Layers in Single Path This is the official repository for our paper: Pruning Self-attentions into Convoluti

Zhuang AI Group 77 Dec 26, 2022
Code for EMNLP2021 paper "Allocating Large Vocabulary Capacity for Cross-lingual Language Model Pre-training"

VoCapXLM Code for EMNLP2021 paper Allocating Large Vocabulary Capacity for Cross-lingual Language Model Pre-training Environment DockerFile: dancingso

Bo Zheng 15 Jul 28, 2022
Facilitating Database Tuning with Hyper-ParameterOptimization: A Comprehensive Experimental Evaluation

A Comprehensive Experimental Evaluation for Database Configuration Tuning This is the source code to the paper "Facilitating Database Tuning with Hype

DAIR Lab 9 Oct 29, 2022
This project aims to be a handler for input creation and running of multiple RICEWQ simulations.

What is autoRICEWQ? This project aims to be a handler for input creation and running of multiple RICEWQ simulations. What is RICEWQ? From the descript

Yass Fuentes 1 Feb 01, 2022
Self-Attention Between Datapoints: Going Beyond Individual Input-Output Pairs in Deep Learning

We challenge a common assumption underlying most supervised deep learning: that a model makes a prediction depending only on its parameters and the features of a single input. To this end, we introdu

OATML 360 Dec 28, 2022
Plenoxels: Radiance Fields without Neural Networks, Code release WIP

Plenoxels: Radiance Fields without Neural Networks Alex Yu*, Sara Fridovich-Keil*, Matthew Tancik, Qinhong Chen, Benjamin Recht, Angjoo Kanazawa UC Be

Alex Yu 2.3k Dec 30, 2022
Dynamic Attentive Graph Learning for Image Restoration, ICCV2021 [PyTorch Code]

Dynamic Attentive Graph Learning for Image Restoration This repository is for GATIR introduced in the following paper: Chong Mou, Jian Zhang, Zhuoyuan

Jian Zhang 84 Dec 09, 2022
Bidimensional Leaderboards: Generate and Evaluate Language Hand in Hand

Bidimensional Leaderboards: Generate and Evaluate Language Hand in Hand Introduction We propose a generalization of leaderboards, bidimensional leader

4 Dec 03, 2022
This repository contains codes of ICCV2021 paper: SO-Pose: Exploiting Self-Occlusion for Direct 6D Pose Estimation

SO-Pose This repository contains codes of ICCV2021 paper: SO-Pose: Exploiting Self-Occlusion for Direct 6D Pose Estimation This paper is basically an

shangbuhuan 52 Nov 25, 2022
基于YoloX目标检测+DeepSort算法实现多目标追踪Baseline

项目简介: 使用YOLOX+Deepsort实现车辆行人追踪和计数,代码封装成一个Detector类,更容易嵌入到自己的项目中。 代码地址(欢迎star): https://github.com/Sharpiless/yolox-deepsort/ 最终效果: 运行demo: python demo

114 Dec 30, 2022
Repository for the Bias Benchmark for QA dataset.

BBQ Repository for the Bias Benchmark for QA dataset. Authors: Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Tho

ML² AT CILVR 18 Nov 18, 2022
Some useful blender add-ons for SMPL skeleton's poses and global translation.

Blender add-ons for SMPL skeleton's poses and trans There are two blender add-ons for SMPL skeleton's poses and trans.The first is for making an offli

犹在镜中 154 Jan 04, 2023
GazeScroller - Using Facial Movements to perform Hands-free Gesture on the system

GazeScroller Using Facial Movements to perform Hands-free Gesture on the system

2 Jan 05, 2022
Code for the paper Relation Prediction as an Auxiliary Training Objective for Improving Multi-Relational Graph Representations (AKBC 2021).

Relation Prediction as an Auxiliary Training Objective for Knowledge Base Completion This repo provides the code for the paper Relation Prediction as

Facebook Research 85 Jan 02, 2023
Match SafeGraph POIs with Data collected through a cultural resource survey in Washington DC.

Match SafeGraph POI data with Cultural Resource Places in Washington DC Match SafeGraph POIs with Data collected through a cultural resource survey in

Changjie Chen 1 Jan 05, 2022
Square Root Bundle Adjustment for Large-Scale Reconstruction

RootBA: Square Root Bundle Adjustment Project Page | Paper | Poster | Video | Code Table of Contents Citation Dependencies Installing dependencies on

Nikolaus Demmel 205 Dec 20, 2022
pytorch, hand(object) detect ,yolo v5,手检测

YOLO V5 物体检测,包括手部检测。 项目介绍 手部检测 手部检测示例如下 : 视频示例: 项目配置 作者开发环境: Python 3.7 PyTorch = 1.5.1 数据集 手部检测数据集 该项目数据集采用 TV-Hand 和 COCO-Hand (COCO-Hand-Big 部分) 进

Eric.Lee 11 Dec 20, 2022
Bottom-up Human Pose Estimation

Introduction This is the official code of Rethinking the Heatmap Regression for Bottom-up Human Pose Estimation. This paper has been accepted to CVPR2

108 Dec 01, 2022