Messages
Update (2022/12/21): we have proposed Emo-CampNet, which can provide emotional options for generated speech on the basis of text-based speech editing, so as to make generated speech more expressive.
Update (2022/3/20): we have added comparative experiments of CampNet between EditSpeech (part 3) and Context-Aware-Prosody-Correction (part 4) method.This page shows the samples in the paper "CampNet: Context-Aware Mask Prediction for End-to-End Text-Based Speech Editing".
Experiments were based on VCTK and LibriTTS dataset. All test samples have not appeared in the training set and validation set.
Arxiv: https://arxiv.org/abs/2202.09950
Content
1: Deletion, insertion and replacement of text-based speech editing based on CampNet
| speaker | Original | Delete some words | Replace some words with new words | Insert some words | Insert long text (word-level autoregressive) | p374 (male) seen speaker |
The rainbow is a division of white light into many beautiful colors. |
The rainbow is a |
The rainbow is a division of black ( |
The rainbow is a just division of white light into many beautiful colors. |
The rainbow is a colorful light and just a division of white light into many beautiful colors. |
|---|---|---|---|---|---|
|
Others have tried to explain the phenomenon physically. |
Others have tried to explain |
Others have tried to translate ( |
Others have tried to explain the very weird phenomenon physically. |
Others have tried to explain the very very very very very weird phenomenon physically. |
p362 (female) seen speaker |
The rainbow is a division of white light into many beautiful colors. |
The rainbow is a |
The rainbow is a division of black ( |
The rainbow is just a division of white light into many beautiful colors. |
The rainbow is a colorful light and just a division of white light into many beautiful colors. |
|
Others have tried to explain the phenomenon physically. |
Others have tried to explain |
Others have tried to translate ( |
Others have tried to explain the very weird phenomenon physically. |
Others have tried to explain the very very very very very weird phenomenon physically. |
p226 (male) unseen speaker one-shot |
The rainbow is a division of white light into many beautiful colors. |
The rainbow is a |
The rainbow is a division of black ( |
The rainbow is just a division of white light into many beautiful colors. |
The rainbow is a colorful light and just a division of white light into many beautiful colors. |
|
Others have tried to explain the phenomenon physically. |
Others have tried to explain |
Others have tried to translate ( |
Others have tried to explain the very weird phenomenon physically. |
Others have tried to explain the very very very very very weird phenomenon physically. |
p225 (female) unseen speaker one-shot |
The rainbow is a division of white light into many beautiful colors. |
The rainbow is a |
The rainbow is a division of black ( |
The rainbow is just a division of white light into many beautiful colors. |
The rainbow is a colorful light and just a division of white light into many beautiful colors. |
|
Others have tried to explain the phenomenon physically. |
Others have tried to explain |
Others have tried to translate ( |
Others have tried to explain the very weird phenomenon physically. |
Others have tried to explain the very very very very very weird phenomenon physically. |
2: Comparison between CampNet and Other Systems (Synth, Voco and Edit on VCTK and LibriTTS dataset) (Replacement operation)
| Original | Synth | VoCo | Edit | CampNet |
|---|---|---|---|---|
|
Ask her to bring these things with her from the store. (VCTK testdata) |
Ask her to bring these things with her from the store. |
Ask her to bring these things with her from the store. |
Ask her to bring these things with her from the store. |
Ask her to bring these things with her from the store. |
|
We also need a small plastic snake and a big toy frog for the kids. (VCTK testdata) |
We also need a small plastic snake and a big toy frog for the kids. |
We also need a small plastic snake and a big toy frog for the kids. |
We also need a small plastic snake and a big toy frog for the kids. |
We also need a small plastic snake and a big toy frog for the kids. |
|
He says your mamma is not there. (LibriTTS testdata) |
He says your mamma is not there. |
He says your mamma is not there. |
He says your mamma is not there. |
He says your mamma is not there. |
|
We walked in the meadows, along the streams that watered them. (LibriTTS testdata) |
We walked in the meadows, along the streams that watered them. |
We walked in the meadows, along the streams that watered them. |
We walked in the meadows, along the streams that watered them. |
We walked in the meadows, along the streams that watered them. |
3: Comparison with EditSpeech (deletion, insertion and replacement)
Note: In this experiment, we take the speech sample on the demo page (Link) of EditSpeech as the standard, then we use CampNet to synthesize the corresponding parallel speech for comparison.
Due to the some difference between CampNet and EditSpeech, for example, different vocoders (CampNet-LpnNet, EditSpeech-HifiGAN), this experiment is only used as a reference for audiometry.
It is worth noting that we directly use CampNet to perform various operations on the speech on the demo page of EditSpeech without retraining or fine-tuneing the model, that is, the following results of CampNet are generated by one shot mode.
| Sample and System | Original | Deletion | Replacement | Insertion | Sample 1 EditSpeech |
For that reason cover should not be given. |
For that reason cover |
For that reason cover is impossible to ( |
for that theoretical and realistic reason cover should not be given. |
|---|---|---|---|---|
| CampNet |
For that reason cover |
For that reason cover is impossible to ( |
for that theoretical and realistic reason cover should not be given. |
Sample 2 EditSpeech |
What do we want to do that for? |
What do we |
What do we want to figure out ( |
What do we this afternoon want to do that for? |
| CampNet |
What do we |
What do we want to figure out ( |
What do we this afternoon want to do that for? |
Sample 3 EditSpeech |
Some have accepted it as a miracle without physical explanation. |
Some have accepted it |
Some have accepted it as an undeniable fact ( |
Some have accepted it as a miracle never seen before without physical explanation. |
| CampNet |
Some have accepted it |
Some have accepted it as an undeniable fact ( |
Some have accepted it as a miracle never seen before without physical explanation. |
4: Comparison with Context-Aware-Prosody-Correction (replacement)
Note: In this experiment, we take the speech sample on the demo page (Link) of Context-Aware-Prosody-Correction as the standard, then we use CampNet to synthesize the corresponding parallel speech for comparison.
Due to the some difference between Context-Aware-Prosody-Correction and CampNet (for example, the speech in the editing area of Context-Aware-Prosody-Correction is ready-made and the Context-Aware-Prosody-Correction model is only used to adjust the prosody of the editing speech, while CampNet directly synthesizes the speech in the editing area; different vocoders (CampNet-LpnNet, Context-Aware-Prosody-Correction-HifiGAN)), this experiment is only used as a reference for audiometry.
Note: In this experiment, all the speakers are unseen in the training set. The TTS system uses global style tokens(GST) to guide the synthesis of target speaker's voice (one-shot learning).
It is worth noting that we directly use CampNet to perform various operations on the speech on the demo page of Context-Aware-Prosody-Correction without retraining or fine-tuneing the model, that is, the following results of CampNet are generated by one shot mode.
Sample
Original
Context-Aware-Prosody-Correction
CampNet
female 0
and of which fewer still would believe that the original is to be found in this enlightened and happy country.
and of which fewer still would believe that the original is to be found in this enlightened and happy country.
and of which fewer still would believe that the original is to be found in this enlightened and happy country.
female 1
On November five, Hosty was traveling near Mrs.Paine's home and took the occasion to stop by to ask whether she had any further information.
On November five, Hosty was traveling near Mrs.Paine's home and took the occasion to stop by to ask whether she had any further information.
On November five, Hosty was traveling near Mrs.Paine's home and took the occasion to stop by to ask whether she had any further information.
male 0
I've seen him walk into a crowd of sawn routes and tell some drool yarn and bust them all up.
I've seen him walk into a crowd of sawn routes and tell some drool yarn and bust them all up.
I've seen him walk into a crowd of sawn routes and tell some drool yarn and bust them all up.
male 1
The steamer was about to put on steam again when I called out to them that they had forgotten to pay me.
The steamer was about to put on steam again when I called out to them that they had forgotten to pay me.
The steamer was about to put on steam again when I called out to them that they had forgotten to pay me.
5: Comparison with full TTS system (replacement operation)
| Original | Full TTS | CampNet |
|---|---|---|
|
Ask her to bring these things with her from the store. |
Ask her to bring these things with her from the store. |
Ask her to bring these things with her from the store. |
|
We also need a small plastic snake and a big toy frog for the kids. |
We also need a small plastic snake and a big toy frog for the kids. |
We also need a small plastic snake and a big toy frog for the kids. |
|
Many complicated ideas about the rainbow have been formed. |
Many complicated ideas about the rainbow have been formed. |
Many complicated ideas about the rainbow have been formed. |
|
The rainbow is a division of white light into many beautiful colors. |
The rainbow is a division of white light into many beautiful colors. |
The rainbow is a division of white light into many beautiful colors. |
6: Comparison between one-step generation and word-level autoregressive generation methods (insert long text) (unseen speaker)
| Original | One-step generation | word-level autoregressive generation |
|---|---|---|
|
The rainbow is a division of white light into many beautiful colors. |
The rainbow is a colorful light and just a division of white light into many beautiful colors. |
The rainbow is a colorful light and just a division of white light into many beautiful colors. |
|
Others have tried to explain the phenomenon physically. |
Others have tried to explain the very very very very very weird phenomenon physically. |
Others have tried to explain the very very very very very weird phenomenon physically. |
|
The rainbow is a division of white light into many beautiful colors. |
The rainbow is a colorful light and just a division of white light into many beautiful colors. |
The rainbow is a colorful light and just a division of white light into many beautiful colors. |
|
Others have tried to explain the phenomenon physically. |
Others have tried to explain the very very very very very weird phenomenon physically. |
Others have tried to explain the very very very very very weird phenomenon physically. |
7: Comparison of coarse-to-fine decoding
| Original | Onedecoder | Coarsedecoder | Finedecoder |
|---|---|---|---|
|
Ask her to bring these things with her from the store. |
Ask her to bring these things with her from the store. |
Ask her to bring these things with her from the store. |
Ask her to bring these things with her from the store. |
|
Many complicated ideas about the rainbow have been formed. |
Many complicated ideas about the rainbow have been formed. |
Many complicated ideas about the rainbow have been formed. |
Many complicated ideas about the rainbow have been formed. |
|
When a man looks for something beyond his reach, his friends say he is looking for the pot of gold at the end of the rainbow. |
When a man looks for something beyond his reach, his friends say he is looking for the pot of gold at the end of the rainbow. |
When a man looks for something beyond his reach, his friends say he is looking for the pot of gold at the end of the rainbow. |
When a man looks for something beyond his reach, his friends say he is looking for the pot of gold at the end of the rainbow. |
|
The rainbow is a division of white light into many beautiful colors. |
The rainbow is a division of white light into many beautiful colors. |
The rainbow is a division of white light into many beautiful colors. |
The rainbow is a division of white light into many beautiful colors. |
8: Few-shot and one-shot based on CampNet
| Speaker | Original | One-shot | One-shot 1utts-finetune | few-shot 50utts-finetune |
|---|---|---|---|---|
| p225 (female) |
She can scoop these things into three red bags, and we will go meet her Wednesday at the train station. |
She can scoop these things into three red bags, and we will go meet her Wednesday at the train station. |
She can scoop these things into three red bags, and we will go meet her Wednesday at the train station. |
She can scoop these things into three red bags, and we will go meet her Wednesday at the train station. |
| p225 (female) |
When a man looks for something beyond his reach, his friends say he is looking for the pot of gold at the end of the rainbow. |
When a man looks for something beyond his reach, his friends say he is looking for the pot of gold at the end of the rainbow. |
When a man looks for something beyond his reach, his friends say he is looking for the pot of gold at the end of the rainbow. |
When a man looks for something beyond his reach, his friends say he is looking for the pot of gold at the end of the rainbow. |
| p226 (male) |
She can scoop these things into three red bags, and we will go meet her Wednesday at the train station. |
She can scoop these things into three red bags, and we will go meet her Wednesday at the train station. |
She can scoop these things into three red bags, and we will go meet her Wednesday at the train station. |
She can scoop these things into three red bags, and we will go meet her Wednesday at the train station. |
| p226 (male) |
When a man looks for something beyond his reach, his friends say he is looking for the pot of gold at the end of the rainbow. |
When a man looks for something beyond his reach, his friends say he is looking for the pot of gold at the end of the rainbow. |
When a man looks for something beyond his reach, his friends say he is looking for the pot of gold at the end of the rainbow. |
When a man looks for something beyond his reach, his friends say he is looking for the pot of gold at the end of the rainbow. |
9: Robustness test of CampNet on the mask area's duration at the inference stage (insertion operation).
Note: In this experiment, we set different duration for the words to be inserted, and the duration range is large (the maximum duration can be close to three times the minimum duration). Through the experiment, we can find that the model has good robustness to the duration, and can automatically adjust the speed of the words to be edited with the change of duration.
| Speaker | Original | duration=0.225s | duration=0.3s | duration=0.375s | duration=0.45s | duration=0.525s | duration=0.6s |
|---|---|---|---|---|---|---|---|
| p225 (unseen speaker) |
The rainbow is a division of white light into many beautiful colors. |
The rainbow is just a division of white light into many beautiful colors. |
The rainbow is just a division of white light into many beautiful colors. |
The rainbow is just a division of white light into many beautiful colors. |
The rainbow is just a division of white light into many beautiful colors. |
The rainbow is just a division of white light into many beautiful colors. |
The rainbow is just a division of white light into many beautiful colors. |
| Speaker | Original | duration=0.375s | duration=0.5s | duration=0.625s | duration=0.75s | duration=0.875s | duration=1s |
| p226 (unseen speaker) |
Others have tried to explain the phenomenon physically. |
Others have tried to explain the very weird phenomenon physically. |
Others have tried to explain the very weird phenomenon physically. |
Others have tried to explain the very weird phenomenon physically. |
Others have tried to explain the very weird phenomenon physically. |
Others have tried to explain the very weird phenomenon physically. |
Others have tried to explain the very weird phenomenon physically. |