julien-c HF staff Ezi commited on
Commit
4e8973e
1 Parent(s): 26d3d1f

restructuring the MC (#4)

Browse files

- restructuring the MC (bc779c1f629bfb8083e590bcec2698f1744915ed)
- update: some edits (e0cc994d667ffa91461912d50241bb589d6ed0a4)
- update: hyperlink (5f71f97a7a9b49b8439c55211cb425b7179f7ea6)
- fixing points 2 & 3 (d015c0470c2a126a975605569450ec2be52f722a)


Co-authored-by: Ezi Ozoani <[email protected]>

Files changed (1) hide show
  1. README.md +172 -5
README.md CHANGED
@@ -15,7 +15,7 @@ This model was introduced in [SpeechT5: Unified-Modal Encoder-Decoder Pre-Traini
15
 
16
  SpeechT5 was first released in [this repository](https://github.com/microsoft/SpeechT5/), [original weights](https://huggingface.co/mechanicalsea/speecht5-tts). The license used is [MIT](https://github.com/microsoft/SpeechT5/blob/main/LICENSE).
17
 
18
- Disclaimer: The team releasing SpeechT5 did not write a model card for this model so this model card has been written by the Hugging Face team.
19
 
20
  ## Model Description
21
 
@@ -25,6 +25,59 @@ Leveraging large-scale unlabeled speech and text data, we pre-train SpeechT5 to
25
 
26
  Extensive evaluations show the superiority of the proposed SpeechT5 framework on a wide variety of spoken language processing tasks, including automatic speech recognition, speech synthesis, speech translation, voice conversion, speech enhancement, and speaker identification.
27
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
28
  ## How to Get Started With the Model
29
 
30
  Use the code below to convert text into a mono 16 kHz speech waveform.
@@ -53,13 +106,106 @@ speech = model.generate_speech(inputs["input_ids"], speaker_embeddings, vocoder=
53
  sf.write("speech.wav", speech.numpy(), samplerate=16000)
54
  ```
55
 
56
- ## Intended Uses & Limitations
57
 
58
- You can use this model for speech synthesis. See the [model hub](https://huggingface.co/models?search=speecht5) to look for fine-tuned versions on a task that interests you.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
59
 
60
- Currently, both the feature extractor and model support PyTorch.
61
 
62
- ## Citation
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
63
 
64
  **BibTeX:**
65
 
@@ -73,3 +219,24 @@ Currently, both the feature extractor and model support PyTorch.
73
  pages={5723--5738},
74
  }
75
  ```
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
15
 
16
  SpeechT5 was first released in [this repository](https://github.com/microsoft/SpeechT5/), [original weights](https://huggingface.co/mechanicalsea/speecht5-tts). The license used is [MIT](https://github.com/microsoft/SpeechT5/blob/main/LICENSE).
17
 
18
+
19
 
20
  ## Model Description
21
 
 
25
 
26
  Extensive evaluations show the superiority of the proposed SpeechT5 framework on a wide variety of spoken language processing tasks, including automatic speech recognition, speech synthesis, speech translation, voice conversion, speech enhancement, and speaker identification.
27
 
28
+ - **Developed by:** Junyi Ao, Rui Wang, Long Zhou, Chengyi Wang, Shuo Ren, Yu Wu, Shujie Liu, Tom Ko, Qing Li, Yu Zhang, Zhihua Wei, Yao Qian, Jinyu Li, Furu Wei.
29
+ - **Shared by [optional]:** [Matthijs Hollemans](https://huggingface.co/Matthijs)
30
+ - **Model type:** text-to-speech
31
+ - **Language(s) (NLP):** [More Information Needed]
32
+ - **License:** [MIT](https://github.com/microsoft/SpeechT5/blob/main/LICENSE)
33
+ - **Finetuned from model [optional]:** [More Information Needed]
34
+
35
+
36
+ ## Model Sources [optional]
37
+
38
+ <!-- Provide the basic links for the model. -->
39
+
40
+ - **Repository:** [https://github.com/microsoft/SpeechT5/]
41
+ - **Paper:** [https://arxiv.org/pdf/2110.07205.pdf]
42
+ - **Blog Post:** [https://huggingface.co/blog/speecht5]
43
+ - **Demo:** [https://huggingface.co/spaces/Matthijs/speecht5-tts-demo]
44
+
45
+
46
+ # Uses
47
+
48
+ <!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
49
+
50
+ ## Direct Use
51
+
52
+ <!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
53
+
54
+ You can use this model for speech synthesis. See the [model hub](https://huggingface.co/models?search=speecht5) to look for fine-tuned versions on a task that interests you.
55
+
56
+ ## Downstream Use [optional]
57
+
58
+ <!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
59
+
60
+ [More Information Needed]
61
+
62
+ ## Out-of-Scope Use
63
+
64
+ <!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
65
+
66
+ [More Information Needed]
67
+
68
+ # Bias, Risks, and Limitations
69
+
70
+ <!-- This section is meant to convey both technical and sociotechnical limitations. -->
71
+
72
+ [More Information Needed]
73
+
74
+ ## Recommendations
75
+
76
+ <!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
77
+
78
+ Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
79
+
80
+
81
  ## How to Get Started With the Model
82
 
83
  Use the code below to convert text into a mono 16 kHz speech waveform.
 
106
  sf.write("speech.wav", speech.numpy(), samplerate=16000)
107
  ```
108
 
109
+ # Training Details
110
 
111
+ ## Training Data
112
+
113
+ <!-- This should link to a Data Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
114
+
115
+ LibriTTS
116
+
117
+ ## Training Procedure
118
+
119
+ <!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
120
+
121
+ ### Preprocessing [optional]
122
+
123
+ Leveraging large-scale unlabeled speech and text data, we pre-train SpeechT5 to learn a unified-modal representation, hoping to improve the modeling capability for both speech and text.
124
+
125
+
126
+ ### Training hyperparameters
127
+ - **Precision:** [More Information Needed] <!--fp16, bf16, fp8, fp32 -->
128
+ - **Regime:** [More Information Needed] <!--mixed precision or not -->
129
+
130
+ ### Speeds, Sizes, Times [optional]
131
+
132
+ <!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
133
+
134
+ [More Information Needed]
135
+
136
+ # Evaluation
137
+
138
+ <!-- This section describes the evaluation protocols and provides the results. -->
139
+
140
+ ## Testing Data, Factors & Metrics
141
+
142
+ ### Testing Data
143
+
144
+ <!-- This should link to a Data Card if possible. -->
145
 
146
+ [More Information Needed]
147
 
148
+ ### Factors
149
+
150
+ <!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
151
+
152
+ [More Information Needed]
153
+
154
+ ### Metrics
155
+
156
+ <!-- These are the evaluation metrics being used, ideally with a description of why. -->
157
+
158
+ [More Information Needed]
159
+
160
+ ## Results
161
+
162
+ [More Information Needed]
163
+
164
+ ### Summary
165
+
166
+
167
+
168
+ # Model Examination [optional]
169
+
170
+ <!-- Relevant interpretability work for the model goes here -->
171
+
172
+ Extensive evaluations show the superiority of the proposed SpeechT5 framework on a wide variety of spoken language processing tasks, including automatic speech recognition, speech synthesis, speech translation, voice conversion, speech enhancement, and speaker identification.
173
+
174
+ # Environmental Impact
175
+
176
+ <!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
177
+
178
+ Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
179
+
180
+ - **Hardware Type:** [More Information Needed]
181
+ - **Hours used:** [More Information Needed]
182
+ - **Cloud Provider:** [More Information Needed]
183
+ - **Compute Region:** [More Information Needed]
184
+ - **Carbon Emitted:** [More Information Needed]
185
+
186
+ # Technical Specifications [optional]
187
+
188
+ ## Model Architecture and Objective
189
+
190
+ The SpeechT5 framework consists of a shared encoder-decoder network and six modal-specific (speech/text) pre/post-nets.
191
+
192
+ After preprocessing the input speech/text through the pre-nets, the shared encoder-decoder network models the sequence-to-sequence transformation, and then the post-nets generate the output in the speech/text modality based on the output of the decoder.
193
+
194
+ ## Compute Infrastructure
195
+
196
+ [More Information Needed]
197
+
198
+ ### Hardware
199
+
200
+ [More Information Needed]
201
+
202
+ ### Software
203
+
204
+ [More Information Needed]
205
+
206
+ # Citation [optional]
207
+
208
+ <!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
209
 
210
  **BibTeX:**
211
 
 
219
  pages={5723--5738},
220
  }
221
  ```
222
+
223
+ # Glossary [optional]
224
+
225
+ <!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
226
+
227
+ - **text-to-speech** to synthesize audio
228
+
229
+ # More Information [optional]
230
+
231
+ [More Information Needed]
232
+
233
+ # Model Card Authors [optional]
234
+
235
+ Disclaimer: The team releasing SpeechT5 did not write a model card for this model so this model card has been written by the Hugging Face team.
236
+
237
+ # Model Card Contact
238
+
239
+ [More Information Needed]
240
+
241
+
242
+