🗣️ Speech Generative Tasks
Benchmarking Results for Generative Tasks
📊 1st Probing Head
| Models/Tasks |
#Q |
SE |
SS (Speech) |
TTS |
| DNSMOS ↑ |
dWER ↓ |
SpkSim ↑ |
DNSMOS ↑ |
dWER ↓ |
SpkSim ↑ |
UTMOS ↑ |
dWER ↓ |
SpkSim ↑ |
| Enc-SMA-24 |
2 |
39.16 ± 1.85 |
63.44 ± 1.35 |
90.90 ± 0.32 |
51.00 ± 0.98 |
45.50 ± 1.40 |
42.90 ± 0.16 |
77.73 ± 3.12 |
89.81 ± 5.46 |
18.33 ± 0.26 |
| 8 |
16.83 ± 1.32 |
38.98 ± 2.16 |
84.53 ± 1.90 |
45.36 ± 0.57 |
44.73 ± 2.23 |
40.03 ± 0.29 |
74.30 ± 1.69 |
94.26 ± 3.99 |
13.54 ± 0.57 |
| 32 |
50.84 ± 34.53 |
66.04 ± 24.79 |
97.39 ± 1.19 |
58.21 ± 0.92 |
42.96 ± 1.75 |
33.66 ± 2.65 |
69.10 ± 3.42 |
91.12 ± 1.92 |
10.12 ± 6.66 |
| DAC-SMA-24 |
2 |
24.57 ± 0.16 |
50.60 ± 0.28 |
95.21 ± 0.84 |
68.93 ± 0.42 |
45.20 ± 1.50 |
29.83 ± 0.19 |
67.27 ± 1.56 |
97.88 ± 0.79 |
21.80 ± 1.00 |
| 8 |
16.29 ± 0.28 |
39.57 ± 0.29 |
97.20 ± 0.14 |
62.45 ± 1.40 |
44.73 ± 1.80 |
23.97 ± 0.41 |
65.27 ± 2.82 |
87.33 ± 10.98 |
15.86 ± 5.26 |
| 32 |
98.40 ± 0.34 |
98.42 ± 0.32 |
98.96 ± 0.18 |
73.57 ± 1.56 |
43.20 ± 2.30 |
44.60 ± 39.19 |
68.67 ± 2.91 |
87.69 ± 4.99 |
17.12 ± 0.76 |
| ST-S-16 |
2 |
3.19 ± 0.02 |
29.98 ± 0.58 |
0.86 ± 0.00 |
3.20 |
84.94 ± 0.63 |
0.87 ± 0.00 |
1.65 ± 0.04 |
4.31 ± 0.77 |
0.89 ± 0.00 |
| 8 |
3.49 ± 0.01 |
21.65 ± 0.57 |
0.87 ± 0.00 |
3.72 |
60.90 ± 0.77 |
0.91 ± 0.00 |
3.37 ± 0.24 |
10.39 ± 1.60 |
0.91 ± 0.00 |
| Mimi-S-24 |
8 |
3.25 ± 0.01 |
67.56 ± 2.21 |
0.85 ± 0.00 |
3.65 |
109.30 ± 3.3 |
0.87 ± 0.00 |
2.62 ± 0.04 |
9.62 ± 0.38 |
0.91 ± 0.00 |
| 32 |
3.18 ± 0.01 |
102.61 ± 2.40 |
0.82 ± 0.00 |
3.72 |
137.00 ± 2.16 |
0.82 ± 0.0 |
1.33 ± 0.00 |
65.31 ± 0.00 |
0.83 ± 0.00 |
| DWavL-S-16 |
2 |
3.56 ± 0.01 |
25.88 ± 2.15 |
0.88 ± 0.00 |
3.57 |
49.57 ± 0.64 |
0.85 ± 0.0 |
2.90 ± 0.01 |
3.32 ± 0.03 |
0.86 ± 0.00 |
| 6 |
3.57 ± 0.01 |
9.43 ± 0.33 |
0.89 ± 0.00 |
3.75 |
30.39 ± 0.45 |
0.91 ± 0.00 |
3.79 ± 0.01 |
3.93 ± 0.14 |
0.90 ± 0.00 |
| DHuBERT-S-16 |
2 |
3.45 ± 0.01 |
59.00 ± 3.92 |
0.85 ± 0.00 |
3.68 |
80.92 ± 1.88 |
0.84 ± 0.00 |
3.36 ± 0.00 |
3.28 ± 0.29 |
0.85 ± 0.00 |
| 6 |
3.49 ± 0.01 |
12.74 ± 0.15 |
0.88 ± 0.00 |
3.72 |
65.8 ± 1.46 |
0.89 ± 0.00 |
3.94 ± 0.00 |
3.70 ± 0.07 |
0.90 ± 0.00 |
| DWav2Vec2-S-16 |
2 |
3.55 ± 0.0 |
30.14 ± 0.44 |
0.88 ± 0.00 |
3.61 |
85.15 ± 0.2 |
0.82 ± 0.00 |
2.82 ± 0.02 |
5.58 ± 0.06 |
0.82 ± 0.00 |
| 6 |
3.57 ± 0.01 |
12.93 ± 0.56 |
0.89 ± 0.00 |
3.72 |
70.99 ± 10.27 |
0.87 ± 0.04 |
3.28 ± 0.03 |
12.21 ± 1.06 |
0.88 ± 0.00 |
| SQ-SMA-16 |
4 |
3.28 ± 0.01 |
122.33 ± 8.74 |
0.83 ± 0.00 |
3.77 |
136.00 ± 3.58 |
0.83 ± 0.00 |
-- |
-- |
-- |
| WT-SMA-24 |
1 |
3.33 ± 0.01 |
67.53 ± 10.65 |
0.85 ± 0.00 |
3.57 |
118.33 ± 4.50 |
0.86 ± 0.0 |
2.86 ± 0.04 |
4.95 ± 0.00 |
0.89 ± 0.00 |
| Mixture |
-- |
-- |
-- |
-- |
3.43 |
-- |
-- |
-- |
-- |
-- |
📊 2nd Probing Head
| Models/Tasks |
#Q |
SE |
SS (Speech) |
TTS |
| DNSMOS ↑ |
dWER ↓ |
SpkSim ↑ |
DNSMOS ↑ |
dWER ↓ |
SpkSim ↑ |
UTMOS ↑ |
dWER ↓ |
| Enc-SMA-24 |
2 |
3.17 ± 0.01 |
36.78 ± 1.15 |
0.85 ± 0.00 |
3.19 |
3.11 ± 0.01 |
86.95 ± 0.74 |
0.87 ± 0.00 |
1.34 ± 0.00 |
53.93 ± 0.99 |
| 8 |
2.94 ± 0.01 |
50.32 ± 2.81 |
0.84 ± 0.01 |
3.54 |
3.05 ± 0.00 |
67.65 ± 2.21 |
0.88 ± 0.00 |
-- |
-- |
| 32 |
3.00 ± 0.02 |
108.65 ± 9.64 |
0.76 ± 0.01 |
3.72 |
3.00 ± 0.00 |
98.82 ± 2.65 |
0.83 ± 0.00 |
-- |
-- |
| DAC-SMA-24 |
2 |
3.31 ± 0.01 |
61.72 ± 2.54 |
0.86 ± 0.00 |
3.16 |
2.97 ± 0.01 |
111.5 ± 0.50 |
0.839 ± 0.00 |
1.79 ± 0.03 |
36.93 ± 6.05 |
| 8 |
2.91 ± 0.10 |
131.33 ± 2.08 |
0.68 ± 0.03 |
3.67 |
3.16 ± 0.01 |
82.33 ± 1.42 |
0.892 ± 0.00 |
1.61 ± 0.05 |
102.08 ± 2.95 |
| 32 |
2.84 ± 0.02 |
103.88 ± 8.79 |
0.78 ± 0.01 |
3.76 |
2.66 ± 0.09 |
178.0 ± 25.0 |
0.80 ± 0.01 |
-- |
-- |
| ST-S-16 |
2 |
3.15 ± 0.01 |
31.76 ± 0.74 |
0.85 ± 0.00 |
3.20 |
3.12 ± 0.01 |
89.9 ± 0.94 |
0.862 ± 0.00 |
2.48 ± 0.01 |
46.59 ± 4.32 |
| 8 |
3.30 ± 0.01 |
33.95 ± 3.29 |
0.85 ± 0.00 |
3.72 |
3.29 ± 0.00 |
70.43 ± 0.75 |
0.87 ± 0.01 |
1.76 ± 0.02 |
89.06 ± 3.16 |
| Mimi-S-24 |
8 |
3.31 ± 0.01 |
107.27 ± 14.68 |
0.81 ± 0.01 |
3.37 |
3.39 ± 0.00 |
109.00 ± 2.16 |
0.806 ± 0.00 |
1.89 ± 0.03 |
25.26 ± 4.40 |
| 32 |
2.93 ± 0.14 |
159.0 ± 35.59 |
0.75 ± 0.01 |
3.65 |
3.20 ± 0.00 |
115.66 ± 1.25 |
0.84 ± 0.00 |
1.24 ± 0.01 |
112.22 ± 8.75 |
| DWavL-S-16 |
2 |
3.57 ± 0.01 |
27.64 ± 4.13 |
0.88 ± 0.00 |
3.57 |
2.97 ± 0.01 |
50.36 ± 1.05 |
0.839 ± 0.00 |
3.01 ± 0.02 |
4.99 ± 4.61 |
| 6 |
3.58 ± 0.00 |
9.39 ± 0.28 |
0.89 ± 0.00 |
3.75 |
3.77 ± 0.00 |
33.51 ± 0.53 |
0.896 ± 0.00 |
3.87 ± 0.02 |
13.35 ± 2.20 |
| DHuBERT-S-16 |
2 |
3.46 ± 0.03 |
59.97 ± 3.33 |
0.85 ± 0.00 |
3.68 |
3.65 ± 0.00 |
83.98 ± 0.18 |
0.826 ± 0.00 |
3.41 ± 0.01 |
1.79 ± 1.79 |
| 6 |
3.50 ± 0.01 |
13.31 ± 0.09 |
0.88 ± 0.00 |
3.72 |
3.69 ± 0.00 |
75.99 ± 1.26 |
0.88 ± 0.00 |
3.80 ± 0.00 |
5.76 ± 0.49 |
| DWav2Vec2-S-16 |
2 |
3.55 ± 0.01 |
30.91 ± 0.32 |
0.88 ± 0.00 |
3.61 |
3.64 ± 0.01 |
94.88 ± 1.98 |
0.805 ± 0.00 |
2.26 ± 0.02 |
26.26 ± 1.26 |
| 6 |
3.57 ± 0.01 |
13.06 ± 0.40 |
0.89 ± 0.00 |
3.77 |
3.74 ± 0.01 |
74.59 ± 1.40 |
0.88 ± 0.00 |
3.07 ± 0.01 |
13.96 ± 0.32 |
| SQ-SMA-16 |
4 |
3.09 ± 0.04 |
328.0 ± 35.00 |
0.76 ± 0.01 |
3.77 |
3.59 ± 0.00 |
72.39 ± 0.6 |
0.90 ± 0.00 |
-- |
-- |
| WT-SMA-24 |
1 |
3.36 ± 0.01 |
62.15 ± 2.52 |
0.85 ± 0.00 |
3.57 |
3.43 ± 0.01 |
128.33 ± 2.49 |
0.85 ± 0.00 |
2.44 ± 0.01 |
14.09 ± 0.84 |
| Mixture |
-- |
-- |
-- |
-- |
-- |
3.43 |
-- |
-- |
-- |
-- |