Mistral’s Voxtral-4B-TTS is a big model, and on paper it should make ElevenLabs look dated. Then you get to the weights. They published them with the encoder truncated, which limits Voxtral Voice Cloning to the presets they approved in advance.
Where are the encoder weights?
Anyone who has worked with autoencoders knows the shape of it: encoder, bottleneck, decoder. Mistral’s 350M audio autoencoder, the Voxtral Codec, produces 37 discrete tokens for every 80ms audio frame. With the encoder weights withheld, there is no native way to feed a new voice in and get the conditioning tokens that cloning needs. Deep learning models are not generally invertible either, so reversing the decoder to find the source codes is not an option.
The backbone is a 3B model based on Ministral, and it generates voice tokens autoregressively. That is good for streaming and not much use when you cannot hand it a reference voice. This is the point where most people give up on the model. There is a way through it, but it involves some optimization work.
A workaround for Voxtral voice cloning
Treat the missing encoder as an optimization problem. Rather than predicting the codes with a model, use gradient descent to train a set of codes that reconstruct the target audio when they pass through the decoder. You initialize a trainable parameter for the audio codes and run a loop that minimizes reconstruction loss.
The catch is that the tokens are discrete, so there is no smooth path from token A to token B to optimize along. That is what a Straight-Through Estimator (STE) is for: gradients flow back through the rounding used in Finite Scalar Quantization (FSQ), so your continuous best guess keeps updating while the forward pass still sees the hard discrete values the decoder expects.
# Example of STE logic for Acoustic Tokens
import torch
import torch.nn as nn
class bbioon_AcousticOptimizer(nn.Module):
def __init__(self, num_frames):
super().__init__()
# 36 acoustic codes per frame
self.values = nn.Parameter(torch.randn(num_frames, 36))
self.levels = 22
def forward(self):
normalized = torch.tanh(self.values)
scaled = ((normalized + 1) / 2) * (self.levels - 1)
quantized = scaled.round()
# The STE trick: forward uses quantized, backward uses scaled
return scaled + (quantized - scaled).detach()Semantic or acoustic tokens?
Semantic tokens are widely assumed to carry the words being spoken. In my own testing they turned out to be fairly robust to noise: corrupt them and the voice stays recognizable, while the speech itself gets harder to make out. So if fidelity is what you want from a Voxtral Voice Cloning run, spend the optimization budget on the acoustic components.
For a clean signal, add Short-Time Fourier Transform (STFT) loss on top of the usual L1 reconstruction loss. That pushes the optimization to respect frequency bins instead of chasing waveform amplitude, which is where high-frequency audio synthesis usually falls apart. On the wider shift going on here, I wrote up WordCamp Asia 2026: AI, Enterprise, and Core Tech.
If Voxtral voice cloning is eating your dev hours, hand it over. I have been working with WordPress and awkward AI integrations since the 4.x days.
Overfitting is the objective here
Overfitting is normally the thing you spend your time avoiding. Here it is the whole point. You are overfitting the parameters to a single 8-second audio clip to pull out the codes Mistral will not give you. It takes about an hour on an M-series Mac, and the result is indistinguishable from the target. Worth reading alongside this: the Mistral AI Voxtral announcement and the Wav2Vec2 paper for the signal processing underneath.
