How AI Stem Separation Actually Works

When you bounce a song to MP3, every instrument is summed into two channels. Mathematically that is a lossy, one-way operation: dozens of sources collapse into a left and a right waveform, and the information about which sound came from where is gone. So the first thing worth being honest about is that stem separation is not undoing the mixdown. Nothing can. What a separation model does is make a very well-informed guess about what each instrument probably sounded like, and then build brand new audio files from that guess.

That distinction matters practically. It tells you why a sparse acoustic duo separates almost perfectly while a dense, heavily-limited festival mix leaves smeared edges. It tells you why a cymbal crash sometimes leaks into the vocal stem. And it tells you which source material to feed the tool if you want usable results.

Step one: from waveform to spectrogram

A raw waveform is a terrible thing to ask a neural network to reason about musically. At 44.1 kHz, three minutes of stereo audio is roughly 16 million numbers, and the thing that identifies a snare drum — its burst of broadband noise around 200 Hz plus a bright transient — is not visible in that list of amplitudes in any simple way.

So the audio is first converted into a spectrogram using a Short-Time Fourier Transform. The STFT chops the signal into short overlapping windows, typically around 1024 to 4096 samples each, and reports how much energy sits in each frequency band during that window. The result is essentially a picture: time runs left to right, frequency runs bottom to top, and brightness represents energy.

On that picture, instruments become visually distinctive in ways a network can learn. A bass line is a thick band of energy hugging the bottom, moving in slow steps. A hi-hat is a thin vertical smear at the top, repeating on a grid. A sung vowel is a stack of evenly-spaced horizontal lines — the harmonics of the vocal fold vibration — that wobble together as the singer applies vibrato. That coordinated wobble is one of the strongest cues a model has for identifying a voice.

Step two: predicting a mask, not a sound

Here is the part that surprises most people. The network does not output audio. It outputs a mask: a value between 0 and 1 for every single time-frequency cell in the spectrogram, for every stem, answering the question how much of the energy in this cell belongs to this instrument?

If the model is confident that the cell at 0.8 seconds and 3.2 kHz is almost entirely vocal, it writes something close to 1.0 in the vocal mask and close to 0 in the drum and bass masks. Multiply the original mixed spectrogram by the vocal mask and you have an estimated vocal spectrogram. Run an inverse STFT on that, borrowing the phase information from the original mix, and you get an audio file.

Step three: the architecture doing the guessing

The dominant design for this task is an encoder-decoder network with skip connections, usually called a U-Net. It was borrowed from medical image segmentation, and the analogy is exact: segmenting a tumour from surrounding tissue in a scan is structurally the same problem as segmenting a vocal from surrounding instruments in a spectrogram.

The encoder repeatedly downsamples the spectrogram, each layer trading fine detail for broader context. Early layers see individual harmonics; deep layers see whole musical phrases and can use the fact that a chorus arrangement usually differs from a verse arrangement. The decoder then upsamples back to full resolution to produce the mask. The skip connections are what make it work: they pass the high-resolution detail from each encoder layer directly across to the matching decoder layer, so the network can make a decision using musical context without losing the ability to place that decision precisely in time and frequency.

VocaSplitter is built on the Spleeter family of models, which uses this spectrogram-masking approach. Newer research systems such as Demucs work partly or entirely in the time domain, predicting waveforms directly rather than masks, which tends to preserve transients better at the cost of considerably more computation. Both approaches share the same core limitation: they can only reproduce what they have learned to recognise.

How the model learned in the first place

Training requires the one thing that is otherwise impossible to obtain: songs where the true separated stems are already known. Researchers use datasets of multitrack recordings — MUSDB18 is the standard benchmark, with 150 songs supplied as isolated vocals, drums, bass and other. The training loop is straightforward:

1. 01Sum the known stems together to synthesise a mix. Now you have an input and the exact correct answer.
2. 02Feed the mix through the network and get its predicted stems.
3. 03Measure the difference between prediction and truth, usually as an L1 or L2 loss on the spectrogram magnitudes.
4. 04Nudge every weight in the network slightly in the direction that reduces that difference.
5. 05Repeat several hundred thousand times, with random crops, gain changes and stem-swapping used to multiply the effective size of the dataset.

This is also the source of the model's blind spots. A few hundred training songs is a tiny sample of recorded music, and it skews heavily toward Western pop, rock and electronic material with conventional four-piece arrangements. Instruments that are rare in the training set get folded into whichever learned category they most resemble.

Where the “other” stem comes from

In a five-stem split you get vocals, drums, bass, piano and other. That last one is not a category the model understands so much as a remainder. Anything that is not confidently one of the first four ends up there: electric guitars, synth pads, strings, brass, mandolin, steel pan, talking drum. If you separate a song built around an instrument the model never learned as its own class, expect to find it in other — and expect some of it to bleed into piano, since both often occupy the same mid-range harmonic territory.

This is why the four- and five-stem modes are not strictly better than the two-stem mode. Every extra class is another decision boundary the model can get wrong. A clean two-way split into vocals and instrumental gives the model an easier problem and often produces a more usable instrumental than summing four separately-estimated stems back together.

Why artifacts sound the way they do

Once you know the process is per-cell masking, the characteristic flaws become predictable rather than mysterious.

What this means for the files you upload

Three practical consequences follow directly from the mechanism, and they are worth more than any settings tweak.

• •Feed the model real bandwidth. A 128 kbps MP3 has already had most content above roughly 16 kHz discarded by the encoder. The model will faithfully separate a song that has no top end, and your stems will sound dull. Use WAV, FLAC, or at minimum a 256 kbps MP3.
• •Prefer dynamic masters. A mix that has been crushed to a very high perceived loudness has less separation between elements for the model to find. If you have access to a pre-master or a vinyl rip, it will usually separate better than the streaming master.
• •Arrangement density predicts quality more than genre does. A sparse gospel recording with one voice and an organ separates beautifully. A wall-of-sound production with sixteen layered synths does not, regardless of how modern it is.

The honest summary

Stem separation in 2026 is genuinely useful and genuinely imperfect. For karaoke tracks, practice loops, transcription, DJ edits and sample sourcing, the quality is well past the threshold where it saves real time. For commercial release of an isolated vocal, it is usually not there yet — you can hear the process if you listen for it on headphones. Knowing that the tool is estimating rather than recovering lets you plan around it: pick the mode with the fewest classes that gets you what you need, feed it the best master you have, and treat the output as raw material rather than a finished asset.