Speech Enhancement in TTS Systems

Text-to-speech (TTS) systems can produce understandable speech with audible hiss, roughness, or other noise-like artifacts. Removing those sounds is useful only if the words, pitch, and character of the voice survive the processing.

In this Winter of Code project with Skit, I worked on speech-enhancement filters and the metrics needed to evaluate them. The project asked two questions:

Can established signal-processing methods improve the audio?
How can we tell whether an output is clearer, rather than simply quieter?

Source code · Published project write-up · Presentation (PDF)

The central finding

The same filter can behave very differently on noisy recordings and on synthesized speech. Our classical filters were more successful on NOIZEUS, a benchmark of speech mixed with environmental noise, than on the Skit TTS dataset. Some filtered TTS outputs lost speech quality instead of improving it.

This pointed to an important assumption mismatch: a method designed to estimate a separate background-noise process may also suppress parts of a synthesized voice. The published project write-up discusses this limitation and the motivation for investigating learned denoisers.

Listen to one example

The clips below are stored outputs from the project repository: one NOIZEUS utterance, its version with station noise at 5 dB SNR, and two filtered versions. A lower signal-to-noise ratio (SNR) means noise is stronger relative to the speech.

Listen for both the background sound and the speech itself. Do quieter pauses come with duller consonants, a changed voice, or other artifacts?

Clean reference

Noisy input · station, 5 dB

Wiener-filtered output

Log-MMSE-filtered output

These are examples from the repository’s audio collection, not a ranking of the methods across the full dataset.

What does the sound look like?

A spectrogram shows how energy at different frequencies changes over time. Brighter regions indicate stronger spectral energy. Speech structure and background noise can occupy overlapping regions, which helps explain why aggressive filtering can damage the voice as well as reduce noise.

Four spectrograms of the same NOIZEUS utterance: clean reference, station-noise input at 5 dB, Wiener-filtered output, and log-MMSE-filtered output. All panels use the same frequency range and color scale. — Spectrograms computed from the four stored clips above, using one shared magnitude reference and fixed color limits. This is a single listening example, not an aggregate benchmark. A darker output alone does not establish better speech quality.

How the enhancement methods work

A common starting model is noisy recording = clean speech + background noise. An enhancer estimates the speech from the observed recording. Its success depends on how closely its assumptions match the actual disturbance.

I implemented and tested five classical approaches:

Method	Intuition	Important trade-off
Wiener filter	Attenuate spectral components according to estimated speech and noise power.	Strong attenuation can also remove weak speech components.
Kalman filter	Recursively estimate an evolving speech signal using a dynamical model and noisy observations.	Performance depends on the signal model and noise estimates.
Spectral subtraction with oversubtraction and flooring	Subtract an estimated noise spectrum; retain a floor to avoid excessively deep spectral valleys.	More subtraction can distort speech; the floor trades residual noise against musical-noise artifacts.
Bayesian MMSE	Estimate the clean short-time spectral amplitude by minimizing expected squared amplitude error.	An amplitude-error objective does not capture every aspect of perceived quality.
Bayesian log-MMSE	Estimate spectral amplitude using a squared-error objective in the log-amplitude domain.	The different objective can improve perceptual behavior, but still relies on a suitable noise model.

The filter implementations and visual examples show how these methods were applied.

Why reducing noise can also damage speech

For a spectral filter, each time–frequency component receives a gain. A gain close to 1 preserves the component; a small gain suppresses it. The filter must infer whether a component is primarily speech or noise from the observed signal and its noise estimate.

If a quiet consonant or a detail of the synthesized voice resembles the estimated noise, it can be attenuated too. Spectral subtraction has a related sensitivity: its oversubtraction factor controls how much estimated noise is removed, while its floor controls how much residual energy remains. The useful setting depends on the audio and disturbance.

This is why lower output energy or a cleaner-looking spectrogram is insufficient evidence of successful enhancement.

Two datasets, two different tests

Dataset	What it contains	What it tests
NOIZEUS	30 speech utterances from six speakers, corrupted with eight environmental noise types at 0, 5, 10, and 15 dB SNR.	How filters handle known additive-noise conditions, with clean references available.
Skit TTS dataset	Speech generated by the in-house TTS systems.	Whether the same methods improve the noise-like artifacts present in synthesized speech.

The environmental conditions included airport, babble, car, exhibition hall, restaurant, street, train, and station noise. The dataset documentation includes examples across noise types and SNR levels; the original benchmark is NOIZEUS.

A filter that works on an artificially mixed recording has not yet demonstrated that it will work on synthesis artifacts. The second dataset was essential to test that transfer.

Measuring improvement from several angles

Speech quality has several dimensions. An output can sound less noisy yet be harder to understand, or preserve the words while altering the pitch and timbre. The project therefore implemented and tested a collection of complementary metrics.

Question	Metrics	How to read them
Does it sound closer to a clean reference?	Perceptual Evaluation of Speech Quality (PESQ, narrowband and wideband)	Higher scores generally indicate better predicted perceptual quality.
Are the words easier to understand?	Short-Time Objective Intelligibility (STOI)	Higher scores indicate better estimated intelligibility relative to a reference.
How much has the spectral character changed?	Mel-Cepstral Distortion (MCD, two implementations) and Mean Speech Distortion (MSD)	Lower distortion is better; interpretation depends on the representation and alignment.
Are pitch and voicing preserved?	Gross Pitch Error (GPE), Voicing Decision Error (VDE), and F0 Frame Error (FFE)	Lower error rates are better. GPE focuses on pitch disagreement in mutually voiced frames; VDE captures voiced/unvoiced mistakes; FFE combines pitch and voicing errors.
Does a recognizer recover the intended words?	Word Error Rate (WER)	Lower is better, but the result also depends on the speech recognizer.

The metric modules and examples provide a reusable evaluation component of the project.

For reference-based comparisons, sample rates, channel handling, and temporal alignment matter. A natural recording of the same sentence is not automatically a perfectly matched reference for synthetic speech: legitimate differences in timing, voice, or prosody can affect a score. Listening remains important when interpreting the metrics.

What we learned

The original experiments and presentation support three practical observations:

The noise condition matters. Wiener and Kalman filtering were strong baselines on NOIZEUS, with their relative performance depending on the noise and SNR.
The metric matters. A gain in one measure does not establish that all aspects of the voice improved. Quality, intelligibility, spectral distortion, and pitch should be examined together.
The target domain matters. The classical methods did not transfer cleanly to the Skit TTS examples. Assumption mismatch was a plausible explanation: synthesis artifacts need not behave like a separate, slowly varying background-noise process.

This motivated follow-up exploration of learned enhancement models, including Facebook Denoiser, SEGAN, and RNNoise. Their usefulness would still need to be assessed on the TTS artifacts of interest; the signal-processing results do not establish a neural-model result.

Explore the work

Get the code

git clone https://github.com/skit-ai/woc-tts-enhancement
cd woc-tts-enhancement
python -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txt

On Windows, activate the environment with .venv\Scripts\activate. The repository contains the filter modules, evaluation metrics, notebooks, and stored example outputs used in this page.