Erasing Your Voice Before It’s Heard:
Training-free Speaker Unlearning for Zero-shot Text-to-Speech

MyungJin Lee^*, Eunji Lee^*, Jiyoung Lee⁺

Ewha Womans University

⁺ Corresponding author: lee.jiyoung@ewha.ac.kr

Abstract

Modern zero-shot text-to-speech (TTS) models offer unprecedented expressivity but also pose serious crime risks, as they can synthesize voices of individuals who never consented. Existing unlearning approaches, reliant on retraining, are costly and limited to speakers seen in the training set. We present TruS, a training-free speaker unlearning framework that shifts the paradigm from data deletion to inference-time control. TruS steers identity-specific hidden activations to suppress target speakers while preserving other attributes (e.g., prosody and emotion). Experiments on F5-TTS show that TruS effectively forgets both seen and unseen speakers without retraining, establishing a scalable safeguard for speech.