This paper outlines a methodology for fine-tuning Text-to-Speech (TTS) models using Unsloth, an optimization framework that claims to achieve 1.5x faster training with 50% less memory due to Flash Attention 2. The primary goal of fine-tuning is to customize TTS models for specific applications, including voice cloning, adaptation of speaking styles and tones, support for new languages, and specialized tasks. The framework also supports Speech-to-Text (STT) models like OpenAI's Whisper.

The core argument emphasizes the superiority of fine-tuning over zero-shot voice cloning. While zero-shot methods, available in models like Orpheus and CSM, can capture the general tone and timbre from brief audio samples, they often fail to replicate crucial expressive elements such as pacing, phrasing, vocal quirks, and subtle prosody, resulting in unnatural or robotic speech. Fine-tuning, conversely, delivers significantly more accurate and realistic voice replication by allowing the model to learn the specific nuances of a speaker's delivery.

The paper highlights several transformer-compatible TTS models supported by Unsloth, including Sesame-CSM (0.5B), Orpheus-TTS (3B), Spark-TTS (0.5B), Llasa-TTS (1B), and Oute-TTS (1B), alongside STT models like Whisper Large V3. For TTS, smaller models (under 3 billion parameters) are generally preferred due to lower latency and faster inference for end-users, with Sesame-CSM (1B) and Orpheus-TTS (3B) being primary examples.

Detailed descriptions of two key models are provided:

1. Sesame-CSM (1B): This is a base model that requires audio context (e.g., reference clips) for each speaker to achieve good performance and consistent voice identity across different generations, as its speaker ID tokens primarily aid consistency *within* a conversation. Fine-tuning from this base model typically demands more computational resources.
2. Orpheus-TTS (3B): A Llama-based speech model pre-trained on a large speech corpus, excelling at realistic speech generation with built-in support for emotional cues (e.g., $<laugh>$, $<sigh>$). Its architecture is designed for ease of use and training, capable of exporting via llama.cpp for broad inference engine compatibility. A key technical feature is that its tokenizer includes special tokens for audio output, meaning it directly outputs audio tokens, eliminating the need for a separate vocoder.

The fine-tuning methodology involves specific steps:

3. Model Loading: Models are loaded with Unsloth, typically enabling LoRA (Low-Rank Adaptation) in 16-bit precision ($load_in_4bit = False$) for higher quality results, or optionally 8-bit ($load_in_8bit = True$) for memory constraints, or full fine-tuning ($full_finetuning = True$) if sufficient VRAM is available.
4. Dataset Preparation: The minimum requirement is a dataset of audio clips (WAV/FLAC) and their corresponding text transcripts. The paper recommends using the Hugging Face datasets library for loading and preprocessing. For Orpheus, datasets can incorporate special emotion tags (e.g., $<laugh>$, $<sigh>$, $<chuckle>$) embedded in the transcripts. These tags, enclosed in angle brackets, are treated as distinct special tokens by Orpheus's tokenizer, allowing the model to learn their associated audio patterns. An example dataset, Elise (~3 hours, single-speaker), is mentioned, available in a base version and an augmented version with emotion tags. Essential dataset preprocessing steps include ensuring proper annotation, normalization of transcripts (no unusual characters), and consistent audio sampling rates (e.g., 24kHz for Orpheus).
5. Data Preprocessing for Training (Technical Detail): For Text-to-Speech, the model is trained in a causal manner. For Orpheus, which is a decoder-only LLM outputting audio, the text serves as the input context, and the audio token IDs serve as labels. This implicitly means that for fine-tuning, the audio data in the dataset must be converted into discrete audio tokens using an audio codec (e.g., Orpheus's internal codec), and these tokens form the actual labels for the model to predict. While Unsloth may abstract this process via an associated processor that automatically encodes audio, manual encoding using the model's specific encode_audio function is also a possibility if automatic tokenization is not supported. This step is crucial for the model to learn the mapping from text to actual audio patterns, beyond just text tokens.
6. Training Setup: This involves configuring transformers.TrainingArguments, specifying parameters such as num_train_epochs or max_steps, per_device_train_batch_size, and logging frequency.
7. Fine-tuning Execution: The training loop commences, with Unsloth's optimizations accelerating the process compared to standard Hugging Face training.
8. Model Saving: After training, only the LoRA adapters are saved by default. Options are provided for saving to 16-bit or GGUF format for broader deployment, including conversion via llama.cpp.

The paper reinforces that for true voice replication that captures a speaker's unique expressive qualities, fine-tuning is indispensable, whereas zero-shot methods offer only an approximation.