SenseVoice is a comprehensive speech foundation model developed by FunAudioLLM, designed to offer multiple speech understanding capabilities including Automatic Speech Recognition (ASR), Spoken Language Identification (LID), Speech Emotion Recognition (SER), and Audio Event Detection (AED).

The core methodology of SenseVoice, particularly the SenseVoice-Small model, relies on a non-autoregressive end-to-end framework, which contributes to its exceptionally low inference latency, processing 10 seconds of audio in approximately 70ms. This architecture significantly outperforms Whisper-Large (15 times faster) and Whisper-Small (5 times faster) in terms of inference speed, while maintaining a similar parameter count to Whisper-Small. The model is trained on an extensive dataset of over 400,000 hours of speech data, covering more than 50 languages, enabling robust multilingual performance. For timestamp generation, SenseVoice leverages CTC alignment.

In terms of performance, SenseVoice exhibits strong capabilities across its various tasks:

• Multilingual Speech Recognition: It surpasses Whisper on open-source benchmarks like AISHELL-1, AISHELL-2, Wenetspeech, LibriSpeech, and Common Voice, demonstrating particular advantages in Chinese and Cantonese recognition.
• Speech Emotion Recognition: SenseVoice achieves and often exceeds the performance of state-of-the-art emotion recognition models on diverse test sets encompassing Chinese and English data from performances, films, and natural conversations, even without specific finetuning on target data. The SenseVoice-Large model generally yields the best results, with SenseVoice-Small also outperforming most other open-source models.
• Audio Event Detection: Despite being primarily trained on speech data, SenseVoice functions as a capable event detection model. It achieves commendable results on the ESC-50 environmental sound classification dataset compared to specialized models like BEATS and PANN, although it acknowledges some performance gaps due to training data and methodological differences.

Key features and functionalities of SenseVoice include:

• Rich Transcribe: Integrates ASR with emotion recognition and audio event detection.
• Speaker Diarization: SenseVoice supports per-sentence speaker labeling when combined with Voice Activity Detection (VAD) models (e.g., fsmn-vad), speaker models (e.g., CAM++), and punctuation models (e.g., ct-punc). This allows for detailed segmentation of audio by speaker and content.
• Timestamp Support: Added based on CTC alignment for more precise event timing.
• Efficient Inference: The AutoModel API in the funasr toolkit provides flexible inference options, supporting various audio formats and durations. It can incorporate VAD for segmenting long audios, handle Inverse Text Normalization (ITN), and utilize dynamic batching (batch_size_s) or fixed batching (batch_size).
• Export and Runtime: The model can be exported to optimized formats such as ONNX (funasr-onnx) and Libtorch (funasr-torch) for deployment, with Python runtimes available.
• Convenient Finetuning: Provides scripts and strategies to allow users to adapt the model to specific datasets and address long-tail sample issues. The finetuning process requires data in jsonl format, which can be generated from wav.scp, text.txt, text_language.txt, emo_target.txt, and event_target.txt files. If language, emotion, or event labels are not provided, SenseVoice can automatically predict them.
• Service Deployment: Offers a service deployment pipeline using FastAPI, supporting multi-concurrent requests and accessible via client-side languages including Python, C++, HTML, Java, and C#.
• Docker Support: Provides Dockerfile and docker-compose.yaml for simplified setup and reproducible deployments on both CPU and GPU.

Related and optimized third-party works further enhance SenseVoice's utility:

• Triton (GPU) Deployment: Achieves significant acceleration (e.g., 526x on V100 GPU for FP32) using Triton Inference Server and TensorRT.
• Sherpa-onnx Deployment: Enables deployment across 10 programming languages and platforms like iOS, Android, and Raspberry Pi.
• SenseVoice.cpp: A pure C/C++ implementation based on GGML, supporting 3-bit, 4-bit, 5-bit, and 8-bit quantization without third-party dependencies.
• Streaming-sensevoice: Allows for pseudo-streaming inference via a truncated attention mechanism, supporting CTC prefix beam search and hot-word boosting.
• OmniSenseVoice: Optimized for lightning-fast inference and batching.
• SenseVoice Hotword: Integrates Neural Network Hotword Enhancement and Contextualized End-to-End Speech Recognition with a Contextual Phrase Prediction Network.