CrisperWhisper 2.0 is an advanced speech recognition framework designed to resolve the ambiguity between "what was said" (verbatim) and "what was meant" (intended) by making transcription style an explicit, controllable parameter. It addresses common ASR shortcomings, specifically inaccurate word-level timing, hallucination loops, and inconsistent handling of disfluencies (fillers, stutters, and repetitions).

Core Methodology and Technical Architecture

1. Dual-Mode Transcription:

The model provides two distinct output modes. The *Verbatim* mode captures the raw, exact signal including all vocal events and disfluencies, while the *Intended* mode produces clean, formatted text suitable for professional documentation. This control is achieved through style-conditioned training across multiple languages.

2. Supervised Alignment:

CrisperWhisper 2.0 utilizes the model’s internal supervised cross-attention mechanisms to derive word-level timestamps. By training these attention heads for high-precision boundary detection, the system achieves a mean absolute boundary error of approximately 29.6 ms on read speech (TIMIT), outperforming standard alignment tools like NeMo-FA and CTC-based segmenters.

3. "Verbatimize" Pipeline:

This feature enables the enrichment of existing clean, text-only transcripts with audio-grounded disfluencies. By taking audio input and a trusted transcript, the model acts as an aligner-plus-inserter, allowing users to transform clean datasets into high-fidelity verbatim corpora, which significantly improves rare-word recall (increasing from 6.8% to 96.1% compared to traditional re-transcription).

4. Longform Inference and Hallucination Mitigation:

The system employs a "conditional continuation" strategy for audio segments exceeding 30 seconds. Instead of relying on fragile, timestamp-token-based stitching, each sliding window is conditioned on the tokens generated by the previous window. This preserves state and eliminates chunk-boundary artifacts (e.g., duplicated or dropped words). Furthermore, it incorporates an active hallucination mitigation layer that detects and suppresses the recursive looping-repetition failure modes common in standard Whisper architectures during the decoding phase.

5. Optimized Production Runtime:

The framework utilizes a CTranslate2 (CT2) backend to enable production-grade performance. It supports speculative decoding, where a smaller "draft" model (e.g., turbo) proposes token sequences that are verified by the larger main model, resulting in a 1.3x to 1.4x speed increase without sacrificing output quality. The architecture also supports quantization (e.g., int8_float16) to balance memory footprint and inference latency.

Benchmark Performance

The project introduces the "Nyra Verbatim Speech Benchmark," which evaluates models based on Disfluency F1—a metric that classifies and scores fillers, repetitions, and cut-offs individually rather than relying on an aggregate Word Error Rate (WER). CrisperWhisper 2.0 achieves a state-of-the-art Disfluency F1 score of 87.8 (or 93.5 for the Pro version), significantly outperforming competitors like ElevenLabs Scribe v2 and Deepgram Nova-3.