Interaction Models, a research preview by Thinking Machines, address the "collaboration bottleneck" in current human-AI interaction by proposing models that handle interactivity natively rather than through external scaffolding. The paper posits that current turn-based AI interfaces, which experience reality in a single thread, create a narrow communication channel that limits real-time collaboration, forcing humans to adapt to AI. In contrast, Interaction Models aim to enable human-AI collaboration akin to human-human interaction, characterized by copresence, contemporality, and simultaneity, allowing continuous input and output across modalities.

The core methodology revolves around a novel system architecture comprising two interconnected components: a time-aware interaction model for maintaining real-time presence and an asynchronous background model for sustained reasoning, tool use, and longer-horizon tasks. This split allows the system to combine real-time responsiveness with comprehensive intelligence.

The interaction model is designed from scratch to be natively multimodal and time-aware, capable of handling concurrent input and output streams. Key design choices include:

1. Time-aligned Micro-turns: Unlike traditional turn-based models that process complete user turns, the interaction model works with continuous input and output streams, processing and generating information in fixed-length 200ms micro-turns. This allows for continuous interleaving of input perception (e.g., audio, video frames, text) and output generation. This eliminates the need for external turn-detection harnesses (e.g., Voice Activity Detection, VAD), which are often less intelligent than the model itself and preclude proactive interactions like interjections or reactions to visual cues. The model internally represents the continuous interaction as a single interleaved token sequence, while preserving the concurrent nature for human perception.

2. Encoder-free Early Fusion: To minimize preprocessing and ensure native multimodality, the model avoids large, standalone encoders common in many omnimodal models. Instead:
   • Audio signals are taken as dMel features and transformed via a lightweight embedding layer.
   • Images are split into $40 \times 40$ patches, which are then encoded by an hMLP.
   • Text inputs are processed via standard token embeddings.
   • For audio decoding, a flow head is used.

All these components are co-trained from scratch together with the core Transformer architecture, enabling deep integration of modalities from the earliest stages of processing. The model takes any subset of text, audio, or video and predicts text and audio outputs.

3. Inference Optimization: Achieving real-time responsiveness for 200ms chunks requires low-latency inference. The authors implemented streaming sessions, where the client sends each 200ms chunk as a separate request, and the inference server appends these chunks into a persistent sequence in GPU memory. This avoids frequent memory reallocations and metadata computations, improving efficiency for small, frequent prefills and decodes. Custom kernel optimizations were also employed, such as a gather+gemv strategy for Mixture-of-Experts (MoE) kernels instead of the standard grouped gemm, tailored for bidirectional serving shapes.

4. Trainer-Sampler Alignment: To ensure training stability and facilitate debugging, bitwise trainer-sampler alignment was maintained. This involved:
   • Implementing batch-invariant kernels with minimal performance overhead.
   • Utilizing NVLS for low-latency communication kernels (e.g., All-reduce, reduce-scatter) that are deterministic and support bitwise alignment across different parallelism strategies (Sequence Parallelism, Tensor Parallelism).
   • Addressing the Split-KV challenge in attention by maintaining a consistent accumulation order between decode and prefill stages (e.g., splitting SMs to process $4096$ tokens at a time, left-aligned).

The background model is delegated tasks requiring deeper reasoning (e.g., complex planning, tool use, web browsing). The interaction model sends a rich context package (full conversation history) to the background model. Results stream back asynchronously and are interleaved into the conversation by the interaction model at appropriate moments, preventing abrupt context switches. Both models share context, allowing the user to benefit from rapid responses and the full extent of the model's intelligence.

This native interactive design unlocks several capabilities: seamless dialog management, verbal and visual interjections, simultaneous speech (e.g., live translation), time-awareness, and simultaneous tool calls, search, and generative UI. The model implicitly tracks user state (thinking, yielding, self-correcting) without separate components.

For safety, the approach emphasizes modality-appropriate refusals (using text-to-speech for colloquial refusals) and long-horizon robustness in extended conversations, utilizing automated red-teaming to generate multi-turn refusal data.

The paper introduces TML-Interaction-Small, a 276B parameter MoE model (12B active). Benchmarking includes:

• Intelligence: Audio MultiChallenge (APR) and other standard text/audio benchmarks (BigBench Audio, IFEval).
• Interactivity: FD-bench v1.5 (measures user interruption, backchannel, concurrent speech) and FD-bench v1 (simple turn-taking latency).

The results show TML-Interaction-Small achieves state-of-the-art responsiveness ($0.40s$ turn-taking latency on FD-bench v1 Audio) and strong interaction quality (77.8 on FD-bench v1.5 Average), while being competitive on intelligence benchmarks.

Furthermore, novel internal benchmarks were developed to quantify qualitative leaps in interactivity:

• TimeSpeak: Measures time awareness, e.g., "remind me to breathe in and out every 4 seconds."
• CueSpeak: Tests proactive speech based on verbal cues, e.g., "Everytime I codeswitch... give me the correct word."
• RepCount-A: Evaluates continuous visual tracking and timely counting, e.g., "count out reps for {action}."
• ProactiveVideoQA: Assesses proactive responses to visual information becoming available, e.g., "stay quiet until a new moment answers the question."
• Charades: Measures temporal action-localization, e.g., "Say 'start' when the person starts doing {action} then say 'Stop'."

TML-Interaction-Small significantly outperforms other models (including GPT-realtime-2.0 minimal) on these benchmarks, demonstrating its unique capabilities in time awareness and visual proactivity, where other models largely fail or remain silent.

Limitations include managing context in very long sessions, the demand for reliable connectivity for low-latency streaming, and ongoing work in alignment and safety for real-time interfaces. Future work involves scaling model size, improving background agents, and fostering community research into interaction models.