View original
GitHub - lablup/mlxcel: High-performance LLM, VLM, embedding, reranking, and audio inference for Apple Silicon and NVIDIA CUDA systems.
Service

GitHub - lablup/mlxcel: High-performance LLM, VLM, embedding, reranking, and audio inference for Apple Silicon and NVIDIA CUDA systems.

lablup
2026.10.17
·GitHub·by igor
#Apple Silicon#CUDA#Inference#LLM#Rust

Key Points

  • 1mlxcel is a high-performance Rust-based runtime for Apple Silicon and NVIDIA GPUs that executes MLX SafeTensors checkpoints natively without requiring a Python environment.
  • 2The software provides a versatile infrastructure for production-grade LLM, VLM, embedding, and audio serving, featuring continuous batching, speculative decoding, and native llama-server compatibility.
  • 3It supports advanced operational controls like router-based model management, live LoRA adaptation, and a bundled WebUI, while offering optimized performance for both dense and mixture-of-experts architectures.

mlxcel is a high-performance, native inference runtime designed for Apple Silicon, NVIDIA CUDA, and experimental AMD ROCm (via HIP) architectures. It acts as both a CLI tool and a production-grade model server, supporting text, multimodal (VLM), embedding, reranking, and audio workloads. A defining characteristic of mlxcel is its architectural design: it executes MLX SafeTensors checkpoints directly through native C++ bindings, bypassing the need for Python interpreters or checkpoint conversion in the production request path.

Core Methodology and Technical Infrastructure

The runtime is built in Rust and serves as a comprehensive engine for inference, utilizing several sophisticated techniques to maximize hardware utilization:

  • Native Engine and Batching: The engine, mlxcel_core::engine::DirectEngine, supports continuous batching, prompt-prefix caching, and automatic prefix caching by default. This minimizes latency and maximizes throughput for multi-user serving environments.
  • Zero-Overhead Serving: The server is compliant with a b10621 manifest, ensuring compatibility with llama-server routes while providing native OpenAI, Anthropic, and Vertex AI-compatible endpoints. It facilitates multi-model routing, live LoRA adapter switching (either fused or unfused), and dynamic memory management with LRU eviction.
  • Memory Management and Surgery: The platform allows for "model surgery" via YAML-defined load-time edits (e.g., scale, add, prune, replace, interpolate). Memory usage is managed through a pre-flight inspect utility that calculates exact byte requirements for weight, KV cache, and activation buffers, preventing runtime OOM failures.
  • Advanced Sampling and Speculative Decoding: mlxcel implements adaptive speculative decoding (e.g., MTP, DFlash) that measures greedy exactness before offloading work to faster paths. The adaptive policy can be queried via internal API endpoints, and sampling controls are provided at both the per-request and global configuration levels.
  • Multimodal and Audio Integration: The runtime natively handles complex input types, including video tiling for VLMs and audio transcription (WAV, MP3, FLAC) where the model architecture allows. It features a full-duplex WebSocket interface (/v1/realtime) for low-latency speech-to-speech tasks like Nemotron VoiceChat.

Technical Performance Enhancements

Performance is optimized through low-level kernel optimizations and specialized precision handling:
  • Precision Management: The system avoids widening half-precision hidden states (f16f16) to f32f32 during activation, preventing unnecessary matmul promotion that historically increased latent overhead and induced NaN results during bfloat16 maximum operations.
  • Kernel Fusion: For AMD ROCm/HIP, the system utilizes fused kernels—such as BitLinear, SSM update steps, and fused add-RMSNorm + RoPE + append—to ensure performance parity with native Metal kernels.
  • Speculative Decoding Policy: The system enforces strict exactness and profitability metrics. The speedup factor, Speedup=Tokens/sspecTokens/sbase\text{Speedup} = \frac{\text{Tokens/s}_{\text{spec}}}{\text{Tokens/s}_{\text{base}}}, is dynamic and dependent on the hardware-specific verify-width and the prompt complexity.

The platform provides a comprehensive ecosystem for model deployment, including an in-binary bundled WebUI, and allows for Python-based orchestration that leverages the native server’s performance without requiring local interpreter-heavy model loading logic.