LiteRT is a Google-developed on-device framework designed for high-performance machine learning (ML) and generative artificial intelligence (GenAI) deployment on edge platforms, serving as a successor and evolution of TensorFlow Lite. It focuses on efficient model conversion, runtime execution, and optimization.

The core methodology of LiteRT revolves around maximizing on-device ML inference performance through specialized APIs and accelerated execution across diverse hardware. Key features include:

• New LiteRT Compiled Model API: This API streamlines development by offering automated accelerator selection, eliminating the need for explicit delegate creation. It supports true asynchronous execution, improving overall inference speed. Furthermore, it facilitates efficient I/O buffer handling, minimizing data transfer overheads between model components and hardware accelerators. This API aims to simplify the developer experience while enhancing performance.
• Unified NPU Acceleration: LiteRT provides seamless and consistent access to Neural Processing Units (NPUs) from major chipset providers (e.g., Google Tensor, Qualcomm, MediaTek S.LSI, Intel). This unified approach ensures a standardized developer experience across different NPU architectures, abstracting hardware-specific complexities. The NPU acceleration, previously in early access, is now generally available.
• Best-in-class GPU Performance: The framework leverages state-of-the-art GPU acceleration for on-device ML. A significant advancement is the new buffer interoperability, which enables zero-copy operations and minimizes latency across various GPU buffer types, optimizing data flow between the CPU and GPU.
• Superior Generative AI Inference: LiteRT is specifically optimized for GenAI models, offering simplified integration and high performance for large language models (LLMs) and diffusion models on resource-constrained edge devices.

LiteRT supports a wide range of platforms:

• Android: CPU, OpenCL/OpenGL (GPU), Google Tensor, Qualcomm, MediaTek S.LSI (NPU).
• iOS: CPU, Metal (GPU), Apple Neural Engine (ANE) (NPU).
• Linux: CPU, WebGPU (GPU).
• macOS: CPU, WebGPU/Metal (GPU), Apple Neural Engine (ANE) (NPU).
• Windows: CPU, WebGPU (GPU), Intel (NPU).
• Web: CPU, WebGPU (GPU).
• IoT: CPU, WebGPU (GPU), Broadcom, Raspberry Pi (NPU, coming soon).

Building LiteRT from source typically involves a Docker-based workflow. Users start a Docker daemon and execute build_with_docker.sh from the docker_build/ directory. This script automatically sets up a Linux Docker image, enabling the compilation of artifacts for Linux and Android (via cross-compilation). Further build instructions for runtime libraries are available for CMake and Bazel within the Docker container environment.

LiteRT offers tailored paths for different developer goals:

• PyTorch Model Conversion:
  • For classic ML models, the LiteRT Torch Converter transforms PyTorch models into the .tflite format. The AI Edge Quantizer can then optimize these models for performance under resource constraints, followed by deployment via the standard LiteRT runtime.
  • For LLMs, the LiteRT Generative Torch API is used to reauthor and convert PyTorch LLMs into an optimized Apache format, which are then deployed using LiteRT LM, a specialized library for efficient LLM execution on edge platforms.
• New to On-device ML:
  • Beginners can follow step-by-step instructions in Android Studio to create real-time segmentation applications utilizing CPU, GPU, or NPU inference.
  • Experienced developers can begin with the comprehensive "Get Started" guide, acquire a pre-trained .tflite model (e.g., from Kaggle Models), and integrate it into Android or iOS applications using the standard LiteRT runtime.
• Maximizing Performance: To accelerate existing models, developers can leverage the LiteRT API for hardware acceleration. For Generative AI models, LiteRT LM provides specialized solutions for performance optimization.
• Working with Generative AI: For deploying LLMs or diffusion models on mobile devices, the recommended path is LiteRT LM. This involves focusing on model quantization and specific architectural optimizations tailored for large models.

The roadmap for LiteRT includes expanding hardware acceleration support (especially NPUs), introducing new optimizations for generative AI models, improving developer tools (debugging, profiling), and enhancing/exploring platform support.

LiteRT is part of a broader ecosystem of Google tools for on-device ML, including:

• LiteRT Samples: A collection of sample applications demonstrating LiteRT usage.
• LiteRT Torch Converter: A tool to convert PyTorch models to the LiteRT .tflite format.
• LiteRT Generative Torch API: A library for reauthoring LLMs to facilitate efficient conversion and on-device inference.
• LiteRT-LM: A library built on LiteRT specifically for efficiently running LLMs across edge platforms.
• XNNPACK: A highly optimized library of neural network inference operators that provides high-performance CPU acceleration for LiteRT on ARM, x86, and WebAssembly architectures.
• MediaPipe: A framework for building cross-platform, customizable ML solutions for live and streaming media.

LiteRT is licensed under the Apache-2.0 License.