KTransformers is a research project focused on achieving efficient inference and fine-tuning of large language models (LLMs) through CPU-GPU heterogeneous computing. The project is structured into two primary modules: kt-kernel for high-performance inference and kt-sft for fine-tuning.

The core methodology of KTransformers revolves around leveraging the strengths of both CPUs and GPUs to handle the computational and memory demands of large-scale LLMs, especially Mixture-of-Experts (MoE) models. This involves strategically placing different parts of the model or different computational tasks on the most suitable hardware.

kt-kernel - High-Performance Inference Kernels:
This module provides CPU-optimized kernel operations for heterogeneous LLM inference. Its technical contributions and features include:

1. AMX/AVX Acceleration: It implements optimized kernels specifically designed to exploit Intel AMX (Advanced Matrix Extensions) and AVX512/AVX2 instruction sets. These optimizations are crucial for accelerating matrix multiplication and other core operations for INT4/INT8 quantized inference on CPUs, significantly boosting throughput for integer precision models.
2. MoE Optimization with NUMA-awareness: For Mixture-of-Experts models, kt-kernel provides efficient inference mechanisms. This includes NUMA (Non-Uniform Memory Access)-aware memory management, which optimizes data placement and access patterns to minimize latency when different experts or their states are distributed across CPU memory nodes. This enables heterogeneous expert placement, where frequently accessed ("hot") experts might reside on the GPU while less frequently accessed ("cold") experts are kept on the CPU to save GPU memory.
3. Quantization Support: It supports CPU-side INT4/INT8 quantized weights for memory efficiency and faster computation on the CPU. Additionally, it integrates with GPU-side GPTQ quantization techniques, allowing for a flexible quantization strategy across the heterogeneous system.
4. Easy Integration: It offers a clean Python API, enabling integration with frameworks like SGLang for production serving scenarios, facilitating CPU-GPU hybrid inference for large MoE models.

Performance examples illustrate its capability, such as achieving 227.85 tokens/s total throughput (87.58 tokens/s output throughput with 8-way concurrency) for DeepSeek-R1-0528 (FP8) utilizing 8x L20 GPUs and a Xeon Gold 6454S CPU.

kt-sft - Fine-Tuning Framework:
This module focuses on resource-efficient fine-tuning, particularly for ultra-large MoE models, by integrating with the popular LLaMA-Factory framework. Its key technical aspects include:

5. Resource Efficiency: kt-sft allows for the fine-tuning of extremely large models, such as the 671B DeepSeek-V3, with significantly reduced GPU memory requirements (e.g., 70GB multi-GPU VRAM) by offloading a substantial portion of the model's parameters or intermediate states to system RAM (e.g., 1.3TB RAM). This heterogeneous memory management is critical for training models that would otherwise exceed typical GPU memory capacities.
6. LoRA Support with Heterogeneous Acceleration: It supports full Low-Rank Adaptation (LoRA) fine-tuning, a parameter-efficient fine-tuning technique. The "heterogeneous acceleration" implies that the LoRA adapters themselves, or the underlying base model computations, might be distributed or optimized across CPU and GPU resources to maximize efficiency during the training process.
7. LLaMA-Factory Integration: The seamless integration with LLaMA-Factory provides a robust and familiar environment for users to conduct fine-tuning, leveraging KTransformers' backend optimizations without requiring extensive changes to their existing LLaMA-Factory workflows.
8. Production Readiness: The framework supports features essential for production environments, including chat, batch inference, and metrics evaluation, demonstrating its applicability beyond research prototyping.

Performance benchmarks highlight its efficiency, such as fine-tuning DeepSeek-V3 (671B) using LoRA and AMX optimizations at approximately 40 tokens/s with 70GB GPU memory, and DeepSeek-V2-Lite (14B) at around 530 tokens/s with only 6GB GPU memory.

In summary, KTransformers provides a comprehensive framework for optimizing LLM workflows by intelligently partitioning and executing tasks across CPU and GPU hardware, with specialized optimizations for quantization, MoE models, and memory-constrained fine-tuning.