tencent/Hy4-flash · Hugging Face
Key Points
- 1Hy4 flash is a cost-effective 220B parameter multimodal model that features a mixture-of-experts architecture, an integrated 0.85B parameter DFly2 drafter for speculative decoding, and a 1M token context window.
- 2The model introduces architectural innovations such as hybrid attention, latent MoE, and N-gram embedding to significantly reduce KV cache requirements and operational costs compared to previous versions.
- 3Performance evaluations demonstrate that Hy4 flash provides high intelligence for everyday productivity tasks while achieving superior cost efficiency and faster turnaround times than larger, more resource-intensive models.
Hy4 flash is an advanced multimodal Mixture-of-Experts (MoE) model developed by Tencent, designed for high-efficiency text and image processing. It features 220B total parameters, including 64B dedicated to Engram components, while maintaining a lean 12B active parameters per token. The model is optimized for a 1M token context window and demonstrates a significant shift toward cost-effective, high-intelligence productivity tasks.
Core Architecture
The language backbone consists of 42 layers, comprising 1 dense feed-forward layer and 41 MoE layers, linked by identity Hyper-Connections (iHC) to facilitate inter-layer information flow. Key architectural innovations include:- Hybrid Attention: The model interleaves sparse attention (IndexPool) and sliding-window attention (SWA) in a 1:3 ratio, optimizing memory and computational overhead for long sequences.
- LatentMoE: Routed experts operate within a lower-dimensional latent space. Each token activates 16 of 512 routed experts plus one shared expert, using a latent dimension of 1,536.
- N-gram Embedding (Engram): The backbone is augmented with learned bigram and trigram representations. These are retrieved via multi-head hash lookups ( heads per order), providing 63.5B specialized parameters stored in hash tables, which effectively encode frequent local sequences to improve language modeling performance.
- DFly2 Speculative Decoding: A 5-layer, 0.85B parameter drafter is integrated to accelerate inference. It utilizes the target model's KV cache () and compressed indices (), ensuring constant-cost speculative generation regardless of context length without requiring additional memory overhead for drafting.
Multimodal and Optimization Features
Hy4 flash utilizes a vision encoder adapted from SigLIP2-So400m-Patch16-Res512, connected via an MLP projector. The model employs Multi-head Latent Attention (MLA) with interleaved Multimodal Rotary Positional Embeddings (MRoPE).To achieve its efficiency, the model relies on several optimization techniques:
- Memory Compression: The architectural improvements reduce the KV cache size by a factor of 24 compared to the Hy3 version.
- Post-Training Efficiency: The model undergoes self-distillation from the larger Hy4 architecture, paired with reinforcement learning that reduces the average end-to-end output token count by 50%.
- Inference Serving: Through the use of megakernels, 4-bit quantization, and the DFly2 drafter, the model achieves decoding speeds exceeding 150 tokens/s.