tencent/Hy4-flash · Hugging Face
News

tencent/Hy4-flash · Hugging Face

2026.11.22
·Hugging Face·by Mineru
#AI#DL#LLM#ML#RAG

Key Points

  • 1Hy4 flash is a cost-effective 220B parameter multimodal model that features a mixture-of-experts architecture, an integrated 0.85B parameter DFly2 drafter for speculative decoding, and a 1M token context window.
  • 2The model introduces architectural innovations such as hybrid attention, latent MoE, and N-gram embedding to significantly reduce KV cache requirements and operational costs compared to previous versions.
  • 3Performance evaluations demonstrate that Hy4 flash provides high intelligence for everyday productivity tasks while achieving superior cost efficiency and faster turnaround times than larger, more resource-intensive models.

Hy4 flash is an advanced multimodal Mixture-of-Experts (MoE) model developed by Tencent, designed for high-efficiency text and image processing. It features 220B total parameters, including 64B dedicated to Engram components, while maintaining a lean 12B active parameters per token. The model is optimized for a 1M token context window and demonstrates a significant shift toward cost-effective, high-intelligence productivity tasks.

Core Architecture

The language backbone consists of 42 layers, comprising 1 dense feed-forward layer and 41 MoE layers, linked by identity Hyper-Connections (iHC) to facilitate inter-layer information flow. Key architectural innovations include:

  • Hybrid Attention: The model interleaves sparse attention (IndexPool) and sliding-window attention (SWA) in a 1:3 ratio, optimizing memory and computational overhead for long sequences.
  • LatentMoE: Routed experts operate within a lower-dimensional latent space. Each token activates 16 of 512 routed experts plus one shared expert, using a latent dimension of 1,536.
  • N-gram Embedding (Engram): The backbone is augmented with learned bigram and trigram representations. These are retrieved via multi-head hash lookups (HH heads per order), providing 63.5B specialized parameters stored in hash tables, which effectively encode frequent local sequences to improve language modeling performance.
  • DFly2 Speculative Decoding: A 5-layer, 0.85B parameter drafter is integrated to accelerate inference. It utilizes the target model's KV cache (KVshareKV_{share}) and compressed indices (IndexshareIndex_{share}), ensuring constant-cost speculative generation regardless of context length without requiring additional memory overhead for drafting.

Multimodal and Optimization Features

Hy4 flash utilizes a vision encoder adapted from SigLIP2-So400m-Patch16-Res512, connected via an MLP projector. The model employs Multi-head Latent Attention (MLA) with interleaved Multimodal Rotary Positional Embeddings (MRoPE).

To achieve its efficiency, the model relies on several optimization techniques:

  1. Memory Compression: The architectural improvements reduce the KV cache size by a factor of 24 compared to the Hy3 version.
  2. Post-Training Efficiency: The model undergoes self-distillation from the larger Hy4 architecture, paired with reinforcement learning that reduces the average end-to-end output token count by 50%.
  3. Inference Serving: Through the use of megakernels, 4-bit quantization, and the DFly2 drafter, the model achieves decoding speeds exceeding 150 tokens/s.

Performance and Utility

Hy4 flash is benchmarked against public and internal metrics, such as the "WorkIndex," which evaluates document processing, slide creation, and coding tasks. It achieves a 42.1 score on this internal index, outperforming the Hy4 preview model (39.7) while maintaining significantly lower operational costs (1/13 of the cost per task compared to the Hy4 preview). User blind evaluations confirm its high utility, matching the satisfaction levels of Claude Haiku while providing faster completion times and reduced costs, making it the recommended choice for daily productivity, while the larger Hy4 model remains the preferred solution for high-complexity software engineering and long-horizon tasks.