MobileSAM2 is a lightweight adaptation of the SAM2 video foundation model designed for resource-constrained edge devices. Addressing the challenges of deploying heavy segmentation models on mobile platforms, the authors introduce a novel distillation framework, Hypergraphical Knowledge Distill (HyperKD), and a family of efficient model architectures, MobileSAM2.

The methodology relies on the premise that SAM2’s performance stems from two primary latent knowledge structures: temporal correlations across video frames and multi-granularity hierarchies of visual concepts. Because these are implicitly encoded within the teacher model’s parameters, the authors propose HyperKD to explicitly model them using hypergraphs.

Hypergraphical Knowledge Distill (HyperKD)

HyperKD consists of two main components:

1. Temporal HyperKD: This module extracts temporal knowledge by constructing patch-wise and instance-wise hypergraphs. For a video embedding $zI_t$, nodes are established via pooling based on mask predictions. Hyperedges are formed using an $\epsilon$-ball distance threshold, where a hyperedge connects all vertices $u$ within a distance $\epsilon$ of a central vertex $v$. By minimizing the difference between the incidence matrices $H_t$ and $H_s$ of the teacher and student models—formulated as $L_{THKD} = \|H^P a_t - H^P a_s\|_F + \|H^{Ins}_t - H^{Ins}_s\|_F$—the student learns to replicate the temporal consistency of the teacher.
2. Granularity HyperKD: This module captures orthogonal multi-granularity knowledge (e.g., objects vs. subparts). Nodes $v \in V^{Gran}_t$ are initialized by masked average pooling of frame embeddings. The resulting hypergraph is encapsulated into the student by minimizing the Frobenius norm of the difference between teacher and student incidence matrices: $L_{GHKD} = \|H^{Gran}_t - H^{Gran}_s\|_F$.

The total objective function is defined as:
$Loss = \|zI_t - zI_s\|_1 + \alpha \cdot L_{THKD} + \beta \cdot L_{GHKD}$

MobileSAM2 Architecture

To achieve efficiency, the authors employ a progressive model contraction strategy on the Hiera vision transformer backbone. By searching a parameter space of contraction factors—including embedding dimensions ($\Gamma_{Emb}$), block depth ($\Gamma_{Blk}$), window sizes ($\Gamma_{WinSiz}$), and attention head expansion ratios ($\Gamma_{HExp}$)—they identified three distinct model configurations (5M, 10M, and 23M parameters). These models retain the memory module, prompt encoder, and mask decoder of the original SAM2, replacing only the heavy image encoder.

Experimental Impact

Extensive benchmarking on VOS datasets (MOSE, DAVIS 2017, LVOS, and SA-V) demonstrates that MobileSAM2 significantly outperforms traditional distillation baselines like FitNet, CIRKD, and FAKD. The approach succeeds in balancing high-fidelity temporal and multi-granularity understanding with the computational requirements of mobile hardware, facilitating real-time spatial intelligence on constrained devices.