MobileSAM2 is a lightweight model family designed to adapt the powerful video segmentation capabilities of SAM2 for resource-constrained edge devices. Recognizing that SAM2’s success stems from its ability to capture large-scale temporal correlations and multi-granular visual hierarchies, the authors introduce Hypergraphical Knowledge Distillation (HyperKD) to transfer this complex knowledge into compact student architectures.

Core Methodology: HyperKD

HyperKD overcomes the limitation of traditional distillation by explicitly modeling the implicit knowledge of the teacher (SAM2) through hypergraphs, which can represent high-order semantic relationships that standard pairwise graphs cannot capture.

1. Temporal HyperKD: This component addresses the need for robust segmentation across video frames. It constructs two types of hypergraphs: Patch-wise and Instance-wise. By treating embeddings as nodes, it forms hyperedges using an $\epsilon$-ball distance threshold, linking nodes that represent related spatial-temporal concepts across frames. The student model is trained to match the teacher's incidence matrix $H$, minimizing:

$L_{THKD} = \|H^{P a}_{t} - H^{P a}_{s} \|_{F} + \|H^{Ins}_{t} - H^{Ins}_{s}\|_{F}$
where $H_{t}$ and $H_{s}$ are the incidence matrices for teacher and student models.

2. Granularity HyperKD: This component captures hierarchical object-part-subpart relationships. Nodes represent segments at different granularity levels, and hyperedges connect these levels to enforce consistent coarse-to-fine understanding. It minimizes:

$L_{GHKD} = \|H^{Gran}_{t} - H^{Gran}_{s}\|_{F}$

The total training objective combines these distillation losses with standard feature-based alignment:
$Loss = \|zI_{t} - zI_{s} \|_{1} + \alpha \cdot L_{THKD} + \beta \cdot L_{GHKD}$

Architecture: Progressive Contraction

To achieve efficiency, the authors introduce a new lightweight model family by searching for optimal architectures using a progressive model contraction strategy. Starting from a Hiera-based backbone, they scale down critical factors:

• $\Gamma_{Emb}$: Embedding dimension.
• $\Gamma_{Blk}$: Depth (number of blocks per stage).
• $\Gamma_{WinSiz}$: Window size for attention.
• $\Gamma_{HExp}$: Expansion ratio of attention heads.

By identifying the best-performing architectures within this search space, the authors developed models ranging from 5.8M to 23.7M parameters. Experimental results on datasets including MOSE, DAVIS 2017, and SA-V demonstrate that MobileSAM2 significantly outperforms baseline distillation approaches (like FitNet and FAKD) in terms of Jaccard and F-measure metrics, successfully enabling high-performance, real-time spatial intelligence on mobile and embedded hardware.