The Institute of Foundation Models (IFM), the research division of MBZUAI, has rolled out a suite of six open-source K2 Horizon models ranging from 0.9B to an impressive 375B parameters. However, the real engineering breakthrough lies not in raw scale, but in its flagship mid-sized model: K2-Horizon-MoVA-36B-A4B. Out of its full 36 billion parameters, it activates just 4 billion per generated token.
Dual sparsity
Standard Mixture-of-Experts (MoE) architectures traditionally sparsify only the feed-forward network (FFN) blocks, leaving the attention mechanism dense. As a result, the quadratic cost of attention computation continues to burn compute cycles as context windows grow. In K2-Horizon-MoVA-36B-A4B, the engineers introduced a second layer of sparsity directly into the attention mechanism, unveiling the Mixture-of-Value Attention (MoVA) concept.
The separation logic is strictly pragmatic. Query and key projections remain dense to preserve the dot-product structure and ensure accurate semantic mapping. Meanwhile, value projections are routed through an independent pool of experts managed by a per-token top-k router.
K2-Horizon-MoVA-36B-A4B simultaneously deploys 100 FFN experts via a top-8+1 scheme and 64 value experts in attention using a top-4 routing setup, activating a combined total of only 4 billion parameters per token.
MoVA attention is driven by a sigmoid router with post-gate softplus activation instead of the default softmax top-k. Crucially, sparsity in the value space integrates natively with FlashAttention and Grouped-Query Attention (GQA), preserving the compute graph without requiring runtime workarounds.
Infrastructure trade-offs
Does the MoVA architecture allow teams to slash GPU server procurement? There are no miracles here: VRAM requirements do not drop by a single gigabyte. Physical memory demands are still dictated by the total model footprint, meaning the full 36 billion parameters must reside entirely in GPU memory.
The real gain is pure computational density. A drastic reduction in FLOPs per token delivers a massive boost in generation speed, driving down the total cost of ownership for enterprise inference—particularly across long-context workloads. Notably, IFM opted out of publishing a full peer-reviewed preprint, releasing only the model weights and a brief technical blog post.
The MoVA architecture demonstrates how to optimize latency without sacrificing model capacity by extending sparsity into attention blocks. For business leaders, the takeaway is strictly utilitarian: computational savings do not eliminate heavy VRAM requirements, and the stability of dual-routing architectures beyond 36B parameters remains to be proven in production.