A newly optimized NInfer inference build has compressed the multimodal 143.84 GB Ornith-1.5-35B-A3B mixture-of-experts model down to a 21.22 GiB artifact, delivering sustained generation speeds north of 250 tokens per second on a single consumer-grade Nvidia GeForce RTX 5090 GPU across 5–8k prefill windows.
Raw quantization alone rarely yields high-throughput speculative decoding. The baseline release from ornith-ai shipped with an untrained Multi-Token Prediction (MTP) head that verified a meager 21.8% of speculative draft tokens under load. Engineers swapped this out for a distilled 15-tensor MTP module sourced from Shisa AI (trained against Qwen3.6-35B-A3B), lifting draft acceptance to 39.0%—a 1.8× efficiency bump that pushed peak single-stream decode throughput up to 336.5 tokens per second.
The artifact balances a 256-expert MoE architecture via mixed groupwise 4-bit, 5-bit, 6-bit, and 8-bit integer quantization, retaining critical attention projections and embeddings in BF16 precision. For engineering teams evaluating edge deployments, matching proprietary cloud API token speeds on a single off-the-shelf desktop card radically reduces the total cost of ownership for private, high-throughput multimodal agent pipelines.