Search engine users expect a concise, accurate summary in mere milliseconds, even under peak loads. To prevent generative features from collapsing under massive query volumes, engineers must overhaul the entire computational pipeline. In its recent release, the Alice AI Search team pulled back the curtain on its search model architecture and released its base weights to the open-source community.
Fast generation relies on the same infrastructure powering the Alice voice assistant. First, the system aggregates data through standard search channels before passing documents to an initial filtering layer. This intermediate stage ruthlessly strips out noise to build a compact context window, which is then fed into the generative network.
Optimizing the context window
Every unnecessary token in the context window inflates hardware spending and drives up response latency. To tackle redundant compute, Yandex launched its Agentic Search initiative six months ago, focusing on next-generation search tech for the Alice AI ecosystem.
Document filtering was assigned to a dedicated 80-million-parameter BERT-style model trained from scratch on 4 trillion tokens from search corpora. To train it to evaluate token-level relevance, the team generated annotations using open-source LLMs. While this produced a solid cold-start extractor, production workloads demanded the absolute minimum text volume required by the generative model.
"To find the optimal context density for our model, we built an algorithm based on Cross-Entropy RL"
The extractor model scores token relevance to produce a hyper-compressed context, while the generative network returns the answer and receives a reward signal. As a result, only mission-critical text chunks reach the generator, slashing latency without sacrificing response quality.
Open weights, proprietary inference
Yandex has released Alice AI-T5-35B-A0.6B Base, trained from scratch. The model combines a classic Encoder-Decoder setup with a sparse Mixture-of-Experts (MoE) architecture: out of 35 billion total parameters, only 600 million are active per token. However, the proprietary high-performance runtime remains internal; external engineers must run the model via standard Hugging Face Transformers.
For enterprise ML teams, open weights offer a solid foundation for building ultra-fast Russian-language enterprise search and RAG pipelines. Yet there is no out-of-the-box miracle: achieving search-engine-grade latency without blowing GPU budgets will require teams to build their own custom runtime optimizations and fine-tune the context-filtering pipeline in-house.