Yandex has open-sourced the streaming engine that feeds its ad models, and the two announcements don’t quite agree on how fast it is. The English press release published on 6 October (Russian version here) says YTsaurus Flow handles more than 100 GB of data per second in Yandex Ads alone. It also says the delay in preparing fresh data to retrain some ad-selection models, which “could take more than 10 hours”, has been “almost entirely eliminated”.
The engineering write-up is more modest, and more useful. In Yandex’s technical post on Medium (there’s a Russian version on Habr), Egor Khairullin says moving one production pipeline from batch to Flow cut end-to-end latency for model retraining from around 10–20 hours to roughly two. One of those two hours is a deliberate wait to see whether an impression turns into a click.
Going from a day to two hours is a big improvement. It just isn’t “almost entirely eliminated”. Advertisers should plan around the engineers’ figure, not the press office’s.
What Yandex actually released
Flow is a stateful stream-processing framework built on YTsaurus, the big-data platform Yandex open-sourced in March 2023. According to the Medium post and the project documentation:
- It processes events exactly once by default, so none are lost or double-counted. Pipelines can relax that to at-least-once or at-most-once to save resources.
- The core is written in C++. Business logic can be written in C++, Python, Go or Kotlin/Java, and simple pipelines can be described entirely in YQL, Yandex’s SQL dialect.
- It’s designed for multi-data-centre deployment. The typical setup uses three sites so processing carries on if one goes down.
- It’s licensed under Apache 2.0, so commercial use is allowed.
The press release names three internal users. Yandex Market uses Flow to collect statistics faster so sellers can react to demand. Antifraud systems use it to detect malicious activity. Yandex Ads uses it to pick more relevant ads. The events it lists are clicks, impressions, likes and orders.
The ad retraining case, in plain English
The Medium post describes the old setup honestly. Event logs and ML features were cut into one-hour tables. To process hour X, the pipeline had to wait for hour X+1, because a click can land in the hour after its impression. It then ran heavy MapReduce joins over features that came to “hundreds of terabytes per hour in compressed form”, even though only a small fraction of them was needed. Yandex says those joins, not the waiting, caused most of the 10–20 hour lag.
Flow reads the event stream straight away and groups each impression with its click by key. It reads the much larger feature stream one hour later, and by then it knows which records are worth keeping. Yandex doesn’t disclose what that did to ad performance: “the precise impact on the advertising system isn’t public.”
Why Direct advertisers should care
None of this is a Yandex Direct product launch, and Yandex doesn’t say Search ranking uses Flow, so don’t read that into it. But it does change one practical thing for anyone spending on Yandex. If the models behind ad selection learn from interactions in about two hours instead of about a day, then whatever you feed them reaches them faster. That includes the mistakes.
Conversion goals that fire twice, Metrica tags that drop on a new checkout, or a bot-inflated landing page used to take most of a day to reach the models. Now they could be working their way in by lunchtime. A faster feedback loop rewards clean tracking and punishes sloppy tracking more quickly. The antifraud mention matters too: invalid clicks should, in theory, be caught closer to when they happen.
For Market sellers, the claim is faster statistics on changes in demand. That is worth testing against your own dashboards before you believe it.
Our take
This is the same Yandex pattern we saw with hybrid BM25-plus-vector search in YDB and the open AliceAI-Foundation base model. Yandex is open-sourcing the plumbing behind its own products and offering it to Russian businesses as an alternative to Western stacks. As infrastructure news, it’s solid. As marketing copy, “almost entirely eliminated” goes further than Yandex’s own engineering post, which says about two hours, one of them on purpose. When a press release and an engineering blog disagree, trust the one with the diagram. More from Moscow in our Yandex News archive.