Media Groups Restrict AI Access to Content for Experimentation

Several prominent media portals have initiated an experimental ban on AI-driven search robots from Yandex and OpenAI, including those used for neural networks, to evaluate the impact on web traffic and content utilization.

Posted by By Dan Taylor 2 Min Read

Several prominent media portals have initiated an experimental ban on AI-driven search robots from Yandex and OpenAI, including those used for neural networks, to evaluate the impact on web traffic and content utilization.

In a notable move within the digital media landscape, several major media portals managed by Skhulev Media Holding, including E1.ru, NGS.ru, and Fontanka, have implemented a ban on the use of their content by artificial intelligence search robots. This restriction specifically targets the YandexAdditional search robot and OpenAI’s robots, GPTBot and ChatGPT-User, prohibiting them from utilizing the portals’ content to power neural networks.

The decision to implement these bans was disclosed through updates to the robots.txt files of the affected media sites. These files are used to regulate how search engines and other web crawlers can interact with site content. Skhulev Media described the initiative as an experimental measure to assess the potential impacts on site traffic and the broader implications of AI integration.

The move comes amid broader considerations within the media industry about the advantages and challenges associated with AI technologies. Experts suggest that the decision by Skhulev Media reflects a cautious approach by site owners, who are currently evaluating the benefits of AI-driven services against the risks of losing direct visitor traffic to their websites.

Yandex has responded to these developments by noting that the exclusion of its search robots from accessing certain content is part of an A/B testing approach by publishers. This testing allows them to analyze different strategies for engaging with AI technologies without fully committing to a permanent change.

This experiment is particularly relevant in light of Yandex’s recent rollout of its Neuro service, which enhances search capabilities by integrating large generative models. These models are designed to sift through extensive online information, extract essential details, and present users with synthesized responses based on the content they access.

As this situation unfolds, it highlights the ongoing dialogue and experimentation within the media and tech industries as they navigate the evolving landscape of AI and its integration into traditional media practices.

Dan Taylor
View More Posts
Dan Taylor is the founder and editor of International Search Insights (ISI), the publication behind ReadIsi, and a Partner and Head of Innovation (Search & AI) at SALT.agency. He has worked in search for more than a decade, in-house and agency-side, and is regarded as one of the West’s leading voices on search engines beyond Google, including Yandex, Baidu, Naver and Cốc Cốc. On ReadIsi, Dan covers the engines and AI answer platforms that English-language coverage tends to miss: Baidu and ERNIE, Yandex and Alice AI, Naver, Cốc Cốc, Shenma, Sogou, WeChat and Weibo search, plus DeepSeek and Alibaba’s Qwen. He also keeps dated records of documented changes, such as the Yandex, Baidu and Naver algorithm update histories and the DeepSeek, ERNIE and Qwen model update histories. His approach is to take learnings from global search engines and apply logic with a high evidence threshold. A claim needs a source, patterns seen in one market are treated as a hypothesis until they are checked in another, and where the evidence is thin he says so. For brands, that means advice built on what can be shown, with measurement that separates being cited from being chosen. Dan is credited with coining the term “edge SEO” and won the inaugural TechSEO Boost research competition in 2018 for his work on CDN and edge changes for SEO. In 2026 he was credited with coining “Infinite Tail” to describe how keyword research is changing with AI search. He is the author of AI Search Marketing For The Travel Sector, writes for Search Engine Journal, Search Engine Land and Travel Marketing Guy, and has advised technology, SaaS, travel and enterprise ecommerce brands, including Cloudflare, GitLab and Airbnb. He has given multiple talks and panels at the International Search Summit in Barcelona, and has also spoken at Optimization Moscow, brightonSEO, Google Search Central Live Deep Dive Asia in Bangkok, Athens SEO and the Asia SEO Virtual Summit 2026, where his talk looked at how Baidu, Naver, DeepSeek, Tencent, Cốc Cốc and Yandex built their own AI answers.
Founder & Editor