TLDR:
Zen, a content platform owned by VK, has blocked OpenAI’s GPTBot from crawling its site, citing the need to reduce server load due to its high-traffic recommendation system. Despite the advisory nature of the robots.txt directive, Zen prioritizes user and author experience by managing technical resources efficiently. This move comes amidst broader discussions about bot access and data use in digital platforms.
In a notable move aimed at safeguarding its digital infrastructure, the Zen content platform, under the ownership of VK, has officially restricted the GPTBot from OpenAI from accessing and collecting data on its website, dzen.ru. The decision, revealed through an update in the website’s robots.txt file, underscores a growing concern among web platforms over the impact of automated data collection on server resources and user experience.
Zen’s action to block GPTBot reflects a strategic measure to manage the platform’s substantial online traffic effectively. With a recommendation system that ranks among the largest in the country, Zen’s servers are under constant demand, handling over 150,000 requests per second. This impressive volume is a testament to the platform’s popularity, where millions of new posts, encompassing both text and video content, are generated regularly.
The VK press service elaborated on the rationale behind this decision, emphasizing the importance of optimizing technical resources for the benefit of its vast user base and content creators. The primary goal is to ensure that the platform’s infrastructure remains robust and responsive, enabling a seamless and high-quality experience for all participants in the Zen ecosystem.
It’s worth noting that the robots.txt file serves as a guideline for crawlers and bots, advising them on which parts of a site they are allowed or disallowed from accessing. However, compliance with these directives is based on good faith, as there are no technical measures in place to enforce these rules strictly. Despite this, Zen’s explicit instruction to block GPTBot highlights the platform’s commitment to controlling its digital environment and safeguarding against unnecessary strain on its systems.
This development occurs against the backdrop of OpenAI’s policy of not providing access to its products in Russia, including blocking site access for users from the Russian Federation. GPTBot is part of OpenAI’s efforts to gather information for the development of neural network products, a practice that has prompted discussions about data use and privacy across the digital landscape.
Previously, Yandex introduced a special directive for site owners to restrict YandexGPT from using their site’s information for generating quick responses, sparking varied reactions within the SEO community. Zen’s recent move adds another layer to the ongoing conversation about the balance between open data access for AI development and the need to protect digital resources and user experiences.
As the digital domain continues to evolve, the actions of platforms like Zen play a crucial role in shaping the future of online content creation, distribution, and consumption. This incident not only highlights the challenges faced by content platforms in managing high traffic and server load but also underscores the importance of strategic resource management in maintaining a quality digital experience for users and creators alike.
