YisouSpider and the Robots.txt Trap: Why Shenma Never Crawls Half of International Sites

Most “we tried Shenma and nothing happened” post-mortems are not ranking mysteries. They are crawl-access mistakes. Shenma’s spider is YisouSpider (you will also see it logged as yisouspider). If your robots.txt allow-lists Baiduspider and Googlebot and then Disallow: / for everyone else, you have locked the Alibaba mobile engine out before a single URL is submitted. A September 2026 practitioner note on A5站长网 put this bluntly, and it matches what we still see on international China builds.

This is not a repeat of the onboarding manual in our Shenma for business guide. That piece is the operating checklist. This one is the failure mode: discovery never starts, so ads, Brand Zone and Taobao packs are irrelevant arguments.

What to allow, and where

The webmaster door remains zhanzhang.sm.cn. Verification is HTML file or DNS TXT; DNS can take anywhere from about ten minutes to a couple of hours, and repeating the TXT record because you were impatient just creates a mess. Until verification sticks, resource submission is a locked room. Official help still lives at the Shenma help centre.

For robots, be explicit rather than clever:

  • User-agent: YisouSpider with Allow: / on anything you actually want in the mobile index.
  • A Sitemap: line at the root, not buried in a subdirectory file Shenma will not fetch.
  • No UA-sniffing that 403s unknown bots. Shenma will not argue with your WAF.

Sitemap rules cited by Chinese webmasters line up with the usual protocol ceiling: 50,000 URLs and about 10MB per file, UTF-8, then an index file if you overflow. Shenma’s manual URL paste is commonly described as 1,000 URLs a time — fine for a repair queue, useless as your only discovery system. Sogou’s dead-link file wants plain text, one URL a line; do not assume Shenma will politely parse a Google-shaped XML dead-link feed either. Check the platform’s sitemap status page for “fetched” versus “skipped as low quality” instead of staring at a site: query for a fortnight.

Mobile adaptation is a crawl issue, not a design slogan

Shenma still prefers pages that are actually mobile. If you run a separate m. host, the spider has to see a clean hop and a 200 on the mobile URL. A PC page that mentions a mobile twin only in a tag your renderer never emits is how you end up indexed on the wrong host. Our older mobile optimisation note still applies; pair it with log splits so YisouSpider is not averaged into “all bots”.

Opinion: international teams over-invest in Baidu rank tracking and under-invest in whether Shenma is allowed to fetch the HTML. A whitelist robots file is not a security posture. It is an accidental boycott of every Chinese crawler you forgot to name. Fix the UA, verify the property, submit a sane sitemap, then argue about content. Until YisouSpider returns 200s in the logs, you do not have a Shenma SEO programme. You have a rumour.

Dan Taylor is the founder and editor of International Search Insights (ISI), the publication behind ReadIsi, and a Partner and Head of Innovation (Search & AI) at SALT.agency. He has worked in search for more than a decade, in-house and agency-side, and is regarded as one of the West’s leading voices on search engines beyond Google, including Yandex, Baidu, Naver and Cốc Cốc. On ReadIsi, Dan covers the engines and AI answer platforms that English-language coverage tends to miss: Baidu and ERNIE, Yandex and Alice AI, Naver, Cốc Cốc, Shenma, Sogou, WeChat and Weibo search, plus DeepSeek and Alibaba’s Qwen. He also keeps dated records of documented changes, such as the Yandex, Baidu and Naver algorithm update histories and the DeepSeek, ERNIE and Qwen model update histories. His approach is to take learnings from global search engines and apply logic with a high evidence threshold. A claim needs a source, patterns seen in one market are treated as a hypothesis until they are checked in another, and where the evidence is thin he says so. For brands, that means advice built on what can be shown, with measurement that separates being cited from being chosen. Dan is credited with coining the term “edge SEO” and won the inaugural TechSEO Boost research competition in 2018 for his work on CDN and edge changes for SEO. In 2026 he was credited with coining “Infinite Tail” to describe how keyword research is changing with AI search. He is the author of AI Search Marketing For The Travel Sector, writes for Search Engine Journal, Search Engine Land and Travel Marketing Guy, and has advised technology, SaaS, travel and enterprise ecommerce brands, including Cloudflare, GitLab and Airbnb. He has given multiple talks and panels at the International Search Summit in Barcelona, and has also spoken at Optimization Moscow, brightonSEO, Google Search Central Live Deep Dive Asia in Bangkok, Athens SEO and the Asia SEO Virtual Summit 2026, where his talk looked at how Baidu, Naver, DeepSeek, Tencent, Cốc Cốc and Yandex built their own AI answers.