YisouSpider and the Robots.txt Trap: Why Shenma Never Crawls Half of International Sites

Most “we tried Shenma and nothing happened” post-mortems are not ranking mysteries. They are crawl-access mistakes. Shenma’s spider is YisouSpider (you will also see it logged as yisouspider). If your robots.txt allow-lists Baiduspider and Googlebot and then Disallow: / for everyone else, you have locked the Alibaba mobile engine out before a single URL is submitted. A September 2026 practitioner note on A5站长网 put this bluntly, and it matches what we still see on international China builds.

This is not a repeat of the onboarding manual in our Shenma for business guide. That piece is the operating checklist. This one is the failure mode: discovery never starts, so ads, Brand Zone and Taobao packs are irrelevant arguments.

What to allow, and where

The webmaster door remains zhanzhang.sm.cn. Verification is HTML file or DNS TXT; DNS can take anywhere from about ten minutes to a couple of hours, and repeating the TXT record because you were impatient just creates a mess. Until verification sticks, resource submission is a locked room. Official help still lives at the Shenma help centre.

For robots, be explicit rather than clever:

  • User-agent: YisouSpider with Allow: / on anything you actually want in the mobile index.
  • A Sitemap: line at the root, not buried in a subdirectory file Shenma will not fetch.
  • No UA-sniffing that 403s unknown bots. Shenma will not argue with your WAF.

Sitemap rules cited by Chinese webmasters line up with the usual protocol ceiling: 50,000 URLs and about 10MB per file, UTF-8, then an index file if you overflow. Shenma’s manual URL paste is commonly described as 1,000 URLs a time — fine for a repair queue, useless as your only discovery system. Sogou’s dead-link file wants plain text, one URL a line; do not assume Shenma will politely parse a Google-shaped XML dead-link feed either. Check the platform’s sitemap status page for “fetched” versus “skipped as low quality” instead of staring at a site: query for a fortnight.

Mobile adaptation is a crawl issue, not a design slogan

Shenma still prefers pages that are actually mobile. If you run a separate m. host, the spider has to see a clean hop and a 200 on the mobile URL. A PC page that mentions a mobile twin only in a tag your renderer never emits is how you end up indexed on the wrong host. Our older mobile optimisation note still applies; pair it with log splits so YisouSpider is not averaged into “all bots”.

Opinion: international teams over-invest in Baidu rank tracking and under-invest in whether Shenma is allowed to fetch the HTML. A whitelist robots file is not a security posture. It is an accidental boycott of every Chinese crawler you forgot to name. Fix the UA, verify the property, submit a sane sitemap, then argue about content. Until YisouSpider returns 200s in the logs, you do not have a Shenma SEO programme. You have a rumour.