How to Test robots.txt for Google, Bing, and AI Search

Define the check you are performing

Start with the robots.txt belonging to the URL’s scheme, host and port. Copy its current plain text and record the date. A file on one host does not establish the rules on another. Inspect what is actually served, especially when a hosting feature can prepend or manage rules.

In the free Robots.txt Tester, enter the exact crawler token, such as Googlebot, Bingbot or OAI-SearchBot. Do not paste a complete browser-style User-Agent header into that field. The result reports the matching token group, line and rule for the supplied input.

The underlying Robots Exclusion Protocol describes cooperative crawling behavior; it is not an access-control system for confidential files. Use actual authentication or authorization for material that must remain private. See RFC 9309.

Paste a deliberately small example

Illustrative fixture: the following is a teaching example, not Jincheng’s live robots file and not a recommendation to publish these restrictions on your site.

User-agent: *
Disallow: /private/
Allow: /private/public/
Disallow: /*.pdf$

User-agent: OAI-SearchBot
Disallow: /private/
Allow: /private/public/
Disallow: /drafts/

Download the example rules. Use URLs beginning with https://example.com for these local tests; the tool does not request that host. The generic group restricts private paths and PDF endings. The specific OAI-SearchBot group repeats the private-path policy and adds a drafts restriction.

The PDF restriction is deliberately absent from the specific group. That omission makes the difference between generic and specific groups visible. If your policy should restrict PDFs for both, you need to express and test that intention in both applicable groups.

Run the expected-result matrix

Download the locally checked results. The fixture and expectations are tested with the same parser used in the tool. This establishes reproducibility for these cases, not exact equivalence to every provider’s crawler implementation.

Try changing /private/ to /Private/ and adding a query after /guide.pdf. Paths are case-sensitive; the end anchor changes what the PDF rule matches. Record what changed instead of treating similar-looking URLs as interchangeable.

Read groups and encoded paths carefully

Under the protocol, matching user-agent groups are compared without case sensitivity and repeated matching groups are combined. Path matching is case-sensitive. The most specific matching rule wins, with Allow preferred for an equivalent Allow/Disallow match. The RFC defines these matching rules.

Our tester uses exact product tokens with a generic fallback, wildcard matching, the end marker and normalization of percent-encoded unreserved ASCII. Its tests include Chinese paths and encoded paths. It does not simulate every provider extension, network failure or robots-fetch policy. A warning about an unsupported directive is a reason to check that provider’s documentation.

Google documents its implementation and supported fields separately. For example, a Sitemap line is useful for discovery but does not change whether an Allow or Disallow path matches. Check Google’s interpretation. Bing also offers its own robots tester in Webmaster Tools for provider-specific investigation.

Separate AI search access from training policy

A crawler’s name is not a generic permission for every AI use. OpenAI documents OAI-SearchBot for search and GPTBot for potential training; those settings are independent. ChatGPT-User is associated with certain user-triggered actions and is not the search-indexing control. Read the current crawler descriptions.

Decide the desired policy before adding a group. A new specific group containing only Allow may accidentally omit restrictions you expected from the generic group. Test both a public URL and a deliberately restricted URL for every token you add.

The example in this article does not change Jincheng’s policy. Nor does naming a crawler guarantee greater visibility. For an editorial decision framework, see SEO, GEO and AEO compared.

Verify the published file and the hosting layer

1. Save the old rules and the outcomes you intend to preserve.
2. Run the test matrix against the proposed text, including restricted and allowed cases.
3. Publish the approved change and fetch the actual robots.txt again.
4. Inspect CDN, firewall and bot controls using provider evidence or official inspection.
5. Record the deployment and later crawler observations separately.

Use the brief builder to document who needs the rule change, evidence still missing and a next verification step. If the page is accessible but absent from an index, switch to the indexing checklist rather than adding more crawler names.

For language annotations, use the hreflang tool. It solves a different task. Robots rules, language mapping and indexing observations should remain separate entries in the change log.