Letting Search Engines Crawl Your Site

A robots.txt rule can stop a search engine’s crawler from fetching a page. Check these rules first when a page is missing from search results, alongside noindex tags, server errors and firewall settings. Being crawlable does not by itself mean a page will be indexed or rank well.

Which Crawlers We Check

The report checks 6 crawlers, in two groups.

The 2 main search engine crawlers decide whether your page can appear in the largest search engines. Blocking one keeps that engine from reading the page, so Headnoted weights their access heavily.

Googlebot - Google Search
Bingbot - Bing, whose index is also used by some other search services

The other 4 are smaller engines and specialist crawlers. You may have a reason to block one, so a block only ever marks the check as Improve.

Googlebot-Image - Google Images
Applebot - Safari suggestions and Spotlight
DuckDuckBot - DuckDuckGo
YandexBot - Yandex Search

Headnoted does not check AI assistant crawlers such as GPTBot or ClaudeBot. Our sister tool, Footnoted, checks those.

Allowing Search Crawlers

robots.txt lives at the root of your site, for example example.com/robots.txt. This example lets every crawler in, asks them not to fetch an admin area and points them at the sitemap. It does not keep that area private: protect private content with a sign-in.

User-agent: *
Disallow: /admin/

Sitemap: https://www.example.com/sitemap.xml

A Named Crawler Ignores the * Group

A group for a specific crawler, such as "User-agent: Googlebot", replaces the "User-agent: *" group for that crawler rather than adding to it. Repeat every restriction you intend in the named group. Google documents its matching rules in the source below.

Other Things That Hide a Page

A noindex robots meta tag or X-Robots-Tag header keeps the page out of search results, even when robots.txt allows it
nosnippet lets the page be listed but stops the search engine showing a text snippet for it, so the report marks it as Improve
A robots.txt block stops the crawler reading the page at all - including a noindex on it - so the address can occasionally still be listed without a description
Some hosting and bot-protection services block crawlers at the firewall, separately from robots.txt. We cannot see those settings, but if our own crawler is turned away, the report's "Page reachable by crawlers" check says so

Content in the HTML

Google can run JavaScript, but it does so in a later step, and not every search engine does. If your page builds its text in the browser, the "Readable without JavaScript" check will flag it - render the main content on the server so every crawler sees it straight away.

Verify a Change

1 Open your site’s /robots.txt and find the group that matches the crawler you want to check.
2 Check the exact page path against its Allow and Disallow rules. A named group may behave differently from the wildcard group.
3 Check the page’s HTTP status, X-Robots-Tag header and robots meta tag. A 200 response can still carry noindex.
4 Check your hosting or firewall logs for the real crawler. A page that opens in your own browser may still be refused to a crawler.
5 Run a new Headnoted check, then use the URL Inspection tool in Google Search Console, or the URL Inspection tool in Bing Webmaster Tools, to see what that engine found.

For example, if /delivery is accidentally under Disallow: /, fix the rule for the search crawlers you want. If /delivery instead carries a noindex, changing robots.txt will not remove it. Keep private account areas protected whatever your crawler settings.

Sources and Limits

Google’s robots.txt specification explains matching and supported rules. Google’s noindex guidance explains why a crawler must be allowed to fetch a page to read its noindex. Robots rules control crawling; they are not a security boundary, and well-behaved crawlers follow them voluntarily.