Robots.txt Checker

Fetch and view the robots.txt file from any website.

Robots.txt is a standard used by websites to communicate with web crawlers and search engine bots. Our robots.txt checker fetches and displays this file from any URL, showing exactly which paths are disallowed and which user-agents are affected.

Result

Enter a URL to fetch robots.txt.
In Action

Robots Checker in action

After a site migration your traffic from Google drops. Run the robots checker and you find the staging server's "Disallow: /" was copied to production. Fixing one line in robots.txt is all it takes — and the checker confirms the fix instantly.

Frequently Asked Questions

What is robots.txt?

Robots.txt is a file placed at the root of a website that tells search engine crawlers which pages they can or cannot access.

How do I check a robots.txt file online?

Enter the site URL and the tool fetches its /robots.txt, displaying all rules in readable form. It also shows whether the file exists at all — a surprisingly common issue.

What does "Disallow: /" in robots.txt mean?

It asks all crawlers to stay off the entire site. Some sites use this during development and forget to remove it — which is why pages mysteriously disappear from search results after a redesign.

Can robots.txt prevent a page from being indexed?

Robots.txt stops crawling but does not guarantee de-indexing. If a page is linked elsewhere, search engines may index it without visiting. To remove already-indexed pages, use noindex tags — robots.txt is the first layer, not the only one.