Robots.txt Checker

Fetch and view the robots.txt file from any website.

Robots.txt is a standard used by websites to communicate with web crawlers and search engine bots. Our robots.txt checker fetches and displays this file from any URL, showing exactly which paths are disallowed and which user-agents are affected.

Result

Enter a URL to fetch robots.txt.

FAQ

What is robots.txt?

Robots.txt is a file placed at the root of a website that tells search engine crawlers which pages they can or cannot access.