robots Policy Inspector

robots Policy Inspector

Fetches site robots.txt, parses User-agent groupings, and immediately verifies whether a path is allowed to crawl.