Analyse robots.txt content and report whether a path may be crawled.
Free
Scanned where there is a package or repository to read, and every release diffed against the tool surface we already hold.
Analyse robots.txt content and report whether a path may be crawled.