Google May Expand Unsupported Robots.txt Rules List via @sejournal, @MattGSouthern
via searchenginejournal.com
Short excerpt below. Read at the original source.
Google may expand its unsupported robots.txt rules list using HTTP Archive data and could broaden how it handles common misspellings of disallow. The post Google May Expand Unsupported Robots.txt Rules List appeared first on Search Engine Journal.