Google going to stop support for many robots.txt commands

Google’s Gary Illyes announced in Google’s official blog that Google is going to drop support for crawl-delay, noindex and nofollow in robots.txt files.

Google no longer supports crawl-delay, noindex, nofollow

“In particular, we focused on rules unsupported by the internet draft, such as crawl-delay, nofollow, and noindex. Since these rules were never documented by Google, naturally, their usage in relation to Googlebot is very low.

Digging further, we saw their usage was contradicted by other rules in all but 0.001% of all robots.txt files on the internet. These mistakes hurt websites’ presence in Google’s search results in ways we don’t think webmasters intended.

In the interest of maintaining a healthy ecosystem and preparing for potential future open source releases, we’re retiring all code that handles unsupported and unpublished rules (such as noindex) on September 1, 2019.”

You can continue to use the nofollow and noindex attribute in the meta robots tag on web pages. If your website delivers a 404 or 410 HTTP status code for a web page, Google will also remove that URL from its index. You can find Google’s robots.txt specifications here.

Bing does the same

Bing’s Frédéric Dubut said on Twitter that Bing also does not support these robots.txt commands:

How to check the HTTP status codes of your web pages

Even if your web pages can be seen in web browsers, they might send the wrong HTTP status codes to search engine robots. Use the website audit tool to check the HTTP status codes of your web pages and find many other issues on your website that prevent search engines from ranking your web pages:

Check your pages

Johannes Selbach

Johannes Selbach is the CEO of SEOprofiler. He blogs about search engine optimization and website marketing topics at "http://blog.seoprofiler.com".