October 7, 2014

Configure Googlebot-Image bot in robots.txt

Googlebot-Image Bot

What is the Googlebot-Image Bot?

From Google we have received an email regarding the Googlebot-Image bot.
This is the Google Image crawler that indexes images appearing in Google's image search results. The Googlebot-Image bot thankfully listens to the robots.txt file - normally located in the root of your public website folder.
The Googlebot-Image bot is pretty flexible. Here are some examples.

Allow indexing of a folder

User-agent: Googlebot-Image 
Disallow: / 
Allow: /path/to/dir/

Remove all images from index

User-agent: Googlebot-Image
Disallow: /

Exclude a file from indexing

This will exclude file image.jpg:
User-agent: Googlebot-Image
Disallow: /image/image.jpg

Exclude a file extension like JPG

This will exclude JPG images and include other formats like GIF:
User-agent: Googlebot-Image
Disallow: /*.jpg$

Exclude files from Google Image Search and Google Web Search

If you want to exclude files not only from Google Image Search, but from Google Web Search as well, you have to replace User-agent: Googlebot-Image with Googlebot. Like this:
User-agent: Googlebot

More information at Google:

1 comment:

  1. I was very impressed by this post, this site has always been pleasant news Thank you very much for such an interesting post, and I meet them more often then I visited this site. txtcart

    ReplyDelete