Google Introduces Two New Crawlers for Enhanced Binary Data Crawling

Google has introduced two new crawlers to the Googlebot family of web crawlers, named GoogleOther-Image and GoogleOther-Video. These crawlers are specialized versions of GoogleOther, optimized for fetching image and video bytes respectively.

The purpose of the GoogleOther-Image and GoogleOther-Video crawlers is to better support crawling of binary data, specifically images and videos, which may be used for research and development. GoogleOther-Image is optimized for fetching publicly accessible image URLs, while GoogleOther-Video is optimized for fetching publicly accessible video URLs.

For GoogleOther-Image, the user agent tokens are GoogleOther-Image and GoogleOther, and the full user agent string is GoogleOther-Image/1.0. Similarly, for GoogleOther-Video, the user agent tokens are GoogleOther-Video and GoogleOther, and the full user agent string is GoogleOther-Video/1.0.

Googlebot crawlers can be categorized into three types:

• Googlebot – The main crawler for Google’s search products, which always respects robots.txt rules.

• Special-case crawlers – Crawlers that perform specific functions, such as AdsBot, which may or may not respect robots.txt rules.

• User-triggered fetchers – Tools and product functions where the end-user triggers a fetch, such as Google Site Verifier or some Google Search Console tools.

To identify the new GoogleOther crawlers in your website logs, you can look for the user agent tokens GoogleOther-Image and GoogleOther-Video or the full user agent strings GoogleOther-Image/1.0 and GoogleOther-Video/1.0. Google has also provided IP address ranges and reverse DNS masks for each type of crawler in their respective JSON files: googlebot.json, special-crawlers.json, and user-triggered-fetchers.json.