Google's Crawler Documentation Now Includes New IP List

The new IP list in Google Crawler Documentation includes a range of IP addresses for bots that are triggered by users of Google products but are not controlled by Google. These IP addresses resolve to a gae.googleusercontent.com hostname and are used for apps on Google Cloud or apps scripts called from Google Sheets. The list is significant for publishers who want to whitelist Google-controlled IP addresses or block scrapers using Google’s cloud and other non-Google-associated crawlers.

The list of IP addresses from Google Cloud and App crawlers that Google doesn’t control can be found at the following URL: https://developers.google.com/static/search/apis/ipranges/user-triggered-fetchers.json.

To differentiate between Google controlled crawlers and those not controlled by Google, you can refer to two different JSON files:

• Google controlled crawlers: These originate from IPs listed in user-triggered-fetchers-google.json and resolve to a google.com hostname. These crawlers ignore robots.txt rules because they are triggered by a user request.

• Non-Google controlled crawlers: These come from IPs listed in user-triggered-fetchers.json and resolve to gae.googleusercontent.com hostnames. These are typically used by apps on Google Cloud or apps scripts that are called from Google Sheets.

Google added an additional list of IP addresses for fetchers in their documentation because it became technically possible to export the ranges. The new list, user-triggered-fetchers-google.json, contains IP ranges that have been in use for a long time and are controlled by Google products, as opposed to user-controlled Apps Scripts. This update helps in distinguishing between the fetchers controlled by Google and those that are not.