Google Introduces Two New Crawlers for Enhanced Binary Data Crawling

May 16, 2024 at 8:02:40 PM

Google Introduces Two New Crawlers for Enhanced Binary Data Crawling

Google has introduced two new additions to its Googlebot web crawler family: GoogleOther-Image and GoogleOther-Video. These crawlers are optimized versions of GoogleOther, which was launched in April 2023 for internal use by Google teams to crawl the public web and free up resources for the main Googlebot crawlers.

  • GoogleOther-Image is designed for fetching publicly accessible image URLs. Its user agent tokens are GoogleOther-Image and GoogleOther, with the full user agent string being GoogleOther-Image/1.0.
  • GoogleOther-Video is optimized for fetching publicly accessible video URLs. Its user agent tokens are GoogleOther-Video and GoogleOther, with the full user agent string being GoogleOther-Video/1.0.

Google launched these new crawlers to better support the crawling of binary data for research and development purposes.

The Googlebot crawler family includes:

  • Googlebot, the main crawler for Google's search products, which always respects robots.txt rules.
  • Special-case crawlers, performing specific functions, which may or may not respect robots.txt rules.
  • User-triggered fetchers, tools and product functions triggered by the end-user.

Google also provides IP address ranges and reverse DNS mask for each type of crawler. When observing crawling activities and bot activity on your website and in your log files, do not be alarmed by the new GoogleOther crawlers, as they are genuine Googlebots.

Q&A

Have more questions on this topic? Ask our AI assistant for in-depth insights.

The Only Digital Marketing Feed You'll Ever Need.

Stay informed your way. Tailored updates when and how you want them. 100% Free.

10,000+ Users

500+ Sources

1000+ Tools

Or

Related Posts

Automate Your GA4 Audit - Say Goodbye to Manual Checks!

Automate Your GA4 Audit - Say Goodbye to Manual Checks!

Sponsored
GA4 Auditor
GA4 Auditor

Verified Sponsor

Verified Sponsor

GA4 Auditor is a Verified Sponsor. Want to get featured here? Contact us.

Verified Sponsor
Strategies to Boost Confidence in Consumers and Improve Metrics

Strategies to Boost Confidence in Consumers and Improve Metrics

Think with Google
Think with Google

Official Source

Official Source

Think with Google is a Official Source. The source has been verified by Swipe Insight team.

Official Source
Google CEO Downplays AI Impact on Search Traffic, Boosts Engagement

Google CEO Downplays AI Impact on Search Traffic, Boosts Engagement

Former Google Employee Claims Company's AI Push Driven by Fear of Falling Behind Trending ️‍🔥

Former Google Employee Claims Company's AI Push Driven by Fear of Falling Behind

Google Possibly Displaying More Review Stars in Mobile Search Results

Google Possibly Displaying More Review Stars in Mobile Search Results

Google Introduces 'Web' Filter for Text-based Links in Search Results

Google Introduces 'Web' Filter for Text-based Links in Search Results

Google I/O 2024: Complete Coverage of All News and Announcements Trending ️‍🔥

Google I/O 2024: Complete Coverage of All News and Announcements

Google
Google

Official Source

Official Source

Google is a Official Source. The source has been verified by Swipe Insight team.

Official Source
Google Integrates Gemini into Android, Enhancing In-App Contextual Awareness

Google Integrates Gemini into Android, Enhancing In-App Contextual Awareness

Google
Google

Official Source

Official Source

Google is a Official Source. The source has been verified by Swipe Insight team.

Official Source

Related Tools

GA4 Auditor logo

GA4 Auditor

Verified Tool

Verified Tool

GA4 Auditor is a Verified Tool. Want to get this badge? Contact us.

Verified Tool

Automated GA4 audits with actionable insights

Get Featured Here

Showcase your tool in this list.

Contact Us
Lighthouse logo

Lighthouse

Automated insights for web performance and SEO

SEO
GTmetrix logo

GTmetrix

Analyze and optimize your website performance

SEO
CanIRank logo

CanIRank

AI-driven SEO insights and action recommendations

SEO
JetOctopus logo

JetOctopus

Fast, limitless SEO crawling and log analysis

SEO