Google has recently expanded its roster of official crawlers with the addition of Google-CloudVertexBot, a specialized crawler designed to support the development of Vertex AI Agents.
Purpose and Functionality
The new Google-CloudVertexBot crawler serves two primary functions:
- It crawls websites at the request of site owners who are building Vertex AI Agents.
- It indexes web pages for search and recommendation purposes within the Vertex AI ecosystem.
Benefits for Site Owners
The introduction of this crawler allows site owners to:
- Identify specific crawler traffic related to Vertex AI Agent development
- Optimize their content for Vertex AI applications
Important Considerations for Website Indexing
When preparing for Google-CloudVertexBot crawling, site owners should:
- Ensure robots.txt is not blocking indexing for targeted pages
- Verify domain ownership for Advanced website indexing
- Implement structured data (meta tags and PageMaps) to enhance indexing
Implementation Process
To utilize the Google-CloudVertexBot crawler, site owners can:
- Create search data stores
- Set up recommendations data stores
This new crawler represents Google's commitment to advancing its AI capabilities while maintaining transparency with website owners about how their content is crawled and indexed for various Google services.
