Google's Gary Illyes Urges Disallowing Crawling of Action URLs to Save Server Resources

June 11, 2024 at 10:57:37 AM

Google's Gary Illyes Urges Disallowing Crawling of Action URLs to Save Server Resources

Gary Illyes from Google advises website owners to disallow the crawling of action URLs, such as "add to cart" and "add to wishlist," to save server resources and improve efficiency. Crawlers do not interact with these URLs in a meaningful way, and their crawling can unnecessarily consume server resources.

Illyes suggests adding disallow rules for these URLs in the robots.txt file or converting them to the HTTP POST method, though some crawlers can still make POST requests. Examples of such URLs include:

  • https://example.com/product/scented-candle-v1?add_to_cart
  • https://example.com/product/scented-candle-v1?add_to_wishlist

Have more questions on this topic? Ask our AI assistant for in-depth insights.

Read more from sources 👇

Want Personalized Digital Marketing Insights at Your Preferred Time?

Our Smart Newsletter brings you the latest insights on the topics you love, delivered at your preferred time and frequency.

Discover More

Google Product Panels Display Site Reviews Without Purchase Links

Google Product Panels Display Site Reviews Without Purchase Links

TikTok Users Can Now Download Videos without Watermark

TikTok Users Can Now Download Videos without Watermark

Looker Studio to Add Native Microsoft Excel Connector

Looker Studio to Add Native Microsoft Excel Connector

Sean Zinsmeister
Sean Zinsmeister

Official Source

Official Source

Sean Zinsmeister is a Official Source. The source has been verified by Swipe Insight team.

Official Source
BigQuery Now Allows Drag-and-Drop Tabs

BigQuery Now Allows Drag-and-Drop Tabs

Google Cloud
Google Cloud

Official Source

Official Source

Google Cloud is a Official Source. The source has been verified by Swipe Insight team.

Official Source
Google Ads Updates Gambling and Games Policy to Include Lottery-Courier Ads in 33 States

Google Ads Updates Gambling and Games Policy to Include Lottery-Courier Ads in 33 States

Google
Google

Official Source

Official Source

Google is a Official Source. The source has been verified by Swipe Insight team.

Official Source
Microsoft Ads Launches Property Center and Global Lodging Campaigns

Microsoft Ads Launches Property Center and Global Lodging Campaigns

Microsoft Advertising
Microsoft Advertising

Official Source

Official Source

Microsoft Advertising is a Official Source. The source has been verified by Swipe Insight team.

Official Source