Google's Gary Illyes Urges Disallowing Crawling of Action URLs to Save Server Resources

June 11, 2024 at 10:57:37 AM

Google's Gary Illyes Urges Disallowing Crawling of Action URLs to Save Server Resources

Gary Illyes from Google advises website owners to disallow the crawling of action URLs, such as "add to cart" and "add to wishlist," to save server resources and improve efficiency. Crawlers do not interact with these URLs in a meaningful way, and their crawling can unnecessarily consume server resources.

Illyes suggests adding disallow rules for these URLs in the robots.txt file or converting them to the HTTP POST method, though some crawlers can still make POST requests. Examples of such URLs include:

  • https://example.com/product/scented-candle-v1?add_to_cart
  • https://example.com/product/scented-candle-v1?add_to_wishlist

Have more questions on this topic? Ask our AI assistant for in-depth insights.

Read more from sources 👇

The Only Digital Marketing Feed You'll Ever Need.

Stay informed your way. Tailored updates when and how you want them. 100% Free.

10,000+ Users

500+ Sources

1000+ Tools

Or

Related Posts

Automate Meta Ads Creative Generation and Uploading

Automate Meta Ads Creative Generation and Uploading

Featured
Markifact
Markifact

Verified Sponsor

Verified Sponsor

Markifact is a Verified Sponsor. Want to get featured here? Contact us.

Verified Sponsor
Google Launches Gemini 3 Pro to Boost AI Search with Interactive Tools and Simulations

Google Launches Gemini 3 Pro to Boost AI Search with Interactive Tools and Simulations

Google
Google

Official Source

Official Source

Google is a Official Source. The source has been verified by Swipe Insight team.

Official Source
Adobe Acquires Semrush for 1.9 Billion to Boost Brand Visibility with AI

Adobe Acquires Semrush for 1.9 Billion to Boost Brand Visibility with AI

Google Search Console adds branded queries filter to analyze site traffic

Google Search Console adds branded queries filter to analyze site traffic

Google for Developers
Google for Developers

Official Source

Official Source

Google for Developers is a Official Source. The source has been verified by Swipe Insight team.

Official Source
Microsoft Copilot enhances AI search with prominent citations and new dedicated search tab

Microsoft Copilot enhances AI search with prominent citations and new dedicated search tab

Microsoft
Microsoft

Official Source

Official Source

Microsoft is a Official Source. The source has been verified by Swipe Insight team.

Official Source
Google Search simplifies search results page for better user experience

Google Search simplifies search results page for better user experience

Google Search Central
Google Search Central

Official Source

Official Source

Google Search Central is a Official Source. The source has been verified by Swipe Insight team.

Official Source
Google adds new Google-CWS user agent for Chrome Web Store fetcher

Google adds new Google-CWS user agent for Chrome Web Store fetcher

Google for Developers
Google for Developers

Official Source

Official Source

Google for Developers is a Official Source. The source has been verified by Swipe Insight team.

Official Source
Google Search Console Launches New Query Groups Feature in Insights

Google Search Console Launches New Query Groups Feature in Insights

Google Search Central
Google Search Central

Official Source

Official Source

Google Search Central is a Official Source. The source has been verified by Swipe Insight team.

Official Source

Related Tools

Markifact logo

Markifact

Verified Tool

Verified Tool

Markifact is a Verified Tool. Want to get this badge? Contact us.

Verified Tool

Marketing Workflows Powered by AI

Featured
Marketing Auditor logo

Marketing Auditor

Verified Tool

Verified Tool

Marketing Auditor is a Verified Tool. Want to get this badge? Contact us.

Verified Tool

Automated audits for Google Ads and Analytics.

Get Featured Here

Showcase your tool in this list.

Contact Us
Ahrefs logo

Ahrefs

SEO tools to boost traffic and rank higher

SEO
Lighthouse logo

Lighthouse

Automated insights for web performance and SEO

SEO
Surfer SEO logo

Surfer SEO

SEO content creation and optimization made easy

SEO
Sitebulb logo

Sitebulb

Efficient website crawler for better SEO audits

SEO
Screpy logo

Screpy

AI-Powered SEO and Web Analysis Simplified

SEO
Blogify logo

Blogify

Convert multimedia to SEO-optimized blogs fast

SEO
Answer the Public logo

Answer the Public

Unlock Consumer Insights for Content Creation

SEO
SEO Writing AI logo

SEO Writing AI

AI-powered SEO content in 1 click

SEO

Get Featured Here

Showcase your tool in this list.

Contact Us