Google Warns of Sudden Crawl Increase: Check for Infinite Spaces, Hacked Content

June 19, 2024 at 8:38:41 AM

Google Warns of Sudden Crawl Increase: Check for Infinite Spaces, Hacked Content

Gary Illyes from Google warns that a sudden increase in crawling can indicate both positive and negative developments. While it might signal positive changes, it can also point to potential issues. Two common problems associated with sharp increases in crawling are:

Infinite Spaces

If your site includes elements like a calendar or infinitely filterable product listings, crawlers may become overly enthusiastic about these infinite spaces. This can lead to excessive crawling. To manage this, utilize robots.txt to exclude such content.

Hacked Content

If your server's file system or content management system is compromised, malicious actors might flood your site with unwanted content. Crawlers may then start indexing these new pages. For guidance on handling such situations, visit web.dev/hacked.

Treat unexpected increases in crawling as a potential issue until proven otherwise.

different-errors-might-i-735533d78f615.gif

Have more questions on this topic? Ask our AI assistant for in-depth insights.

Read more from sources 👇

Want Personalized Digital Marketing Insights at Your Preferred Time?

Our Smart Newsletter brings you the latest insights on the topics you love, delivered at your preferred time and frequency.

Discover More

Google Explains How to Use CDN-Hosted robots.txt Files

Google Explains How to Use CDN-Hosted robots.txt Files

Gary Illyes
Gary Illyes

Official Source

Official Source

Gary Illyes is a Official Source. The source has been verified by Swipe Insight team.

Official Source
Google Shares 4 Image Optimization Tips for SEO

Google Shares 4 Image Optimization Tips for SEO

Google Search Central
Google Search Central

Official Source

Official Source

Google Search Central is a Official Source. The source has been verified by Swipe Insight team.

Official Source
60% of Google Searches Result in No Clicks to External Sites Trending ️‍🔥

60% of Google Searches Result in No Clicks to External Sites

Google Product Panels Display Site Reviews Without Purchase Links

Google Product Panels Display Site Reviews Without Purchase Links

Microsoft Adds Prompt Injection Rule to Bing Webmaster Guidelines

Microsoft Adds Prompt Injection Rule to Bing Webmaster Guidelines

Bing
Bing

Official Source

Official Source

Bing is a Official Source. The source has been verified by Swipe Insight team.

Official Source
robots.txt Turns 30, Why Web Crawlers Ignore Your Typos

robots.txt Turns 30, Why Web Crawlers Ignore Your Typos

Gary Illyes
Gary Illyes

Official Source

Official Source

Gary Illyes is a Official Source. The source has been verified by Swipe Insight team.

Official Source