In a recent post, Google's Gary Illyes highlighted the problems caused by soft 404s and other "soft" or "crypto" errors for both search engines and websites. These issues, often overlooked by site owners, can have a substantial impact on a site's SEO performance and resource allocation.
What Are Soft 404s?
Soft 404s occur when a web server returns a standard 200 OK status code for a page that doesn't actually exist or contains error content. This misleads search engine crawlers, causing them to treat non-existent or error pages as valid content.
Why Soft Errors Are Problematic:
- Wasted Crawl Budget: Crawlers spend time on non-existent pages instead of indexing real, valuable content.
- Poor ROI: These pages are unlikely to appear in search results, wasting the resources used to serve them.
The solution is to use appropriate HTTP status codes for errors.
Illyes used a coffee shop analogy: imagine ordering items from a menu, only to find everything unavailable. This frustration mirrors crawler inefficiency when accessing invalid pages.
He also emphasizes the importance of using appropriate HTTP status codes when errors occur. This helps crawlers understand the true status of a page, allowing for more efficient crawling and indexing.
Additional Notes:
- Rate limiting crawlers with messages like "TOO MANY REQUESTS SLOW DOWN" is ineffective, as crawlers can't interpret these messages.
- The term "crypto" in this context means "hidden," not cryptocurrency.
By addressing these issues, website owners can improve crawlability and potentially enhance search engine visibility.
