Reddit Now Blocks Bing from Crawling Its Site, Grants Exclusive Access to Google

July 25, 2024 at 6:56:36 AM - Trending 🔥

TL;DR Reddit updated its `robots.txt` file on July 1, 2024, blocking Bing and other search engines from crawling the site, while allowing Google to continue. Microsoft confirmed Bing's compliance with this change. Reddit's spokesperson clarified that this move is unrelated to its partnership with Google but stems from failed agreements with other search engines. This raises questions about future impacts on traffic and smaller content producers.

Reddit Now Blocks Bing from Crawling Its Site, Grants Exclusive Access to Google

Reddit has updated its robots.txt file, preventing Bing and many other search engines from crawling the site as of July 1, 2024. Despite this, Google is still allowed to crawl Reddit. This move was confirmed by a Microsoft representative, who stated that Bing respects the robots.txt standard and has ceased crawling Reddit.

What Happened

  • Date: On July 1, 2024, Reddit updated its robots.txt file.
  • Impact: This update prevents many search engines and AI tools from crawling Reddit.
  • Google Exception: Google is still allowed to crawl Reddit, contrary to some earlier beliefs.

Confirmation

A Microsoft spokesperson confirmed that Bing stopped crawling Reddit following the updated robots.txt file, which prohibits all crawling of the site.

  • Initial Notification: Mark Williams-Cook noticed Reddit results disappearing from Bing's search index.
  • IP Detection: Reddit was using IP detection to show different versions of its robots.txt file to search engines and humans.

Reddit's Statement

Reddit spokesperson Tim Rathschmidt clarified that the update is unrelated to Reddit's recent partnership with Google. He mentioned that Reddit has been in discussions with multiple search engines but couldn't reach agreements with all, particularly those unwilling to make enforceable promises regarding the use of Reddit content, including for AI purposes.

Implications

  • Licensing Deal with Google: Reddit's deal with Google allows it to block other search engines and AI tools.
  • Traffic: Google is driving significant traffic to Reddit and is even testing special treatment in its search results.
  • Future Speculations: This raises questions about whether other large websites might follow suit and the potential impact on smaller publishers and content producers.

In summary, Reddit's updated robots.txt file has significantly affected Bing and other search engines, while Google continues to benefit from its partnership with Reddit.

Q&A

Have more questions on this topic? Ask our AI assistant for in-depth insights.

The Only Digital Marketing Feed You'll Ever Need.

Stay informed your way. Tailored updates when and how you want them. 100% Free.

10,000+ Users

500+ Sources

1000+ Tools

Or

Related Posts

Reddit Restricts Moderators' Ability to Switch Subreddits from Public to Private

Reddit Restricts Moderators' Ability to Switch Subreddits from Public to Private

Reddit
Reddit

Official Source

Official Source

Reddit is a Official Source. The source has been verified by Swipe Insight team.

Official Source
Audit your GA4 account in Minutes

Audit your GA4 account in Minutes

Sponsored
GA4 Auditor
GA4 Auditor

Verified Sponsor

Verified Sponsor

GA4 Auditor is a Verified Sponsor. Want to get featured here? Contact us.

Verified Sponsor
Reddit Expands AI-Powered Post Translation to 35+ New Countries

Reddit Expands AI-Powered Post Translation to 35+ New Countries

Reddit
Reddit

Official Source

Official Source

Reddit is a Official Source. The source has been verified by Swipe Insight team.

Official Source
Publishers Boost Reddit Presence After Google's Helpful Content Update Increases Traffic

Publishers Boost Reddit Presence After Google's Helpful Content Update Increases Traffic

Reddit to Test AI-Powered Search Result Pages with Summaries and Recommendations

Reddit to Test AI-Powered Search Result Pages with Summaries and Recommendations

Reddit Acquires Memorable AI to Boost Advertiser Performance with Creative Intelligence

Reddit Acquires Memorable AI to Boost Advertiser Performance with Creative Intelligence

Reddit
Reddit

Official Source

Official Source

Reddit is a Official Source. The source has been verified by Swipe Insight team.

Official Source
Reddit CEO Demands Microsoft Pay for Scraping Site Data

Reddit CEO Demands Microsoft Pay for Scraping Site Data

Microsoft Misses Q2 Expectations Despite 15% Increase in Revenue

Microsoft Misses Q2 Expectations Despite 15% Increase in Revenue

Related Tools

GA4 Auditor logo

GA4 Auditor

Verified Tool

Verified Tool

GA4 Auditor is a Verified Tool. Want to get this badge? Contact us.

Verified Tool

Automated GA4 audits with actionable insights

Get Featured Here

Showcase your tool in this list.

Contact Us
Surfer SEO logo

Surfer SEO

SEO content creation and optimization made easy

SEO
Sitebulb logo

Sitebulb

Efficient website crawler for better SEO audits

SEO
Screpy logo

Screpy

AI-Powered SEO and Web Analysis Simplified

SEO
Blogify logo

Blogify

Convert multimedia to SEO-optimized blogs fast

SEO
Answer the Public logo

Answer the Public

Unlock Consumer Insights for Content Creation

SEO
Ahrefs logo

Ahrefs

SEO tools to boost traffic and rank higher

SEO
SEO Writing AI logo

SEO Writing AI

AI-powered SEO content in 1 click

SEO
SEO Stuff logo

SEO Stuff

Affordable SEO tools without monthly fees

SEO
Screaming Frog logo

Screaming Frog

Comprehensive SEO audits with real-time crawling

SEO

Get Featured Here

Showcase your tool in this list.

Contact Us