Reddit has updated its robots.txt file, preventing Bing and many other search engines from crawling the site as of July 1, 2024. Despite this, Google is still allowed to crawl Reddit. This move was confirmed by a Microsoft representative, who stated that Bing respects the robots.txt standard and has ceased crawling Reddit.
What Happened
- Date: On July 1, 2024, Reddit updated its
robots.txtfile. - Impact: This update prevents many search engines and AI tools from crawling Reddit.
- Google Exception: Google is still allowed to crawl Reddit, contrary to some earlier beliefs.
Confirmation
A Microsoft spokesperson confirmed that Bing stopped crawling Reddit following the updated robots.txt file, which prohibits all crawling of the site.
- Initial Notification: Mark Williams-Cook noticed Reddit results disappearing from Bing's search index.
- IP Detection: Reddit was using IP detection to show different versions of its
robots.txtfile to search engines and humans.
Reddit's Statement
Reddit spokesperson Tim Rathschmidt clarified that the update is unrelated to Reddit's recent partnership with Google. He mentioned that Reddit has been in discussions with multiple search engines but couldn't reach agreements with all, particularly those unwilling to make enforceable promises regarding the use of Reddit content, including for AI purposes.
Implications
- Licensing Deal with Google: Reddit's deal with Google allows it to block other search engines and AI tools.
- Traffic: Google is driving significant traffic to Reddit and is even testing special treatment in its search results.
- Future Speculations: This raises questions about whether other large websites might follow suit and the potential impact on smaller publishers and content producers.
In summary, Reddit's updated robots.txt file has significantly affected Bing and other search engines, while Google continues to benefit from its partnership with Reddit.
