Reddit CEO Demands Microsoft Pay for Scraping Site Data

August 01, 2024 at 3:30:56 PM

TL;DR Reddit CEO Steve Huffman insists Microsoft and others must pay to scrape the site’s data, following deals with Google and OpenAI. Huffman criticized Microsoft, Anthropic, and Perplexity for refusing to negotiate, leading to blocking their crawlers. Reddit updated its robots.txt to block non-paying crawlers, affecting visibility on Bing. Huffman argues public data isn't freeware and seeks licensing deals similar to OpenAI's SearchGPT.

Reddit CEO Demands Microsoft Pay for Scraping Site Data

Reddit CEO Steve Huffman is urging Microsoft and other companies to pay for scraping Reddit's data, following agreements with Google and OpenAI. Huffman criticized Microsoft, Anthropic, and Perplexity for refusing to negotiate, leading Reddit to block these companies from accessing its data. Reddit updated its robots.txt file in July to block unauthorized web crawlers, resulting in Reddit content being visible only in Google search results, where Reddit is compensated.

Huffman accused Microsoft of using Reddit's data to train its AI and summarizing content in Bing results without permission, and selling this data through the Bing API. He referenced Microsoft AI CEO Mustafa Suleyman's comment that public internet data is "freeware" to highlight the issue. Microsoft responded by stating they respect websites' directives not to use their content with generative AI models.

Huffman emphasized that the traditional value exchange between search engines and content providers is changing, with search, summarization, and training merging. He pointed to OpenAI's SearchGPT, which includes Reddit results due to a licensing deal, as a model for future agreements. Reddit aims to join traditional media publishers in seeking payment for their content used in generative AI.

Anthropic confirmed they have blocked Reddit from their web crawler since mid-May, respecting the robots.txt file. Microsoft and Perplexity did not provide further comments.

Have more questions on this topic? Ask our AI assistant for in-depth insights.

The Only Digital Marketing Feed You'll Ever Need.

Stay informed your way. Tailored updates when and how you want them. 100% Free.

10,000+ Users

500+ Sources

1000+ Tools

Or

Related Posts

Audit your GA4 account in Minutes

Audit your GA4 account in Minutes

Sponsored
GA4 Auditor
GA4 Auditor

Verified Sponsor

Verified Sponsor

GA4 Auditor is a Verified Sponsor. Want to get featured here? Contact us.

Verified Sponsor
Reddit Restricts Moderators' Ability to Switch Subreddits from Public to Private

Reddit Restricts Moderators' Ability to Switch Subreddits from Public to Private

Reddit
Reddit

Official Source

Official Source

Reddit is a Official Source. The source has been verified by Swipe Insight team.

Official Source
Reddit Expands AI-Powered Post Translation to 35+ New Countries

Reddit Expands AI-Powered Post Translation to 35+ New Countries

Reddit
Reddit

Official Source

Official Source

Reddit is a Official Source. The source has been verified by Swipe Insight team.

Official Source
Publishers Boost Reddit Presence After Google's Helpful Content Update Increases Traffic

Publishers Boost Reddit Presence After Google's Helpful Content Update Increases Traffic

Reddit to Test AI-Powered Search Result Pages with Summaries and Recommendations

Reddit to Test AI-Powered Search Result Pages with Summaries and Recommendations

Reddit Acquires Memorable AI to Boost Advertiser Performance with Creative Intelligence

Reddit Acquires Memorable AI to Boost Advertiser Performance with Creative Intelligence

Reddit
Reddit

Official Source

Official Source

Reddit is a Official Source. The source has been verified by Swipe Insight team.

Official Source
Reddit Now Blocks Bing from Crawling Its Site, Grants Exclusive Access to Google Trending ️‍🔥

Reddit Now Blocks Bing from Crawling Its Site, Grants Exclusive Access to Google

Bing SEO +1 more
Reddit Launches Sports Program to Boost Engagement

Reddit Launches Sports Program to Boost Engagement

Reddit
Reddit

Official Source

Official Source

Reddit is a Official Source. The source has been verified by Swipe Insight team.

Official Source

Related Tools

GA4 Auditor logo

GA4 Auditor

Verified Tool

Verified Tool

GA4 Auditor is a Verified Tool. Want to get this badge? Contact us.

Verified Tool

Automated GA4 audits with actionable insights

Get Featured Here

Showcase your tool in this list.

Contact Us