Perplexity uses stealth crawlers that change user agents and IPs to bypass no-crawl directives and ignore robots.txt files. Cloudflare blocked these undeclared crawlers after tests showed Perplexity accessed restricted content despite blocks. Unlike OpenAI, Perplexity evades detection and violates crawling norms. Cloudflare added rules to block such AI crawling, protecting customers and promoting transparent, respectful bot behavior.
Cloudflare blocks AI crawlers by default and introduces Pay Per Crawl, letting content creators charge AI services for access. Using HTTP 402 responses and authentication, publishers can allow, charge, or block crawlers with flat prices. This empowers creators to monetize content at scale and supports future dynamic pricing and agent-based access. Pay Per Crawl is in private beta.
Cloudflare has released AI Audit, a tool for website owners to control AI bots accessing their content. It identifies AI bots, provides control to block or allow them, and offers analytics on bot activity. Key features include free bot control, detailed analytics, and legal support. A future feature will allow content pricing for AI access. This helps site owners manage AI scraping, which can impact traffic and content value.
Run an automated Google Ads audit directly from Google Sheets with 40+ built-in checks. Review campaigns, ad groups, keywords, ads, and account settings to catch missed best practices and optimization gaps. Ideal for agencies and in-house teams to standardize audits, save hours of manual work, and turn findings into clear, actionable recommendations.
Cloudflare has launched a free tool to prevent bots from scraping websites for AI model training. Despite some AI vendors allowing site owners to block bots via robots.txt, not all bots comply. Cloudflare's tool uses advanced detection models to identify and block evasive AI bots. The tool is available to all customers, including those on the free tier. AI bots like Bytespider and GPTBot are among the most active and frequently blocked.
Trending ️🔥
