Perplexity uses stealth crawlers that change user agents and IPs to bypass no-crawl directives and ignore robots.txt files. Cloudflare blocked these undeclared crawlers after tests showed Perplexity accessed restricted content despite blocks. Unlike OpenAI, Perplexity evades detection and violates crawling norms. Cloudflare added rules to block such AI crawling, protecting customers and promoting transparent, respectful bot behavior.
Cloudflare blocks AI crawlers by default and introduces Pay Per Crawl, letting content creators charge AI services for access. Using HTTP 402 responses and authentication, publishers can allow, charge, or block crawlers with flat prices. This empowers creators to monetize content at scale and supports future dynamic pricing and agent-based access. Pay Per Crawl is in private beta.
Run a Meta Ads audit with 100+ data points using a structured, automated checklist. Review campaigns, ad sets, ads, creatives, targeting, budgets, and delivery settings to spot hidden issues and missed optimizations. Built for agencies and teams managing multiple accounts who want consistent audits, faster reviews, and clear, actionable insights—without manual digging.
Cloudflare has released AI Audit, a tool for website owners to control AI bots accessing their content. It identifies AI bots, provides control to block or allow them, and offers analytics on bot activity. Key features include free bot control, detailed analytics, and legal support. A future feature will allow content pricing for AI access. This helps site owners manage AI scraping, which can impact traffic and content value.
Cloudflare has launched a free tool to prevent bots from scraping websites for AI model training. Despite some AI vendors allowing site owners to block bots via robots.txt, not all bots comply. Cloudflare's tool uses advanced detection models to identify and block evasive AI bots. The tool is available to all customers, including those on the free tier. AI bots like Bytespider and GPTBot are among the most active and frequently blocked.
Trending ️🔥
