Cloudflare blocks AI crawlers by default and offers Pay Per Crawl payment model

July 02, 2025 at 4:00:17 AM - Trending πŸ”₯

TL;DR Cloudflare blocks AI crawlers by default and introduces Pay Per Crawl, letting content creators charge AI services for access. Using HTTP 402 responses and authentication, publishers can allow, charge, or block crawlers with flat prices. This empowers creators to monetize content at scale and supports future dynamic pricing and agent-based access. Pay Per Crawl is in private beta.

Cloudflare blocks AI crawlers by default and offers Pay Per Crawl payment model

Cloudflare, which powers about 20% of all web pages, is now blocking AI crawlers by default and introducing a new model called Pay Per Crawl that allows AI services to pay content creators for crawling their content. This model offers a third option beyond the binary choice of either fully open access or complete blocking, empowering content creators to control access and monetize their digital assets.

Pay Per Crawl Overview

Pay Per Crawl is currently in private beta and integrates with existing web infrastructure using HTTP status codes, particularly reviving the HTTP 402 Payment Required code. When an AI crawler requests content, it either presents payment intent via request headers to gain access (HTTP 200) or receives a 402 response with pricing details. Cloudflare acts as the Merchant of Record and provides the technical infrastructure for this payment system.

Publisher Controls and Pricing Options

Content owners have full control over monetization, setting a flat, per-request price across their entire site. They can choose to:

  • Allow free access to crawlers,
  • Charge a set price for access,
  • Block access entirely with no payment option.

Even if a crawler lacks a billing relationship with Cloudflare, publishers can still "charge" them, effectively blocking access but signaling the possibility of future payment arrangements. Publishers can also exempt specific crawlers from charges to allow free access or negotiate separate partnerships.

Decisions to allow or charge crawlers are enforced after existing security measures like WAF and bot management are applied.

Technical Implementation and Security

To prevent spoofing, crawlers must:

  • Generate an Ed25519 key pair and host the public key,
  • Register with Cloudflare providing their key directory URL and user agent,
  • Use HTTP Message Signatures with each request, including specific headers (signature-agent, signature-input, and signature).

Access and Payment Flows

There are two main flows for accessing paid content:

  1. Reactive (discovery-first): A crawler requests content and receives a 402 response with a crawler-price header if payment is required. The crawler can then retry with a crawler-exact-price header to agree to pay.

  2. Proactive (intent-first): A crawler includes a crawler-max-price header in the initial request. If the price is acceptable, the content is served with a 200 response and a crawler-charged header confirming the charge.

Only one price declaration header (crawler-exact-price or crawler-max-price) can be used per request. Invalid requests receive a 402 response.

Financial Settlement

Both crawler operators and content owners configure payment details in their Cloudflare accounts. Billing events are logged each time a crawler makes a paid request, and Cloudflare aggregates charges, bills the crawler, and distributes earnings to publishers.

Future Potential and Vision

Pay Per Crawl represents a shift in online content control, offering a programmatic way to value and manage digital assets. Cloudflare anticipates significant evolution, including:

  • Dynamic pricing based on demand or AI user base size,
  • Granular licensing for training, inference, search, or other AI uses,
  • Programmatic agentic paywalls where intelligent agents negotiate content access and budgets autonomously.

By using HTTP 402, Cloudflare envisions a future where intelligent agents can negotiate access to digital resources programmatically.

Getting Started

Pay Per Crawl is in private beta, inviting AI crawler operators interested in paying for content access and content creators wanting to charge for access to participate.

Cloudflare continues to protect networks, accelerate applications, and enhance security, supporting this new monetization approach as part of its broader Internet infrastructure services.

Have more questions on this topic? Ask our AI assistant for in-depth insights.

Read more from sources πŸ‘‡

The Only Digital Marketing Feed You'll Ever Need.

Stay informed your way. Tailored updates when and how you want them. 100% Free.

10,000+ Users

500+ Sources

1000+ Tools

Or

Related Posts

AI marketing workflows made simple

AI marketing workflows made simple

Featured
Markifact
Markifact

Verified Sponsor

Verified Sponsor

Markifact is a Verified Sponsor. Want to get featured here? Contact us.

Verified Sponsor
Google begins June 2025 core update rollout affecting all search rankings Trending ️‍πŸ”₯

Google begins June 2025 core update rollout affecting all search rankings

Google Search Central
Google Search Central

Official Source

Official Source

Google Search Central is a Official Source. The source has been verified by Swipe Insight team.

Official Source
Google Launches New Search Console Insights Report Trending ️‍πŸ”₯

Google Launches New Search Console Insights Report

Google Search Central
Google Search Central

Official Source

Official Source

Google Search Central is a Official Source. The source has been verified by Swipe Insight team.

Official Source
Google launches Doppl app to visualize outfits with AI on iOS and Android

Google launches Doppl app to visualize outfits with AI on iOS and Android

Google
Google

Official Source

Official Source

Google is a Official Source. The source has been verified by Swipe Insight team.

Official Source
Google tests feature to customize Top Stories with preferred news sources

Google tests feature to customize Top Stories with preferred news sources

Google
Google

Official Source

Official Source

Google is a Official Source. The source has been verified by Swipe Insight team.

Official Source
Google Tests New Emoji Answer Feature to Copy Emojis Directly from Search Results

Google Tests New Emoji Answer Feature to Copy Emojis Directly from Search Results

Len
Google launches Search Live with voice input in AI Mode for hands-free searching

Google launches Search Live with voice input in AI Mode for hands-free searching

Google
Google

Official Source

Official Source

Google is a Official Source. The source has been verified by Swipe Insight team.

Official Source
Google Revamps URL Structure Documentation for Better Search Performance

Google Revamps URL Structure Documentation for Better Search Performance

Google for Developers
Google for Developers

Official Source

Official Source

Google for Developers is a Official Source. The source has been verified by Swipe Insight team.

Official Source

Related Tools

Markifact logo

Markifact

Verified Tool

Verified Tool

Markifact is a Verified Tool. Want to get this badge? Contact us.

Verified Tool

Marketing Workflows Powered by AI

Featured
Marketing Auditor logo

Marketing Auditor

Verified Tool

Verified Tool

Marketing Auditor is a Verified Tool. Want to get this badge? Contact us.

Verified Tool

Automated audits for Google Ads and Analytics.

Get Featured Here

Showcase your tool in this list.

Contact Us
Ahrefs logo

Ahrefs

SEO tools to boost traffic and rank higher

SEO
Surfer SEO logo

Surfer SEO

SEO content creation and optimization made easy

SEO
Sitebulb logo

Sitebulb

Efficient website crawler for better SEO audits

SEO
Screpy logo

Screpy

AI-Powered SEO and Web Analysis Simplified

SEO
Blogify logo

Blogify

Convert multimedia to SEO-optimized blogs fast

SEO
Answer the Public logo

Answer the Public

Unlock Consumer Insights for Content Creation

SEO
SEO Writing AI logo

SEO Writing AI

AI-powered SEO content in 1 click

SEO
Thunderbit logo

Thunderbit

No-code AI apps and automations for business users

Workflow Automation

Get Featured Here

Showcase your tool in this list.

Contact Us