The paper introduces "Markdown for Agents," a new Cloudflare service designed to optimize content delivery for AI crawlers and agents. It addresses the challenge of AI systems needing structured data from the predominantly unstructured HTML-based web, which was originally built for human consumption.

The core problem identified is the inefficiency of feeding raw HTML to AI models. HTML, with its extensive tags, wrappers, and non-semantic elements, is highly token-intensive. For instance, an example HTML page is cited as requiring 16,180 tokens, while its Markdown equivalent uses only 3,150 tokens, representing an 80% reduction in token usage. This token bloat leads to increased computational costs, slower processing, and potentially less accurate AI outcomes. While AI pipelines commonly convert HTML to Markdown, this client-side or intermediary conversion adds computational overhead, cost, and complexity, and may not align with the content creator's original intent for AI consumption.

"Markdown for Agents" proposes a server-side solution to this problem by enabling real-time, on-the-fly content conversion at the edge of Cloudflare's network. The core methodology is based on HTTP content negotiation. When an AI system or agent requests a web page from a Cloudflare-enabled zone with "Markdown for Agents" activated, it includes an Accept HTTP header with text/markdown as one of its preferred content types (e.g., Accept: text/markdown, text/html). Upon receiving such a request, Cloudflare's network performs the following steps:

1. Header Detection: Cloudflare identifies the Accept: text/markdown header.
2. Origin Fetch: It fetches the original HTML version of the requested page from the website's origin server.
3. On-the-fly Conversion: Cloudflare's edge infrastructure automatically and efficiently converts the fetched HTML content into its Markdown equivalent.
4. Content Delivery: The converted Markdown content is then served directly to the requesting client.

The response to such a request includes specific HTTP headers:

• $Content-Type: text/markdown; charset=utf-8$: Indicates the content format.
• x-markdown-tokens: [token_count]: A custom header providing an estimated number of tokens in the Markdown document. This value is crucial for AI systems to calculate context window sizes or inform chunking strategies, optimizing downstream processing.
• $Content-Signal: ai-train=yes, search=yes, ai-input=yes$: This header, part of Cloudflare's Content Signals framework, explicitly signals to the agent or crawler that the content is permissible for AI training, search indexing, and general AI input, including agentic use cases. Future enhancements will allow content creators to define custom Content Signal policies.

This approach offers several benefits: significant token reduction for AI processing (leading to cost savings and improved efficiency), better AI results due to structured and concise data, and the ability for content creators to express their content's intended use for AI directly. The feature is already being utilized by popular coding agents like Claude Code and OpenCode. Cloudflare has also enabled this feature for its own Blog and Developer Documentation to encourage adoption.

Beyond this real-time conversion, the paper mentions other Cloudflare tools for Markdown conversion, such as AI.toMarkdown() within Workers AI for various document types, and a /markdown REST API for dynamic page rendering. Cloudflare Radar has also been updated to provide analytics on AI bot and crawler traffic, including content_type insights, allowing users to track the distribution of content types returned to AI agents and monitor the evolution of Markdown usage by specific bots like OAI-Searchbot. "Markdown for Agents" is currently available in Beta for Cloudflare Pro, Business, Enterprise plans, and SSL for SaaS customers, enabled via a simple toggle in the Cloudflare dashboard.