Markdown versions of pages for AI agents: how we would build them automatically

Why markdown at all?

Language models pay and think in tokens. Every <div class="…">, every line of CSS and every hidden menu uses up context that should go to the content. Markdown is the opposite: a heading is #, a list is -, a link is [text](url). The structure stays, the noise goes.

Cloudflare measured this on its own announcement post: the same page is 16,180 tokens as HTML and 3,150 tokens as markdown. That is about 80 % less. For an agent it means a faster answer, a smaller bill and less room for misreading.

tokens as HTML

16,180tokens as HTML

tokens as markdown

3,150tokens as markdown

less to read

−80 %less to read

More important than the saving itself is that agents already ask for markdown. According to Cloudflare, Claude Code and OpenCode send an Accept: text/markdown header in their requests. That is an old, tidy HTTP mechanism called content negotiation: the same URL, and the server picks the form the client prefers.

GET /usluge/hosting HTTP/2
Host: primjer.hr
Accept: text/markdown, text/html;q=0.9

Three approaches that exist today

1. llms.txt: an index for machines

Jeremy Howard proposed llms.txt on 3 September 2024. It is a markdown file at the site root (example.com/llms.txt) with a short description and a list of the most important links, grouped under headings. The idea is that an agent does not have to crawl the whole site to understand what is on it.

An honest status report: no major provider (OpenAI, Google, Anthropic, Meta) has publicly confirmed that its crawlers read llms.txt. Google’s John Mueller compared it to the keywords meta tag, and an Ahrefs log analysis showed that AI crawlers mostly do not even request the file. On the other hand, on 5 May 2026 Google added an llms.txt check to Lighthouse, in the group of audits for agents. If the file is missing, the audit is marked “not applicable”; it only fails when the server returns an error.

2. Conversion at the edge (Cloudflare)

On 12 February 2026 Cloudflare introduced Markdown for Agents. When a request asks for text/markdown, Cloudflare fetches the HTML from the origin, converts it to markdown at the edge and returns it with Content-Type: text/markdown and Vary: Accept. It adds YAML frontmatter at the top (title, description, image), strips navigation, headers, footers and scripts, and keeps JSON-LD in a separate block. An x-markdown-tokens header estimates the token count.

It is elegant, but it has limits that matter a lot for us:

• It is only available on Pro, Business and Enterprise plans. Most of our clients’ zones are on the Free plan.
• If the origin sends no signal of its own, Cloudflare adds Content-Signal: ai-train=yes, search=yes, ai-input=yes. That means the content explicitly allows model training, which not every client wants.
• The conversion is generic. It does not know that a WooCommerce price matters more than a cookie banner, or which sections of a page are important.
• Converted responses lose ETag and Last-Modified, so conditional requests do not work. The limit is 6 MiB of HTML.

3. Markdown from the origin

The third way is for the site itself to generate the markdown. WordPress knows the structure of the content better than any converter: it knows which block is a heading, which is a table, what is a price and what is decoration. If markdown is built from blocks rather than from finished HTML, the result is cleaner and under our control. This is the approach we propose.

How it would work on our platform

The good news is that half of the work already exists. Our MK Cache has a branch (version 1.3) that stores the response on disk by its real Content-Type header: index.html for browsers, index.md for markdown. Nginx picks the file by the Accept header. What is missing is the markdown generator and a rule on Cloudflare.

1. WordPress (a new MK Markdown mu-plugin): when a request asks for text/markdown or the URL ends in .md, the page is built from Gutenberg blocks into markdown. Headings, paragraphs, lists, tables, quotes and code map directly. Images get their alt text and the media subdomain URL. Frontmatter goes on top: title, description, canonical URL, language, modified date and hreflang versions.
2. MK Cache: a response with Content-Type: text/markdown is stored as index.md next to index.html. Clearing the page cache clears both versions.
3. Nginx: map $http_accept $preferred_ext picks .md or .html, and Vary: Accept tells every cache that two versions exist.
4. Cloudflare: since 2 July 2026, Cache Rules honour the Vary header on all plans, including Free. We enable Vary with the normalize action for Accept, so the edge keeps HTML and markdown apart.
5. llms.txt: generated from the sitemap, with the site description from the Hub and links to the .md versions.

# http {}
map $http_accept $preferred_ext {
    default           ".html";
    "~*text/markdown" ".md";
}

# location /
set $cache_file /wp-content/cache/mk-cache/all${uri}index$preferred_ext;
types { text/markdown md; }
add_header Vary "Accept" always;
try_files $cache_file $uri $uri/ /index.php?$args;

Example output

---
title: Lanena košulja Classic
description: Košulja od 100 % lana, šivana u Hrvatskoj.
canonical: https://primjer.hr/proizvod/kosulja-classic/
lang: hr
updated: 2026-10-28
alternate:
  en: https://primjer.hr/en/product/shirt-classic/
---

# Lanena košulja Classic

| Veličina | Cijena (s PDV-om) | Dostupno |
|---|---|---|
| M | 59,00 € | da |
| L | 59,00 € | 2 komada |

![Košulja Classic, bijela](https://media.primjer.hr/…/kosulja.avif)

[Dodaj u košaricu](https://primjer.hr/proizvod/kosulja-classic/)

Who may read: signals and robots

Markdown makes reading easier, so it is fair to ask whether everyone should be allowed to read at all. On 24 September 2025 Cloudflare introduced Content Signals: three statements in robots.txt and headers. search means indexing for a search engine, ai-input means using the content as input for an answer (RAG, grounding), and ai-train means training or fine-tuning a model. On more than 3.8 million domains with managed robots.txt, Cloudflare sends search=yes, ai-train=no by default.

Today our platform blocks known LLM crawlers at the Nginx level, except for sites that have explicitly allowed them. The proposal is to turn this into a clear per-site setting in the Hub, with three switches matching the signals. As the default we would suggest search=yes, ai-input=yes, ai-train=no: an agent may read the page to answer a user, but the content is not training material without the client’s consent.

Content signals are not technical countermeasures against scraping. Some companies may ignore them, so they should be combined with WAF rules and bot management.

Why this is an add-on for the top plan

For a five-page business card site the difference is negligible. It makes sense for content-heavy sites: portals, documentation, catalogues and web shops. There, an agent comparing products needs the price, availability and variants in a tidy table, not a JavaScript gallery. That is why markdown would be part of the Enterprise plan, in three phases:

1. Phase 1: llms.txt and .md versions of pages and posts. Negotiation through the Accept header and storage in MK Cache.
2. Phase 2: WooCommerce. A product in markdown gets its price, VAT, availability and variants as a table, plus JSON-LD.
3. Phase 3: the Hub. An “AI agents” tab shows how many requests asked for markdown, which agents visit (from the Nginx logs) and how many tokens were saved.

What we do not promise

There is no evidence that a markdown version by itself raises your position on Google or in AI assistant answers. What you reliably get is a page an agent reads faster, cheaper and more accurately, with a clear statement about whether the content may be used for training. Once phase one runs on a few client sites, we will publish measured numbers: how many agents actually asked for markdown and how many tokens were saved.