SEJ Live
  1. SEJ
  2.  ⋅ 
  3. SEO

How To Deploy llms.txt When Your CMS Won’t Cooperate

Can AI crawlers follow redirects to find your llms.txt file? Here's how to deploy llms.txt on restrictive CMS platforms using 301 redirects, and what to watch for.

If your CMS blocks root-directory file uploads, a 301 redirect can serve your llms.txt to AI crawlers, but the details matter.

Many CMS platforms, including Shopify, Webflow, and GoHighLevel, don’t allow users to upload files to the site root directory. That creates a problem for anyone trying to deploy an llms.txt file at the standard /llms.txt path where AI crawlers expect to find it.

A recent discussion in r/SEO raised the question directly: will AI agents follow a redirect to reach llms.txt? The short answer is yes, AI crawlers generally follow standard HTTP redirects. But the implementation details, from MIME types to redirect chains, can make or break whether the file actually gets parsed.

Here’s what practitioners need to know about deploying llms.txt via redirect, and where the approach can go wrong.

Do AI Crawlers Follow Redirects?

Standard AI crawlers like GPTBot, ClaudeBot, and PerplexityBot follow HTTP redirects the same way traditional web crawlers do. OpenAI’s documentation on AI agent link safety confirms that its agents follow links and handle redirects as part of normal browsing behavior.

Cloudflare’s AI Crawl Control toolkit provides additional infrastructure-level confirmation. The feature issues 301 redirects to verified AI training crawlers based on canonical tags, which means major AI crawlers are already encountering and following redirects at scale across Cloudflare’s network.

The llmstxt.org specification itself anticipates redirect use cases, particularly for multi-domain setups. The key requirement: the final destination must return a 200 status code with a Content-Type: text/plain header. If the destination serves HTML-wrapped content or returns the wrong MIME type, AI parsers may fail silently.

How To Set Up The Redirect

The workaround is straightforward. Host your llms.txt file in a subdirectory or media folder that your CMS does allow (for example, /assets/llms.txt or a CDN-hosted URL). Then configure a single 301 redirect from /llms.txt to that location.

On platforms like Webflow, this can be done through the built-in redirect settings. On Shopify, you may need a redirect app or a Cloudflare Workers rule. The critical point is keeping the redirect chain to a single hop. A 301 to a 302 to a 200, for example, increases the risk that a bot drops the request before reaching the file.

Once the redirect is in place, verify two things. First, confirm the final URL returns Content-Type: text/plain, not text/html. Second, check your server logs to confirm that GPTBot, ClaudeBot, and PerplexityBot are actually completing the redirect and reaching the file. Search Engine Land recommends log file analysis as a baseline practice for understanding how AI crawlers interact with your site.

Temper Your Expectations

Deploying llms.txt is a low-cost step, but it is not a ranking lever. An SE Ranking study of 300,000 domains found no measurable effect on AI citations from having an llms.txt file in place. A separate Ahrefs study of 1.4 million ChatGPT prompts found that traditional authority signals, not llms.txt, drive citation patterns.

No major AI provider has published official documentation confirming that its systems consume llms.txt files. All current evidence is observational, based on log analysis and crawler behavior rather than stated policy. Google’s John Mueller has publicly called the broader concept of bot-specific markdown delivery “a stupid idea.”

SEJ’s deep dive on the llms.txt proposal noted that adoption remains thin, with under 3% of sites deploying the file and a significant portion of existing files being auto-generated plugin stubs. The architecture beyond llms.txt is still evolving, and practitioners should treat deployment as future-proofing rather than an optimization with proven returns.

What To Do Now

  • If your CMS blocks root-directory uploads, host llms.txt in a subdirectory or media folder and set up a single 301 redirect from /llms.txt to that location.
  • Verify the final destination returns Content-Type: text/plain (not text/html) and a 200 status code. Use curl or a tool like Screaming Frog to check headers.
  • Analyze server logs to confirm GPTBot, ClaudeBot, and PerplexityBot are completing the redirect. Don’t assume the redirect works for bots just because it works in a browser.
  • Avoid redirect chains. Keep it to one 301 hop to minimize the chance of bots dropping the request.
  • Treat llms.txt as low-cost future-proofing, not a citation or ranking strategy. No AI provider has officially committed to consuming the file.

Looking Ahead

The redirect workaround solves a real deployment problem for practitioners on restrictive platforms. AI crawlers follow standard HTTP redirects, and Cloudflare’s infrastructure-level support for AI crawler redirects adds confidence that this approach works at scale.

That said, the broader llms.txt ecosystem remains in early stages. Until a major AI provider officially documents llms.txt consumption, or until independent research shows a measurable effect on AI citations, the file is best understood as a preparatory step. Practitioners who want to understand how AI agents interact with your site architecture should focus on log analysis and structured content as higher-priority investments.

 

AI-generated first-pass scaffolding. This draft was produced by Search Engine Journal’s newsroom automation as a starting point for a writer. Rewrite before publishing.

Research notes (review and remove before publishing)

The bot collected this context while writing. Skim, verify, then delete this whole section before publish.

Headline alternatives

  1. Can AI Agents Follow Redirects To Your llms.txt File?
  2. How To Deploy llms.txt On Restrictive CMS Platforms
  3. llms.txt Redirect Workaround Validated By Cloudflare

Primary sources cited

Suggested internal links (prior SEJ coverage)

Competitor coverage seen

Practitioner pulse

Practitioner opinion is split: technical SEOs see llms.txt as a reasonable future-proofing step (especially with redirect workarounds now validated), but skeptics note no major AI provider officially commits to consuming the file and adoption remains under 3%. Most social posts are from vendors/agencies promoting guides rather than sharing empirical results.

LinkedIn:

X / Twitter:

Background

The llms.txt file is a community-proposed standard (llmstxt.org) that provides AI agents with clean, markdown-formatted site context at a well-known URL path. Adoption remains thin — SE Ranking’s 300K-domain analysis found no measurable effect on AI citations (searchenginejournal.com). Google’s John Mueller has publicly criticized related bot-specific markdown delivery as ‘a stupid idea’ (searchenginejournal.com). Cloudflare recently launched ‘Redirects for AI Training’ within its AI Crawl Control toolkit, which issues 301 redirects to verified AI training crawlers based on canonical tags — providing infrastructure-level confirmation that major AI crawlers (GPTBot, ClaudeBot, ByteSpider) follow standard HTTP redirects (Gemini grounded synthesis). The llmstxt.org spec itself anticipates redirects for multi-domain setups but requires the final destination to return text/plain with a 200 status code.

Open questions for follow-up coverage

  • Does Google’s AI Mode / AI Overviews crawler specifically consume llms.txt, or does Google rely solely on its existing indexing infrastructure?
  • What is the exact Cloudflare ‘Redirects for AI Training’ feature scope — does it apply only to canonical-tag redirects or also to arbitrary redirect rules like /llms.txt → /media/llms.txt?
  • Has any AI provider (OpenAI, Anthropic, Perplexity) published official documentation confirming llms.txt consumption, or is all evidence still observational/log-based?
  • Are there measurable differences in redirect-following behavior between training crawlers (GPTBot, ClaudeBot) and retrieval/answer crawlers (ChatGPT-User, Claude-User)?

⚠ Unknown-tier sources surfaced (vet before quoting)

Image search query

“server redirect diagram web technology”

Flags

degraded research: preflight

Drafter’s writer notes

FACTCHECK_FLAGS_GO_HERE

Research stage was partial (preflight degraded). Verify the following:

  • The Cloudflare AI Crawl Control / ‘Redirects for AI Training’ feature details are synthesized from the research brief’s background section (attributed to Gemini grounded synthesis). Confirm this feature exists and works as described by checking Cloudflare’s blog or docs directly.
  • The Reddit thread that triggered this story had no comments returned in the source scrape. The article avoids quoting the thread directly but links to it for context. Verify whether substantive replies have since appeared.
  • No tier-A primary source provided a verbatim quote suitable for a blockquote. The OpenAI link confirms redirect-following behavior but in the context of link safety, not llms.txt specifically. Writer may want to pull a direct quote from the OpenAI page if one fits.
  • The claim that adoption is under 3% comes from social pulse sources (digitalstrategyforce.com, unknown tier). The SE Ranking study on SEJ is the stronger citation for low adoption/no effect.
  • No AI provider has published official docs confirming llms.txt consumption. This is stated in the article but worth re-verifying at publish time.

Unknown-tier sources not used in body: gracker.ai, digitalstrategyforce.com, infostarbase.com, qwestyon.com, tryanalyze.ai, nothumansearch.ai, just-search.online, presenc.ai. These were filtered per source-tier rules.

Potential follow-up angles:

  • Empirical test: deploy llms.txt via redirect on a test site and monitor AI crawler log data over 30 days.
  • Interview Cloudflare on the scope of their AI Crawl Control redirect feature.
  • Track whether any AI provider officially documents llms.txt support in 2026.

Fact-check pass: No flagged claims.

Category SEO
Read Full Bio
SEJ STAFF Matt G. Southern Senior News Writer at Search Engine Journal

Matt G. Southern, Senior News Writer, has been with Search Engine Journal since 2013. With a bachelor’s degree in communications, ...