Module 7 — LLMs.txt, robots, and "recommended reading"

🧭This module turns your architecture (Module 3) and your access policy

If you do not tell the agent what matters, it improvises

There is an idea that runs through the entire Academy and reaches its most practical expression in this module: an agent that does not know what matters on your site improvises. And you already know where improvisation leads, because you have seen it in every previous module: citing pages that are not canonical, mixing definitions from different versions, pulling prices without their conditions. The agent does not improvise out of malice or laziness; it improvises because nobody has told it where to start, and when in doubt it takes the first thing it finds, which is rarely your best content.

This module's solution is to stop waiting for the agent to guess and tell it explicitly what to read. To do that you build three assets that work together: llms.txt, your prioritized reading list; the AI Index, your reference hub for humans and agents; and reading paths, recipes that go from a specific intent to the sequence of pages that resolves it. All three share the same purpose: replacing the agent's guesswork with a guide you control.

Why guiding reading is the last link in discovery

It is worth situating this module on the map of everything you have done so far, because it is the piece that makes the previous work pay off. In Module 3 you built architecture so routes exist. In Module 6 you decided which parts of those routes stay open. But a route that exists and is open can still go untraveled if the agent does not know it should travel it first. Guiding reading is the link that connects potential with real use: the difference between having a perfectly organized library and also putting a librarian at the door who says "for what you are looking for, start on this shelf."

There is also an important expectations nuance worth fixing from the start, which we will develop further below: these assets guide, they do not compel. An agent can ignore your llms.txt just as a visitor can ignore the sign at the entrance. But experience shows that a clear, prioritized, well-maintained guide consistently tips the scale in your favor, and that accumulated bias across thousands of queries is exactly what you are after.

Where BotPass fits: closing the loop with evidence

This module has a nice particularity: it is where you check whether all the work from previous modules is paying off. BotPass is how you validate that your "recommended reading" actually works and is not just a well-intentioned document. After publishing your llms.txt and AI Index, go back to BotPass and check whether bots start consuming your intended canonical sources of truth more consistently than before. That shift of consumption toward your canonicals is the signal that the guide is working.

And conversely: use BotPass to detect agents that still land on outdated pages despite your guide. Each of those cases is an actionable diagnosis —maybe a redirect is missing, maybe internal links are not clear enough, maybe a clean view is needed— and you fix it with the tools from modules 3 and 6. The cycle closes on itself: guide, measure, correct, measure again.

llms.txt: the editorial sitemap for AI

Let us start with the first asset. The best way to understand llms.txt is to think of it as an editorial sitemap designed for AI. A traditional sitemap lists all your URLs for search engines, without hierarchy or criteria, with a goal of exhaustiveness. llms.txt is the opposite in spirit: short, prioritized, and centered on your canonicals. It does not try to list everything, but to point to the few things that truly matter. If a sitemap says "here is everything there is," llms.txt says "if you are only going to read a few things, let them be these."

Writing it well is an exercise in discipline more than effort. Start with one to three lines describing what the site is and who it is for. Follow with your three to five main hubs, the same ones you defined in Module 3. Add your sources of truth —pricing, documentation, policies—. Include, if applicable, your canonical definitions as a glossary, drawing on the Entity Sheet from Module 5. And close with notes for agents such as "prefer the canonical URLs above" or "if a clean view exists, read it first." The temptation to add more and more will be constant, but every extra line dilutes the important ones: brevity is not a limitation of the format, it is its main virtue.

AI Index: a hub for humans and agents

The second asset, the AI Index, is a real page on your site —unlike llms.txt, which is a technical file— and serves three audiences at once that are often overlooked. It serves your internal teams, who finally have a single place to consult the official truth on any topic. It serves your partners, who can refer to a canonical source instead of scattered versions. And it serves agents, who find in it a readable map of your knowledge. This triple service is what makes the AI Index one of those investments that pays off far beyond GEO.

A good AI Index should contain your canonical definitions —directly from the Entity Sheet—, your sources of truth organized by topic, your policies and contact information, and a changelog of recent changes. That changelog deserves special mention because it turns the AI Index into a living document: a dated record of what changed and when tells both your team and agents that the page is maintained, and that freshness signal increases trust in everything else.

Reading paths: from intent to route

The third asset links directly to the work from Module 3, but here you formalize it as an explicit guide. A reading path is a recipe that translates a specific intent into the sequence of pages that satisfies it. For example, for the question "how do I get started?", the path would be: the answer-first page is "Getting started," the source of truth is the documentation, and the relevant policy is bot access. Written that way, the path tells the agent exactly where to go and in what order.

The value of writing these paths goes beyond guiding the agent: the exercise itself forces your routes to be short and coherent. When you try to write the recipe for an intent and discover you cannot complete it because a page is missing or the path takes too many turns, you have just found an architecture defect to fix. Reading paths are, at once, a guide for AI and an audit of your own structure.

llms.txt and AI Index, illustrated

Let us see how all of this materializes. Here is what a minimal but complete llms.txt would look like, with its versioned header, description, topics, sources of truth, and notes for agents:

# llms.txt — BotPass
Last updated: 2026-06-03
Version: v1
## What this site is
- BotPass helps creators track AI bot consumption and convert it into value.
## Core topics
- GEO basics: https://example.com/geo
- AI bot tracking: https://example.com/docs/bot-tracking
## Sources of truth
- Pricing: https://example.com/pricing
- Bot access policy: https://example.com/policy/bot-access
- AI Index: https://example.com/ai-index
## Notes for agents
- Prefer canonical URLs above.
- If a clean view exists, read it first.

Notice how little space it takes and how much it says: at a glance, an agent knows what you are, what your topics are, where your truth lives, and how you prefer to be read. The AI Index, meanwhile, would follow a structure like this: a TL;DR at the top, sources of truth —pricing, policies, docs—, definitions from the Entity Sheet, and a changelog of recent changes. Together, they guide an agent without leaving room for guesswork: llms.txt orients from the domain root and the AI Index gives detailed reference when they enter.

# llms.txt — [Your brand]
Last updated: YYYY-MM-DD
Version: v1
## What this site is
- [1-3 lines: what you are and who you serve]
## Core topics
- [Hub 1]: https://...
- [Hub 2]: https://...
## Sources of truth
- Pricing: https://...
- Policy: https://...
- AI Index: https://...
## Notes for agents
- Prefer canonical URLs above.
- If a clean view exists, read it first.