Spec-compliant · llmstxt.org

Generate llms.txt
for any website

Crawl a site, extract its key pages, and get a spec-compliant llms.txt in seconds.

llms.txt — example output

# FastHTML
> FastHTML is a Python library for building server-rendered hypermedia applications with HTMX and Starlette.
## Docs
- [Quick start](https://docs.fasthtml.com/quickstart.md) A brief overview of many FastHTML features
- [HTMX reference](https://github.com/bigskysoftware/htmx/blob/master/www/content/reference.md) All HTMX attributes, CSS classes, and events
## Examples
- [Todo list application](https://github.com/AnswerDotAI/fasthtml/blob/main/examples/adv_app.py) Complete CRUD walkthrough showing idiomatic FastHTML patterns
## Optional
- [Starlette docs](https://gist.githubusercontent.com/jph00/.../starlette-sml.md) Starlette subset useful for FastHTML development

How it works

1. Crawl

Traverses the site via sitemap.xml, robots.txt, and link discovery — respecting robots.txt rules and prioritizing high-value pages.

2. Enrich

An LLM classifies each page, scores its importance, assigns it to a meaningful section, and writes a concise description.

3. Generate

The output is a single spec-compliant llms.txt file — ready to copy, download, or host at your own /llms.txt.

Works for any website

The output is always a spec-compliant llms.txt. Learn more about the llms.txt spec here.

Page history

Every llms.txt page you request is saved to your dashboard.

Stay up to date

Download a folder with the latest llms.txt versions in one click.