# ReadByBots

> ReadByBots answers two questions about a website: can AI assistants read it, and do they recommend it? It returns an AI Visibility Score from 14 measured checks, runs real buying questions through a web-connected assistant to see who gets named, and generates the llms.txt, llms-full.txt, agents.json and prompt.txt files that make a site readable. Free for your 5 most important pages; $19 one-time unlocks a full crawl of up to 500 pages.

## What llms.txt is

llms.txt is a plain-text file placed at the root of a website (example.com/llms.txt) that gives large language models a clean, curated map of the site's content. It follows the proposal at llmstxt.org. The companion file llms-full.txt inlines the full text of each page, so an assistant can answer questions about the site without fetching and parsing every page.

## Why a website needs one

Web pages are built for browsers: navigation, cookie banners, scripts and markup surround the actual content. When an AI assistant reads a page, that noise consumes context and buries the useful text. In ReadByBots scans, 95-99% of a typical page's bytes are removed before any content remains — a scan of stripe.com reduced 2.5 MB of HTML to 19 KB of Markdown. An llms.txt file gives assistants the content directly, which makes a site easier to cite accurately.

## Adoption status, honestly

No major AI company documents llms.txt as a signal it reads or ranks on. Google's Search Advocate has called it speculative rather than confirmed. Publishing it costs nothing and follows an open proposal, but it is a bet, not a guarantee — the measurable part of a ReadByBots report is the 24 technical checks and the live recommendation test, not the file's existence alone.

## What llms.txt does not do

Publishing llms.txt does not by itself get a site recommended by ChatGPT. Assistants answer from what web search returns — listicles, review sites, forums and competitors' pages — rather than by visiting a site directly. llms.txt makes a site readable once an assistant arrives. Getting named in the answer means appearing in the pages the assistant reads, which is what the citation check measures.

## The AI Visibility Score

Fourteen weighted checks across five categories, each reporting the measured value rather than a grade:

- LLM readiness: whether llms.txt exists, whether robots.txt blocks GPTBot, ClaudeBot or PerplexityBot, and whether content is present in server-rendered HTML rather than requiring JavaScript.
- Structure: sitemap.xml, canonical URL, hreflang on translated pages, and crawl depth.
- Entities: Schema.org markup, meta description, and page title.
- Headings: a single H1 per page and heading density.
- Noise: content-to-markup ratio and content depth in words.

## The citation check

ReadByBots generates the questions a buyer would ask — never naming the site — and puts them, twice each, to web-connected models built on ChatGPT, Gemini and Perplexity. Asking twice matters: a single query is noisy, so the report states a consistency split (e.g. "1 of 2 runs") rather than a false-certain yes/no. It reports whether the domain was named, which competitors were named instead, the assistant's own words, and every page it read. The paid report runs 5 questions and classifies the sources deciding each answer into listicles, review sites, forums, competitor pages and documentation, with the specific action for each. These are models standing in for the consumer apps, not the literal apps, which add memory and personalisation no API can reproduce.

## How the crawler behaves

It reads robots.txt first and honours the Disallow rules that apply to it, both the wildcard group and any group naming ReadByBotsBot. It enumerates URLs from sitemaps, or crawls same-origin links when no sitemap exists, then ranks pages so documentation, pricing, product and about pages are preferred over legal and archive pages. Translated duplicates are excluded. Extraction is deterministic HTML parsing, not a language model, so results are consistent and fast.

## Pricing

- Free: the 5 highest-value pages, the full visibility score, the crawl simulator, one live citation check, and llms.txt — no signup and no card.
- $19 one-time per site: full crawl up to 500 pages, all four files, 5 citation checks with the competitor list and action plan, and URL submission to Bing, Yahoo, Yandex, Naver, Seznam and Yep via IndexNow. Google does not participate in IndexNow.

## Browser extension

A Chrome extension reads the rendered page, which lets it handle JavaScript-only sites that server-side crawlers cannot read, and saves the current page as clean Markdown.

## Pages

- [ReadByBots — English](https://readbybots.com/): llms.txt generator and AI visibility check in English.
- [ReadByBots — Türkçe](https://readbybots.com/tr): llms.txt generator and AI visibility check in Türkçe.
- [ReadByBots — Deutsch](https://readbybots.com/de): llms.txt generator and AI visibility check in Deutsch.
- [ReadByBots — Français](https://readbybots.com/fr): llms.txt generator and AI visibility check in Français.
- [ReadByBots — Español](https://readbybots.com/es): llms.txt generator and AI visibility check in Español.
- [ReadByBots — Português](https://readbybots.com/pt): llms.txt generator and AI visibility check in Português.
- [ReadByBots — Italiano](https://readbybots.com/it): llms.txt generator and AI visibility check in Italiano.
- [ReadByBots — Nederlands](https://readbybots.com/nl): llms.txt generator and AI visibility check in Nederlands.
- [ReadByBots — Русский](https://readbybots.com/ru): llms.txt generator and AI visibility check in Русский.
- [ReadByBots — 日本語](https://readbybots.com/ja): llms.txt generator and AI visibility check in 日本語.
- [ReadByBots — 中文](https://readbybots.com/zh): llms.txt generator and AI visibility check in 中文.
- [ReadByBots — العربية](https://readbybots.com/ar): llms.txt generator and AI visibility check in العربية.

---
Generated and maintained by readbybots.com