How to Make Your Website Readable by ChatGPT, Claude and Perplexity

To make a website readable by AI assistants, serve your content as plain HTML, allow their crawlers in robots.txt, describe who and what the page is about with structured data, and give every piece of content its own address. None of this guarantees a mention, but it removes the reasons an assistant would skip you.

I went through this while building my own site, so what follows is what I actually did, in the order I would do it again.

Do AI crawlers read websites the same way Google does?

Not quite. Google runs JavaScript before it reads a page. Many AI crawlers do not. If your content only appears after a script runs, some of them see an empty page.

The fix is to send the content in the HTML itself. I built this site with Astro for that reason, but any setup that renders pages on the server or at build time works. A quick test is to open your page with JavaScript switched off. Whatever you can still read is roughly what a simple crawler gets.

Let the crawlers in

AI companies publish the names their crawlers use, and they check robots.txt before reading. If your file blocks them, or your host blocks them by default, you are invisible to that assistant.

My robots.txt allows everything and then names the AI crawlers explicitly, including GPTBot and ChatGPT-User from OpenAI, ClaudeBot from Anthropic, PerplexityBot, and Google-Extended. Naming them is not strictly required when you already allow all agents, but it makes the intent obvious to anyone who reads the file later, including me.

Give every piece of content a real address

My first version showed articles and case studies in overlays on a single page. It looked good and it was a mistake, because an overlay has no URL. A crawler cannot visit it and an assistant cannot cite it.

Each article and case study now has its own page and its own address, and they are all listed in a sitemap. If you want something quoted, it needs somewhere to be quoted from.

Say plainly who you are

Assistants answer questions about people and companies by matching facts across sources. Help them by stating the facts the same way everywhere: your name, what you do, where you are, and what you have built.

On the page, that means a clear sentence near the top instead of a clever tagline. In the code, it means structured data. I add a Person record with my name, role, company, university and profile links, and each article carries a BlogPosting record with its title, date and author. This is the same schema.org markup search engines have used for years.

What about llms.txt?

llms.txt is a proposed convention: a plain text file at the root of your site that summarises who you are and links to your important pages, written for a language model to read. I publish one, generated from the same data as the rest of the site.

I want to be careful here. It is a proposal, not a standard, and I have no evidence of how much any assistant relies on it. It costs almost nothing to add, so I added it, but I would not skip the earlier steps in its favour.

Write things worth citing

This matters more than any file. An assistant has read thousands of general explainers. What it has not read is your specific experience: the project you shipped, the number you measured, the decision you got wrong.

I try to open each article with a direct answer in two sentences, use headings that are phrased the way people ask, and include details that can be checked. Dates help too, because an assistant weighing two sources will often prefer the one it can tell is recent.

What nobody can promise

You cannot make an assistant mention you. These systems change often and none of them publishes a ranking formula. What you can do is make your site easy to read, easy to trust and worth quoting. That is the same work that makes a site good for people, which is probably not a coincidence.