Published
·
5 min
read
How to get your website ready for AI search and AI agents
AI assistants read your site before people do. Here is what actually helps: crawler access, clean structure, schema, answerable content, and an honest verdict on llms.txt.
TL;DR
AI assistants now read your website before many of your customers do, and they read it differently.
Most of the work is technical hygiene you already know: server-rendered HTML, clean headings, structured data.
Choose deliberately which AI crawlers you allow. Blocking all of them removes you from AI answers.
Write answer-first content with specific facts, dates and numbers an assistant can quote.
llms.txt takes twenty minutes and proves nothing yet. Add it if you want, but do not build a strategy on it.
Advertising disclosure: this article contains affiliate links to Framer. If you sign up through them we may earn a commission at no extra cost to you. It does not change what we recommend.
Your website now has two audiences
People still visit your site. So do machines, and far more often than before. Google answers questions directly in AI Overviews and AI Mode, ChatGPT and Perplexity fetch pages while a user waits, and coding and research agents pull pages apart to extract facts.
That changes what a visit is worth. You get fewer clicks on questions an assistant can answer itself, and better-qualified clicks on the rest, because the person arriving has already been told what you do. The site still has to convince a human once they land. It now also has to survive being read by something that cannot see your design.
Decide which AI bots you let in
AI crawlers do different jobs, and the difference matters. Some retrieve pages to cite them in answers. Others collect content to train models. According to OpenAI's crawler documentation, OAI-SearchBot surfaces sites in ChatGPT search results and is not used for model training, while GPTBot is the training crawler, and ChatGPT-User fetches a page when someone asks about it.
For a service business that wants to be recommended, blocking everything is the wrong default. If you block retrieval crawlers, you are not protecting your content. You are removing yourself from the place your buyers are asking questions.
Two practical points:
Treat training access and search access as separate decisions. You can allow one and refuse the other.
Named groups in robots.txt do not inherit rules from User-agent: *. If you add a restriction later, add it to each named group too, or it silently will not apply.
Make the page readable without JavaScript
Assistants read markup, not your layout. If the main content only appears after scripts run, you are gambling on whether a given crawler executes them.
Check the basics on your key pages:
The main copy is in the HTML on first load, not injected afterwards.
One H1 per page, with H2s that describe the section rather than sell it.
Text is real text. Headline copy baked into an image is invisible.
Nothing important hides behind a click, a tab or a hover.
If you build your site in Framer or in Webflow, most of this is handled for you: pages are served as static HTML. The usual failures are self-inflicted. Hero copy exported as an image, key content locked in a component that only renders on interaction, or CMS pages with thin placeholder text left in place.
Structured data is the fastest win
Schema markup tells a machine what your page is about instead of leaving it to inference. For a service business, a small set covers most of it: Organisation, LocalBusiness, Service, Article, FAQPage and BreadcrumbList.
Two rules keep it useful. Schema must match what a visitor can see on the page, and your core facts, such as business name, address and description, must be identical everywhere you publish them. Contradicting yourself across your site, your Google Business Profile and LinkedIn gives a model a reason to trust a competitor instead.
Schema does not buy you a citation. Google's guidance on generative AI features treats it as a way to help systems understand your content, not as a lever that guarantees placement. Anyone promising otherwise is selling something.
Write content an AI can quote
The pages that get cited tend to answer a question in the first two sentences and then explain. That is also better for humans, which is why it is worth doing.
What helps:
Put the answer first, then the reasoning. Do not build up to it.
One question per section, with a heading that matches how someone would ask it.
Specifics over adjectives. "Live in four weeks" can be quoted. "Fast turnaround" cannot.
Dates on anything that changes, so a model can judge whether you are current.
Tables for comparisons, because they survive extraction intact.
llms.txt: cheap to add, unproven so far
llms.txt is a proposed file at the root of your domain listing your important pages in Markdown, on the same logic as robots.txt and sitemap.xml. The idea is sensible. The evidence is thin.
Adoption is real but narrow. SE Ranking's study of 300,000 domains found roughly 10% had one, and no major AI search engine has confirmed that it uses the file for ranking or citation. Developer tools are the exception: coding agents do fetch documentation through it.
Our position: add it if your site has documentation or a large resource library, because it costs almost nothing. Do not treat it as AI-readiness, and do not let it displace the work in the sections above.
Measure it, or you are guessing
Most AI visibility is invisible by default. Someone can be told about your business inside a chat window and never click. You can still see part of the picture.
Separate AI referrals in GA4 so traffic from ChatGPT, Perplexity and similar sources is not buried in direct.
Check server logs for AI user agents to confirm which crawlers actually reach you.
Ask the main assistants what they say about your business and your category, and record the answers so you can spot changes.
Expect small numbers and high intent. Judge the channel on enquiries, not sessions.
Frequently asked questions
Will AI search kill my website traffic?
It reduces clicks on questions an assistant can answer alone, particularly definitions and simple how-tos. Pages tied to a decision, a price or a local supplier hold up better. The site matters more, not less, because fewer visitors arrive and each one is further along.
Should I block AI bots from my site?
Only with a reason. Blocking training crawlers is a defensible choice. Blocking retrieval crawlers takes you out of AI answers entirely, which, for most service businesses, is the opposite of the goal.
Do I need llms.txt?
No. It is optional and its effect is unproven. If you have documentation or many resource pages, add it. Otherwise spend the time on structure and content.
Does schema markup guarantee I get cited?
No. It helps systems understand your content and reduces guesswork, and no one can promise a citation, a rich result or a ranking. Treat any such promise as a warning sign.
Is a Framer or Webflow site fine for AI, or do I need custom code?
Both are fine. They output static HTML and support custom schema. The limiting factor is almost always how the content is written and structured, not the platform.
