How to Optimize Your Website for AI Agents That Browse and Buy
- Level: Advanced
- #AI Search
- #Technical SEO
- #E-commerce
- #Advanced

Optimizing a website for AI agents means making sure a bot acting for a person can get in, understand the page, compare offers and move a task forward. It comes down to three things: not blocking the agent by accident in your firewall, writing HTML that describes buttons and forms without ambiguity, and keeping product data identical on the page, in schema and in the feed. It is the "action" side of SEO for AI search.
Quick summary
- An AI agent is not a crawler: it visits your site because a person asked for a task, not to index pages or train models.
- Bots acting on a user's request often ignore robots.txt. Whether they get in is decided by your firewall or CDN.
- ChatGPT's cloud browser signs its requests, and Google documents the
Google-Agentuser agent. You can verify both. - In-chat buying has pulled back: OpenAI now sends shoppers to finish on the merchant's site, and Google's UCP checkout is in early access in a few countries.
- Semantic HTML (a button that is a
<button>, a field with a<label>) serves screen readers and agents at the same time. - None of this guarantees that an agent will pick your store.
How to read this: if you own a site or a store, read the first three sections and the checklist. If you are on the technical team, the sections on firewalls, HTML, protocols and logs include headers and code. Terms like crawler and WAF are in the SEO, AEO and GEO glossary.
What changes when the visitor is an AI agent?
The purpose of the visit changes. A crawler reads to store, and an agent reads to act: it opens the page, clicks, fills in fields, compares and goes back to the user with an answer or a booking ready for approval. If it cannot get past a step, the task moves on to another site.
In plain English: picture a personal assistant walking into your store with someone else's shopping list. The window display does nothing for them. They read labels, check price and delivery time, and if the door is locked or the label is unreadable, they walk next door. They don't complain. They just don't buy your product.
This is already live. According to OpenAI's help article on the cloud browser, ChatGPT Work, on paid plans other than Free and Go, has its own browser that clicks and fills out forms, including on sites that require sign-in, and asks for confirmation before actions such as booking or paying.
On Google's side, Chrome auto browse lets Gemini carry out multi-step tasks. As of November 2026, it works only on desktop, in the U.S., on paid Google AI plans, using either the person's local Chrome or a remote browser.
For SEO professionals: in auto browse's local mode, Chrome's help page describes no way for a site to identify the traffic, which arrives through the user's own browser. Cloud browsers (ChatGPT Work and
Google-Agent) do have documented identification. Segment your logs and WAF rules with that difference in mind.
Which AI bots visit your site, and how should you treat each one?
There are six groups. You control the first two through robots.txt, the user-action ones through your firewall, and the last one never visits at all. The table reflects what was documented on November 5, 2026.
| Bot type | What it does | How to identify it | What to allow |
|---|---|---|---|
| Training crawler | Collects content that may train models | GPTBot, ClaudeBot, Google-Extended token | A policy decision, in robots.txt |
| Search crawler | Builds the index behind answers with sources | OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot plus official IPs | Allow, if you want to appear in answers |
| On-demand fetcher | Opens a page when a user asks a question | ChatGPT-User, Claude-User, Perplexity-User plus official IPs | Allow public pages |
| Cloud browser agent | Browses, clicks and fills out forms for a user | ChatGPT: signed Signature-Agent header. Google: Google-Agent plus official IPs | Allow in the WAF after verifying the signature or IP |
| Agent in the user's browser | Uses the person's own Chrome and sessions | No documented identification | Nothing to allow: HTML and flow are what count |
| Commerce feed or protocol | Does not visit: the store sends the data | ACP feed (OpenAI), Merchant Center and UCP (Google) | Send data that matches the page |
Google's documentation on user-triggered fetchers says these bots generally ignore robots.txt because a person requested the fetch. Google-Agent is listed there, used by agents hosted on Google infrastructure to navigate and act on a user's request, with IPs published in user-triggered-agents.json. OpenAI says something similar about ChatGPT-User, while Anthropic states that Claude-User respects the file. Each token is covered in the guide to robots.txt and AI crawlers.
Is your firewall blocking agents without you knowing?
It might be. Anti-bot rules, verification challenges and "unknown bot" blocks stop the agent before any content loads, and none of it shows up in Search Console. OpenAI itself warns that some sites block its cloud browser.
To allow agents without opening the door to everyone, use cryptographic identification. According to OpenAI's allowlisting instructions, the cloud browser signs requests with Web Bot Auth, built on the HTTP Message Signatures standard (RFC 9421). Each request carries Signature, Signature-Input and Signature-Agent, the last one set to "https://chatgpt.com", and the public keys live at https://chatgpt.com/.well-known/http-message-signatures-directory.
- Cloudflare: treats the traffic as a signed agent (detection ID
129220581). Create a WAF rule that allows that ID. - Akamai, HUMAN and Vercel: recognize "ChatGPT Agent" and verify the signature for you.
- Others: check
Signature-Agent, fetch the public key and validate the signature before allowing the request.
For Google, the same documentation says Google is experimenting with Web Bot Auth using the https://agent.bot.goog identity. While that remains an experiment, verify Google-Agent by IP.
For SEO professionals: keep "allow access" separate from "allow the action." Allowing the agent in your WAF does not require turning off sign-in, 3-D Secure or rate limits: OpenAI describes the agent pausing so the user can sign in through a secure form or take over the browser. Also check that intermediate proxies keep all three headers, because stripping them breaks verification.
It helps to log the header. In Nginx:
log_format agents '$remote_addr [$time_local] "$request" $status '
'"$http_user_agent" sig_agent="$http_signature_agent"';
access_log /var/log/nginx/agents.log agents;
Anyone can send that header. The log is for measurement, and allowing traffic should depend on a validated signature.
How do you write HTML an agent can use?
Use semantic HTML: anything clickable is a real <button> or <a>, every field has a <label>, and price, shipping and stock appear as text. That is exactly what accessibility asks for. The accessibility tree (the version of the page the browser hands to screen readers, with each element's role, name and state) is a good mirror of what an agent can interpret.
The companies do not say whether their agents read a page through screenshots, the DOM or the accessibility tree. The safe approach is to make all three layers say the same thing.
- Buttons with text. If a button is icon-only, give it an
aria-label. - Native
<select>for size and color, not a clickablediv. - Price, shipping and delivery time as text, not inside an image.
- Error messages written next to the field that failed.
- "Out of stock" in the text and a button with
disabled. - Pop-ups with an obvious close button.
<form action="/checkout/shipping" method="post">
<label for="zip">ZIP code</label>
<input id="zip" name="zip" autocomplete="postal-code" inputmode="numeric" required>
<label for="email">Email</label>
<input id="email" name="email" type="email" autocomplete="email" required>
<button type="submit">Calculate shipping</button>
</form>
In plain English: think of the agent as someone reading the page out loud without seeing colors or pictures. If the buy button is just a drawing of a cart with no name, they have no way of knowing what it does.
For SEO professionals: in Chrome DevTools, the Accessibility pane shows each element's computed name and role, and Lighthouse flags buttons and fields without names. There is no documentation on whether on-demand fetchers execute JavaScript, so ship price and stock in the initial HTML, as covered in technical SEO basics. There is also no public data on agent timeouts, but a multi-step task multiplies every delay; see Core Web Vitals explained.
How do you make checkout and forms completable?
Cut out anything that requires guessing and leave the final step to the person: guest checkout, fields with autocomplete, shipping you can calculate before signing in, and return rules written on the page. OpenAI and Google both describe pauses for confirmation or for the user to take over at payment, and that is a good thing.
The usual blockers are a verification challenge at the cart, a mandatory account before shipping costs, and coupons hidden in pop-ups. They all hurt human customers too.
Feeds, schema and commerce protocols: what matters in 2026?
For an agent to compare your product, the data must match on the page, in schema and in the feed. Commerce protocols between agents and stores do exist, but they changed a lot in 2026 and are limited to a few markets.
OpenAI. The Agentic Commerce Protocol (ACP), created by OpenAI and Stripe, now mainly powers product discovery. On the ChatGPT merchants page, OpenAI says it is moving away from a standalone Instant Checkout experience and that purchases are completed on the merchant's own site or app, with no fees. Shopping in ChatGPT is live only in the U.S., Shopify and Etsy catalogs are already integrated, and a feed is not required if ChatGPT already crawls your site, though it gives you more control. Other merchants apply through a form and join a waitlist.
Google. According to Merchant Center help on UCP, the Universal Commerce Protocol enables a buy button on eligible listings, with checkout on Google's surfaces while the store remains the merchant of record. Eligibility covers the U.S., Canada and Australia in an early access program, and Google names AI Mode and Gemini as surfaces. Per the specification at ucp.dev (version 2026-08-25), the business publishes its UCP profile at /.well-known/ucp. In May 2026, Google announced it will bring AP2, for agent purchases within limits the user sets, to its products. We could not confirm whether that has launched.
If you sell in the U.S., applying for both programs is worth considering. Everywhere else, the safe investment is what already works for search: correct product schema, Merchant Center and pages that help people decide, as in the guide to GEO for e-commerce. Here is an offer with shipping and returns:
{
"@context": "https://schema.org",
"@type": "Product",
"name": "Waterproof Backpack 30L",
"sku": "BPK-30-BLK",
"brand": { "@type": "Brand", "name": "Example Brand" },
"offers": {
"@type": "Offer",
"url": "https://www.yourstore.com/waterproof-backpack-30l",
"price": "59.90",
"priceCurrency": "USD",
"availability": "https://schema.org/InStock",
"shippingDetails": {
"@type": "OfferShippingDetails",
"shippingRate": { "@type": "MonetaryAmount", "value": "0", "currency": "USD" },
"shippingDestination": { "@type": "DefinedRegion", "addressCountry": "US" }
},
"hasMerchantReturnPolicy": {
"@type": "MerchantReturnPolicy",
"applicableCountry": "US",
"returnPolicyCategory": "https://schema.org/MerchantReturnFiniteReturnWindow",
"merchantReturnDays": 30
}
}
}
For SEO professionals: the most expensive mistake is a mismatch. If the feed says $59.90 and the page says $64.90, the agent finds two truths, and nobody knows which one it uses. Generate price and stock from a single source and validate the JSON-LD every time a template changes. Which properties to prioritize is covered in schema markup for AI search.
How do you measure agent visits and tasks?
Through server or CDN logs. JavaScript analytics may count a cloud browser as a person, or not count it at all. The log shows everything that reached the server, including the blocks.
-
Split the groups by
Google-Agent,ChatGPT-User,Claude-User,Perplexity-Userand thesig_agentfield. -
Count status codes. In the combined log format, the status is the ninth field:
grep -E "Google-Agent|ChatGPT-User|Claude-User|Perplexity-User" access.log | awk '{print $9}' | sort | uniq -c -
Follow the path of each session: product, cart, shipping, payment. Where it stops is where the blocker is.
-
Validate the IPs of
Google-Agentagainst the official list.
Whether your brand gets suggested in the conversation is a separate measurement, described in how to measure AI visibility. For that side, Contentor's AI Visibility tracks, for the questions you choose, whether ChatGPT cites your brand (ChatGPT only).
Hypothetical example: an outdoor gear store sees in its logs that Google-Agent gets 200 on product pages and 403 on /cart/shipping. The cause is an old CDN rule that challenges any automated request at checkout. The team allows the official Google-Agent IPs and OpenAI's signed agent on the cart and shipping routes only, leaves sign-in and payment as they were, and notes the date to compare with the following month.
Checklist: an agent-ready website
- Decide in robots.txt what to do with training and search crawlers, without blocking
Googlebot. - Confirm in your CDN or WAF that identified agents get
200on public pages. - Allow OpenAI's agent by signature, not by user agent alone.
- Make sure proxies keep the signature headers.
- Replace fake clickable elements with
<button>,<a>and<select>. - Give every field a
<label>andautocomplete. - Show price, stock, shipping and delivery time as text in the initial HTML.
- Offer guest checkout and shipping costs before sign-in.
- Generate page, JSON-LD and feed from the same data source.
- Review agent status codes and paths in your logs every month.
Frequently asked questions
Do I need a special robots.txt for AI agents?
Not for the agents themselves. Google and OpenAI say that user-triggered bots, such as Google-Agent and ChatGPT-User, may ignore robots.txt because a person requested the visit. The file is still useful for training and search crawlers. For agents, control sits in your firewall or CDN.
Does allowing agents in my firewall make the site less secure?
Not if allowing them depends on verification. OpenAI says to allow its cloud browser only after validating the signature, and Google publishes the Google-Agent IPs. Allowing browsing does not require turning off sign-in, rate limits or payment verification, because the agent hands sensitive steps back to the user.
Does llms.txt help agents use my site?
There is no confirmation that it does. None of the agent documentation reviewed for this article mentions llms.txt: it talks about signatures, IPs and site access. The full discussion is in llms.txt: is it worth creating?. Prioritize clear HTML and consistent product data.
Can I sell inside ChatGPT today?
Only partly, and only in the U.S. as of this review. OpenAI says shopping in ChatGPT is live for U.S. users and that purchases are completed on the merchant's site or app. Google's UCP checkout covers the U.S., Canada and Australia in early access. Stores elsewhere can prepare their data and pages now without depending on either.
How do I know if an agent abandoned a purchase on my site?
Check the logs. Filter identified agents and look at the last URL and status code of each session. Lots of 403 or 429 responses point to a CDN block, and sessions that always stop at the same step point to a blocker in the flow. Agents running in the user's own browser do not show up separately.
Next step
With your site open to agents, look at what happens before the visit: read how AI finds information, with RAG and grounding. To turn all of this into a prioritized plan, move on to the step-by-step GEO audit.
Read also:
Prepare your content for AI answers
Create AEO/GEO briefs, entities, schema, and structured articles for Google AI Overviews, AI Mode, and traditional SEO.
Try Contentor freeStrengthen your authority for AI Search
Combine source-worthy content with contextual backlinks on vetted sites to reinforce trust, discovery, and topical authority.
View contextual backlinks

