Development · Article or course
Make your site readable by AI agents
Practical guide to making websites discoverable and actionable for AI agents.
Published on the Vercel Knowledge Base in collaboration with the ora.ai research lab, this guide outlines technical requirements for AI agent discoverability based on 1,000+ real agent runs and controlled fetch probes. It reveals that server response behavior and server-rendered HTML matter significantly more than emerging conventions like llms.txt. It provides actionable advice on HTTP status codes, SSR/prerendering, markdown endpoints, and API design.
What it's for
- Audit a website to ensure AI agents and crawlers can retrieve content without executing JavaScript
- Configure server responses, HTTP status codes, and rate limits correctly for automated agent traffic
- Expose machine-readable documentation, OpenAPI contracts, and markdown mirrors for AI workflows
- Evaluate domain readiness for AI agents using the is-agentic testing tool
Features
- Analysis of 1,033 real agent runs and 190 controlled fetch probes across 19 site configurations
- Detailed recommendations on server-rendered HTML over client-side JavaScript rendering
- Guidance on configuring llms.txt as a discoverable, linked index rather than relying on guessed paths
- Practical tips for supporting Markdown format negotiation and Link headers
- Checklist for exposing authenticated APIs, OpenAPI specs, and Model Context Protocol (MCP) endpoints
- Integration with domain scanning tools like is-agentic.com
Pricing
Free technical article and public research dataset
Details
- Type
- Article or course
- Category
- Development
- Topic
- AI Readiness
- Open source
- No
- Platforms
- Web
- For
- Web developers, technical SEOs, and API engineers building agent-accessible sites
- Tags
- ai agents, seo, web development, api design, vercel, llms.txt
Why it was saved
- @vercel_dev: Grounded in over 1,000 real agent runs and a controlled experiment with the ora.ai research lab; counters the common assumption that llms.txt is the answer - the way servers respond to requests reportedly matters more.
Related
Open directory and CLI for discovering and installing AI agent skills
Curated React performance rules and coding-agent skills by Vercel
Run browser automation agents inside on-demand isolated sandbox environments
Practical rules for preparing SaaS APIs and infrastructure for AI agents
Curated skills and performance rules for AI coding agents by Vercel.
Universal TypeScript layer for building bots and agents across chat platforms
Updated