Crawler Rules

Cloudflare Adds Search to AI Agents—And Sets the Crawler Rules While It's At It

Cloudflare plugs search APIs into AI Gateway, letting agents search before reading. It also sets rules on how AI crawls the web.