XOOER GEO
Robots.txt GeneratorControl search engine and AI crawler access to your websiteAllowDisallowSEOrobots.txtUser-agent:*Allow:/public/Disallow:/admin/Disallow:/private/Sitemap:https://example.com/sitemap.xmlSCANNING# robots.txt configuration

Robots.txt Generator

Create a robots.txt file to manage search engine and AI bot access to your website. Configure user-agent rules, crawl delays, and sitemap references.

Sitemap URL will be auto-populated based on your domain

Quick Templates

Custom Rules

How to Use

  1. Select a User-agent (crawler type), such as Googlebot, GPTBot, Claude-Web, etc.
  2. Add Allow or Disallow path rules for each crawler.
  3. (Optional) Enter your Sitemap URL and Crawl-delay settings.
  4. Click 'Generate', save the output as robots.txt and upload it to your website's root directory.

Why Use This Tool?

Robots.txt is the key file for controlling how search engines and AI crawlers access your website. Proper configuration prevents sensitive pages from being indexed while ensuring important content remains visible to search engines and AI models.