TOOLS.GAMESLOOP234 Free Online Tools

Robots.txt Generator

Build a valid robots.txt with the right crawl rules.

One path per line. Leading slash added automatically.

Use the absolute URL.

Seconds between requests. Google ignores this.

Disallow does not remove a page from search

robots.txt stops crawling, not indexing. A blocked URL can still appear in results, listed without a description, if other pages link to it. To remove a page, allow crawling and use a noindex meta tag.
The file must be served at exactly /robots.txt on the domain root. A copy in a subdirectory is ignored, and every subdomain needs its own.

What does Robots.txt Generator do?

Generate a robots.txt file with per-crawler rules, sitemap references and sensible defaults. Common mistakes are flagged live — including the classic accidental site-wide block.

How to Use Robots.txt Generator

  1. 1
    Start from a presetOr build the rules yourself.
  2. 2
    Add disallow pathsAdmin areas, API routes, internal search.
  3. 3
    Add your sitemap URLUse the absolute URL.
  4. 4
    Download and uploadIt must sit at your domain root.

Key Features

  • Per-user-agent allow and disallow rules
  • Sitemap and crawl-delay directives
  • Presets for common platforms and AI crawler policies
  • Live warning when a rule would block the entire site
  • Explains why robots.txt is not the way to hide a page from search results
  • Copy or download the finished file

Frequently Asked Questions

Where does robots.txt go?

At the root of the domain, exactly at /robots.txt. Crawlers look only there — a file in a subdirectory is ignored entirely. Each subdomain needs its own.

Does Disallow remove a page from Google?

No, and this trips people up constantly. Disallow stops crawling, not indexing. A blocked URL can still appear in results, listed without a description, if other pages link to it. To remove a page, allow crawling and use a noindex meta tag.

What is the most dangerous mistake?

"Disallow: /" under "User-agent: *", which blocks the entire site. It usually escapes from a staging configuration into production. The generator warns loudly if you create it.

Should I block AI crawlers?

That is a business decision, not a technical one. Blocking them may reduce exposure in AI answers; allowing them may bring referral traffic. The presets include both stances so you can choose deliberately.

Runs in your browser

Analysis runs on the text or HTML you paste, inside your browser. Nothing is fetched from your site and nothing is uploaded, so the tools work on staging environments and pages behind a login.

How the rest of the site handles data

Related Tools

XML Sitemap Generator

Build a valid XML sitemap from a list of URLs.

Use Utility

Canonical URL Checker

Validate canonical tags and spot the mistakes that break them.

Use Utility

Meta Tag Generator

Generate a complete set of HTML meta tags for any page.

Use Utility

HTTP Status Code Lookup

Search every HTTP status code with what it means and when to use it.

Use Utility