Global Araç

Robots Txt Generator

Sitemap URL

Block 1

User-agent

Allow

No paths.

Disallow

Crawl-delay (optional)

robots.txt

User-agent: *
Disallow: /admin/
Disallow: /private/

Sitemap: https://freetoolarena.com/sitemap.xml

Build a valid robots.txt file with multiple user-agent blocks, per-agent allow and disallow paths, an optional crawl-delay, and a sitemap URL. Download it as a text file and upload to your site root. The default starts with a sensible baseline (disallow /admin/, /private/, sitemap pointer).

Pair with meta tag generator and Open Graph generator to round out on-page SEO basics.

Nasıl Kullanılır

Pick a user-agent (* for all) or add specific ones.
Add allow and disallow paths per agent.
Set crawl-delay and sitemap URL if needed.
Copy or download robots.txt.