Robots.txt Editor

Overview

The robots.txt file tells search engines which pages and directories on your site they are allowed to crawl. RndrKit lets you manage your robots.txt through the dashboard, so you do not need to deploy changes to your application every time you want to update crawling rules.

How It Works

When a bot requests https://www.example.com/robots.txt, RndrKit serves the robots.txt content you configured in the dashboard instead of proxying the request to your origin. This gives you full control over crawling directives without modifying your application.

Editing Robots.txt

Go to your domain's detail page and click the Sitemap tab (robots.txt is managed alongside your sitemap).
Edit the robots.txt content in the text editor.
Click Save.

Changes take effect immediately.

Robots.txt Syntax

The robots.txt file uses a simple format with directives:

# Allow all crawlers to access everything
User-agent: *
Allow: /

# Block a specific directory
User-agent: *
Disallow: /admin/
Disallow: /private/

# Reference your sitemap
Sitemap: https://www.example.com/sitemap.xml

Common Directives

Directive	Purpose	Example
`User-agent`	Which bot this rule applies to	`User-agent: Googlebot`
`Allow`	Explicitly allow crawling a path	`Allow: /`
`Disallow`	Prevent crawling a path	`Disallow: /admin/`
`Sitemap`	Location of your XML sitemap	`Sitemap: https://...`
`Crawl-delay`	Seconds between requests (some bots only)	`Crawl-delay: 10`

Targeting Specific Bots

You can create rules for specific crawlers:

# Allow search engines
User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

# Block AI training crawlers
User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

# Default rule for all other bots
User-agent: *
Allow: /
Disallow: /admin/

Recommended Configuration

For most RndrKit users, this robots.txt works well:

User-agent: *
Allow: /
Disallow: /api/
Disallow: /admin/

Sitemap: https://www.example.com/sitemap.xml

This allows all bots to crawl your public pages while blocking internal API and admin routes.

Blocking AI Crawlers

If you want to prevent AI companies from using your content for training, add rules for their known crawlers:

# Block AI/LLM training crawlers
User-agent: GPTBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Bytespider
Disallow: /

Note that blocking these crawlers in robots.txt is a request, not enforcement. Compliant bots will respect these directives, but not all crawlers follow the rules.

Testing Your Robots.txt

After saving changes, verify that your robots.txt is served correctly:

curl https://www.example.com/robots.txt

You should see the content you configured in the dashboard.

You can also test specific rules in Google Search Console's Robots.txt Tester to verify that the right pages are allowed and blocked.

RndrKit's Robots.txt vs. Your Origin's

Your origin server likely already serves a robots.txt file, especially if you are using a modern hosting platform. Here is how to think about when to use RndrKit's robots.txt editor vs. sticking with your origin's version.

When to enable RndrKit's robots.txt

You want to manage crawling rules from the dashboard. If you do not have easy access to your origin's file system (or just prefer a UI), RndrKit's editor lets you update robots.txt without deploying anything.
You want a Sitemap: reference added automatically. When enabled, RndrKit appends a Sitemap: directive pointing to your domain's sitemap URL. This makes sure search engines can always find your sitemap without you having to remember to add it.
Your origin's robots.txt is too permissive or too restrictive. If the default robots.txt from your hosting platform does not match what you want, override it here.

When to leave it disabled

Your origin already has a good robots.txt. If your hosting platform or framework generates the right rules and you are happy with them, there is no need to override. RndrKit will proxy the request to your origin as usual.
You manage robots.txt in your application code. If your app dynamically generates robots.txt (e.g., different rules for staging vs. production), keep that logic in your codebase.

How it works under the hood

When the robots.txt feature is enabled for a domain, RndrKit intercepts any request to /robots.txt and serves the content you have configured in the dashboard. It also automatically appends a Sitemap: reference if you have a sitemap set up. The origin's robots.txt is completely overridden -- bots never see it.

When the feature is disabled, requests for /robots.txt pass straight through to your origin.

Platform notes

Lovable: Always serves its own robots.txt that allows major search engine bots (Googlebot, Bingbot, etc.). If you are happy with Lovable's defaults, you can leave RndrKit's robots.txt editor disabled. Enable it only if you want to customize the rules or add a Sitemap: directive.

Common Mistakes

Blocking your entire site -- A Disallow: / under User-agent: * blocks all crawling. Only use this for specific bots.
Blocking JavaScript/CSS files -- Do not block access to your static assets. Search engines need these to understand your page layout.
Forgetting the sitemap -- Always include a Sitemap: directive so search engines can find your sitemap.
Conflicting rules -- If you have both Allow and Disallow for the same path, the more specific rule wins. Keep your rules clear and non-overlapping.

Next Steps

Sitemaps -- Generate and manage your XML sitemap
URL Redirects -- Set up redirects for moved or renamed pages
SEO Audit -- Audit your pages for SEO issues