How Pre-rendering Works

What Is Pre-rendering?

Pre-rendering is the process of loading a web page in a headless browser, waiting for all JavaScript to execute, and capturing the resulting HTML. This HTML snapshot contains all the content that a human would see -- text, images, meta tags, structured data -- and can be served directly to search engine bots that cannot execute JavaScript on their own.

Unlike server-side rendering (SSR), pre-rendering does not require any changes to your application code. Your SPA continues to work exactly as it does today. RndrKit handles the rendering externally.

The Request Flow

When a request arrives at your domain, it passes through several layers:

1. DNS Resolution

Your domain's CNAME record points to RndrKit's server. When a visitor (human or bot) requests your site, the DNS lookup resolves to RndrKit's infrastructure.

2. Bot Detection at Nginx

The request hits Nginx, which inspects the User-Agent header. RndrKit maintains a list of over 100 known bot user-agents, including:

  • Search engines: Googlebot, Bingbot, DuckDuckBot, Baiduspider, YandexBot
  • Social media: Facebook, Twitter, LinkedIn, Pinterest, Discord, Slack
  • AI crawlers: GPTBot, ClaudeBot, PerplexityBot, Anthropic-AI
  • SEO tools: Semrush, Ahrefs, Screaming Frog, Majestic

Nginx sets an X-Is-Bot header and forwards the request to the Express API.

3. Bot Path: Render and Cache

If the request is from a bot:

  1. The API checks Redis for a cached HTML snapshot of the requested URL.
  2. Cache hit: The cached HTML is returned immediately. Response times are typically under 50ms.
  3. Cache miss: A rendering job is queued via BullMQ. A Puppeteer browser instance loads the page, waits for JavaScript to finish, and captures the HTML. The result is stored in Redis with a one-hour TTL and returned to the bot.

4. Human Path: Proxy to Origin

If the request is from a regular browser, the API proxies the request directly to your origin server (e.g., your-app.lovable.app). The visitor's experience is identical to accessing your site directly. Headers, cookies, and response codes are all preserved.

The Rendering Process

When a page needs to be rendered, the following steps occur:

BullMQ Job Queue
    |
    v
Puppeteer Browser Pool (reuses browsers for up to 50 pages)
    |
    v
New Page Created
    |
    v
Navigate to URL --> Wait for network idle
    |
    v
Capture Full HTML (document.documentElement.outerHTML)
    |
    v
Store in Redis (1-hour TTL) --> Return to API

The browser pool manages multiple Chromium instances. Each instance handles up to 50 pages before being recycled to prevent memory leaks. New rendering requests are distributed across available browsers for optimal performance.

Cache Behavior

Rendered HTML is stored in Redis using the full URL as the cache key. The default time-to-live (TTL) is one hour. This means:

  • The first bot visit to a new page triggers a fresh render (typically 2-5 seconds).
  • Subsequent bot visits within the hour receive the cached version instantly.
  • After the TTL expires, the next bot visit triggers a new render with fresh content.

You can manage caching through the dashboard:

  • Purge individual pages when you update content and want bots to see changes immediately.
  • Purge all cache after a major site update.
  • Warm cache proactively by requesting renders for important pages before bots arrive.

Learn more in How Caching Works.

What Gets Rendered

The rendered HTML includes everything that the browser produces after JavaScript execution:

  • Full DOM content (text, images, links)
  • Meta tags injected by React Helmet, Vue Meta, or similar libraries
  • Open Graph and Twitter Card tags
  • Structured data (JSON-LD schema markup)
  • Canonical URLs
  • Dynamically loaded content (API data, etc.)

Performance Considerations

RndrKit is designed for minimal impact on your site's performance:

  • Human visitors experience no added latency. Proxy overhead is negligible.
  • Bot visitors get cached responses in under 50ms on cache hits.
  • Cache misses take 2-5 seconds for the initial render, which is acceptable for bots.
  • Rate limiting prevents bots from overwhelming the rendering pipeline (2 requests per second per IP for bots).

Next Steps

Previous

What is RndrKit?

Next

Quick Start Guide