Rendering Pipeline

Overview

The rendering pipeline is the core of RndrKit's pre-rendering service. When a bot requests a page that is not in the cache, the pipeline renders the page using a headless Chromium browser, captures the resulting HTML, and stores it for future requests.

Pipeline Architecture

Bot Request (cache miss)
    |
    v
Express API creates render job
    |
    v
BullMQ Job Queue (Redis-backed)
    |
    v
Renderer Service picks up job
    |
    v
Browser Pool assigns a Chromium instance
    |
    v
New page tab opens -> Navigates to URL
    |
    v
Wait for network idle (no pending requests)
    |
    v
Capture document HTML
    |
    v
Store in Redis cache (1hr TTL)
    |
    v
Return HTML to Express API -> Respond to bot

BullMQ Job Queue

RndrKit uses BullMQ (backed by Redis) to manage rendering jobs. The queue provides:

Job Management

• Queuing -- When a cache miss occurs, a render job is added to the queue with the URL to render.
• Deduplication -- If multiple bots request the same URL before it is rendered, only one render job is created. Subsequent requests wait for the first render to complete.
• Retry logic -- Failed render jobs are automatically retried with exponential backoff. If a page fails to render after multiple attempts, it is marked as failed and logged.
• Concurrency control -- The number of concurrent render jobs is limited to prevent overwhelming the server's resources.

Job Lifecycle

1. Waiting -- Job is in the queue, waiting for a worker to pick it up.
2. Active -- A renderer worker is processing the job.
3. Completed -- The page was rendered successfully and cached.
4. Failed -- The render failed after all retry attempts.

Browser Pool

The Renderer service manages a pool of Chromium browser instances via Puppeteer.

Pool Behavior

• Multiple instances -- Several browser instances run concurrently, each capable of handling render requests.
• Page limit -- Each browser instance handles up to 50 page renders before being recycled. This prevents memory leaks and degradation that occur with long-lived browser processes.
• Recycling -- When a browser reaches its page limit, it is gracefully closed and a new instance is launched. In-progress renders complete before the old browser is terminated.
• Health monitoring -- Browser instances are monitored for crashes and unresponsive states. Crashed browsers are automatically replaced.

Why a Pool?

Running a single browser instance creates a bottleneck. If one page takes 10 seconds to render (heavy JavaScript), all other render requests are blocked. A pool of browsers allows parallel rendering, dramatically improving throughput.

The Rendering Process

When a browser instance picks up a render job, it follows these steps:

1. Create a New Page

A new tab (page) is created in the browser instance. Each render gets its own isolated tab to prevent interference between renders.

2. Navigate to the URL

The browser navigates to the target URL. This triggers the full page load process, including:

• HTML download
• CSS parsing and application
• JavaScript download and execution
• API calls and data fetching
• Image and asset loading

3. Wait for Network Idle

After navigation, the renderer waits until the page's network activity settles. "Network idle" means no new network requests have been made for a short period. This ensures that:

• All API calls have completed and data is rendered
• Dynamically loaded content has appeared
• Fonts and critical assets have loaded

4. Capture HTML

Once the page is stable, the renderer captures the full HTML:

const html = await page.evaluate(
  () => document.documentElement.outerHTML
);

This captures everything the browser has rendered, including:

• All DOM elements with their final content
• Inline styles and computed attributes
• Meta tags injected by JavaScript (React Helmet, etc.)
• Structured data (JSON-LD) added dynamically
• Canvas and SVG elements (as markup)

5. Clean Up

The page tab is closed and the browser instance is returned to the pool for the next render job.

Performance Characteristics

Render time depends on the complexity of the page:

• Simple pages with minimal JavaScript and no API calls render fastest.
• Complex SPAs with multiple API calls, heavy frameworks, and dynamic content take longer.
• Pages with large images -- The renderer waits for network idle, so large asset downloads add time.

Error Handling

The pipeline handles several failure scenarios:

Navigation Timeout

If a page does not load within the timeout period (typically 30 seconds), the render job fails and is retried.

JavaScript Errors

Client-side JavaScript errors do not necessarily fail the render. The renderer captures whatever HTML is available when the page reaches network idle. However, if a critical error prevents the page from rendering any content, the resulting HTML will be minimal.

Browser Crash

If a Chromium instance crashes mid-render, the pool detects the crash, replaces the browser instance, and the failed job is retried by BullMQ.

Out of Memory

If the server runs low on memory, browser instances may be terminated. The pool will create new instances when resources become available.

Next Steps

• How Caching Works -- What happens after a page is rendered
• Bot Detection -- How bots are identified before rendering
• Origin Proxy -- How human requests bypass the renderer
• Rendering Failures -- Troubleshoot render problems