How Caching Works

Overview

RndrKit uses Redis to cache pre-rendered HTML pages. When a bot visits a page for the first time, the page is rendered by Puppeteer and the resulting HTML is stored in Redis. Subsequent bot visits to the same page receive the cached HTML instantly, avoiding the overhead of a fresh render.

Cache Architecture

The caching system has two main components:

Redis Cache

Redis stores the rendered HTML for each page as a key-value pair. The key is derived from the full URL (domain + path), and the value is the complete HTML document.

Key:   render:www.example.com:/about
Value: <!DOCTYPE html><html>...full rendered HTML...</html>
TTL:   3600 seconds (1 hour)

BullMQ Job Queue

When a cache miss occurs, a rendering job is added to the BullMQ queue. The queue manages rendering requests across multiple Puppeteer browser instances, ensuring that:

Multiple pages can be rendered concurrently
Duplicate render requests for the same URL are deduplicated
Failed renders are retried automatically
The browser pool is not overwhelmed by too many simultaneous renders

Request Flow

Here is what happens when a bot requests a page:

Bot Request
    |
    v
Nginx (identifies bot via User-Agent)
    |
    v
Express API
    |
    +--> Check Redis for cached HTML
    |       |
    |       +--> Cache HIT: Return cached HTML (< 50ms)
    |       |
    |       +--> Cache MISS: Queue render job
    |                |
    |                v
    |            BullMQ Queue
    |                |
    |                v
    |            Puppeteer renders page (2-5 seconds)
    |                |
    |                v
    |            Store in Redis (1hr TTL)
    |                |
    |                v
    |            Return rendered HTML
    v
Bot receives response

Cache Keys

Cache keys are constructed from the domain and the full URL path:

render:{domain}:{path}

Examples:

URL	Cache Key
`www.example.com/`	`render:www.example.com:/`
`www.example.com/about`	`render:www.example.com:/about`
`www.example.com/blog/my-post`	`render:www.example.com:/blog/my-post`

Query strings are included in the cache key, so ?page=2 and ?page=3 are cached separately.

Time-to-Live (TTL)

The default TTL for cached pages is 1 hour (3600 seconds). This means:

A page rendered at 2:00 PM will be served from cache until 3:00 PM.
At 3:00 PM, the cache entry expires and the next bot request triggers a fresh render.
The fresh render is cached for another hour.

This TTL balances freshness with performance:

Short enough that content updates are reflected within an hour
Long enough that most bot visits within a crawl session hit the cache

Cache Hit vs. Cache Miss

Cache Hit

When the requested page is found in Redis:

Response time: typically under 50ms
No browser resources used
The X-Prerender-Cache header is set to HIT
No render count is consumed (only misses count toward your limit)

Cache Miss

When the requested page is not in Redis or has expired:

A BullMQ job is created for the render
Puppeteer loads the page in a headless browser
JavaScript executes and the page fully renders
Response time: typically 2-5 seconds depending on page complexity
The X-Prerender-Cache header is set to MISS
One render is counted against your monthly limit

Browser Pool

Puppeteer maintains a pool of Chromium browser instances for rendering. Key details:

Each browser instance handles up to 50 page renders before being recycled
Recycling prevents memory leaks from long-lived browser processes
Multiple browser instances run concurrently for parallel rendering
New pages open in fresh tabs within existing browser instances

Cache Refresh Frequency

In addition to the TTL-based expiration, your plan includes automatic cache refresh at a set frequency:

Daily (Starter, Pro plans) -- Cached pages are proactively refreshed once per day.
Hourly (Agency plan) -- Cached pages are refreshed every hour.
Real-time (Agency+ plan) -- Cached pages are refreshed as content changes.

This means higher-tier plans keep cached content fresher without relying solely on TTL expiration.

Cache Eviction

Cache entries are evicted in three ways:

TTL expiration -- Entries automatically expire after 1 hour
Automatic refresh -- Entries are proactively refreshed based on your plan's refresh frequency
Manual purge -- You can purge individual pages or all cache from the dashboard

Redis memory is bounded. If Redis memory is full, the least recently used entries are evicted to make room for new ones.

Monitoring Cache Performance

You can monitor your cache performance from the Analytics page:

Cache hit rate -- Percentage of bot requests served from cache
Total hits and misses -- Raw counts over time
Average render time -- How long cache misses take to render

A healthy setup typically shows a cache hit rate above 70%.

Next Steps

Purging Cache -- Clear cached pages when content changes
Cache Warming -- Pre-render pages before bots arrive
Rendering Pipeline -- Learn more about the Puppeteer rendering process