How Caching Works

Overview

RndrKit caches pre-rendered HTML pages to serve bot traffic quickly. When a bot visits a page for the first time, the page is rendered in a headless browser and the resulting HTML is stored in our cache layer. Subsequent bot visits to the same page receive the cached HTML instantly, avoiding the overhead of a fresh render.

Cache Architecture

The caching system has two main components:

The Cache

The cache stores the rendered HTML for each page as a key-value pair. The key is derived from the full URL (domain + path), and the value is the complete HTML document.

Key:   render:www.example.com:/about
Value: <!DOCTYPE html><html>...full rendered HTML...</html>
TTL:   3600 seconds (1 hour)

Render Job Queue

When a cache miss occurs, a rendering job is added to the render queue. The queue manages rendering requests across multiple headless browser instances, ensuring that:

Multiple pages can be rendered concurrently
Duplicate render requests for the same URL are deduplicated
Failed renders are retried automatically
The browser pool is not overwhelmed by too many simultaneous renders

Request Flow

Here is what happens when a bot requests a page:

Bot Request
    |
    v
Edge (identifies bot via User-Agent)
    |
    v
Express API
    |
    +--> Check cache for rendered HTML
    |       |
    |       +--> Cache HIT: Return cached HTML (< 50ms)
    |       |
    |       +--> Cache MISS: Queue render job
    |                |
    |                v
    |            Render Queue
    |                |
    |                v
    |            Headless browser renders page (2-5 seconds)
    |                |
    |                v
    |            Store in cache (1hr TTL)
    |                |
    |                v
    |            Return rendered HTML
    v
Bot receives response

Cache Keys

Cache keys are constructed from the domain and the full URL path:

render:{domain}:{path}

Examples:

URL	Cache Key
`www.example.com/`	`render:www.example.com:/`
`www.example.com/about`	`render:www.example.com:/about`
`www.example.com/blog/my-post`	`render:www.example.com:/blog/my-post`

Query strings are stripped from cache keys. URLs like /products?page=2 and /products?page=3 resolve to the same cached entry as /products.

Time-to-Live (TTL)

The default TTL for cached pages is 1 hour (3600 seconds). This means:

A page rendered at 2:00 PM will be served from cache until 3:00 PM.
At 3:00 PM, the cache entry expires and the next bot request triggers a fresh render.
The fresh render is cached for another hour.

This TTL balances freshness with performance:

Short enough that content updates are reflected within an hour
Long enough that most bot visits within a crawl session hit the cache

Cache Hit vs. Cache Miss

Cache Hit

When the requested page is found in the cache:

Response time: typically under 50ms
No browser resources used
The X-Prerender-Cache header is set to HIT
No render count is consumed (only misses count toward your limit)

Cache Miss

When the requested page is not in the cache or has expired:

A render job is created for the page
Our renderer loads the page in a headless browser
JavaScript executes and the page fully renders
Response time: typically 2-5 seconds depending on page complexity
The X-Prerender-Cache header is set to MISS
One render is counted against your monthly limit

Browser Pool

Our renderer maintains a pool of headless browser instances for rendering. Key details:

Each browser instance handles up to 20 page renders before being recycled
Recycling prevents memory leaks from long-lived browser processes
Multiple browser instances run concurrently for parallel rendering
New pages open in fresh tabs within existing browser instances

Cache Refresh Frequency

In addition to the TTL-based expiration, your plan includes automatic cache refresh at a set frequency:

Daily (Starter, Pro plans) -- Cached pages are proactively refreshed once per day.
Hourly (Agency plan) -- Cached pages are refreshed every hour.
Real-time (Agency+ plan) -- Cached pages are refreshed as content changes.

This means higher-tier plans keep cached content fresher without relying solely on TTL expiration.

Cache Eviction

Cache entries are evicted in three ways:

TTL expiration -- Entries automatically expire after 1 hour
Automatic refresh -- Entries are proactively refreshed based on your plan's refresh frequency
Manual purge -- You can purge individual pages or all cache from the dashboard

Cache memory is bounded. If the cache runs out of space, the least recently used entries are evicted to make room for new ones.

Monitoring Cache Performance

You can monitor your cache performance from the Analytics page:

Cache hit rate -- Percentage of bot requests served from cache
Total hits and misses -- Raw counts over time
Average render time -- How long cache misses take to render

A healthy setup typically shows a cache hit rate above 70%.

Next Steps

Purging Cache -- Clear cached pages when content changes
Cache Warming -- Pre-render pages before bots arrive
Rendering Pipeline -- Learn more about the rendering process