Session Cache

The shared session cache stores cookies and browser identity from successful browser requests. It supports two reuse paths: Tier 1 can send eligible clearance over HTTP, and Tier 2 can inject the saved cookies into a browser context. Neither path requires creating a named session.

This is a cookie cache, not a page-content cache. Each request fetches the target again. See Performance & Benchmarks for measured results and configuration.

Automatic HTTP clearance reuse

Before an ordinary HTTP request, TRAWL checks whether the saved session can be replayed:

• The request is GET or HEAD and permits Tier 1.
• The connection is direct: no explicit request proxy or configured datacenter proxy pool.
• TLS verification is enabled.
• The entry contains the browser's User-Agent, valid scoped cookies and provenance for the exact requested origin. Cloudflare cf_clearance and supported Anubis authentication cookies qualify directly; other sessions also require the document proof described below.
• The request has no imported cookies, named sessionId, browser capture or content-selector wait.

Cookie domain, path, expiry and secure flags are checked before sending. Same-origin redirects can update cookies; a cross-origin redirect returns to the browser path before credentials are forwarded. Sessions without HTTP origin provenance can still use the browser path.

The HTTP attempt receives at most five seconds or one third of the remaining request budget, whichever is smaller. Rejected clearance, recognized challenge walls, detected JavaScript shells and embedded widget markup fall back through the existing tiers. A successful replay refreshes the cached cookies and browser identity without keeping another browser page open.

Anubis replay recognizes the default authentication cookie names, including their optional settings suffix. Test and referrer cookies do not qualify. Custom cookie names and clearance bound to a browser connection retain browser fallback.

Other direct browser sessions

Sessions without a recognized Cloudflare or Anubis clearance cookie can also use HTTP replay. TRAWL records the exact verified URL and a hash of the normalized rendered HTML document. The HTTP response must match this document before returning it. A partial JavaScript shell, changed document, different URL, empty body or HTML above 262,144 characters retains browser fallback. The original Cloudflare and Anubis paths do not require this comparison.

The API can skip Tier 1 after two consecutive JavaScript-shell or content-mismatch results followed by cached browser success for the same URL. Hints expire after 30 minutes and are bounded to 1,000 URLs. HTTP success, fresh browser solving or failure clears the hint. Only ordinary direct GET requests qualify; explicit skipHttp, HTTP-only requests, custom headers, imported cookies, named sessions and proxy/TLS overrides retain their requested policy. These hints stay in memory and are separate from the persistent session cache.

Proxy-bound browser reuse

Sessions established through a configured datacenter or residential proxy retain an opaque identity for that proxy. Tier 2 resolves it against the current proxy configuration and opens a temporary context on the same pooled browser, with the original proxy, cookies and User-Agent. Optional external CAPTCHA tasks use the same proxy during cached browser requests. The context closes when the request finishes or times out. Direct sessions retain their shared pooled-context path.

Proxy credentials are not copied into the cache. Removing the proxy, changing its credentials or putting it in cooldown invalidates reuse of its cached session. Proxied cookies are never eligible for direct HTTP clearance replay, and explicit per-request proxies and named sessions retain their existing isolation rules.

The same proxy endpoint must still provide a stable exit IP for IP-bound clearance. If the site rejects the cached session, TRAWL invalidates it and attempts a fresh solve as before.

Pool selection policies apply when entering Tier 3 or Tier 4. A Tier 2 cache hit retains the original healthy proxy, including when fresh solves use round-robin or random selection.

Cache driver

TRAWL supports two session cache drivers, selected at startup via SESSION_CACHE_DRIVER:

Driver	Value	Shared across instances	External dependencies
Redis (default)	redis	Yes	Redis
In-memory	memory	No (per-process)	None

SESSION_CACHE_DRIVER=redis   # default, shared across instances
SESSION_CACHE_DRIVER=memory  # in-process Map, zero dependencies

Use redis when running multiple API instances behind a load balancer: sessions solved on one instance are visible to all others. Use memory for single-instance deployments where the operational overhead of Redis is not justified; sessions are scoped to the process and lost on restart.

Both drivers support HTTP clearance reuse and browser fallback through the same SessionCacheStore interface. Redis is not required for reuse in a single API process.

Storage format

Key: session:v2:{hostname} (e.g. session:v2:nowsecure.nl)

Older Redis entries under session:{hostname} are ignored because their proxy origin cannot be established safely. The first browser request after upgrading repopulates the cache; old entries expire with their existing TTL.

Value (JSON):

interface SessionData {
  cookies: Cookie[]
  userAgent: string
  savedAt: number     // unix timestamp ms
  proxyId?: string    // SHA-256 identity of the configured proxy, absent for direct sessions
  httpOrigin?: string // verified direct origin eligible for HTTP reuse
  httpReplay?: { url: string; contentHash: string } // optional proof for other sessions
}

TTL: REDIS_SESSION_TTL_SECONDS (default 3600 seconds / 1 hour) for both drivers. It controls cache retention, not cookie validity; keeping an entry longer does not extend its cookies' expiry.

httpOrigin is recorded after an eligible direct GET/HEAD browser visit with verified TLS and the same requested and final origin. Proxied or unverified visits do not grant HTTP replay provenance.

Session key

The key is the hostname only, without a path, port or protocol. Pages on that hostname share a cache entry, while cookie scope still controls which cookies are sent:

https://example.com/        → session:v2:example.com
https://example.com/page    → session:v2:example.com  (same key)
https://sub.example.com/    → session:v2:sub.example.com  (different key)

Subdomains have separate entries. HTTP replay additionally requires an exact match of scheme, hostname and port with httpOrigin; a shared hostname key alone never authorizes replay to another origin.

Lifecycle

Request permits Tier 1
  |
  +-- eligible cached clearance --> HTTP with saved cookies and User-Agent
  |                                  accepted --> refresh cache, return
  |                                  rejected --> browser fallback
  +-- otherwise -----------------> ordinary HTTP
                                     accepted --> return
                                     blocked  --> browser fallback

Browser fallback
  |
  +-- cached state --> Tier 2: inject cookies and navigate
  |                     accepted --> refresh cache, return
  |                     failed   --> invalidate entry
  +-- no usable state --> Tier 3: fresh solve; cache returned cookies
                          failed --> Tier 4 if a proxy is configured

Requests with imported cookies and named browser sessions retain their separate paths and do not write their login state into this shared cache. Named sessions keep an isolated browser context and storage; they do not use HTTP clearance replay.

Tier limits, proxy routing and the remaining request budget can stop escalation earlier.

Invalidation

If Tier 2 navigates with cached state and still receives a recognized challenge wall, the orchestrator:

1. Calls sessionCache.invalidate(domain) to delete the cache entry
2. Escalates to Tier 3 to get a fresh session

A rejected HTTP replay does not immediately delete the entry: the browser may still be able to use that clearance. Invalidation follows a failed Tier 2 attempt. This handles expiry or rejection before the cache TTL ends.

Redis

When SESSION_CACHE_DRIVER=redis (the default), TRAWL uses new RedisClient(REDIS_URL) from Bun's native Redis client.

The cache is optional. When REDIS_URL is empty or unset with the Redis driver selected, TRAWL does not create a Redis client. Shared clearance reuse and Tier 2 are unavailable; ordinary HTTP and fresh browser solves still work.

Each connection attempt is bounded by REDIS_CONNECT_TIMEOUT_MS (default 5 seconds). If Redis is not ready, scraping continues without Tier 2 while TRAWL retries in the background every REDIS_RETRY_DELAY_MS (default 5 seconds). Set the retry delay to 0 to disable reconnects after a configured Redis endpoint becomes unavailable.

import { RedisClient } from 'bun'

const redis = new RedisClient('redis://localhost:6379')
await redis.set('session:v2:example.com', JSON.stringify(data), 'EX', 3600)
const raw = await redis.get('session:v2:example.com')

In-memory

When SESSION_CACHE_DRIVER=memory, TRAWL uses an in-process Map with TTL-based expiry. There is no network I/O, so the cache is available immediately on startup. Expired entries are removed on read and before writes. MEMORY_SESSION_CACHE_MAX_ENTRIES (default 1000) bounds memory growth; when full, the least recently used session is evicted.

Because the cache lives in the API process, sessions are not shared across instances. A solve on instance A is invisible to instance B, and all sessions disappear on restart. Use this driver only for single-instance deployments. Session contents are never written to application logs.