Advanced Cache Strategies: Preventing Cache Stampede and Thundering Herd
Published on June 2, 2026
Advanced Cache Strategies: Preventing Cache Stampede and Thundering Herd
Every developer knows how to add a cache. You check if the key exists, if it does you return it, if it doesn’t you query the database and store the result. Simple. And it works — until it doesn’t.
The naive TTL-based cache has two failure modes that only appear at scale. They don’t show up in development, they don’t show up in staging, and they have a habit of appearing at 11 PM on Black Friday. Cache Stampede and Thundering Herd are the names for what happens when your caching strategy falls apart exactly when you need it most.
In this article I’ll cover why they happen, how to prevent them, and how to design a caching layer that holds under real production pressure.
The Naive Implementation and Its Problem
async function getUser(id: string): Promise<User> {
const cached = await redis.get(`user:${id}`);
if (cached) return JSON.parse(cached);
const user = await db.users.findById(id);
await redis.set(`user:${id}`, JSON.stringify(user), 'EX', 300);
return user;
}This works at low traffic. At high traffic, it has a specific failure mode: every key has a hard expiration time, and when that TTL hits under concurrent load, every in-flight request misses simultaneously.
Cache Stampede: The Synchronized Miss
Cache Stampede (also called cache miss storm or dog-pile effect) happens when a popular cache key expires while many concurrent requests are waiting for it. All of them check, all of them miss, all of them hit the database simultaneously.
T=0:299 10,000 concurrent requests → all hit cache → hit, return cached
T=0:300 10,000 concurrent requests → key just expired → all miss → 10,000 DB queries
DB receives 10,000 queries for identical data
DB collapses or times out
Requests that wait for DB also start timing out
Cache still empty because the first DB query hasn't completed yet
More requests arriving → more misses → more DB queriesThe problem compounds itself. The database is overwhelmed, latency spikes, the requests in flight time out before they can repopulate the cache, so the next wave of requests also misses. You’re in a positive feedback loop that doesn’t resolve until traffic drops.
Solution 1: Probabilistic Early Expiration (XFetch)
The cleanest solution is to expire the key before it actually expires, with a probability that increases as the key gets closer to expiration. This is the XFetch algorithm, published by researchers at Akamai and widely used in production.
The idea: don’t wait for the TTL to hit. Start refreshing early, probabilistically, so no single moment triggers a synchronized miss.
interface CacheEntry<T> {
value: T;
delta: number; // time it took to compute this value (ms)
expiry: number; // absolute expiry timestamp (ms)
}
async function fetchWithPER<T>(
key: string,
fetcher: () => Promise<T>,
ttlSeconds: number,
beta = 1.0
): Promise<T> {
const raw = await redis.get(key);
if (raw) {
const entry: CacheEntry<T> = JSON.parse(raw);
const now = Date.now();
// XFetch formula: recompute if the random probe exceeds the adjusted expiry
const shouldRefresh = now - entry.delta * beta * Math.log(Math.random()) >= entry.expiry;
if (!shouldRefresh) return entry.value;
// If shouldRefresh: fall through and recompute, but don't block — this single
// request recomputes while others continue to get the stale value
}
const start = Date.now();
const value = await fetcher();
const delta = Date.now() - start;
const entry: CacheEntry<T> = {
value,
delta,
expiry: Date.now() + ttlSeconds * 1000,
};
await redis.set(key, JSON.stringify(entry), 'EX', ttlSeconds + 10);
return value;
}beta controls aggressiveness. beta = 1 is the standard. Higher values refresh earlier. The math: as expiry - now approaches 0, the probability of shouldRefresh returning true approaches 1 — but it spreads that probability across many requests over time, so the refresh is likely to happen before expiration without every request refreshing simultaneously.
Solution 2: Distributed Mutex (Lock-Based)
For cases where PER is too complex or you need strict single-writer semantics, a distributed lock ensures only one request rebuilds the cache while others wait or return stale data.
async function fetchWithLock<T>(
key: string,
fetcher: () => Promise<T>,
ttlSeconds: number
): Promise<T> {
const cached = await redis.get(key);
if (cached) return JSON.parse(cached);
const lockKey = `lock:${key}`;
const lockToken = crypto.randomUUID();
// Atomic SET NX EX — only succeeds for the first requester
const acquired = await redis.set(lockKey, lockToken, 'NX', 'EX', 10);
if (acquired) {
try {
const value = await fetcher();
await redis.set(key, JSON.stringify(value), 'EX', ttlSeconds);
return value;
} finally {
// Release only if we still own the lock (atomic check-and-delete via Lua)
await redis.eval(
`if redis.call("get", KEYS[1]) == ARGV[1] then return redis.call("del", KEYS[1]) else return 0 end`,
1, lockKey, lockToken
);
}
}
// Another process is rebuilding — wait and retry
await sleep(50);
const refreshed = await redis.get(key);
if (refreshed) return JSON.parse(refreshed);
// Lock timeout: fall back to fetching directly (safety valve)
return fetcher();
}The Lua script for the release is critical. Without it, you might delete a lock that was re-acquired by another process after your lock expired — a race condition that’s rare but catastrophic when it hits.
When to use lock-based over PER: when the computation is expensive (>500ms), when you can’t tolerate even brief stale reads, or when the data is write-heavy and PER would constantly trigger refreshes.
Solution 3: Stale-While-Revalidate
The HTTP stale-while-revalidate directive has a server-side analogue. Return the stale value immediately while refreshing asynchronously in the background. No waiting, no lock contention, just a brief window of slightly stale data.
interface SWREntry<T> {
value: T;
freshUntil: number; // serve fresh until this timestamp
staleUntil: number; // serve stale until this timestamp (then it's truly expired)
}
async function fetchWithSWR<T>(
key: string,
fetcher: () => Promise<T>,
freshTTL: number, // seconds to serve fresh
staleTTL: number // additional seconds to serve stale while refreshing
): Promise<T> {
const raw = await redis.get(key);
const now = Date.now();
if (raw) {
const entry: SWREntry<T> = JSON.parse(raw);
if (now < entry.freshUntil) {
return entry.value; // still fresh, serve directly
}
if (now < entry.staleUntil) {
// Stale but not expired — return immediately and refresh in background
refreshInBackground(key, fetcher, freshTTL, staleTTL);
return entry.value;
}
}
// Truly expired — must wait for fresh data
return refreshAndStore(key, fetcher, freshTTL, staleTTL);
}
function refreshInBackground<T>(key: string, fetcher: () => Promise<T>, freshTTL: number, staleTTL: number): void {
// Fire and forget — use a lock to prevent concurrent background refreshes
const lockKey = `bgrefresh:${key}`;
redis.set(lockKey, '1', 'NX', 'EX', 5).then(acquired => {
if (acquired) refreshAndStore(key, fetcher, freshTTL, staleTTL);
});
}This is the most user-friendly pattern: users never wait for a cache miss, and the data is only briefly stale (the window between the background refresh starting and completing).
Thundering Herd: The Restart Problem
Thundering Herd is related but operates at a different level. It describes what happens when a large number of processes that were sleeping wake up simultaneously to compete for a resource — or when your entire cache layer goes cold at once.
The classic scenario: your Redis instance restarts after a deployment. Every single key is gone. Your application receives normal traffic, but now every request misses, hits the database simultaneously, and the database — which was comfortably serving 200 qps through the cache — suddenly receives 50,000 qps of raw queries.
TTL Jitter
The simplest mitigation: never set a fixed TTL. Add random variance so keys expire at different times rather than in synchronized waves.
function jitteredTTL(baseTTL: number, jitterFraction = 0.2): number {
const jitter = baseTTL * jitterFraction;
return Math.floor(baseTTL + (Math.random() * jitter * 2 - jitter));
}
// Instead of: redis.set(key, value, 'EX', 300)
// Use: redis.set(key, value, 'EX', jitteredTTL(300))
// Keys now expire between 240s and 360s — no synchronized expiration waveThis alone eliminates a significant category of stampede problems that come from batch operations (caching results of a cron job, warming caches at startup) where you’d otherwise set thousands of keys with identical TTLs.
Promise Coalescing (In-Process)
For stampedes happening at the application layer — before they even reach Redis — coalesce concurrent requests for the same key into a single in-flight promise.
const inFlight = new Map<string, Promise<unknown>>();
async function fetchCoalesced<T>(
key: string,
fetcher: () => Promise<T>
): Promise<T> {
if (inFlight.has(key)) {
return inFlight.get(key) as Promise<T>;
}
const promise = fetcher().finally(() => inFlight.delete(key));
inFlight.set(key, promise);
return promise;
}This doesn’t replace Redis-level locking — it operates per-process. In a cluster of 20 Node.js instances, you still get 20 concurrent DB queries on a cold miss. But it eliminates the stampede within each process, reducing the multiplier by the number of concurrent requests per instance. Combined with a Redis mutex, you get full protection.
Layered Caching: L1 + L2
For high-read, infrequently-changing data, a two-level cache reduces Redis round-trips and protects against Redis latency spikes.
import NodeCache from 'node-cache';
const l1 = new NodeCache({ stdTTL: 30, checkperiod: 10 }); // in-process, 30s TTL
async function getWithLayeredCache<T>(
key: string,
fetcher: () => Promise<T>,
redisTTL = 300
): Promise<T> {
// L1: in-process memory (microseconds)
const l1Hit = l1.get<T>(key);
if (l1Hit !== undefined) return l1Hit;
// L2: Redis (sub-millisecond local, ~1ms network)
const l2Hit = await redis.get(key);
if (l2Hit) {
const value = JSON.parse(l2Hit) as T;
l1.set(key, value);
return value;
}
// Origin: database
const value = await fetcher();
await redis.set(key, JSON.stringify(value), 'EX', redisTTL);
l1.set(key, value);
return value;
}The L1 TTL should be much shorter than L2 — 30s vs 300s — to limit stale reads in deployments where multiple instances have diverging L1 caches. For data that changes rarely (config, feature flags, static lookups), this pattern can eliminate 90%+ of Redis traffic.
Cache Invalidation: The Hard Part
The famous quote — “there are only two hard things in computer science: cache invalidation and naming things” — is a joke, but it points at a real problem.
Event-driven invalidation is more reliable than TTL-only for mutable data. When a user is updated, publish an invalidation event and delete the cache key directly rather than waiting for TTL expiry.
// In your UserService
async updateUser(id: string, data: Partial<User>): Promise<User> {
const user = await db.users.update(id, data);
// Invalidate immediately — don't wait for TTL
await redis.del(`user:${id}`);
// If you have a pub/sub system, broadcast to other instances
await pubsub.publish('cache:invalidate', { key: `user:${id}` });
return user;
}For distributed systems where multiple services cache the same data, write-through invalidation via a message bus (Kafka, Redis Pub/Sub) ensures all nodes clear their caches synchronously. This is preferable to relying on TTL convergence across services.
My Personal Perspective
Cache Stampede and Thundering Herd are the kind of problems that make you feel like a fool when you finally understand them. The solution to “too many database queries” is a cache. The solution to “the cache is causing too many database queries” is a smarter cache. It feels circular.
The pattern I reach for first in production is Stale-While-Revalidate with a background lock. It gives you the best user experience (no wait on stale reads), prevents duplicate background refreshes, and is easy to reason about. PER is elegant mathematically but harder to explain in a code review — and in production, the code you can explain to your team is usually the code that survives.
The thing I’ve seen go wrong most often in real systems is not choosing the wrong algorithm — it’s not adding jitter. It’s such a cheap fix that you’d think everyone does it by default. They don’t. And then a deployment at 2 AM warms the cache uniformly, every key expires 300 seconds later at exactly the same moment, and someone is paged at 2:05 AM wondering why the database is on fire.
TTL jitter is a one-liner. Add it everywhere. You’ll only thank yourself.