Computer science has a famous saying, usually credited to Phil Karlton: "There are only two hard things in computer science: cache invalidation and naming things." It has been repeated so many times that it lost its sharpness. Most programmers laugh about that. Seasoned engineers, however, who have experienced production incidents due to a wrongly functioning cache no longer laugh. They even cringe.

Caching is one of the strongest weapons that developers have. If implemented correctly, it decreases the response time from hundreds of milliseconds to single digits, helps the database to survive an enormously high load and also makes the application feel like it is running instantaneously. However, when systems increase in size more users, more services, more data, more scale the cache that used to be helpful for your application may very well be its most dangerous liability.

This guide does not aim at introducing Redis or Memcached to you for the first time. It simply reveals the genuine, expensive, and problem areas are often overlooked when memory caches in-complex applications. Plus, it presents an effective engineering framework to solve each of them practically. The problems discussed here are not hypothetical ones, each of them has triggered significant production downtime in the live environment of actual companies. In fact, some of these companies figure among your daily-used brand names.

"Cache invalidation failures can spike database load by 13x in milliseconds. Facebook cut peak DB queries by 92% using lease-based locking alone." - SOURCE · USENIX, 2013 — Facebook's Memcached at Scale

PROBLEM 01 of 06

The Cache Stampede — When Your Safeguard Becomes the Attack

Also known as: Thundering Herd Problem

Think of a typical page of an e-commerce website showcasing a product. Information of that page has been saved in Redis for 10 minutes. If the website traffic is extremely high like during a flash sale, the caching time will be over. In the few milliseconds before the cache is again filled, 1, 000 customers who are using the website at the same time will be asking for the same page. None of them will find the cache. Each of them will send a query to the database. Your database, able to handle only a small load, is suddenly pinpointing one thousand identical queries at the same time.

This is a cache stampede, also the thundering herd problem. Twitter, Reddit, and Instagram have all publicly reported stampede incidents. The pattern is everywhere: high traffic + cache expiration + slow data source = major cascade failure.

Stampedes are extremely dangerous precisely because in development they are invisible. In development or low-traffic environments, you hardly ever see this behavior. Requests are spread out, and cache expirations go unnoticed. When you are dealing with a real production traffic, especially for very popular keys, the scenario changes completely.

REAL-WORLD FAILURE MODE

The Flash Sale Collapse

E-commerce platforms face a flash sale cache stampede when the product availability cache expires, resulting in a flood of database queries as thousands of users check out at the same time. The database fails. The site goes down. The sale is lost.

 VULNERABLE PATTERN (Python / Redis)

# ❌ PROBLEM — Every cache miss fires a DB query
async def get_product(product_id):
    cached = await redis.get(f"product:{product_id}")
    if cached:
        return json.loads(cached)
    # 1000 concurrent requests all hit this line simultaneously
    product = await db.query("SELECT * FROM products WHERE id = %s", product_id)
    await redis.set(f"product:{product_id}", json.dumps(product), ex=600)
    return product

# ✅ FIX 1 — TTL Jitter spreads expiration across a time window
import random
ttl = 600 + random.randint(-60, 60)  # 9–11 min window, not exact 10
await redis.set(key, value, ex=ttl)

# ✅ FIX 2 — Distributed lock allows only ONE rebuild at a time
lock = redis.lock(f"lock:product:{product_id}", timeout=10)
if await lock.acquire(blocking=False):
    try:
        product = await db.query(...)
        await redis.set(key, json.dumps(product), ex=ttl)
    finally:
        await lock.release()
else:
    # Another process is rebuilding — serve stale or wait briefly
    await asyncio.sleep(0.1)
    return await get_product(product_id)

SOLUTIONS FOR CACHE STAMPEDE

• TTL Jitter: A good practice would be adding a bit of randomness (±10, 20%) to the expiration time so that keys almost never expire at the same time. Experiments show that 30-seconds randomness would be enough to stop most of the issues.
• Distributed Locking: The simplest way would be to implement a mutex so at a time only one process is responsible for rebuilding the cache entry while the others wait or serve stale data.
• Probabilistic Early Expiration (XFetch): This is done by refreshment of a cache key randomly before expiration, thus spreading the rebuild over time.
• Request Coalescing (Singleflight): When the multiple requests miss the same key simultaneously one is allowed to hit the database and the rest of the requests wait this one and share the result.
• Background Refresh: Stale cache entry can be served immediately while asynchronously rebuilding the cache in the background.

PROBLEM 02 of 06

Cache Invalidation — The Hardest Problem in Computer Science

Also known as: Stale Data, Dirty Cache, Cache Poisoning

Phil Karlton's well-known saying is not just for fun. Cache invalidation is, in fact, a very difficult problem in software engineering, because there isn't one method that can solve all situations. Each option comes with a compromise between consistency and performance, and the only way to decide the best solution is by looking at the specific situation.

The point is simple: your cache is a copy of some data. That data is updated at the source of truth (database, API, file). However, your cache is still containing the old version. All users who get data from the cache receive the incorrect data, and if no one comes to know, they might even base their actions on it.

In​‍​‌‍​‍‌ a monolithic application, this is pretty challenging. In microservices architecture, the level of complexity goes up even more, because the cache is often spread over several different services that are independently deployed. Maintaining cache coherence in microservices is a harder problem than monolithic applications partly because different services might use different concurrency models and even different communication ​‍​‌‍​‍‌protocols.

REAL-WORLD FAILURE MODE

The Price Update Problem

Consider an e-commerce system which updates a product price in the database. Its cache, having a 5-minute TTL, remains to provide the users with the old price. It means for a span of 5 minutes customers will not only see but also be able to checkout based on the incorrect price. If you multiply it by thousands of items and millions of users, it turns into a huge financial and compliance issue.

The Three Core Invalidation Strategies

• Write-Through: Each time a write operation is performed on the database, it will also synchronizedly update the cache. Cache and database are always consistent. One​‍​‌‍​‍‌ downside is that it takes a lot of time to write and eventually you may cache a lot of data that is hardly ​‍​‌‍​‍‌read.
• Write-Behind (Write-Back): With this method, the writes are made to the cache right away, and then asynchronously the changes are flushed to the database. Writes are super quick but a period is created where the cache and the database are out of sync, if the cache goes down before the flush, the data could be lost.
• Event-Driven Invalidation: Upon each update to data, a notification is sent via a message posted on a queue (Kafka, RabbitMQ, Redis Pub/Sub). Cache nodes listen for these messages and either clear or refresh the associated cache entries. This method is not only the most scalable for distributed systems but also the one that is highly advised for microservices architectures.

SOLUTIONS FOR CACHE INVALIDATION

• Write-Through for read-heavy, important data: User account information, product stock, pricing, anything that delivering stale information would lead to an actual business loss.
• Event-driven invalidation for distributed architectures: Whenever the main data source changes, send out a cache invalidation event. All cache nodes listening will invalidate the corresponding key.
• Short TTL as a safety net: Set a short TTL even if you have an active invalidation process because stale data can not be kept forever in this way.
• Cache versioning: Attach a version number or hash to the cached key. Changing data means changing the key, the old cached entries become outdated naturally without needing explicit invalidation.
• Never cache data without an expiry: An infinite TTL is just a stale data incident waiting to happen.

PROBLEM 03 of 06

Cache Penetration — The Attack That Bypasses Your Cache Entirely

Also known as: Null Key Flooding, Ghost Key Attack

Cache penetration is a situation when people try to get the data that is not in cache and the database. The reason why cache has no record for the one that does not exist in the key, is that every request after that is the database. What happens is that an attacker or some very bad requests suddenly hitting your system for thousands of non-existent keys, then your database gets each one of them.

Howerver, Cache penetration is the situation where a client, possibly a malicious one, is requesting a key that does not exist in both the cache and the database. The database, upon failing to find the key, responds with a null value. These malicious clients then continue to send the requests for the non-existent key that eventually leading to the flood of the queries that is be able to overwhelm the database and also degrade its performance.

 CACHING — Cache the Absence of Data

async def get_user(user_id):
    cache_key = f"user:{user_id}"
    cached = await redis.get(cache_key)

    if cached == "__NULL__":
        # We cached the fact this user doesn't exist
        return None

    if cached:
        return json.loads(cached)

    user = await db.find_user(user_id)

    if user is None:
        # Cache the NULL result with a short TTL — stops DB flooding
        await redis.set(cache_key, "__NULL__", ex=60)
        return None

    await redis.set(cache_key, json.dumps(user), ex=3600)
    return user

SOLUTIONS FOR CACHE PENETRATION

• Negative caching: Cache even a database query which returns null, but with a very short TTL (30, 60 seconds). So, the next time the request for the same non-existent key is made, it will hit the cache and get served very quickly.
• Bloom filters: Use an in-memory probabilistic data structure which can very quickly check whether a key can exist or not. When bloom filter answers the query with a "no, " then the request can completely skip the cache and database.
• Request validation: Make sure to validate and sanitise all cache key inputs at the application layer before these inputs reach the cache or database.
• Rate limiting: Implement per-user or per-IP rate limits for incoming requests that end up missing both the cache and database these are very strongly indicative of either client abuse or a misbehaved client.

PROBLEM 04 of 06

Cache Avalanche — When Everything Expires at Once

Also known as: Mass Expiry, Cache Collapse

A cache avalanche is basically a really massive cache stampede. At the core of a stampede is the expiry of a single hot key, whereas an avalanche means that a large number of cache keys expire at the same time, or a cache node even fails, which causes a bunch of requests to the underlying database all at once.

It is a common scenario of an event occurrence. For example, a server restart will flush all caches in memory. A deployment will also invalidate all cached entries. Or all keys were set with the same TTL fixed value during a bulk loading operation. The end result is the same: the database, which was previously handling only the fraction of total traffic, is suddenly overwhelmed with nothing but the traffic.

Cache avalanche typically occurs when multiple or even all cache entries expire at the same time or within a short time window, thereby triggering a sudden surge in requests hitting the underlying data store. Peak traffic times, like the Black Friday sales, are prime examples when these types of incidents can easily occur.

REAL-WORLD FAILURE MODE

The Post-Deployment Collapse

A development group launches a new edition of their application. As a step of the deployment, they clear all Redis cache entries so that no stale data is served. Traffic immediately goes back up to the full level. Not having any cache warm-up, each request quite naturally reaches the database. The database breaks down as it gets a huge load that it was never designed to handle by its lonely self. The crew decides to undo the deployment, but it turns out the database was in trouble already.

SOLUTIONS FOR CACHE AVALANCHE

• TTL Staggering: Never make all the cache entries expire at the same time. Spread the randomised jitter of TTLs to make the mass expiration not possible.
• Cache warming-up plan: Before redirecting live traffic, after a restart or deployment, fill the critical cache keys with data first. Slowly increase the traffic instead of switching suddenly.
• Multi-layer cache: Have a small cache in the process that serves as L1 cache together with an external L2 cache (Redis). In case Redis is down, the in-process cache serves as a backup for the frequent requests.
• Circuit breaker: Add circuit breakers that notice the spikes in database loading above the normal level and cite to load shedding that is, rejecting or queuing requests rather than allowing the database to be overwhelmed.
• Durable cache (AOF/RDB in Redis): Turn on persistence so that when the server is restarted, the cache is loaded from the disk instead of being cold.

PROBLEM 05 of 06

Cache Inconsistency in Distributed Systems

Also known as: Cache Coherence Problem, Split-Brain Cache

Multiple service instances each having their local in-memory caches in a distributed system would cause consistency issues to the structural problem of the system. The separate service instances will have isolated local in-memory caches. If more instances of the same service are operating behind a load balancer, each one would have its own separate cache, which would lead to data inconsistency among instances. If the cache entry of one instance is modified, other instances will continue to live in ignorance of the change and could use stale data from their caches.

It is not a bug but a natural consequence of architectural side effect of horizontal scaling. Two users accessing the same endpoint might receive different response depending on the instance that will serve them. In financial systems, healthcare, or any other domain where consistency is required, such a situation is unacceptable.

The Microservices Compounding Effect

What complicates the problem even more in microservices architectures is that not only a single service but multiple ones may be involved with data caching. For example, Service A may cache some data about a user, while Service B is the one that actually executes an update on that user's data. In this case, Service A's cache is completely unaware that the updating has happened. Mainly, it's the user who will experience inconsistency in the system different parts of the application show different "truths" about the same data.

SOLUTIONS FOR DISTRIBUTED CACHE INCONSISTENCY

• Centralised distributed cache (Redis/Memcached): Instead of each instance having its own in-memory cache, use a shared external one that all instance access for both reading and writing. This way all the instances will have the same data.
• Event-driven cache synchronisation: Broadcast cache invalidation events using a message bus (Kafka, RabbitMQ). In case of data update, the service commits an event, and all clients consuming that data perform cache invalidation.
• TLRU (Time-aware Least Recently Used): According to studies, distributed TLRU implementations can cut down the delivery of stale data by as much as 35% relative to standard LRU while still achieving similar hit ratios.
• Read-your-writes consistency: Following a write operation, subsequent reads from the same user can be directed to the same instance (sticky sessions) or to the source of truth for a short period of time.
• Accept eventual consistency deliberately: Immediate consistency is not necessary for all types of data. Identify and explicitly state per-data-type consistency requirements. For cases where eventual consistency is allowed, make a note of it and set appropriate TTLs.

PROBLEM 06 of 06

Memory Pressure, Eviction Chaos & Cold Start

Also known as: Cache Thrashing, OOM Kill, Warm-Up Hell

Caches are constrained by a very basic limitation: cache space is limited and the volume of data that can be stored in the cache is fairly small. If data placement in the cache is not adequately controlled, then the hit rate can suffer. Especially, in intricate applications, making a wrong choice for the eviction policy may turn a cache that was originally meant to be a performance-oriented tool into a performance bottleneck by evicting the exactly your application needs most while retaining the rarely accessed ones.

Cache thrashing is a phenomenon that occurs when the cache size is not sufficient to accommodate the working set of data the set of data being actively accessed. The cache keeps on removing the items that are needed again quite soon, leading to a situation where the miss rate is almost 100%, which is often even worse than having no cache at all, since each miss adds to the latency and puts the database under load.

12–18%
Hit ratio improvement using ARC over LRU in production microservices

35%
Reduction in stale data delivery with TLRU vs standard LRU

Cold Start
New instances start with empty caches, spiking DB load on every deploy

SOLUTIONS FOR MEMORY PRESSURE & EVICTION

• Right-size your cache: First, understand the profile of your working set. Then, provide your cache memory accordingly. Too small a cache results in frequent thrashes, while an overly large cache ends up wasting your costly RAM.
• Pick the right eviction policy thoughtfully: On the one hand, LRU (Least Recently Used) is a good fit for bursty access patterns. On the other hand, LFU (Least Frequently Used) is suitable for more stable, popularity-driven access scenarios. But if the workload is mixed, ARC (Adaptive Replacement Cache) will adapt dynamically and even outperform the other two.
• Bound caches by byte size, not item count: A common mistake is to set up the cache size as "1000 items." But items can differ very much in size. Always bounding by bytes is the way to go so as not to lead to memory overruns.
• Implement cache warming: It means that, prior to the main data traffic, you get cache keys with the highest priority already populated at startup. So you don't have a cold start spike with every deployment.
• Compress large values: In order to increase the effective cache capacity without spending more on memory, you can use LZ4 or Snappy compression for large cached objects.
• Monitor hit rate continuously: When the hit rate drops, it means that you are about to have a problem. You should consider it to be a crucial metric together with latency and error rate.

The Six Problems at a Glance

The Real Lesson: Cache Is Architecture, Not an Afterthought

All the issues mentioned in this article have one thing in common: cache was just treated as a simple optimisation that could be added later on, instead of being a major architectural element. You can't simply add a cache to an already complex system as an afterthought and expect it to work properly. You must make design decisions regarding not only the cache but also failure scenarios of the cache.

Those who create systems that can process millions of requests without failing are not the geniuses that others are. They simply understand that a cache is not just a data storage it contains assumptions. Assumptions about data freshness, about legit requests, about the number of users who can come at the same time, and about the consequences of a node failure. Each of those assumptions needs to be scrutinised, tested, and justified.

Have the same kind of attention to detail for cache invalidation as you do for database schema design. Deal with the failure modes of the cache as seriously as you would handle service outages. Track cache hit rate, eviction rate, and memory pressure with the same eagerness as you track CPU and latency. Do these things and your cache will serve its original purpose: providing speed, reliability, and resilience to your application.

Not following these steps will make your cache lie to you at the worst possible moment.