Caching strategies

Protect the slow database by storing frequent results in fast memory.

The idea

A Cache-Aside pattern works like this: The app asks the Cache. If it's there (Hit), return it. If not (Miss), query the DB, put it in the Cache, and return it.

But what if a popular item expires (TTL) and suddenly 1,000 users request it at the exact same millisecond? All 1,000 get a cache miss and hit the DB simultaneously. This is a Cache Stampede. To fix it, we use Single-Flight (Request Coalescing): the first request locks the key and goes to the DB. The other 999 wait for that one DB call to finish, then share the result!

Enable Single-Flight

Normal request hits the cache and returns immediately.

How it works (Single-Flight)

# Without single-flight, 1000 misses = 1000 DB queries.
# With single-flight, 1000 misses = 1 DB query, 999 waiters.

def get_item(key):
    # 1. Try cache
    val = cache.get(key)
    if val: return val
    
    # 2. Cache miss! Use single-flight
    # 'group.do' ensures only ONE concurrent execution for this key
    val, err = single_flight_group.do(key, lambda: db.fetch(key))
    
    # 3. Fill cache and return
    cache.set(key, val)
    return val