Most tutorials treat Redis like a magic box: wrap your endpoint in a SET key value NX check, and congratulations—your payment API is now idempotent.
In production, caches do not fail politely. Containers get OOM-killed mid-request, TCP connection pools hold onto dead sockets, and background disk writes trigger Linux kernel memory spikes that can take down the entire node.
To move beyond happy-path tutorials, I built an empirical lab (redis-durability-lab) using FastAPI, Redis 7+, and PostgreSQL. I deliberately injected SIGKILL crashes, stale socket pools, and heavy write loads to answer three questions:
- What actually happens to in-flight payment retries when Redis dies abruptly?
- How much throughput do you sacrifice for strict disk durability (
appendfsync always)?
- How much hidden RAM does Linux Copy-On-Write (COW) consume during background persistence?
You can explore the full code and reproduction scripts in the repository:
https://github.com/bhargavdas99/redis-durability-lab
The Architecture: Dual-Layer Idempotency
In payment systems, availability and strict consistency pull in opposite directions. If Redis goes down, rejecting every payment request hurts conversion. If you bypass idempotency checks during a Redis outage, a network retry causes a double charge.
The solution is a dual-layer architecture where Redis handles fast-path deduplication and PostgreSQL acts as the ACID source of truth:
Client Request (Idempotency-Key)
│
▼
┌───────────────────────────────┐
│ FastAPI Gateway │
└───────────────┬───────────────┘
│
▼
┌───────────────────────────────┐ Cache Hit ("COMMITTED")
│ Layer 1: Redis Atomic Lock ├──────────────────────────────────► Return 201 (Cached)
│ SET key "PROCESSING" NX EX │
└───────────────┬───────────────┘
│ Lock Acquired OR Redis Unreachable (Graceful Degradation)
▼
┌───────────────────────────────┐ Unique Constraint Violation
│ Layer 2: PostgreSQL Ledger ├──────────────────────────────────► Rollback & Return 201 (DB Fallback)
│ INSERT INTO payment_intents │
└───────────────┬───────────────┘
│ Commit Success
▼
Update Redis -> "COMMITTED"
│
▼
Return 201 (Success)
- Layer 1 (Redis): Uses atomic
SET key "PROCESSING" NX to reject concurrent duplicate requests (409 Conflict) and serve already-completed retries directly from RAM without touching the database.
- Layer 2 (PostgreSQL): Enforces a database-level
UNIQUE constraint on idempotency_key. If Redis loses data or becomes unreachable, the application degrades gracefully to PostgreSQL—guaranteeing zero double-spends.
War Story #1: Two Bugs Exposed by docker kill --signal="SIGKILL"
When I tested graceful degradation by killing the Redis container mid-flight (docker kill --signal="SIGKILL" redis_lab) and restarting it, two subtle bugs surfaced immediately.
1. Stale TCP Sockets After Restart
Even with Append-Only File (appendonly yes) enabled—meaning Redis successfully recovered the COMMITTED key from disk on boot—the very first retry after restart bypassed Redis and hit PostgreSQL anyway.
Why? Async connection pools (redis.asyncio) keep open TCP sockets alive for reuse. When Docker killed the Redis process, the client pool still held dead file descriptors. The first post-crash command tried writing to a broken pipe, raised a ConnectionError, and falsely triggered the "Redis is down" fallback path.
The Fix: Configure explicit connection health checks and exponential retries on the async client so dead sockets are discarded and re-established transparently:
from redis.asyncio import Redis
from redis.asyncio.retry import Retry
from redis.backoff import ExponentialBackoff
from redis.exceptions import ConnectionError, TimeoutError
redis_client = Redis.from_url(
"redis://localhost:6379",
decode_responses=True,
retry=Retry(ExponentialBackoff(), 2),
retry_on_error=[ConnectionError, TimeoutError],
health_check_interval=5,
)
2. The Silent null Response Bug
When Redis was genuinely offline (redis_available = False), retries hit PostgreSQL's UNIQUE constraint as expected—but FastAPI returned an HTTP 201 with a null JSON body.
Tracing the control flow revealed why: inside the except IntegrityError: block, the fallback response return was accidentally nested inside an if redis_available: check. When Redis was unreachable, execution fell off the end of the function and implicitly returned Python's None.
Lesson: Always decouple cache state synchronization from your HTTP return path. Never let a cache outage swallow a database fallback payload.
War Story #2: Linux Copy-On-Write (COW) Memory Spikes
A classic production incident occurs when a Redis container running at 75% memory capacity suddenly gets killed by the Linux OOM Killer during a routine background save (BGSAVE or BGREWRITEAOF).
When Redis forks a child process to write a snapshot to disk, the Linux kernel uses Copy-On-Write (COW): parent and child share the same 4 KB physical memory pages until the parent mutates a key, forcing the kernel to duplicate that page in RAM.
To measure this empirically:
- I seeded ~82 MB of data (
100,000 keys with 1 KB payloads).
- I triggered
BGREWRITEAOF while simultaneously blasting 300,000 pipelined mutations (-P 16 at ~372,670 req/s).
- I inspected
INFO persistence and INFO memory immediately after:
- Baseline
used_memory_human: 81.92 MB (used_memory_rss_human: 93.43 MB)
aof_last_cow_size: 23,990,272 bytes (~22.88 MB — ~28% memory amplification during a sub-second rewrite)
- Peak
used_memory_rss_human: 138.34 MB
Because the Linux OOM Killer evaluates physical Resident Set Size (RSS) rather than Redis's internal used_memory, production Redis instances with persistence enabled should cap maxmemory at 50%–65% of container memory limits.
Benchmarking Persistence Modes & fsync Latency
Redis 7+ uses Multi-Part Hybrid Persistence (aof-use-rdb-preamble yes by default). Inspecting /data/appendonlydir after a rewrite shows how it combines fast RDB boot times with AOF durability:
appendonly.aof.6.base.rdb # Compact binary RDB snapshot for fast startup
appendonly.aof.6.incr.aof # Incremental RESP command log for crash recovery
appendonly.aof.manifest # Sequence metadata linking base and incremental files
The remaining question was the fsync policy: should you flush the AOF buffer to disk once per second (appendfsync everysec) or on every single write (appendfsync always)?
Running redis-benchmark -t set -n 20000 -c 50 -q on NVMe storage revealed the exact cost:
appendfsync Policy | Throughput (SET req/s) | Median Latency (p50) | Relative Throughput | Crash Window |
everysec | 173,913.05 req/s | 0.151 ms | 1.00x (Baseline) | Up to ~1 second on power loss |
always | 35,650.62 req/s | 1.247 ms | 0.20x (~80% drop) | Zero data loss |
Key Takeaways
- Do not force Redis to act like PostgreSQL: Switching to
appendfsync always cuts throughput by ~80% and increases median latency by >8x (0.151 ms to 1.247 ms).
- Pair
everysec with an ACID constraint: Running Redis with Hybrid Persistence + appendfsync everysec gives you 173k+ req/s throughput and survives SIGKILL crashes cleanly. On the rare occasion a hard power loss wipes an unflushed sub-second write, PostgreSQL's UNIQUE constraint catches the duplicate retry with zero financial risk.
- Budget RAM for the kernel, not just the application: Always leave 35%–50% RSS headroom on persistent Redis containers so Linux Copy-On-Write never triggers an OOM cascade during peak traffic.
If you want to run these crash tests and benchmarks locally, clone the repo and spin up the Docker Compose stack:
https://github.com/bhargavdas99/redis-durability-lab