I Killed Redis With SIGKILL to Test Payment Idempotency — Here Is What Broke

●2 ●5
calendar_today ago • schedule4 min read

Most tutorials treat Redis like a magic box: wrap your endpoint in a SET key value NX check, and congratulations—your payment API is now idempotent.

In production, caches do not fail politely. Containers get OOM-killed mid-request, TCP connection pools hold onto dead sockets, and background disk writes trigger Linux kernel memory spikes that can take down the entire node.

To move beyond happy-path tutorials, I built an empirical lab (redis-durability-lab) using FastAPI, Redis 7+, and PostgreSQL. I deliberately injected SIGKILL crashes, stale socket pools, and heavy write loads to answer three questions:

  1. What actually happens to in-flight payment retries when Redis dies abruptly?
  2. How much throughput do you sacrifice for strict disk durability (appendfsync always)?
  3. How much hidden RAM does Linux Copy-On-Write (COW) consume during background persistence?

You can explore the full code and reproduction scripts in the repository:
https://github.com/bhargavdas99/redis-durability-lab


The Architecture: Dual-Layer Idempotency

In payment systems, availability and strict consistency pull in opposite directions. If Redis goes down, rejecting every payment request hurts conversion. If you bypass idempotency checks during a Redis outage, a network retry causes a double charge.

The solution is a dual-layer architecture where Redis handles fast-path deduplication and PostgreSQL acts as the ACID source of truth:

Client Request (Idempotency-Key)
        │
        ▼
┌───────────────────────────────┐
│        FastAPI Gateway        │
└───────────────┬───────────────┘
                │
                ▼
┌───────────────────────────────┐      Cache Hit ("COMMITTED")
│  Layer 1: Redis Atomic Lock   ├──────────────────────────────────► Return 201 (Cached)
│  SET key "PROCESSING" NX EX   │
└───────────────┬───────────────┘
                │ Lock Acquired OR Redis Unreachable (Graceful Degradation)
                ▼
┌───────────────────────────────┐      Unique Constraint Violation
│ Layer 2: PostgreSQL Ledger    ├──────────────────────────────────► Rollback & Return 201 (DB Fallback)
│ INSERT INTO payment_intents   │
└───────────────┬───────────────┘
                │ Commit Success
                ▼
        Update Redis -> "COMMITTED"
                │
                ▼
        Return 201 (Success)

  • Layer 1 (Redis): Uses atomic SET key "PROCESSING" NX to reject concurrent duplicate requests (409 Conflict) and serve already-completed retries directly from RAM without touching the database.
  • Layer 2 (PostgreSQL): Enforces a database-level UNIQUE constraint on idempotency_key. If Redis loses data or becomes unreachable, the application degrades gracefully to PostgreSQL—guaranteeing zero double-spends.

War Story #1: Two Bugs Exposed by docker kill --signal="SIGKILL"

When I tested graceful degradation by killing the Redis container mid-flight (docker kill --signal="SIGKILL" redis_lab) and restarting it, two subtle bugs surfaced immediately.

1. Stale TCP Sockets After Restart

Even with Append-Only File (appendonly yes) enabled—meaning Redis successfully recovered the COMMITTED key from disk on boot—the very first retry after restart bypassed Redis and hit PostgreSQL anyway.

Why? Async connection pools (redis.asyncio) keep open TCP sockets alive for reuse. When Docker killed the Redis process, the client pool still held dead file descriptors. The first post-crash command tried writing to a broken pipe, raised a ConnectionError, and falsely triggered the "Redis is down" fallback path.

The Fix: Configure explicit connection health checks and exponential retries on the async client so dead sockets are discarded and re-established transparently:

from redis.asyncio import Redis
from redis.asyncio.retry import Retry
from redis.backoff import ExponentialBackoff
from redis.exceptions import ConnectionError, TimeoutError

redis_client = Redis.from_url(
    "redis://localhost:6379",
    decode_responses=True,
    retry=Retry(ExponentialBackoff(), 2),
    retry_on_error=[ConnectionError, TimeoutError],
    health_check_interval=5,
)

2. The Silent null Response Bug

When Redis was genuinely offline (redis_available = False), retries hit PostgreSQL's UNIQUE constraint as expected—but FastAPI returned an HTTP 201 with a null JSON body.

Tracing the control flow revealed why: inside the except IntegrityError: block, the fallback response return was accidentally nested inside an if redis_available: check. When Redis was unreachable, execution fell off the end of the function and implicitly returned Python's None.

Lesson: Always decouple cache state synchronization from your HTTP return path. Never let a cache outage swallow a database fallback payload.


War Story #2: Linux Copy-On-Write (COW) Memory Spikes

A classic production incident occurs when a Redis container running at 75% memory capacity suddenly gets killed by the Linux OOM Killer during a routine background save (BGSAVE or BGREWRITEAOF).

When Redis forks a child process to write a snapshot to disk, the Linux kernel uses Copy-On-Write (COW): parent and child share the same 4 KB physical memory pages until the parent mutates a key, forcing the kernel to duplicate that page in RAM.

To measure this empirically:

  1. I seeded ~82 MB of data (100,000 keys with 1 KB payloads).
  2. I triggered BGREWRITEAOF while simultaneously blasting 300,000 pipelined mutations (-P 16 at ~372,670 req/s).
  3. I inspected INFO persistence and INFO memory immediately after:
  • Baseline used_memory_human: 81.92 MB (used_memory_rss_human: 93.43 MB)
  • aof_last_cow_size: 23,990,272 bytes (~22.88 MB — ~28% memory amplification during a sub-second rewrite)
  • Peak used_memory_rss_human: 138.34 MB

Because the Linux OOM Killer evaluates physical Resident Set Size (RSS) rather than Redis's internal used_memory, production Redis instances with persistence enabled should cap maxmemory at 50%–65% of container memory limits.


Benchmarking Persistence Modes & fsync Latency

Redis 7+ uses Multi-Part Hybrid Persistence (aof-use-rdb-preamble yes by default). Inspecting /data/appendonlydir after a rewrite shows how it combines fast RDB boot times with AOF durability:

appendonly.aof.6.base.rdb   # Compact binary RDB snapshot for fast startup
appendonly.aof.6.incr.aof   # Incremental RESP command log for crash recovery
appendonly.aof.manifest     # Sequence metadata linking base and incremental files

The remaining question was the fsync policy: should you flush the AOF buffer to disk once per second (appendfsync everysec) or on every single write (appendfsync always)?

Running redis-benchmark -t set -n 20000 -c 50 -q on NVMe storage revealed the exact cost:

appendfsync Policy Throughput (SET req/s) Median Latency (p50) Relative Throughput Crash Window
everysec 173,913.05 req/s 0.151 ms 1.00x (Baseline) Up to ~1 second on power loss
always 35,650.62 req/s 1.247 ms 0.20x (~80% drop) Zero data loss

Key Takeaways

  1. Do not force Redis to act like PostgreSQL: Switching to appendfsync always cuts throughput by ~80% and increases median latency by >8x (0.151 ms to 1.247 ms).
  2. Pair everysec with an ACID constraint: Running Redis with Hybrid Persistence + appendfsync everysec gives you 173k+ req/s throughput and survives SIGKILL crashes cleanly. On the rare occasion a hard power loss wipes an unflushed sub-second write, PostgreSQL's UNIQUE constraint catches the duplicate retry with zero financial risk.
  3. Budget RAM for the kernel, not just the application: Always leave 35%–50% RSS headroom on persistent Redis containers so Linux Copy-On-Write never triggers an OOM cascade during peak traffic.

If you want to run these crash tests and benchmarks locally, clone the repo and spin up the Docker Compose stack:
https://github.com/bhargavdas99/redis-durability-lab

1 Comment

0 votes
🔥 Join developers growing publicly
Share your knowledge, build in public, and grow your developer presence with a global community.

More Posts

Dashboard Operasional Armada Rental Mobil dengan Python + FastAPI

Masbadar - Mar 12

MCP Is the USB-C of AI. So Why Are You Plugging Everything In?

Ken W. Algerverified - Jun 10

How AI Is Changing Web Development, and What Developers Still Need to Understand

Daniel Decker - Nov 10

Make for Small Business: What It Is, What It Costs, and When to Use It

floxolab - Oct 23

Be Careful with SQL JOINs: Indexes, the Join Cliff and Join Algorithms

Ilja Nevolin - Nov 12
chevron_left
168 Points • 7 Badges
2Posts
1Comments
2Connections
Backend Systems Engineer focused on high-concurrency architecture, Python, FastAPI, and PostgreSQL. ... Show more

Related Jobs

View all jobs →

Commenters (This Week)

1 comment
1 comment

Contribute meaningful comments to climb the leaderboard and earn badges!