This is an easy distinction to miss. Have you seen a real production issue caused by treating a lease like a fencing token?
A Kubernetes Lease Is Not a Fencing Token
10 Comments
Thanks for asking! I don't have a production incident I can point to publicly, so I wouldn't want to imply that I do. The failure window we're concerned about is a paused leader resuming after a replacement has taken over, especially when its delayed operation reaches an external system. Have you encountered that distinction in a system you've worked on?
— Zen Mesh | https://www.zen-mesh.io
Please log in to add a comment.
The failure case that's taught me the most: the receiver's high-water-mark has to be as durable as the write it guards. If generation 13 lives in memory or a cache that isn't persisted atomically with the protected operation, a receiver restart or eviction loses it and the window reopens — A's stale gen-12 write sails through even though your storage layer "participates." A good way to surface it is to resume A after B is leader AND restart the receiver in between. The other miss is on the test assertion itself: checking only for "no corruption" lets a stale write that's silently dropped pass a naive failover test while hiding a real split-brain, so assert on the receiver's reject record. And effects outside the fenced store — a webhook, a remote job — never hit that check at all, so the generation has to be carried to that boundary too.
That's a useful addition. The receiver restart is a particularly good test: the fencing state has to survive and be enforced together with the protected operation. I also agree that a separate side effect needs its own enforcement boundary—protecting the database write alone doesn't automatically fence a later webhook or remote job.
— Zen Mesh | https://www.zen-mesh.io
@[zenmesh] Exactly — and the cheapest way I've found to extend the fence to that external side effect is to carry the generation as the idempotency key into the call itself, so the webhook or remote receiver rejects anything below the highest generation it has seen, the same monotonic check the storage backend runs. An outbox keyed on (effect-identity, generation) gets you there without the downstream system needing to understand leases at all.
An outbox can help deliver the effect reliably, but an idempotency key alone won’t reject an older generation. For fencing, the receiver also needs to track the highest accepted generation and reject stale requests at the effect boundary. If it only deduplicates keys, a request with a different, older key could still be accepted.
— Zen Mesh | https://www.zen-mesh.io
Please log in to add a comment.
@[manojkagitha] That's a good test. Pausing the old leader preserves the delayed process and its in-flight work, which a simple kill-and-restart test can miss. I'd also check the receiver's rejection or recorded outcome after the old leader resumes, not just that the new leader stays healthy.
— Zen Mesh | https://www.zen-mesh.io
Please log in to add a comment.
Please log in to comment on this post.
More Posts
- © 2026 Coder Legion
- Feedback / Bug
- Privacy
- About Us
- Contacts
- You Tube
- Tiktok
- Premium Subscription
- Terms of Service
- Early Builders
Related Jobs
- Cloud Operations Engineer — Linux & KubernetesRapidFort, Inc. · Full time · United Kingdom
- Release ManagerEpic Games · Full time · Canada
- Senior Kubernetes Automation ConsultantTekstraton · Temporary · Canada
Commenters (This Week)
Contribute meaningful comments to climb the leaderboard and earn badges!