Kubernetes leader election answers an important question: which replica should lead now? For many controllers, that is enough. But when leader-only work can change durable state or trigger external effects, another question matters: can the old leader still act after it loses leadership?
The failure window
Imagine replica A holds a Lease and begins preparing a write. A pauses or loses connectivity. Replica B acquires the Lease and takes over. Later, A resumes and sends its delayed write.
The Lease correctly identifies B as the current leader. Whether A’s write is rejected depends on the system receiving that write. A valid workload identity alone does not prove that A still has leader authority.
Election and fencing solve different problems
Leader election chooses a leader. Fencing prevents a former leader from performing an authoritative action after it has lost that authority.
One common approach uses a monotonically increasing authority generation: A acts under generation 12; B takes over under generation 13. The component protecting the resource verifies the generation at the effect boundary and rejects stale operations. If A later becomes leader again, it receives a new generation.
The comparison must be enforced as part of accepting the protected operation. A sender-only check can race: A checks that it is leader, pauses, loses the Lease, then resumes and sends the operation. The receiver or protected storage must participate in rejecting stale authority.
Credential generations, leadership generations, object revisions and session generations have different meanings. A freshly rotated certificate does not make a replica the leader.
Test the old leader, not only the new one
A failover test that stops A and observes B becoming leader proves availability. It does not necessarily prove fencing. Add a test that resumes A or replays its delayed operation after B takes over, and verify that the protected system rejects it. Also test a lower generation, conflicting holders and retries from the current leader.
Fencing and idempotency are separate: fencing rejects stale authority; idempotency prevents a valid leader’s retries from creating duplicate effects.
Not every Kubernetes controller needs another protocol. A Kubernetes update protected by an appropriate resourceVersion check may provide the concurrency guarantee the operation needs. The gap deserves attention when leadership controls effects beyond that boundary—such as a database, remote runtime, external API or durable state machine.
Use the Lease for election, and add enforcement at the effect boundary where a stale leader could still cause harm.
Originally published as part of Beyond Kubernetes Defaults on the Zen Mesh blog. What failure cases have you found most useful when testing leader failover?