Leader Election (Resiliency)

TL;DR

Leader election is a coordination pattern that designates one node in a distributed system as the authoritative decision-maker, preventing conflicts when multiple nodes could perform the same task. It ensures exactly one leader exists at any time, even during network partitions and node failures. Critical for systems requiring coordination like distributed databases (MongoDB replica sets), job schedulers (Kubernetes control plane), and cache invalidation coordinators.

Cheat Sheet: Use Raft for strong consistency + simplicity, Bully for small clusters with stable networks, ZooKeeper/etcd for production-grade external coordination. Always implement lease expiration (typical: 10-30s) and fencing tokens to prevent split-brain. Election time vs consistency is your key trade-off: fast elections (Bully: <1s) sacrifice partition tolerance, consensus-based elections (Raft: 150ms-5s) guarantee safety.

The Problem It Solves

Distributed systems face a fundamental coordination problem: when multiple nodes can perform the same operation, how do you prevent them from stepping on each other’s toes? Consider a distributed cron scheduler running on five servers. Without coordination, all five nodes might trigger the same job simultaneously, causing duplicate charges, race conditions, or resource exhaustion. You can’t just pick a “primary” server at startup because that server will eventually fail, and you need automatic failover.

The naive solution—letting every node check “am I the lowest ID still alive?”—creates race conditions during failures. Two nodes might both think they’re the leader during a network partition (split-brain), leading to data corruption. MongoDB learned this lesson the hard way in 2013 when a network partition caused two primaries to accept writes simultaneously, resulting in divergent data that required manual reconciliation.

Leader election solves this by providing a protocol that guarantees exactly one leader exists, even when networks partition and nodes crash. The leader becomes the single authority for coordination tasks: assigning work, making decisions, or serving as the write master. When the leader fails, the system automatically elects a new one without human intervention. This transforms unreliable distributed systems into resilient ones that self-heal from failures.

Solution Overview

Leader election establishes a single coordinator through a distributed agreement protocol. Nodes participate in an election process that selects one leader based on criteria like node ID, term number, or log completeness. The elected leader maintains its authority through periodic heartbeats or lease renewals, proving it’s still alive. When followers stop receiving heartbeats, they trigger a new election.

The pattern separates into two phases: election (choosing a leader) and maintenance (keeping the leader valid). During election, nodes exchange messages to reach consensus on who should lead. The winner broadcasts its leadership to all nodes. During maintenance, the leader sends regular “I’m alive” signals. If these signals stop, followers assume the leader died and start a new election.

Crucially, the protocol prevents split-brain through versioning mechanisms. Each leadership term gets a monotonically increasing number (epoch, term, or generation). Nodes reject commands from leaders with outdated term numbers, ensuring old leaders can’t cause damage after being replaced. This is similar to how Stripe’s payment processing uses fencing tokens—each leader gets a unique, increasing token, and operations are rejected if they carry an old token.

The pattern requires a quorum (majority) for decisions, meaning elections succeed only when more than half the nodes agree. This prevents two leaders from being elected in different network partitions. If a partition isolates two nodes from three nodes, only the three-node partition can elect a leader because it has the majority.

Leader Election: Split-Brain Prevention with Quorum

graph TB
    subgraph Network Partition Scenario
        subgraph Partition A - Minority
            N1["Node 1"]
            N2["Node 2"]
        end
        
        subgraph Partition B - Majority
            N3["Node 3<br/>(Candidate)"]
            N4["Node 4"]
            N5["Node 5"]
        end
    end

    N1 -."❌ Network partition".-> N3
    N2 -."❌ Network partition".-> N4

    N3 --"RequestVote<br/>Term 6"--> N4
    N3 --"RequestVote<br/>Term 6"--> N5
    N4 --"✓ VoteGranted"--> N3
    N5 --"✓ VoteGranted"--> N3

    N3 --> Result3["✅ Node 3 elected<br/>3/5 votes (majority)<br/>Term 6"]

    N1 --"RequestVote<br/>Term 6"--> N2
    N2 --"✓ VoteGranted"--> N1
    N1 --> Result1["❌ Node 1 CANNOT be elected<br/>2/5 votes (no majority)<br/>Remains follower"]

    subgraph Fencing Token Protection
        Op1["Operation from N1<br/>Token: Term 6"]
        Op3["Operation from N3<br/>Token: Term 6"]
        Service["Execution Service<br/>Current Term: 6"]
    end

    Op1 --"Execute job"--> Service
    Service -."❌ Rejected<br/>(N1 not leader)".-> Op1
    Op3 --"Execute job"--> Service
    Service --"✅ Accepted<br/>(N3 is leader)"--> Op3

Split-brain prevention through quorum-based voting. During a network partition, only Partition B (3 nodes) can elect a leader because it has the majority (3/5 votes). Partition A (2 nodes) cannot reach quorum. Even if both partitions attempt elections, the execution service uses term numbers as fencing tokens to reject operations from non-leaders.

How It Works

Let’s walk through a typical Raft-style leader election in a five-node cluster running a distributed job scheduler.

Step 1: Steady State Operation Node A is the leader with term number 5. Every 100ms, it sends heartbeat messages to nodes B, C, D, and E. These heartbeats say “I’m still the leader for term 5.” Followers reset their election timers when they receive heartbeats. The leader assigns scheduled jobs to followers, who execute them and report results back.

Step 2: Leader Failure Detection Node A crashes. Followers B through E stop receiving heartbeats. Each follower has a randomized election timeout (150-300ms). Node C’s timer fires first at 180ms. It transitions from follower to candidate and increments the term number to 6.

Step 3: Requesting Votes Node C sends “RequestVote” messages to all other nodes: “I want to be leader for term 6. My last log entry is from term 5, index 1247.” Each node can vote for only one candidate per term. Nodes B, D, and E compare their logs with C’s. If C’s log is at least as up-to-date as theirs (same term, equal or higher index), they grant their vote.

Step 4: Winning the Election Node C receives votes from B, D, and E (plus its own vote). That’s 4 out of 5 nodes—a majority. Node C becomes the leader for term 6 and immediately sends heartbeats to all nodes, establishing its authority. Node E’s election timer also fired, but when it sends RequestVote messages with term 6, other nodes respond “I already voted for C in term 6,” so E’s election fails.

Step 5: Handling Stale Leaders Node A recovers and thinks it’s still the leader for term 5. It sends a heartbeat with term 5. Node C receives this message, sees the term is lower than its current term (6), and responds “You’re outdated. I’m the leader for term 6.” Node A updates its term to 6, converts to follower, and accepts C’s leadership.

Step 6: Lease-Based Maintenance In production systems like Kubernetes, leaders hold time-bound leases (typically 15 seconds). Node C must renew its lease with the coordination service (etcd) every 10 seconds. If C crashes, its lease expires after 15 seconds, and other nodes can immediately start a new election without waiting for heartbeat timeouts. This reduces failover time from seconds to milliseconds.

Step 7: Preventing Split-Brain with Fencing Imagine a network partition separates C from the other nodes. C thinks it’s still the leader, but B, D, and E elect a new leader (D) for term 7. When the partition heals, C might try to execute operations. The system prevents this using fencing tokens: every operation includes the term number. When C tries to execute a job with term 6, the job execution service rejects it because it has seen term 7. Only D’s operations with term 7 are accepted.

Raft Leader Election: Complete Failure and Recovery Flow

sequenceDiagram
    participant A as Node A<br/>(Leader, Term 5)
    participant B as Node B<br/>(Follower)
    participant C as Node C<br/>(Follower)
    participant D as Node D<br/>(Follower)
    participant E as Node E<br/>(Follower)

    Note over A,E: Phase 1: Steady State
    A->>B: Heartbeat (Term 5)
    A->>C: Heartbeat (Term 5)
    A->>D: Heartbeat (Term 5)
    A->>E: Heartbeat (Term 5)
    Note over B,E: Followers reset election timers

    Note over A,E: Phase 2: Leader Failure
    A-xA: CRASH
    Note over B,E: No heartbeats received...
    Note over C: Election timeout fires (180ms)
    C->>C: Become Candidate<br/>Increment term to 6

    Note over A,E: Phase 3: Request Votes
    C->>B: RequestVote(Term 6, Log: T5/I1247)
    C->>D: RequestVote(Term 6, Log: T5/I1247)
    C->>E: RequestVote(Term 6, Log: T5/I1247)
    B->>C: VoteGranted ✓
    D->>C: VoteGranted ✓
    E->>C: VoteGranted ✓
    Note over C: Received 4/5 votes (majority!)

    Note over A,E: Phase 4: New Leader Established
    C->>C: Become Leader (Term 6)
    C->>B: Heartbeat (Term 6) - I'm the leader
    C->>D: Heartbeat (Term 6) - I'm the leader
    C->>E: Heartbeat (Term 6) - I'm the leader

    Note over A,E: Phase 5: Stale Leader Recovery
    A->>A: Recovers, thinks it's still leader
    A->>C: Heartbeat (Term 5)
    C->>A: Reject: Term too old. Current term is 6
    A->>A: Update to Term 6<br/>Become Follower
    C->>A: Heartbeat (Term 6)
    Note over A: Accepts C as leader

Complete Raft leader election sequence showing steady state operation, leader failure detection, vote request phase, new leader establishment, and stale leader rejection. Note how term numbers prevent the old leader from causing damage after recovery.

Lease-Based Leadership with Automatic Failover

sequenceDiagram
    participant L as Leader Node C
    participant E as etcd/Coordination Service
    participant F as Follower Nodes
    participant S as Job Execution Service

    Note over L,S: Phase 1: Leader Acquires Lease
    L->>E: AcquireLease(key: /leader, TTL: 15s)
    E->>L: Lease granted (ID: 123, expires: T+15s)
    L->>F: Heartbeat: I'm leader with Lease 123

    Note over L,S: Phase 2: Normal Operation with Renewal
    loop Every 10 seconds
        L->>E: RenewLease(ID: 123)
        E->>L: Lease renewed (expires: T+15s)
        L->>S: Execute job (Lease: 123, Term: 6)
        S->>S: Verify lease is valid
        S->>L: Job executed ✓
    end

    Note over L,S: Phase 3: Leader Crashes
    L-xL: CRASH (no more renewals)
    Note over E: Lease 123 expires after 15s
    E->>E: Lease 123 expired at T+15s

    Note over L,S: Phase 4: New Leader Elected
    F->>E: AcquireLease(key: /leader, TTL: 15s)
    E->>F: Lease granted (ID: 124, expires: T+30s)
    F->>F: Become leader with Lease 124, Term 7
    F->>S: Execute job (Lease: 124, Term: 7)
    S->>F: Job executed ✓

    Note over L,S: Phase 5: Old Leader Recovers (Fencing)
    L->>L: Recovers, thinks it still has Lease 123
    L->>S: Execute job (Lease: 123, Term: 6)
    S->>E: Verify Lease 123
    E->>S: Lease 123 expired
    S->>L: ❌ Rejected: Lease expired
    L->>E: Check current lease
    E->>L: Current lease is 124 (Term 7)
    L->>L: Step down, become follower

Lease-based leadership showing normal operation with 10-second renewals, automatic failover when the leader crashes (lease expires after 15s), and fencing protection when the old leader recovers. The execution service verifies lease validity before accepting operations, preventing split-brain scenarios.

Variants

Bully Algorithm (Aggressive Election)

The Bully algorithm uses node IDs to determine leadership—the highest ID always wins. When a node detects leader failure, it sends “election” messages to all higher-ID nodes. If no higher-ID node responds within a timeout, it declares itself leader and broadcasts a “coordinator” message. If a higher-ID node responds, the initiator backs off.

When to use: Small clusters (3-5 nodes) with stable networks and clear node hierarchies. Works well when you want deterministic leadership (always the same node when available).

Pros: Simple to implement, fast elections (<1 second), deterministic leader selection.

Cons: Vulnerable to network partitions (can elect multiple leaders), generates O(n²) messages, higher-ID nodes can “bully” their way to leadership even if they just joined.

Example: Early versions of Apache Hadoop’s NameNode failover used Bully-style election with ZooKeeper for tie-breaking.

Raft Leader Election (Consensus-Based)

Raft uses term numbers and log completeness to elect leaders. Candidates must receive votes from a majority. Voters grant votes only to candidates whose logs are at least as complete as their own. Randomized timeouts prevent election collisions.

When to use: Systems requiring strong consistency guarantees, especially when leader must have the most up-to-date data (databases, distributed logs).

Pros: Proven safety guarantees, prevents split-brain through quorum requirements, leader always has complete log.

Cons: Slower elections (150ms-5s depending on network), requires majority availability (can’t tolerate n/2 failures), more complex implementation.

Example: etcd, Consul, and CockroachDB use Raft for leader election and log replication.

ZooKeeper Ephemeral Nodes (External Coordination)

Nodes create ephemeral sequential nodes in ZooKeeper. The node with the lowest sequence number becomes leader. When the leader’s session expires (heartbeat failure), its ephemeral node disappears, and the next-lowest node becomes leader. Nodes watch the node immediately before them to detect leadership changes.

When to use: When you want to externalize coordination logic, need sub-second failover, or already use ZooKeeper for other coordination tasks.

Pros: Fast failover (typically 200-500ms), simple client logic, battle-tested at scale (used by Kafka, HBase, Hadoop).

Cons: Adds external dependency, ZooKeeper itself must be highly available, watch storms during cascading failures.

Example: Apache Kafka uses ZooKeeper ephemeral nodes for broker leader election (migrating to KRaft in newer versions).

Bully Algorithm: Election Message Flow

graph TB
    subgraph Initial State
        N1["Node 1<br/>(ID: 1)"]
        N2["Node 2<br/>(ID: 2)"]
        N3["Node 3<br/>(ID: 3)<br/>LEADER"]
        N4["Node 4<br/>(ID: 4)"]
        N5["Node 5<br/>(ID: 5)"]
    end

    N3 -."❌ CRASHES".-> Crash[💥]

    subgraph Election Phase
        N2_E["Node 2<br/>Detects failure"]
        N4_E["Node 4"]
        N5_E["Node 5"]
    end

    N2_E --"1. ELECTION msg"--> N4_E
    N2_E --"1. ELECTION msg"--> N5_E
    N4_E --"2. OK (I'm alive)"--> N2_E
    N5_E --"2. OK (I'm alive)"--> N2_E
    N2_E -."3. Backs off<br/>(higher IDs responded)".-> N2_E

    subgraph Node 4 Continues
        N4_C["Node 4<br/>Sends ELECTION"]
        N5_C["Node 5"]
    end

    N4_C --"4. ELECTION msg"--> N5_C
    N5_C --"5. OK (I'm alive)"--> N4_C
    N4_C -."6. Backs off".-> N4_C

    subgraph Node 5 Wins
        N5_W["Node 5<br/>No higher IDs"]
        N1_W["Node 1"]
        N2_W["Node 2"]
        N4_W["Node 4"]
    end

    N5_W --"7. COORDINATOR<br/>(I'm the leader)"--> N1_W
    N5_W --"7. COORDINATOR"--> N2_W
    N5_W --"7. COORDINATOR"--> N4_W
    N5_W -."8. Becomes LEADER".-> N5_Final["Node 5<br/>NEW LEADER"]

Bully algorithm election flow where Node 3 (leader) crashes. Node 2 detects failure and sends ELECTION to higher-ID nodes. Nodes 4 and 5 respond, causing Node 2 to back off. Node 4 then sends ELECTION to Node 5, which responds. Finally, Node 5 (highest ID) sends COORDINATOR messages declaring itself leader. Note the O(n²) message complexity.

Algorithm Comparison

Election Speed

Bully algorithm completes elections in 1-2 network round trips (typically <1 second) because it uses simple timeout-based decisions. Raft requires 1-3 rounds of RequestVote RPCs plus randomized timeouts to prevent collisions, resulting in 150ms-5s elections depending on timeout configuration. ZooKeeper-based election is fastest at 200-500ms because it leverages ZooKeeper’s existing consensus and session management.

Consistency Guarantees

Bully provides no consistency guarantees during network partitions—multiple nodes can simultaneously believe they’re the leader. Raft guarantees at most one leader per term through quorum-based voting; a leader cannot be elected unless a majority agrees. ZooKeeper provides linearizable consistency for leadership decisions because ZooKeeper itself uses ZAB (similar to Paxos) for consensus.

Partition Tolerance

Bully fails catastrophically during partitions, potentially electing leaders in multiple partitions. Raft tolerates partitions gracefully: only the partition with a majority can elect a leader, and the minority partition remains leaderless. ZooKeeper-based election inherits ZooKeeper’s partition tolerance—only nodes that can reach the ZooKeeper quorum can participate in elections.

Failure Scenarios

In Bully, if the highest-ID node is flaky (crashes and recovers repeatedly), it causes election storms as it repeatedly bullies its way back to leadership. Raft handles flaky nodes through term numbers—a recovering node with an old term is rejected. ZooKeeper handles flaky nodes through session timeouts—a node must maintain a session to hold leadership, preventing rapid leadership changes.

Network Overhead

Bully generates O(n²) messages in the worst case (every node sends to every higher-ID node). Raft generates O(n) messages per election (candidate sends RequestVote to all nodes). ZooKeeper-based election generates minimal client-side traffic—just create/watch operations—but ZooKeeper itself generates significant internal traffic for consensus.

Implementation Complexity

Bully is simplest to implement (200-300 lines of code) but requires careful timeout tuning. Raft is moderately complex (1000-2000 lines) with well-documented edge cases. ZooKeeper-based election is simplest for clients (50-100 lines using ZooKeeper client library) but requires operating a ZooKeeper cluster.

Leader Election Algorithm Comparison Matrix

graph TB
    subgraph Bully Algorithm
        B_Speed["⚡ Election Speed<br/>1-2 RTT (~500ms-1s)"]
        B_Consistency["⚠️ Consistency<br/>No guarantees<br/>Split-brain possible"]
        B_Partition["❌ Partition Tolerance<br/>Multiple leaders in partitions"]
        B_Messages["📊 Message Complexity<br/>O(n²) worst case"]
        B_Use["✓ Use Case<br/>Small stable clusters<br/>3-5 nodes"]
    end

    subgraph Raft Consensus
        R_Speed["🐢 Election Speed<br/>1-3 voting rounds<br/>(150ms-5s)"]
        R_Consistency["✅ Consistency<br/>Strong guarantees<br/>Quorum-based"]
        R_Partition["✅ Partition Tolerance<br/>Only majority elects leader"]
        R_Messages["📊 Message Complexity<br/>O(n) per election"]
        R_Use["✓ Use Case<br/>Databases, distributed logs<br/>Strong consistency needs"]
    end

    subgraph ZooKeeper Ephemeral Nodes
        Z_Speed["⚡⚡ Election Speed<br/>200-500ms<br/>(fastest)"]
        Z_Consistency["✅ Consistency<br/>Linearizable via ZAB<br/>(ZooKeeper consensus)"]
        Z_Partition["✅ Partition Tolerance<br/>Inherits ZooKeeper's<br/>partition handling"]
        Z_Messages["📊 Message Complexity<br/>O(1) client-side<br/>ZK handles internally"]
        Z_Use["✓ Use Case<br/>External coordination<br/>Multiple systems"]
    end

    subgraph Trade-off Summary
        T1["Speed vs Safety<br/>Bully: Fast but unsafe<br/>Raft: Safe but slower<br/>ZK: Fast AND safe (external dep)"]
        T2["Embedded vs External<br/>Bully/Raft: No dependencies<br/>ZK: Requires ZK cluster"]
        T3["Complexity<br/>Bully: Simple (200 LOC)<br/>Raft: Moderate (1500 LOC)<br/>ZK: Simple client (50 LOC)"]
    end

    B_Speed --> T1
    R_Speed --> T1
    Z_Speed --> T1
    R_Consistency --> T2
    Z_Consistency --> T2
    B_Messages --> T3
    Z_Messages --> T3

**

Trade-offs

Election Speed vs Consistency Guarantees

Fast elections (Bully): Complete in <1 second but risk split-brain during partitions. Choose when network is stable and you can tolerate brief inconsistency.

Consistent elections (Raft): Take 150ms-5s but guarantee safety. Choose when correctness is more important than speed, especially for systems managing money or critical state.

Decision framework: If your system can detect and recover from dual-leader scenarios (e.g., through fencing tokens), optimize for speed. If dual leaders cause data corruption, optimize for consistency.

Embedded vs External Coordination

Embedded (Raft): No external dependencies, simpler operations, but every application must implement consensus correctly. Choose for databases or systems where consensus is core functionality.

External (ZooKeeper): Reusable coordination service, proven at scale, but adds operational complexity and network hops. Choose when building multiple distributed systems that need coordination.

Decision framework: If you’re building a database or distributed log, embed consensus. If you’re building application-level coordination (job schedulers, service discovery), use external coordination.

Quorum Size vs Availability

Strict majority (3 of 5 nodes): Strong consistency but can’t tolerate n/2 failures. A 5-node cluster can lose 2 nodes.

Relaxed quorum (2 of 5 nodes): Higher availability but weaker consistency. Used in Dynamo-style systems but not for leader election.

Decision framework: Leader election requires strict majorities for safety. Size your cluster based on failure tolerance needs: 3 nodes for 1 failure, 5 nodes for 2 failures, 7 nodes for 3 failures.

Lease Duration vs Failover Time

Short leases (5-10s): Fast failover but more heartbeat traffic and false positives during GC pauses. Kubernetes uses 15s leases.

Long leases (30-60s): Reduced traffic but slower failover. Acceptable when availability SLA allows 30-60s downtime.

Decision framework: Set lease duration to 2-3x your P99 heartbeat latency plus expected GC pause time. Monitor false leader expiration rates and adjust.

When to Use (and When Not To)

Use leader election when:

You need exactly-once execution of tasks across multiple nodes. Spotify’s job scheduler uses leader election to ensure cron jobs run once, not once-per-server. Without it, their daily playlist generation would create duplicate playlists.

You’re building a distributed database with a write master. MongoDB replica sets elect a primary that accepts writes while secondaries replicate. This prevents conflicting writes and simplifies consistency.

You need a single coordinator for distributed transactions. Google’s Percolator uses leader election to designate a transaction coordinator that manages two-phase commit across thousands of workers.

You’re implementing active-passive failover. Netflix’s Eureka service registry uses leader election to designate which instance handles service registrations, with others standing by as hot backups.

Don’t use leader election when:

All nodes can work independently without coordination. Stateless web servers don’t need leader election—they can all handle requests simultaneously. Adding leader election would create an unnecessary single point of failure.

You need multi-master writes. Leader election enforces single-master patterns. If you need multi-master (like Cassandra), use conflict-free replicated data types (CRDTs) or last-write-wins instead.

Your cluster is geographically distributed across continents. Leader election requires low-latency communication for heartbeats and voting. Cross-continent latencies (100-300ms) make elections slow and prone to false failures. Use regional leaders instead.

You can solve the problem with distributed locking. If you need short-term mutual exclusion (seconds), use distributed locks (see Distributed Locking). Leader election is for long-lived coordination (minutes to hours).

Real-World Examples

company: Spotify system: Luigi Job Scheduler how_they_use_it: Spotify runs thousands of data pipeline jobs daily using Luigi, their workflow orchestration system. Multiple Luigi scheduler instances run across different data centers for redundancy. They use ZooKeeper-based leader election to ensure exactly one scheduler assigns tasks at any time. The leader watches for job dependencies, schedules ready tasks, and monitors execution. When the leader fails, a new scheduler is elected within 500ms, and it resumes scheduling from the last checkpoint in ZooKeeper. interesting_detail: Spotify initially tried running Luigi without leader election, letting all schedulers assign tasks. This caused duplicate job executions that generated incorrect analytics and wasted compute resources. After implementing leader election, they eliminated duplicates and reduced their data pipeline costs by 30% by avoiding redundant computation.

company: MongoDB system: Replica Set Primary Election how_they_use_it: MongoDB replica sets use a Raft-inspired leader election protocol to elect a primary node that accepts writes. When the primary fails or becomes unreachable, remaining nodes hold an election. Candidates must have the most up-to-date oplog (operation log) to win votes. The election typically completes in 2-12 seconds depending on network conditions. The new primary immediately begins accepting writes, and secondaries sync from it. interesting_detail: MongoDB learned about split-brain the hard way in 2013. A network partition caused two primaries to accept writes simultaneously, creating divergent data. They enhanced their protocol with term numbers (called ‘election epochs’) and stricter quorum requirements. Now, even if a partition occurs, only the partition with a majority can elect a primary, guaranteeing at most one primary exists.

company: Kubernetes system: Control Plane Leader Election how_they_use_it: Kubernetes runs multiple instances of its controller manager and scheduler for high availability. These components use leader election via etcd to ensure only one instance actively manages the cluster. The leader holds a lease in etcd with a 15-second TTL, renewing it every 10 seconds. If the leader crashes, its lease expires, and another instance acquires the lease and becomes the new leader. This typically happens within 15-20 seconds. interesting_detail: Kubernetes uses lease-based leader election rather than heartbeat-based because leases provide better behavior during network partitions. If the leader is partitioned from etcd but still running, it stops making changes after its lease expires, preventing split-brain. The leader continuously checks its lease validity before executing any operation, implementing a form of fencing.

---

Interview Essentials

Mid-Level

Explain the split-brain problem and why quorum-based voting prevents it. Walk through a basic leader election scenario: what happens when a leader fails, how nodes detect the failure, and how a new leader is chosen. Understand the difference between leader election and distributed locking—election is for long-lived coordination, locking is for short-term mutual exclusion. Know that production systems use external coordination services (ZooKeeper, etcd) rather than implementing consensus from scratch.

Senior

Compare Raft and Bully algorithms: Raft uses term numbers and quorum voting for safety, Bully uses node IDs and timeouts for simplicity. Explain lease-based leadership: the leader holds a time-bound lease that must be renewed periodically, providing automatic failover when the leader crashes. Discuss fencing tokens: every leader gets a monotonically increasing token, and operations include this token so stale leaders can’t cause damage. Design a leader election system for a specific use case (job scheduler, database primary) and justify your algorithm choice based on consistency requirements, network characteristics, and failover time SLA.

Staff+

Analyze the CAP theorem implications: leader election requires consistency and partition tolerance, sacrificing availability during minority partitions. Discuss the trade-off between election speed and false positives: aggressive timeouts cause faster failover but more false leader expiration during GC pauses or network hiccups. Explain how to handle cascading failures: if leaders keep failing rapidly, implement exponential backoff or circuit breakers to prevent election storms. Design a multi-region leader election system: use regional leaders with a global coordinator, or implement hierarchical consensus. Discuss how to migrate from one leader election algorithm to another in a running system without downtime (dual-write period, gradual rollout, rollback strategy).

Common Interview Questions

How do you prevent split-brain in leader election? Use quorum-based voting (majority required) and fencing tokens (monotonically increasing term numbers). Only one partition can have a majority, and old leaders are rejected via term number checks.

What happens if the leader becomes slow but doesn’t crash? Implement lease-based leadership with health checks. If the leader can’t renew its lease (due to slowness), it expires and a new leader is elected. The old leader must check its lease before executing operations.

How do you choose between Raft and ZooKeeper for leader election? Use Raft when consensus is core to your system (databases, logs) and you want to avoid external dependencies. Use ZooKeeper when you need fast failover (<500ms), already use it for other coordination, or want to externalize complexity.

How do you test leader election in development? Use chaos engineering: randomly kill leaders, partition networks, introduce latency. Tools like Jepsen can verify safety properties. Test scenarios: leader crashes during election, network partition during steady state, all nodes restart simultaneously.

Red Flags to Avoid

Claiming you can implement Paxos or Raft from scratch in production. These algorithms have subtle edge cases that took years to get right. Use proven libraries (etcd, Consul) or services (ZooKeeper).

Not mentioning term numbers or fencing tokens. These are essential for preventing split-brain and stale leader operations.

Suggesting leader election for stateless services. If nodes can work independently, you don’t need a leader—you’re adding unnecessary complexity and a single point of failure.

Ignoring lease expiration and renewal. Without time-bound leases, a partitioned leader can continue operating indefinitely, causing split-brain.

Not discussing quorum requirements. Leader election requires majority agreement for safety. Saying ‘any node can become leader’ shows misunderstanding of distributed consensus.

---

Key Takeaways

Leader election ensures exactly one coordinator exists in a distributed system, preventing conflicts and enabling single-master patterns. It’s essential for job schedulers, database primaries, and distributed transaction coordinators.

Quorum-based voting (Raft, ZooKeeper) prevents split-brain by requiring majority agreement. Only the partition with a majority can elect a leader. Bully algorithm is simpler but vulnerable to partitions.

Lease-based leadership with fencing tokens prevents stale leaders from causing damage. Leaders hold time-bound leases that must be renewed, and operations include monotonically increasing term numbers that allow systems to reject outdated commands.

Trade-off between election speed and consistency: Bully completes in <1s but risks split-brain, Raft takes 150ms-5s but guarantees safety, ZooKeeper provides 200-500ms failover with external coordination.

Use leader election for long-lived coordination (minutes to hours), not short-term mutual exclusion (seconds). For the latter, use distributed locking (see Distributed Locking).

---