Clustering and failover, quorum and split brain
In short: A high availability cluster watches its members with heartbeats and moves services to a healthy node when one fails. Quorum ensures only the majority side acts after a network split, and fencing forcibly isolates a failed node, together preventing split brain and data corruption.
What a failover cluster is
A failover cluster is a group of servers, called nodes, that together run services such as a database, file share or virtual machines. Cluster software, for example Pacemaker with Corosync on Linux or Windows Server Failover Clustering, monitors each node and each service. If a node or service fails, the cluster starts it on another node and moves the virtual IP so clients reconnect to the same address.
Nodes exchange heartbeats over the network. If a node misses several, the others treat it as failed. Use at least two separate heartbeat paths so that one faulty switch does not trigger a false failover.
Split brain
Split brain happens when nodes lose contact with each other but are still running. Each side thinks the other has died and tries to take over. If both mount the same storage or accept writes to the same database, the data is corrupted. This is far worse than a short outage, so clusters are designed to stop rather than risk it.
Quorum and witnesses
Quorum means a majority of votes. After a split, only the side with more than half the votes may run services; the minority stops. With three nodes, any two form quorum. With two nodes there is no majority after a split, so you add a witness (also called a tie-breaker or quorum device): a small vote held on a third site, a file share or cloud storage.
| Votes | Quorum | Can lose | Note |
|---|---|---|---|
| 2 nodes | 2 | 0 | Add a witness |
| 2 nodes + witness | 2 of 3 | 1 | Common small design |
| 3 nodes | 2 | 1 | No witness needed |
| 5 nodes | 3 | 2 | Higher tolerance |
Place the witness in a third location. If it sits in the same site as one half, losing that site removes two votes at once.
Fencing
Quorum stops the minority acting, but a node that is hung, not dead, might wake up and write again. Fencing makes sure the failed node cannot touch shared resources. Methods include powering it off through a management interface (often called STONITH, "shoot the other node in the head"), cutting its storage access, or blocking it at the switch. Only after fencing succeeds should the survivor start the service.
- Give every node redundant heartbeat networks.
- Use an odd number of votes, adding a witness where needed.
- Configure and test fencing; many cluster stacks refuse to run safely without it.
- Set realistic timeouts so brief network blips do not cause needless failovers.
- Test planned and unplanned failover regularly and record how long each takes.
On a Pacemaker cluster you can check quorum and resource status with:
pcs status
pcs quorum status
pcs stonith statusTip: the same majority logic appears in consensus algorithms such as Raft. If you understand quorum here, you understand why etcd clusters have three or five members.
Practice
Why does a two-node cluster need a witness?
What is the purpose of fencing?
Describe split brain in one sentence.
Key takeaways
- Clusters use heartbeats to detect failure and move services and VIPs.
- Split brain risks data corruption and must be prevented.
- Quorum lets only the majority act; add a witness for even node counts.
- Fencing isolates a failed node before failover.
- Test failover regularly and use redundant heartbeat paths.
Original content written by Mohamed Fayas for FM TecHub. © 2026 FM TecHub. You're welcome to share a link to this page; please don't copy the text elsewhere without permission.