How Broadcast
Signaling Works
Placing
this issue in the right perspective requires first understanding something
about how broadcast signaling works. When a member is invoked, it is supposed
to issue a broadcast message to the network. The member does so using its
cluster name as an identifier. If there happens to be a cluster present on the
network with the same cluster name, that cluster is expected to reply to that
broadcast message with its node name. In that case, the joining member should
send a join request to the cluster. The default port being used by the cluster
for issuing broadcast messages is 5405.
You
will see these messages when opening the NIC in promiscuous mode with either
wireshark or tcpdump:
22:25:36.455369
IP server-01.5149 > server-02.netsupport: UDP, length
106
22:25:36.455531
IP server-02.5149 > server-01.netsupport: UDP, length
106
22:25:36.665363
IP server-01.5149 > 255.255.255.255.netsupport: UDP, length 118
22:25:36.852367
IP server-01.5149 > server-02.netsupport: UDP, length
106
22:25:36.852526
IP server-02.5149 > server-01.netsupport: UDP, length
106
How Name resolution works and cluster.conf
Cman (the Cluster Manager) tries hard
to match the local host name(s) to those mentioned in cluster.conf. Here's how
it does it:
1. It looks up $HOSTNAME in
cluster.conf
2. If this fails it strips the
domain name from $HOSTNAME and looks up that in cluster.conf
3. If this fails it looks in
cluster.conf for a fully-qualified name whose short version matches the
short version of $HOSTNAME
4. If all this fails then it will
search the interfaces list for an (ipv4 only) address that matches a
name in cluster.conf
cman
will then bind to the address that it has matched.
Note: we will have to
make sure the settings in /etc/nsswitch.conf are set to:
hosts: files nis dns
nsswitch.conf
is a facility in Linux operating systems that provides a variety of sources for
common configuration databases and name resolution mechanisms.
Split Brain Issues
One of the most dangerous situations that can happen in clusters is that both nodes become active at the same time. This is especially true for clusters that share storage resources. In this case both cluster nodes could be writing to the data on shared storage which will quickly cause data corruption.When both nodes becoming active it is called “split brain” and can happen when a cluster stops receiving heartbeats from its partner node. Since the two nodes are no longer communicating they do not know if the problem is with the other node or if the problem is with itself.
For example say that the passive node stops receiving heartbeats from the active node due to a network failure of the heartbeat network. In this case if the passive node starts the cluster services then you would have a split-brain situation.
Many clusters use a Quorum Disk to prevent this from happening. The Quorum Disk is a small shared disk that both nodes can access at the same time. Whichever node is currently the active node writes to the disk periodically (usually every couple of seconds) and the passive node checks the disk to make sure the active node is keeping it up to date.
When a node stops receiving heartbeats from its partner node it looks at the Quorum Disk to see if it has been updated. If the other node is still updating the Quorum Disk then the passive node knows that the active node is still alive and does not start the cluster services.
Redhat clusters support Quorum Disks, but Redhat support had recommended not to use one since they are difficult to configure and can become problematic. Instead they recommend to relying on Fencing to prevent split brain.
