What is SBD Fencing?

SBD (STONITH Block Device) fencing is a robust and essential node fencing mechanism specifically designed for high-availability (HA) clusters managed by Pacemaker. It plays a critical role in preventing data corruption and ensuring cluster integrity by reliably isolating faulty or unresponsive nodes through the exchange of messages via shared block storage.

The Role of Fencing in High-Availability Clusters

In a high-availability cluster, multiple servers (nodes) work together to provide continuous service. A major challenge for these clusters is preventing a "split-brain" scenario, where communication issues lead to nodes mistakenly believing they are the sole active member, attempting to take control of the same resources simultaneously. This can lead to severe data corruption.

To combat this, fencing (also known as STONITH – Shoot The Other Node In The Head) is employed. Fencing ensures that a non-responsive or faulty node is immediately and definitively powered off or reset, preventing it from accessing shared resources and causing damage.

Deep Dive into SBD (STONITH Block Device) Fencing

SBD fencing provides a reliable and hardware-independent method for isolating nodes. Unlike other fencing methods that might rely on specific network power switches or IPMI (Intelligent Platform Management Interface) cards, SBD leverages the existing shared block storage infrastructure.

How SBD Fencing Works

SBD fencing operates on a principle of shared block storage messaging. Here's a breakdown:

Shared Block Device: Each node in the Pacemaker cluster has access to one or more shared block devices (e.g., SAN LUNs, iSCSI targets, FCoE volumes). These devices are not used for application data but solely for SBD communication.
Heartbeat Messages: The sbd daemon running on each cluster node periodically writes small "heartbeat" messages to the configured shared block device(s). These messages act as a clear indication that the node is alive and functioning.
Watchdog Timer Integration: SBD works in conjunction with a watchdog timer, which can be a hardware-based or software-emulated component. The sbd daemon regularly "pets" (resets) this watchdog timer.
Failure Detection and Fencing:
- If a node becomes unresponsive or crashes, its sbd daemon will stop updating its heartbeat messages on the shared block device.
- Other active nodes detect the absence of these heartbeats within a configured timeout period.
- Crucially, if the sbd daemon on the failing node stops responding or fails to reset its local watchdog timer, the watchdog will automatically trigger a hard reset or power off of that node, effectively fencing it.
- This intelligent mechanism ensures that a problematic node is forcibly removed from the cluster's active resource pool.

A significant advantage of SBD is that this approach isolates the fencing mechanism from changes in firmware version or dependencies on specific firmware controllers. This enhances its reliability and compatibility across diverse hardware environments, reducing the complexity often associated with hardware-specific fencing solutions.

Core Components of SBD Fencing

Implementing SBD fencing typically involves three main components:

sbd daemon: The software process running on each node responsible for writing heartbeats and managing the watchdog timer.
Watchdog timer: A hardware or software component that triggers a node reset if not regularly "petted" by the sbd daemon. It's the ultimate safety net.
Shared block device(s): The dedicated storage volume(s) used for SBD heartbeat communication. For redundancy, it's highly recommended to use multiple independent devices.

Advantages of Using SBD Fencing

SBD fencing offers several compelling benefits for high-availability clusters:

Reliability: By leveraging direct block-level communication and a hardware watchdog, SBD offers a highly dependable and low-latency fencing mechanism.
Hardware Independence: It avoids reliance on specific network cards, power distribution units (PDUs), or firmware versions, making it compatible with a wide range of server hardware.
Cost-Effectiveness: If shared block storage (SAN, iSCSI) is already part of the cluster infrastructure, SBD can be a very economical fencing solution.
Robustness against Network Issues: Since it communicates directly via shared storage, SBD is less susceptible to network latency or minor network outages that might impact IP-based fencing methods.
Simplicity: While requiring careful setup, once configured, SBD tends to be simpler to manage than multiple hardware-specific fencing agents.

Practical Considerations and Setup

Implementing SBD fencing requires careful planning and configuration.

Requirements

To successfully deploy SBD fencing, the following are essential:

Shared Block Device(s): At least one, but preferably three, shared block devices accessible by all cluster nodes. These should be small (e.g., 20 MB each) and dedicated solely to SBD. Examples include:
- LUNs from a Storage Area Network (SAN)
- iSCSI targets
- Fibre Channel over Ethernet (FCoE) volumes
Watchdog Timer: A functional watchdog device must be present and enabled on each node. Modern Linux kernels typically support hardware watchdogs (e.g., iTCO_wdt for Intel chipsets) or a software watchdog.
sbd Package: The sbd software package must be installed on all cluster nodes.
Pacemaker/Corosync: An operational Pacemaker and Corosync cluster environment.

Configuration Highlights

Configuring SBD typically involves editing the /etc/sysconfig/sbd or /etc/default/sbd file and ensuring the watchdog module is loaded.

Configuration Parameter	Description
`SBD_DEVICE`	Specifies the shared block device(s) to use. Can be a space-separated list for redundancy.
`SBD_WATCHDOG_TIMEOUT`	Defines the timeout in seconds for the watchdog. If `sbd` fails to pet the watchdog within this time, the node is reset.
`SBD_TIMEOUT`	Defines the timeout for SBD messages between nodes. If a node doesn't receive messages from a peer within this time, it's considered faulty.
`SBD_DELAY_START`	A delay before SBD starts, useful for systems where storage may take time to initialize.

Example sbd.conf snippet:

SBD_DEVICE="/dev/disk/by-id/scsi-xxxx /dev/disk/by-id/scsi-yyyy" # Use persistent device names
SBD_WATCHDOG_TIMEOUT=5
SBD_TIMEOUT=10

It is crucial to test SBD fencing thoroughly in a non-production environment before deploying it to a live cluster.

SBD Fencing vs. Other Methods

While other fencing methods exist, such as:

IPMI/DRAC/iLO: Out-of-band management interfaces that allow power cycling nodes.
Network Power Switches: Smart PDUs that can cut power to specific outlets.
Virtual Machine Agents: Hypervisor-level agents for fencing guest VMs.

SBD stands out due to its reliance on shared storage and a direct watchdog trigger, making it highly effective even when the network is congested or the operating system is severely compromised. Its independence from specific vendor hardware firmware also provides a layer of resilience and ease of management.