drbd - 入門

由 datahunter 在二, 02/10/2012 - 18:46 發表

最後更新: 2021-02-10

介紹

DRDB = Distributed Replicated Block Device = network based raid-1

特點

* 支援 Checksum-based synchronization (所以支援 online verification)

It is a common use case to have on-line verification managed by the local cron daemon

verification source
        \|/
calculating a cryptographic digest(MD5, SHA-1, or CRC-32C) of every block
(Linux kernel crypto API provides these)
DRBD transmits just the digests, not the full blocks,
online verification uses network bandwidth very efficiently.
        \|/
verification target

* 支援 Fixed-rate / Variable-rate synchronization (Default)

(Variable-rate: DRBD detects the available bandwidth on the synchronization network)

* Synchronously or Asynchronously (Local / Remote)

* Transparently (其他 Program 不知它的存在)

* Driver code in the kernel (Linux kernel's block layer)

* User space tools(drbdadm -call-> drbdsetup -> kernel driver)

Version

Linux DRBD
3.0 - 3.4 8.3.11

Hardware 要求

Disk system 要支援 disk flush, 如

SATA Devices
SCSI
MD
LVM2

如果 disk 有 battery-backed cache 那可以考慮停用 device flushes 去提升 Performance

Feature

V 9

RDMA is now integrated into DRBD9 ( reducing CPU load by 50% )
More than 2-way redundancy
Automatic Promotion of Resources

V 8.4

Defaults to masking I/O errors
Defaults Variable-rate synchronization
multiple volumes (each corresponding to one block device)
-> A single resource may correspond to multiple volumes
-> may share a single replication connection

lrwxrwxrwx 1 root root 11 2011-07-04 09:22 /dev/drbd/by-res/nfs-root/0 ->  ../../drbd2
lrwxrwxrwx 1 root root 11 2011-07-04 09:22 /dev/drbd/by-res/nfs-root/1 ->  ../../drbd3

Boolean configuration options

no-md-flushes -> md-flushes no

syncer section no longer exists

syncer{} 的設定分到 disk{} 及 net{}

在 resource{} 新增了 options{}
replication protocol can be changed on the fly

# temporarily switch a connection to asynchronous replication

drbdadm net-options --protocol=A <resource>

"--force" option to kick off an initial resource synchronization

drbdadm primary --force <resource>

reduce bitmap IO during resync
Align the internal object model with drbd-9.0
New on disk format for the AL: double capacity; 4k aligned IO
Multiple AL changes in a single transaction

V 8.3.10

bandwidth management for mirroring over long distance

V 8.3.2

Resource level fencing script, using Pacemaker's constraints
OCF Heartbeat/Pacemeker resource agent
Support for floating peers
compression of the bitmap exchange

V 8.3.0

off-site node for disaster recovery
Existing file systems can be integrated into new DRBD setups
(add redundancy to existing deployments)
Heartbeat integration to outdate peers with broken replication links
(avoids switchovers to stale data)(Short resynchronization)
Optional data digests to verify the data transfer over the network
Integration scripts for LVM to automatically take a snapshot before a node becomes the target of a resynchronization
Online data verification
Configurable handler scripts for various DRBD events
Dual primary support for use with GFS/OCFS2
Automatic detection of the most up-to-date data after complete failure
Automatic recovery after node, network, or disk failures

Centos7 Installation

Install

# 要安裝 elrepo version, 因為 epel 沒有 Kernel Module

yum install elrepo-release -y

yum install kmod-drbd90 drbd90-utils drbd90-utils-sysvinit # V 9.0

yum install kmod-drbd84 drbd84-utils drbd84-utils-sysvinit # V 8.4

Package

drbd-utils # V9

drbdadm ( drbdsetup 與 drbdmeta 的 font-end )
drbdmon ( Monitor DRBD resources realtime )
drbdmeta ( create, dump, restore, and modify DRBD's meta data )
drbdsetup ( low-level tool 來, 用來控制 kernel module )

設定檔:

/etc/drbd.conf
/etc/drbd.d/global_common.conf
/etc/drbd.d/*.res

Cluster Package

drbd-pacemaker # Pacemaker resource agent for DRBD
drbd-rgmanager # Red Hat Cluster Suite agent for DRBD

Enable Service

systemctl start drbd

lsmod | grep drbd # drbd 後就會載入

cat /proc/drbd # 載入 drbd 後就會有

systemctl enable drbd

DRBD device

Path: /dev/drbd<minor>

ls -l /dev/drbd*

major:minor
   147:N

Role

Single-primary mode (canonical approach for high availability)

在整個 Cluster 內只有 1 個 primary node
(only one cluster node manipulates the data at any moment)

primary role - unrestrictedly for read and write

secondary role - can not be used by applications, neither for read nor write access.

* 可以用任何 file system (ext3, ext4, XFS etc.)

Dual-primary mode (load-balancing clusters)

必須要是以下其中一種 File System 才可以用 Dual primary

GFS
OCFS2 (distributed lock manager)

* Dual primary disabled by default

Replication modes

Protocol A Asynchronous replication protocol
Protocol B Memory synchronous (semi-synchronous)
replication packet has reached the peer node
Protocol C Synchronous replication protocol (fully synchronous)

Protocol A 的優化

原本: write --> local buffer (sndbuf-size)

優化: write --> DRBD Proxy --> secondary node (加入了 proxy 去優化)

Inconsistent and Outdated data

Inconsistent data

data that cannot be expected to be accessible and useful in any manner.

Outdated data

data on a secondary node that is consistent, but no longer in sync with the primary node.
(disconnected secondary node is expected to be clean)

Synchronization & Replication

replication occurs on any write event to a resource in the primary role

Synchronization is necessary if the replication link has been interrupted for any reason
(interruption of the replication link)

* issuing on one node while <resource> is still in the primary role on the peer will result in an error.

Split Brain 問題

情況:

Both nodes switched to the primary role

修復方式:

Discarding modifications made on the “younger” primary.
Discarding modifications made on the “older” primary.
Discarding modifications on the primary with fewer changes.
Graceful recovery from split brain if one host has had no intermediate changes.
(unlikely scenario. Even if both hosts only mounted the file system on the DRBD block device (even read-only))

Doc

man drbd.conf

man drbdadm

https://www.linbit.com/drbd-user-guide/drbd-guide-9_0-en/

https://linbit.com/drbd-user-guide/users-guide-drbd-8-4/

瀏覽次數： 2700

夢想家