首頁 » 所有內容

Keepalived

由 datahunter 在 五, 15/08/2014 - 17:08 發表

最後更新: 2021-07-02

介紹

* written in C
* well-known and widely used lvs
* achieved by VRRPv2 protocol
* 3 distinct processes

lvs

lvs = linuxvirtualserver

HomePage: http://www.linuxvirtualserver.org/

kernel module providing Layer4 loadbalancing

Keepalived 有 3 個 subsystem

  • forked children process monitoring
  • VRRP subsystem (Keepalived_vrrp)
  • healthchecking subsystem. (Keepalived_healthcheckers)

設定一共分3個部份

  1. global (global_defs{})
  2. vrrpd (vrrp_sync_group{}, vrrp_instance{})
  3. lvs ()

 

VRRPv2 protocol

 

Virtual Router Redundancy Protocol (VRRP)

Protocol for failing-over an IP address from one machine to another

* The protocol achieves this by creation of virtual routers

* A virtual router must use 00-00-5E-00-01-XX as its Media Access Control (MAC) address.

  XX: Virtual Router IDentifier (VRID) <-- keepalived 不用設定它

* communicate within themselves using packets with multicast IP address 224.0.0.18 (vrrp_mcast_group4)

* 3 times the advertisement timer causes the backup routers to assume that the master router is dead.

VIP(Virtual IP address)

A VIP address migrates from one LVS router to the other during a failover

Kernel:

VRRP Stack <---> Netlink Reflector <---> NETLINK (Kernel)

Checkers  <---> IPVS wrapper <---> IPVS (Kernel)

 

安裝

 

Install:

yum install keepalived

check version:

keepalived --version

# Centos7

Keepalived v1.3.5 (03/19,2017), git commit v1.3.5-6-g6fa32f2

 

Firewall

 

iptables

iptables -I INPUT -i eth0 -d 224.0.0.0/8 -p vrrp -j ACCEPT
iptables -I OUTPUT -o eth0 -d 224.0.0.0/8 -p vrrp -j ACCEPT

vrrp 相當於 Protocol 112

firewalld

firewall-cmd --add-rich-rule='rule protocol value="vrrp" accept' --permanent

firewall-cmd --reload

 

VIP 設定

 

Diagram

  • Master: LVS-101
  • Slave: LVS-102
  • Virtual IP: 192.168.123.100
  • SMTP: 192.168.123.103

1) sysctl.conf

# vi /etc/sysctl.conf

net.ipv4.ip_forward = 1

sysctl -p

2) Master 的設定

2) 設定檔: /etc/keepalived/keepalived.conf

global_defs {
   notification_email {
     # 一行一個 e-mail
     [email protected]
   }
   notification_email_from [email protected]
   smtp_server 192.168.123.103
   smtp_connect_timeout 30

   # string identifying the machine
   router_id LVS-101
}

vrrp_instance VI_1 {
    # Initial state, MASTER|BACKUP
    # election 在 Initial 後進行 highest "priority" will become MASTER
    state MASTER
    
    # bound by vrrp
    interface eth0
    
    # unique to each VRRP instance
    # arbitary unique number 0~255, 它必須與 backup 一樣 !!
    virtual_router_id 100
    
    # highest priority wins. to be MASTER
    priority 100
    
    # VRRP Advert interval. Unit: secs
    advert_int 1
    
    # FAULT state if any of these go down (Optional)
    track_interface {
        eth0
        eth1
    }

    # 是否使用 login
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    
    virtual_ipaddress {
        # <IPADDR>/<MASK> dev <STRING> label <LABEL>
        # 一行一 IP
        192.168.123.100/24 dev eth0 label eth0:100
    }
}

login

當 password (auth_type PASS)不對時, 在 BACKUP 會見到

... Keepalived_vrrp[2300]: receive an invalid passwd!
... Keepalived_vrrp[2300]: bogus VRRP packet received on eth0 !!!

"auth_type": 不對應時, 兩邊都會起 eth0:100 sub-nic (成為 Master)並碰 IP

"auth_pass": Only the first eight (8) characters are used.

virtual_router_id 值不能相同

... Keepalived_vrrp[2503]: bogus VRRP packet received on eth0 !!!
... Keepalived_vrrp[2503]: VRRP_Instance(VI_1) ignoring received advertisment...

preempt

vrrp_instance VI_1 {
    ...
    # VRRP will normally preempt a lower priority machine
    # nopreempt => lower priority machine to maintain the master role
    # lower priority machine= the initial state must "BACKUP"
    # nopreempt
}

精簡版本

global_defs {
  router_id LVS-101
}
vrrp_instance VI_1 {
  state MASTER
  priority 100
  interface eth0
  virtual_router_id 30
  virtual_ipaddress {
    192.168.88.30/24 dev eth0 label eth0:30
  }
}

Slave 的設定

global_defs {
   notification_email {
     admindatahunter.org
   }
   notification_email_from [email protected]
   smtp_server 192.168.123.103
   smtp_connect_timeout 30
   router_id LVS_102
}

vrrp_instance VI_1 {
    state BACKUP
    interface eth0
    virtual_router_id 111
    priority 50
    advert_int 1
    #authentication {
    #    auth_type PASS
    #    auth_pass 1111
    #}
    nopreempt
    virtual_ipaddress {
        192.168.123.111/24 dev eth0 label eth0:111
    }
}

 

Checking

 

在 master 上會見到有 sub-interface 及 VIP (Backup 上是沒有的 !!)

eth0:111  Link encap:Ethernet  HWaddr 52:54:00:F1:62:1F
          inet addr:192.168.123.111  Bcast:0.0.0.0  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

tcpdump -v

16:07:57.897140 IP (tos 0xc0, ttl 255, id 30, offset 0, flags [none], proto VRRP (112), length 40)
    192.168.123.101 > vrrp.mcast.net: VRRPv2, Advertisement, vrid 111, prio 100,
    authtype simple, intvl 1s, length 20, addrs: 192.168.123.111 auth "1111^@^@^@^@"
16:07:58.897614 IP (tos 0xc0, ttl 255, id 31, offset 0, flags [none], proto VRRP (112), length 40)
    192.168.123.101 > vrrp.mcast.net: VRRPv2, Advertisement, vrid 111, prio 100,
    authtype simple, intvl 1s, length 20, addrs: 192.168.123.111 auth "1111^@^@^@^@"

Remark

  • 每 advert_int 一次
  • ttl 255
  • vrrp.mcast.net = 224.0.0.18

master start 時

dmesg

... IPVS: Registered protocols (TCP, UDP, SCTP, AH, ESP)
... IPVS: Connection hash table configured (size=4096, memory=64Kbytes)
... IPVS: Creating netns size=2048 id=0
... IPVS: Creating netns size=2048 id=1
... IPVS: ipvs loaded.

/var/log/messages

... systemd: Starting LVS and VRRP High Availability Monitor...
... Keepalived[9172]: Starting Keepalived v1.3.5 (03/19,2017), git commit v1.3.5-6-g6fa32f2
... Keepalived[9172]: Opening file '/etc/keepalived/keepalived.conf'.
... systemd: Started LVS and VRRP High Availability Monitor.
... Keepalived[9173]: Starting Healthcheck child process, pid=9174
... Keepalived[9173]: Starting VRRP child process, pid=9175
... Keepalived_healthcheckers[9174]: Opening file '/etc/keepalived/keepalived.conf'.
... Keepalived_vrrp[9175]: Registering Kernel netlink reflector
... Keepalived_vrrp[9175]: Registering Kernel netlink command channel
... Keepalived_vrrp[9175]: Registering gratuitous ARP shared channel
... Keepalived_vrrp[9175]: Opening file '/etc/keepalived/keepalived.conf'.
... Keepalived_vrrp[9175]: VRRP_Instance(VI_1) removing protocol VIPs.
... Keepalived_vrrp[9175]: Using LinkWatch kernel netlink reflector...
... Keepalived_vrrp[9175]: VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(10,11)]
... Keepalived_vrrp[9175]: VRRP_Instance(VI_1) Transition to MASTER STATE
... Keepalived_vrrp[9175]: VRRP_Instance(VI_1) Entering MASTER STATE
... Keepalived_vrrp[9175]: VRRP_Instance(VI_1) setting protocol VIPs.
... Keepalived_vrrp[9175]: Sending gratuitous ARP on eth0 for 192.168.88.30
... avahi-daemon[678]: Registering new address record for 192.168.88.30 on eth0.IPv4.
... Keepalived_vrrp[9175]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on eth0 for 192.168.88.30
... Keepalived_vrrp[9175]: Sending gratuitous ARP on eth0 for 192.168.88.30
... Keepalived_vrrp[9175]: Sending gratuitous ARP on eth0 for 192.168.88.30
... Keepalived_vrrp[9175]: Sending gratuitous ARP on eth0 for 192.168.88.30
... Keepalived_vrrp[9175]: Sending gratuitous ARP on eth0 for 192.168.88.30
... Keepalived_vrrp[9175]: Sending gratuitous ARP on eth0 for 192.168.88.30
... Keepalived_vrrp[9175]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on eth0 for 192.168.88.30
... Keepalived_vrrp[9175]: Sending gratuitous ARP on eth0 for 192.168.88.30
... Keepalived_vrrp[9175]: Sending gratuitous ARP on eth0 for 192.168.88.30
... Keepalived_vrrp[9175]: Sending gratuitous ARP on eth0 for 192.168.88.30
... Keepalived_vrrp[9175]: Sending gratuitous ARP on eth0 for 192.168.88.30

# delay for gratuitous ARP after transition to MASTER, default: 5

garp_master_delay

master stop

... Keepalived[9173]: Stopping
... systemd: Stopping LVS and VRRP High Availability Monitor...
... Keepalived_healthcheckers[9174]: Stopped
... Keepalived_vrrp[9175]: VRRP_Instance(VI_1) sent 0 priority
... Keepalived_vrrp[9175]: VRRP_Instance(VI_1) removing protocol VIPs.
... avahi-daemon[678]: Withdrawing address record for 192.168.88.30 on eth0.
... Keepalived_vrrp[9175]: Stopped
... Keepalived[9173]: Stopped Keepalived v1.3.5 (03/19,2017), git commit v1.3.5-6-g6fa32f2
... systemd: Stopped LVS and VRRP High Availability Monitor.

backup start

... systemd: Starting LVS and VRRP High Availability Monitor...
... Keepalived[1520]: Starting Keepalived v1.3.5 (03/19,2017), git commit v1.3.5-6-g6fa32f2
... Keepalived[1520]: Opening file '/etc/keepalived/keepalived.conf'.
... Keepalived[1521]: Starting Healthcheck child process, pid=1522
... Keepalived[1521]: Starting VRRP child process, pid=1523
... systemd: Started LVS and VRRP High Availability Monitor.
... Keepalived_vrrp[1523]: Registering Kernel netlink reflector
... Keepalived_vrrp[1523]: Registering Kernel netlink command channel
... Keepalived_vrrp[1523]: Registering gratuitous ARP shared channel
... Keepalived_vrrp[1523]: Opening file '/etc/keepalived/keepalived.conf'.
... Keepalived_vrrp[1523]: VRRP_Instance(VI_1) removing protocol VIPs.
... Keepalived_vrrp[1523]: Using LinkWatch kernel netlink reflector...
... Keepalived_vrrp[1523]: VRRP_Instance(VI_1) Entering BACKUP STATE
... Keepalived_vrrp[1523]: VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(10,11)]
... Keepalived_healthcheckers[1522]: Opening file '/etc/keepalived/keepalived.conf'.

FailOver 時

# Backup:

... Keepalived_vrrp[1523]: VRRP_Instance(VI_1) Transition to MASTER STATE
... Keepalived_vrrp[1523]: VRRP_Instance(VI_1) Entering MASTER STATE
... Keepalived_vrrp[1523]: VRRP_Instance(VI_1) setting protocol VIPs.
... Keepalived_vrrp[1523]: Sending gratuitous ARP on eth0 for 192.168.88.30
... Keepalived_vrrp[1523]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on eth0 for 192.168.88.30
... Keepalived_vrrp[1523]: Sending gratuitous ARP on eth0 for 192.168.88.30
... Keepalived_vrrp[1523]: Sending gratuitous ARP on eth0 for 192.168.88.30
... Keepalived_vrrp[1523]: Sending gratuitous ARP on eth0 for 192.168.88.30
... Keepalived_vrrp[1523]: Sending gratuitous ARP on eth0 for 192.168.88.30
... avahi-daemon[675]: Registering new address record for 192.168.88.30 on eth0.IPv4.
... Keepalived_vrrp[1523]: Sending gratuitous ARP on eth0 for 192.168.88.30
... Keepalived_vrrp[1523]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on eth0 for 192.168.88.30
... Keepalived_vrrp[1523]: Sending gratuitous ARP on eth0 for 192.168.88.30
... Keepalived_vrrp[1523]: Sending gratuitous ARP on eth0 for 192.168.88.30
... Keepalived_vrrp[1523]: Sending gratuitous ARP on eth0 for 192.168.88.30
... Keepalived_vrrp[1523]: Sending gratuitous ARP on eth0 for 192.168.88.30

Status

kill -USR2 `cat /var/run/keepalived.pid`

VRRP Instance: VI_1
  Advertisements:
    Received: 1
    Sent: 14443
  Became master: 1
  Released master: 0
  ...

 

其他 Setting

 

no_swap & priority

global_defs
{
  # no swap
  vrrp_no_swap
  checker_no_swap

  # priority (-20 ~ 19)
  vrrp_priority -1
  checker_priority -1
}

Static IP & Route

# keepalived 幫手設定 ip, 它不受 VRRP 影響

static_ipaddress
{
  192.168.123.101/24 dev eth0 scope global
}

static_routes
{
  0.0.0.0/0 via 192.168.123.1 dev eth0
}

Group of IPs that failover together

vrrp_sync_group VG_1 {
    group {
        # 一行一個 vrrp_instance
        inside_network
        outside_network
    }
    # 變成 MASTER 時行
    notify_master <QUOTED-STRING>

    # to BACKUP transition
    notify_backup <QUOTED-STRING>

    # FAULT transition
    notify_fault <QUOTED-STRING>

    # Send email notification during state transition
    smtp_alert
}
vrrp_instance inside_network {
    ...
}
vrrp_instance outside_network {
    ...
}

 

virtual_server Setting

 

設定其中一個 Loadbalane Service

virtual_server <IP> <Port> {
    # delay timer for service polling
    delay_loop 6
    
    # rr|wrr|lc|wlc|lblc|sh|dh
    lb_algo wlc
    
    # NAT|DR|TUN
    lb_kind NAT
    
    nat_mask 255.255.255.0
    persistence_timeout 50
    
    protocol TCP

    # suspend healthchecker’s activity
    # ha_suspend
    
    # HTTP_GET or SSL_GET 時用, virtualhost <STRING>
    # virtualhost www.firewall.loc

    # Script to launch when quorum is gained.
    quorum_up <STRING>

    # Script to launch when quorum is lost.
    quorum_down <STRING>
    
    # Real_server Section
}

lb_kind

Basically supports three methods of package forwarding:

  • NAT
  • Tunneling
  • Direct Routing

NAT:

Web Server 1
Web Server 2  ----> LVS Router
............
Web Server N

Heath Check

HTTP_GET

Working at layer5. Performs a GET HTTP to a specified URL. The get result is
then summed using the MD5 algorithm.

TCP_CHECK

Working at layer4.

MISC_CHECK

/path_to_script/script.sh
result must be 0 or 1

HTTP_GET

genhash -s 192.168.123.101 -p 80 -u /index.html

--server <host>, -s
--url <url>, -u

MD5SUM = 9a4e2ab7c7392d17d511bc4606827fbf

設定

delay loop: specify in seconds the interval between checks.

# Real_server Section

Virtual server definitions

virtual_server (@IP PORT)|(fwmark num) {
    delay_loop num
    lb_algo rr|wrr|lc|wlc|sh|dh|lblc
    lb_kind NAT|DR|TUN
    persistence_timeout num
    persistence_granularity @IP
    virtualhost string
    protocol TCP|UDP
    
    sorry_server @IP PORT
    
    real_server @IP PORT {
        weight num
        TCP_CHECK {
            connect_port num
            connect_timeout num
        }
    }
    real_server @IP PORT {
        weight num
        MISC_CHECK {
            misc_path /path_to_script/script.sh
            (or misc_path “/path_to_script/script.sh <arg_list>”)
        }
    }
    real_server @IP PORT {
        weight num
        HTTP_GET|SSL_GET {
            url { # You can add multiple url block
            path alphanum
            digest alphanum
        }
        connect_port num
        connect_timeout num
        nb_get_retry num
        delay_before_retry num
        }
    }
}

i.e.

real_server <IPADDR> <PORT> {

    weight 1

    #支援這些 check HTTP_GET|SSL_GET|TCP_CHECK|SMTP_CHECK|MISC_CHECK
    HTTP_GET {
        # A url to test
        url {
          path /
          digest ff20ad2481f97b1754ef3e12ecd3a9cc
          status_code 200
        }
        url {
          path /mrtg/
          digest 9b3a0c85a887a256d6939da88aabd8cd
        }

        connect_timeout 3
        # number of get retry
        nb_get_retry 3
        delay_before_retry 3

    }
}

 

Other

 

ipvsadm

 

»
  • 瀏覽次數： 5565

Creative Commons license icon Creative Commons license icon