tc - htb

由 datahunter 在四, 18/07/2024 - 17:39 發表

最後更新: 2024-07-18

介紹

Hierarchical Token Bucket

It based on the Token Bucket Filter algorithm

Allows control of the outbound bandwidth on a given link.

Limit Outgoing (HTB)

HTB ensures that the amount of service provided to each class is
at least the minimum of the amount it requests and the amount assigned to it.

When a class requests less than the amount assigned,
the remaining (excess) bandwidth is distributed to other classes which request service.

* 每層 class 的 rate 的總和一定要小過上層才有效

* With HTB, you should attach all filters to the root !!

* Each node within the tree can have its own filters

* HTB use of the outbound bandwidth on a given link

* each class has a single parent

* each class contains a "leaf" qdisc which by default has pfifo

* one root class cannot borrow from another root class

Doc

Homepage: http://luxik.cdi.cz/~devik/qos/htb/
Doc: http://luxik.cdi.cz/~devik/qos/htb/manual/userg.htm
HTB theory: http://luxik.cdi.cz/~devik/qos/htb/manual/theory.htm

Example 1: Sharing Hierarchy with u32

Diagram

eth0---qdisc (1:)  # To attach htb to NIC, Default: 1:12                       (Step 2)
         |
       _1:1_       # root class. 80mbit,                               Level 3 (Step 3)
      /     \
    1:11     1:12  # child class: 1:11(48mb), leaf class: 1:12(32mb)   Level 2 (Step 4)
   /    \
1:21   1:22        # child class: 1:21(32mb), child class: 1:22(16mb)  Level 1 (Step 5)
  |      |
 21:    22:        # leaf class: 21:, 22:                              Level 0 (Step 6) # Optional

Step

Step 1 ~ 6 建立 Tree
Step 7 加 Rules

Source port map to class

1:12 - *
1:21 - 8021/tcp
1:22 - 8022/tcp

[Step 1] Delete existing rules

tc qdisc del dev eth0 root
iptables -t mangle -F           # 如非必要時勿行 !!

P.S.

tc qdisc del dev eth0 root # 不只 qdisc, 會連 tc filter 一起 delete 埋

當 interface 沒有 qdisc 時, 會見到

Error: Cannot delete qdisc with handle of zero.         # OS: Rocky8

[Step 2] Attaches queue discipline HTB to eth0, and set default class(1:12)

tc qdisc add dev eth0 root handle 1: htb default 12

說明

"handle 1:" => "handle 1:0" # x:y

just a name or identifier with which to refer

Format: major:minor

所以可以係 2:, 3:, 4: ...

The handle for a qdisc must have zero for its y value.

"default minor-id" # default minor = 0

Any traffic that is not otherwise classified will be assigned to class 1:12

Unclassified traffic gets sent to the class with this minor-id.

當沒有設定 "default" 時, 相當於 minor=0 (1:0)
=> unclassified packets are sent "0"
=> completely bypassing any of the classes attached to the root qdisc.
=> dequeued at hardware speed

[3] "root" class, "1:1" under the qdisc "1:"

tc class add dev eth0 parent 1: classid 1:1 htb rate 80mbit

說明

classid major:minor

classes can be named.

* The major number must be equal to the major number of the qdisc to which it belongs.

[Step 4] Create two classes directly under the htb qdisc

tc class add dev eth0 parent 1:1 classid 1:11 htb rate 48mbit ceil 80mbit
tc class add dev eth0 parent 1:1 classid 1:12 htb rate 32mbit

說明

ceil:

Specifies the maximum bandwidth(burst) that a class can use.(borrow)

The default ceil is the same as the rate.

[5] Create two child classes under the "1:11"

tc class add dev eth0 parent 1:11 classid 11:21 htb rate 32mbit
tc class add dev eth0 parent 1:11 classid 11:22 htb rate 16mbit

說明

parent major:minor:

Place of this class within the hierarchy.

[6] Attach queuing disciplines to the leaf classes (沒有設定時, 預設是 pfifo)[非必要 Step]

tc qdisc add dev eth0 parent 1:21 handle 21: sfq perturb 10
tc qdisc add dev eth0 parent 1:22 handle 22: pfifo

[7] Which packets belong in which class

方法1: 使用 tc filter 直接設定

tc_filter

Backup Server IP (n.n.n.n)

# By IP - traffic to backup server (float 100k)
tc filter add dev eth0 protocol ip parent 1:0 prio 50 u32 \
   match ip dst n.n.n.n flowid 1:101

Source Port (8080)

# By Port - tcp port 8080 (fix 300k)
tc filter add dev eth0 protocol ip parent 1:0 prio 49 u32 \
   match ip sport 8080 0xffff flowid 1:102

# By IP & Port

tc filter add dev eth0 protocol ip parent 1:0 prio 48 u32 \
   match ip src 192.168.123.10 match ip sport 1080 0xffff flowid 1:12

prio

* classes with higher priority are offered excess bandwidth first.

What class should you priorize? Generaly those classes where you really need low delays.

1 => highest priority

方法1: filter 駁 iptables

# 有 mark 的 package to 某 qdisc

tc filter add dev eth0 parent 1: protocol ip handle 8021 fw flowid 1:21
tc filter add dev eth0 parent 1: protocol ip handle 8022 fw flowid 1:22

* flowid = classid

# 為 source port 80 的 packet set mark

iptables -t mangle -A OUTPUT -p tcp --sport 8012 -j MARK --set-mark 8021
iptables -t mangle -A OUTPUT -p tcp --sport 8012 -j MARK --set-mark 8022

* mark 只可以用數字. iptables -t mangle 時是 0x????

Checking

(1)

tc qdisc show dev eth0

qdisc htb 1: root refcnt 2 r2q 10 default 0x12 direct_packets_stat 0 direct_qlen 1000
qdisc sfq 21: parent 1:21 limit 127p quantum 1514b depth 127 divisor 1024 perturb 10sec
qdisc pfifo 22: parent 1:22 limit 1000p

(2)

tc class show dev eth0

class htb 1:22 parent 1:11 leaf 22: prio 0 rate 16Mbit ceil 16Mbit burst 1600b cburst 1600b
class htb 1:11 parent 1:1 rate 48Mbit ceil 80Mbit burst 1590b cburst 1600b
class htb 1:1 root rate 80Mbit ceil 80Mbit burst 1600b cburst 1600b
class htb 1:12 parent 1:1 prio 0 rate 32Mbit ceil 32Mbit burst 1600b cburst 1600b
class htb 1:21 parent 1:11 leaf 21: prio 0 rate 32Mbit ceil 32Mbit burst 1600b cburst 1600b

(3)

tc filter show dev eth0

filter parent 1: protocol ip pref 49151 fw chain 0
filter parent 1: protocol ip pref 49151 fw chain 0 handle 0x1f56 classid 1:22
filter parent 1: protocol ip pref 49152 fw chain 0
filter parent 1: protocol ip pref 49152 fw chain 0 handle 0x1f55 classid 1:21

(4)

iptables -v -nL -t mangle | grep MARK

    0     0 MARK       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp spt:8012 MARK set 0x1f55
    0     0 MARK       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp spt:8012 MARK set 0x1f56

Note

* "firewall-cmd --reload" 會清了之前用 iptables 的設定

Testing

# 在 123.10 上面測試

IP=192.168.88.128

wget $IP:8012/test.bin -O /dev/null

wget $IP:8021/test.bin -O /dev/null

wget $IP:8022/test.bin -O /dev/null

Statistics

tc -s -d qdisc show dev eth0

tc -s -d class show dev eth0

-d[etails]

-s[tatistics]

overlimits

how many times the discipline delayed a packet.

level

quantum

don't need to specify quantums manualy as HTB chooses precomputed values.

pps

tells you actual (10 sec averaged) rate going thru class.

giants (Default: 1600 bytes)

number of packets larger than mtu set in tc command.
HTB will work with these but rates will not be accurate at all.

lended

packets donated by this class

borrowed (borrows are transitive)

borrowed from parent.

Other

quantums:

In fact when more classes want to borrow bandwidth they are each given some number of bytes before serving other competing class.

This number is called quantum.

burst

burst bytes: Amount of bytes that can be burst at ceil speed

cburst bytes: Amount of bytes that can be burst at "infinite" speed

why I want bursts. Well it is cheap and simple way how to improve response times on congested link.

* The burst and cburst of a class should always be at least as high as that of any of it children.

i.e.

... burst 2k

Notes

nginx config

http {
    server {
        listen       8012 default_server;
        listen       8021 default_server;
        listen       8022 default_server;
        ...
    }
    ...
}

htb - Over root traffic

Leave Over Root Total

在以下 settings, class 80, 81, 82 它們都有 60kbps, 70kbps, 80kbps,
會無視了 root class 的 100kbps limit !!

parent 1: classid 1:1 htb rate 100kbps

parent 1:1 classid 1:80 htb rate 60kbps
parent 1:1 classid 1:81 htb rate 70kbps
parent 1:1 classid 1:82 htb rate 80kbps

ceil 的應用

在以下 settings, class 80, 81, 82 都有 9x kbps

parent 1: classid 1:1 htb rate 300kbps
parent 1:1 classid 1:80 htb rate 60kbps ceil 100kbps
parent 1:1 classid 1:81 htb rate 70kbps ceil 100kbps
parent 1:1 classid 1:82 htb rate 80kbps ceil 100kbps

瀏覽次數： 91

夢想家

tc - htb

目錄

介紹

Limit Outgoing (HTB)

htb - Over root traffic