Centos6 - libcgroup

最後更新: 2016-06-21


 * system processes are called tasks in cgroup terminology.




# Install

yum -y install libcgroup

# Auto start

service cgconfig restart

chkconfig cgconfig on

# checking cgroup subsystems working

ls /cgroup

blkio  cpu  cpuacct  cpuset  devices  freezer  memory  net_cls

比較特別的 cgroup

devices — this subsystem allows or denies access to devices by tasks in a cgroup

net_cls — this subsystem tags network packets with a class identifier (classid)

                 that allows the Linux traffic controller (tc) to identify packets originating from a particular cgroup task

net_prio — this subsystem provides a way to dynamically set the priority of network traffic per network interface

ns — this is the namespace subsystem




cp -a /etc/cgconfig.conf /etc/cgconfig.conf.orig

建立 limitcpu, limitmem 這兩個 cgroups 去 limit resource


mount {
        cpuset  = /cgroup/cpuset;
        cpu     = /cgroup/cpu;
        cpuacct = /cgroup/cpuacct;
        memory  = /cgroup/memory;
        devices = /cgroup/devices;
        freezer = /cgroup/freezer;
        net_cls = /cgroup/net_cls;
        blkio   = /cgroup/blkio;

group limitcpu{
        cpu {
                cpu.shares = 400;

group limitmem{
        memory {
                # 可用 unit: m
                memory.limit_in_bytes = 512m;

# 每次都要 restart

service cgconfig restart

# checking config




# Starting a Process in a Control Group


* services that support sysconfig


[1] CGROUP_DAEMON="subsystem:control_group"



[2] CGROUP_DAEMON="<space separated list of groups>"


CGROUP_DAEMON="cpu:/limit_httpd memory:/limit_httpd"

[3] GROUP_DAEMON="subsystem1,subsystem2:/control_group"




The daemon() function in /etc/init.d/functions was modified to start the daemon in control group(s) specified by CGROUP_DAEMON environment variable.


Cgred (control group rules engine daemon)


cp /etc/cgrules.conf /etc/cgrules.conf.orig


#<user>                 <controllers>           <destination>
#<user>:<process name>  <controllers>           <destination>

# group: @group
# * for any user or group
# % — represents an item the same as the item in the line above.

<process name>
# a process name
# a full command path of a process

# comma separated controller names

service cgred start

chkconfig cgred on


Launch processes in a cgroup (cgexec)


syntax: cgexec -g subsystems:path_to_cgroup command arguments

cgexec -g cpu:group1 firefox http://www.redhat.com

--sticky option before the command to keep any child processes in the same cgroup.
 * cgred daemon is running 時很有用 ( allocated to cgroups based on the settings found in /etc/cgrules.conf )




# Processes

ps -e -O cgroup | grep httpd

13526 blkio:/;net_cls:/;freezer:/;devices:/;memory:/;cpuacct:/;cpu:/limitcpu;
cpuset:/ S ? 00:00:00 /usr/sbin/httpd
13528 blkio:/;net_cls:/;freezer:/;devices:/;memory:/;cpuacct:/;cpu:/limitcpu;
cpuset:/ S ? 00:00:00 /usr/sbin/httpd

# Subsystem

cat /proc/cgroups

# Hierarchies

yum -y install tree

tree /cgroup

# tester

openload 100

# server


Memory cgroup out of memory: Kill process 16638 (httpd) score 135 or sacrifice child
Killed process 16638, UID 48, (httpd) total-vm:47000kB, anon-rss:5276kB, file-rss:4800kB


cgget & cgset



syntax: cgget [-r <name>] [-g <controller>] [-a] <cgroup_path>

-r, --variable <name>

-g <contoller>

# Display opt

-n     do not print headers, i.e. name of groups.

-v, --values-only


cgget /


cgget -g cpu /


cgget -r memory.failcnt /limit_httpd

memory.failcnt: 0

cgget -nvr memory.memsw.failcnt /limit_httpd


cgget /limit_httpd | grep usage | grep -v max

memory.memsw.usage_in_bytes: 0
memory.usage_in_bytes: 0

cgget /limit_httpd | grep failcnt

memory.memsw.failcnt: 0
memory.failcnt: 0


cgset - set the parameters of given cgroup(s)


Other Tools


cgsnapshot - generate the configuration file for given controllers

cgclassify - move running task(s) to given cgroups

SYNOPSIS: cgclassify [-g <controllers>:<path>] [--sticky | --cancel-sticky] <pidlist>

--sticky    # the daemon of service cgred (cgrulesengd process) does not change both the  specified  pidlist  and  their children tasks.



subsystems setting




lower than 60 decrease the kernel's tendency to swap out process memory

memory.swappiness: 60

# When the OOM killer is disabled, tasks that attempt to use more memory than they are allowed are paused until additional memory is freed.
# To disable it, write 1 to the "memory.oom_control" file



# shares of CPU time are distributed per all CPU cores on multi-core systems.
# Default: cpu.shares: 1024

cpu.shares: 700



# cpuacct.usage

reports the total CPU time (in nanoseconds) consumed by all tasks in this cgroup

# reset:

     echo 0 > /cgroup/cpuacct/cpuacct.usage


    user — CPU time consumed by tasks in user mode.
    system — CPU time consumed by tasks in system (kernel) mode.





cpuset.cpus (mandatory)



cpuset.mems (mandatory)

# memory nodes that tasks in this cgroup are permitted to access.





The net_cls subsystem tags network packets with a class identifier (classid)
that allows the Linux traffic controller (tc) to identify packets originating from a particular cgroup.

net_cls.classid = Dec;

計算 net_cls.classid  的值

01:20 ==> 0x010020 (DEC=1 x 65536 + 20 x 16 =65568 )


net_cls.classid = 65568 ;


net_cls {
    net_cls.classid = 65858;

# 設定 tc filter (parent 1: & handle 20: 相當於 1:20)

tc filter add dev eth0 parent 1: protocol ip handle 20: cgroup


OOM test


cgget -r memory.oom_control  /limit_httpd

    memory.oom_control: oom_kill_disable 0
            under_oom 1

cgget -r memory.oom_control  /

    memory.oom_control: oom_kill_disable 0
            under_oom 0


Example: Apache





mount {
        cpuset  = /cgroup/cpuset;
        cpu     = /cgroup/cpu;
        cpuacct = /cgroup/cpuacct;
        memory  = /cgroup/memory;
        devices = /cgroup/devices;
        freezer = /cgroup/freezer;
        net_cls = /cgroup/net_cls;
        blkio   = /cgroup/blkio;

group limit_httpd{
        cpu {
                cpu.shares = 700;
        cpuset {
                cpuset.cpus = 0;
                cpuset.mems = 0;
        cpuacct {
        net_cls {
                net_cls.classid = 65568;
        memory {
                memory.limit_in_bytes = 700m;
                memory.memsw.limit_in_bytes = 800m;

Apply setting

service cgconfig restart

Check memory usage

cgget -r memory.memsw.usage_in_bytes /limit_httpd

cgget -r memory.stat  /limit_httpd

cgget -r memory.memsw.failcnt /limit_httpd

cgget -r memory.oom_control /limit_httpd

Put httpd inside cgroup

ps -e -o pid,cmd,cgroup | grep [h]ttpd

 2076 /usr/sbin/httpd             blkio:/;net_cls:/;freezer:/;devices:/;memory:/limit_httpd;cpuacct:/limit_httpd;cpu:/limit_httpd;cpuset:/limit_httpd
 2078 /usr/sbin/httpd             blkio:/;net_cls:/;freezer:/;devices:/;memory:/limit_httpd;cpuacct:/limit_httpd;cpu:/limit_httpd;cpuset:/limit_httpd
 2079 /usr/sbin/httpd             blkio:/;net_cls:/;freezer:/;devices:/;memory:/limit_httpd;cpuacct:/limit_httpd;cpu:/limit_httpd;cpuset:/limit_httpd
 2080 /usr/sbin/httpd             blkio:/;net_cls:/;freezer:/;devices:/;memory:/limit_httpd;cpuacct:/limit_httpd;cpu:/limit_httpd;cpuset:/limit_httpd
 2081 /usr/sbin/httpd             blkio:/;net_cls:/;freezer:/;devices:/;memory:/limit_httpd;cpuacct:/limit_httpd;cpu:/limit_httpd;cpuset:/limit_httpd
 2082 /usr/sbin/httpd             blkio:/;net_cls:/;freezer:/;devices:/;memory:/limit_httpd;cpuacct:/limit_httpd;cpu:/limit_httpd;cpuset:/limit_httpd
 2083 /usr/sbin/httpd             blkio:/;net_cls:/;freezer:/;devices:/;memory:/limit_httpd;cpuacct:/limit_httpd;cpu:/limit_httpd;cpuset:/limit_httpd
 2084 /usr/sbin/httpd             blkio:/;net_cls:/;freezer:/;devices:/;memory:/limit_httpd;cpuacct:/limit_httpd;cpu:/limit_httpd;cpuset:/limit_httpd
 2085 /usr/sbin/httpd             blkio:/;net_cls:/;freezer:/;devices:/;memory:/limit_httpd;cpuacct:/limit_httpd;cpu:/limit_httpd;cpuset:/limit_httpd
 2119 grep httpd                  blkio:/;net_cls:/;freezer:/;devices:/;memory:/;cpuacct:/;cpu:/;cpuset:/

Stress Testing

在 dmsg 出現以下 msg 後, apache 會 hang 了, 那時要 killall -9 httpd 才殺到

Memory cgroup out of memory: Kill process 21012 (httpd) score 142 or sacrifice child
Killed process 21012, UID 48, (httpd) total-vm:46664kB, anon-rss:4880kB, file-rss:5464kB



<IfModule prefork.c>
StartServers       5
MinSpareServers    5
MaxSpareServers   10
MaxClients        25
ServerLimit      256
MaxRequestsPerChild 100

ListenBacklog 100

# MaxRequestsPerChild

the limit on the number of requests that an individual child server process will handle
(memory leakage)
# 0=> the process will never expire.

# KeepAlive

only the first request is counted towards this limit.
(In effect, it changes the behavior to limit the number of connections per child.)





* only available in non-root cgroups

    FROZEN — tasks in the cgroup are suspended.
    FREEZING — the system is in the process of suspending tasks in the cgroup.
    THAWED — tasks in the cgroup have resumed.

# To suspend a specific process:

    Move that process to a cgroup in a hierarchy which has the freezer subsystem attached to it.
    Freeze that particular cgroup to suspend the process contained in it.


# all subsystems in one go: "mount -t cgroup none /cgroups"
mount -t cgroup freezer  /freezer -o freezer

# Create a child cgroup:
mkdir /freezer/0

# Put a task into this cgroup:
echo $task_pid > /freezer/0/tasks

# Freeze it:
echo FROZEN > /freezer/0/freezer.state

The freezer allows the checkpoint code to obtain a consistent
image of the tasks by attempting to force the tasks in a cgroup into a
quiescent state.

對比 kill 的 SIGSTOP and SIGCONT

Any programs designed to watch for SIGSTOP and SIGCONT could be broken by
attempting to use SIGSTOP and SIGCONT to stop and resume tasks.

We can demonstrate this problem using nested bash shells:

    $ echo $$

    $ bash

    $ echo $$

    # From a second, unrelated bash shell:

    $ kill -SIGSTOP 16690

    # 行 "jobs" 會見到它

    $ kill -SIGCONT 16690

In contrast, the cgroup freezer uses the kernel freezer code to prevent the freeze/unfreeze cycle from becoming visible to the tasks
being frozen. This allows the bash example above and gdb to run as expected.

The cgroup freezer is hierarchical.

freezer.self_freezing: Read only.

freezer.parent_freezing: Read only.