lxcfs

由 datahunter 在日, 11/01/2015 - 19:41 發表

最後更新: 2020-10-10

介紹

Overlay userspace files(filesystem) for cpuinfo, meminfo, stat and uptime.

=> pretty simple, written in C using libfuse(FUSE filesystem)

A cgroupfs compatible tree allowing unprivileged writes.

=> unprivileged user to interact with cgroupfs

---

It's designed to workaround the shortcomings of procfs, sysfs and cgroupfs

by exporting files which match what a system container user would expect.

---

A set of cgroup-aware files, LXCFS will take care

/proc/cpuinfo
/proc/diskstats
/proc/meminfo
/proc/stat
/proc/swaps
/proc/uptime
/sys/devices/system/cpu/online

---

When a container is started, each filtered hierarchy will be bind-mounted under /sys/fs/cgroup/* in the container

Install

# U16

apt-get install lxcfs

Build From Source

Prepare

# C7

yum groupinstall "Development Tools"

yum install fuse fuse-lib fuse-devel

# U16

apt-get install libcgmanager-dev libnih-dbus-dev libnih-dev libfuse-dev libdbus-1-dev

apt-get install libnih-dbus-dev libnih-dev libfuse-dev

apt-get install automake # 要 aclocal

apt-get install build-essential libtool # 要 libtoolize

apt-get install pkg-config

# Get source

mkdir /usr/src/lxcfs; cd /usr/src/lxcfs

# https://linuxcontainers.org/lxcfs/downloads/

wget https://linuxcontainers.org/downloads/lxcfs/lxcfs-4.0.12.tar.gz

tar -zxf lxcfs-4.0.12.tar.gz

# make

cd lxcfs-4.0.12

./configure --prefix=/opt/lxcfs

make -j

make install

# lib 設定

ls /opt/lxcfs/lib/lxcfs

liblxcfs.la  liblxcfs.so

建立 /etc/ld.so.conf.d/lxc.conf

/opt/lxcfs/lib/lxcfs

ldconfig

# Other

mkdir -p /opt/lxcfs/var/lib/lxcfs # 對應 /opt/lxcfs/share/lxcfs/lxc.mount.hook 設定

ln -s /opt/lxcfs/bin/lxcfs /usr/sbin/lxcfs

# Checking

lxcfs -v

4.0.5

Upgrading

* without restart instance

LXCFS is split into a shared library(liblxcfs) and a simple binary(lxcfs)

lxcfs binary will detect that a new version of the shared library(liblxcfs) is available and

will reload it using dlclose(3) and dlopen(3)

# To force a reload of the shared library

kill -s USR1 $(pidof lxcfs)

Remark

Users of LXCFS on musl(類似 glibc) are advised to restart LXCFS completely and all containers making use of it.

lxcfs Service

/lib/systemd/system/lxcfs.service # Ubuntu 18.04

/etc/systemd/system/lxcfs.service # Centos 7

# lxcfs 4.0
[Unit]
Description=FUSE filesystem for LXC
Before=lxc.service

[Service]
ExecStart=/opt/lxcfs/bin/lxcfs -l -f --enable-cfs --enable-pidfd /opt/lxcfs/var/lib/lxcfs
Restart=on-failure
ExecReload=/bin/kill -s USR1 $MAINPID                       # for upgrade lxcfs

[Install]
WantedBy=multi-user.target

systemctl daemon-reload

systemctl enable lxcfs --now

systemctl status lxcfs

ps aux | grep lxcfs

... /opt/lxcfs/bin/lxcfs -l -f --enable-cfs --enable-pidfd /opt/lxcfs/var/lib/lxcfs

Usage

Options

-h, --help
-f, --foreground
-l, --enable-loadavg
--enable-cfs # Enable CPU virtualization via CPU shares
--enable-pidfd # Use pidfd for process tracking

hook file

ls /opt/lxcfs/share/lxcfs

lxc.mount.hook  lxc.reboot.hook

vps configuration

[方法A] Global Enable lxcfs

Contaner 的 config

lxc.include = /usr/share/lxc/config/ubuntu.common.conf

ubuntu.common.conf

lxc.include = /usr/share/lxc/config/common.conf

common.conf

lxc.include = /usr/share/lxc/config/common.conf.d/

建立 /usr/share/lxc/config/common.conf.d/00-lxcfs.conf

lxc.hook.mount     = /opt/lxcfs/share/lxcfs/lxc.mount.hook
lxc.hook.post-stop = /opt/lxcfs/share/lxcfs/lxc.reboot.hook

[方法B] Per vps config

lxc.hook.mount     = /opt/lxcfs/share/lxcfs/lxc.mount.hook
lxc.hook.post-stop = /opt/lxcfs/share/lxcfs/lxc.reboot.hook

[方法C] 由 Source Compile 的 lxcfs 設定

...
#lxc.mount.auto = cgroup:mixed
lxc.autodev = 1             # 非必須
lxc.kmsg = 0                # 非必須
lxc.include = /opt/lxcfs/share/lxc/config/common.conf.d/00-lxcfs.conf

Inside Instance checking

mount | grep lxcfs

lxcfs on /proc/cpuinfo type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
lxcfs on /proc/diskstats type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
lxcfs on /proc/loadavg type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
lxcfs on /proc/meminfo type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
lxcfs on /proc/stat type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
lxcfs on /proc/swaps type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
lxcfs on /proc/uptime type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
lxcfs on /sys/devices/system/cpu/online type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)

uptime # VPS 有自己的 uptime 了

 19:16:50 up 0 min,  1 user,  load average: 0.82, 0.77, 0.77

Debug

lxc-start --logpriority DEBUG --logfile proxy.log proxy

kmsg

# Enable creating /dev/kmsg as symlink to /dev/console. This defaults to 1.

# Set this to 1 to enable /dev/kmsg symlinking

lxc.kmsg = 1

Version

* Support for the new unified cgroup hierarchy - cgroup2

* /proc/cpuinfo and cpu output in /proc/stat based on cpu shares

* pidfd supported process tracking

LXCFS needs to keep track of each container's init process in order to correctly virtualize various values.

Newer kernels(kernel 5.4) provide the concept of a pidfd which solves pid reuse problems.

* /proc/loadavg virtualization

container-specific /proc/loadavg view based on cgroups

* Compiler based hardening

a set of well-known compiler and linker options are automatically enabled

whenever the compiler or linker support them.