最後更新: 2020-05-21

介紹

把本機的 dmesg 資訊透過 network 送到另一機上

 

---

必要條件

# 要有 Kernel module

grep NETCONSOLE /boot/config-*

CONFIG_NETCONSOLE=m
CONFIG_NETCONSOLE_DYNAMIC=y

ls /lib/modules/`uname -r`/kernel/drivers/net/netconsole.*

/lib/modules/3.10.0-1160.31.1.el7.x86_64/kernel/drivers/net/netconsole.ko.xz

Usage

modprobe netconsole L-Port@L-IP/L-Interface,R-Port@R-IP/MAC

• L-Port - Local port
• L-IP - Local system IP
• L-Interface - Local system interface
• R-Port - Remote System udp port
• R-IP - Remote System IP
• MAC - Remote System Mac

Example

modprobe netconsole [email protected]/eth0,[email protected]/00:13:72:?:?:?

由於 6665 及 6666 都是 Default Port, 所以以上一句相當於以下一句

modprobe netconsole [email protected]/eth0,@192.168.123.10/00:13:72:?:?:?

Remark

 * Source 不能用 br

Fail 例子:

modprobe netconsole [email protected]/enp1s0,@192.168.123.11/00:13:72:?:?:?

... netpoll: netconsole: local port 6665
... netpoll: netconsole: local IPv4 address 192.168.123.10
... netpoll: netconsole: interface 'enp1s0'
... netpoll: netconsole: remote port 6666
... netpoll: netconsole: remote IPv4 address 192.168.123.11
... netpoll: netconsole: remote ethernet address 00:13:72:?:?:?
... netpoll: netconsole: enp1s0 is a slave device, aborting
... netconsole: cleaning up

modprobe netconsole [email protected]/eth0,@r.r.r.r/00:0c:29:?:?:?

... netconsole: local port 6665
... netconsole: local IP l.l.l.l
... netconsole: interface 'br0'
... netconsole: remote port 6666
... netconsole: remote IP r.r.r.r
... netconsole: remote ethernet address 00:0c:29:?:?:?
... netconsole: br0 doesn't support polling, aborting.
... netconsole: cleaning up

Source side output:

dmesg

[ 2569.466312] netconsole: local port 6666
[ 2569.466314] netconsole: local IP 192.168.123.33
[ 2569.466315] netconsole: interface eth1
[ 2569.466316] netconsole: remote port 6666
[ 2569.466318] netconsole: remote IP 192.168.123.10
[ 2569.466319] netconsole: remote ethernet address 00:13:72:?:?:?
[ 2569.477286] console [netcon0] enabled
[ 2569.477290] netconsole: network logging started

 

---

Testing

 

Local:

sysctl -w kernel.printk="7 4 1 7"

echo "Hello Kernel-World" > /dev/kmsg

參考

Remote:

nc -ul 6666

OR

nc -l -u -p 6666 | tee  -a ~/netconsole.log

P.S.

* tee 的 -a = --append

nc 是一次性的 connection 來, 斷了就無了 !!

 

---

永久設置

 

建立檔案:

/etc/modprobe.d/netconsole

內容如下:

options netconsole [email protected]/eth1,@192.168.1.2/00:13:72:?:?:?

在 sysctl.conf 內加上

kernel.printk = 7 4 1 7

 

---

Boot time netconsole

 

/etc/modprobe.d/netconsole.conf

options netconsole [email protected]/br0,@192.168.123.21/xx:xx:xx:xx:xx:xx

/etc/modules

netconsole

# 用 module 載入 netconsole 會失敗

[   48.415628] netpoll: netconsole: local port 6665
[   48.415662] netpoll: netconsole: local IPv4 address 192.168.123.10
[   48.415691] netpoll: netconsole: interface 'br0'
[   48.415720] netpoll: netconsole: remote port 6666
[   48.415749] netpoll: netconsole: remote IPv4 address 192.168.123.21
[   48.415778] netpoll: netconsole: remote ethernet address 50:46:5d:08:80:22
[   48.415808] netpoll: netconsole: br0 doesn't exist, aborting
[   48.415837] netconsole: cleaning up

# 只有在 /etc/rc.local 設定

modprobe netconsole

 

---

printk

 

kernel code 內使用:

printk("My Debugger is Printk\n");

會 output 在 dmesg 內

printk log level 設定

cat /proc/sys/kernel/printk

# current, default, minimum, boot-time-default
4       4       1       7         # <= Default

 * higher priority than this will be printed to console

Value

• 0    KERN_EMERG
• 1    KERN_ALERT
• 4    KERN_WARNING
• 6    KERN_INFO     
• 7    KERN_DEBUG          # Debug messages

Change Level

# 修改 current 的值

[方法1]

echo 5 > /proc/sys/kernel/printk

[方法2]

# "-n" set level of messages printed to console

dmesg -n 5

P.S.

dmesg --help

...
Supported log levels (priorities):
   emerg - system is unusable                  # 0
   alert - action must be taken immediately    # 1
    crit - critical conditions                 # 2
     err - error conditions                    # 3
    warn - warning conditions                  # 4
  notice - normal but significant condition    # 5
    info - informational                       # 6
   debug - debug-level messages                # 7

Printk from userspace

mknod -m 600 /dev/kmsg c 1 11

# default log level

echo "Hello Kernel-World" > /dev/kmsg

# Level: "KERN_CRIT" message

echo "<2>Writing critical printk messages from userspace" >/dev/kmsg

dmesg buffer sizes

# the kernel ring buffer

# grep -a CONFIG_LOG_BUF_SHIFT /boot/config-*               # 17 = 128k

# -s, --buffer-size size

dmesg -s 128000

 

---

Log Server

 

方式1:

/etc/inittab

n1:23:respawn:/usr/bin/nc -ul 6666 >> /var/log/netconsole

telinit q

/etc/logrotate.d/netconsole:

/var/log/netconsole {
    weekly
    rotate 8
    missingok
    compress
    copytruncate
    notifempty
    # Need to restart logger after log file move
    postrotate
        # Below line assumes netcat will be restarted by init
        killall -TERM netcat > /dev/null 2>&1 || true
}

方式2:

Server: /etc/rsyslog.d/netconsole.conf

$ModLoad imudp

$UDPServerRun 6666

if $fromhost-ip == '203.?.?.18' then /var/log/netconsole/203.?.?.18.log
& ~

Client: netconsole.sh

#!/bin/bash

modprobe netconsole netconsole=@203.?.?.18/eth0,@203.?.?.192/00:0c:29:?:?:?

dmesg -n 7

echo "test" > /dev/kmsg

test:

Client:

nc -u 203.?.?.192 6666

Server:

netstat -nlu

 

---