rsyslog

由 datahunter 在一, 04/11/2013 - 15:31 發表

最後更新: 2024-01-08

介紹

Modular architecture

Input Modules
Output Modules
Parser Modules

Check Version

rsyslogd -v

rsyslogd 5.8.12, compiled with:
        FEATURE_REGEXP:                         Yes
        FEATURE_LARGEFILE:                      Yes
        GSSAPI Kerberos 5 support:              Yes
        FEATURE_DEBUG (debug build, slow code): No
        32bit Atomic operations supported:      No
        64bit Atomic operations supported:      No
        Runtime Instrumentation (slow code):    No

* version 3 不支援 ruleset !!

P.S.

CentOS 6.0 = V4
CentOS 6.6 = V5
CentOS 7.X = V8

Opts

-c version # Selects the desired backward compatibility mode.

must always be the first option)

rsyslog v3 native interface = specify -c3

-d # Turns on debug mode.

-N level # Verify configure file

# higher levels will mean more verbosity

rsyslogd -N1

rsyslogd: version 8.1905.0, config validation run (level 1), master config /etc/rsyslog.conf
rsyslogd: End of config validation run. Bye.

Install Latest Version

https://www.rsyslog.com/rhelcentos-rpms/

The Adiscon RPM Repository supports recent rsyslog versions for RHEL/CentOS 6 and 7 including third party packages.

Step

cd /etc/yum.repos.d
wget http://rpms.adiscon.com/v8-stable/rsyslog.repo
yum install rsyslog

Configuration File

#################
#### MODULES ####
#################

/etc/rsyslog.conf

$ModLoad imuxsock

# For Kernel logging
$ModLoad imklog

# provides --MARK-- message capability
# i.e. "May  7 01:05:07 localhost rsyslogd: -- MARK --"
$ModLoad immark
# Default: 1200 (20 minutes)
# mark messages appear every minutes
$MarkMessagePeriod  60 

$ModLoad imudp
$UDPServerRun 514

$ModLoad imtcp
$InputTCPServerRun 514

###########################
#### GLOBAL DIRECTIVES ####
###########################

# Disabled by default
#$ActionFileEnableSync on

$IncludeConfig /etc/rsyslog.d/*.conf

###############
#### RULES ####
###############

................................

一些特別的符號:

#
/* */
~ # discards everything
* # everything
"?" # 用於使用 template name
& # 同時用多個 Action
"\" # 表示同行
$ # global directives

"?"

*.* ?DynFile
$template DynFile,"/var/log/%HOSTNAME%/%programname%.log"

"&"

*.=crit rger
& root
& /var/log/critmsgs

相當於

*.=crit rger
*.=crit root
*.=crit /var/log/critmsgs

Legacy & New Configuration

Legacy Configuration Sample:

# This sets up an UPD server on port 514:
$ModLoad imudp # needs to be done just once
$UDPServerRun 514

New Configuration Sample:

module(load="imudp")
input(type="imudp" port=["10514","10515","10516"])              # InputName = imudp10514, imudp10515, imudp10516

# InputName.AppendPort: since 7.3.9

# three listeners at ports 10514, 10515 and 10516 and
# assign a listner name of "udp10514", "udp10515" (inputname.appendPort="on")  to it
module(load="imudp")
input(type="imudp" port=["10514","10515","10516"]
      inputname="udp" inputname.appendPort="on")

Doc:

imptcp: http://www.rsyslog.com/doc/imtcp.html
impudp: http://www.rsyslog.com/doc/imudp.html

Facility.Severity

The list of Facilities available:

Numerical Facility  Description
0         kern      kernel messages
1         user      user-level messages
2         mail      mail system
3         daemon    system daemons
4         auth      security/authorization messages
5         syslog    messages generated internally by syslogd
6         lpr       line printer subsystem
7         news      network news subsystem
8         uucp      UUCP subsystem
9         cron      clock daemon
10        authpriv  security/authorization messages
11        ftp       FTP daemon
12        ntp       NTP subsystem
13        logaudit  log audit
14        logalert  log alert
15        clock     clock daemon (note 2)
16        local0    local use 0 (local0)
17        local1    local use 1 (local1)
18        local2    local use 2 (local2)
19        local3    local use 3 (local3)
20        local4    local use 4 (local4)
21        local5    local use 5 (local5)
22        local6    local use 6 (local6)
23        local7    local use 7 (local7)

The list of Severity Levels available:

0       Emergency: system is unusable                  emerg
1       Alert: action must be taken immediately        alert
2       Critical: critical conditions                  crit
3       Error: error conditions                        err
4       Warning: warning conditions                    warn
5       Notice: normal but significant condition       notice
6       Informational: informational messages          info
7       Debug: debug-level messages                    debug

The Priority

Priority = Facility * 8 + Severity_Level

authpriv.info = 10*8 + 6 = 86

Filter

rsyslog 有以下西種 Filter(Selectors)

"Traditional" - facility and severity based selectors
property-based filters
expression-based filters
BSD-style blocks (rsyslog v7+ 已經無左)

Traditional Filter(Default)

*.=info;*.=notice;\
mail.none /var/log/messages

Property-Based Filter(很小用)

Format:

:property, [!]compare-operation, "value"

Compare-operation:

contains
isequal
startswith
regex

Property:

msg
programname # i.e. sshd, named ...)
hostname (source)
fromhost #
fromhost-ip #
syslogtag
programname
syslogfacility # in numerical form
syslogfacility-text # in text form
syslogseverity
syslogseverity-text
timestamp
...

i.e.

:msg, contains, "error"                /path/to/file

P.S.

\" - the quote character (e.g. "String with \"Quotes\"")
\\ - the backslash character (e.g. "C:\\tmp")

Expression-Based Filter

Format:

if expr then action-part-of-selector-line

Example

# Basic

if $fromhost-ip == '192.168.1.1' then /var/log/remote_log/router.log

# Or

if $msg contains ['string 1', 'string 2'] then /var/log/somelog

# and, and not

if $syslogfacility-text == 'local0' and $msg startswith 'DEVNAME' and not ($msg contains 'error1' or $msg contains 'error0') then {
    /var/log/somelog
}

[2] fail2ban 的應用

(1)

# vsftpd "syslog_enable=YES"
if ($syslogfacility-text == "ftp") then {
  if ($msg contains "FAIL LOGIN:" or
        $msg contains "OK LOGIN:" or
        $msg contains "CONNECT:") then {
    /var/log/vsftpd.auth
    stop
  } else {
    /var/log/vsftpd.transfer
    stop
  }
}

(2)

Flow Control Statements

if expr then ... else ... - conditional execution
stop - stops processing the current message
call - calls a ruleset (just like a subroutine call)
continue - a NOP, useful e.g. inside the then part of an if

Statements:

"configuration directives" start with dollar sign ($)

Rulesets and Rules

Rules (selector + action)

# selector
# facility and a priority, separated by a period ('.')
# An asterisk ('*') stands for all facilities
# multiple facilities with the same priority pattern in one statement using the comma (',') operator.
# Multiple selectors may be specified for a single action using the semicolon (';') separator.

Outputs:

Outputs are also called "actions"

Nested Rule

if $programname contains 'shadowsocks' then {
       if $msg contains 'update traffic' then {
               action(type="omfile" ioBufferSize="32k"
                        asyncWriting="on"
                        flushInterval="15"
                        flushOnTXEnd="off"
                        file="/var/log/ss-traffic")
               stop
       }
       if $msg contains 'lookup v6 address' then {
               stop
       }
       action(type="omfile" file="/var/log/ss-server")
       stop
}

Remote Log Example(Legacy Sample): imudp

# provides UDP syslog reception. For TCP, load imtcp.

$ModLoad imudp
$UDPServerRun 514

#$InputUDPServerBindRuleset my_remote_ruleset

& ~

# 這兩句要放在很前 !!
if $fromhost-ip startswith '192.168.2.' then /var/log/server.log
& ~
if $fromhost-ip == '192.168.2.1' then /var/log/router.log
& ~

* line ("& ~") is important: it tells rsyslog to stop processing the message after it was written to the log.

rsyslog ver 8

warning: ~ action is deprecated, consider using the 'stop' statement instead

From v7 onward, the stop RainerScript directive is recommended instead.

"& ~" 改成 "& stop"

Ruleset

Doc

# 轉用另一個 Default Ruleset

# $DefaultRuleset <name>
$DefaultRuleset local

i.e.

# Remote logging
if $fromhost-ip == '192.168.152.137' then {
        action(type="omfile" file="/var/log/remotefile02")
        stop
    }

# local logging
authpriv.*  /var/log/secure

Multi Ruleset: Process remote messages

# ... module loading ...

# Process remote messages
# define new ruleset and add rules to it:
ruleset(name="remote"){
    action(type="omfile" file="/var/log/remotefile")
}

# bind ruleset to tcp listener and activate it:
input(type="imptcp" port="10514" ruleset="remote")

One port One hosts:

# ... module loading ...
# process remote messages

ruleset(name="remote10514"){
    action(type="omfile" file="/var/log/remote10514")
}

ruleset(name="remote10515"){
    action(type="omfile" file="/var/log/remote10515")
}

ruleset(name="test1"){
    if prifilt("mail.*") then {
        /var/log/mail10516
        stop
        # note that the stop-command will prevent this message from
        # being written to the remote10516 file - as usual...    
    }
    /var/log/remote10516
}

# and now define listeners bound to the relevant ruleset
input(type="imptcp" port="10514" ruleset="remote10514")
input(type="imptcp" port="10515" ruleset="remote10515")
input(type="imptcp" port="10516" ruleset="remote10516")

優化

/etc/sysconfig/rsyslog

SYSLOGD_OPTIONS="-c 5 -x"

opts:

-x Disable DNS for remote messages.

$programname

the programname is terminated by either (whichever occurs first):

end of tag, nonprintable character, ":", "[", "/"

parser.permitSlashInProgramName

# Version: 8.25.0+; Default: off
global(parser.permitSlashInProgramName="on")

off => $programname is truncated at the first slash

app/foo[1234] will result in a programname of "app"

/app/foo[1234] will be empty ("")

Log to Remote machine

Type:

UDP: @IP:PORT
TCP: @@IP:PORT

設定:

Default: Port 514

*.*        @192.168.0.1

Module: omfile

omfile: File Output Module

omfile is a built-in module that does not need to be loaded.

Example

action(type="omfile" dirCreateMode="0700" FileCreateMode="0644"
       File="/var/log/messages")

if $msg contains 'update traffic' then {
        action(type="omfile" ioBufferSize="8k"
                asyncWriting="on" flushInterval="10"
                file="/var/log/ss-traffic")
        stop
}

ioBufferSize

# Default: 4k (it is quite conservative)

Size of the buffer used to writing output data.

The larger the buffer, the potentially better performance is.

asyncWriting

# Default: off

If turned on, the files will be written in asynchronous mode via a separate thread.

In that case, double buffers will be used so that one buffer can be filled while the other buffer is being written.

Note that in order to enable FlushInterval, AsyncWriting must be set to “on”.

flushInterval

# Default: 1 second

unwritten data is flushed

flushOnTXEnd

# Default: on

"on" => data is written at the end of each transaction

Module: ommysql

ommysql: MySQL Database Output Module

...

Module: ompipe

ompipe: Pipe Output Module

...

Module: omuxsock

omuxsock: Unix sockets Output Module

...

Log to remote server

/etc/rsyslog.conf

# UDP
*.info @192.168.88.10

# TCP
*.info @@192.168.88.10

systemctl restart rsyslog

Test

logger "Test from logclient"

logrotate

sed -i '1i /var/log/vsftpd.log' /etc/logrotate.d/syslog

/var/log/vsftpd.log
/var/log/cron
/var/log/maillog
/var/log/messages
/var/log/secure
/var/log/spooler
{
    missingok
    sharedscripts
    postrotate
        /bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true
    endscript
}

Troubleshoot

[1] /dev/log is missing

As a result, programs relying on the syslog(3) call will not function properly, including, for instance, logger.

Normally, with rsyslogd, the imuxsock module will create the /dev/log socket on its own,

unlinking the previous entry before creating it.

module(load="imuxsock")

Fix on RHEL7

# imuxsock module will actually open and remove /run/systemd/journal/syslog socket.

# "/dev/log" device is created by the system "service-file systemd-journald.socket" which triggers journald

systemctl restart systemd-journald.socket

systemctl start rsyslogd

Fix on Ubuntu

# U16

它是由 systemd-journald-dev-log.socket 建立的

Notes

ls -l /dev/log

lrwxrwxrwx 1 root root 28 Aug  1 16:54 /dev/log -> /run/systemd/journal/dev-log

/lib/systemd/system/systemd-journald-dev-log.socket

[Socket]
Service=systemd-journald.service
ListenDatagram=/run/systemd/journal/dev-log
Symlinks=/dev/log
SocketMode=0666
PassCredentials=yes
PassSecurity=yes

systemctl enable systemd-journald-dev-log.socket --now

由於不成功, 可以嘗試

systemctl restart systemd-journald

Failed to restart systemd-journald.service: Unit systemd-journald.service is masked.

看來找到原因了

Fix

systemctl unmask systemd-journald
systemctl start systemd-journald
systemctl status systemd-journald
systemctl restart rsyslog

Turn off message reception via local log socket

# local messages are retrieved through imjournal now.

#$OmitLocalLogging on

# File to store the position in the journal

$IMJournalStateFile imjournal.state

Shell execute

This executes a program in a subshell.
The program is passed the template-generated message as the only command line parameter.
Rsyslog waits until the program terminates and only then continues to run.

^program-to-execute;template

i.e.

/etc/rsyslog.d/faillock.conf

if $msg contains "test1234" then {
    ^/root/scripts/faillock.sh
}

/root/scripts/faillock.sh

#### Code
Subject="Locked Users Alert (Server: $Server)"
echo "$1" | $MAILX -s "$Subject" $Email

Troubleshoot: RestrictAddressFamilies

當 script 內執行 "mailx" 時會失敗

postfix log

... postfix/sendmail[340423]: fatal: inet_addr_local[getifaddrs]: getifaddrs: Address family not supported by protocol

原因

/usr/lib/systemd/system/rsyslog.service

[Service]
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX

Fix

[Service]
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX AF_NETLINK

template

Syntax

template(parameters)

i.e.

template (name="mailBody" type="string" string="RSYSLOG Alert\\r\\nmsg='%msg%'")

Parameters

name

template 的名稱, 必須 unique

type

list
subtree
string (constant text and replacement variables)
plugin

string type

percent signs (‘%’) is interpreted by the rsyslog property replacer (%msg%)
Escape sequences "\X" (i.e. \n is a newline)

rsyslog property replacer

%property:fromChar:toChar:options%

%msg%
%HOSTNAME%
%syslogtag%
%TIMESTAMP%

Options

lowercase # convert property text to lowercase only
....

Module: omprog

It is similar to the “execute program (^)” action, but offers better security and much higher performance.

feeds log messages to that binary via stdin

module(load="omprog")

action (
        type="omprog"
        binary="/path/to/log.sh p1 p2 --param3=\"value 3\""
        template="RSYSLOG_TraditionalFileFormat"
)

i.e.

/etc/rsyslog.d/faillock.conf

if $syslogfacility-text == 'authpriv' and $msg contains 'pam_faillock' then {
        action (
                type="omprog"
                name="faillock-mail2admin"
                binary="/root/scripts/faillock-mail2admin.sh \"Account locked\""
        )
}

進階使用

output # Default: /dev/null

Full path of a file where the output of the external program will be saved.

If the file already exists, the output is appended to it. If the file does not exist,
it is created with the permissions specified by fileCreateMode.

If confirmMessages is set to “off” (the default),
both the stdout and stderr of the child process are written to the specified file.

If confirmMessages is set to “on”,
only the stderr of the child is written to the specified file (since stdout is used for confirming the messages).

Example

module(load="omprog")

action(type="omprog"
       binary="/usr/share/logging/db_forward.py"
       name="db_forward"
       confirmMessages="on"
       confirmTimeout="30000"
       queue.type="LinkedList"
       queue.saveOnShutdown="on"
       queue.workerThreads="5"
       action.resumeInterval="5"
       killUnresponsive="on"
       output="/var/log/db_forward.log")

原理

rsyslog uses stdin, stdout and parameters to communicate with the external plugin.

Rsyslog pushes messages via stdin.
Each message is terminated by a LF.
So a plugin can obtain a full message simply by reading a line.

The message format is generated by standard rsyslog methods, that is via a template.

feedback mode (default: off)

confirmMessages

O, K and LF (line feed) to stdout.

If the plugin writes a line to stdout containing anything else,
rsyslog will consider that the plugin has not been able to process the message.

When starting the plugin, if confirmMessages is on,
rsyslog will also wait for the plugin to confirm its initialization.

Specifies whether the external program provides feedback to rsyslog via stdout.

When this switch is set to “on”, rsyslog will wait for the program to confirm each received message.

This feature facilitates error handling: instead of having to implement a retry logic,
the external program can rely on the rsyslog queueing capabilities.

To confirm a message, the program must write a line with the word OK to its standard output.

If it writes a line containing anything else, rsyslog considers that the message could not be processed,
keeps it in the action queue, and re-sends it to the program later
(after the period specified by the action.resumeInterval parameter).

confirmTimeout # Default: 10000, Unit: milliseconds

Specifies how long rsyslog must wait for the external program to confirm each message when confirmMessages is set to “on”.

Retry: action.resumeInterval
Log: reportFailures

<= OK
=> log message 1
<= OK
=> log message 2
<= OK

Keep-alive feedback

To provide keep-alive feedback,
the plugin must periodically write a dot (.) to stdout, without ending the line.

Once the plugin completes the processing of the message, it must write the OK word

=> log message 2
<= .......OK

Script terminate

The script must stay active and accept new messages via stdin until it sees EOF on stdin.
It must not terminate after reading one message.

=> When rsyslog ends and send EOF to process specified in omprog

log

... omprog: program '/root/scripts/test.sh' (pid 1143275) terminated; will be restarted
... action 'test1' suspended (module 'omprog'), retry 0. There should be messages before this one giving the reason for suspension.
... action 'test1' resumed (module 'omprog')

Writing to stderr

When multiple instances of the program are running concurrently,
rsyslog guarantees that the lines written to stderr by the various instances will not appear intermingled in the output file

the program writes each line at a time, without buffering multiple lines. (4KB on Linux)

Notes

stdin_with_eof

Script Example

#!/bin/bash

# Read from stdin until EOF
while IFS= read -r line
do
    echo "You entered: $line"
    stdio="$stdin $line"
done

echo $stdio

Module: ommail

i.e. A message is sent at most once every 6 hours (21600 seconds),
any other messages are silently discarded.

module(load="ommail")

template (name="mailBody"  type="string" string="RSYSLOG Alert\\r\\nmsg='%msg%'")
template (name="mailSubject" type="string" string="disk problem on %hostname%")

if $msg contains "hard disk fatal failure" then {
   action(type="ommail" server="localhost" port="25"
          mailfrom="[email protected]"
          mailto="[email protected]"
          subject.template="mailSubject"
          action.execonlyonceeveryinterval="21600")
}

mail server 設定

server (Default: 127.0.0.1)
port (Default: 25)

mailto

mailto="[email protected]"

mailto=["[email protected]", "[email protected]"]

設定 subject

subject.template
subject.text

action.execonlyonceeveryinterval="<seconds>"

* Messages sent more frequently are simply discarded.

Multi-Action

ie.

/etc/rsyslog.d/faillock.conf

if $syslogfacility-text == 'authpriv' and $msg contains 'pam_faillock' then {
        action (
                type="omprog"
                name="faillock-log-msg"
                binary="/usr/bin/logger \"faillock action\""
        )
       action (
               type="omprog"
               name="faillock-mail2admin"
               binary="/root/scripts/faillock-mail2admin.sh \"Account locked\""
       )
}

瀏覽次數： 3764

夢想家

rsyslog

介紹

目錄

Check Version

Opts

Install Latest Version

Configuration File

Legacy & New Configuration

Facility.Severity

Filter

Traditional Filter(Default)

Property-Based Filter(很小用)

Expression-Based Filter

Flow Control Statements

Remote Log Example(Legacy Sample): imudp

優化

$programname

Log to Remote machine

Module: omfile

Module: ommysql

Module: ompipe

Module: omuxsock

Log to remote server

logrotate

Troubleshoot

Shell execute

template

Module: omprog

Module: ommail

Multi-Action