首頁 » Container » 進程管理工具(supervision)

supervisord

由 datahunter 在 二, 02/01/2018 - 13:06 發表

最後更新: 2023-12-13

目錄

 

介紹

HomePage: http://supervisord.org/

Program: Python

Supervisor is a client/server system

    that allows non-root users to monitor and control processes on UNIX-like operating systems.
    (TCP)

Supervisord starts processes as its subprocesses (don’t daemonize)
    and can be configured to automatically restart them on a crash

Subprocesses run under supervisor should not daemonize (foreground)

Starts processes as subprocesses 好處:

  • No need Pidfiles
  • Supervisord always knows the true up/down status of its children
  • allow “normal” users to control processes (Open 80/TCP)

Process Groups

Processes often need to be started and stopped in groups, sometimes even in a “priority order”.

Process Groups (“priority order” + “start all”)

Supervisor Components

  • Configuration file (“Windows-INI” style)
  • supervisord
    (responsible for starting child programs, restarting crashed or exited subprocesseses)
  • supervisorctl
    (talks to supervisord across a UNIX socket or an internet (TCP) socket)
  • Web Server (http://localhost:9001)
  • XML-RPC Interface

 

Installation

 

apt-get install supervisor

# U22

  • /etc/init.d/supervisor
  • /lib/systemd/system/supervisor.service
  • /etc/default/supervisor                           # DAEMON_OPTS=""
  • /etc/supervisor/supervisord.conf

supervisord -v

4.2.1

 

Creating a Configuration File

 

“sample” configuration

# Create a “sample” Supervisor configuration

echo_supervisord_conf > /etc/supervisord.conf

Configuration file locations

First win

  • ../etc/supervisord.conf
  • ../supervisord.conf
  • $CWD/supervisord.conf
  • $CWD/etc/supervisord.conf
  • /etc/supervisord.conf
  • /etc/supervisor/supervisord.conf

supervisord.conf

[supervisord]
logfile=/var/log/supervisor/supervisord.log
pidfile=/var/run/supervisord.pid
childlogdir=/var/log/supervisor

[unix_http_server]
file=/var/run/supervisor.sock
chmod=0700

; the below section must remain in the config file for RPC (supervisorctl/web interface) to work.
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface

[supervisorctl]
serverurl=unix:///var/run/supervisor.sock

[include]
files = /etc/supervisor/conf.d/*.conf

supervisord Opts

  • -n, --nodaemon         executable in the foreground
  • -c FILE                     The path to a supervisord configuration file.
  • -d PATH                    When supervisord is run as a daemon, cd to this directory before daemonizing.

Reload supervisord

# supervisord will stop all processes,
# reload the configuration from the first config file it finds,
# and start all processes.

kill -HUP pid

 

Config Section

 

[supervisord]

supervisord 自身

[unix_http_server] Section

  1. 支援 username & password login
  2. TCP 版: "[inet_http_server]"

[program:x] Section

多數被 "[include]" 引入

i.e.

[program:apache2]
command=/path/to/httpd -c "ErrorLog /dev/stdout" -DFOREGROUND
redirect_stderr=true

redirect_stderr (Default: false)

This is the equivalent of executing /the/program 2>&1

[rpcinterface:supervisor] Section

unix_http_server 及 inet_http_server 使用到它

[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface

[inet_http_server] Section - Web interface(TCP)

 * Disabled by default

功能: start, stop, and monitor your processes.

[inet_http_server]         
port=*:9001
;username=user     ; (Default is no username)
;password=123      ; (Default is no password)

Notes

它須要配合 '[rpcinterface:supervisor]' 使用

 

 

pidproxy

 

Some processes (like mysqld) ignore signals sent to the actual process which is spawned by supervisord

upon the receipt of a signal, sends the signal to the pid provided in a pidfile

[program:mysql]
command=/path/to/pidproxy /path/to/pidfile /path/to/mysqld_safe

 

Subprocess Environment

 

No shell is executed by supervisord when it runs a subprocess,
so environment variables such as USER, PATH, HOME, SHELL, LOGNAME, etc. are not changed from their defaults

[program:apache2]
environment=HOME="/home/chrism",USER="chrism"

 

Process States

 

BACKOFF (30)

The process entered the STARTING state but subsequently exited too quickly
(before the time defined in startsecs) to move to the RUNNING state.

EXITED (100)

The process exited from the RUNNING state (expectedly or unexpectedly).

FATAL (200)

The process could not be started successfully.

 

supervisorctl

 

功能

Control the processes that are currently managed by supervisord.

Connection

supervisorctl -- UNIX Socket/TCP --> supervisord

Usage

supervisorctl Actions

supervisorctl                # go to shell

Opts

  • -s, --serverurl URL
  • -u, --username
  • -p, --password

Help

help [action]

supervisorctl help

default commands (type help <topic>):
=====================================
add    exit      open  reload  restart   start   tail
avail  fg        pid   remove  shutdown  status  update
clear  maintail  quit  reread  signal    stop    version

Actions

Info

supervisorctl status [name]

apache2                          RUNNING   pid 42, uptime 0:37:58

pid [<name>/all]             # Get the PID of supervisord / child / all

Log

  • clear <name> <name>    # Clear multiple process’ log files
  • clear all

Debug

  • fg <process>           # Connect to a process in foreground mode Press Ctrl+C to exit foreground
  • tail [-f] <name> [stdout|stderr]    # Default: stdout. Output the last part of process logs. Ctrl-C to exit.

Processes

  • stop <name|gname:*|all>
  • start <name|gname:*|all>
  • restart <name|gname:*|all>  # restart "Processe" not reread supervisord config files.

config

  • reread                        # Reload the daemon’s configuration files,
                                     # without add/remove (no restarts)
  • add / remove              # Activates any updates in config for process/group
  • update [<gname>]     # Reload config and add/remove as necessary.
                                     # It will restart affected programs

supervisord

reload                    # Restarts the remote supervisord

ie.

supervisorctl reread

apache2: changed

supervisorctl restart apache2

 

Log

 

類型

  • Supervisord Log
  • Child Process Logs

Supervisord Log

It keeps an operations log at $CWD/supervisor.log by default

Signal: SIGUSR2

supervisord will close and reopen the main activity log and all child log files.

Settings

; Default: $CWD/supervisord.log
logfile=/var/log/supervisor/supervisord.log

; error/warn/info/debug
loglevel=info                                  ; Default: info

; Rotation
logfile_maxbytes=50MB                          ; Default: 50MB
logfile_backups=10                             ; Default: 10

Log File Name

supervisord.log.1, supervisord.log.2 etc.

No File Log

logfile=/dev/null
logfile_maxbytes=0

Child Process Logs

當 [program:x] 沒有 logfile 相關設定, 那就會使用 AUTO log mode

=> capture the child proces s’ stdout and stderr output into temporary files
placed in the directory configured as childlogdir of the [supervisord]

Settings

  • stdout_logfile, stdout_logfile_maxbytes, stdout_logfile_backups
  • stderr_logfile, stderr_logfile_maxbytes, stderr_logfile_backups

Notes

nocleanup # Default: false. In [supervisord]

Prevent supervisord from clearing any existing AUTO child log files at startup time.

 

Process Group

 

[group:x] Section
programs=bar,baz
priority=999        ; Default: 999

 

stopasgroup & killasgroup

 

stopasgroup

  • Send the "stopsignal" to the whole process group
  • Implies killasgroup is true

killasgroup

send SIGKILL to the program its whole process group after "stopwaitsecs"

流程

stopsignal(TERM) -> stopwaitsecs(10) -> SIGKILL

應用: Process do not propagate stop signals to their children

i.e. Apache

執行 `supervisorctl restart apache2` 後 apache 死了

supervisorctl status apache2

apache2    FATAL    Exited too quickly (process log may have details)

原因

Apache is launched by calling apache2ctl as root (to bind to port 80),
 which then spawns apache2 processes as the www-data user.

when stopping Apache with supervisor,
 these child processes become orphaned
 (the parent process exits without stopping the children)

Fix

[program:apache2]
command=apachectl -D FOREGROUND
redirect_stderr=true
stopasgroup=true

 

Other

stopsignal (Default: TERM)

TERM, HUP, INT, QUIT, KILL, USR1, or USR2

stopwaitsecs (Default: 10)

 

autostart & autorestart

 

autostart (Default: true)

automatically when supervisord is started

startsecs (Default: 1)

program needs to stay running after a startup to consider the start successful

(State: STARTING -> RUNNING)

 * startsecs 優於 exitcodes => 不夠時間的正常 exit 也算 failure

流程:

autorestart -> startretries
                STARTING <-> BACKOFF -> FATAL
                   |
                RUNNING

autorestart=unexpected    # Default: unexpected

When a process is in the EXITED state, it will automatically restart.

  • true: the process will be unconditionally restarted
  • false: the process will not be autorestarted
  • unexpected: If it exited with an exit code that doesn’t match one of the exit codes defined in the exitcodes

startretries (Default: 3)

supervisord will wait one, two and then three seconds between each restart attempt, for a total of 5 seconds.

 

gzip log file

 

supervisor 本身沒有 gzip log 功能, 所以要透過系統的 cron 及 logrotate 實現

/etc/supervisor/conf.d/my_app.cf

[program:my_app]
...
stderr_logfile=/var/log/supervisor/%(program_name)s_stderr.log
stdout_logfile=/var/log/supervisor/%(program_name)s_stdout.log
stdout_logfile_maxbytes=0
stderr_logfile_maxbytes=0
stdout_logfile_backups=0
stderr_logfile_backups=0

/etc/logrotate.d/supervisor-my_app

/var/log/supervisor/my_app_*.log {
  daily
  rotate 7
  copytruncate
  compress
  missingok
  notifempty
}

 

 

Setup in CT(U22)

 

1. Install supervisor

# U22

apt-get update && apt-get install -y supervisor

Default
 - /etc/supervisor/supervisord.conf
 - /var/run/supervisor.sock
 - /var/run/supervisord.pid
 - /var/log/supervisor/supervisord.log
 - /var/log/supervisor/<APP>
 - /etc/supervisor/conf.d/*.conf

supervisord.conf

[supervisord]
nodaemon=true
user=root
logfile=/dev/null
logfile_maxbytes=0
...

 * supervisord 直接在 FOREGROUND 行, 所以不用 log

 * 不加 "user=root" 會有 warning

apache.sv.conf

; U22
[program:apache]
...
command=apachectl -D FOREGROUND
; apache 'stop' setting
stopasgroup=true

Notes

測試成功後再加設定

apachectl -D FOREGROUND

當 app 直接 log 到 podman 時

[program:app]
stdout_logfile=/dev/fd/1
stdout_logfile_maxbytes=0
redirect_stderr=true

cron.sv.conf

[program:cron]
command=cron -f
autostart=true
autorestart=true
redirect_stderr=true
stdout_logfile=/dev/null
stdout_logfile_maxbytes=0
»
  • 瀏覽次數： 695

Creative Commons license icon Creative Commons license icon