最後更新: 2021-08-30

介紹

HomePage: https://rsync.samba.org/

目錄

• Install & Compile
• Check Version
• Client Usage
   - 用 ssh 去 sync 的方式
   - sync data 到 deamon 的 module
• Source 尾的 '/'
• 常用選項
• 限速
• 減低 bandwith 使用
• 多個 source
• CIFS VFS and Symbolic links
• Partial File
• Find Change
• More atomic
• exclude
• include
• include 與 exclude
• Usage Example
• Troubleshoot
• Performance
• Merge two folders
• --copy-dest 與 --link-dest
• Output checksum
• Other Info.
• Changelog

Other

unison

 

---

Install & Compile

 

Centos X:

yum install rsync

Compile:

# Browse the development history

https://download.samba.org/pub/rsync/NEWS

# 準備 library

# acl, xattrs, iconv

yum install libacl-devel libattr-devel

# make, gcc

yum groupinstall "Development Tools"

Remark

 * There is no package iconv-devel in CentOS7. iconv is part of glibc.

    The program iconv is included in the package glibc-common.

# Get source code

[A]

wget https://download.samba.org/pub/rsync/src/rsync-3.1.3.tar.gz

wget https://download.samba.org/pub/rsync/src/rsync-3.1.3.tar.gz.asc

gpg --verify rsync-3.1.3.tar.gz.asc

[B]

cd /usr/src

git clone git://git.samba.org/rsync.git

cd rsync

./prepare-source

# Compile it

./configure --prefix=/usr --sysconfdir=/etc --disable-ipv6

make -j 2

# backup original rsync

mv /usr/bin/rsync /usr/bin/rsync.orig

make install

 

---

Check Version

 

rsync --version

rsync  version 3.0.9  protocol version 30
Copyright (C) 1996-2011 by Andrew Tridgell, Wayne Davison, and others.
Web site: http://rsync.samba.org/
Capabilities:
    64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
    socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
    append, ACLs, xattrs, iconv, symtimes

 

---

Client Usage

 

rsync 有兩種傳送模式, 分別是

• remote-shell                   <-- 透過 ssh 進行
• rsync module (Daemon)  <-- 873/TCP

用 rsh 去 rsync 的方式

 * Default 只會用 port 22

 * rsync must be installed on both the source and destination machines

Pull: rsync [OPTION...] [USER@]HOST:SRC... [DEST]

Push: rsync [OPTION...] SRC... [USER@]HOST:DEST

Remark: "-e" Option

-e, --rsh=COMMAND           # specify the remote shell to use

This option allows you to choose an alternative remote shell program to use for communication

  between the local and remote copies of rsync.

Typically, rsync is configured to use ssh by default, but you may prefer to use rsh on a local network.

/usr/bin/rsh = /etc/alternatives/rsh = /usr/bin/ssh

Remark: 用 ssh key file to login

rsync -a -e "ssh -l ssh-user -i /home/ssh-user/id_rsa" /var/vmail/ vmail@mailserver:/var/vmail/

-e, --rsh=COMMAND           # specify the remote shell to use

Remark: rsync with ssh change port

rsync .. -e "ssh -p 2222" ..

# 改登入方式及Port一共有3方法

• -e, --rsh=COMMAND
• RSYNC_RSH  environment
• .ssh/config

Remark: Home Directory

USER@SERVER:~/FOLDER

 

Sync data 到 deamon 的 module

查看:

# 查看對方的 DAEMON  有什麼 module 可以 rsync

rsync rsync://<IP>

vmail

log

... module-list request from NUMBER.ctinets.com (R.R.R.R)

# List Folder

rsync rsync://vmail@domain/vmail/

output

drwxr-xr-x          4,096 2021/08/23 12:37:19 .

rsync 過去:

rsync [USER@]host::module   /dest

# "rsync://" 方式可以設定 port

rsync [OPTION]... SRC [SRC]... rsync://[USER@]HOST[:PORT]/DEST

Default 檔案對比方式:

"quick check" <-- size & mtime

當檔案的 mtime 不同時

-rw-------  1 datahunter users 1073741824 Aug 30 17:49  test.bin
-rw-------  1 datahunter users  654049280 Aug 30 17:49  .test.bin.mZN3Pa

rsynd 會抄那檔案一份, 之後用 hash 對比不同部份

對比要用 -vv 才看到

i.e.

total: matches=32768  hash_hits=32768  false_alarms=0 data=0

P.S.

Default rysnc 沒有加上 -a 時, 它只會 rsync 檔案 !!

 

---

Source 尾的 '/'

 

Source 尾部的 "/" 會影響 rsync 目錄還是檔案過去目的地 !!

情況1: foo/

以下 CMD 是有不同的結果

rsync -av foo   /dest               # 會在 dest 會建立目錄 foo

rsync -av foo/ /dest               # 不會建立目錄

情況2: foo/*

rsync -av foo/* rsync://U@D/M

foo 內的隱藏檔(foo/.*)及隱藏目錄不會被 rsync 過去, foo/SubFolder/.* 不受影響, 會 sync

 * 如果想 rsync 隱藏檔: rsync -av foo/ rsync://U@D/M

情況3: 當 Source 是 module

在用 module 時,  rsync 永遠是 module 內的檔案

rsync -av [USER@]host::module       /dest

相同

rsync -av [USER@]host::module/     /dest

rsync -av [USER@]host::module/*   /dest

 

---

---

常用選項

 

• --password-file=FILE

ENV: RSYNC_PASSWORD

• -a, --archive archive mode (相當於 -rlptgoD) (注意 ! no -H,-A,-X,-S)
  • -r, --recursive         # recurse into directories
  • -l, --links                # copy symlinks as symlinks
  • -p, --perms             # preserve permissions
  • -o, --owner             # preserve owner (super-user only)
  • -g, --group              # preserve group
  • -t, --times               # preserve modification times
  • -D                          # same as --devices --specials

* 當沒有加上 -a 參數時, 那 default 只 rsync 單一個檔案

i.e

rsync MyFolder IP:/var/lib/lxc

skipping directory test

* backup 時用 "-avzSH --numeric-ids --delete"

 

• -H, --hard-links               # 當來源是 hardlink 時, 目的地都是 hardlink (不個它們要在同一次 rsync 內出現)

 

• -A, --acls                    # ACLs (implies -p)
                                   # The source and destination systems must have compatible ACL entries for this option to work properly.

 

• -X, --xattrs                 # Preserve extended attributes

 

• -S, --sparse                # turn sequences of nulls into sparse blocks (Conflicts with --inplace)

 

• -W, --whole-file       # copy files whole (w/o delta-xfer algorithm)
                               # [This is the default when both the src and dest are local]

 

• -n, --dry-run

 

• -u, --update                      # skip files that are newer on the receiver (注意)

 

• --delete                        # 不存在於本地, 但存在於目的地的檔案將被刪除 (deletes before transfer)

                                              --delete-during            # when talking to rsync 3.0.0 or newer

                                                  邊 copy 邊 delete

                                              --delete-before           # when talking to an older rsync

                                                  useful if the destination has little storage space as it will first free up more disk space

                                                  whole operation is a two step process and thus slower (1. delete, 2. copy)

                                              --delete-delay            # 相當於 delete-during + delete-after

                                                  It works like "--delete-during", except that it won't delete files immediately

                                                  but after the synchronization is done

                                              --delete-after             # 很小機會用到  

• --numeric-ids                   不用帳戶的名稱去對應 UID 及 GID

 

• -P                                     # same as "--partial --progress" (在 sync 的過程中 partial file 會叫 ".filename.Sql47f")

 

• --progress                         # 顯示每個檔案 rsync 的進度

  
   

• --delete-excluded                        # delete excluded files on the receiving end
                                                    # Default 是不會理會 exclude 了的檔案, 任由它存在

 

• -c, --checksum                            # sendier: 為所有檔案建立 checksum
                                                    # receiver: 為相同 size 的檔案建立 checksum
                                                    # (Compare a 128-bit checksum for each file)
                                                    # => forces the sender to checksum all files (quite slow)

 

• -I, --ignore-times                        # don't skip files that match size and time

Normally rsync will skip any files that are already the same length and have the same time-stamp.

This option turns off this behavior.

means that rsync will checksum every file, even if the timestamps and file sizes match.

This means it will synchronise more files than the default behaviour.

It will include changes to files even where the file size is the same and

the modification date/time has been reset to the original value (resetting the date/time is unlikely to be done in practise, but it could happen).

Checksumming every file means it has to be entirely read from disk, which may be slow.

• --size-only                                 # skip files that match in size

means that rsync will skip files that match in size, even if the timestamps differ.

This means it will synchronise less files than the default behaviour.

It will miss any file with changes that don't affect the overall file size.

• --stats: gives you a nice summary rate, and speed-up rate at the end of the job.

# 加了 --stats 才有
Number of files: 85,471 (reg: 77,705, dir: 5,166, link: 2,515, dev: 41, special: 44)
Number of created files: 40 (reg: 40)
Number of deleted files: 0
Number of regular files transferred: 52
Total file size: 18,802,250,685 bytes
Total transferred file size: 170,354,844 bytes
Literal data: 1,200,040 bytes
Matched data: 169,182,024 bytes
File list size: 1,048,289
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 3,417,903
Total bytes received: 212,521

# Default 有的
sent 3,417,903 bytes  received 212,521 bytes  345,754.67 bytes/sec
total size is 18,802,250,685  speedup is 5,179.08

• -h, --human-readable        output numbers in a human-readable format

 

• -u, --update

This forces rsync to skip any files which exist on the destination and have a modified time that is newer than the source file. (If an existing destination file has a modification time equal to the source file's, it will be updated if the sizes are different.)

Note that this does not affect the copying of dirs, symlinks, or other special files. Also, a difference of file format between the sender and receiver is always considered to be important enough for an update, no matter what date is on the objects. In other words, if the source has a directory where the destination has a file, the transfer would occur regardless of the timestamps.

This option is a transfer rule, not an exclude, so it doesn't affect the data that goes into the file-lists, and thus it doesn't affect deletions. It just limits the files that the receiver requests to be transferred.

• --ignore-existing       # skip updating files that exist on receiver

i.e.

rsync -av --ignore-existing folder1/ folder2

測試

Test prepare

mkdir folder1 folder2
echo test1 > folder1/test1.txt;

rsync -a folder1/ folder2

rsync -avn --ignore-existing folder1/ folder2

sending incremental file list

sent 64 bytes  received 12 bytes  152.00 bytes/sec
total size is 6  speedup is 0.08 (DRY RUN)

rsync -avvn --ignore-existing folder1/ folder2    # 要用 -vv 才看到 exists

sending incremental file list
delta-transmission disabled for local transfer or --whole-file
test1.txt exists
total: matches=0  hash_hits=0  false_alarms=0 data=0

sent 71 bytes  received 107 bytes  356.00 bytes/sec
total size is 6  speedup is 0.03 (DRY RUN)

• --skip-compress=LIST

 * separated by slashes (/)

 * The characters asterisk (*) and question-mark (?) have no special meaning.

# Example

--skip-compress=gz/zip/7z/bz2/jpg/png/mp[34]

# Default list of suffixes

7z ace avi bz2 deb gpg gz iso jpeg jpg lz lzma lzo mov mp3 mp4 ogg png rar rpm rzip tbz tgz tlz txz xz z zip

• --no-motd

suppress daemon-mode MOTD

• -q, --quiet

suppress non-error messages

此 Setting 影響 stdout 與 "-v" 並無關係

加了 -q 後 stdout 會完全沒有 output,

log file ("--log-file") 有 file sync 時:

2020/12/31 11:45:31 [3868] building file list
2020/12/31 11:45:31 [3868] <f.st...... test/adminer-4.7.8-en.php
2020/12/31 11:45:32 [3868] sent 615 bytes  received 2,876 bytes  1,396.40 bytes/sec
2020/12/31 11:45:32 [3868] total size is 330,037  speedup is 94.54

在沒有 file 要 sync 及沒有 Error 的情況下, log file 只會有 summary

2020/12/31 11:44:17 [344] building file list
2020/12/31 11:44:17 [344] sent 123 bytes  received 21 bytes  288.00 bytes/sec
2020/12/31 11:44:17 [344] total size is 330,036  speedup is 2,291.92

• --modify-window

When comparing two timestamps, rsync treats the timestamps as being equal

if they differ by no more than the modify-window value. The default is 0

 

其他有用選項:

• -i, --itemize-changes         # output a change-summary for all updates
• --log-file=FILE

Backup:

• -b, --backup                     # 當沒有指定 DIR 時, 就在原位加上 "~" 來保留 backup file
• --backup-dir=DIR

 

Exclude:

• --exclude=PATTERN
• --exclude-from=FILE            # read exclude patterns from FILE
• --delete-excluded                 # also delete excluded files from dest dirs

 

batch:

• --write-batch=FILE                # write a batched update to FILE
• --only-write-batch=FILE         # like --write-batch but w/o updating dest
• --read-batch=FILE                 # read a batched update from FILE

P.S.

用 daemon  時, 必須要停 SELinux, 否則會傳送失敗 !!

 

---

限速(--bwlimit)

 

 

--bwlimit=N        # rsync 的限速功能, 單位是 KiB/s

 

---

減低 bandwith 使用

 

• --partial                                       # keep partially transferred files
• -z, --compress                              # 壓縮
• -y, --fuzzy                                    # find similar file for basis if no dest file

--fuzzy 解說

Without patching, the rsync utility lacks support to detect when a file was renamed/moved across multiple directories inside the synced tree.

There is a --fuzzy option to save bandwidth by building upon similar files on the target side, but only in the same directory.

 

---

多個 source

 

rsync -av host::modname/file{1,2} host::modname/file3 /dest/

 

---

CIFS VFS and Symbolic links

 

For security reasons Samba does not support symbolic links with absolute paths though
(only relative paths, not paths beginning with / )

遇上 Error:

rsync: symlink .......   failed: Operation not supported (95)

解決方法:

--safe-links

-L, --copy-links            transform symlink into referent file/dir

--copy-unsafe-links     only "unsafe" symlinks are transformed

--safe-links            ignore symlinks that point outside the tree

 

---

Partial File

 

By default, rsync will delete any partially transferred file if the transfer is interrupted.

當 rsync "test.bin" 時會有 ".test.bin.WOvBne"

Opts

• --partial                   # keep partially transferred files
• --partial-dir=DIR      # DIR is relative path (--partial-dir=.rsync-partial)

just tells the receiving end to keep partially transferred files if the sending end disappears

complete the transfer by running rsync again with either --append or --append-verify

 * --inplace 比 --partial 優先

instead of the default method of creating a new copy of the file and moving it into place when it is complete,

rsync instead writes the updated data directly to the destination file.

This option is useful for transferring large files with block-based changes or appended data

It can also help keep a copy-on-write filesystem snapshot from diverging the entire contents of a file that only has minor changes.

WARNING:

 - Hard links are not broken.

 - you  should not use this option to update files that are being accessed by others, so be careful when choosing to use this for a copy.

 - The  files  data will be in an inconsistent state during the transfer and will be left that way if the transfer is interrupted or if an update fails.

 * 在 local --partial 沒有意思 (要 -vv 才看到)

delta-transmission disabled for local transfer or --whole-file

續存

--append

Implies --inplace. Files are written directly to their targets.

when you're using --append, no temporary files are ever created.

If a file needs to be transferred and its size on the receiver is the same or longer than the size on the sender, the file is skipped.

--append-verify

included in the full-file checksum verification step

which will cause a file to be resent if the final verification step fails

(rsync uses a normal,  non-appending --inplace transfer for the resend)

--checksum

compare a 128-bit checksum(MD5) for each file that has a matching size.

different sizes will cause rsync to upload the entire file,  overwriting the target with the same name.

 

---

Find Change

 

# Output a change-summary for all updates

-i, --itemize-changes

Example

YXcstpoguax  path/to/file
|||||||||||
||||||||||╰- x: The extended attribute information changed
|||||||||╰-- a: The ACL information changed
||||||||╰--- u: The u slot is reserved for future use
|||||||╰---- g: Group is different
||||||╰----- o: Owner is different
|||||╰------ p: Permission are different
||||╰------- t: Modification time is different
|||╰-------- s: Size is different
||╰--------- c: Different checksum (for regular files), or
||              changed value (for symlinks, devices, and special files)
|╰---------- the file type:
|            f: for a file,
|            d: for a directory,
|            L: for a symlink,
|            D: for a device,
|            S: for a special file (e.g. named sockets and fifos)
╰----------- the type of update being done::
             <: file is being transferred to the remote host (sent)
             >: file is being transferred to the local host (received)
             c: local change/creation for the item, such as:
                - the creation of a directory
                - the changing of a symlink,
                - etc.
             h: the item is a hard link to another item (requires
                --hard-links).
             .: the item is not being updated (though it might have
                attributes that are being modified)
             *: means that the rest of the itemized-output area contains
                a message (e.g. "deleting")

i.e.

>f+++++++++ centos7.qcow2
>f.st...... centos7.qcow2

 

---

More atomic

 

--delay-updates

puts the updated file into a holding directory until the end of the transfer,
at which time all the files are renamed into place in rapid succession.

This attempts to make the updating of the files a little more atomic.

By default the files are placed into a directory named ".~tmp~" in each file’s destination directory

 * Conflicts with "--inplace" and "--append"

This option uses more memory on the receiving side (one bit per file transferred) and
also requires enough free disk space on the receiving side to hold an additional copy of all the updated files.

Tips

# rsync to cleanup old ".~tmp~" dirs that might be lying around.

--delay-updates --exclude="*.~tmp~"

 

---

exclude

 

# 用一個 file 保存要 exclude 的 path

--exclude-from '/home/backup/exclude.txt'

--exclude 的 format:

 * exclude path 與 dest. path 沒有關係

--exclude="foo"

exclude all filename or foldername matching full foo

(只會中 foo, 不會中 foo1 foo2)

--exclude="*.o"                       # matches a pattern

exclude all filename or foldername matching *.o

wildcard characters:  ’*’, ’?’, and ’[’

• ’*’   matches any path component, but it stops at slashes
• ’**’ to match anything, including slashes
• ’?’   matches any character except a slash (/)
• ’[’   introduces a character class, such as [a-z]

--exclude="foo/"                      # 尾的 "/"

exclude any folder called foo, 在任何層數的 Folder 都有效

--exclude="foo/*"

會有一個空的 Folder 在 remote

exclude 多個 folder & file

--exclude=FOLDER --exclude=FILE

--exclude="/foo"                     # "/" 開頭

exclude a file called foo in the "root path"

 * 用 "--verbose --dry-run"(-vn) 去驗證, 它們可以看到 "root path" 是什麼

說明

mkdir src/folder dst -p; touch src/test1.txt; touch src/folder/test2.txt

tree src

src
├── folder
│   └── test2.txt
└── test1.txt

[1]

rsync -avn src dst

src/
src/test1.txt
src/folder/
src/folder/test2.txt

=> 有 Folder "src" 在 dst Folder 內

[2]

rsync -avn src/ dst

test1.txt
folder/
folder/test2.txt

正確 exclude 的方法

rsync -avn --exclude=/src/test1.txt src dst

src/
src/folder/
src/folder/test2.txt

rsync -avn --exclude=/test1.txt src/ dst

./
folder/
folder/test2.txt

--exclude="/foo/*/bar"

exclude any file called bar two levels below a directory called foo in the transfer-root directory

--exclude="/foo/**/bar"

exclude any file or folder called bar two or more levels below a directory called foo in the transfer-root directory

* paths given need to be relative to the source path

 

---

include

 

# Don't exclude files matching PATTERN

# This option is a simplified form of the "--filter" option

mkdir src/scripts dst -p; touch src/{a..d}.txt; touch src/scripts/{a..d}.sh

tree src

src/
├── a.txt
├── b.txt
├── c.txt
├── d.txt
└── scripts
    ├── a.sh
    ├── b.sh
    ├── c.sh
    └── d.sh

1 directory, 8 files

rsync -avn --include='*.sh' src/ dst/       # Default 係 include ALL 的

a.txt
b.txt
c.txt
d.txt
scripts/
scripts/a.sh
scripts/b.sh
scripts/c.sh
scripts/d.sh

 

---

include 與 exclude

 

[1] "--exclude" 與 "--include" 的係有先後次序之別

 * 不考慮 Folder 的情況(只考慮 file) !!

mkdir src dst; touch src/{1..3}.sh

# 什麼都沒有 rsync 到

rsync -avn --exclude='*' src/ dst/

# 什麼都沒有 rsync 到

rsync -avn --exclude='*' --include='*.sh' src/ dst/

# 有要的檔案了 (*.sh)

rsync -avn --include='*.sh' --exclude='*' src/ dst/

[2] "--include=" 與 "--exclude="係考先慮 folder 然後再考慮 file

ie. 假設某 Folder 內有許多 sh 檔, 現在只想 rsync 此其中一個 sh 檔

mkdir src/folder1 dst -p; touch src/{a..d}.sh; touch src/folder1/{a..d}.sh

echo 'folder1/a.sh' > include.txt

echo '**/folder1/' > exclude.txt

rsync -avn \
    --include-from=include.txt \
    --exclude-from=exclude.txt \
    src/ des/

Result:

即使 include 先過 exclude, 但由於考慮 Folder 先, 所以整個 folder1 沒有 sync 去 dst/

解決方案

# 當 src 及 dst 同是 file 時, 就會考慮 include 與 exclude 的先後次序

echo '**/folder1/*' > exclude.txt

# 改用以下 cmd 就不會 sync folder1/a.sh 了.(exclude 後再 include)

rsync -avn \
    --exclude-from=exclude.txt \
    --include-from=include.txt \
    src/ des/

 

---

filter

 

"protect" filter

i.e.
--filter 'P my-specific-logfile.log'

# -f, --filter=RULE

The filter rules allow for flexible selection of which files
 to transfer (include) and which files to skip (exclude).

As the list of files/directories to transfer is built,
 rsync checks each name to be transferred against the list of include/exclude patterns in turn,
 and the first matching pattern is acted on

FILTER RULES

Syntax:
RULE [PATTERN_OR_FILENAME]
RULE,MODIFIERS [PATTERN_OR_FILENAME]

If you use a short-named rule, the  ’,’  separating the RULE from the MODIFIERS is optional.

exclude, -   # specifies an exclude pattern.
include, +   # specifies an include pattern.
clear, !     # clears the current include/exclude list (takes no arg)

hide, H      # specifies a pattern for hiding files from the transfer.
show, S      # files that match the pattern are not hidden.

protect, P   # specifies a pattern for protecting files from deletion.
risk, R      # files that match the pattern are not protected.

merge, .     # specifies a merge-file to read for more rules.
dir-merge, : # specifies a "per-directory" merge-file.
             # rsync will scan every directory that it traverses for the named file,
             # merging its contents when the file exists into the current list of inherited rules.
             # These per-directory rule files must be created on the sending side

When rules  are  being  read from a file, empty lines are ignored, as are comment lines that start with a "#".

--filter 對應 --include, and --exclude
merge-file syntax of the --filter 對應 --include-from / --exclude-from

 

---

Usage Example

 

[1] Release the website

Production Server:

# backup
cp -a public_html public_html_new

# allow sync
firewall-cmd --add-port=22/tcp
firewall-cmd --reload

# sync it

# use new data
mv public_html public_html_old; mv public_html_new/ public_html

Development Server:(sync it)

cd ????
rsync -rvz --exclude config.ini --delete public_html/ \
[email protected]:/home/vhosts/????/public_html_new/

 

---

Troubleshoot

 

Q1: 有些東西 rsync 唔到

error log:

rsync: opendir "/mysql" (in db) failed: Permission denied (13)
rsync: opendir "/otl_proj" (in db) failed: Permission denied (13)
rsync: opendir "/performance_schema" (in db) failed: Permission denied (13)
rsync: send_files failed to open "/aria_log.00000001" (in db): Permission denied (13)
rsync: send_files failed to open "/aria_log_control" (in db): Permission denied (13)
rsync: send_files failed to open "/ib_logfile0" (in db): Permission denied (13)
rsync: send_files failed to open "/ib_logfile1" (in db): Permission denied (13)
IO error encountered -- skipping file deletion
rsync: send_files failed to open "/ibdata1" (in db): Permission denied (13)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1518) [generator=3.0.9

那些 file 的 permission:

-rw-rw---- 1 mysql mysql     16384 Feb  4 09:58 aria_log.00000001
-rw-rw---- 1 mysql mysql        52 Feb  4 09:58 aria_log_control
-rw-rw---- 1 mysql mysql 211812352 Mar 19 12:24 ibdata1
-rw-rw---- 1 mysql mysql   5242880 Mar 19 12:24 ib_logfile0
-rw-rw---- 1 mysql mysql   5242880 Mar 11 11:35 ib_logfile1

修正的設定:

uid = root
gid = root

Q1: 第2次 rsync 到 NAS 時仍有大量 HDD R/W IO

 

---

sparse

 

--sparse           # To create new files in sparse mode

--inplace          # To update all existing files

 

---

Performance

 

compression & not using inplace is better for doing it over the internet (slow network)

--inplace: rsync writes updated data directly to a file, instead of making a copy and moving it into place.
                   (Default method of creating a new copy of the file and moving it into place when  it  is  complete)

This has several effects:

* Hard  links are not  broken.
* In-use binaries cannot be updated
* The  file’s data will be in an inconsistent state during the transfer and
   will be left that  way if the transfer is interrupted or if an update fails.

This option is useful for transferring large files with block-based changes or appended data

 

---

Create Folder structure

 

# The two -f arguments mean, respectively, "copy all directories" and then "do not copy anything else"."

rsync -a -f"+ */" -f"- *" source/ destination/

 

---

Merge two folders

 

rsync -avh src/ dest/

-u makes rsync transfer skip files which are newer in dest than in src

-v gives more output ("verbose").

-h for human readable.

-a means "archive" and copies everything recursively from source to destination

 

---

--copy-dest 與 --link-dest

 

--copy-dest

它的功能相當於在 remote copy 一份後再 rsync 到它

--link-dest

功能:

unchanged files are hard linked from DIR to the destination directory

which will cause rsync to search the list in the order specified for an exact match.

要求:

The files must be identical in all preserved attributes in order for the files to be linked together.

(e.g. permissions, ownership ...)

 * If file’s aren’t linking, double-check their attributes

Usage:

rsync -av --link-dest=prior_dir host:src_dir/ new_dir/

 * If DIR is a relative path, it is relative to the destination directory

 * Multiple --link-dest directories may be provided

應用: Push

mysrc -> mydest

On Destination (mydest)

mkdir -p /home/test/public_html

On Source (mysrc)

# 建立 1 個 size 為 512M 的 初始測試 file (test.bin)

mkdir /root/public_html

dd if=/dev/zero of=/root/public_html/test.bin bs=1M count=512

# 以 "-z" rsync 到 remote

MyRemotePath=/home/test/public_html

rsync -avz mysrc:/root/public_html $MyRemotePath

On Destination (mydest)

# Backup 它

mkdir _bak

mv public_html _bak

# 用 "--copy-dest" rsync

# 由於 dd 出來的係 "0", 所以無用 "-avz"

rsync -av --copy-dest=$PWD/_bak mysrc:/root/public_html $PWD/public_html/

[3] Checking

dstat -n -d

 

---

logs

 

[1] UNDETERMINED & "reverse lookup"

2022/04/08 12:35:08 [188482] recv UNDETERMINED [F.F.F.F] maildata (USERNAME) path/to/file

reverse lookup

Controls whether the daemon performs a reverse lookup on the client's IP address to determine its hostname,

which is used for "hosts allow" & "hosts deny" checks and the "%h" log escape.

This is enabled by default, but you may wish to disable it to save time if you know the lookup will not return a useful result,

in which case the daemon will use the name "UNDETERMINED" instead.

If this parameter is enabled globally (even by default),

rsync performs the lookup as soon as a client connects, so disabling it for a module will not avoid the lookup.

Thus, you probably want to disable it globally and then enable it for modules that need the information.

 

---

Output checksum

 

rsync --checksum --out-format="%n %C" source_directory/ destination_directory/ > hash_file.txt

A default format of “%n%L” is assumed if -v is specified

%n
the filename (short form; trailing "/" on dir)

%L
the string " -> SYMLINK", " => HARDLINK", or "" (where SYMLINK or HARDLINK is a filename)

%C
the full-file checksum

%l
the length of the file in bytes

md5sum -c /path/to/hash_file.txt

* Make sure to replace /path/to/destination_directory with the actual path to your destination directory

 

---

Other Info.

 

• Window Rysnc
• NAS Migrate Folder(Share)

 

---

Changelog

 

v3.2.3 Now

v3.2.2 (Jul 4th, 2020)

v3.2.1 (June 22th, 2020)

沒有有趣新功能

v3.2.0 (June 19th, 2020)

The default systemd config was made stricter by default.
 - ProtectHome=on (which hides content in /root and /home/USER dirs),
 - ProtectSystem=full (which makes /usr, /boot, & /etc dirs read-only), and
 - PrivateDevices=on (which hides devices)
 - override: systemctl edit rsync

optional use of openssl's MD4 & MD5 checksum algorithms
 - the addition of xxHash checksum support,
 - a negotiation heuristic that ensures that it is easier to add new checksum algorithms in the future.
 - The environment variable RSYNC_CHECKSUM_LIST can be used to customize the preference order of the negotiation,
 - or use ‑‑checksum-choice (‑‑cc) to force a choice.

addition of zstd and lz4 and a negotiation heuristic that picks the best compression option supported by both sides.
 - The environment variable RSYNC_COMPRESS_LIST can be used to customize the preference order of the negotiation,
 - or use ‑‑compress-choice (‑‑zc) to force a choice.

The ‑‑debug=OPTS command-line option is no longer auto-forwarded to the remote rsync
 - which allows for the client and server to have different levels of debug specified.

Added ‑‑open-noatime option to open files using O_NOATIME.

Added openssl & preliminary gnutls support to the rsync-ssl script, which is now installed by default.

Added the proxy protocol daemon parameter that allows your rsyncd to know the real remote IP when it is setup behind a proxy.

Added ‑‑copy-as=USER option to give some extra security to root-run rsync commands into/from untrusted directories
(such as backups and restores).

‑‑fuzzy heuristic to avoid the fuzzy directory scan until all other basis-file options are exhausted (such as ‑‑link-dest)

Added support for RSYNC_SHELL & RSYNC_NO_XFER_EXEC environment variables

The daemon now locks its pid file (when configured to use one) so that it will not fail to start when the file exists but no daemon is running.

rsync 3.1.3 (28 Jan 2018)

- Added the ability for rsync to compare nanosecond times in its file-check comparisons (-@-1)
- Added a short-option "-@" for "--modify-window"
- Added hashing of xattr names (with using -X) to improve the handling of files with large numbers of xattrs
- Added "syslog tag" to the daemon configuration
- Added "daemon chroot|uid|gid" to the daemon config
- Added the --checksum-choice=NAME

rsync 3.1.2 (21 Dec 2015)

- Use usleep() for our msleep() function if it is available.

3.1.1 (22 Jun 2014)

- Added support for a new-compression idiom that does not compress all the matching data in a transfer (--new-compress)

rsync 3.1.0 (28 Sep 2013)

- Added the --usermap/--groupmap/--chown options for manipulating file ownership during the copy.
- Added the "reverse lookup" parameter to the rsync daemon config file to  allow reverse-DNS lookups to be disabled.
- Added the --preallocate command-line option.
- Allow --password-file=- to read the password from stdin
- Improved the speed of some --inplace updates when there are lots of identical checksum blocks that end up being unusable.
- The --chmod option now supports numeric modes, e.g. --chmod=644,D755

rsync 3.0.9 (23 Sep 2011)

...

rsync 3.0.3 (29 Jun 2008) # Centos7