Duplicati

由 datahunter 在三, 23/11/2022 - 10:50 發表

最後更新: 2022-11-24

介紹

HomePage: https://www.duplicati.com/
Code: https://github.com/duplicati/duplicati

License: LGPL

這裡講的是 Duplicati 2, 不是舊版的 Duplicati 1

功能

* Type: Incremental backups
* Storage format: Block-based
* UI: Web or CLI
* Scheduler: Built-in scheduler (不用依賴 Task Scheduler 及 cron )
* No server-side components => support a wide variety of cloud-based storage
* Many Backends (LocalDisk, FTP, SSH ... S3, Dropbox, Google Cloud Storage, OneDrive)
* OS: Windows(.NET 4.7.1, vs2015 run-time), Linux(Mono)

* Compression
    All backup data is compressed before it is encrypted and uploaded.
    Duplicati detects files that are compressed already and
    adds those as they are to the Zip or 7z archives. (mp3, jpeg or mkv)

* Strong encryption(AES-256)

* Deduplication(block-based)

* Online backup verification

Components

Tray Icon (8200/tcp)
Server component
Command Line tools
Service

Tray Icon(Duplicati.GUI.TrayIcon.exe)

The server component is included in the Tray Icon tool
(It perform tasks in the background - Scheduler)

Server component(Duplicati.Server.exe)

The server component is completely included in Duplicati.GUI.TrayIcon.exe
(unless disabled with the --no-hosted-server command line option.)

Command Line tools(Duplicati.CommandLine.exe)

Duplicati can make backups without loading the server component
(用 Windows Task Scheduler (Windows) or Cron (Linux) 去行 backup)

Service(Duplicati.WindowsService.exe)

A small agent starts the server component and pings the server to verify it is running

Backend

The requirements for these backends are very low.

requirements: PUT, GET, LIST, DELETE, CREATEFOLDER

* Duplicati doesn't need to rename files or add data to existing files.

Usage statistics

https://usage-reporter.duplicati.com/

zip 安裝包減磅

刪除它們

Xamarin.Mac.dll # 25M, MacOS 內的
libstorj_uplink.dylib # 15M, storj storage
libstorj_uplink.so # 15M
win-x64 # 8M
win-x86 # 8M
Microsoft.WindowsAzure.Storage.dll #1.1M
runtimes/ # 1.5M, MacOS 內的

40.6M -> 11M

storj

https://www.storj.io/pricing

Storage cost per month $4/TB
(Additional per-segment fee of $0.0000088 applies)
Download bandwidth cost $7/TB

To register Duplicati Server as a Windows service

Services are started by default with the Local System account(SYSTEM)

Basic Setup

A Configuring the Duplicati Tray Icon in Windows

1) C:\ProgramData\Microsoft\Windows\Start Menu\Programs\StartUp
2) properties of the Duplicati 2 shortcut
3) "...Duplicati.GUI.TrayIcon.exe" --no-hosted-server

B Install service

4) administrator run cmd.exe
5) cd C:\Program Files\Duplicati 2
6) 安裝 service 並更改 DB 位置
7) sc start Duplicati

安裝 service 並更改 DB 位置

* 因為原本的位 location 在 upgrading Windows 時會被刪 !!

mkdir c:\Duplicati_DB\log

Duplicati.WindowsService.exe install ^
--server-datafolder=c:\Duplicati_DB --ping-pong-keepalive ^
--log-file=c:\Duplicati_DB\log\log.txt --log-level=Information --log-retention=1D

Log

--log-file=FILE # Log internal information. Append to current log file
--log-file-log-level=Warning
--console-log-level=Warning

--ping-pong-keepalive

When running as a server, the service daemon must verify that the process is responding.
If this option is enabled, the server reads stdin and writes a reply to each line read.

使用 Service 後的 TrayIcon

# 在無 pw 情況下先開到

Duplicati.GUI.TrayIcon.exe --no-hosted-server ^
--hosturl=http://localhost:8200

# 在有 pw 情況, 只可在 DB 讀 PW

Duplicati.GUI.TrayIcon.exe --no-hosted-server ^
--read-config-from-db --server-datafolder=C:\Duplicati_DB

P.S.

TrayIcon 執行方式的 DB: C:\Users\tim\AppData\Local\Duplicati
Service 方式: C:\Windows\System32\config\systemprofile\AppData\Local\Duplicati

CLI - Duplicati.CommandLine.exe

* You can perform all operations that are available in the Graphical User Interface.

=> CLI 不是必須的 ~

Syntax

Duplicati.CommandLine.exe <command> [Storage-URL] [Arguments] [Advanced-options]

Duplicati.CommandLine.exe example # 查看 Usage Example

Duplicati.CommandLine.exe help backup # Help

The Commandline tool supports these commands:

主要: backup, find, restore, test, compact
次要: delete, compare, vacuum,
小用: repair, affected, purge, list-broken-files, purge-broken-files

Debug

... --console-log-level=profiling --console-log-filter=-*.Database.*

加密

--passphrase 是用來設定 AES 加密的 Key (--encryption-module=aes)

當 "--no-encryption=true" 時就不用設定 "--passphrase" 了 (在第一次 backup 時要決定)

建議在 local disk 才用 "--no-encryption=true"

加壓

--compression-module=zip # 每次 Backup 可以用不同方式去 zip
--concurrency-compressors=2
--compression-extension-file=C:\Program Files\Duplicati 2\default_compressed_extensions.txt

Command Usage

使用 CMD 對設定

Set DuplicatiRoot=C:\Duplicati
Set DuplicatiDB=%DuplicatiRoot%\DB\%Name%.sqlite
Set DuplicatiPath=%DuplicatiRoot%\duplicati-2.0.6.105_canary_2023-04-09
Set PATH=%DuplicatiPath%;%PATH%

The BACKUP command

建議在 GUI 生成 CLI 用的 command, 因為在 CLI 上要列出所有 Advanced Options

* 當所有 file 被 backup 時沒有更新, 不會有新 version 的 backup !!

start_backup.bat

@echo off
REM tim@20230420
cls

Set Source="C:\tmp\p1" "C:\tmp\p2" "C:\tmp\p3 p4"
Set Destination="file://D:\tmp\DuplicatiBackup"
Set Name=DB_Backup
Set DuplicatiPath=%DuplicatiRoot%\duplicati-2.0.6.105_canary_2023-04-09

REM Program Settings
Set DuplicatiRoot=C:\Duplicati
Set DuplicatiDB=%DuplicatiRoot%\DB\%Name%.sqlite
Set DuplicatiLog=%DuplicatiRoot%\%Name%.log
SET PATH=%DuplicatiPath%;%PATH%

Del %DuplicatiLog% 2>NUL

REM Main CMD
Duplicati.CommandLine.exe backup %Destination% %Source%^
  --dbpath=%DuplicatiDB% ^
  --parameters-file=%DuplicatiRoot%\config.txt ^
  --backup-name=%Name% --log-file=%DuplicatiLog% --dry-run

config.txt

--retention-policy=1D:0s,2W:1D
--no-encryption=true
--usn-policy=on
--snapshot-policy=required

--backup-name="XXX"

A display name that is attached to this backup.
Can be used to identify the backup when sending mail or running scripts.

The FIND command

Usage: find <storage-URL> ["<filename>"] [<options>]

filename

沒有 filename 時則列出 version
可以用 wildcards (* and ?)
File names in [brackets] are interpreted as regular expression.

version

* Latest backup is searched by default.

想 search 所有 backup version 就要加 "--all-versions=true"

--version=<int> # 在某一 version 內 search

include & exclude

* This is applied before the search is executed.

--include=<string>

--exclude=<string>

i.e. 查看有什麼 version (*.dlist.zip)

Duplicati.CommandLine.exe find %Destination% ^
  --dbpath=%DuplicatiDB% --no-encryption=true

Out

  Listing remote folder ...
  Downloading file (7.33 KB) ...
  Downloading file (7.33 KB) ...
  Downloading file (7.33 KB) ...
  Downloading file (7.24 KB) ...
Listing filesets:
0       : 23/11/2022 5:58:17 pm
1       : 23/11/2022 5:44:28 pm
2       : 23/11/2022 5:42:52 pm
3       : 23/11/2022 1:20:53 pm

The RESTORE command

Duplicati.CommandLine.exe restore <storage-URL> ["<filename>"] [<arguments>] [advanced-options]

[1] 還原當後 version 的 "c:\MyTools\test.txt"

Duplicati.CommandLine.exe restore "file://E:\PC_Backup" ^
  --no-encryption=true --dbpath="c:\Duplicati_DB\ID.sqlite" ^
  "c:\mytools\test.txt"

* 在沒有 overwrite 的情況會建立 "test.2022-11-24.txt"

* 只還原有更新過的 file

* 要 restore 的 filename 沒有分大細階

[2] 還原另一個版本到另一個地方

Duplicati.CommandLine.exe restore "file://E:\PC_Backup" ^
    --dbpath="c:\Duplicati_DB\ID.sqlite --no-encryption=true ^
    --restore-path="D:\test_restore" --version=1

[3] 不使用 local-block 及 local-db 去 restore

Duplicati.CommandLine.exe restore "file://E:\PC_Backup" ^
    --no-encryption=true --no-local-blocks=true --no-local-db=true ^
    --restore-path="D:\test_restore"

The TEST command

Usage: test <storage-URL> <samples> [<options>]

A random sample of dlist, dindex, dblock files is downloaded, verifying the file hash.

<samples> # If "all" is specified, all files in the backup will be tested.

--full-remote-verification # Checks the internal structure of each file (會解密合並成檔案檢查 hash)

--version=<int> # Checks samples from specific versions <- 唔 work

i.e. 限速測試所有 file

Duplicati.CommandLine.exe test %Destination% all ^
  --dbpath=%DuplicatiDB% --no-encryption=true ^
  --throttle-download=4mb

P.S.

--console-log-level=ProfilingProfiling

The DELETE command

刪除 version 2 的 backup

Duplicati.CommandLine.exe delete %Destination% ^
  --dbpath=%DuplicatiDB% --no-encryption=true ^
  --version=2

Output

  Listing remote folder ...
  Deleting file duplicati-DATE.dlist.zip (7.33 KB) ...
These filesets were deleted:
2: 23/11/2022 5:58:17 pm

多個 version 時可以用上 "," 及 "-"

i.e.

--version=2-9

The REPAIR command

If no local database is found or the database is empty, the database is re-created with data from the storage.

If the database is in place but the remote storage is corrupt, the remote storage gets repaired with local data (if available)

Advanced Settings

簡化 parameters

--parameters-file

Each line in the file should be of the format --option=value
The special options --source and --target can be used to override the localpath and the remote destination uri, respectively.
The options in this file take precedence over the options provided on the commandline.

重要設定

--dry-run=false

--snapshot-policy=off

If this is set to off, Duplicati will not attempt to create a disk snapshot. (Default)

Setting this to auto makes Duplicati attempt to create a snapshot,
and fail silently if that was not allowed or supported.

"required" will make Duplicati abort the backup if the snapshot creation fails.

On Windows this uses the Volume Shadow Copy Services (VSS) and requires administrative privileges.
On Linux this uses Logical Volume Management (LVM) and requires root privileges.

--usn-policy=off

This setting controls the usage of NTFS USN numbers

which allows Duplicati to obtain a list of files and folders much faster

required will make Duplicati abort the backup if the USN usage fails.

--dblock-size=50mb

Changing the size can be useful if the backend has a limit on the size of each individual file.

--quota-size=Xmb

This value can be used to set a known upper limit on the amount of space a backend has.

* If the backend reports the size itself, this value is ignored.

--exclude=STRING

* means any number of character, and the special character
? means any single character,
[.*\.txt] regex(hard braces)

verification

--upload-verification-file=false

Upload a verification file(duplicati-verification.json) after changing the remote storage.
contains the size and SHA256 hashes of all the remote files and
can be used to verify the integrity of the files.

--backup-test-percentage=0

After a backup is completed, some (dblock, dindex, dlist) files from the remote backend are selected for verification.

--full-remote-verification=false

true: download, decrypt, compare to database(hash) for individual content files

which will decrypt the files and examine the insides of each volume, instead of simply verifying the external hash

MISC.

--no-auto-compact=false

If a large number of small files are detected during a backup,
or wasted space is found after deleting backups, the remote data will be compacted.
The compacting procedure is triggered after each backup

Restore 時用到

--version=STRING

By default, Duplicati will list and restore files from the most recent backup,
use this option to select another item. (e.g. 0,2-4,7)

0 = last version

--restore-path=<string>

Restores files to instead of their original destination.

--overwrite=false

Use this option to overwrite target files when restoring,
if this option is not set the files will be restored with a timestamp and a number appended.

--full-block-verification=false

Use this option to increase verification by checking the hash of blocks read from a volume
before patching restored files with the data.

--restore-permissions=false

By default permissions are not restored as they might prevent you from accessing your files.

# 用於 verify backup 是否成功

--no-local-blocks=false

Duplicati will attempt to use data from source files to minimize the amount of downloaded data.
Use this option to skip this optimization and only use remote data.

--no-local-db=false

When listing contents or when restoring files, the local database can be skipped. (slower)

Scripting options

--run-script-before (Path)

--run-script-after (Path)

Reporting options

--send-mail-to (String)
--send-mail-from (String)
--send-mail-url (String) # smtp://example.com:25, smtp://example.com:465/?starttls=when-available
--send-mail-username (String)
--send-mail-password (String)
--send-mail-level (String) # Values: Unknown, Success, Warning, Error, Fatal, All(Default)

限速

本機的 upload / download speed

* 這裡的 unit(mb) 是 MiB (1024, 8)來.

--throttle-download=0mb # Default: 0 <- 不限

--throttle-upload=0mb # Default: 0 <- 不限

Performance

--use-background-io-priority=false

To set the current process to use the lowest IO priority level,

which can make operations run slower
but will interfere less with other operations running at the same time.

--use-block-cache=false

Store an in-memory copy of the block table.

This potentially reduces lookup time during backup
as it limits the number of times the database must be accessed when examining blocks.

--blocksize=100kb

blocksize can’t be changed once a backup is going.

=> Duplicati will abort the operation with an error if you attempt to change the chunk size on an existing backup.

* The default 100 KB is good for about 100 GB Backup (ISO, MP4 之類會令到有大量 block)

--dblock-size=50mb

The volume size can be changed after a backup has been created.

Remark

# Data: 1TB (Default: 100kb, 50mb)

file: 20972, block: 10737419

Auto-X

--auto-cleanup=false

If a backup is interrupted there will likely be partial files present on the backend.
Using this flag, Duplicati will automatically remove such files when encountered.

Duplicati.CommandLine.exe compact "file://E:\PC_Backup" ^
--dbpath="c:\Duplicati_DB\ID.sqlite" --no-encryption=true

--auto-vacuum=false

Some operations that manipulate the local database leave unused entries behind.
These entries are not deleted from a hard drive until a VACUUM operation is run.

This operation saves disk space in the long run
but needs to temporarily create a copy of all valid entries in the database.

人手 cleanup

Duplicati.CommandLine.exe vacuum "file://E:\PC_Backup" ^
--dbpath="c:\Duplicati_DB\ID.sqlite" --no-encryption=true

保留多小份 Backup

* 不設定 --keep-time, --keep-versions 及 --retention-policy 即係保留 ALL

--keep-time=7D

保留 7 天 Backup

--keep-versions=7

保留 7 份 Backup

--retention-policy

# time_frame:interval

1D:0s,2W:1D

Other

--block-hash-algorithm=SHA256

--compression-extension-file="C:\Program Files\Duplicati 2\default_compressed_extensions.txt"

--quiet-console=false

當 true 時 console 只顯示基本資料

  Listing remote folder ...
  Uploading file (17.79 KB) ...
  Listing remote folder ...
  Downloading file (1.06 KB) ...
  Downloading file (117.97 KB) ...
  Downloading file (49.91 MB) ...

進階 config.txt

--quiet-console=false
--no-encryption=true
--retention-policy=1D:0s,2W:1D
--quota-size=1000mb
--throttle-download=4mb
--throttle-upload=4mb
--upload-verification-file=true
--usn-policy=on
--snapshot-policy=required
--use-block-cache=true
--auto-cleanup=true

Cleanup Script

The vacuum command

Rebuilds the local database(sqlite), repacking it into a minimal amount of disk space.

vacuum <storage-URL>

echo Running vacuum
Duplicati.CommandLine.exe vacuum %Destination% ^
  --dbpath=%DuplicatiDB% --no-encryption=true

The compact command

針對 dblock-file

--threshold=25 # 25%

dblock file 最多才以包含幾多無用數據

針對 small-file

--small-file-max-count=20

The maximum allowed number of small files.

To avoid filling the remote storage with small files, this value can force grouping small files.

--small-file-size=20 # 20% of <dblock-size>

Files smaller than this size are considered to be small and will be compacted

i.e.

echo Running compact
Duplicati.CommandLine.exe compact %Destination% ^
  --dbpath=%DuplicatiDB% --no-encryption=true ^
  --small-file-max-count=1

Output

  Listing remote folder ...
  Downloading file (7.45 MB) ...
  Downloading file (1.13 KB) ...
  Downloading file (689 bytes) ...
  Uploading file (7.45 MB) ...
  Uploading file (7.04 KB) ...
  Deleting file duplicati-bee7b7fa7cd014166ad9a0bcb2b2b0776.dblock.zip (7.45 MB) ...
  Deleting file duplicati-i993c936558e44b1eafcd1ec18f897162.dindex.zip (7.04 KB) ...
  Deleting file duplicati-bf6e7aa32fa504af486b00d6ae4f6c049.dblock.zip (1.13 KB) ...
  Deleting file duplicati-i5cb5e9c1f2d745a7accca45dd2123b62.dindex.zip (611 bytes) ...
  Deleting file duplicati-b2cde4e1bafe9464482c70f2a4237dc0d.dblock.zip (689 bytes) ...
  Deleting file duplicati-i6c1cc64f76884f9792e7b4e97f5a7326.dindex.zip (608 bytes) ...

Storage Providers

SMB

\\server\folder # UNC format

支援用另一個 user 身份 login server

--auth-password=XXX
--auth-username=XXX
--force-smb-authentication

Any existing authentication against the remote share is dropped before attempting to authenticate.

i.e.

REM Main CMD
Duplicati.CommandLine.exe backup %Destination% %Source% ^
  --dbpath=%DuplicatiDB% ^
  --auth-username=%USER% --auth-password=%PW% ^
  --parameters-file=%DuplicatiRoot%\config.txt ^
  --backup-name=%Name% --log-file=%DuplicatiLog% --dry-run

FTP

--auth-username=XXX
--auth-password=XXX
--ftp-passive
--use-ssl

i.e.

* To protect against network failures, every upload will be attempted verified.
Disable: --disable-upload-verify

SFTP (SSH)

ssh://hostname/folder

--ssh-fingerprint "ssh-rsa 4096 ..."
--auth-username
--auth-password
--ssh-keyfile

Other *.exe

Duplicati.CommandLine.BackendTester.exe
Duplicati.Library.Snapshots.exe
...

Duplicati.CommandLine.BackendTester.exe

Usage:

protocol: file, ssh, ftp, dropbox, googledrive, gcs, azure, onedrive, s3 ...

i.e.

Duplicati.CommandLine.BackendTester.exe D:\tmp\DuplicatiBackup

Opts

--reruns=5 # 行測試 5 次
--number-of-files=10 # 每次建立 10 個 file
--min-file-size=1kb # 每個 file 界乎於 1kb~50mb 之間
--max-file-size=50mb
--min-filename-length=5 # 每個檔名長 5~80 字
--max-filename-length=80
--auto-clean=false # Removes any files found in target folder.

Duplicati.Library.Snapshots.exe

測試 Windows 的 VSS(Volume Shadowcopy Services)

* 此 cli 要用 administrator 身份去行, 它不帶 arguments

Creating file C:\Program Files\Duplicati 2\testfile.bin
Attempting to read locked file C:\Program Files\Duplicati 2\testfile.bin
The file C:\Program Files\Duplicati 2\testfile.bin was correctly locked, message:
 The process cannot access the file 'C:\Program Files\Duplicati 2\testfile.bin'
 because it is being used by another process.
Creating snapshot for folder: C:\Program Files\Duplicati 2
If this fails, try to run as Administrator
Attempting to read locked file via snapshot
Could open locked file C:\Program Files\Duplicati 2\testfile.bin, through snapshot
* Test passed

Duplicati.CommandLine.RecoveryTool.exe

This tool can be used in very specific situations, where you have to restore data from a corrupted backup.

Duplicati.CommandLine.ConfigurationImporter.exe

import backup configurations from the command-line

i.e.

ConfigurationImporter.exe C:\backup-config.json ^
--import-metadata=false ^
--server-datafolder=<absolute path to folder that contains Duplicati-server.sqlite>

Verify backup

1) 方式1

You can copy the db to the remote location, install duplicati, point the config to the db and run full verification.

samples all + --full-remote-verification

2) 方式2

DuplicatiVerify.py script + "--upload-verification-file"

不放心的話

multiple backups

Restoring files if your Duplicati installation is lost

A) import config (JSON ) # contain sensitive information

B) you need to know the backend URL, credentials and the backup passphrase

Before deleting a backup job, it is recommended to export the backup job settings to a file.

If you really want to delete the backup files also, click to select Delete remote files.

The backup process

*.dblock.zip

每個 dblock 的 size 由 "--dblock-size" 設定

dblock = many "block" (每個 block 的 size 由 "--blocksize" 設定)

many "block" => Compression, Encryption => dblock.zip

在 zip 檔內會見到用每個 block, 它們的名稱是 base64 encode sha256

另外會有一個 manifest 檔

{
    "Version": 2,
    "Created": "20230421T075225Z",
    "Encoding": "utf8",
    "Blocksize": 102400,
    "BlockHash": "SHA256",
    "FileHash": "SHA256",
    "AppVersion": "2.0.6.105"
}

*.dlist.zip

內有: filelist.json, fileset, manifest

fileset

{"IsFullBackup":true}

filelist.json

file hash + block hash => *.dlist.zip

SHA256(block), SHA256(file)

P.S.

store around 47 characters(base64 encode hash) for each 100kb(--blocksize) of block

1GB => 482kb of additional data

Indirection block

creates a new block of data with only the hashes.

*.dindex.zip

A file is created to keep track of which dblock files have each hash

瀏覽次數： 895

夢想家

Duplicati

目錄

介紹

zip 安裝包減磅

To register Duplicati Server as a Windows service

CLI - Duplicati.CommandLine.exe

Command Usage

Advanced Settings

Cleanup Script

Storage Providers

Other *.exe

Verify backup

Restoring files if your Duplicati installation is lost

The backup process