imapsync

由 datahunter 在二, 29/07/2014 - 15:20 發表

最後更新: 2022-04-12

imapsync 介紹

Program: Perl

Size: 590K (V1.945)

Memory Usage: 500MB

Link: https://github.com/imapsync/imapsync

* 52 different IMAP server softwares supported with success

* not transferring a given message if it is already on both sides (incremental and recursive)

* All flags are preserved

* stop the transfer at any time and restart

* delete the messages from the source mailbox after a successful transfer
(-delete 包含 --expunge, 安全起見, 加 --noexpunge)

* identification headers are "Message-Id:" and "Received:" lines ( --useheader option )

* Memory consumption ~ 400 MiB

Ctrl+C

once makes imapsync reconnect to both imap servers

twice within two seconds => terminal

Install

# Centos6

wget http://dl.fedoraproject.org/pub/epel/6/i386/epel-release-6-8.noarch.rpm

rpm -Uvh epel-release-6-8.noarch.rpm

yum install imapsync # epel 有 1.684

# U12

apt-get install \
libauthen-ntlm-perl    \
libcrypt-ssleay-perl   \
libdigest-hmac-perl    \
libfile-copy-recursive-perl \
libio-compress-perl     \
libio-socket-inet6-perl \
libio-socket-ssl-perl   \
libio-tee-perl          \
libmodule-scandeps-perl \
libnet-ssleay-perl      \
libpar-packer-perl      \
libreadonly-perl        \
libterm-readkey-perl    \
libtest-pod-perl        \
libtest-simple-perl     \
libunicode-string-perl  \
liburi-perl             \
make                    \
cpanminus

cpanm Data::Uniqid Mail::IMAPClient

# Install Last (On Centos 7)

# imapsync-2.178

yum install imapsync; yum remove imapsync # 為了安裝一大堆依賴包

其他包的依賴

yum install perl-App-cpanminus \
 perl-Dist-CheckConflicts \
 perl-HTML-Parser \
 perl-libwww-perl \
 perl-Module-Implementation \
 perl-Module-ScanDeps \
 perl-Package-Stash \
 perl-Package-Stash-XS \
 perl-PAR-Packer \
 perl-Regexp-Common \
 perl-Sys-MemInfo \
 perl-Test-Fatal \
 perl-Test-Mock-Guard \
 perl-Test-Requires \
 perl-Test-Deep \
 perl-File-Tail \
 perl-Unicode-String \
 perl-Test-NoWarnings \
 perl-Test-Simple \
 perl-Test-Warn \
 perl-Sub-Uplevel \
 ca-certificates

# 2

cpanm Encode::IMAPUTF7

cd /usr/src/

wget https://github.com/imapsync/imapsync/archive/refs/tags/imapsync-2.178.zip

unzip imapsync-2.178.zip

cp imapsync-imapsync-1.945/imapsync /usr/bin/

Check version

imapsync --version

2.178

# Testing

--tests # Run local non-regression tests. Exit code 0 means all ok

( 須要 perl-Test-MockObject perl-CGI perl-HTML-Parser perl-libwww-perl )

( 由於 Centos 6 太舊, 所以用唔到)

# Leaving  tests_infos()
1..1860
# Leaving  tests()
Summary of tests: failed 2 tests, run 1862 tests, expected to run 1860 tests.
List of failed tests:
nb 1606 - mailimapclient_connect ipv6 + ssl: connect to petiteipv6.lamiral.info
nb 1607 - mailimapclient_connect ipv6 + ssl: logout in ssl is ok on petiteipv6.lamiral.info
# Looks like you planned 1860 tests but ran 1862.
# Looks like you failed 2 tests of 1862 run.

--testslive # Run a live test with test1.lamiral.info imap server. (Needs internet connection)

...
The sync is not strict, there are 726 messages in host2 that are not on host1. Use --delete2 to delete them and have a strict sync.
Detected 0 errors

Check if a new imapsync release is available by adding --releasecheck
Homepage: https://imapsync.lamiral.info/
...

Option 介紹

Help

imapsync --help

Testing First:

--justconnect

--justlogin

--justfolders # Do only things about folders (ignore messages).

--justfoldersizes # Exit after having printed the folder sizes.

--dry

* 經過以上 4 個 test 後, 就可以 migrate 了

# Delete

--delete : Deletes messages on host1 server after a successful transfer.

Option --delete has the following behavior:

it marks messages as deleted with the IMAP flag "\Deleted",

then messages are really deleted with an EXPUNGE IMAP command.

--delete2 : Delete messages in host2 that are not in host1 server. Useful for backup or pre-sync.

--delete2folders : Delete folders in host2 that are not in host1 server.

--delete2duplicates : Delete messages in host2 that are duplicates.
( Works only without --useuid since duplicates are detected with header part of each message. )

# Expunge

--noexpunge            : Do not expunge messages on host1. (Expunge really deletes messages marked deleted.)
                                Expunge is made at the beginning, on host1 only.
                            Newly transferred messages are also expunged if option --delete is given.

--expunge1 : Expunge messages on host1 after messages transfer.

--expunge2 : Expunge messages on host2 after messages transfer.
(這 opt 很有用, 不用它, 在 host2 會有好多 Delete 了的信)

# Limit

--maxsize int : Skip messages larger (or equal) than int bytes

# 在同時行幾個 imapsync 時很有用

--tmpdir str # Where to store temporary files and subdirectories.

# Will be created if it doesn't exist. Default is system specific, Unix is /tmp but

# it's often small and deleted at reboot.

# --tmpdir /var/tmp should be better (pidfile 及 cache folder 都會在呢個目錄裡面)

--pidfilelocking # Abort if pidfile already exists. Usefull to avoid concurrent transfers on the same mailbox.

SECURITY

You can use --passfile1 instead of --password1 to give the password since it is safer.

Example

imapsync --host1 imap.src.fr --user1 buddy --password1 secret1 \
--host2 imap.dest.fr --user2 max --password2 secret2

Batch

The file.txt file contains:

user001_1;password001_1;user001_2;password001_2
user002_1;password002_1;user002_2;password002_2

script.sh

{ while IFS=';' read u1 p1 u2 p2; do
    imapsync --host1 imap.side1.org --user1 "$u1" --password1 "$p1" \
             --host2 imap.side2.org --user2 "$u2" --password2 "$p2" ...
done ; } < file.txt

Other Opts

# Window login

--domain1 <string> : Domain on host1 (NTLM authentication).
--domain2 <string> : Domain on host2 (NTLM authentication).

# SSL/TLS

--ssl1                 : Use an SSL connection on host1.
--ssl2                 : Use an SSL connection on host2.
--tls1                 : Use an TLS connection on host1.
--tls2                 : Use an TLS connection on host2.

# Exclude

# --exclude 'fold1|fold2|f3'

--exclude <regex> : Skips folders matching this regular expression

* in case both --include --exclude options are use, include is done before.

# 設定要 sync 什麼 Folder

--folder <string> : Sync this folder.

--folder <string> : and this one, etc.

--folderrec <string> : Sync this folder recursively.

--folderrec <string> : and this one, etc.

--subfolder2 str : Syncs the host1 folders hierarchy under str to the root hierarchy of host2

# prefix

--prefix1     <string> : Remove prefix to all destination folders
                         (usually INBOX. or INBOX/ or an empty string "")
                         you have to use --prefix1 if host1 imap server
                         does not have NAMESPACE capability, all other
                         cases are bad.
--prefix2     <string> : Add prefix to all host2 folders. See --prefix1

log

--logdir /root/imapsync

--logfile log.txt

--nolog

--nofoldersizes

++++ Calculating sizes of 1 folders on Host1
Host1 folder     1/1 [INBOX]                       Size:    0 Messages: 0 Biggest:         0
Host1 Nb folders:                1 folders
Host1 Nb messages:               0 messages
Host1 Total size:                0 bytes (0.000 KiB)
Host1 Biggest message:           0 bytes (0.000 KiB)
Host1 Time spent:              3.3 seconds
++++ Calculating sizes of 1 folders on Host2
Host2 folder     1/1 [Import-liseng]               Size: 1806 Messages: 1 Biggest:      1806
Host2 Nb folders:                1 folders
Host2 Nb messages:               1 messages
Host2 Total size:             1806 bytes (1.764 KiB)
Host2 Biggest message:        1806 bytes (1.764 KiB)
Host2 Time spent:              1.6 seconds

# subscribe

--subscribed : transfers subscribed folders.

--subscribe : subscribe to the folders transferred on the host2 that are subscribed on host1.

--subscribe_all : subscribe to the folders transferred on the host2 even if they are not subscribed on host1.

Other

--usecache : Use cache to speedup.

--debug : Debug mode.

--maxage <int> : Skip messages older than <int> days.

# Selects only messages returned by this IMAP SEARCH command.

--search1 <string> : Same as --search for selecting host1 messages only.

--no-modules_version

Cache

--useuid

--useuid sets --usecache because --useuid does not use headers so a cache is needed to avoid duplicates.

--usecache

--usecache alone use headers to sync messages, store the uids in the cache,

and uses the cache for already transfered messages, fetching headers takes time.

ACL

--syncacls # Synchronises acls (Access Control Lists).

--nosyncacls # Does not synchronize acls. This is the default.

# Acls in IMAP are not standardized, be careful.

Error handler

--errorsmax int # Exit when int number of errors is reached. Default is 50.

Other Info.

# 一共只有 1 個 mail 在 Folder 中, 之前已 sync 了一次
# 沒有 send 東西

Messages transferred              : 1
Messages skipped                  : 1

# 當由 mail 由 1 move 去 2

Messages transferred              : 2
Messages skipped                  : 1

Console Output

++++ Statistics
Transfer started on               : Wed Dec  9 16:38:31 2015
Transfer ended on                 : Wed Dec  9 16:38:41 2015
Transfer time                     : 10.2 sec
Folders synced                    : 10/10 synced
Messages transferred              : 34
Messages skipped                  : 0
Messages found duplicate on host1 : 0
Messages found duplicate on host2 : 0
Messages void (noheader) on host1 : 0
Messages void (noheader) on host2 : 0
Messages deleted on host1         : 0
Messages deleted on host2         : 0
Total bytes transferred           : 15719 (15.351 KiB)
Total bytes duplicate host1       : 0 (0.000 KiB)
Total bytes duplicate host2       : 0 (0.000 KiB)
Total bytes skipped               : 0 (0.000 KiB)
Total bytes error                 : 0 (0.000 KiB)
Message rate                      : 3.3 messages/s
Average bandwidth rate            : 1.5 KiB/s
Reconnections to host1            : 0
Reconnections to host2            : 0
Memory consumption                : 101.8 MiB
Biggest message                   : 1996 bytes
Start difference host2 - host1    : -34 messages, -15719 bytes (-15.351 KiB)
Final difference host2 - host1    : 0 messages, 0 bytes (0.000 KiB)

EXIT STATUS

imapsync will exit with a 0 status (return code) if everything went good.

Otherwise, it exits with a non-zero status.

So if you have an unreliable internet connection, you can use this loop in a Bourne shell:

while ! imapsync ...; do
    echo imapsync not complete
done

一些重要設定

--syncinternaldates : Sets the internal dates on host2 same as host1. (Turned on by default)

Internal date is the date a message arrived on a host (mtime).

Map Folder

--regextrans2 reg

Apply the whole regex to each destination folders.

first add also the safe options --dry --justfolders

Sync Scripts

# Sync one mailbox to O365

#!/bin/bash
# Version: 2.178

_USER1=""
_USER2=$_USER1
_PW1=""
_PW2=""
_SRC="localhost"
_DST="outlook.office365.com"
_TMP="/home/imapsync/${_USER2}"

# For check run time
echo `date`

# Create tmp folder
mkdir $_TMP 2> /dev/null

echo "log file: $_TMP/${now}.log"

imapsync --host1 $_SRC --user1 $_USER1 --password1 $_PW1 --notls1 --nossl1\
         --host2 $_DST --user2 $_USER2 --password2 $_PW2 --office2 \
         --logdir $_TMP --logfile ${now}.log \
         --tmpdir $_TMP --usecache --addheader \
         --pidfilelocking --noreleasecheck --no-modulesversion \
         --noexpunge1 --subscribe_all --justfolders --dry &> /dev/null

echo `date`

P.S.

--subscribeall

Subscribe to the folders transferred on the host2 even if they are not subscribed on host1.

--addheader

When a message has no headers to be identified

The default logfile name is for example

LOG_imapsync/2019_12_22_23_57_59_532_user1_user2.txt

log

msg FROM {92529} copied to TO 0.50 msgs/s 365.669 KiB/s 181.132 MiB copied ETA: Tuesday 12 April 2022-04-12 20:31:38 +0800 HKT  9728 s  4852/5105 msgs left

# Folder

#!/bin/bash

_USER=""
_PW=""
_SRC=""
_DST=""
_LOG=${_USER}.log

echo `date`

mkdir /tmp/$_USER 2> /dev/null

rm -f log_err.log log.txt

imapsync --ssl1 --host1 $_SRC --user1 $_USER --password1 $_PW \
        --host2 $_DST --user2 $_USER --password2 $_PW \

        # 設定只 sync 某些 Folder
        --folderrec "INBOX" \
        --folderrec "Sent Items" \
        --folderrec "Drafts" \

        # Sync 了過去後, 放到 Old_Email Folder 內
        --regextrans2 's/^INBOX(.*)/Old_Email.INBOX$1/' \
        --regextrans2 's/^Sent Items(.*)/Old_Email.Sent Items$1/' \
        --regextrans2 's/^Drafts(.*)/Old_Email.Drafts$1/' \

        --no-modules_version --nolog \
        --tmpdir /tmp \
        --pidfilelocking \
        --dry --justfolders \
        --noexpunge --subscribe_all --usecache \
        --noreleasecheck &> $_LOG
echo `date`

Eport Mail By IMAP

# Run On Server By Cron Job

export_to_ymail.sh

#!/bin/bash

# Source
SHost="localhost"
SPort="143"
SUser="postmaster@MyDomain"
SPWF="pw1.txt"
# Destination
DHost="imap.mail.yahoo.com"
DPort="993"
DUser="[email protected]"
DPWF="pw2.txt"
DFolder="MyDomain"

######## Code ########
echo `date`
rm -f log.txt
imapsync --nossl1 --ssl2 --notls1 --notls2 \
        --justlogin --dry --justfolders \
        --tmpdir /tmp --pidfilelocking \
        --expunge1 --delete \
        --usecache --nofoldersizes\
        --no-modulesversion --nolog --noreleasecheck \
        --host1 $SHost --port1 $SPort --user1 $SUser --passfile1 $SPWF \
        --host2 $DHost --port2 $DPort --user2 $DUser --passfile2 $DPWF \
        --folderrec "INBOX"  --regextrans2 's/^INBOX$/Import-$DFolder/' \
        &> log.txt
echo `date`

# 一個 mail 也沒有時

Transfer time                           : 5.8 sec
Memory consumption at the end           : 64.5 MiB (started with 64.1 MiB)

imapsync 後看 Error

grep Err log.txt

Err 1/1: Could not create folder [&#20013;&#25991;&#28204;&#35430;] 
from [&#20013;&#25991;&#28204;&#35430;]: 70 NO Mailbox name is not valid mUTF-7

中文 Folder 問題

中文測試

&Ti1lh24simY-

namespace

Host1: guessing separator from folder listing: [/]
Host1: separator given by NAMESPACE: [/]
Host2: guessing separator from folder listing: [/]
Host2: separator given by NAMESPACE: [.]
Host1: guessing prefix from folder listing: []
Host1: prefix given by NAMESPACE: []
Host2: guessing prefix from folder listing: []
Host2: prefix given by NAMESPACE: []
Host1 separator and prefix: [/][]
Host2 separator and prefix: [.][]

exchange default: '/'

dovecot default: '.'

M$ Exchange Server

Error

Err 46/50: - msg INBOX/111048 {13385979} couldn't append  (Subject:??????) 
to folder Old_Email/INBOX:
Error sending 'APPEND Old_Email/INBOX (\Seen) "18-Aug-2016 17:55:14 +0800" {13385979}': 
3428 BAD Command Argument Error. 11

在 Exchange 的 IMAP Protocol 最大只能食 15M 的 msg. Default 10M

Duplicated messages issues

Normally and by default, imapsync doesn't generate duplicates.

So if it does generate duplicates it means a problem occurs with message identification.

It happens sometimes with IMAP servers changing the "Message-Id" header line or

one or more of the "Received:" header lines in the header part of messages.

Imapsync uses "Message-Id" header line and "Received:" header lines to identify messages on both sides.

imapsync ... --useheader "Message-Id" --useheader "Received"

--useheader    str : Use this header to compare messages on both sides.
                              Ex: Message-ID or Subject or Date.
--useheader    str    and this one, etc.

Solution:

Use option "--useuid"

imapsync doesn't use header lines to identify and compare messages in folders.

Instead of some headers, --useuid tell imapsync to use the imap UIDs given by imap servers on bith sides.

To avoid duplicates on next runs, imapsync uses a local cache where it keeps UIDs already transfered.

--useheader 'Message-ID' --useheader 'Date' --useheader 'Subject'

* "--useheader" 此方法重可以解決

Messages void (noheader) on host1

isolate messages in a "BUG" folder

imapsync ... --debug --folder BUG

Ver 2.178

Options/specific:

--gmail2 : sets --host2 to Gmail and other options. See FAQ.Gmail.txt

--office2 : sets --host2 to Office365 and other options. See FAQ.Office365.txt

--exchange2 : sets options for Exchange. See FAQ.Exchange.txt

Folder Name

'&UiqWZHaEkPVO9g-' 刪除的郵件

isync 介紹

Homepage: http://isync.sourceforge.net/

While isync is the project name, mbsync is the current executable name;

* Synchronization is based on UIDs (unique message identifiers)

License: GPL v2

* 在 subfolder 上出現問題, 不太好用

瀏覽次數： 4825

夢想家