最後更新: 2024-10-22

介紹

fdupes, duff, jdupes, rdfind

• rdfind - find duplicate files utility
• jdupes - identify and delete or link duplicate files
• fdupes - identifies duplicate files within given directories
• duff - Duplicate file finder

jdupes

The files are found by comparing file sizes, then partial and full file hashes,
 followed by a byte-by-byte comparison.

jdupes -v

jdupes 1.21.3 (2023-02-09) 64-bit

Opts

• -r --recurse
• -A --no-hidden              # exclude hidden files from consideration
• -m --summarize              # summarize duplicate file information
• -u --print-unique           # print only a list of unique files
• -d --delete                 # prompt user for files to preserve
• -L --link-hard              # replace all duplicate files with hardlinks
• -l --link-soft              # replace all duplicate files with symlinks
• -B --dedupe                 # filesystem-level data deduplication

 * link to the first file in each set of duplicates

應用: 在 Openwrt 的 targets 內 kmods/ 與 packages/ 是有相同的 ipk

# V23

20,460 - targets/x86/64/kmods/5.10.221-1-bb372724df40051c0f132fa20fc778d0/kmod-zram_5.10.221-1_x86_64.ipk
20,460 - targets/x86/64/packages/kmod-zram_5.10.221-1_x86_64.ipk

1) 查看 Summary

jdupes -r -m kmods packages

Scanning: 1902 files, 3 items (in 2 specified)
887 duplicate files (in 887 sets), occupying 30 MB

2) 查看 unique 的 file

jdupes -r -u kmods packages

3) hardlink 相同的 file

# "---->" successfully hard linked to the first file
# "-@@->"  successfully symlinked to the first file
# "-//->"  Linking this file failed

jdupes -r -L kmods packages

filesystem-level data deduplication

To call same-extents ioctl or clonefile() to trigger a filesystem-level data deduplication on disk

only a few filesystems support this: btrfs

它們會有不同的 Inode

# "-##->"  This file was successfully cloned from the first file

jdupes -r -B kmods packages