In this article I will describe various ways to identify and manage duplicates on a Mac. Most of the methods described could be extended to any Unix based system.
Preparatory Steps
Install or upgrade fdupes using homebrew
Here are a few example commands:
bash-3.2$ brew update
==> Updating Homebrew...
Already up-to-date.
bash-3.2$ brew install fdupes
Warning: fdupes 2.3.2 is already installed and up-to-date.
To reinstall 2.3.2, run:
brew reinstall fdupes
bash-3.2$ brew reinstall fdupes
==> Downloading https://ghcr.io/v2/homebrew/core/fdupes/manifests/2.3.2
Already downloaded: /Users/radu.parvu/Library/Caches/Homebrew/downloads/aad1302152b3a2221184fdc5302b1cdf5d242c408791eb3f907f6bd5e0292c96--fdupes-2.3.2.bottle_manifest.json
==> Downloading https://ghcr.io/v2/homebrew/core/fdupes/manifests/2.3.2
########################################################################################### 100.0%
==> Fetching fdupes
==> Downloading https://ghcr.io/v2/homebrew/core/fdupes/blobs/sha256:4b36f431443bb485436062a901736
########################################################################################### 100.0%
==> Reinstalling fdupes
==> Pouring fdupes--2.3.2.arm64_sequoia.bottle.tar.gz
🍺 /opt/homebrew/Cellar/fdupes/2.3.2: 8 files, 180.3KB
==> Running `brew cleanup fdupes`...
Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).
==> `brew cleanup` has not been run in the last 30 days, running now...
Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).
Removing: /Users/radu.parvu/Library/Caches/Homebrew/gettext_bottle_manifest--0.22.5... (10KB)
Removing: /Users/radu.parvu/Library/Caches/Homebrew/gmp_bottle_manifest--6.3.0... (7.6KB)
Removing: /Users/radu.parvu/Library/Caches/Homebrew/gmp--6.3.0... (1014.4KB)
Removing: /Users/radu.parvu/Library/Caches/Homebrew/gnutls_bottle_manifest--3.8.4... (17.9KB)
Removing: /Users/radu.parvu/Library/Caches/Homebrew/gnutls--3.8.4... (3.0MB)
Removing: /Users/radu.parvu/Library/Caches/Homebrew/jansson_bottle_manifest--2.14... (10.8KB)
Removing: /Users/radu.parvu/Library/Caches/Homebrew/jansson--2.14... (63.0KB)
Removing: /Users/radu.parvu/Library/Caches/Homebrew/libevent_bottle_manifest--2.1.12_1... (12.1KB)
Removing: /Users/radu.parvu/Library/Caches/Homebrew/libevent--2.1.12_1... (697.7KB)
Removing: /Users/radu.parvu/Library/Caches/Homebrew/libidn2_bottle_manifest--2.3.7... (9KB)
Removing: /Users/radu.parvu/Library/Caches/Homebrew/libidn2--2.3.7... (329KB)
Removing: /Users/radu.parvu/Library/Caches/Homebrew/libtasn1_bottle_manifest--4.19.0... (9.9KB)
Removing: /Users/radu.parvu/Library/Caches/Homebrew/libtasn1--4.19.0... (167.9KB)
Removing: /Users/radu.parvu/Library/Caches/Homebrew/md5sha1sum_bottle_manifest--0.9.5_1-1... (11.3KB)
Removing: /Users/radu.parvu/Library/Caches/Homebrew/md5sha1sum--0.9.5_1... (14.2KB)
Removing: /Users/radu.parvu/Library/Caches/Homebrew/nettle_bottle_manifest--3.10... (8.4KB)
Removing: /Users/radu.parvu/Library/Caches/Homebrew/nettle--3.10... (970.0KB)
Removing: /Users/radu.parvu/Library/Caches/Homebrew/p11-kit_bottle_manifest--0.25.5-1... (10.0KB)
Removing: /Users/radu.parvu/Library/Caches/Homebrew/p11-kit--0.25.5... (874.3KB)
Removing: /Users/radu.parvu/Library/Caches/Homebrew/pcre2_bottle_manifest--10.44... (8.8KB)
Removing: /Users/radu.parvu/Library/Caches/Homebrew/pcre2--10.44... (2MB)
Removing: /Users/radu.parvu/Library/Caches/Homebrew/portable-ruby-3.3.4_1.arm64_big_sur.bottle.tar.gz... (11.1MB)
Removing: /Users/radu.parvu/Library/Caches/Homebrew/colima--0.8.0... (2.2MB)
Removing: /Users/radu.parvu/Library/Caches/Homebrew/docker-completion--27.4.0... (69.3KB)
Removing: /Users/radu.parvu/Library/Caches/Homebrew/lima--1.0.2... (58.3MB)
Removing: /Users/radu.parvu/Library/Caches/Homebrew/docker--27.4.0... (8.5MB)
bash-3.2$Create test case
Create test case for fdupes
Create test directories
bash-3.2$ mkdir test_dir2 test_dir1Populate each directory with test files and data
bash-3.2$ touch test_dir1/file1 test_dir2/file2 test_dir1/file1a test_dir2/file2a
bash-3.2$ echo "a">>test_dir1/file1a
bash-3.2$ echo "a">>test_dir2/file2a
bash-3.2$ echo "1">>test_dir1/file1
bash-3.2$ echo "2">>test_dir2/file2
bash-3.2$ more test_dir1/file1aList Duplicates
Use fdupes to list duplicates
Example: run the following command to find duplicate files between dir1 (originals) and dir2 (potential duplicates):
fdupes -r dir1 dir2Actual example:
bash-3.2$ fdupes -r '/Users/radu.parvu/iCloud Drive (Archive)/books' '/Users/radu.parvu/iCloud Drive (Archive) - 1/books'
...
/Users/radu.parvu/iCloud Drive (Archive)/books/Carti tehnice si utile (din viata de zi cu zi)/Cum-se-monteaza-dalele-flexibile.pdf
/Users/radu.parvu/iCloud Drive (Archive)/books/Carti tehnice si utile (din viata de zi cu zi)/Montarea.Dalelor.Flexibile.Pentru.Acoperis.pdf
/Users/radu.parvu/iCloud Drive (Archive)/books/122.Carti.Tehnice.Si.Utile.In.Romana-sFZ/Cum-se-monteaza-dalele-flexibile.pdf
/Users/radu.parvu/iCloud Drive (Archive)/books/122.Carti.Tehnice.Si.Utile.In.Romana-sFZ/Montarea.Dalelor.Flexibile.Pentru.Acoperis.pdfs
...
bash-3.2$
bash-3.2$ fdupes -r test_dir1/ test_dir2/
test_dir1/file1a
test_dir2/file2a
...Delete duplicates
Delete with prompt:
bash-3.2$ fdupes -rStd test_dir1/ test_dir2/Delete without prompt by adding the N switch:
bash-3.2$ fdupes -rStdN test_dir1/ test_dir2/
[+] test_dir1/file1a
[-] test_dir2/remNow I should be ready to start cleaning duplicates:
bash-3.2$ fdupes -rStdN '/Users/radu.parvu/iCloud Drive (Archive)/books' '/Users/radu.parvu/iCloud Drive (Archive) - 1/books'