How to identify same-content files on Linux (2024)

In a recent post, we looked at how to identify and locate files that are hard links (i.e., that point to the same disk content and share inodes). In this post, we’ll check out commands for finding files that have the same content, but are not otherwise connected.

Hard links are helpful because they allow files to exist in multiple places in the file system while not taking up any additional disk space. Copies of files, on the other hand, sometimes represent a big waste of disk space and run some risk of causing some confusion if you want to make updates. In this post, we’re going to look at multiple ways to identify these files.

Comparing files with the diff command

Probably the easiest way to compare two files is to use the diff command. The output will show you the differences between the two files. The signs indicate whether the extra lines are in the first () file provided as arguments. In this example, the extra lines are in backup.html.

$ diff index.html backup.html2438a2439,2441> 
> That's all there is to report.> 

If diff shows no output, that means the two files are the same.

$ diff home.html index.html$

The only drawbacks to diff are that it can only compare two files at a time, and you have to identify the files to compare. Some commands we will look at in this post can find the duplicate files for you.

Using checksums

The cksum (checksum) command computes checksums for files. Checksums are a mathematical reduction of the contents to a lengthy number (like 2819078353 228029). While not absolutely unique, the chance that files that are not identical in content would result in the same checksum is extremely small.

$ cksum *.html2819078353 228029 backup.html4073570409 227985 home.html4073570409 227985 index.html

In the example above, you can see how the second and third files yield the same checksum and can be assumed to be identical.

Using the find command

While the find command doesn’t have an option for finding duplicate files, it can be used to search files by name or type and run the cksum command. For example:

$ find . -name "*.html" -exec cksum {} ;4073570409 227985 ./home.html2819078353 228029 ./backup.html4073570409 227985 ./index.html

Using the fslint command

The fslint command can be used to specifically find duplicate files. Note that we give it a starting location. The command can take quite some time to complete if it needs to run through a large number of files. Here’s output from a very modest search. Note how it lists the duplicate files and also looks for other issues, such as empty directories and bad IDs.

$ fslint .-----------------------------------file name lint-------------------------------Invalid utf8 names-----------------------------------file case lint----------------------------------DUPlicate fileshome.htmlindex.html-----------------------------------Dangling links--------------------redundant characters in links------------------------------------suspect links--------------------------------Empty Directories./.gnupg----------------------------------Temporary Files----------------------duplicate/conflicting Names------------------------------------------Bad ids-------------------------Non Stripped executables

You may have to install fslint on your system. You will probably have to add it to your search path, as well:

$ export PATH=$PATH:/usr/share/fslint/fslint

Using the rdfind command

The rdfind command will also look for duplicate (same content) files. The name stands for “redundant data find,” and the command is able to determine, based on file dates, which files are the originals — which is helpful if you choose to delete the duplicates, as it will remove the newer files.

$ rdfind ~Now scanning "/home/shark", found 12 files.Now have 12 files in total.Removed 1 files due to nonunique device and inode.Total size is 699498 bytes or 683 KiBRemoved 9 files due to unique sizes from list.2 files left.Now eliminating candidates based on first bytes:removed 0 files from list.2 files left.Now eliminating candidates based on last bytes:removed 0 files from list.2 files left.Now eliminating candidates based on sha1 checksum:removed 0 files from list.2 files left.It seems like you have 2 files that are not uniqueTotally, 223 KiB can be reduced.Now making results file results.txt

You can also run this command in “dryrun” (i.e., only report the changes that might otherwise be made).

$ rdfind -dryrun true ~(DRYRUN MODE) Now scanning "/home/shark", found 12 files.(DRYRUN MODE) Now have 12 files in total.(DRYRUN MODE) Removed 1 files due to nonunique device and inode.(DRYRUN MODE) Total size is 699352 bytes or 683 KiBRemoved 9 files due to unique sizes from list.2 files left.(DRYRUN MODE) Now eliminating candidates based on first bytes:removed 0 files from list.2 files left.(DRYRUN MODE) Now eliminating candidates based on last bytes:removed 0 files from list.2 files left.(DRYRUN MODE) Now eliminating candidates based on sha1 checksum:removed 0 files from list.2 files left.(DRYRUN MODE) It seems like you have 2 files that are not unique(DRYRUN MODE) Totally, 223 KiB can be reduced.(DRYRUN MODE) Now making results file results.txt

The rdfind command also provides options for things such as ignoring empty files (-ignoreempty) and following symbolic links (-followsymlinks). Check out the man page for explanations.

-ignoreemptyignore empty files-minsizeignore files smaller than speficied size-followsymlinksfollow symbolic links-removeidentinoderemove files referring to identical inode-checksumidentify checksum type to be used-deterministicdeterminess how to sort files-makesymlinksturn duplicate files into symbolic links-makehardlinksreplace duplicate files with hard links-makeresultsfilecreate a results file in the current directory-outputnameprovide name for results file-deleteduplicatesdelete/unlink duplicate files-sleepset sleep time between reading files (milliseconds)-n, -dryrundisplay what would have been done, but don't do it

Note that the rdfind command offers an option to delete duplicate files with the -deleteduplicates true setting. Hopefully the command’s modest problem with grammar won’t irritate you. 😉

$ rdfind -deleteduplicates true ....Deleted 1 files.
You will likely have to install the rdfind command on your system. It's probably a good idea to experiment with it to get comfortable with how it works.
Using the fdupes command
The fdupes command also makes it easy to identify duplicate files and provides a large number of useful options — like -r for recursion. In its simplest form, it groups duplicate files together like this:
$ fdupes ~/home/shs/UPGRADE/home/shs/mytwin/home/shs/lp.txt/home/shs/lp.man/home/shs/penguin.png/home/shs/penguin0.png/home/shs/hideme.png
Here's an example using recursion. Note that many of the duplicate files are important (users' .bashrc and .profile files) and should clearly not be deleted.
# fdupes -r /home/home/shark/home.html/home/shark/index.html/home/dory/.bashrc/home/eel/.bashrc/home/nemo/.profile/home/dory/.profile/home/shark/.profile/home/nemo/tryme/home/shs/tryme/home/shs/arrow.png/home/shs/PNGs/arrow.png/home/shs/11/files_11.zip/home/shs/ERIC/file_11.zip/home/shs/penguin0.jpg/home/shs/PNGs/penguin.jpg/home/shs/PNGs/penguin0.jpg/home/shs/Sandra_rotated.png/home/shs/PNGs/Sandra_rotated.png
The fdupe command's many options are listed below. Use the fdupes -h command, or read the man page for more details.
-r --recurse recurse-R --recurse: recurse through specified directories-s --symlinks follow symlinked directories-H --hardlinks treat hard links as duplicates-n --noempty ignore empty files-f --omitfirst omit the first file in each set of matches-A --nohidden ignore hidden files-1 --sameline list matches on a single line-S --size show size of duplicate files-m --summarize summarize duplicate files information-q --quiet hide progress indicator-d --delete prompt user for files to preserve-N --noprompt when used with --delete, preserve the first file in set-I --immediate delete duplicates as they are encountered-p --permissions don't soncider files with different owner/group or permission bits as duplicates-o --order=WORD order files according to specification-i --reverse reverse order while sorting-v --version display fdupes version-h --help displays help
The fdupes command is another one that you're like to have to install and work with for a while to become familiar with its many options.
Wrap-up
Linux systems provide a good selection of tools for locating and potentially removing duplicate files, along with options for where you want to run your search and what you want to do with duplicate files when you find them.

Please enter a valid email address