tar doesn't sort entries on archive creation time. I've tried to use it to compare trees in the past, to notice this caveat. It's a detail most people don't seem to realize about their file system, but files aren't (usually, on most FSes) sorted in any particular order. By creation time order, MAYBE (no guarantee, often not), but as directories get modified and files are deleted an added, this gets clobbered anyhow. Use "ls -f" to see the raw, unsorted list of a directory's contents. Since it is faster for tar to just use this, than to pre-sort entries, that's the order they'll get added in.
What you want is a tar that is deterministically identical regardless of the underlying file system's order. Sadly, no such option exists in GNU tar, but that doesn't quite mean the end of the quest. Behold:
find bin -print0 | sort -Vz | tar -cf bin.tar --no-recursion --null -T -
The first two shouldn't be hard to parse. Find files, separated by NULL characters (so any whitespace or other special characters won't be an issue), pass that onto "sort" that uses a version sort (identical rules to dpkg version sorting, and will disregard the locale sort) and interpret entries as being between NULLs (as well as emit them, likewise). The tar command should be pretty self-explanatory, but the "-T -" tells it to read stdin for a file list, with --null specifying that the input list is separated by NULLs, and --no-recursion prevents it from recursing into directories, instead just adding an entry in the tar for directories -- the find|sort will take care of the directory members already, sorted. Optionally add --numeric-owner to tar, if you want to compare trees on different computers, but results might vary (rsync and tar normally send/store both the id and name, and if the "name" exists on the local machine, uses that instead of id).
Since I develop backup software, this is a problem I needed to solve (e.g., for verifying that restore works). So I wrote a tool: http://liw.fi/summain/
It produces output that is meant to be usefully diffable.
That's a neat looking application - bonus points for a good manpage, too.
That said, how does it handle firstpath/somefile.bin and secondpath/somefile.bin being identical? This breaks your "diffable" output because the paths are different.
The easiest way to verify file checksums is to use "rsync -c". Rsync will usually skip checksum verification of files if both the modification time and size match on the source and destination, the "-c" option tells it to always compute a checksum.
Sometimes I also use the following "one" liner, if it's local.
Another nice rsync tip is that if you aren't using -a, you should at least use -t (preserve modification time). This will make a second rsync faster as it can skip files if the modification time and size match.
> For example, if we rsync -a /source /target, we trust that the contents of /target will exactly match the contents of /source
You might trust that, but you'd be wrong:
$ cd /tmp
$ mkdir foo
$ cd foo
$ mkdir bar
$ mkdir baz
$ touch bar/quux
$ rsync -a ./bar ./baz
$ ls baz
bar
$ ls -R baz
baz:
bar
baz/bar:
quux
You see, without a final '/' rsync will put the source into the target, rather than synchronising the source and the target. Also, if you really want a sync as opposed to just a full copy, you probably want to add --delete (which will delete files in the target which don't exist in the source.
So you probably want rsync -a --delete source/ target/.
Incidentally, given the directory structure above, here's a good way to calculate sha256 checksums over the files. It shells out rather than use the IRONCLAD package to calculate the checksums.
(let ((src #P"/tmp/foo/bar/")
(dst #P"/tmp/foo/baz/"))
(flet ((sha256sum (path)
"Return the SHA256 checksum of PATH as a string (which is
good enough for our purposes here)."
(first (split-sequence:split-sequence
#\Space
(uiop:run-program `("sha256sum" ,(namestring path))
:output :string)))))
(loop for src-path in (directory (uiop:merge-pathnames* "**/*.*" src))
for dst-path = (uiop:merge-pathnames* (uiop:subpathp src-path src) dst)
for file = (uiop:truename* dst-path)
if file
when (uiop:file-pathname-p file)
do (unless (string= (sha256sum src-path) (sha256sum dst-path))
(warn "~a does not match ~a" dst-path src-path))
end
else
do (warn "~a not found" dst-path))))
It would be interesting to see the minimal counter-examples to widely-believed properties that don't hold (not just for rsync but for dfferent CLI tools).
None of the methods described in the article is correct. md5sum does not compare files, it compares checksums. One mostly-correct solution is the 'd' (diff) option to GNU tar.
What you want is a tar that is deterministically identical regardless of the underlying file system's order. Sadly, no such option exists in GNU tar, but that doesn't quite mean the end of the quest. Behold:
find bin -print0 | sort -Vz | tar -cf bin.tar --no-recursion --null -T -
The first two shouldn't be hard to parse. Find files, separated by NULL characters (so any whitespace or other special characters won't be an issue), pass that onto "sort" that uses a version sort (identical rules to dpkg version sorting, and will disregard the locale sort) and interpret entries as being between NULLs (as well as emit them, likewise). The tar command should be pretty self-explanatory, but the "-T -" tells it to read stdin for a file list, with --null specifying that the input list is separated by NULLs, and --no-recursion prevents it from recursing into directories, instead just adding an entry in the tar for directories -- the find|sort will take care of the directory members already, sorted. Optionally add --numeric-owner to tar, if you want to compare trees on different computers, but results might vary (rsync and tar normally send/store both the id and name, and if the "name" exists on the local machine, uses that instead of id).