python3 indexer.py -n 1 /mnt/usb1 /mnt/usb2   # index your drives, using one core only
                                              # (to minimize thrashing with spinning drives)

python3 indexer.py /mnt/usb3ssd           # index external SSD? use all cores available

python3 indexer.py -l 2                   # which files exist on fewer than 2 drives?

python3 indexer.py -v                     # re-hash the drives and verify every checksum
                                          # basically, "ZFS scrub".

I am old enough to remember a time when "backup" meant walking around with a pile of floppy disks, or burning everything onto CD-ROMs (and discovering, years later, that a fair chunk of them had rotted into unreadable coasters). I even used a custom Reed-Solomon to make the process more robust!

...but these days, my approach is very simple; and it grew organically.

Backup, backup, backup - and take it with you

I used to have a ZFS mirror of two USB drives. For quite some time.

And it worked fine; giving me peace of mind, snapshots, and pool scrubs.

And I had all this on an Atomic PI, no less.

So why would you move away from ZFS, the king of filesystems?

(I also know about BTRFS. And mergefs. And snapRAID. And git-annex. Keep reading... Don't rush to comment just yet :-)

Basically, at some point, I realized I don't need any of the complexity involved in any of these solutions. Even the simpler ones, like git-annex, still had more mental overhead than I wanted (not to mention depending on, erm, some knowledge of exotic things like Haskell...)

Why bother with complexity?

...and no, I do NOT need ZFS snapshots on everything.

There were lots of data that I would be fine with just this: multiple copies on multiple external USB drives, with verifiable data integrity.

So I started with an old external USB hard drive, and filled it up with the files I wanted to protect:

# ls -d */
3Blue1Brown/         8-bit-guy/           AI.Donato.Capitella/
Electronics/         Mathologer/          Numberphile/        
Think.Twice/         Thoughty2/           ...
...

# ls -d Electronics/*/
Electronics/Andreas.Spiess/                       
Electronics/Ben.Eater/                            
Electronics/BigClive/                            
Electronics/Bitluni/                             
Electronics/mitxela/
Electronics/MrCarlsonsLab/
Electronics/My.Own/
Electronics/NandLand/
Electronics/Necroware/
Electronics/OpenTechLab/
Electronics/PhilsLab/
...

And when that drive filled up, I used another; splitting the data across the two. When that pair of drives filled up, I added a third drive. And so on.

Not rocket science. Just a growing stable of drives that I have accumulated over the years, put to use. With one, simple rule, and a single number, N:

Anything I keep at all, I keep on at least N drives.

Currently, N=2; but I could easily make that 3 if I needed to, or even higher; just by adding more drives; without caring about balancing ZFS pools, worrying about dealing with advanced filesystems and how to recover when/if their complex datastructures go haywire; with the absolute minimum of control state.

And with just one dependency: Python. Without any external libs of any kind.

Again, let me make this crystal clear: this is not about whole-drive mirroring or RAID-ing. No fancy algorithms. Each drive holds its own contents, but every file exists on at least N of them. If one drive dies (they will always do, eventually - it's not a matter of "if" but "when"), I still have a copy in at least one more drive. And I KNOW that copy is valid and correct.

How? Via checksums stored in a single-file SQLite db.

As simple as it gets.

That's what drove the creation of this standalone Python script.

Danger, Will Robinson

This is a tool I built for me, not a product aimed at the masses. It expects its user to be comfortable with a certain amount of discipline - case in point, in a moment I'll admit to version-controlling the SQLite database itself with git. If that already sounds too fiddly for your taste, it probably is.

But if you're the sort of person who enjoys building glue around your own workflow, read on.

The two serious questions

The simple rule (at least N copies) is still only as good as the ability to answer two questions:

1. Redundancy - is every file actually present on at least N drives? When I add, edit or move a file on one drive and forget to propagate the change to the other drive that's supposed to carry a copy, its count silently drops from 2 to 1. Nobody notices until the day it matters.
2. Integrity - is the copy on drive B identical to the one on drive A? Hard drives do silently corrupt bits (bitrot is real, and more common on large, aging drives) - and especially external USB drives can end up with a corrupted file because of transient cabling issues (a bad cable, a tired connector, a glitch during a transfer). Yet the filesystem is happy to tell you "everything is fine", because it has no idea the content changed.

Syncing (rsync, or your favorite tool du jour) handles the copying, but not the verification of either of these. After every sync, I still couldn't tell you, with a straight face, whether every file really existed in enough places, and whether all those copies had identical checksums.

I wanted a tool that could look at my stable of drives and answer, on demand:

"Which files are missing from some of my drives?"
"Did every single file's checksum match across all copies?"

FileIndexer

The tool is a single Python script, indexer.py - no install, no dependencies, just the standard library. It maintains a small SQLite database that remembers every file it has ever seen: its path, its size, its modification time, and its MD5 checksum. Then:

$ # index your (mechanical) drives - one core to avoid thrashing them
$ indexer.py -n 1 /mnt/usb1 /mnt/usb2

$ # which files exist on fewer than 2 drives?
$ indexer.py -l 2
backup/tax/2024.7z#@#1 1a2b3c4d5...

$ # re-hash every drive and verify each stored checksum
$ indexer.py -v    # aka "ZFS scrub".

That "-l 2" answer reads: backup/tax/2024.7z exists on only one drive, but you asked for two. Go fix that.

And -v re-reads every byte of every file and compares against the checksums stored during the last sync - so a silent bit flip will show up as a cheerful MISMATCH:

[!] MISMATCH: /mnt/usb1/backup/tax/2024.7z (expected=1a2b..., actual=1234...)

That's it. That's all you need, if, like me, you keep a growing stable of external drives and want to...

• be sure every file lives on at least N of them
• before disaster strikes, be sure the redundancy actually holds for REAL.

The utility lives at github.com/ttsiodras/FileIndexer.

What makes it not-completely-trivial

Writing a "file hasher" is easy. Making one that doesn't fool you is where the real work was, and I learned some things the hard way. A few highlights:

Spinning disks vs SSDs. If your external drive is an SSD one, and you have an average consumer CPU, the indexer should be able to spawn MD5 workers all the way up to your machine's CPU count; and exploit the fact that your external SSD drive can deliver hundreds of MBs/sec on file-based random-access reads.

But if your external drive is a spinning one, you want to dial this down to just a few workers (maybe even just one!). Not only because you end up getting slower speeds because of the crazy seeking going all over the place, but also because you are literally abusing your drive's head-moving and seeking machinery. You'll make it die faster.

Hence, the need for the -n option - control how many computation pools will be used. For example, using -n 1 is the safest option for spinning storage.

The parallel hashing is bounded in memory. MD5 computations are spread across all your cores by default - but the number of files in flight is capped at a small multiple of your core count. You don't want to send millions of small files to your computation pool, and run out of memory...

Non-UTF-8 file names are handled correctly. This is the one that drives people nuts. Your filesystem allows bytes that are not valid text - and plenty of tools crash or silently garble them. FileIndexer stores paths as raw bytes, so even a filename with a 0xff byte in it... just works:

$ python3
>>> os.stat(b'/mnt/usb1/caf\xff.txt')   # the \xff just rides along

The database is crash-safe. It uses SQLite in WAL mode with synchronous=NORMAL, and commits after every single file - so if the power dies mid-scan, or the code crashes for whatever reason... you lose at most the very last file's checksum, not the whole afternoon's work.

The failure modes - the real part of this story

You don't learn much from the happy path. You learn from the day a tool silently deletes your index.

On a flaky USB drive, a directory can fail to enumerate once - a bad cable, a tired controller, a transient I/O error. The naive thing happens: the tool can't see inside that directory, so it concludes the files are gone, and deletes their rows from the index:

[-] Deleted (missing): sub/b.txt     <-- but b.txt was STILL THERE on the disk!

False confidence. The whole point of the tool is to tell the truth - and here it was merrily telling me "everything is fine" after quietly wiping out a day's worth of checksums. For an integrity-checker, that is the worst possible bug.

The fix: when a directory can't be scanned, the tool now says so loudly, and refuses to treat anything under it as deleted. The worst case becomes "those files weren't re-verified this run - let's re-check next time" instead of "those files vanished from the index":

[!] Unreadable directory, skipping: /mnt/usb1/sub
[-] Sync complete: 0 inserted, 0 updated, 0 deleted

Other examples of issues:

• What happens if a hashing worker process dies mid-run? An out-of-memory kill is not impossible if you opened dozens of applications doing other work, and forgot that your indexer is throwing 200GB VM files in the parallel MD5 computation pool at the same time...

• Refuse to store the database inside the very folder being scanned. (Yes, some moron (me) may try to do this; and no, the indexer will no longer let you shoot yourself in the foot like this.)

• etc, etc. You can look at the git history of the script to see the subtle ways things can fail even in something as simple as an MD5 hasher.

Note that there's a full test suite - 20 end-to-end tests that actually run the binary against temporary directories and check the resulting database and reports - plus static analysis/linting with flake8, pylint and mypy.

A GitHub Actions CI runs it all on Python 3.9, 3.12 and 3.14 - and so can you:

$ make
[-] Installing VirtualEnv environment...
...
make[1]: Entering directory '/home/ttsiod/Github/FileIndexer'
============================================
 Running flake8...
============================================
.venv/bin/flake8 indexer.py
make[1]: Leaving directory '/home/ttsiod/Github/FileIndexer'
make pylint
make[1]: Entering directory '/home/ttsiod/Github/FileIndexer'
============================================
 Running pylint...
============================================
.venv/bin/pylint --rcfile=pylint.cfg indexer.py

--------------------------------------------------------------------
Your code has been rated at 10.00/10 (previous run: 10.00/10, +0.00)

make[1]: Leaving directory '/home/ttsiod/Github/FileIndexer'
make mypy
make[1]: Entering directory '/home/ttsiod/Github/FileIndexer'
============================================
 Running mypy...
============================================
Success: no issues found in 1 source file
make[1]: Leaving directory '/home/ttsiod/Github/FileIndexer'

$ make test
Test1 passed
Test2 passed
...
Test20 passed
All tests passed successfully.

The real question - which copy is the right one?

There is a moment that quietly exposes the weakness of any "at least N copies" scheme. "-l 2" flags a file, and when you look at the report from the indexer, you find it exists on two drives with two different checksums:

backup/photos/summer.2026/IMG00001.jpg#@#1 12345678...   # on /mnt/usb1
backup/photos/summer.2026/IMG00001.jpg#@#1 ab12cd34...   # on /mnt/usb2

Two copies, two md5s. Which one is the correct file? N=2 just bought you two answers that disagree, and nothing to tell them apart. (Comparing the copies to each other doesn't help - they're both there, they just don't match.)

If you had 3 copies, you'd be easily able to see which one is the right one; the one that has TWO votes! (unless all 3 are different, but then you must be really, really unlucky...)

But if you are cheap and only have enough room to store 2 copies? Or if you are galaxy-class-of-unlucky and you used 3 copies, but get a report of all 3 being different?

My solution to this, is frankly, a bit hacky. I keep the database itself under git revision control.

Here's the workflow in practice:

1. When I first set up a drive, I go through -l 2 until the report comes back empty - meaning every file is present on at least two of my collection of drives, and since the report is empty, all copies have matching checksums. That is my known-good snapshot: the md5s in the database are, by construction, correct.

2. So I then "git commit files.db". The DB repository now holds the authoritative list of checksums, fixed in amber.

3. Later, when -v or a fresh -l 2 shows that foo/bar/baz.txt exists with md5 A on one drive and md5 B on another, I can look the file up in the committed copy of files.db, which tells me what its md5 was when everything agreed:

$ git show HEAD:files.db > /tmp/good.db
$ sqlite3 /tmp/good.db \
  "select md5 from files where full_path like '%/baz.txt'"
1a2b3c4d5...

1. The copy that still matches the committed checksum is the good one; the other is either a silently corrupted drive or a legitimate edit that I forgot to propagate - and one that I now need to fix properly and then re-commit.

Version-controlling a SQLite file is, admittedly, a blunt instrument. But it hands me exactly the thing the tool itself can't: an authoritative record of "what was correct" that survives drives dying and content diverging. The tool covers the "is it there, does it match?" questions; git-history of the database is my answer to the follow-up - "and which copy do I trust?"

And yes, that means I do need to keep this simple repo somewhere else as well. Just "git push" it somewhere; anywhere you want (ideally in a private repo, but... you do you). It will be a small file anyway, since SQLite is very space efficient. If that is not the case because you have bazillions of files... OK, then use git LFS. Or save the sqlite DB file itself (files.db) somewhere else; I am not your mommy, surely you can figure out a way to have your backup script save/rsync a single file somewhere.

(wears helmet) the tool was built with local AI - but wait...

Here's a part that's perhaps interesting - it was to me, at least. This codebase was written with the help of local, private AI models. I try to see past the hype, and the only way to do this is to run AI yourself; not pay for the privilege of using someone else's machines.

This codebase has evolved over months with one local model after another taking a crack at it - reviewing, modifying, improving, bug hunting...

Some, but not all local models used for this:

• GPT OSS 120B
• Qwen 3.5 122B
• Deepseek v4 Flash 0731
• Qwen 3.8 27B

I did not just glue together whatever the models spat out and call it a day. The AI produced the bulk of the structure; I reviewed it, prodded it, poked the failure modes until I found the ones that mattered (the unreadable-directory row-deletion bug above is exactly the kind of thing an AI, and a naive review, will happily miss), and kept sending the whole thing back for another round. My local AI pets and myself, we iterated until the result was something I honestly believe is a good, defensible Python codebase.

I think at some point I did ask the free version of Claude (Sonnet, I believe?) to have a look (the repo is public, after all); and Sonnet did point out some issues. But the code has been overwhelmingly written and worked on by local models. It's a nice demonstration of the workflow I actually want: local models, my own code, my own review, my own hardware. The AI accelerates; it doesn't delegate.

Why MD5?

Why not? :-)

I did consider using something other than MD5; but realized that this process is anyway dominated by I/O (USB I/O in my case); so the checksumming algorithm itself is not really important.

As long as it can detect the issues, and each of my cores can use it on a USB stream without getting CPU bound, it's fine.

If it's not, the code is there for you, in one standalone Python script. Patch it to use your favourite checksumming algo.

But... you just re-invented a much worse snapRAID! BTRFS! ZFS! ...

No, I did not.

Remember the call for simplicity? I dont care about parity codes. I could have done something using parity-based recovery again, as I did 20 years ago; but my goal here was to keep things literally as simple as possible.

To my knowledge, at least, there is no lightweight, dependency-free tool that tracks files across a bunch of independent disks and tells you whether you actually have N copies of each one. There are parity data computation tools, all the way to the ancient par; and there are filesystems with checksums and snapshots; but there is nothing that is as simple as this indexer, watching your back by telling you: "you only have this file once, better make a copy of it in at least one more drive".

I did mention lightweight and dependency-free - so I fully expect an attack from the Haskell crowd about git-annex. What can I say? indexer.py is much, much simpler. You do you, of course.

And no, ZFS "copies=2" is not the same; I can CHOOSE to have a specific file (or a thousand) in 3 drives, not 2. And then I get extra safety JUST for those files alone.

Conclusion

Backups only count when you can prove they exist - that every file really is on at least N drives, and that each of those N copies is byte-identical. FileIndexer is my answer to that, and it's been quietly earning its keep ever since it sprang into existence.

Prior to that, I've used ZFS on two USB drives; which, don't get me wrong, is infinitely superior from a technical standpoint.

Where ZFS loses, is in the inherent complexity. FileIndexer is as simple as can be; I don't have to care about enlarging ZFS pools, about raidz vs mirroring, or whatever. I don't have to worry about digging data out in case of disastrous scenarios; needing insight into the insanely complex data structures involved.

The files are just there, on USB drives; and a SQLite database tells me their MD5s. Easy to figure out, easy to recover.

Let me be clear: I'm not claiming this is bulletproof. But I will claim that it's a good balancing act - deliberately simple logic that I can track, understand and trust, all the way down - even if some of the glue around it (hello, git-versioned database) is cheerfully unconventional. I'd rather have that than a system so clever that its failures are beyond my comprehension (and ability to recover).

Finally, building it with local AI was a very educational exercise in itself (and admittedly, part of the impetus to do this). I saved some of my first prompts and plans in the repo; but be aware that there have been many iterations since that first prompt, with multiple local models taking turns looking at the code.

Anyway - if you want to keep important data under an at-least-N-copies rule, I hope it will be useful to you too: github.com/ttsiodras/FileIndexer.