Member Avatar for evank

I have a collection of (potentially) several hundred thousand image files that I need to generate hash digests for, and I'm unsure of the best algorithm to use. I'll be keying them all to a database based on their hash, so I need the best algorithm for avoiding accidental collisions (where different files end up generating the same hash).

CRC32 is out of the question because I know it's threshold is fairly low, so I'm thinking either MD5 or SHA1, but I don't know which one is better for my purposes or if there's an even better algorithm. Most of what I've found after searching has recommended SHA1 as being 'more secure', but in my case I'm only concerned with accidental collisions rather than intentional malicious ones. Would SHA1 still be the preferable one for my purposes?

  • 3 Contributors
  • 6 Replies
  • 303 Views
  • 6 Days Discussion Span
  • Latest Post Latest Post by sarehu

Dani AI

Generated

A short, practical addendum to the thread: cryptographic weaknesses and a usable pipeline.

No hash gives a mathematical guarantee of uniqueness, as reminded, and some widely used algorithms have known, practical collisions. MD5 collisions were demonstrated in the academic literature, and SHA‑1 collisions were produced in practice (the SHAttered work). Avoid relying on those older hashes for anything where a collision would cause trouble. (eprint.iacr.org)

For a robust, fast dedupe workflow that balances speed and safety: (1) filter by file size (very cheap), (2) group candidates with a fast non‑cryptographic hash (xxHash/XXH3 is a common choice), (3) compute a modern cryptographic fingerprint (BLAKE3 or SHA‑256) for the surviving group, and (4) before any destructive action (delete/replace), perform a final byte‑for‑byte comparison. Many duplicate‑finders follow the size→hash→byte‑compare pattern for exactly this reason. (cyan4973.github.io)

Practical notes for ’s scale (hundreds of thousands of images): store file size plus the fast fingerprint in your index to cheaply eliminate most candidates. Use a fast cryptographic hash (BLAKE3 is both secure and very fast) as the canonical DB fingerprint; that keeps lookups compact while reducing the tiny collision risk to negligible levels. If throughput matters, compute the cheap hash first and only compute BLAKE3 for groups that survive the size+fast‑hash checks. (blake3.io)

Cautions and edge cases: identical visual images may differ in metadata (EXIF) or container bytes — decide whether “duplicate” means exact bytes or identical rendered image (perceptual hashing is a different tool). Always treat any matched pair as a candidate and verify before irreversible operations; logging the original paths + timestamps before any merge/delete is good hygiene. This expands on ’s speed/security tradeoff and gives an operational way to apply it safely. (manpages.ubuntu.com)

Recommended Answers

All 6 Replies

Member Avatar for sarehu

All of the algorithms MD5, SHA-1, SHA-256, ... are good for avoiding accidental collisions. If the running time is an issue for you, MD5 is faster than SHA-1 by a factor of say, 2, and a similar ratio exists for SHA-1 to SHA-256. That ratio is based on reading one website and on running openssl speed md5 and openssl speed sha1 on my computer.

So use md5 for speed, sha-1 for extra paranoia (it's what Git uses for the purpose, after all), and sha-256 for an algorithm for which no collisions have been found yet, period :-)

commented: answered my question and then some, very informative and helpful! +2
Member Avatar for jwenting

No hashing algorithm will guarantee you that there won't be colissions.
All you can do to reduce the occurrance of colissions is to either increase the length of the hash key (and the length of the hash itself) and/or increase the amount of data used to calculate the hash.

Of course either will dramatically increase the amount of time needed to calculate the hash, which may or may not be acceptable.

So the question becomes: how unique do you want your hash to be and why?

Member Avatar for evank

Well, like I said, I'm generating digests for potentially several hundred thousand image files of varying formats, and using the digests to detect and prevent duplicates. I think SHA1 will work, SHA256 seems a bit of overkill.

Of course if I do encounter more collisions than I expected, it would be easy enough to switch over to SHA256 and just redigest all the files.

Member Avatar for sarehu

I think SHA1 will work, SHA256 seems a bit of overkill.

Of course if I do encounter more collisions than I expected,

Ha ha ha ha ha. If you unexpectedly encounter a single collision while hashing different images, I will pay you $1000000.

Member Avatar for jwenting

you can't use hashes to detect duplicates, because as I said they're never guaranteed to be different for different data.

Member Avatar for sarehu

Yes you can.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.