Member Avatar for acezrwild817

VS 2005 does not include the crc32 checksum in their crypto library and I was wondering if anyone knows where I could find a very fast algorithm. I currently have three different algorithms but none of them are much faster than Md5. I know for a fact that the crc32 is much faster than md5 but I can't seem to find one that has been coded in the same manner. It has to be able to hash files.

  • 2 Contributors
  • 2 Replies
  • 115 Views
  • 1 Week Discussion Span
  • Latest Post Latest Post by acezrwild817

Dani AI

Generated

Useful follow-ups to the thread (builds on and )

A native C DLL is a good start — it removes managed bounds checks and runtime overhead — but there are several algorithmic and platform options that often yield much bigger wins than a plain byte-at-a-time table.

Slicing-by-N: start with a 256-entry table algorithm, then move to slicing-by-4 or slicing-by-8. Slicing-by-8 uses eight 256-entry tables and processes eight bytes per loop iteration; that reduces memory traffic and loop overhead and is the common high-throughput approach used in fast libraries.

Hardware intrinsics and CPU features: modern x86 and ARM CPUs provide CRC instructions (SSE4.2 on x86, CRC extensions on ARMv8) that accelerate CRC computations — note these typically target the CRC32C (Castagnoli) variant, not every CRC-32 polynomial. Use compiler intrinsics or runtime CPU feature detection and fall back to software slices if the CPU lacks the feature.

Practical tips that matter in file hashing:

  • Read in large chunks (64KB–1MB) to minimize syscall/IO overhead.
  • Avoid per-byte P/Invoke calls; expose one native function that accepts a buffer and length (or use memory-mapped files) to avoid interop overhead.
  • Compile optimized release builds and align/unroll inner loops where it helps.
  • Benchmark on representative files — small files are dominated by IO and syscall costs; large sequential reads show pure CPU throughput.

Example processing loop (C-style pseudocode):

uint32_t crc = 0;
while ((n = fread(buf, 1, BUFSIZE, f)) > 0) {
    crc = crc_update(crc, buf, n); // crc_update uses slicing or intrinsics
}

Cautions: confirm which CRC polynomial you need (CRC-32 vs CRC-32C) for compatibility, watch endianness if sharing checksums across platforms, and prefer well-tested libraries (zlib/libdeflate or vendor intrinsics) if correctness and portability matter.

Recommended Answers

All 2 Replies

Member Avatar for Lord Soth

Hi,

I can provide you a C# implementation of CRC32. (you can easily find one googling too) But if the speed is of essence (making managed MD5 unusable) you might prefer to use a native code CRC32 DLL through Platform Invoke [P/Invoke].

Loren Soth

Member Avatar for acezrwild817

I have recently written a version in C and imported it through a dll. It was about 3x faster. Thanks for the help.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.