Member Avatar for Tygawr

I need to be able to read and write huge files that nearly weighs 20+GB and I need to do it fast. Any suggestions or tips? Thanks.

  • 4 Contributors
  • 4 Replies
  • 258 Views
  • 17 Hours Discussion Span
  • Latest Post Latest Post by BobS0327

Dani AI

Generated

Below are practical, tested patterns to handle 20+ GB files with good throughput; they expand on points from , , and .

Start checklist: use a 64‑bit build so you won’t fight address-space limits; make sure the volume is NTFS (FAT32 caps at ~4GB); prefer Win32 APIs (CreateFile/ReadFile/WriteFile/GetFileSizeEx or CreateFileMapping) for explicit 64‑bit offsets and flags. Decide first whether access is strictly sequential (streaming) or needs random access — that choice largely determines the fastest approach.

For sequential read/encode/write (common case): open with CreateFile and FILE_FLAG_SEQUENTIAL_SCAN, read large contiguous chunks (experiment 256KB–4MB), push each buffer into a small producer/consumer pipeline (reader thread → N encoder worker threads → writer thread). If encoding is CPU-bound, the pipeline keeps the disk busy while CPU works. For highest IO saturation on fast storage, issue multiple overlapped reads/writes (OVERLAPPED or IOCP) so you have several operations in flight.

For random access: memory-mapped windows often beat ad hoc reads. Use CreateFileMapping/MapViewOfFile and map windows aligned to the system allocation granularity from GetSystemInfo. If you use FILE_FLAG_NO_BUFFERING for raw I/O, you must align buffers and I/O sizes to the sector/cluster size (query with GetDiskFreeSpace) and allocate with _aligned_malloc/VirtualAlloc.

Simple synchronous pattern to get started and measure:

HANDLE h = CreateFile(path, GENERIC_READ, FILE_SHARE_READ, NULL, OPEN_EXISTING,
  FILE_ATTRIBUTE_NORMAL | FILE_FLAG_SEQUENTIAL_SCAN, NULL);

const DWORD BUF = 4*1024*1024;
char* buf = (char*)malloc(BUF);
DWORD n;
while (ReadFile(h, buf, BUF, &n, NULL) && n) {
  // encode buf[0..n)
  // WriteFile(outH, encodedBuf, encodedLen, &w, NULL);
}
CloseHandle(h); free(buf);

Tune buffer size, number of in‑flight ops, and thread counts while measuring real throughput. Also test with antivirus off, write to a temporary file and rename on success, flush (FlushFileBuffers) only if you need durability, and include robust error handling when you implement the final version.

Recommended Answers

All 4 Replies

Member Avatar for Ancient Dragon

1) Use 64-bit version of VC++, I think its version of fstreams support huge files

2) call win32 api read/write functions which support huge files (see ReadFile() and WriteFile())

Member Avatar for rubberman

How you do it depends upon whether you just need serial input/output, or if you need some random access to the data. If you need random access to the data, then as Ancient Dragon said, you need support for files >4GB (maximum a 32-bit offset can handle. Current Linux systems support access to very large files, for both 32-bit and 64-bit systems by defining the type fpos_t appropriately.

Member Avatar for Tygawr

I am trying to read all of the files, encode it, and write it. I need to it in a fast and efficient way. Thanks.

Member Avatar for BobS0327

Here is a I/O profiling link and here is a link to some info on memory mapping file I/O that may be of interest to you.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.