Member Avatar for Tygawr

How can I parse through a large file in a fast and efficient way?

  • 2 Contributors
  • 3 Replies
  • 186 Views
  • 3 Hours Discussion Span
  • Latest Post Latest Post by Ancient Dragon

Dani AI

Generated

noted the file is plain text on Windows/VC++2010; ’s index idea is the right call when many random lookups are needed. For single-pass parsing or heavy tokenization, however, two patterns usually give much higher throughput than line-by-line formatted input: (1) large-block buffered reads with in-place scanning, and (2) memory‑mapped reads that avoid copies. Both reduce per-line allocations and per-character overhead.

A compact, portable pattern (fread + memchr) keeps one reusable buffer and carries partial lines between reads:

const size_t BUFSZ = 1<<20; // 1 MB
std::vector<char> buf(BUFSZ);
std::string carry;
FILE* f = fopen("big.txt","rb");
while (true) {
  size_t n = fread(buf.data(),1,buf.size(),f);
  if (n==0) break;
  const char* p = buf.data();
  const char* end = p + n;
  while (p < end) {
    const char* nl = (const char*)memchr(p, '\n', end - p);
    size_t len = nl ? (nl - p) : (end - p);
    carry.append(p, len);
    if (nl) {
      if (!carry.empty() && carry.back() == '\r') carry.pop_back(); // handle CRLF
      /* process line in 'carry' */
      carry.clear();
      p = nl + 1;
    } else {
      p = end; // partial line saved in carry
    }
  }
}
if (!carry.empty()) { /* process last line */ }
fclose(f);

On Windows, memory mapping often wins for read‑heavy parsing because the kernel pages the file and the code can scan a raw pointer. Skeleton:

HANDLE h = CreateFileA(path, GENERIC_READ, FILE_SHARE_READ, NULL, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, NULL);
LARGE_INTEGER sz; GetFileSizeEx(h, &sz);
HANDLE fm = CreateFileMappingA(h, NULL, PAGE_READONLY, 0, 0, NULL);
const char* base = (const char*)MapViewOfFile(fm, FILE_MAP_READ, 0, 0, 0);
const char* end = base + (size_t)sz.QuadPart;
/* scan base..end with memchr as above */
UnmapViewOfFile(base); CloseHandle(fm); CloseHandle(h);

Practical tips: choose buffer sizes (64KB–4MB) and measure; reserve or reuse a std::string to avoid reallocations; handle CRLF and BOM (UTF-8 vs UTF-16) up front; map in chunks if the file exceeds addressable space; only parallelize by chunk if boundary stitching is handled. For structured formats (CSV/JSON) consider well‑benchmarked parsers rather than raw text handling.

Recommended Answers

All 3 Replies

Member Avatar for Ancient Dragon

Is it text or binary? What operating system and what compiler?

Member Avatar for Tygawr

Text,
Window 7,
VC++ 2010

Member Avatar for Ancient Dragon

The only way to process text files is to read them sequentially, start to finish. Text files generally don't allow for random access. One trick I've used in the past with text files which must be read frequently is to set up a fixed-length record index file which contains line numbers and the offset in the file where the line number begins. This only speeds up frequently reading the file, not writing. You have to completly rewrite the file if the new text is longer or shorter than the existing text. For example you can't just simply overwrite the word "the" with the word "them". But if you want to overwrite "the" with anothr 3-letter word then you can just overwrite it, assuming you know the offset of the start of the word.

A better answer to your question will depend on what you mean by "process the file". It might mean that you need to read a block of lines into memory, process them, and rewrite them to another file, then repeat that until the entire file is processed.

Edited by Ancient Dragon
Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.