Member Avatar for Tygawr

Hello everyone,
I've attempted to use ifstream to open and read a 20GB file but failed to do so (it takes a very very very long period of time...). So, I need find a fast way to open and read a 20GB .txt file in C++. Thank you!

  • 2 Contributors
  • 8 Replies
  • 264 Views
  • 11 Hours Discussion Span
  • Latest Post Latest Post by Tygawr

Dani AI

Generated

Good approach: do not try to load the 20GB file into memory. Treat the file as an addressable byte stream and fetch only the bytes needed for the current "page". As noted, seeking to a byte offset is the right idea. Open the file in binary mode, compute page offsets as (pageIndex * pageSize) in bytes, and serve pages from a modest in-memory buffer that is refilled or prefetched as the user moves forward/back. 's 500-character pages are best implemented as byte ranges unless the file encoding is known to be multi-byte (UTF-8) — if so, take care not to split multi-byte characters at page boundaries.

A simple, correct building block (seek + read):

#include <fstream>
#include <vector>
#include <string>

const std::size_t PAGE = 500;
std::ifstream in("story.txt", std::ios::binary);
in.seekg(pageIndex * PAGE);
std::vector<char> buf(PAGE);
in.read(buf.data(), buf.size());
std::size_t got = static_cast<std::size_t>(in.gcount());
std::string page(buf.data(), got);

This is fine for occasional random jumps. For smooth paging (pressing a key to go next/prev repeatedly), read larger blocks (64KB–1MB) into a buffer and slice pages out of that buffer; keep the previous and next blocks cached so forward/back navigation rarely hits the disk.

Consider memory-mapped files for best performance when doing many small random reads: on Windows use CreateFileMapping/MapViewOfFile (map sliding windows in a 32-bit process; a 64-bit build simplifies mapping large ranges). Also hint the OS about access patterns with FILE_FLAG_SEQUENTIAL_SCAN or FILE_FLAG_RANDOM_ACCESS when opening the file. Measure with wall-clock timing and watch stream error flags.

Final notes: open as binary (no CRLF translation) if offsets matter; if the file is UTF-8, adjust page boundaries to valid code-point starts; prefer larger read sizes to reduce syscall overhead; and keep a tiny cache of recently-read blocks to avoid thrashing on mechanical drives.

Recommended Answers

All 8 Replies

Member Avatar for thines01

Is the file arranged in lines with either a linefeed or a CR/LF delimiter?
20GB (text)? Are you looking for something particular in the file?

Member Avatar for Tygawr

The text file is just a whole story. I am trying to see if its possible for me to be able to read that whole file very fast.

Member Avatar for thines01

I kind-of understand.

Are you keeping any of it in variables or just letting it overwrite the previous buffer?
Is it fixed-format or separated by linefeeds or carriage-returns or something?

Also, if you attempt to load the file with something like TextPad, how long does it take to open and jump to the last line?

That's probably the best you could expect with any software (that will actually do something with the text).

Edited by thines01 because: n/a
Member Avatar for Tygawr

I am trying to make a "console book reader". It opens a text file - which contains the story and its current size is nearly 20GB. It creates a substring for the first 500 characters and I can just press any key to read the next 500 character. Its linefeed.

Member Avatar for thines01

Well, since a stream can access up to 9,223,372,036,854,775,807 bytes, 20 GB would not be a problem.

If your buffer is 500 bytes (characters), you could access this quickly -- especially if each section is treated like a page. You could just "seek" to a given location, whch would provide quick access.

So, it's difficult to answer your question since I don't know how the "speed" will be judged. By access to the first three pages or by paging through the entire file (top to bottom).

Member Avatar for thines01

OK, to access (let's say) the first three pages would be very quick.
After that, you can seek to the next page group and read in enough pages to allow the user to scroll back and forth with minimal disk hits.

If you divide the entire thing into (segments / 500), you can just calculate where you need to seek.

It would be hard to judge the speed of sequentially reading the entire file 500 bytes at a time, but that won't matter with proper segmenting.

Member Avatar for thines01

OK, to access (let's say) the first three pages would be very quick.
After that, you can seek to the next page group and read in enough pages to allow the user to scroll back and forth with minimal disk hits.

If you divide the entire thing into (segments / 500), you can just calculate where you need to seek.

It would be hard to judge the speed of sequentially reading the entire file 500 bytes at a time, but that won't matter with proper segmenting.

Member Avatar for Tygawr

Okay, I will do that then. TY.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.