Hello everyone,
I've attempted to use ifstream to open and read a 20GB file but failed to do so (it takes a very very very long period of time...). So, I need find a fast way to open and read a 20GB .txt file in C++. Thank you!
Tygawr -12 Newbie Poster
thines01 401 Postaholic Team Colleague Featured Poster
Is the file arranged in lines with either a linefeed or a CR/LF delimiter?
20GB (text)? Are you looking for something particular in the file?
Tygawr -12 Newbie Poster
The text file is just a whole story. I am trying to see if its possible for me to be able to read that whole file very fast.
thines01 401 Postaholic Team Colleague Featured Poster
I kind-of understand.
Are you keeping any of it in variables or just letting it overwrite the previous buffer?
Is it fixed-format or separated by linefeeds or carriage-returns or something?
Also, if you attempt to load the file with something like TextPad, how long does it take to open and jump to the last line?
That's probably the best you could expect with any software (that will actually do something with the text).
Tygawr -12 Newbie Poster
I am trying to make a "console book reader". It opens a text file - which contains the story and its current size is nearly 20GB. It creates a substring for the first 500 characters and I can just press any key to read the next 500 character. Its linefeed.
thines01 401 Postaholic Team Colleague Featured Poster
Well, since a stream can access up to 9,223,372,036,854,775,807 bytes, 20 GB would not be a problem.
If your buffer is 500 bytes (characters), you could access this quickly -- especially if each section is treated like a page. You could just "seek" to a given location, whch would provide quick access.
So, it's difficult to answer your question since I don't know how the "speed" will be judged. By access to the first three pages or by paging through the entire file (top to bottom).
thines01 401 Postaholic Team Colleague Featured Poster
OK, to access (let's say) the first three pages would be very quick.
After that, you can seek to the next page group and read in enough pages to allow the user to scroll back and forth with minimal disk hits.
If you divide the entire thing into (segments / 500), you can just calculate where you need to seek.
It would be hard to judge the speed of sequentially reading the entire file 500 bytes at a time, but that won't matter with proper segmenting.
thines01 401 Postaholic Team Colleague Featured Poster
OK, to access (let's say) the first three pages would be very quick.
After that, you can seek to the next page group and read in enough pages to allow the user to scroll back and forth with minimal disk hits.
If you divide the entire thing into (segments / 500), you can just calculate where you need to seek.
It would be hard to judge the speed of sequentially reading the entire file 500 bytes at a time, but that won't matter with proper segmenting.
Tygawr -12 Newbie Poster
Okay, I will do that then. TY.
Be a part of the DaniWeb community
We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.