Member Avatar for freesoft_2000

Hi everyone,

I am trying to to read and write large files about 600M.
The thing is when i use the normal FileOutputStream methods an exception gets thrown saying

java.lang.OutofMemory

This is how i am trying to read the file

FileOutputStream out = new FileOutputStream("C:/my_life_story.zip");
InputStream in = //an input stream from a socket, etc 

int len = 0;
byte[] buffer1 = new byte[1024]; 

while ((len = in.read(buffer1)) > 0)
{ 
out.write(buffer1, 0, len);
}
 
in.close(); 
out.close();

My computer does have sufficient memory

I tried adding this in the while loop but it does not work and i got the same exception thrown by the JVM again

System.gc();

My question is basically how do i increase the memory programatically
or is there away in which i can read the file without having the above exception thrown by the JVM?

Any help i greatly appreciated

Thank You

Yours Sincerely

Richard West

  • 3 Contributors
  • 4 Replies
  • 399 Views
  • 22 Hours Discussion Span
  • Latest Post Latest Post by freesoft_2000

Dani AI

Generated

— the pattern shown in the original post (read a fixed-size buffer and write it out) is the correct streaming approach; it should not force the JVM to hold the entire file in heap. was right that keeping everything in memory is the real problem, and was right that changing the maximum heap on JVM startup is sometimes necessary — but the immediate step is to diagnose where the memory is actually being used before raising heap limits. (developers.google.com)

Troubleshooting checklist: 1) inspect the full OutOfMemoryError stack trace to see which class triggers the error; 2) enable heap-dump-on-OOME or create a live heap dump and inspect it with VisualVM / jcmd / jmap to find large retained objects; 3) verify no code path is collecting bytes (ByteArrayOutputStream, readAllBytes, caching into collections or Strings). Generating and analyzing a heap dump is the most reliable way to see whether the problem is the copy loop or something else. (docs.oracle.com)

If the copy loop is indeed the only code touching the bytes, use the JDK streaming helpers instead of reinventing byte plumbing. Two robust options are Files.copy(InputStream, Path, ...) or using NIO channels (Channels.newChannel + FileChannel.transferFrom) which stream without loading the whole stream into the heap:

// simple, safe streaming API
try (InputStream in = socket.getInputStream()) {
    Files.copy(in, Paths.get("C:\\output.zip"), StandardCopyOption.REPLACE_EXISTING);
}

or

try (ReadableByteChannel rbc = Channels.newChannel(in);
     FileOutputStream fos = new FileOutputStream("C:\\output.zip")) {
    fos.getChannel().transferFrom(rbc, 0, Long.MAX_VALUE);
}

Both avoid large-heap allocations and are implemented to stream efficiently. (docs.oracle.com)

Notes on flushing and heap size: calling GC or repeatedly flushing a FileOutputStream rarely helps — use a BufferedOutputStream when you want Java-level buffering and call flush only to force buffered bytes out. You cannot increase a running JVM’s max heap from inside the program; the max heap is set at JVM start (so for double‑click users distribute a small launcher, a shortcut/batch file that starts java with the desired -Xmx, or use tools like Launch4j / jpackage to bundle startup options). (docs.oracle.com)

Quick checklist summary: 1) inspect stack trace + heap dump; 2) confirm no code buffers the whole file; 3) switch to Files.copy or FileChannel transfer if needed; 4) if real heap growth is required, set the JVM option at startup or ship a launcher.

Recommended Answers

All 4 Replies

Member Avatar for jwenting

Try flushing the buffer after every write.
The only thing I can think of that's eating up memory here is the outputbuffer. It may not be flushing automatically, causing the JVM to attempt to map the entire output into RAM instead of writing it directly to disk (which you seem to be attempting to do).

Member Avatar for NPH

Maybe you should try setting the maximum heap size for the JVM.

java -Xmx256m MyClass

You can put higher values if needed.
For more help,

Member Avatar for jwenting

doesn't matter, having a 256MB heap won't help you contain a 600MB file in memory :)
You don't NEED to retain all that data in memory so the real solution is to not retain the data at all.

Member Avatar for freesoft_2000

Hi everyone,

Is this what you meant NPH

java -Xmx650m MyProgram

Unfortunately i can't use the above code because the thing is the application is in a jar file on windows platform and users that use this application to run it usually double click on the jar file to run it. Is there a way to do what you suggested programatically maybe by the use of properties?

I think maybe jwenting maybe right that i have to flush and i will be trying it out.

On another question do any of you guys know how to increase the heap size programatically?

Richard West

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.