file handling, outof memory error :(

[Miyuki Suzuki]

Hello again,

today is just not my day :(
I had a problem with speed this morning and got help that totally solved it, so here goes hoping for some more help.

background:
I have an ejb that picks up a file, copies it and the has to process the file contents for persistence.

method:
I get the file data in an array and extract the strings from the array. Assign the values to an array list and then add the list to a collection and persist the collection.
*There has to be a better way to do that!

the problem
When I process a file of around 65.956k, I get an outofmemory error and i am stuffed!

Sample code:

/*
 * To change this template, choose Tools | Templates
 * and open the template in the editor.
 */
package stuff;

import java.io.BufferedReader;
import java.io.FileNotFoundException;
import java.io.FileReader;
import java.io.IOException;
import java.text.SimpleDateFormat;
import java.util.ArrayList;
import java.util.Date;
import java.util.List;


public class Read {

    String DATE_FORMAT_NOW = "yyyy/MM/dd hh:mm:ss";
    SimpleDateFormat sdf = new SimpleDateFormat(DATE_FORMAT_NOW);
    long starttime = System.currentTimeMillis();
    String timeS = sdf.format(new Date(starttime));

    void readFile(String fileName) {
        System.out.println(">> time of exe: " + timeS + " milliseconds: " + starttime);
        BufferedReader br = null;
        boolean firstLine = true;
        int size = 32 * 1024; //32k buffer
List<String> stuff = new ArrayList<String>();

        try {
            br = new BufferedReader(new FileReader(fileName), size);
            int count = 0;
            String line = "";
            while ((line = br.readLine()) != null) {
                //strip header
                if (firstLine) {
                    System.out.println("header line :" + line);
                    firstLine = false;
                    continue;
                } else {
                    String[] lineA = line.split("\\|");

                    String id = lineA[0];
                    String flag = lineA[1];
                    String scoreDateS = lineA[2];
                    String flags = lineA[3];
                
                    stuff.add(id);
                    stuff.add(flag);
                    stuff.add(scoreDateS);
                    stuff.add(flags);
                    count++;
                }
            }
            System.out.println(">> rows: " + count);
            System.out.println(">>> elapsed time: " + (System.currentTimeMillis() - starttime));
        } catch (FileNotFoundException fnfe) {
            fnfe.printStackTrace();
        } catch (IOException ioe) {
            ioe.printStackTrace();
        } finally {
            if (br != null) {
                try {
                    br.close();
                } catch (IOException ioe) {
                }
            }
        }
    }

    public static void main(String[] args) {
        String fileName = "\\Score.dat";
        Read r = new Read();
        r.readFile(fileName);
    }
}

the persisting isnt done here since I know i run out of memory before I even get to that point.
It seems like the memory runs out at the point of adding the strings to the arraylist.
Can anyone please help.

Regards,
Miyuki

[masijade]

Why a collection. Why not directly to a DB, or, worse case scenario, a file (i.e. randomaccessfile)?

In any case, 65k is not much. You could simply increase the maximum heap size.

[Miyuki Suzuki]

I already increased the heap size to 1g.
why a collection? - I am using a collection of the entity that i want persisted.
Straight db insert - I dont want this, since I want to use only namedqueries and jpa.

so instead of doing the whole insert into .....[\I]
im doing this

for (Score spp : scoreCollection) {
                            entityManager.persist(spp);
                }

[Miyuki Suzuki]

nevermind,
I solved it.
took array items directly into my Arraylist and then collection.
persistence works., time = 21 secs.

thanks for replying masijade