Member Avatar for Phaelax

Which would be the preferred, or more efficient, way to read in a 40mb XML file? I remember reading (awhile ago) there were different methods of parsing XML, some more efficient than others depending on the need. The entire file would eventually need to be loading into an array, so I don't really need to search the file for a specific entry at random but simply all of them.

  • 4 Contributors
  • 4 Replies
  • 116 Views
  • 3 Days Discussion Span
  • Latest Post Latest Post by jwenting

Dani AI

Generated

Short summary tied to the thread: asked about reading a ~40 MB XML file. Several replies raised memory and parser-choice points (see , , ). Below is a concise, practical guide to the trade-offs and a straightforward Java streaming approach that fits the “read all entries” requirement without blindly loading a heavyweight in-memory tree.

Key trade-offs and strategy

  • In-memory DOM builds a full node graph; a rule-of-thumb is the parsed tree can require several times the file size (often 4–10x) because every element/attribute becomes objects. For a 40 MB document that can easily push heap usage into hundreds of megabytes.
  • If the end result is an array/list of compact records (POJOs with only needed fields), parse the file as a stream and construct each record one at a time. Persist or accumulate only the lightweight records. This avoids holding the XML node graph.
  • A pull-style streaming parser (StAX) gives simpler, linear code compared with callback-based handlers while still keeping memory low. If random access or faster repeated XPath-like queries are needed later, consider specialized libraries (VTD-XML) or a database import.

Minimal StAX pattern (Java)

XMLInputFactory f = XMLInputFactory.newInstance();
try (FileInputStream fis = new FileInputStream("big.xml")) {
    XMLStreamReader r = f.createXMLStreamReader(fis);
    List<Record> list = new ArrayList<>();
    Record cur = null;
    String text = null;
    while (r.hasNext()) {
        int ev = r.next();
        if (ev == XMLStreamConstants.START_ELEMENT) {
            String n = r.getLocalName();
            if ("record".equals(n)) cur = new Record();
        } else if (ev == XMLStreamConstants.CHARACTERS) {
            text = r.getText().trim();
        } else if (ev == XMLStreamConstants.END_ELEMENT) {
            String n = r.getLocalName();
            if ("field".equals(n) && cur != null) cur.addField(text);
            else if ("record".equals(n)) { list.add(cur); cur = null; }
        }
    }
    r.close();
}

Troubleshooting and practical tips

  • Roughly estimate number of records × average object size to size heap or choose a spill-to-disk strategy.
  • Profile actual runs with VisualVM/jmap to verify peak usage; increase -Xmx only if justified.
  • If memory still tight, stream-write records to a lightweight database or serialized binary file and read back as needed rather than keeping everything in heap.

Recommended Answers

All 4 Replies

Member Avatar for jwenting

You don't want to keep that much data in memory...

Use a SAX parser for large amounts of data, as it doesn't need to retain the data in between operations.

Member Avatar for ~s.o.s~

If you are looking for a particular piece of information in that XML file, maybe this would interest you. Also take a look at this.

Member Avatar for hussulinux

You could also try XALAN from apache. Its an open source parser. Available in Java and C++

Member Avatar for jwenting

Xalan won't solve his problems...
For one it's not an XML parser and also it won't limit his memory use to something reasonable...

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.