Loading ...
Sorry, an error occurred while loading the content.

3860Re: [xml-dbms] Out of memory error while importing large xml files

Expand Messages
  • Ronald Bourret
    Jun 3, 2009
      Hello,

      XML-DBMS does not support SAX because linear processing of documents is
      possible only in specific cases. (It is very easy to create an XML
      document whose data cannot be transferred to the database in a linear
      manner.)

      Because of this, you have several choices:

      1) Increase the amount of memory Java uses using the -Xmx switch on the
      java command. For details, see:

      http://java.sun.com/j2se/1.3/docs/tooldocs/solaris/java.html

      Because DOM trees are 5-10 times as large as XML documents, setting
      memory usage to 50MB should solve your problem. This is the easiest
      solution to your problem.

      2) Cut the document into smaller pieces with SAX and pass these pieces
      to XML-DBMS as DOM trees. This only works with certain types of
      documents. For details, see:

      http://www.rpbourret.com/xmldbms/faqs/large.htm

      3) Use a different tool. For example, you could use DataDirect XQuery,
      which can store XML documents in a relational database and also
      implements XQuery over relational data. Other options are HiT Allora and
      XQuare Bridge. (I don't know how well these products work with large XML
      documents.)

      -- Ron

      Ruchika Israni wrote:
      > Hi all,
      >
      > I am using a xml which is 3.08 MB in size. I am running the transfer tool and am getting the following error. I read somewhere in the previous posts that DOM parser creates the entire tree from the XML which poses a problem for large xml files. Is it possible to perhaps use the SAX parser in place of the DOM parser ?
      >
      > Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
      > at org.apache.xerces.dom.DeferredDocumentImpl.getNodeObject(Unknown Sour
      > ce)
      > at org.apache.xerces.dom.DeferredDocumentImpl.synchronizeChildren(Unknow
      > n Source)
      > at org.apache.xerces.dom.DeferredElementNSImpl.synchronizeChildren(Unkno
      > wn Source)
      > at org.apache.xerces.dom.ParentNode.getFirstChild(Unknown Source)
      > at org.xmlmiddleware.xmlutils.DOMNormalizer.getFirstChild(DOMNormalizer.
      > java:123)
      > at org.xmlmiddleware.xmldbms.DOMToDBMS.processChildren(DOMToDBMS.java:60
      > 5)
      > at org.xmlmiddleware.xmldbms.DOMToDBMS.processClassRow(DOMToDBMS.java:54
      > 4)
      > at org.xmlmiddleware.xmldbms.DOMToDBMS.processRow(DOMToDBMS.java:881)
      > at org.xmlmiddleware.xmldbms.DOMToDBMS.processFKNodes(DOMToDBMS.java:946
      > )
      > at org.xmlmiddleware.xmldbms.DOMToDBMS.processClassRow(DOMToDBMS.java:56
      > 6)
      > at org.xmlmiddleware.xmldbms.DOMToDBMS.processRoot(DOMToDBMS.java:447)
      > at org.xmlmiddleware.xmldbms.DOMToDBMS.storeDocument(DOMToDBMS.java:368)
      >
      > at org.xmlmiddleware.xmldbms.DOMToDBMS.storeDocument(DOMToDBMS.java:317)
      >
      > at org.xmlmiddleware.xmldbms.tools.Transfer.storeDocumentInternal(Transf
      > er.java:840)
      > at org.xmlmiddleware.xmldbms.tools.Transfer.storeDocument(Transfer.java:
      > 479)
      > at org.xmlmiddleware.xmldbms.tools.Transfer.dispatchStoreDocument(Transf
      > er.java:696)
      > at org.xmlmiddleware.xmldbms.tools.Transfer.dispatch(Transfer.java:434)
      > at org.xmlmiddleware.xmldbms.tools.Transfer.main(Transfer.java:353)
      >
      > Many thanks in anticipation.
      > Ruchika Israni
    • Show all 3 messages in this topic