Loading ...
Sorry, an error occurred while loading the content.
 

Out of memory error while importing large xml files

Expand Messages
  • Ruchika Israni
    Hi all, I am using a xml which is 3.08 MB in size. I am running the transfer tool and am getting the following error. I read somewhere in the previous posts
    Message 1 of 3 , Jun 3, 2009
      Hi all,

      I am using a xml which is 3.08 MB in size. I am running the transfer tool and am getting the following error. I read somewhere in the previous posts that DOM parser creates the entire tree from the XML which poses a problem for large xml files. Is it possible to perhaps use the SAX parser in place of the DOM parser ?

      Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
      at org.apache.xerces.dom.DeferredDocumentImpl.getNodeObject(Unknown Sour
      ce)
      at org.apache.xerces.dom.DeferredDocumentImpl.synchronizeChildren(Unknow
      n Source)
      at org.apache.xerces.dom.DeferredElementNSImpl.synchronizeChildren(Unkno
      wn Source)
      at org.apache.xerces.dom.ParentNode.getFirstChild(Unknown Source)
      at org.xmlmiddleware.xmlutils.DOMNormalizer.getFirstChild(DOMNormalizer.
      java:123)
      at org.xmlmiddleware.xmldbms.DOMToDBMS.processChildren(DOMToDBMS.java:60
      5)
      at org.xmlmiddleware.xmldbms.DOMToDBMS.processClassRow(DOMToDBMS.java:54
      4)
      at org.xmlmiddleware.xmldbms.DOMToDBMS.processRow(DOMToDBMS.java:881)
      at org.xmlmiddleware.xmldbms.DOMToDBMS.processFKNodes(DOMToDBMS.java:946
      )
      at org.xmlmiddleware.xmldbms.DOMToDBMS.processClassRow(DOMToDBMS.java:56
      6)
      at org.xmlmiddleware.xmldbms.DOMToDBMS.processRoot(DOMToDBMS.java:447)
      at org.xmlmiddleware.xmldbms.DOMToDBMS.storeDocument(DOMToDBMS.java:368)

      at org.xmlmiddleware.xmldbms.DOMToDBMS.storeDocument(DOMToDBMS.java:317)

      at org.xmlmiddleware.xmldbms.tools.Transfer.storeDocumentInternal(Transf
      er.java:840)
      at org.xmlmiddleware.xmldbms.tools.Transfer.storeDocument(Transfer.java:
      479)
      at org.xmlmiddleware.xmldbms.tools.Transfer.dispatchStoreDocument(Transf
      er.java:696)
      at org.xmlmiddleware.xmldbms.tools.Transfer.dispatch(Transfer.java:434)
      at org.xmlmiddleware.xmldbms.tools.Transfer.main(Transfer.java:353)

      Many thanks in anticipation.
      Ruchika Israni
    • Ronald Bourret
      Hello, XML-DBMS does not support SAX because linear processing of documents is possible only in specific cases. (It is very easy to create an XML document
      Message 2 of 3 , Jun 3, 2009
        Hello,

        XML-DBMS does not support SAX because linear processing of documents is
        possible only in specific cases. (It is very easy to create an XML
        document whose data cannot be transferred to the database in a linear
        manner.)

        Because of this, you have several choices:

        1) Increase the amount of memory Java uses using the -Xmx switch on the
        java command. For details, see:

        http://java.sun.com/j2se/1.3/docs/tooldocs/solaris/java.html

        Because DOM trees are 5-10 times as large as XML documents, setting
        memory usage to 50MB should solve your problem. This is the easiest
        solution to your problem.

        2) Cut the document into smaller pieces with SAX and pass these pieces
        to XML-DBMS as DOM trees. This only works with certain types of
        documents. For details, see:

        http://www.rpbourret.com/xmldbms/faqs/large.htm

        3) Use a different tool. For example, you could use DataDirect XQuery,
        which can store XML documents in a relational database and also
        implements XQuery over relational data. Other options are HiT Allora and
        XQuare Bridge. (I don't know how well these products work with large XML
        documents.)

        -- Ron

        Ruchika Israni wrote:
        > Hi all,
        >
        > I am using a xml which is 3.08 MB in size. I am running the transfer tool and am getting the following error. I read somewhere in the previous posts that DOM parser creates the entire tree from the XML which poses a problem for large xml files. Is it possible to perhaps use the SAX parser in place of the DOM parser ?
        >
        > Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
        > at org.apache.xerces.dom.DeferredDocumentImpl.getNodeObject(Unknown Sour
        > ce)
        > at org.apache.xerces.dom.DeferredDocumentImpl.synchronizeChildren(Unknow
        > n Source)
        > at org.apache.xerces.dom.DeferredElementNSImpl.synchronizeChildren(Unkno
        > wn Source)
        > at org.apache.xerces.dom.ParentNode.getFirstChild(Unknown Source)
        > at org.xmlmiddleware.xmlutils.DOMNormalizer.getFirstChild(DOMNormalizer.
        > java:123)
        > at org.xmlmiddleware.xmldbms.DOMToDBMS.processChildren(DOMToDBMS.java:60
        > 5)
        > at org.xmlmiddleware.xmldbms.DOMToDBMS.processClassRow(DOMToDBMS.java:54
        > 4)
        > at org.xmlmiddleware.xmldbms.DOMToDBMS.processRow(DOMToDBMS.java:881)
        > at org.xmlmiddleware.xmldbms.DOMToDBMS.processFKNodes(DOMToDBMS.java:946
        > )
        > at org.xmlmiddleware.xmldbms.DOMToDBMS.processClassRow(DOMToDBMS.java:56
        > 6)
        > at org.xmlmiddleware.xmldbms.DOMToDBMS.processRoot(DOMToDBMS.java:447)
        > at org.xmlmiddleware.xmldbms.DOMToDBMS.storeDocument(DOMToDBMS.java:368)
        >
        > at org.xmlmiddleware.xmldbms.DOMToDBMS.storeDocument(DOMToDBMS.java:317)
        >
        > at org.xmlmiddleware.xmldbms.tools.Transfer.storeDocumentInternal(Transf
        > er.java:840)
        > at org.xmlmiddleware.xmldbms.tools.Transfer.storeDocument(Transfer.java:
        > 479)
        > at org.xmlmiddleware.xmldbms.tools.Transfer.dispatchStoreDocument(Transf
        > er.java:696)
        > at org.xmlmiddleware.xmldbms.tools.Transfer.dispatch(Transfer.java:434)
        > at org.xmlmiddleware.xmldbms.tools.Transfer.main(Transfer.java:353)
        >
        > Many thanks in anticipation.
        > Ruchika Israni
      • Ruchika Israni
        Thank You Ron. I included -Xmx option into the command and it worked. Thanks once again. Ruchika Israni
        Message 3 of 3 , Jun 3, 2009
          Thank You Ron. I included -Xmx option into the command and it worked.

          Thanks once again.

          Ruchika Israni


          --- In xml-dbms@yahoogroups.com, Ronald Bourret <rpbourret@...> wrote:
          >
          > Hello,
          >
          > XML-DBMS does not support SAX because linear processing of documents is
          > possible only in specific cases. (It is very easy to create an XML
          > document whose data cannot be transferred to the database in a linear
          > manner.)
          >
          > Because of this, you have several choices:
          >
          > 1) Increase the amount of memory Java uses using the -Xmx switch on the
          > java command. For details, see:
          >
          > http://java.sun.com/j2se/1.3/docs/tooldocs/solaris/java.html
          >
          > Because DOM trees are 5-10 times as large as XML documents, setting
          > memory usage to 50MB should solve your problem. This is the easiest
          > solution to your problem.
          >
          > 2) Cut the document into smaller pieces with SAX and pass these pieces
          > to XML-DBMS as DOM trees. This only works with certain types of
          > documents. For details, see:
          >
          > http://www.rpbourret.com/xmldbms/faqs/large.htm
          >
          > 3) Use a different tool. For example, you could use DataDirect XQuery,
          > which can store XML documents in a relational database and also
          > implements XQuery over relational data. Other options are HiT Allora and
          > XQuare Bridge. (I don't know how well these products work with large XML
          > documents.)
          >
          > -- Ron
          >
          > Ruchika Israni wrote:
          > > Hi all,
          > >
          > > I am using a xml which is 3.08 MB in size. I am running the transfer tool and am getting the following error. I read somewhere in the previous posts that DOM parser creates the entire tree from the XML which poses a problem for large xml files. Is it possible to perhaps use the SAX parser in place of the DOM parser ?
          > >
          > > Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
          > > at org.apache.xerces.dom.DeferredDocumentImpl.getNodeObject(Unknown Sour
          > > ce)
          > > at org.apache.xerces.dom.DeferredDocumentImpl.synchronizeChildren(Unknow
          > > n Source)
          > > at org.apache.xerces.dom.DeferredElementNSImpl.synchronizeChildren(Unkno
          > > wn Source)
          > > at org.apache.xerces.dom.ParentNode.getFirstChild(Unknown Source)
          > > at org.xmlmiddleware.xmlutils.DOMNormalizer.getFirstChild(DOMNormalizer.
          > > java:123)
          > > at org.xmlmiddleware.xmldbms.DOMToDBMS.processChildren(DOMToDBMS.java:60
          > > 5)
          > > at org.xmlmiddleware.xmldbms.DOMToDBMS.processClassRow(DOMToDBMS.java:54
          > > 4)
          > > at org.xmlmiddleware.xmldbms.DOMToDBMS.processRow(DOMToDBMS.java:881)
          > > at org.xmlmiddleware.xmldbms.DOMToDBMS.processFKNodes(DOMToDBMS.java:946
          > > )
          > > at org.xmlmiddleware.xmldbms.DOMToDBMS.processClassRow(DOMToDBMS.java:56
          > > 6)
          > > at org.xmlmiddleware.xmldbms.DOMToDBMS.processRoot(DOMToDBMS.java:447)
          > > at org.xmlmiddleware.xmldbms.DOMToDBMS.storeDocument(DOMToDBMS.java:368)
          > >
          > > at org.xmlmiddleware.xmldbms.DOMToDBMS.storeDocument(DOMToDBMS.java:317)
          > >
          > > at org.xmlmiddleware.xmldbms.tools.Transfer.storeDocumentInternal(Transf
          > > er.java:840)
          > > at org.xmlmiddleware.xmldbms.tools.Transfer.storeDocument(Transfer.java:
          > > 479)
          > > at org.xmlmiddleware.xmldbms.tools.Transfer.dispatchStoreDocument(Transf
          > > er.java:696)
          > > at org.xmlmiddleware.xmldbms.tools.Transfer.dispatch(Transfer.java:434)
          > > at org.xmlmiddleware.xmldbms.tools.Transfer.main(Transfer.java:353)
          > >
          > > Many thanks in anticipation.
          > > Ruchika Israni
          >
        Your message has been successfully submitted and would be delivered to recipients shortly.