Loading ...
Sorry, an error occurred while loading the content.

Re: [antlr-interest] Error message

Expand Messages
  • Doug Erickson
    How much memory are you giving the JVM? It depends on your JDK, but the default is probably 64Mb, insufficient if you re trying to hold on to the whole 44Mb
    Message 1 of 2 , Jun 1, 2000
    • 0 Attachment
      How much memory are you giving the JVM? It depends on your JDK, but the
      default is probably 64Mb, insufficient if you're trying to hold on to
      the whole 44Mb file (that's probably 88Mb in memory for the text
      alone). Give it lots of memory, like -mx128m.

      Another thing to check is if you are using StringBuffer.setLength(0) and
      toString() a lot. In JDK 1.2.2 and earlier, there's a bug with this
      that creates Strings with a char array large enough to hold the biggest
      string ever seen by the StringBuffer. I replaced an ANTLR lexer with a
      hand-built one that used a StringBuffer to accumulate each token's text,
      and boy did this bug bite me! I had one character tokens that consumed
      over 1k apiece... Use ANTLRStringBuffer instead.

      Although this appears to be happening during lookahead, it seems like
      your lookahead would be a small fixed number, so I'm surprised if it's
      the CharQueue that is actually consuming all of the memory. Profile
      your parser with -Xrunhprof to find out quickly who's asking for all the
      memory.

      jcairns@... wrote:
      >
      > java.lang.OutOfMemoryError
      > at antlr.CharQueue.expand(Compiled Code)
      > at antlr.CharQueue.append(Compiled Code)
      > at antlr.ByteBuffer.fill(Compiled Code)
      > at antlr.InputBuffer.LA(Compiled Code)
      > at antlr.CharScanner.LA(Compiled Code)
      >
      > Would this error reflect a memory leak/problem on my end or within
      > antlr? I am trying to parse a large file (44Mb). I have read some
      > of the 'tricks' to deal with large files- but before I get too
      > involved with that I wanted to see if I had a problem with my parser.
      >
      > The error is the same for different files.
      >
      > Can I control this CharQueue?
      > Does this Queue just keep building throughout the parse - can I flush
      > it?
      > Any ideas why this is happening/ how to try to fix it would be
      > greatly
      > appreciated. Thank you.
      >
      > John
      >
      > ------------------------------------------------------------------------
      > Accurate impartial advice on everything from laptops to table saws.
      > http://click.egroups.com/1/4634/9/_/492272/_/959867612/
      > ------------------------------------------------------------------------
    • jcairns@netgenics.com
      Doug, Thanks for your response! As of right now I am bypassing this by parsing 100 records and then re-instantiating the parser parsing another 100. I forced
      Message 2 of 2 , Jun 2, 2000
      • 0 Attachment
        Doug,

        Thanks for your response! As of right now I am bypassing this by
        parsing 100 records and then re-instantiating the parser parsing
        another 100. I forced to use 1.1.8 which I think is limited to 64M.
        I actually don't want to hold on to the whole file. I can parse a
        record and then build and return an object separately/individually.
        I
        would like to avoid the overhead of re-instantiation since my stream
        could, in theory, be 30 times the size it is now since I only have a
        small chunk for testing purposes. Is there anyway to tell ANTLR not
        to hold on to the data and process the records as they come in (as if
        the next record was just like the first record coming in) ?

        Here is a dump from -prof in case it means something more than the
        obvious to someone- I have never used it before.

        count callee caller time
        439556 antlr/CharQueue.elementAt(I)C antlr/InputBuffer.LA(I)C 539
        439556 antlr/InputBuffer.LA(I)C antlr/CharScanner.LA(I)C 9276
        439556 antlr/ByteBuffer.fill(I)V antlr/InputBuffer.LA(I)C 4879
        439556 antlr/InputBuffer.syncConsume()V antlr/ByteBuffer.fill(I)V 623
        181831 antlr/CharScanner.LA(I)C Lexer.mSYMBOL_PRECEDENCE(Z)V 4384
        65534 antlr/CharQueue.elementAt(I)C antlr/CharQueue.expand()V 90
        57306 antlr/InputBuffer.consume()V antlr/CharScanner.consume()V 53
        57306 antlr/CharScanner.consume()V Lexer.consume()V 1604
        57306 antlr/CharScanner.getLine()I Lexer.consume()V 127
        37104 antlr/CharScanner.LA(I)C antlr/CharScanner.matchRange(CC)V 1254
        36928 java/io/BufferedInputStream.read()I
        java/io/FilterInputStream.read()I 246



        Have a great day!

        John





        --- In antlr-interest@egroups.com, Doug Erickson <doug.erickson@p...>
        wrote:
        > How much memory are you giving the JVM? It depends on your JDK,
        but
        the
        > default is probably 64Mb, insufficient if you're trying to hold on
        to
        > the whole 44Mb file (that's probably 88Mb in memory for the text
        > alone). Give it lots of memory, like -mx128m.
        >
        > Another thing to check is if you are using Strin
        gBuffer.setLength(0)
        and
        > toString() a lot. In JDK 1.2.2 and earlier, there's a bug with this
        > that creates Strings with a char array large enough to hold the
        biggest
        > string ever seen by the StringBuffer. I replaced an ANTLR lexer
        with a
        > hand-built one that used a StringBuffer to accumulate each token's
        text,
        > and boy did this bug bite me! I had one character tokens that
        consumed
        > over 1k apiece... Use ANTLRStringBuffer instead.
        >
        > Although this appears to be happening during lookahead, it seems
        like
        > your lookahead would be a small fixed number, so I'm surprised if

        > the CharQueue that is actually consuming all of the memory. Profile
        > your parser with -Xrunhprof to find out quickly who's asking for
        all
        the
        > memory.
        >
        > jcairns@n... wrote:
        > >
        > > java.lang.OutOfMemoryError
        > > at antlr.CharQueue.expand(Compiled Code)
        > > at antlr.CharQueue.append(Compiled Code)
        > > at antlr.ByteBuffer.fill(Compiled Code)
        > > at antlr.InputBuffer.LA(Compiled Code)
        > > at antlr.CharScanner.LA(Compiled Code)
        > >
        > > Would this error reflect a memory leak/problem on my end or within
        > > antlr? I am trying to parse a large file (44Mb). I have read
        some
        > > of the 'tricks' to deal with large files- but before I get too
        > > involved with that I wanted to see if I had a problem with my
        parser.
        > >
        > > The error is the same for different files.
        > >
        > > Can I control this CharQueue?
        > > Does this Queue just keep building throughout the parse - can I
        flush
        > > it?
        > > Any ideas why this is happening/ how to try to fix it would be
        > > greatly
        > > appreciated. Thank you.
        > >
        > > John
        > >
        > >
        ----------------------------------------------------------------------
        --
        > > Accurate impartial advice on everything from laptops to table
        saws.
        > > http://click.egroups.com/1/4634/9/_/492272/_/959867612/
        > >
        ----------------------------------------------------------------------
        --
      Your message has been successfully submitted and would be delivered to recipients shortly.