Loading ...
Sorry, an error occurred while loading the content.

Re: FW: token's distance in bytes from the b eg

Expand Messages
  • Berhe, Daniel
    Your are right. Thanks a lot. Daniel ... From: mzukowski@bco.com [mailto:mzukowski@bco.com] Sent: 30 April 1999 18:00 To: antlr-interest@onelist.com Subject:
    Message 1 of 3 , Apr 30, 1999
    • 0 Attachment
      Your are right.
      Thanks a lot.
      Daniel

      -----Original Message-----
      From: mzukowski@... [mailto:mzukowski@...]
      Sent: 30 April 1999 18:00
      To: antlr-interest@onelist.com
      Subject: [antlr-interest] Re: FW: token's distance in bytes from the b
      eg


      From: mzukowski@...

      Sounds like you need to cast the token to CharPosToken. The compiler is
      assuming you are using antlr.Token.

      ((CharPosToken) oct).getColumn()

      MOnty

      > -----Original Message-----
      > From: Berhe, Daniel [SMTP:Daniel.Berhe@...]
      > Sent: Friday, April 30, 1999 8:35 AM
      > To: 'mzukowski@...'
      > Cc: 'antlr-interest@onelist.com'
      > Subject: [antlr-interest] Re: FW: token's distance in bytes from the
      > b eg
      >
      > From: "Berhe, Daniel" <Daniel.Berhe@...>
      >
      > Thanks.
      > I am doing a parser for a language used by (our)
      > printers.
      > IS THIS A BUG ?
      > I noticed the overridden makeToken() rule but
      > if I called for ex. getColumn() from the token received
      > by the parser inside the parser I get the following error
      > "Method getColumn() not found in class antlr.Token".
      >
      > Example of the code.
      >
      >
      > _____
      >
      >
      > ///// Start of Grammer my.g
      > class AntlrTagParser extends parser;
      > ...
      > oct:OCTALNUMBER
      > {
      > // here no error with getText()
      > // but with getColumn() there is error.
      > System.out.println(oct.getText() + oct.getColumn());
      > }
      > ...
      > ;
      > class InitalTestLexer extends Lexer;
      > OCTALNUMBER
      > :( '\060'..'\067'
      > ( '\060'..'\067'
      > ('\060'..'\067')?
      > )?
      > )
      > ;
      > //// end of Grammer my.g
      >
      >
      > _____
      >
      >
      > //// CharPosToken.java no change from what you send me
      > class CharPosToken extends CommonToken
      > {
      > ...
      > }
      >
      > _____
      >
      >
      > /// AntlrTagLexer.java same as yours AntlrTagLexer except for the
      > superclass
      > class AntlrTagLexer extends InitalTestLexer // note that the superclass
      > InitalTestLexer is
      > { // generated by my.g
      > ....
      > }
      >
      > _____
      >
      >
      > /// Main.java
      > class Main
      > {
      > ...
      > AntlrTagLexer myLexer = new AntlrTagLexer(inputfile);
      > AntlrTagParser myParser = new AntlrTagParser(myLexer);
      > ...
      > }
      >
      > _____
      >
      >
      > When I commented the action getColumn in the rule OCTALNUMBER in the
      > parser
      > above
      > compilation is ok. And When I run it I see that the makeToken of the class
      > AntlrTagLexer.java is executed not the default one.
      >
      > THUS my question Can this be a bug ?
      >
      > Daniel
      >
      > -----Original Message-----
      > From: mzukowski@... [ mailto:mzukowski@...
      > <mailto:mzukowski@...> ]
      > Sent: 30 April 1999 16:49
      > To: antlr-interest@onelist.com
      > Subject: [antlr-interest] Re: FW: token's distance in bytes from the b
      > eg
      >
      >
      > From: mzukowski@...
      >
      > You will notice the overridden makeToken() rule which puts those values
      > into
      > the Token passed to the parser. That's why I needed the subclass of
      > CommonToken--by default it doesn't remember column or charPos. If you are
      > building trees, your tree nodes will be passed the Token to initialize
      > themselves and you can then extract that information into the tree node.
      >
      > I'm interested to hear what you are building a parser for.
      >
      > Monty
      >
      > > -----Original Message-----
      > > From:Berhe, Daniel [SMTP:Daniel.Berhe@...]
      > > Sent:Friday, April 30, 1999 2:21 AM
      > > To: 'mzukowski@...'
      > > Cc: 'antlr-interest@onelist.com'
      > > Subject: [antlr-interest] Re: FW: token's distance in bytes from
      > the
      > > b egi
      > >
      > > From: "Berhe, Daniel" <Daniel.Berhe@...>
      > >
      > > Nice idea.Thanks.
      > > One more question :
      > > The column number, line number are known inside the Lexer. How
      > > can I propagate this values to the parser ?
      > > Thanks,
      > > Daniel
      > >
      > > -----Original Message-----
      > > From: mzukowski@... [ mailto:mzukowski@...
      > <mailto:mzukowski@...> ]
      > > Sent: 29 April 1999 17:33
      > > To: antlr-interest@onelist.com
      > > Subject: [antlr-interest] Re: FW: token's distance in bytes from the
      > > begi
      > >
      > >
      > > From: mzukowski@...
      > >
      > > I did that just the other day to generate an ETAGS file for .g files.
      > >
      > > > -----Original Message-----
      > > > From: Berhe, Daniel [SMTP:Daniel.Berhe@...]
      > > > Sent: Thursday, April 29, 1999 7:12 AM
      > > > To:'antlr'; 'antlr-interest@onelist.com'
      > > > Subject: [antlr-interest] FW: token's distance in bytes from the
      > > > beginning of the file and its line number from the beginning of the
      > > file
      > > >
      > > > From: "Berhe, Daniel" <Daniel.Berhe@...>
      > > >
      > > > > Hi all,
      > > > > Does some one know how I can get the token's (that has been passed
      > > > > from the lexer to the parser)
      > > > > distance in bytes from the beginning of the file and its line number
      > > > from
      > > > > the beginning of the file using Antlr (Java) ?
      > > > > Many thanks.
      > > > > Daniel
      > > >
      > > You will need to override some methods in the lexer. Since I wanted to
      > > leverage ANTLR's lexer I subclassed it directly. This has some 2.6
      > > specific
      > > names which is easy to fix for older versions. This code makes sure
      > that
      > > the distance is to the first byte of the token, not the last.
      > >
      > > AntlrTagLexer.java:
      > > import antlr.*;
      > > import java.io.*;
      > > import CharPosToken;
      > > class AntlrTagLexer extends ANTLRLexer {
      > > protected int col = 1, tokenStartCol = 1, tokenStartLine = 1,
      > > tokenStartCharPos = 1, charPos = 1;
      > >
      > > public AntlrTagLexer(DataInputStream ib) {
      > > super(ib);
      > > }
      > >
      > > public void consume() throws IOException {
      > > super.consume();
      > > if ( inputState.guessing == 0 ) {
      > > col++;
      > > charPos++;
      > > }
      > > }
      > >
      > > public void newline() {
      > > super.newline(); // increments inputState.line
      > > col = 1;
      > > }
      > >
      > > protected Token makeToken(int t) {
      > > CharPosToken tok = new CharPosToken(t);
      > > tok.setColumn(tokenStartCol);
      > > tok.setLine(tokenStartLine);
      > > tok.setCharPos( tokenStartCharPos );
      > > return tok;
      > > }
      > >
      > > public void resetText() {
      > > super.resetText();
      > > tokenStartCol = col;
      > > tokenStartLine = getLine();
      > > tokenStartCharPos = charPos;
      > > }
      > > }
      > >
      > >
      > >
      > >
      > >
      > > CharPosToken.java:
      > > import antlr.CommonToken;
      > >
      > > class CharPosToken extends CommonToken {
      > > int col = 0;
      > > int charPos = 0;
      > >
      > > public CharPosToken() { super(); }
      > > public CharPosToken(int t) { setType(t); }
      > > public CharPosToken(int t, String txt) { super(t, txt); }
      > >
      > >
      > > public void setColumn( int c ) {
      > > col = c;
      > > }
      > > public int getColumn() {
      > > return col;
      > > }
      > >
      > > public void setCharPos( int c ) {
      > > charPos = c;
      > > }
      > >
      > > public int getCharPos() {
      > > return charPos;
      > > }
      > > }
      > >
      > >
      > > ------------------------------------------------------------------------
      > > Looking for an efficient, fun way to stay in touch with family members?
      > > http://www.onelist.com <http://www.onelist.com>
      > > Create a ONElist community just for your family!
      > >
      > > ------------------------------------------------------------------------
      > > Looking for an easy, effective way to research an important topic?
      > > http://www.onelist.com <http://www.onelist.com>
      > > Joining a ONElist community is your answer.
      >
      > ------------------------------------------------------------------------
      > Did you know that ONElist now offers the richest set of
      > group communications tools on the Internet?
      > http://www.onelist.com <http://www.onelist.com>
      > Check out our homepage for details on these new tools!
      >
      >
      >
      > ------------------------------------------------------------------------
      > Start a new hobby. Meet a new friend.
      > http://www.onelist.com
      > ONElist: The leading provider of free e-mail list services!

      ------------------------------------------------------------------------
      Looking to expand your world?
      http://www.onelist.com
      ONElist has over 130,000 e-mail communities from which to chose!
    Your message has been successfully submitted and would be delivered to recipients shortly.