Loading ...
Sorry, an error occurred while loading the content.

RE: Help Lexer rules & states.

Expand Messages
  • Millaway, John
    ... I didn t actually run that code, it was just sample code. I should have said that up front, sorry. Yes you have to split the lexers into multiple files.
    Message 1 of 3 , Aug 8 1:05 PM
    • 0 Attachment
      > Hi John,
      >
      > Thank you for the reply, I tried to run your simple example through
      > Antlr but i complained about having more that one lexer in the same
      > file. Do you really have to separate them into individual
      > files, and if so
      > in what order do you create them with Antlr.

      I didn't actually run that code, it was just sample code. I should have said
      that up front, sorry.

      Yes you have to split the lexers into multiple files. Easier to maintain, in
      my opinion.

      In what order should you run them?
      Good question. If you notice the exportVocab and importVocab options, you'll
      see that the Parser "controls" the vocabulary. The PARSER EXPORTS the
      vocabulary, and the LEXERS IMPORT the vocabulary. In other words, generate
      the parser first, then generate the lexers in whatever order you want. The
      parser will generate the vocab based on the tokens it expects to find, and
      the lexers will pick up that vocab. There are other ways to do this,
      depending on your needs (and it's in the docs.) Also, I don't even know if
      I'm doing it the "right" way! But this is how I've been doing it for
      multiple lexers, and it's been working fine... Once you setup your makefile
      it's not a pain (just remember to clean the generated TokenType.* files!).

      -John
    • Millaway, John
      ... Wow, that could be disastrous. Thanks for clearing that up for me. In the lexer, are there conditions under which switching would cause trouble? In the
      Message 2 of 3 , Aug 9 7:28 AM
      • 0 Attachment
        > In general, you *can't* choose to let the parser operate the selector for
        > the lexer. The parser is always at least k tokens ahead of the lexer and
        > is possibly already to the end of the file due to "infinite lookahead" aka
        > syntactic predicates. Your lexer will be switched too late, it will
        > already be >=k tokens ahead of where you want the switch to happen. The
        > general rule in antlr is that you can't have any feedback from the parser
        > to the lexer. It is almost guaranteed to not work.
        >

        Wow, that could be disastrous. Thanks for clearing that up for me. In the
        lexer, are there conditions under which switching would cause trouble? In
        the middle of an alternative? In a protected rule?

        -John
      Your message has been successfully submitted and would be delivered to recipients shortly.