Loading ...
Sorry, an error occurred while loading the content.

ANTLR274: Small functional extension proposal: synonym

Expand Messages
  • Francis ANDRE
    Hi Terence & all I would like to propose you the following extension to manage simply synonyms. One could define in the tokens section definition some lexemes
    Message 1 of 7 , Sep 7, 2004
    • 0 Attachment
      Hi Terence & all

      I would like to propose you the following extension to manage simply
      synonyms. One could define in the tokens section definition some
      lexemes that abstract themself to a same token like :

      tokens {
      YES = "YES" | "Y" | "OK" | "YEAP" | "DA";
      }

      Since all literals above should returns the same YES token value,
      IMHO, it should be enough to generate in the Lexer something like:

      void Lexer::initLiterals()
      {
      literals["YES"] = 3;
      literals["Y"]= 3;
      literals["OK"] = 3;
      literals["YEAP"] 3;
      literals["DA"] = 3;
      }


      In fact, this will avoid to have a grammar rule that specifies all
      the alternatives to get a YES which is really the case!!


      What do you think about??

      Cheers

      FA
    • Terence Parr
      Hi. Well, that s pretty specialized. Further, all you have to do is add them to the hashtable of literals with the same token type and it ll just work. :)
      Message 2 of 7 , Sep 7, 2004
      • 0 Attachment
        Hi. Well, that's pretty specialized. Further, all you have to do is
        add them to the hashtable of literals with the same token type and
        it'll just work. :)

        Ter
        On Sep 7, 2004, at 4:51 AM, Francis ANDRE wrote:

        > Hi Terence & all
        >
        > I would like to propose you the following extension to manage simply
        > synonyms. One could define in the tokens section definition some
        > lexemes that abstract themself to a same token like :
        >
        > tokens {
        > YES = "YES" | "Y" | "OK" | "YEAP" | "DA";
        > }
        >
        > Since all literals above should returns the same YES token value,
        > IMHO, it should be enough to generate in the Lexer something like:
        >
        > void Lexer::initLiterals()
        > {
        > literals["YES"] = 3;
        > literals["Y"]= 3;
        > literals["OK"] = 3;
        > literals["YEAP"] 3;
        > literals["DA"] = 3;
        > }
        >
        >
        > In fact, this will avoid to have a grammar rule that specifies all
        > the alternatives to get a YES which is really the case!!
        >
        >
        > What do you think about??
        >
        > Cheers
        >
        > FA
        >
        >
        >
        >
        >
        >
        >
        >
        > Yahoo! Groups Links
        >
        >
        >
        >
        >
        >
        --
        CS Professor & Grad Director, University of San Francisco
        Creator, ANTLR Parser Generator, http://www.antlr.org
        Cofounder, http://www.jguru.com
        Cofounder, http://www.knowspam.net enjoy email again!
      • Francis ANDRE
        ... Yes, it is what I have done for now But the problem is the maintenance and the lisibility of a grammar maintained by a third party. IMHO, it would better
        Message 3 of 7 , Sep 7, 2004
        • 0 Attachment
          > Hi. Well, that's pretty specialized. Further, all you have to do is
          > add them to the hashtable of literals with the same token type and
          > it'll just work. :)

          Yes, it is what I have done for now

          But the problem is the maintenance and the lisibility of a grammar
          maintained by a third party. IMHO, it would better to express the fact that
          a token has more than one lexeme or textual representaiton in the grammar
          itself than in a additional derivative generated lexer. (Just for you to get
          my point, my grammar has more than 260 tokens with at least 20 synonyms)

          Make the evidence evident whenever possible is a quite good rule when
          programming!

          Regards

          FA

          PS: JavaCC is providing such synonym descriptions in the grammar

          ----- Original Message -----
          From: "Terence Parr" <parrt@...>
          To: <antlr-interest@yahoogroups.com>
          Sent: Tuesday, September 07, 2004 7:50 PM
          Subject: Re: [antlr-interest] ANTLR274: Small functional extension proposal:
          synonym


          > Hi. Well, that's pretty specialized. Further, all you have to do is
          > add them to the hashtable of literals with the same token type and
          > it'll just work. :)
          >
          > Ter
          > On Sep 7, 2004, at 4:51 AM, Francis ANDRE wrote:
          >
          > > Hi Terence & all
          > >
          > > I would like to propose you the following extension to manage simply
          > > synonyms. One could define in the tokens section definition some
          > > lexemes that abstract themself to a same token like :
          > >
          > > tokens {
          > > YES = "YES" | "Y" | "OK" | "YEAP" | "DA";
          > > }
          > >
          > > Since all literals above should returns the same YES token value,
          > > IMHO, it should be enough to generate in the Lexer something like:
          > >
          > > void Lexer::initLiterals()
          > > {
          > > literals["YES"] = 3;
          > > literals["Y"]= 3;
          > > literals["OK"] = 3;
          > > literals["YEAP"] 3;
          > > literals["DA"] = 3;
          > > }
          > >
          > >
          > > In fact, this will avoid to have a grammar rule that specifies all
          > > the alternatives to get a YES which is really the case!!
          > >
          > >
          > > What do you think about??
          > >
          > > Cheers
          > >
          > > FA
          > >
          > >
          > >
          > >
          > >
          > >
          > >
          > >
          > > Yahoo! Groups Links
          > >
          > >
          > >
          > >
          > >
          > >
          > --
          > CS Professor & Grad Director, University of San Francisco
          > Creator, ANTLR Parser Generator, http://www.antlr.org
          > Cofounder, http://www.jguru.com
          > Cofounder, http://www.knowspam.net enjoy email again!
          >
          >
          >
          >
          >
          >
          > Yahoo! Groups Links
          >
          >
          >
          >
          >
          >
        • Monty Zukowski
          Actually this comes up more than you might think. It wouldn t be that hard to do, would it? Of course, for now, you can do that manually. Monty ANTLR & Java
          Message 4 of 7 , Sep 7, 2004
          • 0 Attachment
            Actually this comes up more than you might think. It wouldn't be that
            hard to do, would it?

            Of course, for now, you can do that manually.

            Monty

            ANTLR & Java Consultant -- http://www.codetransform.com
            ANSI C/GCC transformation toolkit --
            http://www.codetransform.com/gcc.html
            Embrace the Decay -- http://www.codetransform.com/EmbraceDecay.html


            On Sep 7, 2004, at 10:50 AM, Terence Parr wrote:

            > Hi. Well, that's pretty specialized. Further, all you have to do is
            > add them to the hashtable of literals with the same token type and
            > it'll just work. :)
            >
            > Ter
            > On Sep 7, 2004, at 4:51 AM, Francis ANDRE wrote:
            >
            >> Hi Terence & all
            >>
            >> I would like to propose you the following extension to manage simply
            >> synonyms. One could define in the tokens section definition some
            >> lexemes that abstract themself to a same token like :
            >>
            >> tokens {
            >> YES = "YES" | "Y" | "OK" | "YEAP" | "DA";
            >> }
            >>
            >> Since all literals above should returns the same YES token value,
            >> IMHO, it should be enough to generate in the Lexer something like:
            >>
            >> void Lexer::initLiterals()
            >> {
            >> literals["YES"] = 3;
            >> literals["Y"]= 3;
            >> literals["OK"] = 3;
            >> literals["YEAP"] 3;
            >> literals["DA"] = 3;
            >> }
            >>
            >>
            >> In fact, this will avoid to have a grammar rule that specifies all
            >> the alternatives to get a YES which is really the case!!
            >>
            >>
            >> What do you think about??
            >>
            >> Cheers
            >>
            >> FA
            >>
            >>
            >>
            >>
            >>
            >>
            >>
            >>
            >> Yahoo! Groups Links
            >>
            >>
            >>
            >>
            >>
            >>
            > --
            > CS Professor & Grad Director, University of San Francisco
            > Creator, ANTLR Parser Generator, http://www.antlr.org
            > Cofounder, http://www.jguru.com
            > Cofounder, http://www.knowspam.net enjoy email again!
            >
            >
            >
            >
            >
            >
            > Yahoo! Groups Links
            >
            >
            >
            >
            >
            >
            >
          • Terence Parr
            ... Well, I normally argue that adding a feature (even if easy to do) should require lots of motivation. I ve never run into the problem except perhaps in
            Message 5 of 7 , Sep 7, 2004
            • 0 Attachment
              On Sep 7, 2004, at 12:29 PM, Monty Zukowski wrote:

              > Actually this comes up more than you might think. It wouldn't be that
              > hard to do, would it?

              Well, I normally argue that adding a feature (even if easy to do)
              should require lots of motivation. I've never run into the problem
              except perhaps in some minor way and I've done a few grammars. Can you
              give some more examples?

              Thanks,
              Ter
              --
              CS Professor & Grad Director, University of San Francisco
              Creator, ANTLR Parser Generator, http://www.antlr.org
              Cofounder, http://www.jguru.com
              Cofounder, http://www.knowspam.net enjoy email again!
            • lgcraymer
              Didn t dlg support this sort of thing via token sets? I m with Ter on this one, BTW--this sort of aliasing can be done in the parser (including setting the
              Message 6 of 7 , Sep 7, 2004
              • 0 Attachment
                Didn't dlg support this sort of thing via token sets? I'm with Ter on
                this one, BTW--this sort of aliasing can be done in the parser
                (including setting the token type, if that is desirable) easily and
                with minor effort in the lexer (check the ID tokens to see if they
                match any in the list and setting the token type if they do--that is
                expensive, but no more so than for an automated mechanism). The
                parser approach is the more efficient, so why support it in the lexer?

                --Loring


                --- In antlr-interest@yahoogroups.com, Terence Parr <parrt@c...> wrote:
                >
                > On Sep 7, 2004, at 12:29 PM, Monty Zukowski wrote:
                >
                > > Actually this comes up more than you might think. It wouldn't be that
                > > hard to do, would it?
                >
                > Well, I normally argue that adding a feature (even if easy to do)
                > should require lots of motivation. I've never run into the problem
                > except perhaps in some minor way and I've done a few grammars. Can you
                > give some more examples?
                >
                > Thanks,
                > Ter
                > --
                > CS Professor & Grad Director, University of San Francisco
                > Creator, ANTLR Parser Generator, http://www.antlr.org
                > Cofounder, http://www.jguru.com
                > Cofounder, http://www.knowspam.net enjoy email again!
              • Anakreon
                ... I agree with this opinion although I have never faced a case where synonimes are required. Synonimes is part of the language so it should be expressed in
                Message 7 of 7 , Sep 8, 2004
                • 0 Attachment
                  Francis ANDRE wrote:
                  >>Hi. Well, that's pretty specialized. Further, all you have to do is
                  >>add them to the hashtable of literals with the same token type and
                  >>it'll just work. :)
                  >
                  >
                  > Yes, it is what I have done for now
                  >
                  > But the problem is the maintenance and the lisibility of a grammar
                  > maintained by a third party. IMHO, it would better to express the fact that
                  > a token has more than one lexeme or textual representaiton in the grammar
                  > itself than in a additional derivative generated lexer. (Just for you to get
                  > my point, my grammar has more than 260 tokens with at least 20 synonyms)
                  >
                  > Make the evidence evident whenever possible is a quite good rule when
                  > programming!
                  >
                  I agree with this opinion although I have never faced a case where
                  synonimes are required. Synonimes is part of the language so it should
                  be expressed in antlr instead of doing it via action code.

                  Anakreon
                Your message has been successfully submitted and would be delivered to recipients shortly.