Loading ...
Sorry, an error occurred while loading the content.

RE: [antlr-interest] Build Question

Expand Messages
  • mzukowski@bco.com
    One way to accomplish what you wish would be to do some preprocessing. I m particularly fond of the noweb literate programming tool, which I used to build
    Message 1 of 11 , Mar 12, 2002
    • 0 Attachment
      One way to accomplish what you wish would be to do some preprocessing. I'm
      particularly fond of the "noweb" literate programming tool, which I used to
      build the GCC grammar. It allowed me to reuse grammar fragments and then
      add my own actions. More importantly for me it allowed me to keep related
      rules in the same area of the noweb file, so my declaration rules for the
      ANSI C Parser, GCC Parser, GCC tree walker, GCC tree emitter were all
      together, and while debugging when I modified one rule I was right there to
      make sure the other rules were changed in sync.

      Antlr subclassing is most useful when there are no actions to subclass. So
      I use it for a tree grammar that I subclass with actions for specific passes
      of a translator. Subclassing a grammar with actions would be almost
      useless. In fact I would reorganize my grammars as a literate program so I
      could reuse parts as I wish, it ends up being more flexible than antlr's
      subclassing.

      Monty

      > -----Original Message-----
      > From: Tiller, Michael (M.M.) [mailto:mtiller@...]
      > Sent: Tuesday, March 12, 2002 6:48 AM
      > To: 'antlr-interest@yahoogroups.com'
      > Subject: RE: [antlr-interest] Build Question
      >
      >
      > A few followup comments...
      >
      > I was hoping to support both Java and C++ for future
      > development. It seems like this is impractical for the
      > following reasons:
      >
      > 1) Trying to write a language independent way comment rule
      > that "eats" comments doesn't seem possible.
      > 2) Perhaps for misguided reasons, I'd like to be able to add
      > "child" nodes to my AST during tree construction. It isn't
      > clear to me how that can be done in a language neutral way.
      > It would be nice if ANTLR's tree construction shorthand were
      > extended to allow this.
      > 3) The target language has to be specified in the grammar
      > (couldn't this be a command line option?)
      > 4) The difficulties with trying to do grammar inheritance (see below).
      >
      > This isn't really criticism (ANTLR is a great tool).
      > Consider it ideas for future releases. Because of these
      > problems, I think I'm going to have to abandon my hope of a
      > language neutral grammar and just start plugging in my C++ code.
      >
      > --
      > Mike
      >
      > > -----Original Message-----
      > > From: Ric Klaren [mailto:klaren@...]
      > > Sent: Tuesday, March 12, 2002 3:51 AM
      > > To: antlr-interest@yahoogroups.com
      > > Subject: Re: [antlr-interest] Build Question
      > >
      > >
      > > On Mon, Mar 11, 2002 at 01:03:27PM -0500, Tiller, Michael
      > > (M.M.) wrote:
      > > > So far so good I imagine (although I haven't gotten far
      > > enough to figure
      > > > out if this is a good idea yet). The problem I'm running
      > > into is that I
      > > > imagine these grammars may be shared among several distinct
      > > applications.
      > > > For that reason, I keep the ".g" files away from everything
      > > else. So when
      > > > I run ANTLR, the Makefile rule looks like this:
      > > >
      > > > $(ANTLR) -glib $(ROOTDIR)/base.g -o $(GENDIR)
      > > $(ROOTDIR)/base_java.g
      > > >
      > > > where GENDIR is the location for the generated code and
      > > ROOTDIR is where I
      > > > keep the .g files. This way, I can generate the set of
      > > source files for
      > > > each project based on the grammars.
      > > >
      > > > The problem is that when you use grammar inheritance, ANTLR
      > > expands the
      > > > complete grammar into "expandedbase_java.g". Except that
      > > it actually names
      > > > it "expanded$(ROOTDIR)/base_java.g" (where ROOTDIR is
      > > actually expanded).
      > > > In other words, it assumes that the ".g" is in the current
      > > directory.
      > > >
      > > > Any suggestions about how to avoid this problem? I'm
      > > surprised that I
      > > > can't explicitly specify the name of the expanded grammar.
      > > That would have
      > > > been a nice option.
      > >
      > > Been there done that. Doesn't work. It needs some serious
      > > fixing in antlr
      > > to make this behave well. I already looked at it when I ran
      > > into it, it's
      > > not completely easy to fix in antlr. I currently copy .g
      > > files around with
      > > Make rules to get the right behaviour.
      > >
      > > I personally must say that I found antlr's grammar
      > inheritance pretty
      > > useless (gives to little control over what you want to
      > > override, in the end
      > > you end up copying action code in a complicated way so I
      > > figured I'd better
      > > copy stuff around in an easy way). YMMV though I suspect some
      > > ppl on the
      > > list may have gotten nice results with it.
      > >
      > > Ric
      > > --
      > > -----+++++****************************************************
      > > *+++++++++-------
      > > ---- Ric Klaren ----- klaren@... ----- +31 53
      > > 4893722 ----
      > > -----+++++****************************************************
      > > *+++++++++-------
      > > Time what is time - I wish I knew how to tell You why - It
      > > hurts to know -
      > > Aren't we machines - Time what is time - Unlock the door
      > > - And see the truth - Then time is time again
      > > From: 'Time what is Time' by Blind Guardian
      > >
      > >
      > >
      > >
      > > Your use of Yahoo! Groups is subject to
      > > http://docs.yahoo.com/info/terms/
      > >
      > >
      >
      >
      >
      > Your use of Yahoo! Groups is subject to
      http://docs.yahoo.com/info/terms/
    • Tiller, Michael (M.M.)
      Could you post an example of how this is done? -- Mike
      Message 2 of 11 , Mar 12, 2002
      • 0 Attachment
        Could you post an example of how this is done?

        --
        Mike

        > -----Original Message-----
        > From: mzukowski@... [mailto:mzukowski@...]
        > Sent: Tuesday, March 12, 2002 10:08 AM
        > To: antlr-interest@yahoogroups.com
        > Subject: RE: [antlr-interest] Build Question
        >
        >
        > One way to accomplish what you wish would be to do some
        > preprocessing. I'm
        > particularly fond of the "noweb" literate programming tool,
        > which I used to
        > build the GCC grammar. It allowed me to reuse grammar
        > fragments and then
        > add my own actions. More importantly for me it allowed me to
        > keep related
        > rules in the same area of the noweb file, so my declaration
        > rules for the
        > ANSI C Parser, GCC Parser, GCC tree walker, GCC tree emitter were all
        > together, and while debugging when I modified one rule I was
        > right there to
        > make sure the other rules were changed in sync.
        >
        > Antlr subclassing is most useful when there are no actions to
        > subclass. So
        > I use it for a tree grammar that I subclass with actions for
        > specific passes
        > of a translator. Subclassing a grammar with actions would be almost
        > useless. In fact I would reorganize my grammars as a
        > literate program so I
        > could reuse parts as I wish, it ends up being more flexible
        > than antlr's
        > subclassing.
        >
        > Monty
        >
        > > -----Original Message-----
        > > From: Tiller, Michael (M.M.) [mailto:mtiller@...]
        > > Sent: Tuesday, March 12, 2002 6:48 AM
        > > To: 'antlr-interest@yahoogroups.com'
        > > Subject: RE: [antlr-interest] Build Question
        > >
        > >
        > > A few followup comments...
        > >
        > > I was hoping to support both Java and C++ for future
        > > development. It seems like this is impractical for the
        > > following reasons:
        > >
        > > 1) Trying to write a language independent way comment rule
        > > that "eats" comments doesn't seem possible.
        > > 2) Perhaps for misguided reasons, I'd like to be able to add
        > > "child" nodes to my AST during tree construction. It isn't
        > > clear to me how that can be done in a language neutral way.
        > > It would be nice if ANTLR's tree construction shorthand were
        > > extended to allow this.
        > > 3) The target language has to be specified in the grammar
        > > (couldn't this be a command line option?)
        > > 4) The difficulties with trying to do grammar inheritance
        > (see below).
        > >
        > > This isn't really criticism (ANTLR is a great tool).
        > > Consider it ideas for future releases. Because of these
        > > problems, I think I'm going to have to abandon my hope of a
        > > language neutral grammar and just start plugging in my C++ code.
        > >
        > > --
        > > Mike
        > >
        > > > -----Original Message-----
        > > > From: Ric Klaren [mailto:klaren@...]
        > > > Sent: Tuesday, March 12, 2002 3:51 AM
        > > > To: antlr-interest@yahoogroups.com
        > > > Subject: Re: [antlr-interest] Build Question
        > > >
        > > >
        > > > On Mon, Mar 11, 2002 at 01:03:27PM -0500, Tiller, Michael
        > > > (M.M.) wrote:
        > > > > So far so good I imagine (although I haven't gotten far
        > > > enough to figure
        > > > > out if this is a good idea yet). The problem I'm running
        > > > into is that I
        > > > > imagine these grammars may be shared among several distinct
        > > > applications.
        > > > > For that reason, I keep the ".g" files away from everything
        > > > else. So when
        > > > > I run ANTLR, the Makefile rule looks like this:
        > > > >
        > > > > $(ANTLR) -glib $(ROOTDIR)/base.g -o $(GENDIR)
        > > > $(ROOTDIR)/base_java.g
        > > > >
        > > > > where GENDIR is the location for the generated code and
        > > > ROOTDIR is where I
        > > > > keep the .g files. This way, I can generate the set of
        > > > source files for
        > > > > each project based on the grammars.
        > > > >
        > > > > The problem is that when you use grammar inheritance, ANTLR
        > > > expands the
        > > > > complete grammar into "expandedbase_java.g". Except that
        > > > it actually names
        > > > > it "expanded$(ROOTDIR)/base_java.g" (where ROOTDIR is
        > > > actually expanded).
        > > > > In other words, it assumes that the ".g" is in the current
        > > > directory.
        > > > >
        > > > > Any suggestions about how to avoid this problem? I'm
        > > > surprised that I
        > > > > can't explicitly specify the name of the expanded grammar.
        > > > That would have
        > > > > been a nice option.
        > > >
        > > > Been there done that. Doesn't work. It needs some serious
        > > > fixing in antlr
        > > > to make this behave well. I already looked at it when I ran
        > > > into it, it's
        > > > not completely easy to fix in antlr. I currently copy .g
        > > > files around with
        > > > Make rules to get the right behaviour.
        > > >
        > > > I personally must say that I found antlr's grammar
        > > inheritance pretty
        > > > useless (gives to little control over what you want to
        > > > override, in the end
        > > > you end up copying action code in a complicated way so I
        > > > figured I'd better
        > > > copy stuff around in an easy way). YMMV though I suspect some
        > > > ppl on the
        > > > list may have gotten nice results with it.
        > > >
        > > > Ric
        > > > --
        > > > -----+++++****************************************************
        > > > *+++++++++-------
        > > > ---- Ric Klaren ----- klaren@... ----- +31 53
        > > > 4893722 ----
        > > > -----+++++****************************************************
        > > > *+++++++++-------
        > > > Time what is time - I wish I knew how to tell You why - It
        > > > hurts to know -
        > > > Aren't we machines - Time what is time - Unlock the door
        > > > - And see the truth - Then time is time again
        > > > From: 'Time what is Time' by Blind Guardian
        > > >
        > > >
        > > >
        > > >
        > > > Your use of Yahoo! Groups is subject to
        > > > http://docs.yahoo.com/info/terms/
        > > >
        > > >
        > >
        > >
        > >
        > > Your use of Yahoo! Groups is subject to
        > http://docs.yahoo.com/info/terms/
        >
        >
        >
        >
        >
        > Your use of Yahoo! Groups is subject to
        > http://docs.yahoo.com/info/terms/
        >
        >
      • Ric Klaren
        Hi, ... In current antlr you can plainly forget it to write language independent stuff without reverting to hackery. ... Maybe the fabled new AST construction
        Message 3 of 11 , Mar 12, 2002
        • 0 Attachment
          Hi,

          On Tue, Mar 12, 2002 at 09:47:59AM -0500, Tiller, Michael (M.M.) wrote:
          > I was hoping to support both Java and C++ for future development. It seems
          > like this is impractical for the following reasons:
          >
          > 1) Trying to write a language independent way comment rule that "eats"
          > comments doesn't seem possible.

          In current antlr you can plainly forget it to write language independent
          stuff without reverting to hackery.

          > 2) Perhaps for misguided reasons, I'd like to be able to add "child" nodes
          > to my AST during tree construction. It isn't clear to me how that can be
          > done in a language neutral way. It would be nice if ANTLR's tree
          > construction shorthand were extended to allow this.

          Maybe the fabled new AST construction stuff will make this possible.

          > 3) The target language has to be specified in the grammar (couldn't this be
          > a command line option?)

          You never talked to Ter about commandline options did you ;) There's more
          of these things I personally would like to see as commandline options,
          #line generation options stuff like that. Antlr is because of this quite
          'make' unfriendly.

          Also building different configurations without getting dirty is because of
          this a problem. I'd love commandline options. Terence refuses (I'm told a
          pccts trauma ;) ).

          > 4) The difficulties with trying to do grammar inheritance (see below).

          > This isn't really criticism (ANTLR is a great tool). Consider it ideas for
          > future releases.

          ANTLR's pretty great, but it isn't perfect..

          Cheers,

          Ric
          --
          -----+++++*****************************************************+++++++++-------
          ---- Ric Klaren ----- klaren@... ----- +31 53 4893722 ----
          -----+++++*****************************************************+++++++++-------
          Why don't we just invite them to dinner and massacre them all when they're
          drunk? You heard the man. There's seven hundred thousand of them. Ah? ..
          So it'd have to be something simple with pasta, then.
          From: Interesting Times by Terry Pratchet
        • mzukowski@bco.com
          Yes, I will write up a simple example in the next few days, as I m swamped right now. In the meantime I can send you the noweb file I used for the gcc stuff
          Message 4 of 11 , Mar 12, 2002
          • 0 Attachment
            Yes, I will write up a simple example in the next few days, as I'm swamped
            right now. In the meantime I can send you the noweb file I used for the gcc
            stuff if you like.

            Monty

            > -----Original Message-----
            > From: Tiller, Michael (M.M.) [mailto:mtiller@...]
            > Sent: Tuesday, March 12, 2002 7:13 AM
            > To: 'antlr-interest@yahoogroups.com'
            > Subject: RE: [antlr-interest] Build Question
            >
            >
            > Could you post an example of how this is done?
            >
            > --
            > Mike
            >
            > > -----Original Message-----
            > > From: mzukowski@... [mailto:mzukowski@...]
            > > Sent: Tuesday, March 12, 2002 10:08 AM
            > > To: antlr-interest@yahoogroups.com
            > > Subject: RE: [antlr-interest] Build Question
            > >
            > >
            > > One way to accomplish what you wish would be to do some
            > > preprocessing. I'm
            > > particularly fond of the "noweb" literate programming tool,
            > > which I used to
            > > build the GCC grammar. It allowed me to reuse grammar
            > > fragments and then
            > > add my own actions. More importantly for me it allowed me to
            > > keep related
            > > rules in the same area of the noweb file, so my declaration
            > > rules for the
            > > ANSI C Parser, GCC Parser, GCC tree walker, GCC tree
            > emitter were all
            > > together, and while debugging when I modified one rule I was
            > > right there to
            > > make sure the other rules were changed in sync.
            > >
            > > Antlr subclassing is most useful when there are no actions to
            > > subclass. So
            > > I use it for a tree grammar that I subclass with actions for
            > > specific passes
            > > of a translator. Subclassing a grammar with actions would be almost
            > > useless. In fact I would reorganize my grammars as a
            > > literate program so I
            > > could reuse parts as I wish, it ends up being more flexible
            > > than antlr's
            > > subclassing.
            > >
            > > Monty
            > >
            > > > -----Original Message-----
            > > > From: Tiller, Michael (M.M.) [mailto:mtiller@...]
            > > > Sent: Tuesday, March 12, 2002 6:48 AM
            > > > To: 'antlr-interest@yahoogroups.com'
            > > > Subject: RE: [antlr-interest] Build Question
            > > >
            > > >
            > > > A few followup comments...
            > > >
            > > > I was hoping to support both Java and C++ for future
            > > > development. It seems like this is impractical for the
            > > > following reasons:
            > > >
            > > > 1) Trying to write a language independent way comment rule
            > > > that "eats" comments doesn't seem possible.
            > > > 2) Perhaps for misguided reasons, I'd like to be able to add
            > > > "child" nodes to my AST during tree construction. It isn't
            > > > clear to me how that can be done in a language neutral way.
            > > > It would be nice if ANTLR's tree construction shorthand were
            > > > extended to allow this.
            > > > 3) The target language has to be specified in the grammar
            > > > (couldn't this be a command line option?)
            > > > 4) The difficulties with trying to do grammar inheritance
            > > (see below).
            > > >
            > > > This isn't really criticism (ANTLR is a great tool).
            > > > Consider it ideas for future releases. Because of these
            > > > problems, I think I'm going to have to abandon my hope of a
            > > > language neutral grammar and just start plugging in my C++ code.
            > > >
            > > > --
            > > > Mike
            > > >
            > > > > -----Original Message-----
            > > > > From: Ric Klaren [mailto:klaren@...]
            > > > > Sent: Tuesday, March 12, 2002 3:51 AM
            > > > > To: antlr-interest@yahoogroups.com
            > > > > Subject: Re: [antlr-interest] Build Question
            > > > >
            > > > >
            > > > > On Mon, Mar 11, 2002 at 01:03:27PM -0500, Tiller, Michael
            > > > > (M.M.) wrote:
            > > > > > So far so good I imagine (although I haven't gotten far
            > > > > enough to figure
            > > > > > out if this is a good idea yet). The problem I'm running
            > > > > into is that I
            > > > > > imagine these grammars may be shared among several distinct
            > > > > applications.
            > > > > > For that reason, I keep the ".g" files away from everything
            > > > > else. So when
            > > > > > I run ANTLR, the Makefile rule looks like this:
            > > > > >
            > > > > > $(ANTLR) -glib $(ROOTDIR)/base.g -o $(GENDIR)
            > > > > $(ROOTDIR)/base_java.g
            > > > > >
            > > > > > where GENDIR is the location for the generated code and
            > > > > ROOTDIR is where I
            > > > > > keep the .g files. This way, I can generate the set of
            > > > > source files for
            > > > > > each project based on the grammars.
            > > > > >
            > > > > > The problem is that when you use grammar inheritance, ANTLR
            > > > > expands the
            > > > > > complete grammar into "expandedbase_java.g". Except that
            > > > > it actually names
            > > > > > it "expanded$(ROOTDIR)/base_java.g" (where ROOTDIR is
            > > > > actually expanded).
            > > > > > In other words, it assumes that the ".g" is in the current
            > > > > directory.
            > > > > >
            > > > > > Any suggestions about how to avoid this problem? I'm
            > > > > surprised that I
            > > > > > can't explicitly specify the name of the expanded grammar.
            > > > > That would have
            > > > > > been a nice option.
            > > > >
            > > > > Been there done that. Doesn't work. It needs some serious
            > > > > fixing in antlr
            > > > > to make this behave well. I already looked at it when I ran
            > > > > into it, it's
            > > > > not completely easy to fix in antlr. I currently copy .g
            > > > > files around with
            > > > > Make rules to get the right behaviour.
            > > > >
            > > > > I personally must say that I found antlr's grammar
            > > > inheritance pretty
            > > > > useless (gives to little control over what you want to
            > > > > override, in the end
            > > > > you end up copying action code in a complicated way so I
            > > > > figured I'd better
            > > > > copy stuff around in an easy way). YMMV though I suspect some
            > > > > ppl on the
            > > > > list may have gotten nice results with it.
            > > > >
            > > > > Ric
            > > > > --
            > > > > -----+++++****************************************************
            > > > > *+++++++++-------
            > > > > ---- Ric Klaren ----- klaren@... ----- +31 53
            > > > > 4893722 ----
            > > > > -----+++++****************************************************
            > > > > *+++++++++-------
            > > > > Time what is time - I wish I knew how to tell You why - It
            > > > > hurts to know -
            > > > > Aren't we machines - Time what is time -
            > Unlock the door
            > > > > - And see the truth - Then time is time again
            > > > > From: 'Time what is Time' by Blind Guardian
            > > > >
            > > > >
            > > > >
            > > > >
            > > > > Your use of Yahoo! Groups is subject to
            > > > > http://docs.yahoo.com/info/terms/
            > > > >
            > > > >
            > > >
            > > >
            > > >
            > > > Your use of Yahoo! Groups is subject to
            > > http://docs.yahoo.com/info/terms/
            > >
            > >
            > >
            > >
            > >
            > > Your use of Yahoo! Groups is subject to
            > > http://docs.yahoo.com/info/terms/
            > >
            > >
            >
            >
            >
            > Your use of Yahoo! Groups is subject to
            > http://docs.yahoo.com/info/terms/
            >
            >
            >
          • Matthew Ford
            Yes please Monty, I am interested in you noweb stuff as well matthew ... From: To: Sent: Wednesday, March
            Message 5 of 11 , Mar 12, 2002
            • 0 Attachment
              Yes please Monty,
              I am interested in you noweb stuff as well
              matthew
              ----- Original Message -----
              From: <mzukowski@...>
              To: <antlr-interest@yahoogroups.com>
              Sent: Wednesday, March 13, 2002 3:56 AM
              Subject: RE: [antlr-interest] Build Question


              > Yes, I will write up a simple example in the next few days, as I'm swamped
              > right now. In the meantime I can send you the noweb file I used for the
              gcc
              > stuff if you like.
              >
              > Monty
              >
              > > -----Original Message-----
              > > From: Tiller, Michael (M.M.) [mailto:mtiller@...]
              > > Sent: Tuesday, March 12, 2002 7:13 AM
              > > To: 'antlr-interest@yahoogroups.com'
              > > Subject: RE: [antlr-interest] Build Question
              > >
              > >
              > > Could you post an example of how this is done?
              > >
              > > --
              > > Mike
              > >
              > > > -----Original Message-----
              > > > From: mzukowski@... [mailto:mzukowski@...]
              > > > Sent: Tuesday, March 12, 2002 10:08 AM
              > > > To: antlr-interest@yahoogroups.com
              > > > Subject: RE: [antlr-interest] Build Question
              > > >
              > > >
              > > > One way to accomplish what you wish would be to do some
              > > > preprocessing. I'm
              > > > particularly fond of the "noweb" literate programming tool,
              > > > which I used to
              > > > build the GCC grammar. It allowed me to reuse grammar
              > > > fragments and then
              > > > add my own actions. More importantly for me it allowed me to
              > > > keep related
              > > > rules in the same area of the noweb file, so my declaration
              > > > rules for the
              > > > ANSI C Parser, GCC Parser, GCC tree walker, GCC tree
              > > emitter were all
              > > > together, and while debugging when I modified one rule I was
              > > > right there to
              > > > make sure the other rules were changed in sync.
              > > >
              > > > Antlr subclassing is most useful when there are no actions to
              > > > subclass. So
              > > > I use it for a tree grammar that I subclass with actions for
              > > > specific passes
              > > > of a translator. Subclassing a grammar with actions would be almost
              > > > useless. In fact I would reorganize my grammars as a
              > > > literate program so I
              > > > could reuse parts as I wish, it ends up being more flexible
              > > > than antlr's
              > > > subclassing.
              > > >
              > > > Monty
              > > >
              > > > > -----Original Message-----
              > > > > From: Tiller, Michael (M.M.) [mailto:mtiller@...]
              > > > > Sent: Tuesday, March 12, 2002 6:48 AM
              > > > > To: 'antlr-interest@yahoogroups.com'
              > > > > Subject: RE: [antlr-interest] Build Question
              > > > >
              > > > >
              > > > > A few followup comments...
              > > > >
              > > > > I was hoping to support both Java and C++ for future
              > > > > development. It seems like this is impractical for the
              > > > > following reasons:
              > > > >
              > > > > 1) Trying to write a language independent way comment rule
              > > > > that "eats" comments doesn't seem possible.
              > > > > 2) Perhaps for misguided reasons, I'd like to be able to add
              > > > > "child" nodes to my AST during tree construction. It isn't
              > > > > clear to me how that can be done in a language neutral way.
              > > > > It would be nice if ANTLR's tree construction shorthand were
              > > > > extended to allow this.
              > > > > 3) The target language has to be specified in the grammar
              > > > > (couldn't this be a command line option?)
              > > > > 4) The difficulties with trying to do grammar inheritance
              > > > (see below).
              > > > >
              > > > > This isn't really criticism (ANTLR is a great tool).
              > > > > Consider it ideas for future releases. Because of these
              > > > > problems, I think I'm going to have to abandon my hope of a
              > > > > language neutral grammar and just start plugging in my C++ code.
              > > > >
              > > > > --
              > > > > Mike
              > > > >
              > > > > > -----Original Message-----
              > > > > > From: Ric Klaren [mailto:klaren@...]
              > > > > > Sent: Tuesday, March 12, 2002 3:51 AM
              > > > > > To: antlr-interest@yahoogroups.com
              > > > > > Subject: Re: [antlr-interest] Build Question
              > > > > >
              > > > > >
              > > > > > On Mon, Mar 11, 2002 at 01:03:27PM -0500, Tiller, Michael
              > > > > > (M.M.) wrote:
              > > > > > > So far so good I imagine (although I haven't gotten far
              > > > > > enough to figure
              > > > > > > out if this is a good idea yet). The problem I'm running
              > > > > > into is that I
              > > > > > > imagine these grammars may be shared among several distinct
              > > > > > applications.
              > > > > > > For that reason, I keep the ".g" files away from everything
              > > > > > else. So when
              > > > > > > I run ANTLR, the Makefile rule looks like this:
              > > > > > >
              > > > > > > $(ANTLR) -glib $(ROOTDIR)/base.g -o $(GENDIR)
              > > > > > $(ROOTDIR)/base_java.g
              > > > > > >
              > > > > > > where GENDIR is the location for the generated code and
              > > > > > ROOTDIR is where I
              > > > > > > keep the .g files. This way, I can generate the set of
              > > > > > source files for
              > > > > > > each project based on the grammars.
              > > > > > >
              > > > > > > The problem is that when you use grammar inheritance, ANTLR
              > > > > > expands the
              > > > > > > complete grammar into "expandedbase_java.g". Except that
              > > > > > it actually names
              > > > > > > it "expanded$(ROOTDIR)/base_java.g" (where ROOTDIR is
              > > > > > actually expanded).
              > > > > > > In other words, it assumes that the ".g" is in the current
              > > > > > directory.
              > > > > > >
              > > > > > > Any suggestions about how to avoid this problem? I'm
              > > > > > surprised that I
              > > > > > > can't explicitly specify the name of the expanded grammar.
              > > > > > That would have
              > > > > > > been a nice option.
              > > > > >
              > > > > > Been there done that. Doesn't work. It needs some serious
              > > > > > fixing in antlr
              > > > > > to make this behave well. I already looked at it when I ran
              > > > > > into it, it's
              > > > > > not completely easy to fix in antlr. I currently copy .g
              > > > > > files around with
              > > > > > Make rules to get the right behaviour.
              > > > > >
              > > > > > I personally must say that I found antlr's grammar
              > > > > inheritance pretty
              > > > > > useless (gives to little control over what you want to
              > > > > > override, in the end
              > > > > > you end up copying action code in a complicated way so I
              > > > > > figured I'd better
              > > > > > copy stuff around in an easy way). YMMV though I suspect some
              > > > > > ppl on the
              > > > > > list may have gotten nice results with it.
              > > > > >
              > > > > > Ric
              > > > > > --
              > > > > > -----+++++****************************************************
              > > > > > *+++++++++-------
              > > > > > ---- Ric Klaren ----- klaren@... ----- +31 53
              > > > > > 4893722 ----
              > > > > > -----+++++****************************************************
              > > > > > *+++++++++-------
              > > > > > Time what is time - I wish I knew how to tell You why - It
              > > > > > hurts to know -
              > > > > > Aren't we machines - Time what is time -
              > > Unlock the door
              > > > > > - And see the truth - Then time is time again
              > > > > > From: 'Time what is Time' by Blind Guardian
              > > > > >
              > > > > >
              > > > > >
              > > > > >
              > > > > > Your use of Yahoo! Groups is subject to
              > > > > > http://docs.yahoo.com/info/terms/
              > > > > >
              > > > > >
              > > > >
              > > > >
              > > > >
              > > > > Your use of Yahoo! Groups is subject to
              > > > http://docs.yahoo.com/info/terms/
              > > >
              > > >
              > > >
              > > >
              > > >
              > > > Your use of Yahoo! Groups is subject to
              > > > http://docs.yahoo.com/info/terms/
              > > >
              > > >
              > >
              > >
              > >
              > > Your use of Yahoo! Groups is subject to
              > > http://docs.yahoo.com/info/terms/
              > >
              > >
              > >
              >
              >
              >
              > Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/
              >
              >
            • lgcraymer
              In response to Ric s and Mike Tiller s comments, I thought that I d post something on the AST construction extensions and tree grammar generation. This stuff
              Message 6 of 11 , Mar 12, 2002
              • 0 Attachment
                In response to Ric's and Mike Tiller's comments, I thought that I'd
                post something on the AST construction extensions and tree grammar
                generation. This stuff all works now, and I'm in the process of
                jumping through the necessary hoops so that I can release the code.
                If we get really lucky, Ter will manage to finish his code generation
                stuff in time for a 2.8 release.

                C++/C# support will require under 100 lines of new code to support the
                extensions, excluding the translocation support which takes about 500
                LOC. That enables language neutral AST rewriting.

                My current code base is based on 2.7.1; Ter and I plan to upgrade to a
                2.7.2 base before releasing 2.8.

                More detail:
                As Ter noted a few months back, I have been working on a tree grammar
                generation phase for ANTLR. Since that is nearing completion, now
                seems a good time for a status report and request input.

                Tree grammar generation involves a transformation of loop
                structures--()+ and ()*--containing root elements (elements marked
                with a caret) into recursive rules. Additionally, alternatives need
                to be split out and subtree structure must be tracked.

                The tree rewriting required to transform an input grammar tree into an
                output grammar tree is rather demanding on ANTLR's tree rewrite
                support. In the course of development, I have had to fix a number of
                misfeatures. BANG (!) did not work when applied to parenthesized
                expressions, including ()+ and ()* and ()? blocks. Annotations--ˆ and
                !--did not behave quite as expected in tree grammars.
                #( ROOT! A B )
                led to the output tree
                #( A B )
                rather than the A B list that might have been expected. Further, ^
                had scope limited to the current tree structure within a rule rather
                than having rule-level scope.
                #( A #( B Cˆ D ) )
                would generate
                #( A #( C #( B D ) ) )
                and not
                #( C #( A #( B D ) ) )
                These have been fixed.

                Additionally, roots are limited to single values in 2.7--ranges and
                alternatives are not supported. This may get fixed this in time for
                the 2.8.0 release.

                ANTLR tree restructuring via annotations is powerful, but spartan. I
                found that I needed a facility for reordering AST nodes and explicitly
                respecifying trees within rule definitions. This led to the idea of
                extending the ANTLR meta-language to include generic restructuring.
                The added facilities would include support for long distance
                transposition of tree elements--while not needed for the current
                application, it is essential in other contexts.

                The syntactic approach that I settled on was to mirror the normal tree
                grammar--#( ... )--and add extensions as needed. #{ ... } is a "build
                tree" construct (identifiers and tree expressions within #{ ... }
                define an AST construction rather than grammar elements to be matched)
                and is intended to be mnemonic, with the braces suggestive of an
                action syntax. Tree construction may be conditional; { ... }?
                delineates construction predicates when building trees--the analogy to
                semantic predicates should be clear. Additionally, "COPY" (duplicate
                and insert duplicate into tree and "REUSE" (insert "as is"--usually
                used as an optimization to avoid the overhead of AST node allocation)
                are keywords which precede AST node descriptors. "RETURN" is used to
                restart tree construction--a better name could probably be found, but
                the usual usage is #{ RETURN #( A B C ) } to specify the tree returned
                from a rule.

                One feature that I have found useful in other contexts is to
                translocate (or duplicate) a subtree to another location. To support
                this, an isolated string literal ("literal") marks an insertion point;
                assigning a value to a string literal ("lit" = #( A B C )) marks the
                data to be translocated. Scoping is fairly simple: if an insertion
                point is encountered before an initial assignment of data to that
                insertion point, then subsequent assignments will be to that insertion
                point until a new insertion point is encountered. Similarly, if an
                assignment precedes the first insertion point for that string, then
                the insertion point is used to resolve insertion of the preceding
                assignments and subsequent assignments will be made to the next
                insertion point. Insertions are always done in line: there are no
                special notations for "insert as root" or "insert as child". Any
                such rearrangements are made in subsequent AST transformation passes.

                The simplest loop-to-recursion transformation for tree grammars is
                foo : ( A Bˆ C )+ ;
                to
                foo : #( B foo1 C ) ;
                foo1 : #( B foo1 C A )
                | A
                ;
                but they can quickly become complex when alternative roots or nested
                loops are involved. The tree grammar generation code has been torture
                tested and seems to handle all possible cases, including support for
                the tree rewriting extensions (except for the translocation extension,
                which requires significant effort to support in generated tree
                grammars).

                --Loring Craymer
              • Ric Klaren
                Hi, ... Sounds very good. This probably also means that the current action.g xxxCodeGenerator interface was thoroughly revised? (this would completely overjoy
                Message 7 of 11 , Mar 13, 2002
                • 0 Attachment
                  Hi,

                  On Wed, Mar 13, 2002 at 06:37:03AM -0000, lgcraymer wrote:
                  > C++/C# support will require under 100 lines of new code to support the
                  > extensions, excluding the translocation support which takes about 500
                  > LOC. That enables language neutral AST rewriting.

                  Sounds very good. This probably also means that the current action.g
                  xxxCodeGenerator interface was thoroughly revised? (this would completely
                  overjoy me ;) )

                  The changes need porting to C++ still I presume?

                  > My current code base is based on 2.7.1; Ter and I plan to upgrade to a
                  > 2.7.2 base before releasing 2.8.

                  This means we are going to jump from 2.7.2a1 to 2.8 ?

                  > !--did not behave quite as expected in tree grammars.
                  > #( ROOT! A B )
                  > led to the output tree
                  > #( A B )
                  > rather than the A B list that might have been expected. Further, ^
                  > had scope limited to the current tree structure within a rule rather
                  > than having rule-level scope.
                  > #( A #( B C� D ) )
                  > would generate
                  > #( A #( C #( B D ) ) )
                  > and not
                  > #( C #( A #( B D ) ) )
                  > These have been fixed.

                  Cool.

                  Sounds good all the extensions/changes. I'm still a bit in the unclear as
                  to the exact syntax (maybe a few examples..) but I trust it will be a huge
                  improvement.

                  Well done!

                  Is there already a branch available in the repository with these changes?

                  Cheers,

                  Ric
                  --
                  -----+++++*****************************************************+++++++++-------
                  ---- Ric Klaren ----- klaren@... ----- +31 53 4893722 ----
                  -----+++++*****************************************************+++++++++-------
                  Time what is time - I wish I knew how to tell You why - It hurts to know -
                  Aren't we machines - Time what is time - Unlock the door
                  - And see the truth - Then time is time again
                  From: 'Time what is Time' by Blind Guardian
                • lgcraymer
                  ... the ... 500 ... completely ... No such luck--that will have to wait for 3.0. Basically, I added a few classes to support tree actions and about 30 lines
                  Message 8 of 11 , Mar 13, 2002
                  • 0 Attachment
                    --- In antlr-interest@y..., Ric Klaren <klaren@c...> wrote:
                    > Hi,
                    >
                    > On Wed, Mar 13, 2002 at 06:37:03AM -0000, lgcraymer wrote:
                    > > C++/C# support will require under 100 lines of new code to support
                    the
                    > > extensions, excluding the translocation support which takes about
                    500
                    > > LOC. That enables language neutral AST rewriting.
                    >
                    > Sounds very good. This probably also means that the current action.g
                    > xxxCodeGenerator interface was thoroughly revised? (this would
                    completely
                    > overjoy me ;) )

                    No such luck--that will have to wait for 3.0. Basically, I added a
                    few classes to support "tree actions" and about 30 lines of code in
                    JavaCodeGenerator to interpret the actions.

                    > The changes need porting to C++ still I presume?

                    Yes--something like 50 LOC in CppCodeGenerator, and then another 500
                    LOC for the translocation support library--I have jUnit test code
                    which covers that code, so the translation to C++ might be done
                    quickly.

                    > > My current code base is based on 2.7.1; Ter and I plan to upgrade
                    to a
                    > > 2.7.2 base before releasing 2.8.
                    >
                    > This means we are going to jump from 2.7.2a1 to 2.8 ?

                    More likely at least a 2.7.2b release will precede 2.8.0a--I have to
                    jump through a few hoops to release my code (and I may not be able to
                    release through the jGuru site--stay tuned) and that will take a
                    little time.


                    > Well done!

                    Thanks! This is about the first third of what it takes to
                    dramatically speed up ANTLR development--the second third is Ter's
                    code generation support, and the third component would be a
                    refactoring browser/interactive grammar analyzer tool. I'd dearly
                    like to see something like that done for 3.0.

                    > Is there already a branch available in the repository with these
                    changes?

                    No--I have to work through intellectual property issues with JPL/NASA
                    before I can formally release the code in some fashion.

                    --Loring
                  Your message has been successfully submitted and would be delivered to recipients shortly.