Loading ...
Sorry, an error occurred while loading the content.

[patch] use font-lock

Expand Messages
  • Daniel Colascione
    I ve converted nxml to font-lock. I used the existing fontification machinery and put it inside a cc-mode-style matcher. Efficiency and output are the same,
    Message 1 of 8 , May 23, 2008
    • 0 Attachment
      I've converted nxml to font-lock. I used the existing fontification machinery
      and put it inside a cc-mode-style matcher. Efficiency and output are the
      same, but:

      1) hi-lock-mode works now
      2) all conventional font-locking functions work as expected. For example, you
      can turn fontification on and off with M-x font-lock-mode.
      3) font-lock-add-keywords DTRT
      4) multiple-major-modes modes should be able to use nXML fontification now,
      though this remains untested

      I've also added a new function:
      nxml-debug-region: Interactive function. Activate the region and call
      nxml-debug-region. The new region is what nxml thinks should be re-fontified
      if the original region is changed.

      The new code probably works only on Emacs 22. Lightly tested, but it seems to
      handle corner highlighting cases fine. I removed a bunch of code that
      was "for redisplay", since I'm assuming font-lock handles those funky bits.

      In addition to the patch, I've attached a set of files that demonstrates
      extending nXML mode to work with the Genshi template engine. The examples
      ought to work with some slight modification of the embedded paths.

      (Does the Relax NG compact syntax offer a way to say "include the next file
      for this document type"?)
    • Jason Rumney
      nxml-mode will be distributed with future versions of Emacs. If you are willing to assign these changes to FSF for inclusion in Emacs 23, please send them to
      Message 2 of 8 , May 23, 2008
      • 0 Attachment
        nxml-mode will be distributed with future versions of Emacs. If you are
        willing to assign these changes to FSF for inclusion in Emacs 23, please
        send them to emacs-devel@....


        Daniel Colascione wrote:
        > I've converted nxml to font-lock. I used the existing fontification machinery
        > and put it inside a cc-mode-style matcher. Efficiency and output are the
        > same, but:
        >
        > 1) hi-lock-mode works now
        > 2) all conventional font-locking functions work as expected. For example, you
        > can turn fontification on and off with M-x font-lock-mode.
        > 3) font-lock-add-keywords DTRT
        > 4) multiple-major-modes modes should be able to use nXML fontification now,
        > though this remains untested
        >
        > I've also added a new function:
        > nxml-debug-region: Interactive function. Activate the region and call
        > nxml-debug-region. The new region is what nxml thinks should be re-fontified
        > if the original region is changed.
        >
        > The new code probably works only on Emacs 22. Lightly tested, but it seems to
        > handle corner highlighting cases fine. I removed a bunch of code that
        > was "for redisplay", since I'm assuming font-lock handles those funky bits.
        >
        > In addition to the patch, I've attached a set of files that demonstrates
        > extending nXML mode to work with the Genshi template engine. The examples
        > ought to work with some slight modification of the embedded paths.
        >
        > (Does the Relax NG compact syntax offer a way to say "include the next file
        > for this document type"?)
        >
        >
        >
        > ------------------------------------
        >
        > Yahoo! Groups Links
        >
        >
        >
        >
        > ------------------------------------------------------------------------
        >
        > Hello from hello! Test ${name}
        >
      • Lennart Borgman (gmail)
        ... Hi Daniel, This sounds very interesting. As you might know in nXhtml I am using nxml-mode with mumamo (which is a framework for multiple major modes in a
        Message 3 of 8 , May 23, 2008
        • 0 Attachment
          Daniel Colascione wrote:
          >
          >
          > I've converted nxml to font-lock. I used the existing fontification
          > machinery
          > and put it inside a cc-mode-style matcher. Efficiency and output are the
          > same, but:
          >
          > 1) hi-lock-mode works now
          > 2) all conventional font-locking functions work as expected. For
          > example, you
          > can turn fontification on and off with M-x font-lock-mode.
          > 3) font-lock-add-keywords DTRT
          > 4) multiple-major-modes modes should be able to use nXML fontification now,
          > though this remains untested

          Hi Daniel,

          This sounds very interesting. As you might know in nXhtml I am using
          nxml-mode with mumamo (which is a framework for multiple major modes in
          a buffer). It works good even though mumamo requires that major modes
          uses font-lock.

          You might wonder how that can be the case. To make it work I implemented
          a workaround where I use the parsing capabilities from nxml-mode to
          check that the files follows the DTD specified syntax, but syntax
          highlighting from another mode (xml-mode/html-mode) that supports font-lock.

          There is one very disturbing thing with my solution: I can't stop
          nxml-mode from parsing the whole buffer. It parses also those parts
          where mumamo has assigned another major mode. (I hoped that someone some
          day might have the time and skill to look into this, but I did not have
          them.)

          Does you solution handle this problem? If it does, then how does it
          handle it? Does font-lock-fontify-region-function handle also the
          parsing of the xml code? That would be great, but it seems difficult.

          Another thing that would be great would be integration with CEDET. As
          you have probably seen nxml-mode is a part of CVS Emacs and CEDET will
          hopefully soon be. Eric Ludlam has done very much work on CEDET recently.

          If the completion offered by nxml-mode could be used together with CEDET
          that would be very good. (nXhtml currently offer this in a visible way
          separately, but I believe the long term solution is to go with CEDET -
          at least as an option.)

          BTW, there is a problem with hi-lock. It uses text properties which may
          be hidden by overlays. IMO it should use overlays with high priorities.
          (That seems to be the easiest solution.)

          > I've also added a new function:
          > nxml-debug-region: Interactive function. Activate the region and call
          > nxml-debug-region. The new region is what nxml thinks should be
          > re-fontified
          > if the original region is changed.
          >
          > The new code probably works only on Emacs 22. Lightly tested, but it
          > seems to
          > handle corner highlighting cases fine. I removed a bunch of code that
          > was "for redisplay", since I'm assuming font-lock handles those funky bits.
          >
          > In addition to the patch, I've attached a set of files that demonstrates
          > extending nXML mode to work with the Genshi template engine. The examples
          > ought to work with some slight modification of the embedded paths.

          Genshi was new to me. I will add it to mumamo.el.

          How did you do the integration with xhtml?


          > (Does the Relax NG compact syntax offer a way to say "include the next file
          > for this document type"?)
        • Daniel Colascione
          ... I saw that you used that approach, and decided to just alter nXML-mode instead. I don t really understand why nXML was written to not use font-lock in the
          Message 4 of 8 , May 23, 2008
          • 0 Attachment
            On Friday 23 May 2008, Lennart Borgman (gmail) wrote:
            > You might wonder how that can be the case. To make it work I implemented
            > a workaround where I use the parsing capabilities from nxml-mode to
            > check that the files follows the DTD specified syntax, but syntax
            > highlighting from another mode (xml-mode/html-mode) that supports
            > font-lock.

            I saw that you used that approach, and decided to just alter nXML-mode
            instead.

            I don't really understand why nXML was written to not use font-lock in the
            first place. cc-mode had used the complex-matcher approach for a long time,
            portably and with few problems. But from having read the font-lock
            documentation, one wouldn't suppose this kind of power was available. The
            trick of making a matcher that does all the fontification itself and just
            returns 'nil' is not documented. Perhaps it should be.

            > There is one very disturbing thing with my solution: I can't stop
            > nxml-mode from parsing the whole buffer. It parses also those parts
            > where mumamo has assigned another major mode. (I hoped that someone some
            > day might have the time and skill to look into this, but I did not have
            > them.)

            > Does you solution handle this problem? If it does, then how does it
            > handle it? Does font-lock-fontify-region-function handle also the
            > parsing of the xml code? That would be great, but it seems difficult.

            I haven't looked at mumamo mode in detail. How does mumamo isolate major modes
            to particular areas of the buffer? If you're just narrowing the buffer, I
            think going through nXML and removing all the calls to WIDEN should be
            sufficient. (Or replacing them with some kind of mumamo-widen.)

            cc-mode also widens the buffer. What's the difference?

            > Another thing that would be great would be integration with CEDET. As
            > you have probably seen nxml-mode is a part of CVS Emacs and CEDET will
            > hopefully soon be. Eric Ludlam has done very much work on CEDET recently.

            That's good news. I've followed the "IDE features" thread on emacs-devel with
            excitement.

            > If the completion offered by nxml-mode could be used together with CEDET
            > that would be very good. (nXhtml currently offer this in a visible way
            > separately, but I believe the long term solution is to go with CEDET -
            > at least as an option.)

            I don't use CEDET, but if it's getting into Emacs, I might as well give it
            another whirl. IIRC, CEDET requires a buffer parser to just generate a set of
            tags. What else is required?

            > BTW, there is a problem with hi-lock. It uses text properties which may
            > be hidden by overlays. IMO it should use overlays with high priorities.
            > (That seems to be the easiest solution.)

            IMHO, font-lock itself should use overlays. But ignoring that distinction,
            hi-lock dynamically adding font-lock keywords is the right way to go.

            HIGHLIGHT elements already support a kind of priority through the OVERRIDE
            flag, which can be t, prepend, or append, and through ordering on
            font-lock-keywords.

            > Genshi was new to me. I will add it to mumamo.el.
            >
            > How did you do the integration with xhtml?

            I extended the XHTML schema to support Genshi and XInclude elements and
            attributes; I include the original XHTML schema and augment it.

            It'd be nice if nXML extended Relax NG to support some kind of schema plugin
            mechanism, basically automating what's in qtmstr-xhtml.rnc: e.g. some way of
            saying "elements ns:a, ns:b, and ns:c from mylanguage.rnc are at block level
            and can contain block-level elements, and ns:d is inline, but can't contain
            any children. Oh, and attributes ns:foo and ns:bar can attach to all
            elements".

            As it is, the best approach for Genshi, IMHO, would be some kind of minor
            mode. This mode would add the appropriate font-lock keywords and generate a
            temporary schema like the one in qtmstr-xhtml.rnc, only pointing to the
            correct xhtml.rnc.
          • Lennart Borgman (gmail)
            ... Do you mean font-lock-fontify-region-function? ... Mumamo now defines what I call multi major modes . (Previously there was a minor mode called
            Message 5 of 8 , May 23, 2008
            • 0 Attachment
              Daniel Colascione wrote:
              >
              >
              > On Friday 23 May 2008, Lennart Borgman (gmail) wrote:
              > > You might wonder how that can be the case. To make it work I implemented
              > > a workaround where I use the parsing capabilities from nxml-mode to
              > > check that the files follows the DTD specified syntax, but syntax
              > > highlighting from another mode (xml-mode/html-mode) that supports
              > > font-lock.
              >
              > I saw that you used that approach, and decided to just alter nXML-mode
              > instead.
              >
              > I don't really understand why nXML was written to not use font-lock in the
              > first place. cc-mode had used the complex-matcher approach for a long time,
              > portably and with few problems. But from having read the font-lock
              > documentation, one wouldn't suppose this kind of power was available. The
              > trick of making a matcher that does all the fontification itself and just
              > returns 'nil' is not documented. Perhaps it should be.

              Do you mean font-lock-fontify-region-function?

              > > There is one very disturbing thing with my solution: I can't stop
              > > nxml-mode from parsing the whole buffer. It parses also those parts
              > > where mumamo has assigned another major mode. (I hoped that someone some
              > > day might have the time and skill to look into this, but I did not have
              > > them.)
              >
              > > Does you solution handle this problem? If it does, then how does it
              > > handle it? Does font-lock-fontify-region-function handle also the
              > > parsing of the xml code? That would be great, but it seems difficult.
              >
              > I haven't looked at mumamo mode in detail. How does mumamo isolate major
              > modes
              > to particular areas of the buffer? If you're just narrowing the buffer, I
              > think going through nXML and removing all the calls to WIDEN should be
              > sufficient. (Or replacing them with some kind of mumamo-widen.)

              Mumamo now defines what I call "multi major modes". (Previously there
              was a minor mode called mumamo-mode, but that is obsolete.)

              Mumamo defines its own font-lock-fontify-region-function that does the
              job. In this function it narrows the buffer to chunks where each chunk
              has its own major mode.

              So the main question is: does the parsing happens in the
              font-lock-fontify-region-function that you have defined in nxml-mode? (I
              guess you have defined such a function?)

              One thing I think need to be taken care of is nxml-mode starting state
              in chunks handled by mumamo. Is there a way to do that?


              > cc-mode also widens the buffer. What's the difference?

              Sorry, I do not understand what you mean here.


              > I don't use CEDET, but if it's getting into Emacs, I might as well give it
              > another whirl. IIRC, CEDET requires a buffer parser to just generate a
              > set of
              > tags. What else is required?

              I do not know. Maybe this discussion should be carried over to Emacs
              devel which I think Eric Ludlam reads?


              > > BTW, there is a problem with hi-lock. It uses text properties which may
              > > be hidden by overlays. IMO it should use overlays with high priorities.
              > > (That seems to be the easiest solution.)
              >
              > IMHO, font-lock itself should use overlays.

              That would probably be a performance problem.

              > But ignoring that distinction,
              > hi-lock dynamically adding font-lock keywords is the right way to go.

              Yes, it is very nice.

              > > Genshi was new to me. I will add it to mumamo.el.
              > >
              > > How did you do the integration with xhtml?
              >
              > I extended the XHTML schema to support Genshi and XInclude elements and
              > attributes; I include the original XHTML schema and augment it.

              Thanks, very nice. That mean you can handle <?pyton ... ?> and for
              example py:content="var".

              mumamo can help in two ways here:

              1) inside <?pyton ... ?> it can use a genshi-mode major mode if there is
              one.

              2) inside {% pyton ... %} it can do the same.


              > It'd be nice if nXML extended Relax NG to support some kind of schema
              > plugin
              > mechanism, basically automating what's in qtmstr-xhtml.rnc: e.g. some
              > way of
              > saying "elements ns:a, ns:b, and ns:c from mylanguage.rnc are at block
              > level
              > and can contain block-level elements, and ns:d is inline, but can't contain
              > any children. Oh, and attributes ns:foo and ns:bar can attach to all
              > elements".

              Good idea IMO.


              > As it is, the best approach for Genshi, IMHO, would be some kind of minor
              > mode. This mode would add the appropriate font-lock keywords and generate a
              > temporary schema like the one in qtmstr-xhtml.rnc, only pointing to the
              > correct xhtml.rnc.

              Maybe. But with mumamo it could be a major mode, written with those
              tools that there already are in Emacs.

              This could be combined with the schema extending mechanism you suggest.
            • Dave Pawson
              ... It can. It s called relax NG. Mod the schema an hey presto it works. This is beginning to sound like the M$ approach. Embrace, extend, and exterminate If
              Message 6 of 8 , May 24, 2008
              • 0 Attachment
                2008/5/23 Daniel Colascione <danc@...>:

                >
                > It'd be nice if nXML extended Relax NG to support some kind of schema plugin
                > mechanism, basically automating what's in qtmstr-xhtml.rnc: e.g. some way of
                > saying "elements ns:a, ns:b, and ns:c from mylanguage.rnc are at block level
                > and can contain block-level elements, and ns:d is inline, but can't contain
                > any children. Oh, and attributes ns:foo and ns:bar can attach to all
                > elements".

                It can.
                It's called relax NG.
                Mod the schema an hey presto it works.

                This is beginning to sound like the M$ approach. Embrace, extend, and
                exterminate

                If you want another language go invent one. Don't screw this one.



                --
                Dave Pawson
                XSLT XSL-FO FAQ.
                http://www.dpawson.co.uk
              • Daniel Colascione
                ... Sort of. I m talking about using a function as a matcher keyword in font-lock-keywords. This function always returns nil, but while searching, sets the
                Message 7 of 8 , May 24, 2008
                • 0 Attachment
                  On Friday 23 May 2008 06:50:19 pm Lennart Borgman (gmail) wrote:
                  > Do you mean font-lock-fontify-region-function?

                  Sort of. I'm talking about using a function as a matcher keyword in
                  font-lock-keywords. This function always returns nil, but while searching,
                  sets the face property itself between (point) and its one parameter.

                  The advantage of using the function-matcher approach is that it interacts
                  nicely with other users of font-lock-keywords. cc-mode has used it forever,
                  and it seems to work well.

                  AFAICS, this trick is undocumented.

                  > Mumamo defines its own font-lock-fontify-region-function that does the
                  > job. In this function it narrows the buffer to chunks where each chunk
                  > has its own major mode.
                  >
                  > So the main question is: does the parsing happens in the
                  > font-lock-fontify-region-function that you have defined in nxml-mode? (I
                  > guess you have defined such a function?)

                  Yes, in that the after-change function that updates bookkeeping information is
                  called from the font-lock machinery (via
                  font-lock-extend-after-change-function). But it doesn't matter here, because
                  at least nxml still keeps some buffer-global information, like the end of the
                  XML prolog (which is assume to extend from the start of the buffer to
                  nxml-prolog-end).

                  > One thing I think need to be taken care of is nxml-mode starting state
                  > in chunks handled by mumamo. Is there a way to do that?

                  I don't see an easy way to do that. The problem is that nxml needs to widen
                  the buffer to see whether the prolog has changed, whether it's scanned as far
                  as a given change, and so on. And then it assumes the buffer is contiguous.
                  But I didn't write it; there may be a way to deal with the problem.

                  I'm just speculating here, but what if there were some indirect overlay
                  property? It would have the behavior of using, as the overlay content, the
                  contents of another buffer. Moving the point in the parent would move the
                  point in the child. The child buffer would have its own font-lock setup, and
                  the resulting highlighting would show up in the parent.

                  Using a system like that, you could stitch together multiple child buffers to
                  form a parent that looked like it had multiple major modes. And each major
                  mode of a child buffer wouldn't know or care about the parent.

                  > > IMHO, font-lock itself should use overlays.
                  >
                  > That would probably be a performance problem.

                  And a compatibility nightmare, you're right. But I think it would have been a
                  better conceptual model.
                • Lennart Borgman (gmail)
                  ... Is not that documented in `font-lock-keywords ? But maybe you mean it could be more explicit? ... I am not sure what you mean here. Mumamo calls
                  Message 8 of 8 , May 24, 2008
                  • 0 Attachment
                    Daniel Colascione wrote:
                    > On Friday 23 May 2008 06:50:19 pm Lennart Borgman (gmail) wrote:
                    > > Do you mean font-lock-fontify-region-function?
                    >
                    > Sort of. I'm talking about using a function as a matcher keyword in
                    > font-lock-keywords. This function always returns nil, but while searching,
                    > sets the face property itself between (point) and its one parameter.
                    >
                    > The advantage of using the function-matcher approach is that it interacts
                    > nicely with other users of font-lock-keywords. cc-mode has used it forever,
                    > and it seems to work well.
                    >
                    > AFAICS, this trick is undocumented.

                    Is not that documented in `font-lock-keywords'? But maybe you mean it
                    could be more explicit?


                    > > Mumamo defines its own font-lock-fontify-region-function that does the
                    > > job. In this function it narrows the buffer to chunks where each chunk
                    > > has its own major mode.
                    > >
                    > > So the main question is: does the parsing happens in the
                    > > font-lock-fontify-region-function that you have defined in nxml-mode? (I
                    > > guess you have defined such a function?)
                    >
                    > Yes, in that the after-change function that updates bookkeeping
                    > information is
                    > called from the font-lock machinery (via
                    > font-lock-extend-after-change-function).

                    I am not sure what you mean here. Mumamo calls font-lock-fontify-region
                    (after trying to set the variables needed for the chunks major mode).
                    Does that work together with your approach?


                    > > One thing I think need to be taken care of is nxml-mode starting state
                    > > in chunks handled by mumamo. Is there a way to do that?
                    >
                    > I don't see an easy way to do that. The problem is that nxml needs to widen
                    > the buffer to see whether the prolog has changed, whether it's scanned
                    > as far
                    > as a given change, and so on. And then it assumes the buffer is contiguous.
                    > But I didn't write it; there may be a way to deal with the problem.

                    It is a general problem. Js2 has a similar problem. (I am cc:ing Steve
                    Yegge too ;-)


                    > I'm just speculating here, but what if there were some indirect overlay
                    > property? It would have the behavior of using, as the overlay content, the
                    > contents of another buffer. Moving the point in the parent would move the
                    > point in the child. The child buffer would have its own font-lock setup,
                    > and
                    > the resulting highlighting would show up in the parent.
                    >
                    > Using a system like that, you could stitch together multiple child
                    > buffers to
                    > form a parent that looked like it had multiple major modes. And each major
                    > mode of a child buffer wouldn't know or care about the parent.

                    I have been thinking about different such possibilities. AFAICS it would
                    be much simpler if the parsing modes (like nxml-mode) are aware of what
                    should be parsed in the buffers. Other possibilities tends to be very
                    complex.

                    So I would suggest that something like mumamo divides the buffer into
                    chunks with different major modes. Then the parsing mode and mumamo (or
                    something similar) should communicate to tell what chunks should be
                    parsed and what chunks should not be parsed. (The major mode itself is
                    not enought to tell if a chunk should be parsed or not. For example a
                    <style> ... </style> chunk in xhtml can be parsed by nxml-mode. Ie
                    nxml-mode does not care much about the content of it.)

                    Doing it that way I see two problems:

                    1) The parser must be able to somehow support a "parse region"
                    functionality.

                    2) Parsing states for the different chunks must be initialized somehow.
                    I would suggest carrying over the last state from the previous chunk if
                    needed. (Sometimes assuming that the parsing state at the beginning of
                    the chunk is the "zero state" is enough, sometimes not.)

                    As you noticed the parser must somehow start from the beginning of the file.

                    Font lock could cheat a bit and start in the middle of a file so parsing
                    and font-lock style syntax highlighting do not have the same needs.
                    Maybe that can be handled by doing the syntax highlighting in two
                    passes, the first pass similar to font-lock currents way and the second
                    pass doing parsing?

                    This would allow editing with the benefit of syntax highlighting
                    (font-lock style) while the parser works in the background, doing the
                    second pass. When the support of the parser is needed (for example for
                    completion) the user of course would have to wait for the parser to
                    catch up.



                    How would such a scheme fit js2 and the changed nxml-mode? (I guess
                    there is more work to support it in both modes and a framework similar
                    to jit-lock would have to be used for the parsing.)
                  Your message has been successfully submitted and would be delivered to recipients shortly.