Loading ...
Sorry, an error occurred while loading the content.

VIM-6 - rightleft redisplay

Expand Messages
  • Nadim Shaikli
    Adding Arabic shaping abilities to VIM-6.0 is very close to being done (sans search (ie. /) ability). I have one puzzling hurdle left (I think :-) With the
    Message 1 of 18 , Dec 3, 2001
    • 0 Attachment
      Adding Arabic shaping abilities to VIM-6.0 is very close to being done (sans
      'search' (ie. /) ability). I have one puzzling hurdle left (I think :-)

      With the current code in search.c (function 'screen_line' line #3953) there
      is a 'while' which in essence redisplays everything (which seems to always
      happen upon entering a new character). Here's my question - in "rightleft"
      mode the redrawing is still done from left-to-right instead of right-to-left
      and that is playing havoc with the new Arabic shaping code (Arabic as you
      might know or have guessed by now is a right-to-left language). It seems
      only logical that in "rightleft" mode that the redrawing be done in the
      same direction (ie. from right to left).

      Is someone out there familiar enough with the code to offer an ifdef'ed
      solution that would redraw and traverse the characters from right-to-left
      in "rightleft" mode ? I've attempted to do that myself, but there are bits
      and pieces of the code that I don't understand and can't seem to get right,
      thus rendering my subtle attempts non-functional.

      Many thanks...

      - Nadim


      __________________________________________________
      Do You Yahoo!?
      Buy the perfect holiday gifts at Yahoo! Shopping.
      http://shopping.yahoo.com
    • David Harrison Jr.
      Did you get my previous post about calling screen_update() ? ... You mean in the file screen.c ? ... What about numbers? Are they drawn in the same
      Message 2 of 18 , Dec 4, 2001
      • 0 Attachment
        Did you get my previous post about calling "screen_update()"?

        Nadim Shaikli wrote:

        >With the current code in search.c (function 'screen_line' line #3953)
        >

        You mean in the file "screen.c"?

        > there
        >is a 'while' which in essence redisplays everything (which seems to always
        >happen upon entering a new character). Here's my question - in "rightleft"
        >mode the redrawing is still done from left-to-right instead of right-to-left
        >and that is playing havoc with the new Arabic shaping code
        >

        >(Arabic as you
        >might know or have guessed by now is a right-to-left language).
        >

        What about numbers? Are they drawn in the same direction? I seem to
        remember there was one language that had its numbers drawn in the
        opposite direction from its words, thus making it a combination of
        "rightleft" and "leftright" type of writing. If this is the case, then
        drawing everything in one direction may not be flexible enough, no
        matter which direction you are drawing.

        > It seems
        >only logical that in "rightleft" mode that the redrawing be done in the
        >same direction (ie. from right to left).
        >

        That does seem logical. However, I think Vim is assuming that the
        characters can be laid out in a regular grid.
        If this is the case, then it shouldn't matter which direction you draw
        your characters because the glyphs that are drawn will be self-contained
        (each to it's own cell).

        Actually, I think that the final solution will be something like "draw a
        line at a time". This way, you can account for the direction and draw
        it appropriately. When I say "draw a line", I'm thinking of one line of
        text on the screen.... and drawing it as one long string, possibly. The
        X-Windows FontSet handling/drawing routines should take care of the rest.

        I don't know enough about Unicode (specifically UTF-8) and other
        encodings to really know what the best solution is.

        >
        >Is someone out there familiar enough with the code to offer an ifdef'ed
        >solution that would redraw and traverse the characters from right-to-left
        >in "rightleft" mode ? I've attempted to do that myself, but there are bits
        >and pieces of the code that I don't understand and can't seem to get right,
        >thus rendering my subtle attempts non-functional.
        >

        Unfortunately, I'm not that familiar with the code to offer a solution
        (I need to see Arabic examples.... lots of them).


        --David H. Jr.
      • Yousof Mortazavi
        ... Yes, you are right about numbers rendered as left-to-right. This whole handling of right-to-left and left-to-right gets pretty complicated especially when
        Message 3 of 18 , Dec 4, 2001
        • 0 Attachment
          On Tue, 4 Dec 2001, David Harrison Jr. wrote:

          >Nadim Shaikli wrote:
          >
          >>With the current code in search.c (function 'screen_line' line #3953)
          >>
          >
          >You mean in the file "screen.c"?
          >
          >> there
          >>is a 'while' which in essence redisplays everything (which seems to always
          >>happen upon entering a new character). Here's my question - in "rightleft"
          >>mode the redrawing is still done from left-to-right instead of right-to-left
          >>and that is playing havoc with the new Arabic shaping code
          >>
          >
          >>(Arabic as you
          >>might know or have guessed by now is a right-to-left language).
          >>
          >
          >What about numbers? Are they drawn in the same direction? I seem to
          >remember there was one language that had its numbers drawn in the
          >opposite direction from its words, thus making it a combination of
          >"rightleft" and "leftright" type of writing. If this is the case, then
          >drawing everything in one direction may not be flexible enough, no
          >matter which direction you are drawing.

          Yes, you are right about numbers rendered as left-to-right. This
          whole handling of right-to-left and left-to-right gets pretty
          complicated especially when you start embedding texts from
          left-to-right scripts within right-to-left scripts. It is,
          however, clearly outlined in the Unicode Standard Annex #9: "The
          Bidirectonal Algorithm" available at
          http://www.unicode.org/unicode/reports/tr9/

          As well libraries have been written, the most complete of which
          is "fribidi" available from
          http://fribidi.sourceforge.net

          Noteworthy to say, that implementing BiDi for Vim will ease the
          implementation of right-to-left scripts such as Arabic and
          Hebrew.

          Here I'd like to propose a "BiDi" mode in Vim (just as the
          current RL mode) that when active, characters entered get
          shuffled according to the BiDi Algorithm as described by the
          Unicode Standard. This mode should of course coexist with the
          current RL mode such that when this mode is active, the base
          direction is right-to-left, nevertheless, characters that possess
          left-to-right behaviour are rendered as such.

          I would like to work on this project in conjunction with any
          group that feels the need for proper Bidirectional behaviour. It
          is, after all, part of the Unicode Standard, and the current
          Unicode implementation of Vim would be incomplete without it.

          Bram, do you approve?

          --
          -= Yousof Mortazavi <yousof@...> | Electrical Engineering =-
          -= WWW: http://www.engsoc.org/~yousof/ | Carleton University =-
        • Nadim Shaikli
          On Tue, 04 Dec 2001 12:37:33 -0500 ... Yes I did, but it unfortunately it wasn t what I was looking for; thanks for responding non-the-less. ... Oops, yah it
          Message 4 of 18 , Dec 4, 2001
          • 0 Attachment
            On Tue, 04 Dec 2001 12:37:33 -0500
            "David Harrison Jr." <david at motionrealityinc dot com> wrote:
            >
            > Did you get my previous post about calling "screen_update()"?

            Yes I did, but it unfortunately it wasn't what I was looking for;
            thanks for responding non-the-less.

            > >With the current code in search.c (function 'screen_line' line #3953)
            > >
            >
            > You mean in the file "screen.c"?

            Oops, yah it should have been "screen.c (function 'screen_line' line #3953)".

            > > there
            > >is a 'while' which in essence redisplays everything (which seems to always
            > >happen upon entering a new character). Here's my question - in "rightleft"
            > >mode the redrawing is still done from left-to-right instead of right-to-left
            > >and that is playing havoc with the new Arabic shaping code
            > >
            >
            > >(Arabic as you
            > >might know or have guessed by now is a right-to-left language).
            > >
            >
            > What about numbers? Are they drawn in the same direction? I seem to
            > remember there was one language that had its numbers drawn in the
            > opposite direction from its words, thus making it a combination of
            > "rightleft" and "leftright" type of writing. If this is the case, then
            > drawing everything in one direction may not be flexible enough, no
            > matter which direction you are drawing.

            Numbers aside (and we can live with them not being processed correctly
            until Bidi is adopted). The question still stands on how to modify the
            above noted logic/code to redisplay the characters in "rightleft" mode.

            > > It seems
            > >only logical that in "rightleft" mode that the redrawing be done in the
            > >same direction (ie. from right to left).
            > >
            >
            > That does seem logical. However, I think Vim is assuming that the
            > characters can be laid out in a regular grid.
            > If this is the case, then it shouldn't matter which direction you draw
            > your characters because the glyphs that are drawn will be self-contained
            > (each to it's own cell).
            >
            > Actually, I think that the final solution will be something like "draw a
            > line at a time". This way, you can account for the direction and draw
            > it appropriately. When I say "draw a line", I'm thinking of one line of
            > text on the screen.... and drawing it as one long string, possibly. The
            > X-Windows FontSet handling/drawing routines should take care of the rest.
            >
            > I don't know enough about Unicode (specifically UTF-8) and other
            > encodings to really know what the best solution is.

            As noted, for the new Arabic code to work properly and for it to do
            the shaping correctly, the redisplay needs to be done in a right-to-left
            manner. Let me tell you why - I intercept the displaying of a character
            and have to look at what's before and what's after a certain character for
            me to make a decision on how to process (or shape) that character. In
            ":set rightleft" mode the processing is done correctly when a character is
            entered, but when the redisplay happens - the direction is reversed and the
            new Arabic code is thrown for a loop (instead of it processing characters
            from right-to-left, it gets characters from left-to-right thus rendering
            the previous and next characters all wrong).

            > >Is someone out there familiar enough with the code to offer an ifdef'ed
            > >solution that would redraw and traverse the characters from right-to-left
            > >in "rightleft" mode ? I've attempted to do that myself, but there are bits
            > >and pieces of the code that I don't understand and can't seem to get right,
            > >thus rendering my subtle attempts non-functional.
            > >
            >
            > Unfortunately, I'm not that familiar with the code to offer a solution
            > (I need to see Arabic examples.... lots of them).

            Well, I would suggest downloading our work-in-progress code,

            http://www.arabeyes.org/cgi-bin/cvsweb/arabeyes/vim
            -or-
            http://www.arabeyes.org/~nadim/vim/vim-shaping.tar.gz

            Please CC those that are already included.

            - Nadim


            __________________________________________________
            Do You Yahoo!?
            Buy the perfect holiday gifts at Yahoo! Shopping.
            http://shopping.yahoo.com
          • Nadim Shaikli
            On Tue, 4 Dec 2001 13:25:36 -0500 (EST) ... And other languages as well. ... I agree with you 100% -- Bidi is a MUST for any right-to-left language, as you
            Message 5 of 18 , Dec 4, 2001
            • 0 Attachment
              On Tue, 4 Dec 2001 13:25:36 -0500 (EST)
              "Yousof Mortazavi" <yousof at engsoc dot org> wrote:
              >
              > On Tue, 4 Dec 2001, David Harrison Jr. wrote:
              >
              > >
              > > What about numbers? Are they drawn in the same direction? I seem to
              > > remember there was one language that had its numbers drawn in the
              > > opposite direction from its words, thus making it a combination of
              > > "rightleft" and "leftright" type of writing. If this is the case, then
              > > drawing everything in one direction may not be flexible enough, no
              > > matter which direction you are drawing.
              >
              > Yes, you are right about numbers rendered as left-to-right. This
              > whole handling of right-to-left and left-to-right gets pretty
              > complicated especially when you start embedding texts from
              > left-to-right scripts within right-to-left scripts. It is,
              > however, clearly outlined in the Unicode Standard Annex #9:
              > "The Bidirectonal Algorithm" available at
              >
              > http://www.unicode.org/unicode/reports/tr9/
              >
              > As well libraries have been written, the most complete of which
              > is "fribidi" available from
              >
              > http://fribidi.sourceforge.net
              >
              > Noteworthy to say, that implementing BiDi for Vim will ease the
              > implementation of right-to-left scripts such as Arabic and
              > Hebrew.

              And other languages as well.

              > Here I'd like to propose a "BiDi" mode in Vim (just as the
              > current RL mode) that when active, characters entered get
              > shuffled according to the BiDi Algorithm as described by the
              > Unicode Standard. This mode should of course coexist with the
              > current RL mode such that when this mode is active, the base
              > direction is right-to-left, nevertheless, characters that possess
              > left-to-right behaviour are rendered as such.
              >
              > I would like to work on this project in conjunction with any
              > group that feels the need for proper Bidirectional behaviour. It
              > is, after all, part of the Unicode Standard, and the current
              > Unicode implementation of Vim would be incomplete without it.

              I agree with you 100% -- Bidi is a MUST for any right-to-left
              language, as you note above, if one were to intermix languages.
              I would like to stress, though, that Bidi is an ambitious plan
              that will undoubtedly mean lots of code change (do you have an
              estimate of the work required ?). As such, I hope that shaping
              will not take a back-seat since its almost complete and requires,
              as far as we can see, a potential fix/enhancement. Shaping, like
              Bidi, is also required by most (if not all) right-left languages
              (Arabic, Hebrew, Farsi, etc).

              I am glad to see that both Bidi and Shaping are finally being looked
              into seriously in regards to VIM.

              > Bram, do you approve?

              - Nadim


              __________________________________________________
              Do You Yahoo!?
              Buy the perfect holiday gifts at Yahoo! Shopping.
              http://shopping.yahoo.com
            • Ron Aaron
              ... Just a small correction - Hebrew only has five letters which are differently shaped (at the end of words, only) so people inputting Hebrew just input those
              Message 6 of 18 , Dec 4, 2001
              • 0 Attachment
                Nadim Shaikli <shaikli@...> writes:
                >as far as we can see, a potential fix/enhancement. Shaping, like
                >Bidi, is also required by most (if not all) right-left languages
                >(Arabic, Hebrew, Farsi, etc).

                Just a small correction - Hebrew only has five letters which are differently
                shaped (at the end of words, only) so people inputting Hebrew just input those
                characters directly, rather than relying on the underlying software to reshape
                the input letters. For Arabic, it is almost required to shape, because there
                are so many possibilities.

                Best regards,
                Ron
              • David Harrison Jr.
                ... Agreed. Now that I m thinking about these things, I wonder if certain Asian texts need to be displayed vertically? (topbottom mode?) ... Ok. I ve used it
                Message 7 of 18 , Dec 7, 2001
                • 0 Attachment
                  Nadim Shaikli wrote:

                  >Numbers aside (and we can live with them not being processed correctly
                  >until Bidi is adopted). The question still stands on how to modify the
                  >above noted logic/code to redisplay the characters in "rightleft" mode.
                  >
                  >

                  Agreed. Now that I'm thinking about these things, I wonder if certain
                  Asian texts need to be displayed vertically? (topbottom mode?)

                  >>Unfortunately, I'm not that familiar with the code to offer a solution
                  >>(I need to see Arabic examples.... lots of them).
                  >>
                  >
                  >Well, I would suggest downloading our work-in-progress code,
                  >
                  > http://www.arabeyes.org/cgi-bin/cvsweb/arabeyes/vim
                  >-or-
                  > http://www.arabeyes.org/~nadim/vim/vim-shaping.tar.gz
                  >


                  Ok. I've used it awhile now. Lots of screen refresh problems when you
                  are in an ISO-8859-x type of setup.

                  The Arabic input is interesting. I press 'q','w' which show a slightly
                  different set of characters than typing 'q', ' ', 'w'. I press CTRL-L
                  and the "combined" 'q''w' sequence is indeed drawn differently.

                  Hmmm, I'll take a closer look...

                  --David H. Jr.
                • Nadim Shaikli
                  On Fri, 07 Dec 2001 08:40:03 -0500 ... Actually seems like the refresh is there irrespective of which mode or setting you are in (ie. screen_line seems to
                  Message 8 of 18 , Dec 7, 2001
                  • 0 Attachment
                    On Fri, 07 Dec 2001 08:40:03 -0500
                    "David Harrison Jr." <david at motionrealityinc dot com> wrote:
                    >
                    > Nadim Shaikli wrote:
                    >
                    > >Well, I would suggest downloading our work-in-progress code,
                    > >
                    > > http://www.arabeyes.org/cgi-bin/cvsweb/arabeyes/vim
                    > >-or-
                    > > http://www.arabeyes.org/~nadim/vim/vim-shaping.tar.gz
                    > >
                    >
                    > Ok. I've used it awhile now. Lots of screen refresh problems when
                    > you are in an ISO-8859-x type of setup.

                    Actually seems like the refresh is there irrespective of which mode or
                    setting you are in (ie. screen_line seems to redisplay with each new
                    character even in "default" english).

                    > The Arabic input is interesting. I press 'q','w' which show a slightly
                    > different set of characters than typing 'q', ' ', 'w'. I press CTRL-L
                    > and the "combined" 'q''w' sequence is indeed drawn differently.
                    >
                    > Hmmm, I'll take a closer look...

                    I've solved most of the known problems, shaping is pretty functional
                    in the various different settings (including CTRL-L). As of now, there
                    are no known issues regarding shaping stand-alone (note - "as of now" :-)

                    Here's what's left (as I see it) regarding Arabic Shaping and support:

                    1. Get Bram's blessings/comments :-)
                    2. Fix the screen_line (in screen.c) to redisplay characters from
                    right-to-left in rightleft mode.
                    3. Add 'search' (ie. /) abilities in rightleft mode
                    a. Add rightleft-type of directionality in command-line
                    b. Add Arabic shaping to the command-line
                    c. Search non-shaped characters (ie. 'off_from')
                    4. Add Bidi abilities

                    Please note that the links noted above (in previous post) have been
                    updated and I highly recommend and encourage re-syncing.

                    Thanks for your help and interest.

                    - Nadim


                    __________________________________________________
                    Do You Yahoo!?
                    Send your FREE holiday greetings online!
                    http://greetings.yahoo.com
                  • David Harrison Jr.
                    ... This has me thinking that other languages could benefit from a generalization of this part of the code. Certain Asian texts are shown vertically, thus we
                    Message 9 of 18 , Dec 7, 2001
                    • 0 Attachment
                      Nadim Shaikli wrote:

                      >On Fri, 07 Dec 2001 08:40:03 -0500
                      > "David Harrison Jr." <david at motionrealityinc dot com> wrote:
                      >
                      >>Nadim Shaikli wrote:
                      >>
                      >>>Well, I would suggest downloading our work-in-progress code,
                      >>>
                      >>> http://www.arabeyes.org/cgi-bin/cvsweb/arabeyes/vim
                      >>>-or-
                      >>> http://www.arabeyes.org/~nadim/vim/vim-shaping.tar.gz
                      >>>
                      >>Ok. I've used it awhile now. Lots of screen refresh problems when
                      >>you are in an ISO-8859-x type of setup.
                      >>
                      >
                      >Actually seems like the refresh is there irrespective of which mode or
                      >setting you are in (ie. screen_line seems to redisplay with each new
                      >character even in "default" english).
                      >
                      >>The Arabic input is interesting. I press 'q','w' which show a slightly
                      >>different set of characters than typing 'q', ' ', 'w'. I press CTRL-L
                      >>and the "combined" 'q''w' sequence is indeed drawn differently.
                      >>
                      >>Hmmm, I'll take a closer look...
                      >>
                      >
                      >I've solved most of the known problems, shaping is pretty functional
                      >in the various different settings (including CTRL-L). As of now, there
                      >are no known issues regarding shaping stand-alone (note - "as of now" :-)
                      >
                      >Here's what's left (as I see it) regarding Arabic Shaping and support:
                      >
                      ...

                      >
                      > 2. Fix the screen_line (in screen.c) to redisplay characters from
                      > right-to-left in rightleft mode.
                      >

                      This has me thinking that other languages could benefit from a
                      generalization of this part of the code. Certain Asian texts are shown
                      vertically, thus we may (eventually) have a set of options that allow us
                      to have the main flow of text be either vertical or horizontal. Within
                      each of these two modes the text could be shown one way, or the other
                      "reversed" way (i.e. top-to-bottom or bottom-to-top).

                      I can't even imagine how frustrating it must be for a Vim user that
                      reads vertical text to be forced to read in a horizontal direction (from
                      within Vim). Maybe rotating the screen 90 degrees would be the easiest
                      thing to do... problem solved! :-)

                      >
                      >Please note that the links noted above (in previous post) have been
                      >updated and I highly recommend and encourage re-syncing.
                      >

                      I re-synced. The main redraw problems are now gone.
                      I notice that when I switch back and forth between rightleft and
                      norightleft mode, different Arabic "shaped words" will be displayed
                      (they are the same word, but displayed differently). I assume that they
                      are only correct when in rightleft mode.

                      >Thanks for your help and interest.
                      >

                      I don't think I was of any help, but I'm definitely interested in
                      generalizing Vim's code to handle new languages.

                      --David H. Jr.
                    • Nadim Shaikli
                      On Fri, 07 Dec 2001 16:43:12 -0500 ... BTW: any comment on that list sent as it relates to the tasks at hand ? ... This is WAY beyond what I ve set to do and
                      Message 10 of 18 , Dec 7, 2001
                      • 0 Attachment
                        On Fri, 07 Dec 2001 16:43:12 -0500
                        "David Harrison Jr." <david at motionrealityinc dot com> wrote:
                        >
                        > Nadim Shaikli wrote:
                        > >
                        > > 2. Fix the screen_line (in screen.c) to redisplay characters from
                        > > right-to-left in rightleft mode.
                        > >

                        BTW: any comment on that list sent as it relates to the tasks at hand ?

                        > This has me thinking that other languages could benefit from a
                        > generalization of this part of the code. Certain Asian texts are shown
                        > vertically, thus we may (eventually) have a set of options that allow us
                        > to have the main flow of text be either vertical or horizontal. Within
                        > each of these two modes the text could be shown one way, or the other
                        > "reversed" way (i.e. top-to-bottom or bottom-to-top).
                        >
                        > I can't even imagine how frustrating it must be for a Vim user that
                        > reads vertical text to be forced to read in a horizontal direction (from
                        > within Vim). Maybe rotating the screen 90 degrees would be the easiest
                        > thing to do... problem solved! :-)

                        This is WAY beyond what I've set to do and I'd humbly be way overwhelmed
                        especially with Bidi on the way. Furthermore, I personally have no
                        immediate interest in "topbottom" mode :-) I'm certainly no expert
                        on top-bottom or bottom-top languages, but most allow the text to be
                        read in left-to-right mode (I think Chinese/Mandarin is that way).

                        > I notice that when I switch back and forth between rightleft and
                        > norightleft mode, different Arabic "shaped words" will be displayed
                        > (they are the same word, but displayed differently). I assume that they
                        > are only correct when in rightleft mode.

                        Yes, the only correct form is the one in "rightleft" mode.

                        BTW: just an FYI in case you were curious to delve into an unexplored area.

                        If you enter any shaped Arabic char and try to convert to HTML
                        a core-dump results. For what its worth the same thing happens
                        with hebrew in rightleft mode (with or without the new Arabic code).
                        So it doesn't sound like anything that was recently added or
                        introduced - thought I'd mention it.

                        - Nadim


                        __________________________________________________
                        Do You Yahoo!?
                        Send your FREE holiday greetings online!
                        http://greetings.yahoo.com
                      • Bram Moolenaar
                        ... I have no idea what the impact of implementing BiDi would be. It sounds complicated. In many places in the code the relation between the bytes of text in
                        Message 11 of 18 , Dec 14, 2001
                        • 0 Attachment
                          Yousof Mortazavi wrote:

                          > Noteworthy to say, that implementing BiDi for Vim will ease the
                          > implementation of right-to-left scripts such as Arabic and
                          > Hebrew.
                          >
                          > Here I'd like to propose a "BiDi" mode in Vim (just as the
                          > current RL mode) that when active, characters entered get
                          > shuffled according to the BiDi Algorithm as described by the
                          > Unicode Standard. This mode should of course coexist with the
                          > current RL mode such that when this mode is active, the base
                          > direction is right-to-left, nevertheless, characters that possess
                          > left-to-right behaviour are rendered as such.
                          >
                          > I would like to work on this project in conjunction with any
                          > group that feels the need for proper Bidirectional behaviour. It
                          > is, after all, part of the Unicode Standard, and the current
                          > Unicode implementation of Vim would be incomplete without it.
                          >
                          > Bram, do you approve?

                          I have no idea what the impact of implementing BiDi would be. It sounds
                          complicated. In many places in the code the relation between the bytes
                          of text in a line and the position on the display is used. The simplest
                          example is that the cursor position is on a character in the text, and
                          it's displayed where that character appears on screen. Thus a
                          computation needs to be done from byte number to column number. It gets
                          more complicated when using Tabs and unprintable characters. And the
                          reverse compuation is also needed (e.g., when clicking with the mouse).

                          The best way to find out how to do it is to attempt a few ways to
                          implement it and decide which one works best. Yes, that's a lot of
                          work...

                          --
                          From "know your smileys":
                          % Bike accident (a bit far-fetched, I suppose)

                          /// Bram Moolenaar -- Bram@... -- http://www.moolenaar.net \\\
                          ((( Creator of Vim -- http://vim.sf.net -- ftp://ftp.vim.org/pub/vim )))
                          \\\ Help me helping AIDS orphans in Uganda - http://iccf-holland.org ///
                        • Yousof Mortazavi
                          ... Luckily, this sort of code has been written and there s a library (fribidi, http://fribidi.sourceforge.net) that handles all such issues involving BiDi.
                          Message 12 of 18 , Dec 15, 2001
                          • 0 Attachment
                            On Fri, 14 Dec 2001, Bram Moolenaar wrote:

                            >
                            > Yousof Mortazavi wrote:
                            >
                            > > Noteworthy to say, that implementing BiDi for Vim will ease the
                            > > implementation of right-to-left scripts such as Arabic and
                            > > Hebrew.
                            > >
                            > > Here I'd like to propose a "BiDi" mode in Vim (just as the
                            > > current RL mode) that when active, characters entered get
                            > > shuffled according to the BiDi Algorithm as described by the
                            > > Unicode Standard. This mode should of course coexist with the
                            > > current RL mode such that when this mode is active, the base
                            > > direction is right-to-left, nevertheless, characters that possess
                            > > left-to-right behaviour are rendered as such.
                            > >
                            > > I would like to work on this project in conjunction with any
                            > > group that feels the need for proper Bidirectional behaviour. It
                            > > is, after all, part of the Unicode Standard, and the current
                            > > Unicode implementation of Vim would be incomplete without it.
                            > >
                            > > Bram, do you approve?
                            >
                            > I have no idea what the impact of implementing BiDi would be. It sounds
                            > complicated. In many places in the code the relation between the bytes
                            > of text in a line and the position on the display is used. The simplest
                            > example is that the cursor position is on a character in the text, and
                            > it's displayed where that character appears on screen. Thus a
                            > computation needs to be done from byte number to column number. It gets
                            > more complicated when using Tabs and unprintable characters. And the
                            > reverse compuation is also needed (e.g., when clicking with the mouse).

                            Luckily, this sort of code has been written and there's a library
                            (fribidi, http://fribidi.sourceforge.net) that handles all such
                            issues involving BiDi. The main task is finding the right spots
                            to insert the code. What's your policy on using libraries?

                            Also, is there any sort of documentation on the internals of Vim,
                            besides the code itself?

                            > The best way to find out how to do it is to attempt a few ways to
                            > implement it and decide which one works best. Yes, that's a lot of
                            > work...
                            >

                            It's an effort worth making.

                            --
                            -= Yousof Mortazavi <yousof@...> | Electrical Engineering =-
                            -= WWW: http://www.engsoc.org/~yousof/ | Carleton University =-
                          • Bram Moolenaar
                            ... I suppose that library only provides the algorithm(s). The integration into Vim is probably still a lot of work. There are many places that assume text
                            Message 13 of 18 , Dec 15, 2001
                            • 0 Attachment
                              > > I have no idea what the impact of implementing BiDi would be. It sounds
                              > > complicated. In many places in the code the relation between the bytes
                              > > of text in a line and the position on the display is used. The simplest
                              > > example is that the cursor position is on a character in the text, and
                              > > it's displayed where that character appears on screen. Thus a
                              > > computation needs to be done from byte number to column number. It gets
                              > > more complicated when using Tabs and unprintable characters. And the
                              > > reverse compuation is also needed (e.g., when clicking with the mouse).
                              >
                              > Luckily, this sort of code has been written and there's a library
                              > (fribidi, http://fribidi.sourceforge.net) that handles all such
                              > issues involving BiDi. The main task is finding the right spots
                              > to insert the code. What's your policy on using libraries?

                              I suppose that library only provides the algorithm(s). The integration
                              into Vim is probably still a lot of work. There are many places that
                              assume text is displayed in one direction only, it's unpredicatable what
                              needs to be changed.

                              I wonder, does the reordering need to be done when displaying text? Why
                              not do it when typing the text and store it like how it's displayed?

                              > Also, is there any sort of documentation on the internals of Vim,
                              > besides the code itself?

                              src/README.txt

                              --
                              In a world without walls and borders, who needs windows and gates?

                              /// Bram Moolenaar -- Bram@... -- http://www.moolenaar.net \\\
                              ((( Creator of Vim -- http://vim.sf.net -- ftp://ftp.vim.org/pub/vim )))
                              \\\ Help me helping AIDS orphans in Uganda - http://iccf-holland.org ///
                            • Yousof Mortazavi
                              ... That is true, but I m optimistic that with the help of Vim developers and yourself, I can code the fundamental aspects of BiDi. Then, some time is needed
                              Message 14 of 18 , Dec 15, 2001
                              • 0 Attachment
                                On Sat, 15 Dec 2001, Bram Moolenaar wrote:

                                > > Luckily, this sort of code has been written and there's a library
                                > > (fribidi, http://fribidi.sourceforge.net) that handles all such
                                > > issues involving BiDi. The main task is finding the right spots
                                > > to insert the code. What's your policy on using libraries?
                                >
                                > I suppose that library only provides the algorithm(s). The integration
                                > into Vim is probably still a lot of work. There are many places that
                                > assume text is displayed in one direction only, it's unpredicatable what
                                > needs to be changed.

                                That is true, but I'm optimistic that with the help of Vim
                                developers and yourself, I can code the fundamental aspects of
                                BiDi. Then, some time is needed until users submit bugs and we go
                                towards a complete implementation.

                                > I wonder, does the reordering need to be done when displaying text? Why
                                > not do it when typing the text and store it like how it's displayed?

                                Yes, it's absolutely critical.

                                The way text should be stored in memory is in a logical manner,
                                that is characters that come first must be stored first. As an
                                example, assume capital letters represent a right-to-left script,
                                and small letters a left-to-right script.
                                Then the string of text:
                                car is THE CAR in arabic
                                should be displayed as
                                car is RAC EHT in arabic

                                The former string is the logical order, while the latter is the
                                visual. These two are completely different. Searching and parsing
                                is impossible with visual strings.

                                --
                                -= Yousof Mortazavi <yousof@...> | Electrical Engineering =-
                                -= WWW: http://www.engsoc.org/~yousof/ | Carleton University =-
                              • Bram Moolenaar
                                ... This sounds extremely complicated. You are probably better off ignoring the left-right code, since it doesn t handle this kind of text at all. Suppose
                                Message 15 of 18 , Dec 15, 2001
                                • 0 Attachment
                                  Yousof Mortazavi wrote:

                                  > > I wonder, does the reordering need to be done when displaying text? Why
                                  > > not do it when typing the text and store it like how it's displayed?
                                  >
                                  > Yes, it's absolutely critical.
                                  >
                                  > The way text should be stored in memory is in a logical manner,
                                  > that is characters that come first must be stored first. As an
                                  > example, assume capital letters represent a right-to-left script,
                                  > and small letters a left-to-right script.
                                  > Then the string of text:
                                  > car is THE CAR in arabic
                                  > should be displayed as
                                  > car is RAC EHT in arabic
                                  >
                                  > The former string is the logical order, while the latter is the
                                  > visual. These two are completely different. Searching and parsing
                                  > is impossible with visual strings.

                                  This sounds extremely complicated. You are probably better off ignoring
                                  the left-right code, since it doesn't handle this kind of text at all.

                                  Suppose that when the cursor is moved with "l", the user expects it to
                                  go from "car is" to the 'R' of "RAC". The using "l" will move
                                  rightwards on the display, but downwards in text columns. I'm sure this
                                  is going to cause a lot of trouble. Would it be better to have "l" work
                                  like before (move in text columns), with the result that it moves from
                                  "car is" to the 'T' of "RAC EHT"? And then moves to the left over this
                                  Arabic text?

                                  Just one command and already we have a dilemma...

                                  --
                                  hundred-and-one symptoms of being an internet addict:
                                  112. You are amazed that anyone uses a phone without a modem on it...let
                                  alone hear actual voices.

                                  /// Bram Moolenaar -- Bram@... -- http://www.moolenaar.net \\\
                                  ((( Creator of Vim -- http://vim.sf.net -- ftp://ftp.vim.org/pub/vim )))
                                  \\\ Help me helping AIDS orphans in Uganda - http://iccf-holland.org ///
                                • Yousof Mortazavi
                                  ... I think both the normal and left-right modes should be modified, because BiDi is all about embeding and you can embed Arabic in English, or English in
                                  Message 16 of 18 , Dec 15, 2001
                                  • 0 Attachment
                                    On Sat, 15 Dec 2001, Bram Moolenaar wrote:

                                    > > Then the string of text:
                                    > > car is THE CAR in arabic
                                    > > should be displayed as
                                    > > car is RAC EHT in arabic

                                    > This sounds extremely complicated. You are probably better off ignoring
                                    > the left-right code, since it doesn't handle this kind of text at all.

                                    I think both the normal and left-right modes should be modified,
                                    because BiDi is all about embeding and you can embed Arabic in
                                    English, or English in Arabic.

                                    > Suppose that when the cursor is moved with "l", the user expects it to
                                    > go from "car is" to the 'R' of "RAC". The using "l" will move
                                    > rightwards on the display, but downwards in text columns. I'm sure this
                                    > is going to cause a lot of trouble. Would it be better to have "l" work
                                    > like before (move in text columns), with the result that it moves from
                                    > "car is" to the 'T' of "RAC EHT"? And then moves to the left over this
                                    > Arabic text?

                                    Following the logical string would be best, so yes the second
                                    case is more reasonable. This can be implemented with a map of
                                    the logical string to the visual string. Fribidi has a function
                                    for this.

                                    for example:
                                    logical: car is THE CAR in arabic
                                    visual: car is RAC EHT in arabic
                                    l2v map: 24 1 2 3 4 5 6 13 12 11 10 9 8 7 14 15 16 17 18 19 20 21 22 23
                                    ^^^^^^^^^^^^^^^^^
                                    note reverse order.

                                    So advancing the cursor would be based on the l2v map not the
                                    cursor position.

                                    --
                                    -= Yousof Mortazavi <yousof@...> | Electrical Engineering =-
                                    -= WWW: http://www.engsoc.org/~yousof/ | Carleton University =-
                                  • Nadim Shaikli
                                    ... I m not familiar with fribidi (yet) to know where its involvement starts and where it ends -- sounds like that s the first place we need to spend some time
                                    Message 17 of 18 , Dec 15, 2001
                                    • 0 Attachment
                                      --- Bram Moolenaar wrote:
                                      >
                                      > Yousof Mortazavi wrote:
                                      > > The way text should be stored in memory is in a logical manner,
                                      > > that is characters that come first must be stored first. As an
                                      > > example, assume capital letters represent a right-to-left script,
                                      > > and small letters a left-to-right script.
                                      > > Then the string of text:
                                      > > car is THE CAR in arabic
                                      > > should be displayed as
                                      > > car is RAC EHT in arabic
                                      > >
                                      > > The former string is the logical order, while the latter is the
                                      > > visual. These two are completely different. Searching and parsing
                                      > > is impossible with visual strings.
                                      >
                                      > This sounds extremely complicated. You are probably better off ignoring
                                      > the left-right code, since it doesn't handle this kind of text at all.
                                      >
                                      > Suppose that when the cursor is moved with "l", the user expects it to
                                      > go from "car is" to the 'R' of "RAC". The using "l" will move
                                      > rightwards on the display, but downwards in text columns. I'm sure this
                                      > is going to cause a lot of trouble. Would it be better to have "l" work
                                      > like before (move in text columns), with the result that it moves from
                                      > "car is" to the 'T' of "RAC EHT"? And then moves to the left over this
                                      > Arabic text?

                                      I'm not familiar with fribidi (yet) to know where its involvement starts
                                      and where it ends -- sounds like that's the first place we need to spend
                                      some time on to better define that boundary (of what fribidi will do vs.
                                      what vim will handle).

                                      But to Answer your question Bram - I'd assume an "l" move should end-up on
                                      'T' -- simply put, it seems reasonable to follow the physical/logical order
                                      in cursor movement (physical/logical -> what's stored on disk which
                                      corresponds to the order a user enter characters).

                                      > Just one command and already we have a dilemma...

                                      I don't think these are dilemma, I think this is just the learning curve
                                      (for you and us) to get upto speed - sounds like Yousof has done some
                                      research and might be a great asset to make this a reality (any resources
                                      other than fribidi that's you'd recommend ?).

                                      - Nadim

                                      __________________________________________________
                                      Do You Yahoo!?
                                      Check out Yahoo! Shopping and Yahoo! Auctions for all of
                                      your unique holiday gifts! Buy at http://shopping.yahoo.com
                                      or bid at http://auctions.yahoo.com
                                    • Yousof Mortazavi
                                      ... I agree. The best resource I know of is the Unicode Standard Annex #9 (The BiDi Algorithm) available online at http://www.unicode.org/unicode/reports/tr9/
                                      Message 18 of 18 , Dec 15, 2001
                                      • 0 Attachment
                                        On Sat, 15 Dec 2001, Nadim Shaikli wrote:

                                        > > Just one command and already we have a dilemma...
                                        >
                                        > I don't think these are dilemma, I think this is just the learning curve
                                        > (for you and us) to get upto speed - sounds like Yousof has done some
                                        > research and might be a great asset to make this a reality (any resources
                                        > other than fribidi that's you'd recommend ?).

                                        I agree. The best resource I know of is the Unicode Standard
                                        Annex #9 (The BiDi Algorithm) available online at

                                        http://www.unicode.org/unicode/reports/tr9/

                                        (Let me know if you find anything interesting.)
                                      Your message has been successfully submitted and would be delivered to recipients shortly.