Loading ...
Sorry, an error occurred while loading the content.
 

Re: VIM-6 - rightleft redisplay

Expand Messages
  • Ron Aaron
    ... Just a small correction - Hebrew only has five letters which are differently shaped (at the end of words, only) so people inputting Hebrew just input those
    Message 1 of 18 , Dec 4, 2001
      Nadim Shaikli <shaikli@...> writes:
      >as far as we can see, a potential fix/enhancement. Shaping, like
      >Bidi, is also required by most (if not all) right-left languages
      >(Arabic, Hebrew, Farsi, etc).

      Just a small correction - Hebrew only has five letters which are differently
      shaped (at the end of words, only) so people inputting Hebrew just input those
      characters directly, rather than relying on the underlying software to reshape
      the input letters. For Arabic, it is almost required to shape, because there
      are so many possibilities.

      Best regards,
      Ron
    • David Harrison Jr.
      ... Agreed. Now that I m thinking about these things, I wonder if certain Asian texts need to be displayed vertically? (topbottom mode?) ... Ok. I ve used it
      Message 2 of 18 , Dec 7, 2001
        Nadim Shaikli wrote:

        >Numbers aside (and we can live with them not being processed correctly
        >until Bidi is adopted). The question still stands on how to modify the
        >above noted logic/code to redisplay the characters in "rightleft" mode.
        >
        >

        Agreed. Now that I'm thinking about these things, I wonder if certain
        Asian texts need to be displayed vertically? (topbottom mode?)

        >>Unfortunately, I'm not that familiar with the code to offer a solution
        >>(I need to see Arabic examples.... lots of them).
        >>
        >
        >Well, I would suggest downloading our work-in-progress code,
        >
        > http://www.arabeyes.org/cgi-bin/cvsweb/arabeyes/vim
        >-or-
        > http://www.arabeyes.org/~nadim/vim/vim-shaping.tar.gz
        >


        Ok. I've used it awhile now. Lots of screen refresh problems when you
        are in an ISO-8859-x type of setup.

        The Arabic input is interesting. I press 'q','w' which show a slightly
        different set of characters than typing 'q', ' ', 'w'. I press CTRL-L
        and the "combined" 'q''w' sequence is indeed drawn differently.

        Hmmm, I'll take a closer look...

        --David H. Jr.
      • Nadim Shaikli
        On Fri, 07 Dec 2001 08:40:03 -0500 ... Actually seems like the refresh is there irrespective of which mode or setting you are in (ie. screen_line seems to
        Message 3 of 18 , Dec 7, 2001
          On Fri, 07 Dec 2001 08:40:03 -0500
          "David Harrison Jr." <david at motionrealityinc dot com> wrote:
          >
          > Nadim Shaikli wrote:
          >
          > >Well, I would suggest downloading our work-in-progress code,
          > >
          > > http://www.arabeyes.org/cgi-bin/cvsweb/arabeyes/vim
          > >-or-
          > > http://www.arabeyes.org/~nadim/vim/vim-shaping.tar.gz
          > >
          >
          > Ok. I've used it awhile now. Lots of screen refresh problems when
          > you are in an ISO-8859-x type of setup.

          Actually seems like the refresh is there irrespective of which mode or
          setting you are in (ie. screen_line seems to redisplay with each new
          character even in "default" english).

          > The Arabic input is interesting. I press 'q','w' which show a slightly
          > different set of characters than typing 'q', ' ', 'w'. I press CTRL-L
          > and the "combined" 'q''w' sequence is indeed drawn differently.
          >
          > Hmmm, I'll take a closer look...

          I've solved most of the known problems, shaping is pretty functional
          in the various different settings (including CTRL-L). As of now, there
          are no known issues regarding shaping stand-alone (note - "as of now" :-)

          Here's what's left (as I see it) regarding Arabic Shaping and support:

          1. Get Bram's blessings/comments :-)
          2. Fix the screen_line (in screen.c) to redisplay characters from
          right-to-left in rightleft mode.
          3. Add 'search' (ie. /) abilities in rightleft mode
          a. Add rightleft-type of directionality in command-line
          b. Add Arabic shaping to the command-line
          c. Search non-shaped characters (ie. 'off_from')
          4. Add Bidi abilities

          Please note that the links noted above (in previous post) have been
          updated and I highly recommend and encourage re-syncing.

          Thanks for your help and interest.

          - Nadim


          __________________________________________________
          Do You Yahoo!?
          Send your FREE holiday greetings online!
          http://greetings.yahoo.com
        • David Harrison Jr.
          ... This has me thinking that other languages could benefit from a generalization of this part of the code. Certain Asian texts are shown vertically, thus we
          Message 4 of 18 , Dec 7, 2001
            Nadim Shaikli wrote:

            >On Fri, 07 Dec 2001 08:40:03 -0500
            > "David Harrison Jr." <david at motionrealityinc dot com> wrote:
            >
            >>Nadim Shaikli wrote:
            >>
            >>>Well, I would suggest downloading our work-in-progress code,
            >>>
            >>> http://www.arabeyes.org/cgi-bin/cvsweb/arabeyes/vim
            >>>-or-
            >>> http://www.arabeyes.org/~nadim/vim/vim-shaping.tar.gz
            >>>
            >>Ok. I've used it awhile now. Lots of screen refresh problems when
            >>you are in an ISO-8859-x type of setup.
            >>
            >
            >Actually seems like the refresh is there irrespective of which mode or
            >setting you are in (ie. screen_line seems to redisplay with each new
            >character even in "default" english).
            >
            >>The Arabic input is interesting. I press 'q','w' which show a slightly
            >>different set of characters than typing 'q', ' ', 'w'. I press CTRL-L
            >>and the "combined" 'q''w' sequence is indeed drawn differently.
            >>
            >>Hmmm, I'll take a closer look...
            >>
            >
            >I've solved most of the known problems, shaping is pretty functional
            >in the various different settings (including CTRL-L). As of now, there
            >are no known issues regarding shaping stand-alone (note - "as of now" :-)
            >
            >Here's what's left (as I see it) regarding Arabic Shaping and support:
            >
            ...

            >
            > 2. Fix the screen_line (in screen.c) to redisplay characters from
            > right-to-left in rightleft mode.
            >

            This has me thinking that other languages could benefit from a
            generalization of this part of the code. Certain Asian texts are shown
            vertically, thus we may (eventually) have a set of options that allow us
            to have the main flow of text be either vertical or horizontal. Within
            each of these two modes the text could be shown one way, or the other
            "reversed" way (i.e. top-to-bottom or bottom-to-top).

            I can't even imagine how frustrating it must be for a Vim user that
            reads vertical text to be forced to read in a horizontal direction (from
            within Vim). Maybe rotating the screen 90 degrees would be the easiest
            thing to do... problem solved! :-)

            >
            >Please note that the links noted above (in previous post) have been
            >updated and I highly recommend and encourage re-syncing.
            >

            I re-synced. The main redraw problems are now gone.
            I notice that when I switch back and forth between rightleft and
            norightleft mode, different Arabic "shaped words" will be displayed
            (they are the same word, but displayed differently). I assume that they
            are only correct when in rightleft mode.

            >Thanks for your help and interest.
            >

            I don't think I was of any help, but I'm definitely interested in
            generalizing Vim's code to handle new languages.

            --David H. Jr.
          • Nadim Shaikli
            On Fri, 07 Dec 2001 16:43:12 -0500 ... BTW: any comment on that list sent as it relates to the tasks at hand ? ... This is WAY beyond what I ve set to do and
            Message 5 of 18 , Dec 7, 2001
              On Fri, 07 Dec 2001 16:43:12 -0500
              "David Harrison Jr." <david at motionrealityinc dot com> wrote:
              >
              > Nadim Shaikli wrote:
              > >
              > > 2. Fix the screen_line (in screen.c) to redisplay characters from
              > > right-to-left in rightleft mode.
              > >

              BTW: any comment on that list sent as it relates to the tasks at hand ?

              > This has me thinking that other languages could benefit from a
              > generalization of this part of the code. Certain Asian texts are shown
              > vertically, thus we may (eventually) have a set of options that allow us
              > to have the main flow of text be either vertical or horizontal. Within
              > each of these two modes the text could be shown one way, or the other
              > "reversed" way (i.e. top-to-bottom or bottom-to-top).
              >
              > I can't even imagine how frustrating it must be for a Vim user that
              > reads vertical text to be forced to read in a horizontal direction (from
              > within Vim). Maybe rotating the screen 90 degrees would be the easiest
              > thing to do... problem solved! :-)

              This is WAY beyond what I've set to do and I'd humbly be way overwhelmed
              especially with Bidi on the way. Furthermore, I personally have no
              immediate interest in "topbottom" mode :-) I'm certainly no expert
              on top-bottom or bottom-top languages, but most allow the text to be
              read in left-to-right mode (I think Chinese/Mandarin is that way).

              > I notice that when I switch back and forth between rightleft and
              > norightleft mode, different Arabic "shaped words" will be displayed
              > (they are the same word, but displayed differently). I assume that they
              > are only correct when in rightleft mode.

              Yes, the only correct form is the one in "rightleft" mode.

              BTW: just an FYI in case you were curious to delve into an unexplored area.

              If you enter any shaped Arabic char and try to convert to HTML
              a core-dump results. For what its worth the same thing happens
              with hebrew in rightleft mode (with or without the new Arabic code).
              So it doesn't sound like anything that was recently added or
              introduced - thought I'd mention it.

              - Nadim


              __________________________________________________
              Do You Yahoo!?
              Send your FREE holiday greetings online!
              http://greetings.yahoo.com
            • Bram Moolenaar
              ... I have no idea what the impact of implementing BiDi would be. It sounds complicated. In many places in the code the relation between the bytes of text in
              Message 6 of 18 , Dec 14, 2001
                Yousof Mortazavi wrote:

                > Noteworthy to say, that implementing BiDi for Vim will ease the
                > implementation of right-to-left scripts such as Arabic and
                > Hebrew.
                >
                > Here I'd like to propose a "BiDi" mode in Vim (just as the
                > current RL mode) that when active, characters entered get
                > shuffled according to the BiDi Algorithm as described by the
                > Unicode Standard. This mode should of course coexist with the
                > current RL mode such that when this mode is active, the base
                > direction is right-to-left, nevertheless, characters that possess
                > left-to-right behaviour are rendered as such.
                >
                > I would like to work on this project in conjunction with any
                > group that feels the need for proper Bidirectional behaviour. It
                > is, after all, part of the Unicode Standard, and the current
                > Unicode implementation of Vim would be incomplete without it.
                >
                > Bram, do you approve?

                I have no idea what the impact of implementing BiDi would be. It sounds
                complicated. In many places in the code the relation between the bytes
                of text in a line and the position on the display is used. The simplest
                example is that the cursor position is on a character in the text, and
                it's displayed where that character appears on screen. Thus a
                computation needs to be done from byte number to column number. It gets
                more complicated when using Tabs and unprintable characters. And the
                reverse compuation is also needed (e.g., when clicking with the mouse).

                The best way to find out how to do it is to attempt a few ways to
                implement it and decide which one works best. Yes, that's a lot of
                work...

                --
                From "know your smileys":
                % Bike accident (a bit far-fetched, I suppose)

                /// Bram Moolenaar -- Bram@... -- http://www.moolenaar.net \\\
                ((( Creator of Vim -- http://vim.sf.net -- ftp://ftp.vim.org/pub/vim )))
                \\\ Help me helping AIDS orphans in Uganda - http://iccf-holland.org ///
              • Yousof Mortazavi
                ... Luckily, this sort of code has been written and there s a library (fribidi, http://fribidi.sourceforge.net) that handles all such issues involving BiDi.
                Message 7 of 18 , Dec 15, 2001
                  On Fri, 14 Dec 2001, Bram Moolenaar wrote:

                  >
                  > Yousof Mortazavi wrote:
                  >
                  > > Noteworthy to say, that implementing BiDi for Vim will ease the
                  > > implementation of right-to-left scripts such as Arabic and
                  > > Hebrew.
                  > >
                  > > Here I'd like to propose a "BiDi" mode in Vim (just as the
                  > > current RL mode) that when active, characters entered get
                  > > shuffled according to the BiDi Algorithm as described by the
                  > > Unicode Standard. This mode should of course coexist with the
                  > > current RL mode such that when this mode is active, the base
                  > > direction is right-to-left, nevertheless, characters that possess
                  > > left-to-right behaviour are rendered as such.
                  > >
                  > > I would like to work on this project in conjunction with any
                  > > group that feels the need for proper Bidirectional behaviour. It
                  > > is, after all, part of the Unicode Standard, and the current
                  > > Unicode implementation of Vim would be incomplete without it.
                  > >
                  > > Bram, do you approve?
                  >
                  > I have no idea what the impact of implementing BiDi would be. It sounds
                  > complicated. In many places in the code the relation between the bytes
                  > of text in a line and the position on the display is used. The simplest
                  > example is that the cursor position is on a character in the text, and
                  > it's displayed where that character appears on screen. Thus a
                  > computation needs to be done from byte number to column number. It gets
                  > more complicated when using Tabs and unprintable characters. And the
                  > reverse compuation is also needed (e.g., when clicking with the mouse).

                  Luckily, this sort of code has been written and there's a library
                  (fribidi, http://fribidi.sourceforge.net) that handles all such
                  issues involving BiDi. The main task is finding the right spots
                  to insert the code. What's your policy on using libraries?

                  Also, is there any sort of documentation on the internals of Vim,
                  besides the code itself?

                  > The best way to find out how to do it is to attempt a few ways to
                  > implement it and decide which one works best. Yes, that's a lot of
                  > work...
                  >

                  It's an effort worth making.

                  --
                  -= Yousof Mortazavi <yousof@...> | Electrical Engineering =-
                  -= WWW: http://www.engsoc.org/~yousof/ | Carleton University =-
                • Bram Moolenaar
                  ... I suppose that library only provides the algorithm(s). The integration into Vim is probably still a lot of work. There are many places that assume text
                  Message 8 of 18 , Dec 15, 2001
                    > > I have no idea what the impact of implementing BiDi would be. It sounds
                    > > complicated. In many places in the code the relation between the bytes
                    > > of text in a line and the position on the display is used. The simplest
                    > > example is that the cursor position is on a character in the text, and
                    > > it's displayed where that character appears on screen. Thus a
                    > > computation needs to be done from byte number to column number. It gets
                    > > more complicated when using Tabs and unprintable characters. And the
                    > > reverse compuation is also needed (e.g., when clicking with the mouse).
                    >
                    > Luckily, this sort of code has been written and there's a library
                    > (fribidi, http://fribidi.sourceforge.net) that handles all such
                    > issues involving BiDi. The main task is finding the right spots
                    > to insert the code. What's your policy on using libraries?

                    I suppose that library only provides the algorithm(s). The integration
                    into Vim is probably still a lot of work. There are many places that
                    assume text is displayed in one direction only, it's unpredicatable what
                    needs to be changed.

                    I wonder, does the reordering need to be done when displaying text? Why
                    not do it when typing the text and store it like how it's displayed?

                    > Also, is there any sort of documentation on the internals of Vim,
                    > besides the code itself?

                    src/README.txt

                    --
                    In a world without walls and borders, who needs windows and gates?

                    /// Bram Moolenaar -- Bram@... -- http://www.moolenaar.net \\\
                    ((( Creator of Vim -- http://vim.sf.net -- ftp://ftp.vim.org/pub/vim )))
                    \\\ Help me helping AIDS orphans in Uganda - http://iccf-holland.org ///
                  • Yousof Mortazavi
                    ... That is true, but I m optimistic that with the help of Vim developers and yourself, I can code the fundamental aspects of BiDi. Then, some time is needed
                    Message 9 of 18 , Dec 15, 2001
                      On Sat, 15 Dec 2001, Bram Moolenaar wrote:

                      > > Luckily, this sort of code has been written and there's a library
                      > > (fribidi, http://fribidi.sourceforge.net) that handles all such
                      > > issues involving BiDi. The main task is finding the right spots
                      > > to insert the code. What's your policy on using libraries?
                      >
                      > I suppose that library only provides the algorithm(s). The integration
                      > into Vim is probably still a lot of work. There are many places that
                      > assume text is displayed in one direction only, it's unpredicatable what
                      > needs to be changed.

                      That is true, but I'm optimistic that with the help of Vim
                      developers and yourself, I can code the fundamental aspects of
                      BiDi. Then, some time is needed until users submit bugs and we go
                      towards a complete implementation.

                      > I wonder, does the reordering need to be done when displaying text? Why
                      > not do it when typing the text and store it like how it's displayed?

                      Yes, it's absolutely critical.

                      The way text should be stored in memory is in a logical manner,
                      that is characters that come first must be stored first. As an
                      example, assume capital letters represent a right-to-left script,
                      and small letters a left-to-right script.
                      Then the string of text:
                      car is THE CAR in arabic
                      should be displayed as
                      car is RAC EHT in arabic

                      The former string is the logical order, while the latter is the
                      visual. These two are completely different. Searching and parsing
                      is impossible with visual strings.

                      --
                      -= Yousof Mortazavi <yousof@...> | Electrical Engineering =-
                      -= WWW: http://www.engsoc.org/~yousof/ | Carleton University =-
                    • Bram Moolenaar
                      ... This sounds extremely complicated. You are probably better off ignoring the left-right code, since it doesn t handle this kind of text at all. Suppose
                      Message 10 of 18 , Dec 15, 2001
                        Yousof Mortazavi wrote:

                        > > I wonder, does the reordering need to be done when displaying text? Why
                        > > not do it when typing the text and store it like how it's displayed?
                        >
                        > Yes, it's absolutely critical.
                        >
                        > The way text should be stored in memory is in a logical manner,
                        > that is characters that come first must be stored first. As an
                        > example, assume capital letters represent a right-to-left script,
                        > and small letters a left-to-right script.
                        > Then the string of text:
                        > car is THE CAR in arabic
                        > should be displayed as
                        > car is RAC EHT in arabic
                        >
                        > The former string is the logical order, while the latter is the
                        > visual. These two are completely different. Searching and parsing
                        > is impossible with visual strings.

                        This sounds extremely complicated. You are probably better off ignoring
                        the left-right code, since it doesn't handle this kind of text at all.

                        Suppose that when the cursor is moved with "l", the user expects it to
                        go from "car is" to the 'R' of "RAC". The using "l" will move
                        rightwards on the display, but downwards in text columns. I'm sure this
                        is going to cause a lot of trouble. Would it be better to have "l" work
                        like before (move in text columns), with the result that it moves from
                        "car is" to the 'T' of "RAC EHT"? And then moves to the left over this
                        Arabic text?

                        Just one command and already we have a dilemma...

                        --
                        hundred-and-one symptoms of being an internet addict:
                        112. You are amazed that anyone uses a phone without a modem on it...let
                        alone hear actual voices.

                        /// Bram Moolenaar -- Bram@... -- http://www.moolenaar.net \\\
                        ((( Creator of Vim -- http://vim.sf.net -- ftp://ftp.vim.org/pub/vim )))
                        \\\ Help me helping AIDS orphans in Uganda - http://iccf-holland.org ///
                      • Yousof Mortazavi
                        ... I think both the normal and left-right modes should be modified, because BiDi is all about embeding and you can embed Arabic in English, or English in
                        Message 11 of 18 , Dec 15, 2001
                          On Sat, 15 Dec 2001, Bram Moolenaar wrote:

                          > > Then the string of text:
                          > > car is THE CAR in arabic
                          > > should be displayed as
                          > > car is RAC EHT in arabic

                          > This sounds extremely complicated. You are probably better off ignoring
                          > the left-right code, since it doesn't handle this kind of text at all.

                          I think both the normal and left-right modes should be modified,
                          because BiDi is all about embeding and you can embed Arabic in
                          English, or English in Arabic.

                          > Suppose that when the cursor is moved with "l", the user expects it to
                          > go from "car is" to the 'R' of "RAC". The using "l" will move
                          > rightwards on the display, but downwards in text columns. I'm sure this
                          > is going to cause a lot of trouble. Would it be better to have "l" work
                          > like before (move in text columns), with the result that it moves from
                          > "car is" to the 'T' of "RAC EHT"? And then moves to the left over this
                          > Arabic text?

                          Following the logical string would be best, so yes the second
                          case is more reasonable. This can be implemented with a map of
                          the logical string to the visual string. Fribidi has a function
                          for this.

                          for example:
                          logical: car is THE CAR in arabic
                          visual: car is RAC EHT in arabic
                          l2v map: 24 1 2 3 4 5 6 13 12 11 10 9 8 7 14 15 16 17 18 19 20 21 22 23
                          ^^^^^^^^^^^^^^^^^
                          note reverse order.

                          So advancing the cursor would be based on the l2v map not the
                          cursor position.

                          --
                          -= Yousof Mortazavi <yousof@...> | Electrical Engineering =-
                          -= WWW: http://www.engsoc.org/~yousof/ | Carleton University =-
                        • Nadim Shaikli
                          ... I m not familiar with fribidi (yet) to know where its involvement starts and where it ends -- sounds like that s the first place we need to spend some time
                          Message 12 of 18 , Dec 15, 2001
                            --- Bram Moolenaar wrote:
                            >
                            > Yousof Mortazavi wrote:
                            > > The way text should be stored in memory is in a logical manner,
                            > > that is characters that come first must be stored first. As an
                            > > example, assume capital letters represent a right-to-left script,
                            > > and small letters a left-to-right script.
                            > > Then the string of text:
                            > > car is THE CAR in arabic
                            > > should be displayed as
                            > > car is RAC EHT in arabic
                            > >
                            > > The former string is the logical order, while the latter is the
                            > > visual. These two are completely different. Searching and parsing
                            > > is impossible with visual strings.
                            >
                            > This sounds extremely complicated. You are probably better off ignoring
                            > the left-right code, since it doesn't handle this kind of text at all.
                            >
                            > Suppose that when the cursor is moved with "l", the user expects it to
                            > go from "car is" to the 'R' of "RAC". The using "l" will move
                            > rightwards on the display, but downwards in text columns. I'm sure this
                            > is going to cause a lot of trouble. Would it be better to have "l" work
                            > like before (move in text columns), with the result that it moves from
                            > "car is" to the 'T' of "RAC EHT"? And then moves to the left over this
                            > Arabic text?

                            I'm not familiar with fribidi (yet) to know where its involvement starts
                            and where it ends -- sounds like that's the first place we need to spend
                            some time on to better define that boundary (of what fribidi will do vs.
                            what vim will handle).

                            But to Answer your question Bram - I'd assume an "l" move should end-up on
                            'T' -- simply put, it seems reasonable to follow the physical/logical order
                            in cursor movement (physical/logical -> what's stored on disk which
                            corresponds to the order a user enter characters).

                            > Just one command and already we have a dilemma...

                            I don't think these are dilemma, I think this is just the learning curve
                            (for you and us) to get upto speed - sounds like Yousof has done some
                            research and might be a great asset to make this a reality (any resources
                            other than fribidi that's you'd recommend ?).

                            - Nadim

                            __________________________________________________
                            Do You Yahoo!?
                            Check out Yahoo! Shopping and Yahoo! Auctions for all of
                            your unique holiday gifts! Buy at http://shopping.yahoo.com
                            or bid at http://auctions.yahoo.com
                          • Yousof Mortazavi
                            ... I agree. The best resource I know of is the Unicode Standard Annex #9 (The BiDi Algorithm) available online at http://www.unicode.org/unicode/reports/tr9/
                            Message 13 of 18 , Dec 15, 2001
                              On Sat, 15 Dec 2001, Nadim Shaikli wrote:

                              > > Just one command and already we have a dilemma...
                              >
                              > I don't think these are dilemma, I think this is just the learning curve
                              > (for you and us) to get upto speed - sounds like Yousof has done some
                              > research and might be a great asset to make this a reality (any resources
                              > other than fribidi that's you'd recommend ?).

                              I agree. The best resource I know of is the Unicode Standard
                              Annex #9 (The BiDi Algorithm) available online at

                              http://www.unicode.org/unicode/reports/tr9/

                              (Let me know if you find anything interesting.)
                            Your message has been successfully submitted and would be delivered to recipients shortly.