Loading ...
Sorry, an error occurred while loading the content.

Normalization

Expand Messages
  • Behnam Esfahbod
    Does Vim uses Unicode Normalization (in any way) in its search (and other text tools and filters) ? We started enabling Persian/Arabic support in Vim, that
    Message 1 of 9 , Jul 29, 2003
    • 0 Attachment
      Does Vim uses Unicode Normalization (in any way) in its search (and other
      text tools and filters) ?

      We started enabling Persian/Arabic support in Vim, that many of them,
      implemented in GNU libraries. And so, we are going to read the source
      code ASAP. :)

      --
      Behnam Esfahbod ......[ http://esfahbod.info | behnam(a)esfahbod.info ]
    • Bram Moolenaar
      ... Not much. Ignoring case is mostly handled correctly. You can use the Z item to ignore differences in combining characters, that s about it. ... Vim
      Message 2 of 9 , Jul 29, 2003
      • 0 Attachment
        Behnam Esfahbod wrote:

        > Does Vim uses Unicode Normalization (in any way) in its search (and other
        > text tools and filters) ?

        Not much. Ignoring case is mostly handled correctly. You can use the
        "\Z" item to ignore differences in combining characters, that's about
        it.

        > We started enabling Persian/Arabic support in Vim, that many of them,
        > implemented in GNU libraries. And so, we are going to read the source
        > code ASAP. :)

        Vim can't rely on GNU libraries to be present, it would work only for a
        a few platforms. If some normalization is needed, it will have to be
        implemented in C code.

        --
        If someone questions your market projections, simply point out that your
        target market is "People who are nuts" and "People who will buy any damn
        thing". Nobody is going to tell you there aren't enough of those people
        to go around.
        (Scott Adams - The Dilbert principle)

        /// Bram Moolenaar -- Bram@... -- http://www.Moolenaar.net \\\
        /// Creator of Vim - Vi IMproved -- http://www.Vim.org \\\
        \\\ Project leader for A-A-P -- http://www.A-A-P.org ///
        \\\ Help AIDS victims, buy here: http://ICCF-Holland.org/click1.html ///
      • Behnam Esfahbod
        ... but unicode norm. is more than just latin-combining-characters. ... All of these GNU tools (most in glibc) are implemented in C and are FreeSoftware and
        Message 3 of 9 , Jul 29, 2003
        • 0 Attachment
          > > Does Vim uses Unicode Normalization (in any way) in its search (and other
          > > text tools and filters) ?
          >
          > Not much. Ignoring case is mostly handled correctly. You can use the
          > "\Z" item to ignore differences in combining characters, that's about
          > it.

          but unicode norm. is more than just latin-combining-characters.

          > > We started enabling Persian/Arabic support in Vim, that many of them,
          > > implemented in GNU libraries. And so, we are going to read the source
          > > code ASAP. :)
          >
          > Vim can't rely on GNU libraries to be present, it would work only for a
          > a few platforms. If some normalization is needed, it will have to be
          > implemented in C code.

          All of these GNU tools (most in glibc) are implemented in C and are
          FreeSoftware and OpenSource. We can only use these shared libraries on
          GNU-based OSs and put these libraries in Vim for other OSs.

          As your charity feeling for Vim, there's better to support RTL and Bidi
          for Persian/Arabic/Hebrew users!

          If you don't want to import RTL and Bidi support into main Vim, we have to
          make an Arabic distro. that it doesn't seem good.

          We want to make vim support Persian in Linux, but there's very nice if we
          make it available in all supported OSs.

          --
          Behnam Esfahbod ......[ http://esfahbod.info | behnam(a)esfahbod.info ]
        • Bram Moolenaar
          ... Yes, it s currently limited in functionality. ... I don t want to include GNU code, because it means I am forced to use the GPL. Vim uses a different
          Message 4 of 9 , Jul 29, 2003
          • 0 Attachment
            Behnam Esfahbod wrote:

            > > > Does Vim uses Unicode Normalization (in any way) in its search (and other
            > > > text tools and filters) ?
            > >
            > > Not much. Ignoring case is mostly handled correctly. You can use the
            > > "\Z" item to ignore differences in combining characters, that's about
            > > it.
            >
            > but unicode norm. is more than just latin-combining-characters.

            Yes, it's currently limited in functionality.

            > > > We started enabling Persian/Arabic support in Vim, that many of them,
            > > > implemented in GNU libraries. And so, we are going to read the source
            > > > code ASAP. :)
            > >
            > > Vim can't rely on GNU libraries to be present, it would work only for a
            > > a few platforms. If some normalization is needed, it will have to be
            > > implemented in C code.
            >
            > All of these GNU tools (most in glibc) are implemented in C and are
            > FreeSoftware and OpenSource. We can only use these shared libraries on
            > GNU-based OSs and put these libraries in Vim for other OSs.

            I don't want to include GNU code, because it means I am forced to use
            the GPL. Vim uses a different license. It's exactly this feature of
            the GPL that I have problems with: I want to be free to select the
            license for code that I write myself.

            > As your charity feeling for Vim, there's better to support RTL and Bidi
            > for Persian/Arabic/Hebrew users!
            >
            > If you don't want to import RTL and Bidi support into main Vim, we have to
            > make an Arabic distro. that it doesn't seem good.
            >
            > We want to make vim support Persian in Linux, but there's very nice if we
            > make it available in all supported OSs.

            I'm open to add better support for foreign languages. But since only
            few people will use the functionality, we have to make sure that the
            code is as small as possible.

            --
            The average life of an organization chart is six months. You can safely
            ignore any order from your boss that would take six months to complete.
            (Scott Adams - The Dilbert principle)

            /// Bram Moolenaar -- Bram@... -- http://www.Moolenaar.net \\\
            /// Creator of Vim - Vi IMproved -- http://www.Vim.org \\\
            \\\ Project leader for A-A-P -- http://www.A-A-P.org ///
            \\\ Help AIDS victims, buy here: http://ICCF-Holland.org/click1.html ///
          • Behnam Esfahbod
            I think my prev. reply was very bad-written. FriBibi is an implementation of Unicode Bidi alg. and it s Arabic-script prof. and current mentainer is Behdad, my
            Message 5 of 9 , Jul 29, 2003
            • 0 Attachment
              I think my prev. reply was very bad-written.

              FriBibi is an implementation of Unicode Bidi alg. and it's Arabic-script
              prof. and current mentainer is Behdad, my brother and coworker, and we (I
              and Amir) want to use FriBidi for this work. [as we know many about it,
              and we couldn't write another special as good as it]

              And also something similar for Sort and Normalization.


              On Tue, 29 Jul 2003, Bram Moolenaar wrote:

              > Date: Tue, 29 Jul 2003 18:42:03 +0200
              > From: Bram Moolenaar <Bram@...>
              > To: Behnam Esfahbod <behnam@...>
              > Cc: vim-multibyte@..., Amir Hedayati <amir@...>,
              > Behdad Esfahbod <behdad@...>
              > Subject: Re: Normalization
              >
              >
              > Behnam Esfahbod wrote:
              >
              > > > > Does Vim uses Unicode Normalization (in any way) in its search (and other
              > > > > text tools and filters) ?
              > > >
              > > > Not much. Ignoring case is mostly handled correctly. You can use the
              > > > "\Z" item to ignore differences in combining characters, that's about
              > > > it.
              > >
              > > but unicode norm. is more than just latin-combining-characters.
              >
              > Yes, it's currently limited in functionality.
              >
              > > > > We started enabling Persian/Arabic support in Vim, that many of them,
              > > > > implemented in GNU libraries. And so, we are going to read the source
              > > > > code ASAP. :)
              > > >
              > > > Vim can't rely on GNU libraries to be present, it would work only for a
              > > > a few platforms. If some normalization is needed, it will have to be
              > > > implemented in C code.
              > >
              > > All of these GNU tools (most in glibc) are implemented in C and are
              > > FreeSoftware and OpenSource. We can only use these shared libraries on
              > > GNU-based OSs and put these libraries in Vim for other OSs.
              >
              > I don't want to include GNU code, because it means I am forced to use
              > the GPL. Vim uses a different license. It's exactly this feature of
              > the GPL that I have problems with: I want to be free to select the
              > license for code that I write myself.
              >
              > > As your charity feeling for Vim, there's better to support RTL and Bidi
              > > for Persian/Arabic/Hebrew users!
              > >
              > > If you don't want to import RTL and Bidi support into main Vim, we have to
              > > make an Arabic distro. that it doesn't seem good.
              > >
              > > We want to make vim support Persian in Linux, but there's very nice if we
              > > make it available in all supported OSs.
              >
              > I'm open to add better support for foreign languages. But since only
              > few people will use the functionality, we have to make sure that the
              > code is as small as possible.
              >
              >

              --
              Behnam Esfahbod ......[ http://esfahbod.info | behnam(a)esfahbod.info ]
            • Bram Moolenaar
              ... Oh, if you have written code yourself and are willing to let me include it with Vim under the Vim license, then that part is taken care of. Note that the
              Message 6 of 9 , Jul 29, 2003
              • 0 Attachment
                Behnam Esfahbod wrote:

                > I think my prev. reply was very bad-written.
                >
                > FriBibi is an implementation of Unicode Bidi alg. and it's Arabic-script
                > prof. and current mentainer is Behdad, my brother and coworker, and we (I
                > and Amir) want to use FriBidi for this work. [as we know many about it,
                > and we couldn't write another special as good as it]
                >
                > And also something similar for Sort and Normalization.

                Oh, if you have written code yourself and are willing to let me include
                it with Vim under the Vim license, then that part is taken care of.
                Note that the Vim license is very close to the GPL, mostly a bit more
                free.

                Since I don't know Arabic myself I can hardly do any work on this.
                Hopefully you will be able to find the right place to add new stuff to
                Vim. You can always ask me if you have specific questions.

                --
                You have heard the saying that if you put a thousand monkeys in a room with a
                thousand typewriters and waited long enough, eventually you would have a room
                full of dead monkeys.
                (Scott Adams - The Dilbert principle)

                /// Bram Moolenaar -- Bram@... -- http://www.Moolenaar.net \\\
                /// Creator of Vim - Vi IMproved -- http://www.Vim.org \\\
                \\\ Project leader for A-A-P -- http://www.A-A-P.org ///
                \\\ Help AIDS victims, buy here: http://ICCF-Holland.org/click1.html ///
              • Behnam Esfahbod
                ... I forgot to say that now FriBidi is a GNU Project and the GNU implementation of Unicode BiDi algorithm. So I don t know that we can use it in another
                Message 7 of 9 , Jul 29, 2003
                • 0 Attachment
                  On Tue, 29 Jul 2003, Bram Moolenaar wrote:
                  >
                  > Behnam Esfahbod wrote:
                  > > FriBibi is an implementation of Unicode Bidi alg. and it's Arabic-script
                  > > prof. and current mentainer is Behdad, my brother and coworker, and we (I
                  > > and Amir) want to use FriBidi for this work. [as we know many about it,
                  > > and we couldn't write another special as good as it]
                  >
                  > Oh, if you have written code yourself and are willing to let me include
                  > it with Vim under the Vim license, then that part is taken care of.
                  > Note that the Vim license is very close to the GPL, mostly a bit more
                  > free.
                  >
                  > Since I don't know Arabic myself I can hardly do any work on this.
                  > Hopefully you will be able to find the right place to add new stuff to
                  > Vim. You can always ask me if you have specific questions.
                  >

                  I forgot to say that now FriBidi is a GNU Project and the GNU
                  implementation of Unicode BiDi algorithm. So I don't know that we can use
                  it in another license, or cannot.

                  If not, I think we must use FriBidi to start and make our work on GPL,
                  then make a non-GPL library for the main code.

                  We think the most important parts are:
                  - BiDi alg. for view, and edit;
                  - Making Cursor work on Bidi (in both Logical and Vitual modes);
                  - Normalization and Search Tools;

                  As someone said in freenode.net/#vim, Vim uses bash-tools for filtering,
                  (i.e. GNU sort for sort filter) and so we don't need to do anything for
                  this. :)


                  But ASAP we will update farsi.c for new ISIRI Persian Keyboard Standard
                  and Unicode. (in about a week)

                  --
                  Behnam Esfahbod ......[ http://esfahbod.info | behnam(a)esfahbod.info ]
                • Bram Moolenaar
                  ... [...] ... If you wrote the code, then you can distribute it under several licenses. The GPL doesn t mention this, but in discussions about making the Vim
                  Message 8 of 9 , Jul 30, 2003
                  • 0 Attachment
                    Behnam Esfahbod wrote:

                    > > Oh, if you have written code yourself and are willing to let me include
                    > > it with Vim under the Vim license, then that part is taken care of.
                    > > Note that the Vim license is very close to the GPL, mostly a bit more
                    > > free.

                    [...]
                    > I forgot to say that now FriBidi is a GNU Project and the GNU
                    > implementation of Unicode BiDi algorithm. So I don't know that we can use
                    > it in another license, or cannot.
                    >
                    > If not, I think we must use FriBidi to start and make our work on GPL,
                    > then make a non-GPL library for the main code.

                    If you wrote the code, then you can distribute it under several
                    licenses. The GPL doesn't mention this, but in discussions about making
                    the Vim license GPL compatible this became clear. Thus when you
                    distribute your software under the GPL, you can _also_ distribute it
                    under another license. You can even sell it and keep parts of the code
                    secret (not that this is a logical thing to do).

                    Thus you can give me permission to include the code with Vim under the
                    Vim license. That doesn't change what happens with the same code under
                    the GPL.

                    > We think the most important parts are:
                    > - BiDi alg. for view, and edit;
                    > - Making Cursor work on Bidi (in both Logical and Vitual modes);
                    > - Normalization and Search Tools;

                    That sounds very useful, and at the same time quite complicated to
                    implement. The relation between text in the buffer and the cursor
                    position will no longer be that simple. But since we already have to
                    take care of the 'rightleft' option, we don't have to start from
                    scratch.

                    > As someone said in freenode.net/#vim, Vim uses bash-tools for filtering,
                    > (i.e. GNU sort for sort filter) and so we don't need to do anything for
                    > this. :)

                    You can always use external tools if they are available. But it's slow
                    for normal editing.

                    > But ASAP we will update farsi.c for new ISIRI Persian Keyboard Standard
                    > and Unicode. (in about a week)

                    Is this really useful? Until now I thought this code was a dead end.

                    --
                    hundred-and-one symptoms of being an internet addict:
                    2. You kiss your girlfriend's home page.

                    /// Bram Moolenaar -- Bram@... -- http://www.Moolenaar.net \\\
                    /// Creator of Vim - Vi IMproved -- http://www.Vim.org \\\
                    \\\ Project leader for A-A-P -- http://www.A-A-P.org ///
                    \\\ Help AIDS victims, buy here: http://ICCF-Holland.org/click1.html ///
                  • Bram Moolenaar
                    ... In that case a configure check would need to be used. Don t know how it would work on MS-Windows though. Is there a fribidi.dll available? Then it could
                    Message 9 of 9 , Jul 30, 2003
                    • 0 Attachment
                      Behdad Esfahbod wrote:

                      > > Thus you can give me permission to include the code with Vim under the
                      > > Vim license. That doesn't change what happens with the same code under
                      > > the GPL.
                      >
                      > I am the copyright holder of FriBidi. I think I can give
                      > permissions for inclusion of a single version of the code in Vim,
                      > but dealing with double licensing is a headache. BTW, better
                      > fribidi be used as an external library. Its LGPLed, so no
                      > problem at all.

                      In that case a configure check would need to be used. Don't know how it
                      would work on MS-Windows though. Is there a fribidi.dll available?
                      Then it could be loaded dynamically, like we do with other optional
                      libraries.

                      --
                      hundred-and-one symptoms of being an internet addict:
                      9. All your daydreaming is preoccupied with getting a faster connection to the
                      net: 28.8...ISDN...cable modem...T1...T3.

                      /// Bram Moolenaar -- Bram@... -- http://www.Moolenaar.net \\\
                      /// Creator of Vim - Vi IMproved -- http://www.Vim.org \\\
                      \\\ Project leader for A-A-P -- http://www.A-A-P.org ///
                      \\\ Help AIDS victims, buy here: http://ICCF-Holland.org/click1.html ///
                    Your message has been successfully submitted and would be delivered to recipients shortly.