Loading ...
Sorry, an error occurred while loading the content.

Re: Normalization

Expand Messages
  • Bram Moolenaar
    ... Not much. Ignoring case is mostly handled correctly. You can use the Z item to ignore differences in combining characters, that s about it. ... Vim
    Message 1 of 9 , Jul 29, 2003
    • 0 Attachment
      Behnam Esfahbod wrote:

      > Does Vim uses Unicode Normalization (in any way) in its search (and other
      > text tools and filters) ?

      Not much. Ignoring case is mostly handled correctly. You can use the
      "\Z" item to ignore differences in combining characters, that's about
      it.

      > We started enabling Persian/Arabic support in Vim, that many of them,
      > implemented in GNU libraries. And so, we are going to read the source
      > code ASAP. :)

      Vim can't rely on GNU libraries to be present, it would work only for a
      a few platforms. If some normalization is needed, it will have to be
      implemented in C code.

      --
      If someone questions your market projections, simply point out that your
      target market is "People who are nuts" and "People who will buy any damn
      thing". Nobody is going to tell you there aren't enough of those people
      to go around.
      (Scott Adams - The Dilbert principle)

      /// Bram Moolenaar -- Bram@... -- http://www.Moolenaar.net \\\
      /// Creator of Vim - Vi IMproved -- http://www.Vim.org \\\
      \\\ Project leader for A-A-P -- http://www.A-A-P.org ///
      \\\ Help AIDS victims, buy here: http://ICCF-Holland.org/click1.html ///
    • Behnam Esfahbod
      ... but unicode norm. is more than just latin-combining-characters. ... All of these GNU tools (most in glibc) are implemented in C and are FreeSoftware and
      Message 2 of 9 , Jul 29, 2003
      • 0 Attachment
        > > Does Vim uses Unicode Normalization (in any way) in its search (and other
        > > text tools and filters) ?
        >
        > Not much. Ignoring case is mostly handled correctly. You can use the
        > "\Z" item to ignore differences in combining characters, that's about
        > it.

        but unicode norm. is more than just latin-combining-characters.

        > > We started enabling Persian/Arabic support in Vim, that many of them,
        > > implemented in GNU libraries. And so, we are going to read the source
        > > code ASAP. :)
        >
        > Vim can't rely on GNU libraries to be present, it would work only for a
        > a few platforms. If some normalization is needed, it will have to be
        > implemented in C code.

        All of these GNU tools (most in glibc) are implemented in C and are
        FreeSoftware and OpenSource. We can only use these shared libraries on
        GNU-based OSs and put these libraries in Vim for other OSs.

        As your charity feeling for Vim, there's better to support RTL and Bidi
        for Persian/Arabic/Hebrew users!

        If you don't want to import RTL and Bidi support into main Vim, we have to
        make an Arabic distro. that it doesn't seem good.

        We want to make vim support Persian in Linux, but there's very nice if we
        make it available in all supported OSs.

        --
        Behnam Esfahbod ......[ http://esfahbod.info | behnam(a)esfahbod.info ]
      • Bram Moolenaar
        ... Yes, it s currently limited in functionality. ... I don t want to include GNU code, because it means I am forced to use the GPL. Vim uses a different
        Message 3 of 9 , Jul 29, 2003
        • 0 Attachment
          Behnam Esfahbod wrote:

          > > > Does Vim uses Unicode Normalization (in any way) in its search (and other
          > > > text tools and filters) ?
          > >
          > > Not much. Ignoring case is mostly handled correctly. You can use the
          > > "\Z" item to ignore differences in combining characters, that's about
          > > it.
          >
          > but unicode norm. is more than just latin-combining-characters.

          Yes, it's currently limited in functionality.

          > > > We started enabling Persian/Arabic support in Vim, that many of them,
          > > > implemented in GNU libraries. And so, we are going to read the source
          > > > code ASAP. :)
          > >
          > > Vim can't rely on GNU libraries to be present, it would work only for a
          > > a few platforms. If some normalization is needed, it will have to be
          > > implemented in C code.
          >
          > All of these GNU tools (most in glibc) are implemented in C and are
          > FreeSoftware and OpenSource. We can only use these shared libraries on
          > GNU-based OSs and put these libraries in Vim for other OSs.

          I don't want to include GNU code, because it means I am forced to use
          the GPL. Vim uses a different license. It's exactly this feature of
          the GPL that I have problems with: I want to be free to select the
          license for code that I write myself.

          > As your charity feeling for Vim, there's better to support RTL and Bidi
          > for Persian/Arabic/Hebrew users!
          >
          > If you don't want to import RTL and Bidi support into main Vim, we have to
          > make an Arabic distro. that it doesn't seem good.
          >
          > We want to make vim support Persian in Linux, but there's very nice if we
          > make it available in all supported OSs.

          I'm open to add better support for foreign languages. But since only
          few people will use the functionality, we have to make sure that the
          code is as small as possible.

          --
          The average life of an organization chart is six months. You can safely
          ignore any order from your boss that would take six months to complete.
          (Scott Adams - The Dilbert principle)

          /// Bram Moolenaar -- Bram@... -- http://www.Moolenaar.net \\\
          /// Creator of Vim - Vi IMproved -- http://www.Vim.org \\\
          \\\ Project leader for A-A-P -- http://www.A-A-P.org ///
          \\\ Help AIDS victims, buy here: http://ICCF-Holland.org/click1.html ///
        • Behnam Esfahbod
          I think my prev. reply was very bad-written. FriBibi is an implementation of Unicode Bidi alg. and it s Arabic-script prof. and current mentainer is Behdad, my
          Message 4 of 9 , Jul 29, 2003
          • 0 Attachment
            I think my prev. reply was very bad-written.

            FriBibi is an implementation of Unicode Bidi alg. and it's Arabic-script
            prof. and current mentainer is Behdad, my brother and coworker, and we (I
            and Amir) want to use FriBidi for this work. [as we know many about it,
            and we couldn't write another special as good as it]

            And also something similar for Sort and Normalization.


            On Tue, 29 Jul 2003, Bram Moolenaar wrote:

            > Date: Tue, 29 Jul 2003 18:42:03 +0200
            > From: Bram Moolenaar <Bram@...>
            > To: Behnam Esfahbod <behnam@...>
            > Cc: vim-multibyte@..., Amir Hedayati <amir@...>,
            > Behdad Esfahbod <behdad@...>
            > Subject: Re: Normalization
            >
            >
            > Behnam Esfahbod wrote:
            >
            > > > > Does Vim uses Unicode Normalization (in any way) in its search (and other
            > > > > text tools and filters) ?
            > > >
            > > > Not much. Ignoring case is mostly handled correctly. You can use the
            > > > "\Z" item to ignore differences in combining characters, that's about
            > > > it.
            > >
            > > but unicode norm. is more than just latin-combining-characters.
            >
            > Yes, it's currently limited in functionality.
            >
            > > > > We started enabling Persian/Arabic support in Vim, that many of them,
            > > > > implemented in GNU libraries. And so, we are going to read the source
            > > > > code ASAP. :)
            > > >
            > > > Vim can't rely on GNU libraries to be present, it would work only for a
            > > > a few platforms. If some normalization is needed, it will have to be
            > > > implemented in C code.
            > >
            > > All of these GNU tools (most in glibc) are implemented in C and are
            > > FreeSoftware and OpenSource. We can only use these shared libraries on
            > > GNU-based OSs and put these libraries in Vim for other OSs.
            >
            > I don't want to include GNU code, because it means I am forced to use
            > the GPL. Vim uses a different license. It's exactly this feature of
            > the GPL that I have problems with: I want to be free to select the
            > license for code that I write myself.
            >
            > > As your charity feeling for Vim, there's better to support RTL and Bidi
            > > for Persian/Arabic/Hebrew users!
            > >
            > > If you don't want to import RTL and Bidi support into main Vim, we have to
            > > make an Arabic distro. that it doesn't seem good.
            > >
            > > We want to make vim support Persian in Linux, but there's very nice if we
            > > make it available in all supported OSs.
            >
            > I'm open to add better support for foreign languages. But since only
            > few people will use the functionality, we have to make sure that the
            > code is as small as possible.
            >
            >

            --
            Behnam Esfahbod ......[ http://esfahbod.info | behnam(a)esfahbod.info ]
          • Bram Moolenaar
            ... Oh, if you have written code yourself and are willing to let me include it with Vim under the Vim license, then that part is taken care of. Note that the
            Message 5 of 9 , Jul 29, 2003
            • 0 Attachment
              Behnam Esfahbod wrote:

              > I think my prev. reply was very bad-written.
              >
              > FriBibi is an implementation of Unicode Bidi alg. and it's Arabic-script
              > prof. and current mentainer is Behdad, my brother and coworker, and we (I
              > and Amir) want to use FriBidi for this work. [as we know many about it,
              > and we couldn't write another special as good as it]
              >
              > And also something similar for Sort and Normalization.

              Oh, if you have written code yourself and are willing to let me include
              it with Vim under the Vim license, then that part is taken care of.
              Note that the Vim license is very close to the GPL, mostly a bit more
              free.

              Since I don't know Arabic myself I can hardly do any work on this.
              Hopefully you will be able to find the right place to add new stuff to
              Vim. You can always ask me if you have specific questions.

              --
              You have heard the saying that if you put a thousand monkeys in a room with a
              thousand typewriters and waited long enough, eventually you would have a room
              full of dead monkeys.
              (Scott Adams - The Dilbert principle)

              /// Bram Moolenaar -- Bram@... -- http://www.Moolenaar.net \\\
              /// Creator of Vim - Vi IMproved -- http://www.Vim.org \\\
              \\\ Project leader for A-A-P -- http://www.A-A-P.org ///
              \\\ Help AIDS victims, buy here: http://ICCF-Holland.org/click1.html ///
            • Behnam Esfahbod
              ... I forgot to say that now FriBidi is a GNU Project and the GNU implementation of Unicode BiDi algorithm. So I don t know that we can use it in another
              Message 6 of 9 , Jul 29, 2003
              • 0 Attachment
                On Tue, 29 Jul 2003, Bram Moolenaar wrote:
                >
                > Behnam Esfahbod wrote:
                > > FriBibi is an implementation of Unicode Bidi alg. and it's Arabic-script
                > > prof. and current mentainer is Behdad, my brother and coworker, and we (I
                > > and Amir) want to use FriBidi for this work. [as we know many about it,
                > > and we couldn't write another special as good as it]
                >
                > Oh, if you have written code yourself and are willing to let me include
                > it with Vim under the Vim license, then that part is taken care of.
                > Note that the Vim license is very close to the GPL, mostly a bit more
                > free.
                >
                > Since I don't know Arabic myself I can hardly do any work on this.
                > Hopefully you will be able to find the right place to add new stuff to
                > Vim. You can always ask me if you have specific questions.
                >

                I forgot to say that now FriBidi is a GNU Project and the GNU
                implementation of Unicode BiDi algorithm. So I don't know that we can use
                it in another license, or cannot.

                If not, I think we must use FriBidi to start and make our work on GPL,
                then make a non-GPL library for the main code.

                We think the most important parts are:
                - BiDi alg. for view, and edit;
                - Making Cursor work on Bidi (in both Logical and Vitual modes);
                - Normalization and Search Tools;

                As someone said in freenode.net/#vim, Vim uses bash-tools for filtering,
                (i.e. GNU sort for sort filter) and so we don't need to do anything for
                this. :)


                But ASAP we will update farsi.c for new ISIRI Persian Keyboard Standard
                and Unicode. (in about a week)

                --
                Behnam Esfahbod ......[ http://esfahbod.info | behnam(a)esfahbod.info ]
              • Bram Moolenaar
                ... [...] ... If you wrote the code, then you can distribute it under several licenses. The GPL doesn t mention this, but in discussions about making the Vim
                Message 7 of 9 , Jul 30, 2003
                • 0 Attachment
                  Behnam Esfahbod wrote:

                  > > Oh, if you have written code yourself and are willing to let me include
                  > > it with Vim under the Vim license, then that part is taken care of.
                  > > Note that the Vim license is very close to the GPL, mostly a bit more
                  > > free.

                  [...]
                  > I forgot to say that now FriBidi is a GNU Project and the GNU
                  > implementation of Unicode BiDi algorithm. So I don't know that we can use
                  > it in another license, or cannot.
                  >
                  > If not, I think we must use FriBidi to start and make our work on GPL,
                  > then make a non-GPL library for the main code.

                  If you wrote the code, then you can distribute it under several
                  licenses. The GPL doesn't mention this, but in discussions about making
                  the Vim license GPL compatible this became clear. Thus when you
                  distribute your software under the GPL, you can _also_ distribute it
                  under another license. You can even sell it and keep parts of the code
                  secret (not that this is a logical thing to do).

                  Thus you can give me permission to include the code with Vim under the
                  Vim license. That doesn't change what happens with the same code under
                  the GPL.

                  > We think the most important parts are:
                  > - BiDi alg. for view, and edit;
                  > - Making Cursor work on Bidi (in both Logical and Vitual modes);
                  > - Normalization and Search Tools;

                  That sounds very useful, and at the same time quite complicated to
                  implement. The relation between text in the buffer and the cursor
                  position will no longer be that simple. But since we already have to
                  take care of the 'rightleft' option, we don't have to start from
                  scratch.

                  > As someone said in freenode.net/#vim, Vim uses bash-tools for filtering,
                  > (i.e. GNU sort for sort filter) and so we don't need to do anything for
                  > this. :)

                  You can always use external tools if they are available. But it's slow
                  for normal editing.

                  > But ASAP we will update farsi.c for new ISIRI Persian Keyboard Standard
                  > and Unicode. (in about a week)

                  Is this really useful? Until now I thought this code was a dead end.

                  --
                  hundred-and-one symptoms of being an internet addict:
                  2. You kiss your girlfriend's home page.

                  /// Bram Moolenaar -- Bram@... -- http://www.Moolenaar.net \\\
                  /// Creator of Vim - Vi IMproved -- http://www.Vim.org \\\
                  \\\ Project leader for A-A-P -- http://www.A-A-P.org ///
                  \\\ Help AIDS victims, buy here: http://ICCF-Holland.org/click1.html ///
                • Bram Moolenaar
                  ... In that case a configure check would need to be used. Don t know how it would work on MS-Windows though. Is there a fribidi.dll available? Then it could
                  Message 8 of 9 , Jul 30, 2003
                  • 0 Attachment
                    Behdad Esfahbod wrote:

                    > > Thus you can give me permission to include the code with Vim under the
                    > > Vim license. That doesn't change what happens with the same code under
                    > > the GPL.
                    >
                    > I am the copyright holder of FriBidi. I think I can give
                    > permissions for inclusion of a single version of the code in Vim,
                    > but dealing with double licensing is a headache. BTW, better
                    > fribidi be used as an external library. Its LGPLed, so no
                    > problem at all.

                    In that case a configure check would need to be used. Don't know how it
                    would work on MS-Windows though. Is there a fribidi.dll available?
                    Then it could be loaded dynamically, like we do with other optional
                    libraries.

                    --
                    hundred-and-one symptoms of being an internet addict:
                    9. All your daydreaming is preoccupied with getting a faster connection to the
                    net: 28.8...ISDN...cable modem...T1...T3.

                    /// Bram Moolenaar -- Bram@... -- http://www.Moolenaar.net \\\
                    /// Creator of Vim - Vi IMproved -- http://www.Vim.org \\\
                    \\\ Project leader for A-A-P -- http://www.A-A-P.org ///
                    \\\ Help AIDS victims, buy here: http://ICCF-Holland.org/click1.html ///
                  Your message has been successfully submitted and would be delivered to recipients shortly.