Loading ...
Sorry, an error occurred while loading the content.

Fwd: Vim on OS X, (no)macatsui problem

Expand Messages
  • Nico Weber
    Forwarding this to vim_mac as Bjorn is not subscribed to vim_multibyte as far as i know. Kenneth, I guess it would help if you could post links to screenshots
    Message 1 of 9 , Oct 8, 2007
    • 0 Attachment
      Forwarding this to vim_mac as Bjorn is not subscribed to
      vim_multibyte as far as i know. Kenneth, I guess it would help if you
      could post links to screenshots of the text as it's supposed to look
      and of the garbled look, as well as the font you're using so we can
      reproduce this.

      Nico

      Begin forwarded message:

      > From: Kenneth Beesley <krbeesley@...>
      > Date: October 5, 2007 6:35:27 PM GMT+02:00
      > To: vim_multibyte@...
      > Subject: Re: Vim on OS X, (no)macatsui problem
      > Reply-To: vim_multibyte@...
      >
      >
      >
      > On 5 Oct 2007, at 03:59, Nico Weber wrote:
      >
      >>
      >> Hi Ken,
      >>
      >>>
      >>> In .gvimrc, if I specify
      >>>
      >>> set nomacatsui anti guifont=DejaVuAgain\ Sans\ Mono:h14
      >>>
      >>> then gvim renders Roman glyphs, from the Basic Multilingual Plane,
      >>> well,
      >>> but the Deseret glyphs (from the supplementary area) are rendered as
      >>> sequences of Roman glyphs and spaces. Completely garbled
      >>>
      >>>
      >>> If I change .gvimrc to
      >>>
      >>> set macatsui anti guifont=DejaVuAgain\ Sans\ Mono:h14
      >>>
      >>> (i.e. if I specify macatsui rather than nomacatsui, and this is the
      >>> only change)
      >>> then I see Roman and Deseret glyphs rendered as expected, but all
      >>> the
      >>> glyphs
      >>> look scraggly on the screen.
      >>>
      >>> Can anyone explain to me what is happening here and how I might get
      >>> sharp renderings of both BMP and supplementary glyphs?
      >>
      >> I don't expect this to work at all without 'macatsui'. My experience
      >> is that vim assigns not enough horizontal space to supparea glyphs
      >> (is that what "scraggly" means).
      >
      > Hello Nico,
      >
      > Thanks for the message. With 'nomacatsui' I see sharp, legible
      > glyphs.
      > By "scraggly" I mean thin glyphs, with thin, shaky lines. These
      > "scraggly"
      > glyphs are legible, but they look bad.
      >
      >
      >> This is because vim needs a
      >> monospaced font for correct display, and the supparea glyphs are too
      >> wide for the monospaced width of the current font (this can happen
      >> because the font is not monospaced for all glyphs or because some
      >> glyphs are subsitituted from other fonts, because they are missing in
      >> the current font). This also does happen for some BMP glyphs (U+0E5B
      >> ๛ for example, and many others).
      >
      > The Deseret glyphs (Unicode block starting U+10400) are alphabetic and
      > fit into the same width as the other glyphs. As far as I can tell,
      > the font I'm
      > using is monowidth. When merging the Deseret glyphs, I first reset
      > their
      > width to the width of the characters in the existing font (DejaVu
      > Sans Mono).
      > So whatever my problem is, it is not that the new glyphs I've added
      > are too
      > wide, or wider than the original glyphs.
      >
      >
      >>
      >> One way that _might_ work is to get MacVim ( http://
      >> code.google.com/p/
      >> macvim/ ), set its MMCellWidthMultiplier user default to something a
      >> bit larger than 1 (do `defaults write org.vim.MacVim
      >> MMCellWidthMultiplier 1.3`, see http://code.google.com/p/macvim/wiki/
      >> UserDefaults for more information) and use that. This widens up all
      >> glyphs, but perhaps it's good enough.
      >
      > As just explained, too-wide glyphs are not the problem, as far as I
      > can tell.
      >
      >>
      >> I don't know if it's possible at all to have a monospaced font that
      >> works for all writing systems. The Right Thing is probably to make
      >> vim work with variable width fonts, but I guess that's very very
      >> complicated and won't happen :-\
      >
      > For my current work, I just need a few alphabets (Roman, Shavian,
      > Deseret) that can all fit in a reasonable width. I don't need "all
      > writing
      > systems". I am (as far as I can tell) using a monowidth font. The
      > complication is that Shavian and Deseret are in the supplementary
      > area, and Vim 7.1 just recently added patch 116 that is supposed to
      > allow
      > glyphs from the supplementary area to be rendered, for the first time.
      > Previous to 116, you could edit supplementary characters, but even
      > with a proper font, vim couldn't display the glyphs from the
      > supplementary
      > area.
      >
      > Thanks again for your message,
      >
      > Ken
      >
      >
      >>
      >> Nico
      >>>
      >
      >
      >

      --~--~---------~--~----~------------~-------~--~----~
      You received this message from the "vim_mac" maillist.
      For more information, visit http://www.vim.org/maillist.php
      -~----------~----~----~----~------~----~------~--~---
    • Adam Byrtek
      ... Nico, are you sure this was an issue with MacVim and not Carbon Vim? I have a similar issue with Carbon Vim, either - unicode characters (just accented,
      Message 2 of 9 , Oct 8, 2007
      • 0 Attachment
        On 10/8/07, Nico Weber <nicolasweber@...> wrote:
        > Forwarding this to vim_mac as Bjorn is not subscribed to
        > vim_multibyte as far as i know. Kenneth, I guess it would help if you
        > could post links to screenshots of the text as it's supposed to look
        > and of the garbled look, as well as the font you're using so we can
        > reproduce this.

        Nico, are you sure this was an issue with MacVim and not Carbon Vim? I
        have a similar issue with Carbon Vim, either

        - unicode characters (just accented, not wide) garbled when nomacatsui
        set, antialiasing works,
        - or unicode characters displayed correctly, but in ugly way, with
        macatsui, antialiasing doesn't work, moreover in this case there were
        some refresh problems.

        No problems with MacVim, that is why only when MacVim came out I
        really started to use Vim on Mac.

        Regards,
        Adam


        --
        Adam Byrtek

        --~--~---------~--~----~------------~-------~--~----~
        You received this message from the "vim_mac" maillist.
        For more information, visit http://www.vim.org/maillist.php
        -~----------~----~----~----~------~----~------~--~---
      • Nico Weber
        ... I had the impression Kenneth said it didn t work with MacVim either. *look through mails*. Yes, he did, but in the other mail I forwarded. Sorry. But might
        Message 3 of 9 , Oct 8, 2007
        • 0 Attachment
          > Nico, are you sure this was an issue with MacVim and not Carbon Vim? I
          > have a similar issue with Carbon Vim, either

          I had the impression Kenneth said it didn't work with MacVim either.
          *look through mails*. Yes, he did, but in the other mail I forwarded.
          Sorry. But might still be interesting for reference. ?.

          Nico


          --~--~---------~--~----~------------~-------~--~----~
          You received this message from the "vim_mac" maillist.
          For more information, visit http://www.vim.org/maillist.php
          -~----------~----~----~----~------~----~------~--~---
        • björn
          ... Ugh. I tried sifting through the forwarded posts, but it was kind of hard to understand them. I will try to read the posts on google groups instead,
          Message 4 of 9 , Oct 8, 2007
          • 0 Attachment
            > > Nico, are you sure this was an issue with MacVim and not Carbon Vim? I
            > > have a similar issue with Carbon Vim, either
            >
            > I had the impression Kenneth said it didn't work with MacVim either.
            > *look through mails*. Yes, he did, but in the other mail I forwarded.
            > Sorry. But might still be interesting for reference. ?.

            Ugh. I tried sifting through the forwarded posts, but it was kind of
            hard to understand them. I will try to read the posts on google
            groups instead, unless somebody can summarize the problem(s) for me?


            /Björn

            --~--~---------~--~----~------------~-------~--~----~
            You received this message from the "vim_mac" maillist.
            For more information, visit http://www.vim.org/maillist.php
            -~----------~----~----~----~------~----~------~--~---
          • Nico Weber
            ... Would have been easier if you d Reply all d. Here s what I think ... You can enter desert characters by opening the Character Palette, putting deseret
            Message 5 of 9 , Oct 8, 2007
            • 0 Attachment
              > Ugh. I tried sifting through the forwarded posts, but it was kind of
              > hard to understand them. I will try to read the posts on google
              > groups instead, unless somebody can summarize the problem(s) for me?

              Would have been easier if you'd "Reply all"d. Here's what I think
              Kenneth problems are:

              > I just installed the latest MacVim and tried it with a version of
              > DejaVuSansMono.ttf, augmented
              > with (monowidth) glyphs, the same width as the original
              > DejaVuSansMono.ttf glyphs,
              > for the Deseret Alphabet block (U+10400). It doesn't seem to work
              > for me. When I select my
              > Deseret Alphabet keymap and try to type Deseret Alphabet, I see
              > pseudo glyphs in boxes
              > rendered on the screen.

              You can enter desert characters by opening the Character Palette,
              putting "deseret" in the search box at the bottom and ... well, you
              know the rest. MacVim displays a "character not found" sign which is
              probably the Right Thing as the default DejaVu font seems not to
              include these characters, but Kenneth uses a font that _does_ have
              them. Having access to Kenneth's font would help...

              He also reports that mapping numbers `:map 3 ...` doesn't work. I
              can't reproduce this.

              Nico

              --~--~---------~--~----~------------~-------~--~----~
              You received this message from the "vim_mac" maillist.
              For more information, visit http://www.vim.org/maillist.php
              -~----------~----~----~----~------~----~------~--~---
            • Nico Weber
              ... I got this one wrong. See the other thread for Kenneth s clarification. Sorry. Nico --~--~---------~--~----~------------~-------~--~----~ You received this
              Message 6 of 9 , Oct 8, 2007
              • 0 Attachment
                > He also reports that mapping numbers `:map 3 ...` doesn't work. I
                > can't reproduce this.

                I got this one wrong. See the other thread for Kenneth's
                clarification. Sorry.

                Nico


                --~--~---------~--~----~------------~-------~--~----~
                You received this message from the "vim_mac" maillist.
                For more information, visit http://www.vim.org/maillist.php
                -~----------~----~----~----~------~----~------~--~---
              • björn
                ... Hi Ken, I have looked into why MacVim fails to render the deseret glyphs and I now have an answer, but unfortunately no solution. The problem is that one
                Message 7 of 9 , Oct 13, 2007
                • 0 Attachment
                  > > He also reports that mapping numbers `:map 3 ...` doesn't work. I
                  > > can't reproduce this.
                  >
                  > I got this one wrong. See the other thread for Kenneth's
                  > clarification. Sorry.

                  Hi Ken,

                  I have looked into why MacVim fails to render the deseret glyphs and I
                  now have an answer, but unfortunately no solution.

                  The problem is that one deseret character for some reason takes up
                  _two_ characters when put in the text storage (I guess this have
                  something to do with Unicode?). Specifically, calling "length" on an
                  NSString containing one deseret character returns 2 instead of 1, as I
                  would expect.

                  Now, I do know how to fix this problem, but since Jiang is working on
                  moving his drawing code to MacVim I don't really want to spend any
                  time doing this, since the problem will disappear as soon as he is
                  finished. I'm sorry about that.


                  /Björn

                  --~--~---------~--~----~------------~-------~--~----~
                  You received this message from the "vim_mac" maillist.
                  For more information, visit http://www.vim.org/maillist.php
                  -~----------~----~----~----~------~----~------~--~---
                • Tim Allen
                  ... It sounds like NSString is using UTF-16 internally - contrary to what many programmers think, UTF-16 is a variable-length encoding like UTF-8 (where
                  Message 8 of 9 , Oct 13, 2007
                  • 0 Attachment
                    On Oct 14, 3:57 am, "björn" <bjorn.winck...@...> wrote:
                    > The problem is that one deseret character for some reason takes up
                    > _two_ characters when put in the text storage (I guess this have
                    > something to do with Unicode?). Specifically, calling "length" on an
                    > NSString containing one deseret character returns 2 instead of 1, as I
                    > would expect.

                    It sounds like NSString is using UTF-16 internally - contrary to what
                    many programmers think, UTF-16 is a variable-length encoding like
                    UTF-8 (where different characters are represented by different numbers
                    of bytes). Unicode code-points below U+10000 are represented as two
                    bytes (what NSString seems to be calling a 'character') while U+10000
                    and above are represented as 'surrogate pairs'; that is, a code-point
                    between U+D800 and U+DBFF signifies the most significant 10 bits,
                    followed by a code-point between U+DC00 and U+DFFF to specify the
                    least significant 10 bits.

                    Apparently Apple's stance on the issue is: "don't try to access
                    individual characters":

                    http://www.cocoabuilder.com/archive/message/cocoa/2007/9/7/188858


                    --~--~---------~--~----~------------~-------~--~----~
                    You received this message from the "vim_mac" maillist.
                    For more information, visit http://www.vim.org/maillist.php
                    -~----------~----~----~----~------~----~------~--~---
                  • björn
                    ... I m sorry about the confusion with posting this thread separately on vim_multibyte and vim_mac...I ll try to bring the diverging threads together by
                    Message 9 of 9 , Oct 14, 2007
                    • 0 Attachment
                      > > The problem is that one deseret character for some reason takes up
                      > > _two_ characters when put in the text storage (I guess this have
                      > > something to do with Unicode?). Specifically, calling "length" on an
                      > > NSString containing one deseret character returns 2 instead of 1, as I
                      > > would expect.
                      > >
                      > UTF-8 uses:
                      > 1 byte for each codepoint in the range U+0000 - U+007F
                      > 2 bytes for each codepoint in the range U+0080 - U+07FF
                      > 3 bytes for each codepoint in the range U+0800 - U+FFFF
                      > 4 bytes for each codepoint in the range U+10000 - U+1FFFFF
                      > Actually, current standards mandate that no codepoints higher than U+10FFFD
                      > will "ever" be used. (Vim supports up to U+3FFFFFFF, with up to 6 bytes per
                      > codepoint, following an earlier draft of the standard.)
                      >
                      > Unicode also has the notion of "composing characters", which are characters
                      > which are "superimposed" on the preceding character, possibly changing its
                      > shape. These are usually diacritics: most of the accents of Latin can be
                      > either precomposed or spacing-non-accented + composing-accent, but the
                      > optional vowel marks of Hebrew and Arabic exist only as composing characters.
                      >
                      > Since your Deseret characters are outside the BMP, each of them requires 4
                      > bytes in UTF-8 (also two 16-bit words in UTF-16 and one 32-bit doubleword in
                      > UTF-32); but maybe that's not what your measured "length" means? Does your
                      > NSString include a final null (as C strings do) or an initial bytecount (as
                      > Pascal strings do)? Or do your Deseret characters include "composing" elements?

                      I'm sorry about the confusion with posting this thread separately on
                      vim_multibyte and vim_mac...I'll try to bring the diverging threads
                      together by posting this reply to both groups.

                      Tim Allen replied to the vim_mac thread saying that NSString uses
                      utf-16 internally and this is indeed why it says one deseret char has
                      length 2 (since it needs two 16 bit chars to store one deseret char,
                      as has been pointed out already).

                      I was under the mistaken impression that NSString always returned
                      length 1 for one character (not counting composing characters), which
                      is why I thought MacVim would work in all situations except when
                      composing characters were used. Again, this can be fixed by getting
                      rid of the assumption that each line in the text storage has the same
                      length (as returned by NSString), but this is a rather big code
                      change.

                      Thanks to Tony and Tim for educating me on the finer points of Unicode... :-)


                      /Björn

                      --~--~---------~--~----~------------~-------~--~----~
                      You received this message from the "vim_mac" maillist.
                      For more information, visit http://www.vim.org/maillist.php
                      -~----------~----~----~----~------~----~------~--~---
                    Your message has been successfully submitted and would be delivered to recipients shortly.