Loading ...
Sorry, an error occurred while loading the content.

Re: Unable to display Hebrew correctly on Mac OS/X

Expand Messages
  • björn
    Hi Ron, ... You are not telling us which version of Vim you are using but by the sounds of it you are using the outdated Carbon port. There is a newer port
    Message 1 of 13 , Mar 24, 2009
    • 0 Attachment
      Hi Ron,

      2009/3/24 Ron Aaron:
      >
      > Again: I am using GUI vim, not Console vim.  The 'termbidi' option
      > doesn't apply.
      >
      > The problem seems to be related to the ATSUI renderer, at least there
      > are some hints in 'gui_mac.c' that indicate this might be true.  It
      > may be a good thing to set "kATSULineDirectionTag" so the drawing
      > occurs correctly, but I don't know enough about Mac GUI to know what
      > to do exactly.
      >
      > If noone else chimes in with helpful information, I'll try to fix it
      > myself.

      You are not telling us which version of Vim you are using but by the
      sounds of it you are using the outdated Carbon port. There is a newer
      port called "MacVim" which can be downloaded here:

      http://code.google.com/p/macvim/

      I have also CCed this post to the vim_mac mailing list which is a more
      appropriate list for Mac specific problems.

      Please let me know how MacVim copes with your situation: if it doesn't
      work I'll look into it as soon as I get a chance.

      Björn

      --~--~---------~--~----~------------~-------~--~----~
      You received this message from the "vim_mac" maillist.
      For more information, visit http://www.vim.org/maillist.php
      -~----------~----~----~----~------~----~------~--~---
    • Ron Aaron
      Hi, Björn - I was able to install MacVim (from source) and incorporate my own patches - and it works much better than the Carbon port, thanks! However --
      Message 2 of 13 , Mar 25, 2009
      • 0 Attachment
        Hi, Björn -

        I was able to install MacVim (from source) and incorporate my own
        patches - and it works much better than the Carbon port, thanks!

        However -- there are interesting problems, which perhaps you guys can
        help me with.

        First issue: mixed English/Hebrew text. Set the guifont to whatever -
        the English font changes, the Hebrew font does not! So I did the
        "defaults write org.vim.MacVim MMNoFontSubstitution 1", and now the
        font for Hebrew does change to the 'guifont' setting. However, not
        every font which has Hebrew seems to work, and I truly don't
        understand what is going on here.

        Second issue: if I set guifont to say, "Ezra SIL" (not monospace, but
        MacVim seems to do something ok). If the text has "niqud", which are
        vowel points which are "combining characters", the characters do not
        get composed properly as they do on Linux or Windows vim... I'm trying
        to see what the option should be, but I cannot.

        Third problem: If I bring up hte font selector and click on various
        fonts, eventually the vim window doesn't display any text at all

        I'm using utf8 text.

        On Mar 24, 2:08 pm, björn <bjorn.winck...@...> wrote:
        > Hi Ron,
        >
        > 2009/3/24 Ron Aaron:
        >
        >
        >
        > > Again: I am using GUI vim, not Console vim.  The 'termbidi' option
        > > doesn't apply.
        >
        > > The problem seems to be related to the ATSUI renderer, at least there
        > > are some hints in 'gui_mac.c' that indicate this might be true.  It
        > > may be a good thing to set "kATSULineDirectionTag" so the drawing
        > > occurs correctly, but I don't know enough about Mac GUI to know what
        > > to do exactly.
        >
        > > If noone else chimes in with helpful information, I'll try to fix it
        > > myself.
        >
        > You are not telling us which version of Vim you are using but by the
        > sounds of it you are using the outdated Carbon port.  There is a newer
        > port called "MacVim" which can be downloaded here:
        >
        > http://code.google.com/p/macvim/
        >
        > I have also CCed this post to the vim_mac mailing list which is a more
        > appropriate list for Mac specific problems.
        >
        > Please let me know how MacVim copes with your situation: if it doesn't
        > work I'll look into it as soon as I get a chance.
        >
        > Björn

        --~--~---------~--~----~------------~-------~--~----~
        You received this message from the "vim_mac" maillist.
        For more information, visit http://www.vim.org/maillist.php
        -~----------~----~----~----~------~----~------~--~---
      • björn
        Hi Ron, ... I m glad there is some improvement! ... The font substitution is out of my control -- if you choose a font which does not have certain glyphs then
        Message 3 of 13 , Mar 25, 2009
        • 0 Attachment
          Hi Ron,

          2009/3/25 Ron Aaron:
          >
          > I was able to install MacVim (from source) and incorporate my own
          > patches - and it works much better than the Carbon port, thanks!

          I'm glad there is some improvement!

          > However -- there are interesting problems, which perhaps you guys can
          > help me with.
          >
          > First issue: mixed English/Hebrew text.  Set the guifont to whatever -
          > the English font changes, the Hebrew font does not!  So I did the
          > "defaults write org.vim.MacVim MMNoFontSubstitution 1", and now the
          > font for Hebrew does change to the 'guifont' setting.  However, not
          > every font which has Hebrew seems to work, and I truly don't
          > understand what is going on here.

          The font substitution is out of my control -- if you choose a font
          which does not have certain glyphs then Cocoa will choose a font
          "close" to the one you've selected. In this case it seems the font
          that is "close" is always the same one, regardless which font you set
          using ":set gfn=...". You may have some luck with ":set
          guifontwide=name-of-hebrew-font" in case Hebrew is displayed in a wide
          font (is it?).

          Other than that, there is not much to be done about this at the
          moment. Maybe sometime in the future it will be possible to set a
          list of fonts to fall back on in case a glyph isn't in the font used
          for 'gfn' but don't hold your breath...

          > Second issue: if I set guifont to say, "Ezra SIL" (not monospace, but
          > MacVim seems to do something ok).  If the text has "niqud", which are
          > vowel points which are "combining characters", the characters do not
          > get composed properly as they do on Linux or Windows vim... I'm trying
          > to see what the option should be, but I cannot.
          >
          > Third problem: If I bring up hte font selector and click on various
          > fonts, eventually the vim window doesn't display any text at all

          It would be very helpful if you could send me a file on which you can
          reproduce all of these issues so that I can try it out myself.

          As for using proportional fonts: this is intentionally permitted.
          MacVim will force such fonts to be monospaced by using the width of
          "m" for every glyph. It looks kind of ugly. :)

          Björn

          --~--~---------~--~----~------------~-------~--~----~
          You received this message from the "vim_mac" maillist.
          For more information, visit http://www.vim.org/maillist.php
          -~----------~----~----~----~------~----~------~--~---
        • Ron Aaron
          Hi, Björn - ... No, Hebrew doesn t use a wide font, the characters are all single width. One complication is the composing characters , but that isn t
          Message 4 of 13 , Mar 26, 2009
          • 0 Attachment
            Hi, Björn -

            On Mar 26, 12:05 am, björn <bjorn.winck...@...> wrote:
            > Hi Ron
            > The font substitution is out of my control -- if you choose a font
            > which does not have certain glyphs then Cocoa will choose a font
            > "close" to the one you've selected. In this case it seems the font
            > that is "close" is always the same one, regardless which font you set
            > using ":set gfn=...". You may have some luck with ":set
            > guifontwide=name-of-hebrew-font" in case Hebrew is displayed in a wide
            > font (is it?).

            No, Hebrew doesn't use a wide font, the characters are all single
            width. One
            complication is the 'composing characters', but that isn't unique to
            Hebrew.

            As I mentioned, by telling MacVim not to substitute, I can at least
            display
            Hebrew with different (appropriate) fonts.

            The strange thing is that the fonts *do* have Hebrew characters, but
            it may be
            that they are not tagged in such a way that OS/X can extract the
            correct
            glyphs. For example, the monospace font "Miriam Fixed" has Hebrew
            characters,
            as does "Courier New" (from MS) - but neither seems to be able to be
            used in
            MacVim for Hebrew, nor "Web Hebrew Monospace" nor "SBL Hebrew". But
            the
            non-monospace "Ezra SIL" works, as does "David MF" and various others
            (all
            non-monospace :( )

            > It would be very helpful if you could send me a file on which you can
            > reproduce all of these issues so that I can try it out myself.
            >
            > As for using proportional fonts: this is intentionally permitted.
            > MacVim will force such fonts to be monospaced by using the width of
            > "m" for every glyph. It looks kind of ugly. :)

            I have a set of two files: one with just the base characters and one
            with
            combining (niqud) characters. You can get the file here:

            http://ronware.org/vim-hebrew.zip

            Thank you for all your help!
            Best regards,
            Ron

            --~--~---------~--~----~------------~-------~--~----~
            You received this message from the "vim_mac" maillist.
            For more information, visit http://www.vim.org/maillist.php
            -~----------~----~----~----~------~----~------~--~---
          • björn
            ... Yeah, I don t understand this either. The Cocoa font system is very much like a black box (at least to me) -- it is hard to get any control over aspects
            Message 5 of 13 , Mar 27, 2009
            • 0 Attachment
              2009/3/26 Ron Aaron:
              >
              > The strange thing is that the fonts *do* have Hebrew characters, but
              > it may be
              > that they are not tagged in such a way that OS/X can extract the
              > correct
              > glyphs.  For example, the monospace font "Miriam Fixed" has Hebrew
              > characters,
              > as does "Courier New" (from MS) - but neither seems to be able to be
              > used in
              > MacVim for Hebrew, nor "Web Hebrew Monospace" nor "SBL Hebrew".  But
              > the
              > non-monospace "Ezra SIL" works, as does "David MF" and various others
              > (all
              > non-monospace :( )

              Yeah, I don't understand this either. The Cocoa font system is very
              much like a black box (at least to me) -- it is hard to get any
              control over aspects such as font substitution. There is probably
              some way to "massage" the font files to get this to work but I have no
              idea where you find information on this.

              >> It would be very helpful if you could send me a file on which you can
              >> reproduce all of these issues so that I can try it out myself.
              >
              > I have a set of two files: one with just the base characters and one
              > with
              > combining (niqud) characters.  You can get the file here:
              >
              >   http://ronware.org/vim-hebrew.zip

              Thanks, I see the problem now. I am aware that composing characters
              are problematic and I am not sure how easy it will be to fix with the
              default and ATSUI renderers. Still, I will take another look to
              refresh my memory (and it is good to have some example file to work on
              as a test).

              Failing that there is much more hope for the half-finished CoreText
              renderer I have yet to finish and push to the repo -- I am confident
              that I can get composing characters working properly with it. So, I
              might take another look at that as well. Anyway, I'll post again when
              I have something more to say on this.

              Björn

              --~--~---------~--~----~------------~-------~--~----~
              You received this message from the "vim_mac" maillist.
              For more information, visit http://www.vim.org/maillist.php
              -~----------~----~----~----~------~----~------~--~---
            • björn
              ... I never managed to reproduce this problem -- maybe it has something to do with which fonts are installed. I clicked through all my available fonts (using
              Message 6 of 13 , Mar 27, 2009
              • 0 Attachment
                2009/3/25 Ron Aaron:
                >
                > Third problem: If I bring up hte font selector and click on various
                > fonts, eventually the vim window doesn't display any text at all

                I never managed to reproduce this problem -- maybe it has something to
                do with which fonts are installed. I clicked through all my available
                fonts (using one of the Hebrew text files you sent) 10 times over and
                still no problems. Can you reproduce this problem by only clicking on
                "standard" fonts? Approximately how many fonts do you have to click
                on before the problem occurs?

                Björn

                --~--~---------~--~----~------------~-------~--~----~
                You received this message from the "vim_mac" maillist.
                For more information, visit http://www.vim.org/maillist.php
                -~----------~----~----~----~------~----~------~--~---
              • Ron Aaron
                Hmm. I can t seem to repro it now; perhaps rebooting the Mac helped somehow (there were system updates pending). If I can repro this particular issue, I ll
                Message 7 of 13 , Mar 29, 2009
                • 0 Attachment
                  Hmm. I can't seem to repro it now; perhaps rebooting the Mac helped
                  somehow (there were system updates pending). If I can repro this
                  particular issue, I'll give you specific steps.

                  On Mar 28, 1:15 am, björn <bjorn.winck...@...> wrote:
                  > 2009/3/25 Ron Aaron:
                  >
                  >
                  >
                  > > Third problem: If I bring up hte font selector and click on various
                  > > fonts, eventually the vim window doesn't display any text at all
                  >
                  > I never managed to reproduce this problem -- maybe it has something to
                  > do with which fonts are installed.  I clicked through all my available
                  > fonts (using one of the Hebrew text files you sent) 10 times over and
                  > still no problems.  Can you reproduce this problem by only clicking on
                  > "standard" fonts?  Approximately how many fonts do you have to click
                  > on before the problem occurs?
                  >
                  > Björn
                  --~--~---------~--~----~------------~-------~--~----~
                  You received this message from the "vim_mac" maillist.
                  For more information, visit http://www.vim.org/maillist.php
                  -~----------~----~----~----~------~----~------~--~---
                • Ron Aaron
                  I ll be anxious to test any such changes! Thanks, Ron ... --~--~---------~--~----~------------~-------~--~----~ You received this message from the vim_mac
                  Message 8 of 13 , Mar 29, 2009
                  • 0 Attachment
                    I'll be anxious to test any such changes!

                    Thanks,
                    Ron

                    On Mar 28, 12:41 am, björn <bjorn.winck...@...> wrote:
                    > 2009/3/26 Ron Aaron:
                    >
                    >
                    >
                    > > The strange thing is that the fonts *do* have Hebrew characters, but
                    > > it may be
                    > > that they are not tagged in such a way that OS/X can extract the
                    > > correct
                    > > glyphs.  For example, the monospace font "Miriam Fixed" has Hebrew
                    > > characters,
                    > > as does "Courier New" (from MS) - but neither seems to be able to be
                    > > used in
                    > > MacVim for Hebrew, nor "Web Hebrew Monospace" nor "SBL Hebrew".  But
                    > > the
                    > > non-monospace "Ezra SIL" works, as does "David MF" and various others
                    > > (all
                    > > non-monospace :( )
                    >
                    > Yeah, I don't understand this either.  The Cocoa font system is very
                    > much like a black box (at least to me) -- it is hard to get any
                    > control over aspects such as font substitution.  There is probably
                    > some way to "massage" the font files to get this to work but I have no
                    > idea where you find information on this.
                    >
                    > >> It would be very helpful if you could send me a file on which you can
                    > >> reproduce all of these issues so that I can try it out myself.
                    >
                    > > I have a set of two files: one with just the base characters and one
                    > > with
                    > > combining (niqud) characters.  You can get the file here:
                    >
                    > >  http://ronware.org/vim-hebrew.zip
                    >
                    > Thanks, I see the problem now.  I am aware that composing characters
                    > are problematic and I am not sure how easy it will be to fix with the
                    > default and ATSUI renderers.  Still, I will take another look to
                    > refresh my memory (and it is good to have some example file to work on
                    > as a test).
                    >
                    > Failing that there is much more hope for the half-finished CoreText
                    > renderer I have yet to finish and push to the repo -- I am confident
                    > that I can get composing characters working properly with it.  So, I
                    > might take another look at that as well.  Anyway, I'll post again when
                    > I have something more to say on this.
                    >
                    > Björn
                    --~--~---------~--~----~------------~-------~--~----~
                    You received this message from the "vim_mac" maillist.
                    For more information, visit http://www.vim.org/maillist.php
                    -~----------~----~----~----~------~----~------~--~---
                  • björn
                    ... Please test the Core Text renderer patch I posted in a separate thread and let me know what you think -- your test file renders much better with that
                    Message 9 of 13 , Mar 29, 2009
                    • 0 Attachment
                      2009/3/29 Ron Aaron:
                      > On Mar 28, 12:41 am, björn wrote:
                      >>
                      >> Thanks, I see the problem now.  I am aware that composing characters
                      >> are problematic and I am not sure how easy it will be to fix with the
                      >> default and ATSUI renderers.  Still, I will take another look to
                      >> refresh my memory (and it is good to have some example file to work on
                      >> as a test).
                      >>
                      >> Failing that there is much more hope for the half-finished CoreText
                      >> renderer I have yet to finish and push to the repo -- I am confident
                      >> that I can get composing characters working properly with it.  So, I
                      >> might take another look at that as well.  Anyway, I'll post again when
                      >> I have something more to say on this.
                      >
                      > I'll be anxious to test any such changes!

                      Please test the Core Text renderer patch I posted in a separate thread
                      and let me know what you think -- your test file renders much better
                      with that patch.

                      I was struggling for a couple of hours with the current renderer to
                      get it working with composing characters but I don't understand what
                      is going wrong. The Cocoa text system gets the correct characters
                      (letter + composing char) but then decides to render it as two glyphs:
                      letter first, then composing character next to it (and then the
                      composing character ends up on top of the next letter). I don't know
                      if I have the energy to sort this out at the moment -- I might try
                      asking for help on the cocoa-dev mailing list.

                      Björn

                      --~--~---------~--~----~------------~-------~--~----~
                      You received this message from the "vim_mac" maillist.
                      For more information, visit http://www.vim.org/maillist.php
                      -~----------~----~----~----~------~----~------~--~---
                    • Ron Aaron
                      Hi - The file does render much better with that (series of patches). And now it looks like a reasonable default happens when I select a font which
                      Message 10 of 13 , Mar 29, 2009
                      • 0 Attachment
                        Hi -

                        The file does render much better with that (series of patches). And
                        now it looks like a "reasonable" default happens when I select a font
                        which previously didn't work (some font I don't know which, is
                        substituted - even though the default is set not to substitute).
                        Anyway, the display is much better.

                        Hard to believe, but the display on Windows or GTK on Linux is better
                        still... but at least it is usable now on Mac.

                        Thanks for your hard work; hopefully we can get rid of the other odd
                        issues.

                        Best regards,
                        Ron

                        On Mar 29, 1:51 pm, björn <bjorn.winck...@...> wrote:
                        > 2009/3/29 Ron Aaron:
                        >
                        > > On Mar 28, 12:41 am, björn wrote:
                        >
                        > >> Thanks, I see the problem now.  I am aware that composing characters
                        > >> are problematic and I am not sure how easy it will be to fix with the
                        > >> default and ATSUI renderers.  Still, I will take another look to
                        > >> refresh my memory (and it is good to have some example file to work on
                        > >> as a test).
                        >
                        > >> Failing that there is much more hope for the half-finished CoreText
                        > >> renderer I have yet to finish and push to the repo -- I am confident
                        > >> that I can get composing characters working properly with it.  So, I
                        > >> might take another look at that as well.  Anyway, I'll post again when
                        > >> I have something more to say on this.
                        >
                        > > I'll be anxious to test any such changes!
                        >
                        > Please test the Core Text renderer patch I posted in a separate thread
                        > and let me know what you think -- your test file renders much better
                        > with that patch.
                        >
                        > I was struggling for a couple of hours with the current renderer to
                        > get it working with composing characters but I don't understand what
                        > is going wrong.  The Cocoa text system gets the correct characters
                        > (letter + composing char) but then decides to render it as two glyphs:
                        > letter first, then composing character next to it (and then the
                        > composing character ends up on top of the next letter).  I don't know
                        > if I have the energy to sort this out at the moment -- I might try
                        > asking for help on the cocoa-dev mailing list.
                        >
                        > Björn
                        --~--~---------~--~----~------------~-------~--~----~
                        You received this message from the "vim_mac" maillist.
                        For more information, visit http://www.vim.org/maillist.php
                        -~----------~----~----~----~------~----~------~--~---
                      • björn
                        ... Good. I could quite easily code this renderer so that you could specify which font to fall back on but I just don t know if this is worth it and if so, how
                        Message 11 of 13 , Mar 30, 2009
                        • 0 Attachment
                          2009/3/29 Ron Aaron:
                          >
                          > The file does render much better with that (series of patches).  And
                          > now it looks like a "reasonable" default happens when I select a font
                          > which previously didn't work (some font I don't know which, is
                          > substituted - even though the default is set not to substitute).
                          > Anyway, the display is much better.

                          Good.

                          I could quite easily code this renderer so that you could specify
                          which font to fall back on but I just don't know if this is worth it
                          and if so, how to implement such an option (let 'guifont' take a list
                          of fonts and try the fonts from top to bottom?). I'm open for
                          ideas/suggestions.

                          > Hard to believe, but the display on Windows or GTK on Linux is better
                          > still... but at least it is usable now on Mac.

                          In which way are they better? What do you think I need to do to make
                          MacVim as good as or better than those?


                          Björn

                          --~--~---------~--~----~------------~-------~--~----~
                          You received this message from the "vim_mac" maillist.
                          For more information, visit http://www.vim.org/maillist.php
                          -~----------~----~----~----~------~----~------~--~---
                        • Ron Aaron
                          On the other OSes, combining characters draw correctly (or close to correctly) positioned. On the other OSes, the fonts which have Hebrew characters (like
                          Message 12 of 13 , Mar 30, 2009
                          • 0 Attachment
                            On the other OSes, combining characters draw correctly (or close to
                            correctly) positioned. On the other OSes, the fonts which have Hebrew
                            characters (like Miriam Fixed) can be used to display Hebrew, whereas
                            I still can't do that on the Mac.

                            Perhaps have a "guifont2" option, which is the fallback font? I don't
                            know if that would work, perhaps there is an option yet to be set in
                            the font creation or something? I do know that e.g. OpenOffice has
                            problems getting the positioning correct, while XeLaTex doesn't
                            (because it really uses the font information, so if a font is properly
                            coded it looks good).

                            On Mar 31, 12:14 am, björn <bjorn.winck...@...> wrote:
                            > 2009/3/29 Ron Aaron:
                            >
                            >
                            >
                            > > The file does render much better with that (series of patches).  And
                            > > now it looks like a "reasonable" default happens when I select a font
                            > > which previously didn't work (some font I don't know which, is
                            > > substituted - even though the default is set not to substitute).
                            > > Anyway, the display is much better.
                            >
                            > Good.
                            >
                            > I could quite easily code this renderer so that you could specify
                            > which font to fall back on but I just don't know if this is worth it
                            > and if so, how to implement such an option (let 'guifont' take a list
                            > of fonts and try the fonts from top to bottom?).  I'm open for
                            > ideas/suggestions.
                            >
                            > > Hard to believe, but the display on Windows or GTK on Linux is better
                            > > still... but at least it is usable now on Mac.
                            >
                            > In which way are they better?  What do you think I need to do to make
                            > MacVim as good as or better than those?
                            >
                            > Björn
                            --~--~---------~--~----~------------~-------~--~----~
                            You received this message from the "vim_mac" maillist.
                            For more information, visit http://www.vim.org/maillist.php
                            -~----------~----~----~----~------~----~------~--~---
                          • Tony Mechelynck
                            ... In principle, the guifont option can be a comma-separated list of several fonts, and the first (leftmost) of them which is found will be used. However,
                            Message 13 of 13 , Apr 1, 2009
                            • 0 Attachment
                              On Mar 31, 8:18 am, Ron Aaron <rambam...@...> wrote:
                              > On the other OSes, combining characters draw correctly (or close to
                              > correctly) positioned.  On the other OSes, the fonts which have Hebrew
                              > characters (like Miriam Fixed) can be used to display Hebrew, whereas
                              > I still can't do that on the Mac.
                              >
                              > Perhaps have a "guifont2" option, which is the fallback font?  I don't
                              > know if that would work, perhaps there is an option yet to be set in
                              > the font creation or something?  I do know that e.g. OpenOffice has
                              > problems getting the positioning correct, while XeLaTex doesn't
                              > (because it really uses the font information, so if a font is properly
                              > coded it looks good).

                              In principle, the 'guifont' option can be a comma-separated list of
                              several fonts, and the first (leftmost) of them which is found will be
                              used. However, IIUC Vim makes this choice globally, unlike HTML <font
                              face=...> and CSS { font-family: ... } where the norm is IIUC to make
                              the choice character by character.

                              Maybe a change of how that option works in that respect ought to
                              belong on Vim's todo list? Bram, what do you think?


                              Best regards,
                              Tony.
                              --
                              DENNIS: Oh, very nice. King, eh! I expect you've got a palace and
                              fine
                              clothes and courtiers and plenty of food. And how d'you get
                              that? By
                              exploiting the workers! By hanging on to outdated imperialist
                              dogma
                              which perpetuates the social and economic differences in our
                              society!
                              "Monty Python and the Holy Grail" PYTHON (MONTY)
                              PICTURES LTD

                              --~--~---------~--~----~------------~-------~--~----~
                              You received this message from the "vim_mac" maillist.
                              For more information, visit http://www.vim.org/maillist.php
                              -~----------~----~----~----~------~----~------~--~---
                            Your message has been successfully submitted and would be delivered to recipients shortly.