Loading ...
Sorry, an error occurred while loading the content.
 

Re: bug in experimental renderer

Expand Messages
  • dv1445@wayne.edu
    ... Hi Björn, Thanks for putting this on the TODO list. I just want to clear up a bit of confusion on my part. I thought that (1) the outputs of both :r
    Message 1 of 10 , Jul 7, 2012
      On 03/24/12 at 13:07 +0100, björn wrote:
      > On Tue, Mar 20, 2012 at 11:30 PM, wrote:
      > > I think There is a bug in MacVim regarding multibyte characters.  It
      > > only happens when the Experimental Renderer is turned ON.
      > >
      > > To reproduce:
      > >
      > > 1. make a new folder "bar".
      > > 2. In bar, do "touch föo", where there is an umlaut over the first 'o'.
      > > 3. Start MacVim with mvim -u NONE -U NONE.
      > > 4. In MacVim, do "cd bar".
      > > 5. Now do ":r !ls".
      > > 6. Put a couple of blank lines after the result of that.
      > > 7. Now do :let @a=glob("*"), followed by :put a
      > >
      > > Expected behavior: There should be two instances of "föo" in the buffer.
      > >
      > > Actual behavior: Depending on what your value of 'gfn' is, you might see
      > > that one of the "föo" lacks an umlaut.  For instance, if you use
      > > Consolas or Monaco, the one that was the result of "ls" (i.e., the top
      > > one) will lack an umlaut while the bottom one will look correct.  If you
      > > use Courier New or Menlo, then both words have the umlaut.
      > >
      > > Actual behavior part 2: Here's where it gets weird.  Navigate to each of
      > > the "o" that are supposed to have an umlaut (the first of each pair) and
      > > do "ga".  Vim's feedback for the top-foo (which came from ls) is:
      > >
      > > <o>  111,  Hex 6f,  Octal 157 < ̈> 776, Hex 0308, Octal 1410
      > >
      > > Vim's feedback for the bottom-foo (which came from glob()) is:
      > >
      > > <ö> 246, Hex 00f6, Octal 366
      >
      > This has to do with Unicode normalization [1]. The output from !ls is
      > not composed, whereas when you paste it in Vim it is composed. The
      > problem arises because you are piping characters from an external
      > command.
      >
      > I suppose we could try to normalize all input from external commands
      > but the problem is that it need not be in Unicode format so this may
      > cause new problems and/or be slower. I'll put a note on my todo list.
      >
      > Björn
      >
      > [1] http://en.wikipedia.org/wiki/Unicode_equivalence#Normalization

      Hi Björn,

      Thanks for putting this on the TODO list. I just want to clear up a bit of
      confusion on my part.

      I thought that (1) the outputs of both ":r !ls" and glob() should be
      indentical, and also that (2) they should both do it the way that !ls does it.
      The reason I think this is because that's how "true" console vim (from Bram's
      hg repo) does it, and we want to stay as much like true vim as is humanly
      possible.

      Is this right?

      If it is right, then what would need fixing is the output of glob(), not that
      of !ls. It seemed to me in your message that you were thinking about
      tinkering with !ls (and pipe-ins generally), but I wasn't sure.

      It also now seems that the non-showing of umlauts (depending on the font) is
      really a separate bug, probably in the experimental renderer; this separate
      bug was only *revealed* by MacVim's treating !ls and glob() differently, not
      caused by it.

      -gmn

      --
      You received this message from the "vim_mac" maillist.
      Do not top-post! Type your reply below the text you are replying to.
      For more information, visit http://www.vim.org/maillist.php
    • björn
      ... They are not identical, :r !ls is like typing ls in Terminal whereas glob() is implemented in the Vim source code. The other difference here is that
      Message 2 of 10 , Jul 10, 2012
        On Sat, Jul 7, 2012 at 6:55 PM, <dv1445@...> wrote:
        >
        > I thought that (1) the outputs of both ":r !ls" and glob() should be
        > indentical, and also that (2) they should both do it the way that !ls does it.
        > The reason I think this is because that's how "true" console vim (from Bram's
        > hg repo) does it, and we want to stay as much like true vim as is humanly
        > possible.
        >
        > Is this right?

        They are not identical, ":r !ls" is like typing "ls" in Terminal
        whereas glob() is implemented in the Vim source code.

        The other difference here is that console Vim handles output from
        commands like ":!ls" differently from GUI Vim (specifically, GUI Vim
        does not come with a terminal emulator).

        > If it is right, then what would need fixing is the output of glob(), not that
        > of !ls. It seemed to me in your message that you were thinking about
        > tinkering with !ls (and pipe-ins generally), but I wasn't sure.

        Are you saying that the problem only happens when you use glob() but
        not ":e !ls"?

        > It also now seems that the non-showing of umlauts (depending on the font) is
        > really a separate bug, probably in the experimental renderer; this separate
        > bug was only *revealed* by MacVim's treating !ls and glob() differently, not
        > caused by it.

        I think that there are problems related to rendering decomposed UTF.
        If I remember correctly HFS+ stores filenames in a decomposed format
        (decomposed form NFD) whereas for things to work they need to be in
        normalization form C (some parts of the code uses routines in
        Miscellaneous.h to do this conversion).

        If it is only glob() needs to be patched, then it may be possible to
        do the conversion to NFC to the relevant parts of the Vim source code
        (see f_glob() in eval.c). I'm not 100% sure if I am understanding you
        correctly though.

        Björn

        --
        You received this message from the "vim_mac" maillist.
        Do not top-post! Type your reply below the text you are replying to.
        For more information, visit http://www.vim.org/maillist.php
      • Ben Schmidt
        ... Yes, but that just *reveals* the bug; it isn t the bug. The subject line correctly identifies this as a rendering bug. It should be possible to edit both
        Message 3 of 10 , Jul 15, 2012
          On 8/07/12 2:55 AM, dv1445@... wrote:
          > On 03/24/12 at 13:07 +0100, björn wrote:
          >> On Tue, Mar 20, 2012 at 11:30 PM, wrote:
          >>> I think There is a bug in MacVim regarding multibyte characters. It
          >>> only happens when the Experimental Renderer is turned ON.
          >>>
          >>> To reproduce:
          >>>
          >>> 1. make a new folder "bar".
          >>> 2. In bar, do "touch föo", where there is an umlaut over the first 'o'.
          >>> 3. Start MacVim with mvim -u NONE -U NONE.
          >>> 4. In MacVim, do "cd bar".
          >>> 5. Now do ":r !ls".
          >>> 6. Put a couple of blank lines after the result of that.
          >>> 7. Now do :let @a=glob("*"), followed by :put a
          >>>
          >>> Expected behavior: There should be two instances of "föo" in the buffer.
          >>>
          >>> Actual behavior: Depending on what your value of 'gfn' is, you might see
          >>> that one of the "föo" lacks an umlaut. For instance, if you use
          >>> Consolas or Monaco, the one that was the result of "ls" (i.e., the top
          >>> one) will lack an umlaut while the bottom one will look correct. If you
          >>> use Courier New or Menlo, then both words have the umlaut.
          >>>
          >>> Actual behavior part 2: Here's where it gets weird. Navigate to each of
          >>> the "o" that are supposed to have an umlaut (the first of each pair) and
          >>> do "ga". Vim's feedback for the top-foo (which came from ls) is:
          >>>
          >>> <o> 111, Hex 6f, Octal 157< ̈> 776, Hex 0308, Octal 1410
          >>>
          >>> Vim's feedback for the bottom-foo (which came from glob()) is:
          >>>
          >>> <ö> 246, Hex 00f6, Octal 366
          >>
          >> This has to do with Unicode normalization [1].

          Yes, but that just *reveals* the bug; it isn't the bug.

          The subject line correctly identifies this as a rendering bug.

          It should be possible to edit both composed and non-composed characters
          in Vim. And indeed it is. For example, :set delcombine, and then issue x
          on top of the composed character and it will delete just the umlaut.

          However, you can't see this happen in MacVim because for some reason
          MacVim doesn't overlay the composing characters, but just doesn't
          display them at all. However, console Vim displays them fine in
          Terminal.app.

          >> I suppose we could try to normalize all input from external commands
          >> but the problem is that it need not be in Unicode format so this may
          >> cause new problems and/or be slower. I'll put a note on my todo list.

          This isn't a good idea at all. MacVim shouldn't manipulate its input. It
          should just display it properly. Doing it just for external commands
          wouldn't solve the problem when composing characters are found in a file
          being edited, either.

          We need to find and fix the bug in the renderer.

          (I actually thought the bug had been fixed, or at least made some
          progress, because Hebrew with vowels and accents seems to be displaying
          OK, and it's got composing characters everywhere. It's strange that it's
          working at least to some extent for Hebrew but not for simpler European
          characters.)

          Smiles,

          Ben.



          --
          You received this message from the "vim_mac" maillist.
          Do not top-post! Type your reply below the text you are replying to.
          For more information, visit http://www.vim.org/maillist.php
        • dv1445@wayne.edu
          ... Right, but it seems to me that if at all possible, their *outputs* should be identical, in the following sense: If there is a lone directory or file called
          Message 4 of 10 , Jul 16, 2012
            björn wrote on 07/10/12 at 23:18:38 +0200:
            > On Sat, Jul 7, 2012 at 6:55 PM, <dv1445@...> wrote:
            > > I thought that (1) the outputs of both ":r !ls" and glob() should be
            > > indentical, and also that (2) they should both do it the way that
            > > !ls does it. The reason I think this is because that's how "true"
            > > console vim (from Bram's hg repo) does it, and we want to stay as
            > > much like true vim as is humanly possible.
            > >
            > > Is this right?
            >
            > They are not identical, ":r !ls" is like typing "ls" in Terminal
            > whereas glob() is implemented in the Vim source code.

            Right, but it seems to me that if at all possible, their *outputs*
            should be identical, in the following sense: If there is a lone
            directory or file called 'föo' in the cwd, and we're looking at an empty
            vim buffer, then the two commands

            ":r !ls" and

            "let @a = glob('*') | put a"

            should put the same characters into the buffer. They do not.

            > The other difference here is that console Vim handles output from
            > commands like ":!ls" differently from GUI Vim (specifically, GUI Vim
            > does not come with a terminal emulator).
            >
            > > If it is right, then what would need fixing is the output of glob(),
            > > not that of !ls. It seemed to me in your message that you were
            > > thinking about tinkering with !ls (and pipe-ins generally), but I
            > > wasn't sure.
            >
            > Are you saying that the problem only happens when you use glob() but
            > not ":e !ls"?

            There are two "problems" here. The big problem is that 'ga' over the
            umlauted 'o' yields different values when the characters entered the
            buffer because of glob() than when the chars entered the buffer because
            of !ls. True console vim has their values as identical, and therefore
            if at all possible so should MacVim; also true console vim has them both
            behaving the way :!ls does, and therefore MacVim should too if at all
            possible. Summary: MacVim's glob() is not doing what it should, where
            'should' is understood as 'doing things the way true vim does'.

            The second problem was the titular bug in the experimental renderer;
            this bug (the dropping of the umlaut) only manifests itself when the
            :!ls procedure is used (and only with some fonts). That is, this second
            bug only appears when MacVim is putting chars in the buffer the right
            way. This experimental rendering bug has been masked by MacVim making
            glob() do something different from :!ls.

            > > It also now seems that the non-showing of umlauts (depending on the
            > > font) is really a separate bug, probably in the experimental
            > > renderer; this separate bug was only *revealed* by MacVim's treating
            > > !ls and glob() differently, not caused by it.
            >
            > I think that there are problems related to rendering decomposed UTF. If I
            > remember correctly HFS+ stores filenames in a decomposed format (decomposed
            > form NFD) whereas for things to work they need to be in normalization form C
            > (some parts of the code uses routines in Miscellaneous.h to do this
            > conversion).
            >
            > If it is only glob() needs to be patched, then it may be possible to do the
            > conversion to NFC to the relevant parts of the Vim source code (see f_glob()
            > in eval.c). I'm not 100% sure if I am understanding you correctly though.

            Well, I don't know any programming language other than some VimScript,
            so I can't help with the code. Maybe these two bugs are dinky and it's
            not worth the effort to fix them, for all I know. Maybe they aren't
            even fixable. I just thought it was interesting enough to report.

            It may be worth noting that the two methods of putting 'föo' in the
            buffer, i.e., !ls and glob(), affect certain VimScript functions like
            strlen() and strchars()---though I can't recall which did what. One or
            both of those Vim functions returned different values depending on
            whether its input came from glob() or from !ls.

            -gmn

            --
            You received this message from the "vim_mac" maillist.
            Do not top-post! Type your reply below the text you are replying to.
            For more information, visit http://www.vim.org/maillist.php
          • Ben Schmidt
            ... I wouldn t bother myself. ls might change one day. And other commandline utilities that output filenames might do differently. You re never going to be
            Message 5 of 10 , Jul 18, 2012
              >> They are not identical, ":r !ls" is like typing "ls" in Terminal
              >> whereas glob() is implemented in the Vim source code.
              >
              > Right, but it seems to me that if at all possible, their *outputs*
              > should be identical, in the following sense: If there is a lone
              > directory or file called 'föo' in the cwd, and we're looking at an empty
              > vim buffer, then the two commands
              >
              > ":r !ls" and
              >
              > "let @a = glob('*') | put a"
              >
              > should put the same characters into the buffer. They do not.

              I wouldn't bother myself. ls might change one day. And other commandline
              utilities that output filenames might do differently. You're never going
              to be able to please everybody or match every utility. There's nothing
              wrong with what happens at the moment. Might as well leave it as is. If
              you really want to compose/decompose all characters, find some unicode
              utility that can do it and pipe your data through it.

              > There are two "problems" here. The big problem is that 'ga' over the
              > umlauted 'o' yields different values when the characters entered the
              > buffer because of glob() than when the chars entered the buffer because
              > of !ls. True console vim has their values as identical, and therefore
              > if at all possible so should MacVim; also true console vim has them both
              > behaving the way :!ls does, and therefore MacVim should too if at all
              > possible. Summary: MacVim's glob() is not doing what it should, where
              > 'should' is understood as 'doing things the way true vim does'.

              This is indeed interesting. It is not a console Vim vs. MacVim issue,
              though. My Vim 7.3 compiled by Apple behaves differently to a 7.3.470 I
              compiled myself. I don't know whether it's something about the way it
              was compiled or the version difference that's altered things. Here is my
              full :version output:

              This has glob() and r !ls the same:

              VIM - Vi IMproved 7.3 (2010 Aug 15, compiled Jul 31 2011 19:27:29)
              Compiled by root@...
              Normal version without GUI. Features included (+) or not (-):
              -arabic +autocmd -balloon_eval -browse +builtin_terms +byte_offset +cindent
              -clientserver -clipboard +cmdline_compl +cmdline_hist +cmdline_info +comments
              -conceal +cryptv +cscope +cursorbind +cursorshape +dialog_con +diff +digraphs
              -dnd -ebcdic -emacs_tags +eval +ex_extra +extra_search -farsi +file_in_path
              +find_in_path +float +folding -footer +fork() -gettext -hangul_input +iconv
              +insert_expand +jumplist -keymap -langmap +libcall +linebreak +lispindent
              +listcmds +localmap -lua +menu +mksession +modify_fname +mouse -mouseshape
              -mouse_dec -mouse_gpm -mouse_jsbterm -mouse_netterm -mouse_sysmouse
              +mouse_xterm +multi_byte +multi_lang -mzscheme +netbeans_intg -osfiletype
              +path_extra -perl +persistent_undo +postscript +printer -profile -python
              -python3 +quickfix +reltime -rightleft -ruby +scrollbind +signs +smartindent
              -sniff +startuptime +statusline -sun_workshop +syntax +tag_binary
              +tag_old_static -tag_any_white -tcl +terminfo +termresponse +textobjects +title
              -toolbar +user_commands +vertsplit +virtualedit +visual +visualextra +viminfo
              +vreplace +wildignore +wildmenu +windows +writebackup -X11 -xfontset -xim -xsmp
              -xterm_clipboard -xterm_save
              system vimrc file: "$VIM/vimrc"
              user vimrc file: "$HOME/.vimrc"
              user exrc file: "$HOME/.exrc"
              fall-back for $VIM: "/usr/share/vim"
              Compilation: gcc -c -I. -D_FORTIFY_SOURCE=0 -Iproto -DHAVE_CONFIG_H -arch i386
              -arch x86_64 -g -Os -pipe
              Linking: gcc -arch i386 -arch x86_64 -o vim -lncurses

              These have glob() and r !ls different:

              VIM - Vi IMproved 7.3 (2010 Aug 15, compiled Mar 9 2012 00:45:32)
              MacOS X (unix) version
              Included patches: 1-470
              Modified by Ben Schmidt et al.
              Compiled by ben@...
              Huge version without GUI. Features included (+) or not (-):
              +arabic +autocmd -balloon_eval -browse ++builtin_terms +byte_offset +cindent
              +clientserver +clipboard +cmdline_compl +cmdline_hist +cmdline_info +comments
              +conceal +cryptv +cscope +cursorbind +cursorshape +dialog_con +diff +digraphs
              -dnd -ebcdic +emacs_tags +eval +ex_extra +extra_search +farsi +file_in_path
              +find_in_path +float +folding -footer +fork() -gettext -hangul_input -iconv
              +insert_expand +jumplist +keymap +langmap +libcall +linebreak +lispindent
              +listcmds +localmap -lua +menu +mksession +modify_fname +mouse -mouseshape
              +mouse_dec -mouse_gpm -mouse_jsbterm +mouse_netterm -mouse_sysmouse
              +mouse_xterm +mouse_urxvt +multi_byte +multi_lang -mzscheme +netbeans_intg
              +path_extra +perl +persistent_undo +postscript +printer +profile +python
              -python3 +quickfix +reltime +rightleft +ruby +scrollbind +signs +smartindent
              -sniff +startuptime +statusline -sun_workshop +syntax +tag_binary
              +tag_old_static -tag_any_white +tcl +terminfo +termresponse +textobjects +title
              -toolbar +user_commands +vartabs +vertsplit +virtualedit +visual +visualextra
              +viminfo +vreplace +wildignore +wildmenu +windows +writebackup +X11 +xfontset
              -xim +xsmp_interact +xterm_clipboard -xterm_save
              system vimrc file: "$VIM/vimrc"
              user vimrc file: "$HOME/.vimrc"
              user exrc file: "$HOME/.exrc"
              fall-back for $VIM: "/usr/local/share/vim"
              Compilation: gcc -c -I. -Iproto -DHAVE_CONFIG_H -DMACOS_X_UNIX -no-cpp-precomp
              -g -O2 -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1
              -I/System/Library/Frameworks/Tcl.framework/Headers -D_REENTRANT=1
              -D_THREAD_SAFE=1 -D_DARWIN_C_SOURCE=1
              Linking: gcc -L. -g -L/opt/local/lib -L/usr/local/lib -o vim -lXt
              -lX11 -lSM -lICE -lncurses -framework Cocoa -L/opt/local/lib
              -fstack-protector -L/opt/local/lib/perl5/5.12.3/darwin-multi-2level/CORE -lperl
              -lm -lutil -lc -framework Python -F/System/Library/Frameworks -framework Tcl
              -framework CoreFoundation -lruby

              VIM - Vi IMproved 7.3 (2010 Aug 15, compiled Mar 11 2012 01:07:11)
              MacOS X (unix) version
              Included patches: 1-456
              Modified by Ben Schmidt et al.
              Compiled by ben@...
              Huge version with MacVim GUI. Features included (+) or not (-):
              +arabic +autocmd +balloon_eval +browse ++builtin_terms +byte_offset +cindent
              +clientserver +clipboard +cmdline_compl +cmdline_hist +cmdline_info +comments
              +conceal +cryptv +cscope +cursorbind +cursorshape +dialog_con_gui +diff
              +digraphs +dnd -ebcdic +emacs_tags +eval +ex_extra +extra_search +farsi
              +file_in_path +find_in_path +float +folding -footer +fork() +fullscreen -gettext
              -hangul_input -iconv +insert_expand +jumplist +keymap +langmap +libcall
              +linebreak +lispindent +listcmds +localmap -lua +menu +mksession +modify_fname
              +mouse +mouseshape +mouse_dec -mouse_gpm -mouse_jsbterm +mouse_netterm
              -mouse_sysmouse +mouse_xterm +mouse_urxvt +multi_byte +multi_lang -mzscheme
              +netbeans_intg +odbeditor +path_extra +perl +persistent_undo +postscript
              +printer +profile +python -python3 +quickfix +reltime +rightleft +ruby
              +scrollbind +signs +smartindent -sniff +startuptime +statusline -sun_workshop
              +syntax +tag_binary +tag_old_static -tag_any_white +tcl +terminfo +termresponse
              +textobjects +title +toolbar +transparency +user_commands +vartabs +vertsplit
              +virtualedit +visual +visualextra +viminfo +vreplace +wildignore +wildmenu
              +windows +writebackup -X11 -xfontset +xim -xsmp -xterm_clipboard -xterm_save
              system vimrc file: "$VIM/vimrc"
              user vimrc file: "$HOME/.vimrc"
              user exrc file: "$HOME/.exrc"
              system gvimrc file: "$VIM/gvimrc"
              user gvimrc file: "$HOME/.gvimrc"
              system menu file: "$VIMRUNTIME/menu.vim"
              fall-back for $VIM: "/Applications/MacVim.app/Contents/Resources/vim"
              Compilation: gcc -c -I. -Iproto -DHAVE_CONFIG_H -DFEAT_GUI_MACVIM -Wall
              -Wno-unknown-pragmas -pipe -DMACOS_X_UNIX -no-cpp-precomp -g -O2
              -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1
              -I/System/Library/Frameworks/Tcl.framework/Headers -D_REENTRANT=1
              -D_THREAD_SAFE=1 -D_DARWIN_C_SOURCE=1
              Linking: gcc -L. -L. -L/opt/local/lib -L/usr/local/lib -o Vim
              -framework Cocoa -framework Carbon -lncurses -framework Cocoa
              -L/opt/local/lib -fstack-protector
              -L/opt/local/lib/perl5/5.12.3/darwin-multi-2level/CORE -lperl -lm -lutil -lc
              -framework Python -F/System/Library/Frameworks -framework Tcl -framework
              CoreFoundation -framework Ruby

              What do you get for :version (use :redir to capture it)?

              > The second problem was the titular bug in the experimental renderer;
              > this bug (the dropping of the umlaut) only manifests itself when the
              > :!ls procedure is used (and only with some fonts). That is, this second
              > bug only appears when MacVim is putting chars in the buffer the right
              > way. This experimental rendering bug has been masked by MacVim making
              > glob() do something different from :!ls.

              This seems like a much bigger problem to me. The other issue may be
              inconvenient, but there's nothing actually wrong. In both cases, the
              filename is validly represented. However, this issue affects any time
              Unicode text with composing characters is edited, and the display is
              incorrect as it doesn't represent the contents of the buffer accurately.

              > It may be worth noting that the two methods of putting 'föo' in the
              > buffer, i.e., !ls and glob(), affect certain VimScript functions like
              > strlen() and strchars()---though I can't recall which did what. One or
              > both of those Vim functions returned different values depending on
              > whether its input came from glob() or from !ls.

              That is expected.

              Cheers,

              Ben.



              --
              You received this message from the "vim_mac" maillist.
              Do not top-post! Type your reply below the text you are replying to.
              For more information, visit http://www.vim.org/maillist.php
            • Ben Schmidt
              ... I have tracked down what this is. MacVim is compiled with Darwin support, but Apple s Vim is not (go figure). That means that MacVim calls
              Message 6 of 10 , Jul 19, 2012
                >> There are two "problems" here. The big problem is that 'ga' over the
                >> umlauted 'o' yields different values when the characters entered the
                >> buffer because of glob() than when the chars entered the buffer
                >> because of !ls. True console vim has their values as identical, and
                >> therefore if at all possible so should MacVim; also true console vim
                >> has them both behaving the way :!ls does, and therefore MacVim should
                >> too if at all possible. Summary: MacVim's glob() is not doing what it
                >> should, where 'should' is understood as 'doing things the way true
                >> vim does'.
                >
                > This is indeed interesting. It is not a console Vim vs. MacVim issue,
                > though. My Vim 7.3 compiled by Apple behaves differently to a 7.3.470
                > I compiled myself. I don't know whether it's something about the way
                > it was compiled or the version difference that's altered things.

                I have tracked down what this is.

                MacVim is compiled with Darwin support, but Apple's Vim is not (go
                figure). That means that MacVim calls mac_precompose_path() while
                evaluating glob(), but Apple's Vim doesn't. This function "converts a
                decomposed HFS+ UTF-8 path to precomposed UTF-8". This explains why
                there is a difference between glob() and :r !ls in MacVim but not in
                Apple's Vim: MacVim does some extra processing for glob().

                MacVim won't compile with --disable-darwin, though that may be more
                because of the logic in configure than because it would actually cause
                problems with the codebase. So we are stuck with this for now.

                However, it does raise some questions.

                - Why was it ever thought necessary to do that conversion?
                - Is it still necessary?
                - Why doesn't Apple's Vim have support for their operating system
                compiled in?

                Ben.



                --
                You received this message from the "vim_mac" maillist.
                Do not top-post! Type your reply below the text you are replying to.
                For more information, visit http://www.vim.org/maillist.php
              • björn
                ... Thanks for tracking this down. Precomposing filenames is the right thing to do in this case. In order for Vim to function correctly filenames have to be
                Message 7 of 10 , Jul 19, 2012
                  On Thu, Jul 19, 2012 at 2:31 PM, Ben Schmidt wrote:
                  >>> There are two "problems" here. The big problem is that 'ga' over the
                  >>> umlauted 'o' yields different values when the characters entered the
                  >>> buffer because of glob() than when the chars entered the buffer
                  >>> because of !ls. True console vim has their values as identical, and
                  >>> therefore if at all possible so should MacVim; also true console vim
                  >>> has them both behaving the way :!ls does, and therefore MacVim should
                  >>> too if at all possible. Summary: MacVim's glob() is not doing what it
                  >>> should, where 'should' is understood as 'doing things the way true
                  >>> vim does'.
                  >>
                  >>
                  >> This is indeed interesting. It is not a console Vim vs. MacVim issue,
                  >> though. My Vim 7.3 compiled by Apple behaves differently to a 7.3.470
                  >> I compiled myself. I don't know whether it's something about the way
                  >> it was compiled or the version difference that's altered things.
                  >
                  >
                  > I have tracked down what this is.
                  >
                  > MacVim is compiled with Darwin support, but Apple's Vim is not (go
                  > figure). That means that MacVim calls mac_precompose_path() while
                  > evaluating glob(), but Apple's Vim doesn't. This function "converts a
                  > decomposed HFS+ UTF-8 path to precomposed UTF-8". This explains why
                  > there is a difference between glob() and :r !ls in MacVim but not in
                  > Apple's Vim: MacVim does some extra processing for glob().

                  Thanks for tracking this down.

                  Precomposing filenames is the "right thing" to do in this case. In
                  order for Vim to function correctly filenames have to be precomposed.
                  I don't remember the exact details, but I think e.g. things like
                  comparing filenames fails if they are not precomposed. However, and I
                  admittedly conflated these two issues in a previous post, this should
                  not affect rendering of text.

                  > MacVim won't compile with --disable-darwin, though that may be more
                  > because of the logic in configure than because it would actually cause
                  > problems with the codebase. So we are stuck with this for now.
                  >
                  > However, it does raise some questions.
                  >
                  > - Why was it ever thought necessary to do that conversion?
                  > - Is it still necessary?
                  > - Why doesn't Apple's Vim have support for their operating system
                  > compiled in?

                  I have answered the first two above, the third I cannot answer.

                  So, to conclude and hopefully put this thread to rest:

                  - The experimental renderer has problems rendering decomposed UTF

                  - Vim expects filenames to be precomposed UTF: glob() does the right
                  thing by precomposing, ":!ls" does not precompose since filenames are
                  stored in decomposed form in the filesystem (and we should not /
                  cannot change this). If you are passing filenames to Vim (e.g. to
                  open a file) then they need to be precomposed. The fact that console
                  Vim does not precompose UTF is actually a bug (in part due to Apple
                  not compiling Vim properly).


                  Björn

                  --
                  You received this message from the "vim_mac" maillist.
                  Do not top-post! Type your reply below the text you are replying to.
                  For more information, visit http://www.vim.org/maillist.php
                • dv1445@wayne.edu
                  ... Nice work! ... Ah, my true console vim isn t the factory one from Apple, but compiled myself with a flag of ... sigh ... --disable-darwin! As to why this
                  Message 8 of 10 , Jul 19, 2012
                    Ben Schmidt, on 07/19/12 at 22:31:49 +1000, wrote:
                    > >This is indeed interesting. It is not a console Vim vs. MacVim issue,
                    > >though. My Vim 7.3 compiled by Apple behaves differently to a 7.3.470
                    > >I compiled myself. I don't know whether it's something about the way
                    > >it was compiled or the version difference that's altered things.
                    >
                    > I have tracked down what this is.

                    Nice work!

                    > MacVim is compiled with Darwin support, but Apple's Vim is not (go
                    > figure). That means that MacVim calls mac_precompose_path() while
                    > evaluating glob(), but Apple's Vim doesn't. This function "converts a
                    > decomposed HFS+ UTF-8 path to precomposed UTF-8". This explains why
                    > there is a difference between glob() and :r !ls in MacVim but not in
                    > Apple's Vim: MacVim does some extra processing for glob().

                    Ah, my "true console" vim isn't the factory one from Apple, but compiled
                    myself with a flag of ... sigh ... --disable-darwin!

                    As to why this flag is in my bash alias for running ./configure for
                    console vim, I can no longer recall; I know that I *had* a reason, and I
                    *think* it had something to do with not wanting certain hard-wired
                    mapping-related stuff that --enable-darwin stuck in there.

                    Since --enable-darwin is really the way to go, for precomposing reasons
                    as Björn pointed out, I will try to find out if my reason was actually
                    any good. If it was, perhaps we can fiddle with what --enable-darwin
                    does.

                    Prediction: my reason was not any good. :)

                    -gmn

                    --
                    You received this message from the "vim_mac" maillist.
                    Do not top-post! Type your reply below the text you are replying to.
                    For more information, visit http://www.vim.org/maillist.php
                  Your message has been successfully submitted and would be delivered to recipients shortly.