Loading ...
Sorry, an error occurred while loading the content.

Re: bug in experimental renderer

Expand Messages
  • Ben Schmidt
    ... I wouldn t bother myself. ls might change one day. And other commandline utilities that output filenames might do differently. You re never going to be
    Message 1 of 10 , Jul 18 3:33 PM
    • 0 Attachment
      >> They are not identical, ":r !ls" is like typing "ls" in Terminal
      >> whereas glob() is implemented in the Vim source code.
      >
      > Right, but it seems to me that if at all possible, their *outputs*
      > should be identical, in the following sense: If there is a lone
      > directory or file called 'föo' in the cwd, and we're looking at an empty
      > vim buffer, then the two commands
      >
      > ":r !ls" and
      >
      > "let @a = glob('*') | put a"
      >
      > should put the same characters into the buffer. They do not.

      I wouldn't bother myself. ls might change one day. And other commandline
      utilities that output filenames might do differently. You're never going
      to be able to please everybody or match every utility. There's nothing
      wrong with what happens at the moment. Might as well leave it as is. If
      you really want to compose/decompose all characters, find some unicode
      utility that can do it and pipe your data through it.

      > There are two "problems" here. The big problem is that 'ga' over the
      > umlauted 'o' yields different values when the characters entered the
      > buffer because of glob() than when the chars entered the buffer because
      > of !ls. True console vim has their values as identical, and therefore
      > if at all possible so should MacVim; also true console vim has them both
      > behaving the way :!ls does, and therefore MacVim should too if at all
      > possible. Summary: MacVim's glob() is not doing what it should, where
      > 'should' is understood as 'doing things the way true vim does'.

      This is indeed interesting. It is not a console Vim vs. MacVim issue,
      though. My Vim 7.3 compiled by Apple behaves differently to a 7.3.470 I
      compiled myself. I don't know whether it's something about the way it
      was compiled or the version difference that's altered things. Here is my
      full :version output:

      This has glob() and r !ls the same:

      VIM - Vi IMproved 7.3 (2010 Aug 15, compiled Jul 31 2011 19:27:29)
      Compiled by root@...
      Normal version without GUI. Features included (+) or not (-):
      -arabic +autocmd -balloon_eval -browse +builtin_terms +byte_offset +cindent
      -clientserver -clipboard +cmdline_compl +cmdline_hist +cmdline_info +comments
      -conceal +cryptv +cscope +cursorbind +cursorshape +dialog_con +diff +digraphs
      -dnd -ebcdic -emacs_tags +eval +ex_extra +extra_search -farsi +file_in_path
      +find_in_path +float +folding -footer +fork() -gettext -hangul_input +iconv
      +insert_expand +jumplist -keymap -langmap +libcall +linebreak +lispindent
      +listcmds +localmap -lua +menu +mksession +modify_fname +mouse -mouseshape
      -mouse_dec -mouse_gpm -mouse_jsbterm -mouse_netterm -mouse_sysmouse
      +mouse_xterm +multi_byte +multi_lang -mzscheme +netbeans_intg -osfiletype
      +path_extra -perl +persistent_undo +postscript +printer -profile -python
      -python3 +quickfix +reltime -rightleft -ruby +scrollbind +signs +smartindent
      -sniff +startuptime +statusline -sun_workshop +syntax +tag_binary
      +tag_old_static -tag_any_white -tcl +terminfo +termresponse +textobjects +title
      -toolbar +user_commands +vertsplit +virtualedit +visual +visualextra +viminfo
      +vreplace +wildignore +wildmenu +windows +writebackup -X11 -xfontset -xim -xsmp
      -xterm_clipboard -xterm_save
      system vimrc file: "$VIM/vimrc"
      user vimrc file: "$HOME/.vimrc"
      user exrc file: "$HOME/.exrc"
      fall-back for $VIM: "/usr/share/vim"
      Compilation: gcc -c -I. -D_FORTIFY_SOURCE=0 -Iproto -DHAVE_CONFIG_H -arch i386
      -arch x86_64 -g -Os -pipe
      Linking: gcc -arch i386 -arch x86_64 -o vim -lncurses

      These have glob() and r !ls different:

      VIM - Vi IMproved 7.3 (2010 Aug 15, compiled Mar 9 2012 00:45:32)
      MacOS X (unix) version
      Included patches: 1-470
      Modified by Ben Schmidt et al.
      Compiled by ben@...
      Huge version without GUI. Features included (+) or not (-):
      +arabic +autocmd -balloon_eval -browse ++builtin_terms +byte_offset +cindent
      +clientserver +clipboard +cmdline_compl +cmdline_hist +cmdline_info +comments
      +conceal +cryptv +cscope +cursorbind +cursorshape +dialog_con +diff +digraphs
      -dnd -ebcdic +emacs_tags +eval +ex_extra +extra_search +farsi +file_in_path
      +find_in_path +float +folding -footer +fork() -gettext -hangul_input -iconv
      +insert_expand +jumplist +keymap +langmap +libcall +linebreak +lispindent
      +listcmds +localmap -lua +menu +mksession +modify_fname +mouse -mouseshape
      +mouse_dec -mouse_gpm -mouse_jsbterm +mouse_netterm -mouse_sysmouse
      +mouse_xterm +mouse_urxvt +multi_byte +multi_lang -mzscheme +netbeans_intg
      +path_extra +perl +persistent_undo +postscript +printer +profile +python
      -python3 +quickfix +reltime +rightleft +ruby +scrollbind +signs +smartindent
      -sniff +startuptime +statusline -sun_workshop +syntax +tag_binary
      +tag_old_static -tag_any_white +tcl +terminfo +termresponse +textobjects +title
      -toolbar +user_commands +vartabs +vertsplit +virtualedit +visual +visualextra
      +viminfo +vreplace +wildignore +wildmenu +windows +writebackup +X11 +xfontset
      -xim +xsmp_interact +xterm_clipboard -xterm_save
      system vimrc file: "$VIM/vimrc"
      user vimrc file: "$HOME/.vimrc"
      user exrc file: "$HOME/.exrc"
      fall-back for $VIM: "/usr/local/share/vim"
      Compilation: gcc -c -I. -Iproto -DHAVE_CONFIG_H -DMACOS_X_UNIX -no-cpp-precomp
      -g -O2 -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1
      -I/System/Library/Frameworks/Tcl.framework/Headers -D_REENTRANT=1
      -D_THREAD_SAFE=1 -D_DARWIN_C_SOURCE=1
      Linking: gcc -L. -g -L/opt/local/lib -L/usr/local/lib -o vim -lXt
      -lX11 -lSM -lICE -lncurses -framework Cocoa -L/opt/local/lib
      -fstack-protector -L/opt/local/lib/perl5/5.12.3/darwin-multi-2level/CORE -lperl
      -lm -lutil -lc -framework Python -F/System/Library/Frameworks -framework Tcl
      -framework CoreFoundation -lruby

      VIM - Vi IMproved 7.3 (2010 Aug 15, compiled Mar 11 2012 01:07:11)
      MacOS X (unix) version
      Included patches: 1-456
      Modified by Ben Schmidt et al.
      Compiled by ben@...
      Huge version with MacVim GUI. Features included (+) or not (-):
      +arabic +autocmd +balloon_eval +browse ++builtin_terms +byte_offset +cindent
      +clientserver +clipboard +cmdline_compl +cmdline_hist +cmdline_info +comments
      +conceal +cryptv +cscope +cursorbind +cursorshape +dialog_con_gui +diff
      +digraphs +dnd -ebcdic +emacs_tags +eval +ex_extra +extra_search +farsi
      +file_in_path +find_in_path +float +folding -footer +fork() +fullscreen -gettext
      -hangul_input -iconv +insert_expand +jumplist +keymap +langmap +libcall
      +linebreak +lispindent +listcmds +localmap -lua +menu +mksession +modify_fname
      +mouse +mouseshape +mouse_dec -mouse_gpm -mouse_jsbterm +mouse_netterm
      -mouse_sysmouse +mouse_xterm +mouse_urxvt +multi_byte +multi_lang -mzscheme
      +netbeans_intg +odbeditor +path_extra +perl +persistent_undo +postscript
      +printer +profile +python -python3 +quickfix +reltime +rightleft +ruby
      +scrollbind +signs +smartindent -sniff +startuptime +statusline -sun_workshop
      +syntax +tag_binary +tag_old_static -tag_any_white +tcl +terminfo +termresponse
      +textobjects +title +toolbar +transparency +user_commands +vartabs +vertsplit
      +virtualedit +visual +visualextra +viminfo +vreplace +wildignore +wildmenu
      +windows +writebackup -X11 -xfontset +xim -xsmp -xterm_clipboard -xterm_save
      system vimrc file: "$VIM/vimrc"
      user vimrc file: "$HOME/.vimrc"
      user exrc file: "$HOME/.exrc"
      system gvimrc file: "$VIM/gvimrc"
      user gvimrc file: "$HOME/.gvimrc"
      system menu file: "$VIMRUNTIME/menu.vim"
      fall-back for $VIM: "/Applications/MacVim.app/Contents/Resources/vim"
      Compilation: gcc -c -I. -Iproto -DHAVE_CONFIG_H -DFEAT_GUI_MACVIM -Wall
      -Wno-unknown-pragmas -pipe -DMACOS_X_UNIX -no-cpp-precomp -g -O2
      -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1
      -I/System/Library/Frameworks/Tcl.framework/Headers -D_REENTRANT=1
      -D_THREAD_SAFE=1 -D_DARWIN_C_SOURCE=1
      Linking: gcc -L. -L. -L/opt/local/lib -L/usr/local/lib -o Vim
      -framework Cocoa -framework Carbon -lncurses -framework Cocoa
      -L/opt/local/lib -fstack-protector
      -L/opt/local/lib/perl5/5.12.3/darwin-multi-2level/CORE -lperl -lm -lutil -lc
      -framework Python -F/System/Library/Frameworks -framework Tcl -framework
      CoreFoundation -framework Ruby

      What do you get for :version (use :redir to capture it)?

      > The second problem was the titular bug in the experimental renderer;
      > this bug (the dropping of the umlaut) only manifests itself when the
      > :!ls procedure is used (and only with some fonts). That is, this second
      > bug only appears when MacVim is putting chars in the buffer the right
      > way. This experimental rendering bug has been masked by MacVim making
      > glob() do something different from :!ls.

      This seems like a much bigger problem to me. The other issue may be
      inconvenient, but there's nothing actually wrong. In both cases, the
      filename is validly represented. However, this issue affects any time
      Unicode text with composing characters is edited, and the display is
      incorrect as it doesn't represent the contents of the buffer accurately.

      > It may be worth noting that the two methods of putting 'föo' in the
      > buffer, i.e., !ls and glob(), affect certain VimScript functions like
      > strlen() and strchars()---though I can't recall which did what. One or
      > both of those Vim functions returned different values depending on
      > whether its input came from glob() or from !ls.

      That is expected.

      Cheers,

      Ben.



      --
      You received this message from the "vim_mac" maillist.
      Do not top-post! Type your reply below the text you are replying to.
      For more information, visit http://www.vim.org/maillist.php
    • Ben Schmidt
      ... I have tracked down what this is. MacVim is compiled with Darwin support, but Apple s Vim is not (go figure). That means that MacVim calls
      Message 2 of 10 , Jul 19 5:31 AM
      • 0 Attachment
        >> There are two "problems" here. The big problem is that 'ga' over the
        >> umlauted 'o' yields different values when the characters entered the
        >> buffer because of glob() than when the chars entered the buffer
        >> because of !ls. True console vim has their values as identical, and
        >> therefore if at all possible so should MacVim; also true console vim
        >> has them both behaving the way :!ls does, and therefore MacVim should
        >> too if at all possible. Summary: MacVim's glob() is not doing what it
        >> should, where 'should' is understood as 'doing things the way true
        >> vim does'.
        >
        > This is indeed interesting. It is not a console Vim vs. MacVim issue,
        > though. My Vim 7.3 compiled by Apple behaves differently to a 7.3.470
        > I compiled myself. I don't know whether it's something about the way
        > it was compiled or the version difference that's altered things.

        I have tracked down what this is.

        MacVim is compiled with Darwin support, but Apple's Vim is not (go
        figure). That means that MacVim calls mac_precompose_path() while
        evaluating glob(), but Apple's Vim doesn't. This function "converts a
        decomposed HFS+ UTF-8 path to precomposed UTF-8". This explains why
        there is a difference between glob() and :r !ls in MacVim but not in
        Apple's Vim: MacVim does some extra processing for glob().

        MacVim won't compile with --disable-darwin, though that may be more
        because of the logic in configure than because it would actually cause
        problems with the codebase. So we are stuck with this for now.

        However, it does raise some questions.

        - Why was it ever thought necessary to do that conversion?
        - Is it still necessary?
        - Why doesn't Apple's Vim have support for their operating system
        compiled in?

        Ben.



        --
        You received this message from the "vim_mac" maillist.
        Do not top-post! Type your reply below the text you are replying to.
        For more information, visit http://www.vim.org/maillist.php
      • björn
        ... Thanks for tracking this down. Precomposing filenames is the right thing to do in this case. In order for Vim to function correctly filenames have to be
        Message 3 of 10 , Jul 19 6:37 AM
        • 0 Attachment
          On Thu, Jul 19, 2012 at 2:31 PM, Ben Schmidt wrote:
          >>> There are two "problems" here. The big problem is that 'ga' over the
          >>> umlauted 'o' yields different values when the characters entered the
          >>> buffer because of glob() than when the chars entered the buffer
          >>> because of !ls. True console vim has their values as identical, and
          >>> therefore if at all possible so should MacVim; also true console vim
          >>> has them both behaving the way :!ls does, and therefore MacVim should
          >>> too if at all possible. Summary: MacVim's glob() is not doing what it
          >>> should, where 'should' is understood as 'doing things the way true
          >>> vim does'.
          >>
          >>
          >> This is indeed interesting. It is not a console Vim vs. MacVim issue,
          >> though. My Vim 7.3 compiled by Apple behaves differently to a 7.3.470
          >> I compiled myself. I don't know whether it's something about the way
          >> it was compiled or the version difference that's altered things.
          >
          >
          > I have tracked down what this is.
          >
          > MacVim is compiled with Darwin support, but Apple's Vim is not (go
          > figure). That means that MacVim calls mac_precompose_path() while
          > evaluating glob(), but Apple's Vim doesn't. This function "converts a
          > decomposed HFS+ UTF-8 path to precomposed UTF-8". This explains why
          > there is a difference between glob() and :r !ls in MacVim but not in
          > Apple's Vim: MacVim does some extra processing for glob().

          Thanks for tracking this down.

          Precomposing filenames is the "right thing" to do in this case. In
          order for Vim to function correctly filenames have to be precomposed.
          I don't remember the exact details, but I think e.g. things like
          comparing filenames fails if they are not precomposed. However, and I
          admittedly conflated these two issues in a previous post, this should
          not affect rendering of text.

          > MacVim won't compile with --disable-darwin, though that may be more
          > because of the logic in configure than because it would actually cause
          > problems with the codebase. So we are stuck with this for now.
          >
          > However, it does raise some questions.
          >
          > - Why was it ever thought necessary to do that conversion?
          > - Is it still necessary?
          > - Why doesn't Apple's Vim have support for their operating system
          > compiled in?

          I have answered the first two above, the third I cannot answer.

          So, to conclude and hopefully put this thread to rest:

          - The experimental renderer has problems rendering decomposed UTF

          - Vim expects filenames to be precomposed UTF: glob() does the right
          thing by precomposing, ":!ls" does not precompose since filenames are
          stored in decomposed form in the filesystem (and we should not /
          cannot change this). If you are passing filenames to Vim (e.g. to
          open a file) then they need to be precomposed. The fact that console
          Vim does not precompose UTF is actually a bug (in part due to Apple
          not compiling Vim properly).


          Björn

          --
          You received this message from the "vim_mac" maillist.
          Do not top-post! Type your reply below the text you are replying to.
          For more information, visit http://www.vim.org/maillist.php
        • dv1445@wayne.edu
          ... Nice work! ... Ah, my true console vim isn t the factory one from Apple, but compiled myself with a flag of ... sigh ... --disable-darwin! As to why this
          Message 4 of 10 , Jul 19 9:10 AM
          • 0 Attachment
            Ben Schmidt, on 07/19/12 at 22:31:49 +1000, wrote:
            > >This is indeed interesting. It is not a console Vim vs. MacVim issue,
            > >though. My Vim 7.3 compiled by Apple behaves differently to a 7.3.470
            > >I compiled myself. I don't know whether it's something about the way
            > >it was compiled or the version difference that's altered things.
            >
            > I have tracked down what this is.

            Nice work!

            > MacVim is compiled with Darwin support, but Apple's Vim is not (go
            > figure). That means that MacVim calls mac_precompose_path() while
            > evaluating glob(), but Apple's Vim doesn't. This function "converts a
            > decomposed HFS+ UTF-8 path to precomposed UTF-8". This explains why
            > there is a difference between glob() and :r !ls in MacVim but not in
            > Apple's Vim: MacVim does some extra processing for glob().

            Ah, my "true console" vim isn't the factory one from Apple, but compiled
            myself with a flag of ... sigh ... --disable-darwin!

            As to why this flag is in my bash alias for running ./configure for
            console vim, I can no longer recall; I know that I *had* a reason, and I
            *think* it had something to do with not wanting certain hard-wired
            mapping-related stuff that --enable-darwin stuck in there.

            Since --enable-darwin is really the way to go, for precomposing reasons
            as Björn pointed out, I will try to find out if my reason was actually
            any good. If it was, perhaps we can fiddle with what --enable-darwin
            does.

            Prediction: my reason was not any good. :)

            -gmn

            --
            You received this message from the "vim_mac" maillist.
            Do not top-post! Type your reply below the text you are replying to.
            For more information, visit http://www.vim.org/maillist.php
          Your message has been successfully submitted and would be delivered to recipients shortly.